2024-02-05T07:02:07.409 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-05T07:02:07.409 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546729 --name gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi --owner scheduled_gabrioux@teuthology --verbose --description orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} -- /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546729/orig.config.yaml 2024-02-05T07:02:07.492 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-05T07:02:07.586 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546729 branch: wip-guits-testing-3-2024-02-04-2254 description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} email: gabrioux@ibm.com first_in_suite: false job_id: '7546729' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-testing-3-2024-02-04-2254 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-testing-3-2024-02-04-2254 sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 7317 sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-testing-3-2024-02-04-2254 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_164d3be4a527be775f4d3c343ea591122f1b8921/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCz1R3OHxhVp2mKR6RqPhAvbhtI32CP2dgQ5Xhmx2AFPKED9mGGBpiR+tejO5ppiDdHVd52nqyNd9wPwDeU958k= smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN7CVL30sw/Qt233Zvb9eaojHn2grchuZh5xuuFs7Du40CDTCrKcdHjnGtVfbawCn+dmelh5N7JuCk51oGqODcI= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-05_05:57:01 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-05T07:02:07.587 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_164d3be4a527be775f4d3c343ea591122f1b8921/qa; will attempt to use it 2024-02-05T07:02:07.588 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_164d3be4a527be775f4d3c343ea591122f1b8921/qa/tasks 2024-02-05T07:02:07.588 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-05T07:02:07.593 INFO:teuthology.task.internal:Checking packages... 2024-02-05T07:02:07.619 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '164d3be4a527be775f4d3c343ea591122f1b8921' 2024-02-05T07:02:07.619 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-05T07:02:07.619 INFO:teuthology.packaging:ref: None 2024-02-05T07:02:07.619 INFO:teuthology.packaging:tag: None 2024-02-05T07:02:07.620 INFO:teuthology.packaging:branch: wip-guits-testing-3-2024-02-04-2254 2024-02-05T07:02:07.620 INFO:teuthology.packaging:sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:02:07.620 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-testing-3-2024-02-04-2254 2024-02-05T07:02:07.761 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1241.g164d3be4 2024-02-05T07:02:07.763 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-05T07:02:07.772 INFO:teuthology.task.internal:no buildpackages task found 2024-02-05T07:02:07.772 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-05T07:02:07.787 INFO:teuthology.task.internal:Saving configuration 2024-02-05T07:02:07.801 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-05T07:02:07.812 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T07:02:07.838 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546729', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 06:55:56.963327', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCz1R3OHxhVp2mKR6RqPhAvbhtI32CP2dgQ5Xhmx2AFPKED9mGGBpiR+tejO5ppiDdHVd52nqyNd9wPwDeU958k='} 2024-02-05T07:02:07.863 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546729', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 06:55:56.961345', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN7CVL30sw/Qt233Zvb9eaojHn2grchuZh5xuuFs7Du40CDTCrKcdHjnGtVfbawCn+dmelh5N7JuCk51oGqODcI='} 2024-02-05T07:02:07.864 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-05T07:02:07.875 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-02-05T07:02:07.875 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-02-05T07:02:07.875 INFO:teuthology.run_tasks:Running task console_log... 2024-02-05T07:02:07.957 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f849ccb6f70>, signals=[15]) 2024-02-05T07:02:07.957 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-05T07:02:07.966 INFO:teuthology.task.internal:Opening connections... 2024-02-05T07:02:07.966 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-02-05T07:02:07.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:02:08.073 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-02-05T07:02:08.074 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:02:08.161 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-05T07:02:08.170 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-02-05T07:02:08.188 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-02-05T07:02:08.188 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-02-05T07:02:08.246 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2024-02-05T07:02:08.246 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="9" 2024-02-05T07:02:08.246 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2024-02-05T07:02:08.246 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2024-02-05T07:02:08.247 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="9" 2024-02-05T07:02:08.247 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el9" 2024-02-05T07:02:08.247 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-05T07:02:08.247 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2024-02-05T07:02:08.247 INFO:teuthology.orchestra.run.smithi007.stdout:LOGO="fedora-logo-icon" 2024-02-05T07:02:08.247 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-05T07:02:08.247 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2024-02-05T07:02:08.247 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-05T07:02:08.247 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-05T07:02:08.247 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-05T07:02:08.248 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-02-05T07:02:08.275 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-02-05T07:02:08.292 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-02-05T07:02:08.292 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-02-05T07:02:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2024-02-05T07:02:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="9" 2024-02-05T07:02:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2024-02-05T07:02:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2024-02-05T07:02:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="9" 2024-02-05T07:02:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el9" 2024-02-05T07:02:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-05T07:02:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2024-02-05T07:02:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:LOGO="fedora-logo-icon" 2024-02-05T07:02:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-05T07:02:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2024-02-05T07:02:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-05T07:02:08.350 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-05T07:02:08.351 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-05T07:02:08.351 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-02-05T07:02:08.375 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-05T07:02:08.385 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-05T07:02:08.399 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-05T07:02:08.399 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-02-05T07:02:08.401 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-02-05T07:02:08.417 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-05T07:02:08.423 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-05T07:02:08.423 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-02-05T07:02:08.458 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-02-05T07:02:08.481 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-05T07:02:08.540 INFO:teuthology.run_tasks:Running task kernel... 2024-02-05T07:02:08.561 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-05T07:02:08.561 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-05T07:02:08.561 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-05T07:02:08.562 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-05T07:02:08.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:02:08.578 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-02-05T07:02:08.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:02:08.579 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-02-05T07:02:08.634 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-361.el9.x86_64 2024-02-05T07:02:08.634 INFO:teuthology.task.kernel:Running kernel on smithi090: 5.14.0-361.el9.x86_64 2024-02-05T07:02:08.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum install -y kernel 2024-02-05T07:02:08.637 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-361.el9.x86_64 2024-02-05T07:02:08.638 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.14.0-361.el9.x86_64 2024-02-05T07:02:08.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2024-02-05T07:02:10.827 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 13 MB/s | 6.6 MB 00:00 2024-02-05T07:02:11.116 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 13 MB/s | 6.6 MB 00:00 2024-02-05T07:02:12.294 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-02-05T07:02:12.525 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-02-05T07:02:13.727 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 8.0 MB 00:00 2024-02-05T07:02:14.020 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-02-05T07:02:16.554 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-02-05T07:02:20.944 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 6.5 MB/s | 5.9 MB 00:00 2024-02-05T07:02:21.302 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 3.1 MB/s | 19 MB 00:06 2024-02-05T07:02:22.384 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 15 kB 00:00 2024-02-05T07:02:23.634 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 18 MB/s | 20 MB 00:01 2024-02-05T07:02:26.494 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 5.9 MB 00:01 2024-02-05T07:02:27.895 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 15 kB 00:00 2024-02-05T07:02:28.532 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-02-05T07:02:30.455 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 20 MB 00:02 2024-02-05T07:02:30.714 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-05T07:02:30.715 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-05T07:02:30.829 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:02:30.834 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:02:30.834 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-05T07:02:30.834 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:02:30.835 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:02:30.835 INFO:teuthology.orchestra.run.smithi090.stdout: kernel x86_64 5.14.0-416.el9 CentOS-BaseOS 6.0 M 2024-02-05T07:02:30.835 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-05T07:02:30.835 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core x86_64 5.14.0-416.el9 CentOS-BaseOS 21 M 2024-02-05T07:02:30.835 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules x86_64 5.14.0-416.el9 CentOS-BaseOS 39 M 2024-02-05T07:02:30.835 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core x86_64 5.14.0-416.el9 CentOS-BaseOS 33 M 2024-02-05T07:02:30.835 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:02:30.835 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:02:30.835 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:02:30.835 INFO:teuthology.orchestra.run.smithi090.stdout:Install 4 Packages 2024-02-05T07:02:30.835 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:02:30.836 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 99 M 2024-02-05T07:02:30.836 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 122 M 2024-02-05T07:02:30.836 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:02:31.367 INFO:teuthology.orchestra.run.smithi090.stdout:(1/4): kernel-5.14.0-416.el9.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-02-05T07:02:31.751 INFO:teuthology.orchestra.run.smithi090.stdout:(2/4): kernel-core-5.14.0-416.el9.x86_64.rpm 23 MB/s | 21 MB 00:00 2024-02-05T07:02:32.443 INFO:teuthology.orchestra.run.smithi090.stdout:(3/4): kernel-modules-5.14.0-416.el9.x86_64.rpm 24 MB/s | 39 MB 00:01 2024-02-05T07:02:32.943 INFO:teuthology.orchestra.run.smithi090.stdout:(4/4): kernel-modules-core-5.14.0-416.el9.x86_6 21 MB/s | 33 MB 00:01 2024-02-05T07:02:32.946 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:02:32.946 INFO:teuthology.orchestra.run.smithi090.stdout:Total 47 MB/s | 99 MB 00:02 2024-02-05T07:02:32.947 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:02:33.331 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:02:33.331 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:02:34.382 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:02:34.382 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:02:35.688 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-02-05T07:02:36.856 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:02:37.587 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-core-5.14.0-416.el9.x86_64 1/4 2024-02-05T07:02:37.745 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T07:02:38.003 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-05T07:02:38.003 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-05T07:02:38.097 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:02:38.113 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:02:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-02-05T07:02:38.113 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:02:38.113 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:02:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 5.14.0-416.el9 CentOS-BaseOS 6.0 M 2024-02-05T07:02:38.113 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-02-05T07:02:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core x86_64 5.14.0-416.el9 CentOS-BaseOS 21 M 2024-02-05T07:02:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules x86_64 5.14.0-416.el9 CentOS-BaseOS 39 M 2024-02-05T07:02:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core x86_64 5.14.0-416.el9 CentOS-BaseOS 33 M 2024-02-05T07:02:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:02:38.114 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:02:38.114 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:02:38.114 INFO:teuthology.orchestra.run.smithi007.stdout:Install 4 Packages 2024-02-05T07:02:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:02:38.115 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 99 M 2024-02-05T07:02:38.115 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 122 M 2024-02-05T07:02:38.115 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:02:38.506 INFO:teuthology.orchestra.run.smithi007.stdout:(1/4): kernel-5.14.0-416.el9.x86_64.rpm 15 MB/s | 6.0 MB 00:00 2024-02-05T07:02:39.189 INFO:teuthology.orchestra.run.smithi007.stdout:(2/4): kernel-core-5.14.0-416.el9.x86_64.rpm 19 MB/s | 21 MB 00:01 2024-02-05T07:02:39.606 INFO:teuthology.orchestra.run.smithi007.stdout:(3/4): kernel-modules-core-5.14.0-416.el9.x86_6 30 MB/s | 33 MB 00:01 2024-02-05T07:02:40.240 INFO:teuthology.orchestra.run.smithi007.stdout:(4/4): kernel-modules-5.14.0-416.el9.x86_64.rpm 18 MB/s | 39 MB 00:02 2024-02-05T07:02:40.243 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:02:40.243 INFO:teuthology.orchestra.run.smithi007.stdout:Total 47 MB/s | 99 MB 00:02 2024-02-05T07:02:40.243 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:02:40.538 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T07:02:40.697 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:02:40.697 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:02:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T07:02:41.798 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:02:41.798 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:02:44.336 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:02:45.142 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-core-5.14.0-416.el9.x86_64 1/4 2024-02-05T07:02:45.277 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T07:02:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T07:02:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:02:48.138 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T07:02:48.330 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T07:02:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:02:53.378 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T07:02:53.485 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:02:58.565 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:03:20.504 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:03:20.505 INFO:teuthology.orchestra.run.smithi090.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-05T07:03:20.505 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:03:20.573 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:03:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:03:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-5.14.0-416.el9.x86_64 1/4 2024-02-05T07:03:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T07:03:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T07:03:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:03:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:03:21.621 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:03:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-5.14.0-416.el9.x86_64 2024-02-05T07:03:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core-5.14.0-416.el9.x86_64 2024-02-05T07:03:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-5.14.0-416.el9.x86_64 2024-02-05T07:03:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core-5.14.0-416.el9.x86_64 2024-02-05T07:03:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:03:21.621 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:03:22.179 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T07:03:22.321 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T07:03:22.322 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T07:03:22.322 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-416.el9.x86_64 2024-02-05T07:03:22.322 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-05T07:03:22.322 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-05T07:03:22.323 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-05T07:03:22.323 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=None, version=distro) 2024-02-05T07:03:22.323 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T07:03:22.430 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T07:03:22.430 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T07:03:22.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -qi grub2-tools 2024-02-05T07:03:22.463 INFO:teuthology.orchestra.run.smithi090.stdout:Name : grub2-tools 2024-02-05T07:03:22.463 INFO:teuthology.orchestra.run.smithi090.stdout:Epoch : 1 2024-02-05T07:03:22.463 INFO:teuthology.orchestra.run.smithi090.stdout:Version : 2.06 2024-02-05T07:03:22.463 INFO:teuthology.orchestra.run.smithi090.stdout:Release : 68.el9 2024-02-05T07:03:22.463 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: x86_64 2024-02-05T07:03:22.463 INFO:teuthology.orchestra.run.smithi090.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-05T07:03:22.463 INFO:teuthology.orchestra.run.smithi090.stdout:Group : Unspecified 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout:Size : 8264254 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout:License : GPLv3+ 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout:Packager : builder@centos.org 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout:Vendor : CentOS 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout:Summary : Support tools for GRUB. 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout:Description : 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-05T07:03:22.464 INFO:teuthology.orchestra.run.smithi090.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-05T07:03:22.465 INFO:teuthology.orchestra.run.smithi090.stdout:hardware devices. 2024-02-05T07:03:22.465 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:03:22.465 INFO:teuthology.orchestra.run.smithi090.stdout:This subpackage provides tools for support of all platforms. 2024-02-05T07:03:22.465 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-05T07:03:22.465 INFO:teuthology.task.kernel:Updating grub on smithi090 to boot 5.14.0-416.el9.x86_64 2024-02-05T07:03:22.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-05T07:03:23.116 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-02-05T07:03:25.000 INFO:teuthology.orchestra.run.smithi090.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-05T07:03:25.038 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-02-05T07:03:25.041 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-02-05T07:03:25.063 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.x9pWb9rjao 2024-02-05T07:03:25.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.x9pWb9rjao 2024-02-05T07:03:25.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.x9pWb9rjao 2024-02-05T07:03:25.286 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.x9pWb9rjao is 6KB 2024-02-05T07:03:25.334 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.x9pWb9rjao 2024-02-05T07:03:25.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo /bin/ls /boot/loader/entries || true 2024-02-05T07:03:25.417 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-05T07:03:25.417 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-05T07:03:25.417 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-05T07:03:25.417 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64.conf 2024-02-05T07:03:25.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64 2024-02-05T07:03:25.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2024-02-05T07:03:29.505 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:03:29.505 INFO:teuthology.orchestra.run.smithi007.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-05T07:03:29.506 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:03:29.578 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:03:30.041 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:03:30.041 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-5.14.0-416.el9.x86_64 1/4 2024-02-05T07:03:30.041 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T07:03:30.041 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T07:03:30.500 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:03:30.500 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:03:30.500 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:03:30.500 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-5.14.0-416.el9.x86_64 2024-02-05T07:03:30.500 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core-5.14.0-416.el9.x86_64 2024-02-05T07:03:30.500 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-5.14.0-416.el9.x86_64 2024-02-05T07:03:30.500 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core-5.14.0-416.el9.x86_64 2024-02-05T07:03:30.500 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:03:30.500 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:03:31.003 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T07:03:31.140 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T07:03:31.140 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T07:03:31.141 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-416.el9.x86_64 2024-02-05T07:03:31.141 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-05T07:03:31.141 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-05T07:03:31.141 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-05T07:03:31.141 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2024-02-05T07:03:31.141 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T07:03:31.254 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T07:03:31.255 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T07:03:31.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2024-02-05T07:03:31.292 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2024-02-05T07:03:31.292 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2024-02-05T07:03:31.292 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.06 2024-02-05T07:03:31.292 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 68.el9 2024-02-05T07:03:31.292 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout:Group : Unspecified 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 8264254 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : builder@centos.org 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : CentOS 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:03:31.293 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-05T07:03:31.294 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-05T07:03:31.294 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-05T07:03:31.294 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2024-02-05T07:03:31.294 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:03:31.294 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2024-02-05T07:03:31.295 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-05T07:03:31.295 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 5.14.0-416.el9.x86_64 2024-02-05T07:03:31.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-05T07:03:31.930 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2024-02-05T07:03:33.368 INFO:teuthology.orchestra.run.smithi007.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-05T07:03:33.414 INFO:teuthology.orchestra.run.smithi007.stderr:done 2024-02-05T07:03:33.416 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-02-05T07:03:33.431 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.05lPA07S53 2024-02-05T07:03:33.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.05lPA07S53 2024-02-05T07:03:33.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.05lPA07S53 2024-02-05T07:03:33.646 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.05lPA07S53 is 6KB 2024-02-05T07:03:33.666 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.05lPA07S53 2024-02-05T07:03:33.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2024-02-05T07:03:33.757 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-05T07:03:33.757 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-05T07:03:33.757 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-05T07:03:33.757 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64.conf 2024-02-05T07:03:33.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64 2024-02-05T07:03:33.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2024-02-05T07:03:55.588 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-05T07:03:55.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-05T07:03:55.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:04:03.933 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-05T07:04:03.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-02-05T07:04:03.934 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:04:30.269 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-05T07:04:39.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-05T07:04:39.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:04:42.332 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-05T07:04:54.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-05T07:04:54.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:05:03.941 DEBUG:teuthology.orchestra.remote:timed out 2024-02-05T07:05:12.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-02-05T07:05:12.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:05:13.557 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-02-05T07:05:13.984 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-02-05T07:05:13.985 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-416.el9.x86_64"... 2024-02-05T07:05:13.985 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-02-05T07:05:13.999 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-416.el9.x86_64 2024-02-05T07:05:13.999 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-416.el9.x86_64 vs 5.14.0-416.el9.x86_64 2024-02-05T07:05:13.999 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-05T07:05:13.999 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-05T07:05:15.000 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-05T07:05:15.001 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-05T07:05:15.059 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2024-02-05T07:05:15.113 DEBUG:teuthology.parallel:result is None 2024-02-05T07:05:26.455 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-02-05T07:05:26.974 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-05T07:05:26.975 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-416.el9.x86_64"... 2024-02-05T07:05:26.975 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-02-05T07:05:26.989 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-416.el9.x86_64 2024-02-05T07:05:26.989 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-416.el9.x86_64 vs 5.14.0-416.el9.x86_64 2024-02-05T07:05:26.989 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-05T07:05:26.990 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-05T07:05:27.991 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-05T07:05:27.992 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-05T07:05:28.095 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2024-02-05T07:05:28.135 DEBUG:teuthology.parallel:result is None 2024-02-05T07:05:28.135 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-05T07:05:28.142 INFO:teuthology.task.internal:Creating test directory... 2024-02-05T07:05:28.142 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-05T07:05:28.145 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-05T07:05:28.161 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-05T07:05:28.167 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-05T07:05:28.177 INFO:teuthology.task.internal:Creating archive directory... 2024-02-05T07:05:28.177 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-05T07:05:28.203 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-05T07:05:28.246 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-05T07:05:28.254 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-05T07:05:28.255 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-05T07:05:28.281 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-05T07:05:28.316 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T07:05:28.317 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T07:05:28.328 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T07:05:28.331 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T07:05:28.333 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-05T07:05:28.340 INFO:teuthology.task.internal:Configuring sudo... 2024-02-05T07:05:28.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-05T07:05:28.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-05T07:05:28.403 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-05T07:05:28.412 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-05T07:05:28.412 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-05T07:05:28.446 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-05T07:05:28.464 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T07:05:28.545 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T07:05:28.589 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T07:05:28.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T07:05:28.714 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:05:28.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-05T07:05:28.782 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T07:05:28.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T07:05:28.883 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T07:05:28.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T07:05:29.012 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-05T07:05:29.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-05T07:05:29.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-02-05T07:05:29.113 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-02-05T07:05:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T07:05:29.202 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T07:05:29.516 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-05T07:05:29.524 INFO:teuthology.task.internal:Starting timer... 2024-02-05T07:05:29.524 INFO:teuthology.run_tasks:Running task pcp... 2024-02-05T07:05:29.545 INFO:teuthology.run_tasks:Running task selinux... 2024-02-05T07:05:29.556 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-05T07:05:29.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2024-02-05T07:05:29.627 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2024-02-05T07:05:29.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2024-02-05T07:05:29.723 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2024-02-05T07:05:29.725 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-05T07:05:29.725 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2024-02-05T07:05:29.764 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2024-02-05T07:05:29.765 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2024-02-05T07:05:29.792 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2024-02-05T07:05:29.792 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi090.front.sepia.ceph.com': 'permissive'} 2024-02-05T07:05:29.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T07:05:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:05:29.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T07:05:29.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:05:29.863 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-05T07:05:29.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2024-02-05T07:05:29.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2024-02-05T07:05:29.950 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-05T07:05:29.963 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-05T07:05:29.985 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-05T07:05:29.986 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi090.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-05T07:09:41.288 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi090.front.sepia.ceph.com')] 2024-02-05T07:09:41.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-02-05T07:09:41.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:09:41.384 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-02-05T07:09:41.466 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-02-05T07:09:41.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-05T07:09:41.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:09:41.556 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-02-05T07:09:41.639 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-05T07:09:41.639 INFO:teuthology.run_tasks:Running task clock... 2024-02-05T07:09:41.651 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-05T07:09:41.651 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-05T07:09:41.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T07:09:41.654 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-05T07:09:41.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T07:09:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-05T07:09:41.701 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-05T07:09:41.710 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-05T07:09:41.726 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-05T07:09:41.730 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2024-02-05T07:09:41.744 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2024-02-05T07:09:41.755 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2024-02-05T07:09:41.761 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-05T07:09:41.769 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2024-02-05T07:09:41.777 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-05T07:09:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-05T07:09:41.801 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-05T07:09:41.828 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-02-05T07:09:41.831 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T07:09:41.831 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-02-05T07:09:41.831 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:09:41.831 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:09:41.831 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:09:41.831 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:09:41.851 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-02-05T07:09:41.854 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T07:09:41.854 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-02-05T07:09:41.854 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:09:41.855 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:09:41.855 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:09:41.855 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:09:41.856 INFO:teuthology.run_tasks:Running task install... 2024-02-05T07:09:41.865 DEBUG:teuthology.task.install:project ceph 2024-02-05T07:09:41.865 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'}} 2024-02-05T07:09:41.865 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'} 2024-02-05T07:09:41.865 INFO:teuthology.task.install:Using flavor: default 2024-02-05T07:09:41.871 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-05T07:09:41.871 INFO:teuthology.task.install:extra packages: [] 2024-02-05T07:09:41.872 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921', 'tag': None, 'wait_for_package': False} 2024-02-05T07:09:41.872 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:09:41.874 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921', 'tag': None, 'wait_for_package': False} 2024-02-05T07:09:41.874 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:09:42.021 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/ 2024-02-05T07:09:42.021 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1241.g164d3be4 2024-02-05T07:09:42.092 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/ 2024-02-05T07:09:42.093 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1241.g164d3be4 2024-02-05T07:09:42.311 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-05T07:09:42.311 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:09:42.311 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-05T07:09:42.340 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-05T07:09:42.340 DEBUG:teuthology.orchestra.run.smithi007:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/164d3be4a527be775f4d3c343ea591122f1b8921/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-05T07:09:42.396 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-05T07:09:42.396 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-05T07:09:42.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-05T07:09:42.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-05T07:09:42.425 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-05T07:09:42.425 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/164d3be4a527be775f4d3c343ea591122f1b8921/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-05T07:09:42.490 DEBUG:teuthology.orchestra.run.smithi007:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-05T07:09:42.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-05T07:09:42.557 INFO:teuthology.orchestra.run.smithi007.stdout:check_obsoletes = 1 2024-02-05T07:09:42.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum clean all 2024-02-05T07:09:42.572 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-05T07:09:42.639 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2024-02-05T07:09:42.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2024-02-05T07:09:42.811 INFO:teuthology.orchestra.run.smithi007.stdout:74 files removed 2024-02-05T07:09:42.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw 2024-02-05T07:09:42.888 INFO:teuthology.orchestra.run.smithi090.stdout:74 files removed 2024-02-05T07:09:42.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw 2024-02-05T07:09:43.576 INFO:teuthology.orchestra.run.smithi007.stdout:ceph packages for x86_64 193 kB/s | 80 kB 00:00 2024-02-05T07:09:43.591 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 226 kB/s | 80 kB 00:00 2024-02-05T07:09:43.992 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 28 kB/s | 10 kB 00:00 2024-02-05T07:09:44.009 INFO:teuthology.orchestra.run.smithi007.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-02-05T07:09:44.327 INFO:teuthology.orchestra.run.smithi007.stdout:ceph source packages 6.8 kB/s | 1.9 kB 00:00 2024-02-05T07:09:44.342 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-02-05T07:09:44.820 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 15 MB/s | 6.6 MB 00:00 2024-02-05T07:09:44.837 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 14 MB/s | 6.6 MB 00:00 2024-02-05T07:09:46.254 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-02-05T07:09:46.355 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-02-05T07:09:49.053 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 8.0 MB 00:02 2024-02-05T07:09:49.897 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.0 MB 00:03 2024-02-05T07:09:53.546 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 19 MB 00:03 2024-02-05T07:09:53.640 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 19 MB 00:02 2024-02-05T07:09:57.549 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 14 MB/s | 5.9 MB 00:00 2024-02-05T07:09:58.942 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 15 kB 00:00 2024-02-05T07:09:59.355 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 5.9 MB 00:02 2024-02-05T07:10:00.703 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2024-02-05T07:10:00.730 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 15 kB 00:00 2024-02-05T07:10:01.701 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 24 MB/s | 20 MB 00:00 2024-02-05T07:10:05.620 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-02-05T07:10:06.790 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-02-05T07:10:08.175 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:10:08.189 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:10:08.189 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-05T07:10:08.190 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:10:08.190 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:10:08.190 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 9.7 M 2024-02-05T07:10:08.190 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-02-05T07:10:08.190 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.4 M 2024-02-05T07:10:08.190 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.1 M 2024-02-05T07:10:08.190 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-05T07:10:08.190 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options 2024-02-05T07:10:08.190 INFO:teuthology.orchestra.run.smithi090.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-05T07:10:08.190 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 5.1 M 2024-02-05T07:10:08.190 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 20 M 2024-02-05T07:10:08.190 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 25 k 2024-02-05T07:10:08.190 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-05T07:10:08.191 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-05T07:10:08.191 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-05T07:10:08.191 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 743 k 2024-02-05T07:10:08.191 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-05T07:10:08.191 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-05T07:10:08.191 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-05T07:10:08.191 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 498 k 2024-02-05T07:10:08.191 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-05T07:10:08.191 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 5.1 M 2024-02-05T07:10:08.191 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-05T07:10:08.191 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-05T07:10:08.191 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-05T07:10:08.191 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse 2024-02-05T07:10:08.192 INFO:teuthology.orchestra.run.smithi090.stdout: x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 45 k 2024-02-05T07:10:08.192 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 131 k 2024-02-05T07:10:08.192 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 162 k 2024-02-05T07:10:08.192 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-05T07:10:08.192 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 320 k 2024-02-05T07:10:08.192 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 298 k 2024-02-05T07:10:08.192 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 99 k 2024-02-05T07:10:08.192 INFO:teuthology.orchestra.run.smithi090.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-05T07:10:08.192 INFO:teuthology.orchestra.run.smithi090.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-05T07:10:08.192 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:10:08.192 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:10:08.192 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:10:08.192 INFO:teuthology.orchestra.run.smithi090.stdout:Install 27 Packages 2024-02-05T07:10:08.192 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 2 Packages 2024-02-05T07:10:08.193 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:10:08.193 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 57 M 2024-02-05T07:10:08.193 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:10:08.906 INFO:teuthology.orchestra.run.smithi090.stdout:(1/29): ceph-base-19.0.0-1241.g164d3be4.el9.x86 7.9 MB/s | 5.1 MB 00:00 2024-02-05T07:10:08.982 INFO:teuthology.orchestra.run.smithi090.stdout:(2/29): ceph-selinux-19.0.0-1241.g164d3be4.el9. 335 kB/s | 25 kB 00:00 2024-02-05T07:10:09.090 INFO:teuthology.orchestra.run.smithi090.stdout:(3/29): libcephfs2-19.0.0-1241.g164d3be4.el9.x8 6.7 MB/s | 743 kB 00:00 2024-02-05T07:10:09.174 INFO:teuthology.orchestra.run.smithi090.stdout:(4/29): libradosstriper1-19.0.0-1241.g164d3be4. 5.8 MB/s | 498 kB 00:00 2024-02-05T07:10:09.479 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:10:09.496 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:10:09.497 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-02-05T07:10:09.497 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:10:09.497 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:10:09.497 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 9.7 M 2024-02-05T07:10:09.497 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2024-02-05T07:10:09.497 INFO:teuthology.orchestra.run.smithi007.stdout: librados2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.4 M 2024-02-05T07:10:09.498 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.1 M 2024-02-05T07:10:09.498 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-02-05T07:10:09.498 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options 2024-02-05T07:10:09.498 INFO:teuthology.orchestra.run.smithi007.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-05T07:10:09.498 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 5.1 M 2024-02-05T07:10:09.498 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 20 M 2024-02-05T07:10:09.499 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 25 k 2024-02-05T07:10:09.499 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-05T07:10:09.499 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-05T07:10:09.499 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-05T07:10:09.499 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 743 k 2024-02-05T07:10:09.499 INFO:teuthology.orchestra.run.smithi007.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-05T07:10:09.500 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-05T07:10:09.500 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-05T07:10:09.500 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 498 k 2024-02-05T07:10:09.500 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-05T07:10:09.500 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 5.1 M 2024-02-05T07:10:09.500 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-05T07:10:09.500 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-05T07:10:09.500 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-05T07:10:09.500 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse 2024-02-05T07:10:09.500 INFO:teuthology.orchestra.run.smithi007.stdout: x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 45 k 2024-02-05T07:10:09.501 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 131 k 2024-02-05T07:10:09.501 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 162 k 2024-02-05T07:10:09.501 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-05T07:10:09.501 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 320 k 2024-02-05T07:10:09.501 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 298 k 2024-02-05T07:10:09.501 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 99 k 2024-02-05T07:10:09.501 INFO:teuthology.orchestra.run.smithi007.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-05T07:10:09.501 INFO:teuthology.orchestra.run.smithi007.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-05T07:10:09.501 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:09.502 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:10:09.502 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:10:09.502 INFO:teuthology.orchestra.run.smithi007.stdout:Install 27 Packages 2024-02-05T07:10:09.502 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 2 Packages 2024-02-05T07:10:09.502 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:09.502 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 57 M 2024-02-05T07:10:09.503 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:10:09.741 INFO:teuthology.orchestra.run.smithi090.stdout:(5/29): librgw2-19.0.0-1241.g164d3be4.el9.x86_6 9.0 MB/s | 5.1 MB 00:00 2024-02-05T07:10:10.014 INFO:teuthology.orchestra.run.smithi090.stdout:(6/29): ceph-common-19.0.0-1241.g164d3be4.el9.x 11 MB/s | 20 MB 00:01 2024-02-05T07:10:10.042 INFO:teuthology.orchestra.run.smithi090.stdout:(7/29): python3-ceph-argparse-19.0.0-1241.g164d 150 kB/s | 45 kB 00:00 2024-02-05T07:10:10.067 INFO:teuthology.orchestra.run.smithi090.stdout:(8/29): python3-ceph-common-19.0.0-1241.g164d3b 2.4 MB/s | 131 kB 00:00 2024-02-05T07:10:10.093 INFO:teuthology.orchestra.run.smithi090.stdout:(9/29): python3-cephfs-19.0.0-1241.g164d3be4.el 3.2 MB/s | 162 kB 00:00 2024-02-05T07:10:10.126 INFO:teuthology.orchestra.run.smithi090.stdout:(10/29): python3-rados-19.0.0-1241.g164d3be4.el 5.3 MB/s | 320 kB 00:00 2024-02-05T07:10:10.152 INFO:teuthology.orchestra.run.smithi090.stdout:(11/29): python3-rbd-19.0.0-1241.g164d3be4.el9. 5.0 MB/s | 298 kB 00:00 2024-02-05T07:10:10.177 INFO:teuthology.orchestra.run.smithi090.stdout:(12/29): python3-rgw-19.0.0-1241.g164d3be4.el9. 1.9 MB/s | 99 kB 00:00 2024-02-05T07:10:10.327 INFO:teuthology.orchestra.run.smithi090.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 603 kB/s | 106 kB 00:00 2024-02-05T07:10:10.353 INFO:teuthology.orchestra.run.smithi090.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-02-05T07:10:10.370 INFO:teuthology.orchestra.run.smithi090.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-02-05T07:10:10.453 INFO:teuthology.orchestra.run.smithi090.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 661 kB 00:00 2024-02-05T07:10:10.487 INFO:teuthology.orchestra.run.smithi090.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 282 kB 00:00 2024-02-05T07:10:10.513 INFO:teuthology.orchestra.run.smithi090.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 703 kB/s | 41 kB 00:00 2024-02-05T07:10:10.841 INFO:teuthology.orchestra.run.smithi007.stdout:(1/29): ceph-base-19.0.0-1241.g164d3be4.el9.x86 5.1 MB/s | 5.1 MB 00:00 2024-02-05T07:10:10.892 INFO:teuthology.orchestra.run.smithi007.stdout:(2/29): ceph-selinux-19.0.0-1241.g164d3be4.el9. 507 kB/s | 25 kB 00:00 2024-02-05T07:10:11.000 INFO:teuthology.orchestra.run.smithi007.stdout:(3/29): libcephfs2-19.0.0-1241.g164d3be4.el9.x8 6.7 MB/s | 743 kB 00:00 2024-02-05T07:10:11.100 INFO:teuthology.orchestra.run.smithi007.stdout:(4/29): libradosstriper1-19.0.0-1241.g164d3be4. 4.9 MB/s | 498 kB 00:00 2024-02-05T07:10:11.309 INFO:teuthology.orchestra.run.smithi007.stdout:(5/29): ceph-radosgw-19.0.0-1241.g164d3be4.el9. 6.7 MB/s | 9.7 MB 00:01 2024-02-05T07:10:11.359 INFO:teuthology.orchestra.run.smithi007.stdout:(6/29): python3-ceph-argparse-19.0.0-1241.g164d 904 kB/s | 45 kB 00:00 2024-02-05T07:10:11.426 INFO:teuthology.orchestra.run.smithi007.stdout:(7/29): python3-ceph-common-19.0.0-1241.g164d3b 1.9 MB/s | 131 kB 00:00 2024-02-05T07:10:11.476 INFO:teuthology.orchestra.run.smithi007.stdout:(8/29): python3-cephfs-19.0.0-1241.g164d3be4.el 3.2 MB/s | 162 kB 00:00 2024-02-05T07:10:11.543 INFO:teuthology.orchestra.run.smithi007.stdout:(9/29): python3-rados-19.0.0-1241.g164d3be4.el9 4.7 MB/s | 320 kB 00:00 2024-02-05T07:10:11.564 INFO:teuthology.orchestra.run.smithi090.stdout:(19/29): ceph-radosgw-19.0.0-1241.g164d3be4.el9 2.9 MB/s | 9.7 MB 00:03 2024-02-05T07:10:11.619 INFO:teuthology.orchestra.run.smithi007.stdout:(10/29): librgw2-19.0.0-1241.g164d3be4.el9.x86_ 9.8 MB/s | 5.1 MB 00:00 2024-02-05T07:10:11.644 INFO:teuthology.orchestra.run.smithi007.stdout:(11/29): python3-rbd-19.0.0-1241.g164d3be4.el9. 2.9 MB/s | 298 kB 00:00 2024-02-05T07:10:11.670 INFO:teuthology.orchestra.run.smithi007.stdout:(12/29): python3-rgw-19.0.0-1241.g164d3be4.el9. 1.9 MB/s | 99 kB 00:00 2024-02-05T07:10:11.811 INFO:teuthology.orchestra.run.smithi007.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 633 kB/s | 106 kB 00:00 2024-02-05T07:10:11.837 INFO:teuthology.orchestra.run.smithi007.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 954 kB/s | 159 kB 00:00 2024-02-05T07:10:11.862 INFO:teuthology.orchestra.run.smithi007.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-02-05T07:10:11.929 INFO:teuthology.orchestra.run.smithi007.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-02-05T07:10:11.955 INFO:teuthology.orchestra.run.smithi007.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-02-05T07:10:11.980 INFO:teuthology.orchestra.run.smithi007.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2024-02-05T07:10:12.064 INFO:teuthology.orchestra.run.smithi090.stdout:(20/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 196 kB/s | 309 kB 00:01 2024-02-05T07:10:12.156 INFO:teuthology.orchestra.run.smithi090.stdout:(21/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 44 kB/s | 26 kB 00:00 2024-02-05T07:10:12.365 INFO:teuthology.orchestra.run.smithi090.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 162 kB/s | 49 kB 00:00 2024-02-05T07:10:12.515 INFO:teuthology.orchestra.run.smithi090.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 188 kB/s | 67 kB 00:00 2024-02-05T07:10:12.905 INFO:teuthology.orchestra.run.smithi007.stdout:(19/29): ceph-common-19.0.0-1241.g164d3be4.el9. 6.5 MB/s | 20 MB 00:03 2024-02-05T07:10:12.938 INFO:teuthology.orchestra.run.smithi007.stdout:(20/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 314 kB/s | 309 kB 00:00 2024-02-05T07:10:13.105 INFO:teuthology.orchestra.run.smithi007.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 292 kB/s | 49 kB 00:00 2024-02-05T07:10:13.206 INFO:teuthology.orchestra.run.smithi007.stdout:(22/29): libarrow-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 4.4 MB 00:01 2024-02-05T07:10:13.231 INFO:teuthology.orchestra.run.smithi007.stdout:(23/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 80 kB/s | 26 kB 00:00 2024-02-05T07:10:13.264 INFO:teuthology.orchestra.run.smithi007.stdout:(24/29): libunwind-1.6.2-1.el9.x86_64.rpm 424 kB/s | 67 kB 00:00 2024-02-05T07:10:13.348 INFO:teuthology.orchestra.run.smithi007.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-02-05T07:10:13.565 INFO:teuthology.orchestra.run.smithi007.stdout:(26/29): re2-20211101-3.el9.x86_64.rpm 574 kB/s | 192 kB 00:00 2024-02-05T07:10:13.665 INFO:teuthology.orchestra.run.smithi007.stdout:(27/29): librados2-19.0.0-1241.g164d3be4.el9.x8 11 MB/s | 3.4 MB 00:00 2024-02-05T07:10:13.707 INFO:teuthology.orchestra.run.smithi007.stdout:(28/29): thrift-0.15.0-2.el9.x86_64.rpm 3.6 MB/s | 1.6 MB 00:00 2024-02-05T07:10:14.224 INFO:teuthology.orchestra.run.smithi007.stdout:(29/29): librbd1-19.0.0-1241.g164d3be4.el9.x86_ 4.8 MB/s | 3.1 MB 00:00 2024-02-05T07:10:14.228 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:10:14.228 INFO:teuthology.orchestra.run.smithi007.stdout:Total 12 MB/s | 57 MB 00:04 2024-02-05T07:10:14.229 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:10:14.269 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:10:14.270 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:10:14.776 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:10:14.777 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:10:15.316 INFO:teuthology.orchestra.run.smithi090.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 68 kB/s | 192 kB 00:02 2024-02-05T07:10:15.491 INFO:teuthology.orchestra.run.smithi090.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 268 kB/s | 838 kB 00:03 2024-02-05T07:10:15.825 INFO:teuthology.orchestra.run.smithi090.stdout:(26/29): librados2-19.0.0-1241.g164d3be4.el9.x8 10 MB/s | 3.4 MB 00:00 2024-02-05T07:10:16.050 INFO:teuthology.orchestra.run.smithi090.stdout:(27/29): librbd1-19.0.0-1241.g164d3be4.el9.x86_ 14 MB/s | 3.1 MB 00:00 2024-02-05T07:10:16.191 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:10:16.278 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-02-05T07:10:16.312 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-02-05T07:10:16.363 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-02-05T07:10:16.772 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-02-05T07:10:16.803 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T07:10:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T07:10:16.993 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 6/31 2024-02-05T07:10:17.021 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T07:10:17.083 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T07:10:17.112 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-02-05T07:10:17.157 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9. 9/31 2024-02-05T07:10:17.227 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 10/31 2024-02-05T07:10:17.248 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 11/31 2024-02-05T07:10:17.314 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 11/31 2024-02-05T07:10:17.425 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-02-05T07:10:17.464 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-02-05T07:10:17.505 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-02-05T07:10:17.537 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-02-05T07:10:17.566 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-02-05T07:10:17.599 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-02-05T07:10:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-02-05T07:10:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 19/31 2024-02-05T07:10:18.052 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 19/31 2024-02-05T07:10:18.083 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 20/31 2024-02-05T07:10:18.128 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-02-05T07:10:18.608 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-02-05T07:10:18.639 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 23/31 2024-02-05T07:10:18.711 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 23/31 2024-02-05T07:10:18.790 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 24/31 2024-02-05T07:10:18.827 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x8 25/31 2024-02-05T07:10:19.551 INFO:teuthology.orchestra.run.smithi090.stdout:(28/29): thrift-0.15.0-2.el9.x86_64.rpm 382 kB/s | 1.6 MB 00:04 2024-02-05T07:10:21.144 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T07:10:21.197 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T07:10:21.752 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T07:10:21.799 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 27/31 2024-02-05T07:10:21.892 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 27/31 2024-02-05T07:10:21.893 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-05T07:10:21.893 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-05T07:10:21.893 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:21.934 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T07:10:22.592 INFO:teuthology.orchestra.run.smithi090.stdout:(29/29): libarrow-9.0.0-10.el9.x86_64.rpm 374 kB/s | 4.4 MB 00:12 2024-02-05T07:10:22.600 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:10:22.600 INFO:teuthology.orchestra.run.smithi090.stdout:Total 4.0 MB/s | 57 MB 00:14 2024-02-05T07:10:22.600 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:10:22.637 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:10:22.637 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:10:23.112 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:10:23.112 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:10:24.518 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:10:24.603 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-02-05T07:10:24.631 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-02-05T07:10:24.681 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-02-05T07:10:25.017 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-02-05T07:10:25.090 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T07:10:25.181 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T07:10:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 6/31 2024-02-05T07:10:25.317 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T07:10:25.379 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T07:10:25.417 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-02-05T07:10:25.460 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9. 9/31 2024-02-05T07:10:25.529 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 10/31 2024-02-05T07:10:25.553 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 11/31 2024-02-05T07:10:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 11/31 2024-02-05T07:10:25.726 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-02-05T07:10:25.760 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-02-05T07:10:25.809 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-02-05T07:10:25.844 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-02-05T07:10:25.881 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-02-05T07:10:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-02-05T07:10:26.223 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-02-05T07:10:26.252 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 19/31 2024-02-05T07:10:26.334 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 19/31 2024-02-05T07:10:26.365 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 20/31 2024-02-05T07:10:26.408 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-02-05T07:10:26.857 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-02-05T07:10:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 23/31 2024-02-05T07:10:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 23/31 2024-02-05T07:10:27.037 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 24/31 2024-02-05T07:10:27.068 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x8 25/31 2024-02-05T07:10:29.292 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T07:10:29.349 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T07:10:29.928 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T07:10:29.979 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 27/31 2024-02-05T07:10:30.071 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 27/31 2024-02-05T07:10:30.071 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-05T07:10:30.072 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-05T07:10:30.072 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:10:30.114 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T07:10:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T07:10:31.998 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /sys 2024-02-05T07:10:31.998 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /proc 2024-02-05T07:10:31.998 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /mnt 2024-02-05T07:10:31.998 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /var/tmp 2024-02-05T07:10:31.998 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /home 2024-02-05T07:10:31.998 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /root 2024-02-05T07:10:31.998 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /tmp 2024-02-05T07:10:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:32.941 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 29/31 2024-02-05T07:10:32.968 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 29/31 2024-02-05T07:10:32.968 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:10:32.968 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-05T07:10:32.968 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T07:10:32.968 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T07:10:32.969 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:32.970 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T07:10:33.053 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T07:10:33.053 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T07:10:34.188 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T07:10:34.188 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 1/31 2024-02-05T07:10:34.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2/31 2024-02-05T07:10:34.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 3/31 2024-02-05T07:10:34.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 4/31 2024-02-05T07:10:34.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T07:10:34.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 6/31 2024-02-05T07:10:34.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T07:10:34.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9. 8/31 2024-02-05T07:10:34.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x8 9/31 2024-02-05T07:10:34.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 10/31 2024-02-05T07:10:34.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 11/31 2024-02-05T07:10:34.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 12/31 2024-02-05T07:10:34.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 13/31 2024-02-05T07:10:34.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-02-05T07:10:34.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-02-05T07:10:34.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-02-05T07:10:34.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-02-05T07:10:34.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-02-05T07:10:34.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-02-05T07:10:34.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-02-05T07:10:34.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-02-05T07:10:34.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-02-05T07:10:34.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-02-05T07:10:34.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-02-05T07:10:34.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-02-05T07:10:34.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-02-05T07:10:34.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-02-05T07:10:34.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T07:10:34.191 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-02-05T07:10:34.191 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 30/31 2024-02-05T07:10:35.231 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T07:10:35.231 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:35.231 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2024-02-05T07:10:35.231 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.231 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.231 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:10:35.231 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-05T07:10:35.231 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.231 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.231 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.232 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.232 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-05T07:10:35.232 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-05T07:10:35.232 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-05T07:10:35.232 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.232 INFO:teuthology.orchestra.run.smithi007.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-05T07:10:35.232 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-05T07:10:35.232 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-05T07:10:35.232 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.232 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-05T07:10:35.232 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.232 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-05T07:10:35.233 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-05T07:10:35.233 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-05T07:10:35.233 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.233 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.233 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.233 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-05T07:10:35.233 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.233 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.233 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:35.233 INFO:teuthology.orchestra.run.smithi007.stdout: re2-1:20211101-3.el9.x86_64 2024-02-05T07:10:35.233 INFO:teuthology.orchestra.run.smithi007.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-05T07:10:35.233 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:35.233 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:10:35.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-test 2024-02-05T07:10:36.170 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:30 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:10:36.332 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:10:36.333 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:10:36.333 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-02-05T07:10:36.333 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:10:36.333 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:10:36.333 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 47 M 2024-02-05T07:10:36.334 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-02-05T07:10:36.334 INFO:teuthology.orchestra.run.smithi007.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-05T07:10:36.334 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 163 k 2024-02-05T07:10:36.334 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-05T07:10:36.334 INFO:teuthology.orchestra.run.smithi007.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-05T07:10:36.334 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-05T07:10:36.334 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:36.334 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:10:36.334 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:10:36.334 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2024-02-05T07:10:36.334 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:36.335 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 48 M 2024-02-05T07:10:36.335 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 204 M 2024-02-05T07:10:36.335 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:10:36.539 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-02-05T07:10:36.573 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-02-05T07:10:36.623 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-02-05T07:10:36.648 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): libcephsqlite-19.0.0-1241.g164d3be4.el9. 522 kB/s | 163 kB 00:00 2024-02-05T07:10:36.682 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-02-05T07:10:38.849 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): ceph-test-19.0.0-1241.g164d3be4.el9.x86_ 19 MB/s | 47 MB 00:02 2024-02-05T07:10:38.851 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:10:38.852 INFO:teuthology.orchestra.run.smithi007.stdout:Total 19 MB/s | 48 MB 00:02 2024-02-05T07:10:38.852 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:10:38.875 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:10:38.876 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:10:39.277 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:10:39.278 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:10:39.759 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T07:10:39.759 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2024-02-05T07:10:39.759 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2024-02-05T07:10:39.759 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2024-02-05T07:10:39.760 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2024-02-05T07:10:39.760 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2024-02-05T07:10:39.760 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /root 2024-02-05T07:10:39.760 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2024-02-05T07:10:39.760 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:10:39.870 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:10:39.942 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-05T07:10:39.982 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-05T07:10:40.027 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-05T07:10:40.076 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-05T07:10:40.102 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 5/6 2024-02-05T07:10:40.691 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 29/31 2024-02-05T07:10:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 29/31 2024-02-05T07:10:40.717 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:10:40.717 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-05T07:10:40.717 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T07:10:40.717 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T07:10:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:10:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T07:10:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T07:10:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T07:10:42.161 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T07:10:42.161 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 1/31 2024-02-05T07:10:42.161 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2/31 2024-02-05T07:10:42.161 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 3/31 2024-02-05T07:10:42.161 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 4/31 2024-02-05T07:10:42.161 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T07:10:42.161 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 6/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9. 8/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x8 9/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 10/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 11/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 12/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 13/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-02-05T07:10:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-02-05T07:10:42.163 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-02-05T07:10:42.163 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-02-05T07:10:42.163 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-02-05T07:10:42.163 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-02-05T07:10:42.163 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-02-05T07:10:42.163 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-02-05T07:10:42.163 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T07:10:42.163 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-02-05T07:10:42.163 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 30/31 2024-02-05T07:10:43.237 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T07:10:43.237 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:10:43.237 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-02-05T07:10:43.237 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.237 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.237 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:10:43.237 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-05T07:10:43.237 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-05T07:10:43.238 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-05T07:10:43.239 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.239 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.239 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.239 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-05T07:10:43.239 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.239 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.239 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:43.239 INFO:teuthology.orchestra.run.smithi090.stdout: re2-1:20211101-3.el9.x86_64 2024-02-05T07:10:43.239 INFO:teuthology.orchestra.run.smithi090.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-05T07:10:43.239 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:10:43.239 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:10:43.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-test 2024-02-05T07:10:44.233 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:39 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:10:44.397 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:10:44.398 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:10:44.398 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-05T07:10:44.398 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:10:44.398 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:10:44.398 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 47 M 2024-02-05T07:10:44.398 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-05T07:10:44.398 INFO:teuthology.orchestra.run.smithi090.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-05T07:10:44.398 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 163 k 2024-02-05T07:10:44.398 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-05T07:10:44.398 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-05T07:10:44.399 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-05T07:10:44.399 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:10:44.399 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:10:44.399 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:10:44.399 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-02-05T07:10:44.399 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:10:44.399 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 48 M 2024-02-05T07:10:44.400 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 204 M 2024-02-05T07:10:44.400 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:10:44.610 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-02-05T07:10:44.652 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-02-05T07:10:44.703 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-02-05T07:10:44.762 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-02-05T07:10:44.796 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): libcephsqlite-19.0.0-1241.g164d3be4.el9. 413 kB/s | 163 kB 00:00 2024-02-05T07:10:44.856 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 5/6 2024-02-05T07:10:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 6/6 2024-02-05T07:10:46.863 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): ceph-test-19.0.0-1241.g164d3be4.el9.x86_ 19 MB/s | 47 MB 00:02 2024-02-05T07:10:46.866 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:10:46.866 INFO:teuthology.orchestra.run.smithi090.stdout:Total 20 MB/s | 48 MB 00:02 2024-02-05T07:10:46.867 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:10:46.886 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:10:46.886 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:10:47.266 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:10:47.267 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:10:47.522 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 6/6 2024-02-05T07:10:47.522 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 1/6 2024-02-05T07:10:47.522 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 2/6 2024-02-05T07:10:47.522 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-05T07:10:47.522 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-05T07:10:47.522 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-05T07:10:47.857 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:10:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-05T07:10:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:47.884 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:10:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: jq-1.6-16.el9.x86_64 2024-02-05T07:10:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-05T07:10:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-05T07:10:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-05T07:10:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:47.885 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:10:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-05T07:10:47.988 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-05T07:10:48.040 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-05T07:10:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-05T07:10:48.126 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 5/6 2024-02-05T07:10:48.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph 2024-02-05T07:10:48.719 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:42 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:10:48.885 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:10:48.887 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2024-02-05T07:10:48.887 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-02-05T07:10:48.888 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2024-02-05T07:10:48.888 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:10:48.888 INFO:teuthology.orchestra.run.smithi007.stdout: ceph x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 6.6 k 2024-02-05T07:10:48.888 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-02-05T07:10:48.888 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 2.1 M 2024-02-05T07:10:48.888 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 1.5 M 2024-02-05T07:10:48.888 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 247 k 2024-02-05T07:10:48.888 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 4.7 M 2024-02-05T07:10:48.888 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 17 M 2024-02-05T07:10:48.888 INFO:teuthology.orchestra.run.smithi007.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-05T07:10:48.888 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-05T07:10:48.888 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-05T07:10:48.888 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-05T07:10:48.888 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-05T07:10:48.889 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-05T07:10:48.889 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-05T07:10:48.889 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-05T07:10:48.889 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-05T07:10:48.889 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-05T07:10:48.889 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-05T07:10:48.889 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-05T07:10:48.889 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-05T07:10:48.889 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-05T07:10:48.889 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-05T07:10:48.889 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-05T07:10:48.889 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-05T07:10:48.889 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-05T07:10:48.889 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2024-02-05T07:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout:Install 36 Packages 2024-02-05T07:10:48.891 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:48.891 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 30 M 2024-02-05T07:10:48.891 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 107 M 2024-02-05T07:10:48.891 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:10:49.582 INFO:teuthology.orchestra.run.smithi007.stdout:(1/36): ceph-19.0.0-1241.g164d3be4.el9.x86_64.r 30 kB/s | 6.6 kB 00:00 2024-02-05T07:10:49.891 INFO:teuthology.orchestra.run.smithi007.stdout:(2/36): ceph-mds-19.0.0-1241.g164d3be4.el9.x86_ 4.0 MB/s | 2.1 MB 00:00 2024-02-05T07:10:49.924 INFO:teuthology.orchestra.run.smithi007.stdout:(3/36): ceph-mgr-19.0.0-1241.g164d3be4.el9.x86_ 2.7 MB/s | 1.5 MB 00:00 2024-02-05T07:10:50.041 INFO:teuthology.orchestra.run.smithi007.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1241.g164d 2.1 MB/s | 247 kB 00:00 2024-02-05T07:10:50.200 INFO:teuthology.orchestra.run.smithi007.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-02-05T07:10:50.275 INFO:teuthology.orchestra.run.smithi007.stdout:(6/36): ceph-mon-19.0.0-1241.g164d3be4.el9.x86_ 6.8 MB/s | 4.7 MB 00:00 2024-02-05T07:10:50.300 INFO:teuthology.orchestra.run.smithi007.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2024-02-05T07:10:50.317 INFO:teuthology.orchestra.run.smithi007.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 759 kB/s | 32 kB 00:00 2024-02-05T07:10:50.335 INFO:teuthology.orchestra.run.smithi007.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-02-05T07:10:50.351 INFO:teuthology.orchestra.run.smithi007.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-02-05T07:10:50.385 INFO:teuthology.orchestra.run.smithi007.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-02-05T07:10:50.444 INFO:teuthology.orchestra.run.smithi007.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2024-02-05T07:10:50.469 INFO:teuthology.orchestra.run.smithi007.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-02-05T07:10:50.494 INFO:teuthology.orchestra.run.smithi007.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-02-05T07:10:50.520 INFO:teuthology.orchestra.run.smithi007.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-02-05T07:10:50.545 INFO:teuthology.orchestra.run.smithi007.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-02-05T07:10:50.687 INFO:teuthology.orchestra.run.smithi007.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-02-05T07:10:50.729 INFO:teuthology.orchestra.run.smithi007.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-02-05T07:10:50.796 INFO:teuthology.orchestra.run.smithi007.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-02-05T07:10:50.813 INFO:teuthology.orchestra.run.smithi007.stdout:(20/36): lua-devel-5.4.4-4.el9.x86_64.rpm 76 kB/s | 22 kB 00:00 2024-02-05T07:10:50.863 INFO:teuthology.orchestra.run.smithi007.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.2 MB/s | 358 kB 00:00 2024-02-05T07:10:50.880 INFO:teuthology.orchestra.run.smithi007.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-02-05T07:10:51.048 INFO:teuthology.orchestra.run.smithi007.stdout:(23/36): ceph-osd-19.0.0-1241.g164d3be4.el9.x86 15 MB/s | 17 MB 00:01 2024-02-05T07:10:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 88 kB/s | 18 kB 00:00 2024-02-05T07:10:51.081 INFO:teuthology.orchestra.run.smithi007.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 115 kB/s | 23 kB 00:00 2024-02-05T07:10:51.115 INFO:teuthology.orchestra.run.smithi007.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 593 kB/s | 20 kB 00:00 2024-02-05T07:10:51.140 INFO:teuthology.orchestra.run.smithi007.stdout:(27/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-02-05T07:10:51.157 INFO:teuthology.orchestra.run.smithi007.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-02-05T07:10:51.182 INFO:teuthology.orchestra.run.smithi007.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-02-05T07:10:51.210 INFO:teuthology.orchestra.run.smithi007.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 309 kB/s | 16 kB 00:00 2024-02-05T07:10:51.233 INFO:teuthology.orchestra.run.smithi007.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-02-05T07:10:51.283 INFO:teuthology.orchestra.run.smithi007.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 494 kB/s | 36 kB 00:00 2024-02-05T07:10:51.342 INFO:teuthology.orchestra.run.smithi007.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-02-05T07:10:51.367 INFO:teuthology.orchestra.run.smithi007.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-02-05T07:10:51.392 INFO:teuthology.orchestra.run.smithi007.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 397 kB/s | 20 kB 00:00 2024-02-05T07:10:51.443 INFO:teuthology.orchestra.run.smithi007.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-02-05T07:10:51.446 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:10:51.446 INFO:teuthology.orchestra.run.smithi007.stdout:Total 12 MB/s | 30 MB 00:02 2024-02-05T07:10:51.468 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:10:51.511 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:10:51.511 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:10:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:10:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:10:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 5/6 2024-02-05T07:10:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 6/6 2024-02-05T07:10:53.427 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:10:53.487 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-05T07:10:53.520 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-05T07:10:53.557 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-05T07:10:53.631 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-05T07:10:53.789 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-05T07:10:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-05T07:10:53.918 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-05T07:10:53.996 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-05T07:10:54.042 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-05T07:10:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-05T07:10:54.177 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-05T07:10:54.214 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-05T07:10:54.266 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-05T07:10:54.336 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-05T07:10:54.528 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-05T07:10:54.586 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-05T07:10:54.642 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-05T07:10:54.684 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-05T07:10:54.713 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-05T07:10:54.791 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-05T07:10:54.949 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T07:10:54.982 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-05T07:10:55.023 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-05T07:10:55.058 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-05T07:10:55.100 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-05T07:10:55.193 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-05T07:10:55.231 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-05T07:10:55.307 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-05T07:10:55.455 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-05T07:10:55.579 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T07:10:55.755 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9. 31/36 2024-02-05T07:10:55.786 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 32/36 2024-02-05T07:10:55.811 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 32/36 2024-02-05T07:10:55.811 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:10:55.811 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-05T07:10:55.811 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T07:10:55.811 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T07:10:55.811 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 6/6 2024-02-05T07:10:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 1/6 2024-02-05T07:10:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 2/6 2024-02-05T07:10:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-05T07:10:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-05T07:10:55.860 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-05T07:10:56.285 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-05T07:10:56.285 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:10:56.285 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:10:56.285 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:56.285 INFO:teuthology.orchestra.run.smithi090.stdout: jq-1.6-16.el9.x86_64 2024-02-05T07:10:56.286 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:10:56.286 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-05T07:10:56.286 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-05T07:10:56.286 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-05T07:10:56.286 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:10:56.286 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:10:56.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph 2024-02-05T07:10:57.191 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:52 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:10:57.320 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 33/36 2024-02-05T07:10:57.346 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 33/36 2024-02-05T07:10:57.346 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:10:57.346 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-05T07:10:57.346 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T07:10:57.346 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T07:10:57.347 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:57.360 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:10:57.362 INFO:teuthology.orchestra.run.smithi090.stdout:========================================================================================= 2024-02-05T07:10:57.362 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-05T07:10:57.363 INFO:teuthology.orchestra.run.smithi090.stdout:========================================================================================= 2024-02-05T07:10:57.363 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:10:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 6.6 k 2024-02-05T07:10:57.363 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-05T07:10:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 2.1 M 2024-02-05T07:10:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 1.5 M 2024-02-05T07:10:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 247 k 2024-02-05T07:10:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 4.7 M 2024-02-05T07:10:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 17 M 2024-02-05T07:10:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-05T07:10:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-05T07:10:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-05T07:10:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-05T07:10:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-05T07:10:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-05T07:10:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-05T07:10:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-05T07:10:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-05T07:10:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-05T07:10:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-05T07:10:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-05T07:10:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-05T07:10:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-05T07:10:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-05T07:10:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-05T07:10:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-05T07:10:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-05T07:10:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-05T07:10:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-05T07:10:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-05T07:10:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-05T07:10:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-05T07:10:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-05T07:10:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-05T07:10:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-05T07:10:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-05T07:10:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-05T07:10:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-05T07:10:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-05T07:10:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:10:57.365 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:10:57.365 INFO:teuthology.orchestra.run.smithi090.stdout:========================================================================================= 2024-02-05T07:10:57.366 INFO:teuthology.orchestra.run.smithi090.stdout:Install 36 Packages 2024-02-05T07:10:57.366 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:10:57.366 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 30 M 2024-02-05T07:10:57.366 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 107 M 2024-02-05T07:10:57.366 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:10:57.821 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 34/36 2024-02-05T07:10:57.826 INFO:teuthology.orchestra.run.smithi090.stdout:(1/36): ceph-19.0.0-1241.g164d3be4.el9.x86_64.r 29 kB/s | 6.6 kB 00:00 2024-02-05T07:10:57.844 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 34/36 2024-02-05T07:10:57.844 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:10:57.844 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-05T07:10:57.844 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T07:10:57.844 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T07:10:57.844 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:57.993 INFO:teuthology.orchestra.run.smithi090.stdout:(2/36): ceph-mds-19.0.0-1241.g164d3be4.el9.x86_ 5.4 MB/s | 2.1 MB 00:00 2024-02-05T07:10:58.044 INFO:teuthology.orchestra.run.smithi090.stdout:(3/36): ceph-mgr-19.0.0-1241.g164d3be4.el9.x86_ 3.3 MB/s | 1.5 MB 00:00 2024-02-05T07:10:58.063 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 35/36 2024-02-05T07:10:58.086 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 35/36 2024-02-05T07:10:58.086 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:10:58.086 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-05T07:10:58.086 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T07:10:58.086 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T07:10:58.087 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:10:58.136 INFO:teuthology.orchestra.run.smithi090.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1241.g164d 2.6 MB/s | 247 kB 00:00 2024-02-05T07:10:58.155 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 36/36 2024-02-05T07:10:58.286 INFO:teuthology.orchestra.run.smithi090.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-02-05T07:10:58.337 INFO:teuthology.orchestra.run.smithi090.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-02-05T07:10:58.379 INFO:teuthology.orchestra.run.smithi090.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 760 kB/s | 32 kB 00:00 2024-02-05T07:10:58.421 INFO:teuthology.orchestra.run.smithi090.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-02-05T07:10:58.463 INFO:teuthology.orchestra.run.smithi090.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 926 kB/s | 39 kB 00:00 2024-02-05T07:10:58.523 INFO:teuthology.orchestra.run.smithi090.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.9 MB/s | 241 kB 00:00 2024-02-05T07:10:58.623 INFO:teuthology.orchestra.run.smithi090.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-02-05T07:10:58.824 INFO:teuthology.orchestra.run.smithi090.stdout:(12/36): ceph-osd-19.0.0-1241.g164d3be4.el9.x86 20 MB/s | 17 MB 00:00 2024-02-05T07:10:58.849 INFO:teuthology.orchestra.run.smithi090.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 456 kB/s | 103 kB 00:00 2024-02-05T07:10:58.874 INFO:teuthology.orchestra.run.smithi090.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-02-05T07:10:58.900 INFO:teuthology.orchestra.run.smithi090.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-02-05T07:10:58.933 INFO:teuthology.orchestra.run.smithi090.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-02-05T07:10:59.034 INFO:teuthology.orchestra.run.smithi090.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-02-05T07:10:59.084 INFO:teuthology.orchestra.run.smithi090.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-02-05T07:10:59.143 INFO:teuthology.orchestra.run.smithi090.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 400 kB/s | 43 kB 00:00 2024-02-05T07:10:59.168 INFO:teuthology.orchestra.run.smithi090.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-02-05T07:10:59.210 INFO:teuthology.orchestra.run.smithi090.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 255 kB/s | 11 kB 00:00 2024-02-05T07:10:59.253 INFO:teuthology.orchestra.run.smithi090.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 423 kB/s | 18 kB 00:00 2024-02-05T07:10:59.294 INFO:teuthology.orchestra.run.smithi090.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 557 kB/s | 23 kB 00:00 2024-02-05T07:10:59.328 INFO:teuthology.orchestra.run.smithi090.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.9 MB/s | 358 kB 00:00 2024-02-05T07:10:59.354 INFO:teuthology.orchestra.run.smithi090.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 329 kB/s | 19 kB 00:00 2024-02-05T07:10:59.379 INFO:teuthology.orchestra.run.smithi090.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-02-05T07:10:59.404 INFO:teuthology.orchestra.run.smithi090.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-02-05T07:10:59.438 INFO:teuthology.orchestra.run.smithi090.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-02-05T07:10:59.480 INFO:teuthology.orchestra.run.smithi090.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-02-05T07:10:59.505 INFO:teuthology.orchestra.run.smithi090.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 244 kB/s | 16 kB 00:00 2024-02-05T07:10:59.531 INFO:teuthology.orchestra.run.smithi090.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-02-05T07:10:59.556 INFO:teuthology.orchestra.run.smithi090.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-02-05T07:10:59.641 INFO:teuthology.orchestra.run.smithi090.stdout:(33/36): ceph-mon-19.0.0-1241.g164d3be4.el9.x86 2.6 MB/s | 4.7 MB 00:01 2024-02-05T07:10:59.675 INFO:teuthology.orchestra.run.smithi090.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-02-05T07:10:59.742 INFO:teuthology.orchestra.run.smithi090.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2024-02-05T07:10:59.767 INFO:teuthology.orchestra.run.smithi090.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 159 kB/s | 20 kB 00:00 2024-02-05T07:10:59.771 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:10:59.771 INFO:teuthology.orchestra.run.smithi090.stdout:Total 12 MB/s | 30 MB 00:02 2024-02-05T07:10:59.789 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:10:59.832 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:10:59.832 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:10:59.964 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 36/36 2024-02-05T07:10:59.964 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 1/36 2024-02-05T07:10:59.964 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 2/36 2024-02-05T07:10:59.964 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 3/36 2024-02-05T07:10:59.964 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 4/36 2024-02-05T07:10:59.964 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 5/36 2024-02-05T07:10:59.964 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9. 6/36 2024-02-05T07:10:59.964 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-05T07:10:59.964 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-05T07:10:59.964 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-05T07:10:59.964 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-05T07:10:59.965 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-05T07:10:59.966 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-05T07:10:59.966 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-05T07:10:59.967 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-05T07:10:59.967 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-05T07:10:59.967 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T07:10:59.967 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-05T07:10:59.967 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-05T07:10:59.967 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-05T07:10:59.967 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-05T07:10:59.967 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-05T07:11:00.290 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:11:00.291 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:11:00.794 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-05T07:11:00.794 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: lua-5.4.4-4.el9.x86_64 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-05T07:11:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-05T07:11:00.796 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-05T07:11:00.796 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-05T07:11:00.796 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-05T07:11:00.796 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-05T07:11:00.796 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-05T07:11:00.796 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-05T07:11:00.796 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-05T07:11:00.796 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-05T07:11:00.796 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-05T07:11:00.796 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-05T07:11:00.796 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply-3.11-14.el9.noarch 2024-02-05T07:11:00.796 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-05T07:11:00.796 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-05T07:11:00.797 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-05T07:11:00.797 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-05T07:11:00.797 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-05T07:11:00.797 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-05T07:11:00.797 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-05T07:11:00.797 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-05T07:11:00.797 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-05T07:11:00.797 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-05T07:11:00.797 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-05T07:11:00.797 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:00.797 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:01.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base 2024-02-05T07:11:01.610 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:55 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:01.640 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:11:01.719 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:11:01.720 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-05T07:11:01.761 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-05T07:11:01.774 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:01.775 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-02-05T07:11:01.775 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:01.798 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-05T07:11:01.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install cephadm 2024-02-05T07:11:01.878 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-05T07:11:01.922 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-05T07:11:01.955 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-05T07:11:02.046 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-05T07:11:02.126 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-05T07:11:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-05T07:11:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-05T07:11:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-05T07:11:02.341 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-05T07:11:02.399 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-05T07:11:02.402 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:56 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:02.465 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-05T07:11:02.564 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:02.565 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:02.565 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-02-05T07:11:02.565 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:02.565 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:11:02.565 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 748 k 2024-02-05T07:11:02.565 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:02.565 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:11:02.565 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:02.565 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-02-05T07:11:02.565 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:02.565 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 748 k 2024-02-05T07:11:02.565 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 754 k 2024-02-05T07:11:02.565 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:11:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-05T07:11:02.713 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-05T07:11:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-05T07:11:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-05T07:11:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-05T07:11:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-05T07:11:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T07:11:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-05T07:11:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-05T07:11:03.021 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm-19.0.0-1241.g164d3be4.el9.noarch.rpm 1.6 MB/s | 748 kB 00:00 2024-02-05T07:11:03.021 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:03.022 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.6 MB/s | 748 kB 00:00 2024-02-05T07:11:03.022 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:11:03.027 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:11:03.027 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:11:03.033 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:11:03.034 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:11:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-05T07:11:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-05T07:11:03.110 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:11:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-05T07:11:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-05T07:11:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-05T07:11:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-05T07:11:03.445 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:11:03.528 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:11:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T07:11:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9. 31/36 2024-02-05T07:11:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 32/36 2024-02-05T07:11:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 32/36 2024-02-05T07:11:03.776 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:11:03.776 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-05T07:11:03.776 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T07:11:03.776 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T07:11:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:03.926 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:11:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:11:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:04.232 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:11:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:04.233 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:04.421 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-immutable-object-cache 2024-02-05T07:11:05.002 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:58 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:05.166 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:05.167 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:05.167 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-02-05T07:11:05.167 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:05.167 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:11:05.167 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 145 k 2024-02-05T07:11:05.167 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:05.167 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:11:05.167 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:05.167 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-02-05T07:11:05.167 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:05.167 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 145 k 2024-02-05T07:11:05.167 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 436 k 2024-02-05T07:11:05.167 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:11:05.246 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 33/36 2024-02-05T07:11:05.272 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 33/36 2024-02-05T07:11:05.272 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:11:05.272 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-05T07:11:05.272 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T07:11:05.272 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T07:11:05.272 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:05.403 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-immutable-object-cache-19.0.0-1241.g164d3b 618 kB/s | 145 kB 00:00 2024-02-05T07:11:05.403 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:05.403 INFO:teuthology.orchestra.run.smithi007.stdout:Total 615 kB/s | 145 kB 00:00 2024-02-05T07:11:05.403 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:11:05.411 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:11:05.411 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:11:05.465 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:11:05.466 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:11:05.617 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:11:05.679 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T07:11:05.703 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T07:11:05.703 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:11:05.703 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-05T07:11:05.703 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:05.722 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 34/36 2024-02-05T07:11:05.744 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 34/36 2024-02-05T07:11:05.745 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:11:05.745 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-05T07:11:05.745 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T07:11:05.745 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T07:11:05.745 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:05.956 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 35/36 2024-02-05T07:11:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 35/36 2024-02-05T07:11:05.979 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:11:05.979 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-05T07:11:05.979 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T07:11:05.979 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T07:11:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:06.045 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 36/36 2024-02-05T07:11:06.779 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T07:11:06.779 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:06.780 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:11:06.780 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:06.780 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:06.780 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:07.081 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr 2024-02-05T07:11:07.659 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:01 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:07.767 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:11:07.820 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:07.822 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-02-05T07:11:07.822 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:07.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-dashboard 2024-02-05T07:11:08.296 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 36/36 2024-02-05T07:11:08.297 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 1/36 2024-02-05T07:11:08.297 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 2/36 2024-02-05T07:11:08.297 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 3/36 2024-02-05T07:11:08.297 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 4/36 2024-02-05T07:11:08.297 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 5/36 2024-02-05T07:11:08.297 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9. 6/36 2024-02-05T07:11:08.297 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-05T07:11:08.297 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-05T07:11:08.297 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-05T07:11:08.297 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-05T07:11:08.297 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-05T07:11:08.298 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-05T07:11:08.298 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-05T07:11:08.298 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-05T07:11:08.298 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-05T07:11:08.298 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-05T07:11:08.298 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-05T07:11:08.298 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-05T07:11:08.298 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-05T07:11:08.298 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-05T07:11:08.298 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T07:11:08.298 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-05T07:11:08.298 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-05T07:11:08.298 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-05T07:11:08.299 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-05T07:11:08.299 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-05T07:11:08.299 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-05T07:11:08.299 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-05T07:11:08.299 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-05T07:11:08.299 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T07:11:08.299 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-05T07:11:08.299 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-05T07:11:08.299 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-05T07:11:08.299 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-05T07:11:08.299 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-05T07:11:08.454 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:02 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:08.617 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:08.618 INFO:teuthology.orchestra.run.smithi007.stdout:====================================================================================== 2024-02-05T07:11:08.618 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-02-05T07:11:08.618 INFO:teuthology.orchestra.run.smithi007.stdout:====================================================================================== 2024-02-05T07:11:08.618 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:11:08.618 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 1.8 M 2024-02-05T07:11:08.618 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-02-05T07:11:08.618 INFO:teuthology.orchestra.run.smithi007.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-05T07:11:08.618 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 28 k 2024-02-05T07:11:08.618 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 15 k 2024-02-05T07:11:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-05T07:11:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-05T07:11:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-05T07:11:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-05T07:11:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-05T07:11:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-05T07:11:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-05T07:11:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-05T07:11:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:08.619 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:11:08.619 INFO:teuthology.orchestra.run.smithi007.stdout:====================================================================================== 2024-02-05T07:11:08.619 INFO:teuthology.orchestra.run.smithi007.stdout:Install 12 Packages 2024-02-05T07:11:08.619 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:08.620 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 6.8 M 2024-02-05T07:11:08.620 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 26 M 2024-02-05T07:11:08.620 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:11:09.245 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-05T07:11:09.245 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:09.245 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:11:09.245 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:09.245 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:09.245 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:09.245 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:09.245 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:09.245 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:09.245 INFO:teuthology.orchestra.run.smithi090.stdout: lua-5.4.4-4.el9.x86_64 2024-02-05T07:11:09.246 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-05T07:11:09.246 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-05T07:11:09.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-05T07:11:09.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-05T07:11:09.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-05T07:11:09.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-05T07:11:09.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-05T07:11:09.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-05T07:11:09.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-05T07:11:09.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-05T07:11:09.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-05T07:11:09.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-05T07:11:09.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-05T07:11:09.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-05T07:11:09.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-05T07:11:09.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-05T07:11:09.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-05T07:11:09.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply-3.11-14.el9.noarch 2024-02-05T07:11:09.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-05T07:11:09.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-05T07:11:09.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-05T07:11:09.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-05T07:11:09.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-05T07:11:09.247 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-05T07:11:09.248 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-05T07:11:09.248 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-05T07:11:09.248 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-05T07:11:09.248 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-05T07:11:09.248 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-05T07:11:09.248 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:09.248 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:11:09.340 INFO:teuthology.orchestra.run.smithi007.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1241.g16 114 kB/s | 28 kB 00:00 2024-02-05T07:11:09.366 INFO:teuthology.orchestra.run.smithi007.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1241.g164 56 kB/s | 15 kB 00:00 2024-02-05T07:11:09.541 INFO:teuthology.orchestra.run.smithi007.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-02-05T07:11:09.579 INFO:teuthology.orchestra.run.smithi007.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2024-02-05T07:11:09.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base 2024-02-05T07:11:09.745 INFO:teuthology.orchestra.run.smithi007.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.2 MB/s | 548 kB 00:00 2024-02-05T07:11:09.779 INFO:teuthology.orchestra.run.smithi007.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 585 kB/s | 19 kB 00:00 2024-02-05T07:11:09.871 INFO:teuthology.orchestra.run.smithi007.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 22 MB/s | 2.0 MB 00:00 2024-02-05T07:11:09.905 INFO:teuthology.orchestra.run.smithi007.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.3 MB/s | 863 kB 00:00 2024-02-05T07:11:09.930 INFO:teuthology.orchestra.run.smithi007.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-02-05T07:11:09.955 INFO:teuthology.orchestra.run.smithi007.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 617 kB/s | 31 kB 00:00 2024-02-05T07:11:09.980 INFO:teuthology.orchestra.run.smithi007.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-02-05T07:11:10.064 INFO:teuthology.orchestra.run.smithi007.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1241.g164d3b 1.8 MB/s | 1.8 MB 00:00 2024-02-05T07:11:10.069 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:10.069 INFO:teuthology.orchestra.run.smithi007.stdout:Total 4.7 MB/s | 6.8 MB 00:01 2024-02-05T07:11:10.087 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:11:10.104 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:11:10.104 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:11:10.290 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:11:10.291 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:11:10.296 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:05 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:11:10.409 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:11:10.463 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:11:10.464 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-05T07:11:10.464 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:11:10.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install cephadm 2024-02-05T07:11:10.824 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:11:10.916 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-05T07:11:10.950 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-05T07:11:10.998 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-05T07:11:11.036 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-05T07:11:11.115 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:06 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:11:11.145 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-05T07:11:11.273 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:11:11.273 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:11.273 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-05T07:11:11.273 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:11.273 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:11:11.273 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 748 k 2024-02-05T07:11:11.273 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:11.274 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:11:11.274 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:11.274 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-05T07:11:11.274 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:11.274 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 748 k 2024-02-05T07:11:11.274 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 754 k 2024-02-05T07:11:11.274 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:11:11.292 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-05T07:11:11.386 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-05T07:11:11.442 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-05T07:11:11.469 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-05T07:11:11.500 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9 10/12 2024-02-05T07:11:11.762 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el 11/12 2024-02-05T07:11:11.851 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 12/12 2024-02-05T07:11:11.871 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm-19.0.0-1241.g164d3be4.el9.noarch.rpm 1.2 MB/s | 748 kB 00:00 2024-02-05T07:11:11.871 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:11.871 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.2 MB/s | 748 kB 00:00 2024-02-05T07:11:11.871 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:11:11.876 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:11:11.876 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:11:11.882 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:11:11.882 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:11:11.953 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:11:12.437 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:11:12.527 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:11:12.553 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 12/12 2024-02-05T07:11:12.553 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el 1/12 2024-02-05T07:11:12.553 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 2/12 2024-02-05T07:11:12.553 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9 3/12 2024-02-05T07:11:12.553 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-05T07:11:12.553 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-05T07:11:12.553 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-05T07:11:12.553 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-05T07:11:12.553 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-05T07:11:12.553 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-05T07:11:12.553 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-05T07:11:12.553 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-05T07:11:12.928 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-05T07:11:12.928 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:12.928 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:11:12.928 INFO:teuthology.orchestra.run.smithi007.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-05T07:11:12.928 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:12.928 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:12.928 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:12.929 INFO:teuthology.orchestra.run.smithi007.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-05T07:11:12.929 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-05T07:11:12.929 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-05T07:11:12.929 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-05T07:11:12.929 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-05T07:11:12.929 INFO:teuthology.orchestra.run.smithi007.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-05T07:11:12.929 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-05T07:11:12.929 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-05T07:11:12.929 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:12.929 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:11:13.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-05T07:11:13.313 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:11:13.313 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:13.313 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:11:13.314 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:13.314 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:13.314 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:11:13.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-immutable-object-cache 2024-02-05T07:11:13.737 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:07 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:13.903 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:13.904 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-02-05T07:11:13.904 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-02-05T07:11:13.904 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-02-05T07:11:13.904 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:11:13.904 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 7.4 M 2024-02-05T07:11:13.904 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-02-05T07:11:13.904 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-05T07:11:13.905 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-05T07:11:13.905 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-05T07:11:13.905 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-05T07:11:13.905 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-05T07:11:13.905 INFO:teuthology.orchestra.run.smithi007.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-05T07:11:13.905 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-05T07:11:13.905 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-05T07:11:13.905 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-05T07:11:13.905 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-05T07:11:13.905 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-05T07:11:13.905 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:13.905 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:11:13.905 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-02-05T07:11:13.905 INFO:teuthology.orchestra.run.smithi007.stdout:Install 12 Packages 2024-02-05T07:11:13.906 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:13.906 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 37 M 2024-02-05T07:11:13.906 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 209 M 2024-02-05T07:11:13.906 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:11:14.123 INFO:teuthology.orchestra.run.smithi007.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 201 kB/s | 32 kB 00:00 2024-02-05T07:11:14.144 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:09 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:11:14.165 INFO:teuthology.orchestra.run.smithi007.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 424 kB/s | 17 kB 00:00 2024-02-05T07:11:14.206 INFO:teuthology.orchestra.run.smithi007.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 847 kB/s | 34 kB 00:00 2024-02-05T07:11:14.274 INFO:teuthology.orchestra.run.smithi007.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.7 MB/s | 3.0 MB 00:00 2024-02-05T07:11:14.304 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:11:14.305 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:14.305 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-02-05T07:11:14.305 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:14.305 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:11:14.305 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 145 k 2024-02-05T07:11:14.305 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:14.305 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:11:14.305 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:14.305 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-05T07:11:14.305 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:14.305 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 145 k 2024-02-05T07:11:14.305 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 436 k 2024-02-05T07:11:14.305 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:11:14.491 INFO:teuthology.orchestra.run.smithi007.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-02-05T07:11:14.531 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-immutable-object-cache-19.0.0-1241.g164d3b 643 kB/s | 145 kB 00:00 2024-02-05T07:11:14.532 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:14.532 INFO:teuthology.orchestra.run.smithi090.stdout:Total 639 kB/s | 145 kB 00:00 2024-02-05T07:11:14.532 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:11:14.540 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:11:14.540 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:11:14.566 INFO:teuthology.orchestra.run.smithi007.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-02-05T07:11:14.591 INFO:teuthology.orchestra.run.smithi007.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2024-02-05T07:11:14.592 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:11:14.592 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:11:14.733 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:11:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T07:11:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T07:11:14.815 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:11:14.815 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-05T07:11:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:14.900 INFO:teuthology.orchestra.run.smithi007.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-02-05T07:11:14.925 INFO:teuthology.orchestra.run.smithi007.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 802 kB 00:00 2024-02-05T07:11:14.951 INFO:teuthology.orchestra.run.smithi007.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 194 kB 00:00 2024-02-05T07:11:15.084 INFO:teuthology.orchestra.run.smithi007.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-02-05T07:11:15.718 INFO:teuthology.orchestra.run.smithi007.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.2 MB/s | 7.4 MB 00:01 2024-02-05T07:11:15.721 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:15.721 INFO:teuthology.orchestra.run.smithi007.stdout:Total 20 MB/s | 37 MB 00:01 2024-02-05T07:11:15.734 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:11:15.759 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:11:15.759 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:11:15.907 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T07:11:15.908 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:15.908 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:11:15.908 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:15.908 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:15.908 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:11:16.137 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:11:16.137 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:11:16.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr 2024-02-05T07:11:16.781 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:11 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:11:16.884 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:11:16.936 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:11:16.937 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-05T07:11:16.937 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:11:16.993 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:11:17.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-dashboard 2024-02-05T07:11:17.050 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-05T07:11:17.080 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-05T07:11:17.180 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-05T07:11:17.209 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-05T07:11:17.386 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T07:11:17.416 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T07:11:17.490 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-05T07:11:17.560 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:12 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:11:17.717 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:11:17.720 INFO:teuthology.orchestra.run.smithi090.stdout:====================================================================================== 2024-02-05T07:11:17.720 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-05T07:11:17.720 INFO:teuthology.orchestra.run.smithi090.stdout:====================================================================================== 2024-02-05T07:11:17.720 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:11:17.720 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 1.8 M 2024-02-05T07:11:17.720 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-05T07:11:17.720 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-05T07:11:17.720 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 28 k 2024-02-05T07:11:17.720 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 15 k 2024-02-05T07:11:17.721 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-05T07:11:17.721 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-05T07:11:17.721 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-05T07:11:17.721 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-05T07:11:17.721 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-05T07:11:17.721 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-05T07:11:17.721 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-05T07:11:17.721 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-05T07:11:17.721 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:17.721 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:11:17.721 INFO:teuthology.orchestra.run.smithi090.stdout:====================================================================================== 2024-02-05T07:11:17.721 INFO:teuthology.orchestra.run.smithi090.stdout:Install 12 Packages 2024-02-05T07:11:17.721 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:17.721 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 6.8 M 2024-02-05T07:11:17.722 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 26 M 2024-02-05T07:11:17.722 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:11:18.210 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-05T07:11:18.326 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-05T07:11:18.399 INFO:teuthology.orchestra.run.smithi090.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1241.g16 98 kB/s | 28 kB 00:00 2024-02-05T07:11:18.424 INFO:teuthology.orchestra.run.smithi090.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1241.g164 49 kB/s | 15 kB 00:00 2024-02-05T07:11:18.600 INFO:teuthology.orchestra.run.smithi090.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-02-05T07:11:18.650 INFO:teuthology.orchestra.run.smithi090.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-02-05T07:11:18.834 INFO:teuthology.orchestra.run.smithi090.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1241.g164d3be 2.4 MB/s | 1.8 MB 00:00 2024-02-05T07:11:19.184 INFO:teuthology.orchestra.run.smithi090.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 56 kB/s | 19 kB 00:00 2024-02-05T07:11:19.334 INFO:teuthology.orchestra.run.smithi090.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 801 kB/s | 548 kB 00:00 2024-02-05T07:11:19.443 INFO:teuthology.orchestra.run.smithi090.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-02-05T07:11:19.543 INFO:teuthology.orchestra.run.smithi090.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 308 kB/s | 31 kB 00:00 2024-02-05T07:11:19.645 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-05T07:11:19.660 INFO:teuthology.orchestra.run.smithi090.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-02-05T07:11:19.786 INFO:teuthology.orchestra.run.smithi090.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.4 MB/s | 2.0 MB 00:00 2024-02-05T07:11:20.714 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-05T07:11:20.750 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 12/12 2024-02-05T07:11:21.795 INFO:teuthology.orchestra.run.smithi090.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 270 kB/s | 863 kB 00:03 2024-02-05T07:11:21.800 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:21.801 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.7 MB/s | 6.8 MB 00:04 2024-02-05T07:11:21.820 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:11:21.839 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:11:21.839 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:11:22.022 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:11:22.022 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:11:22.559 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:11:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-05T07:11:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-05T07:11:22.736 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-05T07:11:22.776 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-05T07:11:22.886 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-05T07:11:23.017 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-05T07:11:23.104 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-05T07:11:23.157 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-05T07:11:23.195 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-05T07:11:23.233 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9 10/12 2024-02-05T07:11:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el 11/12 2024-02-05T07:11:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 12/12 2024-02-05T07:11:23.595 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 12/12 2024-02-05T07:11:23.595 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 1/12 2024-02-05T07:11:23.595 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-05T07:11:23.595 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-05T07:11:23.595 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-05T07:11:23.595 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T07:11:23.595 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T07:11:23.595 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-05T07:11:23.595 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-05T07:11:23.596 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-05T07:11:23.596 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-05T07:11:23.596 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-05T07:11:24.034 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-05T07:11:24.034 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:24.034 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:11:24.034 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:24.034 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-05T07:11:24.034 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-05T07:11:24.034 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-05T07:11:24.035 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-05T07:11:24.035 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-05T07:11:24.035 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-05T07:11:24.035 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-05T07:11:24.035 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-05T07:11:24.035 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-05T07:11:24.035 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-05T07:11:24.035 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-05T07:11:24.035 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:24.035 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:24.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-rook 2024-02-05T07:11:24.598 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 12/12 2024-02-05T07:11:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el 1/12 2024-02-05T07:11:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 2/12 2024-02-05T07:11:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9 3/12 2024-02-05T07:11:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-05T07:11:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-05T07:11:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-05T07:11:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-05T07:11:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-05T07:11:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-05T07:11:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-05T07:11:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-05T07:11:24.960 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-05T07:11:24.960 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:24.961 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:11:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-05T07:11:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-05T07:11:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-05T07:11:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-05T07:11:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-05T07:11:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-05T07:11:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-05T07:11:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-05T07:11:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-05T07:11:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:24.962 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:11:25.031 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:19 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:25.194 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:25.195 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-02-05T07:11:25.196 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-02-05T07:11:25.196 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-02-05T07:11:25.196 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:11:25.196 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 52 k 2024-02-05T07:11:25.196 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-02-05T07:11:25.196 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-05T07:11:25.196 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-05T07:11:25.196 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-05T07:11:25.196 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-05T07:11:25.196 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-05T07:11:25.196 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-05T07:11:25.196 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-05T07:11:25.197 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-05T07:11:25.197 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-05T07:11:25.197 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-05T07:11:25.197 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-05T07:11:25.197 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-05T07:11:25.197 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:25.197 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:11:25.197 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-02-05T07:11:25.197 INFO:teuthology.orchestra.run.smithi007.stdout:Install 13 Packages 2024-02-05T07:11:25.197 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:25.198 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.0 M 2024-02-05T07:11:25.198 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 26 M 2024-02-05T07:11:25.198 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:11:25.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-05T07:11:25.434 INFO:teuthology.orchestra.run.smithi007.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 173 kB/s | 24 kB 00:00 2024-02-05T07:11:25.459 INFO:teuthology.orchestra.run.smithi007.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 109 kB/s | 18 kB 00:00 2024-02-05T07:11:25.526 INFO:teuthology.orchestra.run.smithi007.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-02-05T07:11:25.552 INFO:teuthology.orchestra.run.smithi007.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-02-05T07:11:25.585 INFO:teuthology.orchestra.run.smithi007.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-02-05T07:11:25.611 INFO:teuthology.orchestra.run.smithi007.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 735 kB/s | 43 kB 00:00 2024-02-05T07:11:25.636 INFO:teuthology.orchestra.run.smithi007.stdout:(7/13): ceph-mgr-rook-19.0.0-1241.g164d3be4.el9 152 kB/s | 52 kB 00:00 2024-02-05T07:11:25.818 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:20 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:11:25.853 INFO:teuthology.orchestra.run.smithi007.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-02-05T07:11:25.878 INFO:teuthology.orchestra.run.smithi007.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-02-05T07:11:25.974 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:11:25.975 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-02-05T07:11:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-05T07:11:25.975 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-02-05T07:11:25.975 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:11:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 7.4 M 2024-02-05T07:11:25.976 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-05T07:11:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-05T07:11:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-05T07:11:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-05T07:11:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-05T07:11:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-05T07:11:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-05T07:11:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-05T07:11:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-05T07:11:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-05T07:11:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-05T07:11:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-05T07:11:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:25.977 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:11:25.977 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-02-05T07:11:25.977 INFO:teuthology.orchestra.run.smithi090.stdout:Install 12 Packages 2024-02-05T07:11:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:25.977 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 37 M 2024-02-05T07:11:25.977 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 209 M 2024-02-05T07:11:25.978 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:11:26.095 INFO:teuthology.orchestra.run.smithi007.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 442 kB/s | 203 kB 00:00 2024-02-05T07:11:26.112 INFO:teuthology.orchestra.run.smithi007.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 253 kB/s | 59 kB 00:00 2024-02-05T07:11:26.212 INFO:teuthology.orchestra.run.smithi007.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 768 kB/s | 90 kB 00:00 2024-02-05T07:11:26.250 INFO:teuthology.orchestra.run.smithi090.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 166 kB/s | 32 kB 00:00 2024-02-05T07:11:26.271 INFO:teuthology.orchestra.run.smithi007.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.4 MB/s | 1.0 MB 00:00 2024-02-05T07:11:26.273 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:26.273 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-02-05T07:11:26.274 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:11:26.290 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:11:26.290 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:11:26.292 INFO:teuthology.orchestra.run.smithi090.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 425 kB/s | 17 kB 00:00 2024-02-05T07:11:26.343 INFO:teuthology.orchestra.run.smithi090.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 694 kB/s | 34 kB 00:00 2024-02-05T07:11:26.404 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:11:26.404 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:11:26.410 INFO:teuthology.orchestra.run.smithi090.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.5 MB/s | 3.0 MB 00:00 2024-02-05T07:11:26.644 INFO:teuthology.orchestra.run.smithi090.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2024-02-05T07:11:26.719 INFO:teuthology.orchestra.run.smithi090.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-02-05T07:11:26.753 INFO:teuthology.orchestra.run.smithi090.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-02-05T07:11:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:11:26.961 INFO:teuthology.orchestra.run.smithi090.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 802 kB 00:00 2024-02-05T07:11:27.100 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-05T07:11:27.145 INFO:teuthology.orchestra.run.smithi090.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-02-05T07:11:27.173 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-05T07:11:27.179 INFO:teuthology.orchestra.run.smithi090.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 901 kB/s | 194 kB 00:00 2024-02-05T07:11:27.238 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-05T07:11:27.274 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-05T07:11:27.315 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-05T07:11:27.387 INFO:teuthology.orchestra.run.smithi090.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-02-05T07:11:27.412 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-05T07:11:27.521 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-05T07:11:27.576 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-05T07:11:27.746 INFO:teuthology.orchestra.run.smithi090.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.4 MB/s | 7.4 MB 00:01 2024-02-05T07:11:27.748 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:27.748 INFO:teuthology.orchestra.run.smithi090.stdout:Total 21 MB/s | 37 MB 00:01 2024-02-05T07:11:27.761 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:11:27.785 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:11:27.785 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:11:28.141 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:11:28.142 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:11:28.345 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-05T07:11:28.398 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-05T07:11:28.431 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-05T07:11:28.481 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-05T07:11:28.510 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 13/13 2024-02-05T07:11:28.841 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:11:28.927 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-05T07:11:28.959 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-05T07:11:29.054 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-05T07:11:29.096 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-05T07:11:29.273 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T07:11:29.313 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T07:11:29.326 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 13/13 2024-02-05T07:11:29.326 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 1/13 2024-02-05T07:11:29.326 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-05T07:11:29.326 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-05T07:11:29.326 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-05T07:11:29.327 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-05T07:11:29.327 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-05T07:11:29.327 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-05T07:11:29.327 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-05T07:11:29.327 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-05T07:11:29.327 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-05T07:11:29.327 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-05T07:11:29.327 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-05T07:11:29.438 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-05T07:11:29.703 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-05T07:11:29.703 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:29.703 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:11:29.703 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:29.703 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-05T07:11:29.704 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-05T07:11:29.704 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-05T07:11:29.704 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-05T07:11:29.704 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-05T07:11:29.704 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-05T07:11:29.704 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-05T07:11:29.704 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-05T07:11:29.704 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-05T07:11:29.704 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-05T07:11:29.704 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-05T07:11:29.704 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-05T07:11:29.704 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:29.704 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:29.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-cephadm 2024-02-05T07:11:30.094 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-05T07:11:30.218 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-05T07:11:30.522 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:24 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:30.684 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:30.685 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-02-05T07:11:30.685 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-02-05T07:11:30.685 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-02-05T07:11:30.685 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:11:30.685 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 138 k 2024-02-05T07:11:30.685 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-02-05T07:11:30.685 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-05T07:11:30.685 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-05T07:11:30.685 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-02-05T07:11:30.685 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-05T07:11:30.686 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-05T07:11:30.686 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:30.686 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:11:30.686 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-02-05T07:11:30.686 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2024-02-05T07:11:30.686 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 6.8 M 2024-02-05T07:11:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 33 M 2024-02-05T07:11:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:11:30.933 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.1 MB/s | 228 kB 00:00 2024-02-05T07:11:31.017 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1241.g164d3be4.e 482 kB/s | 138 kB 00:00 2024-02-05T07:11:31.117 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-02-05T07:11:31.184 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2024-02-05T07:11:31.201 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 318 kB/s | 58 kB 00:00 2024-02-05T07:11:31.217 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 509 kB/s | 51 kB 00:00 2024-02-05T07:11:31.219 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:31.220 INFO:teuthology.orchestra.run.smithi007.stdout:Total 13 MB/s | 6.8 MB 00:00 2024-02-05T07:11:31.220 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:11:31.229 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:11:31.229 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:11:31.324 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:11:31.324 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:11:31.451 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-05T07:11:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:11:31.780 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-05T07:11:31.846 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-05T07:11:32.406 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-05T07:11:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-05T07:11:32.500 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-05T07:11:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 12/12 2024-02-05T07:11:32.578 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-02-05T07:11:32.607 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 6/6 2024-02-05T07:11:33.460 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 6/6 2024-02-05T07:11:33.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/6 2024-02-05T07:11:33.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-05T07:11:33.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-02-05T07:11:33.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-05T07:11:33.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-05T07:11:33.776 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-05T07:11:33.776 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:33.776 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:11:33.776 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:33.776 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-05T07:11:33.776 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-05T07:11:33.776 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-02-05T07:11:33.776 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-05T07:11:33.776 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-05T07:11:33.776 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:33.777 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:34.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-fuse 2024-02-05T07:11:34.612 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:28 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:34.777 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:34.778 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:34.778 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-02-05T07:11:34.778 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:34.778 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:11:34.778 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 885 k 2024-02-05T07:11:34.778 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-02-05T07:11:34.778 INFO:teuthology.orchestra.run.smithi007.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-05T07:11:34.778 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:34.779 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:11:34.779 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:34.779 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-02-05T07:11:34.779 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:34.779 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 964 k 2024-02-05T07:11:34.779 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.8 M 2024-02-05T07:11:34.779 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:11:34.920 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 561 kB/s | 79 kB 00:00 2024-02-05T07:11:35.120 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-fuse-19.0.0-1241.g164d3be4.el9.x86_ 2.5 MB/s | 885 kB 00:00 2024-02-05T07:11:35.123 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:35.124 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.7 MB/s | 964 kB 00:00 2024-02-05T07:11:35.124 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:11:35.143 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:11:35.143 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:11:35.222 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:11:35.223 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:11:35.424 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:11:35.567 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-05T07:11:35.648 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2/2 2024-02-05T07:11:35.830 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 12/12 2024-02-05T07:11:35.831 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 1/12 2024-02-05T07:11:35.831 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-05T07:11:35.831 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-05T07:11:35.831 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-05T07:11:35.831 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T07:11:35.831 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T07:11:35.831 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-05T07:11:35.831 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-05T07:11:35.831 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-05T07:11:35.831 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-05T07:11:35.831 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-05T07:11:36.222 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-05T07:11:36.222 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:36.222 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:11:36.222 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:36.222 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-05T07:11:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-05T07:11:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-05T07:11:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-05T07:11:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-05T07:11:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-05T07:11:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-05T07:11:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-05T07:11:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-05T07:11:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-05T07:11:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-05T07:11:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:36.223 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:11:36.403 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2/2 2024-02-05T07:11:36.403 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/2 2024-02-05T07:11:36.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-rook 2024-02-05T07:11:36.726 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-05T07:11:36.727 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:36.727 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:11:36.727 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-05T07:11:36.727 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:36.727 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:36.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-volume 2024-02-05T07:11:37.227 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:32 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:11:37.387 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:11:37.388 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-02-05T07:11:37.388 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-05T07:11:37.388 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-02-05T07:11:37.388 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:11:37.388 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 52 k 2024-02-05T07:11:37.389 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-05T07:11:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-05T07:11:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-05T07:11:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-05T07:11:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-05T07:11:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-05T07:11:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-05T07:11:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-05T07:11:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-05T07:11:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-05T07:11:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-05T07:11:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-05T07:11:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-05T07:11:37.390 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:37.390 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:11:37.390 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-02-05T07:11:37.390 INFO:teuthology.orchestra.run.smithi090.stdout:Install 13 Packages 2024-02-05T07:11:37.390 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:37.390 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.0 M 2024-02-05T07:11:37.390 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 26 M 2024-02-05T07:11:37.390 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:11:37.559 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:31 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:37.722 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:37.723 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:37.723 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-02-05T07:11:37.723 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:37.723 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:11:37.723 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 263 k 2024-02-05T07:11:37.724 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-02-05T07:11:37.724 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-05T07:11:37.724 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:37.724 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:11:37.724 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:37.724 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-02-05T07:11:37.724 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:37.724 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 332 k 2024-02-05T07:11:37.725 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.5 M 2024-02-05T07:11:37.725 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:11:37.861 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 504 kB/s | 69 kB 00:00 2024-02-05T07:11:37.910 INFO:teuthology.orchestra.run.smithi090.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 135 kB/s | 24 kB 00:00 2024-02-05T07:11:37.935 INFO:teuthology.orchestra.run.smithi090.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 87 kB/s | 18 kB 00:00 2024-02-05T07:11:37.986 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-volume-19.0.0-1241.g164d3be4.el9.no 1.0 MB/s | 263 kB 00:00 2024-02-05T07:11:37.987 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:37.987 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-02-05T07:11:37.988 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:11:37.997 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:11:37.997 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:11:38.002 INFO:teuthology.orchestra.run.smithi090.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-02-05T07:11:38.028 INFO:teuthology.orchestra.run.smithi090.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-02-05T07:11:38.043 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:11:38.043 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:11:38.053 INFO:teuthology.orchestra.run.smithi090.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-05T07:11:38.070 INFO:teuthology.orchestra.run.smithi090.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-02-05T07:11:38.087 INFO:teuthology.orchestra.run.smithi090.stdout:(7/13): ceph-mgr-rook-19.0.0-1241.g164d3be4.el9 146 kB/s | 52 kB 00:00 2024-02-05T07:11:38.203 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:11:38.386 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-05T07:11:38.422 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2/2 2024-02-05T07:11:38.437 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2/2 2024-02-05T07:11:38.438 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:11:38.438 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-05T07:11:38.438 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:38.521 INFO:teuthology.orchestra.run.smithi090.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 69 kB/s | 32 kB 00:00 2024-02-05T07:11:38.555 INFO:teuthology.orchestra.run.smithi090.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 31 kB/s | 15 kB 00:00 2024-02-05T07:11:38.738 INFO:teuthology.orchestra.run.smithi090.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-02-05T07:11:38.855 INFO:teuthology.orchestra.run.smithi090.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 264 kB/s | 203 kB 00:00 2024-02-05T07:11:38.922 INFO:teuthology.orchestra.run.smithi090.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 488 kB/s | 90 kB 00:00 2024-02-05T07:11:39.156 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 1/2 2024-02-05T07:11:39.550 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-05T07:11:39.550 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:39.550 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:11:39.550 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:39.550 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-05T07:11:39.550 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:39.550 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:39.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados-devel 2024-02-05T07:11:40.468 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:34 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:40.630 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:40.631 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:40.631 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-02-05T07:11:40.631 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:40.631 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:11:40.631 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 127 k 2024-02-05T07:11:40.631 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:40.631 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:11:40.631 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:40.631 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-02-05T07:11:40.631 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:40.631 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 127 k 2024-02-05T07:11:40.631 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 456 k 2024-02-05T07:11:40.631 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:11:40.831 INFO:teuthology.orchestra.run.smithi090.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 453 kB/s | 1.0 MB 00:02 2024-02-05T07:11:40.837 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:40.837 INFO:teuthology.orchestra.run.smithi090.stdout:Total 609 kB/s | 2.0 MB 00:03 2024-02-05T07:11:40.837 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:11:40.853 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:11:40.853 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:11:40.961 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:11:40.962 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:11:40.975 INFO:teuthology.orchestra.run.smithi007.stdout:librados-devel-19.0.0-1241.g164d3be4.el9.x86_64 369 kB/s | 127 kB 00:00 2024-02-05T07:11:40.976 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:40.976 INFO:teuthology.orchestra.run.smithi007.stdout:Total 367 kB/s | 127 kB 00:00 2024-02-05T07:11:40.976 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:11:40.982 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:11:40.982 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:11:41.041 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:11:41.041 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:11:41.205 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:11:41.310 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:41.486 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:11:41.655 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-05T07:11:41.717 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-05T07:11:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-05T07:11:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-05T07:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:41.868 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-05T07:11:41.971 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-05T07:11:42.089 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-05T07:11:42.118 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:42.118 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:42.118 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:11:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:42.119 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:42.145 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-05T07:11:42.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs2 2024-02-05T07:11:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-05T07:11:42.920 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-05T07:11:42.938 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:36 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:42.954 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-05T07:11:43.002 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-05T07:11:43.032 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 13/13 2024-02-05T07:11:43.049 INFO:teuthology.orchestra.run.smithi007.stdout:Package libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:11:43.106 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:43.107 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-02-05T07:11:43.107 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:43.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs-devel 2024-02-05T07:11:43.758 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:37 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:43.920 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:43.921 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:43.921 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-02-05T07:11:43.921 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:43.921 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:11:43.921 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 31 k 2024-02-05T07:11:43.921 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:43.921 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:11:43.921 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:43.922 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-02-05T07:11:43.922 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:43.922 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 31 k 2024-02-05T07:11:43.922 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 140 k 2024-02-05T07:11:43.922 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:11:44.071 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 13/13 2024-02-05T07:11:44.072 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 1/13 2024-02-05T07:11:44.072 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-05T07:11:44.072 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-05T07:11:44.072 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-05T07:11:44.072 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-05T07:11:44.072 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-05T07:11:44.072 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-05T07:11:44.072 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-05T07:11:44.072 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-05T07:11:44.072 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-05T07:11:44.072 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-05T07:11:44.072 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-05T07:11:44.163 INFO:teuthology.orchestra.run.smithi007.stdout:libcephfs-devel-19.0.0-1241.g164d3be4.el9.x86_6 129 kB/s | 31 kB 00:00 2024-02-05T07:11:44.164 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:44.164 INFO:teuthology.orchestra.run.smithi007.stdout:Total 128 kB/s | 31 kB 00:00 2024-02-05T07:11:44.164 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:11:44.168 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:11:44.168 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:11:44.192 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:11:44.193 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:11:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:11:44.390 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:44.456 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-05T07:11:44.456 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:44.456 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:11:44.456 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:44.456 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-05T07:11:44.456 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-05T07:11:44.456 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-05T07:11:44.456 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-05T07:11:44.456 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-05T07:11:44.456 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-05T07:11:44.456 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-05T07:11:44.456 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-05T07:11:44.456 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-05T07:11:44.457 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-05T07:11:44.457 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-05T07:11:44.457 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-05T07:11:44.457 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:44.457 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:11:44.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-cephadm 2024-02-05T07:11:44.810 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:45.077 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:45.077 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:45.077 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:11:45.077 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:45.077 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:45.077 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:45.269 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados2 2024-02-05T07:11:45.304 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:40 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:11:45.460 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:11:45.461 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-02-05T07:11:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-05T07:11:45.461 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-02-05T07:11:45.461 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:11:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 138 k 2024-02-05T07:11:45.461 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-05T07:11:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-05T07:11:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-05T07:11:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-02-05T07:11:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-05T07:11:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-05T07:11:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:45.462 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:11:45.462 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-02-05T07:11:45.462 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-02-05T07:11:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:45.463 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 6.8 M 2024-02-05T07:11:45.463 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 33 M 2024-02-05T07:11:45.463 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:11:45.742 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 967 kB/s | 228 kB 00:00 2024-02-05T07:11:45.843 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:39 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:45.926 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-02-05T07:11:45.951 INFO:teuthology.orchestra.run.smithi007.stdout:Package librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:11:45.993 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1241.g164d3be4.e 283 kB/s | 138 kB 00:00 2024-02-05T07:11:46.007 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:46.008 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-02-05T07:11:46.008 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:46.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librbd1 2024-02-05T07:11:46.076 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-02-05T07:11:46.101 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 329 kB/s | 58 kB 00:00 2024-02-05T07:11:46.135 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 358 kB/s | 51 kB 00:00 2024-02-05T07:11:46.137 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:46.137 INFO:teuthology.orchestra.run.smithi090.stdout:Total 10 MB/s | 6.8 MB 00:00 2024-02-05T07:11:46.137 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:11:46.147 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:11:46.147 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:11:46.236 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:11:46.236 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:11:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:11:46.644 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:40 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:46.709 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-05T07:11:46.753 INFO:teuthology.orchestra.run.smithi007.stdout:Package librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:11:46.764 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-05T07:11:46.808 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:46.809 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-02-05T07:11:46.809 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:46.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rados 2024-02-05T07:11:47.283 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-05T07:11:47.392 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-05T07:11:47.456 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:41 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:47.462 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-02-05T07:11:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 6/6 2024-02-05T07:11:47.563 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:11:47.618 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:47.620 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-02-05T07:11:47.620 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:47.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rgw 2024-02-05T07:11:48.258 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:42 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:48.365 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:11:48.420 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:48.421 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-02-05T07:11:48.421 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:48.485 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 6/6 2024-02-05T07:11:48.485 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/6 2024-02-05T07:11:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-05T07:11:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-02-05T07:11:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-05T07:11:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-05T07:11:48.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-cephfs 2024-02-05T07:11:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-05T07:11:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:48.797 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:11:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-05T07:11:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-05T07:11:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-02-05T07:11:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-05T07:11:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-05T07:11:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:48.798 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:11:49.063 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:43 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:49.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-fuse 2024-02-05T07:11:49.169 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:11:49.225 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:49.226 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-02-05T07:11:49.226 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:49.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rbd 2024-02-05T07:11:49.645 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:44 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:11:49.805 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:11:49.805 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:49.805 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-05T07:11:49.805 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:49.805 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:11:49.805 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 885 k 2024-02-05T07:11:49.805 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-05T07:11:49.805 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-05T07:11:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:49.806 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:11:49.806 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:49.806 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2024-02-05T07:11:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:49.806 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 964 k 2024-02-05T07:11:49.806 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 2.8 M 2024-02-05T07:11:49.807 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:11:49.883 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:43 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:49.967 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 490 kB/s | 79 kB 00:00 2024-02-05T07:11:49.991 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:11:50.047 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:50.048 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-02-05T07:11:50.048 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:50.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-fuse 2024-02-05T07:11:50.325 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): ceph-fuse-19.0.0-1241.g164d3be4.el9.x86_ 1.7 MB/s | 885 kB 00:00 2024-02-05T07:11:50.327 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:50.327 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.8 MB/s | 964 kB 00:00 2024-02-05T07:11:50.327 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:11:50.347 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:11:50.347 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:11:50.424 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:11:50.424 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:11:50.618 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:11:50.713 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:44 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:50.768 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-05T07:11:50.851 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2/2 2024-02-05T07:11:50.879 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:50.880 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:50.880 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-02-05T07:11:50.880 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:50.880 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:11:50.880 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 85 k 2024-02-05T07:11:50.880 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:50.880 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:11:50.880 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:50.881 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-02-05T07:11:50.881 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:50.881 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 85 k 2024-02-05T07:11:50.881 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 231 k 2024-02-05T07:11:50.881 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:11:51.083 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-fuse-19.0.0-1241.g164d3be4.el9.x86_64.rpm 421 kB/s | 85 kB 00:00 2024-02-05T07:11:51.084 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:51.084 INFO:teuthology.orchestra.run.smithi007.stdout:Total 418 kB/s | 85 kB 00:00 2024-02-05T07:11:51.084 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:11:51.092 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:11:51.092 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:11:51.150 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:11:51.150 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:11:51.289 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:11:51.393 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2/2 2024-02-05T07:11:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/2 2024-02-05T07:11:51.818 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-05T07:11:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:52.071 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:11:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-05T07:11:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:52.071 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:11:52.231 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:52.232 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:52.232 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:11:52.232 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:52.232 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:52.232 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:52.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-volume 2024-02-05T07:11:52.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-mirror 2024-02-05T07:11:52.988 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:47 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:11:53.016 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:47 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:53.144 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:11:53.144 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:53.145 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-05T07:11:53.145 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:53.145 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:11:53.145 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 263 k 2024-02-05T07:11:53.145 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-05T07:11:53.145 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-05T07:11:53.145 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:53.145 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:11:53.145 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:53.145 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2024-02-05T07:11:53.145 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:53.146 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 332 k 2024-02-05T07:11:53.146 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 1.5 M 2024-02-05T07:11:53.146 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:11:53.180 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:53.181 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:53.181 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-02-05T07:11:53.181 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:53.181 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:11:53.181 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.1 M 2024-02-05T07:11:53.181 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:53.181 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:11:53.181 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:53.181 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-02-05T07:11:53.182 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:53.182 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 3.1 M 2024-02-05T07:11:53.182 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 13 M 2024-02-05T07:11:53.182 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:11:53.286 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 492 kB/s | 69 kB 00:00 2024-02-05T07:11:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): ceph-volume-19.0.0-1241.g164d3be4.el9.no 690 kB/s | 263 kB 00:00 2024-02-05T07:11:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:Total 864 kB/s | 332 kB 00:00 2024-02-05T07:11:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:11:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:11:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:11:53.581 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:11:53.581 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:11:53.735 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:11:53.770 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-mirror-19.0.0-1241.g164d3be4.el9.x86_64.rpm 5.2 MB/s | 3.1 MB 00:00 2024-02-05T07:11:53.771 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:53.771 INFO:teuthology.orchestra.run.smithi007.stdout:Total 5.2 MB/s | 3.1 MB 00:00 2024-02-05T07:11:53.772 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:11:53.781 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:11:53.782 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:11:53.859 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:11:53.860 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:11:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-05T07:11:53.956 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2/2 2024-02-05T07:11:53.971 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2/2 2024-02-05T07:11:53.972 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:11:53.972 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-05T07:11:53.972 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:54.297 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:11:54.363 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:54.387 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:54.387 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:11:54.387 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-05T07:11:54.387 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T07:11:54.387 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T07:11:54.387 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:54.812 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 1/2 2024-02-05T07:11:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-05T07:11:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:55.189 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:11:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:11:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-05T07:11:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:55.190 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:11:55.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados-devel 2024-02-05T07:11:55.664 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:55.664 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:55.664 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:11:55.664 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:55.664 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:55.664 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:55.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-nbd 2024-02-05T07:11:56.111 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:51 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:11:56.267 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:11:56.267 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:56.267 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-02-05T07:11:56.268 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:56.268 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:11:56.268 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 127 k 2024-02-05T07:11:56.268 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:56.268 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:11:56.268 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:56.268 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-05T07:11:56.268 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:56.268 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 127 k 2024-02-05T07:11:56.268 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 456 k 2024-02-05T07:11:56.268 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:11:56.487 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:50 ago on Mon 05 Feb 2024 07:10:06 AM UTC. 2024-02-05T07:11:56.498 INFO:teuthology.orchestra.run.smithi090.stdout:librados-devel-19.0.0-1241.g164d3be4.el9.x86_64 552 kB/s | 127 kB 00:00 2024-02-05T07:11:56.498 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:56.499 INFO:teuthology.orchestra.run.smithi090.stdout:Total 549 kB/s | 127 kB 00:00 2024-02-05T07:11:56.499 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:11:56.504 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:11:56.504 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:11:56.560 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:11:56.561 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:11:56.651 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-02-05T07:11:56.652 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-02-05T07:11:56.652 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:56.652 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-02-05T07:11:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 171 k 2024-02-05T07:11:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:56.652 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-02-05T07:11:56.652 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-02-05T07:11:56.653 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-02-05T07:11:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:56.653 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 171 k 2024-02-05T07:11:56.653 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 494 k 2024-02-05T07:11:56.653 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-02-05T07:11:56.702 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:11:56.814 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:56.875 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-nbd-19.0.0-1241.g164d3be4.el9.x86_64.rpm 773 kB/s | 171 kB 00:00 2024-02-05T07:11:56.875 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:56.875 INFO:teuthology.orchestra.run.smithi007.stdout:Total 768 kB/s | 171 kB 00:00 2024-02-05T07:11:56.875 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-02-05T07:11:56.884 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-02-05T07:11:56.884 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-02-05T07:11:56.942 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-02-05T07:11:56.943 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-02-05T07:11:57.137 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-02-05T07:11:57.242 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:57.383 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:57.707 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:57.736 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:57.737 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:11:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:57.737 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:11:58.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs2 2024-02-05T07:11:58.112 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:11:58.112 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:58.112 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-02-05T07:11:58.112 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:11:58.112 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:11:58.112 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-02-05T07:11:58.341 DEBUG:teuthology.parallel:result is None 2024-02-05T07:11:58.610 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:53 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:11:58.717 INFO:teuthology.orchestra.run.smithi090.stdout:Package libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:11:58.770 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:11:58.771 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-05T07:11:58.771 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:11:58.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs-devel 2024-02-05T07:11:59.414 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:54 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:11:59.570 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:11:59.570 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-02-05T07:11:59.571 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:59.571 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:11:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 31 k 2024-02-05T07:11:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:59.571 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:11:59.571 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:11:59.571 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-05T07:11:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:11:59.572 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 31 k 2024-02-05T07:11:59.572 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 140 k 2024-02-05T07:11:59.572 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:11:59.757 INFO:teuthology.orchestra.run.smithi090.stdout:libcephfs-devel-19.0.0-1241.g164d3be4.el9.x86_6 168 kB/s | 31 kB 00:00 2024-02-05T07:11:59.757 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:11:59.758 INFO:teuthology.orchestra.run.smithi090.stdout:Total 167 kB/s | 31 kB 00:00 2024-02-05T07:11:59.758 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:11:59.761 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:11:59.761 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:11:59.792 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:11:59.793 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:11:59.887 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:12:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:12:00.520 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:12:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:12:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:12:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:12:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:12:01.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados2 2024-02-05T07:12:01.601 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:56 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:12:01.707 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:12:01.758 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:12:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-05T07:12:01.760 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:12:01.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librbd1 2024-02-05T07:12:02.407 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:57 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:12:02.512 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:12:02.567 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:12:02.567 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-05T07:12:02.568 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:12:02.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rados 2024-02-05T07:12:03.219 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:58 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:12:03.322 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:12:03.374 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:12:03.375 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-05T07:12:03.375 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:12:03.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rgw 2024-02-05T07:12:04.017 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:59 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:12:04.120 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:12:04.175 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:12:04.175 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-05T07:12:04.175 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:12:04.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-cephfs 2024-02-05T07:12:04.838 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:59 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:12:04.941 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:12:04.993 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:12:04.994 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-05T07:12:04.995 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:12:05.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rbd 2024-02-05T07:12:05.661 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:00 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:12:05.764 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:12:05.816 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:12:05.817 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-05T07:12:05.817 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:12:05.887 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-fuse 2024-02-05T07:12:06.453 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:01 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:12:06.610 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:12:06.611 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:12:06.611 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-05T07:12:06.611 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:12:06.611 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:12:06.612 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 85 k 2024-02-05T07:12:06.612 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:06.612 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:12:06.612 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:12:06.612 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-05T07:12:06.612 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:06.612 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 85 k 2024-02-05T07:12:06.612 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 231 k 2024-02-05T07:12:06.612 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:12:06.830 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-fuse-19.0.0-1241.g164d3be4.el9.x86_64.rpm 392 kB/s | 85 kB 00:00 2024-02-05T07:12:06.830 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:12:06.830 INFO:teuthology.orchestra.run.smithi090.stdout:Total 390 kB/s | 85 kB 00:00 2024-02-05T07:12:06.831 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:12:06.837 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:12:06.837 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:12:06.892 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:12:06.892 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:12:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:12:07.144 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:12:07.643 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:12:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:12:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:07.903 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:12:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:12:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:07.903 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:12:08.263 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-mirror 2024-02-05T07:12:08.842 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:03 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:12:09.000 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-02-05T07:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.1 M 2024-02-05T07:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-05T07:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 3.1 M 2024-02-05T07:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 13 M 2024-02-05T07:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:12:09.445 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-mirror-19.0.0-1241.g164d3be4.el9.x86_64.rpm 7.0 MB/s | 3.1 MB 00:00 2024-02-05T07:12:09.446 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:12:09.446 INFO:teuthology.orchestra.run.smithi090.stdout:Total 6.9 MB/s | 3.1 MB 00:00 2024-02-05T07:12:09.446 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:12:09.456 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:12:09.456 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:12:09.531 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:12:09.531 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:12:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:12:10.005 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:12:10.028 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:12:10.028 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:12:10.028 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-05T07:12:10.028 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T07:12:10.028 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T07:12:10.029 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:11.243 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:12:11.244 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:11.244 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:12:11.244 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:12:11.244 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:11.244 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:12:11.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-nbd 2024-02-05T07:12:12.135 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:07 ago on Mon 05 Feb 2024 07:10:05 AM UTC. 2024-02-05T07:12:12.291 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-05T07:12:12.292 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:12:12.292 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-05T07:12:12.292 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:12:12.292 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-05T07:12:12.292 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 171 k 2024-02-05T07:12:12.292 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:12.292 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-05T07:12:12.293 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-05T07:12:12.293 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-05T07:12:12.293 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:12.293 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 171 k 2024-02-05T07:12:12.293 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 494 k 2024-02-05T07:12:12.293 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-05T07:12:12.637 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-nbd-19.0.0-1241.g164d3be4.el9.x86_64.rpm 495 kB/s | 171 kB 00:00 2024-02-05T07:12:12.638 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:12:12.638 INFO:teuthology.orchestra.run.smithi090.stdout:Total 494 kB/s | 171 kB 00:00 2024-02-05T07:12:12.638 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-05T07:12:12.646 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-05T07:12:12.646 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-05T07:12:12.701 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-05T07:12:12.701 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-05T07:12:12.896 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-05T07:12:13.002 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:12:13.592 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:12:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:12:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:13.922 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-05T07:12:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:12:13.923 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:12:13.923 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-05T07:12:14.229 DEBUG:teuthology.parallel:result is None 2024-02-05T07:12:14.230 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:12:14.371 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-05T07:12:14.396 INFO:teuthology.orchestra.run.smithi007.stdout:19.0.0-1241.g164d3be4.el9 2024-02-05T07:12:14.396 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1241.g164d3be4.el9 2024-02-05T07:12:14.396 INFO:teuthology.task.install:The correct ceph version 19.0.0-1241.g164d3be4 is installed. 2024-02-05T07:12:14.398 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:12:14.536 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-05T07:12:14.557 INFO:teuthology.orchestra.run.smithi090.stdout:19.0.0-1241.g164d3be4.el9 2024-02-05T07:12:14.557 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1241.g164d3be4.el9 2024-02-05T07:12:14.557 INFO:teuthology.task.install:The correct ceph version 19.0.0-1241.g164d3be4 is installed. 2024-02-05T07:12:14.559 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-05T07:12:14.559 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:12:14.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-05T07:12:14.591 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-05T07:12:14.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-05T07:12:14.625 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-05T07:12:14.625 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:12:14.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2024-02-05T07:12:14.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-05T07:12:14.732 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-05T07:12:14.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-02-05T07:12:14.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-05T07:12:14.825 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-05T07:12:14.825 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:12:14.825 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-05T07:12:14.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-05T07:12:14.923 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-05T07:12:14.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-05T07:12:14.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-05T07:12:15.017 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-05T07:12:15.017 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:12:15.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2024-02-05T07:12:15.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-05T07:12:15.111 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-05T07:12:15.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-02-05T07:12:15.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-05T07:12:15.205 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-05T07:12:15.303 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921', 'cephadm_mode': 'root'} 2024-02-05T07:12:15.303 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:12:15.303 INFO:tasks.cephadm:Cluster fsid is e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:12:15.303 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-05T07:12:15.304 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.7', 'mon.c': '[v2:172.21.15.7:3301,v1:172.21.15.7:6790]', 'mon.b': '172.21.15.90'} 2024-02-05T07:12:15.304 INFO:tasks.cephadm:First mon is mon.a on smithi007 2024-02-05T07:12:15.304 INFO:tasks.cephadm:First mgr is y 2024-02-05T07:12:15.304 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-05T07:12:15.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo hostname $(hostname -s) 2024-02-05T07:12:15.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo hostname $(hostname -s) 2024-02-05T07:12:15.359 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-02-05T07:12:15.359 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:12:15.573 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/76505/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1241-g164d3be4', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1241.g164d3be4'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-02-05 00:32:51.050045', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-testing-3-2024-02-04-2254', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-02-05T07:12:15.785 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-guits-testing-3-2024-02-04-2254, sha1 164d3be4a527be775f4d3c343ea591122f1b8921 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:12:15.786 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/x86_64/flavors/default/cephadm 2024-02-05T07:12:15.787 DEBUG:teuthology.orchestra.run.smithi007:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-05T07:12:16.355 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r--. 1 ubuntu ubuntu 766928 Feb 5 07:12 /home/ubuntu/cephtest/cephadm 2024-02-05T07:12:16.355 DEBUG:teuthology.orchestra.run.smithi090:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-05T07:12:16.953 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r--. 1 ubuntu ubuntu 766928 Feb 5 07:12 /home/ubuntu/cephtest/cephadm 2024-02-05T07:12:16.953 DEBUG:teuthology.orchestra.run.smithi007:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-05T07:12:16.972 DEBUG:teuthology.orchestra.run.smithi090:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-05T07:12:17.003 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 on all hosts... 2024-02-05T07:12:17.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 pull 2024-02-05T07:12:17.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 pull 2024-02-05T07:12:17.934 INFO:teuthology.orchestra.run.smithi007.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921... 2024-02-05T07:12:18.090 INFO:teuthology.orchestra.run.smithi090.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921... 2024-02-05T07:13:00.271 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:13:00.271 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev)", 2024-02-05T07:13:00.272 INFO:teuthology.orchestra.run.smithi090.stdout: "image_id": "8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1", 2024-02-05T07:13:00.272 INFO:teuthology.orchestra.run.smithi090.stdout: "repo_digests": [ 2024-02-05T07:13:00.272 INFO:teuthology.orchestra.run.smithi090.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0" 2024-02-05T07:13:00.272 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:13:00.272 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:13:04.653 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:13:04.653 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev)", 2024-02-05T07:13:04.653 INFO:teuthology.orchestra.run.smithi007.stdout: "image_id": "8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1", 2024-02-05T07:13:04.653 INFO:teuthology.orchestra.run.smithi007.stdout: "repo_digests": [ 2024-02-05T07:13:04.653 INFO:teuthology.orchestra.run.smithi007.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0" 2024-02-05T07:13:04.653 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:13:04.653 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:13:04.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph 2024-02-05T07:13:04.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph 2024-02-05T07:13:04.752 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /etc/ceph 2024-02-05T07:13:04.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /etc/ceph 2024-02-05T07:13:04.822 INFO:tasks.cephadm:Writing seed config... 2024-02-05T07:13:04.823 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-05T07:13:04.823 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-05T07:13:04.823 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-02-05T07:13:04.823 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-02-05T07:13:04.823 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-02-05T07:13:04.823 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-02-05T07:13:04.823 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-02-05T07:13:04.823 INFO:tasks.cephadm: override: [global] ms type = async 2024-02-05T07:13:04.823 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-05T07:13:04.824 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-02-05T07:13:04.825 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:13:04.825 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-05T07:13:04.839 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e3e7059e-c3f5-11ee-95b6-87774f69a715 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-02-05T07:13:04.840 DEBUG:teuthology.orchestra.run.smithi007:mon.a> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mon.a.service 2024-02-05T07:13:04.882 DEBUG:teuthology.orchestra.run.smithi007:mgr.y> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mgr.y.service 2024-02-05T07:13:04.924 INFO:tasks.cephadm:Bootstrapping... 2024-02-05T07:13:04.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 -v bootstrap --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.7 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-05T07:13:05.118 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:13:05.118 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921', '-v', 'bootstrap', '--fsid', 'e3e7059e-c3f5-11ee-95b6-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.7', '--skip-admin-label'] 2024-02-05T07:13:05.144 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4.9.0 2024-02-05T07:13:05.144 INFO:teuthology.orchestra.run.smithi007.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-05T07:13:05.144 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying podman|docker is present... 2024-02-05T07:13:05.168 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4.9.0 2024-02-05T07:13:05.168 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying lvm2 is present... 2024-02-05T07:13:05.168 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying time synchronization is in place... 2024-02-05T07:13:05.176 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-05T07:13:05.176 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-05T07:13:05.182 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-05T07:13:05.182 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-02-05T07:13:05.188 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout enabled 2024-02-05T07:13:05.194 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout active 2024-02-05T07:13:05.194 INFO:teuthology.orchestra.run.smithi007.stdout:Unit chronyd.service is enabled and running 2024-02-05T07:13:05.194 INFO:teuthology.orchestra.run.smithi007.stdout:Repeating the final host check... 2024-02-05T07:13:05.216 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4.9.0 2024-02-05T07:13:05.216 INFO:teuthology.orchestra.run.smithi007.stdout:podman (/bin/podman) version 4.9.0 is present 2024-02-05T07:13:05.217 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl is present 2024-02-05T07:13:05.217 INFO:teuthology.orchestra.run.smithi007.stdout:lvcreate is present 2024-02-05T07:13:05.223 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-05T07:13:05.223 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-05T07:13:05.229 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-05T07:13:05.229 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-02-05T07:13:05.235 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout enabled 2024-02-05T07:13:05.241 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout active 2024-02-05T07:13:05.241 INFO:teuthology.orchestra.run.smithi007.stdout:Unit chronyd.service is enabled and running 2024-02-05T07:13:05.241 INFO:teuthology.orchestra.run.smithi007.stdout:Host looks OK 2024-02-05T07:13:05.241 INFO:teuthology.orchestra.run.smithi007.stdout:Cluster fsid: e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:13:05.241 INFO:teuthology.orchestra.run.smithi007.stdout:Acquiring lock 140008586437200 on /run/cephadm/e3e7059e-c3f5-11ee-95b6-87774f69a715.lock 2024-02-05T07:13:05.241 INFO:teuthology.orchestra.run.smithi007.stdout:Lock 140008586437200 acquired on /run/cephadm/e3e7059e-c3f5-11ee-95b6-87774f69a715.lock 2024-02-05T07:13:05.242 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying IP 172.21.15.7 port 3300 ... 2024-02-05T07:13:05.242 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying IP 172.21.15.7 port 6789 ... 2024-02-05T07:13:05.242 INFO:teuthology.orchestra.run.smithi007.stdout:Base mon IP(s) is [172.21.15.7:3300, 172.21.15.7:6789], mon addrv is [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-02-05T07:13:05.245 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.7 metric 100 2024-02-05T07:13:05.245 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.7 metric 100 2024-02-05T07:13:05.248 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-05T07:13:05.248 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-05T07:13:05.248 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-05T07:13:05.250 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-05T07:13:05.250 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-05T07:13:05.250 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-05T07:13:05.250 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-05T07:13:05.250 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15ea/64 scope link noprefixroute 2024-02-05T07:13:05.250 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-05T07:13:05.251 INFO:teuthology.orchestra.run.smithi007.stdout:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2024-02-05T07:13:05.251 INFO:teuthology.orchestra.run.smithi007.stdout:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2024-02-05T07:13:05.251 INFO:teuthology.orchestra.run.smithi007.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-05T07:13:05.251 INFO:teuthology.orchestra.run.smithi007.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-05T07:13:05.252 INFO:teuthology.orchestra.run.smithi007.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921... 2024-02-05T07:13:06.323 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 2024-02-05T07:13:06.323 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921... 2024-02-05T07:13:06.323 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Getting image source signatures 2024-02-05T07:13:06.323 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying blob sha256:48ac04de0ee67de328b3ea21477b81735e6e7c0783ebcda067ca4695a8266750 2024-02-05T07:13:06.323 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-05T07:13:06.323 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying config sha256:8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 2024-02-05T07:13:06.323 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-05T07:13:07.252 INFO:teuthology.orchestra.run.smithi007.stdout:ceph: stdout ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev) 2024-02-05T07:13:07.253 INFO:teuthology.orchestra.run.smithi007.stdout:Ceph version: ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev) 2024-02-05T07:13:07.253 INFO:teuthology.orchestra.run.smithi007.stdout:Extracting ceph user uid/gid from container image... 2024-02-05T07:13:08.250 INFO:teuthology.orchestra.run.smithi007.stdout:stat: stdout 167 167 2024-02-05T07:13:08.250 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial keys... 2024-02-05T07:13:09.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQCEisBlXtzAIhAAm+94BwtPC51xGGrNSxxsyA== 2024-02-05T07:13:10.213 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQCFisBlaHNUIxAAKhdhfzjlcFQ+2BTf3221Uw== 2024-02-05T07:13:11.111 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQCGisBlycwMIBAASQX3B6ZQi8HLo4UzUPtDHw== 2024-02-05T07:13:11.111 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial monmap... 2024-02-05T07:13:12.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-05T07:13:12.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-02-05T07:13:12.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:13:12.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-05T07:13:12.000 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool for a [v2:172.21.15.7:3300,v1:172.21.15.7:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-05T07:13:12.000 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = quincy 2024-02-05T07:13:12.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: set fsid to e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:13:12.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-05T07:13:12.000 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:13:12.000 INFO:teuthology.orchestra.run.smithi007.stdout:Creating mon... 2024-02-05T07:13:13.071 INFO:teuthology.orchestra.run.smithi007.stdout:create mon.a on 2024-02-05T07:13:13.281 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-05T07:13:13.430 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-05T07:13:13.715 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715.target → /etc/systemd/system/ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715.target. 2024-02-05T07:13:13.716 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715.target → /etc/systemd/system/ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715.target. 2024-02-05T07:13:13.967 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mon.a 2024-02-05T07:13:13.968 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mon.a.service: Unit ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mon.a.service not loaded. 2024-02-05T07:13:14.134 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715.target.wants/ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mon.a.service → /etc/systemd/system/ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@.service. 2024-02-05T07:13:14.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:14 smithi007 podman[28710]: 2024-02-05 07:13:14.277407711 +0000 UTC m=+0.019943746 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:13:14.621 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-05T07:13:14.621 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-02-05T07:13:14.627 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-05T07:13:14.627 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-02-05T07:13:14.627 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-02-05T07:13:14.627 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to enable service . firewalld.service is not available 2024-02-05T07:13:14.628 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mon to start... 2024-02-05T07:13:14.628 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mon... 2024-02-05T07:13:14.741 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:14 smithi007 podman[28710]: 2024-02-05 07:13:14.538594243 +0000 UTC m=+0.281130250 container init e3cb26ba652abc122a86a3116250bd7b3c1f414eaeb3ddc637a8fe1168af0069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T07:13:14.741 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:14 smithi007 podman[28710]: 2024-02-05 07:13:14.541845268 +0000 UTC m=+0.284381285 container start e3cb26ba652abc122a86a3116250bd7b3c1f414eaeb3ddc637a8fe1168af0069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T07:13:14.741 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:14 smithi007 bash[28710]: e3cb26ba652abc122a86a3116250bd7b3c1f414eaeb3ddc637a8fe1168af0069 2024-02-05T07:13:14.741 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:14 smithi007 ceph-mon[28729]: mkfs e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:13:14.741 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:14 smithi007 systemd[1]: Started Ceph mon.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:13:14.742 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:14 smithi007 ceph-mon[28729]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-05T07:13:15.634 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout cluster: 2024-02-05T07:13:15.634 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout id: e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:13:15.634 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-05T07:13:15.634 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-02-05T07:13:15.634 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout services: 2024-02-05T07:13:15.634 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.508176s) 2024-02-05T07:13:15.634 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-05T07:13:15.635 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-05T07:13:15.635 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-02-05T07:13:15.635 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout data: 2024-02-05T07:13:15.635 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-05T07:13:15.635 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-05T07:13:15.635 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-05T07:13:15.635 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout pgs: 2024-02-05T07:13:15.635 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-02-05T07:13:15.635 INFO:teuthology.orchestra.run.smithi007.stdout:mon is available 2024-02-05T07:13:15.635 INFO:teuthology.orchestra.run.smithi007.stdout:Assimilating anything we can from ceph.conf... 2024-02-05T07:13:15.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:15 smithi007 ceph-mon[28729]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-05T07:13:15.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:15 smithi007 ceph-mon[28729]: monmap e1: 1 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T07:13:15.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:15 smithi007 ceph-mon[28729]: fsmap 2024-02-05T07:13:15.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:15 smithi007 ceph-mon[28729]: osdmap e1: 0 total, 0 up, 0 in 2024-02-05T07:13:15.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:15 smithi007 ceph-mon[28729]: mgrmap e1: no daemons active 2024-02-05T07:13:15.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:15 smithi007 ceph-mon[28729]: from='client.? 172.21.15.7:0/526653647' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-05T07:13:16.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [global] 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout fsid = e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [osd] 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-05T07:13:16.607 INFO:teuthology.orchestra.run.smithi007.stdout:Generating new minimal ceph.conf... 2024-02-05T07:13:16.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:16 smithi007 ceph-mon[28729]: from='client.? 172.21.15.7:0/3956658459' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-05T07:13:16.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:16 smithi007 ceph-mon[28729]: from='client.? 172.21.15.7:0/3956658459' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-05T07:13:17.586 INFO:teuthology.orchestra.run.smithi007.stdout:Restarting the monitor... 2024-02-05T07:13:17.845 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:17 smithi007 systemd[1]: Stopping Ceph mon.a for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:13:17.845 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:17 smithi007 ceph-mon[28729]: from='client.? 172.21.15.7:0/4126832842' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:13:18.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:17 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a[28724]: 2024-02-05T07:13:17.842+0000 7fcb3ac9f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:13:18.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:17 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a[28724]: 2024-02-05T07:13:17.842+0000 7fcb3ac9f700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-05T07:13:18.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:18 smithi007 podman[28972]: 2024-02-05 07:13:18.079087011 +0000 UTC m=+0.333818129 container died e3cb26ba652abc122a86a3116250bd7b3c1f414eaeb3ddc637a8fe1168af0069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T07:13:21.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:21 smithi007 podman[28972]: 2024-02-05 07:13:21.524692583 +0000 UTC m=+3.779423700 container cleanup e3cb26ba652abc122a86a3116250bd7b3c1f414eaeb3ddc637a8fe1168af0069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-05T07:13:21.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:21 smithi007 bash[28972]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a 2024-02-05T07:13:23.490 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:23 smithi007 podman[28987]: 2024-02-05 07:13:23.205729368 +0000 UTC m=+5.119701527 container remove e3cb26ba652abc122a86a3116250bd7b3c1f414eaeb3ddc637a8fe1168af0069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2) 2024-02-05T07:13:23.810 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:23 smithi007 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-05T07:13:23.811 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:23 smithi007 systemd[1]: Stopped Ceph mon.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:13:23.811 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:23 smithi007 systemd[1]: Starting Ceph mon.a for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:13:23.811 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:23 smithi007 podman[29080]: 2024-02-05 07:13:23.627909959 +0000 UTC m=+0.018392599 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:13:23.811 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:23 smithi007 podman[29080]: 2024-02-05 07:13:23.809615201 +0000 UTC m=+0.200097845 container create 0693f30f1833f925d469b3e73e4fbf8ca84c0dd9c2efcb09d41965be0d1662b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True) 2024-02-05T07:13:24.059 INFO:teuthology.orchestra.run.smithi007.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-05T07:13:24.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:23 smithi007 podman[29080]: 2024-02-05 07:13:23.978548518 +0000 UTC m=+0.369031154 container init 0693f30f1833f925d469b3e73e4fbf8ca84c0dd9c2efcb09d41965be0d1662b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-05T07:13:24.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:23 smithi007 podman[29080]: 2024-02-05 07:13:23.981754367 +0000 UTC m=+0.372237003 container start 0693f30f1833f925d469b3e73e4fbf8ca84c0dd9c2efcb09d41965be0d1662b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-05T07:13:24.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: set uid:gid to 167:167 (ceph:ceph) 2024-02-05T07:13:24.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev), process ceph-mon, pid 2 2024-02-05T07:13:24.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: pidfile_write: ignore empty --pid-file 2024-02-05T07:13:24.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: load: jerasure load: lrc 2024-02-05T07:13:24.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: RocksDB version: 7.9.2 2024-02-05T07:13:24.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Git sha 0 2024-02-05T07:13:24.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Compile date 2024-02-04 23:11:06 2024-02-05T07:13:24.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: DB SUMMARY 2024-02-05T07:13:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: DB Session ID: OC549KH1O49QXZQDF5G2 2024-02-05T07:13:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: CURRENT file: CURRENT 2024-02-05T07:13:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: IDENTITY file: IDENTITY 2024-02-05T07:13:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-02-05T07:13:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-05T07:13:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 91671 ; 2024-02-05T07:13:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.error_if_exists: 0 2024-02-05T07:13:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.create_if_missing: 0 2024-02-05T07:13:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.paranoid_checks: 1 2024-02-05T07:13:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-05T07:13:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-05T07:13:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-05T07:13:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.env: 0x5614fe8d8ae0 2024-02-05T07:13:24.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.fs: PosixFileSystem 2024-02-05T07:13:24.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.info_log: 0x5614ffc498c0 2024-02-05T07:13:24.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_file_opening_threads: 16 2024-02-05T07:13:24.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.statistics: (nil) 2024-02-05T07:13:24.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.use_fsync: 0 2024-02-05T07:13:24.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_log_file_size: 0 2024-02-05T07:13:24.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-05T07:13:24.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-05T07:13:24.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.keep_log_file_num: 1000 2024-02-05T07:13:24.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.recycle_log_file_num: 0 2024-02-05T07:13:24.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.allow_fallocate: 1 2024-02-05T07:13:24.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.allow_mmap_reads: 0 2024-02-05T07:13:24.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.allow_mmap_writes: 0 2024-02-05T07:13:24.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.use_direct_reads: 0 2024-02-05T07:13:24.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-05T07:13:24.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.create_missing_column_families: 0 2024-02-05T07:13:24.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.db_log_dir: 2024-02-05T07:13:24.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.wal_dir: 2024-02-05T07:13:24.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-05T07:13:24.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-05T07:13:24.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-05T07:13:24.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-05T07:13:24.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-05T07:13:24.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-05T07:13:24.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.advise_random_on_open: 1 2024-02-05T07:13:24.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.db_write_buffer_size: 0 2024-02-05T07:13:24.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.write_buffer_manager: 0x5614ffc1d400 2024-02-05T07:13:24.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-05T07:13:24.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-05T07:13:24.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-05T07:13:24.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.rate_limiter: (nil) 2024-02-05T07:13:24.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-05T07:13:24.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.wal_recovery_mode: 2 2024-02-05T07:13:24.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.enable_thread_tracking: 0 2024-02-05T07:13:24.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.enable_pipelined_write: 0 2024-02-05T07:13:24.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.unordered_write: 0 2024-02-05T07:13:24.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-05T07:13:24.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-05T07:13:24.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-05T07:13:24.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-05T07:13:24.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.row_cache: None 2024-02-05T07:13:24.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.wal_filter: None 2024-02-05T07:13:24.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-05T07:13:24.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.allow_ingest_behind: 0 2024-02-05T07:13:24.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.two_write_queues: 0 2024-02-05T07:13:24.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.manual_wal_flush: 0 2024-02-05T07:13:24.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.wal_compression: 0 2024-02-05T07:13:24.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.atomic_flush: 0 2024-02-05T07:13:24.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-05T07:13:24.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-05T07:13:24.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-05T07:13:24.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.log_readahead_size: 0 2024-02-05T07:13:24.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-05T07:13:24.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.best_efforts_recovery: 0 2024-02-05T07:13:24.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-05T07:13:24.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-05T07:13:24.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.allow_data_in_errors: 0 2024-02-05T07:13:24.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.db_host_id: __hostname__ 2024-02-05T07:13:24.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_background_jobs: 2 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_background_compactions: -1 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_subcompactions: 1 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_total_wal_size: 0 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_open_files: -1 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.bytes_per_sync: 0 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-05T07:13:24.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compaction_readahead_size: 0 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_background_flushes: -1 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Compression algorithms supported: 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: kZSTD supported: 0 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: kXpressCompression supported: 0 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: kLZ4HCCompression supported: 1 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: kZlibCompression supported: 1 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: kSnappyCompression supported: 1 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: kLZ4Compression supported: 1 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: kBZip2Compression supported: 0 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-05T07:13:24.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.merge_operator: 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compaction_filter: None 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compaction_filter_factory: None 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.sst_partitioner_factory: None 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5614ffc499c0) 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks: 1 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_top_level_index_and_filter: 1 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_type: 0 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_index_type: 0 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_shortening: 1 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout: checksum: 4 2024-02-05T07:13:24.107 INFO:journalctl@ceph.mon.a.smithi007.stdout: no_block_cache: 0 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache: 0x5614ffc4d090 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_name: BinnedLRUCache 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_options: 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: capacity : 536870912 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_shard_bits : 4 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: strict_capacity_limit : 0 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: high_pri_pool_ratio: 0.000 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_compressed: (nil) 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: persistent_cache: (nil) 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size: 4096 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size_deviation: 10 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_restart_interval: 16 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_block_restart_interval: 1 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: metadata_block_size: 4096 2024-02-05T07:13:24.108 INFO:journalctl@ceph.mon.a.smithi007.stdout: partition_filters: 0 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout: use_delta_encoding: 1 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout: filter_policy: bloomfilter 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout: whole_key_filtering: 1 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout: verify_compression: 0 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout: read_amp_bytes_per_bit: 0 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout: format_version: 5 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_index_compression: 1 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_align: 0 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout: max_auto_readahead_size: 262144 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout: prepopulate_block_cache: 0 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout: initial_auto_readahead_size: 8192 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_file_reads_for_auto_readahead: 2 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.write_buffer_size: 33554432 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_write_buffer_number: 2 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compression: NoCompression 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.bottommost_compression: Disabled 2024-02-05T07:13:24.109 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.prefix_extractor: nullptr 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.num_levels: 7 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-05T07:13:24.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compression_opts.level: 32767 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compression_opts.strategy: 0 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compression_opts.enabled: false 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.target_file_size_base: 67108864 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-05T07:13:24.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-05T07:13:24.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-05T07:13:24.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-05T07:13:24.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-05T07:13:24.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-05T07:13:24.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-05T07:13:24.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-05T07:13:24.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-05T07:13:24.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-05T07:13:24.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-05T07:13:24.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.arena_block_size: 1048576 2024-02-05T07:13:24.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-05T07:13:24.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-05T07:13:24.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.disable_auto_compactions: 0 2024-02-05T07:13:24.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.table_properties_collectors: 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.inplace_update_support: 0 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-05T07:13:24.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.bloom_locality: 0 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.max_successive_merges: 0 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.paranoid_file_checks: 0 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.force_consistency_checks: 1 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.report_bg_io_stats: 0 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.ttl: 2592000 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.enable_blob_files: false 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.min_blob_size: 0 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.blob_file_size: 268435456 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-05T07:13:24.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-05T07:13:24.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-05T07:13:24.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.blob_file_starting_level: 0 2024-02-05T07:13:24.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-05T07:13:24.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-02-05T07:13:24.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-02-05T07:13:24.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 90eb6965-4d27-4834-b5fc-41217ee3ff87 2024-02-05T07:13:24.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707117204012382, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-02-05T07:13:24.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-02-05T07:13:24.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707117204013265, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 87544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 282, "table_properties": {"data_size": 85632, "index_size": 258, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11802, "raw_average_key_size": 48, "raw_value_size": 79167, "raw_average_value_size": 328, "num_data_blocks": 11, "num_entries": 241, "num_filter_entries": 241, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707117204, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "90eb6965-4d27-4834-b5fc-41217ee3ff87", "db_session_id": "OC549KH1O49QXZQDF5G2", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-02-05T07:13:24.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707117204013323, "job": 1, "event": "recovery_finished"} 2024-02-05T07:13:24.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-02-05T07:13:24.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-05T07:13:24.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5614ffd3e000 2024-02-05T07:13:24.116 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: DB pointer 0x5614ffd28000 2024-02-05T07:13:24.116 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-05T07:13:24.116 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-05T07:13:24.116 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** DB Stats ** 2024-02-05T07:13:24.116 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T07:13:24.116 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-05T07:13:24.116 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T07:13:24.116 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T07:13:24.116 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-05T07:13:24.116 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T07:13:24.116 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T07:13:24.117 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-02-05T07:13:24.117 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-02-05T07:13:24.117 INFO:journalctl@ceph.mon.a.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T07:13:24.117 INFO:journalctl@ceph.mon.a.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-05T07:13:24.117 INFO:journalctl@ceph.mon.a.smithi007.stdout: L0 2/0 87.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T07:13:24.117 INFO:journalctl@ceph.mon.a.smithi007.stdout: Sum 2/0 87.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T07:13:24.117 INFO:journalctl@ceph.mon.a.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T07:13:24.117 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-02-05T07:13:24.117 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-02-05T07:13:24.117 INFO:journalctl@ceph.mon.a.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T07:13:24.117 INFO:journalctl@ceph.mon.a.smithi007.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-05T07:13:24.117 INFO:journalctl@ceph.mon.a.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T07:13:24.117 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-02-05T07:13:24.117 INFO:journalctl@ceph.mon.a.smithi007.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-05T07:13:24.118 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-02-05T07:13:24.118 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T07:13:24.118 INFO:journalctl@ceph.mon.a.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-05T07:13:24.118 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-05T07:13:24.118 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-05T07:13:24.118 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-05T07:13:24.118 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-05T07:13:24.118 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative compaction: 0.00 GB write, 29.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T07:13:24.118 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval compaction: 0.00 GB write, 29.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T07:13:24.118 INFO:journalctl@ceph.mon.a.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-05T07:13:24.118 INFO:journalctl@ceph.mon.a.smithi007.stdout: Block cache BinnedLRUCache@0x5614ffc4d090#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-05T07:13:24.118 INFO:journalctl@ceph.mon.a.smithi007.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-02-05T07:13:24.118 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-02-05T07:13:24.118 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: starting mon.a rank 0 at public addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] at bind addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: mon.a@-1(???) e1 preinit fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: mon.a@-1(???).mds e1 new map 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: mon.a@-1(???).mds e1 print_map 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout: e1 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout: legacy client fscid: -1 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout: No filesystems configured 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T07:13:24.119 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-05T07:13:24.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 bash[29080]: 0693f30f1833f925d469b3e73e4fbf8ca84c0dd9c2efcb09d41965be0d1662b2 2024-02-05T07:13:24.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 systemd[1]: Started Ceph mon.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:13:24.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-05T07:13:24.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: monmap e1: 1 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T07:13:24.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: fsmap 2024-02-05T07:13:24.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: osdmap e1: 0 total, 0 up, 0 in 2024-02-05T07:13:24.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:24 smithi007 ceph-mon[29098]: mgrmap e1: no daemons active 2024-02-05T07:13:25.139 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-05T07:13:25.161 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-05T07:13:25.161 INFO:teuthology.orchestra.run.smithi007.stdout:Creating mgr... 2024-02-05T07:13:25.162 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying port 0.0.0.0:9283 ... 2024-02-05T07:13:25.162 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying port 0.0.0.0:8765 ... 2024-02-05T07:13:25.410 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mgr.y 2024-02-05T07:13:25.410 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mgr.y.service: Unit ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mgr.y.service not loaded. 2024-02-05T07:13:25.558 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715.target.wants/ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@.service. 2024-02-05T07:13:25.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:25 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2463284098' entity='client.admin' 2024-02-05T07:13:26.049 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:25 smithi007 podman[29313]: 2024-02-05 07:13:25.975485635 +0000 UTC m=+0.304018256 container start 763e883e587bb6d98d1552f781ef34b72d20855a080e9c9d8e0c89a94ab6f86e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-05T07:13:26.049 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:26 smithi007 bash[29313]: 763e883e587bb6d98d1552f781ef34b72d20855a080e9c9d8e0c89a94ab6f86e 2024-02-05T07:13:26.064 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-05T07:13:26.064 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-02-05T07:13:26.071 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-05T07:13:26.072 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-02-05T07:13:26.072 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-02-05T07:13:26.072 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to enable service . firewalld.service is not available 2024-02-05T07:13:26.079 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-05T07:13:26.079 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-02-05T07:13:26.087 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-05T07:13:26.087 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-02-05T07:13:26.087 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-02-05T07:13:26.087 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-02-05T07:13:26.087 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr to start... 2024-02-05T07:13:26.087 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr... 2024-02-05T07:13:26.385 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:26 smithi007 systemd[1]: Started Ceph mgr.y for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:13:26.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:26 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3077695783' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-05T07:13:26.651 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:26 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:26.547+0000 7fb151504200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T07:13:26.651 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:26 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:26.648+0000 7fb151504200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T07:13:27.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-02-05T07:13:27.169 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-02-05T07:13:27.169 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "e3e7059e-c3f5-11ee-95b6-87774f69a715", 2024-02-05T07:13:27.169 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-02-05T07:13:27.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-05T07:13:27.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-05T07:13:27.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-05T07:13:27.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:27.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-05T07:13:27.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-05T07:13:27.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-02-05T07:13:27.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:13:27.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-05T07:13:27.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-02-05T07:13:27.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:13:27.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-02-05T07:13:27.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-05T07:13:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:13:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-05T07:13:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-05T07:13:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-05T07:13:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:13:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-05T07:13:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-05T07:13:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-05T07:13:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-05T07:13:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-05T07:13:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-05T07:13:27.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:27.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-05T07:13:27.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-05T07:13:27.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-05T07:13:27.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-05T07:13:27.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-05T07:13:27.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-05T07:13:27.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-05T07:13:27.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-05T07:13:27.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-05T07:13:27.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:27.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-05T07:13:27.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:13:27.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-05T07:13:27.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-05T07:13:27.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:27.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-05T07:13:27.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-05T07:13:27.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-05T07:13:27.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-05T07:13:27.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-05T07:13:27.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-05T07:13:27.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-02-05T07:13:27.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:13:27.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T07:13:27.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:27.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-05T07:13:27.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:13:27.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-02-05T07:13:14.604359+0000", 2024-02-05T07:13:27.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T07:13:27.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:27.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-05T07:13:27.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-02-05T07:13:27.177 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (1/15)... 2024-02-05T07:13:27.654 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:27 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:27.316+0000 7fb151504200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T07:13:27.655 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:27 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:27.411+0000 7fb151504200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T07:13:27.655 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:27 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:27.555+0000 7fb151504200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T07:13:27.655 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:27 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:27.652+0000 7fb151504200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T07:13:27.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:27 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:27.737+0000 7fb151504200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T07:13:28.304 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:28 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:28.005+0000 7fb151504200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T07:13:28.304 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:28 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:28.117+0000 7fb151504200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T07:13:28.304 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:28 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:28.205+0000 7fb151504200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T07:13:28.304 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:28 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:28.302+0000 7fb151504200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T07:13:28.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:28 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:28.480+0000 7fb151504200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T07:13:29.629 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:29 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:29.492+0000 7fb151504200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T07:13:29.629 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:29 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:29.627+0000 7fb151504200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T07:13:30.260 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:29 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:29.966+0000 7fb151504200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T07:13:30.261 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:30 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:30.058+0000 7fb151504200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T07:13:30.261 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:30 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:30.258+0000 7fb151504200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:13:30.532 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:30 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:30.447+0000 7fb151504200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T07:13:30.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:30 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:30.739+0000 7fb151504200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:13:31.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:31 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:31.001+0000 7fb151504200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T07:13:31.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:31 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:31.244+0000 7fb151504200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T07:13:32.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:32 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:32.080+0000 7fb151504200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T07:13:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:32 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/685922124' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-05T07:13:32.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:32 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:32.665+0000 7fb151504200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T07:13:32.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:32 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:32.769+0000 7fb151504200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T07:13:33.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:32 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:32.994+0000 7fb151504200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T07:13:34.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:33 smithi007 ceph-mon[29098]: Activating manager daemon y 2024-02-05T07:13:34.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:33 smithi007 ceph-mon[29098]: mgrmap e2: y(active, starting, since 0.00304096s) 2024-02-05T07:13:34.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:33 smithi007 ceph-mon[29098]: from='mgr.14100 172.21.15.7:0/251022889' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T07:13:34.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:33 smithi007 ceph-mon[29098]: from='mgr.14100 172.21.15.7:0/251022889' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T07:13:34.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:33 smithi007 ceph-mon[29098]: from='mgr.14100 172.21.15.7:0/251022889' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T07:13:34.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:33 smithi007 ceph-mon[29098]: from='mgr.14100 172.21.15.7:0/251022889' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:13:34.121 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:33 smithi007 ceph-mon[29098]: from='mgr.14100 172.21.15.7:0/251022889' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T07:13:34.121 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:33 smithi007 ceph-mon[29098]: Manager daemon y is now available 2024-02-05T07:13:34.121 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:33 smithi007 ceph-mon[29098]: from='mgr.14100 172.21.15.7:0/251022889' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T07:13:34.121 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:33 smithi007 ceph-mon[29098]: from='mgr.14100 172.21.15.7:0/251022889' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T07:13:34.121 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:33 smithi007 ceph-mon[29098]: from='mgr.14100 172.21.15.7:0/251022889' entity='mgr.y' 2024-02-05T07:13:34.121 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:33 smithi007 ceph-mon[29098]: from='mgr.14100 172.21.15.7:0/251022889' entity='mgr.y' 2024-02-05T07:13:34.121 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:33 smithi007 ceph-mon[29098]: from='mgr.14100 172.21.15.7:0/251022889' entity='mgr.y' 2024-02-05T07:13:34.146 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-02-05T07:13:34.146 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-02-05T07:13:34.146 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "e3e7059e-c3f5-11ee-95b6-87774f69a715", 2024-02-05T07:13:34.147 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-02-05T07:13:34.147 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-05T07:13:34.147 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-05T07:13:34.147 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-05T07:13:34.147 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:34.147 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-05T07:13:34.147 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-05T07:13:34.147 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-02-05T07:13:34.147 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:13:34.147 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-05T07:13:34.147 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-02-05T07:13:34.147 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:13:34.147 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-02-05T07:13:34.148 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-05T07:13:34.148 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:13:34.148 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-05T07:13:34.148 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-05T07:13:34.148 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:34.148 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-05T07:13:34.148 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:13:34.148 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-05T07:13:34.148 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-05T07:13:34.148 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-05T07:13:34.148 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-05T07:13:34.148 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-05T07:13:34.148 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-05T07:13:34.152 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:34.152 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-05T07:13:34.152 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-05T07:13:34.152 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-05T07:13:34.153 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-05T07:13:34.153 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-05T07:13:34.153 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-05T07:13:34.153 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-05T07:13:34.153 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-05T07:13:34.153 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-05T07:13:34.153 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:34.153 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-05T07:13:34.153 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:13:34.153 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-05T07:13:34.153 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-05T07:13:34.153 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:34.154 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-05T07:13:34.154 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-05T07:13:34.154 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-05T07:13:34.154 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-05T07:13:34.154 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-05T07:13:34.154 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-05T07:13:34.154 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-02-05T07:13:34.154 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:13:34.154 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T07:13:34.154 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:34.154 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-05T07:13:34.154 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:13:34.154 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-02-05T07:13:14.604359+0000", 2024-02-05T07:13:34.155 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T07:13:34.155 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:34.155 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-05T07:13:34.155 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-02-05T07:13:34.155 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (2/15)... 2024-02-05T07:13:35.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:35 smithi007 ceph-mon[29098]: mgrmap e3: y(active, since 1.00704s) 2024-02-05T07:13:37.160 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:37 smithi007 ceph-mon[29098]: mgrmap e4: y(active, since 3s) 2024-02-05T07:13:37.160 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:37 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3455684096' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-05T07:13:37.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-02-05T07:13:37.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-02-05T07:13:37.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "e3e7059e-c3f5-11ee-95b6-87774f69a715", 2024-02-05T07:13:37.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-02-05T07:13:37.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-05T07:13:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-05T07:13:37.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:37.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-05T07:13:37.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:13:37.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-05T07:13:37.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-05T07:13:37.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-05T07:13:37.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-05T07:13:37.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-05T07:13:37.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-05T07:13:37.446 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:37.446 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-05T07:13:37.446 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-05T07:13:37.446 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-05T07:13:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-05T07:13:37.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-05T07:13:37.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-05T07:13:37.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-05T07:13:37.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-02-05T07:13:37.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:13:37.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T07:13:37.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:37.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-05T07:13:37.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:13:37.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-02-05T07:13:14.604359+0000", 2024-02-05T07:13:37.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T07:13:37.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:13:37.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-05T07:13:37.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-02-05T07:13:37.449 INFO:teuthology.orchestra.run.smithi007.stdout:mgr is available 2024-02-05T07:13:38.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:38 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3231093332' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-05T07:13:38.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-02-05T07:13:38.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [global] 2024-02-05T07:13:38.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout fsid = e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:13:38.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-05T07:13:38.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-05T07:13:38.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-05T07:13:38.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-05T07:13:38.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-02-05T07:13:38.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-05T07:13:38.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-05T07:13:38.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-02-05T07:13:38.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [osd] 2024-02-05T07:13:38.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-05T07:13:38.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-05T07:13:38.621 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling cephadm module... 2024-02-05T07:13:40.331 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:40 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/896768209' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-05T07:13:40.331 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:40 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ignoring --setuser ceph since I am not root 2024-02-05T07:13:40.331 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:40 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ignoring --setgroup ceph since I am not root 2024-02-05T07:13:40.875 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:40 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:40.600+0000 7f7aec6d7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T07:13:40.875 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:40 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:40.698+0000 7f7aec6d7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T07:13:41.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:41 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/896768209' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-05T07:13:41.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:41 smithi007 ceph-mon[29098]: mgrmap e5: y(active, since 7s) 2024-02-05T07:13:41.605 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:41 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:41.363+0000 7f7aec6d7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T07:13:41.605 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:41 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:41.459+0000 7f7aec6d7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T07:13:41.605 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:41 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:41.603+0000 7f7aec6d7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T07:13:41.748 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-02-05T07:13:41.748 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-05T07:13:41.748 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-05T07:13:41.749 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-05T07:13:41.749 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-05T07:13:41.749 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-02-05T07:13:41.749 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for the mgr to restart... 2024-02-05T07:13:41.749 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr epoch 5... 2024-02-05T07:13:41.869 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:41 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:41.701+0000 7f7aec6d7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T07:13:41.869 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:41 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:41.790+0000 7f7aec6d7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T07:13:42.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:42 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1699365821' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-05T07:13:42.148 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:42 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:42.064+0000 7f7aec6d7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T07:13:42.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:42 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:42.179+0000 7f7aec6d7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T07:13:42.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:42 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:42.267+0000 7f7aec6d7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T07:13:42.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:42 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:42.363+0000 7f7aec6d7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T07:13:42.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:42 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:42.541+0000 7f7aec6d7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T07:13:43.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:43.524+0000 7f7aec6d7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T07:13:43.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:43.658+0000 7f7aec6d7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T07:13:44.287 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:43.994+0000 7f7aec6d7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T07:13:44.287 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:44 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:44.086+0000 7f7aec6d7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T07:13:44.287 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:44 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:44.285+0000 7f7aec6d7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:13:44.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:44 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:44.473+0000 7f7aec6d7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T07:13:45.027 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:44 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:44.762+0000 7f7aec6d7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:13:45.027 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:45 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:45.025+0000 7f7aec6d7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T07:13:45.397 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:45 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:45.265+0000 7f7aec6d7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T07:13:46.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:46 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:46.090+0000 7f7aec6d7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T07:13:46.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:46 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:46.670+0000 7f7aec6d7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T07:13:46.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:46 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:46.770+0000 7f7aec6d7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T07:13:47.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:46 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:13:46.991+0000 7f7aec6d7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T07:13:47.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:47 smithi007 ceph-mon[29098]: Active manager daemon y restarted 2024-02-05T07:13:47.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:47 smithi007 ceph-mon[29098]: Activating manager daemon y 2024-02-05T07:13:47.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:47 smithi007 ceph-mon[29098]: osdmap e2: 0 total, 0 up, 0 in 2024-02-05T07:13:47.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:47 smithi007 ceph-mon[29098]: mgrmap e6: y(active, starting, since 0.00330179s) 2024-02-05T07:13:47.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:47 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:13:47.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:47 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T07:13:47.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:47 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T07:13:47.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:47 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T07:13:47.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:47 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T07:13:47.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:47 smithi007 ceph-mon[29098]: Manager daemon y is now available 2024-02-05T07:13:47.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:47 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' 2024-02-05T07:13:47.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:47 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' 2024-02-05T07:13:47.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:47 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:13:47.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:47 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:13:48.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:48 smithi007 ceph-mon[29098]: Found migration_current of "None". Setting to last migration. 2024-02-05T07:13:48.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:48 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:13:48.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:48 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T07:13:48.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:48 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T07:13:48.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:48 smithi007 ceph-mon[29098]: mgrmap e7: y(active, since 1.00514s) 2024-02-05T07:13:48.602 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-02-05T07:13:48.602 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-05T07:13:48.602 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-05T07:13:48.602 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-02-05T07:13:48.603 INFO:teuthology.orchestra.run.smithi007.stdout:mgr epoch 5 is available 2024-02-05T07:13:48.603 INFO:teuthology.orchestra.run.smithi007.stdout:Setting orchestrator backend to cephadm... 2024-02-05T07:13:49.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:49 smithi007 ceph-mon[29098]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-05T07:13:49.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:49 smithi007 ceph-mon[29098]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-05T07:13:49.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:49 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' 2024-02-05T07:13:49.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:49 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' 2024-02-05T07:13:49.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:49 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:13:50.343 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:50 smithi007 ceph-mon[29098]: [05/Feb/2024:07:13:48] ENGINE Bus STARTING 2024-02-05T07:13:50.344 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:50 smithi007 ceph-mon[29098]: [05/Feb/2024:07:13:48] ENGINE Serving on http://172.21.15.7:8765 2024-02-05T07:13:50.344 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:50 smithi007 ceph-mon[29098]: [05/Feb/2024:07:13:49] ENGINE Serving on https://172.21.15.7:7150 2024-02-05T07:13:50.344 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:50 smithi007 ceph-mon[29098]: [05/Feb/2024:07:13:49] ENGINE Bus STARTED 2024-02-05T07:13:50.344 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:50 smithi007 ceph-mon[29098]: mgrmap e8: y(active, since 2s) 2024-02-05T07:13:50.344 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:50 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' 2024-02-05T07:13:50.344 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:50 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:13:51.221 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:51 smithi007 ceph-mon[29098]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:13:51.224 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-05T07:13:51.224 INFO:teuthology.orchestra.run.smithi007.stdout:Generating ssh key... 2024-02-05T07:13:52.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-mon[29098]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:13:52.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: Generating public/private rsa key pair. 2024-02-05T07:13:52.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: Your identification has been saved in /tmp/tmppiimv3ov/key. 2024-02-05T07:13:52.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: Your public key has been saved in /tmp/tmppiimv3ov/key.pub. 2024-02-05T07:13:52.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: The key fingerprint is: 2024-02-05T07:13:52.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: SHA256:dgd4g8bBOhTzyU2I98DRRCZeT6Vj4CwcD1GXbrNcn1o ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:13:52.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: The key's randomart image is: 2024-02-05T07:13:52.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: +---[RSA 3072]----+ 2024-02-05T07:13:52.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: | o=BOO oo. | 2024-02-05T07:13:52.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: | oB*^.=.. | 2024-02-05T07:13:52.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: | . o%oO.= | 2024-02-05T07:13:52.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: | o. o.++.. | 2024-02-05T07:13:52.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: | .S .o.+ ..| 2024-02-05T07:13:52.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: | . . .o E.| 2024-02-05T07:13:52.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: | o | 2024-02-05T07:13:52.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: | . | 2024-02-05T07:13:52.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: | | 2024-02-05T07:13:52.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:13:52 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: +----[SHA256]-----+ 2024-02-05T07:13:53.147 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:53 smithi007 ceph-mon[29098]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:13:53.147 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:53 smithi007 ceph-mon[29098]: Generating ssh key... 2024-02-05T07:13:53.147 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:53 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' 2024-02-05T07:13:53.147 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:53 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' 2024-02-05T07:13:53.895 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9+aEAWMpp3drRu/DD6s/Rdzu+RbOPjvlvXXLOZJbOI46T2o1KUsQcYQ1ZLBTXHeK5XLX7iHIFsje92FuOy2562tZwm9jdK4m2w/QNCmt98OOGhzr6TkaLhGrvjJyZ1fosNH2Jx0Lk+Obg/T8gkvIIEUnP0nvQAK9yBJCX5bL85BaiMljP+kDQDSKTYX51GGjPymneCHDCLIv5Y0E6Lb0V4hUCe963S6iza1Jbi0diFaV6XMUkm5h8/KnhChM4MZsNF1i0Zm3Vc36nJr5V1EWdsTq0YEK+vQipoc+ueDvRz87WfbJWsNbXlIdQdqv3xzPnTZLA6YPJs2NpmeGdyxOIQ0q/KFklS+OjVj++miKxj9eq9bPckX88ahMJxu74DqaBRt/9lxtEy96AYgjraoSyHmR+yF3hYQo+jUClceR5fNiZgEQmSXniKkxUDjwbc6j5XuKuqHcFUEQfDi4IFn+mU0f6E/CGrU17jJ6Oy94SpDI0Oxleb4oTw5qQLnFo+Kc= ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:13:53.895 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-05T07:13:53.895 INFO:teuthology.orchestra.run.smithi007.stdout:Adding key to root@localhost authorized_keys... 2024-02-05T07:13:53.896 INFO:teuthology.orchestra.run.smithi007.stdout:Adding host smithi007... 2024-02-05T07:13:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:55 smithi007 ceph-mon[29098]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:13:56.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:56 smithi007 ceph-mon[29098]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi007", "addr": "172.21.15.7", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:13:57.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:57 smithi007 ceph-mon[29098]: Deploying cephadm binary to smithi007 2024-02-05T07:13:59.551 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Added host 'smithi007' with addr '172.21.15.7' 2024-02-05T07:13:59.551 INFO:teuthology.orchestra.run.smithi007.stdout:Deploying unmanaged mon service... 2024-02-05T07:14:00.164 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:59 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' 2024-02-05T07:14:00.186 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:59 smithi007 ceph-mon[29098]: Added host smithi007 2024-02-05T07:14:00.186 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:13:59 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:14:01.150 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-05T07:14:01.151 INFO:teuthology.orchestra.run.smithi007.stdout:Deploying unmanaged mgr service... 2024-02-05T07:14:01.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:01 smithi007 ceph-mon[29098]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:14:01.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:01 smithi007 ceph-mon[29098]: Saving service mon spec with placement count:5 2024-02-05T07:14:01.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:01 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' 2024-02-05T07:14:02.864 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-05T07:14:03.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:02 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' 2024-02-05T07:14:03.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:02 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' 2024-02-05T07:14:03.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:02 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' 2024-02-05T07:14:04.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:03 smithi007 ceph-mon[29098]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:14:04.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:03 smithi007 ceph-mon[29098]: Saving service mgr spec with placement count:2 2024-02-05T07:14:04.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:03 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3019855105' entity='client.admin' 2024-02-05T07:14:05.497 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling the dashboard module... 2024-02-05T07:14:05.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:05 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2402043005' entity='client.admin' 2024-02-05T07:14:07.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:07 smithi007 ceph-mon[29098]: from='mgr.14118 172.21.15.7:0/301083030' entity='mgr.y' 2024-02-05T07:14:07.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:07 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/58813061' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-05T07:14:07.334 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:07 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ignoring --setuser ceph since I am not root 2024-02-05T07:14:07.334 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:07 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ignoring --setgroup ceph since I am not root 2024-02-05T07:14:07.873 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:07 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:07.606+0000 7f7bfd618200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T07:14:07.873 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:07 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:07.706+0000 7f7bfd618200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T07:14:08.145 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:08 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/58813061' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-05T07:14:08.145 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:08 smithi007 ceph-mon[29098]: mgrmap e9: y(active, since 20s) 2024-02-05T07:14:08.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:08.393+0000 7f7bfd618200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T07:14:08.674 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:08.489+0000 7f7bfd618200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T07:14:08.674 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:08.639+0000 7f7bfd618200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T07:14:09.007 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:08.739+0000 7f7bfd618200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T07:14:09.007 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:08.825+0000 7f7bfd618200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T07:14:09.011 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-02-05T07:14:09.011 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-05T07:14:09.011 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-05T07:14:09.011 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-05T07:14:09.011 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-05T07:14:09.011 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-02-05T07:14:09.011 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for the mgr to restart... 2024-02-05T07:14:09.011 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr epoch 9... 2024-02-05T07:14:09.310 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:09 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1327016445' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-05T07:14:09.310 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:09 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:09.100+0000 7f7bfd618200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T07:14:09.311 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:09 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:09.215+0000 7f7bfd618200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T07:14:09.311 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:09 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:09.308+0000 7f7bfd618200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T07:14:09.592 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:09 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:09.409+0000 7f7bfd618200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T07:14:09.592 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:09 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:09.590+0000 7f7bfd618200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T07:14:10.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:10 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:10.595+0000 7f7bfd618200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T07:14:10.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:10 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:10.732+0000 7f7bfd618200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T07:14:11.370 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:11 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:11.072+0000 7f7bfd618200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T07:14:11.370 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:11 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:11.166+0000 7f7bfd618200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T07:14:11.371 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:11 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:11.368+0000 7f7bfd618200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:14:11.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:11 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:11.559+0000 7f7bfd618200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T07:14:12.118 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:11 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:11.853+0000 7f7bfd618200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:14:12.118 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:12 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:12.116+0000 7f7bfd618200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T07:14:12.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:12 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:12.360+0000 7f7bfd618200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T07:14:13.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:13.204+0000 7f7bfd618200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T07:14:14.152 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:13.822+0000 7f7bfd618200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T07:14:14.153 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:13.926+0000 7f7bfd618200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T07:14:14.153 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:14:14 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:14:14.150+0000 7f7bfd618200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T07:14:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:15 smithi007 ceph-mon[29098]: Active manager daemon y restarted 2024-02-05T07:14:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:15 smithi007 ceph-mon[29098]: Activating manager daemon y 2024-02-05T07:14:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:15 smithi007 ceph-mon[29098]: osdmap e3: 0 total, 0 up, 0 in 2024-02-05T07:14:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:15 smithi007 ceph-mon[29098]: mgrmap e10: y(active, starting, since 0.00325996s) 2024-02-05T07:14:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:14:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T07:14:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T07:14:15.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T07:14:15.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T07:14:15.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:15 smithi007 ceph-mon[29098]: Manager daemon y is now available 2024-02-05T07:14:15.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:14:15.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:14:15.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T07:14:15.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T07:14:16.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:16 smithi007 ceph-mon[29098]: [05/Feb/2024:07:14:14] ENGINE Bus STARTING 2024-02-05T07:14:16.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:16 smithi007 ceph-mon[29098]: [05/Feb/2024:07:14:14] ENGINE Serving on http://172.21.15.7:8765 2024-02-05T07:14:16.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:16 smithi007 ceph-mon[29098]: [05/Feb/2024:07:14:14] ENGINE Serving on https://172.21.15.7:7150 2024-02-05T07:14:16.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:16 smithi007 ceph-mon[29098]: [05/Feb/2024:07:14:14] ENGINE Bus STARTED 2024-02-05T07:14:16.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:16 smithi007 ceph-mon[29098]: mgrmap e11: y(active, since 1.00636s) 2024-02-05T07:14:16.160 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-02-05T07:14:16.160 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-05T07:14:16.161 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-05T07:14:16.161 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-02-05T07:14:16.161 INFO:teuthology.orchestra.run.smithi007.stdout:mgr epoch 9 is available 2024-02-05T07:14:16.161 INFO:teuthology.orchestra.run.smithi007.stdout:Generating a dashboard self-signed certificate... 2024-02-05T07:14:17.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:17 smithi007 ceph-mon[29098]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-05T07:14:17.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:17 smithi007 ceph-mon[29098]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-05T07:14:17.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:17.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:17.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:17.558 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-05T07:14:17.558 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial admin user... 2024-02-05T07:14:18.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:18 smithi007 ceph-mon[29098]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:14:18.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:18 smithi007 ceph-mon[29098]: mgrmap e12: y(active, since 2s) 2024-02-05T07:14:18.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$xvRXBUQlxyfDVhUktT2YHeJTaN7u16BCyRukqaFKRBCxvmMEZ1Yha", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707117258, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-05T07:14:18.897 INFO:teuthology.orchestra.run.smithi007.stdout:Fetching dashboard port number... 2024-02-05T07:14:19.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:19 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:19 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:20.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:20 smithi007 ceph-mon[29098]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:14:20.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:20 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2405783184' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-05T07:14:20.398 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 8443 2024-02-05T07:14:20.407 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-05T07:14:20.407 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-02-05T07:14:20.414 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-05T07:14:20.414 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-02-05T07:14:20.414 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-02-05T07:14:20.414 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-05T07:14:20.416 INFO:teuthology.orchestra.run.smithi007.stdout:Ceph Dashboard is now available at: 2024-02-05T07:14:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:14:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: URL: https://smithi007.front.sepia.ceph.com:8443/ 2024-02-05T07:14:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: User: admin 2024-02-05T07:14:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: Password: qqozfqa7dn 2024-02-05T07:14:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:14:20.417 INFO:teuthology.orchestra.run.smithi007.stdout:Saving cluster configuration to /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config directory 2024-02-05T07:14:20.417 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling autotune for osd_memory_target 2024-02-05T07:14:22.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:22 smithi007 ceph-mon[29098]: mgrmap e13: y(active, since 6s) 2024-02-05T07:14:23.517 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-05T07:14:23.517 INFO:teuthology.orchestra.run.smithi007.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-05T07:14:23.517 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:14:23.517 INFO:teuthology.orchestra.run.smithi007.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-05T07:14:23.517 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:14:23.517 INFO:teuthology.orchestra.run.smithi007.stdout:Or, if you are only running a single cluster on this host: 2024-02-05T07:14:23.517 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:14:23.517 INFO:teuthology.orchestra.run.smithi007.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-05T07:14:23.517 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:14:23.517 INFO:teuthology.orchestra.run.smithi007.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-05T07:14:23.517 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:14:23.518 INFO:teuthology.orchestra.run.smithi007.stdout: ceph telemetry on 2024-02-05T07:14:23.518 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:14:23.518 INFO:teuthology.orchestra.run.smithi007.stdout:For more information see: 2024-02-05T07:14:23.518 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:14:23.518 INFO:teuthology.orchestra.run.smithi007.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-02-05T07:14:23.518 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:14:23.518 INFO:teuthology.orchestra.run.smithi007.stdout:Bootstrap complete. 2024-02-05T07:14:23.560 INFO:tasks.cephadm:Fetching config... 2024-02-05T07:14:23.560 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:14:23.560 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-05T07:14:23.575 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-05T07:14:23.575 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:14:23.575 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-05T07:14:23.631 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-05T07:14:23.631 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:14:23.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-05T07:14:23.698 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-05T07:14:23.698 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:14:23.698 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-05T07:14:23.753 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-05T07:14:23.753 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9+aEAWMpp3drRu/DD6s/Rdzu+RbOPjvlvXXLOZJbOI46T2o1KUsQcYQ1ZLBTXHeK5XLX7iHIFsje92FuOy2562tZwm9jdK4m2w/QNCmt98OOGhzr6TkaLhGrvjJyZ1fosNH2Jx0Lk+Obg/T8gkvIIEUnP0nvQAK9yBJCX5bL85BaiMljP+kDQDSKTYX51GGjPymneCHDCLIv5Y0E6Lb0V4hUCe963S6iza1Jbi0diFaV6XMUkm5h8/KnhChM4MZsNF1i0Zm3Vc36nJr5V1EWdsTq0YEK+vQipoc+ueDvRz87WfbJWsNbXlIdQdqv3xzPnTZLA6YPJs2NpmeGdyxOIQ0q/KFklS+OjVj++miKxj9eq9bPckX88ahMJxu74DqaBRt/9lxtEy96AYgjraoSyHmR+yF3hYQo+jUClceR5fNiZgEQmSXniKkxUDjwbc6j5XuKuqHcFUEQfDi4IFn+mU0f6E/CGrU17jJ6Oy94SpDI0Oxleb4oTw5qQLnFo+Kc= ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-05T07:14:23.833 INFO:teuthology.orchestra.run.smithi007.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9+aEAWMpp3drRu/DD6s/Rdzu+RbOPjvlvXXLOZJbOI46T2o1KUsQcYQ1ZLBTXHeK5XLX7iHIFsje92FuOy2562tZwm9jdK4m2w/QNCmt98OOGhzr6TkaLhGrvjJyZ1fosNH2Jx0Lk+Obg/T8gkvIIEUnP0nvQAK9yBJCX5bL85BaiMljP+kDQDSKTYX51GGjPymneCHDCLIv5Y0E6Lb0V4hUCe963S6iza1Jbi0diFaV6XMUkm5h8/KnhChM4MZsNF1i0Zm3Vc36nJr5V1EWdsTq0YEK+vQipoc+ueDvRz87WfbJWsNbXlIdQdqv3xzPnTZLA6YPJs2NpmeGdyxOIQ0q/KFklS+OjVj++miKxj9eq9bPckX88ahMJxu74DqaBRt/9lxtEy96AYgjraoSyHmR+yF3hYQo+jUClceR5fNiZgEQmSXniKkxUDjwbc6j5XuKuqHcFUEQfDi4IFn+mU0f6E/CGrU17jJ6Oy94SpDI0Oxleb4oTw5qQLnFo+Kc= ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:14:23.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9+aEAWMpp3drRu/DD6s/Rdzu+RbOPjvlvXXLOZJbOI46T2o1KUsQcYQ1ZLBTXHeK5XLX7iHIFsje92FuOy2562tZwm9jdK4m2w/QNCmt98OOGhzr6TkaLhGrvjJyZ1fosNH2Jx0Lk+Obg/T8gkvIIEUnP0nvQAK9yBJCX5bL85BaiMljP+kDQDSKTYX51GGjPymneCHDCLIv5Y0E6Lb0V4hUCe963S6iza1Jbi0diFaV6XMUkm5h8/KnhChM4MZsNF1i0Zm3Vc36nJr5V1EWdsTq0YEK+vQipoc+ueDvRz87WfbJWsNbXlIdQdqv3xzPnTZLA6YPJs2NpmeGdyxOIQ0q/KFklS+OjVj++miKxj9eq9bPckX88ahMJxu74DqaBRt/9lxtEy96AYgjraoSyHmR+yF3hYQo+jUClceR5fNiZgEQmSXniKkxUDjwbc6j5XuKuqHcFUEQfDi4IFn+mU0f6E/CGrU17jJ6Oy94SpDI0Oxleb4oTw5qQLnFo+Kc= ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-05T07:14:23.894 INFO:teuthology.orchestra.run.smithi090.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9+aEAWMpp3drRu/DD6s/Rdzu+RbOPjvlvXXLOZJbOI46T2o1KUsQcYQ1ZLBTXHeK5XLX7iHIFsje92FuOy2562tZwm9jdK4m2w/QNCmt98OOGhzr6TkaLhGrvjJyZ1fosNH2Jx0Lk+Obg/T8gkvIIEUnP0nvQAK9yBJCX5bL85BaiMljP+kDQDSKTYX51GGjPymneCHDCLIv5Y0E6Lb0V4hUCe963S6iza1Jbi0diFaV6XMUkm5h8/KnhChM4MZsNF1i0Zm3Vc36nJr5V1EWdsTq0YEK+vQipoc+ueDvRz87WfbJWsNbXlIdQdqv3xzPnTZLA6YPJs2NpmeGdyxOIQ0q/KFklS+OjVj++miKxj9eq9bPckX88ahMJxu74DqaBRt/9lxtEy96AYgjraoSyHmR+yF3hYQo+jUClceR5fNiZgEQmSXniKkxUDjwbc6j5XuKuqHcFUEQfDi4IFn+mU0f6E/CGrU17jJ6Oy94SpDI0Oxleb4oTw5qQLnFo+Kc= ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:14:23.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-05T07:14:24.153 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:14:24.274 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:23 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2417448200' entity='client.admin' 2024-02-05T07:14:26.921 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-05T07:14:26.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-05T07:14:27.160 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:14:27.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:27 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1651636262' entity='client.admin' 2024-02-05T07:14:27.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:27 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:27.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:27 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:27.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:27 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:14:27.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:27 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:27.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:27 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:14:27.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:27 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:29.614 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi090 2024-02-05T07:14:29.614 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-05T07:14:29.614 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/etc/ceph/ceph.conf 2024-02-05T07:14:29.632 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-05T07:14:29.633 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-05T07:14:29.690 INFO:tasks.cephadm:Adding host smithi090 to orchestrator... 2024-02-05T07:14:29.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch host add smithi090 2024-02-05T07:14:29.929 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:14:30.139 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:29 smithi007 ceph-mon[29098]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:14:30.139 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:29 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:30.139 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:29 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:14:30.139 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:29 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:14:30.139 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:29 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:14:30.139 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:29 smithi007 ceph-mon[29098]: Updating smithi007:/etc/ceph/ceph.conf 2024-02-05T07:14:30.139 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:29 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:30.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:29 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:30.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:29 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:31.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:30 smithi007 ceph-mon[29098]: Updating smithi007:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:14:31.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:30 smithi007 ceph-mon[29098]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-02-05T07:14:31.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:30 smithi007 ceph-mon[29098]: Updating smithi007:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T07:14:32.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:32 smithi007 ceph-mon[29098]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi090", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:14:33.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:33 smithi007 ceph-mon[29098]: Deploying cephadm binary to smithi090 2024-02-05T07:14:36.116 INFO:teuthology.orchestra.run.smithi007.stdout:Added host 'smithi090' with addr '172.21.15.90' 2024-02-05T07:14:36.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:36 smithi007 ceph-mon[29098]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:36.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch host ls --format=json 2024-02-05T07:14:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:14:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:37 smithi007 ceph-mon[29098]: Added host smithi090 2024-02-05T07:14:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:14:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:37 smithi007 ceph-mon[29098]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:37.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:38.330 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:38 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:39.089 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:14:39.089 INFO:teuthology.orchestra.run.smithi007.stdout:[{"addr": "172.21.15.7", "hostname": "smithi007", "labels": [], "status": ""}, {"addr": "172.21.15.90", "hostname": "smithi090", "labels": [], "status": ""}] 2024-02-05T07:14:39.664 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-05T07:14:39.665 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd crush tunables default 2024-02-05T07:14:39.901 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:14:39.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:39 smithi007 ceph-mon[29098]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:39.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:39 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:41.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:40 smithi007 ceph-mon[29098]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:14:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-02-05T07:14:42.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:41 smithi007 ceph-mon[29098]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:42.034 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:41 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1655488462' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-05T07:14:42.309 INFO:tasks.cephadm:Adding mon.a on smithi007 2024-02-05T07:14:42.310 INFO:tasks.cephadm:Adding mon.c on smithi007 2024-02-05T07:14:42.310 INFO:tasks.cephadm:Adding mon.b on smithi090 2024-02-05T07:14:42.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch apply mon '3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi090:172.21.15.90=b' 2024-02-05T07:14:42.542 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:14:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:42 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1655488462' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-05T07:14:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:42 smithi007 ceph-mon[29098]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T07:14:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi090", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:14:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:14:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:14:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:42 smithi007 ceph-mon[29098]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-05T07:14:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:42 smithi007 ceph-mon[29098]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:14:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:43.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:44.151 INFO:teuthology.orchestra.run.smithi090.stdout:Scheduled mon update... 2024-02-05T07:14:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:43 smithi007 ceph-mon[29098]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-02-05T07:14:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:43 smithi007 ceph-mon[29098]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:43 smithi007 ceph-mon[29098]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T07:14:44.654 DEBUG:teuthology.orchestra.run.smithi007:mon.c> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mon.c.service 2024-02-05T07:14:44.657 DEBUG:teuthology.orchestra.run.smithi090:mon.b> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mon.b.service 2024-02-05T07:14:44.660 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-05T07:14:44.660 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph mon dump -f json 2024-02-05T07:14:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:45 smithi007 ceph-mon[29098]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi090:172.21.15.90=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:14:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:45 smithi007 ceph-mon[29098]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi090:172.21.15.90=b;count:3 2024-02-05T07:14:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:14:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:14:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:14:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:14:45.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:14:45.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:45 smithi007 ceph-mon[29098]: Deploying daemon mon.b on smithi090 2024-02-05T07:14:45.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:45 smithi007 ceph-mon[29098]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:46.234 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:14:47.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:47 smithi007 ceph-mon[29098]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:48.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:48 smithi090 systemd[1]: Starting Ceph mon.b for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:14:48.572 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:14:48.572 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"e3e7059e-c3f5-11ee-95b6-87774f69a715","modified":"2024-02-05T07:13:11.432869Z","created":"2024-02-05T07:13:11.432869Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-05T07:14:48.574 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-05T07:14:48.866 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:48 smithi090 podman[33178]: 2024-02-05 07:14:48.526256686 +0000 UTC m=+0.114271282 container create f4c0e28b73aeff5d516794b188f91ef8e84a102d54f9fb8cdc06fdca33ca57cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-b, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T07:14:48.866 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:48 smithi090 podman[33178]: 2024-02-05 07:14:48.429926684 +0000 UTC m=+0.017941279 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T07:14:49.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:49 smithi007 ceph-mon[29098]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:49.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:49 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/482978831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:14:49.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 podman[33178]: 2024-02-05 07:14:49.459596354 +0000 UTC m=+1.047610950 container init f4c0e28b73aeff5d516794b188f91ef8e84a102d54f9fb8cdc06fdca33ca57cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-05T07:14:49.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 podman[33178]: 2024-02-05 07:14:49.463052739 +0000 UTC m=+1.051067337 container start f4c0e28b73aeff5d516794b188f91ef8e84a102d54f9fb8cdc06fdca33ca57cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-05T07:14:49.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: set uid:gid to 167:167 (ceph:ceph) 2024-02-05T07:14:49.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev), process ceph-mon, pid 2 2024-02-05T07:14:49.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: pidfile_write: ignore empty --pid-file 2024-02-05T07:14:49.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: load: jerasure load: lrc 2024-02-05T07:14:49.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: RocksDB version: 7.9.2 2024-02-05T07:14:49.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Git sha 0 2024-02-05T07:14:49.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Compile date 2024-02-04 23:11:06 2024-02-05T07:14:49.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: DB SUMMARY 2024-02-05T07:14:49.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: DB Session ID: R7TH7CXSO03MOXJGYIQV 2024-02-05T07:14:49.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: CURRENT file: CURRENT 2024-02-05T07:14:49.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: IDENTITY file: IDENTITY 2024-02-05T07:14:49.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-05T07:14:49.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-05T07:14:49.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-02-05T07:14:49.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.error_if_exists: 0 2024-02-05T07:14:49.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.create_if_missing: 0 2024-02-05T07:14:49.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.paranoid_checks: 1 2024-02-05T07:14:49.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-05T07:14:49.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-05T07:14:49.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-05T07:14:49.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.env: 0x565149a11ae0 2024-02-05T07:14:49.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.fs: PosixFileSystem 2024-02-05T07:14:49.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.info_log: 0x56514b6678c0 2024-02-05T07:14:49.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_file_opening_threads: 16 2024-02-05T07:14:49.778 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.statistics: (nil) 2024-02-05T07:14:49.779 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.use_fsync: 0 2024-02-05T07:14:49.779 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_log_file_size: 0 2024-02-05T07:14:49.779 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-05T07:14:49.779 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-05T07:14:49.779 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.keep_log_file_num: 1000 2024-02-05T07:14:49.779 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.recycle_log_file_num: 0 2024-02-05T07:14:49.779 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.allow_fallocate: 1 2024-02-05T07:14:49.779 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.allow_mmap_reads: 0 2024-02-05T07:14:49.779 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.allow_mmap_writes: 0 2024-02-05T07:14:49.779 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.use_direct_reads: 0 2024-02-05T07:14:49.779 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-05T07:14:49.779 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.create_missing_column_families: 0 2024-02-05T07:14:49.779 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.db_log_dir: 2024-02-05T07:14:49.779 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.wal_dir: 2024-02-05T07:14:49.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-05T07:14:49.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-05T07:14:49.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-05T07:14:49.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-05T07:14:49.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-05T07:14:49.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-05T07:14:49.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.advise_random_on_open: 1 2024-02-05T07:14:49.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.db_write_buffer_size: 0 2024-02-05T07:14:49.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.write_buffer_manager: 0x56514b63b400 2024-02-05T07:14:49.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-05T07:14:49.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-05T07:14:49.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-05T07:14:49.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.rate_limiter: (nil) 2024-02-05T07:14:49.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-05T07:14:49.781 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.wal_recovery_mode: 2 2024-02-05T07:14:49.781 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.enable_thread_tracking: 0 2024-02-05T07:14:49.781 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.enable_pipelined_write: 0 2024-02-05T07:14:49.781 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.unordered_write: 0 2024-02-05T07:14:49.781 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-05T07:14:49.781 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-05T07:14:49.781 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-05T07:14:49.781 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-05T07:14:49.781 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.row_cache: None 2024-02-05T07:14:49.781 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.wal_filter: None 2024-02-05T07:14:49.781 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-05T07:14:49.781 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.allow_ingest_behind: 0 2024-02-05T07:14:49.781 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.two_write_queues: 0 2024-02-05T07:14:49.781 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.manual_wal_flush: 0 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.wal_compression: 0 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.atomic_flush: 0 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.log_readahead_size: 0 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.best_efforts_recovery: 0 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.allow_data_in_errors: 0 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.db_host_id: __hostname__ 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_background_jobs: 2 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_background_compactions: -1 2024-02-05T07:14:49.782 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_subcompactions: 1 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_total_wal_size: 0 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_open_files: -1 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.bytes_per_sync: 0 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compaction_readahead_size: 0 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_background_flushes: -1 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Compression algorithms supported: 2024-02-05T07:14:49.783 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-05T07:14:49.784 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: kZSTD supported: 0 2024-02-05T07:14:49.784 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: kXpressCompression supported: 0 2024-02-05T07:14:49.784 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: kLZ4HCCompression supported: 1 2024-02-05T07:14:49.784 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: kZlibCompression supported: 1 2024-02-05T07:14:49.784 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: kSnappyCompression supported: 1 2024-02-05T07:14:49.784 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: kLZ4Compression supported: 1 2024-02-05T07:14:49.784 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: kBZip2Compression supported: 0 2024-02-05T07:14:49.784 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-05T07:14:49.784 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-05T07:14:49.784 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-02-05T07:14:49.784 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-05T07:14:49.784 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-05T07:14:49.784 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.merge_operator: 2024-02-05T07:14:49.784 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compaction_filter: None 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compaction_filter_factory: None 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.sst_partitioner_factory: None 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56514b6679c0) 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout: cache_index_and_filter_blocks: 1 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout: pin_top_level_index_and_filter: 1 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout: index_type: 0 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout: data_block_index_type: 0 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout: index_shortening: 1 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout: checksum: 4 2024-02-05T07:14:49.785 INFO:journalctl@ceph.mon.b.smithi090.stdout: no_block_cache: 0 2024-02-05T07:14:49.786 INFO:journalctl@ceph.mon.b.smithi090.stdout: block_cache: 0x56514b66b090 2024-02-05T07:14:49.786 INFO:journalctl@ceph.mon.b.smithi090.stdout: block_cache_name: BinnedLRUCache 2024-02-05T07:14:49.786 INFO:journalctl@ceph.mon.b.smithi090.stdout: block_cache_options: 2024-02-05T07:14:49.786 INFO:journalctl@ceph.mon.b.smithi090.stdout: capacity : 536870912 2024-02-05T07:14:49.786 INFO:journalctl@ceph.mon.b.smithi090.stdout: num_shard_bits : 4 2024-02-05T07:14:49.786 INFO:journalctl@ceph.mon.b.smithi090.stdout: strict_capacity_limit : 0 2024-02-05T07:14:49.786 INFO:journalctl@ceph.mon.b.smithi090.stdout: high_pri_pool_ratio: 0.000 2024-02-05T07:14:49.786 INFO:journalctl@ceph.mon.b.smithi090.stdout: block_cache_compressed: (nil) 2024-02-05T07:14:49.786 INFO:journalctl@ceph.mon.b.smithi090.stdout: persistent_cache: (nil) 2024-02-05T07:14:49.786 INFO:journalctl@ceph.mon.b.smithi090.stdout: block_size: 4096 2024-02-05T07:14:49.786 INFO:journalctl@ceph.mon.b.smithi090.stdout: block_size_deviation: 10 2024-02-05T07:14:49.786 INFO:journalctl@ceph.mon.b.smithi090.stdout: block_restart_interval: 16 2024-02-05T07:14:49.786 INFO:journalctl@ceph.mon.b.smithi090.stdout: index_block_restart_interval: 1 2024-02-05T07:14:49.786 INFO:journalctl@ceph.mon.b.smithi090.stdout: metadata_block_size: 4096 2024-02-05T07:14:49.787 INFO:journalctl@ceph.mon.b.smithi090.stdout: partition_filters: 0 2024-02-05T07:14:49.787 INFO:journalctl@ceph.mon.b.smithi090.stdout: use_delta_encoding: 1 2024-02-05T07:14:49.787 INFO:journalctl@ceph.mon.b.smithi090.stdout: filter_policy: bloomfilter 2024-02-05T07:14:49.787 INFO:journalctl@ceph.mon.b.smithi090.stdout: whole_key_filtering: 1 2024-02-05T07:14:49.787 INFO:journalctl@ceph.mon.b.smithi090.stdout: verify_compression: 0 2024-02-05T07:14:49.787 INFO:journalctl@ceph.mon.b.smithi090.stdout: read_amp_bytes_per_bit: 0 2024-02-05T07:14:49.787 INFO:journalctl@ceph.mon.b.smithi090.stdout: format_version: 5 2024-02-05T07:14:49.787 INFO:journalctl@ceph.mon.b.smithi090.stdout: enable_index_compression: 1 2024-02-05T07:14:49.787 INFO:journalctl@ceph.mon.b.smithi090.stdout: block_align: 0 2024-02-05T07:14:49.787 INFO:journalctl@ceph.mon.b.smithi090.stdout: max_auto_readahead_size: 262144 2024-02-05T07:14:49.787 INFO:journalctl@ceph.mon.b.smithi090.stdout: prepopulate_block_cache: 0 2024-02-05T07:14:49.787 INFO:journalctl@ceph.mon.b.smithi090.stdout: initial_auto_readahead_size: 8192 2024-02-05T07:14:49.787 INFO:journalctl@ceph.mon.b.smithi090.stdout: num_file_reads_for_auto_readahead: 2 2024-02-05T07:14:49.787 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.write_buffer_size: 33554432 2024-02-05T07:14:49.788 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_write_buffer_number: 2 2024-02-05T07:14:49.788 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compression: NoCompression 2024-02-05T07:14:49.788 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.bottommost_compression: Disabled 2024-02-05T07:14:49.788 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.prefix_extractor: nullptr 2024-02-05T07:14:49.788 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-05T07:14:49.788 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.num_levels: 7 2024-02-05T07:14:49.788 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-05T07:14:49.788 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-05T07:14:49.788 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-05T07:14:49.788 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-05T07:14:49.788 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-05T07:14:49.788 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-05T07:14:49.788 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-05T07:14:49.789 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-05T07:14:49.789 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-05T07:14:49.789 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-05T07:14:49.789 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-05T07:14:49.789 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-05T07:14:49.789 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-05T07:14:49.789 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compression_opts.level: 32767 2024-02-05T07:14:49.789 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compression_opts.strategy: 0 2024-02-05T07:14:49.789 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-05T07:14:49.789 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-05T07:14:49.789 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-05T07:14:49.789 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-05T07:14:49.789 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compression_opts.enabled: false 2024-02-05T07:14:49.789 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-05T07:14:49.790 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-05T07:14:49.790 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-05T07:14:49.790 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-05T07:14:49.790 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.target_file_size_base: 67108864 2024-02-05T07:14:49.790 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-05T07:14:49.790 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-05T07:14:49.790 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-05T07:14:49.790 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-05T07:14:49.790 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-05T07:14:49.790 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-05T07:14:49.790 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-05T07:14:49.790 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-05T07:14:49.790 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-05T07:14:49.790 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-05T07:14:49.791 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-05T07:14:49.791 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-05T07:14:49.791 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-05T07:14:49.791 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-05T07:14:49.791 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.arena_block_size: 1048576 2024-02-05T07:14:49.791 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-05T07:14:49.791 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-05T07:14:49.791 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.disable_auto_compactions: 0 2024-02-05T07:14:49.791 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-05T07:14:49.791 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-05T07:14:49.791 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-05T07:14:49.791 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-05T07:14:49.791 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-05T07:14:49.791 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-05T07:14:49.792 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-05T07:14:49.792 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-05T07:14:49.792 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-05T07:14:49.792 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-05T07:14:49.792 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.table_properties_collectors: 2024-02-05T07:14:49.792 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.inplace_update_support: 0 2024-02-05T07:14:49.792 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-05T07:14:49.792 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-05T07:14:49.792 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-05T07:14:49.792 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-05T07:14:49.792 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.bloom_locality: 0 2024-02-05T07:14:49.792 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.max_successive_merges: 0 2024-02-05T07:14:49.792 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-05T07:14:49.792 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.paranoid_file_checks: 0 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.force_consistency_checks: 1 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.report_bg_io_stats: 0 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.ttl: 2592000 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.enable_blob_files: false 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.min_blob_size: 0 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.blob_file_size: 268435456 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.blob_file_starting_level: 0 2024-02-05T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-05T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-05T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-05T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a5ef6bd2-fdf1-4ebf-a83a-642bd1d43ded 2024-02-05T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707117289504854, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-05T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-05T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707117289505296, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707117289, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a5ef6bd2-fdf1-4ebf-a83a-642bd1d43ded", "db_session_id": "R7TH7CXSO03MOXJGYIQV", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-05T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707117289505377, "job": 1, "event": "recovery_finished"} 2024-02-05T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-05T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-05T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56514b75c000 2024-02-05T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: DB pointer 0x56514b746000 2024-02-05T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-05T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-05T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi090.stdout: ** DB Stats ** 2024-02-05T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi090.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-05T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi090.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi090.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi090.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-05T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi090.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi090.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi090.stdout: 2024-02-05T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi090.stdout: ** Compaction Stats [default] ** 2024-02-05T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi090.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi090.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-05T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi090.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi090.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi090.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T07:14:49.796 INFO:journalctl@ceph.mon.b.smithi090.stdout: 2024-02-05T07:14:49.796 INFO:journalctl@ceph.mon.b.smithi090.stdout: ** Compaction Stats [default] ** 2024-02-05T07:14:49.796 INFO:journalctl@ceph.mon.b.smithi090.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T07:14:49.796 INFO:journalctl@ceph.mon.b.smithi090.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-05T07:14:49.796 INFO:journalctl@ceph.mon.b.smithi090.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T07:14:49.796 INFO:journalctl@ceph.mon.b.smithi090.stdout: 2024-02-05T07:14:49.796 INFO:journalctl@ceph.mon.b.smithi090.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-05T07:14:49.796 INFO:journalctl@ceph.mon.b.smithi090.stdout: 2024-02-05T07:14:49.796 INFO:journalctl@ceph.mon.b.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T07:14:49.796 INFO:journalctl@ceph.mon.b.smithi090.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-05T07:14:49.796 INFO:journalctl@ceph.mon.b.smithi090.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-05T07:14:49.796 INFO:journalctl@ceph.mon.b.smithi090.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-05T07:14:49.796 INFO:journalctl@ceph.mon.b.smithi090.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-05T07:14:49.796 INFO:journalctl@ceph.mon.b.smithi090.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout: Block cache BinnedLRUCache@0x56514b66b090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout: 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: using public_addr v2:172.21.15.90:0/0 -> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: starting mon.b rank -1 at public addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] at bind addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b@-1(???) e0 preinit fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b@-1(synchronizing).mds e1 new map 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b@-1(synchronizing).mds e1 print_map 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout: e1 2024-02-05T07:14:49.797 INFO:journalctl@ceph.mon.b.smithi090.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-05T07:14:49.798 INFO:journalctl@ceph.mon.b.smithi090.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-05T07:14:49.798 INFO:journalctl@ceph.mon.b.smithi090.stdout: legacy client fscid: -1 2024-02-05T07:14:49.798 INFO:journalctl@ceph.mon.b.smithi090.stdout: 2024-02-05T07:14:49.798 INFO:journalctl@ceph.mon.b.smithi090.stdout: No filesystems configured 2024-02-05T07:14:49.798 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-05T07:14:49.798 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-05T07:14:49.798 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-05T07:14:49.798 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-05T07:14:49.798 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-05T07:14:49.798 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-05T07:14:49.798 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-05T07:14:49.798 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T07:14:49.798 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T07:14:49.798 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T07:14:49.799 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:14:49.799 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:49.799 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1655488462' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-05T07:14:49.799 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1655488462' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-05T07:14:49.799 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T07:14:49.799 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:49.799 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:49.799 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:49.799 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:49.799 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi090", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:14:49.799 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:14:49.799 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:14:49.799 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-05T07:14:49.799 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:14:49.800 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:49.800 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:49.800 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:49.800 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-02-05T07:14:49.800 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:49.800 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T07:14:49.800 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi090:172.21.15.90=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:14:49.800 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi090:172.21.15.90=b;count:3 2024-02-05T07:14:49.800 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:49.800 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:14:49.800 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:14:49.800 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:14:49.800 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:49.800 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:14:49.801 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:14:49.801 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: Deploying daemon mon.b on smithi090 2024-02-05T07:14:49.801 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:49.801 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:49.801 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:49.801 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/482978831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:14:49.801 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 ceph-mon[33213]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-05T07:14:49.801 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 bash[33178]: f4c0e28b73aeff5d516794b188f91ef8e84a102d54f9fb8cdc06fdca33ca57cd 2024-02-05T07:14:49.801 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:49 smithi090 systemd[1]: Started Ceph mon.b for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:14:50.380 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-05T07:14:50.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph mon dump -f json 2024-02-05T07:14:50.629 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:14:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:14:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: mon.a calling monitor election 2024-02-05T07:14:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: mon.b calling monitor election 2024-02-05T07:14:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:54.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-05T07:14:54.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T07:14:54.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: fsmap 2024-02-05T07:14:54.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T07:14:54.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: mgrmap e13: y(active, since 40s) 2024-02-05T07:14:54.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: overall HEALTH_OK 2024-02-05T07:14:54.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:54.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:54 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:54.977 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:14:54.978 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:54.978 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: mon.a calling monitor election 2024-02-05T07:14:54.978 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:54.978 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:54.978 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:54.978 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: mon.b calling monitor election 2024-02-05T07:14:54.978 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:54.978 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:54.978 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:54.978 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:54.978 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-05T07:14:54.978 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T07:14:54.978 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: fsmap 2024-02-05T07:14:54.978 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T07:14:54.979 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: mgrmap e13: y(active, since 40s) 2024-02-05T07:14:54.979 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: overall HEALTH_OK 2024-02-05T07:14:54.979 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:54.979 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:54 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:55.321 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:14:55.321 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":2,"fsid":"e3e7059e-c3f5-11ee-95b6-87774f69a715","modified":"2024-02-05T07:14:49.528452Z","created":"2024-02-05T07:13:11.432869Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-05T07:14:55.323 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 2 2024-02-05T07:14:55.878 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:55 smithi090 ceph-mon[33213]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:55.878 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:55 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:55.878 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:55 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:14:55.878 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:55 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:14:55.878 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:55 smithi090 ceph-mon[33213]: Deploying daemon mon.c on smithi007 2024-02-05T07:14:55.878 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:55 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/3759483065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:14:55.878 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:55 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:55.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:55 smithi007 ceph-mon[29098]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:55.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:55 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:55.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:55 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:14:55.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:55 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:14:55.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:55 smithi007 ceph-mon[29098]: Deploying daemon mon.c on smithi007 2024-02-05T07:14:55.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:55 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/3759483065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:14:55.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:55 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:56.903 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-05T07:14:56.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph mon dump -f json 2024-02-05T07:14:57.148 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:14:57.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:14:57 smithi007 ceph-mon[29098]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:58.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:14:57 smithi090 ceph-mon[33213]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:58.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 podman[37131]: 2024-02-05 07:14:58.162705214 +0000 UTC m=+0.361423934 container start 5ab95d0c35ea1725621fd116554fbbf6f7592a8f64a66d8c734ea45039ad4f74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-c, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T07:14:58.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: set uid:gid to 167:167 (ceph:ceph) 2024-02-05T07:14:58.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev), process ceph-mon, pid 2 2024-02-05T07:14:58.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: pidfile_write: ignore empty --pid-file 2024-02-05T07:14:58.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: load: jerasure load: lrc 2024-02-05T07:14:58.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: RocksDB version: 7.9.2 2024-02-05T07:14:58.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Git sha 0 2024-02-05T07:14:58.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Compile date 2024-02-04 23:11:06 2024-02-05T07:14:58.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: DB SUMMARY 2024-02-05T07:14:58.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: DB Session ID: S4VTV11ZUR7R1VEQOJ57 2024-02-05T07:14:58.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: CURRENT file: CURRENT 2024-02-05T07:14:58.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: IDENTITY file: IDENTITY 2024-02-05T07:14:58.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-05T07:14:58.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-05T07:14:58.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-02-05T07:14:58.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.error_if_exists: 0 2024-02-05T07:14:58.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.create_if_missing: 0 2024-02-05T07:14:58.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.paranoid_checks: 1 2024-02-05T07:14:58.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-05T07:14:58.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-05T07:14:58.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-05T07:14:58.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.env: 0x55eba67b2ae0 2024-02-05T07:14:58.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.fs: PosixFileSystem 2024-02-05T07:14:58.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.info_log: 0x55eba8eb78c0 2024-02-05T07:14:58.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_file_opening_threads: 16 2024-02-05T07:14:58.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.statistics: (nil) 2024-02-05T07:14:58.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.use_fsync: 0 2024-02-05T07:14:58.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_log_file_size: 0 2024-02-05T07:14:58.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-05T07:14:58.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-05T07:14:58.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.keep_log_file_num: 1000 2024-02-05T07:14:58.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.recycle_log_file_num: 0 2024-02-05T07:14:58.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.allow_fallocate: 1 2024-02-05T07:14:58.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.allow_mmap_reads: 0 2024-02-05T07:14:58.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.allow_mmap_writes: 0 2024-02-05T07:14:58.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.use_direct_reads: 0 2024-02-05T07:14:58.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-05T07:14:58.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.create_missing_column_families: 0 2024-02-05T07:14:58.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.db_log_dir: 2024-02-05T07:14:58.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.wal_dir: 2024-02-05T07:14:58.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-05T07:14:58.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-05T07:14:58.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-05T07:14:58.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-05T07:14:58.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-05T07:14:58.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-05T07:14:58.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.advise_random_on_open: 1 2024-02-05T07:14:58.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.db_write_buffer_size: 0 2024-02-05T07:14:58.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.write_buffer_manager: 0x55eba8e8b400 2024-02-05T07:14:58.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-05T07:14:58.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-05T07:14:58.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-05T07:14:58.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.rate_limiter: (nil) 2024-02-05T07:14:58.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-05T07:14:58.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.wal_recovery_mode: 2 2024-02-05T07:14:58.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.enable_thread_tracking: 0 2024-02-05T07:14:58.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.enable_pipelined_write: 0 2024-02-05T07:14:58.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.unordered_write: 0 2024-02-05T07:14:58.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-05T07:14:58.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-05T07:14:58.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-05T07:14:58.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-05T07:14:58.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.row_cache: None 2024-02-05T07:14:58.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.wal_filter: None 2024-02-05T07:14:58.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-05T07:14:58.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.allow_ingest_behind: 0 2024-02-05T07:14:58.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.two_write_queues: 0 2024-02-05T07:14:58.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.manual_wal_flush: 0 2024-02-05T07:14:58.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.wal_compression: 0 2024-02-05T07:14:58.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.atomic_flush: 0 2024-02-05T07:14:58.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-05T07:14:58.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-05T07:14:58.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-05T07:14:58.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.log_readahead_size: 0 2024-02-05T07:14:58.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-05T07:14:58.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.best_efforts_recovery: 0 2024-02-05T07:14:58.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-05T07:14:58.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-05T07:14:58.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.allow_data_in_errors: 0 2024-02-05T07:14:58.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.db_host_id: __hostname__ 2024-02-05T07:14:58.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-05T07:14:58.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_background_jobs: 2 2024-02-05T07:14:58.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_background_compactions: -1 2024-02-05T07:14:58.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_subcompactions: 1 2024-02-05T07:14:58.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-05T07:14:58.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-05T07:14:58.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-05T07:14:58.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_total_wal_size: 0 2024-02-05T07:14:58.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-05T07:14:58.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-05T07:14:58.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-05T07:14:58.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-05T07:14:58.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_open_files: -1 2024-02-05T07:14:58.683 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.bytes_per_sync: 0 2024-02-05T07:14:58.683 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-05T07:14:58.683 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-05T07:14:58.683 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compaction_readahead_size: 0 2024-02-05T07:14:58.683 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_background_flushes: -1 2024-02-05T07:14:58.683 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Compression algorithms supported: 2024-02-05T07:14:58.683 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-05T07:14:58.683 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: kZSTD supported: 0 2024-02-05T07:14:58.683 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: kXpressCompression supported: 0 2024-02-05T07:14:58.683 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: kLZ4HCCompression supported: 1 2024-02-05T07:14:58.683 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: kZlibCompression supported: 1 2024-02-05T07:14:58.683 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: kSnappyCompression supported: 1 2024-02-05T07:14:58.683 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: kLZ4Compression supported: 1 2024-02-05T07:14:58.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: kBZip2Compression supported: 0 2024-02-05T07:14:58.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-05T07:14:58.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-05T07:14:58.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-02-05T07:14:58.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-05T07:14:58.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-05T07:14:58.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.merge_operator: 2024-02-05T07:14:58.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compaction_filter: None 2024-02-05T07:14:58.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compaction_filter_factory: None 2024-02-05T07:14:58.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.sst_partitioner_factory: None 2024-02-05T07:14:58.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-05T07:14:58.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-05T07:14:58.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eba8eb79c0) 2024-02-05T07:14:58.684 INFO:journalctl@ceph.mon.c.smithi007.stdout: cache_index_and_filter_blocks: 1 2024-02-05T07:14:58.685 INFO:journalctl@ceph.mon.c.smithi007.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-05T07:14:58.685 INFO:journalctl@ceph.mon.c.smithi007.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-05T07:14:58.685 INFO:journalctl@ceph.mon.c.smithi007.stdout: pin_top_level_index_and_filter: 1 2024-02-05T07:14:58.685 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_type: 0 2024-02-05T07:14:58.685 INFO:journalctl@ceph.mon.c.smithi007.stdout: data_block_index_type: 0 2024-02-05T07:14:58.685 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_shortening: 1 2024-02-05T07:14:58.685 INFO:journalctl@ceph.mon.c.smithi007.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-05T07:14:58.685 INFO:journalctl@ceph.mon.c.smithi007.stdout: checksum: 4 2024-02-05T07:14:58.685 INFO:journalctl@ceph.mon.c.smithi007.stdout: no_block_cache: 0 2024-02-05T07:14:58.685 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache: 0x55eba8ebb090 2024-02-05T07:14:58.685 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_name: BinnedLRUCache 2024-02-05T07:14:58.685 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_options: 2024-02-05T07:14:58.685 INFO:journalctl@ceph.mon.c.smithi007.stdout: capacity : 536870912 2024-02-05T07:14:58.685 INFO:journalctl@ceph.mon.c.smithi007.stdout: num_shard_bits : 4 2024-02-05T07:14:58.686 INFO:journalctl@ceph.mon.c.smithi007.stdout: strict_capacity_limit : 0 2024-02-05T07:14:58.686 INFO:journalctl@ceph.mon.c.smithi007.stdout: high_pri_pool_ratio: 0.000 2024-02-05T07:14:58.686 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_compressed: (nil) 2024-02-05T07:14:58.686 INFO:journalctl@ceph.mon.c.smithi007.stdout: persistent_cache: (nil) 2024-02-05T07:14:58.686 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_size: 4096 2024-02-05T07:14:58.686 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_size_deviation: 10 2024-02-05T07:14:58.686 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_restart_interval: 16 2024-02-05T07:14:58.686 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_block_restart_interval: 1 2024-02-05T07:14:58.686 INFO:journalctl@ceph.mon.c.smithi007.stdout: metadata_block_size: 4096 2024-02-05T07:14:58.686 INFO:journalctl@ceph.mon.c.smithi007.stdout: partition_filters: 0 2024-02-05T07:14:58.686 INFO:journalctl@ceph.mon.c.smithi007.stdout: use_delta_encoding: 1 2024-02-05T07:14:58.686 INFO:journalctl@ceph.mon.c.smithi007.stdout: filter_policy: bloomfilter 2024-02-05T07:14:58.686 INFO:journalctl@ceph.mon.c.smithi007.stdout: whole_key_filtering: 1 2024-02-05T07:14:58.686 INFO:journalctl@ceph.mon.c.smithi007.stdout: verify_compression: 0 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout: read_amp_bytes_per_bit: 0 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout: format_version: 5 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout: enable_index_compression: 1 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_align: 0 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout: max_auto_readahead_size: 262144 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout: prepopulate_block_cache: 0 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout: initial_auto_readahead_size: 8192 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout: num_file_reads_for_auto_readahead: 2 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.write_buffer_size: 33554432 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_write_buffer_number: 2 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compression: NoCompression 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.bottommost_compression: Disabled 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.prefix_extractor: nullptr 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.num_levels: 7 2024-02-05T07:14:58.687 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-05T07:14:58.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-05T07:14:58.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-05T07:14:58.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-05T07:14:58.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-05T07:14:58.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-05T07:14:58.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-05T07:14:58.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-05T07:14:58.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-05T07:14:58.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-05T07:14:58.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-05T07:14:58.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-05T07:14:58.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-05T07:14:58.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compression_opts.level: 32767 2024-02-05T07:14:58.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compression_opts.strategy: 0 2024-02-05T07:14:58.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-05T07:14:58.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-05T07:14:58.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-05T07:14:58.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-05T07:14:58.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compression_opts.enabled: false 2024-02-05T07:14:58.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-05T07:14:58.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-05T07:14:58.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-05T07:14:58.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-05T07:14:58.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.target_file_size_base: 67108864 2024-02-05T07:14:58.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.arena_block_size: 1048576 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-05T07:14:58.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.disable_auto_compactions: 0 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.table_properties_collectors: 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.inplace_update_support: 0 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-05T07:14:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-05T07:14:58.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-05T07:14:58.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-05T07:14:58.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.bloom_locality: 0 2024-02-05T07:14:58.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.max_successive_merges: 0 2024-02-05T07:14:58.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-05T07:14:58.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.paranoid_file_checks: 0 2024-02-05T07:14:58.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.force_consistency_checks: 1 2024-02-05T07:14:58.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.report_bg_io_stats: 0 2024-02-05T07:14:58.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.ttl: 2592000 2024-02-05T07:14:58.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-05T07:14:58.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-05T07:14:58.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-05T07:14:58.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.enable_blob_files: false 2024-02-05T07:14:58.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.min_blob_size: 0 2024-02-05T07:14:58.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.blob_file_size: 268435456 2024-02-05T07:14:58.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-05T07:14:58.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-05T07:14:58.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-05T07:14:58.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-05T07:14:58.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-05T07:14:58.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.blob_file_starting_level: 0 2024-02-05T07:14:58.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-05T07:14:58.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-05T07:14:58.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-05T07:14:58.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cb5d1470-4edf-4d1f-b173-7bb82d95deb9 2024-02-05T07:14:58.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707117298192766, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-05T07:14:58.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-05T07:14:58.694 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707117298193319, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707117298, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cb5d1470-4edf-4d1f-b173-7bb82d95deb9", "db_session_id": "S4VTV11ZUR7R1VEQOJ57", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-05T07:14:58.694 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707117298193428, "job": 1, "event": "recovery_finished"} 2024-02-05T07:14:58.694 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-05T07:14:58.694 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-05T07:14:58.694 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55eba8fac000 2024-02-05T07:14:58.694 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: DB pointer 0x55eba8f96000 2024-02-05T07:14:58.694 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-05T07:14:58.694 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-05T07:14:58.694 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** DB Stats ** 2024-02-05T07:14:58.694 INFO:journalctl@ceph.mon.c.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T07:14:58.694 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-05T07:14:58.694 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T07:14:58.694 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T07:14:58.694 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-05T07:14:58.695 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T07:14:58.695 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T07:14:58.695 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-02-05T07:14:58.695 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** Compaction Stats [default] ** 2024-02-05T07:14:58.695 INFO:journalctl@ceph.mon.c.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T07:14:58.695 INFO:journalctl@ceph.mon.c.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-05T07:14:58.695 INFO:journalctl@ceph.mon.c.smithi007.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T07:14:58.695 INFO:journalctl@ceph.mon.c.smithi007.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T07:14:58.695 INFO:journalctl@ceph.mon.c.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T07:14:58.695 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-02-05T07:14:58.695 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** Compaction Stats [default] ** 2024-02-05T07:14:58.695 INFO:journalctl@ceph.mon.c.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T07:14:58.695 INFO:journalctl@ceph.mon.c.smithi007.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-05T07:14:58.696 INFO:journalctl@ceph.mon.c.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T07:14:58.696 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-02-05T07:14:58.696 INFO:journalctl@ceph.mon.c.smithi007.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-05T07:14:58.696 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-02-05T07:14:58.696 INFO:journalctl@ceph.mon.c.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T07:14:58.696 INFO:journalctl@ceph.mon.c.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-05T07:14:58.696 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-05T07:14:58.696 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-05T07:14:58.696 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-05T07:14:58.696 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-05T07:14:58.696 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T07:14:58.696 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T07:14:58.696 INFO:journalctl@ceph.mon.c.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout: Block cache BinnedLRUCache@0x55eba8ebb090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: using public_addrv [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: starting mon.c rank -1 at public addrs [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] at bind addrs [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c@-1(???) e0 preinit fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c@-1(synchronizing).mds e1 new map 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c@-1(synchronizing).mds e1 print_map 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout: e1 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout: legacy client fscid: -1 2024-02-05T07:14:58.697 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout: No filesystems configured 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1655488462' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-05T07:14:58.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1655488462' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi090", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:58.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi090:172.21.15.90=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi090:172.21.15.90=b;count:3 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: Deploying daemon mon.b on smithi090 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/482978831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:14:58.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:14:58.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:58.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.a calling monitor election 2024-02-05T07:14:58.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:58.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:58.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:58.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.b calling monitor election 2024-02-05T07:14:58.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:58.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:58.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:58.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:58.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-05T07:14:58.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T07:14:58.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: fsmap 2024-02-05T07:14:58.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mgrmap e13: y(active, since 40s) 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: overall HEALTH_OK 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: Deploying daemon mon.c on smithi007 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/3759483065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 ceph-mon[37158]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 bash[37131]: 5ab95d0c35ea1725621fd116554fbbf6f7592a8f64a66d8c734ea45039ad4f74 2024-02-05T07:14:58.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:14:58 smithi007 systemd[1]: Started Ceph mon.c for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:15:03.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:03.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:15:03.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: mon.a calling monitor election 2024-02-05T07:15:03.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: mon.b calling monitor election 2024-02-05T07:15:03.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:15:03.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:03.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: mon.c calling monitor election 2024-02-05T07:15:03.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:03.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-05T07:15:03.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T07:15:03.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: fsmap 2024-02-05T07:15:03.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T07:15:03.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: mgrmap e13: y(active, since 49s) 2024-02-05T07:15:03.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: overall HEALTH_OK 2024-02-05T07:15:03.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:03.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:03.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:03.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:03.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:15:03.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: mon.a calling monitor election 2024-02-05T07:15:03.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: mon.b calling monitor election 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: mon.c calling monitor election 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: fsmap 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T07:15:03.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: mgrmap e13: y(active, since 49s) 2024-02-05T07:15:03.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: overall HEALTH_OK 2024-02-05T07:15:03.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:03.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:03.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: mon.a calling monitor election 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: mon.b calling monitor election 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: mon.c calling monitor election 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:03.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-05T07:15:03.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T07:15:03.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: fsmap 2024-02-05T07:15:03.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T07:15:03.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: mgrmap e13: y(active, since 49s) 2024-02-05T07:15:03.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: overall HEALTH_OK 2024-02-05T07:15:03.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:03.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:03.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:04.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:04 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:04.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:04 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:04.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:04 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:04.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:04 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:04 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:04 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:04 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:04 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:04.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:04 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:04.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:04 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:04.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:04 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:15:04.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:04 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:04.882 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-05T07:15:04.882 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":3,"fsid":"e3e7059e-c3f5-11ee-95b6-87774f69a715","modified":"2024-02-05T07:14:58.215445Z","created":"2024-02-05T07:13:11.432869Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3301","nonce":0},{"type":"v1","addr":"172.21.15.7:6790","nonce":0}]},"addr":"172.21.15.7:6790/0","public_addr":"172.21.15.7:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-02-05T07:15:04.884 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 3 2024-02-05T07:15:05.569 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-05T07:15:05.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph config generate-minimal-conf 2024-02-05T07:15:05.819 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:15:05.854 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:05.854 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.855 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.855 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.855 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.855 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:05.855 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:05.855 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: Updating smithi007:/etc/ceph/ceph.conf 2024-02-05T07:15:05.855 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-05T07:15:05.855 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/1774668247' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:15:05.855 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: Updating smithi007:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:15:05.855 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:15:05.855 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.855 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.856 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.856 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.856 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.856 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.856 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.856 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.856 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.856 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:15:05.856 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T07:15:05.856 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:05.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:05.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:05.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:05.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: Updating smithi007:/etc/ceph/ceph.conf 2024-02-05T07:15:05.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-05T07:15:05.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/1774668247' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:15:05.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: Updating smithi007:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:15:05.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:15:05.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:15:05.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T07:15:05.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:05.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:05.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:05.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:05.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: Updating smithi007:/etc/ceph/ceph.conf 2024-02-05T07:15:05.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-05T07:15:05.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/1774668247' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:15:05.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: Updating smithi007:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:15:05.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:15:05.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:05.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:15:05.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T07:15:05.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:06.823 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:06 smithi007 ceph-mon[29098]: Reconfiguring mon.a (unknown last config time)... 2024-02-05T07:15:06.823 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:06 smithi007 ceph-mon[29098]: Reconfiguring daemon mon.a on smithi007 2024-02-05T07:15:06.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:06 smithi007 ceph-mon[37158]: Reconfiguring mon.a (unknown last config time)... 2024-02-05T07:15:06.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:06 smithi007 ceph-mon[37158]: Reconfiguring daemon mon.a on smithi007 2024-02-05T07:15:07.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:06 smithi090 ceph-mon[33213]: Reconfiguring mon.a (unknown last config time)... 2024-02-05T07:15:07.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:06 smithi090 ceph-mon[33213]: Reconfiguring daemon mon.a on smithi007 2024-02-05T07:15:07.913 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:07 smithi007 ceph-mon[29098]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:07.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:07 smithi007 ceph-mon[37158]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:08.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:07 smithi090 ceph-mon[33213]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:08.452 INFO:teuthology.orchestra.run.smithi007.stdout:# minimal ceph.conf for e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:15:08.452 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-02-05T07:15:08.452 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:15:08.452 INFO:teuthology.orchestra.run.smithi007.stdout: mon_host = [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-02-05T07:15:09.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:09.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:09.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[29098]: Reconfiguring mon.c (monmap changed)... 2024-02-05T07:15:09.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:15:09.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T07:15:09.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:09.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[29098]: Reconfiguring daemon mon.c on smithi007 2024-02-05T07:15:09.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2729941894' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:09.106 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:09.106 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:09.106 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[37158]: Reconfiguring mon.c (monmap changed)... 2024-02-05T07:15:09.106 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:15:09.106 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T07:15:09.106 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:09.106 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[37158]: Reconfiguring daemon mon.c on smithi007 2024-02-05T07:15:09.106 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:08 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2729941894' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:08 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:08 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:08 smithi090 ceph-mon[33213]: Reconfiguring mon.c (monmap changed)... 2024-02-05T07:15:09.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:08 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:15:09.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:08 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T07:15:09.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:08 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:09.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:08 smithi090 ceph-mon[33213]: Reconfiguring daemon mon.c on smithi007 2024-02-05T07:15:09.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:08 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2729941894' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:09.624 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-05T07:15:09.624 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:15:09.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.conf 2024-02-05T07:15:09.656 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:15:09.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-05T07:15:09.726 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-05T07:15:09.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.conf 2024-02-05T07:15:09.760 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-05T07:15:09.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-05T07:15:09.831 INFO:tasks.cephadm:Adding mgr.y on smithi007 2024-02-05T07:15:09.832 INFO:tasks.cephadm:Adding mgr.x on smithi090 2024-02-05T07:15:09.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch apply mgr '2;smithi007=y;smithi090=x' 2024-02-05T07:15:10.112 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:15:10.338 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:10 smithi090 ceph-mon[33213]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:10.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:10 smithi007 ceph-mon[29098]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:10.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:10 smithi007 ceph-mon[37158]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:11.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:11 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:11.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:11 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:11.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:11 smithi090 ceph-mon[33213]: Reconfiguring mon.b (monmap changed)... 2024-02-05T07:15:11.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:11 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:15:11.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:11 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T07:15:11.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:11 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:11.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:11 smithi090 ceph-mon[33213]: Reconfiguring daemon mon.b on smithi090 2024-02-05T07:15:11.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:11 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:11.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:11 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:11.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:11 smithi007 ceph-mon[29098]: Reconfiguring mon.b (monmap changed)... 2024-02-05T07:15:11.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:11 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:15:11.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:11 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T07:15:11.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:11 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:11.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:11 smithi007 ceph-mon[29098]: Reconfiguring daemon mon.b on smithi090 2024-02-05T07:15:11.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:11 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:11.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:11 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:11.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:11 smithi007 ceph-mon[37158]: Reconfiguring mon.b (monmap changed)... 2024-02-05T07:15:11.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:11 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T07:15:11.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:11 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T07:15:11.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:11 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:11.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:11 smithi007 ceph-mon[37158]: Reconfiguring daemon mon.b on smithi090 2024-02-05T07:15:12.212 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:12 smithi090 ceph-mon[33213]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:12.212 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:12.212 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:12.213 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:12.213 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:12.213 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:12.213 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:12.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:12 smithi007 ceph-mon[29098]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:12.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:12.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:12.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:12.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:12.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:12.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:12.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:12 smithi007 ceph-mon[37158]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:12.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:12.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:12.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:12.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:12.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:12.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:12.612 INFO:teuthology.orchestra.run.smithi090.stdout:Scheduled mgr update... 2024-02-05T07:15:13.197 DEBUG:teuthology.orchestra.run.smithi090:mgr.x> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mgr.x.service 2024-02-05T07:15:13.199 INFO:tasks.cephadm:Deploying OSDs... 2024-02-05T07:15:13.200 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:15:13.200 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-02-05T07:15:13.218 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-05T07:15:13.219 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-02-05T07:15:13.276 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-05T07:15:13.276 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T07:15:13.276 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-05T07:15:13.276 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T07:15:13.276 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T07:15:13.276 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-02-05 07:14:57.633060117 +0000 2024-02-05T07:15:13.276 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-02-05 07:09:17.898853466 +0000 2024-02-05T07:15:13.276 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-02-05 07:09:17.898853466 +0000 2024-02-05T07:15:13.276 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-02-05 07:09:17.898853466 +0000 2024-02-05T07:15:13.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-05T07:15:13.345 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-02-05T07:15:13.345 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-02-05T07:15:13.345 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000119887 s, 4.3 MB/s 2024-02-05T07:15:13.346 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-05T07:15:13.404 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-02-05T07:15:13.460 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-05T07:15:13.460 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T07:15:13.461 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-05T07:15:13.461 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T07:15:13.461 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T07:15:13.461 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-02-05 07:14:57.633060117 +0000 2024-02-05T07:15:13.461 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-02-05 07:09:18.238848434 +0000 2024-02-05T07:15:13.461 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-02-05 07:09:18.239848419 +0000 2024-02-05T07:15:13.461 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-02-05 07:09:18.238848434 +0000 2024-02-05T07:15:13.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-05T07:15:13.528 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-02-05T07:15:13.528 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-02-05T07:15:13.528 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000114586 s, 4.5 MB/s 2024-02-05T07:15:13.529 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-05T07:15:13.586 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-02-05T07:15:13.643 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-05T07:15:13.643 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T07:15:13.643 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-05T07:15:13.643 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T07:15:13.643 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T07:15:13.643 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-02-05 07:14:57.633060117 +0000 2024-02-05T07:15:13.643 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-02-05 07:09:18.575843446 +0000 2024-02-05T07:15:13.643 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-02-05 07:09:18.575843446 +0000 2024-02-05T07:15:13.643 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-02-05 07:09:18.575843446 +0000 2024-02-05T07:15:13.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-05T07:15:13.710 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-02-05T07:15:13.710 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-02-05T07:15:13.710 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000113646 s, 4.5 MB/s 2024-02-05T07:15:13.711 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-05T07:15:13.768 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-02-05T07:15:13.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:13 smithi090 ceph-mon[33213]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:13.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:13 smithi090 ceph-mon[33213]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi090=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:15:13.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:13 smithi090 ceph-mon[33213]: Saving service mgr spec with placement smithi007=y;smithi090=x;count:2 2024-02-05T07:15:13.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:13 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:13.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:13 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:13.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:13 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:13.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:13 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:13.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:13 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:13.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:13 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T07:15:13.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:13 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-05T07:15:13.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:13 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T07:15:13.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:13 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:13.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:13 smithi090 ceph-mon[33213]: Deploying daemon mgr.x on smithi090 2024-02-05T07:15:13.824 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-05T07:15:13.825 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T07:15:13.825 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-05T07:15:13.825 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T07:15:13.825 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T07:15:13.825 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-02-05 07:14:57.634060101 +0000 2024-02-05T07:15:13.825 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-02-05 07:09:18.902838606 +0000 2024-02-05T07:15:13.825 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-02-05 07:09:18.902838606 +0000 2024-02-05T07:15:13.825 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-02-05 07:09:18.902838606 +0000 2024-02-05T07:15:13.825 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-05T07:15:13.889 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[37158]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:13.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[37158]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi090=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:15:13.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[37158]: Saving service mgr spec with placement smithi007=y;smithi090=x;count:2 2024-02-05T07:15:13.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:13.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:13.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:13.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:13.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:13.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T07:15:13.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-05T07:15:13.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T07:15:13.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:13.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[37158]: Deploying daemon mgr.x on smithi090 2024-02-05T07:15:13.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[29098]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:13.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[29098]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi090=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:15:13.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[29098]: Saving service mgr spec with placement smithi007=y;smithi090=x;count:2 2024-02-05T07:15:13.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:13.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:13.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:13.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:13.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:13.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T07:15:13.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-05T07:15:13.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T07:15:13.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:13.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:13 smithi007 ceph-mon[29098]: Deploying daemon mgr.x on smithi090 2024-02-05T07:15:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-02-05T07:15:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-02-05T07:15:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000105304 s, 4.9 MB/s 2024-02-05T07:15:13.893 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-05T07:15:13.950 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-05T07:15:13.950 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-02-05T07:15:13.967 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-05T07:15:13.967 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-02-05T07:15:14.023 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-05T07:15:14.023 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T07:15:14.023 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-05T07:15:14.023 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T07:15:14.023 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T07:15:14.023 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-02-05 07:14:48.223262562 +0000 2024-02-05T07:15:14.023 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-02-05 07:09:17.896969246 +0000 2024-02-05T07:15:14.023 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-02-05 07:09:17.896969246 +0000 2024-02-05T07:15:14.023 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-02-05 07:09:17.896969246 +0000 2024-02-05T07:15:14.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-05T07:15:14.092 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-02-05T07:15:14.092 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-02-05T07:15:14.092 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.00024273 s, 2.1 MB/s 2024-02-05T07:15:14.093 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-05T07:15:14.149 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-02-05T07:15:14.204 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-05T07:15:14.204 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T07:15:14.204 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-05T07:15:14.204 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T07:15:14.204 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T07:15:14.204 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-02-05 07:14:48.224262546 +0000 2024-02-05T07:15:14.204 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-02-05 07:09:18.240963929 +0000 2024-02-05T07:15:14.204 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-02-05 07:09:18.240963929 +0000 2024-02-05T07:15:14.204 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-02-05 07:09:18.240963929 +0000 2024-02-05T07:15:14.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-05T07:15:14.335 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-02-05T07:15:14.335 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-02-05T07:15:14.335 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000189102 s, 2.7 MB/s 2024-02-05T07:15:14.336 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-05T07:15:14.351 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-02-05T07:15:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-05T07:15:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T07:15:14.413 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-05T07:15:14.413 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T07:15:14.413 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T07:15:14.413 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-02-05 07:15:14.289854436 +0000 2024-02-05T07:15:14.413 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-02-05 07:09:18.571958813 +0000 2024-02-05T07:15:14.413 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-02-05 07:09:18.571958813 +0000 2024-02-05T07:15:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-02-05 07:09:18.571958813 +0000 2024-02-05T07:15:14.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-05T07:15:14.517 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-02-05T07:15:14.517 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-02-05T07:15:14.518 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000214973 s, 2.4 MB/s 2024-02-05T07:15:14.519 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-05T07:15:14.534 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-02-05T07:15:14.591 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-05T07:15:14.591 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T07:15:14.591 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-05T07:15:14.591 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T07:15:14.591 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T07:15:14.591 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-02-05 07:15:14.473851573 +0000 2024-02-05T07:15:14.591 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-02-05 07:09:18.896953790 +0000 2024-02-05T07:15:14.592 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-02-05 07:09:18.896953790 +0000 2024-02-05T07:15:14.592 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-02-05 07:09:18.896953790 +0000 2024-02-05T07:15:14.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-05T07:15:14.657 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:14 smithi090 systemd[1]: Starting Ceph mgr.x for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:15:14.659 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-02-05T07:15:14.659 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-02-05T07:15:14.660 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000179833 s, 2.8 MB/s 2024-02-05T07:15:14.661 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-05T07:15:14.716 INFO:tasks.cephadm:Deploying osd.0 on smithi007 with /dev/vg_nvme/lv_4... 2024-02-05T07:15:14.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-05T07:15:14.953 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:14 smithi090 podman[35048]: 2024-02-05 07:15:14.65188016 +0000 UTC m=+0.018029107 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T07:15:14.953 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:14 smithi090 podman[35048]: 2024-02-05 07:15:14.756661936 +0000 UTC m=+0.122810929 container create e809b7c6434d4b19e9aff1e760bfcc7ed08aff88fa65f79e76ed62bddecdce50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_BRANCH=HEAD) 2024-02-05T07:15:14.953 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:14 smithi090 podman[35048]: 2024-02-05 07:15:14.952269044 +0000 UTC m=+0.318417995 container init e809b7c6434d4b19e9aff1e760bfcc7ed08aff88fa65f79e76ed62bddecdce50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T07:15:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:15:15.274 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:14 smithi090 podman[35048]: 2024-02-05 07:15:14.955107734 +0000 UTC m=+0.321256684 container start e809b7c6434d4b19e9aff1e760bfcc7ed08aff88fa65f79e76ed62bddecdce50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T07:15:15.274 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:14 smithi090 ceph-mgr[35104]: -- 172.21.15.90:0/507427034 <== mon.1 v2:172.21.15.90:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d63d6d1520 con 0x55d63d6a6800 2024-02-05T07:15:15.274 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:15 smithi090 bash[35048]: e809b7c6434d4b19e9aff1e760bfcc7ed08aff88fa65f79e76ed62bddecdce50 2024-02-05T07:15:15.275 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:15 smithi090 systemd[1]: Started Ceph mgr.x for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:15:15.774 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:15 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:15.512+0000 7f04880fd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T07:15:15.775 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:15 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:15.609+0000 7f04880fd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T07:15:16.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:16 smithi007 ceph-mon[29098]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:16.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:16 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:16.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:16 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:16.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:16 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:16.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:16 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:16.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:16 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:16.244 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:16 smithi007 ceph-mon[37158]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:16.244 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:16 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:16.244 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:16 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:16.244 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:16 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:16.244 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:16 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:16.244 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:16 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:16.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:16 smithi090 ceph-mon[33213]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:16.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:16 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:16.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:16 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:16.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:16 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:16.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:16 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:16.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:16 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:16.559 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:16 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:16.292+0000 7f04880fd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T07:15:16.559 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:16 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:16.399+0000 7f04880fd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T07:15:16.827 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-05T07:15:16.827 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-05T07:15:16.827 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-02-05T07:15:16.827 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-02-05T07:15:16.827 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00836192 s, 1.3 GB/s 2024-02-05T07:15:16.827 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-02-05T07:15:17.011 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:16 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:16.556+0000 7f04880fd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T07:15:17.016 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:16 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:16.656+0000 7f04880fd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T07:15:17.016 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:16 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:16.743+0000 7f04880fd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T07:15:17.016 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:17.008+0000 7f04880fd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T07:15:17.274 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:17.118+0000 7f04880fd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T07:15:17.275 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:17.205+0000 7f04880fd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T07:15:17.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi007:vg_nvme/lv_4 2024-02-05T07:15:17.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[29098]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:17.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:17.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:17.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:17.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:17.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:17.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[29098]: Reconfiguring mgr.y (unknown last config time)... 2024-02-05T07:15:17.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T07:15:17.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T07:15:17.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:17.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[29098]: Reconfiguring daemon mgr.y on smithi007 2024-02-05T07:15:17.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[37158]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:17.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:17.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:17.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:17.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:17.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:17.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[37158]: Reconfiguring mgr.y (unknown last config time)... 2024-02-05T07:15:17.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T07:15:17.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T07:15:17.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:17.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:17 smithi007 ceph-mon[37158]: Reconfiguring daemon mgr.y on smithi007 2024-02-05T07:15:17.774 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:17.299+0000 7f04880fd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T07:15:17.774 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:17.474+0000 7f04880fd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T07:15:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-mon[33213]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-mon[33213]: Reconfiguring mgr.y (unknown last config time)... 2024-02-05T07:15:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T07:15:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T07:15:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:17.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:17 smithi090 ceph-mon[33213]: Reconfiguring daemon mgr.y on smithi007 2024-02-05T07:15:18.045 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:15:18.774 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:18 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:18.432+0000 7f04880fd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T07:15:18.774 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:18 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:18.560+0000 7f04880fd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T07:15:19.170 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:18 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:18.888+0000 7f04880fd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T07:15:19.170 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:18 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:18.975+0000 7f04880fd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T07:15:19.170 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:19 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:19.167+0000 7f04880fd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:15:19.524 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:19 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:19.348+0000 7f04880fd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T07:15:19.885 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:19 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:19.625+0000 7f04880fd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:15:19.885 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:19 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:19.882+0000 7f04880fd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T07:15:20.274 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:20.115+0000 7f04880fd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T07:15:20.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:20 smithi090 ceph-mon[33213]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:20.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:20 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:20.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:20 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:20.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:20 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:20.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:20 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:20.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:20 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:20.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:20 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:20.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:20 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:20.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[29098]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:20.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:20.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:20.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:20.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:20.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:20.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:20.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:20.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[37158]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:20.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:20.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:20.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:20.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:20.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:20.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:20.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:20 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:21.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:21 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:15:21.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:21 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:15:21.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:21 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:21.275 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:20.899+0000 7f04880fd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T07:15:21.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:21 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:15:21.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:21 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:15:21.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:21 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:21.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:21 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:15:21.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:21 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:15:21.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:21 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:21.774 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:21.466+0000 7f04880fd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T07:15:21.774 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:21.562+0000 7f04880fd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T07:15:22.047 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:15:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:15:21.775+0000 7f04880fd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T07:15:22.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:22 smithi007 ceph-mon[29098]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:15:22.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:22 smithi007 ceph-mon[29098]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:22.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:22 smithi007 ceph-mon[29098]: Standby manager daemon x started 2024-02-05T07:15:22.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:22 smithi007 ceph-mon[29098]: from='mgr.? 172.21.15.90:0/638931645' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T07:15:22.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:22 smithi007 ceph-mon[29098]: from='mgr.? 172.21.15.90:0/638931645' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T07:15:22.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:22 smithi007 ceph-mon[29098]: from='mgr.? 172.21.15.90:0/638931645' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T07:15:22.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:22 smithi007 ceph-mon[29098]: from='mgr.? 172.21.15.90:0/638931645' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T07:15:22.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:22 smithi007 ceph-mon[37158]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:15:22.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:22 smithi007 ceph-mon[37158]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:22.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:22 smithi007 ceph-mon[37158]: Standby manager daemon x started 2024-02-05T07:15:22.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:22 smithi007 ceph-mon[37158]: from='mgr.? 172.21.15.90:0/638931645' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T07:15:22.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:22 smithi007 ceph-mon[37158]: from='mgr.? 172.21.15.90:0/638931645' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T07:15:22.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:22 smithi007 ceph-mon[37158]: from='mgr.? 172.21.15.90:0/638931645' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T07:15:22.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:22 smithi007 ceph-mon[37158]: from='mgr.? 172.21.15.90:0/638931645' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T07:15:22.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:22 smithi090 ceph-mon[33213]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:15:22.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:22 smithi090 ceph-mon[33213]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:22.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:22 smithi090 ceph-mon[33213]: Standby manager daemon x started 2024-02-05T07:15:22.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:22 smithi090 ceph-mon[33213]: from='mgr.? 172.21.15.90:0/638931645' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T07:15:22.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:22 smithi090 ceph-mon[33213]: from='mgr.? 172.21.15.90:0/638931645' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T07:15:22.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:22 smithi090 ceph-mon[33213]: from='mgr.? 172.21.15.90:0/638931645' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T07:15:22.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:22 smithi090 ceph-mon[33213]: from='mgr.? 172.21.15.90:0/638931645' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T07:15:23.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:23 smithi007 ceph-mon[29098]: mgrmap e14: y(active, since 67s), standbys: x 2024-02-05T07:15:23.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:23 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T07:15:23.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:23 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3240587656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "703c3948-8003-4015-9dd8-21019cf0eeb8"}]: dispatch 2024-02-05T07:15:23.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:23 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3240587656' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "703c3948-8003-4015-9dd8-21019cf0eeb8"}]': finished 2024-02-05T07:15:23.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:23 smithi007 ceph-mon[29098]: osdmap e5: 1 total, 0 up, 1 in 2024-02-05T07:15:23.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:23 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:23.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:23 smithi007 ceph-mon[37158]: mgrmap e14: y(active, since 67s), standbys: x 2024-02-05T07:15:23.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:23 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T07:15:23.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:23 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3240587656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "703c3948-8003-4015-9dd8-21019cf0eeb8"}]: dispatch 2024-02-05T07:15:23.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:23 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3240587656' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "703c3948-8003-4015-9dd8-21019cf0eeb8"}]': finished 2024-02-05T07:15:23.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:23 smithi007 ceph-mon[37158]: osdmap e5: 1 total, 0 up, 1 in 2024-02-05T07:15:23.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:23 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:23.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:23 smithi090 ceph-mon[33213]: mgrmap e14: y(active, since 67s), standbys: x 2024-02-05T07:15:23.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:23 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T07:15:23.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:23 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3240587656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "703c3948-8003-4015-9dd8-21019cf0eeb8"}]: dispatch 2024-02-05T07:15:23.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:23 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3240587656' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "703c3948-8003-4015-9dd8-21019cf0eeb8"}]': finished 2024-02-05T07:15:23.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:23 smithi090 ceph-mon[33213]: osdmap e5: 1 total, 0 up, 1 in 2024-02-05T07:15:23.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:23 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:24.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:24 smithi007 ceph-mon[29098]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:24.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:24 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3874754362' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:15:24.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:24 smithi007 ceph-mon[37158]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:24.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:24 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3874754362' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:15:24.425 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:24 smithi090 ceph-mon[33213]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:24.426 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:24 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3874754362' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:15:26.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:26 smithi007 ceph-mon[29098]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:26.291 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:26 smithi007 ceph-mon[37158]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:26.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:26 smithi090 ceph-mon[33213]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:28.167 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:28 smithi007 ceph-mon[29098]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:28.167 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:28 smithi007 ceph-mon[37158]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:28 smithi090 ceph-mon[33213]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:30.150 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:30 smithi007 ceph-mon[37158]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:30.150 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:30 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-05T07:15:30.150 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:30 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:30.150 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:30 smithi007 ceph-mon[29098]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:30.150 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:30 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-05T07:15:30.150 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:30 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:30.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:30 smithi090 ceph-mon[33213]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:30 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-05T07:15:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:30 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:31.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:31 smithi007 ceph-mon[29098]: Deploying daemon osd.0 on smithi007 2024-02-05T07:15:31.416 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:31 smithi007 ceph-mon[37158]: Deploying daemon osd.0 on smithi007 2024-02-05T07:15:31.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:31 smithi090 ceph-mon[33213]: Deploying daemon osd.0 on smithi007 2024-02-05T07:15:32.323 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:32 smithi007 ceph-mon[29098]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:32.323 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:32 smithi007 ceph-mon[37158]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:32.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:32 smithi090 ceph-mon[33213]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:34.289 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:34 smithi007 ceph-mon[29098]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:34.289 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:34 smithi007 ceph-mon[37158]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:34.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:34 smithi090 ceph-mon[33213]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:35.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:35 smithi090 ceph-mon[33213]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:35 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:35 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:35 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:35.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:35 smithi007 ceph-mon[29098]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:35.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:35 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:35.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:35 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:35.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:35 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:35.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:35 smithi007 ceph-mon[37158]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:35.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:35 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:35.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:35 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:35.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:35 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:38.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:38 smithi007 ceph-mon[29098]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:38.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:38 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:38.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:38 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:38.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:38 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:38.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:38 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:38.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:38 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:38.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:38 smithi007 ceph-mon[29098]: from='osd.0 v2:172.21.15.7:6801/2069016689' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-05T07:15:38.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:38 smithi007 ceph-mon[37158]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:38.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:38 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:38.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:38 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:38.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:38 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:38.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:38 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:38.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:38 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:38.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:38 smithi007 ceph-mon[37158]: from='osd.0 v2:172.21.15.7:6801/2069016689' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-05T07:15:38.123 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 0 on host 'smithi007' 2024-02-05T07:15:38.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:38 smithi090 ceph-mon[33213]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:38.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:38 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:38.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:38 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:38.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:38 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:38.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:38 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:38.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:38 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:38.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:38 smithi090 ceph-mon[33213]: from='osd.0 v2:172.21.15.7:6801/2069016689' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-05T07:15:38.729 DEBUG:teuthology.orchestra.run.smithi007:osd.0> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.0.service 2024-02-05T07:15:38.731 INFO:tasks.cephadm:Deploying osd.1 on smithi007 with /dev/vg_nvme/lv_3... 2024-02-05T07:15:38.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-05T07:15:39.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-mon[37158]: from='osd.0 v2:172.21.15.7:6801/2069016689' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-05T07:15:39.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-mon[37158]: osdmap e6: 1 total, 0 up, 1 in 2024-02-05T07:15:39.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-mon[37158]: from='osd.0 v2:172.21.15.7:6801/2069016689' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:15:39.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:39.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:39.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:39.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:39.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-mon[29098]: from='osd.0 v2:172.21.15.7:6801/2069016689' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-05T07:15:39.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-mon[29098]: osdmap e6: 1 total, 0 up, 1 in 2024-02-05T07:15:39.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-mon[29098]: from='osd.0 v2:172.21.15.7:6801/2069016689' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:15:39.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:39.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:39.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:39.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:39.425 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-0[41180]: 2024-02-05T07:15:39.087+0000 7fc74c63f700 -1 osd.0 0 waiting for initial osdmap 2024-02-05T07:15:39.425 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:15:39 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-0[41180]: 2024-02-05T07:15:39.089+0000 7fc748439700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T07:15:39.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:39 smithi090 ceph-mon[33213]: from='osd.0 v2:172.21.15.7:6801/2069016689' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-05T07:15:39.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:39 smithi090 ceph-mon[33213]: osdmap e6: 1 total, 0 up, 1 in 2024-02-05T07:15:39.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:39 smithi090 ceph-mon[33213]: from='osd.0 v2:172.21.15.7:6801/2069016689' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:15:39.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:39 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:39.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:39 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:39.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:39 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:39.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:39 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:15:40.340 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:40 smithi007 ceph-mon[29098]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:40.340 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:40 smithi007 ceph-mon[29098]: from='osd.0 v2:172.21.15.7:6801/2069016689' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-02-05T07:15:40.340 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:40 smithi007 ceph-mon[29098]: osdmap e7: 1 total, 0 up, 1 in 2024-02-05T07:15:40.340 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:40 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:40.340 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:40 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:40.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:40 smithi007 ceph-mon[37158]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:40.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:40 smithi007 ceph-mon[37158]: from='osd.0 v2:172.21.15.7:6801/2069016689' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-02-05T07:15:40.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:40 smithi007 ceph-mon[37158]: osdmap e7: 1 total, 0 up, 1 in 2024-02-05T07:15:40.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:40 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:40.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:40 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:40.486 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:15:40.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:40 smithi090 ceph-mon[33213]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:40.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:40 smithi090 ceph-mon[33213]: from='osd.0 v2:172.21.15.7:6801/2069016689' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-02-05T07:15:40.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:40 smithi090 ceph-mon[33213]: osdmap e7: 1 total, 0 up, 1 in 2024-02-05T07:15:40.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:40 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:40.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:40 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:41.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:41 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:15:41.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:41 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:15:41.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:41 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:41.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:41 smithi007 ceph-mon[29098]: osd.0 v2:172.21.15.7:6801/2069016689 boot 2024-02-05T07:15:41.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:41 smithi007 ceph-mon[29098]: osdmap e8: 1 total, 1 up, 1 in 2024-02-05T07:15:41.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:41 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:41.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:41 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:15:41.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:41 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:15:41.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:41 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:41.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:41 smithi007 ceph-mon[37158]: osd.0 v2:172.21.15.7:6801/2069016689 boot 2024-02-05T07:15:41.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:41 smithi007 ceph-mon[37158]: osdmap e8: 1 total, 1 up, 1 in 2024-02-05T07:15:41.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:41 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:41.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:41 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:15:41.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:41 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:15:41.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:41 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:41.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:41 smithi090 ceph-mon[33213]: osd.0 v2:172.21.15.7:6801/2069016689 boot 2024-02-05T07:15:41.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:41 smithi090 ceph-mon[33213]: osdmap e8: 1 total, 1 up, 1 in 2024-02-05T07:15:41.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:41 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:15:42.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[29098]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:42.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[29098]: osdmap e9: 1 total, 1 up, 1 in 2024-02-05T07:15:42.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:42.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:42.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:15:42.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:42.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:42.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:42.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:42.414 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[37158]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:42.414 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[37158]: osdmap e9: 1 total, 1 up, 1 in 2024-02-05T07:15:42.414 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:42.414 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:42.414 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:15:42.414 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:42.414 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:42.414 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:42.415 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:42 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:42.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:42 smithi090 ceph-mon[33213]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T07:15:42.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:42 smithi090 ceph-mon[33213]: osdmap e9: 1 total, 1 up, 1 in 2024-02-05T07:15:42.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:42 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:42.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:42 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:42.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:42 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:15:42.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:42 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:42.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:42 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:42.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:42 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:15:42.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:42 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:15:42.959 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-05T07:15:42.959 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-05T07:15:42.959 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-02-05T07:15:42.959 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-02-05T07:15:42.959 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00855322 s, 1.2 GB/s 2024-02-05T07:15:42.959 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 2024-02-05T07:15:42.959 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-02-05T07:15:43.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:43 smithi007 ceph-mon[29098]: Detected new or changed devices on smithi007 2024-02-05T07:15:43.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:43 smithi007 ceph-mon[29098]: Adjusting osd_memory_target on smithi007 to 16069M 2024-02-05T07:15:43.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:43 smithi007 ceph-mon[37158]: Detected new or changed devices on smithi007 2024-02-05T07:15:43.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:43 smithi007 ceph-mon[37158]: Adjusting osd_memory_target on smithi007 to 16069M 2024-02-05T07:15:43.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:43 smithi090 ceph-mon[33213]: Detected new or changed devices on smithi007 2024-02-05T07:15:43.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:43 smithi090 ceph-mon[33213]: Adjusting osd_memory_target on smithi007 to 16069M 2024-02-05T07:15:43.639 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi007:vg_nvme/lv_3 2024-02-05T07:15:43.884 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:15:44.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:44 smithi007 ceph-mon[29098]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:44.258 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:44 smithi007 ceph-mon[37158]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:44.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:44 smithi090 ceph-mon[33213]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:46.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:46 smithi007 ceph-mon[29098]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:46.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:46 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:15:46.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:46 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:15:46.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:46 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:46.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:46 smithi007 ceph-mon[37158]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:46.386 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:46 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:15:46.386 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:46 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:15:46.386 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:46 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:46.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:46 smithi090 ceph-mon[33213]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:46.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:46 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:15:46.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:46 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:15:46.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:46 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:47.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:47 smithi007 ceph-mon[29098]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:15:47.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:47 smithi007 ceph-mon[37158]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:15:47.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:47 smithi090 ceph-mon[33213]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:15:48.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:48 smithi007 ceph-mon[29098]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:48.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:48 smithi007 ceph-mon[37158]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:48.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:48 smithi090 ceph-mon[33213]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:49 smithi007 ceph-mon[29098]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:49 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1549318672' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c3e7d1e-a61d-44b9-b21e-f91c99a096bc"}]: dispatch 2024-02-05T07:15:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:49 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1549318672' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c3e7d1e-a61d-44b9-b21e-f91c99a096bc"}]': finished 2024-02-05T07:15:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:49 smithi007 ceph-mon[29098]: osdmap e10: 2 total, 1 up, 2 in 2024-02-05T07:15:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:49 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:15:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:49 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1528581535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:15:49.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:49 smithi007 ceph-mon[37158]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:49.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:49 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1549318672' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c3e7d1e-a61d-44b9-b21e-f91c99a096bc"}]: dispatch 2024-02-05T07:15:49.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:49 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1549318672' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c3e7d1e-a61d-44b9-b21e-f91c99a096bc"}]': finished 2024-02-05T07:15:49.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:49 smithi007 ceph-mon[37158]: osdmap e10: 2 total, 1 up, 2 in 2024-02-05T07:15:49.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:49 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:15:49.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:49 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1528581535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:15:49.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:49 smithi090 ceph-mon[33213]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:49.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:49 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1549318672' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c3e7d1e-a61d-44b9-b21e-f91c99a096bc"}]: dispatch 2024-02-05T07:15:49.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:49 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1549318672' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c3e7d1e-a61d-44b9-b21e-f91c99a096bc"}]': finished 2024-02-05T07:15:49.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:49 smithi090 ceph-mon[33213]: osdmap e10: 2 total, 1 up, 2 in 2024-02-05T07:15:49.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:15:49.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:49 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1528581535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:15:51.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:51 smithi007 ceph-mon[29098]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:51.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:51 smithi007 ceph-mon[37158]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:51.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:51 smithi090 ceph-mon[33213]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:53.246 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:53 smithi007 ceph-mon[29098]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:53.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:53 smithi007 ceph-mon[37158]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:53.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:53 smithi090 ceph-mon[33213]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:55.310 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:55 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-05T07:15:55.310 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:55 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:55.310 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:55 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-05T07:15:55.310 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:55 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:55.327 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:55 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-05T07:15:55.327 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:55 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:15:56.273 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:56 smithi007 ceph-mon[29098]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:56.273 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:56 smithi007 ceph-mon[29098]: Deploying daemon osd.1 on smithi007 2024-02-05T07:15:56.273 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:56 smithi007 ceph-mon[37158]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:56.273 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:56 smithi007 ceph-mon[37158]: Deploying daemon osd.1 on smithi007 2024-02-05T07:15:56.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:56 smithi090 ceph-mon[33213]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:56.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:56 smithi090 ceph-mon[33213]: Deploying daemon osd.1 on smithi007 2024-02-05T07:15:58.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:15:58 smithi007 ceph-mon[29098]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:58.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:15:58 smithi007 ceph-mon[37158]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:15:58.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:15:58 smithi090 ceph-mon[33213]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:00.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:00 smithi007 ceph-mon[29098]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:00.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:00 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:00.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:00 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:00.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:00 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:00.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:00 smithi007 ceph-mon[37158]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:00.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:00 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:00.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:00 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:00.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:00 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:00 smithi090 ceph-mon[33213]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:00 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:00 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:00 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:02 smithi007 ceph-mon[29098]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:02 smithi007 ceph-mon[37158]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:02.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:02 smithi090 ceph-mon[33213]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:03.772 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 1 on host 'smithi007' 2024-02-05T07:16:04.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[37158]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:04.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:04.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:04.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[37158]: from='osd.1 v2:172.21.15.7:6805/865486532' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T07:16:04.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[37158]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T07:16:04.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:04.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[29098]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:04.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:04.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:04.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[29098]: from='osd.1 v2:172.21.15.7:6805/865486532' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T07:16:04.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[29098]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T07:16:04.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:03 smithi090 ceph-mon[33213]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:03 smithi090 ceph-mon[33213]: from='osd.1 v2:172.21.15.7:6805/865486532' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T07:16:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:03 smithi090 ceph-mon[33213]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T07:16:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:04.386 DEBUG:teuthology.orchestra.run.smithi007:osd.1> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.1.service 2024-02-05T07:16:04.388 INFO:tasks.cephadm:Deploying osd.2 on smithi007 with /dev/vg_nvme/lv_2... 2024-02-05T07:16:04.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-05T07:16:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:16:05.381 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:05 smithi007 ceph-mon[29098]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-05T07:16:05.381 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:05 smithi007 ceph-mon[29098]: osdmap e11: 2 total, 1 up, 2 in 2024-02-05T07:16:05.381 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:05.381 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:05 smithi007 ceph-mon[29098]: from='osd.1 v2:172.21.15.7:6805/865486532' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:16:05.381 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:05 smithi007 ceph-mon[29098]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:16:05.381 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:05 smithi007 ceph-mon[37158]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-05T07:16:05.381 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:05 smithi007 ceph-mon[37158]: osdmap e11: 2 total, 1 up, 2 in 2024-02-05T07:16:05.381 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:05.382 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:05 smithi007 ceph-mon[37158]: from='osd.1 v2:172.21.15.7:6805/865486532' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:16:05.382 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:05 smithi007 ceph-mon[37158]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:16:05.382 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:16:05 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-1[47043]: 2024-02-05T07:16:05.070+0000 7f21ac71a700 -1 osd.1 0 waiting for initial osdmap 2024-02-05T07:16:05.382 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:16:05 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-1[47043]: 2024-02-05T07:16:05.074+0000 7f21a8514700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T07:16:05.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:05 smithi090 ceph-mon[33213]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-05T07:16:05.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:05 smithi090 ceph-mon[33213]: osdmap e11: 2 total, 1 up, 2 in 2024-02-05T07:16:05.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:05.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:05 smithi090 ceph-mon[33213]: from='osd.1 v2:172.21.15.7:6805/865486532' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:16:05.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:05 smithi090 ceph-mon[33213]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:16:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:06 smithi007 ceph-mon[29098]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:06 smithi007 ceph-mon[29098]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-02-05T07:16:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:06 smithi007 ceph-mon[29098]: osdmap e12: 2 total, 1 up, 2 in 2024-02-05T07:16:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:06 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:06 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:06.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:06 smithi007 ceph-mon[37158]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:06.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:06 smithi007 ceph-mon[37158]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-02-05T07:16:06.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:06 smithi007 ceph-mon[37158]: osdmap e12: 2 total, 1 up, 2 in 2024-02-05T07:16:06.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:06 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:06.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:06 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:06 smithi090 ceph-mon[33213]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:06 smithi090 ceph-mon[33213]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-02-05T07:16:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:06 smithi090 ceph-mon[33213]: osdmap e12: 2 total, 1 up, 2 in 2024-02-05T07:16:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:06 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:06 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:07.344 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:07 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:16:07.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:07 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:16:07.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:07 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:07.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:07 smithi007 ceph-mon[29098]: osd.1 v2:172.21.15.7:6805/865486532 boot 2024-02-05T07:16:07.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:07 smithi007 ceph-mon[29098]: osdmap e13: 2 total, 2 up, 2 in 2024-02-05T07:16:07.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:07 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:07.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:07 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:16:07.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:07 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:16:07.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:07 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:07.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:07 smithi007 ceph-mon[37158]: osd.1 v2:172.21.15.7:6805/865486532 boot 2024-02-05T07:16:07.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:07 smithi007 ceph-mon[37158]: osdmap e13: 2 total, 2 up, 2 in 2024-02-05T07:16:07.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:07 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:07.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:07 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:16:07.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:07 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:16:07.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:07 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:07.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:07 smithi090 ceph-mon[33213]: osd.1 v2:172.21.15.7:6805/865486532 boot 2024-02-05T07:16:07.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:07 smithi090 ceph-mon[33213]: osdmap e13: 2 total, 2 up, 2 in 2024-02-05T07:16:07.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:07 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:16:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-05T07:16:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-05T07:16:07.738 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-02-05T07:16:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-02-05T07:16:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00956667 s, 1.1 GB/s 2024-02-05T07:16:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-02-05T07:16:08.366 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:08 smithi007 ceph-mon[29098]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:08.366 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:08 smithi007 ceph-mon[29098]: osdmap e14: 2 total, 2 up, 2 in 2024-02-05T07:16:08.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:08 smithi007 ceph-mon[37158]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:08.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:08 smithi007 ceph-mon[37158]: osdmap e14: 2 total, 2 up, 2 in 2024-02-05T07:16:08.394 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi007:vg_nvme/lv_2 2024-02-05T07:16:08.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:08 smithi090 ceph-mon[33213]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T07:16:08.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:08 smithi090 ceph-mon[33213]: osdmap e14: 2 total, 2 up, 2 in 2024-02-05T07:16:09.089 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:16:10.387 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[29098]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:10.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[29098]: Detected new or changed devices on smithi007 2024-02-05T07:16:10.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:10.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:10.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:10.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:10.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[29098]: Adjusting osd_memory_target on smithi007 to 8034M 2024-02-05T07:16:10.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:10.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:10.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:10.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:10.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[37158]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:10.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[37158]: Detected new or changed devices on smithi007 2024-02-05T07:16:10.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:10.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:10.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:10.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:10.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[37158]: Adjusting osd_memory_target on smithi007 to 8034M 2024-02-05T07:16:10.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:10.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:10.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:10.390 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:10 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:10 smithi090 ceph-mon[33213]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:10 smithi090 ceph-mon[33213]: Detected new or changed devices on smithi007 2024-02-05T07:16:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:10 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:10 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:10 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:10 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:10 smithi090 ceph-mon[33213]: Adjusting osd_memory_target on smithi007 to 8034M 2024-02-05T07:16:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:10 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:10 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:10 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:10 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:11.422 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:11 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:16:11.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:11 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:16:11.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:11 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:11.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:11 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:16:11.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:11 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:16:11.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:11 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:11.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:11 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:16:11.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:11 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:16:11.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:11 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:12.398 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:12 smithi007 ceph-mon[29098]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:12.398 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:12 smithi007 ceph-mon[29098]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:16:12.399 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:12 smithi007 ceph-mon[37158]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:12.399 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:12 smithi007 ceph-mon[37158]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:16:12.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:12 smithi090 ceph-mon[33213]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:12.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:12 smithi090 ceph-mon[33213]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:16:14.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:14 smithi007 ceph-mon[37158]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:14.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:14 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3774622817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4fd930f-7bb2-46ca-9964-98235bd47b3a"}]: dispatch 2024-02-05T07:16:14.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:14 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3774622817' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4fd930f-7bb2-46ca-9964-98235bd47b3a"}]': finished 2024-02-05T07:16:14.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:14 smithi007 ceph-mon[37158]: osdmap e15: 3 total, 2 up, 3 in 2024-02-05T07:16:14.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:14 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:14.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:14 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3752498309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:16:14.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:14 smithi007 ceph-mon[29098]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:14.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:14 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3774622817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4fd930f-7bb2-46ca-9964-98235bd47b3a"}]: dispatch 2024-02-05T07:16:14.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:14 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3774622817' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4fd930f-7bb2-46ca-9964-98235bd47b3a"}]': finished 2024-02-05T07:16:14.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:14 smithi007 ceph-mon[29098]: osdmap e15: 3 total, 2 up, 3 in 2024-02-05T07:16:14.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:14 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:14.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:14 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3752498309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:16:14.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:14 smithi090 ceph-mon[33213]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:14.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:14 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3774622817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4fd930f-7bb2-46ca-9964-98235bd47b3a"}]: dispatch 2024-02-05T07:16:14.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:14 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3774622817' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4fd930f-7bb2-46ca-9964-98235bd47b3a"}]': finished 2024-02-05T07:16:14.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:14 smithi090 ceph-mon[33213]: osdmap e15: 3 total, 2 up, 3 in 2024-02-05T07:16:14.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:14 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:14.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:14 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3752498309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:16:16.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:16 smithi007 ceph-mon[37158]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:16 smithi007 ceph-mon[29098]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:16.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:16 smithi090 ceph-mon[33213]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:18.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:18 smithi007 ceph-mon[29098]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:18.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:18 smithi007 ceph-mon[37158]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:18.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:18 smithi090 ceph-mon[33213]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:20.378 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:20 smithi007 ceph-mon[37158]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:20.379 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:20 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-05T07:16:20.379 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:20 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:20.379 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:20 smithi007 ceph-mon[29098]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:20.379 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:20 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-05T07:16:20.379 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:20 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:20.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:20 smithi090 ceph-mon[33213]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:20.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:20 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-05T07:16:20.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:20 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:21.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:21 smithi007 ceph-mon[29098]: Deploying daemon osd.2 on smithi007 2024-02-05T07:16:21.393 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:21 smithi007 ceph-mon[37158]: Deploying daemon osd.2 on smithi007 2024-02-05T07:16:21.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:21 smithi090 ceph-mon[33213]: Deploying daemon osd.2 on smithi007 2024-02-05T07:16:22.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:22 smithi007 ceph-mon[29098]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:22.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:22 smithi007 ceph-mon[37158]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:22.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:22 smithi090 ceph-mon[33213]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:24.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:24 smithi007 ceph-mon[29098]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:24.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:24 smithi007 ceph-mon[37158]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:24.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:24 smithi090 ceph-mon[33213]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:26.327 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:26 smithi090 ceph-mon[33213]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:26.327 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:26 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:26.328 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:26 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:26.328 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:26 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:26.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:26 smithi007 ceph-mon[29098]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:26.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:26 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:26.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:26 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:26.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:26 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:26.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:26 smithi007 ceph-mon[37158]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:26.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:26 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:26.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:26 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:26.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:26 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:28.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:28 smithi007 ceph-mon[29098]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:28.377 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:28 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:28.377 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:28 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:28.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:28 smithi007 ceph-mon[37158]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:28.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:28 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:28.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:28 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:28 smithi090 ceph-mon[33213]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:28.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:28 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:28.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:28 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:28.721 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 2 on host 'smithi007' 2024-02-05T07:16:29.306 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:29 smithi007 ceph-mon[29098]: from='osd.2 v2:172.21.15.7:6809/904871370' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T07:16:29.307 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:29 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:29.307 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:29 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:29.307 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:29 smithi007 ceph-mon[37158]: from='osd.2 v2:172.21.15.7:6809/904871370' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T07:16:29.307 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:29 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:29.307 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:29 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:29.338 DEBUG:teuthology.orchestra.run.smithi007:osd.2> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.2.service 2024-02-05T07:16:29.340 INFO:tasks.cephadm:Deploying osd.3 on smithi007 with /dev/vg_nvme/lv_1... 2024-02-05T07:16:29.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-05T07:16:29.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:29 smithi090 ceph-mon[33213]: from='osd.2 v2:172.21.15.7:6809/904871370' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T07:16:29.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:29 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:29.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:29 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:29.977 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:16:30.389 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:30 smithi007 ceph-mon[29098]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:30.390 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:30 smithi007 ceph-mon[29098]: from='osd.2 v2:172.21.15.7:6809/904871370' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-05T07:16:30.390 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:30 smithi007 ceph-mon[29098]: osdmap e16: 3 total, 2 up, 3 in 2024-02-05T07:16:30.390 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:30 smithi007 ceph-mon[29098]: from='osd.2 v2:172.21.15.7:6809/904871370' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:16:30.390 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:30 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:30.390 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:30 smithi007 ceph-mon[37158]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:30.391 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:30 smithi007 ceph-mon[37158]: from='osd.2 v2:172.21.15.7:6809/904871370' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-05T07:16:30.391 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:30 smithi007 ceph-mon[37158]: osdmap e16: 3 total, 2 up, 3 in 2024-02-05T07:16:30.391 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:30 smithi007 ceph-mon[37158]: from='osd.2 v2:172.21.15.7:6809/904871370' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:16:30.391 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:30 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:30.391 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:16:30 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-2[52839]: 2024-02-05T07:16:30.074+0000 7fab31a02700 -1 osd.2 0 waiting for initial osdmap 2024-02-05T07:16:30.391 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:16:30 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-2[52839]: 2024-02-05T07:16:30.077+0000 7fab2d7fc700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T07:16:30.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:30 smithi090 ceph-mon[33213]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:30 smithi090 ceph-mon[33213]: from='osd.2 v2:172.21.15.7:6809/904871370' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-05T07:16:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:30 smithi090 ceph-mon[33213]: osdmap e16: 3 total, 2 up, 3 in 2024-02-05T07:16:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:30 smithi090 ceph-mon[33213]: from='osd.2 v2:172.21.15.7:6809/904871370' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:16:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:30 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:31.354 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:31 smithi007 ceph-mon[29098]: from='osd.2 v2:172.21.15.7:6809/904871370' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-02-05T07:16:31.354 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:31 smithi007 ceph-mon[29098]: osdmap e17: 3 total, 2 up, 3 in 2024-02-05T07:16:31.354 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:31 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:31.354 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:31 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:31.354 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:31 smithi007 ceph-mon[37158]: from='osd.2 v2:172.21.15.7:6809/904871370' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-02-05T07:16:31.354 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:31 smithi007 ceph-mon[37158]: osdmap e17: 3 total, 2 up, 3 in 2024-02-05T07:16:31.354 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:31 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:31.355 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:31 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:31.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:31 smithi090 ceph-mon[33213]: from='osd.2 v2:172.21.15.7:6809/904871370' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-02-05T07:16:31.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:31 smithi090 ceph-mon[33213]: osdmap e17: 3 total, 2 up, 3 in 2024-02-05T07:16:31.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:31 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:31.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:31 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:32.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:32 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:16:32.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:32 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:16:32.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:32 smithi007 ceph-mon[29098]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:32.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:32 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:32.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:32 smithi007 ceph-mon[29098]: osd.2 v2:172.21.15.7:6809/904871370 boot 2024-02-05T07:16:32.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:32 smithi007 ceph-mon[29098]: osdmap e18: 3 total, 3 up, 3 in 2024-02-05T07:16:32.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:32 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:32.234 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-05T07:16:32.234 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-05T07:16:32.234 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-02-05T07:16:32.234 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-02-05T07:16:32.234 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00881603 s, 1.2 GB/s 2024-02-05T07:16:32.234 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-02-05T07:16:32.234 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:32 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:16:32.235 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:32 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:16:32.235 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:32 smithi007 ceph-mon[37158]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:32.235 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:32 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:32.235 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:32 smithi007 ceph-mon[37158]: osd.2 v2:172.21.15.7:6809/904871370 boot 2024-02-05T07:16:32.235 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:32 smithi007 ceph-mon[37158]: osdmap e18: 3 total, 3 up, 3 in 2024-02-05T07:16:32.235 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:32 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:32 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:16:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:32 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:16:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:32 smithi090 ceph-mon[33213]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T07:16:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:32 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:32 smithi090 ceph-mon[33213]: osd.2 v2:172.21.15.7:6809/904871370 boot 2024-02-05T07:16:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:32 smithi090 ceph-mon[33213]: osdmap e18: 3 total, 3 up, 3 in 2024-02-05T07:16:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:32 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:16:32.821 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi007:vg_nvme/lv_1 2024-02-05T07:16:33.381 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:33 smithi007 ceph-mon[29098]: osdmap e19: 3 total, 3 up, 3 in 2024-02-05T07:16:33.381 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:33 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:16:33.381 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:33 smithi007 ceph-mon[37158]: osdmap e19: 3 total, 3 up, 3 in 2024-02-05T07:16:33.381 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:33 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:16:33.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:33 smithi090 ceph-mon[33213]: osdmap e19: 3 total, 3 up, 3 in 2024-02-05T07:16:33.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:33 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:16:33.542 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:16:34.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: pgmap v78: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:34.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-05T07:16:34.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: osdmap e20: 3 total, 3 up, 3 in 2024-02-05T07:16:34.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:16:34.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: pgmap v78: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: osdmap e20: 3 total, 3 up, 3 in 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:34.427 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: pgmap v78: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-05T07:16:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: osdmap e20: 3 total, 3 up, 3 in 2024-02-05T07:16:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:16:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:35.101 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:16:34 smithi007 sudo[56853]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T07:16:35.101 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:16:34 smithi007 sudo[56853]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T07:16:35.101 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:16:34 smithi007 sudo[56853]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T07:16:35.101 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:16:34 smithi007 sudo[56853]: pam_unix(sudo:session): session closed for user root 2024-02-05T07:16:35.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:35 smithi007 ceph-mon[29098]: Detected new or changed devices on smithi007 2024-02-05T07:16:35.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:35 smithi007 ceph-mon[29098]: Adjusting osd_memory_target on smithi007 to 5356M 2024-02-05T07:16:35.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:35 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:16:35.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:35 smithi007 ceph-mon[29098]: osdmap e21: 3 total, 3 up, 3 in 2024-02-05T07:16:35.424 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[56876]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T07:16:35.424 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[56876]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T07:16:35.424 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[56876]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T07:16:35.425 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[56876]: pam_unix(sudo:session): session closed for user root 2024-02-05T07:16:35.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:35 smithi007 ceph-mon[37158]: Detected new or changed devices on smithi007 2024-02-05T07:16:35.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:35 smithi007 ceph-mon[37158]: Adjusting osd_memory_target on smithi007 to 5356M 2024-02-05T07:16:35.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:35 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:16:35.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:35 smithi007 ceph-mon[37158]: osdmap e21: 3 total, 3 up, 3 in 2024-02-05T07:16:35.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:35 smithi090 ceph-mon[33213]: Detected new or changed devices on smithi007 2024-02-05T07:16:35.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:35 smithi090 ceph-mon[33213]: Adjusting osd_memory_target on smithi007 to 5356M 2024-02-05T07:16:35.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:35 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:16:35.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:35 smithi090 ceph-mon[33213]: osdmap e21: 3 total, 3 up, 3 in 2024-02-05T07:16:35.703 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[56946]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T07:16:35.704 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[56946]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T07:16:35.704 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[56946]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T07:16:35.704 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[56946]: pam_unix(sudo:session): session closed for user root 2024-02-05T07:16:35.967 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[56993]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T07:16:35.967 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[56993]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T07:16:35.967 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[56993]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T07:16:35.967 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[56993]: pam_unix(sudo:session): session closed for user root 2024-02-05T07:16:36.237 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[57014]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T07:16:36.237 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[57014]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T07:16:36.237 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[57014]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T07:16:36.238 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:16:35 smithi007 sudo[57014]: pam_unix(sudo:session): session closed for user root 2024-02-05T07:16:36.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 ceph-mon[29098]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:36.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 ceph-mon[29098]: osdmap e22: 3 total, 3 up, 3 in 2024-02-05T07:16:36.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 ceph-mon[29098]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-05T07:16:36.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:36.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:16:36.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:16:36.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:36.239 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:36 smithi007 ceph-mon[37158]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:36.239 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:36 smithi007 ceph-mon[37158]: osdmap e22: 3 total, 3 up, 3 in 2024-02-05T07:16:36.239 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:36 smithi007 ceph-mon[37158]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-05T07:16:36.239 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:36 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:36.239 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:36 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:16:36.239 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:36 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:16:36.239 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:36 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:36.498 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:16:36 smithi007 sudo[57031]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T07:16:36.498 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:16:36 smithi007 sudo[57031]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T07:16:36.498 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:16:36 smithi007 sudo[57031]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T07:16:36.498 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:16:36 smithi007 sudo[57031]: pam_unix(sudo:session): session closed for user root 2024-02-05T07:16:36.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:36 smithi090 ceph-mon[33213]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:36.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:36 smithi090 ceph-mon[33213]: osdmap e22: 3 total, 3 up, 3 in 2024-02-05T07:16:36.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:36 smithi090 ceph-mon[33213]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-05T07:16:36.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:36 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:36.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:36 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:16:36.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:36 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:16:36.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:36 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:36.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 sudo[57036]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T07:16:36.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 sudo[57036]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T07:16:36.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 sudo[57036]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T07:16:36.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 sudo[57036]: pam_unix(sudo:session): session closed for user root 2024-02-05T07:16:37.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 sudo[57057]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T07:16:37.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 sudo[57057]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T07:16:37.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 sudo[57057]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T07:16:37.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:36 smithi007 sudo[57057]: pam_unix(sudo:session): session closed for user root 2024-02-05T07:16:37.265 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 sudo[35816]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T07:16:37.265 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 sudo[35816]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T07:16:37.265 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 sudo[35816]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T07:16:37.265 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 sudo[35816]: pam_unix(sudo:session): session closed for user root 2024-02-05T07:16:37.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[37158]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:16:37.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[37158]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T07:16:37.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[37158]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T07:16:37.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:16:37.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:16:37.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:16:37.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:16:37.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:16:37.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:16:37.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[29098]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:16:37.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[29098]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T07:16:37.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[29098]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T07:16:37.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:16:37.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:16:37.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:16:37.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:16:37.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:16:37.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:16:37.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 sudo[35820]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T07:16:37.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 sudo[35820]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T07:16:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 sudo[35820]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T07:16:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 sudo[35820]: pam_unix(sudo:session): session closed for user root 2024-02-05T07:16:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 ceph-mon[33213]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:16:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 ceph-mon[33213]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T07:16:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 ceph-mon[33213]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T07:16:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:16:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:16:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:16:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:16:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:16:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:16:37.812 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 sudo[57113]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T07:16:37.812 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 sudo[57113]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T07:16:37.812 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 sudo[57113]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T07:16:37.812 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 sudo[57113]: pam_unix(sudo:session): session closed for user root 2024-02-05T07:16:38.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 sudo[57117]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T07:16:38.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 sudo[57117]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T07:16:38.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 sudo[57117]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T07:16:38.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:37 smithi007 sudo[57117]: pam_unix(sudo:session): session closed for user root 2024-02-05T07:16:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[29098]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[29098]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T07:16:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[29098]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T07:16:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[29098]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T07:16:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:16:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:16:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:16:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1530740465' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "149ed0d6-b049-4478-9819-cafc446e94d3"}]: dispatch 2024-02-05T07:16:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[29098]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T07:16:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1530740465' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "149ed0d6-b049-4478-9819-cafc446e94d3"}]': finished 2024-02-05T07:16:38.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[29098]: osdmap e23: 4 total, 3 up, 4 in 2024-02-05T07:16:38.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[37158]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[37158]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T07:16:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[37158]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T07:16:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[37158]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T07:16:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:16:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:16:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:16:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1530740465' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "149ed0d6-b049-4478-9819-cafc446e94d3"}]: dispatch 2024-02-05T07:16:38.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[37158]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T07:16:38.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1530740465' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "149ed0d6-b049-4478-9819-cafc446e94d3"}]': finished 2024-02-05T07:16:38.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[37158]: osdmap e23: 4 total, 3 up, 4 in 2024-02-05T07:16:38.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:38 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:38.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:38 smithi090 ceph-mon[33213]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:38.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:38 smithi090 ceph-mon[33213]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T07:16:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:38 smithi090 ceph-mon[33213]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T07:16:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:38 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:38 smithi090 ceph-mon[33213]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T07:16:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:38 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:16:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:38 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:16:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:38 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:16:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:38 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1530740465' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "149ed0d6-b049-4478-9819-cafc446e94d3"}]: dispatch 2024-02-05T07:16:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:38 smithi090 ceph-mon[33213]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T07:16:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:38 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1530740465' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "149ed0d6-b049-4478-9819-cafc446e94d3"}]': finished 2024-02-05T07:16:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:38 smithi090 ceph-mon[33213]: osdmap e23: 4 total, 3 up, 4 in 2024-02-05T07:16:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:38 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:39.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:39 smithi007 ceph-mon[29098]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:39.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:39 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/606979604' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:16:39.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:39 smithi007 ceph-mon[37158]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:39.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:39 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/606979604' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:16:39.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:39 smithi090 ceph-mon[33213]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:39.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:39 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/606979604' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:16:41.364 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:41 smithi007 ceph-mon[29098]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:41.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:41 smithi007 ceph-mon[37158]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:41.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:41 smithi090 ceph-mon[33213]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:43.612 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:43 smithi007 ceph-mon[29098]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.1 KiB/s rd, 179 KiB/s wr, 12 op/s 2024-02-05T07:16:43.612 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:43 smithi007 ceph-mon[37158]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.1 KiB/s rd, 179 KiB/s wr, 12 op/s 2024-02-05T07:16:43.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:43 smithi090 ceph-mon[33213]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.1 KiB/s rd, 179 KiB/s wr, 12 op/s 2024-02-05T07:16:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-05T07:16:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:45.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:45 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-05T07:16:45.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:45 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:45.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:45 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-05T07:16:45.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:45 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:46.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:46 smithi007 ceph-mon[29098]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 123 KiB/s wr, 8 op/s 2024-02-05T07:16:46.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:46 smithi007 ceph-mon[29098]: Deploying daemon osd.3 on smithi007 2024-02-05T07:16:46.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:46 smithi007 ceph-mon[37158]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 123 KiB/s wr, 8 op/s 2024-02-05T07:16:46.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:46 smithi007 ceph-mon[37158]: Deploying daemon osd.3 on smithi007 2024-02-05T07:16:46.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:46 smithi090 ceph-mon[33213]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 123 KiB/s wr, 8 op/s 2024-02-05T07:16:46.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:46 smithi090 ceph-mon[33213]: Deploying daemon osd.3 on smithi007 2024-02-05T07:16:48.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:48 smithi007 ceph-mon[29098]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 94 KiB/s wr, 6 op/s 2024-02-05T07:16:48.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:48 smithi007 ceph-mon[37158]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 94 KiB/s wr, 6 op/s 2024-02-05T07:16:48.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:48 smithi090 ceph-mon[33213]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 94 KiB/s wr, 6 op/s 2024-02-05T07:16:50.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:50 smithi007 ceph-mon[29098]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-02-05T07:16:50.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:50 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:50.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:50 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:50.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:50 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:50.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:50 smithi007 ceph-mon[37158]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-02-05T07:16:50.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:50 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:50.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:50 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:50.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:50 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:50.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:50 smithi090 ceph-mon[33213]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-02-05T07:16:50.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:50 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:50.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:50 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:50.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:50 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:52.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:52 smithi007 ceph-mon[29098]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-05T07:16:52.244 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:52 smithi007 ceph-mon[37158]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-05T07:16:52.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:52 smithi090 ceph-mon[33213]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-05T07:16:52.892 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 3 on host 'smithi007' 2024-02-05T07:16:53.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:53 smithi007 ceph-mon[29098]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T07:16:53.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:53 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:53.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:53 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:53.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:53 smithi007 ceph-mon[37158]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T07:16:53.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:53 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:53.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:53 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:53.492 DEBUG:teuthology.orchestra.run.smithi007:osd.3> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.3.service 2024-02-05T07:16:53.494 INFO:tasks.cephadm:Deploying osd.4 on smithi090 with /dev/vg_nvme/lv_4... 2024-02-05T07:16:53.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-05T07:16:53.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:53 smithi090 ceph-mon[33213]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T07:16:53.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:53 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:53.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:53 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:53.741 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:16:54.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-mon[29098]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-05T07:16:54.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-mon[29098]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-05T07:16:54.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-mon[29098]: osdmap e24: 4 total, 3 up, 4 in 2024-02-05T07:16:54.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-mon[29098]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:16:54.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:54.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:54.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:54.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-mon[37158]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-05T07:16:54.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-mon[37158]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-05T07:16:54.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-mon[37158]: osdmap e24: 4 total, 3 up, 4 in 2024-02-05T07:16:54.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-mon[37158]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:16:54.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:54.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:54.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:54.419 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3[58793]: 2024-02-05T07:16:54.334+0000 7fba29535700 -1 osd.3 0 waiting for initial osdmap 2024-02-05T07:16:54.419 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:16:54 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3[58793]: 2024-02-05T07:16:54.337+0000 7fba21327700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T07:16:54.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:54 smithi090 ceph-mon[33213]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-05T07:16:54.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:54 smithi090 ceph-mon[33213]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-05T07:16:54.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:54 smithi090 ceph-mon[33213]: osdmap e24: 4 total, 3 up, 4 in 2024-02-05T07:16:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:54 smithi090 ceph-mon[33213]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-02-05T07:16:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:54 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:54 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:54 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:55 smithi090 ceph-mon[33213]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-02-05T07:16:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:55 smithi090 ceph-mon[33213]: osdmap e25: 4 total, 3 up, 4 in 2024-02-05T07:16:55.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:55 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:55.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:55 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:55.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:55 smithi090 ceph-mon[33213]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:55.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:55 smithi090 ceph-mon[33213]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' 2024-02-05T07:16:55.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:55 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:55.380 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:55 smithi007 ceph-mon[29098]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-02-05T07:16:55.381 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:55 smithi007 ceph-mon[29098]: osdmap e25: 4 total, 3 up, 4 in 2024-02-05T07:16:55.381 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:55 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:55.381 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:55 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:55.381 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:55 smithi007 ceph-mon[29098]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:55.381 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:55 smithi007 ceph-mon[29098]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' 2024-02-05T07:16:55.381 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:55 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:55.381 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:55 smithi007 ceph-mon[37158]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-02-05T07:16:55.381 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:55 smithi007 ceph-mon[37158]: osdmap e25: 4 total, 3 up, 4 in 2024-02-05T07:16:55.381 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:55 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:55.381 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:55 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:55.381 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:55 smithi007 ceph-mon[37158]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T07:16:55.381 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:55 smithi007 ceph-mon[37158]: from='osd.3 v2:172.21.15.7:6813/2758586242' entity='osd.3' 2024-02-05T07:16:55.381 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:55 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:55.583 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-05T07:16:55.583 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-05T07:16:55.583 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-02-05T07:16:55.583 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-02-05T07:16:55.584 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01142 s, 918 MB/s 2024-02-05T07:16:55.584 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-02-05T07:16:56.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi090:vg_nvme/lv_4 2024-02-05T07:16:56.473 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:16:56.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:56 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:16:56.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:56 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:16:56.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:56 smithi090 ceph-mon[33213]: osd.3 v2:172.21.15.7:6813/2758586242 boot 2024-02-05T07:16:56.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:56 smithi090 ceph-mon[33213]: osdmap e26: 4 total, 4 up, 4 in 2024-02-05T07:16:56.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:56 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:56.661 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:56 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:16:56.661 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:56 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:16:56.661 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:56 smithi007 ceph-mon[29098]: osd.3 v2:172.21.15.7:6813/2758586242 boot 2024-02-05T07:16:56.661 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:56 smithi007 ceph-mon[29098]: osdmap e26: 4 total, 4 up, 4 in 2024-02-05T07:16:56.662 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:56 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:56.662 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:56 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:16:56.662 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:56 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:16:56.662 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:56 smithi007 ceph-mon[37158]: osd.3 v2:172.21.15.7:6813/2758586242 boot 2024-02-05T07:16:56.662 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:56 smithi007 ceph-mon[37158]: osdmap e26: 4 total, 4 up, 4 in 2024-02-05T07:16:56.662 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:56 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:16:57.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:16:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: osdmap e27: 4 total, 4 up, 4 in 2024-02-05T07:16:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: Detected new or changed devices on smithi007 2024-02-05T07:16:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: Adjusting osd_memory_target on smithi007 to 4017M 2024-02-05T07:16:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:57.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:57.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:57.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:16:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: osdmap e27: 4 total, 4 up, 4 in 2024-02-05T07:16:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: Detected new or changed devices on smithi007 2024-02-05T07:16:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:57.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:57.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: Adjusting osd_memory_target on smithi007 to 4017M 2024-02-05T07:16:57.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:57.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:57.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:57.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:57.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:57.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:57.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:57.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:57.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:16:57.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: osdmap e27: 4 total, 4 up, 4 in 2024-02-05T07:16:57.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: Detected new or changed devices on smithi007 2024-02-05T07:16:57.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:57.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:57.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:57.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:57.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:57.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:16:57.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: Adjusting osd_memory_target on smithi007 to 4017M 2024-02-05T07:16:57.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:57.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:57.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:57.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:57.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:16:57.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:57.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:16:57.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:16:58.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:58 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:16:58.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:58 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:16:58.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:16:58 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:58.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:16:58.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:16:58.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:16:58 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:16:58.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:58 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:16:58.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:58 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:16:58.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:16:58 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:00.226 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:00 smithi090 ceph-mon[33213]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:00.227 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:00 smithi090 ceph-mon[33213]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:17:00.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:00 smithi007 ceph-mon[29098]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:00.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:00 smithi007 ceph-mon[29098]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:17:00.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:00 smithi007 ceph-mon[37158]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:00.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:00 smithi007 ceph-mon[37158]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:17:01.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:01 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/4093259015' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d"}]: dispatch 2024-02-05T07:17:01.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:01 smithi007 ceph-mon[29098]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d"}]: dispatch 2024-02-05T07:17:01.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:01 smithi007 ceph-mon[29098]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d"}]': finished 2024-02-05T07:17:01.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:01 smithi007 ceph-mon[29098]: osdmap e28: 5 total, 4 up, 5 in 2024-02-05T07:17:01.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:01 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:01.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:01 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/4093259015' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d"}]: dispatch 2024-02-05T07:17:01.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:01 smithi007 ceph-mon[37158]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d"}]: dispatch 2024-02-05T07:17:01.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:01 smithi007 ceph-mon[37158]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d"}]': finished 2024-02-05T07:17:01.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:01 smithi007 ceph-mon[37158]: osdmap e28: 5 total, 4 up, 5 in 2024-02-05T07:17:01.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:01 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:01.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:01 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/4093259015' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d"}]: dispatch 2024-02-05T07:17:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:01 smithi090 ceph-mon[33213]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d"}]: dispatch 2024-02-05T07:17:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:01 smithi090 ceph-mon[33213]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d"}]': finished 2024-02-05T07:17:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:01 smithi090 ceph-mon[33213]: osdmap e28: 5 total, 4 up, 5 in 2024-02-05T07:17:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:01 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:02 smithi007 ceph-mon[29098]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:02 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/3526924245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:17:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:02 smithi007 ceph-mon[37158]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:02 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/3526924245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:17:02.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:02 smithi090 ceph-mon[33213]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:02.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:02 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/3526924245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:17:04.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:04 smithi090 ceph-mon[33213]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:04 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:17:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:04 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:17:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:04 smithi090 ceph-mon[33213]: osdmap e29: 5 total, 4 up, 5 in 2024-02-05T07:17:04.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:04 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:04 smithi007 ceph-mon[29098]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:04 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:17:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:04 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:17:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:04 smithi007 ceph-mon[29098]: osdmap e29: 5 total, 4 up, 5 in 2024-02-05T07:17:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:04 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:04 smithi007 ceph-mon[37158]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:04 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:17:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:04 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:17:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:04 smithi007 ceph-mon[37158]: osdmap e29: 5 total, 4 up, 5 in 2024-02-05T07:17:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:04 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:06.413 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:06 smithi090 ceph-mon[33213]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:06 smithi007 ceph-mon[29098]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:06.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:06 smithi007 ceph-mon[37158]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:08.332 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:08 smithi090 ceph-mon[33213]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:08.332 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:08 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-05T07:17:08.332 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:08 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:08.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:08 smithi007 ceph-mon[29098]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:08.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:08 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-05T07:17:08.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:08 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:08.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:08 smithi007 ceph-mon[37158]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:08.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:08 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-05T07:17:08.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:08 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:09.377 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:09 smithi090 ceph-mon[33213]: Deploying daemon osd.4 on smithi090 2024-02-05T07:17:09.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:09 smithi007 ceph-mon[29098]: Deploying daemon osd.4 on smithi090 2024-02-05T07:17:09.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:09 smithi007 ceph-mon[37158]: Deploying daemon osd.4 on smithi090 2024-02-05T07:17:10.321 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:10 smithi090 ceph-mon[33213]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:10.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:10 smithi007 ceph-mon[29098]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:10.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:10 smithi007 ceph-mon[37158]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:12.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:12 smithi090 ceph-mon[33213]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:12.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:12 smithi007 ceph-mon[29098]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:12.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:12 smithi007 ceph-mon[37158]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:13 smithi090 ceph-mon[33213]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:14.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:13 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:14.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:13 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:17:14.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:13 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:14.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:13 smithi007 ceph-mon[29098]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:14.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:13 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:14.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:13 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:17:14.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:13 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:14.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:13 smithi007 ceph-mon[37158]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:14.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:13 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:14.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:13 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:17:14.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:13 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:16.411 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:16 smithi090 ceph-mon[33213]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:16.412 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:16 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:16.412 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:16 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:16.412 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:16 smithi090 ceph-mon[33213]: from='osd.4 v2:172.21.15.90:6800/1154929470' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T07:17:16.412 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:16 smithi090 ceph-mon[33213]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T07:17:16.412 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:16 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:16.412 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:16 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:17:16.412 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:16 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[29098]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[29098]: from='osd.4 v2:172.21.15.90:6800/1154929470' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T07:17:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[29098]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T07:17:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:17:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:16.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[37158]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:16.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:16.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:16.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[37158]: from='osd.4 v2:172.21.15.90:6800/1154929470' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T07:17:16.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[37158]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T07:17:16.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:16.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:17:16.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:16 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:16.496 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 4 on host 'smithi090' 2024-02-05T07:17:17.094 DEBUG:teuthology.orchestra.run.smithi090:osd.4> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.4.service 2024-02-05T07:17:17.096 INFO:tasks.cephadm:Deploying osd.5 on smithi090 with /dev/vg_nvme/lv_3... 2024-02-05T07:17:17.096 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-05T07:17:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[29098]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[29098]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-05T07:17:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[29098]: from='osd.4 v2:172.21.15.90:6800/1154929470' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:17:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[29098]: osdmap e30: 5 total, 4 up, 5 in 2024-02-05T07:17:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[29098]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:17:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:17.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:17:17.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[37158]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:17.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[37158]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-05T07:17:17.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[37158]: from='osd.4 v2:172.21.15.90:6800/1154929470' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:17:17.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[37158]: osdmap e30: 5 total, 4 up, 5 in 2024-02-05T07:17:17.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:17.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[37158]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:17:17.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:17.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:17.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:17 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:17:17.751 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:17:17.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:17 smithi090 ceph-mon[33213]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:17.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:17 smithi090 ceph-mon[33213]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-05T07:17:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:17 smithi090 ceph-mon[33213]: from='osd.4 v2:172.21.15.90:6800/1154929470' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:17:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:17 smithi090 ceph-mon[33213]: osdmap e30: 5 total, 4 up, 5 in 2024-02-05T07:17:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:17 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:17 smithi090 ceph-mon[33213]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:17:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:17 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:17 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:17 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:17:18.110 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:17:17 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:17:17.784+0000 7f006c974700 -1 osd.4 0 waiting for initial osdmap 2024-02-05T07:17:18.110 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:17:17 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:17:17.788+0000 7f0063f65700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T07:17:18.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:18 smithi090 ceph-mon[33213]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-05T07:17:18.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:18 smithi090 ceph-mon[33213]: osdmap e31: 5 total, 4 up, 5 in 2024-02-05T07:17:18.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:18 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:18.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:18 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:18.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:18 smithi090 ceph-mon[33213]: from='osd.4 ' entity='osd.4' 2024-02-05T07:17:18.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:18 smithi007 ceph-mon[29098]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-05T07:17:18.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:18 smithi007 ceph-mon[29098]: osdmap e31: 5 total, 4 up, 5 in 2024-02-05T07:17:18.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:18 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:18.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:18 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:18.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:18 smithi007 ceph-mon[29098]: from='osd.4 ' entity='osd.4' 2024-02-05T07:17:18.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:18 smithi007 ceph-mon[37158]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-05T07:17:18.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:18 smithi007 ceph-mon[37158]: osdmap e31: 5 total, 4 up, 5 in 2024-02-05T07:17:18.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:18 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:18.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:18 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:18.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:18 smithi007 ceph-mon[37158]: from='osd.4 ' entity='osd.4' 2024-02-05T07:17:19.471 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:19 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:17:19.471 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:19 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:17:19.471 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:19 smithi090 ceph-mon[33213]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:19.471 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:19 smithi090 ceph-mon[33213]: osd.4 v2:172.21.15.90:6800/1154929470 boot 2024-02-05T07:17:19.472 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:19 smithi090 ceph-mon[33213]: osdmap e32: 5 total, 5 up, 5 in 2024-02-05T07:17:19.472 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:19 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:19.472 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:19 smithi090 ceph-mon[33213]: osdmap e33: 5 total, 5 up, 5 in 2024-02-05T07:17:19.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:19 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:17:19.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:19 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:17:19.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:19 smithi007 ceph-mon[29098]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:19.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:19 smithi007 ceph-mon[29098]: osd.4 v2:172.21.15.90:6800/1154929470 boot 2024-02-05T07:17:19.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:19 smithi007 ceph-mon[29098]: osdmap e32: 5 total, 5 up, 5 in 2024-02-05T07:17:19.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:19 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:19.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:19 smithi007 ceph-mon[29098]: osdmap e33: 5 total, 5 up, 5 in 2024-02-05T07:17:19.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:19 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:17:19.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:19 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:17:19.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:19 smithi007 ceph-mon[37158]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-05T07:17:19.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:19 smithi007 ceph-mon[37158]: osd.4 v2:172.21.15.90:6800/1154929470 boot 2024-02-05T07:17:19.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:19 smithi007 ceph-mon[37158]: osdmap e32: 5 total, 5 up, 5 in 2024-02-05T07:17:19.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:19 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:17:19.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:19 smithi007 ceph-mon[37158]: osdmap e33: 5 total, 5 up, 5 in 2024-02-05T07:17:20.508 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-05T07:17:20.509 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-05T07:17:20.509 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-02-05T07:17:20.509 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-02-05T07:17:20.509 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0149138 s, 703 MB/s 2024-02-05T07:17:20.509 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-02-05T07:17:21.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:21 smithi007 ceph-mon[29098]: osdmap e34: 5 total, 5 up, 5 in 2024-02-05T07:17:21.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:21 smithi007 ceph-mon[37158]: osdmap e34: 5 total, 5 up, 5 in 2024-02-05T07:17:21.437 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:21 smithi090 ceph-mon[33213]: osdmap e34: 5 total, 5 up, 5 in 2024-02-05T07:17:21.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi090:vg_nvme/lv_3 2024-02-05T07:17:22.016 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:17:22.351 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:22 smithi090 ceph-mon[33213]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:22.351 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:22 smithi090 ceph-mon[33213]: osdmap e35: 5 total, 5 up, 5 in 2024-02-05T07:17:22.351 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:22 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:22.351 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:22 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:22.351 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:22 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:17:22.352 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:22 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:22.352 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:22 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:22.352 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:22 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:17:22.352 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:22 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:22.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[29098]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:22.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[29098]: osdmap e35: 5 total, 5 up, 5 in 2024-02-05T07:17:22.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:22.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:22.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:17:22.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:22.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:22.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:17:22.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:22.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[37158]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:22.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[37158]: osdmap e35: 5 total, 5 up, 5 in 2024-02-05T07:17:22.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:22.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:22.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:17:22.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:22.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:22.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:17:22.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:22 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:23.364 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:23 smithi090 ceph-mon[33213]: Detected new or changed devices on smithi090 2024-02-05T07:17:23.365 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:23 smithi090 ceph-mon[33213]: Adjusting osd_memory_target on smithi090 to 17088M 2024-02-05T07:17:23.365 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:23 smithi090 ceph-mon[33213]: osdmap e36: 5 total, 5 up, 5 in 2024-02-05T07:17:23.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:23 smithi007 ceph-mon[29098]: Detected new or changed devices on smithi090 2024-02-05T07:17:23.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:23 smithi007 ceph-mon[29098]: Adjusting osd_memory_target on smithi090 to 17088M 2024-02-05T07:17:23.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:23 smithi007 ceph-mon[29098]: osdmap e36: 5 total, 5 up, 5 in 2024-02-05T07:17:23.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:23 smithi007 ceph-mon[37158]: Detected new or changed devices on smithi090 2024-02-05T07:17:23.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:23 smithi007 ceph-mon[37158]: Adjusting osd_memory_target on smithi090 to 17088M 2024-02-05T07:17:23.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:23 smithi007 ceph-mon[37158]: osdmap e36: 5 total, 5 up, 5 in 2024-02-05T07:17:24.389 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:24 smithi090 ceph-mon[33213]: pgmap v120: 1 pgs: 1 activating; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 150 KiB/s, 0 objects/s recovering 2024-02-05T07:17:24.390 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:24 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:17:24.390 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:24 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:17:24.390 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:24 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:24.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:24 smithi007 ceph-mon[29098]: pgmap v120: 1 pgs: 1 activating; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 150 KiB/s, 0 objects/s recovering 2024-02-05T07:17:24.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:24 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:17:24.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:24 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:17:24.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:24 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:24.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:24 smithi007 ceph-mon[37158]: pgmap v120: 1 pgs: 1 activating; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 150 KiB/s, 0 objects/s recovering 2024-02-05T07:17:24.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:24 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:17:24.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:24 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:17:24.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:24 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:25 smithi090 ceph-mon[33213]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:17:25.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:25 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:17:25.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:25 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:17:25.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:25 smithi007 ceph-mon[29098]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:17:25.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:25 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:17:25.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:25 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:17:25.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:25 smithi007 ceph-mon[37158]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:17:25.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:25 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:17:25.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:25 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:17:26.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:26 smithi090 ceph-mon[33213]: pgmap v121: 1 pgs: 1 activating; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-02-05T07:17:26.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:26 smithi090 ceph-mon[33213]: osdmap e37: 5 total, 5 up, 5 in 2024-02-05T07:17:26.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:26 smithi007 ceph-mon[29098]: pgmap v121: 1 pgs: 1 activating; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-02-05T07:17:26.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:26 smithi007 ceph-mon[29098]: osdmap e37: 5 total, 5 up, 5 in 2024-02-05T07:17:26.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:26 smithi007 ceph-mon[37158]: pgmap v121: 1 pgs: 1 activating; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-02-05T07:17:26.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:26 smithi007 ceph-mon[37158]: osdmap e37: 5 total, 5 up, 5 in 2024-02-05T07:17:27.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:27 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/2172435739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "244a7930-c22c-413e-86e1-11e0749f7367"}]: dispatch 2024-02-05T07:17:27.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:27 smithi007 ceph-mon[37158]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "244a7930-c22c-413e-86e1-11e0749f7367"}]: dispatch 2024-02-05T07:17:27.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:27 smithi007 ceph-mon[37158]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "244a7930-c22c-413e-86e1-11e0749f7367"}]': finished 2024-02-05T07:17:27.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:27 smithi007 ceph-mon[37158]: osdmap e38: 6 total, 5 up, 6 in 2024-02-05T07:17:27.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:27 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:27.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:27 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/2172435739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "244a7930-c22c-413e-86e1-11e0749f7367"}]: dispatch 2024-02-05T07:17:27.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:27 smithi007 ceph-mon[29098]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "244a7930-c22c-413e-86e1-11e0749f7367"}]: dispatch 2024-02-05T07:17:27.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:27 smithi007 ceph-mon[29098]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "244a7930-c22c-413e-86e1-11e0749f7367"}]': finished 2024-02-05T07:17:27.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:27 smithi007 ceph-mon[29098]: osdmap e38: 6 total, 5 up, 6 in 2024-02-05T07:17:27.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:27 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:27.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:27 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/2172435739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "244a7930-c22c-413e-86e1-11e0749f7367"}]: dispatch 2024-02-05T07:17:27.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:27 smithi090 ceph-mon[33213]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "244a7930-c22c-413e-86e1-11e0749f7367"}]: dispatch 2024-02-05T07:17:27.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:27 smithi090 ceph-mon[33213]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "244a7930-c22c-413e-86e1-11e0749f7367"}]': finished 2024-02-05T07:17:27.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:27 smithi090 ceph-mon[33213]: osdmap e38: 6 total, 5 up, 6 in 2024-02-05T07:17:27.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:27 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:28.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:28 smithi007 ceph-mon[29098]: pgmap v123: 1 pgs: 1 activating; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T07:17:28.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:28 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/2933005243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:17:28.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:28 smithi007 ceph-mon[37158]: pgmap v123: 1 pgs: 1 activating; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T07:17:28.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:28 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/2933005243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:17:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:28 smithi090 ceph-mon[33213]: pgmap v123: 1 pgs: 1 activating; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T07:17:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:28 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/2933005243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:17:30.256 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:30 smithi090 ceph-mon[33213]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-02-05T07:17:30.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:30 smithi007 ceph-mon[29098]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-02-05T07:17:30.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:30 smithi007 ceph-mon[37158]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-02-05T07:17:32.169 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:32 smithi090 ceph-mon[33213]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:32.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:32 smithi007 ceph-mon[29098]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:32.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:32 smithi007 ceph-mon[37158]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:34.172 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:34 smithi090 ceph-mon[33213]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:34.172 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-05T07:17:34.172 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:34.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:34 smithi007 ceph-mon[29098]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:34.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-05T07:17:34.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:34.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:34 smithi007 ceph-mon[37158]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:34.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-05T07:17:34.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:35.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:35 smithi007 ceph-mon[29098]: Deploying daemon osd.5 on smithi090 2024-02-05T07:17:35.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:35 smithi007 ceph-mon[37158]: Deploying daemon osd.5 on smithi090 2024-02-05T07:17:35.458 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:35 smithi090 ceph-mon[33213]: Deploying daemon osd.5 on smithi090 2024-02-05T07:17:36.399 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:36 smithi090 ceph-mon[33213]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:36.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:36 smithi007 ceph-mon[29098]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:36.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:36 smithi007 ceph-mon[37158]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:38.378 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:38 smithi090 ceph-mon[33213]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:38.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:38 smithi007 ceph-mon[29098]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:38.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:38 smithi007 ceph-mon[37158]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:39 smithi090 ceph-mon[33213]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:39 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:39 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:39 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:17:39.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:39 smithi007 ceph-mon[29098]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:39.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:39 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:39.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:39 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:39.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:39 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:17:39.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:39 smithi007 ceph-mon[37158]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:39.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:39 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:39.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:39 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:39.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:39 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:17:41.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:41 smithi007 ceph-mon[29098]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:41.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:41 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:41.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:41 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:41.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:41 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:41.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:41 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:17:41.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:41 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:41.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:41 smithi007 ceph-mon[37158]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:41.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:41 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:41.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:41 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:41.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:41 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:41.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:41 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:17:41.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:41 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:41.974 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:41 smithi090 ceph-mon[33213]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:41.974 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:41 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:41.974 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:41 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:41.974 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:41 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:41.975 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:41 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:17:41.975 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:41 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:42.304 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 5 on host 'smithi090' 2024-02-05T07:17:42.875 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:42 smithi090 ceph-mon[33213]: from='osd.5 v2:172.21.15.90:6804/398113577' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T07:17:42.875 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:42 smithi090 ceph-mon[33213]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T07:17:42.875 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:42 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:42.875 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:42 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:42.875 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:42 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:17:42.901 DEBUG:teuthology.orchestra.run.smithi090:osd.5> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.5.service 2024-02-05T07:17:42.903 INFO:tasks.cephadm:Deploying osd.6 on smithi090 with /dev/vg_nvme/lv_2... 2024-02-05T07:17:42.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-05T07:17:42.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:42 smithi007 ceph-mon[29098]: from='osd.5 v2:172.21.15.90:6804/398113577' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T07:17:42.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:42 smithi007 ceph-mon[29098]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T07:17:42.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:42.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:42.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:42 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:17:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:42 smithi007 ceph-mon[37158]: from='osd.5 v2:172.21.15.90:6804/398113577' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T07:17:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:42 smithi007 ceph-mon[37158]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T07:17:42.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:42 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:42.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:42 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:42.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:42 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:17:43.647 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:17:43.888 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:43 smithi090 ceph-mon[33213]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:43.888 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:43 smithi090 ceph-mon[33213]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-05T07:17:43.888 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:43 smithi090 ceph-mon[33213]: osdmap e39: 6 total, 5 up, 6 in 2024-02-05T07:17:43.888 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:43 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:43.888 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:43 smithi090 ceph-mon[33213]: from='osd.5 v2:172.21.15.90:6804/398113577' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:17:43.888 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:43 smithi090 ceph-mon[33213]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:17:43.888 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:17:43 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:17:43.635+0000 7ff1720b7700 -1 osd.5 0 waiting for initial osdmap 2024-02-05T07:17:43.888 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:17:43 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:17:43.640+0000 7ff16deb1700 -1 osd.5 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T07:17:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:43 smithi007 ceph-mon[29098]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:43 smithi007 ceph-mon[29098]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-05T07:17:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:43 smithi007 ceph-mon[29098]: osdmap e39: 6 total, 5 up, 6 in 2024-02-05T07:17:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:43 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:43 smithi007 ceph-mon[29098]: from='osd.5 v2:172.21.15.90:6804/398113577' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:17:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:43 smithi007 ceph-mon[29098]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:17:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:43 smithi007 ceph-mon[37158]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:43 smithi007 ceph-mon[37158]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-05T07:17:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:43 smithi007 ceph-mon[37158]: osdmap e39: 6 total, 5 up, 6 in 2024-02-05T07:17:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:43 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:43 smithi007 ceph-mon[37158]: from='osd.5 v2:172.21.15.90:6804/398113577' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:17:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:43 smithi007 ceph-mon[37158]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:17:44.824 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:44 smithi090 ceph-mon[33213]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-05T07:17:44.824 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:44 smithi090 ceph-mon[33213]: osdmap e40: 6 total, 5 up, 6 in 2024-02-05T07:17:44.824 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:44 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:44.824 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:44 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:44.824 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:44 smithi090 ceph-mon[33213]: osd.5 v2:172.21.15.90:6804/398113577 boot 2024-02-05T07:17:44.824 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:44 smithi090 ceph-mon[33213]: osdmap e41: 6 total, 6 up, 6 in 2024-02-05T07:17:44.825 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:44 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:44 smithi007 ceph-mon[29098]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-05T07:17:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:44 smithi007 ceph-mon[29098]: osdmap e40: 6 total, 5 up, 6 in 2024-02-05T07:17:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:44 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:44 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:44 smithi007 ceph-mon[29098]: osd.5 v2:172.21.15.90:6804/398113577 boot 2024-02-05T07:17:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:44 smithi007 ceph-mon[29098]: osdmap e41: 6 total, 6 up, 6 in 2024-02-05T07:17:44.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:44 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:44.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:44 smithi007 ceph-mon[37158]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-05T07:17:44.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:44 smithi007 ceph-mon[37158]: osdmap e40: 6 total, 5 up, 6 in 2024-02-05T07:17:44.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:44 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:44.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:44 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:44.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:44 smithi007 ceph-mon[37158]: osd.5 v2:172.21.15.90:6804/398113577 boot 2024-02-05T07:17:44.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:44 smithi007 ceph-mon[37158]: osdmap e41: 6 total, 6 up, 6 in 2024-02-05T07:17:44.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:44 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:17:45.673 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:45 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:17:45.673 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:45 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:17:45.673 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:45 smithi090 ceph-mon[33213]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:45.673 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:45 smithi090 ceph-mon[33213]: osdmap e42: 6 total, 6 up, 6 in 2024-02-05T07:17:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:45 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:17:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:45 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:17:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:45 smithi007 ceph-mon[29098]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:45 smithi007 ceph-mon[29098]: osdmap e42: 6 total, 6 up, 6 in 2024-02-05T07:17:45.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:45 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:17:45.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:45 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:17:45.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:45 smithi007 ceph-mon[37158]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-05T07:17:45.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:45 smithi007 ceph-mon[37158]: osdmap e42: 6 total, 6 up, 6 in 2024-02-05T07:17:45.981 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-05T07:17:45.981 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-05T07:17:45.981 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-02-05T07:17:45.981 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-02-05T07:17:45.981 INFO:teuthology.orchestra.run.smithi090.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140812 s, 745 MB/s 2024-02-05T07:17:45.981 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-02-05T07:17:46.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi090:vg_nvme/lv_2 2024-02-05T07:17:47.254 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:47 smithi090 ceph-mon[33213]: osdmap e43: 6 total, 6 up, 6 in 2024-02-05T07:17:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:17:47.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:47 smithi007 ceph-mon[29098]: osdmap e43: 6 total, 6 up, 6 in 2024-02-05T07:17:47.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:47 smithi007 ceph-mon[37158]: osdmap e43: 6 total, 6 up, 6 in 2024-02-05T07:17:48.338 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:48 smithi090 ceph-mon[33213]: pgmap v139: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:48.338 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:48 smithi090 ceph-mon[33213]: osdmap e44: 6 total, 6 up, 6 in 2024-02-05T07:17:48.338 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:48 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:48.338 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:48 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:48.339 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:48 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:17:48.339 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:48 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:17:48.339 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:48 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:48.339 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:48 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:48.339 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:48 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:17:48.339 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:48 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:48.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[29098]: pgmap v139: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:48.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[29098]: osdmap e44: 6 total, 6 up, 6 in 2024-02-05T07:17:48.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:48.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:48.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:17:48.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:17:48.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:48.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:48.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:17:48.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:48.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[37158]: pgmap v139: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:48.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[37158]: osdmap e44: 6 total, 6 up, 6 in 2024-02-05T07:17:48.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:48.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:48.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:17:48.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:17:48.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:48.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:48.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:17:48.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:48 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:17:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:49 smithi007 ceph-mon[29098]: Detected new or changed devices on smithi090 2024-02-05T07:17:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:49 smithi007 ceph-mon[29098]: Adjusting osd_memory_target on smithi090 to 8544M 2024-02-05T07:17:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:49 smithi007 ceph-mon[29098]: osdmap e45: 6 total, 6 up, 6 in 2024-02-05T07:17:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:49 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:17:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:49 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:17:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:49 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:49.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:49 smithi007 ceph-mon[37158]: Detected new or changed devices on smithi090 2024-02-05T07:17:49.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:49 smithi007 ceph-mon[37158]: Adjusting osd_memory_target on smithi090 to 8544M 2024-02-05T07:17:49.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:49 smithi007 ceph-mon[37158]: osdmap e45: 6 total, 6 up, 6 in 2024-02-05T07:17:49.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:49 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:17:49.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:49 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:17:49.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:49 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:49.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:49 smithi090 ceph-mon[33213]: Detected new or changed devices on smithi090 2024-02-05T07:17:49.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:49 smithi090 ceph-mon[33213]: Adjusting osd_memory_target on smithi090 to 8544M 2024-02-05T07:17:49.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:49 smithi090 ceph-mon[33213]: osdmap e45: 6 total, 6 up, 6 in 2024-02-05T07:17:49.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:17:49.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:17:49.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:50.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:50 smithi090 ceph-mon[33213]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:50.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:50 smithi090 ceph-mon[33213]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:17:50.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:50 smithi007 ceph-mon[29098]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:50.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:50 smithi007 ceph-mon[29098]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:17:50.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:50 smithi007 ceph-mon[37158]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:50.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:50 smithi007 ceph-mon[37158]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:17:52.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:52 smithi007 ceph-mon[29098]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:52.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:52 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/1192227118' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7"}]: dispatch 2024-02-05T07:17:52.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:52 smithi007 ceph-mon[29098]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7"}]: dispatch 2024-02-05T07:17:52.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:52 smithi007 ceph-mon[29098]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7"}]': finished 2024-02-05T07:17:52.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:52 smithi007 ceph-mon[29098]: osdmap e46: 7 total, 6 up, 7 in 2024-02-05T07:17:52.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:52 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:17:52.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:52 smithi007 ceph-mon[37158]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:52.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:52 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/1192227118' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7"}]: dispatch 2024-02-05T07:17:52.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:52 smithi007 ceph-mon[37158]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7"}]: dispatch 2024-02-05T07:17:52.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:52 smithi007 ceph-mon[37158]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7"}]': finished 2024-02-05T07:17:52.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:52 smithi007 ceph-mon[37158]: osdmap e46: 7 total, 6 up, 7 in 2024-02-05T07:17:52.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:52 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:17:52.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:52 smithi090 ceph-mon[33213]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:52.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:52 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/1192227118' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7"}]: dispatch 2024-02-05T07:17:52.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:52 smithi090 ceph-mon[33213]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7"}]: dispatch 2024-02-05T07:17:52.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:52 smithi090 ceph-mon[33213]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7"}]': finished 2024-02-05T07:17:52.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:52 smithi090 ceph-mon[33213]: osdmap e46: 7 total, 6 up, 7 in 2024-02-05T07:17:52.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:52 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:17:53.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:53 smithi007 ceph-mon[29098]: 1.0 scrub starts 2024-02-05T07:17:53.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:53 smithi007 ceph-mon[29098]: 1.0 scrub ok 2024-02-05T07:17:53.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:53 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/3264271649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:17:53.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:53 smithi007 ceph-mon[37158]: 1.0 scrub starts 2024-02-05T07:17:53.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:53 smithi007 ceph-mon[37158]: 1.0 scrub ok 2024-02-05T07:17:53.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:53 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/3264271649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:17:53.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:53 smithi090 ceph-mon[33213]: 1.0 scrub starts 2024-02-05T07:17:53.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:53 smithi090 ceph-mon[33213]: 1.0 scrub ok 2024-02-05T07:17:53.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:53 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/3264271649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:17:54.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:54 smithi007 ceph-mon[29098]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:54.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:54 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:17:54.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:54 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:17:54.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:54 smithi007 ceph-mon[29098]: osdmap e47: 7 total, 6 up, 7 in 2024-02-05T07:17:54.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:54 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:17:54.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:54 smithi007 ceph-mon[37158]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:54.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:54 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:17:54.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:54 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:17:54.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:54 smithi007 ceph-mon[37158]: osdmap e47: 7 total, 6 up, 7 in 2024-02-05T07:17:54.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:54 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:17:54.500 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:54 smithi090 ceph-mon[33213]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:54.500 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:54 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:17:54.500 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:54 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:17:54.500 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:54 smithi090 ceph-mon[33213]: osdmap e47: 7 total, 6 up, 7 in 2024-02-05T07:17:54.500 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:54 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:17:56.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:56 smithi007 ceph-mon[29098]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:56.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:56 smithi007 ceph-mon[37158]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:56.431 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:56 smithi090 ceph-mon[33213]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:58.151 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:58 smithi090 ceph-mon[33213]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:58.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:58 smithi007 ceph-mon[29098]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:58.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:58 smithi007 ceph-mon[37158]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:17:59.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:59 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-05T07:17:59.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:17:59 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:59.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:59 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-05T07:17:59.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:17:59 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:17:59.443 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:59 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-05T07:17:59.443 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:17:59 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:00.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:00 smithi007 ceph-mon[29098]: Deploying daemon osd.6 on smithi090 2024-02-05T07:18:00.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:00 smithi007 ceph-mon[29098]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:00.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:00 smithi007 ceph-mon[37158]: Deploying daemon osd.6 on smithi090 2024-02-05T07:18:00.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:00 smithi007 ceph-mon[37158]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:00.465 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:00 smithi090 ceph-mon[33213]: Deploying daemon osd.6 on smithi090 2024-02-05T07:18:00.465 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:00 smithi090 ceph-mon[33213]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:02 smithi007 ceph-mon[29098]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:02 smithi007 ceph-mon[37158]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:02.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:02 smithi090 ceph-mon[33213]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:04.380 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:04 smithi090 ceph-mon[33213]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:04.380 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:04 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:04.380 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:04 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:04.380 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:04 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:04 smithi007 ceph-mon[29098]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:04 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:04 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:04 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:04 smithi007 ceph-mon[37158]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:04.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:04 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:04.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:04 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:04.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:04 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:05.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:05 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:18:05.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:05 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:18:05.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:05 smithi007 ceph-mon[29098]: osdmap e48: 7 total, 6 up, 7 in 2024-02-05T07:18:05.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:05.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:05 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:18:05.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:05 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:18:05.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:05 smithi007 ceph-mon[37158]: osdmap e48: 7 total, 6 up, 7 in 2024-02-05T07:18:05.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:05.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:05 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:18:05.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:05 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:18:05.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:05 smithi090 ceph-mon[33213]: osdmap e48: 7 total, 6 up, 7 in 2024-02-05T07:18:05.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:06.244 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:06 smithi090 ceph-mon[33213]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:06 smithi007 ceph-mon[29098]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:06.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:06 smithi007 ceph-mon[37158]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:07.408 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 6 on host 'smithi090' 2024-02-05T07:18:07.702 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:07 smithi090 ceph-mon[33213]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:07.703 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:07 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.703 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:07 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.703 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:07 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:07.703 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:07 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:07.703 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:07 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.703 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:07 smithi090 ceph-mon[33213]: from='osd.6 v2:172.21.15.90:6808/2371506925' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T07:18:07.703 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:07 smithi090 ceph-mon[33213]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T07:18:07.703 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:07 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.703 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:07 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.703 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:07 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[29098]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[29098]: from='osd.6 v2:172.21.15.90:6808/2371506925' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T07:18:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[29098]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T07:18:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:07.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[37158]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:07.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:07.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:07.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[37158]: from='osd.6 v2:172.21.15.90:6808/2371506925' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T07:18:07.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[37158]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T07:18:07.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:07.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:07 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:08.024 DEBUG:teuthology.orchestra.run.smithi090:osd.6> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.6.service 2024-02-05T07:18:08.026 INFO:tasks.cephadm:Deploying osd.7 on smithi090 with /dev/vg_nvme/lv_1... 2024-02-05T07:18:08.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-05T07:18:08.906 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:08 smithi090 ceph-mon[33213]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-05T07:18:08.906 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:08 smithi090 ceph-mon[33213]: osdmap e49: 7 total, 6 up, 7 in 2024-02-05T07:18:08.906 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:08 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:08.907 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:08 smithi090 ceph-mon[33213]: from='osd.6 v2:172.21.15.90:6808/2371506925' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:18:08.907 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:08 smithi090 ceph-mon[33213]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:18:08.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:08 smithi007 ceph-mon[29098]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-05T07:18:08.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:08 smithi007 ceph-mon[29098]: osdmap e49: 7 total, 6 up, 7 in 2024-02-05T07:18:08.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:08 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:08.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:08 smithi007 ceph-mon[29098]: from='osd.6 v2:172.21.15.90:6808/2371506925' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:18:08.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:08 smithi007 ceph-mon[29098]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:18:08.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:08 smithi007 ceph-mon[37158]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-05T07:18:08.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:08 smithi007 ceph-mon[37158]: osdmap e49: 7 total, 6 up, 7 in 2024-02-05T07:18:08.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:08 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:08.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:08 smithi007 ceph-mon[37158]: from='osd.6 v2:172.21.15.90:6808/2371506925' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:18:08.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:08 smithi007 ceph-mon[37158]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:18:09.175 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:18:09 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[49515]: 2024-02-05T07:18:09.070+0000 7f9b04d33700 -1 osd.6 0 waiting for initial osdmap 2024-02-05T07:18:09.175 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:18:09 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[49515]: 2024-02-05T07:18:09.077+0000 7f9afcb25700 -1 osd.6 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T07:18:09.656 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:18:09.763 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:09 smithi090 ceph-mon[33213]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:09.763 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:09 smithi090 ceph-mon[33213]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-05T07:18:09.764 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:09 smithi090 ceph-mon[33213]: osdmap e50: 7 total, 6 up, 7 in 2024-02-05T07:18:09.764 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:09 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:09.764 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:09 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:09.764 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:09 smithi090 ceph-mon[33213]: from='osd.6 ' entity='osd.6' 2024-02-05T07:18:09.764 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:09 smithi090 ceph-mon[33213]: osdmap e51: 7 total, 6 up, 7 in 2024-02-05T07:18:09.764 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:09 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[37158]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[37158]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-05T07:18:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[37158]: osdmap e50: 7 total, 6 up, 7 in 2024-02-05T07:18:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[37158]: from='osd.6 ' entity='osd.6' 2024-02-05T07:18:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[37158]: osdmap e51: 7 total, 6 up, 7 in 2024-02-05T07:18:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:09.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[29098]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:09.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[29098]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-05T07:18:09.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[29098]: osdmap e50: 7 total, 6 up, 7 in 2024-02-05T07:18:09.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:09.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:09.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[29098]: from='osd.6 ' entity='osd.6' 2024-02-05T07:18:09.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[29098]: osdmap e51: 7 total, 6 up, 7 in 2024-02-05T07:18:09.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:09 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:10.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:10 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:18:10.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:10 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:18:10.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:10 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:10.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:10 smithi007 ceph-mon[29098]: osd.6 v2:172.21.15.90:6808/2371506925 boot 2024-02-05T07:18:10.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:10 smithi007 ceph-mon[29098]: osdmap e52: 7 total, 7 up, 7 in 2024-02-05T07:18:10.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:10 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:10.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:10 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:18:10.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:10 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:18:10.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:10 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:10.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:10 smithi007 ceph-mon[37158]: osd.6 v2:172.21.15.90:6808/2371506925 boot 2024-02-05T07:18:10.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:10 smithi007 ceph-mon[37158]: osdmap e52: 7 total, 7 up, 7 in 2024-02-05T07:18:10.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:10 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:10.956 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:10 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:18:10.957 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:10 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:18:10.957 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:10 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:10.957 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:10 smithi090 ceph-mon[33213]: osd.6 v2:172.21.15.90:6808/2371506925 boot 2024-02-05T07:18:10.957 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:10 smithi090 ceph-mon[33213]: osdmap e52: 7 total, 7 up, 7 in 2024-02-05T07:18:10.957 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:10 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:18:12.286 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-05T07:18:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-05T07:18:12.287 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-02-05T07:18:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-02-05T07:18:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115391 s, 909 MB/s 2024-02-05T07:18:12.287 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 2024-02-05T07:18:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-02-05T07:18:12.310 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:12 smithi090 ceph-mon[33213]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:12.311 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:12 smithi090 ceph-mon[33213]: Detected new or changed devices on smithi090 2024-02-05T07:18:12.311 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:12.311 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:12.311 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:12.311 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:12.311 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:12.311 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:12 smithi090 ceph-mon[33213]: Adjusting osd_memory_target on smithi090 to 5696M 2024-02-05T07:18:12.312 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:12.312 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:12.312 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:12.312 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:12 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:12.312 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:12 smithi090 ceph-mon[33213]: osdmap e53: 7 total, 7 up, 7 in 2024-02-05T07:18:12.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[37158]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:12.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[37158]: Detected new or changed devices on smithi090 2024-02-05T07:18:12.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:12.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:12.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:12.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:12.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:12.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[37158]: Adjusting osd_memory_target on smithi090 to 5696M 2024-02-05T07:18:12.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:12.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:12.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:12.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:12.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[37158]: osdmap e53: 7 total, 7 up, 7 in 2024-02-05T07:18:12.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[29098]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-05T07:18:12.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[29098]: Detected new or changed devices on smithi090 2024-02-05T07:18:12.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:12.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:12.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:12.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:12.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:12.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[29098]: Adjusting osd_memory_target on smithi090 to 5696M 2024-02-05T07:18:12.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:12.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:12.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:12.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:12.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:12 smithi007 ceph-mon[29098]: osdmap e53: 7 total, 7 up, 7 in 2024-02-05T07:18:12.916 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi090:vg_nvme/lv_1 2024-02-05T07:18:13.154 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:18:13.376 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:13 smithi090 ceph-mon[33213]: osdmap e54: 7 total, 7 up, 7 in 2024-02-05T07:18:13.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:13 smithi007 ceph-mon[29098]: osdmap e54: 7 total, 7 up, 7 in 2024-02-05T07:18:13.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:13 smithi007 ceph-mon[37158]: osdmap e54: 7 total, 7 up, 7 in 2024-02-05T07:18:14.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:14 smithi090 ceph-mon[33213]: pgmap v163: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:14.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:14 smithi090 ceph-mon[33213]: osdmap e55: 7 total, 7 up, 7 in 2024-02-05T07:18:14.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:14 smithi007 ceph-mon[29098]: pgmap v163: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:14.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:14 smithi007 ceph-mon[29098]: osdmap e55: 7 total, 7 up, 7 in 2024-02-05T07:18:14.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:14 smithi007 ceph-mon[37158]: pgmap v163: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:14.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:14 smithi007 ceph-mon[37158]: osdmap e55: 7 total, 7 up, 7 in 2024-02-05T07:18:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:15 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:18:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:15 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:18:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:15 smithi007 ceph-mon[29098]: osdmap e56: 7 total, 7 up, 7 in 2024-02-05T07:18:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:18:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:18:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:15.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:15 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:18:15.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:15 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:18:15.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:15 smithi007 ceph-mon[37158]: osdmap e56: 7 total, 7 up, 7 in 2024-02-05T07:18:15.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:15 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:18:15.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:15 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:18:15.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:15 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:15.493 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:15 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:18:15.493 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:15 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:18:15.493 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:15 smithi090 ceph-mon[33213]: osdmap e56: 7 total, 7 up, 7 in 2024-02-05T07:18:15.493 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:15 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T07:18:15.494 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:15 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T07:18:15.494 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:15 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:16 smithi007 ceph-mon[29098]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:16 smithi007 ceph-mon[29098]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:18:16.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:16 smithi007 ceph-mon[37158]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:16.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:16 smithi007 ceph-mon[37158]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:18:16.434 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:16 smithi090 ceph-mon[33213]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:16.434 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:16 smithi090 ceph-mon[33213]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:18:18.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:18 smithi007 ceph-mon[29098]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-02-05T07:18:18.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:18 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/3035709245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9"}]: dispatch 2024-02-05T07:18:18.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:18 smithi007 ceph-mon[29098]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9"}]: dispatch 2024-02-05T07:18:18.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:18 smithi007 ceph-mon[29098]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9"}]': finished 2024-02-05T07:18:18.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:18 smithi007 ceph-mon[29098]: osdmap e57: 8 total, 7 up, 8 in 2024-02-05T07:18:18.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:18 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:18.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:18 smithi007 ceph-mon[37158]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-02-05T07:18:18.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:18 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/3035709245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9"}]: dispatch 2024-02-05T07:18:18.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:18 smithi007 ceph-mon[37158]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9"}]: dispatch 2024-02-05T07:18:18.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:18 smithi007 ceph-mon[37158]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9"}]': finished 2024-02-05T07:18:18.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:18 smithi007 ceph-mon[37158]: osdmap e57: 8 total, 7 up, 8 in 2024-02-05T07:18:18.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:18 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:18.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:18 smithi090 ceph-mon[33213]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-02-05T07:18:18.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:18 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/3035709245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9"}]: dispatch 2024-02-05T07:18:18.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:18 smithi090 ceph-mon[33213]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9"}]: dispatch 2024-02-05T07:18:18.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:18 smithi090 ceph-mon[33213]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9"}]': finished 2024-02-05T07:18:18.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:18 smithi090 ceph-mon[33213]: osdmap e57: 8 total, 7 up, 8 in 2024-02-05T07:18:18.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:18 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:19.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:19 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/573119374' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:18:19.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:19 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/573119374' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:18:19.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:19 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/573119374' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T07:18:20.399 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:20 smithi090 ceph-mon[33213]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T07:18:20.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:20 smithi007 ceph-mon[29098]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T07:18:20.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:20 smithi007 ceph-mon[37158]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T07:18:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:22 smithi090 ceph-mon[33213]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-02-05T07:18:22.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:22 smithi007 ceph-mon[29098]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-02-05T07:18:22.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:22 smithi007 ceph-mon[37158]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-02-05T07:18:24.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:24 smithi090 ceph-mon[33213]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-02-05T07:18:24.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:24 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-05T07:18:24.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:24 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:24.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:24 smithi007 ceph-mon[29098]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-02-05T07:18:24.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:24 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-05T07:18:24.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:24 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:24.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:24 smithi007 ceph-mon[37158]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-02-05T07:18:24.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:24 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-05T07:18:24.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:24 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:25 smithi090 ceph-mon[33213]: Deploying daemon osd.7 on smithi090 2024-02-05T07:18:25.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:25 smithi007 ceph-mon[29098]: Deploying daemon osd.7 on smithi090 2024-02-05T07:18:25.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:25 smithi007 ceph-mon[37158]: Deploying daemon osd.7 on smithi090 2024-02-05T07:18:26.201 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:26 smithi090 ceph-mon[33213]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T07:18:26.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:26 smithi007 ceph-mon[29098]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T07:18:26.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:26 smithi007 ceph-mon[37158]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T07:18:28.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:28 smithi090 ceph-mon[33213]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:28.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:28 smithi007 ceph-mon[29098]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:28.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:28 smithi007 ceph-mon[37158]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:30.170 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:30 smithi090 ceph-mon[33213]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:30.170 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:30 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:30.170 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:30 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:30.170 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:30 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:30.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:30 smithi007 ceph-mon[29098]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:30.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:30 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:30.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:30 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:30.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:30 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:30.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:30 smithi007 ceph-mon[37158]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:30.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:30 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:30.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:30 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:30.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:30 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:32.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:32 smithi007 ceph-mon[29098]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:32.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:32 smithi007 ceph-mon[37158]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:32.486 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:32 smithi090 ceph-mon[33213]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:33.203 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 7 on host 'smithi090' 2024-02-05T07:18:33.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:33 smithi090 ceph-mon[33213]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:33.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:33 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:33.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:33 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:33.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:33 smithi090 ceph-mon[33213]: from='osd.7 v2:172.21.15.90:6812/2983605980' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T07:18:33.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:33 smithi090 ceph-mon[33213]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T07:18:33.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:33 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:33.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:33 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:33.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:33 smithi007 ceph-mon[29098]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:33.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:33 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:33.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:33 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:33.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:33 smithi007 ceph-mon[29098]: from='osd.7 v2:172.21.15.90:6812/2983605980' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T07:18:33.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:33 smithi007 ceph-mon[29098]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T07:18:33.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:33 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:33.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:33 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:33.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:33 smithi007 ceph-mon[37158]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:33.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:33 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:33.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:33 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:33.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:33 smithi007 ceph-mon[37158]: from='osd.7 v2:172.21.15.90:6812/2983605980' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T07:18:33.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:33 smithi007 ceph-mon[37158]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T07:18:33.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:33 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:33.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:33 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:33.957 DEBUG:teuthology.orchestra.run.smithi090:osd.7> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.7.service 2024-02-05T07:18:33.959 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-05T07:18:33.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd stat -f json 2024-02-05T07:18:34.219 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:18:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:34 smithi090 ceph-mon[33213]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-05T07:18:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:34 smithi090 ceph-mon[33213]: osdmap e58: 8 total, 7 up, 8 in 2024-02-05T07:18:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:34 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:34 smithi090 ceph-mon[33213]: from='osd.7 v2:172.21.15.90:6812/2983605980' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:18:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:34 smithi090 ceph-mon[33213]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:18:34.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:18:34 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:18:34.442+0000 7f8a2b6f8700 -1 osd.7 0 waiting for initial osdmap 2024-02-05T07:18:34.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:18:34 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:18:34.447+0000 7f8a274f2700 -1 osd.7 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T07:18:34.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:34 smithi007 ceph-mon[29098]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-05T07:18:34.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:34 smithi007 ceph-mon[29098]: osdmap e58: 8 total, 7 up, 8 in 2024-02-05T07:18:34.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:34 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:34.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:34 smithi007 ceph-mon[29098]: from='osd.7 v2:172.21.15.90:6812/2983605980' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:18:34.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:34 smithi007 ceph-mon[29098]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:18:34.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:34 smithi007 ceph-mon[37158]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-05T07:18:34.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:34 smithi007 ceph-mon[37158]: osdmap e58: 8 total, 7 up, 8 in 2024-02-05T07:18:34.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:34 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:34.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:34 smithi007 ceph-mon[37158]: from='osd.7 v2:172.21.15.90:6812/2983605980' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:18:34.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:34 smithi007 ceph-mon[37158]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:18:35.762 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:35 smithi007 ceph-mon[29098]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:35.763 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:35 smithi007 ceph-mon[29098]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-05T07:18:35.763 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:35 smithi007 ceph-mon[29098]: osdmap e59: 8 total, 7 up, 8 in 2024-02-05T07:18:35.763 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:35 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:35.763 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:35 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:35.763 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:35 smithi007 ceph-mon[37158]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:35.763 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:35 smithi007 ceph-mon[37158]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-05T07:18:35.763 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:35 smithi007 ceph-mon[37158]: osdmap e59: 8 total, 7 up, 8 in 2024-02-05T07:18:35.763 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:35 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:35.763 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:35 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:35 smithi090 ceph-mon[33213]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-05T07:18:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:35 smithi090 ceph-mon[33213]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-05T07:18:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:35 smithi090 ceph-mon[33213]: osdmap e59: 8 total, 7 up, 8 in 2024-02-05T07:18:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:35 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:35 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:18:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:36 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:18:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:36 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:18:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:36 smithi090 ceph-mon[33213]: osd.7 v2:172.21.15.90:6812/2983605980 boot 2024-02-05T07:18:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:36 smithi090 ceph-mon[33213]: osdmap e60: 8 total, 8 up, 8 in 2024-02-05T07:18:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:36 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:36 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2713520859' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-05T07:18:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:36 smithi090 ceph-mon[33213]: osdmap e61: 8 total, 8 up, 8 in 2024-02-05T07:18:36.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:36 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:18:36.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:36 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:18:36.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:36 smithi007 ceph-mon[29098]: osd.7 v2:172.21.15.90:6812/2983605980 boot 2024-02-05T07:18:36.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:36 smithi007 ceph-mon[29098]: osdmap e60: 8 total, 8 up, 8 in 2024-02-05T07:18:36.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:36 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:36.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:36 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2713520859' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-05T07:18:36.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:36 smithi007 ceph-mon[29098]: osdmap e61: 8 total, 8 up, 8 in 2024-02-05T07:18:36.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:36 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:18:36.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:36 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:18:36.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:36 smithi007 ceph-mon[37158]: osd.7 v2:172.21.15.90:6812/2983605980 boot 2024-02-05T07:18:36.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:36 smithi007 ceph-mon[37158]: osdmap e60: 8 total, 8 up, 8 in 2024-02-05T07:18:36.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:36 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:18:36.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:36 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2713520859' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-05T07:18:36.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:36 smithi007 ceph-mon[37158]: osdmap e61: 8 total, 8 up, 8 in 2024-02-05T07:18:36.856 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":60,"num_osds":8,"num_up_osds":8,"osd_up_since":1707117515,"num_in_osds":8,"osd_in_since":1707117497,"num_remapped_pgs":1} 2024-02-05T07:18:36.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T07:18:37.108 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:18:38.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:18:38.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: Detected new or changed devices on smithi090 2024-02-05T07:18:38.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:38.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:38.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:38.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:38.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: Adjusting osd_memory_target on smithi090 to 4272M 2024-02-05T07:18:38.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:38.147 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:38.147 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.147 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:38.147 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:38.147 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:38.147 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.147 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[37158]: osdmap e62: 8 total, 8 up, 8 in 2024-02-05T07:18:38.147 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:18:38.147 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: Detected new or changed devices on smithi090 2024-02-05T07:18:38.147 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.147 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.147 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:38.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:38.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:38.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:38.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: Adjusting osd_memory_target on smithi090 to 4272M 2024-02-05T07:18:38.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:38.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:38.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:38.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:38.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:38.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:37 smithi007 ceph-mon[29098]: osdmap e62: 8 total, 8 up, 8 in 2024-02-05T07:18:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:18:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: Detected new or changed devices on smithi090 2024-02-05T07:18:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:38.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:18:38.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: Adjusting osd_memory_target on smithi090 to 4272M 2024-02-05T07:18:38.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:38.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:38.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:38.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:38.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:38.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:38.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:37 smithi090 ceph-mon[33213]: osdmap e62: 8 total, 8 up, 8 in 2024-02-05T07:18:38.794 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:18:38.794 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":63,"fsid":"e3e7059e-c3f5-11ee-95b6-87774f69a715","created":"2024-02-05T07:13:14.603885+0000","modified":"2024-02-05T07:18:38.438966+0000","last_up_change":"2024-02-05T07:18:35.434647+0000","last_in_change":"2024-02-05T07:18:17.533340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T07:16:32.505567+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"703c3948-8003-4015-9dd8-21019cf0eeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6801","nonce":2069016689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2069016689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2069016689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":2069016689}]},"public_addr":"172.21.15.7:6801/2069016689","cluster_addr":"172.21.15.7:6802/2069016689","heartbeat_back_addr":"172.21.15.7:6804/2069016689","heartbeat_front_addr":"172.21.15.7:6803/2069016689","state":["exists","up"]},{"osd":1,"uuid":"5c3e7d1e-a61d-44b9-b21e-f91c99a096bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":865486532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":865486532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":865486532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6807","nonce":865486532}]},"public_addr":"172.21.15.7:6805/865486532","cluster_addr":"172.21.15.7:6806/865486532","heartbeat_back_addr":"172.21.15.7:6808/865486532","heartbeat_front_addr":"172.21.15.7:6807/865486532","state":["exists","up"]},{"osd":2,"uuid":"f4fd930f-7bb2-46ca-9964-98235bd47b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":904871370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":904871370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":904871370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":904871370}]},"public_addr":"172.21.15.7:6809/904871370","cluster_addr":"172.21.15.7:6810/904871370","heartbeat_back_addr":"172.21.15.7:6812/904871370","heartbeat_front_addr":"172.21.15.7:6811/904871370","state":["exists","up"]},{"osd":3,"uuid":"149ed0d6-b049-4478-9819-cafc446e94d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":2758586242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2758586242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2758586242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2758586242}]},"public_addr":"172.21.15.7:6813/2758586242","cluster_addr":"172.21.15.7:6814/2758586242","heartbeat_back_addr":"172.21.15.7:6816/2758586242","heartbeat_front_addr":"172.21.15.7:6815/2758586242","state":["exists","up"]},{"osd":4,"uuid":"3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":1154929470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6801","nonce":1154929470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6803","nonce":1154929470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1154929470}]},"public_addr":"172.21.15.90:6800/1154929470","cluster_addr":"172.21.15.90:6801/1154929470","heartbeat_back_addr":"172.21.15.90:6803/1154929470","heartbeat_front_addr":"172.21.15.90:6802/1154929470","state":["exists","up"]},{"osd":5,"uuid":"244a7930-c22c-413e-86e1-11e0749f7367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":398113577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6805","nonce":398113577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6807","nonce":398113577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":398113577}]},"public_addr":"172.21.15.90:6804/398113577","cluster_addr":"172.21.15.90:6805/398113577","heartbeat_back_addr":"172.21.15.90:6807/398113577","heartbeat_front_addr":"172.21.15.90:6806/398113577","state":["exists","up"]},{"osd":6,"uuid":"0adbb562-3012-4f60-b5ec-02550a10b2c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2371506925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2371506925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6811","nonce":2371506925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2371506925}]},"public_addr":"172.21.15.90:6808/2371506925","cluster_addr":"172.21.15.90:6809/2371506925","heartbeat_back_addr":"172.21.15.90:6811/2371506925","heartbeat_front_addr":"172.21.15.90:6810/2371506925","state":["exists","up"]},{"osd":7,"uuid":"dbc26c36-a0e9-4f21-9e98-739270cea6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2983605980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":2983605980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":2983605980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2983605980}]},"public_addr":"172.21.15.90:6812/2983605980","cluster_addr":"172.21.15.90:6813/2983605980","heartbeat_back_addr":"172.21.15.90:6815/2983605980","heartbeat_front_addr":"172.21.15.90:6814/2983605980","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:18:03.668956+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:17:24.096439+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:17:52.664591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:18:13.951048+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:17:16.921301+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:17:42.679552+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:18:07.899397+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:18:33.632182+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3418781614":"2024-02-06T07:14:14.164720+0000","172.21.15.7:6800/1518592084":"2024-02-06T07:13:47.004765+0000","172.21.15.7:0/2208590106":"2024-02-06T07:13:47.004765+0000","172.21.15.7:0/223579766":"2024-02-06T07:13:47.004765+0000","172.21.15.7:6800/2634895574":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/1462812719":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/2823976078":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/1905520582":"2024-02-06T07:13:47.004765+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T07:18:39.462 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T07:16:32.505567+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-05T07:18:39.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-05T07:18:39.707 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:18:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:39 smithi090 ceph-mon[33213]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:18:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:39 smithi090 ceph-mon[33213]: osdmap e63: 8 total, 8 up, 8 in 2024-02-05T07:18:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:39 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3973765700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:18:39.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:39 smithi007 ceph-mon[29098]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:18:39.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:39 smithi007 ceph-mon[29098]: osdmap e63: 8 total, 8 up, 8 in 2024-02-05T07:18:39.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:39 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3973765700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:18:39.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:39 smithi007 ceph-mon[37158]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:18:39.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:39 smithi007 ceph-mon[37158]: osdmap e63: 8 total, 8 up, 8 in 2024-02-05T07:18:39.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:39 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3973765700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:18:40.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:40 smithi090 ceph-mon[33213]: osdmap e64: 8 total, 8 up, 8 in 2024-02-05T07:18:40.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:40 smithi007 ceph-mon[29098]: osdmap e64: 8 total, 8 up, 8 in 2024-02-05T07:18:40.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:40 smithi007 ceph-mon[37158]: osdmap e64: 8 total, 8 up, 8 in 2024-02-05T07:18:41.587 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-02-05T07:18:41.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:41 smithi090 ceph-mon[33213]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:18:41.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:41 smithi007 ceph-mon[29098]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:18:41.883 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:41 smithi007 ceph-mon[37158]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:18:42.264 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi007 2024-02-05T07:18:42.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi007=foo.a' 2024-02-05T07:18:42.514 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:18:42.635 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:42 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3448816750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-05T07:18:42.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:42 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3448816750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-05T07:18:42.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:42 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3448816750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-05T07:18:43.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:43 smithi090 ceph-mon[33213]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-02-05T07:18:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:43 smithi007 ceph-mon[29098]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-02-05T07:18:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:43 smithi007 ceph-mon[37158]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-02-05T07:18:44.403 INFO:teuthology.orchestra.run.smithi090.stdout:Scheduled rgw.foo.a update... 2024-02-05T07:18:45.019 DEBUG:teuthology.orchestra.run.smithi007:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@rgw.foo.a.service 2024-02-05T07:18:45.022 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi090 2024-02-05T07:18:45.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-02-05T07:18:45.260 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:18:45.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[37158]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-05T07:18:45.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[37158]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi007=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:18:45.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[37158]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-02-05T07:18:45.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:45.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:45.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:45.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:45.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:45.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-05T07:18:45.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-05T07:18:45.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:45.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:45.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[37158]: Deploying daemon rgw.foo.a on smithi007 2024-02-05T07:18:45.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[29098]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-05T07:18:45.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[29098]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi007=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:18:45.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[29098]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-02-05T07:18:45.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:45.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:45.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:45.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:45.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:45.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-05T07:18:45.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-05T07:18:45.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:45.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:45.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:45 smithi007 ceph-mon[29098]: Deploying daemon rgw.foo.a on smithi007 2024-02-05T07:18:45.693 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:45 smithi090 ceph-mon[33213]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-05T07:18:45.694 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:45 smithi090 ceph-mon[33213]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi007=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:18:45.694 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:45 smithi090 ceph-mon[33213]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-02-05T07:18:45.694 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:45 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:45.694 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:45 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:45.694 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:45 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:45.694 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:45 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:45.694 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:45 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:45.694 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:45 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-05T07:18:45.694 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:45 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-05T07:18:45.694 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:45 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:45.694 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:45 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:45.694 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:45 smithi090 ceph-mon[33213]: Deploying daemon rgw.foo.a on smithi007 2024-02-05T07:18:46.599 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:46 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:18:46.599 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:46 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:18:46.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:46 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:18:46.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:46 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:18:46.665 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:46 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:18:46.665 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:46 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:18:47.416 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'datapool' created 2024-02-05T07:18:47.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:47 smithi007 ceph-mon[29098]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T07:18:47.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:47 smithi007 ceph-mon[29098]: osdmap e65: 8 total, 8 up, 8 in 2024-02-05T07:18:47.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:47 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/1542600563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T07:18:47.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:47 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T07:18:47.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:47 smithi007 ceph-mon[37158]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T07:18:47.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:47 smithi007 ceph-mon[37158]: osdmap e65: 8 total, 8 up, 8 in 2024-02-05T07:18:47.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:47 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/1542600563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T07:18:47.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:47 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T07:18:47.675 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Feb 05 07:18:47 smithi007 systemd[1]: Started Ceph rgw.foo.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:18:47.723 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:47 smithi090 ceph-mon[33213]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T07:18:47.723 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:47 smithi090 ceph-mon[33213]: osdmap e65: 8 total, 8 up, 8 in 2024-02-05T07:18:47.723 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:47 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/1542600563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T07:18:47.723 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:47 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T07:18:48.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- rbd pool init datapool 2024-02-05T07:18:48.254 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:18:48.671 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:48 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-05T07:18:48.671 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:48 smithi090 ceph-mon[33213]: osdmap e66: 8 total, 8 up, 8 in 2024-02-05T07:18:48.671 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:48 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.671 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:48 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.671 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:48 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.671 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:48 smithi090 ceph-mon[33213]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-02-05T07:18:48.671 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:48 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.672 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:48 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.672 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:48 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-05T07:18:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[29098]: osdmap e66: 8 total, 8 up, 8 in 2024-02-05T07:18:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[29098]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-02-05T07:18:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:48.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-05T07:18:48.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[37158]: osdmap e66: 8 total, 8 up, 8 in 2024-02-05T07:18:48.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[37158]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-02-05T07:18:48.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:48.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:48 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:18:49.655 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:49 smithi090 ceph-mon[33213]: pgmap v193: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T07:18:49.655 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:49 smithi090 ceph-mon[33213]: osdmap e67: 8 total, 8 up, 8 in 2024-02-05T07:18:49.655 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:49 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/17554859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T07:18:49.655 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:49 smithi090 ceph-mon[33213]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:18:49.655 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:49 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:49.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:49 smithi007 ceph-mon[29098]: pgmap v193: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T07:18:49.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:49 smithi007 ceph-mon[29098]: osdmap e67: 8 total, 8 up, 8 in 2024-02-05T07:18:49.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:49 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/17554859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T07:18:49.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:49 smithi007 ceph-mon[29098]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:18:49.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:49 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:49.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:49 smithi007 ceph-mon[37158]: pgmap v193: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T07:18:49.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:49 smithi007 ceph-mon[37158]: osdmap e67: 8 total, 8 up, 8 in 2024-02-05T07:18:49.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:49 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/17554859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T07:18:49.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:49 smithi007 ceph-mon[37158]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:18:49.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:49 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/17554859' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-05T07:18:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[29098]: osdmap e68: 8 total, 8 up, 8 in 2024-02-05T07:18:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[29098]: Checking dashboard <-> RGW credentials 2024-02-05T07:18:50.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/2664223046' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T07:18:50.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T07:18:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/17554859' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-05T07:18:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[37158]: osdmap e68: 8 total, 8 up, 8 in 2024-02-05T07:18:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[37158]: Checking dashboard <-> RGW credentials 2024-02-05T07:18:50.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/2664223046' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T07:18:50.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:50 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T07:18:50.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:50 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/17554859' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-05T07:18:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:50 smithi090 ceph-mon[33213]: osdmap e68: 8 total, 8 up, 8 in 2024-02-05T07:18:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:50 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:50 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:50 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:50 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:18:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:50 smithi090 ceph-mon[33213]: Checking dashboard <-> RGW credentials 2024-02-05T07:18:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:50 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/2664223046' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T07:18:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:50 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T07:18:51.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:51 smithi090 ceph-mon[33213]: pgmap v196: 36 pgs: 29 unknown, 7 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:18:51.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:51 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-05T07:18:51.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:51 smithi090 ceph-mon[33213]: osdmap e69: 8 total, 8 up, 8 in 2024-02-05T07:18:51.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:51 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2716729371' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T07:18:51.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:51 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/74545789' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T07:18:51.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:51 smithi090 ceph-mon[33213]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T07:18:51.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:51 smithi090 ceph-mon[33213]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T07:18:51.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:51 smithi007 ceph-mon[29098]: pgmap v196: 36 pgs: 29 unknown, 7 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:18:51.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:51 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-05T07:18:51.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:51 smithi007 ceph-mon[29098]: osdmap e69: 8 total, 8 up, 8 in 2024-02-05T07:18:51.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:51 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2716729371' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T07:18:51.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:51 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/74545789' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T07:18:51.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:51 smithi007 ceph-mon[29098]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T07:18:51.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:51 smithi007 ceph-mon[29098]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T07:18:51.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:51 smithi007 ceph-mon[37158]: pgmap v196: 36 pgs: 29 unknown, 7 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:18:51.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:51 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-05T07:18:51.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:51 smithi007 ceph-mon[37158]: osdmap e69: 8 total, 8 up, 8 in 2024-02-05T07:18:51.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:51 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2716729371' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T07:18:51.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:51 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/74545789' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T07:18:51.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:51 smithi007 ceph-mon[37158]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T07:18:51.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:51 smithi007 ceph-mon[37158]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T07:18:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:52 smithi090 ceph-mon[33213]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T07:18:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:52 smithi090 ceph-mon[33213]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T07:18:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:52 smithi090 ceph-mon[33213]: osdmap e70: 8 total, 8 up, 8 in 2024-02-05T07:18:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:52 smithi007 ceph-mon[29098]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T07:18:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:52 smithi007 ceph-mon[29098]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T07:18:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:52 smithi007 ceph-mon[29098]: osdmap e70: 8 total, 8 up, 8 in 2024-02-05T07:18:52.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:52 smithi007 ceph-mon[37158]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T07:18:52.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:52 smithi007 ceph-mon[37158]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T07:18:52.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:52 smithi007 ceph-mon[37158]: osdmap e70: 8 total, 8 up, 8 in 2024-02-05T07:18:53.118 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.90 --placement '1;smithi090=iscsi.a' 2024-02-05T07:18:53.356 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:18:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:53 smithi090 ceph-mon[33213]: pgmap v199: 68 pgs: 15 creating+peering, 38 active+clean, 15 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-05T07:18:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:53 smithi090 ceph-mon[33213]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:18:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:53 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:18:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:53 smithi090 ceph-mon[33213]: osdmap e71: 8 total, 8 up, 8 in 2024-02-05T07:18:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:53 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2716729371' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T07:18:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:53 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/74545789' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T07:18:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:53 smithi090 ceph-mon[33213]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T07:18:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:53 smithi090 ceph-mon[33213]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T07:18:53.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[29098]: pgmap v199: 68 pgs: 15 creating+peering, 38 active+clean, 15 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-05T07:18:53.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[29098]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:18:53.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:18:53.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[29098]: osdmap e71: 8 total, 8 up, 8 in 2024-02-05T07:18:53.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2716729371' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T07:18:53.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/74545789' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T07:18:53.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[29098]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T07:18:53.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[29098]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T07:18:53.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[37158]: pgmap v199: 68 pgs: 15 creating+peering, 38 active+clean, 15 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-05T07:18:53.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[37158]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:18:53.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:18:53.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[37158]: osdmap e71: 8 total, 8 up, 8 in 2024-02-05T07:18:53.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2716729371' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T07:18:53.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/74545789' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T07:18:53.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[37158]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T07:18:53.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:53 smithi007 ceph-mon[37158]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T07:18:54.659 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:54 smithi090 ceph-mon[33213]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T07:18:54.659 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:54 smithi090 ceph-mon[33213]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T07:18:54.660 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:54 smithi090 ceph-mon[33213]: osdmap e72: 8 total, 8 up, 8 in 2024-02-05T07:18:54.660 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:54 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:18:54.660 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:54 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:18:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:54 smithi007 ceph-mon[29098]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T07:18:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:54 smithi007 ceph-mon[29098]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T07:18:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:54 smithi007 ceph-mon[29098]: osdmap e72: 8 total, 8 up, 8 in 2024-02-05T07:18:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:54 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:18:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:54 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:18:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:54 smithi007 ceph-mon[37158]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T07:18:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:54 smithi007 ceph-mon[37158]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T07:18:54.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:54 smithi007 ceph-mon[37158]: osdmap e72: 8 total, 8 up, 8 in 2024-02-05T07:18:54.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:54 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:18:54.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:54 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:18:55.165 INFO:teuthology.orchestra.run.smithi090.stdout:Scheduled iscsi.datapool update... 2024-02-05T07:18:55.740 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:55 smithi090 ceph-mon[33213]: pgmap v202: 100 pgs: 19 creating+peering, 51 active+clean, 30 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-05T07:18:55.741 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:55 smithi090 ceph-mon[33213]: osdmap e73: 8 total, 8 up, 8 in 2024-02-05T07:18:55.741 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:55 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/74545789' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T07:18:55.741 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:55 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2716729371' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T07:18:55.741 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:55 smithi090 ceph-mon[33213]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T07:18:55.741 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:55 smithi090 ceph-mon[33213]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T07:18:55.741 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:55 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:55.767 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-02-05T07:18:55.767 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:18:55.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-02-05T07:18:55.798 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:55 smithi007 ceph-mon[29098]: pgmap v202: 100 pgs: 19 creating+peering, 51 active+clean, 30 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-05T07:18:55.798 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:55 smithi007 ceph-mon[29098]: osdmap e73: 8 total, 8 up, 8 in 2024-02-05T07:18:55.798 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:55 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/74545789' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T07:18:55.798 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:55 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2716729371' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T07:18:55.798 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:55 smithi007 ceph-mon[29098]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T07:18:55.798 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:55 smithi007 ceph-mon[29098]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T07:18:55.798 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:55 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:55.798 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:55 smithi007 ceph-mon[37158]: pgmap v202: 100 pgs: 19 creating+peering, 51 active+clean, 30 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-05T07:18:55.799 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:55 smithi007 ceph-mon[37158]: osdmap e73: 8 total, 8 up, 8 in 2024-02-05T07:18:55.799 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:55 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/74545789' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T07:18:55.799 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:55 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2716729371' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T07:18:55.799 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:55 smithi007 ceph-mon[37158]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T07:18:55.799 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:55 smithi007 ceph-mon[37158]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T07:18:55.799 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:55 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:55.803 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-05T07:18:55.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-02-05T07:18:55.832 DEBUG:teuthology.orchestra.run.smithi090:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@iscsi.iscsi.a.service 2024-02-05T07:18:55.875 INFO:tasks.cephadm:Adding prometheus.a on smithi090 2024-02-05T07:18:55.875 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch apply prometheus '1;smithi090=a' 2024-02-05T07:18:56.148 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:18:56.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:56 smithi090 ceph-mon[33213]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.90", "placement": "1;smithi090=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:18:56.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:56 smithi090 ceph-mon[33213]: Saving service iscsi.datapool spec with placement smithi090=iscsi.a;count:1 2024-02-05T07:18:56.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:56 smithi090 ceph-mon[33213]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T07:18:56.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:56 smithi090 ceph-mon[33213]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T07:18:56.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:56 smithi090 ceph-mon[33213]: osdmap e74: 8 total, 8 up, 8 in 2024-02-05T07:18:56.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:56 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/74545789' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T07:18:56.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:56 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2716729371' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T07:18:56.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:56 smithi090 ceph-mon[33213]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T07:18:56.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:56 smithi090 ceph-mon[33213]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T07:18:56.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[37158]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.90", "placement": "1;smithi090=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:18:56.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[37158]: Saving service iscsi.datapool spec with placement smithi090=iscsi.a;count:1 2024-02-05T07:18:56.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[37158]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T07:18:56.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[37158]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T07:18:56.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[37158]: osdmap e74: 8 total, 8 up, 8 in 2024-02-05T07:18:56.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/74545789' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T07:18:56.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2716729371' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T07:18:56.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[37158]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T07:18:56.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[37158]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T07:18:56.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[29098]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.90", "placement": "1;smithi090=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:18:56.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[29098]: Saving service iscsi.datapool spec with placement smithi090=iscsi.a;count:1 2024-02-05T07:18:56.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[29098]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T07:18:56.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[29098]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T07:18:56.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[29098]: osdmap e74: 8 total, 8 up, 8 in 2024-02-05T07:18:56.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/74545789' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T07:18:56.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2716729371' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T07:18:56.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[29098]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T07:18:56.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:56 smithi007 ceph-mon[29098]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T07:18:57.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:57 smithi090 ceph-mon[33213]: pgmap v205: 132 pgs: 14 creating+peering, 99 active+clean, 19 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 767 B/s wr, 2 op/s 2024-02-05T07:18:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:57 smithi090 ceph-mon[33213]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T07:18:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:57 smithi090 ceph-mon[33213]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T07:18:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:57 smithi090 ceph-mon[33213]: osdmap e75: 8 total, 8 up, 8 in 2024-02-05T07:18:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-05T07:18:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-05T07:18:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:57 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:57 smithi090 ceph-mon[33213]: Deploying daemon iscsi.iscsi.a on smithi090 2024-02-05T07:18:57.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[29098]: pgmap v205: 132 pgs: 14 creating+peering, 99 active+clean, 19 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 767 B/s wr, 2 op/s 2024-02-05T07:18:57.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[29098]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T07:18:57.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[29098]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T07:18:57.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[29098]: osdmap e75: 8 total, 8 up, 8 in 2024-02-05T07:18:57.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:57.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:57.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:57.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-05T07:18:57.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-05T07:18:57.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:57.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[29098]: Deploying daemon iscsi.iscsi.a on smithi090 2024-02-05T07:18:57.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[37158]: pgmap v205: 132 pgs: 14 creating+peering, 99 active+clean, 19 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 767 B/s wr, 2 op/s 2024-02-05T07:18:57.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[37158]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T07:18:57.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[37158]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T07:18:57.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[37158]: osdmap e75: 8 total, 8 up, 8 in 2024-02-05T07:18:57.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:57.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:57.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:57.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-05T07:18:57.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-05T07:18:57.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:18:57.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:57 smithi007 ceph-mon[37158]: Deploying daemon iscsi.iscsi.a on smithi090 2024-02-05T07:18:58.099 INFO:teuthology.orchestra.run.smithi090.stdout:Scheduled prometheus update... 2024-02-05T07:18:58.902 DEBUG:teuthology.orchestra.run.smithi090:prometheus.a> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@prometheus.a.service 2024-02-05T07:18:58.905 INFO:tasks.cephadm:Adding node-exporter.a on smithi007 2024-02-05T07:18:58.905 INFO:tasks.cephadm:Adding node-exporter.b on smithi090 2024-02-05T07:18:58.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch apply node-exporter '2;smithi007=a;smithi090=b' 2024-02-05T07:18:59.176 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:59 smithi090 ceph-mon[33213]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi090=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:18:59.176 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:59 smithi090 ceph-mon[33213]: Saving service prometheus spec with placement smithi090=a;count:1 2024-02-05T07:18:59.176 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:18:59 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:18:59.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:59 smithi007 ceph-mon[29098]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi090=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:18:59.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:59 smithi007 ceph-mon[29098]: Saving service prometheus spec with placement smithi090=a;count:1 2024-02-05T07:18:59.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:18:59 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:18:59.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:59 smithi007 ceph-mon[37158]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi090=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:18:59.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:59 smithi007 ceph-mon[37158]: Saving service prometheus spec with placement smithi090=a;count:1 2024-02-05T07:18:59.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:18:59 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:00.388 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:00 smithi090 ceph-mon[33213]: pgmap v207: 132 pgs: 14 creating+peering, 113 active+clean, 5 unknown; 579 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 2.9 KiB/s wr, 56 op/s 2024-02-05T07:19:00.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:00 smithi007 ceph-mon[29098]: pgmap v207: 132 pgs: 14 creating+peering, 113 active+clean, 5 unknown; 579 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 2.9 KiB/s wr, 56 op/s 2024-02-05T07:19:00.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:00 smithi007 ceph-mon[37158]: pgmap v207: 132 pgs: 14 creating+peering, 113 active+clean, 5 unknown; 579 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 2.9 KiB/s wr, 56 op/s 2024-02-05T07:19:02.288 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:02 smithi090 ceph-mon[33213]: pgmap v208: 132 pgs: 6 creating+peering, 126 active+clean; 580 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 4.0 KiB/s wr, 103 op/s 2024-02-05T07:19:02.289 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:02 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:02.289 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:02 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:02.289 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:02 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:02.289 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:02 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:02 smithi007 ceph-mon[29098]: pgmap v208: 132 pgs: 6 creating+peering, 126 active+clean; 580 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 4.0 KiB/s wr, 103 op/s 2024-02-05T07:19:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:02 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:02 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:02 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:02 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:02.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:02 smithi007 ceph-mon[37158]: pgmap v208: 132 pgs: 6 creating+peering, 126 active+clean; 580 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 4.0 KiB/s wr, 103 op/s 2024-02-05T07:19:02.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:02 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:02.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:02 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:02.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:02 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:02.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:02 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:02.433 INFO:teuthology.orchestra.run.smithi090.stdout:Scheduled node-exporter update... 2024-02-05T07:19:03.172 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:03 smithi090 ceph-mon[33213]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-05T07:19:03.172 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:19:03.173 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:03 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/3467854671' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-05T07:19:03.173 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:03 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:03.202 DEBUG:teuthology.orchestra.run.smithi007:node-exporter.a> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@node-exporter.a.service 2024-02-05T07:19:03.205 DEBUG:teuthology.orchestra.run.smithi090:node-exporter.b> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@node-exporter.b.service 2024-02-05T07:19:03.208 INFO:tasks.cephadm:Adding alertmanager.a on smithi007 2024-02-05T07:19:03.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch apply alertmanager '1;smithi007=a' 2024-02-05T07:19:03.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:03 smithi007 ceph-mon[29098]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-05T07:19:03.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:19:03.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:03 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/3467854671' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-05T07:19:03.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:03 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:03.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:03 smithi007 ceph-mon[37158]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-05T07:19:03.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:19:03.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:03 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/3467854671' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-05T07:19:03.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:03 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:04.009 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:19:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:04 smithi007 ceph-mon[29098]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.9 KiB/s wr, 145 op/s 2024-02-05T07:19:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:04 smithi007 ceph-mon[29098]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi007=a;smithi090=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:19:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:04 smithi007 ceph-mon[29098]: Saving service node-exporter spec with placement smithi007=a;smithi090=b;count:2 2024-02-05T07:19:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:04 smithi007 ceph-mon[29098]: osdmap e76: 8 total, 8 up, 8 in 2024-02-05T07:19:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:04 smithi007 ceph-mon[37158]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.9 KiB/s wr, 145 op/s 2024-02-05T07:19:04.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:04 smithi007 ceph-mon[37158]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi007=a;smithi090=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:19:04.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:04 smithi007 ceph-mon[37158]: Saving service node-exporter spec with placement smithi007=a;smithi090=b;count:2 2024-02-05T07:19:04.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:04 smithi007 ceph-mon[37158]: osdmap e76: 8 total, 8 up, 8 in 2024-02-05T07:19:04.457 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:04 smithi090 ceph-mon[33213]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.9 KiB/s wr, 145 op/s 2024-02-05T07:19:04.457 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:04 smithi090 ceph-mon[33213]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi007=a;smithi090=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:19:04.457 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:04 smithi090 ceph-mon[33213]: Saving service node-exporter spec with placement smithi007=a;smithi090=b;count:2 2024-02-05T07:19:04.457 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:04 smithi090 ceph-mon[33213]: osdmap e76: 8 total, 8 up, 8 in 2024-02-05T07:19:05.363 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:05 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:19:05.363 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:05 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:19:05.363 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:05 smithi090 ceph-mon[33213]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-05T07:19:05.363 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:05 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:05.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:05 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:19:05.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:05 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:19:05.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:05 smithi007 ceph-mon[29098]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-05T07:19:05.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:05 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:05.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:05 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:19:05.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:05 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:19:05.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:05 smithi007 ceph-mon[37158]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-05T07:19:05.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:05 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:06.263 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:06 smithi090 ceph-mon[33213]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 139 op/s 2024-02-05T07:19:06.263 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:06 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:19:06.263 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:06 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:19:06.263 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:06 smithi090 ceph-mon[33213]: osdmap e77: 8 total, 8 up, 8 in 2024-02-05T07:19:06.263 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:06 smithi090 ceph-mon[33213]: 1.0 scrub starts 2024-02-05T07:19:06.263 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:06 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:06.263 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:06 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:06.263 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:06 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:19:06.263 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:06 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:19:06.263 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:06 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[29098]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 139 op/s 2024-02-05T07:19:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:19:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:19:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[29098]: osdmap e77: 8 total, 8 up, 8 in 2024-02-05T07:19:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[29098]: 1.0 scrub starts 2024-02-05T07:19:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:06.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:19:06.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:19:06.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:06.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[37158]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 139 op/s 2024-02-05T07:19:06.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:19:06.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:19:06.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[37158]: osdmap e77: 8 total, 8 up, 8 in 2024-02-05T07:19:06.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[37158]: 1.0 scrub starts 2024-02-05T07:19:06.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:06.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:06.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:19:06.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:19:06.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:06 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:06.488 INFO:teuthology.orchestra.run.smithi090.stdout:Scheduled alertmanager update... 2024-02-05T07:19:07.137 DEBUG:teuthology.orchestra.run.smithi007:alertmanager.a> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@alertmanager.a.service 2024-02-05T07:19:07.140 INFO:tasks.cephadm:Adding grafana.a on smithi090 2024-02-05T07:19:07.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph orch apply grafana '1;smithi090=a' 2024-02-05T07:19:07.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:07 smithi090 ceph-mon[33213]: 1.0 scrub ok 2024-02-05T07:19:07.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:07 smithi090 ceph-mon[33213]: Deploying daemon prometheus.a on smithi090 2024-02-05T07:19:07.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:07 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:07.392 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:19:07.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:07 smithi007 ceph-mon[29098]: 1.0 scrub ok 2024-02-05T07:19:07.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:07 smithi007 ceph-mon[29098]: Deploying daemon prometheus.a on smithi090 2024-02-05T07:19:07.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:07 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:07.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:07 smithi007 ceph-mon[37158]: 1.0 scrub ok 2024-02-05T07:19:07.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:07 smithi007 ceph-mon[37158]: Deploying daemon prometheus.a on smithi090 2024-02-05T07:19:07.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:07 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:08.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:08 smithi007 ceph-mon[29098]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 3.1 KiB/s wr, 109 op/s 2024-02-05T07:19:08.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:08 smithi007 ceph-mon[29098]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:19:08.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:08 smithi007 ceph-mon[29098]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-02-05T07:19:08.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:08 smithi007 ceph-mon[37158]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 3.1 KiB/s wr, 109 op/s 2024-02-05T07:19:08.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:08 smithi007 ceph-mon[37158]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:19:08.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:08 smithi007 ceph-mon[37158]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-02-05T07:19:08.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:08 smithi090 ceph-mon[33213]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 3.1 KiB/s wr, 109 op/s 2024-02-05T07:19:08.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:08 smithi090 ceph-mon[33213]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:19:08.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:08 smithi090 ceph-mon[33213]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-02-05T07:19:09.826 INFO:teuthology.orchestra.run.smithi090.stdout:Scheduled grafana update... 2024-02-05T07:19:10.350 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:10 smithi090 ceph-mon[33213]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 1.9 KiB/s wr, 65 op/s 2024-02-05T07:19:10.350 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:10 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:10.350 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:10 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:10.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:10 smithi007 ceph-mon[29098]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 1.9 KiB/s wr, 65 op/s 2024-02-05T07:19:10.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:10 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:10.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:10 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:10.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:10 smithi007 ceph-mon[37158]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 1.9 KiB/s wr, 65 op/s 2024-02-05T07:19:10.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:10 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:10.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:10 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:10.722 DEBUG:teuthology.orchestra.run.smithi090:grafana.a> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@grafana.a.service 2024-02-05T07:19:10.725 INFO:tasks.cephadm:Setting up client nodes... 2024-02-05T07:19:10.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-05T07:19:10.979 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:11.361 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:11 smithi007 ceph-mon[29098]: from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi090=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:19:11.361 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:11 smithi007 ceph-mon[29098]: Saving service grafana spec with placement smithi090=a;count:1 2024-02-05T07:19:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:11 smithi007 ceph-mon[37158]: from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi090=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:19:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:11 smithi007 ceph-mon[37158]: Saving service grafana spec with placement smithi090=a;count:1 2024-02-05T07:19:11.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:11 smithi090 ceph-mon[33213]: from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi090=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:19:11.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:11 smithi090 ceph-mon[33213]: Saving service grafana spec with placement smithi090=a;count:1 2024-02-05T07:19:12.346 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:12 smithi007 ceph-mon[29098]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 1 op/s 2024-02-05T07:19:12.346 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:12 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:19:12.346 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:12 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:19:12.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:12 smithi007 ceph-mon[37158]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 1 op/s 2024-02-05T07:19:12.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:12 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:19:12.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:12 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:19:12.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:12 smithi090 ceph-mon[33213]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 1 op/s 2024-02-05T07:19:12.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:12 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:19:12.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:12 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:19:13.014 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-02-05T07:19:13.015 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDxi8BllTqMABAACI1vWLAFZjAajkH43Nsj7A== 2024-02-05T07:19:13.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:13 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:19:13.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:13 smithi007 ceph-mon[29098]: osdmap e78: 8 total, 8 up, 8 in 2024-02-05T07:19:13.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:13 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/179472210' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T07:19:13.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:13 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T07:19:13.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:13 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T07:19:13.291 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:13 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:19:13.291 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:13 smithi007 ceph-mon[37158]: osdmap e78: 8 total, 8 up, 8 in 2024-02-05T07:19:13.291 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:13 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/179472210' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T07:19:13.291 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:13 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T07:19:13.291 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:13 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T07:19:13.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:13 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:19:13.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:13 smithi090 ceph-mon[33213]: osdmap e78: 8 total, 8 up, 8 in 2024-02-05T07:19:13.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:13 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/179472210' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T07:19:13.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:13 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T07:19:13.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:13 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T07:19:13.577 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-05T07:19:13.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-05T07:19:13.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-05T07:19:13.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-05T07:19:14.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:14 smithi007 ceph-mon[29098]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-02-05T07:19:14.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:14 smithi007 ceph-mon[29098]: osdmap e79: 8 total, 8 up, 8 in 2024-02-05T07:19:14.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:14 smithi007 ceph-mon[37158]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-02-05T07:19:14.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:14 smithi007 ceph-mon[37158]: osdmap e79: 8 total, 8 up, 8 in 2024-02-05T07:19:14.441 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:19:14.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:14 smithi090 ceph-mon[33213]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-02-05T07:19:14.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:14 smithi090 ceph-mon[33213]: osdmap e79: 8 total, 8 up, 8 in 2024-02-05T07:19:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-05T07:19:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-05T07:19:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:15 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-05T07:19:15.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:15 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-05T07:19:15.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:15 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-05T07:19:15.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:15 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-05T07:19:15.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:15 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-05T07:19:15.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:15 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-05T07:19:15.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:15 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-05T07:19:16.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:16 smithi090 ceph-mon[33213]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:19:16.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:16 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-05T07:19:16.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:16 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-05T07:19:16.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:16 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-05T07:19:16.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:16 smithi090 ceph-mon[33213]: osdmap e80: 8 total, 8 up, 8 in 2024-02-05T07:19:16.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:16 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:19:16.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:16 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:19:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:16 smithi007 ceph-mon[29098]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:19:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:16 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-05T07:19:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:16 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-05T07:19:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:16 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-05T07:19:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:16 smithi007 ceph-mon[29098]: osdmap e80: 8 total, 8 up, 8 in 2024-02-05T07:19:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:16 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:19:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:16 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:19:16.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:16 smithi007 ceph-mon[37158]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:19:16.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:16 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-05T07:19:16.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:16 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-05T07:19:16.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:16 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-05T07:19:16.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:16 smithi007 ceph-mon[37158]: osdmap e80: 8 total, 8 up, 8 in 2024-02-05T07:19:16.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:16 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:19:16.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:16 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:19:17.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:17 smithi007 ceph-mon[29098]: osdmap e81: 8 total, 8 up, 8 in 2024-02-05T07:19:17.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:17 smithi007 ceph-mon[37158]: osdmap e81: 8 total, 8 up, 8 in 2024-02-05T07:19:17.454 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:17 smithi090 ceph-mon[33213]: osdmap e81: 8 total, 8 up, 8 in 2024-02-05T07:19:18.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:18 smithi007 ceph-mon[29098]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:19:18.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:18 smithi007 ceph-mon[37158]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:19:18.506 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:18 smithi090 ceph-mon[33213]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:19:18.914 INFO:teuthology.orchestra.run.smithi090.stdout:[client.1] 2024-02-05T07:19:18.915 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQD2i8BlQ8c9NhAAA75ZorfjC1mWnqMs+Jr8bQ== 2024-02-05T07:19:19.396 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:19 smithi090 ceph-mon[33213]: from='client.? 172.21.15.90:0/787013989' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T07:19:19.396 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:19 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T07:19:19.396 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:19 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T07:19:19.396 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:19 smithi090 ceph-mon[33213]: osdmap e82: 8 total, 8 up, 8 in 2024-02-05T07:19:19.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:19 smithi007 ceph-mon[29098]: from='client.? 172.21.15.90:0/787013989' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T07:19:19.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:19 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T07:19:19.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:19 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T07:19:19.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:19 smithi007 ceph-mon[29098]: osdmap e82: 8 total, 8 up, 8 in 2024-02-05T07:19:19.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:19 smithi007 ceph-mon[37158]: from='client.? 172.21.15.90:0/787013989' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T07:19:19.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:19 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T07:19:19.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:19 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T07:19:19.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:19 smithi007 ceph-mon[37158]: osdmap e82: 8 total, 8 up, 8 in 2024-02-05T07:19:19.853 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-05T07:19:19.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-05T07:19:19.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-05T07:19:19.899 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-05T07:19:19.899 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-05T07:19:19.899 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph mgr dump --format=json 2024-02-05T07:19:20.135 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 podman[61529]: 2024-02-05 07:19:20.020625234 +0000 UTC m=+0.118684024 container create 7c2c4aabf01150ac95291ddf2e556238c03df157074d2c567298b9a4c8286c54 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T07:19:20.153 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:20.274 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:20 smithi007 ceph-mon[29098]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:19:20.275 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:20 smithi007 ceph-mon[37158]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:19:20.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-mon[33213]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:19:20.525 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 podman[61529]: 2024-02-05 07:19:20.24007913 +0000 UTC m=+0.338137913 container init 7c2c4aabf01150ac95291ddf2e556238c03df157074d2c567298b9a4c8286c54 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T07:19:20.525 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 podman[61529]: 2024-02-05 07:19:20.242783374 +0000 UTC m=+0.340842160 container start 7c2c4aabf01150ac95291ddf2e556238c03df157074d2c567298b9a4c8286c54 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T07:19:20.525 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.265Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-02-05T07:19:20.525 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.265Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-02-05T07:19:20.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.265Z caller=main.go:570 level=info host_details="(Linux 5.14.0-416.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jan 31 22:52:48 UTC 2024 x86_64 smithi090 (none))" 2024-02-05T07:19:20.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.266Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-05T07:19:20.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.266Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-05T07:19:20.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.267Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-02-05T07:19:20.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.267Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-02-05T07:19:20.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.268Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-02-05T07:19:20.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.268Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-02-05T07:19:20.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.269Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-05T07:19:20.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.269Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.928µs 2024-02-05T07:19:20.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.269Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-05T07:19:20.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.271Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-05T07:19:20.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.271Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.251µs wal_replay_duration=1.117729ms wbl_replay_duration=119ns total_replay_duration=1.236805ms 2024-02-05T07:19:20.527 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.271Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-02-05T07:19:20.527 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.271Z caller=main.go:1029 level=info msg="TSDB started" 2024-02-05T07:19:20.527 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.271Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-05T07:19:20.527 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.288Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.971717ms db_storage=733ns remote_storage=1.078µs web_handler=367ns query_engine=506ns scrape=417.482µs scrape_sd=89.099µs notify=760ns notify_sd=1.881µs rules=16.28708ms tracing=8.186µs 2024-02-05T07:19:20.527 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.288Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-02-05T07:19:20.527 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:19:20.288Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-02-05T07:19:20.527 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 bash[61529]: 7c2c4aabf01150ac95291ddf2e556238c03df157074d2c567298b9a4c8286c54 2024-02-05T07:19:20.527 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:19:20 smithi090 systemd[1]: Started Ceph prometheus.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:19:21.382 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:21 smithi007 ceph-mon[29098]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 73 B/s, 2 objects/s recovering 2024-02-05T07:19:21.382 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:21 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:21.382 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:21 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:21.382 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:21 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:21.382 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:21 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-05T07:19:21.383 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:21 smithi007 ceph-mon[37158]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 73 B/s, 2 objects/s recovering 2024-02-05T07:19:21.383 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:21 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:21.383 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:21 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:21.383 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:21 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:21.383 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:21 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-05T07:19:21.674 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:21 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ignoring --setuser ceph since I am not root 2024-02-05T07:19:21.674 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:21 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ignoring --setgroup ceph since I am not root 2024-02-05T07:19:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:21 smithi090 ceph-mon[33213]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 73 B/s, 2 objects/s recovering 2024-02-05T07:19:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:21 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:21 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:21 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' 2024-02-05T07:19:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:21 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-05T07:19:21.776 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: ignoring --setuser ceph since I am not root 2024-02-05T07:19:21.776 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: ignoring --setgroup ceph since I am not root 2024-02-05T07:19:21.776 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:21 smithi090 ceph-mgr[35104]: -- 172.21.15.90:0/1106006852 <== mon.2 v2:172.21.15.7:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ada111b520 con 0x55ada10f0c00 2024-02-05T07:19:21.906 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:19:22.174 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:22 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:22.017+0000 7fdd5ef7a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T07:19:22.174 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:22 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:22.115+0000 7fdd5ef7a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T07:19:22.274 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:21.986+0000 7f0611d80200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T07:19:22.275 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:22.079+0000 7f0611d80200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T07:19:22.492 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:22 smithi007 ceph-mon[29098]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-05T07:19:22.492 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:22 smithi007 ceph-mon[29098]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-05T07:19:22.492 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:22 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4271881654' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-05T07:19:22.493 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:22 smithi007 ceph-mon[37158]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-05T07:19:22.493 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:22 smithi007 ceph-mon[37158]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-05T07:19:22.493 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:22 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4271881654' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-05T07:19:22.518 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":3380493069}]},"active_addr":"172.21.15.7:6800/3380493069","active_change":"2024-02-05T07:14:14.164829+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[{"gid":24118,"name":"x","mgr_features":4540701272860131327,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.7:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":2066685086}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":4100020626}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3203189162}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":1953351182}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":2180087772}]}]} 2024-02-05T07:19:22.523 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-05T07:19:22.523 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-05T07:19:22.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T07:19:22.712 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:22 smithi090 ceph-mon[33213]: from='mgr.14150 172.21.15.7:0/2331414933' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-05T07:19:22.712 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:22 smithi090 ceph-mon[33213]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-05T07:19:22.712 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:22 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4271881654' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-05T07:19:22.774 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:23.024 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:22.709+0000 7f0611d80200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T07:19:23.024 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:22.802+0000 7f0611d80200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T07:19:23.024 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:22.942+0000 7f0611d80200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T07:19:23.077 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:22 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:22.795+0000 7fdd5ef7a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T07:19:23.077 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:22 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:22.896+0000 7fdd5ef7a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T07:19:23.077 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:23.044+0000 7fdd5ef7a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T07:19:23.383 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:23.037+0000 7f0611d80200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T07:19:23.383 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:23.120+0000 7f0611d80200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T07:19:23.384 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:23.381+0000 7f0611d80200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T07:19:23.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:23.149+0000 7fdd5ef7a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T07:19:23.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:23.245+0000 7fdd5ef7a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T07:19:23.672 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:23.490+0000 7f0611d80200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T07:19:23.672 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:23.576+0000 7f0611d80200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T07:19:23.672 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:23.669+0000 7f0611d80200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T07:19:23.819 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:23.517+0000 7fdd5ef7a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T07:19:23.819 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:23.631+0000 7fdd5ef7a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T07:19:23.819 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:23.719+0000 7fdd5ef7a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T07:19:24.024 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:23.844+0000 7f0611d80200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T07:19:24.116 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:23.817+0000 7fdd5ef7a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T07:19:24.116 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:24 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:24.000+0000 7fdd5ef7a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T07:19:24.720 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:19:24.720 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":84,"fsid":"e3e7059e-c3f5-11ee-95b6-87774f69a715","created":"2024-02-05T07:13:14.603885+0000","modified":"2024-02-05T07:19:24.090126+0000","last_up_change":"2024-02-05T07:18:35.434647+0000","last_in_change":"2024-02-05T07:18:17.533340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T07:16:32.505567+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-05T07:18:47.259462+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-05T07:18:47.762336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-05T07:18:49.736599+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-05T07:18:51.696948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-05T07:18:53.768555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"703c3948-8003-4015-9dd8-21019cf0eeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6801","nonce":2069016689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2069016689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2069016689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":2069016689}]},"public_addr":"172.21.15.7:6801/2069016689","cluster_addr":"172.21.15.7:6802/2069016689","heartbeat_back_addr":"172.21.15.7:6804/2069016689","heartbeat_front_addr":"172.21.15.7:6803/2069016689","state":["exists","up"]},{"osd":1,"uuid":"5c3e7d1e-a61d-44b9-b21e-f91c99a096bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":865486532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":865486532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":865486532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6807","nonce":865486532}]},"public_addr":"172.21.15.7:6805/865486532","cluster_addr":"172.21.15.7:6806/865486532","heartbeat_back_addr":"172.21.15.7:6808/865486532","heartbeat_front_addr":"172.21.15.7:6807/865486532","state":["exists","up"]},{"osd":2,"uuid":"f4fd930f-7bb2-46ca-9964-98235bd47b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":904871370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":904871370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":904871370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":904871370}]},"public_addr":"172.21.15.7:6809/904871370","cluster_addr":"172.21.15.7:6810/904871370","heartbeat_back_addr":"172.21.15.7:6812/904871370","heartbeat_front_addr":"172.21.15.7:6811/904871370","state":["exists","up"]},{"osd":3,"uuid":"149ed0d6-b049-4478-9819-cafc446e94d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":2758586242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2758586242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2758586242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2758586242}]},"public_addr":"172.21.15.7:6813/2758586242","cluster_addr":"172.21.15.7:6814/2758586242","heartbeat_back_addr":"172.21.15.7:6816/2758586242","heartbeat_front_addr":"172.21.15.7:6815/2758586242","state":["exists","up"]},{"osd":4,"uuid":"3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":1154929470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6801","nonce":1154929470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6803","nonce":1154929470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1154929470}]},"public_addr":"172.21.15.90:6800/1154929470","cluster_addr":"172.21.15.90:6801/1154929470","heartbeat_back_addr":"172.21.15.90:6803/1154929470","heartbeat_front_addr":"172.21.15.90:6802/1154929470","state":["exists","up"]},{"osd":5,"uuid":"244a7930-c22c-413e-86e1-11e0749f7367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":398113577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6805","nonce":398113577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6807","nonce":398113577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":398113577}]},"public_addr":"172.21.15.90:6804/398113577","cluster_addr":"172.21.15.90:6805/398113577","heartbeat_back_addr":"172.21.15.90:6807/398113577","heartbeat_front_addr":"172.21.15.90:6806/398113577","state":["exists","up"]},{"osd":6,"uuid":"0adbb562-3012-4f60-b5ec-02550a10b2c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2371506925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2371506925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6811","nonce":2371506925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2371506925}]},"public_addr":"172.21.15.90:6808/2371506925","cluster_addr":"172.21.15.90:6809/2371506925","heartbeat_back_addr":"172.21.15.90:6811/2371506925","heartbeat_front_addr":"172.21.15.90:6810/2371506925","state":["exists","up"]},{"osd":7,"uuid":"dbc26c36-a0e9-4f21-9e98-739270cea6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2983605980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":2983605980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":2983605980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2983605980}]},"public_addr":"172.21.15.90:6812/2983605980","cluster_addr":"172.21.15.90:6813/2983605980","heartbeat_back_addr":"172.21.15.90:6815/2983605980","heartbeat_front_addr":"172.21.15.90:6814/2983605980","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:19:11.460565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:18:53.860218+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:19:04.572025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:19:15.384158+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:18:45.753265+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:19:03.796302+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:19:23.927654+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:18:33.632182+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3418781614":"2024-02-06T07:14:14.164720+0000","172.21.15.7:6800/1518592084":"2024-02-06T07:13:47.004765+0000","172.21.15.7:0/2208590106":"2024-02-06T07:13:47.004765+0000","172.21.15.7:0/223579766":"2024-02-06T07:13:47.004765+0000","172.21.15.7:6800/2634895574":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/1462812719":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/2823976078":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/1905520582":"2024-02-06T07:13:47.004765+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T07:19:25.272 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:24 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:24.809+0000 7f0611d80200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T07:19:25.273 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:24 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:24.939+0000 7f0611d80200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T07:19:25.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:24 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:19:25.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:24 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:19:25.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:24 smithi090 ceph-mon[33213]: osdmap e83: 8 total, 8 up, 8 in 2024-02-05T07:19:25.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:24 smithi090 ceph-mon[33213]: osdmap e84: 8 total, 8 up, 8 in 2024-02-05T07:19:25.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:24 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/90990157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:19:25.324 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:25 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:25.025+0000 7fdd5ef7a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T07:19:25.324 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:25 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:25.160+0000 7fdd5ef7a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T07:19:25.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:24 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:19:25.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:24 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:19:25.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:24 smithi007 ceph-mon[29098]: osdmap e83: 8 total, 8 up, 8 in 2024-02-05T07:19:25.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:24 smithi007 ceph-mon[29098]: osdmap e84: 8 total, 8 up, 8 in 2024-02-05T07:19:25.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:24 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/90990157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:19:25.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:24 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:19:25.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:24 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:19:25.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:24 smithi007 ceph-mon[37158]: osdmap e83: 8 total, 8 up, 8 in 2024-02-05T07:19:25.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:24 smithi007 ceph-mon[37158]: osdmap e84: 8 total, 8 up, 8 in 2024-02-05T07:19:25.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:24 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/90990157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:19:25.356 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-05T07:19:25.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T07:19:25.524 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:25 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:25.270+0000 7f0611d80200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T07:19:25.524 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:25 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:25.360+0000 7f0611d80200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T07:19:25.611 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:25.615 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:25 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:25.509+0000 7fdd5ef7a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T07:19:25.615 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:25 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:25.613+0000 7fdd5ef7a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T07:19:25.918 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:25 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:25.826+0000 7fdd5ef7a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:19:26.023 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:25 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:25.558+0000 7f0611d80200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:19:26.023 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:25 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:25.741+0000 7f0611d80200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T07:19:26.173 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:26 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:26.020+0000 7fdd5ef7a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T07:19:26.274 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:26 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:26.020+0000 7f0611d80200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:19:26.581 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:26 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:26.316+0000 7fdd5ef7a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:19:26.774 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:26 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:26.291+0000 7f0611d80200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T07:19:26.775 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:26 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:26.539+0000 7f0611d80200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T07:19:26.851 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:26 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:26.579+0000 7fdd5ef7a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T07:19:27.109 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:26 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:26.849+0000 7fdd5ef7a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T07:19:27.424 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:19:27.425 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":84,"fsid":"e3e7059e-c3f5-11ee-95b6-87774f69a715","created":"2024-02-05T07:13:14.603885+0000","modified":"2024-02-05T07:19:24.090126+0000","last_up_change":"2024-02-05T07:18:35.434647+0000","last_in_change":"2024-02-05T07:18:17.533340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T07:16:32.505567+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-05T07:18:47.259462+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-05T07:18:47.762336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-05T07:18:49.736599+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-05T07:18:51.696948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-05T07:18:53.768555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"703c3948-8003-4015-9dd8-21019cf0eeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6801","nonce":2069016689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2069016689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2069016689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":2069016689}]},"public_addr":"172.21.15.7:6801/2069016689","cluster_addr":"172.21.15.7:6802/2069016689","heartbeat_back_addr":"172.21.15.7:6804/2069016689","heartbeat_front_addr":"172.21.15.7:6803/2069016689","state":["exists","up"]},{"osd":1,"uuid":"5c3e7d1e-a61d-44b9-b21e-f91c99a096bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":865486532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":865486532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":865486532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6807","nonce":865486532}]},"public_addr":"172.21.15.7:6805/865486532","cluster_addr":"172.21.15.7:6806/865486532","heartbeat_back_addr":"172.21.15.7:6808/865486532","heartbeat_front_addr":"172.21.15.7:6807/865486532","state":["exists","up"]},{"osd":2,"uuid":"f4fd930f-7bb2-46ca-9964-98235bd47b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":904871370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":904871370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":904871370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":904871370}]},"public_addr":"172.21.15.7:6809/904871370","cluster_addr":"172.21.15.7:6810/904871370","heartbeat_back_addr":"172.21.15.7:6812/904871370","heartbeat_front_addr":"172.21.15.7:6811/904871370","state":["exists","up"]},{"osd":3,"uuid":"149ed0d6-b049-4478-9819-cafc446e94d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":2758586242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2758586242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2758586242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2758586242}]},"public_addr":"172.21.15.7:6813/2758586242","cluster_addr":"172.21.15.7:6814/2758586242","heartbeat_back_addr":"172.21.15.7:6816/2758586242","heartbeat_front_addr":"172.21.15.7:6815/2758586242","state":["exists","up"]},{"osd":4,"uuid":"3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":1154929470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6801","nonce":1154929470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6803","nonce":1154929470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1154929470}]},"public_addr":"172.21.15.90:6800/1154929470","cluster_addr":"172.21.15.90:6801/1154929470","heartbeat_back_addr":"172.21.15.90:6803/1154929470","heartbeat_front_addr":"172.21.15.90:6802/1154929470","state":["exists","up"]},{"osd":5,"uuid":"244a7930-c22c-413e-86e1-11e0749f7367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":398113577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6805","nonce":398113577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6807","nonce":398113577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":398113577}]},"public_addr":"172.21.15.90:6804/398113577","cluster_addr":"172.21.15.90:6805/398113577","heartbeat_back_addr":"172.21.15.90:6807/398113577","heartbeat_front_addr":"172.21.15.90:6806/398113577","state":["exists","up"]},{"osd":6,"uuid":"0adbb562-3012-4f60-b5ec-02550a10b2c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2371506925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2371506925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6811","nonce":2371506925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2371506925}]},"public_addr":"172.21.15.90:6808/2371506925","cluster_addr":"172.21.15.90:6809/2371506925","heartbeat_back_addr":"172.21.15.90:6811/2371506925","heartbeat_front_addr":"172.21.15.90:6810/2371506925","state":["exists","up"]},{"osd":7,"uuid":"dbc26c36-a0e9-4f21-9e98-739270cea6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2983605980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":2983605980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":2983605980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2983605980}]},"public_addr":"172.21.15.90:6812/2983605980","cluster_addr":"172.21.15.90:6813/2983605980","heartbeat_back_addr":"172.21.15.90:6815/2983605980","heartbeat_front_addr":"172.21.15.90:6814/2983605980","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:19:11.460565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:18:53.860218+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:19:04.572025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:19:15.384158+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:18:45.753265+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:19:03.796302+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:19:23.927654+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:18:33.632182+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3418781614":"2024-02-06T07:14:14.164720+0000","172.21.15.7:6800/1518592084":"2024-02-06T07:13:47.004765+0000","172.21.15.7:0/2208590106":"2024-02-06T07:13:47.004765+0000","172.21.15.7:0/223579766":"2024-02-06T07:13:47.004765+0000","172.21.15.7:6800/2634895574":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/1462812719":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/2823976078":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/1905520582":"2024-02-06T07:13:47.004765+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T07:19:27.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:27 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2748159455' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:19:27.525 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:27 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:27.366+0000 7f0611d80200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T07:19:27.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:27 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2748159455' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:19:27.706 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:27 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2748159455' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:19:28.174 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:27 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:27.703+0000 7fdd5ef7a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T07:19:28.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-05T07:19:28.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-05T07:19:28.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-05T07:19:28.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-05T07:19:28.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-05T07:19:28.218 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-05T07:19:28.218 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-05T07:19:28.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-05T07:19:28.248 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:27 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:27.934+0000 7f0611d80200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T07:19:28.248 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:28 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:28.031+0000 7f0611d80200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T07:19:28.248 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:28 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:28.245+0000 7f0611d80200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T07:19:28.484 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:28.357+0000 7fdd5ef7a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T07:19:28.486 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-mon[29098]: from='mgr.? 172.21.15.90:0/2527373337' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T07:19:28.486 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-mon[29098]: Standby manager daemon x restarted 2024-02-05T07:19:28.487 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-mon[29098]: Standby manager daemon x started 2024-02-05T07:19:28.487 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-mon[29098]: from='mgr.? 172.21.15.90:0/2527373337' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T07:19:28.487 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-mon[29098]: from='mgr.? 172.21.15.90:0/2527373337' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T07:19:28.487 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-mon[29098]: from='mgr.? 172.21.15.90:0/2527373337' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T07:19:28.487 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-mon[37158]: from='mgr.? 172.21.15.90:0/2527373337' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T07:19:28.487 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-mon[37158]: Standby manager daemon x restarted 2024-02-05T07:19:28.487 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-mon[37158]: Standby manager daemon x started 2024-02-05T07:19:28.487 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-mon[37158]: from='mgr.? 172.21.15.90:0/2527373337' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T07:19:28.487 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-mon[37158]: from='mgr.? 172.21.15.90:0/2527373337' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T07:19:28.487 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-mon[37158]: from='mgr.? 172.21.15.90:0/2527373337' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T07:19:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:28 smithi090 ceph-mon[33213]: from='mgr.? 172.21.15.90:0/2527373337' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T07:19:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:28 smithi090 ceph-mon[33213]: Standby manager daemon x restarted 2024-02-05T07:19:28.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:28 smithi090 ceph-mon[33213]: Standby manager daemon x started 2024-02-05T07:19:28.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:28 smithi090 ceph-mon[33213]: from='mgr.? 172.21.15.90:0/2527373337' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T07:19:28.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:28 smithi090 ceph-mon[33213]: from='mgr.? 172.21.15.90:0/2527373337' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T07:19:28.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:28 smithi090 ceph-mon[33213]: from='mgr.? 172.21.15.90:0/2527373337' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T07:19:28.525 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:28 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: [05/Feb/2024:07:19:28] ENGINE Bus STARTING 2024-02-05T07:19:28.525 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:28 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: CherryPy Checker: 2024-02-05T07:19:28.525 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:28 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: The Application mounted at '' has an empty config. 2024-02-05T07:19:28.525 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:28 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: 2024-02-05T07:19:28.525 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:28 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: [05/Feb/2024:07:19:28] ENGINE Serving on http://:::9283 2024-02-05T07:19:28.525 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:19:28 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x[35100]: [05/Feb/2024:07:19:28] ENGINE Bus STARTED 2024-02-05T07:19:28.691 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:28.788 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:28.789 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:28.798 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:28.802 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:28.802 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:28.811 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:28.816 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:28.883 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:28.542+0000 7fdd5ef7a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T07:19:29.174 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:28.881+0000 7fdd5ef7a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T07:19:29.174 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:28 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:19:28] ENGINE Bus STARTING 2024-02-05T07:19:29.174 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: CherryPy Checker: 2024-02-05T07:19:29.174 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: The Application mounted at '' has an empty config. 2024-02-05T07:19:29.174 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:19:29.174 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:19:29] ENGINE Serving on http://:::9283 2024-02-05T07:19:29.174 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:19:29] ENGINE Bus STARTED 2024-02-05T07:19:29.746 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-05T07:19:29.746 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: Active manager daemon y restarted 2024-02-05T07:19:29.746 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: Activating manager daemon y 2024-02-05T07:19:29.746 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: osdmap e85: 8 total, 8 up, 8 in 2024-02-05T07:19:29.746 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: mgrmap e19: y(active, starting, since 0.00892792s), standbys: x 2024-02-05T07:19:29.746 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:19:29.746 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:19:29.746 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:19:29.747 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T07:19:29.747 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T07:19:29.747 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:19:29.747 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:19:29.747 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:19:29.747 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:19:29.747 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:19:29.747 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:19:29.747 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:19:29.747 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:19:29.747 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T07:19:29.747 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T07:19:29.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T07:19:29.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: Manager daemon y is now available 2024-02-05T07:19:29.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:19:29.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:19:29.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:29.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:19:29.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T07:19:29.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T07:19:29.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T07:19:29.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T07:19:29.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[29098]: osdmap e86: 8 total, 8 up, 8 in 2024-02-05T07:19:29.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-05T07:19:29.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: Active manager daemon y restarted 2024-02-05T07:19:29.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: Activating manager daemon y 2024-02-05T07:19:29.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: osdmap e85: 8 total, 8 up, 8 in 2024-02-05T07:19:29.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: mgrmap e19: y(active, starting, since 0.00892792s), standbys: x 2024-02-05T07:19:29.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:19:29.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:19:29.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:19:29.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T07:19:29.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T07:19:29.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:19:29.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: Manager daemon y is now available 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:19:29.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T07:19:29.751 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T07:19:29.751 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T07:19:29.751 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T07:19:29.751 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:29 smithi007 ceph-mon[37158]: osdmap e86: 8 total, 8 up, 8 in 2024-02-05T07:19:29.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-05T07:19:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: Active manager daemon y restarted 2024-02-05T07:19:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: Activating manager daemon y 2024-02-05T07:19:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: osdmap e85: 8 total, 8 up, 8 in 2024-02-05T07:19:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: mgrmap e19: y(active, starting, since 0.00892792s), standbys: x 2024-02-05T07:19:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:19:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T07:19:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T07:19:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T07:19:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T07:19:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T07:19:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T07:19:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T07:19:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T07:19:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T07:19:29.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T07:19:29.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:19:29.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T07:19:29.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T07:19:29.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T07:19:29.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T07:19:29.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: Manager daemon y is now available 2024-02-05T07:19:29.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:19:29.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:19:29.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:29.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:19:29.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T07:19:29.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T07:19:29.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T07:19:29.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T07:19:29.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:29 smithi090 ceph-mon[33213]: osdmap e86: 8 total, 8 up, 8 in 2024-02-05T07:19:31.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:30 smithi007 ceph-mon[29098]: [05/Feb/2024:07:19:29] ENGINE Bus STARTING 2024-02-05T07:19:31.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:30 smithi007 ceph-mon[29098]: [05/Feb/2024:07:19:29] ENGINE Serving on http://172.21.15.7:8765 2024-02-05T07:19:31.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:30 smithi007 ceph-mon[29098]: [05/Feb/2024:07:19:29] ENGINE Serving on https://172.21.15.7:7150 2024-02-05T07:19:31.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:30 smithi007 ceph-mon[29098]: [05/Feb/2024:07:19:29] ENGINE Bus STARTED 2024-02-05T07:19:31.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:30 smithi007 ceph-mon[29098]: mgrmap e20: y(active, since 1.01351s), standbys: x 2024-02-05T07:19:31.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:30 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:19:31.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:30 smithi007 ceph-mon[29098]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:19:31.005 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:30 smithi007 ceph-mon[37158]: [05/Feb/2024:07:19:29] ENGINE Bus STARTING 2024-02-05T07:19:31.005 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:30 smithi007 ceph-mon[37158]: [05/Feb/2024:07:19:29] ENGINE Serving on http://172.21.15.7:8765 2024-02-05T07:19:31.005 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:30 smithi007 ceph-mon[37158]: [05/Feb/2024:07:19:29] ENGINE Serving on https://172.21.15.7:7150 2024-02-05T07:19:31.005 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:30 smithi007 ceph-mon[37158]: [05/Feb/2024:07:19:29] ENGINE Bus STARTED 2024-02-05T07:19:31.005 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:30 smithi007 ceph-mon[37158]: mgrmap e20: y(active, since 1.01351s), standbys: x 2024-02-05T07:19:31.005 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:30 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:19:31.006 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:30 smithi007 ceph-mon[37158]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:19:31.180 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:30 smithi090 ceph-mon[33213]: [05/Feb/2024:07:19:29] ENGINE Bus STARTING 2024-02-05T07:19:31.180 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:30 smithi090 ceph-mon[33213]: [05/Feb/2024:07:19:29] ENGINE Serving on http://172.21.15.7:8765 2024-02-05T07:19:31.180 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:30 smithi090 ceph-mon[33213]: [05/Feb/2024:07:19:29] ENGINE Serving on https://172.21.15.7:7150 2024-02-05T07:19:31.180 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:30 smithi090 ceph-mon[33213]: [05/Feb/2024:07:19:29] ENGINE Bus STARTED 2024-02-05T07:19:31.180 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:30 smithi090 ceph-mon[33213]: mgrmap e20: y(active, since 1.01351s), standbys: x 2024-02-05T07:19:31.180 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:30 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:19:31.181 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:30 smithi090 ceph-mon[33213]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:19:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:31 smithi090 ceph-mon[33213]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:19:32.137 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:31 smithi007 ceph-mon[29098]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:19:32.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:31 smithi007 ceph-mon[37158]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:19:33.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:32 smithi007 ceph-mon[29098]: mgrmap e21: y(active, since 3s), standbys: x 2024-02-05T07:19:33.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:32 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:33.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:32 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:33.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:32 smithi007 ceph-mon[37158]: mgrmap e21: y(active, since 3s), standbys: x 2024-02-05T07:19:33.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:32 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:33.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:32 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:32 smithi090 ceph-mon[33213]: mgrmap e21: y(active, since 3s), standbys: x 2024-02-05T07:19:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:32 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:33.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:32 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:33.829 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:19:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:19:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[29098]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:19:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[29098]: Adjusting osd_memory_target on smithi090 to 3760M 2024-02-05T07:19:34.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:34.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[29098]: mgrmap e22: y(active, since 5s), standbys: x 2024-02-05T07:19:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[37158]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:19:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.427 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.427 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.427 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.427 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.427 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[37158]: Adjusting osd_memory_target on smithi090 to 3760M 2024-02-05T07:19:34.427 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:34.427 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:34 smithi007 ceph-mon[37158]: mgrmap e22: y(active, since 5s), standbys: x 2024-02-05T07:19:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:34 smithi090 ceph-mon[33213]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:19:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:34 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:34 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:34 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:34 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:34 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:34 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:34 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:34 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:34 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:34 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:34 smithi090 ceph-mon[33213]: Adjusting osd_memory_target on smithi090 to 3760M 2024-02-05T07:19:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:34 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:34.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:34 smithi090 ceph-mon[33213]: mgrmap e22: y(active, since 5s), standbys: x 2024-02-05T07:19:36.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:36 smithi007 ceph-mon[29098]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:19:36.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:36 smithi007 ceph-mon[37158]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:19:36.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:36 smithi090 ceph-mon[33213]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-05T07:19:38.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:38 smithi007 ceph-mon[29098]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-05T07:19:38.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:38 smithi007 ceph-mon[37158]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-05T07:19:38.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:38 smithi090 ceph-mon[33213]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-05T07:19:38.588 INFO:teuthology.orchestra.run.smithi007.stdout:137438953501 2024-02-05T07:19:38.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-05T07:19:39.469 INFO:teuthology.orchestra.run.smithi007.stdout:77309411367 2024-02-05T07:19:39.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-05T07:19:40.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:19:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:19:40.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:40 smithi007 ceph-mon[29098]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-05T07:19:40.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:40 smithi007 ceph-mon[37158]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-05T07:19:40.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:40 smithi090 ceph-mon[33213]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-05T07:19:41.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:41 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:19:41.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:41 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:19:41.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:41 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:19:41.846 INFO:teuthology.orchestra.run.smithi007.stdout:111669149730 2024-02-05T07:19:41.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-05T07:19:42.498 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:42 smithi007 ceph-mon[29098]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-05T07:19:42.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:42 smithi007 ceph-mon[37158]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-05T07:19:42.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:42 smithi090 ceph-mon[33213]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-05T07:19:42.531 INFO:teuthology.orchestra.run.smithi007.stdout:176093659160 2024-02-05T07:19:42.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-05T07:19:43.175 INFO:teuthology.orchestra.run.smithi007.stdout:223338299413 2024-02-05T07:19:43.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-05T07:19:43.550 INFO:teuthology.orchestra.run.smithi007.stdout:34359738416 2024-02-05T07:19:43.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-05T07:19:43.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:19:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:19:44.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:44 smithi007 ceph-mon[29098]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-05T07:19:44.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:44 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:19:44.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:44 smithi007 ceph-mon[37158]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-05T07:19:44.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:44 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:19:44.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:44 smithi090 ceph-mon[33213]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-05T07:19:44.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:44 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:19:44.598 INFO:teuthology.orchestra.run.smithi007.stdout:257698037775 2024-02-05T07:19:44.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-05T07:19:44.955 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:44.955 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:44.960 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:44.961 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:44.961 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:44.962 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:46.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:46 smithi007 ceph-mon[29098]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-05T07:19:46.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:46 smithi007 ceph-mon[37158]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-05T07:19:46.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:46 smithi090 ceph-mon[33213]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-05T07:19:48.451 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:48 smithi007 ceph-mon[29098]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-05T07:19:48.451 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:48 smithi007 ceph-mon[37158]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-05T07:19:48.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:48 smithi090 ceph-mon[33213]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-05T07:19:50.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:19:49 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:19:50.414 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:19:50.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:50 smithi090 ceph-mon[33213]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:19:50.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:50 smithi090 ceph-mon[33213]: 3.16 scrub starts 2024-02-05T07:19:50.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:50 smithi090 ceph-mon[33213]: 3.16 scrub ok 2024-02-05T07:19:50.588 INFO:teuthology.orchestra.run.smithi007.stdout:34359738419 2024-02-05T07:19:50.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:50 smithi007 ceph-mon[29098]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:19:50.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:50 smithi007 ceph-mon[29098]: 3.16 scrub starts 2024-02-05T07:19:50.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:50 smithi007 ceph-mon[29098]: 3.16 scrub ok 2024-02-05T07:19:50.589 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:50 smithi007 ceph-mon[37158]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:19:50.589 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:50 smithi007 ceph-mon[37158]: 3.16 scrub starts 2024-02-05T07:19:50.589 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:50 smithi007 ceph-mon[37158]: 3.16 scrub ok 2024-02-05T07:19:51.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:51 smithi007 ceph-mon[29098]: 3.18 scrub starts 2024-02-05T07:19:51.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:51 smithi007 ceph-mon[29098]: 3.18 scrub ok 2024-02-05T07:19:51.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:51 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:19:51.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:51 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1070553399' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T07:19:51.287 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:51 smithi007 ceph-mon[37158]: 3.18 scrub starts 2024-02-05T07:19:51.287 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:51 smithi007 ceph-mon[37158]: 3.18 scrub ok 2024-02-05T07:19:51.287 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:51 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:19:51.287 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:51 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1070553399' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T07:19:51.313 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738416 got 34359738419 for osd.0 2024-02-05T07:19:51.313 DEBUG:teuthology.parallel:result is None 2024-02-05T07:19:51.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:51 smithi090 ceph-mon[33213]: 3.18 scrub starts 2024-02-05T07:19:51.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:51 smithi090 ceph-mon[33213]: 3.18 scrub ok 2024-02-05T07:19:51.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:51 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:19:51.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:51 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1070553399' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T07:19:52.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[37158]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:19:52.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[37158]: 3.6 scrub starts 2024-02-05T07:19:52.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[37158]: 3.6 scrub ok 2024-02-05T07:19:52.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:52.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:52.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[37158]: 4.d scrub starts 2024-02-05T07:19:52.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[37158]: 4.d scrub ok 2024-02-05T07:19:52.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:19:52.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:19:52.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[29098]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:19:52.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[29098]: 3.6 scrub starts 2024-02-05T07:19:52.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[29098]: 3.6 scrub ok 2024-02-05T07:19:52.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:52.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:52.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[29098]: 4.d scrub starts 2024-02-05T07:19:52.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[29098]: 4.d scrub ok 2024-02-05T07:19:52.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:19:52.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:52 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:19:52.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:52 smithi090 ceph-mon[33213]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:19:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:52 smithi090 ceph-mon[33213]: 3.6 scrub starts 2024-02-05T07:19:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:52 smithi090 ceph-mon[33213]: 3.6 scrub ok 2024-02-05T07:19:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:52 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:52 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:52 smithi090 ceph-mon[33213]: 4.d scrub starts 2024-02-05T07:19:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:52 smithi090 ceph-mon[33213]: 4.d scrub ok 2024-02-05T07:19:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:52 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:19:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:52 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:19:52.965 INFO:teuthology.orchestra.run.smithi007.stdout:77309411370 2024-02-05T07:19:53.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: 3.e scrub starts 2024-02-05T07:19:53.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: 3.e scrub ok 2024-02-05T07:19:53.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: osdmap e87: 8 total, 8 up, 8 in 2024-02-05T07:19:53.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:53.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:53.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:53.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:19:53.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:19:53.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1003687188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T07:19:53.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: 3.e scrub starts 2024-02-05T07:19:53.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: 3.e scrub ok 2024-02-05T07:19:53.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: osdmap e87: 8 total, 8 up, 8 in 2024-02-05T07:19:53.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:53.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:53.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:53.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:19:53.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:19:53.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1003687188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T07:19:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: 3.e scrub starts 2024-02-05T07:19:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: 3.e scrub ok 2024-02-05T07:19:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: osdmap e87: 8 total, 8 up, 8 in 2024-02-05T07:19:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:19:53.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:53.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:19:53.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:19:53.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:53 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1003687188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T07:19:53.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:19:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:19:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:19:54.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: Adjusting osd_memory_target on smithi007 to 3761M 2024-02-05T07:19:54.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: Updating smithi007:/etc/ceph/ceph.conf 2024-02-05T07:19:54.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-05T07:19:54.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-02-05T07:19:54.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: Updating smithi007:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:19:54.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:19:54.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-02-05T07:19:54.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-02-05T07:19:54.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: Updating smithi007:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T07:19:54.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[29098]: osdmap e88: 8 total, 8 up, 8 in 2024-02-05T07:19:54.515 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: Adjusting osd_memory_target on smithi007 to 3761M 2024-02-05T07:19:54.515 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: Updating smithi007:/etc/ceph/ceph.conf 2024-02-05T07:19:54.515 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-05T07:19:54.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-02-05T07:19:54.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: Updating smithi007:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:19:54.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:19:54.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-02-05T07:19:54.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-02-05T07:19:54.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: Updating smithi007:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T07:19:54.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:54 smithi007 ceph-mon[37158]: osdmap e88: 8 total, 8 up, 8 in 2024-02-05T07:19:54.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: Adjusting osd_memory_target on smithi007 to 3761M 2024-02-05T07:19:54.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: Updating smithi007:/etc/ceph/ceph.conf 2024-02-05T07:19:54.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-05T07:19:54.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-02-05T07:19:54.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: Updating smithi007:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:19:54.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T07:19:54.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-02-05T07:19:54.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-02-05T07:19:54.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: Updating smithi007:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T07:19:54.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:19:54.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:54 smithi090 ceph-mon[33213]: osdmap e88: 8 total, 8 up, 8 in 2024-02-05T07:19:55.319 INFO:teuthology.orchestra.run.smithi007.stdout:223338299415 2024-02-05T07:19:55.523 INFO:teuthology.orchestra.run.smithi007.stdout:137438953504 2024-02-05T07:19:55.523 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:55 smithi007 ceph-mon[29098]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T07:19:55.523 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:55 smithi007 ceph-mon[29098]: Deploying daemon alertmanager.a on smithi007 2024-02-05T07:19:55.523 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:55 smithi007 ceph-mon[29098]: 5.18 scrub starts 2024-02-05T07:19:55.524 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:55 smithi007 ceph-mon[29098]: 5.18 scrub ok 2024-02-05T07:19:55.524 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:55 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3444513109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T07:19:55.524 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:55 smithi007 ceph-mon[37158]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T07:19:55.524 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:55 smithi007 ceph-mon[37158]: Deploying daemon alertmanager.a on smithi007 2024-02-05T07:19:55.524 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:55 smithi007 ceph-mon[37158]: 5.18 scrub starts 2024-02-05T07:19:55.524 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:55 smithi007 ceph-mon[37158]: 5.18 scrub ok 2024-02-05T07:19:55.524 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:55 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3444513109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T07:19:55.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:55 smithi090 ceph-mon[33213]: Updating smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T07:19:55.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:55 smithi090 ceph-mon[33213]: Deploying daemon alertmanager.a on smithi007 2024-02-05T07:19:55.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:55 smithi090 ceph-mon[33213]: 5.18 scrub starts 2024-02-05T07:19:55.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:55 smithi090 ceph-mon[33213]: 5.18 scrub ok 2024-02-05T07:19:55.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:55 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3444513109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T07:19:55.785 INFO:teuthology.orchestra.run.smithi007.stdout:55834574894 2024-02-05T07:19:55.786 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-05T07:19:56.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:56 smithi007 ceph-mon[29098]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:19:56.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:56 smithi007 ceph-mon[29098]: 3.10 scrub starts 2024-02-05T07:19:56.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:56 smithi007 ceph-mon[29098]: 3.10 scrub ok 2024-02-05T07:19:56.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:56 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2215367814' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T07:19:56.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:56 smithi007 ceph-mon[37158]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:19:56.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:56 smithi007 ceph-mon[37158]: 3.10 scrub starts 2024-02-05T07:19:56.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:56 smithi007 ceph-mon[37158]: 3.10 scrub ok 2024-02-05T07:19:56.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:56 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2215367814' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T07:19:56.676 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:56 smithi090 ceph-mon[33213]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:19:56.676 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:56 smithi090 ceph-mon[33213]: 3.10 scrub starts 2024-02-05T07:19:56.676 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:56 smithi090 ceph-mon[33213]: 3.10 scrub ok 2024-02-05T07:19:56.676 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:56 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2215367814' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T07:19:57.256 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411370 for osd.2 2024-02-05T07:19:57.256 DEBUG:teuthology.parallel:result is None 2024-02-05T07:19:57.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:57 smithi007 ceph-mon[29098]: 3.d scrub starts 2024-02-05T07:19:57.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:57 smithi007 ceph-mon[29098]: 3.d scrub ok 2024-02-05T07:19:57.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:57 smithi007 ceph-mon[29098]: 5.2 deep-scrub starts 2024-02-05T07:19:57.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:57 smithi007 ceph-mon[29098]: 5.2 deep-scrub ok 2024-02-05T07:19:57.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:57 smithi007 ceph-mon[29098]: 2.1 scrub starts 2024-02-05T07:19:57.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:57 smithi007 ceph-mon[29098]: 2.1 scrub ok 2024-02-05T07:19:57.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:57 smithi007 ceph-mon[37158]: 3.d scrub starts 2024-02-05T07:19:57.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:57 smithi007 ceph-mon[37158]: 3.d scrub ok 2024-02-05T07:19:57.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:57 smithi007 ceph-mon[37158]: 5.2 deep-scrub starts 2024-02-05T07:19:57.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:57 smithi007 ceph-mon[37158]: 5.2 deep-scrub ok 2024-02-05T07:19:57.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:57 smithi007 ceph-mon[37158]: 2.1 scrub starts 2024-02-05T07:19:57.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:57 smithi007 ceph-mon[37158]: 2.1 scrub ok 2024-02-05T07:19:57.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:57 smithi090 ceph-mon[33213]: 3.d scrub starts 2024-02-05T07:19:57.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:57 smithi090 ceph-mon[33213]: 3.d scrub ok 2024-02-05T07:19:57.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:57 smithi090 ceph-mon[33213]: 5.2 deep-scrub starts 2024-02-05T07:19:57.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:57 smithi090 ceph-mon[33213]: 5.2 deep-scrub ok 2024-02-05T07:19:57.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:57 smithi090 ceph-mon[33213]: 2.1 scrub starts 2024-02-05T07:19:57.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:57 smithi090 ceph-mon[33213]: 2.1 scrub ok 2024-02-05T07:19:58.049 INFO:teuthology.orchestra.run.smithi007.stdout:257698037778 2024-02-05T07:19:58.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:58 smithi007 ceph-mon[29098]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:19:58.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:58 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3714986563' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T07:19:58.514 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:58 smithi007 ceph-mon[37158]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:19:58.514 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:58 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3714986563' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T07:19:58.694 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299413 got 223338299415 for osd.6 2024-02-05T07:19:58.694 DEBUG:teuthology.parallel:result is None 2024-02-05T07:19:58.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:58 smithi090 ceph-mon[33213]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:19:58.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:58 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3714986563' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T07:19:59.524 INFO:teuthology.orchestra.run.smithi007.stdout:111669149734 2024-02-05T07:19:59.524 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:19:59 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:19:59.525 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:19:59 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:19:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:19:59 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:20:00.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:19:59 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:20:00.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:00 smithi007 ceph-mon[37158]: 4.8 scrub starts 2024-02-05T07:20:00.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:00 smithi007 ceph-mon[37158]: 4.8 scrub ok 2024-02-05T07:20:00.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:00 smithi007 ceph-mon[37158]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:20:00.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:00 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3865999241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T07:20:00.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:00 smithi007 ceph-mon[37158]: 4.1c scrub starts 2024-02-05T07:20:00.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:00 smithi007 ceph-mon[37158]: 4.1c scrub ok 2024-02-05T07:20:00.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:00 smithi007 ceph-mon[37158]: overall HEALTH_OK 2024-02-05T07:20:00.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:00 smithi007 ceph-mon[29098]: 4.8 scrub starts 2024-02-05T07:20:00.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:00 smithi007 ceph-mon[29098]: 4.8 scrub ok 2024-02-05T07:20:00.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:00 smithi007 ceph-mon[29098]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:20:00.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:00 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3865999241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T07:20:00.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:00 smithi007 ceph-mon[29098]: 4.1c scrub starts 2024-02-05T07:20:00.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:00 smithi007 ceph-mon[29098]: 4.1c scrub ok 2024-02-05T07:20:00.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:00 smithi007 ceph-mon[29098]: overall HEALTH_OK 2024-02-05T07:20:00.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:00 smithi090 ceph-mon[33213]: 4.8 scrub starts 2024-02-05T07:20:00.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:00 smithi090 ceph-mon[33213]: 4.8 scrub ok 2024-02-05T07:20:00.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:00 smithi090 ceph-mon[33213]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:20:00.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:00 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3865999241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T07:20:00.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:00 smithi090 ceph-mon[33213]: 4.1c scrub starts 2024-02-05T07:20:00.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:00 smithi090 ceph-mon[33213]: 4.1c scrub ok 2024-02-05T07:20:00.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:00 smithi090 ceph-mon[33213]: overall HEALTH_OK 2024-02-05T07:20:01.662 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[29098]: 3.1d scrub starts 2024-02-05T07:20:01.662 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[29098]: 3.1d scrub ok 2024-02-05T07:20:01.662 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:01.662 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:20:01.663 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:20:01.663 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[29098]: 4.a scrub starts 2024-02-05T07:20:01.663 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[29098]: 4.a scrub ok 2024-02-05T07:20:01.663 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[29098]: osdmap e89: 8 total, 8 up, 8 in 2024-02-05T07:20:01.663 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[29098]: 6.1e scrub starts 2024-02-05T07:20:01.663 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[29098]: 6.1e scrub ok 2024-02-05T07:20:01.663 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[37158]: 3.1d scrub starts 2024-02-05T07:20:01.663 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[37158]: 3.1d scrub ok 2024-02-05T07:20:01.663 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:01.663 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:20:01.663 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:20:01.663 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[37158]: 4.a scrub starts 2024-02-05T07:20:01.664 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[37158]: 4.a scrub ok 2024-02-05T07:20:01.664 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[37158]: osdmap e89: 8 total, 8 up, 8 in 2024-02-05T07:20:01.664 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[37158]: 6.1e scrub starts 2024-02-05T07:20:01.664 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:01 smithi007 ceph-mon[37158]: 6.1e scrub ok 2024-02-05T07:20:01.721 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149730 got 111669149734 for osd.3 2024-02-05T07:20:01.721 DEBUG:teuthology.parallel:result is None 2024-02-05T07:20:01.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:01 smithi090 ceph-mon[33213]: 3.1d scrub starts 2024-02-05T07:20:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:01 smithi090 ceph-mon[33213]: 3.1d scrub ok 2024-02-05T07:20:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:01 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:20:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:01 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:20:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:01 smithi090 ceph-mon[33213]: 4.a scrub starts 2024-02-05T07:20:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:01 smithi090 ceph-mon[33213]: 4.a scrub ok 2024-02-05T07:20:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:01 smithi090 ceph-mon[33213]: osdmap e89: 8 total, 8 up, 8 in 2024-02-05T07:20:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:01 smithi090 ceph-mon[33213]: 6.1e scrub starts 2024-02-05T07:20:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:01 smithi090 ceph-mon[33213]: 6.1e scrub ok 2024-02-05T07:20:01.812 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037775 got 257698037778 for osd.7 2024-02-05T07:20:01.812 DEBUG:teuthology.parallel:result is None 2024-02-05T07:20:02.327 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:02.355 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953501 got 137438953504 for osd.4 2024-02-05T07:20:02.356 DEBUG:teuthology.parallel:result is None 2024-02-05T07:20:02.513 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:02 smithi007 ceph-mon[29098]: 5.9 deep-scrub starts 2024-02-05T07:20:02.513 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:02 smithi007 ceph-mon[29098]: 5.9 deep-scrub ok 2024-02-05T07:20:02.513 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:02 smithi007 ceph-mon[29098]: 4.0 scrub starts 2024-02-05T07:20:02.513 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:02 smithi007 ceph-mon[29098]: 4.0 scrub ok 2024-02-05T07:20:02.513 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:02 smithi007 ceph-mon[29098]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:02.514 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:02 smithi007 ceph-mon[37158]: 5.9 deep-scrub starts 2024-02-05T07:20:02.514 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:02 smithi007 ceph-mon[37158]: 5.9 deep-scrub ok 2024-02-05T07:20:02.514 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:02 smithi007 ceph-mon[37158]: 4.0 scrub starts 2024-02-05T07:20:02.514 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:02 smithi007 ceph-mon[37158]: 4.0 scrub ok 2024-02-05T07:20:02.514 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:02 smithi007 ceph-mon[37158]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:02.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:02 smithi090 ceph-mon[33213]: 5.9 deep-scrub starts 2024-02-05T07:20:02.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:02 smithi090 ceph-mon[33213]: 5.9 deep-scrub ok 2024-02-05T07:20:02.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:02 smithi090 ceph-mon[33213]: 4.0 scrub starts 2024-02-05T07:20:02.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:02 smithi090 ceph-mon[33213]: 4.0 scrub ok 2024-02-05T07:20:02.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:02 smithi090 ceph-mon[33213]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:03.643 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:20:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:20:03.643 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[37158]: 6.12 scrub starts 2024-02-05T07:20:03.643 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[37158]: 6.12 scrub ok 2024-02-05T07:20:03.643 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[37158]: 4.1d scrub starts 2024-02-05T07:20:03.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[37158]: 4.1d scrub ok 2024-02-05T07:20:03.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[37158]: 5.8 scrub starts 2024-02-05T07:20:03.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[37158]: 5.8 scrub ok 2024-02-05T07:20:03.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[37158]: 4.1a deep-scrub starts 2024-02-05T07:20:03.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[37158]: 4.1a deep-scrub ok 2024-02-05T07:20:03.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[29098]: 6.12 scrub starts 2024-02-05T07:20:03.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[29098]: 6.12 scrub ok 2024-02-05T07:20:03.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[29098]: 4.1d scrub starts 2024-02-05T07:20:03.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[29098]: 4.1d scrub ok 2024-02-05T07:20:03.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[29098]: 5.8 scrub starts 2024-02-05T07:20:03.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[29098]: 5.8 scrub ok 2024-02-05T07:20:03.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[29098]: 4.1a deep-scrub starts 2024-02-05T07:20:03.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:03 smithi007 ceph-mon[29098]: 4.1a deep-scrub ok 2024-02-05T07:20:03.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:03 smithi090 ceph-mon[33213]: 6.12 scrub starts 2024-02-05T07:20:03.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:03 smithi090 ceph-mon[33213]: 6.12 scrub ok 2024-02-05T07:20:03.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:03 smithi090 ceph-mon[33213]: 4.1d scrub starts 2024-02-05T07:20:03.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:03 smithi090 ceph-mon[33213]: 4.1d scrub ok 2024-02-05T07:20:03.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:03 smithi090 ceph-mon[33213]: 5.8 scrub starts 2024-02-05T07:20:03.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:03 smithi090 ceph-mon[33213]: 5.8 scrub ok 2024-02-05T07:20:03.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:03 smithi090 ceph-mon[33213]: 4.1a deep-scrub starts 2024-02-05T07:20:03.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:03 smithi090 ceph-mon[33213]: 4.1a deep-scrub ok 2024-02-05T07:20:04.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[29098]: 3.11 deep-scrub starts 2024-02-05T07:20:04.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[29098]: 3.11 deep-scrub ok 2024-02-05T07:20:04.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[29098]: 5.17 scrub starts 2024-02-05T07:20:04.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[29098]: 5.17 scrub ok 2024-02-05T07:20:04.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[29098]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:20:04.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[29098]: 6.1d scrub starts 2024-02-05T07:20:04.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[29098]: 6.1d scrub ok 2024-02-05T07:20:04.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[29098]: osdmap e90: 8 total, 8 up, 8 in 2024-02-05T07:20:04.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[37158]: 3.11 deep-scrub starts 2024-02-05T07:20:04.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[37158]: 3.11 deep-scrub ok 2024-02-05T07:20:04.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[37158]: 5.17 scrub starts 2024-02-05T07:20:04.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[37158]: 5.17 scrub ok 2024-02-05T07:20:04.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[37158]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:20:04.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[37158]: 6.1d scrub starts 2024-02-05T07:20:04.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[37158]: 6.1d scrub ok 2024-02-05T07:20:04.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:04 smithi007 ceph-mon[37158]: osdmap e90: 8 total, 8 up, 8 in 2024-02-05T07:20:04.421 INFO:teuthology.orchestra.run.smithi007.stdout:176093659165 2024-02-05T07:20:04.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:04 smithi090 ceph-mon[33213]: 3.11 deep-scrub starts 2024-02-05T07:20:04.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:04 smithi090 ceph-mon[33213]: 3.11 deep-scrub ok 2024-02-05T07:20:04.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:04 smithi090 ceph-mon[33213]: 5.17 scrub starts 2024-02-05T07:20:04.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:04 smithi090 ceph-mon[33213]: 5.17 scrub ok 2024-02-05T07:20:04.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:04 smithi090 ceph-mon[33213]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:20:04.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:04 smithi090 ceph-mon[33213]: 6.1d scrub starts 2024-02-05T07:20:04.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:04 smithi090 ceph-mon[33213]: 6.1d scrub ok 2024-02-05T07:20:04.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:04 smithi090 ceph-mon[33213]: osdmap e90: 8 total, 8 up, 8 in 2024-02-05T07:20:05.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:05 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1227142622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T07:20:05.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:05 smithi007 ceph-mon[29098]: 4.9 scrub starts 2024-02-05T07:20:05.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:05 smithi007 ceph-mon[29098]: 4.9 scrub ok 2024-02-05T07:20:05.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:05 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1227142622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T07:20:05.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:05 smithi007 ceph-mon[37158]: 4.9 scrub starts 2024-02-05T07:20:05.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:05 smithi007 ceph-mon[37158]: 4.9 scrub ok 2024-02-05T07:20:05.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:05 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1227142622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T07:20:05.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:05 smithi090 ceph-mon[33213]: 4.9 scrub starts 2024-02-05T07:20:05.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:05 smithi090 ceph-mon[33213]: 4.9 scrub ok 2024-02-05T07:20:06.000 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659160 got 176093659165 for osd.5 2024-02-05T07:20:06.001 DEBUG:teuthology.parallel:result is None 2024-02-05T07:20:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:06 smithi007 ceph-mon[29098]: 3.b scrub starts 2024-02-05T07:20:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:06 smithi007 ceph-mon[29098]: 3.b scrub ok 2024-02-05T07:20:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:06 smithi007 ceph-mon[29098]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:06 smithi007 ceph-mon[29098]: 3.9 scrub starts 2024-02-05T07:20:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:06 smithi007 ceph-mon[29098]: 3.9 scrub ok 2024-02-05T07:20:06.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:06 smithi007 ceph-mon[37158]: 3.b scrub starts 2024-02-05T07:20:06.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:06 smithi007 ceph-mon[37158]: 3.b scrub ok 2024-02-05T07:20:06.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:06 smithi007 ceph-mon[37158]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:06.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:06 smithi007 ceph-mon[37158]: 3.9 scrub starts 2024-02-05T07:20:06.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:06 smithi007 ceph-mon[37158]: 3.9 scrub ok 2024-02-05T07:20:06.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:06 smithi090 ceph-mon[33213]: 3.b scrub starts 2024-02-05T07:20:06.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:06 smithi090 ceph-mon[33213]: 3.b scrub ok 2024-02-05T07:20:06.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:06 smithi090 ceph-mon[33213]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:06.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:06 smithi090 ceph-mon[33213]: 3.9 scrub starts 2024-02-05T07:20:06.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:06 smithi090 ceph-mon[33213]: 3.9 scrub ok 2024-02-05T07:20:06.998 INFO:teuthology.orchestra.run.smithi007.stdout:55834574898 2024-02-05T07:20:07.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[29098]: 5.10 deep-scrub starts 2024-02-05T07:20:07.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[29098]: 5.10 deep-scrub ok 2024-02-05T07:20:07.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[29098]: 4.1f scrub starts 2024-02-05T07:20:07.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[29098]: 4.1f scrub ok 2024-02-05T07:20:07.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[29098]: 5.c scrub starts 2024-02-05T07:20:07.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[29098]: 5.c scrub ok 2024-02-05T07:20:07.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[29098]: 4.13 scrub starts 2024-02-05T07:20:07.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[29098]: 4.13 scrub ok 2024-02-05T07:20:07.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2978385110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T07:20:07.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[37158]: 5.10 deep-scrub starts 2024-02-05T07:20:07.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[37158]: 5.10 deep-scrub ok 2024-02-05T07:20:07.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[37158]: 4.1f scrub starts 2024-02-05T07:20:07.602 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[37158]: 4.1f scrub ok 2024-02-05T07:20:07.602 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[37158]: 5.c scrub starts 2024-02-05T07:20:07.602 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[37158]: 5.c scrub ok 2024-02-05T07:20:07.602 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[37158]: 4.13 scrub starts 2024-02-05T07:20:07.602 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[37158]: 4.13 scrub ok 2024-02-05T07:20:07.602 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:07 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2978385110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T07:20:07.626 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574894 got 55834574898 for osd.1 2024-02-05T07:20:07.627 DEBUG:teuthology.parallel:result is None 2024-02-05T07:20:07.627 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-05T07:20:07.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:20:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:07 smithi090 ceph-mon[33213]: 5.10 deep-scrub starts 2024-02-05T07:20:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:07 smithi090 ceph-mon[33213]: 5.10 deep-scrub ok 2024-02-05T07:20:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:07 smithi090 ceph-mon[33213]: 4.1f scrub starts 2024-02-05T07:20:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:07 smithi090 ceph-mon[33213]: 4.1f scrub ok 2024-02-05T07:20:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:07 smithi090 ceph-mon[33213]: 5.c scrub starts 2024-02-05T07:20:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:07 smithi090 ceph-mon[33213]: 5.c scrub ok 2024-02-05T07:20:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:07 smithi090 ceph-mon[33213]: 4.13 scrub starts 2024-02-05T07:20:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:07 smithi090 ceph-mon[33213]: 4.13 scrub ok 2024-02-05T07:20:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:07 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2978385110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T07:20:07.924 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:07 smithi007 podman[71893]: 2024-02-05 07:20:07.842258244 +0000 UTC m=+0.135575206 volume create ecda5a5d6fc515d2203ceaa095a9ed2c44a35aa087b6cfa21c6664e1b4b37f90 2024-02-05T07:20:08.247 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:07 smithi007 podman[71893]: 2024-02-05 07:20:07.951074627 +0000 UTC m=+0.244391587 container create 9e76a975cda0902668349e41e7862feeffe8f1e8ab75b39bcb73ac0fb40334e6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T07:20:08.453 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:08.529 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-mon[29098]: 6.5 deep-scrub starts 2024-02-05T07:20:08.529 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-mon[29098]: 6.5 deep-scrub ok 2024-02-05T07:20:08.529 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-mon[29098]: 4.15 scrub starts 2024-02-05T07:20:08.529 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-mon[29098]: 4.15 scrub ok 2024-02-05T07:20:08.529 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-mon[29098]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:08.529 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:08.529 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:08.529 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-mon[37158]: 6.5 deep-scrub starts 2024-02-05T07:20:08.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-mon[37158]: 6.5 deep-scrub ok 2024-02-05T07:20:08.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-mon[37158]: 4.15 scrub starts 2024-02-05T07:20:08.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-mon[37158]: 4.15 scrub ok 2024-02-05T07:20:08.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-mon[37158]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:08.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:08.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:08.530 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:08 smithi007 podman[71893]: 2024-02-05 07:20:08.246083079 +0000 UTC m=+0.539400049 container init 9e76a975cda0902668349e41e7862feeffe8f1e8ab75b39bcb73ac0fb40334e6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T07:20:08.530 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:08 smithi007 podman[71893]: 2024-02-05 07:20:08.248893413 +0000 UTC m=+0.542210382 container start 9e76a975cda0902668349e41e7862feeffe8f1e8ab75b39bcb73ac0fb40334e6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T07:20:08.530 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[71934]: ts=2024-02-05T07:20:08.261Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-05T07:20:08.530 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[71934]: ts=2024-02-05T07:20:08.261Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-05T07:20:08.530 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[71934]: ts=2024-02-05T07:20:08.262Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.7 port=9094 2024-02-05T07:20:08.530 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[71934]: ts=2024-02-05T07:20:08.263Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-05T07:20:08.531 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[71934]: ts=2024-02-05T07:20:08.286Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-05T07:20:08.531 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[71934]: ts=2024-02-05T07:20:08.287Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-05T07:20:08.531 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[71934]: ts=2024-02-05T07:20:08.289Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-05T07:20:08.531 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[71934]: ts=2024-02-05T07:20:08.289Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-05T07:20:08.531 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:08 smithi007 bash[71893]: 9e76a975cda0902668349e41e7862feeffe8f1e8ab75b39bcb73ac0fb40334e6 2024-02-05T07:20:08.531 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:08 smithi007 systemd[1]: Started Ceph alertmanager.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:20:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:08 smithi090 ceph-mon[33213]: 6.5 deep-scrub starts 2024-02-05T07:20:08.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:08 smithi090 ceph-mon[33213]: 6.5 deep-scrub ok 2024-02-05T07:20:08.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:08 smithi090 ceph-mon[33213]: 4.15 scrub starts 2024-02-05T07:20:08.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:08 smithi090 ceph-mon[33213]: 4.15 scrub ok 2024-02-05T07:20:08.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:08 smithi090 ceph-mon[33213]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:08.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:08 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:08.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:08 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:08.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:20:08] ENGINE Bus STOPPING 2024-02-05T07:20:08.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:20:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T07:20:08.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:20:08] ENGINE Bus STOPPED 2024-02-05T07:20:08.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:20:08] ENGINE Bus STARTING 2024-02-05T07:20:08.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:20:08] ENGINE Serving on http://:::9283 2024-02-05T07:20:08.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:20:08] ENGINE Bus STARTED 2024-02-05T07:20:09.630 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.630 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.630 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[29098]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-05T07:20:09.631 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.631 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.631 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T07:20:09.631 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.631 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.631 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.631 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.631 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[37158]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-05T07:20:09.631 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.631 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.631 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T07:20:09.631 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.632 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:09 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:09 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:09 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:09 smithi090 ceph-mon[33213]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-05T07:20:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:09 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:09 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:09 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T07:20:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:09 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:09 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:10.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:20:09 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:20:10.348 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:20:10.351 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:20:10.674 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[71934]: ts=2024-02-05T07:20:10.264Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000501702s 2024-02-05T07:20:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[29098]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T07:20:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[29098]: Deploying daemon grafana.a on smithi090 2024-02-05T07:20:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[29098]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-02-05T07:20:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[29098]: 5.11 deep-scrub starts 2024-02-05T07:20:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[29098]: 5.11 deep-scrub ok 2024-02-05T07:20:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[29098]: 3.0 scrub starts 2024-02-05T07:20:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[29098]: 3.0 scrub ok 2024-02-05T07:20:10.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[29098]: 5.a scrub starts 2024-02-05T07:20:10.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[29098]: 5.a scrub ok 2024-02-05T07:20:10.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[29098]: 4.e deep-scrub starts 2024-02-05T07:20:10.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[29098]: 4.e deep-scrub ok 2024-02-05T07:20:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[37158]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T07:20:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[37158]: Deploying daemon grafana.a on smithi090 2024-02-05T07:20:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[37158]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-02-05T07:20:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[37158]: 5.11 deep-scrub starts 2024-02-05T07:20:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[37158]: 5.11 deep-scrub ok 2024-02-05T07:20:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[37158]: 3.0 scrub starts 2024-02-05T07:20:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[37158]: 3.0 scrub ok 2024-02-05T07:20:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[37158]: 5.a scrub starts 2024-02-05T07:20:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[37158]: 5.a scrub ok 2024-02-05T07:20:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[37158]: 4.e deep-scrub starts 2024-02-05T07:20:10.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:10 smithi007 ceph-mon[37158]: 4.e deep-scrub ok 2024-02-05T07:20:10.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:10 smithi090 ceph-mon[33213]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T07:20:10.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:10 smithi090 ceph-mon[33213]: Deploying daemon grafana.a on smithi090 2024-02-05T07:20:10.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:10 smithi090 ceph-mon[33213]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-02-05T07:20:10.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:10 smithi090 ceph-mon[33213]: 5.11 deep-scrub starts 2024-02-05T07:20:10.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:10 smithi090 ceph-mon[33213]: 5.11 deep-scrub ok 2024-02-05T07:20:10.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:10 smithi090 ceph-mon[33213]: 3.0 scrub starts 2024-02-05T07:20:10.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:10 smithi090 ceph-mon[33213]: 3.0 scrub ok 2024-02-05T07:20:10.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:10 smithi090 ceph-mon[33213]: 5.a scrub starts 2024-02-05T07:20:10.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:10 smithi090 ceph-mon[33213]: 5.a scrub ok 2024-02-05T07:20:10.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:10 smithi090 ceph-mon[33213]: 4.e deep-scrub starts 2024-02-05T07:20:10.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:10 smithi090 ceph-mon[33213]: 4.e deep-scrub ok 2024-02-05T07:20:10.975 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-02-05T07:20:08.918400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1002,"num_read_kb":848,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":605,"ondisk_log_size":605,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236712,"kb_used_data":8076,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724504,"statfs":{"total":767960285184,"available":767717892096,"internally_reserved":0,"allocated":8269824,"data_stored":4581032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.521576"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828002+0000","last_change":"2024-02-05T07:18:55.466290+0000","last_active":"2024-02-05T07:20:04.828002+0000","last_peered":"2024-02-05T07:20:04.828002+0000","last_clean":"2024-02-05T07:20:04.828002+0000","last_became_active":"2024-02-05T07:18:55.466001+0000","last_became_peered":"2024-02-05T07:18:55.466001+0000","last_unstale":"2024-02-05T07:20:04.828002+0000","last_undegraded":"2024-02-05T07:20:04.828002+0000","last_fullsized":"2024-02-05T07:20:04.828002+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:09.550888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.102929+0000","last_change":"2024-02-05T07:18:53.461117+0000","last_active":"2024-02-05T07:20:04.102929+0000","last_peered":"2024-02-05T07:20:04.102929+0000","last_clean":"2024-02-05T07:20:04.102929+0000","last_became_active":"2024-02-05T07:18:53.460820+0000","last_became_peered":"2024-02-05T07:18:53.460820+0000","last_unstale":"2024-02-05T07:20:04.102929+0000","last_undegraded":"2024-02-05T07:20:04.102929+0000","last_fullsized":"2024-02-05T07:20:04.102929+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:14.324555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105373+0000","last_change":"2024-02-05T07:18:49.434741+0000","last_active":"2024-02-05T07:20:04.105373+0000","last_peered":"2024-02-05T07:20:04.105373+0000","last_clean":"2024-02-05T07:20:04.105373+0000","last_became_active":"2024-02-05T07:18:49.434456+0000","last_became_peered":"2024-02-05T07:18:49.434456+0000","last_unstale":"2024-02-05T07:20:04.105373+0000","last_undegraded":"2024-02-05T07:20:04.105373+0000","last_fullsized":"2024-02-05T07:20:04.105373+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:12.107995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.827906+0000","last_change":"2024-02-05T07:18:51.449261+0000","last_active":"2024-02-05T07:20:04.827906+0000","last_peered":"2024-02-05T07:20:04.827906+0000","last_clean":"2024-02-05T07:20:04.827906+0000","last_became_active":"2024-02-05T07:18:51.448924+0000","last_became_peered":"2024-02-05T07:18:51.448924+0000","last_unstale":"2024-02-05T07:20:04.827906+0000","last_undegraded":"2024-02-05T07:20:04.827906+0000","last_fullsized":"2024-02-05T07:20:04.827906+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.547504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826011+0000","last_change":"2024-02-05T07:18:55.506169+0000","last_active":"2024-02-05T07:20:04.826011+0000","last_peered":"2024-02-05T07:20:04.826011+0000","last_clean":"2024-02-05T07:20:04.826011+0000","last_became_active":"2024-02-05T07:18:55.505718+0000","last_became_peered":"2024-02-05T07:18:55.505718+0000","last_unstale":"2024-02-05T07:20:04.826011+0000","last_undegraded":"2024-02-05T07:20:04.826011+0000","last_fullsized":"2024-02-05T07:20:04.826011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.751562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.827917+0000","last_change":"2024-02-05T07:19:54.746304+0000","last_active":"2024-02-05T07:20:04.827917+0000","last_peered":"2024-02-05T07:20:04.827917+0000","last_clean":"2024-02-05T07:20:04.827917+0000","last_became_active":"2024-02-05T07:18:53.461506+0000","last_became_peered":"2024-02-05T07:18:53.461506+0000","last_unstale":"2024-02-05T07:20:04.827917+0000","last_undegraded":"2024-02-05T07:20:04.827917+0000","last_fullsized":"2024-02-05T07:20:04.827917+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:54.746206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:19:54.746206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:57.230561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826409+0000","last_change":"2024-02-05T07:18:49.436786+0000","last_active":"2024-02-05T07:20:04.826409+0000","last_peered":"2024-02-05T07:20:04.826409+0000","last_clean":"2024-02-05T07:20:04.826409+0000","last_became_active":"2024-02-05T07:18:49.436490+0000","last_became_peered":"2024-02-05T07:18:49.436490+0000","last_unstale":"2024-02-05T07:20:04.826409+0000","last_undegraded":"2024-02-05T07:20:04.826409+0000","last_fullsized":"2024-02-05T07:20:04.826409+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825941+0000","last_change":"2024-02-05T07:18:51.445160+0000","last_active":"2024-02-05T07:20:04.825941+0000","last_peered":"2024-02-05T07:20:04.825941+0000","last_clean":"2024-02-05T07:20:04.825941+0000","last_became_active":"2024-02-05T07:18:51.444783+0000","last_became_peered":"2024-02-05T07:18:51.444783+0000","last_unstale":"2024-02-05T07:20:04.825941+0000","last_undegraded":"2024-02-05T07:20:04.825941+0000","last_fullsized":"2024-02-05T07:20:04.825941+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:05.434052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.102877+0000","last_change":"2024-02-05T07:18:55.477022+0000","last_active":"2024-02-05T07:20:04.102877+0000","last_peered":"2024-02-05T07:20:04.102877+0000","last_clean":"2024-02-05T07:20:04.102877+0000","last_became_active":"2024-02-05T07:18:55.474002+0000","last_became_peered":"2024-02-05T07:18:55.474002+0000","last_unstale":"2024-02-05T07:20:04.102877+0000","last_undegraded":"2024-02-05T07:20:04.102877+0000","last_fullsized":"2024-02-05T07:20:04.102877+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.705323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105906+0000","last_change":"2024-02-05T07:18:53.464788+0000","last_active":"2024-02-05T07:20:04.105906+0000","last_peered":"2024-02-05T07:20:04.105906+0000","last_clean":"2024-02-05T07:20:04.105906+0000","last_became_active":"2024-02-05T07:18:53.464520+0000","last_became_peered":"2024-02-05T07:18:53.464520+0000","last_unstale":"2024-02-05T07:20:04.105906+0000","last_undegraded":"2024-02-05T07:20:04.105906+0000","last_fullsized":"2024-02-05T07:20:04.105906+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.843776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106234+0000","last_change":"2024-02-05T07:19:59.782950+0000","last_active":"2024-02-05T07:20:04.106234+0000","last_peered":"2024-02-05T07:20:04.106234+0000","last_clean":"2024-02-05T07:20:04.106234+0000","last_became_active":"2024-02-05T07:18:49.435454+0000","last_became_peered":"2024-02-05T07:18:49.435454+0000","last_unstale":"2024-02-05T07:20:04.106234+0000","last_undegraded":"2024-02-05T07:20:04.106234+0000","last_fullsized":"2024-02-05T07:20:04.106234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:59.782841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:59.782841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.410088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"89'13","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828200+0000","last_change":"2024-02-05T07:20:02.798288+0000","last_active":"2024-02-05T07:20:04.828200+0000","last_peered":"2024-02-05T07:20:04.828200+0000","last_clean":"2024-02-05T07:20:04.828200+0000","last_became_active":"2024-02-05T07:18:51.444592+0000","last_became_peered":"2024-02-05T07:18:51.444592+0000","last_unstale":"2024-02-05T07:20:04.828200+0000","last_undegraded":"2024-02-05T07:20:04.828200+0000","last_fullsized":"2024-02-05T07:20:04.828200+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.798180+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:11.151835+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105609+0000","last_change":"2024-02-05T07:18:55.473667+0000","last_active":"2024-02-05T07:20:04.105609+0000","last_peered":"2024-02-05T07:20:04.105609+0000","last_clean":"2024-02-05T07:20:04.105609+0000","last_became_active":"2024-02-05T07:18:55.473368+0000","last_became_peered":"2024-02-05T07:18:55.473368+0000","last_unstale":"2024-02-05T07:20:04.105609+0000","last_undegraded":"2024-02-05T07:20:04.105609+0000","last_fullsized":"2024-02-05T07:20:04.105609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:15.915465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.805983+0000","last_change":"2024-02-05T07:18:53.457259+0000","last_active":"2024-02-05T07:20:04.805983+0000","last_peered":"2024-02-05T07:20:04.805983+0000","last_clean":"2024-02-05T07:20:04.805983+0000","last_became_active":"2024-02-05T07:18:53.456881+0000","last_became_peered":"2024-02-05T07:18:53.456881+0000","last_unstale":"2024-02-05T07:20:04.805983+0000","last_undegraded":"2024-02-05T07:20:04.805983+0000","last_fullsized":"2024-02-05T07:20:04.805983+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.078811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106066+0000","last_change":"2024-02-05T07:18:49.431652+0000","last_active":"2024-02-05T07:20:04.106066+0000","last_peered":"2024-02-05T07:20:04.106066+0000","last_clean":"2024-02-05T07:20:04.106066+0000","last_became_active":"2024-02-05T07:18:49.431220+0000","last_became_peered":"2024-02-05T07:18:49.431220+0000","last_unstale":"2024-02-05T07:20:04.106066+0000","last_undegraded":"2024-02-05T07:20:04.106066+0000","last_fullsized":"2024-02-05T07:20:04.106066+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828154+0000","last_change":"2024-02-05T07:18:51.448820+0000","last_active":"2024-02-05T07:20:04.828154+0000","last_peered":"2024-02-05T07:20:04.828154+0000","last_clean":"2024-02-05T07:20:04.828154+0000","last_became_active":"2024-02-05T07:18:51.448462+0000","last_became_peered":"2024-02-05T07:18:51.448462+0000","last_unstale":"2024-02-05T07:20:04.828154+0000","last_undegraded":"2024-02-05T07:20:04.828154+0000","last_fullsized":"2024-02-05T07:20:04.828154+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:17.724462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828818+0000","last_change":"2024-02-05T07:20:00.760332+0000","last_active":"2024-02-05T07:20:04.828818+0000","last_peered":"2024-02-05T07:20:04.828818+0000","last_clean":"2024-02-05T07:20:04.828818+0000","last_became_active":"2024-02-05T07:18:55.505678+0000","last_became_peered":"2024-02-05T07:18:55.505678+0000","last_unstale":"2024-02-05T07:20:04.828818+0000","last_undegraded":"2024-02-05T07:20:04.828818+0000","last_fullsized":"2024-02-05T07:20:04.828818+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:00.760230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.760230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:19.923028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.104093+0000","last_change":"2024-02-05T07:18:53.461703+0000","last_active":"2024-02-05T07:20:04.104093+0000","last_peered":"2024-02-05T07:20:04.104093+0000","last_clean":"2024-02-05T07:20:04.104093+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:20:04.104093+0000","last_undegraded":"2024-02-05T07:20:04.104093+0000","last_fullsized":"2024-02-05T07:20:04.104093+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105004+0000","last_change":"2024-02-05T07:18:49.432729+0000","last_active":"2024-02-05T07:20:04.105004+0000","last_peered":"2024-02-05T07:20:04.105004+0000","last_clean":"2024-02-05T07:20:04.105004+0000","last_became_active":"2024-02-05T07:18:49.432449+0000","last_became_peered":"2024-02-05T07:18:49.432449+0000","last_unstale":"2024-02-05T07:20:04.105004+0000","last_undegraded":"2024-02-05T07:20:04.105004+0000","last_fullsized":"2024-02-05T07:20:04.105004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:17.889196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.628904+0000","last_change":"2024-02-05T07:19:59.624459+0000","last_active":"2024-02-05T07:20:04.628904+0000","last_peered":"2024-02-05T07:20:04.628904+0000","last_clean":"2024-02-05T07:20:04.628904+0000","last_became_active":"2024-02-05T07:18:51.447728+0000","last_became_peered":"2024-02-05T07:18:51.447728+0000","last_unstale":"2024-02-05T07:20:04.628904+0000","last_undegraded":"2024-02-05T07:20:04.628904+0000","last_fullsized":"2024-02-05T07:20:04.628904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:19:59.624369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:59.624369+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:28.126457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826616+0000","last_change":"2024-02-05T07:18:55.506354+0000","last_active":"2024-02-05T07:20:04.826616+0000","last_peered":"2024-02-05T07:20:04.826616+0000","last_clean":"2024-02-05T07:20:04.826616+0000","last_became_active":"2024-02-05T07:18:55.505375+0000","last_became_peered":"2024-02-05T07:18:55.505375+0000","last_unstale":"2024-02-05T07:20:04.826616+0000","last_undegraded":"2024-02-05T07:20:04.826616+0000","last_fullsized":"2024-02-05T07:20:04.826616+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.462745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828427+0000","last_change":"2024-02-05T07:18:53.459361+0000","last_active":"2024-02-05T07:20:04.828427+0000","last_peered":"2024-02-05T07:20:04.828427+0000","last_clean":"2024-02-05T07:20:04.828427+0000","last_became_active":"2024-02-05T07:18:53.459077+0000","last_became_peered":"2024-02-05T07:18:53.459077+0000","last_unstale":"2024-02-05T07:20:04.828427+0000","last_undegraded":"2024-02-05T07:20:04.828427+0000","last_fullsized":"2024-02-05T07:20:04.828427+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.124069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828252+0000","last_change":"2024-02-05T07:18:49.428575+0000","last_active":"2024-02-05T07:20:04.828252+0000","last_peered":"2024-02-05T07:20:04.828252+0000","last_clean":"2024-02-05T07:20:04.828252+0000","last_became_active":"2024-02-05T07:18:49.427837+0000","last_became_peered":"2024-02-05T07:18:49.427837+0000","last_unstale":"2024-02-05T07:20:04.828252+0000","last_undegraded":"2024-02-05T07:20:04.828252+0000","last_fullsized":"2024-02-05T07:20:04.828252+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:15.656514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826104+0000","last_change":"2024-02-05T07:20:01.841591+0000","last_active":"2024-02-05T07:20:04.826104+0000","last_peered":"2024-02-05T07:20:04.826104+0000","last_clean":"2024-02-05T07:20:04.826104+0000","last_became_active":"2024-02-05T07:18:51.447761+0000","last_became_peered":"2024-02-05T07:18:51.447761+0000","last_unstale":"2024-02-05T07:20:04.826104+0000","last_undegraded":"2024-02-05T07:20:04.826104+0000","last_fullsized":"2024-02-05T07:20:04.826104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:01.841489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:01.841489+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:19.277134+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806379+0000","last_change":"2024-02-05T07:18:55.476374+0000","last_active":"2024-02-05T07:20:04.806379+0000","last_peered":"2024-02-05T07:20:04.806379+0000","last_clean":"2024-02-05T07:20:04.806379+0000","last_became_active":"2024-02-05T07:18:55.476071+0000","last_became_peered":"2024-02-05T07:18:55.476071+0000","last_unstale":"2024-02-05T07:20:04.806379+0000","last_undegraded":"2024-02-05T07:20:04.806379+0000","last_fullsized":"2024-02-05T07:20:04.806379+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.860100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.301356+0000","last_change":"2024-02-05T07:18:53.461297+0000","last_active":"2024-02-05T07:20:04.301356+0000","last_peered":"2024-02-05T07:20:04.301356+0000","last_clean":"2024-02-05T07:20:04.301356+0000","last_became_active":"2024-02-05T07:18:53.461011+0000","last_became_peered":"2024-02-05T07:18:53.461011+0000","last_unstale":"2024-02-05T07:20:04.301356+0000","last_undegraded":"2024-02-05T07:20:04.301356+0000","last_fullsized":"2024-02-05T07:20:04.301356+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:14.390445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103048+0000","last_change":"2024-02-05T07:18:49.431334+0000","last_active":"2024-02-05T07:20:04.103048+0000","last_peered":"2024-02-05T07:20:04.103048+0000","last_clean":"2024-02-05T07:20:04.103048+0000","last_became_active":"2024-02-05T07:18:49.431031+0000","last_became_peered":"2024-02-05T07:18:49.431031+0000","last_unstale":"2024-02-05T07:20:04.103048+0000","last_undegraded":"2024-02-05T07:20:04.103048+0000","last_fullsized":"2024-02-05T07:20:04.103048+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:10.104803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105230+0000","last_change":"2024-02-05T07:18:51.510293+0000","last_active":"2024-02-05T07:20:04.105230+0000","last_peered":"2024-02-05T07:20:04.105230+0000","last_clean":"2024-02-05T07:20:04.105230+0000","last_became_active":"2024-02-05T07:18:51.509918+0000","last_became_peered":"2024-02-05T07:18:51.509918+0000","last_unstale":"2024-02-05T07:20:04.105230+0000","last_undegraded":"2024-02-05T07:20:04.105230+0000","last_fullsized":"2024-02-05T07:20:04.105230+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.104275+0000","last_change":"2024-02-05T07:20:03.444274+0000","last_active":"2024-02-05T07:20:04.104275+0000","last_peered":"2024-02-05T07:20:04.104275+0000","last_clean":"2024-02-05T07:20:04.104275+0000","last_became_active":"2024-02-05T07:18:55.466164+0000","last_became_peered":"2024-02-05T07:18:55.466164+0000","last_unstale":"2024-02-05T07:20:04.104275+0000","last_undegraded":"2024-02-05T07:20:04.104275+0000","last_fullsized":"2024-02-05T07:20:04.104275+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:03.444153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:03.444153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:32.621342+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105378+0000","last_change":"2024-02-05T07:18:53.457928+0000","last_active":"2024-02-05T07:20:04.105378+0000","last_peered":"2024-02-05T07:20:04.105378+0000","last_clean":"2024-02-05T07:20:04.105378+0000","last_became_active":"2024-02-05T07:18:53.457587+0000","last_became_peered":"2024-02-05T07:18:53.457587+0000","last_unstale":"2024-02-05T07:20:04.105378+0000","last_undegraded":"2024-02-05T07:20:04.105378+0000","last_fullsized":"2024-02-05T07:20:04.105378+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:14.500310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825630+0000","last_change":"2024-02-05T07:19:49.788355+0000","last_active":"2024-02-05T07:20:04.825630+0000","last_peered":"2024-02-05T07:20:04.825630+0000","last_clean":"2024-02-05T07:20:04.825630+0000","last_became_active":"2024-02-05T07:18:49.430988+0000","last_became_peered":"2024-02-05T07:18:49.430988+0000","last_unstale":"2024-02-05T07:20:04.825630+0000","last_undegraded":"2024-02-05T07:20:04.825630+0000","last_fullsized":"2024-02-05T07:20:04.825630+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:49.788275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:49.788275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:50.100908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.301318+0000","last_change":"2024-02-05T07:18:51.446975+0000","last_active":"2024-02-05T07:20:04.301318+0000","last_peered":"2024-02-05T07:20:04.301318+0000","last_clean":"2024-02-05T07:20:04.301318+0000","last_became_active":"2024-02-05T07:18:51.446604+0000","last_became_peered":"2024-02-05T07:18:51.446604+0000","last_unstale":"2024-02-05T07:20:04.301318+0000","last_undegraded":"2024-02-05T07:20:04.301318+0000","last_fullsized":"2024-02-05T07:20:04.301318+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.258988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.805981+0000","last_change":"2024-02-05T07:20:01.693975+0000","last_active":"2024-02-05T07:20:04.805981+0000","last_peered":"2024-02-05T07:20:04.805981+0000","last_clean":"2024-02-05T07:20:04.805981+0000","last_became_active":"2024-02-05T07:18:55.468086+0000","last_became_peered":"2024-02-05T07:18:55.468086+0000","last_unstale":"2024-02-05T07:20:04.805981+0000","last_undegraded":"2024-02-05T07:20:04.805981+0000","last_fullsized":"2024-02-05T07:20:04.805981+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:01.693871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:01.693871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:12.004988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.300523+0000","last_change":"2024-02-05T07:18:53.460572+0000","last_active":"2024-02-05T07:20:04.300523+0000","last_peered":"2024-02-05T07:20:04.300523+0000","last_clean":"2024-02-05T07:20:04.300523+0000","last_became_active":"2024-02-05T07:18:53.460307+0000","last_became_peered":"2024-02-05T07:18:53.460307+0000","last_unstale":"2024-02-05T07:20:04.300523+0000","last_undegraded":"2024-02-05T07:20:04.300523+0000","last_fullsized":"2024-02-05T07:20:04.300523+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:08.545644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105269+0000","last_change":"2024-02-05T07:18:49.434712+0000","last_active":"2024-02-05T07:20:04.105269+0000","last_peered":"2024-02-05T07:20:04.105269+0000","last_clean":"2024-02-05T07:20:04.105269+0000","last_became_active":"2024-02-05T07:18:49.434408+0000","last_became_peered":"2024-02-05T07:18:49.434408+0000","last_unstale":"2024-02-05T07:20:04.105269+0000","last_undegraded":"2024-02-05T07:20:04.105269+0000","last_fullsized":"2024-02-05T07:20:04.105269+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825638+0000","last_change":"2024-02-05T07:18:51.449489+0000","last_active":"2024-02-05T07:20:04.825638+0000","last_peered":"2024-02-05T07:20:04.825638+0000","last_clean":"2024-02-05T07:20:04.825638+0000","last_became_active":"2024-02-05T07:18:51.449083+0000","last_became_peered":"2024-02-05T07:18:51.449083+0000","last_unstale":"2024-02-05T07:20:04.825638+0000","last_undegraded":"2024-02-05T07:20:04.825638+0000","last_fullsized":"2024-02-05T07:20:04.825638+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825873+0000","last_change":"2024-02-05T07:18:55.507549+0000","last_active":"2024-02-05T07:20:04.825873+0000","last_peered":"2024-02-05T07:20:04.825873+0000","last_clean":"2024-02-05T07:20:04.825873+0000","last_became_active":"2024-02-05T07:18:55.506968+0000","last_became_peered":"2024-02-05T07:18:55.506968+0000","last_unstale":"2024-02-05T07:20:04.825873+0000","last_undegraded":"2024-02-05T07:20:04.825873+0000","last_fullsized":"2024-02-05T07:20:04.825873+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806249+0000","last_change":"2024-02-05T07:18:53.461952+0000","last_active":"2024-02-05T07:20:04.806249+0000","last_peered":"2024-02-05T07:20:04.806249+0000","last_clean":"2024-02-05T07:20:04.806249+0000","last_became_active":"2024-02-05T07:18:53.460977+0000","last_became_peered":"2024-02-05T07:18:53.460977+0000","last_unstale":"2024-02-05T07:20:04.806249+0000","last_undegraded":"2024-02-05T07:20:04.806249+0000","last_fullsized":"2024-02-05T07:20:04.806249+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106407+0000","last_change":"2024-02-05T07:19:48.944128+0000","last_active":"2024-02-05T07:20:04.106407+0000","last_peered":"2024-02-05T07:20:04.106407+0000","last_clean":"2024-02-05T07:20:04.106407+0000","last_became_active":"2024-02-05T07:18:49.434560+0000","last_became_peered":"2024-02-05T07:18:49.434560+0000","last_unstale":"2024-02-05T07:20:04.106407+0000","last_undegraded":"2024-02-05T07:20:04.106407+0000","last_fullsized":"2024-02-05T07:20:04.106407+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:48.944028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:48.944028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:56.156526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825839+0000","last_change":"2024-02-05T07:18:51.510217+0000","last_active":"2024-02-05T07:20:04.825839+0000","last_peered":"2024-02-05T07:20:04.825839+0000","last_clean":"2024-02-05T07:20:04.825839+0000","last_became_active":"2024-02-05T07:18:51.509911+0000","last_became_peered":"2024-02-05T07:18:51.509911+0000","last_unstale":"2024-02-05T07:20:04.825839+0000","last_undegraded":"2024-02-05T07:20:04.825839+0000","last_fullsized":"2024-02-05T07:20:04.825839+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.102411+0000","last_change":"2024-02-05T07:18:55.477626+0000","last_active":"2024-02-05T07:20:04.102411+0000","last_peered":"2024-02-05T07:20:04.102411+0000","last_clean":"2024-02-05T07:20:04.102411+0000","last_became_active":"2024-02-05T07:18:55.477312+0000","last_became_peered":"2024-02-05T07:18:55.477312+0000","last_unstale":"2024-02-05T07:20:04.102411+0000","last_undegraded":"2024-02-05T07:20:04.102411+0000","last_fullsized":"2024-02-05T07:20:04.102411+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.934928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826060+0000","last_change":"2024-02-05T07:18:53.468306+0000","last_active":"2024-02-05T07:20:04.826060+0000","last_peered":"2024-02-05T07:20:04.826060+0000","last_clean":"2024-02-05T07:20:04.826060+0000","last_became_active":"2024-02-05T07:18:53.467986+0000","last_became_peered":"2024-02-05T07:18:53.467986+0000","last_unstale":"2024-02-05T07:20:04.826060+0000","last_undegraded":"2024-02-05T07:20:04.826060+0000","last_fullsized":"2024-02-05T07:20:04.826060+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:21.663256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806406+0000","last_change":"2024-02-05T07:18:49.438840+0000","last_active":"2024-02-05T07:20:04.806406+0000","last_peered":"2024-02-05T07:20:04.806406+0000","last_clean":"2024-02-05T07:20:04.806406+0000","last_became_active":"2024-02-05T07:18:49.438565+0000","last_became_peered":"2024-02-05T07:18:49.438565+0000","last_unstale":"2024-02-05T07:20:04.806406+0000","last_undegraded":"2024-02-05T07:20:04.806406+0000","last_fullsized":"2024-02-05T07:20:04.806406+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:15.720884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105721+0000","last_change":"2024-02-05T07:19:16.128305+0000","last_active":"2024-02-05T07:20:04.105721+0000","last_peered":"2024-02-05T07:20:04.105721+0000","last_clean":"2024-02-05T07:20:04.105721+0000","last_became_active":"2024-02-05T07:19:16.122059+0000","last_became_peered":"2024-02-05T07:19:16.122059+0000","last_unstale":"2024-02-05T07:20:04.105721+0000","last_undegraded":"2024-02-05T07:20:04.105721+0000","last_fullsized":"2024-02-05T07:20:04.105721+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:16.076653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825624+0000","last_change":"2024-02-05T07:18:55.478367+0000","last_active":"2024-02-05T07:20:04.825624+0000","last_peered":"2024-02-05T07:20:04.825624+0000","last_clean":"2024-02-05T07:20:04.825624+0000","last_became_active":"2024-02-05T07:18:55.478087+0000","last_became_peered":"2024-02-05T07:18:55.478087+0000","last_unstale":"2024-02-05T07:20:04.825624+0000","last_undegraded":"2024-02-05T07:20:04.825624+0000","last_fullsized":"2024-02-05T07:20:04.825624+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.104389+0000","last_change":"2024-02-05T07:18:53.463942+0000","last_active":"2024-02-05T07:20:04.104389+0000","last_peered":"2024-02-05T07:20:04.104389+0000","last_clean":"2024-02-05T07:20:04.104389+0000","last_became_active":"2024-02-05T07:18:53.463620+0000","last_became_peered":"2024-02-05T07:18:53.463620+0000","last_unstale":"2024-02-05T07:20:04.104389+0000","last_undegraded":"2024-02-05T07:20:04.104389+0000","last_fullsized":"2024-02-05T07:20:04.104389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.598964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828650+0000","last_change":"2024-02-05T07:18:49.431084+0000","last_active":"2024-02-05T07:20:04.828650+0000","last_peered":"2024-02-05T07:20:04.828650+0000","last_clean":"2024-02-05T07:20:04.828650+0000","last_became_active":"2024-02-05T07:18:49.430766+0000","last_became_peered":"2024-02-05T07:18:49.430766+0000","last_unstale":"2024-02-05T07:20:04.828650+0000","last_undegraded":"2024-02-05T07:20:04.828650+0000","last_fullsized":"2024-02-05T07:20:04.828650+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:14.154245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":87,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828419+0000","last_change":"2024-02-05T07:19:16.129619+0000","last_active":"2024-02-05T07:20:04.828419+0000","last_peered":"2024-02-05T07:20:04.828419+0000","last_clean":"2024-02-05T07:20:04.828419+0000","last_became_active":"2024-02-05T07:19:16.121696+0000","last_became_peered":"2024-02-05T07:19:16.121696+0000","last_unstale":"2024-02-05T07:20:04.828419+0000","last_undegraded":"2024-02-05T07:20:04.828419+0000","last_fullsized":"2024-02-05T07:20:04.828419+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103711+0000","last_change":"2024-02-05T07:18:55.478247+0000","last_active":"2024-02-05T07:20:04.103711+0000","last_peered":"2024-02-05T07:20:04.103711+0000","last_clean":"2024-02-05T07:20:04.103711+0000","last_became_active":"2024-02-05T07:18:55.477896+0000","last_became_peered":"2024-02-05T07:18:55.477896+0000","last_unstale":"2024-02-05T07:20:04.103711+0000","last_undegraded":"2024-02-05T07:20:04.103711+0000","last_fullsized":"2024-02-05T07:20:04.103711+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106289+0000","last_change":"2024-02-05T07:18:53.465570+0000","last_active":"2024-02-05T07:20:04.106289+0000","last_peered":"2024-02-05T07:20:04.106289+0000","last_clean":"2024-02-05T07:20:04.106289+0000","last_became_active":"2024-02-05T07:18:53.465287+0000","last_became_peered":"2024-02-05T07:18:53.465287+0000","last_unstale":"2024-02-05T07:20:04.106289+0000","last_undegraded":"2024-02-05T07:20:04.106289+0000","last_fullsized":"2024-02-05T07:20:04.106289+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:13.807259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806132+0000","last_change":"2024-02-05T07:18:49.436416+0000","last_active":"2024-02-05T07:20:04.806132+0000","last_peered":"2024-02-05T07:20:04.806132+0000","last_clean":"2024-02-05T07:20:04.806132+0000","last_became_active":"2024-02-05T07:18:49.436131+0000","last_became_peered":"2024-02-05T07:18:49.436131+0000","last_unstale":"2024-02-05T07:20:04.806132+0000","last_undegraded":"2024-02-05T07:20:04.806132+0000","last_fullsized":"2024-02-05T07:20:04.806132+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:08.178865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826559+0000","last_change":"2024-02-05T07:18:51.507793+0000","last_active":"2024-02-05T07:20:04.826559+0000","last_peered":"2024-02-05T07:20:04.826559+0000","last_clean":"2024-02-05T07:20:04.826559+0000","last_became_active":"2024-02-05T07:18:51.507491+0000","last_became_peered":"2024-02-05T07:18:51.507491+0000","last_unstale":"2024-02-05T07:20:04.826559+0000","last_undegraded":"2024-02-05T07:20:04.826559+0000","last_fullsized":"2024-02-05T07:20:04.826559+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.940131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828875+0000","last_change":"2024-02-05T07:18:55.467356+0000","last_active":"2024-02-05T07:20:04.828875+0000","last_peered":"2024-02-05T07:20:04.828875+0000","last_clean":"2024-02-05T07:20:04.828875+0000","last_became_active":"2024-02-05T07:18:55.467044+0000","last_became_peered":"2024-02-05T07:18:55.467044+0000","last_unstale":"2024-02-05T07:20:04.828875+0000","last_undegraded":"2024-02-05T07:20:04.828875+0000","last_fullsized":"2024-02-05T07:20:04.828875+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:12.322358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826261+0000","last_change":"2024-02-05T07:18:53.465551+0000","last_active":"2024-02-05T07:20:04.826261+0000","last_peered":"2024-02-05T07:20:04.826261+0000","last_clean":"2024-02-05T07:20:04.826261+0000","last_became_active":"2024-02-05T07:18:53.465270+0000","last_became_peered":"2024-02-05T07:18:53.465270+0000","last_unstale":"2024-02-05T07:20:04.826261+0000","last_undegraded":"2024-02-05T07:20:04.826261+0000","last_fullsized":"2024-02-05T07:20:04.826261+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:15.539033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103163+0000","last_change":"2024-02-05T07:18:49.435653+0000","last_active":"2024-02-05T07:20:04.103163+0000","last_peered":"2024-02-05T07:20:04.103163+0000","last_clean":"2024-02-05T07:20:04.103163+0000","last_became_active":"2024-02-05T07:18:49.434402+0000","last_became_peered":"2024-02-05T07:18:49.434402+0000","last_unstale":"2024-02-05T07:20:04.103163+0000","last_undegraded":"2024-02-05T07:20:04.103163+0000","last_fullsized":"2024-02-05T07:20:04.103163+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.792373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106398+0000","last_change":"2024-02-05T07:18:51.508944+0000","last_active":"2024-02-05T07:20:04.106398+0000","last_peered":"2024-02-05T07:20:04.106398+0000","last_clean":"2024-02-05T07:20:04.106398+0000","last_became_active":"2024-02-05T07:18:51.508551+0000","last_became_peered":"2024-02-05T07:18:51.508551+0000","last_unstale":"2024-02-05T07:20:04.106398+0000","last_undegraded":"2024-02-05T07:20:04.106398+0000","last_fullsized":"2024-02-05T07:20:04.106398+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.629037+0000","last_change":"2024-02-05T07:18:55.469355+0000","last_active":"2024-02-05T07:20:04.629037+0000","last_peered":"2024-02-05T07:20:04.629037+0000","last_clean":"2024-02-05T07:20:04.629037+0000","last_became_active":"2024-02-05T07:18:55.469044+0000","last_became_peered":"2024-02-05T07:18:55.469044+0000","last_unstale":"2024-02-05T07:20:04.629037+0000","last_undegraded":"2024-02-05T07:20:04.629037+0000","last_fullsized":"2024-02-05T07:20:04.629037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:10.874735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826572+0000","last_change":"2024-02-05T07:20:02.813432+0000","last_active":"2024-02-05T07:20:04.826572+0000","last_peered":"2024-02-05T07:20:04.826572+0000","last_clean":"2024-02-05T07:20:04.826572+0000","last_became_active":"2024-02-05T07:18:53.463607+0000","last_became_peered":"2024-02-05T07:18:53.463607+0000","last_unstale":"2024-02-05T07:20:04.826572+0000","last_undegraded":"2024-02-05T07:20:04.826572+0000","last_fullsized":"2024-02-05T07:20:04.826572+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.813344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.813344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:30.005634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806163+0000","last_change":"2024-02-05T07:20:02.673707+0000","last_active":"2024-02-05T07:20:04.806163+0000","last_peered":"2024-02-05T07:20:04.806163+0000","last_clean":"2024-02-05T07:20:04.806163+0000","last_became_active":"2024-02-05T07:18:49.437732+0000","last_became_peered":"2024-02-05T07:18:49.437732+0000","last_unstale":"2024-02-05T07:20:04.806163+0000","last_undegraded":"2024-02-05T07:20:04.806163+0000","last_fullsized":"2024-02-05T07:20:04.806163+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.673592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.665799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105276+0000","last_change":"2024-02-05T07:18:51.510254+0000","last_active":"2024-02-05T07:20:04.105276+0000","last_peered":"2024-02-05T07:20:04.105276+0000","last_clean":"2024-02-05T07:20:04.105276+0000","last_became_active":"2024-02-05T07:18:51.509917+0000","last_became_peered":"2024-02-05T07:18:51.509917+0000","last_unstale":"2024-02-05T07:20:04.105276+0000","last_undegraded":"2024-02-05T07:20:04.105276+0000","last_fullsized":"2024-02-05T07:20:04.105276+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:17.917819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806435+0000","last_change":"2024-02-05T07:18:55.507824+0000","last_active":"2024-02-05T07:20:04.806435+0000","last_peered":"2024-02-05T07:20:04.806435+0000","last_clean":"2024-02-05T07:20:04.806435+0000","last_became_active":"2024-02-05T07:18:55.507555+0000","last_became_peered":"2024-02-05T07:18:55.507555+0000","last_unstale":"2024-02-05T07:20:04.806435+0000","last_undegraded":"2024-02-05T07:20:04.806435+0000","last_fullsized":"2024-02-05T07:20:04.806435+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.017979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106495+0000","last_change":"2024-02-05T07:18:53.465851+0000","last_active":"2024-02-05T07:20:04.106495+0000","last_peered":"2024-02-05T07:20:04.106495+0000","last_clean":"2024-02-05T07:20:04.106495+0000","last_became_active":"2024-02-05T07:18:53.465591+0000","last_became_peered":"2024-02-05T07:18:53.465591+0000","last_unstale":"2024-02-05T07:20:04.106495+0000","last_undegraded":"2024-02-05T07:20:04.106495+0000","last_fullsized":"2024-02-05T07:20:04.106495+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:16.779901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.300528+0000","last_change":"2024-02-05T07:19:55.369939+0000","last_active":"2024-02-05T07:20:04.300528+0000","last_peered":"2024-02-05T07:20:04.300528+0000","last_clean":"2024-02-05T07:20:04.300528+0000","last_became_active":"2024-02-05T07:18:49.433121+0000","last_became_peered":"2024-02-05T07:18:49.433121+0000","last_unstale":"2024-02-05T07:20:04.300528+0000","last_undegraded":"2024-02-05T07:20:04.300528+0000","last_fullsized":"2024-02-05T07:20:04.300528+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:19:55.369800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:55.369800+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:04.984273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826639+0000","last_change":"2024-02-05T07:18:51.508489+0000","last_active":"2024-02-05T07:20:04.826639+0000","last_peered":"2024-02-05T07:20:04.826639+0000","last_clean":"2024-02-05T07:20:04.826639+0000","last_became_active":"2024-02-05T07:18:51.507451+0000","last_became_peered":"2024-02-05T07:18:51.507451+0000","last_unstale":"2024-02-05T07:20:04.826639+0000","last_undegraded":"2024-02-05T07:20:04.826639+0000","last_fullsized":"2024-02-05T07:20:04.826639+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:18.887208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106057+0000","last_change":"2024-02-05T07:18:55.505712+0000","last_active":"2024-02-05T07:20:04.106057+0000","last_peered":"2024-02-05T07:20:04.106057+0000","last_clean":"2024-02-05T07:20:04.106057+0000","last_became_active":"2024-02-05T07:18:55.505401+0000","last_became_peered":"2024-02-05T07:18:55.505401+0000","last_unstale":"2024-02-05T07:20:04.106057+0000","last_undegraded":"2024-02-05T07:20:04.106057+0000","last_fullsized":"2024-02-05T07:20:04.106057+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:13.385778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806823+0000","last_change":"2024-02-05T07:18:49.437997+0000","last_active":"2024-02-05T07:20:04.806823+0000","last_peered":"2024-02-05T07:20:04.806823+0000","last_clean":"2024-02-05T07:20:04.806823+0000","last_became_active":"2024-02-05T07:18:49.437663+0000","last_became_peered":"2024-02-05T07:18:49.437663+0000","last_unstale":"2024-02-05T07:20:04.806823+0000","last_undegraded":"2024-02-05T07:20:04.806823+0000","last_fullsized":"2024-02-05T07:20:04.806823+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.060603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105633+0000","last_change":"2024-02-05T07:19:58.753612+0000","last_active":"2024-02-05T07:20:04.105633+0000","last_peered":"2024-02-05T07:20:04.105633+0000","last_clean":"2024-02-05T07:20:04.105633+0000","last_became_active":"2024-02-05T07:18:51.508097+0000","last_became_peered":"2024-02-05T07:18:51.508097+0000","last_unstale":"2024-02-05T07:20:04.105633+0000","last_undegraded":"2024-02-05T07:20:04.105633+0000","last_fullsized":"2024-02-05T07:20:04.105633+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:19:58.753472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:58.753472+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:26.081872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806325+0000","last_change":"2024-02-05T07:20:00.685371+0000","last_active":"2024-02-05T07:20:04.806325+0000","last_peered":"2024-02-05T07:20:04.806325+0000","last_clean":"2024-02-05T07:20:04.806325+0000","last_became_active":"2024-02-05T07:18:53.456287+0000","last_became_peered":"2024-02-05T07:18:53.456287+0000","last_unstale":"2024-02-05T07:20:04.806325+0000","last_undegraded":"2024-02-05T07:20:04.806325+0000","last_fullsized":"2024-02-05T07:20:04.806325+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.685282+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:11.368545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825965+0000","last_change":"2024-02-05T07:18:55.474988+0000","last_active":"2024-02-05T07:20:04.825965+0000","last_peered":"2024-02-05T07:20:04.825965+0000","last_clean":"2024-02-05T07:20:04.825965+0000","last_became_active":"2024-02-05T07:18:55.474663+0000","last_became_peered":"2024-02-05T07:18:55.474663+0000","last_unstale":"2024-02-05T07:20:04.825965+0000","last_undegraded":"2024-02-05T07:20:04.825965+0000","last_fullsized":"2024-02-05T07:20:04.825965+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806593+0000","last_change":"2024-02-05T07:19:51.758261+0000","last_active":"2024-02-05T07:20:04.806593+0000","last_peered":"2024-02-05T07:20:04.806593+0000","last_clean":"2024-02-05T07:20:04.806593+0000","last_became_active":"2024-02-05T07:18:49.435621+0000","last_became_peered":"2024-02-05T07:18:49.435621+0000","last_unstale":"2024-02-05T07:20:04.806593+0000","last_undegraded":"2024-02-05T07:20:04.806593+0000","last_fullsized":"2024-02-05T07:20:04.806593+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:51.758163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.758163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:15.549152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":99,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.885848+0000","last_change":"2024-02-05T07:20:04.885890+0000","last_active":"2024-02-05T07:20:04.885848+0000","last_peered":"2024-02-05T07:20:04.885848+0000","last_clean":"2024-02-05T07:20:04.885848+0000","last_became_active":"2024-02-05T07:19:16.123650+0000","last_became_peered":"2024-02-05T07:19:16.123650+0000","last_unstale":"2024-02-05T07:20:04.885848+0000","last_undegraded":"2024-02-05T07:20:04.885848+0000","last_fullsized":"2024-02-05T07:20:04.885848+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:04.885778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:04.885778+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:07.176995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.628647+0000","last_change":"2024-02-05T07:20:02.694603+0000","last_active":"2024-02-05T07:20:04.628647+0000","last_peered":"2024-02-05T07:20:04.628647+0000","last_clean":"2024-02-05T07:20:04.628647+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:20:04.628647+0000","last_undegraded":"2024-02-05T07:20:04.628647+0000","last_fullsized":"2024-02-05T07:20:04.628647+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.694506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.694506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.700377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.805796+0000","last_change":"2024-02-05T07:18:55.473523+0000","last_active":"2024-02-05T07:20:04.805796+0000","last_peered":"2024-02-05T07:20:04.805796+0000","last_clean":"2024-02-05T07:20:04.805796+0000","last_became_active":"2024-02-05T07:18:55.473032+0000","last_became_peered":"2024-02-05T07:18:55.473032+0000","last_unstale":"2024-02-05T07:20:04.805796+0000","last_undegraded":"2024-02-05T07:20:04.805796+0000","last_fullsized":"2024-02-05T07:20:04.805796+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:12.882475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.628877+0000","last_change":"2024-02-05T07:18:53.465189+0000","last_active":"2024-02-05T07:20:04.628877+0000","last_peered":"2024-02-05T07:20:04.628877+0000","last_clean":"2024-02-05T07:20:04.628877+0000","last_became_active":"2024-02-05T07:18:53.464867+0000","last_became_peered":"2024-02-05T07:18:53.464867+0000","last_unstale":"2024-02-05T07:20:04.628877+0000","last_undegraded":"2024-02-05T07:20:04.628877+0000","last_fullsized":"2024-02-05T07:20:04.628877+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:15.717864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806645+0000","last_change":"2024-02-05T07:19:55.788134+0000","last_active":"2024-02-05T07:20:04.806645+0000","last_peered":"2024-02-05T07:20:04.806645+0000","last_clean":"2024-02-05T07:20:04.806645+0000","last_became_active":"2024-02-05T07:18:49.434451+0000","last_became_peered":"2024-02-05T07:18:49.434451+0000","last_unstale":"2024-02-05T07:20:04.806645+0000","last_undegraded":"2024-02-05T07:20:04.806645+0000","last_fullsized":"2024-02-05T07:20:04.806645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:55.788005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:55.788005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.126482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.300887+0000","last_change":"2024-02-05T07:20:00.384918+0000","last_active":"2024-02-05T07:20:04.300887+0000","last_peered":"2024-02-05T07:20:04.300887+0000","last_clean":"2024-02-05T07:20:04.300887+0000","last_became_active":"2024-02-05T07:18:51.507672+0000","last_became_peered":"2024-02-05T07:18:51.507672+0000","last_unstale":"2024-02-05T07:20:04.300887+0000","last_undegraded":"2024-02-05T07:20:04.300887+0000","last_fullsized":"2024-02-05T07:20:04.300887+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:20:00.384813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.384813+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:12.230867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.102967+0000","last_change":"2024-02-05T07:18:55.476043+0000","last_active":"2024-02-05T07:20:04.102967+0000","last_peered":"2024-02-05T07:20:04.102967+0000","last_clean":"2024-02-05T07:20:04.102967+0000","last_became_active":"2024-02-05T07:18:55.475702+0000","last_became_peered":"2024-02-05T07:18:55.475702+0000","last_unstale":"2024-02-05T07:20:04.102967+0000","last_undegraded":"2024-02-05T07:20:04.102967+0000","last_fullsized":"2024-02-05T07:20:04.102967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:20.177474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.628811+0000","last_change":"2024-02-05T07:18:53.459884+0000","last_active":"2024-02-05T07:20:04.628811+0000","last_peered":"2024-02-05T07:20:04.628811+0000","last_clean":"2024-02-05T07:20:04.628811+0000","last_became_active":"2024-02-05T07:18:53.459560+0000","last_became_peered":"2024-02-05T07:18:53.459560+0000","last_unstale":"2024-02-05T07:20:04.628811+0000","last_undegraded":"2024-02-05T07:20:04.628811+0000","last_fullsized":"2024-02-05T07:20:04.628811+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.014402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105948+0000","last_change":"2024-02-05T07:18:49.434599+0000","last_active":"2024-02-05T07:20:04.105948+0000","last_peered":"2024-02-05T07:20:04.105948+0000","last_clean":"2024-02-05T07:20:04.105948+0000","last_became_active":"2024-02-05T07:18:49.434238+0000","last_became_peered":"2024-02-05T07:18:49.434238+0000","last_unstale":"2024-02-05T07:20:04.105948+0000","last_undegraded":"2024-02-05T07:20:04.105948+0000","last_fullsized":"2024-02-05T07:20:04.105948+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105194+0000","last_change":"2024-02-05T07:18:51.441099+0000","last_active":"2024-02-05T07:20:04.105194+0000","last_peered":"2024-02-05T07:20:04.105194+0000","last_clean":"2024-02-05T07:20:04.105194+0000","last_became_active":"2024-02-05T07:18:51.440744+0000","last_became_peered":"2024-02-05T07:18:51.440744+0000","last_unstale":"2024-02-05T07:20:04.105194+0000","last_undegraded":"2024-02-05T07:20:04.105194+0000","last_fullsized":"2024-02-05T07:20:04.105194+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:14.420348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826260+0000","last_change":"2024-02-05T07:18:55.476663+0000","last_active":"2024-02-05T07:20:04.826260+0000","last_peered":"2024-02-05T07:20:04.826260+0000","last_clean":"2024-02-05T07:20:04.826260+0000","last_became_active":"2024-02-05T07:18:55.476313+0000","last_became_peered":"2024-02-05T07:18:55.476313+0000","last_unstale":"2024-02-05T07:20:04.826260+0000","last_undegraded":"2024-02-05T07:20:04.826260+0000","last_fullsized":"2024-02-05T07:20:04.826260+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:20.640694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105845+0000","last_change":"2024-02-05T07:18:52.445271+0000","last_active":"2024-02-05T07:20:04.105845+0000","last_peered":"2024-02-05T07:20:04.105845+0000","last_clean":"2024-02-05T07:20:04.105845+0000","last_became_active":"2024-02-05T07:18:48.423032+0000","last_became_peered":"2024-02-05T07:18:48.423032+0000","last_unstale":"2024-02-05T07:20:04.105845+0000","last_undegraded":"2024-02-05T07:20:04.105845+0000","last_fullsized":"2024-02-05T07:20:04.105845+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:18:47.405257+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:14.430841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828156+0000","last_change":"2024-02-05T07:18:49.430550+0000","last_active":"2024-02-05T07:20:04.828156+0000","last_peered":"2024-02-05T07:20:04.828156+0000","last_clean":"2024-02-05T07:20:04.828156+0000","last_became_active":"2024-02-05T07:18:49.430208+0000","last_became_peered":"2024-02-05T07:18:49.430208+0000","last_unstale":"2024-02-05T07:20:04.828156+0000","last_undegraded":"2024-02-05T07:20:04.828156+0000","last_fullsized":"2024-02-05T07:20:04.828156+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:10.597988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":115,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103813+0000","last_change":"2024-02-05T07:18:51.442632+0000","last_active":"2024-02-05T07:20:04.103813+0000","last_peered":"2024-02-05T07:20:04.103813+0000","last_clean":"2024-02-05T07:20:04.103813+0000","last_became_active":"2024-02-05T07:18:51.442110+0000","last_became_peered":"2024-02-05T07:18:51.442110+0000","last_unstale":"2024-02-05T07:20:04.103813+0000","last_undegraded":"2024-02-05T07:20:04.103813+0000","last_fullsized":"2024-02-05T07:20:04.103813+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:08.345704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103912+0000","last_change":"2024-02-05T07:18:53.461030+0000","last_active":"2024-02-05T07:20:04.103912+0000","last_peered":"2024-02-05T07:20:04.103912+0000","last_clean":"2024-02-05T07:20:04.103912+0000","last_became_active":"2024-02-05T07:18:53.460715+0000","last_became_peered":"2024-02-05T07:18:53.460715+0000","last_unstale":"2024-02-05T07:20:04.103912+0000","last_undegraded":"2024-02-05T07:20:04.103912+0000","last_fullsized":"2024-02-05T07:20:04.103912+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.356767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806853+0000","last_change":"2024-02-05T07:18:55.507105+0000","last_active":"2024-02-05T07:20:04.806853+0000","last_peered":"2024-02-05T07:20:04.806853+0000","last_clean":"2024-02-05T07:20:04.806853+0000","last_became_active":"2024-02-05T07:18:55.506803+0000","last_became_peered":"2024-02-05T07:18:55.506803+0000","last_unstale":"2024-02-05T07:20:04.806853+0000","last_undegraded":"2024-02-05T07:20:04.806853+0000","last_fullsized":"2024-02-05T07:20:04.806853+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:05.049069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103259+0000","last_change":"2024-02-05T07:18:49.429953+0000","last_active":"2024-02-05T07:20:04.103259+0000","last_peered":"2024-02-05T07:20:04.103259+0000","last_clean":"2024-02-05T07:20:04.103259+0000","last_became_active":"2024-02-05T07:18:49.429651+0000","last_became_peered":"2024-02-05T07:18:49.429651+0000","last_unstale":"2024-02-05T07:20:04.103259+0000","last_undegraded":"2024-02-05T07:20:04.103259+0000","last_fullsized":"2024-02-05T07:20:04.103259+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.628755+0000","last_change":"2024-02-05T07:19:56.661601+0000","last_active":"2024-02-05T07:20:04.628755+0000","last_peered":"2024-02-05T07:20:04.628755+0000","last_clean":"2024-02-05T07:20:04.628755+0000","last_became_active":"2024-02-05T07:18:48.423639+0000","last_became_peered":"2024-02-05T07:18:48.423639+0000","last_unstale":"2024-02-05T07:20:04.628755+0000","last_undegraded":"2024-02-05T07:20:04.628755+0000","last_fullsized":"2024-02-05T07:20:04.628755+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:56.661490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:19:56.661490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.585463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.104456+0000","last_change":"2024-02-05T07:18:51.436432+0000","last_active":"2024-02-05T07:20:04.104456+0000","last_peered":"2024-02-05T07:20:04.104456+0000","last_clean":"2024-02-05T07:20:04.104456+0000","last_became_active":"2024-02-05T07:18:51.436097+0000","last_became_peered":"2024-02-05T07:18:51.436097+0000","last_unstale":"2024-02-05T07:20:04.104456+0000","last_undegraded":"2024-02-05T07:20:04.104456+0000","last_fullsized":"2024-02-05T07:20:04.104456+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:11.611640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.629033+0000","last_change":"2024-02-05T07:18:53.461134+0000","last_active":"2024-02-05T07:20:04.629033+0000","last_peered":"2024-02-05T07:20:04.629033+0000","last_clean":"2024-02-05T07:20:04.629033+0000","last_became_active":"2024-02-05T07:18:53.460771+0000","last_became_peered":"2024-02-05T07:18:53.460771+0000","last_unstale":"2024-02-05T07:20:04.629033+0000","last_undegraded":"2024-02-05T07:20:04.629033+0000","last_fullsized":"2024-02-05T07:20:04.629033+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:09.923504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106539+0000","last_change":"2024-02-05T07:18:55.473873+0000","last_active":"2024-02-05T07:20:04.106539+0000","last_peered":"2024-02-05T07:20:04.106539+0000","last_clean":"2024-02-05T07:20:04.106539+0000","last_became_active":"2024-02-05T07:18:55.473358+0000","last_became_peered":"2024-02-05T07:18:55.473358+0000","last_unstale":"2024-02-05T07:20:04.106539+0000","last_undegraded":"2024-02-05T07:20:04.106539+0000","last_fullsized":"2024-02-05T07:20:04.106539+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:09.714221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":311,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.805927+0000","last_change":"2024-02-05T07:19:05.594948+0000","last_active":"2024-02-05T07:20:04.805927+0000","last_peered":"2024-02-05T07:20:04.805927+0000","last_clean":"2024-02-05T07:20:04.805927+0000","last_became_active":"2024-02-05T07:18:39.444898+0000","last_became_peered":"2024-02-05T07:18:39.444898+0000","last_unstale":"2024-02-05T07:20:04.805927+0000","last_undegraded":"2024-02-05T07:20:04.805927+0000","last_fullsized":"2024-02-05T07:20:04.805927+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-02-05T07:19:05.594812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:19:05.594812+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.994559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826034+0000","last_change":"2024-02-05T07:18:49.439534+0000","last_active":"2024-02-05T07:20:04.826034+0000","last_peered":"2024-02-05T07:20:04.826034+0000","last_clean":"2024-02-05T07:20:04.826034+0000","last_became_active":"2024-02-05T07:18:49.439249+0000","last_became_peered":"2024-02-05T07:18:49.439249+0000","last_unstale":"2024-02-05T07:20:04.826034+0000","last_undegraded":"2024-02-05T07:20:04.826034+0000","last_fullsized":"2024-02-05T07:20:04.826034+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.301169+0000","last_change":"2024-02-05T07:18:51.508007+0000","last_active":"2024-02-05T07:20:04.301169+0000","last_peered":"2024-02-05T07:20:04.301169+0000","last_clean":"2024-02-05T07:20:04.301169+0000","last_became_active":"2024-02-05T07:18:51.507714+0000","last_became_peered":"2024-02-05T07:18:51.507714+0000","last_unstale":"2024-02-05T07:20:04.301169+0000","last_undegraded":"2024-02-05T07:20:04.301169+0000","last_fullsized":"2024-02-05T07:20:04.301169+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.846104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806605+0000","last_change":"2024-02-05T07:18:53.460344+0000","last_active":"2024-02-05T07:20:04.806605+0000","last_peered":"2024-02-05T07:20:04.806605+0000","last_clean":"2024-02-05T07:20:04.806605+0000","last_became_active":"2024-02-05T07:18:53.459971+0000","last_became_peered":"2024-02-05T07:18:53.459971+0000","last_unstale":"2024-02-05T07:20:04.806605+0000","last_undegraded":"2024-02-05T07:20:04.806605+0000","last_fullsized":"2024-02-05T07:20:04.806605+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:13.408456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103426+0000","last_change":"2024-02-05T07:18:55.473808+0000","last_active":"2024-02-05T07:20:04.103426+0000","last_peered":"2024-02-05T07:20:04.103426+0000","last_clean":"2024-02-05T07:20:04.103426+0000","last_became_active":"2024-02-05T07:18:55.473509+0000","last_became_peered":"2024-02-05T07:18:55.473509+0000","last_unstale":"2024-02-05T07:20:04.103426+0000","last_undegraded":"2024-02-05T07:20:04.103426+0000","last_fullsized":"2024-02-05T07:20:04.103426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:22.190977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103258+0000","last_change":"2024-02-05T07:18:49.434806+0000","last_active":"2024-02-05T07:20:04.103258+0000","last_peered":"2024-02-05T07:20:04.103258+0000","last_clean":"2024-02-05T07:20:04.103258+0000","last_became_active":"2024-02-05T07:18:49.434400+0000","last_became_peered":"2024-02-05T07:18:49.434400+0000","last_unstale":"2024-02-05T07:20:04.103258+0000","last_undegraded":"2024-02-05T07:20:04.103258+0000","last_fullsized":"2024-02-05T07:20:04.103258+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":134,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.805863+0000","last_change":"2024-02-05T07:18:52.450728+0000","last_active":"2024-02-05T07:20:04.805863+0000","last_peered":"2024-02-05T07:20:04.805863+0000","last_clean":"2024-02-05T07:20:04.805863+0000","last_became_active":"2024-02-05T07:18:48.424620+0000","last_became_peered":"2024-02-05T07:18:48.424620+0000","last_unstale":"2024-02-05T07:20:04.805863+0000","last_undegraded":"2024-02-05T07:20:04.805863+0000","last_fullsized":"2024-02-05T07:20:04.805863+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:18:47.405257+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:12.026892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":63,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103815+0000","last_change":"2024-02-05T07:18:51.442488+0000","last_active":"2024-02-05T07:20:04.103815+0000","last_peered":"2024-02-05T07:20:04.103815+0000","last_clean":"2024-02-05T07:20:04.103815+0000","last_became_active":"2024-02-05T07:18:51.442184+0000","last_became_peered":"2024-02-05T07:18:51.442184+0000","last_unstale":"2024-02-05T07:20:04.103815+0000","last_undegraded":"2024-02-05T07:20:04.103815+0000","last_fullsized":"2024-02-05T07:20:04.103815+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106719+0000","last_change":"2024-02-05T07:18:53.468712+0000","last_active":"2024-02-05T07:20:04.106719+0000","last_peered":"2024-02-05T07:20:04.106719+0000","last_clean":"2024-02-05T07:20:04.106719+0000","last_became_active":"2024-02-05T07:18:53.468512+0000","last_became_peered":"2024-02-05T07:18:53.468512+0000","last_unstale":"2024-02-05T07:20:04.106719+0000","last_undegraded":"2024-02-05T07:20:04.106719+0000","last_fullsized":"2024-02-05T07:20:04.106719+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.283544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.102894+0000","last_change":"2024-02-05T07:18:55.505742+0000","last_active":"2024-02-05T07:20:04.102894+0000","last_peered":"2024-02-05T07:20:04.102894+0000","last_clean":"2024-02-05T07:20:04.102894+0000","last_became_active":"2024-02-05T07:18:55.505360+0000","last_became_peered":"2024-02-05T07:18:55.505360+0000","last_unstale":"2024-02-05T07:20:04.102894+0000","last_undegraded":"2024-02-05T07:20:04.102894+0000","last_fullsized":"2024-02-05T07:20:04.102894+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103681+0000","last_change":"2024-02-05T07:18:49.428750+0000","last_active":"2024-02-05T07:20:04.103681+0000","last_peered":"2024-02-05T07:20:04.103681+0000","last_clean":"2024-02-05T07:20:04.103681+0000","last_became_active":"2024-02-05T07:18:49.428470+0000","last_became_peered":"2024-02-05T07:18:49.428470+0000","last_unstale":"2024-02-05T07:20:04.103681+0000","last_undegraded":"2024-02-05T07:20:04.103681+0000","last_fullsized":"2024-02-05T07:20:04.103681+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105124+0000","last_change":"2024-02-05T07:18:51.507836+0000","last_active":"2024-02-05T07:20:04.105124+0000","last_peered":"2024-02-05T07:20:04.105124+0000","last_clean":"2024-02-05T07:20:04.105124+0000","last_became_active":"2024-02-05T07:18:51.507507+0000","last_became_peered":"2024-02-05T07:18:51.507507+0000","last_unstale":"2024-02-05T07:20:04.105124+0000","last_undegraded":"2024-02-05T07:20:04.105124+0000","last_fullsized":"2024-02-05T07:20:04.105124+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.300896+0000","last_change":"2024-02-05T07:19:56.330173+0000","last_active":"2024-02-05T07:20:04.300896+0000","last_peered":"2024-02-05T07:20:04.300896+0000","last_clean":"2024-02-05T07:20:04.300896+0000","last_became_active":"2024-02-05T07:18:53.459474+0000","last_became_peered":"2024-02-05T07:18:53.459474+0000","last_unstale":"2024-02-05T07:20:04.300896+0000","last_undegraded":"2024-02-05T07:20:04.300896+0000","last_fullsized":"2024-02-05T07:20:04.300896+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:56.330068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:19:56.330068+0000","last_clean_scrub_stamp":"2024-02-05T07:19:56.330068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:22.066765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.102416+0000","last_change":"2024-02-05T07:18:55.474472+0000","last_active":"2024-02-05T07:20:04.102416+0000","last_peered":"2024-02-05T07:20:04.102416+0000","last_clean":"2024-02-05T07:20:04.102416+0000","last_became_active":"2024-02-05T07:18:55.474029+0000","last_became_peered":"2024-02-05T07:18:55.474029+0000","last_unstale":"2024-02-05T07:20:04.102416+0000","last_undegraded":"2024-02-05T07:20:04.102416+0000","last_fullsized":"2024-02-05T07:20:04.102416+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:16.669709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106371+0000","last_change":"2024-02-05T07:18:49.435665+0000","last_active":"2024-02-05T07:20:04.106371+0000","last_peered":"2024-02-05T07:20:04.106371+0000","last_clean":"2024-02-05T07:20:04.106371+0000","last_became_active":"2024-02-05T07:18:49.435299+0000","last_became_peered":"2024-02-05T07:18:49.435299+0000","last_unstale":"2024-02-05T07:20:04.106371+0000","last_undegraded":"2024-02-05T07:20:04.106371+0000","last_fullsized":"2024-02-05T07:20:04.106371+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.603669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.104480+0000","last_change":"2024-02-05T07:18:51.448399+0000","last_active":"2024-02-05T07:20:04.104480+0000","last_peered":"2024-02-05T07:20:04.104480+0000","last_clean":"2024-02-05T07:20:04.104480+0000","last_became_active":"2024-02-05T07:18:51.448044+0000","last_became_peered":"2024-02-05T07:18:51.448044+0000","last_unstale":"2024-02-05T07:20:04.104480+0000","last_undegraded":"2024-02-05T07:20:04.104480+0000","last_fullsized":"2024-02-05T07:20:04.104480+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.940142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105027+0000","last_change":"2024-02-05T07:18:53.463629+0000","last_active":"2024-02-05T07:20:04.105027+0000","last_peered":"2024-02-05T07:20:04.105027+0000","last_clean":"2024-02-05T07:20:04.105027+0000","last_became_active":"2024-02-05T07:18:53.462890+0000","last_became_peered":"2024-02-05T07:18:53.462890+0000","last_unstale":"2024-02-05T07:20:04.105027+0000","last_undegraded":"2024-02-05T07:20:04.105027+0000","last_fullsized":"2024-02-05T07:20:04.105027+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.827616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806343+0000","last_change":"2024-02-05T07:18:55.508071+0000","last_active":"2024-02-05T07:20:04.806343+0000","last_peered":"2024-02-05T07:20:04.806343+0000","last_clean":"2024-02-05T07:20:04.806343+0000","last_became_active":"2024-02-05T07:18:55.507791+0000","last_became_peered":"2024-02-05T07:18:55.507791+0000","last_unstale":"2024-02-05T07:20:04.806343+0000","last_undegraded":"2024-02-05T07:20:04.806343+0000","last_fullsized":"2024-02-05T07:20:04.806343+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:24.052431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105110+0000","last_change":"2024-02-05T07:19:51.364668+0000","last_active":"2024-02-05T07:20:04.105110+0000","last_peered":"2024-02-05T07:20:04.105110+0000","last_clean":"2024-02-05T07:20:04.105110+0000","last_became_active":"2024-02-05T07:18:49.430123+0000","last_became_peered":"2024-02-05T07:18:49.430123+0000","last_unstale":"2024-02-05T07:20:04.105110+0000","last_undegraded":"2024-02-05T07:20:04.105110+0000","last_fullsized":"2024-02-05T07:20:04.105110+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:19:51.364585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.364585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:57.656562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828254+0000","last_change":"2024-02-05T07:18:51.438840+0000","last_active":"2024-02-05T07:20:04.828254+0000","last_peered":"2024-02-05T07:20:04.828254+0000","last_clean":"2024-02-05T07:20:04.828254+0000","last_became_active":"2024-02-05T07:18:51.438507+0000","last_became_peered":"2024-02-05T07:18:51.438507+0000","last_unstale":"2024-02-05T07:20:04.828254+0000","last_undegraded":"2024-02-05T07:20:04.828254+0000","last_fullsized":"2024-02-05T07:20:04.828254+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:18.455164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825875+0000","last_change":"2024-02-05T07:18:53.464137+0000","last_active":"2024-02-05T07:20:04.825875+0000","last_peered":"2024-02-05T07:20:04.825875+0000","last_clean":"2024-02-05T07:20:04.825875+0000","last_became_active":"2024-02-05T07:18:53.463651+0000","last_became_peered":"2024-02-05T07:18:53.463651+0000","last_unstale":"2024-02-05T07:20:04.825875+0000","last_undegraded":"2024-02-05T07:20:04.825875+0000","last_fullsized":"2024-02-05T07:20:04.825875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:18.813121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828117+0000","last_change":"2024-02-05T07:18:55.474782+0000","last_active":"2024-02-05T07:20:04.828117+0000","last_peered":"2024-02-05T07:20:04.828117+0000","last_clean":"2024-02-05T07:20:04.828117+0000","last_became_active":"2024-02-05T07:18:55.474447+0000","last_became_peered":"2024-02-05T07:18:55.474447+0000","last_unstale":"2024-02-05T07:20:04.828117+0000","last_undegraded":"2024-02-05T07:20:04.828117+0000","last_fullsized":"2024-02-05T07:20:04.828117+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:09.902392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826459+0000","last_change":"2024-02-05T07:18:49.435071+0000","last_active":"2024-02-05T07:20:04.826459+0000","last_peered":"2024-02-05T07:20:04.826459+0000","last_clean":"2024-02-05T07:20:04.826459+0000","last_became_active":"2024-02-05T07:18:49.434719+0000","last_became_peered":"2024-02-05T07:18:49.434719+0000","last_unstale":"2024-02-05T07:20:04.826459+0000","last_undegraded":"2024-02-05T07:20:04.826459+0000","last_fullsized":"2024-02-05T07:20:04.826459+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.568029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825770+0000","last_change":"2024-02-05T07:20:00.797048+0000","last_active":"2024-02-05T07:20:04.825770+0000","last_peered":"2024-02-05T07:20:04.825770+0000","last_clean":"2024-02-05T07:20:04.825770+0000","last_became_active":"2024-02-05T07:18:51.507524+0000","last_became_peered":"2024-02-05T07:18:51.507524+0000","last_unstale":"2024-02-05T07:20:04.825770+0000","last_undegraded":"2024-02-05T07:20:04.825770+0000","last_fullsized":"2024-02-05T07:20:04.825770+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:20:00.796905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.796905+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:13.158158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828000+0000","last_change":"2024-02-05T07:18:53.461489+0000","last_active":"2024-02-05T07:20:04.828000+0000","last_peered":"2024-02-05T07:20:04.828000+0000","last_clean":"2024-02-05T07:20:04.828000+0000","last_became_active":"2024-02-05T07:18:53.461213+0000","last_became_peered":"2024-02-05T07:18:53.461213+0000","last_unstale":"2024-02-05T07:20:04.828000+0000","last_undegraded":"2024-02-05T07:20:04.828000+0000","last_fullsized":"2024-02-05T07:20:04.828000+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:08.272501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106085+0000","last_change":"2024-02-05T07:18:55.472546+0000","last_active":"2024-02-05T07:20:04.106085+0000","last_peered":"2024-02-05T07:20:04.106085+0000","last_clean":"2024-02-05T07:20:04.106085+0000","last_became_active":"2024-02-05T07:18:55.472172+0000","last_became_peered":"2024-02-05T07:18:55.472172+0000","last_unstale":"2024-02-05T07:20:04.106085+0000","last_undegraded":"2024-02-05T07:20:04.106085+0000","last_fullsized":"2024-02-05T07:20:04.106085+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828918+0000","last_change":"2024-02-05T07:18:53.460159+0000","last_active":"2024-02-05T07:20:04.828918+0000","last_peered":"2024-02-05T07:20:04.828918+0000","last_clean":"2024-02-05T07:20:04.828918+0000","last_became_active":"2024-02-05T07:18:53.459832+0000","last_became_peered":"2024-02-05T07:18:53.459832+0000","last_unstale":"2024-02-05T07:20:04.828918+0000","last_undegraded":"2024-02-05T07:20:04.828918+0000","last_fullsized":"2024-02-05T07:20:04.828918+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:17.337266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825761+0000","last_change":"2024-02-05T07:18:49.431779+0000","last_active":"2024-02-05T07:20:04.825761+0000","last_peered":"2024-02-05T07:20:04.825761+0000","last_clean":"2024-02-05T07:20:04.825761+0000","last_became_active":"2024-02-05T07:18:49.431369+0000","last_became_peered":"2024-02-05T07:18:49.431369+0000","last_unstale":"2024-02-05T07:20:04.825761+0000","last_undegraded":"2024-02-05T07:20:04.825761+0000","last_fullsized":"2024-02-05T07:20:04.825761+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.014547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.104621+0000","last_change":"2024-02-05T07:18:51.447767+0000","last_active":"2024-02-05T07:20:04.104621+0000","last_peered":"2024-02-05T07:20:04.104621+0000","last_clean":"2024-02-05T07:20:04.104621+0000","last_became_active":"2024-02-05T07:18:51.447469+0000","last_became_peered":"2024-02-05T07:18:51.447469+0000","last_unstale":"2024-02-05T07:20:04.104621+0000","last_undegraded":"2024-02-05T07:20:04.104621+0000","last_fullsized":"2024-02-05T07:20:04.104621+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.124394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826160+0000","last_change":"2024-02-05T07:18:55.506254+0000","last_active":"2024-02-05T07:20:04.826160+0000","last_peered":"2024-02-05T07:20:04.826160+0000","last_clean":"2024-02-05T07:20:04.826160+0000","last_became_active":"2024-02-05T07:18:55.505340+0000","last_became_peered":"2024-02-05T07:18:55.505340+0000","last_unstale":"2024-02-05T07:20:04.826160+0000","last_undegraded":"2024-02-05T07:20:04.826160+0000","last_fullsized":"2024-02-05T07:20:04.826160+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106708+0000","last_change":"2024-02-05T07:18:53.462879+0000","last_active":"2024-02-05T07:20:04.106708+0000","last_peered":"2024-02-05T07:20:04.106708+0000","last_clean":"2024-02-05T07:20:04.106708+0000","last_became_active":"2024-02-05T07:18:53.462591+0000","last_became_peered":"2024-02-05T07:18:53.462591+0000","last_unstale":"2024-02-05T07:20:04.106708+0000","last_undegraded":"2024-02-05T07:20:04.106708+0000","last_fullsized":"2024-02-05T07:20:04.106708+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:20.782612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828117+0000","last_change":"2024-02-05T07:18:49.434399+0000","last_active":"2024-02-05T07:20:04.828117+0000","last_peered":"2024-02-05T07:20:04.828117+0000","last_clean":"2024-02-05T07:20:04.828117+0000","last_became_active":"2024-02-05T07:18:49.434074+0000","last_became_peered":"2024-02-05T07:18:49.434074+0000","last_unstale":"2024-02-05T07:20:04.828117+0000","last_undegraded":"2024-02-05T07:20:04.828117+0000","last_fullsized":"2024-02-05T07:20:04.828117+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828765+0000","last_change":"2024-02-05T07:18:51.448261+0000","last_active":"2024-02-05T07:20:04.828765+0000","last_peered":"2024-02-05T07:20:04.828765+0000","last_clean":"2024-02-05T07:20:04.828765+0000","last_became_active":"2024-02-05T07:18:51.447922+0000","last_became_peered":"2024-02-05T07:18:51.447922+0000","last_unstale":"2024-02-05T07:20:04.828765+0000","last_undegraded":"2024-02-05T07:20:04.828765+0000","last_fullsized":"2024-02-05T07:20:04.828765+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.629123+0000","last_change":"2024-02-05T07:18:55.475305+0000","last_active":"2024-02-05T07:20:04.629123+0000","last_peered":"2024-02-05T07:20:04.629123+0000","last_clean":"2024-02-05T07:20:04.629123+0000","last_became_active":"2024-02-05T07:18:55.474992+0000","last_became_peered":"2024-02-05T07:18:55.474992+0000","last_unstale":"2024-02-05T07:20:04.629123+0000","last_undegraded":"2024-02-05T07:20:04.629123+0000","last_fullsized":"2024-02-05T07:20:04.629123+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:18.245654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103938+0000","last_change":"2024-02-05T07:18:53.457727+0000","last_active":"2024-02-05T07:20:04.103938+0000","last_peered":"2024-02-05T07:20:04.103938+0000","last_clean":"2024-02-05T07:20:04.103938+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:20:04.103938+0000","last_undegraded":"2024-02-05T07:20:04.103938+0000","last_fullsized":"2024-02-05T07:20:04.103938+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.300777+0000","last_change":"2024-02-05T07:18:49.431227+0000","last_active":"2024-02-05T07:20:04.300777+0000","last_peered":"2024-02-05T07:20:04.300777+0000","last_clean":"2024-02-05T07:20:04.300777+0000","last_became_active":"2024-02-05T07:18:49.430518+0000","last_became_peered":"2024-02-05T07:18:49.430518+0000","last_unstale":"2024-02-05T07:20:04.300777+0000","last_undegraded":"2024-02-05T07:20:04.300777+0000","last_fullsized":"2024-02-05T07:20:04.300777+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:14.159632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828840+0000","last_change":"2024-02-05T07:19:51.808315+0000","last_active":"2024-02-05T07:20:04.828840+0000","last_peered":"2024-02-05T07:20:04.828840+0000","last_clean":"2024-02-05T07:20:04.828840+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:20:04.828840+0000","last_undegraded":"2024-02-05T07:20:04.828840+0000","last_fullsized":"2024-02-05T07:20:04.828840+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:19:51.808218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.808218+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:55.279939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828919+0000","last_change":"2024-02-05T07:18:55.466723+0000","last_active":"2024-02-05T07:20:04.828919+0000","last_peered":"2024-02-05T07:20:04.828919+0000","last_clean":"2024-02-05T07:20:04.828919+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:20:04.828919+0000","last_undegraded":"2024-02-05T07:20:04.828919+0000","last_fullsized":"2024-02-05T07:20:04.828919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:13.884922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.628981+0000","last_change":"2024-02-05T07:18:53.462179+0000","last_active":"2024-02-05T07:20:04.628981+0000","last_peered":"2024-02-05T07:20:04.628981+0000","last_clean":"2024-02-05T07:20:04.628981+0000","last_became_active":"2024-02-05T07:18:53.461852+0000","last_became_peered":"2024-02-05T07:18:53.461852+0000","last_unstale":"2024-02-05T07:20:04.628981+0000","last_undegraded":"2024-02-05T07:20:04.628981+0000","last_fullsized":"2024-02-05T07:20:04.628981+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:17.991283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.827237+0000","last_change":"2024-02-05T07:20:04.827259+0000","last_active":"2024-02-05T07:20:04.827237+0000","last_peered":"2024-02-05T07:20:04.827237+0000","last_clean":"2024-02-05T07:20:04.827237+0000","last_became_active":"2024-02-05T07:18:49.435368+0000","last_became_peered":"2024-02-05T07:18:49.435368+0000","last_unstale":"2024-02-05T07:20:04.827237+0000","last_undegraded":"2024-02-05T07:20:04.827237+0000","last_fullsized":"2024-02-05T07:20:04.827237+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:04.827180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:04.827180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:17.606501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828539+0000","last_change":"2024-02-05T07:18:51.443126+0000","last_active":"2024-02-05T07:20:04.828539+0000","last_peered":"2024-02-05T07:20:04.828539+0000","last_clean":"2024-02-05T07:20:04.828539+0000","last_became_active":"2024-02-05T07:18:51.442711+0000","last_became_peered":"2024-02-05T07:18:51.442711+0000","last_unstale":"2024-02-05T07:20:04.828539+0000","last_undegraded":"2024-02-05T07:20:04.828539+0000","last_fullsized":"2024-02-05T07:20:04.828539+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.627110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":397,"ondisk_log_size":397,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":63,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037780,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28380,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716772,"statfs":{"total":95995035648,"available":95965974528,"internally_reserved":0,"allocated":1392640,"data_stored":938253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.65100000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.63400000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.61399999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.59499999999999997}]},{"osd":4,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.41599999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.439},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.52700000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.49199999999999999}]}]},{"osd":6,"up_from":52,"seq":223338299418,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32528,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712624,"statfs":{"total":95995035648,"available":95961726976,"internally_reserved":0,"allocated":1380352,"data_stored":938474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.503},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.65100000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.70099999999999996}]},{"osd":2,"last update":"Mon Feb 5 07:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.56499999999999995}]},{"osd":3,"last update":"Mon Feb 5 07:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.67000000000000004}]},{"osd":4,"last update":"Mon Feb 5 07:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Feb 5 07:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.63900000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.58299999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":847872,"data_stored":355525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.35999999999999999,"15min":0.33300000000000002},"min":{"1min":0.24199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60799999999999998,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.433},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.35999999999999999,"15min":0.33200000000000002},"min":{"1min":0.28599999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.502}]},{"osd":2,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.442,"15min":0.44600000000000001},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.60799999999999998,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.88,"15min":1.0740000000000001},"min":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.61299999999999999,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.45800000000000002,"15min":0.47499999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.63800000000000001,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.47999999999999998,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68100000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59099999999999997,"15min":0.58999999999999997},"min":{"1min":0.44600000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.745},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.72999999999999998,"15min":0.753},"min":{"1min":0.41899999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.0069999999999999,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.88900000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61699999999999999,"15min":0.622},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.75700000000000001,"15min":0.78100000000000003},"min":{"1min":0.46500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.04,"5min":5.0289999999999999,"15min":5.0289999999999999},"last":0.94999999999999996}]},{"osd":6,"last update":"Mon Feb 5 07:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":0.91000000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.98499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28400,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716752,"statfs":{"total":95995035648,"available":95965954048,"internally_reserved":0,"allocated":1413120,"data_stored":944853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.32600000000000001,"15min":0.30399999999999999},"min":{"1min":0.26300000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.50700000000000001,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.31900000000000001},"min":{"1min":0.28399999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.47599999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.435}]},{"osd":2,"last update":"Mon Feb 5 07:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.439,"15min":0.42799999999999999},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.42799999999999999,"15min":0.42099999999999999},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.63100000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:20:00 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42899999999999999,"15min":0.43099999999999999},"min":{"1min":0.29899999999999999,"5min":0.246,"15min":0.246},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41399999999999998,"15min":0.41799999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.46999999999999997}]},{"osd":4,"last update":"Mon Feb 5 07:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.47599999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66000000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.54900000000000004},"min":{"1min":0.41999999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.83899999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56000000000000005,"15min":0.55400000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.497},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.55800000000000005}]},{"osd":6,"last update":"Mon Feb 5 07:19:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.65000000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.48399999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717360,"statfs":{"total":95995035648,"available":95966576640,"internally_reserved":0,"allocated":790528,"data_stored":348152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.377,"15min":0.36899999999999999},"min":{"1min":0.23499999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.40100000000000002,"15min":0.38600000000000001},"min":{"1min":0.27400000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.40999999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.39100000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.44500000000000001,"15min":0.42699999999999999},"min":{"1min":0.29099999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.45900000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.46700000000000003,"15min":0.46899999999999997},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.56000000000000005,"5min":1.631,"15min":1.631},"last":0.437},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46600000000000003,"15min":0.46400000000000002},"min":{"1min":0.32500000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.55000000000000004,"5min":1.391,"15min":1.391},"last":0.39800000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54100000000000004,"15min":0.53200000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.56899999999999995,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:19:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.61699999999999999,"15min":0.60699999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.66000000000000003}]},{"osd":6,"last update":"Mon Feb 5 07:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.8319999999999999,"5min":2.8319999999999999,"15min":2.8319999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.624}]},{"osd":7,"last update":"Mon Feb 5 07:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.61099999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":819200,"data_stored":347738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.40100000000000002,"15min":0.40699999999999997},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.51500000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45400000000000001,"15min":0.46000000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.56000000000000005,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.26000000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.46600000000000003,"15min":0.47399999999999998},"min":{"1min":0.28999999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.57199999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.499,"15min":0.50900000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.57299999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.46600000000000003,"15min":0.47199999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.59399999999999997,"5min":0.75,"15min":0.75},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.56899999999999995,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.375}]},{"osd":4,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.58899999999999997,"15min":0.59599999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.70999999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60199999999999998,"15min":0.59599999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.5779999999999998,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":0.60299999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79400000000000004,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59699999999999998,"15min":0.59499999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.747}]},{"osd":6,"last update":"Mon Feb 5 07:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.67100000000000004}]},{"osd":7,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.68700000000000006}]}]},{"osd":4,"up_from":32,"seq":137438953507,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31980,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713172,"statfs":{"total":95995035648,"available":95962288128,"internally_reserved":0,"allocated":819200,"data_stored":353898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.74199999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.65000000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.36599999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75700000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71499999999999997,"5min":0.754,"15min":0.754},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53300000000000003,"15min":0.54000000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.36399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69899999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.41999999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55600000000000005,"15min":0.56100000000000005},"min":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72999999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.67200000000000004,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.66300000000000003}]},{"osd":5,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.376,"15min":0.36599999999999999},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.39000000000000001,"15min":0.378},"min":{"1min":0.309,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.57499999999999996,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.58599999999999997}]},{"osd":6,"last update":"Mon Feb 5 07:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.68400000000000005}]},{"osd":7,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.50700000000000001}]}]},{"osd":5,"up_from":41,"seq":176093659166,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":806912,"data_stored":354139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66800000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71199999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.44800000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.55400000000000005,"15min":0.56299999999999994},"min":{"1min":0.35399999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.69699999999999995,"5min":0.753,"15min":0.753},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.38200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.68200000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.57699999999999996}]},{"osd":2,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68000000000000005,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.35099999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69999999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.51000000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64200000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71999999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.55700000000000005}]},{"osd":4,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.35999999999999999,"15min":0.34799999999999998},"min":{"1min":0.26800000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.38900000000000001,"15min":0.373},"min":{"1min":0.33200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.73499999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.62},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.70899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:20:10.983 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:20:11.232 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:11.659 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:11 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:11.659 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:11 smithi007 ceph-mon[29098]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:11.659 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:11 smithi007 ceph-mon[29098]: 3.4 deep-scrub starts 2024-02-05T07:20:11.659 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:11 smithi007 ceph-mon[29098]: 3.4 deep-scrub ok 2024-02-05T07:20:11.660 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:11 smithi007 ceph-mon[29098]: 5.6 scrub starts 2024-02-05T07:20:11.660 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:11 smithi007 ceph-mon[29098]: 5.6 scrub ok 2024-02-05T07:20:11.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:11 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:11.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:11 smithi007 ceph-mon[37158]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:11.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:11 smithi007 ceph-mon[37158]: 3.4 deep-scrub starts 2024-02-05T07:20:11.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:11 smithi007 ceph-mon[37158]: 3.4 deep-scrub ok 2024-02-05T07:20:11.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:11 smithi007 ceph-mon[37158]: 5.6 scrub starts 2024-02-05T07:20:11.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:11 smithi007 ceph-mon[37158]: 5.6 scrub ok 2024-02-05T07:20:11.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:11 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:11.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:11 smithi090 ceph-mon[33213]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:11.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:11 smithi090 ceph-mon[33213]: 3.4 deep-scrub starts 2024-02-05T07:20:11.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:11 smithi090 ceph-mon[33213]: 3.4 deep-scrub ok 2024-02-05T07:20:11.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:11 smithi090 ceph-mon[33213]: 5.6 scrub starts 2024-02-05T07:20:11.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:11 smithi090 ceph-mon[33213]: 5.6 scrub ok 2024-02-05T07:20:12.611 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:12 smithi007 ceph-mon[29098]: 6.c scrub starts 2024-02-05T07:20:12.611 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:12 smithi007 ceph-mon[29098]: 6.c scrub ok 2024-02-05T07:20:12.611 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:12 smithi007 ceph-mon[29098]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:20:12.611 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:12 smithi007 ceph-mon[29098]: 6.14 scrub starts 2024-02-05T07:20:12.611 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:12 smithi007 ceph-mon[29098]: 6.14 scrub ok 2024-02-05T07:20:12.612 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:12 smithi007 ceph-mon[37158]: 6.c scrub starts 2024-02-05T07:20:12.612 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:12 smithi007 ceph-mon[37158]: 6.c scrub ok 2024-02-05T07:20:12.612 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:12 smithi007 ceph-mon[37158]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:20:12.612 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:12 smithi007 ceph-mon[37158]: 6.14 scrub starts 2024-02-05T07:20:12.612 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:12 smithi007 ceph-mon[37158]: 6.14 scrub ok 2024-02-05T07:20:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:12 smithi090 ceph-mon[33213]: 6.c scrub starts 2024-02-05T07:20:12.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:12 smithi090 ceph-mon[33213]: 6.c scrub ok 2024-02-05T07:20:12.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:12 smithi090 ceph-mon[33213]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:20:12.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:12 smithi090 ceph-mon[33213]: 6.14 scrub starts 2024-02-05T07:20:12.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:12 smithi090 ceph-mon[33213]: 6.14 scrub ok 2024-02-05T07:20:13.051 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:20:13.053 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:20:13.655 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:13 smithi007 ceph-mon[29098]: 4.3 scrub starts 2024-02-05T07:20:13.655 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:13 smithi007 ceph-mon[29098]: 4.3 scrub ok 2024-02-05T07:20:13.655 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:13 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:20:13.655 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:13 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:20:13.656 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:20:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:20:13.656 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:13 smithi007 ceph-mon[37158]: 4.3 scrub starts 2024-02-05T07:20:13.656 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:13 smithi007 ceph-mon[37158]: 4.3 scrub ok 2024-02-05T07:20:13.656 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:13 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:20:13.656 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:13 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:20:13.683 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-02-05T07:20:12.919564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1007,"num_read_kb":853,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":610,"ondisk_log_size":610,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236740,"kb_used_data":8124,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724476,"statfs":{"total":767960285184,"available":767717863424,"internally_reserved":0,"allocated":8318976,"data_stored":4606868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002412"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828002+0000","last_change":"2024-02-05T07:18:55.466290+0000","last_active":"2024-02-05T07:20:04.828002+0000","last_peered":"2024-02-05T07:20:04.828002+0000","last_clean":"2024-02-05T07:20:04.828002+0000","last_became_active":"2024-02-05T07:18:55.466001+0000","last_became_peered":"2024-02-05T07:18:55.466001+0000","last_unstale":"2024-02-05T07:20:04.828002+0000","last_undegraded":"2024-02-05T07:20:04.828002+0000","last_fullsized":"2024-02-05T07:20:04.828002+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:09.550888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.102929+0000","last_change":"2024-02-05T07:18:53.461117+0000","last_active":"2024-02-05T07:20:04.102929+0000","last_peered":"2024-02-05T07:20:04.102929+0000","last_clean":"2024-02-05T07:20:04.102929+0000","last_became_active":"2024-02-05T07:18:53.460820+0000","last_became_peered":"2024-02-05T07:18:53.460820+0000","last_unstale":"2024-02-05T07:20:04.102929+0000","last_undegraded":"2024-02-05T07:20:04.102929+0000","last_fullsized":"2024-02-05T07:20:04.102929+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:14.324555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105373+0000","last_change":"2024-02-05T07:18:49.434741+0000","last_active":"2024-02-05T07:20:04.105373+0000","last_peered":"2024-02-05T07:20:04.105373+0000","last_clean":"2024-02-05T07:20:04.105373+0000","last_became_active":"2024-02-05T07:18:49.434456+0000","last_became_peered":"2024-02-05T07:18:49.434456+0000","last_unstale":"2024-02-05T07:20:04.105373+0000","last_undegraded":"2024-02-05T07:20:04.105373+0000","last_fullsized":"2024-02-05T07:20:04.105373+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:12.107995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.827906+0000","last_change":"2024-02-05T07:18:51.449261+0000","last_active":"2024-02-05T07:20:04.827906+0000","last_peered":"2024-02-05T07:20:04.827906+0000","last_clean":"2024-02-05T07:20:04.827906+0000","last_became_active":"2024-02-05T07:18:51.448924+0000","last_became_peered":"2024-02-05T07:18:51.448924+0000","last_unstale":"2024-02-05T07:20:04.827906+0000","last_undegraded":"2024-02-05T07:20:04.827906+0000","last_fullsized":"2024-02-05T07:20:04.827906+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.547504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826011+0000","last_change":"2024-02-05T07:18:55.506169+0000","last_active":"2024-02-05T07:20:04.826011+0000","last_peered":"2024-02-05T07:20:04.826011+0000","last_clean":"2024-02-05T07:20:04.826011+0000","last_became_active":"2024-02-05T07:18:55.505718+0000","last_became_peered":"2024-02-05T07:18:55.505718+0000","last_unstale":"2024-02-05T07:20:04.826011+0000","last_undegraded":"2024-02-05T07:20:04.826011+0000","last_fullsized":"2024-02-05T07:20:04.826011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.751562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.827917+0000","last_change":"2024-02-05T07:19:54.746304+0000","last_active":"2024-02-05T07:20:04.827917+0000","last_peered":"2024-02-05T07:20:04.827917+0000","last_clean":"2024-02-05T07:20:04.827917+0000","last_became_active":"2024-02-05T07:18:53.461506+0000","last_became_peered":"2024-02-05T07:18:53.461506+0000","last_unstale":"2024-02-05T07:20:04.827917+0000","last_undegraded":"2024-02-05T07:20:04.827917+0000","last_fullsized":"2024-02-05T07:20:04.827917+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:54.746206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:19:54.746206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:57.230561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826409+0000","last_change":"2024-02-05T07:18:49.436786+0000","last_active":"2024-02-05T07:20:04.826409+0000","last_peered":"2024-02-05T07:20:04.826409+0000","last_clean":"2024-02-05T07:20:04.826409+0000","last_became_active":"2024-02-05T07:18:49.436490+0000","last_became_peered":"2024-02-05T07:18:49.436490+0000","last_unstale":"2024-02-05T07:20:04.826409+0000","last_undegraded":"2024-02-05T07:20:04.826409+0000","last_fullsized":"2024-02-05T07:20:04.826409+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825941+0000","last_change":"2024-02-05T07:18:51.445160+0000","last_active":"2024-02-05T07:20:04.825941+0000","last_peered":"2024-02-05T07:20:04.825941+0000","last_clean":"2024-02-05T07:20:04.825941+0000","last_became_active":"2024-02-05T07:18:51.444783+0000","last_became_peered":"2024-02-05T07:18:51.444783+0000","last_unstale":"2024-02-05T07:20:04.825941+0000","last_undegraded":"2024-02-05T07:20:04.825941+0000","last_fullsized":"2024-02-05T07:20:04.825941+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:05.434052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.102877+0000","last_change":"2024-02-05T07:18:55.477022+0000","last_active":"2024-02-05T07:20:04.102877+0000","last_peered":"2024-02-05T07:20:04.102877+0000","last_clean":"2024-02-05T07:20:04.102877+0000","last_became_active":"2024-02-05T07:18:55.474002+0000","last_became_peered":"2024-02-05T07:18:55.474002+0000","last_unstale":"2024-02-05T07:20:04.102877+0000","last_undegraded":"2024-02-05T07:20:04.102877+0000","last_fullsized":"2024-02-05T07:20:04.102877+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.705323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105906+0000","last_change":"2024-02-05T07:18:53.464788+0000","last_active":"2024-02-05T07:20:04.105906+0000","last_peered":"2024-02-05T07:20:04.105906+0000","last_clean":"2024-02-05T07:20:04.105906+0000","last_became_active":"2024-02-05T07:18:53.464520+0000","last_became_peered":"2024-02-05T07:18:53.464520+0000","last_unstale":"2024-02-05T07:20:04.105906+0000","last_undegraded":"2024-02-05T07:20:04.105906+0000","last_fullsized":"2024-02-05T07:20:04.105906+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.843776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106234+0000","last_change":"2024-02-05T07:19:59.782950+0000","last_active":"2024-02-05T07:20:04.106234+0000","last_peered":"2024-02-05T07:20:04.106234+0000","last_clean":"2024-02-05T07:20:04.106234+0000","last_became_active":"2024-02-05T07:18:49.435454+0000","last_became_peered":"2024-02-05T07:18:49.435454+0000","last_unstale":"2024-02-05T07:20:04.106234+0000","last_undegraded":"2024-02-05T07:20:04.106234+0000","last_fullsized":"2024-02-05T07:20:04.106234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:59.782841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:59.782841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.410088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"89'13","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828200+0000","last_change":"2024-02-05T07:20:02.798288+0000","last_active":"2024-02-05T07:20:04.828200+0000","last_peered":"2024-02-05T07:20:04.828200+0000","last_clean":"2024-02-05T07:20:04.828200+0000","last_became_active":"2024-02-05T07:18:51.444592+0000","last_became_peered":"2024-02-05T07:18:51.444592+0000","last_unstale":"2024-02-05T07:20:04.828200+0000","last_undegraded":"2024-02-05T07:20:04.828200+0000","last_fullsized":"2024-02-05T07:20:04.828200+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.798180+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:11.151835+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105609+0000","last_change":"2024-02-05T07:18:55.473667+0000","last_active":"2024-02-05T07:20:04.105609+0000","last_peered":"2024-02-05T07:20:04.105609+0000","last_clean":"2024-02-05T07:20:04.105609+0000","last_became_active":"2024-02-05T07:18:55.473368+0000","last_became_peered":"2024-02-05T07:18:55.473368+0000","last_unstale":"2024-02-05T07:20:04.105609+0000","last_undegraded":"2024-02-05T07:20:04.105609+0000","last_fullsized":"2024-02-05T07:20:04.105609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:15.915465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.805983+0000","last_change":"2024-02-05T07:18:53.457259+0000","last_active":"2024-02-05T07:20:04.805983+0000","last_peered":"2024-02-05T07:20:04.805983+0000","last_clean":"2024-02-05T07:20:04.805983+0000","last_became_active":"2024-02-05T07:18:53.456881+0000","last_became_peered":"2024-02-05T07:18:53.456881+0000","last_unstale":"2024-02-05T07:20:04.805983+0000","last_undegraded":"2024-02-05T07:20:04.805983+0000","last_fullsized":"2024-02-05T07:20:04.805983+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.078811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106066+0000","last_change":"2024-02-05T07:18:49.431652+0000","last_active":"2024-02-05T07:20:04.106066+0000","last_peered":"2024-02-05T07:20:04.106066+0000","last_clean":"2024-02-05T07:20:04.106066+0000","last_became_active":"2024-02-05T07:18:49.431220+0000","last_became_peered":"2024-02-05T07:18:49.431220+0000","last_unstale":"2024-02-05T07:20:04.106066+0000","last_undegraded":"2024-02-05T07:20:04.106066+0000","last_fullsized":"2024-02-05T07:20:04.106066+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828154+0000","last_change":"2024-02-05T07:18:51.448820+0000","last_active":"2024-02-05T07:20:04.828154+0000","last_peered":"2024-02-05T07:20:04.828154+0000","last_clean":"2024-02-05T07:20:04.828154+0000","last_became_active":"2024-02-05T07:18:51.448462+0000","last_became_peered":"2024-02-05T07:18:51.448462+0000","last_unstale":"2024-02-05T07:20:04.828154+0000","last_undegraded":"2024-02-05T07:20:04.828154+0000","last_fullsized":"2024-02-05T07:20:04.828154+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:17.724462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828818+0000","last_change":"2024-02-05T07:20:00.760332+0000","last_active":"2024-02-05T07:20:04.828818+0000","last_peered":"2024-02-05T07:20:04.828818+0000","last_clean":"2024-02-05T07:20:04.828818+0000","last_became_active":"2024-02-05T07:18:55.505678+0000","last_became_peered":"2024-02-05T07:18:55.505678+0000","last_unstale":"2024-02-05T07:20:04.828818+0000","last_undegraded":"2024-02-05T07:20:04.828818+0000","last_fullsized":"2024-02-05T07:20:04.828818+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:00.760230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.760230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:19.923028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.104093+0000","last_change":"2024-02-05T07:18:53.461703+0000","last_active":"2024-02-05T07:20:04.104093+0000","last_peered":"2024-02-05T07:20:04.104093+0000","last_clean":"2024-02-05T07:20:04.104093+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:20:04.104093+0000","last_undegraded":"2024-02-05T07:20:04.104093+0000","last_fullsized":"2024-02-05T07:20:04.104093+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105004+0000","last_change":"2024-02-05T07:18:49.432729+0000","last_active":"2024-02-05T07:20:04.105004+0000","last_peered":"2024-02-05T07:20:04.105004+0000","last_clean":"2024-02-05T07:20:04.105004+0000","last_became_active":"2024-02-05T07:18:49.432449+0000","last_became_peered":"2024-02-05T07:18:49.432449+0000","last_unstale":"2024-02-05T07:20:04.105004+0000","last_undegraded":"2024-02-05T07:20:04.105004+0000","last_fullsized":"2024-02-05T07:20:04.105004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:17.889196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.628904+0000","last_change":"2024-02-05T07:19:59.624459+0000","last_active":"2024-02-05T07:20:04.628904+0000","last_peered":"2024-02-05T07:20:04.628904+0000","last_clean":"2024-02-05T07:20:04.628904+0000","last_became_active":"2024-02-05T07:18:51.447728+0000","last_became_peered":"2024-02-05T07:18:51.447728+0000","last_unstale":"2024-02-05T07:20:04.628904+0000","last_undegraded":"2024-02-05T07:20:04.628904+0000","last_fullsized":"2024-02-05T07:20:04.628904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:19:59.624369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:59.624369+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:28.126457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826616+0000","last_change":"2024-02-05T07:18:55.506354+0000","last_active":"2024-02-05T07:20:04.826616+0000","last_peered":"2024-02-05T07:20:04.826616+0000","last_clean":"2024-02-05T07:20:04.826616+0000","last_became_active":"2024-02-05T07:18:55.505375+0000","last_became_peered":"2024-02-05T07:18:55.505375+0000","last_unstale":"2024-02-05T07:20:04.826616+0000","last_undegraded":"2024-02-05T07:20:04.826616+0000","last_fullsized":"2024-02-05T07:20:04.826616+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.462745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828427+0000","last_change":"2024-02-05T07:18:53.459361+0000","last_active":"2024-02-05T07:20:04.828427+0000","last_peered":"2024-02-05T07:20:04.828427+0000","last_clean":"2024-02-05T07:20:04.828427+0000","last_became_active":"2024-02-05T07:18:53.459077+0000","last_became_peered":"2024-02-05T07:18:53.459077+0000","last_unstale":"2024-02-05T07:20:04.828427+0000","last_undegraded":"2024-02-05T07:20:04.828427+0000","last_fullsized":"2024-02-05T07:20:04.828427+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.124069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828252+0000","last_change":"2024-02-05T07:18:49.428575+0000","last_active":"2024-02-05T07:20:04.828252+0000","last_peered":"2024-02-05T07:20:04.828252+0000","last_clean":"2024-02-05T07:20:04.828252+0000","last_became_active":"2024-02-05T07:18:49.427837+0000","last_became_peered":"2024-02-05T07:18:49.427837+0000","last_unstale":"2024-02-05T07:20:04.828252+0000","last_undegraded":"2024-02-05T07:20:04.828252+0000","last_fullsized":"2024-02-05T07:20:04.828252+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:15.656514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826104+0000","last_change":"2024-02-05T07:20:01.841591+0000","last_active":"2024-02-05T07:20:04.826104+0000","last_peered":"2024-02-05T07:20:04.826104+0000","last_clean":"2024-02-05T07:20:04.826104+0000","last_became_active":"2024-02-05T07:18:51.447761+0000","last_became_peered":"2024-02-05T07:18:51.447761+0000","last_unstale":"2024-02-05T07:20:04.826104+0000","last_undegraded":"2024-02-05T07:20:04.826104+0000","last_fullsized":"2024-02-05T07:20:04.826104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:01.841489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:01.841489+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:19.277134+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806379+0000","last_change":"2024-02-05T07:18:55.476374+0000","last_active":"2024-02-05T07:20:04.806379+0000","last_peered":"2024-02-05T07:20:04.806379+0000","last_clean":"2024-02-05T07:20:04.806379+0000","last_became_active":"2024-02-05T07:18:55.476071+0000","last_became_peered":"2024-02-05T07:18:55.476071+0000","last_unstale":"2024-02-05T07:20:04.806379+0000","last_undegraded":"2024-02-05T07:20:04.806379+0000","last_fullsized":"2024-02-05T07:20:04.806379+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.860100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.489895+0000","last_change":"2024-02-05T07:18:53.461297+0000","last_active":"2024-02-05T07:20:04.489895+0000","last_peered":"2024-02-05T07:20:04.489895+0000","last_clean":"2024-02-05T07:20:04.489895+0000","last_became_active":"2024-02-05T07:18:53.461011+0000","last_became_peered":"2024-02-05T07:18:53.461011+0000","last_unstale":"2024-02-05T07:20:04.489895+0000","last_undegraded":"2024-02-05T07:20:04.489895+0000","last_fullsized":"2024-02-05T07:20:04.489895+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:14.390445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103048+0000","last_change":"2024-02-05T07:18:49.431334+0000","last_active":"2024-02-05T07:20:04.103048+0000","last_peered":"2024-02-05T07:20:04.103048+0000","last_clean":"2024-02-05T07:20:04.103048+0000","last_became_active":"2024-02-05T07:18:49.431031+0000","last_became_peered":"2024-02-05T07:18:49.431031+0000","last_unstale":"2024-02-05T07:20:04.103048+0000","last_undegraded":"2024-02-05T07:20:04.103048+0000","last_fullsized":"2024-02-05T07:20:04.103048+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:10.104803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105230+0000","last_change":"2024-02-05T07:18:51.510293+0000","last_active":"2024-02-05T07:20:04.105230+0000","last_peered":"2024-02-05T07:20:04.105230+0000","last_clean":"2024-02-05T07:20:04.105230+0000","last_became_active":"2024-02-05T07:18:51.509918+0000","last_became_peered":"2024-02-05T07:18:51.509918+0000","last_unstale":"2024-02-05T07:20:04.105230+0000","last_undegraded":"2024-02-05T07:20:04.105230+0000","last_fullsized":"2024-02-05T07:20:04.105230+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.104275+0000","last_change":"2024-02-05T07:20:03.444274+0000","last_active":"2024-02-05T07:20:04.104275+0000","last_peered":"2024-02-05T07:20:04.104275+0000","last_clean":"2024-02-05T07:20:04.104275+0000","last_became_active":"2024-02-05T07:18:55.466164+0000","last_became_peered":"2024-02-05T07:18:55.466164+0000","last_unstale":"2024-02-05T07:20:04.104275+0000","last_undegraded":"2024-02-05T07:20:04.104275+0000","last_fullsized":"2024-02-05T07:20:04.104275+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:03.444153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:03.444153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:32.621342+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105378+0000","last_change":"2024-02-05T07:18:53.457928+0000","last_active":"2024-02-05T07:20:04.105378+0000","last_peered":"2024-02-05T07:20:04.105378+0000","last_clean":"2024-02-05T07:20:04.105378+0000","last_became_active":"2024-02-05T07:18:53.457587+0000","last_became_peered":"2024-02-05T07:18:53.457587+0000","last_unstale":"2024-02-05T07:20:04.105378+0000","last_undegraded":"2024-02-05T07:20:04.105378+0000","last_fullsized":"2024-02-05T07:20:04.105378+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:14.500310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825630+0000","last_change":"2024-02-05T07:19:49.788355+0000","last_active":"2024-02-05T07:20:04.825630+0000","last_peered":"2024-02-05T07:20:04.825630+0000","last_clean":"2024-02-05T07:20:04.825630+0000","last_became_active":"2024-02-05T07:18:49.430988+0000","last_became_peered":"2024-02-05T07:18:49.430988+0000","last_unstale":"2024-02-05T07:20:04.825630+0000","last_undegraded":"2024-02-05T07:20:04.825630+0000","last_fullsized":"2024-02-05T07:20:04.825630+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:49.788275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:49.788275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:50.100908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:06.366769+0000","last_change":"2024-02-05T07:20:06.366811+0000","last_active":"2024-02-05T07:20:06.366769+0000","last_peered":"2024-02-05T07:20:06.366769+0000","last_clean":"2024-02-05T07:20:06.366769+0000","last_became_active":"2024-02-05T07:18:51.446604+0000","last_became_peered":"2024-02-05T07:18:51.446604+0000","last_unstale":"2024-02-05T07:20:06.366769+0000","last_undegraded":"2024-02-05T07:20:06.366769+0000","last_fullsized":"2024-02-05T07:20:06.366769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:20:06.366699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.366699+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.671090+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.805981+0000","last_change":"2024-02-05T07:20:01.693975+0000","last_active":"2024-02-05T07:20:04.805981+0000","last_peered":"2024-02-05T07:20:04.805981+0000","last_clean":"2024-02-05T07:20:04.805981+0000","last_became_active":"2024-02-05T07:18:55.468086+0000","last_became_peered":"2024-02-05T07:18:55.468086+0000","last_unstale":"2024-02-05T07:20:04.805981+0000","last_undegraded":"2024-02-05T07:20:04.805981+0000","last_fullsized":"2024-02-05T07:20:04.805981+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:01.693871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:01.693871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:12.004988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:09.349444+0000","last_change":"2024-02-05T07:20:09.349490+0000","last_active":"2024-02-05T07:20:09.349444+0000","last_peered":"2024-02-05T07:20:09.349444+0000","last_clean":"2024-02-05T07:20:09.349444+0000","last_became_active":"2024-02-05T07:18:53.460307+0000","last_became_peered":"2024-02-05T07:18:53.460307+0000","last_unstale":"2024-02-05T07:20:09.349444+0000","last_undegraded":"2024-02-05T07:20:09.349444+0000","last_fullsized":"2024-02-05T07:20:09.349444+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.349342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.451852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105269+0000","last_change":"2024-02-05T07:18:49.434712+0000","last_active":"2024-02-05T07:20:04.105269+0000","last_peered":"2024-02-05T07:20:04.105269+0000","last_clean":"2024-02-05T07:20:04.105269+0000","last_became_active":"2024-02-05T07:18:49.434408+0000","last_became_peered":"2024-02-05T07:18:49.434408+0000","last_unstale":"2024-02-05T07:20:04.105269+0000","last_undegraded":"2024-02-05T07:20:04.105269+0000","last_fullsized":"2024-02-05T07:20:04.105269+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825638+0000","last_change":"2024-02-05T07:18:51.449489+0000","last_active":"2024-02-05T07:20:04.825638+0000","last_peered":"2024-02-05T07:20:04.825638+0000","last_clean":"2024-02-05T07:20:04.825638+0000","last_became_active":"2024-02-05T07:18:51.449083+0000","last_became_peered":"2024-02-05T07:18:51.449083+0000","last_unstale":"2024-02-05T07:20:04.825638+0000","last_undegraded":"2024-02-05T07:20:04.825638+0000","last_fullsized":"2024-02-05T07:20:04.825638+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825873+0000","last_change":"2024-02-05T07:18:55.507549+0000","last_active":"2024-02-05T07:20:04.825873+0000","last_peered":"2024-02-05T07:20:04.825873+0000","last_clean":"2024-02-05T07:20:04.825873+0000","last_became_active":"2024-02-05T07:18:55.506968+0000","last_became_peered":"2024-02-05T07:18:55.506968+0000","last_unstale":"2024-02-05T07:20:04.825873+0000","last_undegraded":"2024-02-05T07:20:04.825873+0000","last_fullsized":"2024-02-05T07:20:04.825873+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:05.674073+0000","last_change":"2024-02-05T07:20:05.674115+0000","last_active":"2024-02-05T07:20:05.674073+0000","last_peered":"2024-02-05T07:20:05.674073+0000","last_clean":"2024-02-05T07:20:05.674073+0000","last_became_active":"2024-02-05T07:18:53.460977+0000","last_became_peered":"2024-02-05T07:18:53.460977+0000","last_unstale":"2024-02-05T07:20:05.674073+0000","last_undegraded":"2024-02-05T07:20:05.674073+0000","last_fullsized":"2024-02-05T07:20:05.674073+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_clean_scrub_stamp":"2024-02-05T07:20:05.673983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:06.714441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106407+0000","last_change":"2024-02-05T07:19:48.944128+0000","last_active":"2024-02-05T07:20:04.106407+0000","last_peered":"2024-02-05T07:20:04.106407+0000","last_clean":"2024-02-05T07:20:04.106407+0000","last_became_active":"2024-02-05T07:18:49.434560+0000","last_became_peered":"2024-02-05T07:18:49.434560+0000","last_unstale":"2024-02-05T07:20:04.106407+0000","last_undegraded":"2024-02-05T07:20:04.106407+0000","last_fullsized":"2024-02-05T07:20:04.106407+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:48.944028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:48.944028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:56.156526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825839+0000","last_change":"2024-02-05T07:18:51.510217+0000","last_active":"2024-02-05T07:20:04.825839+0000","last_peered":"2024-02-05T07:20:04.825839+0000","last_clean":"2024-02-05T07:20:04.825839+0000","last_became_active":"2024-02-05T07:18:51.509911+0000","last_became_peered":"2024-02-05T07:18:51.509911+0000","last_unstale":"2024-02-05T07:20:04.825839+0000","last_undegraded":"2024-02-05T07:20:04.825839+0000","last_fullsized":"2024-02-05T07:20:04.825839+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.102411+0000","last_change":"2024-02-05T07:18:55.477626+0000","last_active":"2024-02-05T07:20:04.102411+0000","last_peered":"2024-02-05T07:20:04.102411+0000","last_clean":"2024-02-05T07:20:04.102411+0000","last_became_active":"2024-02-05T07:18:55.477312+0000","last_became_peered":"2024-02-05T07:18:55.477312+0000","last_unstale":"2024-02-05T07:20:04.102411+0000","last_undegraded":"2024-02-05T07:20:04.102411+0000","last_fullsized":"2024-02-05T07:20:04.102411+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.934928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826060+0000","last_change":"2024-02-05T07:18:53.468306+0000","last_active":"2024-02-05T07:20:04.826060+0000","last_peered":"2024-02-05T07:20:04.826060+0000","last_clean":"2024-02-05T07:20:04.826060+0000","last_became_active":"2024-02-05T07:18:53.467986+0000","last_became_peered":"2024-02-05T07:18:53.467986+0000","last_unstale":"2024-02-05T07:20:04.826060+0000","last_undegraded":"2024-02-05T07:20:04.826060+0000","last_fullsized":"2024-02-05T07:20:04.826060+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:21.663256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806406+0000","last_change":"2024-02-05T07:18:49.438840+0000","last_active":"2024-02-05T07:20:04.806406+0000","last_peered":"2024-02-05T07:20:04.806406+0000","last_clean":"2024-02-05T07:20:04.806406+0000","last_became_active":"2024-02-05T07:18:49.438565+0000","last_became_peered":"2024-02-05T07:18:49.438565+0000","last_unstale":"2024-02-05T07:20:04.806406+0000","last_undegraded":"2024-02-05T07:20:04.806406+0000","last_fullsized":"2024-02-05T07:20:04.806406+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:15.720884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105721+0000","last_change":"2024-02-05T07:19:16.128305+0000","last_active":"2024-02-05T07:20:04.105721+0000","last_peered":"2024-02-05T07:20:04.105721+0000","last_clean":"2024-02-05T07:20:04.105721+0000","last_became_active":"2024-02-05T07:19:16.122059+0000","last_became_peered":"2024-02-05T07:19:16.122059+0000","last_unstale":"2024-02-05T07:20:04.105721+0000","last_undegraded":"2024-02-05T07:20:04.105721+0000","last_fullsized":"2024-02-05T07:20:04.105721+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:16.076653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825624+0000","last_change":"2024-02-05T07:18:55.478367+0000","last_active":"2024-02-05T07:20:04.825624+0000","last_peered":"2024-02-05T07:20:04.825624+0000","last_clean":"2024-02-05T07:20:04.825624+0000","last_became_active":"2024-02-05T07:18:55.478087+0000","last_became_peered":"2024-02-05T07:18:55.478087+0000","last_unstale":"2024-02-05T07:20:04.825624+0000","last_undegraded":"2024-02-05T07:20:04.825624+0000","last_fullsized":"2024-02-05T07:20:04.825624+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.104389+0000","last_change":"2024-02-05T07:18:53.463942+0000","last_active":"2024-02-05T07:20:04.104389+0000","last_peered":"2024-02-05T07:20:04.104389+0000","last_clean":"2024-02-05T07:20:04.104389+0000","last_became_active":"2024-02-05T07:18:53.463620+0000","last_became_peered":"2024-02-05T07:18:53.463620+0000","last_unstale":"2024-02-05T07:20:04.104389+0000","last_undegraded":"2024-02-05T07:20:04.104389+0000","last_fullsized":"2024-02-05T07:20:04.104389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.598964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828650+0000","last_change":"2024-02-05T07:18:49.431084+0000","last_active":"2024-02-05T07:20:04.828650+0000","last_peered":"2024-02-05T07:20:04.828650+0000","last_clean":"2024-02-05T07:20:04.828650+0000","last_became_active":"2024-02-05T07:18:49.430766+0000","last_became_peered":"2024-02-05T07:18:49.430766+0000","last_unstale":"2024-02-05T07:20:04.828650+0000","last_undegraded":"2024-02-05T07:20:04.828650+0000","last_fullsized":"2024-02-05T07:20:04.828650+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:14.154245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":95,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:06.870083+0000","last_change":"2024-02-05T07:20:06.870132+0000","last_active":"2024-02-05T07:20:06.870083+0000","last_peered":"2024-02-05T07:20:06.870083+0000","last_clean":"2024-02-05T07:20:06.870083+0000","last_became_active":"2024-02-05T07:19:16.121696+0000","last_became_peered":"2024-02-05T07:19:16.121696+0000","last_unstale":"2024-02-05T07:20:06.870083+0000","last_undegraded":"2024-02-05T07:20:06.870083+0000","last_fullsized":"2024-02-05T07:20:06.870083+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:20:06.870006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.870006+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:12.414942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103711+0000","last_change":"2024-02-05T07:18:55.478247+0000","last_active":"2024-02-05T07:20:04.103711+0000","last_peered":"2024-02-05T07:20:04.103711+0000","last_clean":"2024-02-05T07:20:04.103711+0000","last_became_active":"2024-02-05T07:18:55.477896+0000","last_became_peered":"2024-02-05T07:18:55.477896+0000","last_unstale":"2024-02-05T07:20:04.103711+0000","last_undegraded":"2024-02-05T07:20:04.103711+0000","last_fullsized":"2024-02-05T07:20:04.103711+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.489933+0000","last_change":"2024-02-05T07:18:53.465570+0000","last_active":"2024-02-05T07:20:04.489933+0000","last_peered":"2024-02-05T07:20:04.489933+0000","last_clean":"2024-02-05T07:20:04.489933+0000","last_became_active":"2024-02-05T07:18:53.465287+0000","last_became_peered":"2024-02-05T07:18:53.465287+0000","last_unstale":"2024-02-05T07:20:04.489933+0000","last_undegraded":"2024-02-05T07:20:04.489933+0000","last_fullsized":"2024-02-05T07:20:04.489933+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:13.807259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806132+0000","last_change":"2024-02-05T07:18:49.436416+0000","last_active":"2024-02-05T07:20:04.806132+0000","last_peered":"2024-02-05T07:20:04.806132+0000","last_clean":"2024-02-05T07:20:04.806132+0000","last_became_active":"2024-02-05T07:18:49.436131+0000","last_became_peered":"2024-02-05T07:18:49.436131+0000","last_unstale":"2024-02-05T07:20:04.806132+0000","last_undegraded":"2024-02-05T07:20:04.806132+0000","last_fullsized":"2024-02-05T07:20:04.806132+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:08.178865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826559+0000","last_change":"2024-02-05T07:18:51.507793+0000","last_active":"2024-02-05T07:20:04.826559+0000","last_peered":"2024-02-05T07:20:04.826559+0000","last_clean":"2024-02-05T07:20:04.826559+0000","last_became_active":"2024-02-05T07:18:51.507491+0000","last_became_peered":"2024-02-05T07:18:51.507491+0000","last_unstale":"2024-02-05T07:20:04.826559+0000","last_undegraded":"2024-02-05T07:20:04.826559+0000","last_fullsized":"2024-02-05T07:20:04.826559+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.940131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828875+0000","last_change":"2024-02-05T07:18:55.467356+0000","last_active":"2024-02-05T07:20:04.828875+0000","last_peered":"2024-02-05T07:20:04.828875+0000","last_clean":"2024-02-05T07:20:04.828875+0000","last_became_active":"2024-02-05T07:18:55.467044+0000","last_became_peered":"2024-02-05T07:18:55.467044+0000","last_unstale":"2024-02-05T07:20:04.828875+0000","last_undegraded":"2024-02-05T07:20:04.828875+0000","last_fullsized":"2024-02-05T07:20:04.828875+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:12.322358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826304+0000","last_change":"2024-02-05T07:18:53.465551+0000","last_active":"2024-02-05T07:20:04.826304+0000","last_peered":"2024-02-05T07:20:04.826304+0000","last_clean":"2024-02-05T07:20:04.826304+0000","last_became_active":"2024-02-05T07:18:53.465270+0000","last_became_peered":"2024-02-05T07:18:53.465270+0000","last_unstale":"2024-02-05T07:20:04.826304+0000","last_undegraded":"2024-02-05T07:20:04.826304+0000","last_fullsized":"2024-02-05T07:20:04.826304+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:15.539033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103163+0000","last_change":"2024-02-05T07:18:49.435653+0000","last_active":"2024-02-05T07:20:04.103163+0000","last_peered":"2024-02-05T07:20:04.103163+0000","last_clean":"2024-02-05T07:20:04.103163+0000","last_became_active":"2024-02-05T07:18:49.434402+0000","last_became_peered":"2024-02-05T07:18:49.434402+0000","last_unstale":"2024-02-05T07:20:04.103163+0000","last_undegraded":"2024-02-05T07:20:04.103163+0000","last_fullsized":"2024-02-05T07:20:04.103163+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.792373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:06.852671+0000","last_change":"2024-02-05T07:20:06.852726+0000","last_active":"2024-02-05T07:20:06.852671+0000","last_peered":"2024-02-05T07:20:06.852671+0000","last_clean":"2024-02-05T07:20:06.852671+0000","last_became_active":"2024-02-05T07:18:51.508551+0000","last_became_peered":"2024-02-05T07:18:51.508551+0000","last_unstale":"2024-02-05T07:20:06.852671+0000","last_undegraded":"2024-02-05T07:20:06.852671+0000","last_fullsized":"2024-02-05T07:20:06.852671+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:20:06.852562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.852562+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:20.949048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.629037+0000","last_change":"2024-02-05T07:18:55.469355+0000","last_active":"2024-02-05T07:20:04.629037+0000","last_peered":"2024-02-05T07:20:04.629037+0000","last_clean":"2024-02-05T07:20:04.629037+0000","last_became_active":"2024-02-05T07:18:55.469044+0000","last_became_peered":"2024-02-05T07:18:55.469044+0000","last_unstale":"2024-02-05T07:20:04.629037+0000","last_undegraded":"2024-02-05T07:20:04.629037+0000","last_fullsized":"2024-02-05T07:20:04.629037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:10.874735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826572+0000","last_change":"2024-02-05T07:20:02.813432+0000","last_active":"2024-02-05T07:20:04.826572+0000","last_peered":"2024-02-05T07:20:04.826572+0000","last_clean":"2024-02-05T07:20:04.826572+0000","last_became_active":"2024-02-05T07:18:53.463607+0000","last_became_peered":"2024-02-05T07:18:53.463607+0000","last_unstale":"2024-02-05T07:20:04.826572+0000","last_undegraded":"2024-02-05T07:20:04.826572+0000","last_fullsized":"2024-02-05T07:20:04.826572+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.813344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.813344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:30.005634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806163+0000","last_change":"2024-02-05T07:20:02.673707+0000","last_active":"2024-02-05T07:20:04.806163+0000","last_peered":"2024-02-05T07:20:04.806163+0000","last_clean":"2024-02-05T07:20:04.806163+0000","last_became_active":"2024-02-05T07:18:49.437732+0000","last_became_peered":"2024-02-05T07:18:49.437732+0000","last_unstale":"2024-02-05T07:20:04.806163+0000","last_undegraded":"2024-02-05T07:20:04.806163+0000","last_fullsized":"2024-02-05T07:20:04.806163+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.673592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.665799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105276+0000","last_change":"2024-02-05T07:18:51.510254+0000","last_active":"2024-02-05T07:20:04.105276+0000","last_peered":"2024-02-05T07:20:04.105276+0000","last_clean":"2024-02-05T07:20:04.105276+0000","last_became_active":"2024-02-05T07:18:51.509917+0000","last_became_peered":"2024-02-05T07:18:51.509917+0000","last_unstale":"2024-02-05T07:20:04.105276+0000","last_undegraded":"2024-02-05T07:20:04.105276+0000","last_fullsized":"2024-02-05T07:20:04.105276+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:17.917819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806435+0000","last_change":"2024-02-05T07:18:55.507824+0000","last_active":"2024-02-05T07:20:04.806435+0000","last_peered":"2024-02-05T07:20:04.806435+0000","last_clean":"2024-02-05T07:20:04.806435+0000","last_became_active":"2024-02-05T07:18:55.507555+0000","last_became_peered":"2024-02-05T07:18:55.507555+0000","last_unstale":"2024-02-05T07:20:04.806435+0000","last_undegraded":"2024-02-05T07:20:04.806435+0000","last_fullsized":"2024-02-05T07:20:04.806435+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.017979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106495+0000","last_change":"2024-02-05T07:18:53.465851+0000","last_active":"2024-02-05T07:20:04.106495+0000","last_peered":"2024-02-05T07:20:04.106495+0000","last_clean":"2024-02-05T07:20:04.106495+0000","last_became_active":"2024-02-05T07:18:53.465591+0000","last_became_peered":"2024-02-05T07:18:53.465591+0000","last_unstale":"2024-02-05T07:20:04.106495+0000","last_undegraded":"2024-02-05T07:20:04.106495+0000","last_fullsized":"2024-02-05T07:20:04.106495+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:16.779901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.300528+0000","last_change":"2024-02-05T07:19:55.369939+0000","last_active":"2024-02-05T07:20:04.300528+0000","last_peered":"2024-02-05T07:20:04.300528+0000","last_clean":"2024-02-05T07:20:04.300528+0000","last_became_active":"2024-02-05T07:18:49.433121+0000","last_became_peered":"2024-02-05T07:18:49.433121+0000","last_unstale":"2024-02-05T07:20:04.300528+0000","last_undegraded":"2024-02-05T07:20:04.300528+0000","last_fullsized":"2024-02-05T07:20:04.300528+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:19:55.369800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:55.369800+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:04.984273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826639+0000","last_change":"2024-02-05T07:18:51.508489+0000","last_active":"2024-02-05T07:20:04.826639+0000","last_peered":"2024-02-05T07:20:04.826639+0000","last_clean":"2024-02-05T07:20:04.826639+0000","last_became_active":"2024-02-05T07:18:51.507451+0000","last_became_peered":"2024-02-05T07:18:51.507451+0000","last_unstale":"2024-02-05T07:20:04.826639+0000","last_undegraded":"2024-02-05T07:20:04.826639+0000","last_fullsized":"2024-02-05T07:20:04.826639+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:18.887208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106057+0000","last_change":"2024-02-05T07:18:55.505712+0000","last_active":"2024-02-05T07:20:04.106057+0000","last_peered":"2024-02-05T07:20:04.106057+0000","last_clean":"2024-02-05T07:20:04.106057+0000","last_became_active":"2024-02-05T07:18:55.505401+0000","last_became_peered":"2024-02-05T07:18:55.505401+0000","last_unstale":"2024-02-05T07:20:04.106057+0000","last_undegraded":"2024-02-05T07:20:04.106057+0000","last_fullsized":"2024-02-05T07:20:04.106057+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:13.385778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806823+0000","last_change":"2024-02-05T07:18:49.437997+0000","last_active":"2024-02-05T07:20:04.806823+0000","last_peered":"2024-02-05T07:20:04.806823+0000","last_clean":"2024-02-05T07:20:04.806823+0000","last_became_active":"2024-02-05T07:18:49.437663+0000","last_became_peered":"2024-02-05T07:18:49.437663+0000","last_unstale":"2024-02-05T07:20:04.806823+0000","last_undegraded":"2024-02-05T07:20:04.806823+0000","last_fullsized":"2024-02-05T07:20:04.806823+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.060603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105633+0000","last_change":"2024-02-05T07:19:58.753612+0000","last_active":"2024-02-05T07:20:04.105633+0000","last_peered":"2024-02-05T07:20:04.105633+0000","last_clean":"2024-02-05T07:20:04.105633+0000","last_became_active":"2024-02-05T07:18:51.508097+0000","last_became_peered":"2024-02-05T07:18:51.508097+0000","last_unstale":"2024-02-05T07:20:04.105633+0000","last_undegraded":"2024-02-05T07:20:04.105633+0000","last_fullsized":"2024-02-05T07:20:04.105633+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:19:58.753472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:58.753472+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:26.081872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806378+0000","last_change":"2024-02-05T07:20:00.685371+0000","last_active":"2024-02-05T07:20:04.806378+0000","last_peered":"2024-02-05T07:20:04.806378+0000","last_clean":"2024-02-05T07:20:04.806378+0000","last_became_active":"2024-02-05T07:18:53.456287+0000","last_became_peered":"2024-02-05T07:18:53.456287+0000","last_unstale":"2024-02-05T07:20:04.806378+0000","last_undegraded":"2024-02-05T07:20:04.806378+0000","last_fullsized":"2024-02-05T07:20:04.806378+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.685282+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:11.368545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825965+0000","last_change":"2024-02-05T07:18:55.474988+0000","last_active":"2024-02-05T07:20:04.825965+0000","last_peered":"2024-02-05T07:20:04.825965+0000","last_clean":"2024-02-05T07:20:04.825965+0000","last_became_active":"2024-02-05T07:18:55.474663+0000","last_became_peered":"2024-02-05T07:18:55.474663+0000","last_unstale":"2024-02-05T07:20:04.825965+0000","last_undegraded":"2024-02-05T07:20:04.825965+0000","last_fullsized":"2024-02-05T07:20:04.825965+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806593+0000","last_change":"2024-02-05T07:19:51.758261+0000","last_active":"2024-02-05T07:20:04.806593+0000","last_peered":"2024-02-05T07:20:04.806593+0000","last_clean":"2024-02-05T07:20:04.806593+0000","last_became_active":"2024-02-05T07:18:49.435621+0000","last_became_peered":"2024-02-05T07:18:49.435621+0000","last_unstale":"2024-02-05T07:20:04.806593+0000","last_undegraded":"2024-02-05T07:20:04.806593+0000","last_fullsized":"2024-02-05T07:20:04.806593+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:51.758163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.758163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:15.549152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":99,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.885848+0000","last_change":"2024-02-05T07:20:04.885890+0000","last_active":"2024-02-05T07:20:04.885848+0000","last_peered":"2024-02-05T07:20:04.885848+0000","last_clean":"2024-02-05T07:20:04.885848+0000","last_became_active":"2024-02-05T07:19:16.123650+0000","last_became_peered":"2024-02-05T07:19:16.123650+0000","last_unstale":"2024-02-05T07:20:04.885848+0000","last_undegraded":"2024-02-05T07:20:04.885848+0000","last_fullsized":"2024-02-05T07:20:04.885848+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:04.885778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:04.885778+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:07.176995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.628647+0000","last_change":"2024-02-05T07:20:02.694603+0000","last_active":"2024-02-05T07:20:04.628647+0000","last_peered":"2024-02-05T07:20:04.628647+0000","last_clean":"2024-02-05T07:20:04.628647+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:20:04.628647+0000","last_undegraded":"2024-02-05T07:20:04.628647+0000","last_fullsized":"2024-02-05T07:20:04.628647+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.694506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.694506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.700377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.805796+0000","last_change":"2024-02-05T07:18:55.473523+0000","last_active":"2024-02-05T07:20:04.805796+0000","last_peered":"2024-02-05T07:20:04.805796+0000","last_clean":"2024-02-05T07:20:04.805796+0000","last_became_active":"2024-02-05T07:18:55.473032+0000","last_became_peered":"2024-02-05T07:18:55.473032+0000","last_unstale":"2024-02-05T07:20:04.805796+0000","last_undegraded":"2024-02-05T07:20:04.805796+0000","last_fullsized":"2024-02-05T07:20:04.805796+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:12.882475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.628877+0000","last_change":"2024-02-05T07:18:53.465189+0000","last_active":"2024-02-05T07:20:04.628877+0000","last_peered":"2024-02-05T07:20:04.628877+0000","last_clean":"2024-02-05T07:20:04.628877+0000","last_became_active":"2024-02-05T07:18:53.464867+0000","last_became_peered":"2024-02-05T07:18:53.464867+0000","last_unstale":"2024-02-05T07:20:04.628877+0000","last_undegraded":"2024-02-05T07:20:04.628877+0000","last_fullsized":"2024-02-05T07:20:04.628877+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:15.717864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806645+0000","last_change":"2024-02-05T07:19:55.788134+0000","last_active":"2024-02-05T07:20:04.806645+0000","last_peered":"2024-02-05T07:20:04.806645+0000","last_clean":"2024-02-05T07:20:04.806645+0000","last_became_active":"2024-02-05T07:18:49.434451+0000","last_became_peered":"2024-02-05T07:18:49.434451+0000","last_unstale":"2024-02-05T07:20:04.806645+0000","last_undegraded":"2024-02-05T07:20:04.806645+0000","last_fullsized":"2024-02-05T07:20:04.806645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:55.788005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:55.788005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.126482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.300887+0000","last_change":"2024-02-05T07:20:00.384918+0000","last_active":"2024-02-05T07:20:04.300887+0000","last_peered":"2024-02-05T07:20:04.300887+0000","last_clean":"2024-02-05T07:20:04.300887+0000","last_became_active":"2024-02-05T07:18:51.507672+0000","last_became_peered":"2024-02-05T07:18:51.507672+0000","last_unstale":"2024-02-05T07:20:04.300887+0000","last_undegraded":"2024-02-05T07:20:04.300887+0000","last_fullsized":"2024-02-05T07:20:04.300887+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:20:00.384813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.384813+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:12.230867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.102967+0000","last_change":"2024-02-05T07:18:55.476043+0000","last_active":"2024-02-05T07:20:04.102967+0000","last_peered":"2024-02-05T07:20:04.102967+0000","last_clean":"2024-02-05T07:20:04.102967+0000","last_became_active":"2024-02-05T07:18:55.475702+0000","last_became_peered":"2024-02-05T07:18:55.475702+0000","last_unstale":"2024-02-05T07:20:04.102967+0000","last_undegraded":"2024-02-05T07:20:04.102967+0000","last_fullsized":"2024-02-05T07:20:04.102967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:20.177474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:09.589001+0000","last_change":"2024-02-05T07:20:09.589044+0000","last_active":"2024-02-05T07:20:09.589001+0000","last_peered":"2024-02-05T07:20:09.589001+0000","last_clean":"2024-02-05T07:20:09.589001+0000","last_became_active":"2024-02-05T07:18:53.459560+0000","last_became_peered":"2024-02-05T07:18:53.459560+0000","last_unstale":"2024-02-05T07:20:09.589001+0000","last_undegraded":"2024-02-05T07:20:09.589001+0000","last_fullsized":"2024-02-05T07:20:09.589001+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:09.588893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.588893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:27.848596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105948+0000","last_change":"2024-02-05T07:18:49.434599+0000","last_active":"2024-02-05T07:20:04.105948+0000","last_peered":"2024-02-05T07:20:04.105948+0000","last_clean":"2024-02-05T07:20:04.105948+0000","last_became_active":"2024-02-05T07:18:49.434238+0000","last_became_peered":"2024-02-05T07:18:49.434238+0000","last_unstale":"2024-02-05T07:20:04.105948+0000","last_undegraded":"2024-02-05T07:20:04.105948+0000","last_fullsized":"2024-02-05T07:20:04.105948+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105194+0000","last_change":"2024-02-05T07:18:51.441099+0000","last_active":"2024-02-05T07:20:04.105194+0000","last_peered":"2024-02-05T07:20:04.105194+0000","last_clean":"2024-02-05T07:20:04.105194+0000","last_became_active":"2024-02-05T07:18:51.440744+0000","last_became_peered":"2024-02-05T07:18:51.440744+0000","last_unstale":"2024-02-05T07:20:04.105194+0000","last_undegraded":"2024-02-05T07:20:04.105194+0000","last_fullsized":"2024-02-05T07:20:04.105194+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:14.420348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826260+0000","last_change":"2024-02-05T07:18:55.476663+0000","last_active":"2024-02-05T07:20:04.826260+0000","last_peered":"2024-02-05T07:20:04.826260+0000","last_clean":"2024-02-05T07:20:04.826260+0000","last_became_active":"2024-02-05T07:18:55.476313+0000","last_became_peered":"2024-02-05T07:18:55.476313+0000","last_unstale":"2024-02-05T07:20:04.826260+0000","last_undegraded":"2024-02-05T07:20:04.826260+0000","last_fullsized":"2024-02-05T07:20:04.826260+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:20.640694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105845+0000","last_change":"2024-02-05T07:18:52.445271+0000","last_active":"2024-02-05T07:20:04.105845+0000","last_peered":"2024-02-05T07:20:04.105845+0000","last_clean":"2024-02-05T07:20:04.105845+0000","last_became_active":"2024-02-05T07:18:48.423032+0000","last_became_peered":"2024-02-05T07:18:48.423032+0000","last_unstale":"2024-02-05T07:20:04.105845+0000","last_undegraded":"2024-02-05T07:20:04.105845+0000","last_fullsized":"2024-02-05T07:20:04.105845+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:18:47.405257+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:14.430841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828156+0000","last_change":"2024-02-05T07:18:49.430550+0000","last_active":"2024-02-05T07:20:04.828156+0000","last_peered":"2024-02-05T07:20:04.828156+0000","last_clean":"2024-02-05T07:20:04.828156+0000","last_became_active":"2024-02-05T07:18:49.430208+0000","last_became_peered":"2024-02-05T07:18:49.430208+0000","last_unstale":"2024-02-05T07:20:04.828156+0000","last_undegraded":"2024-02-05T07:20:04.828156+0000","last_fullsized":"2024-02-05T07:20:04.828156+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:10.597988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":116,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.489550+0000","last_change":"2024-02-05T07:18:51.442632+0000","last_active":"2024-02-05T07:20:04.489550+0000","last_peered":"2024-02-05T07:20:04.489550+0000","last_clean":"2024-02-05T07:20:04.489550+0000","last_became_active":"2024-02-05T07:18:51.442110+0000","last_became_peered":"2024-02-05T07:18:51.442110+0000","last_unstale":"2024-02-05T07:20:04.489550+0000","last_undegraded":"2024-02-05T07:20:04.489550+0000","last_fullsized":"2024-02-05T07:20:04.489550+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103912+0000","last_change":"2024-02-05T07:18:53.461030+0000","last_active":"2024-02-05T07:20:04.103912+0000","last_peered":"2024-02-05T07:20:04.103912+0000","last_clean":"2024-02-05T07:20:04.103912+0000","last_became_active":"2024-02-05T07:18:53.460715+0000","last_became_peered":"2024-02-05T07:18:53.460715+0000","last_unstale":"2024-02-05T07:20:04.103912+0000","last_undegraded":"2024-02-05T07:20:04.103912+0000","last_fullsized":"2024-02-05T07:20:04.103912+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.356767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:06.673156+0000","last_change":"2024-02-05T07:20:06.673182+0000","last_active":"2024-02-05T07:20:06.673156+0000","last_peered":"2024-02-05T07:20:06.673156+0000","last_clean":"2024-02-05T07:20:06.673156+0000","last_became_active":"2024-02-05T07:18:55.506803+0000","last_became_peered":"2024-02-05T07:18:55.506803+0000","last_unstale":"2024-02-05T07:20:06.673156+0000","last_undegraded":"2024-02-05T07:20:06.673156+0000","last_fullsized":"2024-02-05T07:20:06.673156+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.673081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:21.893489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:09.406791+0000","last_change":"2024-02-05T07:20:09.406817+0000","last_active":"2024-02-05T07:20:09.406791+0000","last_peered":"2024-02-05T07:20:09.406791+0000","last_clean":"2024-02-05T07:20:09.406791+0000","last_became_active":"2024-02-05T07:18:49.429651+0000","last_became_peered":"2024-02-05T07:18:49.429651+0000","last_unstale":"2024-02-05T07:20:09.406791+0000","last_undegraded":"2024-02-05T07:20:09.406791+0000","last_fullsized":"2024-02-05T07:20:09.406791+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:20:09.406718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.406718+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:33.441463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.628755+0000","last_change":"2024-02-05T07:19:56.661601+0000","last_active":"2024-02-05T07:20:04.628755+0000","last_peered":"2024-02-05T07:20:04.628755+0000","last_clean":"2024-02-05T07:20:04.628755+0000","last_became_active":"2024-02-05T07:18:48.423639+0000","last_became_peered":"2024-02-05T07:18:48.423639+0000","last_unstale":"2024-02-05T07:20:04.628755+0000","last_undegraded":"2024-02-05T07:20:04.628755+0000","last_fullsized":"2024-02-05T07:20:04.628755+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:56.661490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:19:56.661490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.585463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.104456+0000","last_change":"2024-02-05T07:18:51.436432+0000","last_active":"2024-02-05T07:20:04.104456+0000","last_peered":"2024-02-05T07:20:04.104456+0000","last_clean":"2024-02-05T07:20:04.104456+0000","last_became_active":"2024-02-05T07:18:51.436097+0000","last_became_peered":"2024-02-05T07:18:51.436097+0000","last_unstale":"2024-02-05T07:20:04.104456+0000","last_undegraded":"2024-02-05T07:20:04.104456+0000","last_fullsized":"2024-02-05T07:20:04.104456+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:11.611640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.629033+0000","last_change":"2024-02-05T07:18:53.461134+0000","last_active":"2024-02-05T07:20:04.629033+0000","last_peered":"2024-02-05T07:20:04.629033+0000","last_clean":"2024-02-05T07:20:04.629033+0000","last_became_active":"2024-02-05T07:18:53.460771+0000","last_became_peered":"2024-02-05T07:18:53.460771+0000","last_unstale":"2024-02-05T07:20:04.629033+0000","last_undegraded":"2024-02-05T07:20:04.629033+0000","last_fullsized":"2024-02-05T07:20:04.629033+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:09.923504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106539+0000","last_change":"2024-02-05T07:18:55.473873+0000","last_active":"2024-02-05T07:20:04.106539+0000","last_peered":"2024-02-05T07:20:04.106539+0000","last_clean":"2024-02-05T07:20:04.106539+0000","last_became_active":"2024-02-05T07:18:55.473358+0000","last_became_peered":"2024-02-05T07:18:55.473358+0000","last_unstale":"2024-02-05T07:20:04.106539+0000","last_undegraded":"2024-02-05T07:20:04.106539+0000","last_fullsized":"2024-02-05T07:20:04.106539+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:09.714221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":311,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.805927+0000","last_change":"2024-02-05T07:19:05.594948+0000","last_active":"2024-02-05T07:20:04.805927+0000","last_peered":"2024-02-05T07:20:04.805927+0000","last_clean":"2024-02-05T07:20:04.805927+0000","last_became_active":"2024-02-05T07:18:39.444898+0000","last_became_peered":"2024-02-05T07:18:39.444898+0000","last_unstale":"2024-02-05T07:20:04.805927+0000","last_undegraded":"2024-02-05T07:20:04.805927+0000","last_fullsized":"2024-02-05T07:20:04.805927+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-02-05T07:19:05.594812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:19:05.594812+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.994559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826034+0000","last_change":"2024-02-05T07:18:49.439534+0000","last_active":"2024-02-05T07:20:04.826034+0000","last_peered":"2024-02-05T07:20:04.826034+0000","last_clean":"2024-02-05T07:20:04.826034+0000","last_became_active":"2024-02-05T07:18:49.439249+0000","last_became_peered":"2024-02-05T07:18:49.439249+0000","last_unstale":"2024-02-05T07:20:04.826034+0000","last_undegraded":"2024-02-05T07:20:04.826034+0000","last_fullsized":"2024-02-05T07:20:04.826034+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.489880+0000","last_change":"2024-02-05T07:18:51.508007+0000","last_active":"2024-02-05T07:20:04.489880+0000","last_peered":"2024-02-05T07:20:04.489880+0000","last_clean":"2024-02-05T07:20:04.489880+0000","last_became_active":"2024-02-05T07:18:51.507714+0000","last_became_peered":"2024-02-05T07:18:51.507714+0000","last_unstale":"2024-02-05T07:20:04.489880+0000","last_undegraded":"2024-02-05T07:20:04.489880+0000","last_fullsized":"2024-02-05T07:20:04.489880+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.846104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806605+0000","last_change":"2024-02-05T07:18:53.460344+0000","last_active":"2024-02-05T07:20:04.806605+0000","last_peered":"2024-02-05T07:20:04.806605+0000","last_clean":"2024-02-05T07:20:04.806605+0000","last_became_active":"2024-02-05T07:18:53.459971+0000","last_became_peered":"2024-02-05T07:18:53.459971+0000","last_unstale":"2024-02-05T07:20:04.806605+0000","last_undegraded":"2024-02-05T07:20:04.806605+0000","last_fullsized":"2024-02-05T07:20:04.806605+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:13.408456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103426+0000","last_change":"2024-02-05T07:18:55.473808+0000","last_active":"2024-02-05T07:20:04.103426+0000","last_peered":"2024-02-05T07:20:04.103426+0000","last_clean":"2024-02-05T07:20:04.103426+0000","last_became_active":"2024-02-05T07:18:55.473509+0000","last_became_peered":"2024-02-05T07:18:55.473509+0000","last_unstale":"2024-02-05T07:20:04.103426+0000","last_undegraded":"2024-02-05T07:20:04.103426+0000","last_fullsized":"2024-02-05T07:20:04.103426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:22.190977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103258+0000","last_change":"2024-02-05T07:18:49.434806+0000","last_active":"2024-02-05T07:20:04.103258+0000","last_peered":"2024-02-05T07:20:04.103258+0000","last_clean":"2024-02-05T07:20:04.103258+0000","last_became_active":"2024-02-05T07:18:49.434400+0000","last_became_peered":"2024-02-05T07:18:49.434400+0000","last_unstale":"2024-02-05T07:20:04.103258+0000","last_undegraded":"2024-02-05T07:20:04.103258+0000","last_fullsized":"2024-02-05T07:20:04.103258+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":139,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:08.140891+0000","last_change":"2024-02-05T07:18:52.450728+0000","last_active":"2024-02-05T07:20:08.140891+0000","last_peered":"2024-02-05T07:20:08.140891+0000","last_clean":"2024-02-05T07:20:08.140891+0000","last_became_active":"2024-02-05T07:18:48.424620+0000","last_became_peered":"2024-02-05T07:18:48.424620+0000","last_unstale":"2024-02-05T07:20:08.140891+0000","last_undegraded":"2024-02-05T07:20:08.140891+0000","last_fullsized":"2024-02-05T07:20:08.140891+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:18:47.405257+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:12.026892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103815+0000","last_change":"2024-02-05T07:18:51.442488+0000","last_active":"2024-02-05T07:20:04.103815+0000","last_peered":"2024-02-05T07:20:04.103815+0000","last_clean":"2024-02-05T07:20:04.103815+0000","last_became_active":"2024-02-05T07:18:51.442184+0000","last_became_peered":"2024-02-05T07:18:51.442184+0000","last_unstale":"2024-02-05T07:20:04.103815+0000","last_undegraded":"2024-02-05T07:20:04.103815+0000","last_fullsized":"2024-02-05T07:20:04.103815+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106719+0000","last_change":"2024-02-05T07:18:53.468712+0000","last_active":"2024-02-05T07:20:04.106719+0000","last_peered":"2024-02-05T07:20:04.106719+0000","last_clean":"2024-02-05T07:20:04.106719+0000","last_became_active":"2024-02-05T07:18:53.468512+0000","last_became_peered":"2024-02-05T07:18:53.468512+0000","last_unstale":"2024-02-05T07:20:04.106719+0000","last_undegraded":"2024-02-05T07:20:04.106719+0000","last_fullsized":"2024-02-05T07:20:04.106719+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.283544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.102894+0000","last_change":"2024-02-05T07:18:55.505742+0000","last_active":"2024-02-05T07:20:04.102894+0000","last_peered":"2024-02-05T07:20:04.102894+0000","last_clean":"2024-02-05T07:20:04.102894+0000","last_became_active":"2024-02-05T07:18:55.505360+0000","last_became_peered":"2024-02-05T07:18:55.505360+0000","last_unstale":"2024-02-05T07:20:04.102894+0000","last_undegraded":"2024-02-05T07:20:04.102894+0000","last_fullsized":"2024-02-05T07:20:04.102894+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.103681+0000","last_change":"2024-02-05T07:18:49.428750+0000","last_active":"2024-02-05T07:20:04.103681+0000","last_peered":"2024-02-05T07:20:04.103681+0000","last_clean":"2024-02-05T07:20:04.103681+0000","last_became_active":"2024-02-05T07:18:49.428470+0000","last_became_peered":"2024-02-05T07:18:49.428470+0000","last_unstale":"2024-02-05T07:20:04.103681+0000","last_undegraded":"2024-02-05T07:20:04.103681+0000","last_fullsized":"2024-02-05T07:20:04.103681+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105124+0000","last_change":"2024-02-05T07:18:51.507836+0000","last_active":"2024-02-05T07:20:04.105124+0000","last_peered":"2024-02-05T07:20:04.105124+0000","last_clean":"2024-02-05T07:20:04.105124+0000","last_became_active":"2024-02-05T07:18:51.507507+0000","last_became_peered":"2024-02-05T07:18:51.507507+0000","last_unstale":"2024-02-05T07:20:04.105124+0000","last_undegraded":"2024-02-05T07:20:04.105124+0000","last_fullsized":"2024-02-05T07:20:04.105124+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.300896+0000","last_change":"2024-02-05T07:19:56.330173+0000","last_active":"2024-02-05T07:20:04.300896+0000","last_peered":"2024-02-05T07:20:04.300896+0000","last_clean":"2024-02-05T07:20:04.300896+0000","last_became_active":"2024-02-05T07:18:53.459474+0000","last_became_peered":"2024-02-05T07:18:53.459474+0000","last_unstale":"2024-02-05T07:20:04.300896+0000","last_undegraded":"2024-02-05T07:20:04.300896+0000","last_fullsized":"2024-02-05T07:20:04.300896+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:56.330068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:19:56.330068+0000","last_clean_scrub_stamp":"2024-02-05T07:19:56.330068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:22.066765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.102416+0000","last_change":"2024-02-05T07:18:55.474472+0000","last_active":"2024-02-05T07:20:04.102416+0000","last_peered":"2024-02-05T07:20:04.102416+0000","last_clean":"2024-02-05T07:20:04.102416+0000","last_became_active":"2024-02-05T07:18:55.474029+0000","last_became_peered":"2024-02-05T07:18:55.474029+0000","last_unstale":"2024-02-05T07:20:04.102416+0000","last_undegraded":"2024-02-05T07:20:04.102416+0000","last_fullsized":"2024-02-05T07:20:04.102416+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:16.669709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106371+0000","last_change":"2024-02-05T07:18:49.435665+0000","last_active":"2024-02-05T07:20:04.106371+0000","last_peered":"2024-02-05T07:20:04.106371+0000","last_clean":"2024-02-05T07:20:04.106371+0000","last_became_active":"2024-02-05T07:18:49.435299+0000","last_became_peered":"2024-02-05T07:18:49.435299+0000","last_unstale":"2024-02-05T07:20:04.106371+0000","last_undegraded":"2024-02-05T07:20:04.106371+0000","last_fullsized":"2024-02-05T07:20:04.106371+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.603669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.104480+0000","last_change":"2024-02-05T07:18:51.448399+0000","last_active":"2024-02-05T07:20:04.104480+0000","last_peered":"2024-02-05T07:20:04.104480+0000","last_clean":"2024-02-05T07:20:04.104480+0000","last_became_active":"2024-02-05T07:18:51.448044+0000","last_became_peered":"2024-02-05T07:18:51.448044+0000","last_unstale":"2024-02-05T07:20:04.104480+0000","last_undegraded":"2024-02-05T07:20:04.104480+0000","last_fullsized":"2024-02-05T07:20:04.104480+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.940142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.489451+0000","last_change":"2024-02-05T07:18:53.463629+0000","last_active":"2024-02-05T07:20:04.489451+0000","last_peered":"2024-02-05T07:20:04.489451+0000","last_clean":"2024-02-05T07:20:04.489451+0000","last_became_active":"2024-02-05T07:18:53.462890+0000","last_became_peered":"2024-02-05T07:18:53.462890+0000","last_unstale":"2024-02-05T07:20:04.489451+0000","last_undegraded":"2024-02-05T07:20:04.489451+0000","last_fullsized":"2024-02-05T07:20:04.489451+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.827616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.806343+0000","last_change":"2024-02-05T07:18:55.508071+0000","last_active":"2024-02-05T07:20:04.806343+0000","last_peered":"2024-02-05T07:20:04.806343+0000","last_clean":"2024-02-05T07:20:04.806343+0000","last_became_active":"2024-02-05T07:18:55.507791+0000","last_became_peered":"2024-02-05T07:18:55.507791+0000","last_unstale":"2024-02-05T07:20:04.806343+0000","last_undegraded":"2024-02-05T07:20:04.806343+0000","last_fullsized":"2024-02-05T07:20:04.806343+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:24.052431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.105110+0000","last_change":"2024-02-05T07:19:51.364668+0000","last_active":"2024-02-05T07:20:04.105110+0000","last_peered":"2024-02-05T07:20:04.105110+0000","last_clean":"2024-02-05T07:20:04.105110+0000","last_became_active":"2024-02-05T07:18:49.430123+0000","last_became_peered":"2024-02-05T07:18:49.430123+0000","last_unstale":"2024-02-05T07:20:04.105110+0000","last_undegraded":"2024-02-05T07:20:04.105110+0000","last_fullsized":"2024-02-05T07:20:04.105110+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:19:51.364585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.364585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:57.656562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828254+0000","last_change":"2024-02-05T07:18:51.438840+0000","last_active":"2024-02-05T07:20:04.828254+0000","last_peered":"2024-02-05T07:20:04.828254+0000","last_clean":"2024-02-05T07:20:04.828254+0000","last_became_active":"2024-02-05T07:18:51.438507+0000","last_became_peered":"2024-02-05T07:18:51.438507+0000","last_unstale":"2024-02-05T07:20:04.828254+0000","last_undegraded":"2024-02-05T07:20:04.828254+0000","last_fullsized":"2024-02-05T07:20:04.828254+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:18.455164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825940+0000","last_change":"2024-02-05T07:18:53.464137+0000","last_active":"2024-02-05T07:20:04.825940+0000","last_peered":"2024-02-05T07:20:04.825940+0000","last_clean":"2024-02-05T07:20:04.825940+0000","last_became_active":"2024-02-05T07:18:53.463651+0000","last_became_peered":"2024-02-05T07:18:53.463651+0000","last_unstale":"2024-02-05T07:20:04.825940+0000","last_undegraded":"2024-02-05T07:20:04.825940+0000","last_fullsized":"2024-02-05T07:20:04.825940+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:18.813121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828117+0000","last_change":"2024-02-05T07:18:55.474782+0000","last_active":"2024-02-05T07:20:04.828117+0000","last_peered":"2024-02-05T07:20:04.828117+0000","last_clean":"2024-02-05T07:20:04.828117+0000","last_became_active":"2024-02-05T07:18:55.474447+0000","last_became_peered":"2024-02-05T07:18:55.474447+0000","last_unstale":"2024-02-05T07:20:04.828117+0000","last_undegraded":"2024-02-05T07:20:04.828117+0000","last_fullsized":"2024-02-05T07:20:04.828117+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:09.902392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826459+0000","last_change":"2024-02-05T07:18:49.435071+0000","last_active":"2024-02-05T07:20:04.826459+0000","last_peered":"2024-02-05T07:20:04.826459+0000","last_clean":"2024-02-05T07:20:04.826459+0000","last_became_active":"2024-02-05T07:18:49.434719+0000","last_became_peered":"2024-02-05T07:18:49.434719+0000","last_unstale":"2024-02-05T07:20:04.826459+0000","last_undegraded":"2024-02-05T07:20:04.826459+0000","last_fullsized":"2024-02-05T07:20:04.826459+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.568029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825770+0000","last_change":"2024-02-05T07:20:00.797048+0000","last_active":"2024-02-05T07:20:04.825770+0000","last_peered":"2024-02-05T07:20:04.825770+0000","last_clean":"2024-02-05T07:20:04.825770+0000","last_became_active":"2024-02-05T07:18:51.507524+0000","last_became_peered":"2024-02-05T07:18:51.507524+0000","last_unstale":"2024-02-05T07:20:04.825770+0000","last_undegraded":"2024-02-05T07:20:04.825770+0000","last_fullsized":"2024-02-05T07:20:04.825770+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:20:00.796905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.796905+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:13.158158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828000+0000","last_change":"2024-02-05T07:18:53.461489+0000","last_active":"2024-02-05T07:20:04.828000+0000","last_peered":"2024-02-05T07:20:04.828000+0000","last_clean":"2024-02-05T07:20:04.828000+0000","last_became_active":"2024-02-05T07:18:53.461213+0000","last_became_peered":"2024-02-05T07:18:53.461213+0000","last_unstale":"2024-02-05T07:20:04.828000+0000","last_undegraded":"2024-02-05T07:20:04.828000+0000","last_fullsized":"2024-02-05T07:20:04.828000+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:08.272501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106085+0000","last_change":"2024-02-05T07:18:55.472546+0000","last_active":"2024-02-05T07:20:04.106085+0000","last_peered":"2024-02-05T07:20:04.106085+0000","last_clean":"2024-02-05T07:20:04.106085+0000","last_became_active":"2024-02-05T07:18:55.472172+0000","last_became_peered":"2024-02-05T07:18:55.472172+0000","last_unstale":"2024-02-05T07:20:04.106085+0000","last_undegraded":"2024-02-05T07:20:04.106085+0000","last_fullsized":"2024-02-05T07:20:04.106085+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828954+0000","last_change":"2024-02-05T07:18:53.460159+0000","last_active":"2024-02-05T07:20:04.828954+0000","last_peered":"2024-02-05T07:20:04.828954+0000","last_clean":"2024-02-05T07:20:04.828954+0000","last_became_active":"2024-02-05T07:18:53.459832+0000","last_became_peered":"2024-02-05T07:18:53.459832+0000","last_unstale":"2024-02-05T07:20:04.828954+0000","last_undegraded":"2024-02-05T07:20:04.828954+0000","last_fullsized":"2024-02-05T07:20:04.828954+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:17.337266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.825761+0000","last_change":"2024-02-05T07:18:49.431779+0000","last_active":"2024-02-05T07:20:04.825761+0000","last_peered":"2024-02-05T07:20:04.825761+0000","last_clean":"2024-02-05T07:20:04.825761+0000","last_became_active":"2024-02-05T07:18:49.431369+0000","last_became_peered":"2024-02-05T07:18:49.431369+0000","last_unstale":"2024-02-05T07:20:04.825761+0000","last_undegraded":"2024-02-05T07:20:04.825761+0000","last_fullsized":"2024-02-05T07:20:04.825761+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:07.014547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.104621+0000","last_change":"2024-02-05T07:18:51.447767+0000","last_active":"2024-02-05T07:20:04.104621+0000","last_peered":"2024-02-05T07:20:04.104621+0000","last_clean":"2024-02-05T07:20:04.104621+0000","last_became_active":"2024-02-05T07:18:51.447469+0000","last_became_peered":"2024-02-05T07:18:51.447469+0000","last_unstale":"2024-02-05T07:20:04.104621+0000","last_undegraded":"2024-02-05T07:20:04.104621+0000","last_fullsized":"2024-02-05T07:20:04.104621+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.124394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.826160+0000","last_change":"2024-02-05T07:18:55.506254+0000","last_active":"2024-02-05T07:20:04.826160+0000","last_peered":"2024-02-05T07:20:04.826160+0000","last_clean":"2024-02-05T07:20:04.826160+0000","last_became_active":"2024-02-05T07:18:55.505340+0000","last_became_peered":"2024-02-05T07:18:55.505340+0000","last_unstale":"2024-02-05T07:20:04.826160+0000","last_undegraded":"2024-02-05T07:20:04.826160+0000","last_fullsized":"2024-02-05T07:20:04.826160+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.106708+0000","last_change":"2024-02-05T07:18:53.462879+0000","last_active":"2024-02-05T07:20:04.106708+0000","last_peered":"2024-02-05T07:20:04.106708+0000","last_clean":"2024-02-05T07:20:04.106708+0000","last_became_active":"2024-02-05T07:18:53.462591+0000","last_became_peered":"2024-02-05T07:18:53.462591+0000","last_unstale":"2024-02-05T07:20:04.106708+0000","last_undegraded":"2024-02-05T07:20:04.106708+0000","last_fullsized":"2024-02-05T07:20:04.106708+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:20.782612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:05.883917+0000","last_change":"2024-02-05T07:20:05.883941+0000","last_active":"2024-02-05T07:20:05.883917+0000","last_peered":"2024-02-05T07:20:05.883917+0000","last_clean":"2024-02-05T07:20:05.883917+0000","last_became_active":"2024-02-05T07:18:49.434074+0000","last_became_peered":"2024-02-05T07:18:49.434074+0000","last_unstale":"2024-02-05T07:20:05.883917+0000","last_undegraded":"2024-02-05T07:20:05.883917+0000","last_fullsized":"2024-02-05T07:20:05.883917+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:05.883840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:05.883840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:30.747182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"90'16","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:09.873471+0000","last_change":"2024-02-05T07:20:09.873512+0000","last_active":"2024-02-05T07:20:09.873471+0000","last_peered":"2024-02-05T07:20:09.873471+0000","last_clean":"2024-02-05T07:20:09.873471+0000","last_became_active":"2024-02-05T07:18:51.447922+0000","last_became_peered":"2024-02-05T07:18:51.447922+0000","last_unstale":"2024-02-05T07:20:09.873471+0000","last_undegraded":"2024-02-05T07:20:09.873471+0000","last_fullsized":"2024-02-05T07:20:09.873471+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.873388+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.334535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.629123+0000","last_change":"2024-02-05T07:18:55.475305+0000","last_active":"2024-02-05T07:20:04.629123+0000","last_peered":"2024-02-05T07:20:04.629123+0000","last_clean":"2024-02-05T07:20:04.629123+0000","last_became_active":"2024-02-05T07:18:55.474992+0000","last_became_peered":"2024-02-05T07:18:55.474992+0000","last_unstale":"2024-02-05T07:20:04.629123+0000","last_undegraded":"2024-02-05T07:20:04.629123+0000","last_fullsized":"2024-02-05T07:20:04.629123+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:18.245654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:06.398751+0000","last_change":"2024-02-05T07:20:06.398771+0000","last_active":"2024-02-05T07:20:06.398751+0000","last_peered":"2024-02-05T07:20:06.398751+0000","last_clean":"2024-02-05T07:20:06.398751+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:20:06.398751+0000","last_undegraded":"2024-02-05T07:20:06.398751+0000","last_fullsized":"2024-02-05T07:20:06.398751+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:06.398697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.398697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.245750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.300777+0000","last_change":"2024-02-05T07:18:49.431227+0000","last_active":"2024-02-05T07:20:04.300777+0000","last_peered":"2024-02-05T07:20:04.300777+0000","last_clean":"2024-02-05T07:20:04.300777+0000","last_became_active":"2024-02-05T07:18:49.430518+0000","last_became_peered":"2024-02-05T07:18:49.430518+0000","last_unstale":"2024-02-05T07:20:04.300777+0000","last_undegraded":"2024-02-05T07:20:04.300777+0000","last_fullsized":"2024-02-05T07:20:04.300777+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:14.159632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828840+0000","last_change":"2024-02-05T07:19:51.808315+0000","last_active":"2024-02-05T07:20:04.828840+0000","last_peered":"2024-02-05T07:20:04.828840+0000","last_clean":"2024-02-05T07:20:04.828840+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:20:04.828840+0000","last_undegraded":"2024-02-05T07:20:04.828840+0000","last_fullsized":"2024-02-05T07:20:04.828840+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:19:51.808218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.808218+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:55.279939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828919+0000","last_change":"2024-02-05T07:18:55.466723+0000","last_active":"2024-02-05T07:20:04.828919+0000","last_peered":"2024-02-05T07:20:04.828919+0000","last_clean":"2024-02-05T07:20:04.828919+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:20:04.828919+0000","last_undegraded":"2024-02-05T07:20:04.828919+0000","last_fullsized":"2024-02-05T07:20:04.828919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:13.884922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.629035+0000","last_change":"2024-02-05T07:18:53.462179+0000","last_active":"2024-02-05T07:20:04.629035+0000","last_peered":"2024-02-05T07:20:04.629035+0000","last_clean":"2024-02-05T07:20:04.629035+0000","last_became_active":"2024-02-05T07:18:53.461852+0000","last_became_peered":"2024-02-05T07:18:53.461852+0000","last_unstale":"2024-02-05T07:20:04.629035+0000","last_undegraded":"2024-02-05T07:20:04.629035+0000","last_fullsized":"2024-02-05T07:20:04.629035+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:17.991283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.827237+0000","last_change":"2024-02-05T07:20:04.827259+0000","last_active":"2024-02-05T07:20:04.827237+0000","last_peered":"2024-02-05T07:20:04.827237+0000","last_clean":"2024-02-05T07:20:04.827237+0000","last_became_active":"2024-02-05T07:18:49.435368+0000","last_became_peered":"2024-02-05T07:18:49.435368+0000","last_unstale":"2024-02-05T07:20:04.827237+0000","last_undegraded":"2024-02-05T07:20:04.827237+0000","last_fullsized":"2024-02-05T07:20:04.827237+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:04.827180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:04.827180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:17.606501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-05T07:20:04.828539+0000","last_change":"2024-02-05T07:18:51.443126+0000","last_active":"2024-02-05T07:20:04.828539+0000","last_peered":"2024-02-05T07:20:04.828539+0000","last_clean":"2024-02-05T07:20:04.828539+0000","last_became_active":"2024-02-05T07:18:51.442711+0000","last_became_peered":"2024-02-05T07:18:51.442711+0000","last_unstale":"2024-02-05T07:20:04.828539+0000","last_undegraded":"2024-02-05T07:20:04.828539+0000","last_fullsized":"2024-02-05T07:20:04.828539+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:06.627110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":402,"ondisk_log_size":402,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037781,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28388,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716764,"statfs":{"total":95995035648,"available":95965966336,"internally_reserved":0,"allocated":1404928,"data_stored":944712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.51600000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.57599999999999996}]},{"osd":2,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.47399999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.56100000000000005}]},{"osd":4,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.58499999999999996}]},{"osd":5,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.54200000000000004}]},{"osd":6,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.498},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.66600000000000004}]}]},{"osd":6,"up_from":52,"seq":223338299419,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32536,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712616,"statfs":{"total":95995035648,"available":95961718784,"internally_reserved":0,"allocated":1392640,"data_stored":944933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.629}]},{"osd":1,"last update":"Mon Feb 5 07:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.56200000000000006}]},{"osd":2,"last update":"Mon Feb 5 07:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.442},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.58799999999999997}]},{"osd":3,"last update":"Mon Feb 5 07:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.48599999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.54800000000000004}]},{"osd":5,"last update":"Mon Feb 5 07:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.51000000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.57499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":847872,"data_stored":355525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.35999999999999999,"15min":0.33300000000000002},"min":{"1min":0.24199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60799999999999998,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.316},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.35999999999999999,"15min":0.33200000000000002},"min":{"1min":0.28599999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.442,"15min":0.44600000000000001},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.60799999999999998,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.88,"15min":1.0740000000000001},"min":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.61299999999999999,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.38300000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.45800000000000002,"15min":0.47499999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.63800000000000001,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.47999999999999998,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68100000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.42699999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59099999999999997,"15min":0.58999999999999997},"min":{"1min":0.44600000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.72999999999999998,"15min":0.753},"min":{"1min":0.41899999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.0069999999999999,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.503}]},{"osd":5,"last update":"Mon Feb 5 07:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61699999999999999,"15min":0.622},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.75700000000000001,"15min":0.78100000000000003},"min":{"1min":0.46500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.04,"5min":5.0289999999999999,"15min":5.0289999999999999},"last":0.61199999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":0.47699999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.55400000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28400,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716752,"statfs":{"total":95995035648,"available":95965954048,"internally_reserved":0,"allocated":1413120,"data_stored":944853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.32600000000000001,"15min":0.30399999999999999},"min":{"1min":0.26300000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.50700000000000001,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.31900000000000001},"min":{"1min":0.28399999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.47599999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.439,"15min":0.42799999999999999},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.42799999999999999,"15min":0.42099999999999999},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:20:00 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42899999999999999,"15min":0.43099999999999999},"min":{"1min":0.29899999999999999,"5min":0.246,"15min":0.246},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41399999999999998,"15min":0.41799999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.47599999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66000000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.54900000000000004},"min":{"1min":0.41999999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.61799999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56000000000000005,"15min":0.55400000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.49199999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:19:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.53300000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.504},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.52000000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":802816,"data_stored":354611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.377,"15min":0.36899999999999999},"min":{"1min":0.23499999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.317},{"interface":"front","average":{"1min":0.433,"5min":0.40100000000000002,"15min":0.38600000000000001},"min":{"1min":0.27400000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.248}]},{"osd":1,"last update":"Mon Feb 5 07:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.39100000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.44500000000000001,"15min":0.42699999999999999},"min":{"1min":0.29099999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.39000000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.46700000000000003,"15min":0.46899999999999997},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.56000000000000005,"5min":1.631,"15min":1.631},"last":0.371},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46600000000000003,"15min":0.46400000000000002},"min":{"1min":0.32500000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.55000000000000004,"5min":1.391,"15min":1.391},"last":0.35899999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54100000000000004,"15min":0.53200000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.621},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.56899999999999995,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.47599999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:19:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.61699999999999999,"15min":0.60699999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.49399999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.8319999999999999,"5min":2.8319999999999999,"15min":2.8319999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55900000000000005}]},{"osd":7,"last update":"Mon Feb 5 07:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.63500000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":831488,"data_stored":354197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39300000000000002,"15min":0.40500000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.52500000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.44400000000000001,"15min":0.45700000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.36799999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45000000000000001,"15min":0.46800000000000003},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":0.63500000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48199999999999998,"15min":0.503},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57999999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.45600000000000002,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.61899999999999999,"5min":0.75,"15min":0.75},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.47499999999999998,"15min":0.495},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.40799999999999997}]},{"osd":4,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.58899999999999997,"15min":0.59599999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.70999999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60199999999999998,"15min":0.59599999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.5779999999999998,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":0.64500000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79400000000000004,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59699999999999998,"15min":0.59499999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.57899999999999996}]},{"osd":6,"last update":"Mon Feb 5 07:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.52700000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.60399999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953508,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":819200,"data_stored":353898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.74199999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.65000000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.41599999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.36599999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75700000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71499999999999997,"5min":0.754,"15min":0.754},"last":0.35299999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53300000000000003,"15min":0.54000000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.36399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69899999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.496}]},{"osd":3,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55600000000000005,"15min":0.56100000000000005},"min":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72999999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.67200000000000004,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.42699999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.376,"15min":0.36599999999999999},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.39000000000000001,"15min":0.378},"min":{"1min":0.309,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.57499999999999996,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.315}]},{"osd":6,"last update":"Mon Feb 5 07:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.45300000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.439}]}]},{"osd":5,"up_from":41,"seq":176093659167,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27804,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717348,"statfs":{"total":95995035648,"available":95966564352,"internally_reserved":0,"allocated":806912,"data_stored":354139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66800000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71199999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.374}]},{"osd":1,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.55400000000000005,"15min":0.56299999999999994},"min":{"1min":0.35399999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.69699999999999995,"5min":0.753,"15min":0.753},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.38200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.68200000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.48399999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68000000000000005,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.35099999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69999999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.43099999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64200000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71999999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.40899999999999997}]},{"osd":4,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.35999999999999999,"15min":0.34799999999999998},"min":{"1min":0.26800000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.38900000000000001,"15min":0.373},"min":{"1min":0.33200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.25800000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.41799999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.44400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:20:13.689 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-05T07:20:13.689 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-05T07:20:13.689 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-05T07:20:13.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph health --format=json 2024-02-05T07:20:13.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:13 smithi090 ceph-mon[33213]: 4.3 scrub starts 2024-02-05T07:20:13.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:13 smithi090 ceph-mon[33213]: 4.3 scrub ok 2024-02-05T07:20:13.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:13 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:20:13.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:13 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:20:13.936 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[29098]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:20:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[29098]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[29098]: 3.1c scrub starts 2024-02-05T07:20:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[29098]: 3.1c scrub ok 2024-02-05T07:20:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[29098]: osdmap e91: 8 total, 8 up, 8 in 2024-02-05T07:20:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[29098]: 4.c scrub starts 2024-02-05T07:20:14.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[29098]: 4.c scrub ok 2024-02-05T07:20:14.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:20:14.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[29098]: osdmap e92: 8 total, 8 up, 8 in 2024-02-05T07:20:14.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[37158]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:20:14.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[37158]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:14.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[37158]: 3.1c scrub starts 2024-02-05T07:20:14.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[37158]: 3.1c scrub ok 2024-02-05T07:20:14.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[37158]: osdmap e91: 8 total, 8 up, 8 in 2024-02-05T07:20:14.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[37158]: 4.c scrub starts 2024-02-05T07:20:14.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[37158]: 4.c scrub ok 2024-02-05T07:20:14.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:20:14.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:14 smithi007 ceph-mon[37158]: osdmap e92: 8 total, 8 up, 8 in 2024-02-05T07:20:14.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:14 smithi090 ceph-mon[33213]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:20:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:14 smithi090 ceph-mon[33213]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:14 smithi090 ceph-mon[33213]: 3.1c scrub starts 2024-02-05T07:20:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:14 smithi090 ceph-mon[33213]: 3.1c scrub ok 2024-02-05T07:20:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:14 smithi090 ceph-mon[33213]: osdmap e91: 8 total, 8 up, 8 in 2024-02-05T07:20:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:14 smithi090 ceph-mon[33213]: 4.c scrub starts 2024-02-05T07:20:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:14 smithi090 ceph-mon[33213]: 4.c scrub ok 2024-02-05T07:20:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:14 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:20:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:14 smithi090 ceph-mon[33213]: osdmap e92: 8 total, 8 up, 8 in 2024-02-05T07:20:15.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:15 smithi007 ceph-mon[29098]: 3.f scrub starts 2024-02-05T07:20:15.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:15 smithi007 ceph-mon[29098]: 3.f scrub ok 2024-02-05T07:20:15.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:15 smithi007 ceph-mon[29098]: 4.6 scrub starts 2024-02-05T07:20:15.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:15 smithi007 ceph-mon[29098]: 4.6 scrub ok 2024-02-05T07:20:15.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:15 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:20:15.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:15 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:20:15.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:15 smithi007 ceph-mon[37158]: 3.f scrub starts 2024-02-05T07:20:15.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:15 smithi007 ceph-mon[37158]: 3.f scrub ok 2024-02-05T07:20:15.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:15 smithi007 ceph-mon[37158]: 4.6 scrub starts 2024-02-05T07:20:15.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:15 smithi007 ceph-mon[37158]: 4.6 scrub ok 2024-02-05T07:20:15.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:15 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:20:15.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:15 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:20:15.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:15 smithi090 ceph-mon[33213]: 3.f scrub starts 2024-02-05T07:20:15.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:15 smithi090 ceph-mon[33213]: 3.f scrub ok 2024-02-05T07:20:15.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:15 smithi090 ceph-mon[33213]: 4.6 scrub starts 2024-02-05T07:20:15.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:15 smithi090 ceph-mon[33213]: 4.6 scrub ok 2024-02-05T07:20:15.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:15 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:20:15.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:15 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:20:15.827 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:20:15.827 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-05T07:20:16.584 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-05T07:20:16.585 INFO:tasks.cephadm:Setup complete, yielding 2024-02-05T07:20:16.585 INFO:teuthology.run_tasks:Running task thrashosds... 2024-02-05T07:20:16.599 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-05T07:20:16.599 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-testing-3-2024-02-04-2254'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-guits-testing-3-2024-02-04-2254', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'}} 2024-02-05T07:20:16.600 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-05T07:20:16.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-02-05T07:20:16.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:16 smithi090 ceph-mon[33213]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:16.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:16 smithi090 ceph-mon[33213]: 3.c scrub starts 2024-02-05T07:20:16.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:16 smithi090 ceph-mon[33213]: 3.c scrub ok 2024-02-05T07:20:16.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:16 smithi090 ceph-mon[33213]: osdmap e93: 8 total, 8 up, 8 in 2024-02-05T07:20:16.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:16 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1308886487' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-05T07:20:16.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:16 smithi007 ceph-mon[29098]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:16.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:16 smithi007 ceph-mon[29098]: 3.c scrub starts 2024-02-05T07:20:16.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:16 smithi007 ceph-mon[29098]: 3.c scrub ok 2024-02-05T07:20:16.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:16 smithi007 ceph-mon[29098]: osdmap e93: 8 total, 8 up, 8 in 2024-02-05T07:20:16.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:16 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1308886487' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-05T07:20:16.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:16 smithi007 ceph-mon[37158]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:16.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:16 smithi007 ceph-mon[37158]: 3.c scrub starts 2024-02-05T07:20:16.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:16 smithi007 ceph-mon[37158]: 3.c scrub ok 2024-02-05T07:20:16.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:16 smithi007 ceph-mon[37158]: osdmap e93: 8 total, 8 up, 8 in 2024-02-05T07:20:16.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:16 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1308886487' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-05T07:20:16.848 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:17.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:17 smithi090 ceph-mon[33213]: 5.b scrub starts 2024-02-05T07:20:17.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:17 smithi090 ceph-mon[33213]: 5.b scrub ok 2024-02-05T07:20:17.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:17 smithi007 ceph-mon[29098]: 5.b scrub starts 2024-02-05T07:20:17.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:17 smithi007 ceph-mon[29098]: 5.b scrub ok 2024-02-05T07:20:17.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:17 smithi007 ceph-mon[37158]: 5.b scrub starts 2024-02-05T07:20:17.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:17 smithi007 ceph-mon[37158]: 5.b scrub ok 2024-02-05T07:20:18.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:18 smithi007 ceph-mon[29098]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:20:18.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:18 smithi007 ceph-mon[37158]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:20:18.674 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:20:18 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[71934]: ts=2024-02-05T07:20:18.270Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.006559636s 2024-02-05T07:20:18.764 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:18 smithi090 ceph-mon[33213]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:20:19.013 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:18 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:20:18] ENGINE Bus STOPPING 2024-02-05T07:20:19.013 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:18 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:20:18] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T07:20:19.013 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:18 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:20:18] ENGINE Bus STOPPED 2024-02-05T07:20:19.013 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:18 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:20:18] ENGINE Bus STARTING 2024-02-05T07:20:19.013 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:18 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:20:18] ENGINE Serving on http://:::9283 2024-02-05T07:20:19.014 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:18 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:20:18] ENGINE Bus STARTED 2024-02-05T07:20:19.443 INFO:tasks.thrashosds:Beginning thrashosds... 2024-02-05T07:20:19.443 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-05T07:20:19.443 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:20:19.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[29098]: 6.1c scrub starts 2024-02-05T07:20:19.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[29098]: 6.1c scrub ok 2024-02-05T07:20:19.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[29098]: 3.a scrub starts 2024-02-05T07:20:19.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[29098]: 3.a scrub ok 2024-02-05T07:20:19.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[29098]: 5.d scrub starts 2024-02-05T07:20:19.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[29098]: 5.d scrub ok 2024-02-05T07:20:19.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[29098]: 5.1d scrub starts 2024-02-05T07:20:19.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[29098]: 5.1d scrub ok 2024-02-05T07:20:19.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3915841605' entity='client.admin' 2024-02-05T07:20:19.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[29098]: 5.1c scrub starts 2024-02-05T07:20:19.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[29098]: 5.1c scrub ok 2024-02-05T07:20:19.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[29098]: osdmap e94: 8 total, 8 up, 8 in 2024-02-05T07:20:19.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[37158]: 6.1c scrub starts 2024-02-05T07:20:19.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[37158]: 6.1c scrub ok 2024-02-05T07:20:19.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[37158]: 3.a scrub starts 2024-02-05T07:20:19.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[37158]: 3.a scrub ok 2024-02-05T07:20:19.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[37158]: 5.d scrub starts 2024-02-05T07:20:19.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[37158]: 5.d scrub ok 2024-02-05T07:20:19.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[37158]: 5.1d scrub starts 2024-02-05T07:20:19.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[37158]: 5.1d scrub ok 2024-02-05T07:20:19.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3915841605' entity='client.admin' 2024-02-05T07:20:19.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[37158]: 5.1c scrub starts 2024-02-05T07:20:19.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[37158]: 5.1c scrub ok 2024-02-05T07:20:19.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:19 smithi007 ceph-mon[37158]: osdmap e94: 8 total, 8 up, 8 in 2024-02-05T07:20:19.697 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:19.748 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:19 smithi090 ceph-mon[33213]: 6.1c scrub starts 2024-02-05T07:20:19.748 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:19 smithi090 ceph-mon[33213]: 6.1c scrub ok 2024-02-05T07:20:19.748 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:19 smithi090 ceph-mon[33213]: 3.a scrub starts 2024-02-05T07:20:19.748 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:19 smithi090 ceph-mon[33213]: 3.a scrub ok 2024-02-05T07:20:19.748 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:19 smithi090 ceph-mon[33213]: 5.d scrub starts 2024-02-05T07:20:19.748 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:19 smithi090 ceph-mon[33213]: 5.d scrub ok 2024-02-05T07:20:19.748 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:19 smithi090 ceph-mon[33213]: 5.1d scrub starts 2024-02-05T07:20:19.748 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:19 smithi090 ceph-mon[33213]: 5.1d scrub ok 2024-02-05T07:20:19.748 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:19 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3915841605' entity='client.admin' 2024-02-05T07:20:19.749 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:19 smithi090 ceph-mon[33213]: 5.1c scrub starts 2024-02-05T07:20:19.749 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:19 smithi090 ceph-mon[33213]: 5.1c scrub ok 2024-02-05T07:20:19.749 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:19 smithi090 ceph-mon[33213]: osdmap e94: 8 total, 8 up, 8 in 2024-02-05T07:20:20.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:20:19 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:20:20.711 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:20 smithi007 ceph-mon[29098]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:20:20.711 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:20 smithi007 ceph-mon[29098]: 3.17 scrub starts 2024-02-05T07:20:20.711 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:20 smithi007 ceph-mon[29098]: 3.17 scrub ok 2024-02-05T07:20:20.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:20 smithi007 ceph-mon[37158]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:20:20.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:20 smithi007 ceph-mon[37158]: 3.17 scrub starts 2024-02-05T07:20:20.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:20 smithi007 ceph-mon[37158]: 3.17 scrub ok 2024-02-05T07:20:20.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:20 smithi090 ceph-mon[33213]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:20:20.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:20 smithi090 ceph-mon[33213]: 3.17 scrub starts 2024-02-05T07:20:20.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:20 smithi090 ceph-mon[33213]: 3.17 scrub ok 2024-02-05T07:20:21.462 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 systemd[1]: Starting Ceph grafana.a for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:20:21.512 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:20:21.515 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:20:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:21 smithi007 ceph-mon[29098]: 6.15 scrub starts 2024-02-05T07:20:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:21 smithi007 ceph-mon[29098]: 6.15 scrub ok 2024-02-05T07:20:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:21 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:21 smithi007 ceph-mon[29098]: 6.16 scrub starts 2024-02-05T07:20:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:21 smithi007 ceph-mon[29098]: 6.16 scrub ok 2024-02-05T07:20:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:21 smithi007 ceph-mon[29098]: 6.1 scrub starts 2024-02-05T07:20:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:21 smithi007 ceph-mon[29098]: 6.1 scrub ok 2024-02-05T07:20:21.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:21 smithi007 ceph-mon[37158]: 6.15 scrub starts 2024-02-05T07:20:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:21 smithi007 ceph-mon[37158]: 6.15 scrub ok 2024-02-05T07:20:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:21 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:21 smithi007 ceph-mon[37158]: 6.16 scrub starts 2024-02-05T07:20:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:21 smithi007 ceph-mon[37158]: 6.16 scrub ok 2024-02-05T07:20:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:21 smithi007 ceph-mon[37158]: 6.1 scrub starts 2024-02-05T07:20:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:21 smithi007 ceph-mon[37158]: 6.1 scrub ok 2024-02-05T07:20:21.716 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-mon[33213]: 6.15 scrub starts 2024-02-05T07:20:21.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-mon[33213]: 6.15 scrub ok 2024-02-05T07:20:21.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:21.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-mon[33213]: 6.16 scrub starts 2024-02-05T07:20:21.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-mon[33213]: 6.16 scrub ok 2024-02-05T07:20:21.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-mon[33213]: 6.1 scrub starts 2024-02-05T07:20:21.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-mon[33213]: 6.1 scrub ok 2024-02-05T07:20:21.717 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 podman[63427]: 2024-02-05 07:20:21.472527039 +0000 UTC m=+0.019590221 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-02-05T07:20:21.717 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 podman[63427]: 2024-02-05 07:20:21.592948432 +0000 UTC m=+0.140011613 container create dfa4c185051b59779667b9443621e705ee93d165c0a39e0a11739a5fb29d764c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, io.buildah.version=1.32.0, distribution-scope=public, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, com.redhat.component=ubi8-container, io.openshift.expose-services=) 2024-02-05T07:20:21.984 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 podman[63427]: 2024-02-05 07:20:21.821971673 +0000 UTC m=+0.369034854 container init dfa4c185051b59779667b9443621e705ee93d165c0a39e0a11739a5fb29d764c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a, io.openshift.expose-services=, io.openshift.tags=base rhel8, architecture=x86_64, maintainer=Paul Cuzner , distribution-scope=public, build-date=2022-04-29T11:48:11.055897, vcs-type=git, release=239.1651231664, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, io.buildah.version=1.32.0, version=8.5, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-02-05T07:20:21.985 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 podman[63427]: 2024-02-05 07:20:21.824736879 +0000 UTC m=+0.371800063 container start dfa4c185051b59779667b9443621e705ee93d165c0a39e0a11739a5fb29d764c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , architecture=x86_64, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, distribution-scope=public, io.buildah.version=1.32.0, release=239.1651231664, name=ubi8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-02-05T07:20:21.985 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 bash[63427]: dfa4c185051b59779667b9443621e705ee93d165c0a39e0a11739a5fb29d764c 2024-02-05T07:20:21.985 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 systemd[1]: Started Ceph grafana.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:20:21.985 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=settings t=2024-02-05T07:20:21.983209273Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-02-05T07:20:21.985 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=settings t=2024-02-05T07:20:21.983431532Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-02-05T07:20:21.985 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=settings t=2024-02-05T07:20:21.983498847Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-02-05T07:20:21.985 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=settings t=2024-02-05T07:20:21.98352085Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-02-05T07:20:21.985 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=settings t=2024-02-05T07:20:21.98353315Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-05T07:20:21.985 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=settings t=2024-02-05T07:20:21.983543936Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-05T07:20:21.986 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=settings t=2024-02-05T07:20:21.983553039Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-05T07:20:21.986 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=settings t=2024-02-05T07:20:21.983561819Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-05T07:20:21.986 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=settings t=2024-02-05T07:20:21.983570355Z level=info msg="Path Home" path=/usr/share/grafana 2024-02-05T07:20:21.986 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=settings t=2024-02-05T07:20:21.983579097Z level=info msg="Path Data" path=/var/lib/grafana 2024-02-05T07:20:21.986 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=settings t=2024-02-05T07:20:21.983588278Z level=info msg="Path Logs" path=/var/log/grafana 2024-02-05T07:20:21.986 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=settings t=2024-02-05T07:20:21.983597927Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-02-05T07:20:21.986 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=settings t=2024-02-05T07:20:21.983610103Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-02-05T07:20:21.986 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=settings t=2024-02-05T07:20:21.983623095Z level=info msg="App mode production" 2024-02-05T07:20:21.986 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=sqlstore t=2024-02-05T07:20:21.983693073Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-02-05T07:20:21.986 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=sqlstore t=2024-02-05T07:20:21.983724322Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-05T07:20:22.235 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.984885621Z level=info msg="Starting DB migrations" 2024-02-05T07:20:22.235 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.984962123Z level=info msg="Executing migration" id="create migration_log table" 2024-02-05T07:20:22.235 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.985473519Z level=info msg="Executing migration" id="create user table" 2024-02-05T07:20:22.235 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.985915055Z level=info msg="Executing migration" id="add unique index user.login" 2024-02-05T07:20:22.235 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.986396991Z level=info msg="Executing migration" id="add unique index user.email" 2024-02-05T07:20:22.235 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.986834652Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-02-05T07:20:22.236 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.987359231Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-02-05T07:20:22.236 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.987776527Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-02-05T07:20:22.236 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.988507112Z level=info msg="Executing migration" id="create user table v2" 2024-02-05T07:20:22.236 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.988954267Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-02-05T07:20:22.236 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.989341982Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-02-05T07:20:22.236 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.989879832Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-02-05T07:20:22.236 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.990235659Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-02-05T07:20:22.236 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.990561331Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-02-05T07:20:22.237 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.990975059Z level=info msg="Executing migration" id="Update user table charset" 2024-02-05T07:20:22.237 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.991238335Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-02-05T07:20:22.237 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.991655416Z level=info msg="Executing migration" id="Add missing user data" 2024-02-05T07:20:22.237 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.992003461Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-02-05T07:20:22.237 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.992395088Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-02-05T07:20:22.237 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.992825259Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-02-05T07:20:22.237 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.993249814Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-02-05T07:20:22.237 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.994349761Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-02-05T07:20:22.237 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.994843464Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-02-05T07:20:22.237 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.995310501Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-02-05T07:20:22.237 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.995746888Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-02-05T07:20:22.237 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.99616954Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-02-05T07:20:22.237 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.996593451Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-02-05T07:20:22.238 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.996909859Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-02-05T07:20:22.238 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.997277814Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-02-05T07:20:22.238 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.997682593Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-02-05T07:20:22.238 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.998077911Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-02-05T07:20:22.238 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.998446392Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-05T07:20:22.238 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.999103362Z level=info msg="Executing migration" id="create temp_user v2" 2024-02-05T07:20:22.238 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.99949831Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-02-05T07:20:22.238 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:21.999910682Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-02-05T07:20:22.238 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.000302609Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-02-05T07:20:22.238 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.000729463Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-02-05T07:20:22.238 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.00122158Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-02-05T07:20:22.238 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.001998042Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-02-05T07:20:22.240 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.002401064Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-02-05T07:20:22.240 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.002876128Z level=info msg="Executing migration" id="create star table" 2024-02-05T07:20:22.240 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.003770991Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-02-05T07:20:22.240 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.004450183Z level=info msg="Executing migration" id="create org table v1" 2024-02-05T07:20:22.240 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.004903845Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-02-05T07:20:22.240 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.005778754Z level=info msg="Executing migration" id="create org_user table v1" 2024-02-05T07:20:22.240 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.006614992Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-02-05T07:20:22.240 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.00756592Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-02-05T07:20:22.240 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.008258347Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-02-05T07:20:22.240 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.009185946Z level=info msg="Executing migration" id="Update org table charset" 2024-02-05T07:20:22.240 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.009903548Z level=info msg="Executing migration" id="Update org_user table charset" 2024-02-05T07:20:22.240 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.010790896Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-02-05T07:20:22.240 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.01151049Z level=info msg="Executing migration" id="create dashboard table" 2024-02-05T07:20:22.241 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.012491586Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-02-05T07:20:22.241 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.013208344Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-02-05T07:20:22.241 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.014178521Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-02-05T07:20:22.241 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.014910325Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-02-05T07:20:22.241 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.015769245Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-05T07:20:22.241 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.016631597Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-02-05T07:20:22.241 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.017525419Z level=info msg="Executing migration" id="create dashboard v2" 2024-02-05T07:20:22.241 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.018366127Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-02-05T07:20:22.241 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.019345047Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-02-05T07:20:22.241 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.020138608Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-02-05T07:20:22.241 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.021038565Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-02-05T07:20:22.241 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.021541795Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-02-05T07:20:22.241 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.021862271Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-02-05T07:20:22.241 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.02269284Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.023224313Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.023750383Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.024595851Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.025494775Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.026445614Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.029382333Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.032182106Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.035005828Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.037884494Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.040581021Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.043443839Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.046283109Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.049169016Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.052097839Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-02-05T07:20:22.242 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.054979566Z level=info msg="Executing migration" id="Update dashboard title length" 2024-02-05T07:20:22.243 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.057765311Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-05T07:20:22.243 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.060572325Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-02-05T07:20:22.243 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.063347484Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-05T07:20:22.243 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.066035731Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-02-05T07:20:22.243 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.068868619Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-05T07:20:22.243 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.071588145Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-05T07:20:22.243 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.074364966Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-02-05T07:20:22.243 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.077185664Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-02-05T07:20:22.243 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.079927942Z level=info msg="Executing migration" id="Add check_sum column" 2024-02-05T07:20:22.243 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.082751512Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-02-05T07:20:22.243 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.085390519Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-02-05T07:20:22.243 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.08810213Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-02-05T07:20:22.243 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.09082252Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-02-05T07:20:22.243 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.093687498Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-02-05T07:20:22.244 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.096305467Z level=info msg="Executing migration" id="create data_source table" 2024-02-05T07:20:22.244 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.099183352Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-02-05T07:20:22.244 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.101871895Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-02-05T07:20:22.244 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.104695139Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-02-05T07:20:22.244 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.107442667Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-02-05T07:20:22.244 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.110143838Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-02-05T07:20:22.244 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.112769244Z level=info msg="Executing migration" id="create data_source table v2" 2024-02-05T07:20:22.244 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.115495646Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-02-05T07:20:22.244 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.118138066Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-02-05T07:20:22.244 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.120926699Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-02-05T07:20:22.244 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.123768976Z level=info msg="Executing migration" id="Add column with_credentials" 2024-02-05T07:20:22.244 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.124485043Z level=info msg="Executing migration" id="Add secure json data column" 2024-02-05T07:20:22.244 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.125326363Z level=info msg="Executing migration" id="Update data_source table charset" 2024-02-05T07:20:22.244 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.126292482Z level=info msg="Executing migration" id="Update initial version to 1" 2024-02-05T07:20:22.245 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.127000047Z level=info msg="Executing migration" id="Add read_only data column" 2024-02-05T07:20:22.245 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.128011666Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-02-05T07:20:22.245 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.128697901Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-02-05T07:20:22.245 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.129564925Z level=info msg="Executing migration" id="Add uid column" 2024-02-05T07:20:22.245 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.130419023Z level=info msg="Executing migration" id="Update uid value" 2024-02-05T07:20:22.245 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.131251888Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-02-05T07:20:22.245 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.13213199Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-02-05T07:20:22.245 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.133047307Z level=info msg="Executing migration" id="create api_key table" 2024-02-05T07:20:22.245 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.133754219Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-02-05T07:20:22.245 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.134547218Z level=info msg="Executing migration" id="add index api_key.key" 2024-02-05T07:20:22.245 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.135396459Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-02-05T07:20:22.245 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.136273745Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-02-05T07:20:22.245 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.136940623Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-02-05T07:20:22.245 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.137370797Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.138151444Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.139221093Z level=info msg="Executing migration" id="create api_key table v2" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.140078917Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.14055249Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.141435253Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.142259869Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.143220144Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.143791059Z level=info msg="Executing migration" id="Update api_key table charset" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.144634088Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.145555577Z level=info msg="Executing migration" id="Add service account foreign key" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.146526675Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.146952684Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.147813076Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.14843796Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-02-05T07:20:22.246 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.148921085Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-02-05T07:20:22.247 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.149804459Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-02-05T07:20:22.247 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.150304408Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-02-05T07:20:22.247 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.150759761Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-05T07:20:22.247 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.153701957Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-05T07:20:22.247 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.156381686Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-02-05T07:20:22.247 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.15930514Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-02-05T07:20:22.247 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.159760591Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-02-05T07:20:22.247 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.162767392Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-02-05T07:20:22.247 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.163397545Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-05T07:20:22.247 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.165601918Z level=info msg="Executing migration" id="create quota table v1" 2024-02-05T07:20:22.248 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.168332204Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-05T07:20:22.248 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.171328772Z level=info msg="Executing migration" id="Update quota table charset" 2024-02-05T07:20:22.248 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.174027599Z level=info msg="Executing migration" id="create plugin_setting table" 2024-02-05T07:20:22.248 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.17671025Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-05T07:20:22.248 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.179526609Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-02-05T07:20:22.248 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.180155499Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-02-05T07:20:22.248 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.182927447Z level=info msg="Executing migration" id="create session table" 2024-02-05T07:20:22.248 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.183384781Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-02-05T07:20:22.248 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.185843901Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-02-05T07:20:22.248 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.188712578Z level=info msg="Executing migration" id="create playlist table v2" 2024-02-05T07:20:22.248 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.189301492Z level=info msg="Executing migration" id="create playlist item table v2" 2024-02-05T07:20:22.248 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.192122814Z level=info msg="Executing migration" id="Update playlist table charset" 2024-02-05T07:20:22.249 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.194774279Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-02-05T07:20:22.249 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.197618627Z level=info msg="Executing migration" id="drop preferences table v2" 2024-02-05T07:20:22.249 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.197928112Z level=info msg="Executing migration" id="drop preferences table v3" 2024-02-05T07:20:22.249 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.200686792Z level=info msg="Executing migration" id="create preferences table v3" 2024-02-05T07:20:22.249 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.201134562Z level=info msg="Executing migration" id="Update preferences table charset" 2024-02-05T07:20:22.249 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.203392925Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-02-05T07:20:22.249 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.204083857Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-02-05T07:20:22.249 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.20607398Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-02-05T07:20:22.249 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.20672126Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-02-05T07:20:22.249 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.207347605Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-02-05T07:20:22.249 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.210253833Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-02-05T07:20:22.249 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.213026786Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-02-05T07:20:22.250 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.215584962Z level=info msg="Executing migration" id="create alert table v1" 2024-02-05T07:20:22.250 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.216078264Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-02-05T07:20:22.250 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.218851154Z level=info msg="Executing migration" id="add index alert state" 2024-02-05T07:20:22.250 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.221650272Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-02-05T07:20:22.250 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.222127586Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-02-05T07:20:22.250 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.222513814Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-05T07:20:22.250 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.222969095Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-05T07:20:22.250 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.223349199Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-05T07:20:22.250 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.224802157Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-02-05T07:20:22.250 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.225197913Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-05T07:20:22.250 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.225640955Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-02-05T07:20:22.250 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.22595753Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-02-05T07:20:22.250 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.226275173Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-02-05T07:20:22.250 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.226694288Z level=info msg="Executing migration" id="Add column is_default" 2024-02-05T07:20:22.251 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.227306185Z level=info msg="Executing migration" id="Add column frequency" 2024-02-05T07:20:22.251 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.227978515Z level=info msg="Executing migration" id="Add column send_reminder" 2024-02-05T07:20:22.251 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.228642785Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-02-05T07:20:22.251 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.229293412Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-02-05T07:20:22.251 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.229719555Z level=info msg="Executing migration" id="Update alert table charset" 2024-02-05T07:20:22.251 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.229997946Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-02-05T07:20:22.251 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.230255313Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-02-05T07:20:22.251 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.230644981Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-05T07:20:22.251 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.231067889Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-02-05T07:20:22.251 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.231412376Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-02-05T07:20:22.251 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.231831351Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-05T07:20:22.251 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.232204303Z level=info msg="Executing migration" id="Add for to alert table" 2024-02-05T07:20:22.251 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.232854036Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-02-05T07:20:22.252 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.233470843Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-02-05T07:20:22.252 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.233795908Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-02-05T07:20:22.252 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.234249421Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-02-05T07:20:22.252 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.234696454Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-02-05T07:20:22.252 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.235404057Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-02-05T07:20:22.252 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.23574598Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-02-05T07:20:22.252 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.236183909Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-02-05T07:20:22.252 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.236652141Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-02-05T07:20:22.253 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.236996683Z level=info msg="Executing migration" id="create annotation table v5" 2024-02-05T07:20:22.253 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.237454065Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-02-05T07:20:22.256 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-02-05T07:20:21.462026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1017,"num_read_kb":863,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":610,"ondisk_log_size":610,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":237120,"kb_used_data":8484,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724096,"statfs":{"total":767960285184,"available":767717474304,"internally_reserved":0,"allocated":8687616,"data_stored":4801438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.542052"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107306+0000","last_change":"2024-02-05T07:18:55.466290+0000","last_active":"2024-02-05T07:20:19.107306+0000","last_peered":"2024-02-05T07:20:19.107306+0000","last_clean":"2024-02-05T07:20:19.107306+0000","last_became_active":"2024-02-05T07:18:55.466001+0000","last_became_peered":"2024-02-05T07:18:55.466001+0000","last_unstale":"2024-02-05T07:20:19.107306+0000","last_undegraded":"2024-02-05T07:20:19.107306+0000","last_fullsized":"2024-02-05T07:20:19.107306+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106942+0000","last_change":"2024-02-05T07:18:53.461117+0000","last_active":"2024-02-05T07:20:19.106942+0000","last_peered":"2024-02-05T07:20:19.106942+0000","last_clean":"2024-02-05T07:20:19.106942+0000","last_became_active":"2024-02-05T07:18:53.460820+0000","last_became_peered":"2024-02-05T07:18:53.460820+0000","last_unstale":"2024-02-05T07:20:19.106942+0000","last_undegraded":"2024-02-05T07:20:19.106942+0000","last_fullsized":"2024-02-05T07:20:19.106942+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264513+0000","last_change":"2024-02-05T07:18:49.434741+0000","last_active":"2024-02-05T07:20:19.264513+0000","last_peered":"2024-02-05T07:20:19.264513+0000","last_clean":"2024-02-05T07:20:19.264513+0000","last_became_active":"2024-02-05T07:18:49.434456+0000","last_became_peered":"2024-02-05T07:18:49.434456+0000","last_unstale":"2024-02-05T07:20:19.264513+0000","last_undegraded":"2024-02-05T07:20:19.264513+0000","last_fullsized":"2024-02-05T07:20:19.264513+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107210+0000","last_change":"2024-02-05T07:18:51.449261+0000","last_active":"2024-02-05T07:20:19.107210+0000","last_peered":"2024-02-05T07:20:19.107210+0000","last_clean":"2024-02-05T07:20:19.107210+0000","last_became_active":"2024-02-05T07:18:51.448924+0000","last_became_peered":"2024-02-05T07:18:51.448924+0000","last_unstale":"2024-02-05T07:20:19.107210+0000","last_undegraded":"2024-02-05T07:20:19.107210+0000","last_fullsized":"2024-02-05T07:20:19.107210+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106916+0000","last_change":"2024-02-05T07:18:55.506169+0000","last_active":"2024-02-05T07:20:19.106916+0000","last_peered":"2024-02-05T07:20:19.106916+0000","last_clean":"2024-02-05T07:20:19.106916+0000","last_became_active":"2024-02-05T07:18:55.505718+0000","last_became_peered":"2024-02-05T07:18:55.505718+0000","last_unstale":"2024-02-05T07:20:19.106916+0000","last_undegraded":"2024-02-05T07:20:19.106916+0000","last_fullsized":"2024-02-05T07:20:19.106916+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.751562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107221+0000","last_change":"2024-02-05T07:19:54.746304+0000","last_active":"2024-02-05T07:20:19.107221+0000","last_peered":"2024-02-05T07:20:19.107221+0000","last_clean":"2024-02-05T07:20:19.107221+0000","last_became_active":"2024-02-05T07:18:53.461506+0000","last_became_peered":"2024-02-05T07:18:53.461506+0000","last_unstale":"2024-02-05T07:20:19.107221+0000","last_undegraded":"2024-02-05T07:20:19.107221+0000","last_fullsized":"2024-02-05T07:20:19.107221+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:54.746206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:19:54.746206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:57.230561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107547+0000","last_change":"2024-02-05T07:18:49.436786+0000","last_active":"2024-02-05T07:20:19.107547+0000","last_peered":"2024-02-05T07:20:19.107547+0000","last_clean":"2024-02-05T07:20:19.107547+0000","last_became_active":"2024-02-05T07:18:49.436490+0000","last_became_peered":"2024-02-05T07:18:49.436490+0000","last_unstale":"2024-02-05T07:20:19.107547+0000","last_undegraded":"2024-02-05T07:20:19.107547+0000","last_fullsized":"2024-02-05T07:20:19.107547+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106497+0000","last_change":"2024-02-05T07:18:51.445160+0000","last_active":"2024-02-05T07:20:19.106497+0000","last_peered":"2024-02-05T07:20:19.106497+0000","last_clean":"2024-02-05T07:20:19.106497+0000","last_became_active":"2024-02-05T07:18:51.444783+0000","last_became_peered":"2024-02-05T07:18:51.444783+0000","last_unstale":"2024-02-05T07:20:19.106497+0000","last_undegraded":"2024-02-05T07:20:19.106497+0000","last_fullsized":"2024-02-05T07:20:19.106497+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.263859+0000","last_change":"2024-02-05T07:18:55.477022+0000","last_active":"2024-02-05T07:20:19.263859+0000","last_peered":"2024-02-05T07:20:19.263859+0000","last_clean":"2024-02-05T07:20:19.263859+0000","last_became_active":"2024-02-05T07:18:55.474002+0000","last_became_peered":"2024-02-05T07:18:55.474002+0000","last_unstale":"2024-02-05T07:20:19.263859+0000","last_undegraded":"2024-02-05T07:20:19.263859+0000","last_fullsized":"2024-02-05T07:20:19.263859+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324626+0000","last_change":"2024-02-05T07:18:53.464788+0000","last_active":"2024-02-05T07:20:19.324626+0000","last_peered":"2024-02-05T07:20:19.324626+0000","last_clean":"2024-02-05T07:20:19.324626+0000","last_became_active":"2024-02-05T07:18:53.464520+0000","last_became_peered":"2024-02-05T07:18:53.464520+0000","last_unstale":"2024-02-05T07:20:19.324626+0000","last_undegraded":"2024-02-05T07:20:19.324626+0000","last_fullsized":"2024-02-05T07:20:19.324626+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.843776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324814+0000","last_change":"2024-02-05T07:19:59.782950+0000","last_active":"2024-02-05T07:20:19.324814+0000","last_peered":"2024-02-05T07:20:19.324814+0000","last_clean":"2024-02-05T07:20:19.324814+0000","last_became_active":"2024-02-05T07:18:49.435454+0000","last_became_peered":"2024-02-05T07:18:49.435454+0000","last_unstale":"2024-02-05T07:20:19.324814+0000","last_undegraded":"2024-02-05T07:20:19.324814+0000","last_fullsized":"2024-02-05T07:20:19.324814+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:59.782841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:59.782841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.410088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"89'13","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107477+0000","last_change":"2024-02-05T07:20:02.798288+0000","last_active":"2024-02-05T07:20:19.107477+0000","last_peered":"2024-02-05T07:20:19.107477+0000","last_clean":"2024-02-05T07:20:19.107477+0000","last_became_active":"2024-02-05T07:18:51.444592+0000","last_became_peered":"2024-02-05T07:18:51.444592+0000","last_unstale":"2024-02-05T07:20:19.107477+0000","last_undegraded":"2024-02-05T07:20:19.107477+0000","last_fullsized":"2024-02-05T07:20:19.107477+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.798180+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:11.151835+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324397+0000","last_change":"2024-02-05T07:18:55.473667+0000","last_active":"2024-02-05T07:20:19.324397+0000","last_peered":"2024-02-05T07:20:19.324397+0000","last_clean":"2024-02-05T07:20:19.324397+0000","last_became_active":"2024-02-05T07:18:55.473368+0000","last_became_peered":"2024-02-05T07:18:55.473368+0000","last_unstale":"2024-02-05T07:20:19.324397+0000","last_undegraded":"2024-02-05T07:20:19.324397+0000","last_fullsized":"2024-02-05T07:20:19.324397+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537402+0000","last_change":"2024-02-05T07:18:53.457259+0000","last_active":"2024-02-05T07:20:19.537402+0000","last_peered":"2024-02-05T07:20:19.537402+0000","last_clean":"2024-02-05T07:20:19.537402+0000","last_became_active":"2024-02-05T07:18:53.456881+0000","last_became_peered":"2024-02-05T07:18:53.456881+0000","last_unstale":"2024-02-05T07:20:19.537402+0000","last_undegraded":"2024-02-05T07:20:19.537402+0000","last_fullsized":"2024-02-05T07:20:19.537402+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324830+0000","last_change":"2024-02-05T07:20:13.048248+0000","last_active":"2024-02-05T07:20:19.324830+0000","last_peered":"2024-02-05T07:20:19.324830+0000","last_clean":"2024-02-05T07:20:19.324830+0000","last_became_active":"2024-02-05T07:18:49.431220+0000","last_became_peered":"2024-02-05T07:18:49.431220+0000","last_unstale":"2024-02-05T07:20:19.324830+0000","last_undegraded":"2024-02-05T07:20:19.324830+0000","last_fullsized":"2024-02-05T07:20:19.324830+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:13.048126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:13.048126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:28.009618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107501+0000","last_change":"2024-02-05T07:18:51.448820+0000","last_active":"2024-02-05T07:20:19.107501+0000","last_peered":"2024-02-05T07:20:19.107501+0000","last_clean":"2024-02-05T07:20:19.107501+0000","last_became_active":"2024-02-05T07:18:51.448462+0000","last_became_peered":"2024-02-05T07:18:51.448462+0000","last_unstale":"2024-02-05T07:20:19.107501+0000","last_undegraded":"2024-02-05T07:20:19.107501+0000","last_fullsized":"2024-02-05T07:20:19.107501+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107996+0000","last_change":"2024-02-05T07:20:00.760332+0000","last_active":"2024-02-05T07:20:19.107996+0000","last_peered":"2024-02-05T07:20:19.107996+0000","last_clean":"2024-02-05T07:20:19.107996+0000","last_became_active":"2024-02-05T07:18:55.505678+0000","last_became_peered":"2024-02-05T07:18:55.505678+0000","last_unstale":"2024-02-05T07:20:19.107996+0000","last_undegraded":"2024-02-05T07:20:19.107996+0000","last_fullsized":"2024-02-05T07:20:19.107996+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:00.760230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.760230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:19.923028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.108178+0000","last_change":"2024-02-05T07:20:18.552686+0000","last_active":"2024-02-05T07:20:19.108178+0000","last_peered":"2024-02-05T07:20:19.108178+0000","last_clean":"2024-02-05T07:20:19.108178+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:20:19.108178+0000","last_undegraded":"2024-02-05T07:20:19.108178+0000","last_fullsized":"2024-02-05T07:20:19.108178+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:18.552608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.552608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:43.361846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264189+0000","last_change":"2024-02-05T07:18:49.432729+0000","last_active":"2024-02-05T07:20:19.264189+0000","last_peered":"2024-02-05T07:20:19.264189+0000","last_clean":"2024-02-05T07:20:19.264189+0000","last_became_active":"2024-02-05T07:18:49.432449+0000","last_became_peered":"2024-02-05T07:18:49.432449+0000","last_unstale":"2024-02-05T07:20:19.264189+0000","last_undegraded":"2024-02-05T07:20:19.264189+0000","last_fullsized":"2024-02-05T07:20:19.264189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":91,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536091+0000","last_change":"2024-02-05T07:19:59.624459+0000","last_active":"2024-02-05T07:20:19.536091+0000","last_peered":"2024-02-05T07:20:19.536091+0000","last_clean":"2024-02-05T07:20:19.536091+0000","last_became_active":"2024-02-05T07:18:51.447728+0000","last_became_peered":"2024-02-05T07:18:51.447728+0000","last_unstale":"2024-02-05T07:20:19.536091+0000","last_undegraded":"2024-02-05T07:20:19.536091+0000","last_fullsized":"2024-02-05T07:20:19.536091+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:19:59.624369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:59.624369+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:28.126457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107817+0000","last_change":"2024-02-05T07:18:55.506354+0000","last_active":"2024-02-05T07:20:19.107817+0000","last_peered":"2024-02-05T07:20:19.107817+0000","last_clean":"2024-02-05T07:20:19.107817+0000","last_became_active":"2024-02-05T07:18:55.505375+0000","last_became_peered":"2024-02-05T07:18:55.505375+0000","last_unstale":"2024-02-05T07:20:19.107817+0000","last_undegraded":"2024-02-05T07:20:19.107817+0000","last_fullsized":"2024-02-05T07:20:19.107817+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107504+0000","last_change":"2024-02-05T07:20:18.795431+0000","last_active":"2024-02-05T07:20:19.107504+0000","last_peered":"2024-02-05T07:20:19.107504+0000","last_clean":"2024-02-05T07:20:19.107504+0000","last_became_active":"2024-02-05T07:18:53.459077+0000","last_became_peered":"2024-02-05T07:18:53.459077+0000","last_unstale":"2024-02-05T07:20:19.107504+0000","last_undegraded":"2024-02-05T07:20:19.107504+0000","last_fullsized":"2024-02-05T07:20:19.107504+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:18.795341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.795341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:22.388371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107593+0000","last_change":"2024-02-05T07:18:49.428575+0000","last_active":"2024-02-05T07:20:19.107593+0000","last_peered":"2024-02-05T07:20:19.107593+0000","last_clean":"2024-02-05T07:20:19.107593+0000","last_became_active":"2024-02-05T07:18:49.427837+0000","last_became_peered":"2024-02-05T07:18:49.427837+0000","last_unstale":"2024-02-05T07:20:19.107593+0000","last_undegraded":"2024-02-05T07:20:19.107593+0000","last_fullsized":"2024-02-05T07:20:19.107593+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107370+0000","last_change":"2024-02-05T07:20:01.841591+0000","last_active":"2024-02-05T07:20:19.107370+0000","last_peered":"2024-02-05T07:20:19.107370+0000","last_clean":"2024-02-05T07:20:19.107370+0000","last_became_active":"2024-02-05T07:18:51.447761+0000","last_became_peered":"2024-02-05T07:18:51.447761+0000","last_unstale":"2024-02-05T07:20:19.107370+0000","last_undegraded":"2024-02-05T07:20:19.107370+0000","last_fullsized":"2024-02-05T07:20:19.107370+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:01.841489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:01.841489+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:19.277134+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537318+0000","last_change":"2024-02-05T07:20:17.755329+0000","last_active":"2024-02-05T07:20:19.537318+0000","last_peered":"2024-02-05T07:20:19.537318+0000","last_clean":"2024-02-05T07:20:19.537318+0000","last_became_active":"2024-02-05T07:18:55.476071+0000","last_became_peered":"2024-02-05T07:18:55.476071+0000","last_unstale":"2024-02-05T07:20:19.537318+0000","last_undegraded":"2024-02-05T07:20:19.537318+0000","last_fullsized":"2024-02-05T07:20:19.537318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:20:17.755246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:17.755246+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:22.075178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.538485+0000","last_change":"2024-02-05T07:18:53.461297+0000","last_active":"2024-02-05T07:20:19.538485+0000","last_peered":"2024-02-05T07:20:19.538485+0000","last_clean":"2024-02-05T07:20:19.538485+0000","last_became_active":"2024-02-05T07:18:53.461011+0000","last_became_peered":"2024-02-05T07:18:53.461011+0000","last_unstale":"2024-02-05T07:20:19.538485+0000","last_undegraded":"2024-02-05T07:20:19.538485+0000","last_fullsized":"2024-02-05T07:20:19.538485+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107026+0000","last_change":"2024-02-05T07:18:49.431334+0000","last_active":"2024-02-05T07:20:19.107026+0000","last_peered":"2024-02-05T07:20:19.107026+0000","last_clean":"2024-02-05T07:20:19.107026+0000","last_became_active":"2024-02-05T07:18:49.431031+0000","last_became_peered":"2024-02-05T07:18:49.431031+0000","last_unstale":"2024-02-05T07:20:19.107026+0000","last_undegraded":"2024-02-05T07:20:19.107026+0000","last_fullsized":"2024-02-05T07:20:19.107026+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264433+0000","last_change":"2024-02-05T07:18:51.510293+0000","last_active":"2024-02-05T07:20:19.264433+0000","last_peered":"2024-02-05T07:20:19.264433+0000","last_clean":"2024-02-05T07:20:19.264433+0000","last_became_active":"2024-02-05T07:18:51.509918+0000","last_became_peered":"2024-02-05T07:18:51.509918+0000","last_unstale":"2024-02-05T07:20:19.264433+0000","last_undegraded":"2024-02-05T07:20:19.264433+0000","last_fullsized":"2024-02-05T07:20:19.264433+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.108225+0000","last_change":"2024-02-05T07:20:03.444274+0000","last_active":"2024-02-05T07:20:19.108225+0000","last_peered":"2024-02-05T07:20:19.108225+0000","last_clean":"2024-02-05T07:20:19.108225+0000","last_became_active":"2024-02-05T07:18:55.466164+0000","last_became_peered":"2024-02-05T07:18:55.466164+0000","last_unstale":"2024-02-05T07:20:19.108225+0000","last_undegraded":"2024-02-05T07:20:19.108225+0000","last_fullsized":"2024-02-05T07:20:19.108225+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:03.444153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:03.444153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:32.621342+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264610+0000","last_change":"2024-02-05T07:18:53.457928+0000","last_active":"2024-02-05T07:20:19.264610+0000","last_peered":"2024-02-05T07:20:19.264610+0000","last_clean":"2024-02-05T07:20:19.264610+0000","last_became_active":"2024-02-05T07:18:53.457587+0000","last_became_peered":"2024-02-05T07:18:53.457587+0000","last_unstale":"2024-02-05T07:20:19.264610+0000","last_undegraded":"2024-02-05T07:20:19.264610+0000","last_fullsized":"2024-02-05T07:20:19.264610+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.105892+0000","last_change":"2024-02-05T07:19:49.788355+0000","last_active":"2024-02-05T07:20:19.105892+0000","last_peered":"2024-02-05T07:20:19.105892+0000","last_clean":"2024-02-05T07:20:19.105892+0000","last_became_active":"2024-02-05T07:18:49.430988+0000","last_became_peered":"2024-02-05T07:18:49.430988+0000","last_unstale":"2024-02-05T07:20:19.105892+0000","last_undegraded":"2024-02-05T07:20:19.105892+0000","last_fullsized":"2024-02-05T07:20:19.105892+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:49.788275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:49.788275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:50.100908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":85,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536243+0000","last_change":"2024-02-05T07:20:06.366811+0000","last_active":"2024-02-05T07:20:19.536243+0000","last_peered":"2024-02-05T07:20:19.536243+0000","last_clean":"2024-02-05T07:20:19.536243+0000","last_became_active":"2024-02-05T07:18:51.446604+0000","last_became_peered":"2024-02-05T07:18:51.446604+0000","last_unstale":"2024-02-05T07:20:19.536243+0000","last_undegraded":"2024-02-05T07:20:19.536243+0000","last_fullsized":"2024-02-05T07:20:19.536243+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:20:06.366699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.366699+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.671090+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537248+0000","last_change":"2024-02-05T07:20:01.693975+0000","last_active":"2024-02-05T07:20:19.537248+0000","last_peered":"2024-02-05T07:20:19.537248+0000","last_clean":"2024-02-05T07:20:19.537248+0000","last_became_active":"2024-02-05T07:18:55.468086+0000","last_became_peered":"2024-02-05T07:18:55.468086+0000","last_unstale":"2024-02-05T07:20:19.537248+0000","last_undegraded":"2024-02-05T07:20:19.537248+0000","last_fullsized":"2024-02-05T07:20:19.537248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:01.693871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:01.693871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:12.004988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536302+0000","last_change":"2024-02-05T07:20:09.349490+0000","last_active":"2024-02-05T07:20:19.536302+0000","last_peered":"2024-02-05T07:20:19.536302+0000","last_clean":"2024-02-05T07:20:19.536302+0000","last_became_active":"2024-02-05T07:18:53.460307+0000","last_became_peered":"2024-02-05T07:18:53.460307+0000","last_unstale":"2024-02-05T07:20:19.536302+0000","last_undegraded":"2024-02-05T07:20:19.536302+0000","last_fullsized":"2024-02-05T07:20:19.536302+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.349342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.451852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.326467+0000","last_change":"2024-02-05T07:20:19.326489+0000","last_active":"2024-02-05T07:20:19.326467+0000","last_peered":"2024-02-05T07:20:19.326467+0000","last_clean":"2024-02-05T07:20:19.326467+0000","last_became_active":"2024-02-05T07:18:49.434408+0000","last_became_peered":"2024-02-05T07:18:49.434408+0000","last_unstale":"2024-02-05T07:20:19.326467+0000","last_undegraded":"2024-02-05T07:20:19.326467+0000","last_fullsized":"2024-02-05T07:20:19.326467+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:19.326400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:19.326400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:45.594433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.105998+0000","last_change":"2024-02-05T07:18:51.449489+0000","last_active":"2024-02-05T07:20:19.105998+0000","last_peered":"2024-02-05T07:20:19.105998+0000","last_clean":"2024-02-05T07:20:19.105998+0000","last_became_active":"2024-02-05T07:18:51.449083+0000","last_became_peered":"2024-02-05T07:18:51.449083+0000","last_unstale":"2024-02-05T07:20:19.105998+0000","last_undegraded":"2024-02-05T07:20:19.105998+0000","last_fullsized":"2024-02-05T07:20:19.105998+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106745+0000","last_change":"2024-02-05T07:18:55.507549+0000","last_active":"2024-02-05T07:20:19.106745+0000","last_peered":"2024-02-05T07:20:19.106745+0000","last_clean":"2024-02-05T07:20:19.106745+0000","last_became_active":"2024-02-05T07:18:55.506968+0000","last_became_peered":"2024-02-05T07:18:55.506968+0000","last_unstale":"2024-02-05T07:20:19.106745+0000","last_undegraded":"2024-02-05T07:20:19.106745+0000","last_fullsized":"2024-02-05T07:20:19.106745+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537956+0000","last_change":"2024-02-05T07:20:05.674115+0000","last_active":"2024-02-05T07:20:19.537956+0000","last_peered":"2024-02-05T07:20:19.537956+0000","last_clean":"2024-02-05T07:20:19.537956+0000","last_became_active":"2024-02-05T07:18:53.460977+0000","last_became_peered":"2024-02-05T07:18:53.460977+0000","last_unstale":"2024-02-05T07:20:19.537956+0000","last_undegraded":"2024-02-05T07:20:19.537956+0000","last_fullsized":"2024-02-05T07:20:19.537956+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_clean_scrub_stamp":"2024-02-05T07:20:05.673983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:06.714441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324989+0000","last_change":"2024-02-05T07:19:48.944128+0000","last_active":"2024-02-05T07:20:19.324989+0000","last_peered":"2024-02-05T07:20:19.324989+0000","last_clean":"2024-02-05T07:20:19.324989+0000","last_became_active":"2024-02-05T07:18:49.434560+0000","last_became_peered":"2024-02-05T07:18:49.434560+0000","last_unstale":"2024-02-05T07:20:19.324989+0000","last_undegraded":"2024-02-05T07:20:19.324989+0000","last_fullsized":"2024-02-05T07:20:19.324989+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:48.944028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:48.944028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:56.156526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106467+0000","last_change":"2024-02-05T07:18:51.510217+0000","last_active":"2024-02-05T07:20:19.106467+0000","last_peered":"2024-02-05T07:20:19.106467+0000","last_clean":"2024-02-05T07:20:19.106467+0000","last_became_active":"2024-02-05T07:18:51.509911+0000","last_became_peered":"2024-02-05T07:18:51.509911+0000","last_unstale":"2024-02-05T07:20:19.106467+0000","last_undegraded":"2024-02-05T07:20:19.106467+0000","last_fullsized":"2024-02-05T07:20:19.106467+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.263725+0000","last_change":"2024-02-05T07:18:55.477626+0000","last_active":"2024-02-05T07:20:19.263725+0000","last_peered":"2024-02-05T07:20:19.263725+0000","last_clean":"2024-02-05T07:20:19.263725+0000","last_became_active":"2024-02-05T07:18:55.477312+0000","last_became_peered":"2024-02-05T07:18:55.477312+0000","last_unstale":"2024-02-05T07:20:19.263725+0000","last_undegraded":"2024-02-05T07:20:19.263725+0000","last_fullsized":"2024-02-05T07:20:19.263725+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106996+0000","last_change":"2024-02-05T07:18:53.468306+0000","last_active":"2024-02-05T07:20:19.106996+0000","last_peered":"2024-02-05T07:20:19.106996+0000","last_clean":"2024-02-05T07:20:19.106996+0000","last_became_active":"2024-02-05T07:18:53.467986+0000","last_became_peered":"2024-02-05T07:18:53.467986+0000","last_unstale":"2024-02-05T07:20:19.106996+0000","last_undegraded":"2024-02-05T07:20:19.106996+0000","last_fullsized":"2024-02-05T07:20:19.106996+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:21.663256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537858+0000","last_change":"2024-02-05T07:18:49.438840+0000","last_active":"2024-02-05T07:20:19.537858+0000","last_peered":"2024-02-05T07:20:19.537858+0000","last_clean":"2024-02-05T07:20:19.537858+0000","last_became_active":"2024-02-05T07:18:49.438565+0000","last_became_peered":"2024-02-05T07:18:49.438565+0000","last_unstale":"2024-02-05T07:20:19.537858+0000","last_undegraded":"2024-02-05T07:20:19.537858+0000","last_fullsized":"2024-02-05T07:20:19.537858+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324390+0000","last_change":"2024-02-05T07:19:16.128305+0000","last_active":"2024-02-05T07:20:19.324390+0000","last_peered":"2024-02-05T07:20:19.324390+0000","last_clean":"2024-02-05T07:20:19.324390+0000","last_became_active":"2024-02-05T07:19:16.122059+0000","last_became_peered":"2024-02-05T07:19:16.122059+0000","last_unstale":"2024-02-05T07:20:19.324390+0000","last_undegraded":"2024-02-05T07:20:19.324390+0000","last_fullsized":"2024-02-05T07:20:19.324390+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.105988+0000","last_change":"2024-02-05T07:18:55.478367+0000","last_active":"2024-02-05T07:20:19.105988+0000","last_peered":"2024-02-05T07:20:19.105988+0000","last_clean":"2024-02-05T07:20:19.105988+0000","last_became_active":"2024-02-05T07:18:55.478087+0000","last_became_peered":"2024-02-05T07:18:55.478087+0000","last_unstale":"2024-02-05T07:20:19.105988+0000","last_undegraded":"2024-02-05T07:20:19.105988+0000","last_fullsized":"2024-02-05T07:20:19.105988+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107481+0000","last_change":"2024-02-05T07:18:53.463942+0000","last_active":"2024-02-05T07:20:19.107481+0000","last_peered":"2024-02-05T07:20:19.107481+0000","last_clean":"2024-02-05T07:20:19.107481+0000","last_became_active":"2024-02-05T07:18:53.463620+0000","last_became_peered":"2024-02-05T07:18:53.463620+0000","last_unstale":"2024-02-05T07:20:19.107481+0000","last_undegraded":"2024-02-05T07:20:19.107481+0000","last_fullsized":"2024-02-05T07:20:19.107481+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107923+0000","last_change":"2024-02-05T07:18:49.431084+0000","last_active":"2024-02-05T07:20:19.107923+0000","last_peered":"2024-02-05T07:20:19.107923+0000","last_clean":"2024-02-05T07:20:19.107923+0000","last_became_active":"2024-02-05T07:18:49.430766+0000","last_became_peered":"2024-02-05T07:18:49.430766+0000","last_unstale":"2024-02-05T07:20:19.107923+0000","last_undegraded":"2024-02-05T07:20:19.107923+0000","last_fullsized":"2024-02-05T07:20:19.107923+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":103,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107790+0000","last_change":"2024-02-05T07:20:06.870132+0000","last_active":"2024-02-05T07:20:19.107790+0000","last_peered":"2024-02-05T07:20:19.107790+0000","last_clean":"2024-02-05T07:20:19.107790+0000","last_became_active":"2024-02-05T07:19:16.121696+0000","last_became_peered":"2024-02-05T07:19:16.121696+0000","last_unstale":"2024-02-05T07:20:19.107790+0000","last_undegraded":"2024-02-05T07:20:19.107790+0000","last_fullsized":"2024-02-05T07:20:19.107790+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:20:06.870006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.870006+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:12.414942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264309+0000","last_change":"2024-02-05T07:18:55.478247+0000","last_active":"2024-02-05T07:20:19.264309+0000","last_peered":"2024-02-05T07:20:19.264309+0000","last_clean":"2024-02-05T07:20:19.264309+0000","last_became_active":"2024-02-05T07:18:55.477896+0000","last_became_peered":"2024-02-05T07:18:55.477896+0000","last_unstale":"2024-02-05T07:20:19.264309+0000","last_undegraded":"2024-02-05T07:20:19.264309+0000","last_fullsized":"2024-02-05T07:20:19.264309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324979+0000","last_change":"2024-02-05T07:18:53.465570+0000","last_active":"2024-02-05T07:20:19.324979+0000","last_peered":"2024-02-05T07:20:19.324979+0000","last_clean":"2024-02-05T07:20:19.324979+0000","last_became_active":"2024-02-05T07:18:53.465287+0000","last_became_peered":"2024-02-05T07:18:53.465287+0000","last_unstale":"2024-02-05T07:20:19.324979+0000","last_undegraded":"2024-02-05T07:20:19.324979+0000","last_fullsized":"2024-02-05T07:20:19.324979+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536950+0000","last_change":"2024-02-05T07:18:49.436416+0000","last_active":"2024-02-05T07:20:19.536950+0000","last_peered":"2024-02-05T07:20:19.536950+0000","last_clean":"2024-02-05T07:20:19.536950+0000","last_became_active":"2024-02-05T07:18:49.436131+0000","last_became_peered":"2024-02-05T07:18:49.436131+0000","last_unstale":"2024-02-05T07:20:19.536950+0000","last_undegraded":"2024-02-05T07:20:19.536950+0000","last_fullsized":"2024-02-05T07:20:19.536950+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107106+0000","last_change":"2024-02-05T07:18:51.507793+0000","last_active":"2024-02-05T07:20:19.107106+0000","last_peered":"2024-02-05T07:20:19.107106+0000","last_clean":"2024-02-05T07:20:19.107106+0000","last_became_active":"2024-02-05T07:18:51.507491+0000","last_became_peered":"2024-02-05T07:18:51.507491+0000","last_unstale":"2024-02-05T07:20:19.107106+0000","last_undegraded":"2024-02-05T07:20:19.107106+0000","last_fullsized":"2024-02-05T07:20:19.107106+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.108318+0000","last_change":"2024-02-05T07:18:55.467356+0000","last_active":"2024-02-05T07:20:19.108318+0000","last_peered":"2024-02-05T07:20:19.108318+0000","last_clean":"2024-02-05T07:20:19.108318+0000","last_became_active":"2024-02-05T07:18:55.467044+0000","last_became_peered":"2024-02-05T07:18:55.467044+0000","last_unstale":"2024-02-05T07:20:19.108318+0000","last_undegraded":"2024-02-05T07:20:19.108318+0000","last_fullsized":"2024-02-05T07:20:19.108318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106935+0000","last_change":"2024-02-05T07:18:53.465551+0000","last_active":"2024-02-05T07:20:19.106935+0000","last_peered":"2024-02-05T07:20:19.106935+0000","last_clean":"2024-02-05T07:20:19.106935+0000","last_became_active":"2024-02-05T07:18:53.465270+0000","last_became_peered":"2024-02-05T07:18:53.465270+0000","last_unstale":"2024-02-05T07:20:19.106935+0000","last_undegraded":"2024-02-05T07:20:19.106935+0000","last_fullsized":"2024-02-05T07:20:19.106935+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.263969+0000","last_change":"2024-02-05T07:18:49.435653+0000","last_active":"2024-02-05T07:20:19.263969+0000","last_peered":"2024-02-05T07:20:19.263969+0000","last_clean":"2024-02-05T07:20:19.263969+0000","last_became_active":"2024-02-05T07:18:49.434402+0000","last_became_peered":"2024-02-05T07:18:49.434402+0000","last_unstale":"2024-02-05T07:20:19.263969+0000","last_undegraded":"2024-02-05T07:20:19.263969+0000","last_fullsized":"2024-02-05T07:20:19.263969+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.325064+0000","last_change":"2024-02-05T07:20:06.852726+0000","last_active":"2024-02-05T07:20:19.325064+0000","last_peered":"2024-02-05T07:20:19.325064+0000","last_clean":"2024-02-05T07:20:19.325064+0000","last_became_active":"2024-02-05T07:18:51.508551+0000","last_became_peered":"2024-02-05T07:18:51.508551+0000","last_unstale":"2024-02-05T07:20:19.325064+0000","last_undegraded":"2024-02-05T07:20:19.325064+0000","last_fullsized":"2024-02-05T07:20:19.325064+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:20:06.852562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.852562+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:20.949048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536815+0000","last_change":"2024-02-05T07:20:11.543111+0000","last_active":"2024-02-05T07:20:19.536815+0000","last_peered":"2024-02-05T07:20:19.536815+0000","last_clean":"2024-02-05T07:20:19.536815+0000","last_became_active":"2024-02-05T07:18:55.469044+0000","last_became_peered":"2024-02-05T07:18:55.469044+0000","last_unstale":"2024-02-05T07:20:19.536815+0000","last_undegraded":"2024-02-05T07:20:19.536815+0000","last_fullsized":"2024-02-05T07:20:19.536815+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:11.543036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:11.543036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:38.671608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107792+0000","last_change":"2024-02-05T07:20:02.813432+0000","last_active":"2024-02-05T07:20:19.107792+0000","last_peered":"2024-02-05T07:20:19.107792+0000","last_clean":"2024-02-05T07:20:19.107792+0000","last_became_active":"2024-02-05T07:18:53.463607+0000","last_became_peered":"2024-02-05T07:18:53.463607+0000","last_unstale":"2024-02-05T07:20:19.107792+0000","last_undegraded":"2024-02-05T07:20:19.107792+0000","last_fullsized":"2024-02-05T07:20:19.107792+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.813344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.813344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:30.005634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537804+0000","last_change":"2024-02-05T07:20:02.673707+0000","last_active":"2024-02-05T07:20:19.537804+0000","last_peered":"2024-02-05T07:20:19.537804+0000","last_clean":"2024-02-05T07:20:19.537804+0000","last_became_active":"2024-02-05T07:18:49.437732+0000","last_became_peered":"2024-02-05T07:18:49.437732+0000","last_unstale":"2024-02-05T07:20:19.537804+0000","last_undegraded":"2024-02-05T07:20:19.537804+0000","last_fullsized":"2024-02-05T07:20:19.537804+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.673592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.665799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264540+0000","last_change":"2024-02-05T07:18:51.510254+0000","last_active":"2024-02-05T07:20:19.264540+0000","last_peered":"2024-02-05T07:20:19.264540+0000","last_clean":"2024-02-05T07:20:19.264540+0000","last_became_active":"2024-02-05T07:18:51.509917+0000","last_became_peered":"2024-02-05T07:18:51.509917+0000","last_unstale":"2024-02-05T07:20:19.264540+0000","last_undegraded":"2024-02-05T07:20:19.264540+0000","last_fullsized":"2024-02-05T07:20:19.264540+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.732419+0000","last_change":"2024-02-05T07:20:19.732465+0000","last_active":"2024-02-05T07:20:19.732419+0000","last_peered":"2024-02-05T07:20:19.732419+0000","last_clean":"2024-02-05T07:20:19.732419+0000","last_became_active":"2024-02-05T07:18:55.507555+0000","last_became_peered":"2024-02-05T07:18:55.507555+0000","last_unstale":"2024-02-05T07:20:19.732419+0000","last_undegraded":"2024-02-05T07:20:19.732419+0000","last_fullsized":"2024-02-05T07:20:19.732419+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:19.732337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:19.732337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:28.044159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.325001+0000","last_change":"2024-02-05T07:18:53.465851+0000","last_active":"2024-02-05T07:20:19.325001+0000","last_peered":"2024-02-05T07:20:19.325001+0000","last_clean":"2024-02-05T07:20:19.325001+0000","last_became_active":"2024-02-05T07:18:53.465591+0000","last_became_peered":"2024-02-05T07:18:53.465591+0000","last_unstale":"2024-02-05T07:20:19.325001+0000","last_undegraded":"2024-02-05T07:20:19.325001+0000","last_fullsized":"2024-02-05T07:20:19.325001+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":76,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536559+0000","last_change":"2024-02-05T07:19:55.369939+0000","last_active":"2024-02-05T07:20:19.536559+0000","last_peered":"2024-02-05T07:20:19.536559+0000","last_clean":"2024-02-05T07:20:19.536559+0000","last_became_active":"2024-02-05T07:18:49.433121+0000","last_became_peered":"2024-02-05T07:18:49.433121+0000","last_unstale":"2024-02-05T07:20:19.536559+0000","last_undegraded":"2024-02-05T07:20:19.536559+0000","last_fullsized":"2024-02-05T07:20:19.536559+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:19:55.369800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:55.369800+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:04.984273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107918+0000","last_change":"2024-02-05T07:18:51.508489+0000","last_active":"2024-02-05T07:20:19.107918+0000","last_peered":"2024-02-05T07:20:19.107918+0000","last_clean":"2024-02-05T07:20:19.107918+0000","last_became_active":"2024-02-05T07:18:51.507451+0000","last_became_peered":"2024-02-05T07:18:51.507451+0000","last_unstale":"2024-02-05T07:20:19.107918+0000","last_undegraded":"2024-02-05T07:20:19.107918+0000","last_fullsized":"2024-02-05T07:20:19.107918+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324748+0000","last_change":"2024-02-05T07:18:55.505712+0000","last_active":"2024-02-05T07:20:19.324748+0000","last_peered":"2024-02-05T07:20:19.324748+0000","last_clean":"2024-02-05T07:20:19.324748+0000","last_became_active":"2024-02-05T07:18:55.505401+0000","last_became_peered":"2024-02-05T07:18:55.505401+0000","last_unstale":"2024-02-05T07:20:19.324748+0000","last_undegraded":"2024-02-05T07:20:19.324748+0000","last_fullsized":"2024-02-05T07:20:19.324748+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537620+0000","last_change":"2024-02-05T07:20:13.701938+0000","last_active":"2024-02-05T07:20:19.537620+0000","last_peered":"2024-02-05T07:20:19.537620+0000","last_clean":"2024-02-05T07:20:19.537620+0000","last_became_active":"2024-02-05T07:18:49.437663+0000","last_became_peered":"2024-02-05T07:18:49.437663+0000","last_unstale":"2024-02-05T07:20:19.537620+0000","last_undegraded":"2024-02-05T07:20:19.537620+0000","last_fullsized":"2024-02-05T07:20:19.537620+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:20:13.701852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:13.701852+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:32.496072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":91,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324432+0000","last_change":"2024-02-05T07:19:58.753612+0000","last_active":"2024-02-05T07:20:19.324432+0000","last_peered":"2024-02-05T07:20:19.324432+0000","last_clean":"2024-02-05T07:20:19.324432+0000","last_became_active":"2024-02-05T07:18:51.508097+0000","last_became_peered":"2024-02-05T07:18:51.508097+0000","last_unstale":"2024-02-05T07:20:19.324432+0000","last_undegraded":"2024-02-05T07:20:19.324432+0000","last_fullsized":"2024-02-05T07:20:19.324432+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:19:58.753472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:58.753472+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:26.081872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":90,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.538002+0000","last_change":"2024-02-05T07:20:00.685371+0000","last_active":"2024-02-05T07:20:19.538002+0000","last_peered":"2024-02-05T07:20:19.538002+0000","last_clean":"2024-02-05T07:20:19.538002+0000","last_became_active":"2024-02-05T07:18:53.456287+0000","last_became_peered":"2024-02-05T07:18:53.456287+0000","last_unstale":"2024-02-05T07:20:19.538002+0000","last_undegraded":"2024-02-05T07:20:19.538002+0000","last_fullsized":"2024-02-05T07:20:19.538002+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.685282+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:11.368545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106830+0000","last_change":"2024-02-05T07:18:55.474988+0000","last_active":"2024-02-05T07:20:19.106830+0000","last_peered":"2024-02-05T07:20:19.106830+0000","last_clean":"2024-02-05T07:20:19.106830+0000","last_became_active":"2024-02-05T07:18:55.474663+0000","last_became_peered":"2024-02-05T07:18:55.474663+0000","last_unstale":"2024-02-05T07:20:19.106830+0000","last_undegraded":"2024-02-05T07:20:19.106830+0000","last_fullsized":"2024-02-05T07:20:19.106830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537017+0000","last_change":"2024-02-05T07:19:51.758261+0000","last_active":"2024-02-05T07:20:19.537017+0000","last_peered":"2024-02-05T07:20:19.537017+0000","last_clean":"2024-02-05T07:20:19.537017+0000","last_became_active":"2024-02-05T07:18:49.435621+0000","last_became_peered":"2024-02-05T07:18:49.435621+0000","last_unstale":"2024-02-05T07:20:19.537017+0000","last_undegraded":"2024-02-05T07:20:19.537017+0000","last_fullsized":"2024-02-05T07:20:19.537017+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:51.758163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.758163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:15.549152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":107,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107970+0000","last_change":"2024-02-05T07:20:04.885890+0000","last_active":"2024-02-05T07:20:19.107970+0000","last_peered":"2024-02-05T07:20:19.107970+0000","last_clean":"2024-02-05T07:20:19.107970+0000","last_became_active":"2024-02-05T07:19:16.123650+0000","last_became_peered":"2024-02-05T07:19:16.123650+0000","last_unstale":"2024-02-05T07:20:19.107970+0000","last_undegraded":"2024-02-05T07:20:19.107970+0000","last_fullsized":"2024-02-05T07:20:19.107970+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:04.885778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:04.885778+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:07.176995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536286+0000","last_change":"2024-02-05T07:20:02.694603+0000","last_active":"2024-02-05T07:20:19.536286+0000","last_peered":"2024-02-05T07:20:19.536286+0000","last_clean":"2024-02-05T07:20:19.536286+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:20:19.536286+0000","last_undegraded":"2024-02-05T07:20:19.536286+0000","last_fullsized":"2024-02-05T07:20:19.536286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.694506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.694506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.700377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536905+0000","last_change":"2024-02-05T07:18:55.473523+0000","last_active":"2024-02-05T07:20:19.536905+0000","last_peered":"2024-02-05T07:20:19.536905+0000","last_clean":"2024-02-05T07:20:19.536905+0000","last_became_active":"2024-02-05T07:18:55.473032+0000","last_became_peered":"2024-02-05T07:18:55.473032+0000","last_unstale":"2024-02-05T07:20:19.536905+0000","last_undegraded":"2024-02-05T07:20:19.536905+0000","last_fullsized":"2024-02-05T07:20:19.536905+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536595+0000","last_change":"2024-02-05T07:20:16.476649+0000","last_active":"2024-02-05T07:20:19.536595+0000","last_peered":"2024-02-05T07:20:19.536595+0000","last_clean":"2024-02-05T07:20:19.536595+0000","last_became_active":"2024-02-05T07:18:53.464867+0000","last_became_peered":"2024-02-05T07:18:53.464867+0000","last_unstale":"2024-02-05T07:20:19.536595+0000","last_undegraded":"2024-02-05T07:20:19.536595+0000","last_fullsized":"2024-02-05T07:20:19.536595+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:16.476554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:16.476554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:31.284238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.538140+0000","last_change":"2024-02-05T07:19:55.788134+0000","last_active":"2024-02-05T07:20:19.538140+0000","last_peered":"2024-02-05T07:20:19.538140+0000","last_clean":"2024-02-05T07:20:19.538140+0000","last_became_active":"2024-02-05T07:18:49.434451+0000","last_became_peered":"2024-02-05T07:18:49.434451+0000","last_unstale":"2024-02-05T07:20:19.538140+0000","last_undegraded":"2024-02-05T07:20:19.538140+0000","last_fullsized":"2024-02-05T07:20:19.538140+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:55.788005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:55.788005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.126482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":97,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536108+0000","last_change":"2024-02-05T07:20:00.384918+0000","last_active":"2024-02-05T07:20:19.536108+0000","last_peered":"2024-02-05T07:20:19.536108+0000","last_clean":"2024-02-05T07:20:19.536108+0000","last_became_active":"2024-02-05T07:18:51.507672+0000","last_became_peered":"2024-02-05T07:18:51.507672+0000","last_unstale":"2024-02-05T07:20:19.536108+0000","last_undegraded":"2024-02-05T07:20:19.536108+0000","last_fullsized":"2024-02-05T07:20:19.536108+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:20:00.384813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.384813+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:12.230867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.263790+0000","last_change":"2024-02-05T07:18:55.476043+0000","last_active":"2024-02-05T07:20:19.263790+0000","last_peered":"2024-02-05T07:20:19.263790+0000","last_clean":"2024-02-05T07:20:19.263790+0000","last_became_active":"2024-02-05T07:18:55.475702+0000","last_became_peered":"2024-02-05T07:18:55.475702+0000","last_unstale":"2024-02-05T07:20:19.263790+0000","last_undegraded":"2024-02-05T07:20:19.263790+0000","last_fullsized":"2024-02-05T07:20:19.263790+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:20.177474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536487+0000","last_change":"2024-02-05T07:20:09.589044+0000","last_active":"2024-02-05T07:20:19.536487+0000","last_peered":"2024-02-05T07:20:19.536487+0000","last_clean":"2024-02-05T07:20:19.536487+0000","last_became_active":"2024-02-05T07:18:53.459560+0000","last_became_peered":"2024-02-05T07:18:53.459560+0000","last_unstale":"2024-02-05T07:20:19.536487+0000","last_undegraded":"2024-02-05T07:20:19.536487+0000","last_fullsized":"2024-02-05T07:20:19.536487+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:09.588893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.588893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:27.848596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324652+0000","last_change":"2024-02-05T07:20:15.066894+0000","last_active":"2024-02-05T07:20:19.324652+0000","last_peered":"2024-02-05T07:20:19.324652+0000","last_clean":"2024-02-05T07:20:19.324652+0000","last_became_active":"2024-02-05T07:18:49.434238+0000","last_became_peered":"2024-02-05T07:18:49.434238+0000","last_unstale":"2024-02-05T07:20:19.324652+0000","last_undegraded":"2024-02-05T07:20:19.324652+0000","last_fullsized":"2024-02-05T07:20:19.324652+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:15.066768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:15.066768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:41.133131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264329+0000","last_change":"2024-02-05T07:18:51.441099+0000","last_active":"2024-02-05T07:20:19.264329+0000","last_peered":"2024-02-05T07:20:19.264329+0000","last_clean":"2024-02-05T07:20:19.264329+0000","last_became_active":"2024-02-05T07:18:51.440744+0000","last_became_peered":"2024-02-05T07:18:51.440744+0000","last_unstale":"2024-02-05T07:20:19.264329+0000","last_undegraded":"2024-02-05T07:20:19.264329+0000","last_fullsized":"2024-02-05T07:20:19.264329+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106503+0000","last_change":"2024-02-05T07:18:55.476663+0000","last_active":"2024-02-05T07:20:19.106503+0000","last_peered":"2024-02-05T07:20:19.106503+0000","last_clean":"2024-02-05T07:20:19.106503+0000","last_became_active":"2024-02-05T07:18:55.476313+0000","last_became_peered":"2024-02-05T07:18:55.476313+0000","last_unstale":"2024-02-05T07:20:19.106503+0000","last_undegraded":"2024-02-05T07:20:19.106503+0000","last_fullsized":"2024-02-05T07:20:19.106503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:20.640694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324529+0000","last_change":"2024-02-05T07:18:52.445271+0000","last_active":"2024-02-05T07:20:19.324529+0000","last_peered":"2024-02-05T07:20:19.324529+0000","last_clean":"2024-02-05T07:20:19.324529+0000","last_became_active":"2024-02-05T07:18:48.423032+0000","last_became_peered":"2024-02-05T07:18:48.423032+0000","last_unstale":"2024-02-05T07:20:19.324529+0000","last_undegraded":"2024-02-05T07:20:19.324529+0000","last_fullsized":"2024-02-05T07:20:19.324529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:18:47.405257+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107452+0000","last_change":"2024-02-05T07:18:49.430550+0000","last_active":"2024-02-05T07:20:19.107452+0000","last_peered":"2024-02-05T07:20:19.107452+0000","last_clean":"2024-02-05T07:20:19.107452+0000","last_became_active":"2024-02-05T07:18:49.430208+0000","last_became_peered":"2024-02-05T07:18:49.430208+0000","last_unstale":"2024-02-05T07:20:19.107452+0000","last_undegraded":"2024-02-05T07:20:19.107452+0000","last_fullsized":"2024-02-05T07:20:19.107452+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":126,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107146+0000","last_change":"2024-02-05T07:18:51.442632+0000","last_active":"2024-02-05T07:20:19.107146+0000","last_peered":"2024-02-05T07:20:19.107146+0000","last_clean":"2024-02-05T07:20:19.107146+0000","last_became_active":"2024-02-05T07:18:51.442110+0000","last_became_peered":"2024-02-05T07:18:51.442110+0000","last_unstale":"2024-02-05T07:20:19.107146+0000","last_undegraded":"2024-02-05T07:20:19.107146+0000","last_fullsized":"2024-02-05T07:20:19.107146+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264365+0000","last_change":"2024-02-05T07:18:53.461030+0000","last_active":"2024-02-05T07:20:19.264365+0000","last_peered":"2024-02-05T07:20:19.264365+0000","last_clean":"2024-02-05T07:20:19.264365+0000","last_became_active":"2024-02-05T07:18:53.460715+0000","last_became_peered":"2024-02-05T07:18:53.460715+0000","last_unstale":"2024-02-05T07:20:19.264365+0000","last_undegraded":"2024-02-05T07:20:19.264365+0000","last_fullsized":"2024-02-05T07:20:19.264365+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.356767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.538271+0000","last_change":"2024-02-05T07:20:06.673182+0000","last_active":"2024-02-05T07:20:19.538271+0000","last_peered":"2024-02-05T07:20:19.538271+0000","last_clean":"2024-02-05T07:20:19.538271+0000","last_became_active":"2024-02-05T07:18:55.506803+0000","last_became_peered":"2024-02-05T07:18:55.506803+0000","last_unstale":"2024-02-05T07:20:19.538271+0000","last_undegraded":"2024-02-05T07:20:19.538271+0000","last_fullsized":"2024-02-05T07:20:19.538271+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.673081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:21.893489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106435+0000","last_change":"2024-02-05T07:20:09.406817+0000","last_active":"2024-02-05T07:20:19.106435+0000","last_peered":"2024-02-05T07:20:19.106435+0000","last_clean":"2024-02-05T07:20:19.106435+0000","last_became_active":"2024-02-05T07:18:49.429651+0000","last_became_peered":"2024-02-05T07:18:49.429651+0000","last_unstale":"2024-02-05T07:20:19.106435+0000","last_undegraded":"2024-02-05T07:20:19.106435+0000","last_fullsized":"2024-02-05T07:20:19.106435+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:20:09.406718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.406718+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:33.441463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536606+0000","last_change":"2024-02-05T07:19:56.661601+0000","last_active":"2024-02-05T07:20:19.536606+0000","last_peered":"2024-02-05T07:20:19.536606+0000","last_clean":"2024-02-05T07:20:19.536606+0000","last_became_active":"2024-02-05T07:18:48.423639+0000","last_became_peered":"2024-02-05T07:18:48.423639+0000","last_unstale":"2024-02-05T07:20:19.536606+0000","last_undegraded":"2024-02-05T07:20:19.536606+0000","last_fullsized":"2024-02-05T07:20:19.536606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:56.661490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:19:56.661490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.585463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107965+0000","last_change":"2024-02-05T07:18:51.436432+0000","last_active":"2024-02-05T07:20:19.107965+0000","last_peered":"2024-02-05T07:20:19.107965+0000","last_clean":"2024-02-05T07:20:19.107965+0000","last_became_active":"2024-02-05T07:18:51.436097+0000","last_became_peered":"2024-02-05T07:18:51.436097+0000","last_unstale":"2024-02-05T07:20:19.107965+0000","last_undegraded":"2024-02-05T07:20:19.107965+0000","last_fullsized":"2024-02-05T07:20:19.107965+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536967+0000","last_change":"2024-02-05T07:20:10.558042+0000","last_active":"2024-02-05T07:20:19.536967+0000","last_peered":"2024-02-05T07:20:19.536967+0000","last_clean":"2024-02-05T07:20:19.536967+0000","last_became_active":"2024-02-05T07:18:53.460771+0000","last_became_peered":"2024-02-05T07:18:53.460771+0000","last_unstale":"2024-02-05T07:20:19.536967+0000","last_undegraded":"2024-02-05T07:20:19.536967+0000","last_fullsized":"2024-02-05T07:20:19.536967+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:10.557948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:10.557948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:27.157314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.325061+0000","last_change":"2024-02-05T07:18:55.473873+0000","last_active":"2024-02-05T07:20:19.325061+0000","last_peered":"2024-02-05T07:20:19.325061+0000","last_clean":"2024-02-05T07:20:19.325061+0000","last_became_active":"2024-02-05T07:18:55.473358+0000","last_became_peered":"2024-02-05T07:18:55.473358+0000","last_unstale":"2024-02-05T07:20:19.325061+0000","last_undegraded":"2024-02-05T07:20:19.325061+0000","last_fullsized":"2024-02-05T07:20:19.325061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":319,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537224+0000","last_change":"2024-02-05T07:19:05.594948+0000","last_active":"2024-02-05T07:20:19.537224+0000","last_peered":"2024-02-05T07:20:19.537224+0000","last_clean":"2024-02-05T07:20:19.537224+0000","last_became_active":"2024-02-05T07:18:39.444898+0000","last_became_peered":"2024-02-05T07:18:39.444898+0000","last_unstale":"2024-02-05T07:20:19.537224+0000","last_undegraded":"2024-02-05T07:20:19.537224+0000","last_fullsized":"2024-02-05T07:20:19.537224+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-02-05T07:19:05.594812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:19:05.594812+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106628+0000","last_change":"2024-02-05T07:18:49.439534+0000","last_active":"2024-02-05T07:20:19.106628+0000","last_peered":"2024-02-05T07:20:19.106628+0000","last_clean":"2024-02-05T07:20:19.106628+0000","last_became_active":"2024-02-05T07:18:49.439249+0000","last_became_peered":"2024-02-05T07:18:49.439249+0000","last_unstale":"2024-02-05T07:20:19.106628+0000","last_undegraded":"2024-02-05T07:20:19.106628+0000","last_fullsized":"2024-02-05T07:20:19.106628+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":99,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536349+0000","last_change":"2024-02-05T07:18:51.508007+0000","last_active":"2024-02-05T07:20:19.536349+0000","last_peered":"2024-02-05T07:20:19.536349+0000","last_clean":"2024-02-05T07:20:19.536349+0000","last_became_active":"2024-02-05T07:18:51.507714+0000","last_became_peered":"2024-02-05T07:18:51.507714+0000","last_unstale":"2024-02-05T07:20:19.536349+0000","last_undegraded":"2024-02-05T07:20:19.536349+0000","last_fullsized":"2024-02-05T07:20:19.536349+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.846104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537612+0000","last_change":"2024-02-05T07:18:53.460344+0000","last_active":"2024-02-05T07:20:19.537612+0000","last_peered":"2024-02-05T07:20:19.537612+0000","last_clean":"2024-02-05T07:20:19.537612+0000","last_became_active":"2024-02-05T07:18:53.459971+0000","last_became_peered":"2024-02-05T07:18:53.459971+0000","last_unstale":"2024-02-05T07:20:19.537612+0000","last_undegraded":"2024-02-05T07:20:19.537612+0000","last_fullsized":"2024-02-05T07:20:19.537612+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106796+0000","last_change":"2024-02-05T07:18:55.473808+0000","last_active":"2024-02-05T07:20:19.106796+0000","last_peered":"2024-02-05T07:20:19.106796+0000","last_clean":"2024-02-05T07:20:19.106796+0000","last_became_active":"2024-02-05T07:18:55.473509+0000","last_became_peered":"2024-02-05T07:18:55.473509+0000","last_unstale":"2024-02-05T07:20:19.106796+0000","last_undegraded":"2024-02-05T07:20:19.106796+0000","last_fullsized":"2024-02-05T07:20:19.106796+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:22.190977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.263976+0000","last_change":"2024-02-05T07:18:49.434806+0000","last_active":"2024-02-05T07:20:19.263976+0000","last_peered":"2024-02-05T07:20:19.263976+0000","last_clean":"2024-02-05T07:20:19.263976+0000","last_became_active":"2024-02-05T07:18:49.434400+0000","last_became_peered":"2024-02-05T07:18:49.434400+0000","last_unstale":"2024-02-05T07:20:19.263976+0000","last_undegraded":"2024-02-05T07:20:19.263976+0000","last_fullsized":"2024-02-05T07:20:19.263976+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":157,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536907+0000","last_change":"2024-02-05T07:18:52.450728+0000","last_active":"2024-02-05T07:20:19.536907+0000","last_peered":"2024-02-05T07:20:19.536907+0000","last_clean":"2024-02-05T07:20:19.536907+0000","last_became_active":"2024-02-05T07:18:48.424620+0000","last_became_peered":"2024-02-05T07:18:48.424620+0000","last_unstale":"2024-02-05T07:20:19.536907+0000","last_undegraded":"2024-02-05T07:20:19.536907+0000","last_fullsized":"2024-02-05T07:20:19.536907+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:18:47.405257+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264313+0000","last_change":"2024-02-05T07:20:14.321610+0000","last_active":"2024-02-05T07:20:19.264313+0000","last_peered":"2024-02-05T07:20:19.264313+0000","last_clean":"2024-02-05T07:20:19.264313+0000","last_became_active":"2024-02-05T07:18:51.442184+0000","last_became_peered":"2024-02-05T07:18:51.442184+0000","last_unstale":"2024-02-05T07:20:19.264313+0000","last_undegraded":"2024-02-05T07:20:19.264313+0000","last_fullsized":"2024-02-05T07:20:19.264313+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:14.321523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:14.321523+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.745540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.325243+0000","last_change":"2024-02-05T07:18:53.468712+0000","last_active":"2024-02-05T07:20:19.325243+0000","last_peered":"2024-02-05T07:20:19.325243+0000","last_clean":"2024-02-05T07:20:19.325243+0000","last_became_active":"2024-02-05T07:18:53.468512+0000","last_became_peered":"2024-02-05T07:18:53.468512+0000","last_unstale":"2024-02-05T07:20:19.325243+0000","last_undegraded":"2024-02-05T07:20:19.325243+0000","last_fullsized":"2024-02-05T07:20:19.325243+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106842+0000","last_change":"2024-02-05T07:18:55.505742+0000","last_active":"2024-02-05T07:20:19.106842+0000","last_peered":"2024-02-05T07:20:19.106842+0000","last_clean":"2024-02-05T07:20:19.106842+0000","last_became_active":"2024-02-05T07:18:55.505360+0000","last_became_peered":"2024-02-05T07:18:55.505360+0000","last_unstale":"2024-02-05T07:20:19.106842+0000","last_undegraded":"2024-02-05T07:20:19.106842+0000","last_fullsized":"2024-02-05T07:20:19.106842+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107019+0000","last_change":"2024-02-05T07:20:10.436628+0000","last_active":"2024-02-05T07:20:19.107019+0000","last_peered":"2024-02-05T07:20:19.107019+0000","last_clean":"2024-02-05T07:20:19.107019+0000","last_became_active":"2024-02-05T07:18:49.428470+0000","last_became_peered":"2024-02-05T07:18:49.428470+0000","last_unstale":"2024-02-05T07:20:19.107019+0000","last_undegraded":"2024-02-05T07:20:19.107019+0000","last_fullsized":"2024-02-05T07:20:19.107019+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:20:10.436535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:36.573911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":102,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264236+0000","last_change":"2024-02-05T07:20:12.309384+0000","last_active":"2024-02-05T07:20:19.264236+0000","last_peered":"2024-02-05T07:20:19.264236+0000","last_clean":"2024-02-05T07:20:19.264236+0000","last_became_active":"2024-02-05T07:18:51.507507+0000","last_became_peered":"2024-02-05T07:18:51.507507+0000","last_unstale":"2024-02-05T07:20:19.264236+0000","last_undegraded":"2024-02-05T07:20:19.264236+0000","last_fullsized":"2024-02-05T07:20:19.264236+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:20:12.309238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:12.309238+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:40.996099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536526+0000","last_change":"2024-02-05T07:19:56.330173+0000","last_active":"2024-02-05T07:20:19.536526+0000","last_peered":"2024-02-05T07:20:19.536526+0000","last_clean":"2024-02-05T07:20:19.536526+0000","last_became_active":"2024-02-05T07:18:53.459474+0000","last_became_peered":"2024-02-05T07:18:53.459474+0000","last_unstale":"2024-02-05T07:20:19.536526+0000","last_undegraded":"2024-02-05T07:20:19.536526+0000","last_fullsized":"2024-02-05T07:20:19.536526+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:56.330068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:19:56.330068+0000","last_clean_scrub_stamp":"2024-02-05T07:19:56.330068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:22.066765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.263736+0000","last_change":"2024-02-05T07:18:55.474472+0000","last_active":"2024-02-05T07:20:19.263736+0000","last_peered":"2024-02-05T07:20:19.263736+0000","last_clean":"2024-02-05T07:20:19.263736+0000","last_became_active":"2024-02-05T07:18:55.474029+0000","last_became_peered":"2024-02-05T07:18:55.474029+0000","last_unstale":"2024-02-05T07:20:19.263736+0000","last_undegraded":"2024-02-05T07:20:19.263736+0000","last_fullsized":"2024-02-05T07:20:19.263736+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324927+0000","last_change":"2024-02-05T07:18:49.435665+0000","last_active":"2024-02-05T07:20:19.324927+0000","last_peered":"2024-02-05T07:20:19.324927+0000","last_clean":"2024-02-05T07:20:19.324927+0000","last_became_active":"2024-02-05T07:18:49.435299+0000","last_became_peered":"2024-02-05T07:18:49.435299+0000","last_unstale":"2024-02-05T07:20:19.324927+0000","last_undegraded":"2024-02-05T07:20:19.324927+0000","last_fullsized":"2024-02-05T07:20:19.324927+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107565+0000","last_change":"2024-02-05T07:18:51.448399+0000","last_active":"2024-02-05T07:20:19.107565+0000","last_peered":"2024-02-05T07:20:19.107565+0000","last_clean":"2024-02-05T07:20:19.107565+0000","last_became_active":"2024-02-05T07:18:51.448044+0000","last_became_peered":"2024-02-05T07:18:51.448044+0000","last_unstale":"2024-02-05T07:20:19.107565+0000","last_undegraded":"2024-02-05T07:20:19.107565+0000","last_fullsized":"2024-02-05T07:20:19.107565+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264174+0000","last_change":"2024-02-05T07:18:53.463629+0000","last_active":"2024-02-05T07:20:19.264174+0000","last_peered":"2024-02-05T07:20:19.264174+0000","last_clean":"2024-02-05T07:20:19.264174+0000","last_became_active":"2024-02-05T07:18:53.462890+0000","last_became_peered":"2024-02-05T07:18:53.462890+0000","last_unstale":"2024-02-05T07:20:19.264174+0000","last_undegraded":"2024-02-05T07:20:19.264174+0000","last_fullsized":"2024-02-05T07:20:19.264174+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.827616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537132+0000","last_change":"2024-02-05T07:18:55.508071+0000","last_active":"2024-02-05T07:20:19.537132+0000","last_peered":"2024-02-05T07:20:19.537132+0000","last_clean":"2024-02-05T07:20:19.537132+0000","last_became_active":"2024-02-05T07:18:55.507791+0000","last_became_peered":"2024-02-05T07:18:55.507791+0000","last_unstale":"2024-02-05T07:20:19.537132+0000","last_undegraded":"2024-02-05T07:20:19.537132+0000","last_fullsized":"2024-02-05T07:20:19.537132+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:24.052431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264427+0000","last_change":"2024-02-05T07:19:51.364668+0000","last_active":"2024-02-05T07:20:19.264427+0000","last_peered":"2024-02-05T07:20:19.264427+0000","last_clean":"2024-02-05T07:20:19.264427+0000","last_became_active":"2024-02-05T07:18:49.430123+0000","last_became_peered":"2024-02-05T07:18:49.430123+0000","last_unstale":"2024-02-05T07:20:19.264427+0000","last_undegraded":"2024-02-05T07:20:19.264427+0000","last_fullsized":"2024-02-05T07:20:19.264427+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:19:51.364585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.364585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:57.656562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107487+0000","last_change":"2024-02-05T07:18:51.438840+0000","last_active":"2024-02-05T07:20:19.107487+0000","last_peered":"2024-02-05T07:20:19.107487+0000","last_clean":"2024-02-05T07:20:19.107487+0000","last_became_active":"2024-02-05T07:18:51.438507+0000","last_became_peered":"2024-02-05T07:18:51.438507+0000","last_unstale":"2024-02-05T07:20:19.107487+0000","last_undegraded":"2024-02-05T07:20:19.107487+0000","last_fullsized":"2024-02-05T07:20:19.107487+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106217+0000","last_change":"2024-02-05T07:18:53.464137+0000","last_active":"2024-02-05T07:20:19.106217+0000","last_peered":"2024-02-05T07:20:19.106217+0000","last_clean":"2024-02-05T07:20:19.106217+0000","last_became_active":"2024-02-05T07:18:53.463651+0000","last_became_peered":"2024-02-05T07:18:53.463651+0000","last_unstale":"2024-02-05T07:20:19.106217+0000","last_undegraded":"2024-02-05T07:20:19.106217+0000","last_fullsized":"2024-02-05T07:20:19.106217+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107359+0000","last_change":"2024-02-05T07:18:55.474782+0000","last_active":"2024-02-05T07:20:19.107359+0000","last_peered":"2024-02-05T07:20:19.107359+0000","last_clean":"2024-02-05T07:20:19.107359+0000","last_became_active":"2024-02-05T07:18:55.474447+0000","last_became_peered":"2024-02-05T07:18:55.474447+0000","last_unstale":"2024-02-05T07:20:19.107359+0000","last_undegraded":"2024-02-05T07:20:19.107359+0000","last_fullsized":"2024-02-05T07:20:19.107359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107628+0000","last_change":"2024-02-05T07:18:49.435071+0000","last_active":"2024-02-05T07:20:19.107628+0000","last_peered":"2024-02-05T07:20:19.107628+0000","last_clean":"2024-02-05T07:20:19.107628+0000","last_became_active":"2024-02-05T07:18:49.434719+0000","last_became_peered":"2024-02-05T07:18:49.434719+0000","last_unstale":"2024-02-05T07:20:19.107628+0000","last_undegraded":"2024-02-05T07:20:19.107628+0000","last_fullsized":"2024-02-05T07:20:19.107628+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":98,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106115+0000","last_change":"2024-02-05T07:20:00.797048+0000","last_active":"2024-02-05T07:20:19.106115+0000","last_peered":"2024-02-05T07:20:19.106115+0000","last_clean":"2024-02-05T07:20:19.106115+0000","last_became_active":"2024-02-05T07:18:51.507524+0000","last_became_peered":"2024-02-05T07:18:51.507524+0000","last_unstale":"2024-02-05T07:20:19.106115+0000","last_undegraded":"2024-02-05T07:20:19.106115+0000","last_fullsized":"2024-02-05T07:20:19.106115+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:20:00.796905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.796905+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:13.158158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107321+0000","last_change":"2024-02-05T07:18:53.461489+0000","last_active":"2024-02-05T07:20:19.107321+0000","last_peered":"2024-02-05T07:20:19.107321+0000","last_clean":"2024-02-05T07:20:19.107321+0000","last_became_active":"2024-02-05T07:18:53.461213+0000","last_became_peered":"2024-02-05T07:18:53.461213+0000","last_unstale":"2024-02-05T07:20:19.107321+0000","last_undegraded":"2024-02-05T07:20:19.107321+0000","last_fullsized":"2024-02-05T07:20:19.107321+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324763+0000","last_change":"2024-02-05T07:18:55.472546+0000","last_active":"2024-02-05T07:20:19.324763+0000","last_peered":"2024-02-05T07:20:19.324763+0000","last_clean":"2024-02-05T07:20:19.324763+0000","last_became_active":"2024-02-05T07:18:55.472172+0000","last_became_peered":"2024-02-05T07:18:55.472172+0000","last_unstale":"2024-02-05T07:20:19.324763+0000","last_undegraded":"2024-02-05T07:20:19.324763+0000","last_fullsized":"2024-02-05T07:20:19.324763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.108120+0000","last_change":"2024-02-05T07:18:53.460159+0000","last_active":"2024-02-05T07:20:19.108120+0000","last_peered":"2024-02-05T07:20:19.108120+0000","last_clean":"2024-02-05T07:20:19.108120+0000","last_became_active":"2024-02-05T07:18:53.459832+0000","last_became_peered":"2024-02-05T07:18:53.459832+0000","last_unstale":"2024-02-05T07:20:19.108120+0000","last_undegraded":"2024-02-05T07:20:19.108120+0000","last_fullsized":"2024-02-05T07:20:19.108120+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106090+0000","last_change":"2024-02-05T07:18:49.431779+0000","last_active":"2024-02-05T07:20:19.106090+0000","last_peered":"2024-02-05T07:20:19.106090+0000","last_clean":"2024-02-05T07:20:19.106090+0000","last_became_active":"2024-02-05T07:18:49.431369+0000","last_became_peered":"2024-02-05T07:18:49.431369+0000","last_unstale":"2024-02-05T07:20:19.106090+0000","last_undegraded":"2024-02-05T07:20:19.106090+0000","last_fullsized":"2024-02-05T07:20:19.106090+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.108153+0000","last_change":"2024-02-05T07:18:51.447767+0000","last_active":"2024-02-05T07:20:19.108153+0000","last_peered":"2024-02-05T07:20:19.108153+0000","last_clean":"2024-02-05T07:20:19.108153+0000","last_became_active":"2024-02-05T07:18:51.447469+0000","last_became_peered":"2024-02-05T07:18:51.447469+0000","last_unstale":"2024-02-05T07:20:19.108153+0000","last_undegraded":"2024-02-05T07:20:19.108153+0000","last_fullsized":"2024-02-05T07:20:19.108153+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.124394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106723+0000","last_change":"2024-02-05T07:20:10.764040+0000","last_active":"2024-02-05T07:20:19.106723+0000","last_peered":"2024-02-05T07:20:19.106723+0000","last_clean":"2024-02-05T07:20:19.106723+0000","last_became_active":"2024-02-05T07:18:55.505340+0000","last_became_peered":"2024-02-05T07:18:55.505340+0000","last_unstale":"2024-02-05T07:20:19.106723+0000","last_undegraded":"2024-02-05T07:20:19.106723+0000","last_fullsized":"2024-02-05T07:20:19.106723+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:10.763957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:10.763957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:35.465090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.325203+0000","last_change":"2024-02-05T07:18:53.462879+0000","last_active":"2024-02-05T07:20:19.325203+0000","last_peered":"2024-02-05T07:20:19.325203+0000","last_clean":"2024-02-05T07:20:19.325203+0000","last_became_active":"2024-02-05T07:18:53.462591+0000","last_became_peered":"2024-02-05T07:18:53.462591+0000","last_unstale":"2024-02-05T07:20:19.325203+0000","last_undegraded":"2024-02-05T07:20:19.325203+0000","last_fullsized":"2024-02-05T07:20:19.325203+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:20.782612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107356+0000","last_change":"2024-02-05T07:20:05.883941+0000","last_active":"2024-02-05T07:20:19.107356+0000","last_peered":"2024-02-05T07:20:19.107356+0000","last_clean":"2024-02-05T07:20:19.107356+0000","last_became_active":"2024-02-05T07:18:49.434074+0000","last_became_peered":"2024-02-05T07:18:49.434074+0000","last_unstale":"2024-02-05T07:20:19.107356+0000","last_undegraded":"2024-02-05T07:20:19.107356+0000","last_fullsized":"2024-02-05T07:20:19.107356+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:05.883840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:05.883840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:30.747182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"90'16","reported_seq":90,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107864+0000","last_change":"2024-02-05T07:20:09.873512+0000","last_active":"2024-02-05T07:20:19.107864+0000","last_peered":"2024-02-05T07:20:19.107864+0000","last_clean":"2024-02-05T07:20:19.107864+0000","last_became_active":"2024-02-05T07:18:51.447922+0000","last_became_peered":"2024-02-05T07:18:51.447922+0000","last_unstale":"2024-02-05T07:20:19.107864+0000","last_undegraded":"2024-02-05T07:20:19.107864+0000","last_fullsized":"2024-02-05T07:20:19.107864+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.873388+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.334535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537235+0000","last_change":"2024-02-05T07:18:55.475305+0000","last_active":"2024-02-05T07:20:19.537235+0000","last_peered":"2024-02-05T07:20:19.537235+0000","last_clean":"2024-02-05T07:20:19.537235+0000","last_became_active":"2024-02-05T07:18:55.474992+0000","last_became_peered":"2024-02-05T07:18:55.474992+0000","last_unstale":"2024-02-05T07:20:19.537235+0000","last_undegraded":"2024-02-05T07:20:19.537235+0000","last_fullsized":"2024-02-05T07:20:19.537235+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107260+0000","last_change":"2024-02-05T07:20:06.398771+0000","last_active":"2024-02-05T07:20:19.107260+0000","last_peered":"2024-02-05T07:20:19.107260+0000","last_clean":"2024-02-05T07:20:19.107260+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:20:19.107260+0000","last_undegraded":"2024-02-05T07:20:19.107260+0000","last_fullsized":"2024-02-05T07:20:19.107260+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:06.398697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.398697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.245750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536412+0000","last_change":"2024-02-05T07:20:18.399191+0000","last_active":"2024-02-05T07:20:19.536412+0000","last_peered":"2024-02-05T07:20:19.536412+0000","last_clean":"2024-02-05T07:20:19.536412+0000","last_became_active":"2024-02-05T07:18:49.430518+0000","last_became_peered":"2024-02-05T07:18:49.430518+0000","last_unstale":"2024-02-05T07:20:19.536412+0000","last_undegraded":"2024-02-05T07:20:19.536412+0000","last_fullsized":"2024-02-05T07:20:19.536412+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:18.399075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.399075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:36.367477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":94,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.108030+0000","last_change":"2024-02-05T07:19:51.808315+0000","last_active":"2024-02-05T07:20:19.108030+0000","last_peered":"2024-02-05T07:20:19.108030+0000","last_clean":"2024-02-05T07:20:19.108030+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:20:19.108030+0000","last_undegraded":"2024-02-05T07:20:19.108030+0000","last_fullsized":"2024-02-05T07:20:19.108030+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:19:51.808218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.808218+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:55.279939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.108177+0000","last_change":"2024-02-05T07:18:55.466723+0000","last_active":"2024-02-05T07:20:19.108177+0000","last_peered":"2024-02-05T07:20:19.108177+0000","last_clean":"2024-02-05T07:20:19.108177+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:20:19.108177+0000","last_undegraded":"2024-02-05T07:20:19.108177+0000","last_fullsized":"2024-02-05T07:20:19.108177+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536797+0000","last_change":"2024-02-05T07:20:18.512034+0000","last_active":"2024-02-05T07:20:19.536797+0000","last_peered":"2024-02-05T07:20:19.536797+0000","last_clean":"2024-02-05T07:20:19.536797+0000","last_became_active":"2024-02-05T07:18:53.461852+0000","last_became_peered":"2024-02-05T07:18:53.461852+0000","last_unstale":"2024-02-05T07:20:19.536797+0000","last_undegraded":"2024-02-05T07:20:19.536797+0000","last_fullsized":"2024-02-05T07:20:19.536797+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:20:18.511957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.511957+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:23.253319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107076+0000","last_change":"2024-02-05T07:20:04.827259+0000","last_active":"2024-02-05T07:20:19.107076+0000","last_peered":"2024-02-05T07:20:19.107076+0000","last_clean":"2024-02-05T07:20:19.107076+0000","last_became_active":"2024-02-05T07:18:49.435368+0000","last_became_peered":"2024-02-05T07:18:49.435368+0000","last_unstale":"2024-02-05T07:20:19.107076+0000","last_undegraded":"2024-02-05T07:20:19.107076+0000","last_fullsized":"2024-02-05T07:20:19.107076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:04.827180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:04.827180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:17.606501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107566+0000","last_change":"2024-02-05T07:20:13.843192+0000","last_active":"2024-02-05T07:20:19.107566+0000","last_peered":"2024-02-05T07:20:19.107566+0000","last_clean":"2024-02-05T07:20:19.107566+0000","last_became_active":"2024-02-05T07:18:51.442711+0000","last_became_peered":"2024-02-05T07:18:51.442711+0000","last_unstale":"2024-02-05T07:20:19.107566+0000","last_undegraded":"2024-02-05T07:20:19.107566+0000","last_fullsized":"2024-02-05T07:20:19.107566+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:20:13.843051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:13.843051+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:41.856009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":402,"ondisk_log_size":402,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037783,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28440,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716712,"statfs":{"total":95995035648,"available":95965913088,"internally_reserved":0,"allocated":1454080,"data_stored":970648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.623},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.76800000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.60999999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.67600000000000005}]},{"osd":3,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.63800000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.496}]},{"osd":5,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46100000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.55900000000000005}]}]},{"osd":6,"up_from":52,"seq":223338299421,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32572,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712580,"statfs":{"total":95995035648,"available":95961681920,"internally_reserved":0,"allocated":1429504,"data_stored":964410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.35399999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70299999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.503},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.40000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.67100000000000004,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.66300000000000003}]},{"osd":1,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68700000000000006,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38500000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.747,"5min":0.751,"15min":0.751},"last":0.64500000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.42199999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77000000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.53000000000000003,"15min":0.53100000000000003},"min":{"1min":0.36899999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.41199999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68999999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38200000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.54900000000000004}]},{"osd":4,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.41299999999999998,"15min":0.40500000000000003},"min":{"1min":0.28699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.45300000000000001},"min":{"1min":0.314,"5min":0.313,"15min":0.313},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45200000000000001,"15min":0.44700000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.63},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.46899999999999997,"15min":0.45800000000000002},"min":{"1min":0.314,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.53600000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.61599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":897024,"data_stored":381461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.36199999999999999,"15min":0.33400000000000002},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.45200000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.318},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.376,"15min":0.33700000000000002},"min":{"1min":0.23000000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55100000000000005,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.46400000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.442,"15min":0.44600000000000001},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.60799999999999998,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.88,"15min":1.0740000000000001},"min":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.61299999999999999,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.28399999999999997}]},{"osd":3,"last update":"Mon Feb 5 07:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.45800000000000002,"15min":0.47499999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.63800000000000001,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.47999999999999998,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68100000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.34499999999999997}]},{"osd":4,"last update":"Mon Feb 5 07:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59099999999999997,"15min":0.58999999999999997},"min":{"1min":0.44600000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.72999999999999998,"15min":0.753},"min":{"1min":0.41899999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.0069999999999999,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.44800000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61699999999999999,"15min":0.622},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.435},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.75700000000000001,"15min":0.78100000000000003},"min":{"1min":0.46500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.04,"5min":5.0289999999999999,"15min":5.0289999999999999},"last":0.53300000000000003}]},{"osd":6,"last update":"Mon Feb 5 07:20:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.66900000000000004,"15min":0.68100000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.97299999999999998,"5min":2.48,"15min":2.48},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.66200000000000003,"15min":0.67700000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0569999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":0.40999999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.42599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28448,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716704,"statfs":{"total":95995035648,"available":95965904896,"internally_reserved":0,"allocated":1462272,"data_stored":970789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.34300000000000003,"15min":0.31},"min":{"1min":0.23499999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.32500000000000001},"min":{"1min":0.26700000000000002,"5min":0.214,"15min":0.214},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.439,"15min":0.42799999999999999},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.42799999999999999,"15min":0.42099999999999999},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.28000000000000003}]},{"osd":3,"last update":"Mon Feb 5 07:20:00 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42899999999999999,"15min":0.43099999999999999},"min":{"1min":0.29899999999999999,"5min":0.246,"15min":0.246},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.375},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41399999999999998,"15min":0.41799999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.42499999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.47599999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66000000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.54900000000000004},"min":{"1min":0.41999999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.46400000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56000000000000005,"15min":0.55400000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.496}]},{"osd":6,"last update":"Mon Feb 5 07:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.54000000000000004}]},{"osd":7,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.59499999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717300,"statfs":{"total":95995035648,"available":95966515200,"internally_reserved":0,"allocated":851968,"data_stored":380547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.377,"15min":0.36899999999999999},"min":{"1min":0.23499999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.40100000000000002,"15min":0.38600000000000001},"min":{"1min":0.27400000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.28499999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.39100000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.44500000000000001,"15min":0.42699999999999999},"min":{"1min":0.29099999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.438}]},{"osd":3,"last update":"Mon Feb 5 07:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.46700000000000003,"15min":0.46899999999999997},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.56000000000000005,"5min":1.631,"15min":1.631},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46600000000000003,"15min":0.46400000000000002},"min":{"1min":0.32500000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.55000000000000004,"5min":1.391,"15min":1.391},"last":0.52100000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54100000000000004,"15min":0.53200000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.375},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.56899999999999995,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.73999999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:19:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.61699999999999999,"15min":0.60699999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.72799999999999998,"15min":0.75700000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86099999999999999,"5min":2.8319999999999999,"15min":2.8319999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.59799999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.72599999999999998,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.41199999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"last":0.5},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.59799999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32040,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713112,"statfs":{"total":95995035648,"available":95962226688,"internally_reserved":0,"allocated":880640,"data_stored":380133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39300000000000002,"15min":0.40500000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.52500000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.25},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.44400000000000001,"15min":0.45700000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.44}]},{"osd":1,"last update":"Mon Feb 5 07:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45000000000000001,"15min":0.46800000000000003},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":0.63500000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48199999999999998,"15min":0.503},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57999999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.34699999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.45600000000000002,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.61899999999999999,"5min":0.75,"15min":0.75},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.47499999999999998,"15min":0.495},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.38800000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.58899999999999997,"15min":0.59599999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.70999999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60199999999999998,"15min":0.59599999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.5779999999999998,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":0.58899999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79400000000000004,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59699999999999998,"15min":0.59499999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56699999999999995}]},{"osd":6,"last update":"Mon Feb 5 07:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58599999999999997,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.752,"5min":1.046,"15min":1.046},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.81100000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.53300000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.629},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.51800000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953510,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32028,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713124,"statfs":{"total":95995035648,"available":95962238976,"internally_reserved":0,"allocated":868352,"data_stored":379834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.74199999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.498},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.65000000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57199999999999995}]},{"osd":1,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.36599999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75700000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71499999999999997,"5min":0.754,"15min":0.754},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53300000000000003,"15min":0.54000000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.36399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69899999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.59299999999999997}]},{"osd":3,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55600000000000005,"15min":0.56100000000000005},"min":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72999999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.67200000000000004,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.55700000000000005}]},{"osd":5,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.376,"15min":0.36599999999999999},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.39000000000000001,"15min":0.378},"min":{"1min":0.309,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.57499999999999996,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.42899999999999999,"15min":0.41499999999999998},"min":{"1min":0.30599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.439},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.42699999999999999,"15min":0.40799999999999997},"min":{"1min":0.34999999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.54400000000000004}]},{"osd":7,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64400000000000002}]}]},{"osd":5,"up_from":41,"seq":176093659169,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":843776,"data_stored":373616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66800000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71199999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.56200000000000006}]},{"osd":1,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.55400000000000005,"15min":0.56299999999999994},"min":{"1min":0.35399999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.69699999999999995,"5min":0.753,"15min":0.753},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.5,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.38200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.68200000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.53100000000000003}]},{"osd":2,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68000000000000005,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.35099999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69999999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.61299999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64200000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71999999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.67900000000000005}]},{"osd":4,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.35999999999999999,"15min":0.34799999999999998},"min":{"1min":0.26800000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.38900000000000001,"15min":0.373},"min":{"1min":0.33200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Feb 5 07:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.42099999999999999,"15min":0.40200000000000002},"min":{"1min":0.31900000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.47699999999999998,"15min":0.45300000000000001},"min":{"1min":0.41799999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Feb 5 07:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:20:22.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:20:22.464 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[29098]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T07:20:22.464 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[29098]: 3.1 deep-scrub starts 2024-02-05T07:20:22.464 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[29098]: 3.1 deep-scrub ok 2024-02-05T07:20:22.464 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:20:22.464 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[37158]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T07:20:22.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[37158]: 3.1 deep-scrub starts 2024-02-05T07:20:22.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[37158]: 3.1 deep-scrub ok 2024-02-05T07:20:22.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:20:22.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:20:22.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[37158]: osdmap e95: 8 total, 8 up, 8 in 2024-02-05T07:20:22.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[37158]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:22.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[37158]: 5.12 scrub starts 2024-02-05T07:20:22.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[37158]: 5.12 scrub ok 2024-02-05T07:20:22.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:22.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:22.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:22.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:22.485 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-mon[33213]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T07:20:22.485 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-mon[33213]: 3.1 deep-scrub starts 2024-02-05T07:20:22.485 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-mon[33213]: 3.1 deep-scrub ok 2024-02-05T07:20:22.485 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:20:22.485 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:20:22.486 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-mon[33213]: osdmap e95: 8 total, 8 up, 8 in 2024-02-05T07:20:22.486 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-mon[33213]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:22.486 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-mon[33213]: 5.12 scrub starts 2024-02-05T07:20:22.486 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-mon[33213]: 5.12 scrub ok 2024-02-05T07:20:22.486 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:22.486 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:22.486 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:22.486 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:22.489 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.240214234Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-02-05T07:20:22.489 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.24068026Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-02-05T07:20:22.489 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.24114268Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-02-05T07:20:22.489 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.241676444Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-02-05T07:20:22.489 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.244446864Z level=info msg="Executing migration" id="Update annotation table charset" 2024-02-05T07:20:22.490 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.244746634Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-02-05T07:20:22.490 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.245369457Z level=info msg="Executing migration" id="Drop category_id index" 2024-02-05T07:20:22.490 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.245786634Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-02-05T07:20:22.490 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.246421741Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-02-05T07:20:22.490 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.246833819Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-05T07:20:22.491 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.247269146Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-05T07:20:22.491 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.247688116Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-05T07:20:22.491 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.250432343Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-02-05T07:20:22.491 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.253354578Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-05T07:20:22.491 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.25592575Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-02-05T07:20:22.491 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.256267276Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-02-05T07:20:22.491 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.258970173Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-02-05T07:20:22.492 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.261747855Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-02-05T07:20:22.492 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.264446854Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-02-05T07:20:22.492 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.265182159Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-02-05T07:20:22.493 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.267980896Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-02-05T07:20:22.493 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.270780869Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-02-05T07:20:22.493 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.273357611Z level=info msg="Executing migration" id="Add epoch_end column" 2024-02-05T07:20:22.493 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.274172198Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-02-05T07:20:22.493 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.276924443Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-02-05T07:20:22.493 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.279609851Z level=info msg="Executing migration" id="Move region to single row" 2024-02-05T07:20:22.493 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.282246721Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-02-05T07:20:22.493 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.28274313Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-05T07:20:22.494 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.285610054Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-05T07:20:22.494 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.286276543Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-05T07:20:22.494 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.287159463Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-05T07:20:22.494 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.287629174Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-02-05T07:20:22.494 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.288641135Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-02-05T07:20:22.494 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.289398559Z level=info msg="Executing migration" id="create test_data table" 2024-02-05T07:20:22.495 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.289867886Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-02-05T07:20:22.495 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.290294415Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-02-05T07:20:22.495 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.291073252Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-05T07:20:22.495 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.291890407Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-02-05T07:20:22.495 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.292692864Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-02-05T07:20:22.495 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.293055973Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-02-05T07:20:22.495 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.293828432Z level=info msg="Executing migration" id="create team table" 2024-02-05T07:20:22.496 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.2942787Z level=info msg="Executing migration" id="add index team.org_id" 2024-02-05T07:20:22.496 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.294762373Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-02-05T07:20:22.496 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.295794334Z level=info msg="Executing migration" id="create team member table" 2024-02-05T07:20:22.497 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.29633335Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-02-05T07:20:22.497 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.297178376Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-02-05T07:20:22.497 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.298155205Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-02-05T07:20:22.497 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.298840877Z level=info msg="Executing migration" id="Add column email to team table" 2024-02-05T07:20:22.497 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.299587642Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-02-05T07:20:22.497 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.300467537Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-02-05T07:20:22.497 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.30131824Z level=info msg="Executing migration" id="create dashboard acl table" 2024-02-05T07:20:22.498 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.302392104Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-02-05T07:20:22.498 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.302906784Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-05T07:20:22.498 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.303729731Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-05T07:20:22.498 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.304233936Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-02-05T07:20:22.498 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.305075103Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-02-05T07:20:22.498 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.305580061Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-02-05T07:20:22.499 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.306443816Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-02-05T07:20:22.499 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.307406043Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-02-05T07:20:22.499 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.308131953Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-02-05T07:20:22.499 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.309010079Z level=info msg="Executing migration" id="create tag table" 2024-02-05T07:20:22.499 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.309575476Z level=info msg="Executing migration" id="add index tag.key_value" 2024-02-05T07:20:22.499 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.310066448Z level=info msg="Executing migration" id="create login attempt table" 2024-02-05T07:20:22.499 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.310887128Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-02-05T07:20:22.499 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.311912396Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-02-05T07:20:22.499 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.312674617Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-05T07:20:22.500 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.315527457Z level=info msg="Executing migration" id="create login_attempt v2" 2024-02-05T07:20:22.500 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.318637391Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-02-05T07:20:22.500 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.321281839Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-02-05T07:20:22.500 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.324255668Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-02-05T07:20:22.500 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.327065894Z level=info msg="Executing migration" id="create user auth table" 2024-02-05T07:20:22.500 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.329977896Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-05T07:20:22.500 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.332805741Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-02-05T07:20:22.500 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.33563515Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-02-05T07:20:22.500 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.338316996Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.340967103Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.343788717Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.346653964Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.349392254Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.352348449Z level=info msg="Executing migration" id="create server_lock table" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.355094728Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.357851145Z level=info msg="Executing migration" id="create user auth token table" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.360662028Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.363502172Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.366301534Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.369126877Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.371827226Z level=info msg="Executing migration" id="create cache_data table" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.3745639Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.377267781Z level=info msg="Executing migration" id="create short_url table v1" 2024-02-05T07:20:22.501 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.379991354Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-02-05T07:20:22.502 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.382785833Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-02-05T07:20:22.502 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.385394451Z level=info msg="Executing migration" id="delete alert_definition table" 2024-02-05T07:20:22.502 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.385765073Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-02-05T07:20:22.502 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.386223502Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-02-05T07:20:22.502 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.389068328Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-02-05T07:20:22.502 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.389731948Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-02-05T07:20:22.502 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.390025613Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-02-05T07:20:22.502 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.390399071Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-02-05T07:20:22.502 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.390810206Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-02-05T07:20:22.502 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.391222827Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-02-05T07:20:22.502 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.391650074Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-02-05T07:20:22.502 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.393337497Z level=info msg="Executing migration" id="drop alert_definition table" 2024-02-05T07:20:22.502 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.395996644Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-02-05T07:20:22.502 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.398610618Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.401291561Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.403944653Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.406650783Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.407680841Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.408424411Z level=info msg="Executing migration" id="create alert_instance table" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.409238156Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.410124658Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.410977113Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.411774103Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.412740339Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.413495156Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.416821646Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.420104788Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.42091382Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-02-05T07:20:22.503 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.421812201Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-02-05T07:20:22.504 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.422682079Z level=info msg="Executing migration" id="create alert_rule table" 2024-02-05T07:20:22.504 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.423518763Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-02-05T07:20:22.504 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.424336159Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-02-05T07:20:22.504 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.425144676Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-05T07:20:22.504 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.425956386Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-02-05T07:20:22.504 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.426780014Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-02-05T07:20:22.504 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.427634942Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-02-05T07:20:22.504 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.428491801Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-02-05T07:20:22.504 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.429377215Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-02-05T07:20:22.504 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.430264919Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-05T07:20:22.504 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.43111065Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-02-05T07:20:22.504 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.432030815Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-02-05T07:20:22.504 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.432893392Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-05T07:20:22.504 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.433730744Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.434592744Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.435436451Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.436422963Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.437159955Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.438026027Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.438868091Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.43979126Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.440893454Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.441770807Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.444821731Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.447605829Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.450581479Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.453375166Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.456089688Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-02-05T07:20:22.505 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.458693496Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-05T07:20:22.506 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.461469076Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-02-05T07:20:22.506 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.46430779Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-02-05T07:20:22.506 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.467425785Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-02-05T07:20:22.506 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.470184016Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-02-05T07:20:22.506 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.472940739Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-02-05T07:20:22.506 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.475818517Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-02-05T07:20:22.506 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.478676156Z level=info msg="Executing migration" id="create provenance_type table" 2024-02-05T07:20:22.506 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.481411074Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-02-05T07:20:22.506 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.484083168Z level=info msg="Executing migration" id="create alert_image table" 2024-02-05T07:20:22.506 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.486945961Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-02-05T07:20:22.518 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:22.753 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.489658993Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-02-05T07:20:22.754 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.492408113Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-02-05T07:20:22.754 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.495080858Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-02-05T07:20:22.754 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.497907024Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-02-05T07:20:22.754 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.498112021Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-02-05T07:20:22.754 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.500608415Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-02-05T07:20:22.754 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.503536499Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-02-05T07:20:22.754 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.506065783Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-02-05T07:20:22.754 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.506201905Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-02-05T07:20:22.755 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.506311382Z level=info msg="alerts found to migrate" alerts=0 2024-02-05T07:20:22.755 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.508906882Z level=info msg="Executing migration" id="create library_element table v1" 2024-02-05T07:20:22.755 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.511533023Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-02-05T07:20:22.755 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.514375743Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-02-05T07:20:22.755 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.517121252Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-02-05T07:20:22.755 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.519758667Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-02-05T07:20:22.755 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.522601124Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-02-05T07:20:22.755 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.525322034Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-02-05T07:20:22.756 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.528110367Z level=info msg="Executing migration" id="create data_keys table" 2024-02-05T07:20:22.757 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.530616053Z level=info msg="Executing migration" id="create secrets table" 2024-02-05T07:20:22.757 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.533266411Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-02-05T07:20:22.757 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.537542315Z level=info msg="Executing migration" id="add name column into data_keys" 2024-02-05T07:20:22.757 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.540274271Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-02-05T07:20:22.757 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.541258436Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-02-05T07:20:22.758 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.545499114Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-02-05T07:20:22.758 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.549677922Z level=info msg="Executing migration" id="create kv_store table v1" 2024-02-05T07:20:22.758 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.550553899Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-02-05T07:20:22.758 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.55148687Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-02-05T07:20:22.758 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.552160999Z level=info msg="Executing migration" id="create permission table" 2024-02-05T07:20:22.758 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.55302211Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-02-05T07:20:22.758 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.553921374Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-02-05T07:20:22.758 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.554754107Z level=info msg="Executing migration" id="create role table" 2024-02-05T07:20:22.759 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.555684833Z level=info msg="Executing migration" id="add column display_name" 2024-02-05T07:20:22.759 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.556798503Z level=info msg="Executing migration" id="add column group_name" 2024-02-05T07:20:22.759 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.557806954Z level=info msg="Executing migration" id="add index role.org_id" 2024-02-05T07:20:22.759 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.558735531Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-02-05T07:20:22.759 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.559594518Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-02-05T07:20:22.759 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.560269803Z level=info msg="Executing migration" id="create team role table" 2024-02-05T07:20:22.759 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.56108278Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-02-05T07:20:22.759 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.561907474Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-02-05T07:20:22.759 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.562718202Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-02-05T07:20:22.759 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.563632812Z level=info msg="Executing migration" id="create user role table" 2024-02-05T07:20:22.759 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.564513387Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-02-05T07:20:22.760 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.565506996Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-02-05T07:20:22.760 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.566246693Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-02-05T07:20:22.760 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.567184922Z level=info msg="Executing migration" id="create builtin role table" 2024-02-05T07:20:22.760 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.567926387Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-02-05T07:20:22.760 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.568760322Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-02-05T07:20:22.760 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.569721822Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-02-05T07:20:22.760 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.57085726Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-02-05T07:20:22.760 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.571743161Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-02-05T07:20:22.760 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.572761917Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-02-05T07:20:22.760 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.573434133Z level=info msg="Executing migration" id="add unique index role.uid" 2024-02-05T07:20:22.760 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.57431472Z level=info msg="Executing migration" id="create seed assignment table" 2024-02-05T07:20:22.760 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.575157188Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-02-05T07:20:22.760 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.576051615Z level=info msg="Executing migration" id="add column hidden to role table" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.577065995Z level=info msg="Executing migration" id="create query_history table v1" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.577959182Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.580910697Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.583766882Z level=info msg="Executing migration" id="teams permissions migration" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.586690641Z level=info msg="Executing migration" id="dashboard permissions" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.58951967Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.592536733Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.595380711Z level=info msg="Executing migration" id="alerting notification permissions" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.598140019Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.600942014Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.603985768Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.606570455Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.606910509Z level=info msg="Executing migration" id="create correlation table v1" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.607379581Z level=info msg="Executing migration" id="add index correlations.uid" 2024-02-05T07:20:22.761 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.607853604Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-02-05T07:20:22.762 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.608307762Z level=info msg="Executing migration" id="add correlation config column" 2024-02-05T07:20:22.762 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.609316206Z level=info msg="Executing migration" id="create entity_events table" 2024-02-05T07:20:22.762 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.609785751Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-02-05T07:20:22.762 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.612629284Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-05T07:20:22.762 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.612757306Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-05T07:20:22.762 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.613130315Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-05T07:20:22.762 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.613223009Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-05T07:20:22.762 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.613496005Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-02-05T07:20:22.762 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.613910172Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-02-05T07:20:22.762 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.614294878Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-02-05T07:20:22.762 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.61474621Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-05T07:20:22.762 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.615184663Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-02-05T07:20:22.762 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.615597304Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-05T07:20:22.763 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.616015205Z level=info msg="Executing migration" id="Drop public config table" 2024-02-05T07:20:22.763 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.616392696Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-02-05T07:20:22.763 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.616840183Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-02-05T07:20:22.763 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.617240106Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-05T07:20:22.763 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.617693231Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-02-05T07:20:22.763 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.618115503Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-02-05T07:20:22.763 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.621090781Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-02-05T07:20:22.763 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.622157692Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-02-05T07:20:22.763 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.623177526Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-02-05T07:20:22.763 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.62351951Z level=info msg="Executing migration" id="add share column" 2024-02-05T07:20:22.763 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.624510903Z level=info msg="Executing migration" id="create default alerting folders" 2024-02-05T07:20:22.763 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.624945757Z level=info msg="Executing migration" id="create file table" 2024-02-05T07:20:22.763 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.625370644Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-02-05T07:20:22.763 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.625819719Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-02-05T07:20:22.764 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.628174741Z level=info msg="Executing migration" id="create file_meta table" 2024-02-05T07:20:22.764 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.631039846Z level=info msg="Executing migration" id="file table idx: path key" 2024-02-05T07:20:22.764 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.633777384Z level=info msg="Executing migration" id="set path collation in file table" 2024-02-05T07:20:22.764 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.636422401Z level=info msg="Executing migration" id="managed permissions migration" 2024-02-05T07:20:22.764 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.639284055Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-02-05T07:20:22.764 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.642134662Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-02-05T07:20:22.764 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.644999489Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-02-05T07:20:22.764 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.646214464Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-02-05T07:20:22.764 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.649000921Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-02-05T07:20:22.764 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.651822379Z level=info msg="Executing migration" id="update group index for alert rules" 2024-02-05T07:20:22.764 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.654689784Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-02-05T07:20:22.764 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.655152124Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-02-05T07:20:22.764 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.657923132Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-02-05T07:20:22.764 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.659101981Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-02-05T07:20:22.765 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.66071762Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-02-05T07:20:22.765 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.663645549Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-02-05T07:20:22.765 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.673293757Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-02-05T07:20:22.765 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.676092434Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-02-05T07:20:22.765 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.678802098Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-02-05T07:20:22.765 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.682143152Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-02-05T07:20:22.765 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.682511993Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-02-05T07:20:22.765 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.685384807Z level=info msg="Executing migration" id="create folder table" 2024-02-05T07:20:22.765 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.687900268Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-02-05T07:20:22.766 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.69048173Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-02-05T07:20:22.766 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.690949822Z level=info msg="Executing migration" id="Update folder title length" 2024-02-05T07:20:22.766 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.693742928Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-02-05T07:20:22.766 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=migrator t=2024-02-05T07:20:22.694253757Z level=info msg="migrations completed" performed=485 skipped=0 duration=709.315508ms 2024-02-05T07:20:22.766 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=sqlstore t=2024-02-05T07:20:22.694655755Z level=info msg="Created default organization" 2024-02-05T07:20:22.766 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=plugin.loader t=2024-02-05T07:20:22.751734913Z level=info msg="Plugin registered" pluginID=input 2024-02-05T07:20:22.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:20:22.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[29098]: osdmap e95: 8 total, 8 up, 8 in 2024-02-05T07:20:22.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[29098]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:22.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[29098]: 5.12 scrub starts 2024-02-05T07:20:22.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[29098]: 5.12 scrub ok 2024-02-05T07:20:22.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:22.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:22.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:22.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:22 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:23.024 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=plugin.loader t=2024-02-05T07:20:22.780495865Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-02-05T07:20:23.025 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=plugin.loader t=2024-02-05T07:20:22.780520961Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-02-05T07:20:23.025 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=secrets t=2024-02-05T07:20:22.780581971Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-02-05T07:20:23.025 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=query_data t=2024-02-05T07:20:22.78435799Z level=info msg="Query Service initialization" 2024-02-05T07:20:23.025 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=live.push_http t=2024-02-05T07:20:22.786751782Z level=info msg="Live Push Gateway initialization" 2024-02-05T07:20:23.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-mon[29098]: Deploying daemon node-exporter.a on smithi007 2024-02-05T07:20:23.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-mon[29098]: 4.1e scrub starts 2024-02-05T07:20:23.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-mon[29098]: 4.1e scrub ok 2024-02-05T07:20:23.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-mon[29098]: 5.1f scrub starts 2024-02-05T07:20:23.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-mon[29098]: 5.1f scrub ok 2024-02-05T07:20:23.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-mon[29098]: 4.2 scrub starts 2024-02-05T07:20:23.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-mon[29098]: 4.2 scrub ok 2024-02-05T07:20:23.675 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:20:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:20:23.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-mon[37158]: Deploying daemon node-exporter.a on smithi007 2024-02-05T07:20:23.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-mon[37158]: 4.1e scrub starts 2024-02-05T07:20:23.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-mon[37158]: 4.1e scrub ok 2024-02-05T07:20:23.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-mon[37158]: 5.1f scrub starts 2024-02-05T07:20:23.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-mon[37158]: 5.1f scrub ok 2024-02-05T07:20:23.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-mon[37158]: 4.2 scrub starts 2024-02-05T07:20:23.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:23 smithi007 ceph-mon[37158]: 4.2 scrub ok 2024-02-05T07:20:23.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-mon[33213]: Deploying daemon node-exporter.a on smithi007 2024-02-05T07:20:23.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-mon[33213]: 4.1e scrub starts 2024-02-05T07:20:23.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-mon[33213]: 4.1e scrub ok 2024-02-05T07:20:23.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-mon[33213]: 5.1f scrub starts 2024-02-05T07:20:23.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-mon[33213]: 5.1f scrub ok 2024-02-05T07:20:23.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-mon[33213]: 4.2 scrub starts 2024-02-05T07:20:23.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-mon[33213]: 4.2 scrub ok 2024-02-05T07:20:23.775 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=infra.usagestats.collector t=2024-02-05T07:20:23.388818691Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-02-05T07:20:23.775 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=provisioning.datasources t=2024-02-05T07:20:23.389930078Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-05T07:20:23.775 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=provisioning.datasources t=2024-02-05T07:20:23.400694689Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-02-05T07:20:23.775 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=provisioning.alerting t=2024-02-05T07:20:23.411242376Z level=info msg="starting to provision alerting" 2024-02-05T07:20:23.776 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=provisioning.alerting t=2024-02-05T07:20:23.41126042Z level=info msg="finished to provision alerting" 2024-02-05T07:20:23.776 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=ngalert.state.manager t=2024-02-05T07:20:23.411348843Z level=info msg="Warming state cache for startup" 2024-02-05T07:20:23.776 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=grafanaStorageLogger t=2024-02-05T07:20:23.411423934Z level=info msg="storage starting" 2024-02-05T07:20:23.776 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=ngalert.state.manager t=2024-02-05T07:20:23.411551506Z level=info msg="State cache has been initialized" states=0 duration=201.781µs 2024-02-05T07:20:23.776 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=ticker t=2024-02-05T07:20:23.411698764Z level=info msg=starting first_tick=2024-02-05T07:20:30Z 2024-02-05T07:20:23.776 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=ngalert.multiorg.alertmanager t=2024-02-05T07:20:23.411758231Z level=info msg="starting MultiOrg Alertmanager" 2024-02-05T07:20:23.776 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=http.server t=2024-02-05T07:20:23.413161936Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-02-05T07:20:23.776 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=sqlstore.transactions t=2024-02-05T07:20:23.43173438Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-05T07:20:23.776 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=sqlstore.transactions t=2024-02-05T07:20:23.45036494Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-02-05T07:20:23.776 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:20:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=sqlstore.transactions t=2024-02-05T07:20:23.461533622Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-02-05T07:20:24.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:24 smithi007 ceph-mon[29098]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-02-05T07:20:24.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:24 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:20:24.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:24 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:20:24.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:24 smithi007 ceph-mon[29098]: osdmap e96: 8 total, 8 up, 8 in 2024-02-05T07:20:24.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:24 smithi007 ceph-mon[29098]: 4.18 scrub starts 2024-02-05T07:20:24.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:24 smithi007 ceph-mon[29098]: 4.18 scrub ok 2024-02-05T07:20:24.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:24 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:24.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:24 smithi007 ceph-mon[37158]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-02-05T07:20:24.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:24 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:20:24.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:24 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:20:24.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:24 smithi007 ceph-mon[37158]: osdmap e96: 8 total, 8 up, 8 in 2024-02-05T07:20:24.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:24 smithi007 ceph-mon[37158]: 4.18 scrub starts 2024-02-05T07:20:24.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:24 smithi007 ceph-mon[37158]: 4.18 scrub ok 2024-02-05T07:20:24.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:24 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:24.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:24 smithi090 ceph-mon[33213]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-02-05T07:20:24.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:24 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:20:24.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:24 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:20:24.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:24 smithi090 ceph-mon[33213]: osdmap e96: 8 total, 8 up, 8 in 2024-02-05T07:20:24.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:24 smithi090 ceph-mon[33213]: 4.18 scrub starts 2024-02-05T07:20:24.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:24 smithi090 ceph-mon[33213]: 4.18 scrub ok 2024-02-05T07:20:24.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:24 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:24.903 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:20:24.906 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:20:25.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:25 smithi090 ceph-mon[33213]: 5.1a scrub starts 2024-02-05T07:20:25.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:25 smithi090 ceph-mon[33213]: 5.1a scrub ok 2024-02-05T07:20:25.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:25 smithi090 ceph-mon[33213]: 4.4 scrub starts 2024-02-05T07:20:25.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:25 smithi090 ceph-mon[33213]: 4.4 scrub ok 2024-02-05T07:20:25.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:25 smithi090 ceph-mon[33213]: 5.1 scrub starts 2024-02-05T07:20:25.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:25 smithi090 ceph-mon[33213]: 5.1 scrub ok 2024-02-05T07:20:25.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:25 smithi007 ceph-mon[29098]: 5.1a scrub starts 2024-02-05T07:20:25.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:25 smithi007 ceph-mon[29098]: 5.1a scrub ok 2024-02-05T07:20:25.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:25 smithi007 ceph-mon[29098]: 4.4 scrub starts 2024-02-05T07:20:25.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:25 smithi007 ceph-mon[29098]: 4.4 scrub ok 2024-02-05T07:20:25.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:25 smithi007 ceph-mon[29098]: 5.1 scrub starts 2024-02-05T07:20:25.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:25 smithi007 ceph-mon[29098]: 5.1 scrub ok 2024-02-05T07:20:25.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:25 smithi007 ceph-mon[37158]: 5.1a scrub starts 2024-02-05T07:20:25.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:25 smithi007 ceph-mon[37158]: 5.1a scrub ok 2024-02-05T07:20:25.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:25 smithi007 ceph-mon[37158]: 4.4 scrub starts 2024-02-05T07:20:25.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:25 smithi007 ceph-mon[37158]: 4.4 scrub ok 2024-02-05T07:20:25.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:25 smithi007 ceph-mon[37158]: 5.1 scrub starts 2024-02-05T07:20:25.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:25 smithi007 ceph-mon[37158]: 5.1 scrub ok 2024-02-05T07:20:26.128 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-02-05T07:20:23.466342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1017,"num_read_kb":863,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":610,"ondisk_log_size":610,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":237120,"kb_used_data":8484,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724096,"statfs":{"total":767960285184,"available":767717474304,"internally_reserved":0,"allocated":8687616,"data_stored":4801438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.545601"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107306+0000","last_change":"2024-02-05T07:18:55.466290+0000","last_active":"2024-02-05T07:20:19.107306+0000","last_peered":"2024-02-05T07:20:19.107306+0000","last_clean":"2024-02-05T07:20:19.107306+0000","last_became_active":"2024-02-05T07:18:55.466001+0000","last_became_peered":"2024-02-05T07:18:55.466001+0000","last_unstale":"2024-02-05T07:20:19.107306+0000","last_undegraded":"2024-02-05T07:20:19.107306+0000","last_fullsized":"2024-02-05T07:20:19.107306+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106942+0000","last_change":"2024-02-05T07:18:53.461117+0000","last_active":"2024-02-05T07:20:19.106942+0000","last_peered":"2024-02-05T07:20:19.106942+0000","last_clean":"2024-02-05T07:20:19.106942+0000","last_became_active":"2024-02-05T07:18:53.460820+0000","last_became_peered":"2024-02-05T07:18:53.460820+0000","last_unstale":"2024-02-05T07:20:19.106942+0000","last_undegraded":"2024-02-05T07:20:19.106942+0000","last_fullsized":"2024-02-05T07:20:19.106942+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264513+0000","last_change":"2024-02-05T07:18:49.434741+0000","last_active":"2024-02-05T07:20:19.264513+0000","last_peered":"2024-02-05T07:20:19.264513+0000","last_clean":"2024-02-05T07:20:19.264513+0000","last_became_active":"2024-02-05T07:18:49.434456+0000","last_became_peered":"2024-02-05T07:18:49.434456+0000","last_unstale":"2024-02-05T07:20:19.264513+0000","last_undegraded":"2024-02-05T07:20:19.264513+0000","last_fullsized":"2024-02-05T07:20:19.264513+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107210+0000","last_change":"2024-02-05T07:18:51.449261+0000","last_active":"2024-02-05T07:20:19.107210+0000","last_peered":"2024-02-05T07:20:19.107210+0000","last_clean":"2024-02-05T07:20:19.107210+0000","last_became_active":"2024-02-05T07:18:51.448924+0000","last_became_peered":"2024-02-05T07:18:51.448924+0000","last_unstale":"2024-02-05T07:20:19.107210+0000","last_undegraded":"2024-02-05T07:20:19.107210+0000","last_fullsized":"2024-02-05T07:20:19.107210+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106916+0000","last_change":"2024-02-05T07:18:55.506169+0000","last_active":"2024-02-05T07:20:19.106916+0000","last_peered":"2024-02-05T07:20:19.106916+0000","last_clean":"2024-02-05T07:20:19.106916+0000","last_became_active":"2024-02-05T07:18:55.505718+0000","last_became_peered":"2024-02-05T07:18:55.505718+0000","last_unstale":"2024-02-05T07:20:19.106916+0000","last_undegraded":"2024-02-05T07:20:19.106916+0000","last_fullsized":"2024-02-05T07:20:19.106916+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.751562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107221+0000","last_change":"2024-02-05T07:19:54.746304+0000","last_active":"2024-02-05T07:20:19.107221+0000","last_peered":"2024-02-05T07:20:19.107221+0000","last_clean":"2024-02-05T07:20:19.107221+0000","last_became_active":"2024-02-05T07:18:53.461506+0000","last_became_peered":"2024-02-05T07:18:53.461506+0000","last_unstale":"2024-02-05T07:20:19.107221+0000","last_undegraded":"2024-02-05T07:20:19.107221+0000","last_fullsized":"2024-02-05T07:20:19.107221+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:54.746206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:19:54.746206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:57.230561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107547+0000","last_change":"2024-02-05T07:18:49.436786+0000","last_active":"2024-02-05T07:20:19.107547+0000","last_peered":"2024-02-05T07:20:19.107547+0000","last_clean":"2024-02-05T07:20:19.107547+0000","last_became_active":"2024-02-05T07:18:49.436490+0000","last_became_peered":"2024-02-05T07:18:49.436490+0000","last_unstale":"2024-02-05T07:20:19.107547+0000","last_undegraded":"2024-02-05T07:20:19.107547+0000","last_fullsized":"2024-02-05T07:20:19.107547+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106497+0000","last_change":"2024-02-05T07:18:51.445160+0000","last_active":"2024-02-05T07:20:19.106497+0000","last_peered":"2024-02-05T07:20:19.106497+0000","last_clean":"2024-02-05T07:20:19.106497+0000","last_became_active":"2024-02-05T07:18:51.444783+0000","last_became_peered":"2024-02-05T07:18:51.444783+0000","last_unstale":"2024-02-05T07:20:19.106497+0000","last_undegraded":"2024-02-05T07:20:19.106497+0000","last_fullsized":"2024-02-05T07:20:19.106497+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.263859+0000","last_change":"2024-02-05T07:18:55.477022+0000","last_active":"2024-02-05T07:20:19.263859+0000","last_peered":"2024-02-05T07:20:19.263859+0000","last_clean":"2024-02-05T07:20:19.263859+0000","last_became_active":"2024-02-05T07:18:55.474002+0000","last_became_peered":"2024-02-05T07:18:55.474002+0000","last_unstale":"2024-02-05T07:20:19.263859+0000","last_undegraded":"2024-02-05T07:20:19.263859+0000","last_fullsized":"2024-02-05T07:20:19.263859+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324626+0000","last_change":"2024-02-05T07:18:53.464788+0000","last_active":"2024-02-05T07:20:19.324626+0000","last_peered":"2024-02-05T07:20:19.324626+0000","last_clean":"2024-02-05T07:20:19.324626+0000","last_became_active":"2024-02-05T07:18:53.464520+0000","last_became_peered":"2024-02-05T07:18:53.464520+0000","last_unstale":"2024-02-05T07:20:19.324626+0000","last_undegraded":"2024-02-05T07:20:19.324626+0000","last_fullsized":"2024-02-05T07:20:19.324626+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.843776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324814+0000","last_change":"2024-02-05T07:19:59.782950+0000","last_active":"2024-02-05T07:20:19.324814+0000","last_peered":"2024-02-05T07:20:19.324814+0000","last_clean":"2024-02-05T07:20:19.324814+0000","last_became_active":"2024-02-05T07:18:49.435454+0000","last_became_peered":"2024-02-05T07:18:49.435454+0000","last_unstale":"2024-02-05T07:20:19.324814+0000","last_undegraded":"2024-02-05T07:20:19.324814+0000","last_fullsized":"2024-02-05T07:20:19.324814+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:59.782841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:59.782841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.410088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"89'13","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107477+0000","last_change":"2024-02-05T07:20:02.798288+0000","last_active":"2024-02-05T07:20:19.107477+0000","last_peered":"2024-02-05T07:20:19.107477+0000","last_clean":"2024-02-05T07:20:19.107477+0000","last_became_active":"2024-02-05T07:18:51.444592+0000","last_became_peered":"2024-02-05T07:18:51.444592+0000","last_unstale":"2024-02-05T07:20:19.107477+0000","last_undegraded":"2024-02-05T07:20:19.107477+0000","last_fullsized":"2024-02-05T07:20:19.107477+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.798180+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:11.151835+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324397+0000","last_change":"2024-02-05T07:18:55.473667+0000","last_active":"2024-02-05T07:20:19.324397+0000","last_peered":"2024-02-05T07:20:19.324397+0000","last_clean":"2024-02-05T07:20:19.324397+0000","last_became_active":"2024-02-05T07:18:55.473368+0000","last_became_peered":"2024-02-05T07:18:55.473368+0000","last_unstale":"2024-02-05T07:20:19.324397+0000","last_undegraded":"2024-02-05T07:20:19.324397+0000","last_fullsized":"2024-02-05T07:20:19.324397+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537402+0000","last_change":"2024-02-05T07:18:53.457259+0000","last_active":"2024-02-05T07:20:19.537402+0000","last_peered":"2024-02-05T07:20:19.537402+0000","last_clean":"2024-02-05T07:20:19.537402+0000","last_became_active":"2024-02-05T07:18:53.456881+0000","last_became_peered":"2024-02-05T07:18:53.456881+0000","last_unstale":"2024-02-05T07:20:19.537402+0000","last_undegraded":"2024-02-05T07:20:19.537402+0000","last_fullsized":"2024-02-05T07:20:19.537402+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324830+0000","last_change":"2024-02-05T07:20:13.048248+0000","last_active":"2024-02-05T07:20:19.324830+0000","last_peered":"2024-02-05T07:20:19.324830+0000","last_clean":"2024-02-05T07:20:19.324830+0000","last_became_active":"2024-02-05T07:18:49.431220+0000","last_became_peered":"2024-02-05T07:18:49.431220+0000","last_unstale":"2024-02-05T07:20:19.324830+0000","last_undegraded":"2024-02-05T07:20:19.324830+0000","last_fullsized":"2024-02-05T07:20:19.324830+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:13.048126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:13.048126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:28.009618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107501+0000","last_change":"2024-02-05T07:18:51.448820+0000","last_active":"2024-02-05T07:20:19.107501+0000","last_peered":"2024-02-05T07:20:19.107501+0000","last_clean":"2024-02-05T07:20:19.107501+0000","last_became_active":"2024-02-05T07:18:51.448462+0000","last_became_peered":"2024-02-05T07:18:51.448462+0000","last_unstale":"2024-02-05T07:20:19.107501+0000","last_undegraded":"2024-02-05T07:20:19.107501+0000","last_fullsized":"2024-02-05T07:20:19.107501+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107996+0000","last_change":"2024-02-05T07:20:00.760332+0000","last_active":"2024-02-05T07:20:19.107996+0000","last_peered":"2024-02-05T07:20:19.107996+0000","last_clean":"2024-02-05T07:20:19.107996+0000","last_became_active":"2024-02-05T07:18:55.505678+0000","last_became_peered":"2024-02-05T07:18:55.505678+0000","last_unstale":"2024-02-05T07:20:19.107996+0000","last_undegraded":"2024-02-05T07:20:19.107996+0000","last_fullsized":"2024-02-05T07:20:19.107996+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:00.760230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.760230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:19.923028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.108178+0000","last_change":"2024-02-05T07:20:18.552686+0000","last_active":"2024-02-05T07:20:19.108178+0000","last_peered":"2024-02-05T07:20:19.108178+0000","last_clean":"2024-02-05T07:20:19.108178+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:20:19.108178+0000","last_undegraded":"2024-02-05T07:20:19.108178+0000","last_fullsized":"2024-02-05T07:20:19.108178+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:18.552608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.552608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:43.361846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264189+0000","last_change":"2024-02-05T07:18:49.432729+0000","last_active":"2024-02-05T07:20:19.264189+0000","last_peered":"2024-02-05T07:20:19.264189+0000","last_clean":"2024-02-05T07:20:19.264189+0000","last_became_active":"2024-02-05T07:18:49.432449+0000","last_became_peered":"2024-02-05T07:18:49.432449+0000","last_unstale":"2024-02-05T07:20:19.264189+0000","last_undegraded":"2024-02-05T07:20:19.264189+0000","last_fullsized":"2024-02-05T07:20:19.264189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":91,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536091+0000","last_change":"2024-02-05T07:19:59.624459+0000","last_active":"2024-02-05T07:20:19.536091+0000","last_peered":"2024-02-05T07:20:19.536091+0000","last_clean":"2024-02-05T07:20:19.536091+0000","last_became_active":"2024-02-05T07:18:51.447728+0000","last_became_peered":"2024-02-05T07:18:51.447728+0000","last_unstale":"2024-02-05T07:20:19.536091+0000","last_undegraded":"2024-02-05T07:20:19.536091+0000","last_fullsized":"2024-02-05T07:20:19.536091+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:19:59.624369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:59.624369+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:28.126457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107817+0000","last_change":"2024-02-05T07:18:55.506354+0000","last_active":"2024-02-05T07:20:19.107817+0000","last_peered":"2024-02-05T07:20:19.107817+0000","last_clean":"2024-02-05T07:20:19.107817+0000","last_became_active":"2024-02-05T07:18:55.505375+0000","last_became_peered":"2024-02-05T07:18:55.505375+0000","last_unstale":"2024-02-05T07:20:19.107817+0000","last_undegraded":"2024-02-05T07:20:19.107817+0000","last_fullsized":"2024-02-05T07:20:19.107817+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107504+0000","last_change":"2024-02-05T07:20:18.795431+0000","last_active":"2024-02-05T07:20:19.107504+0000","last_peered":"2024-02-05T07:20:19.107504+0000","last_clean":"2024-02-05T07:20:19.107504+0000","last_became_active":"2024-02-05T07:18:53.459077+0000","last_became_peered":"2024-02-05T07:18:53.459077+0000","last_unstale":"2024-02-05T07:20:19.107504+0000","last_undegraded":"2024-02-05T07:20:19.107504+0000","last_fullsized":"2024-02-05T07:20:19.107504+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:18.795341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.795341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:22.388371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107593+0000","last_change":"2024-02-05T07:18:49.428575+0000","last_active":"2024-02-05T07:20:19.107593+0000","last_peered":"2024-02-05T07:20:19.107593+0000","last_clean":"2024-02-05T07:20:19.107593+0000","last_became_active":"2024-02-05T07:18:49.427837+0000","last_became_peered":"2024-02-05T07:18:49.427837+0000","last_unstale":"2024-02-05T07:20:19.107593+0000","last_undegraded":"2024-02-05T07:20:19.107593+0000","last_fullsized":"2024-02-05T07:20:19.107593+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107370+0000","last_change":"2024-02-05T07:20:01.841591+0000","last_active":"2024-02-05T07:20:19.107370+0000","last_peered":"2024-02-05T07:20:19.107370+0000","last_clean":"2024-02-05T07:20:19.107370+0000","last_became_active":"2024-02-05T07:18:51.447761+0000","last_became_peered":"2024-02-05T07:18:51.447761+0000","last_unstale":"2024-02-05T07:20:19.107370+0000","last_undegraded":"2024-02-05T07:20:19.107370+0000","last_fullsized":"2024-02-05T07:20:19.107370+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:01.841489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:01.841489+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:19.277134+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537318+0000","last_change":"2024-02-05T07:20:17.755329+0000","last_active":"2024-02-05T07:20:19.537318+0000","last_peered":"2024-02-05T07:20:19.537318+0000","last_clean":"2024-02-05T07:20:19.537318+0000","last_became_active":"2024-02-05T07:18:55.476071+0000","last_became_peered":"2024-02-05T07:18:55.476071+0000","last_unstale":"2024-02-05T07:20:19.537318+0000","last_undegraded":"2024-02-05T07:20:19.537318+0000","last_fullsized":"2024-02-05T07:20:19.537318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:20:17.755246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:17.755246+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:22.075178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.538485+0000","last_change":"2024-02-05T07:18:53.461297+0000","last_active":"2024-02-05T07:20:19.538485+0000","last_peered":"2024-02-05T07:20:19.538485+0000","last_clean":"2024-02-05T07:20:19.538485+0000","last_became_active":"2024-02-05T07:18:53.461011+0000","last_became_peered":"2024-02-05T07:18:53.461011+0000","last_unstale":"2024-02-05T07:20:19.538485+0000","last_undegraded":"2024-02-05T07:20:19.538485+0000","last_fullsized":"2024-02-05T07:20:19.538485+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107026+0000","last_change":"2024-02-05T07:18:49.431334+0000","last_active":"2024-02-05T07:20:19.107026+0000","last_peered":"2024-02-05T07:20:19.107026+0000","last_clean":"2024-02-05T07:20:19.107026+0000","last_became_active":"2024-02-05T07:18:49.431031+0000","last_became_peered":"2024-02-05T07:18:49.431031+0000","last_unstale":"2024-02-05T07:20:19.107026+0000","last_undegraded":"2024-02-05T07:20:19.107026+0000","last_fullsized":"2024-02-05T07:20:19.107026+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264433+0000","last_change":"2024-02-05T07:18:51.510293+0000","last_active":"2024-02-05T07:20:19.264433+0000","last_peered":"2024-02-05T07:20:19.264433+0000","last_clean":"2024-02-05T07:20:19.264433+0000","last_became_active":"2024-02-05T07:18:51.509918+0000","last_became_peered":"2024-02-05T07:18:51.509918+0000","last_unstale":"2024-02-05T07:20:19.264433+0000","last_undegraded":"2024-02-05T07:20:19.264433+0000","last_fullsized":"2024-02-05T07:20:19.264433+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.108225+0000","last_change":"2024-02-05T07:20:03.444274+0000","last_active":"2024-02-05T07:20:19.108225+0000","last_peered":"2024-02-05T07:20:19.108225+0000","last_clean":"2024-02-05T07:20:19.108225+0000","last_became_active":"2024-02-05T07:18:55.466164+0000","last_became_peered":"2024-02-05T07:18:55.466164+0000","last_unstale":"2024-02-05T07:20:19.108225+0000","last_undegraded":"2024-02-05T07:20:19.108225+0000","last_fullsized":"2024-02-05T07:20:19.108225+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:03.444153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:03.444153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:32.621342+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264610+0000","last_change":"2024-02-05T07:18:53.457928+0000","last_active":"2024-02-05T07:20:19.264610+0000","last_peered":"2024-02-05T07:20:19.264610+0000","last_clean":"2024-02-05T07:20:19.264610+0000","last_became_active":"2024-02-05T07:18:53.457587+0000","last_became_peered":"2024-02-05T07:18:53.457587+0000","last_unstale":"2024-02-05T07:20:19.264610+0000","last_undegraded":"2024-02-05T07:20:19.264610+0000","last_fullsized":"2024-02-05T07:20:19.264610+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.105892+0000","last_change":"2024-02-05T07:19:49.788355+0000","last_active":"2024-02-05T07:20:19.105892+0000","last_peered":"2024-02-05T07:20:19.105892+0000","last_clean":"2024-02-05T07:20:19.105892+0000","last_became_active":"2024-02-05T07:18:49.430988+0000","last_became_peered":"2024-02-05T07:18:49.430988+0000","last_unstale":"2024-02-05T07:20:19.105892+0000","last_undegraded":"2024-02-05T07:20:19.105892+0000","last_fullsized":"2024-02-05T07:20:19.105892+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:49.788275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:49.788275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:50.100908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":85,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536243+0000","last_change":"2024-02-05T07:20:06.366811+0000","last_active":"2024-02-05T07:20:19.536243+0000","last_peered":"2024-02-05T07:20:19.536243+0000","last_clean":"2024-02-05T07:20:19.536243+0000","last_became_active":"2024-02-05T07:18:51.446604+0000","last_became_peered":"2024-02-05T07:18:51.446604+0000","last_unstale":"2024-02-05T07:20:19.536243+0000","last_undegraded":"2024-02-05T07:20:19.536243+0000","last_fullsized":"2024-02-05T07:20:19.536243+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:20:06.366699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.366699+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.671090+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537248+0000","last_change":"2024-02-05T07:20:01.693975+0000","last_active":"2024-02-05T07:20:19.537248+0000","last_peered":"2024-02-05T07:20:19.537248+0000","last_clean":"2024-02-05T07:20:19.537248+0000","last_became_active":"2024-02-05T07:18:55.468086+0000","last_became_peered":"2024-02-05T07:18:55.468086+0000","last_unstale":"2024-02-05T07:20:19.537248+0000","last_undegraded":"2024-02-05T07:20:19.537248+0000","last_fullsized":"2024-02-05T07:20:19.537248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:01.693871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:01.693871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:12.004988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536302+0000","last_change":"2024-02-05T07:20:09.349490+0000","last_active":"2024-02-05T07:20:19.536302+0000","last_peered":"2024-02-05T07:20:19.536302+0000","last_clean":"2024-02-05T07:20:19.536302+0000","last_became_active":"2024-02-05T07:18:53.460307+0000","last_became_peered":"2024-02-05T07:18:53.460307+0000","last_unstale":"2024-02-05T07:20:19.536302+0000","last_undegraded":"2024-02-05T07:20:19.536302+0000","last_fullsized":"2024-02-05T07:20:19.536302+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.349342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.451852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.326467+0000","last_change":"2024-02-05T07:20:19.326489+0000","last_active":"2024-02-05T07:20:19.326467+0000","last_peered":"2024-02-05T07:20:19.326467+0000","last_clean":"2024-02-05T07:20:19.326467+0000","last_became_active":"2024-02-05T07:18:49.434408+0000","last_became_peered":"2024-02-05T07:18:49.434408+0000","last_unstale":"2024-02-05T07:20:19.326467+0000","last_undegraded":"2024-02-05T07:20:19.326467+0000","last_fullsized":"2024-02-05T07:20:19.326467+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:19.326400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:19.326400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:45.594433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.105998+0000","last_change":"2024-02-05T07:18:51.449489+0000","last_active":"2024-02-05T07:20:19.105998+0000","last_peered":"2024-02-05T07:20:19.105998+0000","last_clean":"2024-02-05T07:20:19.105998+0000","last_became_active":"2024-02-05T07:18:51.449083+0000","last_became_peered":"2024-02-05T07:18:51.449083+0000","last_unstale":"2024-02-05T07:20:19.105998+0000","last_undegraded":"2024-02-05T07:20:19.105998+0000","last_fullsized":"2024-02-05T07:20:19.105998+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106745+0000","last_change":"2024-02-05T07:18:55.507549+0000","last_active":"2024-02-05T07:20:19.106745+0000","last_peered":"2024-02-05T07:20:19.106745+0000","last_clean":"2024-02-05T07:20:19.106745+0000","last_became_active":"2024-02-05T07:18:55.506968+0000","last_became_peered":"2024-02-05T07:18:55.506968+0000","last_unstale":"2024-02-05T07:20:19.106745+0000","last_undegraded":"2024-02-05T07:20:19.106745+0000","last_fullsized":"2024-02-05T07:20:19.106745+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537956+0000","last_change":"2024-02-05T07:20:05.674115+0000","last_active":"2024-02-05T07:20:19.537956+0000","last_peered":"2024-02-05T07:20:19.537956+0000","last_clean":"2024-02-05T07:20:19.537956+0000","last_became_active":"2024-02-05T07:18:53.460977+0000","last_became_peered":"2024-02-05T07:18:53.460977+0000","last_unstale":"2024-02-05T07:20:19.537956+0000","last_undegraded":"2024-02-05T07:20:19.537956+0000","last_fullsized":"2024-02-05T07:20:19.537956+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_clean_scrub_stamp":"2024-02-05T07:20:05.673983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:06.714441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324989+0000","last_change":"2024-02-05T07:19:48.944128+0000","last_active":"2024-02-05T07:20:19.324989+0000","last_peered":"2024-02-05T07:20:19.324989+0000","last_clean":"2024-02-05T07:20:19.324989+0000","last_became_active":"2024-02-05T07:18:49.434560+0000","last_became_peered":"2024-02-05T07:18:49.434560+0000","last_unstale":"2024-02-05T07:20:19.324989+0000","last_undegraded":"2024-02-05T07:20:19.324989+0000","last_fullsized":"2024-02-05T07:20:19.324989+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:48.944028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:48.944028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:56.156526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106467+0000","last_change":"2024-02-05T07:18:51.510217+0000","last_active":"2024-02-05T07:20:19.106467+0000","last_peered":"2024-02-05T07:20:19.106467+0000","last_clean":"2024-02-05T07:20:19.106467+0000","last_became_active":"2024-02-05T07:18:51.509911+0000","last_became_peered":"2024-02-05T07:18:51.509911+0000","last_unstale":"2024-02-05T07:20:19.106467+0000","last_undegraded":"2024-02-05T07:20:19.106467+0000","last_fullsized":"2024-02-05T07:20:19.106467+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.263725+0000","last_change":"2024-02-05T07:18:55.477626+0000","last_active":"2024-02-05T07:20:19.263725+0000","last_peered":"2024-02-05T07:20:19.263725+0000","last_clean":"2024-02-05T07:20:19.263725+0000","last_became_active":"2024-02-05T07:18:55.477312+0000","last_became_peered":"2024-02-05T07:18:55.477312+0000","last_unstale":"2024-02-05T07:20:19.263725+0000","last_undegraded":"2024-02-05T07:20:19.263725+0000","last_fullsized":"2024-02-05T07:20:19.263725+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106996+0000","last_change":"2024-02-05T07:18:53.468306+0000","last_active":"2024-02-05T07:20:19.106996+0000","last_peered":"2024-02-05T07:20:19.106996+0000","last_clean":"2024-02-05T07:20:19.106996+0000","last_became_active":"2024-02-05T07:18:53.467986+0000","last_became_peered":"2024-02-05T07:18:53.467986+0000","last_unstale":"2024-02-05T07:20:19.106996+0000","last_undegraded":"2024-02-05T07:20:19.106996+0000","last_fullsized":"2024-02-05T07:20:19.106996+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:21.663256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537858+0000","last_change":"2024-02-05T07:18:49.438840+0000","last_active":"2024-02-05T07:20:19.537858+0000","last_peered":"2024-02-05T07:20:19.537858+0000","last_clean":"2024-02-05T07:20:19.537858+0000","last_became_active":"2024-02-05T07:18:49.438565+0000","last_became_peered":"2024-02-05T07:18:49.438565+0000","last_unstale":"2024-02-05T07:20:19.537858+0000","last_undegraded":"2024-02-05T07:20:19.537858+0000","last_fullsized":"2024-02-05T07:20:19.537858+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324390+0000","last_change":"2024-02-05T07:19:16.128305+0000","last_active":"2024-02-05T07:20:19.324390+0000","last_peered":"2024-02-05T07:20:19.324390+0000","last_clean":"2024-02-05T07:20:19.324390+0000","last_became_active":"2024-02-05T07:19:16.122059+0000","last_became_peered":"2024-02-05T07:19:16.122059+0000","last_unstale":"2024-02-05T07:20:19.324390+0000","last_undegraded":"2024-02-05T07:20:19.324390+0000","last_fullsized":"2024-02-05T07:20:19.324390+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.105988+0000","last_change":"2024-02-05T07:18:55.478367+0000","last_active":"2024-02-05T07:20:19.105988+0000","last_peered":"2024-02-05T07:20:19.105988+0000","last_clean":"2024-02-05T07:20:19.105988+0000","last_became_active":"2024-02-05T07:18:55.478087+0000","last_became_peered":"2024-02-05T07:18:55.478087+0000","last_unstale":"2024-02-05T07:20:19.105988+0000","last_undegraded":"2024-02-05T07:20:19.105988+0000","last_fullsized":"2024-02-05T07:20:19.105988+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107481+0000","last_change":"2024-02-05T07:18:53.463942+0000","last_active":"2024-02-05T07:20:19.107481+0000","last_peered":"2024-02-05T07:20:19.107481+0000","last_clean":"2024-02-05T07:20:19.107481+0000","last_became_active":"2024-02-05T07:18:53.463620+0000","last_became_peered":"2024-02-05T07:18:53.463620+0000","last_unstale":"2024-02-05T07:20:19.107481+0000","last_undegraded":"2024-02-05T07:20:19.107481+0000","last_fullsized":"2024-02-05T07:20:19.107481+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107923+0000","last_change":"2024-02-05T07:18:49.431084+0000","last_active":"2024-02-05T07:20:19.107923+0000","last_peered":"2024-02-05T07:20:19.107923+0000","last_clean":"2024-02-05T07:20:19.107923+0000","last_became_active":"2024-02-05T07:18:49.430766+0000","last_became_peered":"2024-02-05T07:18:49.430766+0000","last_unstale":"2024-02-05T07:20:19.107923+0000","last_undegraded":"2024-02-05T07:20:19.107923+0000","last_fullsized":"2024-02-05T07:20:19.107923+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":103,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107790+0000","last_change":"2024-02-05T07:20:06.870132+0000","last_active":"2024-02-05T07:20:19.107790+0000","last_peered":"2024-02-05T07:20:19.107790+0000","last_clean":"2024-02-05T07:20:19.107790+0000","last_became_active":"2024-02-05T07:19:16.121696+0000","last_became_peered":"2024-02-05T07:19:16.121696+0000","last_unstale":"2024-02-05T07:20:19.107790+0000","last_undegraded":"2024-02-05T07:20:19.107790+0000","last_fullsized":"2024-02-05T07:20:19.107790+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:20:06.870006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.870006+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:12.414942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264309+0000","last_change":"2024-02-05T07:18:55.478247+0000","last_active":"2024-02-05T07:20:19.264309+0000","last_peered":"2024-02-05T07:20:19.264309+0000","last_clean":"2024-02-05T07:20:19.264309+0000","last_became_active":"2024-02-05T07:18:55.477896+0000","last_became_peered":"2024-02-05T07:18:55.477896+0000","last_unstale":"2024-02-05T07:20:19.264309+0000","last_undegraded":"2024-02-05T07:20:19.264309+0000","last_fullsized":"2024-02-05T07:20:19.264309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324979+0000","last_change":"2024-02-05T07:18:53.465570+0000","last_active":"2024-02-05T07:20:19.324979+0000","last_peered":"2024-02-05T07:20:19.324979+0000","last_clean":"2024-02-05T07:20:19.324979+0000","last_became_active":"2024-02-05T07:18:53.465287+0000","last_became_peered":"2024-02-05T07:18:53.465287+0000","last_unstale":"2024-02-05T07:20:19.324979+0000","last_undegraded":"2024-02-05T07:20:19.324979+0000","last_fullsized":"2024-02-05T07:20:19.324979+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536950+0000","last_change":"2024-02-05T07:18:49.436416+0000","last_active":"2024-02-05T07:20:19.536950+0000","last_peered":"2024-02-05T07:20:19.536950+0000","last_clean":"2024-02-05T07:20:19.536950+0000","last_became_active":"2024-02-05T07:18:49.436131+0000","last_became_peered":"2024-02-05T07:18:49.436131+0000","last_unstale":"2024-02-05T07:20:19.536950+0000","last_undegraded":"2024-02-05T07:20:19.536950+0000","last_fullsized":"2024-02-05T07:20:19.536950+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107106+0000","last_change":"2024-02-05T07:18:51.507793+0000","last_active":"2024-02-05T07:20:19.107106+0000","last_peered":"2024-02-05T07:20:19.107106+0000","last_clean":"2024-02-05T07:20:19.107106+0000","last_became_active":"2024-02-05T07:18:51.507491+0000","last_became_peered":"2024-02-05T07:18:51.507491+0000","last_unstale":"2024-02-05T07:20:19.107106+0000","last_undegraded":"2024-02-05T07:20:19.107106+0000","last_fullsized":"2024-02-05T07:20:19.107106+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.108318+0000","last_change":"2024-02-05T07:18:55.467356+0000","last_active":"2024-02-05T07:20:19.108318+0000","last_peered":"2024-02-05T07:20:19.108318+0000","last_clean":"2024-02-05T07:20:19.108318+0000","last_became_active":"2024-02-05T07:18:55.467044+0000","last_became_peered":"2024-02-05T07:18:55.467044+0000","last_unstale":"2024-02-05T07:20:19.108318+0000","last_undegraded":"2024-02-05T07:20:19.108318+0000","last_fullsized":"2024-02-05T07:20:19.108318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106935+0000","last_change":"2024-02-05T07:18:53.465551+0000","last_active":"2024-02-05T07:20:19.106935+0000","last_peered":"2024-02-05T07:20:19.106935+0000","last_clean":"2024-02-05T07:20:19.106935+0000","last_became_active":"2024-02-05T07:18:53.465270+0000","last_became_peered":"2024-02-05T07:18:53.465270+0000","last_unstale":"2024-02-05T07:20:19.106935+0000","last_undegraded":"2024-02-05T07:20:19.106935+0000","last_fullsized":"2024-02-05T07:20:19.106935+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.263969+0000","last_change":"2024-02-05T07:18:49.435653+0000","last_active":"2024-02-05T07:20:19.263969+0000","last_peered":"2024-02-05T07:20:19.263969+0000","last_clean":"2024-02-05T07:20:19.263969+0000","last_became_active":"2024-02-05T07:18:49.434402+0000","last_became_peered":"2024-02-05T07:18:49.434402+0000","last_unstale":"2024-02-05T07:20:19.263969+0000","last_undegraded":"2024-02-05T07:20:19.263969+0000","last_fullsized":"2024-02-05T07:20:19.263969+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.325064+0000","last_change":"2024-02-05T07:20:06.852726+0000","last_active":"2024-02-05T07:20:19.325064+0000","last_peered":"2024-02-05T07:20:19.325064+0000","last_clean":"2024-02-05T07:20:19.325064+0000","last_became_active":"2024-02-05T07:18:51.508551+0000","last_became_peered":"2024-02-05T07:18:51.508551+0000","last_unstale":"2024-02-05T07:20:19.325064+0000","last_undegraded":"2024-02-05T07:20:19.325064+0000","last_fullsized":"2024-02-05T07:20:19.325064+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:20:06.852562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.852562+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:20.949048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536815+0000","last_change":"2024-02-05T07:20:11.543111+0000","last_active":"2024-02-05T07:20:19.536815+0000","last_peered":"2024-02-05T07:20:19.536815+0000","last_clean":"2024-02-05T07:20:19.536815+0000","last_became_active":"2024-02-05T07:18:55.469044+0000","last_became_peered":"2024-02-05T07:18:55.469044+0000","last_unstale":"2024-02-05T07:20:19.536815+0000","last_undegraded":"2024-02-05T07:20:19.536815+0000","last_fullsized":"2024-02-05T07:20:19.536815+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:11.543036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:11.543036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:38.671608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107792+0000","last_change":"2024-02-05T07:20:02.813432+0000","last_active":"2024-02-05T07:20:19.107792+0000","last_peered":"2024-02-05T07:20:19.107792+0000","last_clean":"2024-02-05T07:20:19.107792+0000","last_became_active":"2024-02-05T07:18:53.463607+0000","last_became_peered":"2024-02-05T07:18:53.463607+0000","last_unstale":"2024-02-05T07:20:19.107792+0000","last_undegraded":"2024-02-05T07:20:19.107792+0000","last_fullsized":"2024-02-05T07:20:19.107792+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.813344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.813344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:30.005634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537804+0000","last_change":"2024-02-05T07:20:02.673707+0000","last_active":"2024-02-05T07:20:19.537804+0000","last_peered":"2024-02-05T07:20:19.537804+0000","last_clean":"2024-02-05T07:20:19.537804+0000","last_became_active":"2024-02-05T07:18:49.437732+0000","last_became_peered":"2024-02-05T07:18:49.437732+0000","last_unstale":"2024-02-05T07:20:19.537804+0000","last_undegraded":"2024-02-05T07:20:19.537804+0000","last_fullsized":"2024-02-05T07:20:19.537804+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.673592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.665799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264540+0000","last_change":"2024-02-05T07:18:51.510254+0000","last_active":"2024-02-05T07:20:19.264540+0000","last_peered":"2024-02-05T07:20:19.264540+0000","last_clean":"2024-02-05T07:20:19.264540+0000","last_became_active":"2024-02-05T07:18:51.509917+0000","last_became_peered":"2024-02-05T07:18:51.509917+0000","last_unstale":"2024-02-05T07:20:19.264540+0000","last_undegraded":"2024-02-05T07:20:19.264540+0000","last_fullsized":"2024-02-05T07:20:19.264540+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.732419+0000","last_change":"2024-02-05T07:20:19.732465+0000","last_active":"2024-02-05T07:20:19.732419+0000","last_peered":"2024-02-05T07:20:19.732419+0000","last_clean":"2024-02-05T07:20:19.732419+0000","last_became_active":"2024-02-05T07:18:55.507555+0000","last_became_peered":"2024-02-05T07:18:55.507555+0000","last_unstale":"2024-02-05T07:20:19.732419+0000","last_undegraded":"2024-02-05T07:20:19.732419+0000","last_fullsized":"2024-02-05T07:20:19.732419+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:19.732337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:19.732337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:28.044159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.325001+0000","last_change":"2024-02-05T07:18:53.465851+0000","last_active":"2024-02-05T07:20:19.325001+0000","last_peered":"2024-02-05T07:20:19.325001+0000","last_clean":"2024-02-05T07:20:19.325001+0000","last_became_active":"2024-02-05T07:18:53.465591+0000","last_became_peered":"2024-02-05T07:18:53.465591+0000","last_unstale":"2024-02-05T07:20:19.325001+0000","last_undegraded":"2024-02-05T07:20:19.325001+0000","last_fullsized":"2024-02-05T07:20:19.325001+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":76,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536559+0000","last_change":"2024-02-05T07:19:55.369939+0000","last_active":"2024-02-05T07:20:19.536559+0000","last_peered":"2024-02-05T07:20:19.536559+0000","last_clean":"2024-02-05T07:20:19.536559+0000","last_became_active":"2024-02-05T07:18:49.433121+0000","last_became_peered":"2024-02-05T07:18:49.433121+0000","last_unstale":"2024-02-05T07:20:19.536559+0000","last_undegraded":"2024-02-05T07:20:19.536559+0000","last_fullsized":"2024-02-05T07:20:19.536559+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:19:55.369800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:55.369800+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:04.984273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107918+0000","last_change":"2024-02-05T07:18:51.508489+0000","last_active":"2024-02-05T07:20:19.107918+0000","last_peered":"2024-02-05T07:20:19.107918+0000","last_clean":"2024-02-05T07:20:19.107918+0000","last_became_active":"2024-02-05T07:18:51.507451+0000","last_became_peered":"2024-02-05T07:18:51.507451+0000","last_unstale":"2024-02-05T07:20:19.107918+0000","last_undegraded":"2024-02-05T07:20:19.107918+0000","last_fullsized":"2024-02-05T07:20:19.107918+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324748+0000","last_change":"2024-02-05T07:18:55.505712+0000","last_active":"2024-02-05T07:20:19.324748+0000","last_peered":"2024-02-05T07:20:19.324748+0000","last_clean":"2024-02-05T07:20:19.324748+0000","last_became_active":"2024-02-05T07:18:55.505401+0000","last_became_peered":"2024-02-05T07:18:55.505401+0000","last_unstale":"2024-02-05T07:20:19.324748+0000","last_undegraded":"2024-02-05T07:20:19.324748+0000","last_fullsized":"2024-02-05T07:20:19.324748+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537620+0000","last_change":"2024-02-05T07:20:13.701938+0000","last_active":"2024-02-05T07:20:19.537620+0000","last_peered":"2024-02-05T07:20:19.537620+0000","last_clean":"2024-02-05T07:20:19.537620+0000","last_became_active":"2024-02-05T07:18:49.437663+0000","last_became_peered":"2024-02-05T07:18:49.437663+0000","last_unstale":"2024-02-05T07:20:19.537620+0000","last_undegraded":"2024-02-05T07:20:19.537620+0000","last_fullsized":"2024-02-05T07:20:19.537620+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:20:13.701852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:13.701852+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:32.496072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":91,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324432+0000","last_change":"2024-02-05T07:19:58.753612+0000","last_active":"2024-02-05T07:20:19.324432+0000","last_peered":"2024-02-05T07:20:19.324432+0000","last_clean":"2024-02-05T07:20:19.324432+0000","last_became_active":"2024-02-05T07:18:51.508097+0000","last_became_peered":"2024-02-05T07:18:51.508097+0000","last_unstale":"2024-02-05T07:20:19.324432+0000","last_undegraded":"2024-02-05T07:20:19.324432+0000","last_fullsized":"2024-02-05T07:20:19.324432+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:19:58.753472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:58.753472+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:26.081872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":90,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.538002+0000","last_change":"2024-02-05T07:20:00.685371+0000","last_active":"2024-02-05T07:20:19.538002+0000","last_peered":"2024-02-05T07:20:19.538002+0000","last_clean":"2024-02-05T07:20:19.538002+0000","last_became_active":"2024-02-05T07:18:53.456287+0000","last_became_peered":"2024-02-05T07:18:53.456287+0000","last_unstale":"2024-02-05T07:20:19.538002+0000","last_undegraded":"2024-02-05T07:20:19.538002+0000","last_fullsized":"2024-02-05T07:20:19.538002+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.685282+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:11.368545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106830+0000","last_change":"2024-02-05T07:18:55.474988+0000","last_active":"2024-02-05T07:20:19.106830+0000","last_peered":"2024-02-05T07:20:19.106830+0000","last_clean":"2024-02-05T07:20:19.106830+0000","last_became_active":"2024-02-05T07:18:55.474663+0000","last_became_peered":"2024-02-05T07:18:55.474663+0000","last_unstale":"2024-02-05T07:20:19.106830+0000","last_undegraded":"2024-02-05T07:20:19.106830+0000","last_fullsized":"2024-02-05T07:20:19.106830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537017+0000","last_change":"2024-02-05T07:19:51.758261+0000","last_active":"2024-02-05T07:20:19.537017+0000","last_peered":"2024-02-05T07:20:19.537017+0000","last_clean":"2024-02-05T07:20:19.537017+0000","last_became_active":"2024-02-05T07:18:49.435621+0000","last_became_peered":"2024-02-05T07:18:49.435621+0000","last_unstale":"2024-02-05T07:20:19.537017+0000","last_undegraded":"2024-02-05T07:20:19.537017+0000","last_fullsized":"2024-02-05T07:20:19.537017+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:51.758163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.758163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:15.549152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":107,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107970+0000","last_change":"2024-02-05T07:20:04.885890+0000","last_active":"2024-02-05T07:20:19.107970+0000","last_peered":"2024-02-05T07:20:19.107970+0000","last_clean":"2024-02-05T07:20:19.107970+0000","last_became_active":"2024-02-05T07:19:16.123650+0000","last_became_peered":"2024-02-05T07:19:16.123650+0000","last_unstale":"2024-02-05T07:20:19.107970+0000","last_undegraded":"2024-02-05T07:20:19.107970+0000","last_fullsized":"2024-02-05T07:20:19.107970+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:04.885778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:04.885778+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:07.176995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536286+0000","last_change":"2024-02-05T07:20:02.694603+0000","last_active":"2024-02-05T07:20:19.536286+0000","last_peered":"2024-02-05T07:20:19.536286+0000","last_clean":"2024-02-05T07:20:19.536286+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:20:19.536286+0000","last_undegraded":"2024-02-05T07:20:19.536286+0000","last_fullsized":"2024-02-05T07:20:19.536286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.694506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.694506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.700377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536905+0000","last_change":"2024-02-05T07:18:55.473523+0000","last_active":"2024-02-05T07:20:19.536905+0000","last_peered":"2024-02-05T07:20:19.536905+0000","last_clean":"2024-02-05T07:20:19.536905+0000","last_became_active":"2024-02-05T07:18:55.473032+0000","last_became_peered":"2024-02-05T07:18:55.473032+0000","last_unstale":"2024-02-05T07:20:19.536905+0000","last_undegraded":"2024-02-05T07:20:19.536905+0000","last_fullsized":"2024-02-05T07:20:19.536905+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536595+0000","last_change":"2024-02-05T07:20:16.476649+0000","last_active":"2024-02-05T07:20:19.536595+0000","last_peered":"2024-02-05T07:20:19.536595+0000","last_clean":"2024-02-05T07:20:19.536595+0000","last_became_active":"2024-02-05T07:18:53.464867+0000","last_became_peered":"2024-02-05T07:18:53.464867+0000","last_unstale":"2024-02-05T07:20:19.536595+0000","last_undegraded":"2024-02-05T07:20:19.536595+0000","last_fullsized":"2024-02-05T07:20:19.536595+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:16.476554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:16.476554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:31.284238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.538140+0000","last_change":"2024-02-05T07:19:55.788134+0000","last_active":"2024-02-05T07:20:19.538140+0000","last_peered":"2024-02-05T07:20:19.538140+0000","last_clean":"2024-02-05T07:20:19.538140+0000","last_became_active":"2024-02-05T07:18:49.434451+0000","last_became_peered":"2024-02-05T07:18:49.434451+0000","last_unstale":"2024-02-05T07:20:19.538140+0000","last_undegraded":"2024-02-05T07:20:19.538140+0000","last_fullsized":"2024-02-05T07:20:19.538140+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:55.788005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:55.788005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.126482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":97,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536108+0000","last_change":"2024-02-05T07:20:00.384918+0000","last_active":"2024-02-05T07:20:19.536108+0000","last_peered":"2024-02-05T07:20:19.536108+0000","last_clean":"2024-02-05T07:20:19.536108+0000","last_became_active":"2024-02-05T07:18:51.507672+0000","last_became_peered":"2024-02-05T07:18:51.507672+0000","last_unstale":"2024-02-05T07:20:19.536108+0000","last_undegraded":"2024-02-05T07:20:19.536108+0000","last_fullsized":"2024-02-05T07:20:19.536108+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:20:00.384813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.384813+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:12.230867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.263790+0000","last_change":"2024-02-05T07:18:55.476043+0000","last_active":"2024-02-05T07:20:19.263790+0000","last_peered":"2024-02-05T07:20:19.263790+0000","last_clean":"2024-02-05T07:20:19.263790+0000","last_became_active":"2024-02-05T07:18:55.475702+0000","last_became_peered":"2024-02-05T07:18:55.475702+0000","last_unstale":"2024-02-05T07:20:19.263790+0000","last_undegraded":"2024-02-05T07:20:19.263790+0000","last_fullsized":"2024-02-05T07:20:19.263790+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:20.177474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536487+0000","last_change":"2024-02-05T07:20:09.589044+0000","last_active":"2024-02-05T07:20:19.536487+0000","last_peered":"2024-02-05T07:20:19.536487+0000","last_clean":"2024-02-05T07:20:19.536487+0000","last_became_active":"2024-02-05T07:18:53.459560+0000","last_became_peered":"2024-02-05T07:18:53.459560+0000","last_unstale":"2024-02-05T07:20:19.536487+0000","last_undegraded":"2024-02-05T07:20:19.536487+0000","last_fullsized":"2024-02-05T07:20:19.536487+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:09.588893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.588893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:27.848596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324652+0000","last_change":"2024-02-05T07:20:15.066894+0000","last_active":"2024-02-05T07:20:19.324652+0000","last_peered":"2024-02-05T07:20:19.324652+0000","last_clean":"2024-02-05T07:20:19.324652+0000","last_became_active":"2024-02-05T07:18:49.434238+0000","last_became_peered":"2024-02-05T07:18:49.434238+0000","last_unstale":"2024-02-05T07:20:19.324652+0000","last_undegraded":"2024-02-05T07:20:19.324652+0000","last_fullsized":"2024-02-05T07:20:19.324652+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:15.066768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:15.066768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:41.133131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264329+0000","last_change":"2024-02-05T07:18:51.441099+0000","last_active":"2024-02-05T07:20:19.264329+0000","last_peered":"2024-02-05T07:20:19.264329+0000","last_clean":"2024-02-05T07:20:19.264329+0000","last_became_active":"2024-02-05T07:18:51.440744+0000","last_became_peered":"2024-02-05T07:18:51.440744+0000","last_unstale":"2024-02-05T07:20:19.264329+0000","last_undegraded":"2024-02-05T07:20:19.264329+0000","last_fullsized":"2024-02-05T07:20:19.264329+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106503+0000","last_change":"2024-02-05T07:18:55.476663+0000","last_active":"2024-02-05T07:20:19.106503+0000","last_peered":"2024-02-05T07:20:19.106503+0000","last_clean":"2024-02-05T07:20:19.106503+0000","last_became_active":"2024-02-05T07:18:55.476313+0000","last_became_peered":"2024-02-05T07:18:55.476313+0000","last_unstale":"2024-02-05T07:20:19.106503+0000","last_undegraded":"2024-02-05T07:20:19.106503+0000","last_fullsized":"2024-02-05T07:20:19.106503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:20.640694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324529+0000","last_change":"2024-02-05T07:18:52.445271+0000","last_active":"2024-02-05T07:20:19.324529+0000","last_peered":"2024-02-05T07:20:19.324529+0000","last_clean":"2024-02-05T07:20:19.324529+0000","last_became_active":"2024-02-05T07:18:48.423032+0000","last_became_peered":"2024-02-05T07:18:48.423032+0000","last_unstale":"2024-02-05T07:20:19.324529+0000","last_undegraded":"2024-02-05T07:20:19.324529+0000","last_fullsized":"2024-02-05T07:20:19.324529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:18:47.405257+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107452+0000","last_change":"2024-02-05T07:18:49.430550+0000","last_active":"2024-02-05T07:20:19.107452+0000","last_peered":"2024-02-05T07:20:19.107452+0000","last_clean":"2024-02-05T07:20:19.107452+0000","last_became_active":"2024-02-05T07:18:49.430208+0000","last_became_peered":"2024-02-05T07:18:49.430208+0000","last_unstale":"2024-02-05T07:20:19.107452+0000","last_undegraded":"2024-02-05T07:20:19.107452+0000","last_fullsized":"2024-02-05T07:20:19.107452+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":126,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107146+0000","last_change":"2024-02-05T07:18:51.442632+0000","last_active":"2024-02-05T07:20:19.107146+0000","last_peered":"2024-02-05T07:20:19.107146+0000","last_clean":"2024-02-05T07:20:19.107146+0000","last_became_active":"2024-02-05T07:18:51.442110+0000","last_became_peered":"2024-02-05T07:18:51.442110+0000","last_unstale":"2024-02-05T07:20:19.107146+0000","last_undegraded":"2024-02-05T07:20:19.107146+0000","last_fullsized":"2024-02-05T07:20:19.107146+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264365+0000","last_change":"2024-02-05T07:18:53.461030+0000","last_active":"2024-02-05T07:20:19.264365+0000","last_peered":"2024-02-05T07:20:19.264365+0000","last_clean":"2024-02-05T07:20:19.264365+0000","last_became_active":"2024-02-05T07:18:53.460715+0000","last_became_peered":"2024-02-05T07:18:53.460715+0000","last_unstale":"2024-02-05T07:20:19.264365+0000","last_undegraded":"2024-02-05T07:20:19.264365+0000","last_fullsized":"2024-02-05T07:20:19.264365+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.356767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.538271+0000","last_change":"2024-02-05T07:20:06.673182+0000","last_active":"2024-02-05T07:20:19.538271+0000","last_peered":"2024-02-05T07:20:19.538271+0000","last_clean":"2024-02-05T07:20:19.538271+0000","last_became_active":"2024-02-05T07:18:55.506803+0000","last_became_peered":"2024-02-05T07:18:55.506803+0000","last_unstale":"2024-02-05T07:20:19.538271+0000","last_undegraded":"2024-02-05T07:20:19.538271+0000","last_fullsized":"2024-02-05T07:20:19.538271+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.673081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:21.893489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106435+0000","last_change":"2024-02-05T07:20:09.406817+0000","last_active":"2024-02-05T07:20:19.106435+0000","last_peered":"2024-02-05T07:20:19.106435+0000","last_clean":"2024-02-05T07:20:19.106435+0000","last_became_active":"2024-02-05T07:18:49.429651+0000","last_became_peered":"2024-02-05T07:18:49.429651+0000","last_unstale":"2024-02-05T07:20:19.106435+0000","last_undegraded":"2024-02-05T07:20:19.106435+0000","last_fullsized":"2024-02-05T07:20:19.106435+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:20:09.406718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.406718+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:33.441463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536606+0000","last_change":"2024-02-05T07:19:56.661601+0000","last_active":"2024-02-05T07:20:19.536606+0000","last_peered":"2024-02-05T07:20:19.536606+0000","last_clean":"2024-02-05T07:20:19.536606+0000","last_became_active":"2024-02-05T07:18:48.423639+0000","last_became_peered":"2024-02-05T07:18:48.423639+0000","last_unstale":"2024-02-05T07:20:19.536606+0000","last_undegraded":"2024-02-05T07:20:19.536606+0000","last_fullsized":"2024-02-05T07:20:19.536606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:56.661490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:19:56.661490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.585463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107965+0000","last_change":"2024-02-05T07:18:51.436432+0000","last_active":"2024-02-05T07:20:19.107965+0000","last_peered":"2024-02-05T07:20:19.107965+0000","last_clean":"2024-02-05T07:20:19.107965+0000","last_became_active":"2024-02-05T07:18:51.436097+0000","last_became_peered":"2024-02-05T07:18:51.436097+0000","last_unstale":"2024-02-05T07:20:19.107965+0000","last_undegraded":"2024-02-05T07:20:19.107965+0000","last_fullsized":"2024-02-05T07:20:19.107965+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536967+0000","last_change":"2024-02-05T07:20:10.558042+0000","last_active":"2024-02-05T07:20:19.536967+0000","last_peered":"2024-02-05T07:20:19.536967+0000","last_clean":"2024-02-05T07:20:19.536967+0000","last_became_active":"2024-02-05T07:18:53.460771+0000","last_became_peered":"2024-02-05T07:18:53.460771+0000","last_unstale":"2024-02-05T07:20:19.536967+0000","last_undegraded":"2024-02-05T07:20:19.536967+0000","last_fullsized":"2024-02-05T07:20:19.536967+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:10.557948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:10.557948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:27.157314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.325061+0000","last_change":"2024-02-05T07:18:55.473873+0000","last_active":"2024-02-05T07:20:19.325061+0000","last_peered":"2024-02-05T07:20:19.325061+0000","last_clean":"2024-02-05T07:20:19.325061+0000","last_became_active":"2024-02-05T07:18:55.473358+0000","last_became_peered":"2024-02-05T07:18:55.473358+0000","last_unstale":"2024-02-05T07:20:19.325061+0000","last_undegraded":"2024-02-05T07:20:19.325061+0000","last_fullsized":"2024-02-05T07:20:19.325061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":319,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537224+0000","last_change":"2024-02-05T07:19:05.594948+0000","last_active":"2024-02-05T07:20:19.537224+0000","last_peered":"2024-02-05T07:20:19.537224+0000","last_clean":"2024-02-05T07:20:19.537224+0000","last_became_active":"2024-02-05T07:18:39.444898+0000","last_became_peered":"2024-02-05T07:18:39.444898+0000","last_unstale":"2024-02-05T07:20:19.537224+0000","last_undegraded":"2024-02-05T07:20:19.537224+0000","last_fullsized":"2024-02-05T07:20:19.537224+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-02-05T07:19:05.594812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:19:05.594812+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106628+0000","last_change":"2024-02-05T07:18:49.439534+0000","last_active":"2024-02-05T07:20:19.106628+0000","last_peered":"2024-02-05T07:20:19.106628+0000","last_clean":"2024-02-05T07:20:19.106628+0000","last_became_active":"2024-02-05T07:18:49.439249+0000","last_became_peered":"2024-02-05T07:18:49.439249+0000","last_unstale":"2024-02-05T07:20:19.106628+0000","last_undegraded":"2024-02-05T07:20:19.106628+0000","last_fullsized":"2024-02-05T07:20:19.106628+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":99,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536349+0000","last_change":"2024-02-05T07:18:51.508007+0000","last_active":"2024-02-05T07:20:19.536349+0000","last_peered":"2024-02-05T07:20:19.536349+0000","last_clean":"2024-02-05T07:20:19.536349+0000","last_became_active":"2024-02-05T07:18:51.507714+0000","last_became_peered":"2024-02-05T07:18:51.507714+0000","last_unstale":"2024-02-05T07:20:19.536349+0000","last_undegraded":"2024-02-05T07:20:19.536349+0000","last_fullsized":"2024-02-05T07:20:19.536349+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.846104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537612+0000","last_change":"2024-02-05T07:18:53.460344+0000","last_active":"2024-02-05T07:20:19.537612+0000","last_peered":"2024-02-05T07:20:19.537612+0000","last_clean":"2024-02-05T07:20:19.537612+0000","last_became_active":"2024-02-05T07:18:53.459971+0000","last_became_peered":"2024-02-05T07:18:53.459971+0000","last_unstale":"2024-02-05T07:20:19.537612+0000","last_undegraded":"2024-02-05T07:20:19.537612+0000","last_fullsized":"2024-02-05T07:20:19.537612+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106796+0000","last_change":"2024-02-05T07:18:55.473808+0000","last_active":"2024-02-05T07:20:19.106796+0000","last_peered":"2024-02-05T07:20:19.106796+0000","last_clean":"2024-02-05T07:20:19.106796+0000","last_became_active":"2024-02-05T07:18:55.473509+0000","last_became_peered":"2024-02-05T07:18:55.473509+0000","last_unstale":"2024-02-05T07:20:19.106796+0000","last_undegraded":"2024-02-05T07:20:19.106796+0000","last_fullsized":"2024-02-05T07:20:19.106796+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:22.190977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.263976+0000","last_change":"2024-02-05T07:18:49.434806+0000","last_active":"2024-02-05T07:20:19.263976+0000","last_peered":"2024-02-05T07:20:19.263976+0000","last_clean":"2024-02-05T07:20:19.263976+0000","last_became_active":"2024-02-05T07:18:49.434400+0000","last_became_peered":"2024-02-05T07:18:49.434400+0000","last_unstale":"2024-02-05T07:20:19.263976+0000","last_undegraded":"2024-02-05T07:20:19.263976+0000","last_fullsized":"2024-02-05T07:20:19.263976+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":157,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536907+0000","last_change":"2024-02-05T07:18:52.450728+0000","last_active":"2024-02-05T07:20:19.536907+0000","last_peered":"2024-02-05T07:20:19.536907+0000","last_clean":"2024-02-05T07:20:19.536907+0000","last_became_active":"2024-02-05T07:18:48.424620+0000","last_became_peered":"2024-02-05T07:18:48.424620+0000","last_unstale":"2024-02-05T07:20:19.536907+0000","last_undegraded":"2024-02-05T07:20:19.536907+0000","last_fullsized":"2024-02-05T07:20:19.536907+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:18:47.405257+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264313+0000","last_change":"2024-02-05T07:20:14.321610+0000","last_active":"2024-02-05T07:20:19.264313+0000","last_peered":"2024-02-05T07:20:19.264313+0000","last_clean":"2024-02-05T07:20:19.264313+0000","last_became_active":"2024-02-05T07:18:51.442184+0000","last_became_peered":"2024-02-05T07:18:51.442184+0000","last_unstale":"2024-02-05T07:20:19.264313+0000","last_undegraded":"2024-02-05T07:20:19.264313+0000","last_fullsized":"2024-02-05T07:20:19.264313+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:14.321523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:14.321523+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.745540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.325243+0000","last_change":"2024-02-05T07:18:53.468712+0000","last_active":"2024-02-05T07:20:19.325243+0000","last_peered":"2024-02-05T07:20:19.325243+0000","last_clean":"2024-02-05T07:20:19.325243+0000","last_became_active":"2024-02-05T07:18:53.468512+0000","last_became_peered":"2024-02-05T07:18:53.468512+0000","last_unstale":"2024-02-05T07:20:19.325243+0000","last_undegraded":"2024-02-05T07:20:19.325243+0000","last_fullsized":"2024-02-05T07:20:19.325243+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106842+0000","last_change":"2024-02-05T07:18:55.505742+0000","last_active":"2024-02-05T07:20:19.106842+0000","last_peered":"2024-02-05T07:20:19.106842+0000","last_clean":"2024-02-05T07:20:19.106842+0000","last_became_active":"2024-02-05T07:18:55.505360+0000","last_became_peered":"2024-02-05T07:18:55.505360+0000","last_unstale":"2024-02-05T07:20:19.106842+0000","last_undegraded":"2024-02-05T07:20:19.106842+0000","last_fullsized":"2024-02-05T07:20:19.106842+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107019+0000","last_change":"2024-02-05T07:20:10.436628+0000","last_active":"2024-02-05T07:20:19.107019+0000","last_peered":"2024-02-05T07:20:19.107019+0000","last_clean":"2024-02-05T07:20:19.107019+0000","last_became_active":"2024-02-05T07:18:49.428470+0000","last_became_peered":"2024-02-05T07:18:49.428470+0000","last_unstale":"2024-02-05T07:20:19.107019+0000","last_undegraded":"2024-02-05T07:20:19.107019+0000","last_fullsized":"2024-02-05T07:20:19.107019+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:20:10.436535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:36.573911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":102,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264236+0000","last_change":"2024-02-05T07:20:12.309384+0000","last_active":"2024-02-05T07:20:19.264236+0000","last_peered":"2024-02-05T07:20:19.264236+0000","last_clean":"2024-02-05T07:20:19.264236+0000","last_became_active":"2024-02-05T07:18:51.507507+0000","last_became_peered":"2024-02-05T07:18:51.507507+0000","last_unstale":"2024-02-05T07:20:19.264236+0000","last_undegraded":"2024-02-05T07:20:19.264236+0000","last_fullsized":"2024-02-05T07:20:19.264236+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:20:12.309238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:12.309238+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:40.996099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536526+0000","last_change":"2024-02-05T07:19:56.330173+0000","last_active":"2024-02-05T07:20:19.536526+0000","last_peered":"2024-02-05T07:20:19.536526+0000","last_clean":"2024-02-05T07:20:19.536526+0000","last_became_active":"2024-02-05T07:18:53.459474+0000","last_became_peered":"2024-02-05T07:18:53.459474+0000","last_unstale":"2024-02-05T07:20:19.536526+0000","last_undegraded":"2024-02-05T07:20:19.536526+0000","last_fullsized":"2024-02-05T07:20:19.536526+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:56.330068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:19:56.330068+0000","last_clean_scrub_stamp":"2024-02-05T07:19:56.330068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:22.066765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.263736+0000","last_change":"2024-02-05T07:18:55.474472+0000","last_active":"2024-02-05T07:20:19.263736+0000","last_peered":"2024-02-05T07:20:19.263736+0000","last_clean":"2024-02-05T07:20:19.263736+0000","last_became_active":"2024-02-05T07:18:55.474029+0000","last_became_peered":"2024-02-05T07:18:55.474029+0000","last_unstale":"2024-02-05T07:20:19.263736+0000","last_undegraded":"2024-02-05T07:20:19.263736+0000","last_fullsized":"2024-02-05T07:20:19.263736+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324927+0000","last_change":"2024-02-05T07:18:49.435665+0000","last_active":"2024-02-05T07:20:19.324927+0000","last_peered":"2024-02-05T07:20:19.324927+0000","last_clean":"2024-02-05T07:20:19.324927+0000","last_became_active":"2024-02-05T07:18:49.435299+0000","last_became_peered":"2024-02-05T07:18:49.435299+0000","last_unstale":"2024-02-05T07:20:19.324927+0000","last_undegraded":"2024-02-05T07:20:19.324927+0000","last_fullsized":"2024-02-05T07:20:19.324927+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107565+0000","last_change":"2024-02-05T07:18:51.448399+0000","last_active":"2024-02-05T07:20:19.107565+0000","last_peered":"2024-02-05T07:20:19.107565+0000","last_clean":"2024-02-05T07:20:19.107565+0000","last_became_active":"2024-02-05T07:18:51.448044+0000","last_became_peered":"2024-02-05T07:18:51.448044+0000","last_unstale":"2024-02-05T07:20:19.107565+0000","last_undegraded":"2024-02-05T07:20:19.107565+0000","last_fullsized":"2024-02-05T07:20:19.107565+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264174+0000","last_change":"2024-02-05T07:18:53.463629+0000","last_active":"2024-02-05T07:20:19.264174+0000","last_peered":"2024-02-05T07:20:19.264174+0000","last_clean":"2024-02-05T07:20:19.264174+0000","last_became_active":"2024-02-05T07:18:53.462890+0000","last_became_peered":"2024-02-05T07:18:53.462890+0000","last_unstale":"2024-02-05T07:20:19.264174+0000","last_undegraded":"2024-02-05T07:20:19.264174+0000","last_fullsized":"2024-02-05T07:20:19.264174+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.827616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537132+0000","last_change":"2024-02-05T07:18:55.508071+0000","last_active":"2024-02-05T07:20:19.537132+0000","last_peered":"2024-02-05T07:20:19.537132+0000","last_clean":"2024-02-05T07:20:19.537132+0000","last_became_active":"2024-02-05T07:18:55.507791+0000","last_became_peered":"2024-02-05T07:18:55.507791+0000","last_unstale":"2024-02-05T07:20:19.537132+0000","last_undegraded":"2024-02-05T07:20:19.537132+0000","last_fullsized":"2024-02-05T07:20:19.537132+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:24.052431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.264427+0000","last_change":"2024-02-05T07:19:51.364668+0000","last_active":"2024-02-05T07:20:19.264427+0000","last_peered":"2024-02-05T07:20:19.264427+0000","last_clean":"2024-02-05T07:20:19.264427+0000","last_became_active":"2024-02-05T07:18:49.430123+0000","last_became_peered":"2024-02-05T07:18:49.430123+0000","last_unstale":"2024-02-05T07:20:19.264427+0000","last_undegraded":"2024-02-05T07:20:19.264427+0000","last_fullsized":"2024-02-05T07:20:19.264427+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:19:51.364585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.364585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:57.656562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107487+0000","last_change":"2024-02-05T07:18:51.438840+0000","last_active":"2024-02-05T07:20:19.107487+0000","last_peered":"2024-02-05T07:20:19.107487+0000","last_clean":"2024-02-05T07:20:19.107487+0000","last_became_active":"2024-02-05T07:18:51.438507+0000","last_became_peered":"2024-02-05T07:18:51.438507+0000","last_unstale":"2024-02-05T07:20:19.107487+0000","last_undegraded":"2024-02-05T07:20:19.107487+0000","last_fullsized":"2024-02-05T07:20:19.107487+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106217+0000","last_change":"2024-02-05T07:18:53.464137+0000","last_active":"2024-02-05T07:20:19.106217+0000","last_peered":"2024-02-05T07:20:19.106217+0000","last_clean":"2024-02-05T07:20:19.106217+0000","last_became_active":"2024-02-05T07:18:53.463651+0000","last_became_peered":"2024-02-05T07:18:53.463651+0000","last_unstale":"2024-02-05T07:20:19.106217+0000","last_undegraded":"2024-02-05T07:20:19.106217+0000","last_fullsized":"2024-02-05T07:20:19.106217+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107359+0000","last_change":"2024-02-05T07:18:55.474782+0000","last_active":"2024-02-05T07:20:19.107359+0000","last_peered":"2024-02-05T07:20:19.107359+0000","last_clean":"2024-02-05T07:20:19.107359+0000","last_became_active":"2024-02-05T07:18:55.474447+0000","last_became_peered":"2024-02-05T07:18:55.474447+0000","last_unstale":"2024-02-05T07:20:19.107359+0000","last_undegraded":"2024-02-05T07:20:19.107359+0000","last_fullsized":"2024-02-05T07:20:19.107359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107628+0000","last_change":"2024-02-05T07:18:49.435071+0000","last_active":"2024-02-05T07:20:19.107628+0000","last_peered":"2024-02-05T07:20:19.107628+0000","last_clean":"2024-02-05T07:20:19.107628+0000","last_became_active":"2024-02-05T07:18:49.434719+0000","last_became_peered":"2024-02-05T07:18:49.434719+0000","last_unstale":"2024-02-05T07:20:19.107628+0000","last_undegraded":"2024-02-05T07:20:19.107628+0000","last_fullsized":"2024-02-05T07:20:19.107628+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":98,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106115+0000","last_change":"2024-02-05T07:20:00.797048+0000","last_active":"2024-02-05T07:20:19.106115+0000","last_peered":"2024-02-05T07:20:19.106115+0000","last_clean":"2024-02-05T07:20:19.106115+0000","last_became_active":"2024-02-05T07:18:51.507524+0000","last_became_peered":"2024-02-05T07:18:51.507524+0000","last_unstale":"2024-02-05T07:20:19.106115+0000","last_undegraded":"2024-02-05T07:20:19.106115+0000","last_fullsized":"2024-02-05T07:20:19.106115+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:20:00.796905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.796905+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:13.158158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107321+0000","last_change":"2024-02-05T07:18:53.461489+0000","last_active":"2024-02-05T07:20:19.107321+0000","last_peered":"2024-02-05T07:20:19.107321+0000","last_clean":"2024-02-05T07:20:19.107321+0000","last_became_active":"2024-02-05T07:18:53.461213+0000","last_became_peered":"2024-02-05T07:18:53.461213+0000","last_unstale":"2024-02-05T07:20:19.107321+0000","last_undegraded":"2024-02-05T07:20:19.107321+0000","last_fullsized":"2024-02-05T07:20:19.107321+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.324763+0000","last_change":"2024-02-05T07:18:55.472546+0000","last_active":"2024-02-05T07:20:19.324763+0000","last_peered":"2024-02-05T07:20:19.324763+0000","last_clean":"2024-02-05T07:20:19.324763+0000","last_became_active":"2024-02-05T07:18:55.472172+0000","last_became_peered":"2024-02-05T07:18:55.472172+0000","last_unstale":"2024-02-05T07:20:19.324763+0000","last_undegraded":"2024-02-05T07:20:19.324763+0000","last_fullsized":"2024-02-05T07:20:19.324763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.108120+0000","last_change":"2024-02-05T07:18:53.460159+0000","last_active":"2024-02-05T07:20:19.108120+0000","last_peered":"2024-02-05T07:20:19.108120+0000","last_clean":"2024-02-05T07:20:19.108120+0000","last_became_active":"2024-02-05T07:18:53.459832+0000","last_became_peered":"2024-02-05T07:18:53.459832+0000","last_unstale":"2024-02-05T07:20:19.108120+0000","last_undegraded":"2024-02-05T07:20:19.108120+0000","last_fullsized":"2024-02-05T07:20:19.108120+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106090+0000","last_change":"2024-02-05T07:18:49.431779+0000","last_active":"2024-02-05T07:20:19.106090+0000","last_peered":"2024-02-05T07:20:19.106090+0000","last_clean":"2024-02-05T07:20:19.106090+0000","last_became_active":"2024-02-05T07:18:49.431369+0000","last_became_peered":"2024-02-05T07:18:49.431369+0000","last_unstale":"2024-02-05T07:20:19.106090+0000","last_undegraded":"2024-02-05T07:20:19.106090+0000","last_fullsized":"2024-02-05T07:20:19.106090+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.108153+0000","last_change":"2024-02-05T07:18:51.447767+0000","last_active":"2024-02-05T07:20:19.108153+0000","last_peered":"2024-02-05T07:20:19.108153+0000","last_clean":"2024-02-05T07:20:19.108153+0000","last_became_active":"2024-02-05T07:18:51.447469+0000","last_became_peered":"2024-02-05T07:18:51.447469+0000","last_unstale":"2024-02-05T07:20:19.108153+0000","last_undegraded":"2024-02-05T07:20:19.108153+0000","last_fullsized":"2024-02-05T07:20:19.108153+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:19.124394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.106723+0000","last_change":"2024-02-05T07:20:10.764040+0000","last_active":"2024-02-05T07:20:19.106723+0000","last_peered":"2024-02-05T07:20:19.106723+0000","last_clean":"2024-02-05T07:20:19.106723+0000","last_became_active":"2024-02-05T07:18:55.505340+0000","last_became_peered":"2024-02-05T07:18:55.505340+0000","last_unstale":"2024-02-05T07:20:19.106723+0000","last_undegraded":"2024-02-05T07:20:19.106723+0000","last_fullsized":"2024-02-05T07:20:19.106723+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:10.763957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:10.763957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:35.465090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.325203+0000","last_change":"2024-02-05T07:18:53.462879+0000","last_active":"2024-02-05T07:20:19.325203+0000","last_peered":"2024-02-05T07:20:19.325203+0000","last_clean":"2024-02-05T07:20:19.325203+0000","last_became_active":"2024-02-05T07:18:53.462591+0000","last_became_peered":"2024-02-05T07:18:53.462591+0000","last_unstale":"2024-02-05T07:20:19.325203+0000","last_undegraded":"2024-02-05T07:20:19.325203+0000","last_fullsized":"2024-02-05T07:20:19.325203+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:20.782612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107356+0000","last_change":"2024-02-05T07:20:05.883941+0000","last_active":"2024-02-05T07:20:19.107356+0000","last_peered":"2024-02-05T07:20:19.107356+0000","last_clean":"2024-02-05T07:20:19.107356+0000","last_became_active":"2024-02-05T07:18:49.434074+0000","last_became_peered":"2024-02-05T07:18:49.434074+0000","last_unstale":"2024-02-05T07:20:19.107356+0000","last_undegraded":"2024-02-05T07:20:19.107356+0000","last_fullsized":"2024-02-05T07:20:19.107356+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:05.883840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:05.883840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:30.747182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"90'16","reported_seq":90,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107864+0000","last_change":"2024-02-05T07:20:09.873512+0000","last_active":"2024-02-05T07:20:19.107864+0000","last_peered":"2024-02-05T07:20:19.107864+0000","last_clean":"2024-02-05T07:20:19.107864+0000","last_became_active":"2024-02-05T07:18:51.447922+0000","last_became_peered":"2024-02-05T07:18:51.447922+0000","last_unstale":"2024-02-05T07:20:19.107864+0000","last_undegraded":"2024-02-05T07:20:19.107864+0000","last_fullsized":"2024-02-05T07:20:19.107864+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.873388+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.334535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.537235+0000","last_change":"2024-02-05T07:18:55.475305+0000","last_active":"2024-02-05T07:20:19.537235+0000","last_peered":"2024-02-05T07:20:19.537235+0000","last_clean":"2024-02-05T07:20:19.537235+0000","last_became_active":"2024-02-05T07:18:55.474992+0000","last_became_peered":"2024-02-05T07:18:55.474992+0000","last_unstale":"2024-02-05T07:20:19.537235+0000","last_undegraded":"2024-02-05T07:20:19.537235+0000","last_fullsized":"2024-02-05T07:20:19.537235+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107260+0000","last_change":"2024-02-05T07:20:06.398771+0000","last_active":"2024-02-05T07:20:19.107260+0000","last_peered":"2024-02-05T07:20:19.107260+0000","last_clean":"2024-02-05T07:20:19.107260+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:20:19.107260+0000","last_undegraded":"2024-02-05T07:20:19.107260+0000","last_fullsized":"2024-02-05T07:20:19.107260+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:06.398697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.398697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.245750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536412+0000","last_change":"2024-02-05T07:20:18.399191+0000","last_active":"2024-02-05T07:20:19.536412+0000","last_peered":"2024-02-05T07:20:19.536412+0000","last_clean":"2024-02-05T07:20:19.536412+0000","last_became_active":"2024-02-05T07:18:49.430518+0000","last_became_peered":"2024-02-05T07:18:49.430518+0000","last_unstale":"2024-02-05T07:20:19.536412+0000","last_undegraded":"2024-02-05T07:20:19.536412+0000","last_fullsized":"2024-02-05T07:20:19.536412+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:18.399075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.399075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:36.367477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":94,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.108030+0000","last_change":"2024-02-05T07:19:51.808315+0000","last_active":"2024-02-05T07:20:19.108030+0000","last_peered":"2024-02-05T07:20:19.108030+0000","last_clean":"2024-02-05T07:20:19.108030+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:20:19.108030+0000","last_undegraded":"2024-02-05T07:20:19.108030+0000","last_fullsized":"2024-02-05T07:20:19.108030+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:19:51.808218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.808218+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:55.279939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.108177+0000","last_change":"2024-02-05T07:18:55.466723+0000","last_active":"2024-02-05T07:20:19.108177+0000","last_peered":"2024-02-05T07:20:19.108177+0000","last_clean":"2024-02-05T07:20:19.108177+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:20:19.108177+0000","last_undegraded":"2024-02-05T07:20:19.108177+0000","last_fullsized":"2024-02-05T07:20:19.108177+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.536797+0000","last_change":"2024-02-05T07:20:18.512034+0000","last_active":"2024-02-05T07:20:19.536797+0000","last_peered":"2024-02-05T07:20:19.536797+0000","last_clean":"2024-02-05T07:20:19.536797+0000","last_became_active":"2024-02-05T07:18:53.461852+0000","last_became_peered":"2024-02-05T07:18:53.461852+0000","last_unstale":"2024-02-05T07:20:19.536797+0000","last_undegraded":"2024-02-05T07:20:19.536797+0000","last_fullsized":"2024-02-05T07:20:19.536797+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:20:18.511957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.511957+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:23.253319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107076+0000","last_change":"2024-02-05T07:20:04.827259+0000","last_active":"2024-02-05T07:20:19.107076+0000","last_peered":"2024-02-05T07:20:19.107076+0000","last_clean":"2024-02-05T07:20:19.107076+0000","last_became_active":"2024-02-05T07:18:49.435368+0000","last_became_peered":"2024-02-05T07:18:49.435368+0000","last_unstale":"2024-02-05T07:20:19.107076+0000","last_undegraded":"2024-02-05T07:20:19.107076+0000","last_fullsized":"2024-02-05T07:20:19.107076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:04.827180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:04.827180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:17.606501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-05T07:20:19.107566+0000","last_change":"2024-02-05T07:20:13.843192+0000","last_active":"2024-02-05T07:20:19.107566+0000","last_peered":"2024-02-05T07:20:19.107566+0000","last_clean":"2024-02-05T07:20:19.107566+0000","last_became_active":"2024-02-05T07:18:51.442711+0000","last_became_peered":"2024-02-05T07:18:51.442711+0000","last_unstale":"2024-02-05T07:20:19.107566+0000","last_undegraded":"2024-02-05T07:20:19.107566+0000","last_fullsized":"2024-02-05T07:20:19.107566+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:20:13.843051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:13.843051+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:41.856009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":402,"ondisk_log_size":402,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037783,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28440,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716712,"statfs":{"total":95995035648,"available":95965913088,"internally_reserved":0,"allocated":1454080,"data_stored":970648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.623},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.76800000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.60999999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.67600000000000005}]},{"osd":3,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.63800000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.496}]},{"osd":5,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46100000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.55900000000000005}]}]},{"osd":6,"up_from":52,"seq":223338299421,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32572,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712580,"statfs":{"total":95995035648,"available":95961681920,"internally_reserved":0,"allocated":1429504,"data_stored":964410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.35399999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70299999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.503},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.40000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.67100000000000004,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.66300000000000003}]},{"osd":1,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68700000000000006,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38500000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.747,"5min":0.751,"15min":0.751},"last":0.64500000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.42199999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77000000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.53000000000000003,"15min":0.53100000000000003},"min":{"1min":0.36899999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.41199999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68999999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38200000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.54900000000000004}]},{"osd":4,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.41299999999999998,"15min":0.40500000000000003},"min":{"1min":0.28699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.45300000000000001},"min":{"1min":0.314,"5min":0.313,"15min":0.313},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45200000000000001,"15min":0.44700000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.63},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.46899999999999997,"15min":0.45800000000000002},"min":{"1min":0.314,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.53600000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.61599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":897024,"data_stored":381461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.36199999999999999,"15min":0.33400000000000002},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.45200000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.318},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.376,"15min":0.33700000000000002},"min":{"1min":0.23000000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55100000000000005,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.46400000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.442,"15min":0.44600000000000001},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.60799999999999998,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.88,"15min":1.0740000000000001},"min":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.61299999999999999,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.28399999999999997}]},{"osd":3,"last update":"Mon Feb 5 07:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.45800000000000002,"15min":0.47499999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.63800000000000001,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.47999999999999998,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68100000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.34499999999999997}]},{"osd":4,"last update":"Mon Feb 5 07:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59099999999999997,"15min":0.58999999999999997},"min":{"1min":0.44600000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.72999999999999998,"15min":0.753},"min":{"1min":0.41899999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.0069999999999999,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.44800000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61699999999999999,"15min":0.622},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.435},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.75700000000000001,"15min":0.78100000000000003},"min":{"1min":0.46500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.04,"5min":5.0289999999999999,"15min":5.0289999999999999},"last":0.53300000000000003}]},{"osd":6,"last update":"Mon Feb 5 07:20:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.66900000000000004,"15min":0.68100000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.97299999999999998,"5min":2.48,"15min":2.48},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.66200000000000003,"15min":0.67700000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0569999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":0.40999999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.42599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28448,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716704,"statfs":{"total":95995035648,"available":95965904896,"internally_reserved":0,"allocated":1462272,"data_stored":970789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.34300000000000003,"15min":0.31},"min":{"1min":0.23499999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.32500000000000001},"min":{"1min":0.26700000000000002,"5min":0.214,"15min":0.214},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.439,"15min":0.42799999999999999},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.42799999999999999,"15min":0.42099999999999999},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.28000000000000003}]},{"osd":3,"last update":"Mon Feb 5 07:20:00 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42899999999999999,"15min":0.43099999999999999},"min":{"1min":0.29899999999999999,"5min":0.246,"15min":0.246},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.375},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41399999999999998,"15min":0.41799999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.42499999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.47599999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66000000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.54900000000000004},"min":{"1min":0.41999999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.46400000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56000000000000005,"15min":0.55400000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.496}]},{"osd":6,"last update":"Mon Feb 5 07:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.54000000000000004}]},{"osd":7,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.59499999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717300,"statfs":{"total":95995035648,"available":95966515200,"internally_reserved":0,"allocated":851968,"data_stored":380547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.377,"15min":0.36899999999999999},"min":{"1min":0.23499999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.40100000000000002,"15min":0.38600000000000001},"min":{"1min":0.27400000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.28499999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.39100000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.44500000000000001,"15min":0.42699999999999999},"min":{"1min":0.29099999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.438}]},{"osd":3,"last update":"Mon Feb 5 07:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.46700000000000003,"15min":0.46899999999999997},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.56000000000000005,"5min":1.631,"15min":1.631},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46600000000000003,"15min":0.46400000000000002},"min":{"1min":0.32500000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.55000000000000004,"5min":1.391,"15min":1.391},"last":0.52100000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54100000000000004,"15min":0.53200000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.375},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.56899999999999995,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.73999999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:19:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.61699999999999999,"15min":0.60699999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.72799999999999998,"15min":0.75700000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86099999999999999,"5min":2.8319999999999999,"15min":2.8319999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.59799999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.72599999999999998,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.41199999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"last":0.5},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.59799999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32040,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713112,"statfs":{"total":95995035648,"available":95962226688,"internally_reserved":0,"allocated":880640,"data_stored":380133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39300000000000002,"15min":0.40500000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.52500000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.25},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.44400000000000001,"15min":0.45700000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.44}]},{"osd":1,"last update":"Mon Feb 5 07:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45000000000000001,"15min":0.46800000000000003},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":0.63500000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48199999999999998,"15min":0.503},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57999999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.34699999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.45600000000000002,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.61899999999999999,"5min":0.75,"15min":0.75},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.47499999999999998,"15min":0.495},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.38800000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.58899999999999997,"15min":0.59599999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.70999999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60199999999999998,"15min":0.59599999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.5779999999999998,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":0.58899999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79400000000000004,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59699999999999998,"15min":0.59499999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56699999999999995}]},{"osd":6,"last update":"Mon Feb 5 07:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58599999999999997,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.752,"5min":1.046,"15min":1.046},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.81100000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.53300000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.629},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.51800000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953510,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32028,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713124,"statfs":{"total":95995035648,"available":95962238976,"internally_reserved":0,"allocated":868352,"data_stored":379834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.74199999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.498},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.65000000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57199999999999995}]},{"osd":1,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.36599999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75700000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71499999999999997,"5min":0.754,"15min":0.754},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53300000000000003,"15min":0.54000000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.36399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69899999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.59299999999999997}]},{"osd":3,"last update":"Mon Feb 5 07:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55600000000000005,"15min":0.56100000000000005},"min":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72999999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.67200000000000004,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.55700000000000005}]},{"osd":5,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.376,"15min":0.36599999999999999},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.39000000000000001,"15min":0.378},"min":{"1min":0.309,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.57499999999999996,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.42899999999999999,"15min":0.41499999999999998},"min":{"1min":0.30599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.439},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.42699999999999999,"15min":0.40799999999999997},"min":{"1min":0.34999999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.54400000000000004}]},{"osd":7,"last update":"Mon Feb 5 07:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64400000000000002}]}]},{"osd":5,"up_from":41,"seq":176093659169,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":843776,"data_stored":373616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66800000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71199999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.56200000000000006}]},{"osd":1,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.55400000000000005,"15min":0.56299999999999994},"min":{"1min":0.35399999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.69699999999999995,"5min":0.753,"15min":0.753},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.5,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.38200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.68200000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.53100000000000003}]},{"osd":2,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68000000000000005,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.35099999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69999999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.61299999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64200000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71999999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.67900000000000005}]},{"osd":4,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.35999999999999999,"15min":0.34799999999999998},"min":{"1min":0.26800000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.38900000000000001,"15min":0.373},"min":{"1min":0.33200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Feb 5 07:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.42099999999999999,"15min":0.40200000000000002},"min":{"1min":0.31900000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.47699999999999998,"15min":0.45300000000000001},"min":{"1min":0.41799999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Feb 5 07:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:20:26.135 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-05T07:20:26.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd dump 2024-02-05T07:20:26.389 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:26.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:26 smithi007 ceph-mon[29098]: from='client.24644 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:26.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:26 smithi007 ceph-mon[29098]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T07:20:26.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:26 smithi007 ceph-mon[29098]: 4.5 scrub starts 2024-02-05T07:20:26.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:26 smithi007 ceph-mon[29098]: 4.5 scrub ok 2024-02-05T07:20:26.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:26 smithi007 ceph-mon[29098]: 3.19 scrub starts 2024-02-05T07:20:26.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:26 smithi007 ceph-mon[29098]: 3.19 scrub ok 2024-02-05T07:20:26.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:26 smithi007 ceph-mon[37158]: from='client.24644 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:26.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:26 smithi007 ceph-mon[37158]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T07:20:26.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:26 smithi007 ceph-mon[37158]: 4.5 scrub starts 2024-02-05T07:20:26.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:26 smithi007 ceph-mon[37158]: 4.5 scrub ok 2024-02-05T07:20:26.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:26 smithi007 ceph-mon[37158]: 3.19 scrub starts 2024-02-05T07:20:26.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:26 smithi007 ceph-mon[37158]: 3.19 scrub ok 2024-02-05T07:20:26.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:26 smithi090 ceph-mon[33213]: from='client.24644 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:26.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:26 smithi090 ceph-mon[33213]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T07:20:26.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:26 smithi090 ceph-mon[33213]: 4.5 scrub starts 2024-02-05T07:20:26.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:26 smithi090 ceph-mon[33213]: 4.5 scrub ok 2024-02-05T07:20:26.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:26 smithi090 ceph-mon[33213]: 3.19 scrub starts 2024-02-05T07:20:26.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:26 smithi090 ceph-mon[33213]: 3.19 scrub ok 2024-02-05T07:20:27.561 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:27 smithi007 ceph-mon[29098]: 6.1a scrub starts 2024-02-05T07:20:27.561 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:27 smithi007 ceph-mon[29098]: 6.1a scrub ok 2024-02-05T07:20:27.561 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:27 smithi007 ceph-mon[37158]: 6.1a scrub starts 2024-02-05T07:20:27.561 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:27 smithi007 ceph-mon[37158]: 6.1a scrub ok 2024-02-05T07:20:27.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:27 smithi090 ceph-mon[33213]: 6.1a scrub starts 2024-02-05T07:20:27.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:27 smithi090 ceph-mon[33213]: 6.1a scrub ok 2024-02-05T07:20:28.312 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Feb 05 07:20:28 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-a[73267]: ts=2024-02-05T07:20:28.220Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-05T07:20:28.581 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:28 smithi007 ceph-mon[29098]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:20:28.581 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:28 smithi007 ceph-mon[29098]: 6.10 scrub starts 2024-02-05T07:20:28.581 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:28 smithi007 ceph-mon[29098]: 6.10 scrub ok 2024-02-05T07:20:28.581 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:28 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:28.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:28 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:28.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:28 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:28.582 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Feb 05 07:20:28 smithi007 bash[73059]: 339bd22ee8613d57b78888bbb648af1ce01210d252588a1086736877d85b0de4 2024-02-05T07:20:28.582 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Feb 05 07:20:28 smithi007 systemd[1]: Started Ceph node-exporter.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:20:28.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:28 smithi007 ceph-mon[37158]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:20:28.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:28 smithi007 ceph-mon[37158]: 6.10 scrub starts 2024-02-05T07:20:28.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:28 smithi007 ceph-mon[37158]: 6.10 scrub ok 2024-02-05T07:20:28.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:28 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:28.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:28 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:28.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:28 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:28.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:28 smithi090 ceph-mon[33213]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:20:28.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:28 smithi090 ceph-mon[33213]: 6.10 scrub starts 2024-02-05T07:20:28.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:28 smithi090 ceph-mon[33213]: 6.10 scrub ok 2024-02-05T07:20:28.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:28 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:28.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:28 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:28.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:28 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:29.047 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 96 2024-02-05T07:20:29.047 INFO:teuthology.orchestra.run.smithi007.stdout:fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:20:29.047 INFO:teuthology.orchestra.run.smithi007.stdout:created 2024-02-05T07:13:14.603885+0000 2024-02-05T07:20:29.047 INFO:teuthology.orchestra.run.smithi007.stdout:modified 2024-02-05T07:20:23.462444+0000 2024-02-05T07:20:29.047 INFO:teuthology.orchestra.run.smithi007.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-05T07:20:29.048 INFO:teuthology.orchestra.run.smithi007.stdout:crush_version 18 2024-02-05T07:20:29.048 INFO:teuthology.orchestra.run.smithi007.stdout:full_ratio 0.95 2024-02-05T07:20:29.048 INFO:teuthology.orchestra.run.smithi007.stdout:backfillfull_ratio 0.9 2024-02-05T07:20:29.048 INFO:teuthology.orchestra.run.smithi007.stdout:nearfull_ratio 0.85 2024-02-05T07:20:29.048 INFO:teuthology.orchestra.run.smithi007.stdout:require_min_compat_client luminous 2024-02-05T07:20:29.048 INFO:teuthology.orchestra.run.smithi007.stdout:min_compat_client luminous 2024-02-05T07:20:29.048 INFO:teuthology.orchestra.run.smithi007.stdout:require_osd_release squid 2024-02-05T07:20:29.048 INFO:teuthology.orchestra.run.smithi007.stdout:stretch_mode_enabled false 2024-02-05T07:20:29.048 INFO:teuthology.orchestra.run.smithi007.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-02-05T07:20:29.048 INFO:teuthology.orchestra.run.smithi007.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 71 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-02-05T07:20:29.048 INFO:teuthology.orchestra.run.smithi007.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 69 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-02-05T07:20:29.048 INFO:teuthology.orchestra.run.smithi007.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-02-05T07:20:29.048 INFO:teuthology.orchestra.run.smithi007.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 73 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-02-05T07:20:29.049 INFO:teuthology.orchestra.run.smithi007.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 75 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-02-05T07:20:29.049 INFO:teuthology.orchestra.run.smithi007.stdout:max_osd 8 2024-02-05T07:20:29.049 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 up in weight 1 up_from 8 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6801/2069016689 v2:172.21.15.7:6802/2069016689 exists,up 703c3948-8003-4015-9dd8-21019cf0eeb8 2024-02-05T07:20:29.049 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 up in weight 1 up_from 13 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6805/865486532 v2:172.21.15.7:6806/865486532 exists,up 5c3e7d1e-a61d-44b9-b21e-f91c99a096bc 2024-02-05T07:20:29.049 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 up in weight 1 up_from 18 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6809/904871370 v2:172.21.15.7:6810/904871370 exists,up f4fd930f-7bb2-46ca-9964-98235bd47b3a 2024-02-05T07:20:29.049 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 up in weight 1 up_from 26 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6813/2758586242 v2:172.21.15.7:6814/2758586242 exists,up 149ed0d6-b049-4478-9819-cafc446e94d3 2024-02-05T07:20:29.049 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 up in weight 1 up_from 32 up_thru 80 down_at 0 last_clean_interval [0,0) v2:172.21.15.90:6800/1154929470 v2:172.21.15.90:6801/1154929470 exists,up 3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d 2024-02-05T07:20:29.049 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 up in weight 1 up_from 41 up_thru 80 down_at 0 last_clean_interval [0,0) v2:172.21.15.90:6804/398113577 v2:172.21.15.90:6805/398113577 exists,up 244a7930-c22c-413e-86e1-11e0749f7367 2024-02-05T07:20:29.049 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 up in weight 1 up_from 52 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.90:6808/2371506925 v2:172.21.15.90:6809/2371506925 exists,up 0adbb562-3012-4f60-b5ec-02550a10b2c7 2024-02-05T07:20:29.049 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 up in weight 1 up_from 60 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.90:6812/2983605980 v2:172.21.15.90:6813/2983605980 exists,up dbc26c36-a0e9-4f21-9e98-739270cea6d9 2024-02-05T07:20:29.049 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 4.9 [1,2] 2024-02-05T07:20:29.049 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 4.12 [1,5] 2024-02-05T07:20:29.049 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 4.13 [1,2] 2024-02-05T07:20:29.049 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/2180087772 expires 2024-02-06T07:19:28.895331+0000 2024-02-05T07:20:29.050 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/2066685086 expires 2024-02-06T07:19:28.895331+0000 2024-02-05T07:20:29.050 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/4100020626 expires 2024-02-06T07:19:28.895331+0000 2024-02-05T07:20:29.050 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/1953351182 expires 2024-02-06T07:19:28.895331+0000 2024-02-05T07:20:29.050 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/3203189162 expires 2024-02-06T07:19:28.895331+0000 2024-02-05T07:20:29.050 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/3418781614 expires 2024-02-06T07:14:14.164720+0000 2024-02-05T07:20:29.050 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:6800/1518592084 expires 2024-02-06T07:13:47.004765+0000 2024-02-05T07:20:29.050 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:6800/3380493069 expires 2024-02-06T07:19:28.895331+0000 2024-02-05T07:20:29.050 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/2208590106 expires 2024-02-06T07:13:47.004765+0000 2024-02-05T07:20:29.050 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/223579766 expires 2024-02-06T07:13:47.004765+0000 2024-02-05T07:20:29.050 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:6800/2634895574 expires 2024-02-06T07:14:14.164720+0000 2024-02-05T07:20:29.050 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/1462812719 expires 2024-02-06T07:14:14.164720+0000 2024-02-05T07:20:29.050 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/2823976078 expires 2024-02-06T07:14:14.164720+0000 2024-02-05T07:20:29.050 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/1905520582 expires 2024-02-06T07:13:47.004765+0000 2024-02-05T07:20:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:29 smithi090 ceph-mon[33213]: 6.11 scrub starts 2024-02-05T07:20:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:29 smithi090 ceph-mon[33213]: 6.11 scrub ok 2024-02-05T07:20:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:29 smithi090 ceph-mon[33213]: 4.7 deep-scrub starts 2024-02-05T07:20:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:29 smithi090 ceph-mon[33213]: Deploying daemon node-exporter.b on smithi090 2024-02-05T07:20:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:29 smithi090 ceph-mon[33213]: 4.7 deep-scrub ok 2024-02-05T07:20:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:20:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:29 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/25821109' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-05T07:20:29.776 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:29 smithi090 bash[63710]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-05T07:20:29.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:29 smithi007 ceph-mon[29098]: 6.11 scrub starts 2024-02-05T07:20:29.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:29 smithi007 ceph-mon[29098]: 6.11 scrub ok 2024-02-05T07:20:29.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:29 smithi007 ceph-mon[29098]: 4.7 deep-scrub starts 2024-02-05T07:20:29.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:29 smithi007 ceph-mon[29098]: Deploying daemon node-exporter.b on smithi090 2024-02-05T07:20:29.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:29 smithi007 ceph-mon[29098]: 4.7 deep-scrub ok 2024-02-05T07:20:29.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:20:29.867 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:29 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/25821109' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-05T07:20:29.867 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:29 smithi007 ceph-mon[37158]: 6.11 scrub starts 2024-02-05T07:20:29.867 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:29 smithi007 ceph-mon[37158]: 6.11 scrub ok 2024-02-05T07:20:29.867 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:29 smithi007 ceph-mon[37158]: 4.7 deep-scrub starts 2024-02-05T07:20:29.867 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:29 smithi007 ceph-mon[37158]: Deploying daemon node-exporter.b on smithi090 2024-02-05T07:20:29.867 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:29 smithi007 ceph-mon[37158]: 4.7 deep-scrub ok 2024-02-05T07:20:29.867 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:20:29.867 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:29 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/25821109' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-05T07:20:29.896 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6801/2069016689 v2:172.21.15.7:6802/2069016689 exists,up 703c3948-8003-4015-9dd8-21019cf0eeb8', 'osd.1 up in weight 1 up_from 13 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6805/865486532 v2:172.21.15.7:6806/865486532 exists,up 5c3e7d1e-a61d-44b9-b21e-f91c99a096bc', 'osd.2 up in weight 1 up_from 18 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6809/904871370 v2:172.21.15.7:6810/904871370 exists,up f4fd930f-7bb2-46ca-9964-98235bd47b3a', 'osd.3 up in weight 1 up_from 26 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6813/2758586242 v2:172.21.15.7:6814/2758586242 exists,up 149ed0d6-b049-4478-9819-cafc446e94d3', 'osd.4 up in weight 1 up_from 32 up_thru 80 down_at 0 last_clean_interval [0,0) v2:172.21.15.90:6800/1154929470 v2:172.21.15.90:6801/1154929470 exists,up 3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d', 'osd.5 up in weight 1 up_from 41 up_thru 80 down_at 0 last_clean_interval [0,0) v2:172.21.15.90:6804/398113577 v2:172.21.15.90:6805/398113577 exists,up 244a7930-c22c-413e-86e1-11e0749f7367', 'osd.6 up in weight 1 up_from 52 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.90:6808/2371506925 v2:172.21.15.90:6809/2371506925 exists,up 0adbb562-3012-4f60-b5ec-02550a10b2c7', 'osd.7 up in weight 1 up_from 60 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.90:6812/2983605980 v2:172.21.15.90:6813/2983605980 exists,up dbc26c36-a0e9-4f21-9e98-739270cea6d9'] 2024-02-05T07:20:29.897 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon mon.a config show 2024-02-05T07:20:30.149 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:30.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:20:29 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:20:30.275 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:30 smithi090 bash[63710]: Getting image source signatures 2024-02-05T07:20:30.275 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:30 smithi090 bash[63710]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-05T07:20:30.275 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:30 smithi090 bash[63710]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-05T07:20:30.275 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:30 smithi090 bash[63710]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-05T07:20:30.773 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:30 smithi007 ceph-mon[29098]: 6.b scrub starts 2024-02-05T07:20:30.773 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:30 smithi007 ceph-mon[29098]: 6.b scrub ok 2024-02-05T07:20:30.773 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:30 smithi007 ceph-mon[29098]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:20:30.773 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:30 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:20:30.773 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:30 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:20:30.773 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:30 smithi007 ceph-mon[29098]: 6.f scrub starts 2024-02-05T07:20:30.774 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:30 smithi007 ceph-mon[29098]: 6.f scrub ok 2024-02-05T07:20:30.774 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:30 smithi007 ceph-mon[37158]: 6.b scrub starts 2024-02-05T07:20:30.774 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:30 smithi007 ceph-mon[37158]: 6.b scrub ok 2024-02-05T07:20:30.774 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:30 smithi007 ceph-mon[37158]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:20:30.774 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:30 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:20:30.774 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:30 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:20:30.775 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:30 smithi007 ceph-mon[37158]: 6.f scrub starts 2024-02-05T07:20:30.775 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:30 smithi007 ceph-mon[37158]: 6.f scrub ok 2024-02-05T07:20:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:30 smithi090 ceph-mon[33213]: 6.b scrub starts 2024-02-05T07:20:30.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:30 smithi090 ceph-mon[33213]: 6.b scrub ok 2024-02-05T07:20:30.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:30 smithi090 ceph-mon[33213]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:20:30.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:30 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:20:30.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:30 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:20:30.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:30 smithi090 ceph-mon[33213]: 6.f scrub starts 2024-02-05T07:20:30.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:30 smithi090 ceph-mon[33213]: 6.f scrub ok 2024-02-05T07:20:31.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[29098]: 3.2 scrub starts 2024-02-05T07:20:31.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[29098]: 3.2 scrub ok 2024-02-05T07:20:31.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:31.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[29098]: 6.d scrub starts 2024-02-05T07:20:31.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[29098]: 6.d scrub ok 2024-02-05T07:20:31.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[29098]: osdmap e97: 8 total, 8 up, 8 in 2024-02-05T07:20:31.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[29098]: 6.2 scrub starts 2024-02-05T07:20:31.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[29098]: 6.2 scrub ok 2024-02-05T07:20:31.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[37158]: 3.2 scrub starts 2024-02-05T07:20:31.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[37158]: 3.2 scrub ok 2024-02-05T07:20:31.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:31.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[37158]: 6.d scrub starts 2024-02-05T07:20:31.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[37158]: 6.d scrub ok 2024-02-05T07:20:31.655 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[37158]: osdmap e97: 8 total, 8 up, 8 in 2024-02-05T07:20:31.655 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[37158]: 6.2 scrub starts 2024-02-05T07:20:31.655 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:31 smithi007 ceph-mon[37158]: 6.2 scrub ok 2024-02-05T07:20:31.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:31 smithi090 ceph-mon[33213]: 3.2 scrub starts 2024-02-05T07:20:31.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:31 smithi090 ceph-mon[33213]: 3.2 scrub ok 2024-02-05T07:20:31.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:31 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:31.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:31 smithi090 ceph-mon[33213]: 6.d scrub starts 2024-02-05T07:20:31.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:31 smithi090 ceph-mon[33213]: 6.d scrub ok 2024-02-05T07:20:31.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:31 smithi090 ceph-mon[33213]: osdmap e97: 8 total, 8 up, 8 in 2024-02-05T07:20:31.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:31 smithi090 ceph-mon[33213]: 6.2 scrub starts 2024-02-05T07:20:31.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:31 smithi090 ceph-mon[33213]: 6.2 scrub ok 2024-02-05T07:20:31.922 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:20:31.923 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mon.a", 2024-02-05T07:20:31.923 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster": "ceph", 2024-02-05T07:20:31.923 INFO:teuthology.orchestra.run.smithi007.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-02-05T07:20:31.923 INFO:teuthology.orchestra.run.smithi007.stdout: "admin_socket_mode": "", 2024-02-05T07:20:31.923 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-02-05T07:20:31.923 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_client_required": "cephx, none", 2024-02-05T07:20:31.923 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_cluster_required": "cephx", 2024-02-05T07:20:31.923 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_debug": "true", 2024-02-05T07:20:31.923 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-02-05T07:20:31.923 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-02-05T07:20:31.923 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_service_required": "cephx", 2024-02-05T07:20:31.923 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_service_ticket_ttl": "240.000000", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_supported": "", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_aio": "true", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_aio_max_queue_depth": "1024", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_aio_poll_ms": "250", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_aio_reap_max": "16", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_async_discard": "false", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_block_size": "4096", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_debug_aio": "false", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_debug_inflight_ios": "false", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_enable_discard": "false", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_flock_retry": "3", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_flock_retry_interval": "0.100000", 2024-02-05T07:20:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_inject_crash": "0", 2024-02-05T07:20:31.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_inject_crash_flush_delay": "2", 2024-02-05T07:20:31.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_ioring": "false", 2024-02-05T07:20:31.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_ioring_hipri": "false", 2024-02-05T07:20:31.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_ioring_sqthread_poll": "false", 2024-02-05T07:20:31.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-02-05T07:20:31.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_read_buffer_alignment": "4096", 2024-02-05T07:20:31.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-02-05T07:20:31.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_type": "", 2024-02-05T07:20:31.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_alloc_size": "1048576", 2024-02-05T07:20:31.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_allocator": "hybrid", 2024-02-05T07:20:31.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_buffered_io": "true", 2024-02-05T07:20:31.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_check_for_zeros": "false", 2024-02-05T07:20:31.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_check_volume_selector_often": "false", 2024-02-05T07:20:31.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-02-05T07:20:31.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_compact_log_sync": "false", 2024-02-05T07:20:31.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-02-05T07:20:31.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-02-05T07:20:31.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_log_compact_min_size": "16777216", 2024-02-05T07:20:31.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_log_replay_check_allocations": "true", 2024-02-05T07:20:31.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_max_log_runway": "4194304", 2024-02-05T07:20:31.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_max_prefetch": "1048576", 2024-02-05T07:20:31.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_min_flush_size": "524288", 2024-02-05T07:20:31.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_min_log_runway": "1048576", 2024-02-05T07:20:31.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_replay_recovery": "false", 2024-02-05T07:20:31.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-02-05T07:20:31.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_shared_alloc_size": "65536", 2024-02-05T07:20:31.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_sync_write": "false", 2024-02-05T07:20:31.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-02-05T07:20:31.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-02-05T07:20:31.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-02-05T07:20:31.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_allocation_from_file": "true", 2024-02-05T07:20:31.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_allocator": "hybrid", 2024-02-05T07:20:31.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-02-05T07:20:31.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-02-05T07:20:31.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-02-05T07:20:31.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-02-05T07:20:31.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-02-05T07:20:31.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-02-05T07:20:31.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blobid_prealloc": "10240", 2024-02-05T07:20:31.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_create": "true", 2024-02-05T07:20:31.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_db_create": "false", 2024-02-05T07:20:31.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_db_path": "", 2024-02-05T07:20:31.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_db_size": "0", 2024-02-05T07:20:31.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_path": "", 2024-02-05T07:20:31.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_preallocate_file": "false", 2024-02-05T07:20:31.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_size": "107374182400", 2024-02-05T07:20:31.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_wal_create": "false", 2024-02-05T07:20:31.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_wal_path": "", 2024-02-05T07:20:31.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_wal_size": "100663296", 2024-02-05T07:20:31.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bluefs": "true", 2024-02-05T07:20:31.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-02-05T07:20:31.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bluefs_env_mirror": "false", 2024-02-05T07:20:31.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-02-05T07:20:31.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-02-05T07:20:31.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-02-05T07:20:31.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_autotune": "true", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_size": "0", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_type": "2q", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-02-05T07:20:31.929 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_clone_cow": "true", 2024-02-05T07:20:31.930 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_algorithm": "snappy", 2024-02-05T07:20:31.930 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_max_blob_size": "0", 2024-02-05T07:20:31.930 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-02-05T07:20:31.930 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-02-05T07:20:31.930 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_min_blob_size": "0", 2024-02-05T07:20:31.930 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-02-05T07:20:31.930 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-02-05T07:20:31.930 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_mode": "none", 2024-02-05T07:20:31.930 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-02-05T07:20:31.930 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_csum_type": "crc32c", 2024-02-05T07:20:31.930 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_enforce_settings": "default", 2024-02-05T07:20:31.930 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_freelist": "false", 2024-02-05T07:20:31.930 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_fsck_abort": "false", 2024-02-05T07:20:31.930 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_inject_read_err": "false", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_legacy_omap": "false", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_prefill": "0.000000", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_small_allocations": "0", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_default_buffered_read": "true", 2024-02-05T07:20:31.931 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_default_buffered_write": "false", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_deferred_batch_ops": "0", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_elastic_shared_blobs": "true", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fail_eio": "false", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-02-05T07:20:31.932 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_on_mkfs": "true", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_on_mount": "false", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_on_umount": "false", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_ignore_data_csum": "false", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-02-05T07:20:31.933 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_kvbackend": "rocksdb", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_log_op_age": "5.000000", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_max_alloc_size": "0", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_max_blob_size": "0", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_max_defer_interval": "3.000000", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_max_deferred_txc": "32", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_min_alloc_size": "0", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_nid_prealloc": "1024", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_prefer_deferred_size": "0", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-02-05T07:20:31.934 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-02-05T07:20:31.935 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_qfsck_on_mount": "true", 2024-02-05T07:20:31.935 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_retry_disk_reads": "3", 2024-02-05T07:20:31.935 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_rocksdb_cf": "true", 2024-02-05T07:20:31.935 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-02-05T07:20:31.936 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-02-05T07:20:31.936 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_rocksdb_options_annex": "", 2024-02-05T07:20:31.936 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_spdk_coremask": "0x1", 2024-02-05T07:20:31.936 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_spdk_io_sleep": "5", 2024-02-05T07:20:31.936 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_spdk_max_io_completion": "0", 2024-02-05T07:20:31.936 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_spdk_mem": "512", 2024-02-05T07:20:31.936 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_sync_submit_transaction": "false", 2024-02-05T07:20:31.936 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_throttle_bytes": "67108864", 2024-02-05T07:20:31.936 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_throttle_cost_per_io": "0", 2024-02-05T07:20:31.936 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-02-05T07:20:31.936 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-02-05T07:20:31.936 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-02-05T07:20:31.936 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_tracing": "false", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_volume_selection_reserved": "0", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_zero_block_detection": "false", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_action_update_interval": "2", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-02-05T07:20:31.937 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_mount_timeout": "10", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephx_cluster_require_signatures": "false", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephx_cluster_require_version": "2", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephx_require_signatures": "false", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephx_require_version": "2", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephx_service_require_signatures": "false", 2024-02-05T07:20:31.938 INFO:teuthology.orchestra.run.smithi007.stdout: "cephx_service_require_version": "2", 2024-02-05T07:20:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "cephx_sign_messages": "true", 2024-02-05T07:20:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "chdir": "", 2024-02-05T07:20:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "client_acl_type": "", 2024-02-05T07:20:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "client_asio_thread_count": "2", 2024-02-05T07:20:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "client_cache_mid": "0.750000", 2024-02-05T07:20:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "client_cache_size": "16384", 2024-02-05T07:20:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "client_caps_release_delay": "5", 2024-02-05T07:20:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "client_check_pool_perm": "true", 2024-02-05T07:20:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "client_collect_and_send_global_metrics": "false", 2024-02-05T07:20:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "client_debug_force_sync_read": "false", 2024-02-05T07:20:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "client_debug_getattr_caps": "false", 2024-02-05T07:20:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "client_debug_inject_tick_delay": "0", 2024-02-05T07:20:31.939 INFO:teuthology.orchestra.run.smithi007.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_die_on_failed_remount": "false", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_dirsize_rbytes": "true", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_force_lazyio": "false", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_fs": "", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_inject_fixed_oldest_tid": "false", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_inject_release_failure": "false", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_max_inline_size": "4096", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_max_retries_on_remount_failure": "5", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_mds_namespace": "", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_metadata": "", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_mount_gid": "-1", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_mount_timeout": "300", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_mount_uid": "-1", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_mountpoint": "/", 2024-02-05T07:20:31.940 INFO:teuthology.orchestra.run.smithi007.stdout: "client_notify_timeout": "10", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_oc": "true", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_oc_max_dirty": "104857600", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_oc_max_dirty_age": "5.000000", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_oc_max_objects": "1000", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_oc_size": "209715200", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_oc_target_dirty": "8388608", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_permissions": "true", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_quota": "true", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_quota_df": "true", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_readahead_max_bytes": "0", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_readahead_max_periods": "4", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_readahead_min": "131072", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_reconnect_stale": "false", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_shutdown_timeout": "30", 2024-02-05T07:20:31.941 INFO:teuthology.orchestra.run.smithi007.stdout: "client_snapdir": ".snap", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "client_tick_interval": "1", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "client_trace": "", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "client_try_dentry_invalidate": "false", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "client_use_faked_inos": "false", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "client_use_random_mds": "false", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "clog_to_graylog": "false", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "clog_to_graylog_port": "12201", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "clog_to_monitors": "default=true", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "clog_to_syslog": "false", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "clog_to_syslog_level": "info", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "-", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_network": "", 2024-02-05T07:20:31.942 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_network_interface": "", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "compressor_zlib_isal": "false", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "compressor_zlib_level": "5", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "compressor_zlib_winsize": "-15", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "compressor_zstd_level": "1", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "crimson_alien_op_num_threads": "6", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "crimson_alien_thread_cpu_cores": "", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "crimson_osd_obc_lru_size": "512", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "crimson_seastar_cpu_cores": "", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_location": "", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_location_hook": "", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_location_hook_timeout": "10", 2024-02-05T07:20:31.943 INFO:teuthology.orchestra.run.smithi007.stdout: "daemonize": "false", 2024-02-05T07:20:31.944 INFO:teuthology.orchestra.run.smithi007.stdout: "daos_pool": "tank", 2024-02-05T07:20:31.944 INFO:teuthology.orchestra.run.smithi007.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-02-05T07:20:31.944 INFO:teuthology.orchestra.run.smithi007.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-02-05T07:20:31.944 INFO:teuthology.orchestra.run.smithi007.stdout: "dbstore_db_name_prefix": "dbstore", 2024-02-05T07:20:31.944 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_alienstore": "0/5", 2024-02-05T07:20:31.944 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_allow_any_pool_priority": "false", 2024-02-05T07:20:31.944 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_asok": "1/5", 2024-02-05T07:20:31.944 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_asok_assert_abort": "false", 2024-02-05T07:20:31.944 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_asserts_on_shutdown": "true", 2024-02-05T07:20:31.944 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_auth": "1/5", 2024-02-05T07:20:31.944 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_bdev": "1/3", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_bluefs": "1/5", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_bluestore": "1/5", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_buffer": "0/1", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_ceph_exporter": "1/5", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_cephfs_mirror": "0/5", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_cephsqlite": "0/5", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_client": "0/5", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_compressor": "1/5", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_context": "0/1", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_crimson_interrupt": "0/5", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_crush": "1/1", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_crypto": "1/5", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_cyanstore": "0/5", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_deliberately_leak_memory": "false", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_disable_randomized_ping": "false", 2024-02-05T07:20:31.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_dpdk": "1/5", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_eventtrace": "1/5", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_filer": "0/1", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_filestore": "1/3", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_finisher": "1/1", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_fuse": "1/5", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_heartbeat_testing_span": "0", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_heartbeatmap": "1/5", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_immutable_obj_cache": "0/5", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_javaclient": "1/5", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_journal": "1/3", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_journaler": "0/5", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_kstore": "1/5", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_lockdep": "0/1", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mclock": "1/5", 2024-02-05T07:20:31.946 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mds": "1/5", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mds_balancer": "1/5", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mds_locker": "1/5", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mds_log": "1/5", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mds_log_expire": "1/5", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mds_migrator": "1/5", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_memstore": "1/5", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mgr": "2/5", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mgrc": "1/5", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mon": "20/20", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_monc": "0/10", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_ms": "1/1", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_none": "0/5", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_objclass": "0/5", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_objectcacher": "0/5", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_objecter": "0/1", 2024-02-05T07:20:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_optracker": "0/5", 2024-02-05T07:20:31.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_osd": "1/5", 2024-02-05T07:20:31.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_paxos": "20/20", 2024-02-05T07:20:31.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_perfcounter": "1/5", 2024-02-05T07:20:31.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_prioritycache": "1/5", 2024-02-05T07:20:31.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rados": "0/5", 2024-02-05T07:20:31.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rbd": "0/5", 2024-02-05T07:20:31.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rbd_mirror": "0/5", 2024-02-05T07:20:31.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rbd_pwl": "0/5", 2024-02-05T07:20:31.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rbd_replay": "0/5", 2024-02-05T07:20:31.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_refs": "0/0", 2024-02-05T07:20:31.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_reserver": "1/1", 2024-02-05T07:20:31.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw": "1/5", 2024-02-05T07:20:31.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw_access": "1/5", 2024-02-05T07:20:31.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw_datacache": "1/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw_dbstore": "1/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw_flight": "1/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw_lifecycle": "1/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw_sync": "1/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rocksdb": "4/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore": "0/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_backref": "0/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_cache": "0/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_cleaner": "0/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_device": "0/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_epm": "0/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_journal": "0/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_lba": "0/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_odata": "0/5", 2024-02-05T07:20:31.949 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_omap": "0/5", 2024-02-05T07:20:31.950 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_onode": "0/5", 2024-02-05T07:20:31.950 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_t": "0/5", 2024-02-05T07:20:31.950 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_tm": "0/5", 2024-02-05T07:20:31.950 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_striper": "0/1", 2024-02-05T07:20:31.950 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_test": "0/5", 2024-02-05T07:20:31.950 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_throttle": "1/1", 2024-02-05T07:20:31.950 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_timer": "0/1", 2024-02-05T07:20:31.950 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_tp": "0/5", 2024-02-05T07:20:31.950 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_trace": "1/5", 2024-02-05T07:20:31.950 INFO:teuthology.orchestra.run.smithi007.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-02-05T07:20:31.950 INFO:teuthology.orchestra.run.smithi007.stdout: "device_failure_prediction_mode": "none", 2024-02-05T07:20:31.950 INFO:teuthology.orchestra.run.smithi007.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-02-05T07:20:31.950 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-02-05T07:20:31.950 INFO:teuthology.orchestra.run.smithi007.stdout: "err_to_graylog": "false", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "err_to_journald": "false", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "err_to_stderr": "true", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "err_to_syslog": "false", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "event_tracing": "false", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "exporter_addr": "0.0.0.0", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "exporter_http_port": "9926", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "exporter_prio_limit": "5", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "exporter_sort_metrics": "true", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "exporter_stats_period": "5", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "fake_statfs_for_testing": "0", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "fatal_signal_handlers": "true", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "filer_max_purge_ops": "10", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "filer_max_truncate_ops": "128", 2024-02-05T07:20:31.951 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_apply_finisher_threads": "1", 2024-02-05T07:20:31.952 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_blackhole": "false", 2024-02-05T07:20:31.952 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_btrfs_clone_range": "true", 2024-02-05T07:20:31.952 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_btrfs_snap": "true", 2024-02-05T07:20:31.952 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_caller_concurrency": "10", 2024-02-05T07:20:31.952 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_collect_device_partition_information": "true", 2024-02-05T07:20:31.952 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_commit_timeout": "600.000000", 2024-02-05T07:20:31.952 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_debug_inject_read_err": "false", 2024-02-05T07:20:31.952 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_debug_omap_check": "false", 2024-02-05T07:20:31.952 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_debug_random_read_err": "0.000000", 2024-02-05T07:20:31.952 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_debug_verify_split": "false", 2024-02-05T07:20:31.952 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_dump_file": "", 2024-02-05T07:20:31.952 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-02-05T07:20:31.952 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-02-05T07:20:31.952 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_fadvise": "true", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_fail_eio": "true", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_fd_cache_shards": "16", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_fd_cache_size": "128", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_fiemap": "false", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_fiemap_threshold": "4096", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_index_retry_probability": "0.000000", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_inject_stall": "0", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_journal_parallel": "false", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_journal_trailing": "false", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_journal_writeahead": "false", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_kill_at": "0", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattr_size": "0", 2024-02-05T07:20:31.953 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattrs": "0", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattrs_other": "2", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_sync_interval": "5.000000", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_xattr_value_size": "0", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_merge_threshold": "-10", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_min_sync_interval": "0.010000", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_odsync_write": "false", 2024-02-05T07:20:31.954 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_omap_backend": "rocksdb", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_omap_backend_path": "", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_omap_header_cache_size": "1024", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_ondisk_finisher_threads": "1", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_op_thread_timeout": "60", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_op_threads": "2", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_punch_hole": "false", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_max_bytes": "104857600", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-02-05T07:20:31.955 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_max_ops": "50", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_seek_data_hole": "false", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_sloppy_crc": "false", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_splice": "false", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_split_multiple": "2", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_split_rand_factor": "20", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_update_to": "1000", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-02-05T07:20:31.956 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-02-05T07:20:31.957 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-02-05T07:20:31.957 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_enable": "true", 2024-02-05T07:20:31.957 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-02-05T07:20:31.957 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-02-05T07:20:31.957 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-02-05T07:20:31.957 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-02-05T07:20:31.957 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-02-05T07:20:31.957 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-02-05T07:20:31.957 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_xfs_extsize": "false", 2024-02-05T07:20:31.957 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_zfs_snap": "false", 2024-02-05T07:20:31.957 INFO:teuthology.orchestra.run.smithi007.stdout: "fio_dir": "/tmp/fio", 2024-02-05T07:20:31.957 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "e3e7059e-c3f5-11ee-95b6-87774f69a715", 2024-02-05T07:20:31.957 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_allow_other": "true", 2024-02-05T07:20:31.957 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_atomic_o_trunc": "true", 2024-02-05T07:20:31.958 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_big_writes": "true", 2024-02-05T07:20:31.958 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_debug": "false", 2024-02-05T07:20:31.958 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_default_permissions": "false", 2024-02-05T07:20:31.958 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_disable_pagecache": "false", 2024-02-05T07:20:31.958 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_max_write": "0", 2024-02-05T07:20:31.958 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_multithreaded": "true", 2024-02-05T07:20:31.958 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_require_active_mds": "true", 2024-02-05T07:20:31.958 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_set_user_groups": "true", 2024-02-05T07:20:31.958 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_splice_move": "true", 2024-02-05T07:20:31.958 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_splice_read": "true", 2024-02-05T07:20:31.958 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_splice_write": "true", 2024-02-05T07:20:31.958 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_syncfs_on_mksnap": "true", 2024-02-05T07:20:31.958 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_use_invalidate_cb": "true", 2024-02-05T07:20:31.958 INFO:teuthology.orchestra.run.smithi007.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "gss_target_name": "ceph", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_file": "", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_inject_failure": "0", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_interval": "5", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "host": "smithi007", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_max_size": "1073741824", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-02-05T07:20:31.959 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-02-05T07:20:31.960 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-02-05T07:20:31.960 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_watermark": "0.900000", 2024-02-05T07:20:31.960 INFO:teuthology.orchestra.run.smithi007.stdout: "inject_early_sigterm": "false", 2024-02-05T07:20:31.960 INFO:teuthology.orchestra.run.smithi007.stdout: "jaeger_agent_port": "6799", 2024-02-05T07:20:31.960 INFO:teuthology.orchestra.run.smithi007.stdout: "jaeger_tracing_enable": "false", 2024-02-05T07:20:31.960 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_aio": "true", 2024-02-05T07:20:31.960 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_align_min_size": "65536", 2024-02-05T07:20:31.960 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_block_align": "true", 2024-02-05T07:20:31.960 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_block_size": "4096", 2024-02-05T07:20:31.960 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_dio": "true", 2024-02-05T07:20:31.960 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_discard": "false", 2024-02-05T07:20:31.960 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_force_aio": "false", 2024-02-05T07:20:31.960 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_ignore_corruption": "false", 2024-02-05T07:20:31.960 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_max_write_bytes": "10485760", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_max_write_entries": "100", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_replay_from": "0", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_throttle_high_multiple": "0.000000", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_throttle_max_multiple": "0.000000", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_write_header_frequency": "0", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_zero_on_create": "false", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "journaler_prefetch_periods": "10", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "journaler_prezero_periods": "5", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "journaler_write_head_interval": "15", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "keyfile": "", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-02-05T07:20:31.961 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_backend": "rocksdb", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_default_stripe_size": "65536", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_fsck_on_mount": "false", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_fsck_on_mount_deep": "true", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_max_bytes": "67108864", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_max_ops": "512", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_nid_prealloc": "1024", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_onode_map_size": "1024", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_sync_submit_transaction": "false", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_sync_transaction": "false", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "librados_thread_count": "2", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "lockdep": "false", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "lockdep_force_backtrace": "false", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "log_coarse_timestamps": "true", 2024-02-05T07:20:31.962 INFO:teuthology.orchestra.run.smithi007.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-02-05T07:20:31.963 INFO:teuthology.orchestra.run.smithi007.stdout: "log_flush_on_exit": "false", 2024-02-05T07:20:31.963 INFO:teuthology.orchestra.run.smithi007.stdout: "log_graylog_host": "127.0.0.1", 2024-02-05T07:20:31.963 INFO:teuthology.orchestra.run.smithi007.stdout: "log_graylog_port": "12201", 2024-02-05T07:20:31.963 INFO:teuthology.orchestra.run.smithi007.stdout: "log_max_new": "1000", 2024-02-05T07:20:31.963 INFO:teuthology.orchestra.run.smithi007.stdout: "log_max_recent": "10000", 2024-02-05T07:20:31.963 INFO:teuthology.orchestra.run.smithi007.stdout: "log_stderr_prefix": "", 2024-02-05T07:20:31.963 INFO:teuthology.orchestra.run.smithi007.stdout: "log_stop_at_utilization": "0.970000", 2024-02-05T07:20:31.963 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": "true", 2024-02-05T07:20:31.963 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_graylog": "false", 2024-02-05T07:20:31.963 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_journald": "false", 2024-02-05T07:20:31.963 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_stderr": "false", 2024-02-05T07:20:31.963 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_syslog": "false", 2024-02-05T07:20:31.963 INFO:teuthology.orchestra.run.smithi007.stdout: "mandatory_topic_permissions": "false", 2024-02-05T07:20:31.963 INFO:teuthology.orchestra.run.smithi007.stdout: "max_rotating_auth_attempts": "10", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_action_on_write_error": "1", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_alternate_name_max": "8192", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_asio_thread_count": "2", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_export_pin": "true", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_fragment_dirs": "true", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_fragment_interval": "5", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_fragment_size_max": "100000", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_idle_threshold": "0.000000", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_interval": "10", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_max": "-1", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_max_until": "-1", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_merge_size": "50", 2024-02-05T07:20:31.964 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_midchunk": "0.300000", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_min_rebalance": "0.100000", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_min_start": "0.200000", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_minchunk": "0.001000", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_mode": "0", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_need_max": "1.200000", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_need_min": "0.800000", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_overload_epochs": "2", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_sample_interval": "3.000000", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_split_bits": "3", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_split_rd": "25000.000000", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_split_size": "10000", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_split_wr": "10000.000000", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_target_decay": "10.000000", 2024-02-05T07:20:31.965 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_beacon_grace": "15.000000", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_beacon_interval": "4.000000", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_beacon_mon_down_grace": "60", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_memory_limit": "4294967296", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_mid": "0.700000", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_release_free_interval": "10", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_reservation": "0.050000", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_trim_interval": "1", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_trim_threshold": "262144", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_client_delegate_inos_pct": "50", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_client_prealloc_inos": "1000", 2024-02-05T07:20:31.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_connect_bootstrapping": "false", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_damage_table_max_entries": "10000", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_debug_auth_pins": "false", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_debug_frag": "false", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_debug_scatterstat": "false", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_debug_subtrees": "false", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_decay_halflife": "5.000000", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_default_dir_hash": "2", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_defer_session_stale": "true", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_deny_all_reconnect": "false", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dir_keys_per_op": "16384", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dir_max_commit_size": "10", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dir_max_entries": "0", 2024-02-05T07:20:31.967 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dir_prefetch": "true", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dirstat_min_interval": "1.000000", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dump_cache_after_rejoin": "false", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dump_cache_on_map": "false", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dump_cache_threshold_file": "0", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_early_reply": "true", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_enable_op_tracker": "true", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_enforce_unique_name": "true", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_export_ephemeral_distributed": "true", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_export_ephemeral_random": "true", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_forward_all_requests_to_auth": "false", 2024-02-05T07:20:31.968 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-02-05T07:20:31.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-02-05T07:20:31.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-02-05T07:20:31.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-02-05T07:20:31.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_health_cache_threshold": "1.500000", 2024-02-05T07:20:31.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_health_summarize_threshold": "10", 2024-02-05T07:20:31.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_heartbeat_grace": "15.000000", 2024-02-05T07:20:31.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_heartbeat_reset_grace": "1000", 2024-02-05T07:20:31.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_inject_health_dummy": "false", 2024-02-05T07:20:31.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-02-05T07:20:31.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_inject_migrator_session_race": "false", 2024-02-05T07:20:31.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-02-05T07:20:31.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-02-05T07:20:31.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-02-05T07:20:31.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_join_fs": "", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_journal_format": "1", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_create_at": "0", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_export_at": "0", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_import_at": "0", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_journal_at": "0", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_journal_expire_at": "0", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_journal_replay_at": "0", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_link_at": "0", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_mdstable_at": "0", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_openc_at": "0", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_rename_at": "0", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_shutdown_at": "0", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_event_large_threshold": "524288", 2024-02-05T07:20:31.970 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_events_per_segment": "1024", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_major_segment_event_ratio": "12", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_max_events": "-1", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_max_segments": "128", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_pause": "false", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_segment_size": "0", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_skip_corrupt_events": "false", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_skip_unbounded_events": "false", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_trim_threshold": "128", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_warn_factor": "2.000000", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_caps_per_client": "1048576", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_completed_flushes": "100000", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_completed_requests": "100000", 2024-02-05T07:20:31.971 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_export_size": "20971520", 2024-02-05T07:20:31.972 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_file_recover": "32", 2024-02-05T07:20:31.972 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_purge_files": "64", 2024-02-05T07:20:31.972 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_purge_ops": "8192", 2024-02-05T07:20:31.972 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-02-05T07:20:31.972 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-02-05T07:20:31.972 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_snaps_per_dir": "100", 2024-02-05T07:20:31.972 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_metrics_update_interval": "2", 2024-02-05T07:20:31.972 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_min_caps_per_client": "100", 2024-02-05T07:20:31.972 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_min_caps_working_set": "10000", 2024-02-05T07:20:31.972 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-02-05T07:20:31.972 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_numa_node": "-1", 2024-02-05T07:20:31.972 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-02-05T07:20:31.972 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_op_complaint_time": "30.000000", 2024-02-05T07:20:31.972 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_op_history_duration": "600", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_op_history_size": "20", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_op_history_slow_op_size": "20", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_op_history_slow_op_threshold": "10", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_op_log_threshold": "5", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_ping_grace": "15", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_ping_interval": "5", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_recall_max_caps": "30000", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_recall_max_decay_threshold": "131072", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_recall_warning_threshold": "262144", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_reconnect_timeout": "45.000000", 2024-02-05T07:20:31.973 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_replay_interval": "1.000000", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_root_ino_gid": "0", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_root_ino_uid": "0", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_blocklist_on_evict": "true", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_blocklist_on_timeout": "true", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_metadata_threshold": "16777216", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-02-05T07:20:31.974 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_shutdown_check": "0", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_skip_ino": "0", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_sleep_rank_change": "0.000000", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_snap_max_uid": "4294967294", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_snap_min_uid": "0", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_snap_rstat": "false", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_standby_replay_damaged": "false", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_symlink_recovery": "true", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_task_status_update_interval": "2.000000", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_thrash_exports": "0", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_thrash_fragments": "0", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_tick_interval": "5.000000", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_valgrind_exit": "false", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_verify_backtrace": "1", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_verify_scatter": "false", 2024-02-05T07:20:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_wipe_ino_prealloc": "false", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_wipe_sessions": "false", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "mempool_debug": "false", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "memstore_debug_omit_block_device_write": "false", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "memstore_device_bytes": "1073741824", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "memstore_page_set": "false", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "memstore_page_size": "65536", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_client_bytes": "134217728", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_client_messages": "512", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_connect_retry_interval": "1.000000", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_disabled_modules": "", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-02-05T07:20:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_max_pg_num_change": "128", 2024-02-05T07:20:31.977 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_mds_bytes": "134217728", 2024-02-05T07:20:31.977 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_mds_messages": "128", 2024-02-05T07:20:31.977 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-02-05T07:20:31.977 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_mon_bytes": "134217728", 2024-02-05T07:20:31.978 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_mon_messages": "128", 2024-02-05T07:20:31.978 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_osd_bytes": "536870912", 2024-02-05T07:20:31.978 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_osd_messages": "8192", 2024-02-05T07:20:31.978 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_pool": "true", 2024-02-05T07:20:31.979 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_service_beacon_grace": "60.000000", 2024-02-05T07:20:31.979 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_standby_modules": "true", 2024-02-05T07:20:31.979 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_stats_period": "5", 2024-02-05T07:20:31.979 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_stats_threshold": "5", 2024-02-05T07:20:31.979 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_tick_period": "2", 2024-02-05T07:20:31.979 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-02-05T07:20:31.979 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_accept_timeout_factor": "2.000000", 2024-02-05T07:20:31.979 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_allow_pool_delete": "true", 2024-02-05T07:20:31.979 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_allow_pool_size_one": "false", 2024-02-05T07:20:31.979 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_auth_validate_all_caps": "true", 2024-02-05T07:20:31.979 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-02-05T07:20:31.979 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-02-05T07:20:31.979 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_bytes": "104857600", 2024-02-05T07:20:31.979 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_directed_command_retry": "2", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_hunt_interval": "3.000000", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_hunt_parallel": "3", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_log_interval": "1.000000", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_ping_interval": "10.000000", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_ping_timeout": "30.000000", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_target_rank": "-1", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_clock_drift_allowed": "1.000000", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_file_level": "debug", 2024-02-05T07:20:31.980 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_file": "true", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_graylog": "false", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_journald": "true", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_stderr": "false", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_compact_on_bootstrap": "false", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_compact_on_start": "false", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_compact_on_trim": "true", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_con_tracker_persist_interval": "10", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_con_tracker_score_halflife": "43200", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_config_key_max_entry_size": "65536", 2024-02-05T07:20:31.981 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cpu_threads": "4", 2024-02-05T07:20:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_crush_min_required_version": "hammer", 2024-02-05T07:20:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_daemon_bytes": "419430400", 2024-02-05T07:20:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-02-05T07:20:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_data_avail_crit": "5", 2024-02-05T07:20:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_data_avail_warn": "5", 2024-02-05T07:20:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_data_size_warn": "16106127360", 2024-02-05T07:20:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_block_osdmap_trim": "false", 2024-02-05T07:20:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-02-05T07:20:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_dump_json": "false", 2024-02-05T07:20:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-02-05T07:20:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_dump_transactions": "false", 2024-02-05T07:20:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_extra_checks": "false", 2024-02-05T07:20:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-02-05T07:20:31.982 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_no_require_reef": "false", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_no_require_squid": "false", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_delta_reset_interval": "10.000000", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_dns_srv_name": "ceph-mon", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_down_mkfs_grace": "60", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_election_default_strategy": "1", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_election_timeout": "5.000000", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_elector_ping_divisor": "2", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_elector_ping_timeout": "2.000000", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_enable_op_tracker": "true", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_fake_pool_delete": "false", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_force_quorum_join": "false", 2024-02-05T07:20:31.983 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_globalid_prealloc": "10000", 2024-02-05T07:20:31.984 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_health_detail_to_clog": "true", 2024-02-05T07:20:31.984 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_health_log_update_period": "5", 2024-02-05T07:20:31.984 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_health_max_detail": "50", 2024-02-05T07:20:31.984 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_health_to_clog": "true", 2024-02-05T07:20:31.984 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_health_to_clog_interval": "600", 2024-02-05T07:20:31.984 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-02-05T07:20:31.984 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_host": "[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]", 2024-02-05T07:20:31.984 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_host_override": "", 2024-02-05T07:20:31.984 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_initial_members": "", 2024-02-05T07:20:31.984 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-02-05T07:20:31.984 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-02-05T07:20:31.984 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-02-05T07:20:31.984 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-02-05T07:20:31.984 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_keyvaluedb": "rocksdb", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_lease": "5.000000", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_log_full_interval": "50", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_log_max": "10000", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_log_max_summary": "50", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_log_entries_per_event": "4096", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_log_epochs": "500", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_mdsmap_epochs": "500", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_mgrmap_epochs": "500", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_osd": "10000", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_pg_per_osd": "10000", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_pool_pg_num": "65536", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-02-05T07:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mds_force_trim_to": "0", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mds_skip_sanity": "false", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_memory_autotune": "true", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_memory_target": "2147483648", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mgr_beacon_grace": "30", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mgr_digest_period": "5", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mgr_inactive_grace": "60", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mgr_mkfs_grace": "240", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_min_osdmap_epochs": "50", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_op_complaint_time": "30", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_op_history_duration": "600", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_op_history_size": "20", 2024-02-05T07:20:31.986 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_op_history_slow_op_size": "20", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_op_history_slow_op_threshold": "10", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_op_log_threshold": "5", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_auto_mark_in": "false", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_auto_mark_new_in": "true", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_cache_size": "500", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_cache_size_min": "134217728", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_crush_smoke_test": "true", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_destroyed_out_interval": "600", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_down_out_interval": "600", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-02-05T07:20:31.987 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_force_trim_to": "0", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_full_ratio": "0.950000", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_laggy_halflife": "3600", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_laggy_max_interval": "300", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_laggy_weight": "0.300000", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_max_creating_pgs": "1024", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_max_initial_pgs": "1024", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_min_down_reporters": "2", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_prime_pg_temp": "true", 2024-02-05T07:20:31.988 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-02-05T07:20:31.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-02-05T07:20:31.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_report_timeout": "900", 2024-02-05T07:20:31.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-02-05T07:20:31.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-02-05T07:20:31.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_warn_num_repaired": "10", 2024-02-05T07:20:31.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_warn_op_age": "32.000000", 2024-02-05T07:20:31.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-02-05T07:20:31.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osdmap_full_prune_interval": "2", 2024-02-05T07:20:31.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osdmap_full_prune_min": "15", 2024-02-05T07:20:31.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-02-05T07:20:31.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-02-05T07:20:31.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pg_stuck_threshold": "60", 2024-02-05T07:20:31.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-02-05T07:20:31.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pg_warn_min_objects": "10000", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pg_warn_min_per_osd": "0", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pool_quota_crit_threshold": "0", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pool_quota_warn_threshold": "0", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_probe_timeout": "2.000000", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_reweight_max_change": "0.050000", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_reweight_max_osds": "4", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_scrub_interval": "86400", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_scrub_max_keys": "100", 2024-02-05T07:20:31.990 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_scrub_timeout": "300", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_session_timeout": "300", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_smart_report_timeout": "5", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_stat_smooth_intervals": "6", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_stretch_pool_min_size": "2", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_stretch_pool_size": "4", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_subscribe_interval": "86400.000000", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_sync_debug": "false", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_sync_max_payload_keys": "2000", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_sync_max_payload_size": "1048576", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_sync_provider_kill_at": "0", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_sync_requester_kill_at": "0", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_sync_timeout": "60.000000", 2024-02-05T07:20:31.991 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_target_pg_per_osd": "100", 2024-02-05T07:20:31.992 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_tick_interval": "5", 2024-02-05T07:20:31.992 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_timecheck_interval": "300.000000", 2024-02-05T07:20:31.992 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-02-05T07:20:31.992 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_use_min_delay_socket": "false", 2024-02-05T07:20:31.992 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_older_version_delay": "604800", 2024-02-05T07:20:31.992 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-02-05T07:20:31.992 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-02-05T07:20:31.992 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-02-05T07:20:31.992 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_filestore_osds": "true", 2024-02-05T07:20:31.992 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-02-05T07:20:31.992 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-02-05T07:20:31.992 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-02-05T07:20:31.992 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_misplaced": "false", 2024-02-05T07:20:31.992 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_older_version": "true", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_pool_no_app": "true", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_too_few_osds": "false", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "monmap": "", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-02-05T07:20:31.993 INFO:teuthology.orchestra.run.smithi007.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "motr_tracing_enabled": "false", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_op_threads": "3", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_buffer_size": "131072", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_cm": "false", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_device_name": "", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_dscp": "96", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_gid_idx": "0", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_local_gid": "", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_polling_us": "1000", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_port_num": "1", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-02-05T07:20:31.994 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_roce_ver": "1", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_send_buffers": "1024", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_sl": "3", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_support_srq": "true", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_type": "ib", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_reap_threshold": "5", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_before_connect": "false", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_ipv4": "true", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_ipv6": "false", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_msgr1": "false", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_msgr2": "true", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_port_max": "7568", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_port_min": "6800", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_prefer_ipv4": "false", 2024-02-05T07:20:31.995 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_retry_count": "3", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_retry_delay": "5", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_blackhole_client": "false", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_blackhole_mds": "false", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_blackhole_mgr": "false", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_blackhole_mon": "false", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_blackhole_osd": "false", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_client_mode": "crc secure", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_cluster_mode": "crc secure", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_cluster_type": "", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_compress_secure": "false", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_connection_idle_timeout": "900", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_connection_ready_timeout": "10", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_crc_data": "true", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_crc_header": "true", 2024-02-05T07:20:31.996 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_die_on_bad_msg": "false", 2024-02-05T07:20:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_die_on_bug": "true", 2024-02-05T07:20:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_die_on_old_message": "true", 2024-02-05T07:20:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_die_on_skipped_message": "false", 2024-02-05T07:20:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_die_on_unhandled_msg": "false", 2024-02-05T07:20:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-02-05T07:20:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_coremask": "0xF", 2024-02-05T07:20:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-02-05T07:20:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_devs_allowlist": "", 2024-02-05T07:20:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_enable_tso": "true", 2024-02-05T07:20:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-02-05T07:20:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-02-05T07:20:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_hugepages": "", 2024-02-05T07:20:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_hw_flow_control": "true", 2024-02-05T07:20:31.997 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-02-05T07:20:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_lro": "true", 2024-02-05T07:20:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_memory_channel": "4", 2024-02-05T07:20:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-02-05T07:20:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_pmd": "", 2024-02-05T07:20:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_port_id": "0", 2024-02-05T07:20:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-02-05T07:20:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dump_corrupt_message_level": "1", 2024-02-05T07:20:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dump_on_send": "false", 2024-02-05T07:20:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_initial_backoff": "0.200000", 2024-02-05T07:20:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_inject_delay_max": "1.000000", 2024-02-05T07:20:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_inject_delay_probability": "0.000000", 2024-02-05T07:20:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_inject_delay_type": "", 2024-02-05T07:20:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_inject_internal_delays": "0.000000", 2024-02-05T07:20:31.998 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_inject_network_congestion": "0", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_inject_socket_failures": "0", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_learn_addr_from_peer": "true", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_max_accept_failures": "4", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_max_backoff": "15.000000", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_mon_client_mode": "secure crc", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_mon_cluster_mode": "secure crc", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_mon_service_mode": "secure crc", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_osd_compress_min_size": "1024", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_osd_compress_mode": "none", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_osd_compression_algorithm": "snappy", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_pq_min_cost": "65536", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_public_type": "", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_service_mode": "crc secure", 2024-02-05T07:20:31.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_tcp_listen_backlog": "512", 2024-02-05T07:20:32.000 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_tcp_nodelay": "true", 2024-02-05T07:20:32.000 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-02-05T07:20:32.000 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_tcp_rcvbuf": "0", 2024-02-05T07:20:32.000 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_type": "async+posix", 2024-02-05T07:20:32.000 INFO:teuthology.orchestra.run.smithi007.stdout: "no_config_file": "false", 2024-02-05T07:20:32.000 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_completion_locks_per_session": "32", 2024-02-05T07:20:32.000 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_debug_inject_relock_delay": "false", 2024-02-05T07:20:32.000 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_inflight_op_bytes": "104857600", 2024-02-05T07:20:32.000 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_inflight_ops": "1024", 2024-02-05T07:20:32.000 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_inject_no_watch_ping": "false", 2024-02-05T07:20:32.000 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-02-05T07:20:32.000 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_tick_interval": "5.000000", 2024-02-05T07:20:32.000 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_timeout": "10.000000", 2024-02-05T07:20:32.000 INFO:teuthology.orchestra.run.smithi007.stdout: "objectstore_blackhole": "false", 2024-02-05T07:20:32.001 INFO:teuthology.orchestra.run.smithi007.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-02-05T07:20:32.001 INFO:teuthology.orchestra.run.smithi007.stdout: "openssl_engine_opts": "", 2024-02-05T07:20:32.001 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_agent_delay_time": "5.000000", 2024-02-05T07:20:32.001 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_agent_hist_halflife": "1000", 2024-02-05T07:20:32.003 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_agent_max_low_ops": "2", 2024-02-05T07:20:32.003 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_agent_max_ops": "4", 2024-02-05T07:20:32.003 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-02-05T07:20:32.003 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_agent_quantize_effort": "0.100000", 2024-02-05T07:20:32.003 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_agent_slop": "0.020000", 2024-02-05T07:20:32.003 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-02-05T07:20:32.004 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_allow_recovery_below_min_size": "true", 2024-02-05T07:20:32.004 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_asio_thread_count": "2", 2024-02-05T07:20:32.004 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_async_recovery_min_cost": "100", 2024-02-05T07:20:32.004 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_auto_mark_unfound_lost": "false", 2024-02-05T07:20:32.004 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_backfill_retry_interval": "30.000000", 2024-02-05T07:20:32.004 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_backfill_scan_max": "512", 2024-02-05T07:20:32.004 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_backfill_scan_min": "64", 2024-02-05T07:20:32.004 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_backoff_on_degraded": "false", 2024-02-05T07:20:32.004 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_backoff_on_peering": "false", 2024-02-05T07:20:32.004 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_backoff_on_unfound": "true", 2024-02-05T07:20:32.004 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_beacon_report_interval": "300", 2024-02-05T07:20:32.004 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bench_duration": "30", 2024-02-05T07:20:32.004 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-02-05T07:20:32.004 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bench_max_block_size": "67108864", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bench_small_size_max_iops": "100", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_blkin_trace_all": "false", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_blocked_scrub_grace_period": "120", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_check_for_log_corruption": "false", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_class_update_on_start": "true", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_client_message_cap": "256", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_client_message_size_cap": "524288000", 2024-02-05T07:20:32.005 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_client_op_priority": "63", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_client_watch_timeout": "30", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_command_max_records": "256", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_compact_on_start": "false", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_crush_chooseleaf_type": "1", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_crush_initial_weight": "-1.000000", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_crush_update_on_start": "true", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_crush_update_weight_set": "true", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_drop_ping_duration": "0", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_feed_pullee": "-1", 2024-02-05T07:20:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-02-05T07:20:32.007 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-02-05T07:20:32.007 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-02-05T07:20:32.007 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_misdirected_ops": "false", 2024-02-05T07:20:32.007 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_no_acting_change": "false", 2024-02-05T07:20:32.007 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_no_purge_strays": "false", 2024-02-05T07:20:32.007 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_op_order": "false", 2024-02-05T07:20:32.007 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_pg_log_writeout": "false", 2024-02-05T07:20:32.007 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_pretend_recovery_active": "false", 2024-02-05T07:20:32.007 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-02-05T07:20:32.007 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-02-05T07:20:32.007 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_shutdown": "false", 2024-02-05T07:20:32.007 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-02-05T07:20:32.007 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-02-05T07:20:32.007 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_trim_objects": "false", 2024-02-05T07:20:32.008 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_verify_cached_snaps": "false", 2024-02-05T07:20:32.008 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_verify_missing_on_start": "false", 2024-02-05T07:20:32.008 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_verify_snaps": "false", 2024-02-05T07:20:32.008 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-02-05T07:20:32.008 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-02-05T07:20:32.008 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_deep_scrub_keys": "1024", 2024-02-05T07:20:32.008 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-02-05T07:20:32.008 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-02-05T07:20:32.008 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-02-05T07:20:32.008 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_deep_scrub_stride": "524288", 2024-02-05T07:20:32.008 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-02-05T07:20:32.008 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_default_data_pool_replay_window": "45", 2024-02-05T07:20:32.008 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_default_notify_timeout": "30", 2024-02-05T07:20:32.008 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_delete_sleep": "0.000000", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_discard_disconnected_ops": "true", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_enable_op_tracker": "true", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_extblkdev_plugins": "vdo", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_fast_info": "true", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_fast_shutdown": "true", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_fast_shutdown_timeout": "15", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-02-05T07:20:32.009 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-02-05T07:20:32.010 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-02-05T07:20:32.010 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_function_tracing": "false", 2024-02-05T07:20:32.010 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_heartbeat_grace": "20", 2024-02-05T07:20:32.010 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_heartbeat_interval": "6", 2024-02-05T07:20:32.010 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-02-05T07:20:32.010 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_heartbeat_min_peers": "10", 2024-02-05T07:20:32.010 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_heartbeat_min_size": "2000", 2024-02-05T07:20:32.010 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_heartbeat_stale": "600", 2024-02-05T07:20:32.010 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-02-05T07:20:32.010 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_hit_set_max_size": "100000", 2024-02-05T07:20:32.010 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_hit_set_min_size": "1000", 2024-02-05T07:20:32.010 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-02-05T07:20:32.010 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-02-05T07:20:32.010 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_journal_flush_on_shutdown": "true", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_journal_size": "5120", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_kill_backfill_at": "0", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_loop_before_reset_tphandle": "64", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_map_cache_size": "50", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_map_dedup": "true", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_map_message_max": "40", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_map_message_max_bytes": "10485760", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_map_share_max_epochs": "40", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_attr_name_len": "100", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_attr_size": "0", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_backfills": "1", 2024-02-05T07:20:32.011 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_markdown_count": "5", 2024-02-05T07:20:32.012 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_markdown_period": "600", 2024-02-05T07:20:32.012 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_object_name_len": "2048", 2024-02-05T07:20:32.012 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_object_namespace_len": "256", 2024-02-05T07:20:32.012 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_object_size": "134217728", 2024-02-05T07:20:32.012 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-02-05T07:20:32.012 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_omap_entries_per_request": "1024", 2024-02-05T07:20:32.012 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_pg_blocked_by": "16", 2024-02-05T07:20:32.012 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_pg_log_entries": "10000", 2024-02-05T07:20:32.012 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-02-05T07:20:32.012 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_pgls": "1024", 2024-02-05T07:20:32.012 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_push_cost": "8388608", 2024-02-05T07:20:32.012 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_push_objects": "10", 2024-02-05T07:20:32.012 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_scrubs": "3", 2024-02-05T07:20:32.012 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_trimming_pgs": "2", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_write_op_reply_len": "64", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_write_size": "90", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_override_recovery_settings": "false", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_profile": "balanced", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-02-05T07:20:32.013 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-02-05T07:20:32.014 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-02-05T07:20:32.014 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-02-05T07:20:32.014 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-02-05T07:20:32.014 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-02-05T07:20:32.014 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-02-05T07:20:32.014 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-02-05T07:20:32.014 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-02-05T07:20:32.014 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_skip_benchmark": "false", 2024-02-05T07:20:32.014 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_memory_base": "805306368", 2024-02-05T07:20:32.014 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_memory_cache_min": "134217728", 2024-02-05T07:20:32.014 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-02-05T07:20:32.014 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-02-05T07:20:32.014 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_memory_target": "4294967296", 2024-02-05T07:20:32.014 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_memory_target_autotune": "false", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_min_pg_log_entries": "250", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_min_recovery_priority": "0", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mon_heartbeat_interval": "30", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mon_report_interval": "5", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mon_report_max_in_flight": "2", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_num_cache_shards": "32", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_num_op_tracker_shard": "32", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_numa_auto_affinity": "true", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_numa_node": "-1", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_numa_prefer_iface": "true", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-02-05T07:20:32.015 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_objecter_finishers": "1", 2024-02-05T07:20:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_objectstore": "bluestore", 2024-02-05T07:20:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_objectstore_fuse": "false", 2024-02-05T07:20:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_objectstore_tracing": "false", 2024-02-05T07:20:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_complaint_time": "30.000000", 2024-02-05T07:20:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_history_duration": "600", 2024-02-05T07:20:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_history_size": "20", 2024-02-05T07:20:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_history_slow_op_size": "20", 2024-02-05T07:20:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-02-05T07:20:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_log_threshold": "5", 2024-02-05T07:20:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_num_shards": "0", 2024-02-05T07:20:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_num_shards_hdd": "5", 2024-02-05T07:20:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_num_shards_ssd": "8", 2024-02-05T07:20:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_num_threads_per_shard": "0", 2024-02-05T07:20:32.016 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_pq_min_cost": "65536", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_queue": "mclock_scheduler", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_queue_cut_off": "high", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_thread_suicide_timeout": "150", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_thread_timeout": "15", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_open_classes_on_start": "true", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_os_flags": "0", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_peering_op_priority": "255", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_delete_cost": "1048576", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_delete_priority": "5", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_log_dups_tracked": "3000", 2024-02-05T07:20:32.017 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_log_trim_max": "10000", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_log_trim_min": "100", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_object_context_cache_count": "64", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_stat_report_interval_max_seconds": "500", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_crimson": "false", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_crush_rule": "-1", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_ec_fast_read": "false", 2024-02-05T07:20:32.018 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_flag_bulk": "false", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_flag_nodelete": "false", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_flags": "0", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_min_size": "0", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_pg_num": "32", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_pgp_num": "0", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_read_ratio": "70", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_size": "3", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_type": "replicated", 2024-02-05T07:20:32.019 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_use_gmt_hitset": "true", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_push_per_object_cost": "1000", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_read_ec_check_for_errors": "false", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recover_clone_overlap": "true", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recover_clone_overlap_limit": "10", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_cost": "20971520", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_delay_start": "0.000000", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_max_active": "0", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_max_active_hdd": "3", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_max_active_ssd": "10", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_max_chunk": "8388608", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_max_single_start": "1", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_op_priority": "3", 2024-02-05T07:20:32.020 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_op_warn_multiple": "16", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_priority": "5", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_retry_interval": "30.000000", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_sleep": "0.000000", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_repair_during_recovery": "false", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_requested_scrub_priority": "120", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_rollback_to_cluster_snap": "", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_auto_repair": "false", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_begin_hour": "0", 2024-02-05T07:20:32.021 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_begin_week_day": "0", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_chunk_max": "25", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_chunk_min": "5", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_cost": "52428800", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_during_recovery": "false", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_end_hour": "0", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_end_week_day": "0", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_event_cost": "4096", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_invalid_stats": "true", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_load_threshold": "0.500000", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_max_interval": "604800.000000", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_max_preemptions": "5", 2024-02-05T07:20:32.022 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_min_interval": "86400.000000", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_priority": "5", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_reservation_timeout": "300000", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_sleep": "0.000000", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_shutdown_pgref_assert": "false", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_skip_data_digest": "false", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_smart_report_timeout": "5", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_snap_trim_cost": "1048576", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_snap_trim_priority": "5", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_snap_trim_sleep": "0.000000", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-02-05T07:20:32.023 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_stats_update_period_scrubbing": "15", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_target_transaction_size": "30", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_mode": "writeback", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-02-05T07:20:32.024 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tracing": "false", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_use_stale_snap": "false", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "osdc_blkin_trace_all": "false", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_kill_at": "0", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_max_join_drift": "10", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_min": "500", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_min_wait": "0.050000", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_propose_interval": "1.000000", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_service_trim_max": "500", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_service_trim_max_multiplier": "20", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_service_trim_min": "10", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_stash_full_interval": "25", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_trim_max": "500", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_trim_min": "250", 2024-02-05T07:20:32.025 INFO:teuthology.orchestra.run.smithi007.stdout: "perf": "true", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "pid_file": "", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "-", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrv": "", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "public_bind_addr": "-", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "public_network": "", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "public_network_interface": "", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "qat_compressor_enabled": "false", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "qat_compressor_session_max_number": "256", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "rados_mon_op_timeout": "0", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "rados_osd_op_timeout": "0", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "rados_tracing": "false", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_atime_update_interval": "60", 2024-02-05T07:20:32.026 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-02-05T07:20:32.027 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_balance_parent_reads": "false", 2024-02-05T07:20:32.027 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_balance_snap_reads": "false", 2024-02-05T07:20:32.027 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_blkin_trace_all": "false", 2024-02-05T07:20:32.027 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_blocklist_expire_seconds": "0", 2024-02-05T07:20:32.027 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_blocklist_on_break_lock": "true", 2024-02-05T07:20:32.027 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache": "true", 2024-02-05T07:20:32.027 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_block_writes_upfront": "false", 2024-02-05T07:20:32.027 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_max_dirty": "25165824", 2024-02-05T07:20:32.027 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-02-05T07:20:32.027 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_max_dirty_object": "0", 2024-02-05T07:20:32.027 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_policy": "writearound", 2024-02-05T07:20:32.027 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_size": "33554432", 2024-02-05T07:20:32.027 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_target_dirty": "16777216", 2024-02-05T07:20:32.027 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_clone_copy_on_read": "false", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_compression_hint": "none", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_concurrent_management_ops": "10", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_clone_format": "auto", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_data_pool": "", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_features": "61", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_format": "2", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_map_options": "", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_order": "22", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_pool": "rbd", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_stripe_count": "0", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_stripe_unit": "0", 2024-02-05T07:20:32.028 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_disable_zero_copy_writes": "true", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_discard_granularity_bytes": "65536", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_enable_alloc_hint": "true", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_io_scheduler": "simple", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_commit_age": "5.000000", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_object_flush_interval": "0", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-02-05T07:20:32.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_order": "24", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_pool": "", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_splay_width": "4", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_localize_parent_reads": "false", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_localize_snap_reads": "false", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_image_policy_type": "simple", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-02-05T07:20:32.030 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_memory_autotune": "true", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_memory_base": "805306368", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_memory_target": "4294967296", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirroring_delete_delay": "0", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-02-05T07:20:32.031 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirroring_replay_delay": "0", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_move_to_trash_on_remove": "false", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mtime_update_interval": "60", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_non_blocking_aio": "true", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_op_thread_timeout": "60", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_op_threads": "1", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_parent_cache_enabled": "false", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_persistent_cache_mode": "disabled", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_persistent_cache_path": "/tmp", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_persistent_cache_size": "1073741824", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_plugins": "", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_bps_burst": "0", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-02-05T07:20:32.032 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_bps_limit": "0", 2024-02-05T07:20:32.033 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_exclude_ops": "0", 2024-02-05T07:20:32.033 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_iops_burst": "0", 2024-02-05T07:20:32.033 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-02-05T07:20:32.033 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_iops_limit": "0", 2024-02-05T07:20:32.033 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_read_bps_burst": "0", 2024-02-05T07:20:32.033 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-02-05T07:20:32.033 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_read_bps_limit": "0", 2024-02-05T07:20:32.033 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_read_iops_burst": "0", 2024-02-05T07:20:32.033 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-02-05T07:20:32.033 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_read_iops_limit": "0", 2024-02-05T07:20:32.033 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_schedule_tick_min": "50", 2024-02-05T07:20:32.033 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_write_bps_burst": "0", 2024-02-05T07:20:32.033 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-02-05T07:20:32.033 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_write_bps_limit": "0", 2024-02-05T07:20:32.034 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_write_iops_burst": "0", 2024-02-05T07:20:32.034 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-02-05T07:20:32.034 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_write_iops_limit": "0", 2024-02-05T07:20:32.034 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_quiesce_notification_attempts": "10", 2024-02-05T07:20:32.034 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_read_from_replica_policy": "default", 2024-02-05T07:20:32.034 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-02-05T07:20:32.034 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_readahead_max_bytes": "524288", 2024-02-05T07:20:32.034 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_readahead_trigger_requests": "10", 2024-02-05T07:20:32.034 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_request_timed_out_seconds": "30", 2024-02-05T07:20:32.034 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_skip_partial_discard": "true", 2024-02-05T07:20:32.034 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-02-05T07:20:32.034 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_tracing": "false", 2024-02-05T07:20:32.034 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_validate_names": "true", 2024-02-05T07:20:32.034 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_validate_pool": "true", 2024-02-05T07:20:32.035 INFO:teuthology.orchestra.run.smithi007.stdout: "restapi_base_url": "", 2024-02-05T07:20:32.035 INFO:teuthology.orchestra.run.smithi007.stdout: "restapi_log_level": "", 2024-02-05T07:20:32.035 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_acl_grants_max_num": "100", 2024-02-05T07:20:32.035 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_admin_entry": "admin", 2024-02-05T07:20:32.035 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-02-05T07:20:32.035 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_backend_store": "rados", 2024-02-05T07:20:32.035 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_barbican_url": "", 2024-02-05T07:20:32.035 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_beast_enable_async": "true", 2024-02-05T07:20:32.035 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_counters_cache": "false", 2024-02-05T07:20:32.035 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-02-05T07:20:32.035 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-02-05T07:20:32.035 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-02-05T07:20:32.035 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_index_max_aio": "128", 2024-02-05T07:20:32.035 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_quota_ttl": "600", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_cache_enabled": "true", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_cache_expiry_interval": "900", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_cache_lru_size": "10000", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_config_store": "rados", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_content_length_compat": "false", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_copy_obj_progress": "true", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_cors_rules_max_num": "100", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_cross_domain_policy": "", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_default_encryption_key": "", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_addr": "", 2024-02-05T07:20:32.036 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_ca_path": "", 2024-02-05T07:20:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_client_cert": "", 2024-02-05T07:20:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_client_key": "", 2024-02-05T07:20:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-02-05T07:20:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_password": "", 2024-02-05T07:20:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-02-05T07:20:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_username": "", 2024-02-05T07:20:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_require_ssl": "true", 2024-02-05T07:20:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-02-05T07:20:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-02-05T07:20:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-02-05T07:20:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-02-05T07:20:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-02-05T07:20:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-02-05T07:20:32.037 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_suppress_logs": "true", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_addr": "", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_auth": "token", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_namespace": "", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_prefix": "", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-02-05T07:20:32.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_token_file": "", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_curl_buffersize": "524288", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_curl_low_speed_limit": "1024", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_curl_low_speed_time": "300", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_curl_tcp_keepalive": "0", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_l1_fadvise": "4", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-02-05T07:20:32.039 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-02-05T07:20:32.040 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-02-05T07:20:32.040 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d4n_host": "127.0.0.1", 2024-02-05T07:20:32.040 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d4n_port": "6379", 2024-02-05T07:20:32.040 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-02-05T07:20:32.040 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data_log_changes_size": "1000", 2024-02-05T07:20:32.040 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data_log_num_shards": "128", 2024-02-05T07:20:32.040 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-02-05T07:20:32.040 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data_log_window": "30", 2024-02-05T07:20:32.040 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data_notify_interval_msec": "0", 2024-02-05T07:20:32.040 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data_sync_poll_interval": "20", 2024-02-05T07:20:32.040 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data_sync_spawn_window": "20", 2024-02-05T07:20:32.040 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-02-05T07:20:32.040 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-02-05T07:20:32.040 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_default_data_log_backing": "fifo", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_default_region_info_oid": "default.region", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_defer_to_bucket_acls": "", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-02-05T07:20:32.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_data_res": "500.000000", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dns_name": "", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dns_s3website_name": "", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dynamic_resharding": "true", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_gc_threads": "true", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_lc_threads": "true", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_ops_log": "false", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_quota_threads": "true", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_static_website": "false", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_usage_log": "false", 2024-02-05T07:20:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enforce_swift_acls": "true", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_exit_timeout_secs": "120", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_expose_bucket": "false", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_extended_http_attrs": "", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_filter": "none", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_frontends": "beast port=7480", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_max_concurrent_io": "10", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_max_deferred": "50", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_max_objs": "32", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_max_queue_size": "134213632", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_max_trim_chunk": "16", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_obj_min_wait": "7200", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_processor_max_time": "3600", 2024-02-05T07:20:32.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_processor_period": "3600", 2024-02-05T07:20:32.044 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-02-05T07:20:32.044 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_get_obj_window_size": "16777216", 2024-02-05T07:20:32.044 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_healthcheck_disabling_path": "", 2024-02-05T07:20:32.044 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ignore_get_invalid_range": "false", 2024-02-05T07:20:32.044 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_init_timeout": "300", 2024-02-05T07:20:32.044 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-02-05T07:20:32.044 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-02-05T07:20:32.044 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_kafka_connection_idle": "30", 2024-02-05T07:20:32.044 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_kafka_sleep_timeout": "10", 2024-02-05T07:20:32.044 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-02-05T07:20:32.044 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-02-05T07:20:32.044 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-02-05T07:20:32.044 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_domain": "", 2024-02-05T07:20:32.044 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_password": "", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_password_path": "", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_project": "", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_tenant": "", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_token": "", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_token_path": "", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_user": "", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_api_version": "2", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_barbican_domain": "", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_barbican_password": "", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_barbican_project": "", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_barbican_tenant": "", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_barbican_user": "", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_implicit_tenants": "false", 2024-02-05T07:20:32.045 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-02-05T07:20:32.046 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_service_token_enabled": "false", 2024-02-05T07:20:32.046 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_token_cache_size": "10000", 2024-02-05T07:20:32.046 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_url": "", 2024-02-05T07:20:32.046 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_verify_ssl": "true", 2024-02-05T07:20:32.046 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lc_debug_interval": "-1", 2024-02-05T07:20:32.046 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lc_lock_max_time": "90", 2024-02-05T07:20:32.046 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lc_max_objs": "32", 2024-02-05T07:20:32.046 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lc_max_rules": "1000", 2024-02-05T07:20:32.046 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lc_max_worker": "3", 2024-02-05T07:20:32.046 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lc_max_wp_worker": "3", 2024-02-05T07:20:32.046 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lc_thread_delay": "0", 2024-02-05T07:20:32.046 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-02-05T07:20:32.046 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ldap_dnattr": "uid", 2024-02-05T07:20:32.046 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ldap_searchfilter": "", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ldap_uri": "ldaps://", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_log_http_headers": "", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_log_nonexistent_bucket": "false", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_log_object_name_utc": "false", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_attr_name_len": "0", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_attr_size": "0", 2024-02-05T07:20:32.047 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_attrs_num_in_req": "0", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_chunk_size": "4194304", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_concurrent_requests": "1024", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_dynamic_shards": "1999", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_listing_results": "1000", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_notify_retries": "10", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_objs_per_shard": "100000", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_put_param_size": "1048576", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_put_size": "5368709120", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_slo_entries": "1000", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_md_log_max_shards": "64", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_md_notify_interval_msec": "200", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_meta_sync_poll_interval": "20", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_meta_sync_spawn_window": "20", 2024-02-05T07:20:32.048 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_mp_lock_max_time": "600", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_multipart_min_part_size": "5242880", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_fhcache_size": "2017", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_lru_lanes": "5", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_max_gc": "300", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_run_gc_threads": "false", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_run_lc_threads": "false", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_run_quota_threads": "false", 2024-02-05T07:20:32.049 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_run_sync_thread": "false", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_num_async_rados_threads": "32", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_num_control_oids": "8", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_numa_node": "-1", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_obj_stripe_size": "4194304", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_objexp_chunk_size": "100", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_objexp_gc_interval": "600", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_objexp_hints_num_shards": "127", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_op_thread_timeout": "600", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_op_tracing": "false", 2024-02-05T07:20:32.050 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_opa_token": "", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_opa_url": "", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_opa_verify_ssl": "true", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ops_log_rados": "false", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ops_log_socket_path": "", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_period_push_interval": "2.000000", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_period_push_interval_max": "30.000000", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_period_root_pool": ".rgw.root", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-02-05T07:20:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_posix_cache_lanes": "3", 2024-02-05T07:20:32.052 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-02-05T07:20:32.052 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_posix_cache_max_buckets": "100", 2024-02-05T07:20:32.052 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_posix_cache_partitions": "3", 2024-02-05T07:20:32.052 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-02-05T07:20:32.052 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_print_continue": "true", 2024-02-05T07:20:32.052 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_print_prohibited_content_length": "false", 2024-02-05T07:20:32.052 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-02-05T07:20:32.052 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-02-05T07:20:32.052 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-02-05T07:20:32.052 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-02-05T07:20:32.052 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_rados_tracing": "false", 2024-02-05T07:20:32.052 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_realm": "", 2024-02-05T07:20:32.052 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_realm_id": "", 2024-02-05T07:20:32.052 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_region": "", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_region_root_pool": ".rgw.root", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_relaxed_region_enforcement": "false", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_request_uri": "", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_reshard_batch_size": "64", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_reshard_max_aio": "128", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_reshard_num_logs": "16", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_reshard_thread_interval": "600", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_resolve_cname": "false", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_rest_getusage_op_compat": "false", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_run_sync_thread": "true", 2024-02-05T07:20:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-02-05T07:20:32.054 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_auth_use_keystone": "false", 2024-02-05T07:20:32.054 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_auth_use_ldap": "false", 2024-02-05T07:20:32.054 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_auth_use_rados": "true", 2024-02-05T07:20:32.054 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_auth_use_sts": "false", 2024-02-05T07:20:32.054 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-02-05T07:20:32.054 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_success_create_obj_status": "0", 2024-02-05T07:20:32.054 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-02-05T07:20:32.054 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_scheduler_type": "throttler", 2024-02-05T07:20:32.054 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_script_uri": "", 2024-02-05T07:20:32.054 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_service_provider_name": "", 2024-02-05T07:20:32.054 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-02-05T07:20:32.054 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sts_client_id": "", 2024-02-05T07:20:32.054 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sts_client_secret": "", 2024-02-05T07:20:32.054 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sts_entry": "sts", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sts_key": "sts", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sts_max_session_duration": "43200", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sts_min_session_duration": "900", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sts_token_introspection_url": "", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_account_in_url": "false", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_auth_entry": "auth", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_auth_url": "", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_custom_header": "", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_enforce_content_length": "false", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_need_stats": "true", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_tenant_name": "", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_token_expiration": "86400", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_url": "", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_url_prefix": "swift", 2024-02-05T07:20:32.055 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_versioning_enabled": "false", 2024-02-05T07:20:32.056 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-02-05T07:20:32.056 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-02-05T07:20:32.056 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_lease_period": "120", 2024-02-05T07:20:32.056 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-02-05T07:20:32.056 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_log_trim_interval": "1200", 2024-02-05T07:20:32.056 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-02-05T07:20:32.056 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-02-05T07:20:32.056 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-02-05T07:20:32.056 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_obj_etag_verify": "false", 2024-02-05T07:20:32.056 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_trace_history_size": "4096", 2024-02-05T07:20:32.056 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-02-05T07:20:32.056 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-02-05T07:20:32.056 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_thread_pool_size": "512", 2024-02-05T07:20:32.056 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_topic_persistency_max_retries": "0", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_comment": "", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_createby": "", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_encoding": "", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_flag": "false", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_max_size": "5368709120", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_origin": "", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_sha_unit": "524288", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_tracker": "", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_trust_forwarded_https": "false", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_usage_log_tick_interval": "30", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_usage_max_shards": "32", 2024-02-05T07:20:32.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_usage_max_user_shards": "1", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_use_opa_authz": "false", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_counters_cache": "false", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_counters_cache_size": "10000", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_default_quota_max_size": "-1", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_max_buckets": "1000", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_policies_max_num": "100", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_quota_sync_interval": "86400", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_unique_email": "true", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_verify_ssl": "true", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_website_routing_rules_max_num": "50", 2024-02-05T07:20:32.058 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_zone": "", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_zone_id": "", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_zonegroup": "", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_zonegroup_id": "", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rgwlc_auto_session_clear": "true", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rgwlc_skip_bucket_step": "false", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_block_size": "4096", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cache_shard_bits": "4", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cache_size": "536870912", 2024-02-05T07:20:32.059 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cache_type": "binned_lru", 2024-02-05T07:20:32.060 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-02-05T07:20:32.060 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-02-05T07:20:32.060 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-02-05T07:20:32.060 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_collect_compaction_stats": "false", 2024-02-05T07:20:32.060 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_collect_extended_stats": "false", 2024-02-05T07:20:32.060 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_collect_memory_stats": "false", 2024-02-05T07:20:32.060 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-02-05T07:20:32.060 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_index_type": "binary_search", 2024-02-05T07:20:32.060 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_log_to_ceph_log": "true", 2024-02-05T07:20:32.060 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_metadata_block_size": "4096", 2024-02-05T07:20:32.060 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_partition_filters": "false", 2024-02-05T07:20:32.060 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_perf": "false", 2024-02-05T07:20:32.060 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-02-05T07:20:32.060 INFO:teuthology.orchestra.run.smithi007.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "rotating_keys_renewal_timeout": "10", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "run_dir": "/var/run/ceph", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_block_create": "true", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_cache_lru_size": "67108864", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_cbjournal_size": "5368709120", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_data_delta_based_overwrite": "0", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_default_max_object_size": "16777216", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_device_size": "53687091200", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_journal_batch_capacity": "16", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_journal_iodepth_limit": "5", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_main_device_type": "SSD", 2024-02-05T07:20:32.061 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_max_concurrent_transactions": "8", 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_segment_size": "67108864", 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout: "set_keepcaps": "false", 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout: "setgroup": "ceph", 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout: "setuser": "ceph", 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout: "setuser_match_path": "", 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_misplaced_ratio": "0.050000", 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout: "thp": "false", 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout: "threadpool_default_timeout": "60", 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout: "threadpool_empty_queue_max_wait": "2", 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout: "throttler_perf_counter": "true" 2024-02-05T07:20:32.062 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:20:32.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-02-05T07:20:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:32 smithi090 ceph-mon[33213]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:32 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:20:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:32 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:20:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:32 smithi090 ceph-mon[33213]: 5.7 scrub starts 2024-02-05T07:20:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:32 smithi090 ceph-mon[33213]: 5.7 scrub ok 2024-02-05T07:20:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:32 smithi090 ceph-mon[33213]: 6.18 scrub starts 2024-02-05T07:20:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:32 smithi090 ceph-mon[33213]: 6.18 scrub ok 2024-02-05T07:20:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:32 smithi090 ceph-mon[33213]: 5.19 scrub starts 2024-02-05T07:20:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:32 smithi090 ceph-mon[33213]: 5.19 scrub ok 2024-02-05T07:20:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:32 smithi090 ceph-mon[33213]: osdmap e98: 8 total, 8 up, 8 in 2024-02-05T07:20:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[29098]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:20:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:20:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[29098]: 5.7 scrub starts 2024-02-05T07:20:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[29098]: 5.7 scrub ok 2024-02-05T07:20:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[29098]: 6.18 scrub starts 2024-02-05T07:20:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[29098]: 6.18 scrub ok 2024-02-05T07:20:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[29098]: 5.19 scrub starts 2024-02-05T07:20:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[29098]: 5.19 scrub ok 2024-02-05T07:20:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[29098]: osdmap e98: 8 total, 8 up, 8 in 2024-02-05T07:20:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[37158]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:20:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:20:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[37158]: 5.7 scrub starts 2024-02-05T07:20:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[37158]: 5.7 scrub ok 2024-02-05T07:20:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[37158]: 6.18 scrub starts 2024-02-05T07:20:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[37158]: 6.18 scrub ok 2024-02-05T07:20:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[37158]: 5.19 scrub starts 2024-02-05T07:20:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[37158]: 5.19 scrub ok 2024-02-05T07:20:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:32 smithi007 ceph-mon[37158]: osdmap e98: 8 total, 8 up, 8 in 2024-02-05T07:20:33.537 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:33 smithi007 ceph-mon[29098]: 1.0 scrub starts 2024-02-05T07:20:33.537 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:33 smithi007 ceph-mon[29098]: 1.0 scrub ok 2024-02-05T07:20:33.537 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:33 smithi007 ceph-mon[29098]: 3.5 deep-scrub starts 2024-02-05T07:20:33.537 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:33 smithi007 ceph-mon[29098]: 3.5 deep-scrub ok 2024-02-05T07:20:33.537 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:33 smithi007 ceph-mon[29098]: 3.3 scrub starts 2024-02-05T07:20:33.537 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:33 smithi007 ceph-mon[29098]: 3.3 scrub ok 2024-02-05T07:20:33.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:33 smithi007 ceph-mon[37158]: 1.0 scrub starts 2024-02-05T07:20:33.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:33 smithi007 ceph-mon[37158]: 1.0 scrub ok 2024-02-05T07:20:33.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:33 smithi007 ceph-mon[37158]: 3.5 deep-scrub starts 2024-02-05T07:20:33.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:33 smithi007 ceph-mon[37158]: 3.5 deep-scrub ok 2024-02-05T07:20:33.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:33 smithi007 ceph-mon[37158]: 3.3 scrub starts 2024-02-05T07:20:33.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:33 smithi007 ceph-mon[37158]: 3.3 scrub ok 2024-02-05T07:20:33.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:33 smithi090 ceph-mon[33213]: 1.0 scrub starts 2024-02-05T07:20:33.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:33 smithi090 ceph-mon[33213]: 1.0 scrub ok 2024-02-05T07:20:33.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:33 smithi090 ceph-mon[33213]: 3.5 deep-scrub starts 2024-02-05T07:20:33.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:33 smithi090 ceph-mon[33213]: 3.5 deep-scrub ok 2024-02-05T07:20:33.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:33 smithi090 ceph-mon[33213]: 3.3 scrub starts 2024-02-05T07:20:33.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:33 smithi090 ceph-mon[33213]: 3.3 scrub ok 2024-02-05T07:20:33.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:20:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:20:34.448 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a: {} 2024-02-05T07:20:34.459 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b: {} 2024-02-05T07:20:34.469 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c: {} 2024-02-05T07:20:34.470 INFO:teuthology.orchestra.run.smithi007.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T07:20:34.470 INFO:teuthology.orchestra.run.smithi007.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T07:20:34.470 INFO:teuthology.orchestra.run.smithi007.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T07:20:34.763 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[29098]: 3.13 scrub starts 2024-02-05T07:20:34.763 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[29098]: 3.13 scrub ok 2024-02-05T07:20:34.764 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[29098]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:20:34.764 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[29098]: 3.12 scrub starts 2024-02-05T07:20:34.764 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[29098]: 3.12 scrub ok 2024-02-05T07:20:34.764 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[29098]: 6.17 scrub starts 2024-02-05T07:20:34.764 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[29098]: 6.17 scrub ok 2024-02-05T07:20:34.764 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3506712729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:20:34.764 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[37158]: 3.13 scrub starts 2024-02-05T07:20:34.764 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[37158]: 3.13 scrub ok 2024-02-05T07:20:34.764 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[37158]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:20:34.764 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[37158]: 3.12 scrub starts 2024-02-05T07:20:34.764 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[37158]: 3.12 scrub ok 2024-02-05T07:20:34.764 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[37158]: 6.17 scrub starts 2024-02-05T07:20:34.765 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[37158]: 6.17 scrub ok 2024-02-05T07:20:34.765 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:34 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3506712729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:20:34.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:34 smithi090 ceph-mon[33213]: 3.13 scrub starts 2024-02-05T07:20:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:34 smithi090 ceph-mon[33213]: 3.13 scrub ok 2024-02-05T07:20:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:34 smithi090 ceph-mon[33213]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:20:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:34 smithi090 ceph-mon[33213]: 3.12 scrub starts 2024-02-05T07:20:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:34 smithi090 ceph-mon[33213]: 3.12 scrub ok 2024-02-05T07:20:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:34 smithi090 ceph-mon[33213]: 6.17 scrub starts 2024-02-05T07:20:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:34 smithi090 ceph-mon[33213]: 6.17 scrub ok 2024-02-05T07:20:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:34 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3506712729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:20:35.066 INFO:teuthology.run_tasks:Running task full_sequential... 2024-02-05T07:20:35.076 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T07:20:35.085 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T07:20:35.086 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-02-05T07:20:35.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-02-05T07:20:35.086 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-02-05T07:20:35.087 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T07:20:35.087 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:20:35.087 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-05T07:20:35.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T07:20:35.087 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-02-05T07:20:35.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:35.088 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-02-05T07:20:35.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:20:35.089 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-02-05T07:20:35.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T07:20:35.089 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-02-05T07:20:35.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:20:35.124 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:35.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:35.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:35.256 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:35.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:35.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:35.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:35.384 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:35.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:35.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:35.397 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:35.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:35.513 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:35.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:35.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:35.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:35.639 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:35.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:35.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:35.765 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:35.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:35.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:35.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:35.890 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:35.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:35.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:36.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:36.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:36.146 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:36.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:36.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:36.270 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:36.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:36.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:36.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:36.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:36.400 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:36.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:36.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:36.527 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:36.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:36.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:36.630 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:36.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:36.655 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:36.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:36.662 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:36.662 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:36.663 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:36 smithi007 ceph-mon[29098]: 2.0 scrub starts 2024-02-05T07:20:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:36 smithi007 ceph-mon[29098]: 2.0 scrub ok 2024-02-05T07:20:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:36 smithi007 ceph-mon[29098]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:36 smithi007 ceph-mon[29098]: 6.7 scrub starts 2024-02-05T07:20:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:36 smithi007 ceph-mon[29098]: 6.7 scrub ok 2024-02-05T07:20:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:36 smithi007 ceph-mon[29098]: 3.1f scrub starts 2024-02-05T07:20:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:36 smithi007 ceph-mon[29098]: 3.1f scrub ok 2024-02-05T07:20:36.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:36 smithi007 ceph-mon[37158]: 2.0 scrub starts 2024-02-05T07:20:36.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:36 smithi007 ceph-mon[37158]: 2.0 scrub ok 2024-02-05T07:20:36.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:36 smithi007 ceph-mon[37158]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:36.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:36 smithi007 ceph-mon[37158]: 6.7 scrub starts 2024-02-05T07:20:36.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:36 smithi007 ceph-mon[37158]: 6.7 scrub ok 2024-02-05T07:20:36.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:36 smithi007 ceph-mon[37158]: 3.1f scrub starts 2024-02-05T07:20:36.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:36 smithi007 ceph-mon[37158]: 3.1f scrub ok 2024-02-05T07:20:36.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:36 smithi090 ceph-mon[33213]: 2.0 scrub starts 2024-02-05T07:20:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:36 smithi090 ceph-mon[33213]: 2.0 scrub ok 2024-02-05T07:20:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:36 smithi090 ceph-mon[33213]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:36 smithi090 ceph-mon[33213]: 6.7 scrub starts 2024-02-05T07:20:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:36 smithi090 ceph-mon[33213]: 6.7 scrub ok 2024-02-05T07:20:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:36 smithi090 ceph-mon[33213]: 3.1f scrub starts 2024-02-05T07:20:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:36 smithi090 ceph-mon[33213]: 3.1f scrub ok 2024-02-05T07:20:36.780 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:36.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:36.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:36.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:36.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:36.910 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:36.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:37.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:37.036 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:37.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:37.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:37.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:37.165 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:37.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:37.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:37.294 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:37.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:37.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:37.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:37.420 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:37.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:37.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:37.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:37.545 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:37.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:37.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:37.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:37.670 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:37.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:37.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:37.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:37.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:37 smithi090 ceph-mon[33213]: 4.f deep-scrub starts 2024-02-05T07:20:37.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:37 smithi090 ceph-mon[33213]: 4.f deep-scrub ok 2024-02-05T07:20:37.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:37 smithi090 ceph-mon[33213]: 6.e scrub starts 2024-02-05T07:20:37.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:37 smithi090 ceph-mon[33213]: 6.e scrub ok 2024-02-05T07:20:37.795 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:37.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:37.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:37.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:37 smithi007 ceph-mon[29098]: 4.f deep-scrub starts 2024-02-05T07:20:37.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:37 smithi007 ceph-mon[29098]: 4.f deep-scrub ok 2024-02-05T07:20:37.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:37 smithi007 ceph-mon[29098]: 6.e scrub starts 2024-02-05T07:20:37.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:37 smithi007 ceph-mon[29098]: 6.e scrub ok 2024-02-05T07:20:37.806 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:37 smithi007 ceph-mon[37158]: 4.f deep-scrub starts 2024-02-05T07:20:37.806 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:37 smithi007 ceph-mon[37158]: 4.f deep-scrub ok 2024-02-05T07:20:37.806 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:37 smithi007 ceph-mon[37158]: 6.e scrub starts 2024-02-05T07:20:37.807 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:37 smithi007 ceph-mon[37158]: 6.e scrub ok 2024-02-05T07:20:37.897 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:37.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:37.923 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:37.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:38.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:38.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:38.049 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:38.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:38.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:38.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:38.175 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:38.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:38.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:38.301 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:38.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:38.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:38.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:38.427 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:38.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:38.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:38.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:38.552 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:38.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:38.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:38.576 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:38 smithi007 ceph-mon[29098]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:38.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:38 smithi007 ceph-mon[29098]: 6.a deep-scrub starts 2024-02-05T07:20:38.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:38 smithi007 ceph-mon[29098]: 6.a deep-scrub ok 2024-02-05T07:20:38.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:38 smithi007 ceph-mon[29098]: 6.4 scrub starts 2024-02-05T07:20:38.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:38 smithi007 ceph-mon[29098]: 6.4 scrub ok 2024-02-05T07:20:38.577 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:38 smithi007 ceph-mon[37158]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:38.577 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:38 smithi007 ceph-mon[37158]: 6.a deep-scrub starts 2024-02-05T07:20:38.577 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:38 smithi007 ceph-mon[37158]: 6.a deep-scrub ok 2024-02-05T07:20:38.577 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:38 smithi007 ceph-mon[37158]: 6.4 scrub starts 2024-02-05T07:20:38.577 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:38 smithi007 ceph-mon[37158]: 6.4 scrub ok 2024-02-05T07:20:38.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:38.678 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:38.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:38.774 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:38 smithi090 bash[63710]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-05T07:20:38.775 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:38 smithi090 bash[63710]: Writing manifest to image destination 2024-02-05T07:20:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:38 smithi090 ceph-mon[33213]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:38 smithi090 ceph-mon[33213]: 6.a deep-scrub starts 2024-02-05T07:20:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:38 smithi090 ceph-mon[33213]: 6.a deep-scrub ok 2024-02-05T07:20:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:38 smithi090 ceph-mon[33213]: 6.4 scrub starts 2024-02-05T07:20:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:38 smithi090 ceph-mon[33213]: 6.4 scrub ok 2024-02-05T07:20:38.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:38.804 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:38.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:38.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:38.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:38.929 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:38.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:38.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:39.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:39.055 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:39.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:39.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:39.138 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:38 smithi090 podman[63710]: 2024-02-05 07:20:38.794933643 +0000 UTC m=+9.319318362 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-05T07:20:39.138 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:38 smithi090 podman[63710]: 2024-02-05 07:20:38.892870922 +0000 UTC m=+9.417255633 container create 7d496c9d2a54e69cdbe652d1d2447c2de25a03dd54e1d5a3fd88bde7ced5dd00 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T07:20:39.138 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 podman[63710]: 2024-02-05 07:20:39.137208973 +0000 UTC m=+9.661593690 container init 7d496c9d2a54e69cdbe652d1d2447c2de25a03dd54e1d5a3fd88bde7ced5dd00 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T07:20:39.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:39.180 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:39.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:39.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:39.307 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:39.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:39.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:39.432 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:39.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:39.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:39.513 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 podman[63710]: 2024-02-05 07:20:39.139851858 +0000 UTC m=+9.664236578 container start 7d496c9d2a54e69cdbe652d1d2447c2de25a03dd54e1d5a3fd88bde7ced5dd00 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T07:20:39.513 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.143Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-05T07:20:39.513 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.143Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-05T07:20:39.513 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.143Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-05T07:20:39.513 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-05T07:20:39.513 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-05T07:20:39.513 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-05T07:20:39.513 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-05T07:20:39.514 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=arp 2024-02-05T07:20:39.514 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-05T07:20:39.514 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-05T07:20:39.514 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-05T07:20:39.514 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-05T07:20:39.514 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-05T07:20:39.514 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-05T07:20:39.514 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-05T07:20:39.514 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-05T07:20:39.515 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=edac 2024-02-05T07:20:39.515 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-05T07:20:39.515 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-05T07:20:39.515 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-05T07:20:39.515 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-05T07:20:39.515 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-05T07:20:39.515 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-05T07:20:39.515 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-05T07:20:39.515 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-05T07:20:39.515 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-05T07:20:39.515 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-05T07:20:39.515 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-05T07:20:39.515 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-05T07:20:39.516 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-05T07:20:39.516 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-05T07:20:39.516 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-05T07:20:39.516 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-05T07:20:39.516 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=os 2024-02-05T07:20:39.516 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-05T07:20:39.516 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-05T07:20:39.516 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-05T07:20:39.516 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-05T07:20:39.516 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-05T07:20:39.516 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-05T07:20:39.516 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-05T07:20:39.516 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=stat 2024-02-05T07:20:39.516 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-05T07:20:39.517 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-05T07:20:39.517 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-05T07:20:39.517 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=time 2024-02-05T07:20:39.517 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-05T07:20:39.517 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=uname 2024-02-05T07:20:39.517 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-05T07:20:39.517 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-05T07:20:39.517 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.144Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-05T07:20:39.517 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.145Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-05T07:20:39.517 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b[64085]: ts=2024-02-05T07:20:39.145Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-05T07:20:39.517 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 bash[63710]: 7d496c9d2a54e69cdbe652d1d2447c2de25a03dd54e1d5a3fd88bde7ced5dd00 2024-02-05T07:20:39.517 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:20:39 smithi090 systemd[1]: Started Ceph node-exporter.b for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:20:39.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:39.557 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:39.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:39.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:39.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:39.684 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:39.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:39.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-mon[33213]: 5.15 scrub starts 2024-02-05T07:20:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-mon[33213]: 5.15 scrub ok 2024-02-05T07:20:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-mon[33213]: 3.14 deep-scrub starts 2024-02-05T07:20:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-mon[33213]: 3.14 deep-scrub ok 2024-02-05T07:20:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:20:39.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:39.809 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:39.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:39.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[29098]: 5.15 scrub starts 2024-02-05T07:20:39.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[29098]: 5.15 scrub ok 2024-02-05T07:20:39.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[29098]: 3.14 deep-scrub starts 2024-02-05T07:20:39.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[29098]: 3.14 deep-scrub ok 2024-02-05T07:20:39.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:20:39.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[37158]: 5.15 scrub starts 2024-02-05T07:20:39.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[37158]: 5.15 scrub ok 2024-02-05T07:20:39.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[37158]: 3.14 deep-scrub starts 2024-02-05T07:20:39.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[37158]: 3.14 deep-scrub ok 2024-02-05T07:20:39.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:39.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:39 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:20:39.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:39.934 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:39.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:40.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:40.060 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:40.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:40.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:40.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:40.189 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:40.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:40.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:40.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:20:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:20:40.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:40.314 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:40.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:40.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:40.439 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:40.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:40.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:40.524 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:20:40.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:40.565 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:40.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:40.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:40.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:40.691 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:40.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:40 smithi090 ceph-mon[33213]: 6.8 scrub starts 2024-02-05T07:20:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:40 smithi090 ceph-mon[33213]: 6.8 scrub ok 2024-02-05T07:20:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:40 smithi090 ceph-mon[33213]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-02-05T07:20:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:40 smithi090 ceph-mon[33213]: 2.2 deep-scrub starts 2024-02-05T07:20:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:40 smithi090 ceph-mon[33213]: 2.2 deep-scrub ok 2024-02-05T07:20:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:40 smithi090 ceph-mon[33213]: 3.1a scrub starts 2024-02-05T07:20:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:40 smithi090 ceph-mon[33213]: 3.1a scrub ok 2024-02-05T07:20:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:40 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1280824490' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:20:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:40 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:20:40.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:40.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:40.818 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:40.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:40.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[29098]: 6.8 scrub starts 2024-02-05T07:20:40.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[29098]: 6.8 scrub ok 2024-02-05T07:20:40.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[29098]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-02-05T07:20:40.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[29098]: 2.2 deep-scrub starts 2024-02-05T07:20:40.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[29098]: 2.2 deep-scrub ok 2024-02-05T07:20:40.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[29098]: 3.1a scrub starts 2024-02-05T07:20:40.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[29098]: 3.1a scrub ok 2024-02-05T07:20:40.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1280824490' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:20:40.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:20:40.884 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[37158]: 6.8 scrub starts 2024-02-05T07:20:40.885 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[37158]: 6.8 scrub ok 2024-02-05T07:20:40.885 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[37158]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-02-05T07:20:40.885 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[37158]: 2.2 deep-scrub starts 2024-02-05T07:20:40.885 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[37158]: 2.2 deep-scrub ok 2024-02-05T07:20:40.885 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[37158]: 3.1a scrub starts 2024-02-05T07:20:40.886 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[37158]: 3.1a scrub ok 2024-02-05T07:20:40.886 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1280824490' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:20:40.886 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:40 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:20:40.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:40.943 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:40.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:40.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:41.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:41.069 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:41.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:41.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:41.195 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:41.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:41.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:41.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:41.321 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:41.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:41.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:41.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:41.447 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:41.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:41.549 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:41.573 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:41.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:41.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:41 smithi007 ceph-mon[29098]: 3.1e scrub starts 2024-02-05T07:20:41.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:41 smithi007 ceph-mon[29098]: 3.1e scrub ok 2024-02-05T07:20:41.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:41 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:41.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:41 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:20:41.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:41 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:20:41.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:41 smithi007 ceph-mon[29098]: osdmap e99: 8 total, 8 up, 8 in 2024-02-05T07:20:41.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:41 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/180992809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:20:41.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:41 smithi007 ceph-mon[37158]: 3.1e scrub starts 2024-02-05T07:20:41.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:41 smithi007 ceph-mon[37158]: 3.1e scrub ok 2024-02-05T07:20:41.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:41 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:41.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:41 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:20:41.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:41 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:20:41.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:41 smithi007 ceph-mon[37158]: osdmap e99: 8 total, 8 up, 8 in 2024-02-05T07:20:41.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:41 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/180992809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:20:41.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:41.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:41.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:41.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:41 smithi090 ceph-mon[33213]: 3.1e scrub starts 2024-02-05T07:20:41.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:41 smithi090 ceph-mon[33213]: 3.1e scrub ok 2024-02-05T07:20:41.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:41 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:41.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:41 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:20:41.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:41 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:20:41.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:41 smithi090 ceph-mon[33213]: osdmap e99: 8 total, 8 up, 8 in 2024-02-05T07:20:41.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:41 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/180992809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:20:41.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:41.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:41.826 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:41.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:41.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:41.950 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:41.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:42.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:42.076 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:42.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:42.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:42.201 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:42.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:42.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:42.295 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:20:42.295 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:20:42.296 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:20:42.296 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:20:42.296 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:20:42.296 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:20:42.296 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:20:42.296 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:20:42.296 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:20:42.296 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:20:42.296 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:20:42.296 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:20:42.296 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:20:42.296 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:20:42.297 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:20:42.297 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:20:42.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:42.326 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:42.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:42.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:42.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:42.451 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:42.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:42.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:42.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:42.576 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:42.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:42.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:42.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:42.702 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:42.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:42.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:42 smithi090 ceph-mon[33213]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:20:42.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:42.828 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:42.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:42.916 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:42 smithi007 ceph-mon[29098]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:20:42.916 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:42 smithi007 ceph-mon[37158]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:20:42.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:42.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:42.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:43.057 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:43.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:43.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:43.206 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:43.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:43.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:43.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:43.333 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:43.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:43.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:43.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:43.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:43.586 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:43.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:43.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:43.712 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:43.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:43.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:43.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:20:43.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:43.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:43 smithi007 ceph-mon[29098]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:20:43.825 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:20:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:20:43.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:43 smithi007 ceph-mon[37158]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:20:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:43.850 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:43.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:43.874 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:20:43.874 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:20:43.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:43.974 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:43.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:43 smithi090 ceph-mon[33213]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:20:44.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:44.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:44.100 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:44.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:44.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:44.225 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:44.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:44.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:44.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:44.350 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:44.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:44.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:44.475 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:44.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:44.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:44.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:44.601 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:44.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:44.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:44.725 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:44.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:44.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:44.736 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:20:44.736 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:20:44.736 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 99, 2024-02-05T07:20:44.736 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "e3e7059e-c3f5-11ee-95b6-87774f69a715", 2024-02-05T07:20:44.736 INFO:teuthology.orchestra.run.smithi007.stdout: "created": "2024-02-05T07:13:14.603885+0000", 2024-02-05T07:20:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "modified": "2024-02-05T07:20:40.511575+0000", 2024-02-05T07:20:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "last_up_change": "2024-02-05T07:18:35.434647+0000", 2024-02-05T07:20:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "last_in_change": "2024-02-05T07:18:17.533340+0000", 2024-02-05T07:20:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T07:20:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_num": 5801984, 2024-02-05T07:20:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_set": [ 2024-02-05T07:20:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "noscrub", 2024-02-05T07:20:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "pglog_hardlimit", 2024-02-05T07:20:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "purged_snapdirs", 2024-02-05T07:20:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_deletes", 2024-02-05T07:20:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "sortbitwise" 2024-02-05T07:20:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:20:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_version": 18, 2024-02-05T07:20:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T07:20:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T07:20:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T07:20:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_snapshot": "", 2024-02-05T07:20:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_max": 6, 2024-02-05T07:20:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd": 8, 2024-02-05T07:20:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "require_min_compat_client": "luminous", 2024-02-05T07:20:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "min_compat_client": "luminous", 2024-02-05T07:20:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "require_osd_release": "squid", 2024-02-05T07:20:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_crimson": false, 2024-02-05T07:20:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pools": [ 2024-02-05T07:20:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 1, 2024-02-05T07:20:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".mgr", 2024-02-05T07:20:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:16:32.505567+0000", 2024-02-05T07:20:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:20:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:20:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:20:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:20:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:20:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:20:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:20:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:20:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:20:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:20:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:20:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:20:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 1, 2024-02-05T07:20:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 1, 2024-02-05T07:20:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 1, 2024-02-05T07:20:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 1, 2024-02-05T07:20:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 1, 2024-02-05T07:20:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:20:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:20:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:20:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:20:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:20:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:20:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:20:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "22", 2024-02-05T07:20:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:20:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:20:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:20:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:20:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:20:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:20:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:20:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:20:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:20:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:20:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:20:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:20:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:20:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:20:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:20:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:20:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:20:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:20:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:20:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:20:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:20:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:20:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:20:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:20:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:20:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:20:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:20:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:20:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:20:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:20:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:20:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:20:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:20:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:20:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:20:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:20:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:20:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-02-05T07:20:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_max": 32, 2024-02-05T07:20:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_min": 1 2024-02-05T07:20:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:20:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": {} 2024-02-05T07:20:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:20:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:20:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 7.8899998664855957, 2024-02-05T07:20:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 7.8899998664855957, 2024-02-05T07:20:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.37999999523162842, 2024-02-05T07:20:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 3, 2024-02-05T07:20:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 3, 2024-02-05T07:20:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 1, 2024-02-05T07:20:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 1, 2024-02-05T07:20:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 1 2024-02-05T07:20:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 2, 2024-02-05T07:20:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "datapool", 2024-02-05T07:20:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:47.259462+0000", 2024-02-05T07:20:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 8193, 2024-02-05T07:20:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T07:20:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:20:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:20:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:20:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:20:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:20:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:20:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:20:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:20:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:20:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:20:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 3, 2024-02-05T07:20:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 3, 2024-02-05T07:20:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 3, 2024-02-05T07:20:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 3, 2024-02-05T07:20:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 3, 2024-02-05T07:20:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:20:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:20:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "71", 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 2, 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 71, 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:20:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:20:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:20:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:20:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:20:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:20:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:20:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:20:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:20:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:20:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:20:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:20:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:20:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:20:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:20:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:20:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:20:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:20:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:20:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:20:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:20:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:20:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:20:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:20:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:20:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:20:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:20:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:20:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:20:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:20:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:20:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd": {} 2024-02-05T07:20:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:20:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:20:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.6500000953674316, 2024-02-05T07:20:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.6500000953674316, 2024-02-05T07:20:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T07:20:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.3299999237060547, 2024-02-05T07:20:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.3299999237060547, 2024-02-05T07:20:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 1, 2024-02-05T07:20:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 1, 2024-02-05T07:20:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 1 2024-02-05T07:20:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 3, 2024-02-05T07:20:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".rgw.root", 2024-02-05T07:20:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:47.762336+0000", 2024-02-05T07:20:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:20:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:20:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:20:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:20:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:20:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:20:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:20:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:20:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:20:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:20:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:20:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:20:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:20:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:20:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:20:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:20:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:20:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:20:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:20:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:20:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:20:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:20:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:20:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:20:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "69", 2024-02-05T07:20:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:20:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:20:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:20:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:20:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:20:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:20:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:20:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:20:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:20:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:20:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:20:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:20:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:20:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:20:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:20:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:20:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:20:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:20:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:20:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:20:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:20:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:20:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:20:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:20:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:20:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:20:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:20:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:20:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:20:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.5, 2024-02-05T07:20:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.5, 2024-02-05T07:20:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-02-05T07:20:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.5, 2024-02-05T07:20:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.5, 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 1, 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 1, 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 1 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 4, 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.log", 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:49.736599+0000", 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:20:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:20:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:20:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:20:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:20:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:20:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:20:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:20:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:20:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:20:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:20:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:20:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:20:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:20:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:20:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:20:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:20:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:20:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:20:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:20:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "71", 2024-02-05T07:20:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:20:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:20:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:20:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:20:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:20:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:20:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:20:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:20:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:20:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:20:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:20:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:20:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:20:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:20:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:20:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.25, 2024-02-05T07:20:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.25, 2024-02-05T07:20:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-02-05T07:20:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.25, 2024-02-05T07:20:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.25, 2024-02-05T07:20:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 1, 2024-02-05T07:20:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 1, 2024-02-05T07:20:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 1 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 5, 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.control", 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:51.696948+0000", 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:20:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:20:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:20:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:20:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:20:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:20:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:20:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:20:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:20:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:20:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:20:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:20:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:20:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:20:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:20:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "73", 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:20:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:20:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:20:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:20:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:20:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:20:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:20:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:20:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:20:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:20:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:20:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:20:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:20:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:20:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:20:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.25, 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.25, 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.25, 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.25, 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 1, 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 1, 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 1 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 6, 2024-02-05T07:20:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.meta", 2024-02-05T07:20:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:53.768555+0000", 2024-02-05T07:20:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:20:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:20:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:20:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:20:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:20:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:20:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:20:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:20:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:20:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:20:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:20:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:20:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:20:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:20:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:20:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:20:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:20:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:20:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:20:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:20:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:20:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:20:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:20:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:20:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "75", 2024-02-05T07:20:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:20:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:20:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:20:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:20:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:20:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:20:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:20:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:20:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:20:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:20:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:20:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:20:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:20:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:20:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:20:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:20:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:20:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:20:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:20:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:20:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:20:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:20:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:20:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:20:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:20:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:20:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:20:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:20:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:20:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:20:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:20:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:20:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:20:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:20:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:20:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:20:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:20:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-02-05T07:20:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_bias": 4 2024-02-05T07:20:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:20:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:20:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:20:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:20:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.75, 2024-02-05T07:20:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.75, 2024-02-05T07:20:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-02-05T07:20:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.75, 2024-02-05T07:20:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.75, 2024-02-05T07:20:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 1, 2024-02-05T07:20:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 1, 2024-02-05T07:20:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 1 2024-02-05T07:20:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:20:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:20:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-02-05T07:20:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "703c3948-8003-4015-9dd8-21019cf0eeb8", 2024-02-05T07:20:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:20:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:20:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:20:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:20:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:20:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 8, 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 73, 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:20:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6802", 2024-02-05T07:20:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:20:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:20:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6804", 2024-02-05T07:20:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:20:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:20:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6803", 2024-02-05T07:20:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:20:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6801/2069016689", 2024-02-05T07:20:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6802/2069016689", 2024-02-05T07:20:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6804/2069016689", 2024-02-05T07:20:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6803/2069016689", 2024-02-05T07:20:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:20:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:20:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:20:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-02-05T07:20:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "5c3e7d1e-a61d-44b9-b21e-f91c99a096bc", 2024-02-05T07:20:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:20:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:20:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:20:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:20:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:20:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:20:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 13, 2024-02-05T07:20:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 73, 2024-02-05T07:20:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:20:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:20:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:20:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6805", 2024-02-05T07:20:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:20:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6806", 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6808", 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6807", 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6805/865486532", 2024-02-05T07:20:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6806/865486532", 2024-02-05T07:20:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6808/865486532", 2024-02-05T07:20:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6807/865486532", 2024-02-05T07:20:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:20:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:20:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:20:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-02-05T07:20:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "f4fd930f-7bb2-46ca-9964-98235bd47b3a", 2024-02-05T07:20:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:20:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:20:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:20:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 18, 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 73, 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6809", 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6810", 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6812", 2024-02-05T07:20:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:20:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:20:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6811", 2024-02-05T07:20:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:20:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6809/904871370", 2024-02-05T07:20:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6810/904871370", 2024-02-05T07:20:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6812/904871370", 2024-02-05T07:20:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6811/904871370", 2024-02-05T07:20:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:20:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:20:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:20:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-02-05T07:20:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "149ed0d6-b049-4478-9819-cafc446e94d3", 2024-02-05T07:20:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:20:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:20:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:20:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:20:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:20:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:20:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 26, 2024-02-05T07:20:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 73, 2024-02-05T07:20:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:20:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:20:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:20:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6813", 2024-02-05T07:20:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:20:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:20:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6814", 2024-02-05T07:20:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:20:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:20:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6816", 2024-02-05T07:20:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:20:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:20:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6815", 2024-02-05T07:20:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:20:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6813/2758586242", 2024-02-05T07:20:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6814/2758586242", 2024-02-05T07:20:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6816/2758586242", 2024-02-05T07:20:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6815/2758586242", 2024-02-05T07:20:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:20:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:20:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:20:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-02-05T07:20:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d", 2024-02-05T07:20:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:20:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:20:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:20:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:20:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:20:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:20:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 32, 2024-02-05T07:20:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 80, 2024-02-05T07:20:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:20:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:20:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:20:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6800", 2024-02-05T07:20:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:20:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:20:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6801", 2024-02-05T07:20:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:20:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:20:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6803", 2024-02-05T07:20:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:20:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:20:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6802", 2024-02-05T07:20:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:20:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6800/1154929470", 2024-02-05T07:20:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6801/1154929470", 2024-02-05T07:20:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6803/1154929470", 2024-02-05T07:20:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6802/1154929470", 2024-02-05T07:20:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:20:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:20:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:20:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-02-05T07:20:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "244a7930-c22c-413e-86e1-11e0749f7367", 2024-02-05T07:20:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:20:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:20:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:20:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:20:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:20:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:20:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 41, 2024-02-05T07:20:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 80, 2024-02-05T07:20:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:20:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:20:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:20:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6804", 2024-02-05T07:20:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:20:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:20:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6805", 2024-02-05T07:20:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:20:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:20:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6807", 2024-02-05T07:20:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:20:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:20:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6806", 2024-02-05T07:20:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:20:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6804/398113577", 2024-02-05T07:20:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6805/398113577", 2024-02-05T07:20:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6807/398113577", 2024-02-05T07:20:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6806/398113577", 2024-02-05T07:20:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:20:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:20:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:20:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-02-05T07:20:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7", 2024-02-05T07:20:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:20:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:20:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:20:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:20:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:20:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:20:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 52, 2024-02-05T07:20:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 71, 2024-02-05T07:20:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:20:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:20:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:20:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6808", 2024-02-05T07:20:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2371506925 2024-02-05T07:20:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:20:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6809", 2024-02-05T07:20:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2371506925 2024-02-05T07:20:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:20:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6811", 2024-02-05T07:20:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2371506925 2024-02-05T07:20:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:20:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6810", 2024-02-05T07:20:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2371506925 2024-02-05T07:20:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6808/2371506925", 2024-02-05T07:20:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6809/2371506925", 2024-02-05T07:20:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6811/2371506925", 2024-02-05T07:20:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6810/2371506925", 2024-02-05T07:20:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:20:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:20:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:20:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-02-05T07:20:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9", 2024-02-05T07:20:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:20:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:20:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:20:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:20:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:20:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:20:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 60, 2024-02-05T07:20:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 73, 2024-02-05T07:20:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:20:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:20:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:20:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6812", 2024-02-05T07:20:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:20:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.810 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.810 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:20:44.810 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.810 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.810 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6813", 2024-02-05T07:20:44.811 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_0' created 2024-02-05T07:20:44.811 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:20:44.811 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.811 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.811 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:20:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6815", 2024-02-05T07:20:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:20:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:20:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:20:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:20:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6814", 2024-02-05T07:20:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:20:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6812/2983605980", 2024-02-05T07:20:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6813/2983605980", 2024-02-05T07:20:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2983605980", 2024-02-05T07:20:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6814/2983605980", 2024-02-05T07:20:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:20:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:20:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:20:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:20:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_xinfo": [ 2024-02-05T07:20:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-02-05T07:20:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:20:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:20:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:20:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:20:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:20:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:20:22.989141+0000", 2024-02-05T07:20:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:20:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-02-05T07:20:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:20:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:20:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:20:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:20:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:20:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:20:15.017561+0000", 2024-02-05T07:20:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:20:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-02-05T07:20:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:20:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:20:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:20:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:20:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:20:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:20:12.457285+0000", 2024-02-05T07:20:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:20:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-02-05T07:20:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:20:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:20:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:20:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:20:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:20:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:20:21.324779+0000", 2024-02-05T07:20:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:20:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-02-05T07:20:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:20:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:20:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:20:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:20:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:20:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:19:51.847229+0000", 2024-02-05T07:20:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:20:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-02-05T07:20:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:20:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:20:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:20:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:20:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:20:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:20:29.526367+0000", 2024-02-05T07:20:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:20:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-02-05T07:20:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:20:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:20:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:20:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:20:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:20:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:20:31.130645+0000", 2024-02-05T07:20:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:20:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-02-05T07:20:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:20:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:20:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:20:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:20:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:20:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:19:59.950217+0000", 2024-02-05T07:20:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:20:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:20:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap": [], 2024-02-05T07:20:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_items": [ 2024-02-05T07:20:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.9", 2024-02-05T07:20:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:20:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:20:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:20:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.12", 2024-02-05T07:20:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:20:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:20:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 5 2024-02-05T07:20:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.13", 2024-02-05T07:20:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:20:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:20:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:20:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:20:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:20:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:20:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_primaries": [], 2024-02-05T07:20:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_temp": [], 2024-02-05T07:20:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_temp": [], 2024-02-05T07:20:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklist": { 2024-02-05T07:20:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2180087772": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:20:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2066685086": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:20:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/4100020626": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:20:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1953351182": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:20:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3203189162": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:20:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3418781614": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:20:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1518592084": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:20:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/3380493069": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:20:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2208590106": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:20:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/223579766": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:20:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/2634895574": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:20:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1462812719": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:20:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2823976078": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:20:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1905520582": "2024-02-06T07:13:47.004765+0000" 2024-02-05T07:20:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "range_blocklist": {}, 2024-02-05T07:20:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profiles": { 2024-02-05T07:20:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "default": { 2024-02-05T07:20:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-failure-domain": "osd", 2024-02-05T07:20:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "k": "2", 2024-02-05T07:20:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "m": "1", 2024-02-05T07:20:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin": "jerasure", 2024-02-05T07:20:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "technique": "reed_sol_van" 2024-02-05T07:20:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:20:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps_queue": [], 2024-02-05T07:20:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "new_removed_snaps": [], 2024-02-05T07:20:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "new_purged_snaps": [], 2024-02-05T07:20:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_node_flags": {}, 2024-02-05T07:20:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "device_class_flags": {}, 2024-02-05T07:20:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode": { 2024-02-05T07:20:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_enabled": false, 2024-02-05T07:20:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_bucket_count": 0, 2024-02-05T07:20:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: "degraded_stretch_mode": 0, 2024-02-05T07:20:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: "recovering_stretch_mode": 0, 2024-02-05T07:20:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_bucket": 0 2024-02-05T07:20:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:20:44.827 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:20:44.827 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:44.852 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:44.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:44.947 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:20:44.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:44.978 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:44.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:44 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:44 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:44 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4055904967' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T07:20:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:44 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T07:20:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:44 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:44 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:44 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:20:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:44 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2450287281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:20:45.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:45.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:45.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4055904967' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T07:20:45.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T07:20:45.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:45.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:45.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:20:45.039 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2450287281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:20:45.039 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:45.039 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:45.039 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4055904967' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T07:20:45.039 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T07:20:45.039 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:45.039 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:20:45.039 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:20:45.039 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:44 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2450287281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:20:45.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:45.104 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:45.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:45.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:45.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:45.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T07:20:45.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:45.230 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:45.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:45.332 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:45.355 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:45.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:45.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:45.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:45.482 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:45.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:45.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:45.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:45.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:45.737 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:45.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:45.760 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:45 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-05T07:20:45.760 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:45 smithi007 ceph-mon[29098]: osdmap e100: 8 total, 8 up, 8 in 2024-02-05T07:20:45.760 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:45 smithi007 ceph-mon[29098]: pgmap v56: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:45.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:45.864 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:45.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:45.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:45.994 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:45.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:45 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-05T07:20:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:45 smithi090 ceph-mon[33213]: osdmap e100: 8 total, 8 up, 8 in 2024-02-05T07:20:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:45 smithi090 ceph-mon[33213]: pgmap v56: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:46.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:46.120 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:46.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:46.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:46.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:45 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-05T07:20:46.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:45 smithi007 ceph-mon[37158]: osdmap e100: 8 total, 8 up, 8 in 2024-02-05T07:20:46.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:45 smithi007 ceph-mon[37158]: pgmap v56: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:20:46.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:46.250 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:46.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:46.312 INFO:tasks.thrashosds.thrasher:j is {'epoch': 99, 'fsid': 'e3e7059e-c3f5-11ee-95b6-87774f69a715', 'created': '2024-02-05T07:13:14.603885+0000', 'modified': '2024-02-05T07:20:40.511575+0000', 'last_up_change': '2024-02-05T07:18:35.434647+0000', 'last_in_change': '2024-02-05T07:18:17.533340+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T07:16:32.505567+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T07:18:47.259462+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T07:18:47.762336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T07:18:49.736599+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T07:18:51.696948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T07:18:53.768555+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '703c3948-8003-4015-9dd8-21019cf0eeb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6801', 'nonce': 2069016689}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6802', 'nonce': 2069016689}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6804', 'nonce': 2069016689}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6803', 'nonce': 2069016689}]}, 'public_addr': '172.21.15.7:6801/2069016689', 'cluster_addr': '172.21.15.7:6802/2069016689', 'heartbeat_back_addr': '172.21.15.7:6804/2069016689', 'heartbeat_front_addr': '172.21.15.7:6803/2069016689', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '5c3e7d1e-a61d-44b9-b21e-f91c99a096bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6805', 'nonce': 865486532}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6806', 'nonce': 865486532}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6808', 'nonce': 865486532}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6807', 'nonce': 865486532}]}, 'public_addr': '172.21.15.7:6805/865486532', 'cluster_addr': '172.21.15.7:6806/865486532', 'heartbeat_back_addr': '172.21.15.7:6808/865486532', 'heartbeat_front_addr': '172.21.15.7:6807/865486532', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4fd930f-7bb2-46ca-9964-98235bd47b3a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6809', 'nonce': 904871370}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6810', 'nonce': 904871370}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6812', 'nonce': 904871370}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6811', 'nonce': 904871370}]}, 'public_addr': '172.21.15.7:6809/904871370', 'cluster_addr': '172.21.15.7:6810/904871370', 'heartbeat_back_addr': '172.21.15.7:6812/904871370', 'heartbeat_front_addr': '172.21.15.7:6811/904871370', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '149ed0d6-b049-4478-9819-cafc446e94d3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6813', 'nonce': 2758586242}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6814', 'nonce': 2758586242}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6816', 'nonce': 2758586242}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6815', 'nonce': 2758586242}]}, 'public_addr': '172.21.15.7:6813/2758586242', 'cluster_addr': '172.21.15.7:6814/2758586242', 'heartbeat_back_addr': '172.21.15.7:6816/2758586242', 'heartbeat_front_addr': '172.21.15.7:6815/2758586242', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 80, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 1154929470}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6801', 'nonce': 1154929470}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6803', 'nonce': 1154929470}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 1154929470}]}, 'public_addr': '172.21.15.90:6800/1154929470', 'cluster_addr': '172.21.15.90:6801/1154929470', 'heartbeat_back_addr': '172.21.15.90:6803/1154929470', 'heartbeat_front_addr': '172.21.15.90:6802/1154929470', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '244a7930-c22c-413e-86e1-11e0749f7367', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 80, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 398113577}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6805', 'nonce': 398113577}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6807', 'nonce': 398113577}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 398113577}]}, 'public_addr': '172.21.15.90:6804/398113577', 'cluster_addr': '172.21.15.90:6805/398113577', 'heartbeat_back_addr': '172.21.15.90:6807/398113577', 'heartbeat_front_addr': '172.21.15.90:6806/398113577', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0adbb562-3012-4f60-b5ec-02550a10b2c7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 71, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 2371506925}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 2371506925}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6811', 'nonce': 2371506925}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6810', 'nonce': 2371506925}]}, 'public_addr': '172.21.15.90:6808/2371506925', 'cluster_addr': '172.21.15.90:6809/2371506925', 'heartbeat_back_addr': '172.21.15.90:6811/2371506925', 'heartbeat_front_addr': '172.21.15.90:6810/2371506925', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dbc26c36-a0e9-4f21-9e98-739270cea6d9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 2983605980}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 2983605980}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6815', 'nonce': 2983605980}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6814', 'nonce': 2983605980}]}, 'public_addr': '172.21.15.90:6812/2983605980', 'cluster_addr': '172.21.15.90:6813/2983605980', 'heartbeat_back_addr': '172.21.15.90:6815/2983605980', 'heartbeat_front_addr': '172.21.15.90:6814/2983605980', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:20:22.989141+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:20:15.017561+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:20:12.457285+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:20:21.324779+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:19:51.847229+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:20:29.526367+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:20:31.130645+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:19:59.950217+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.7:0/2180087772': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/2066685086': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/4100020626': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/1953351182': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/3203189162': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/3418781614': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:6800/1518592084': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:6800/3380493069': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/2208590106': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:0/223579766': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:6800/2634895574': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/1462812719': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/2823976078': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/1905520582': '2024-02-06T07:13:47.004765+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T07:20:46.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:20:46.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:46.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:46.381 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:46.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:46.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:46.508 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:46.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:46.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:46.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:46.635 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:46.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:46.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:46.760 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:46.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:46.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:46.862 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:46.890 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:46.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:46.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:46 smithi007 ceph-mon[29098]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:20:46.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:46 smithi007 ceph-mon[29098]: osdmap e101: 8 total, 8 up, 8 in 2024-02-05T07:20:46.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:46 smithi007 ceph-mon[37158]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:20:46.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:46 smithi007 ceph-mon[37158]: osdmap e101: 8 total, 8 up, 8 in 2024-02-05T07:20:46.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:47.019 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:47.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:47.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:46 smithi090 ceph-mon[33213]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:20:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:46 smithi090 ceph-mon[33213]: osdmap e101: 8 total, 8 up, 8 in 2024-02-05T07:20:47.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:47.145 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:47.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:47.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:47.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:47.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:47.378 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:47.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-02-05T07:20:47.385 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:47.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:47.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:47.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:47.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:47.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:47.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:47.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:47.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:47.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:47.679 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:47.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:47.792 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:47.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:47.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:47.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:47.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:47.919 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:47.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:47.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:47 smithi007 ceph-mon[29098]: osdmap e102: 8 total, 8 up, 8 in 2024-02-05T07:20:47.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:47 smithi007 ceph-mon[29098]: pgmap v59: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:20:47.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:47 smithi007 ceph-mon[37158]: osdmap e102: 8 total, 8 up, 8 in 2024-02-05T07:20:47.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:47 smithi007 ceph-mon[37158]: pgmap v59: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:20:48.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:47 smithi090 ceph-mon[33213]: osdmap e102: 8 total, 8 up, 8 in 2024-02-05T07:20:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:47 smithi090 ceph-mon[33213]: pgmap v59: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:20:48.043 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:48.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:48.079 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:48.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:48.173 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:48.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:48.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:48.302 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:48.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:48.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:48.428 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:48.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:48.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:48.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:48.557 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:48.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:48.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:48.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:48.683 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:48.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:48.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:48.739 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:48.739 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:48.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:48.813 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:48.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:48.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:48.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:48.942 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:48.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:48.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:49.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:49.069 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:49.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:49.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:49.199 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:49.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:49.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:49.325 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:49.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:49.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:49.451 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:49.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:49.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:49.553 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:49.581 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:49.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:49.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:49.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:49.712 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:49.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:49.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:49.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:49.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:49.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:49.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:50.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:50.093 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:50.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:50.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:50.195 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:50.223 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:50.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:50.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:50 smithi090 ceph-mon[33213]: pgmap v60: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:20:50.275 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:20:49 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:20:50.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:50 smithi007 ceph-mon[29098]: pgmap v60: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:20:50.304 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:50 smithi007 ceph-mon[37158]: pgmap v60: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:20:50.325 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:50.350 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:50.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:50.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:50.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:50.469 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:20:50.472 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:20:50.480 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:50.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:50.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:50.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:50.605 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:50.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:50.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:50.735 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:50.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:50.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:50.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:50.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:50.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:50.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:50.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:50.990 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:50.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:51.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:51.120 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:51.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:51.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:51.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:51.246 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:51.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:51.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:51.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:51.372 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:51.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:51.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:51 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:51.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:51 smithi007 ceph-mon[29098]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:51.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:51 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:51.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:51 smithi007 ceph-mon[37158]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:51.474 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:51.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:51.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:51 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:20:51.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:51 smithi090 ceph-mon[33213]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:20:51.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:51.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:51.624 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:51.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:51.725 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:51.753 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:51.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:51.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:51.855 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:51.883 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:51.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:51.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:51.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:52.012 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:52.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:52.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:52.141 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:52.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:52.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:52.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:52 smithi007 ceph-mon[29098]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-05T07:20:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:52 smithi007 ceph-mon[37158]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-05T07:20:52.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:52.272 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:52.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:52.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:52.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:52.403 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:52.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:52.505 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:52.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:52 smithi090 ceph-mon[33213]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-05T07:20:52.529 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:52.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:52.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:52.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:52.654 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:52.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:52.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:52.779 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:52.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:52.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:52.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:52.908 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:52.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:53.009 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:53.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:53.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:53.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:53.158 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:53.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:53.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:53.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:53.288 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:53.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:53.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:53.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:53.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:53.544 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-02-05T07:20:48.927976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1042,"num_read_kb":888,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":623,"ondisk_log_size":623,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":399,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":239868,"kb_used_data":6876,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749721348,"statfs":{"total":767960285184,"available":767714660352,"internally_reserved":0,"allocated":7041024,"data_stored":4200988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001330"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761527+0000","last_change":"2024-02-05T07:20:25.913055+0000","last_active":"2024-02-05T07:20:44.761527+0000","last_peered":"2024-02-05T07:20:44.761527+0000","last_clean":"2024-02-05T07:20:44.761527+0000","last_became_active":"2024-02-05T07:18:55.466001+0000","last_became_peered":"2024-02-05T07:18:55.466001+0000","last_unstale":"2024-02-05T07:20:44.761527+0000","last_undegraded":"2024-02-05T07:20:44.761527+0000","last_fullsized":"2024-02-05T07:20:44.761527+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:25.912938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:25.912938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:53.255881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756308+0000","last_change":"2024-02-05T07:20:31.382191+0000","last_active":"2024-02-05T07:20:44.756308+0000","last_peered":"2024-02-05T07:20:44.756308+0000","last_clean":"2024-02-05T07:20:44.756308+0000","last_became_active":"2024-02-05T07:18:53.460820+0000","last_became_peered":"2024-02-05T07:18:53.460820+0000","last_unstale":"2024-02-05T07:20:44.756308+0000","last_undegraded":"2024-02-05T07:20:44.756308+0000","last_fullsized":"2024-02-05T07:20:44.756308+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:31.382106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:31.382106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:54.397321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756717+0000","last_change":"2024-02-05T07:20:35.243411+0000","last_active":"2024-02-05T07:20:44.756717+0000","last_peered":"2024-02-05T07:20:44.756717+0000","last_clean":"2024-02-05T07:20:44.756717+0000","last_became_active":"2024-02-05T07:18:49.434456+0000","last_became_peered":"2024-02-05T07:18:49.434456+0000","last_unstale":"2024-02-05T07:20:44.756717+0000","last_undegraded":"2024-02-05T07:20:44.756717+0000","last_fullsized":"2024-02-05T07:20:44.756717+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:35.243281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:35.243281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:44.816513+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":94,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761269+0000","last_change":"2024-02-05T07:20:23.852115+0000","last_active":"2024-02-05T07:20:44.761269+0000","last_peered":"2024-02-05T07:20:44.761269+0000","last_clean":"2024-02-05T07:20:44.761269+0000","last_became_active":"2024-02-05T07:18:51.448924+0000","last_became_peered":"2024-02-05T07:18:51.448924+0000","last_unstale":"2024-02-05T07:20:44.761269+0000","last_undegraded":"2024-02-05T07:20:44.761269+0000","last_fullsized":"2024-02-05T07:20:44.761269+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:20:23.851993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:23.851993+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:48.656007+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.764053+0000","last_change":"2024-02-05T07:18:55.506169+0000","last_active":"2024-02-05T07:20:44.764053+0000","last_peered":"2024-02-05T07:20:44.764053+0000","last_clean":"2024-02-05T07:20:44.764053+0000","last_became_active":"2024-02-05T07:18:55.505718+0000","last_became_peered":"2024-02-05T07:18:55.505718+0000","last_unstale":"2024-02-05T07:20:44.764053+0000","last_undegraded":"2024-02-05T07:20:44.764053+0000","last_fullsized":"2024-02-05T07:20:44.764053+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.995669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761361+0000","last_change":"2024-02-05T07:19:54.746304+0000","last_active":"2024-02-05T07:20:44.761361+0000","last_peered":"2024-02-05T07:20:44.761361+0000","last_clean":"2024-02-05T07:20:44.761361+0000","last_became_active":"2024-02-05T07:18:53.461506+0000","last_became_peered":"2024-02-05T07:18:53.461506+0000","last_unstale":"2024-02-05T07:20:44.761361+0000","last_undegraded":"2024-02-05T07:20:44.761361+0000","last_fullsized":"2024-02-05T07:20:44.761361+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:54.746206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:19:54.746206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:57.230561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763949+0000","last_change":"2024-02-05T07:20:39.926066+0000","last_active":"2024-02-05T07:20:44.763949+0000","last_peered":"2024-02-05T07:20:44.763949+0000","last_clean":"2024-02-05T07:20:44.763949+0000","last_became_active":"2024-02-05T07:18:49.436490+0000","last_became_peered":"2024-02-05T07:18:49.436490+0000","last_unstale":"2024-02-05T07:20:44.763949+0000","last_undegraded":"2024-02-05T07:20:44.763949+0000","last_fullsized":"2024-02-05T07:20:44.763949+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:39.925994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:39.925994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:57.408803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":95,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763860+0000","last_change":"2024-02-05T07:18:51.445160+0000","last_active":"2024-02-05T07:20:44.763860+0000","last_peered":"2024-02-05T07:20:44.763860+0000","last_clean":"2024-02-05T07:20:44.763860+0000","last_became_active":"2024-02-05T07:18:51.444783+0000","last_became_peered":"2024-02-05T07:18:51.444783+0000","last_unstale":"2024-02-05T07:20:44.763860+0000","last_undegraded":"2024-02-05T07:20:44.763860+0000","last_fullsized":"2024-02-05T07:20:44.763860+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.995438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755400+0000","last_change":"2024-02-05T07:20:31.252764+0000","last_active":"2024-02-05T07:20:44.755400+0000","last_peered":"2024-02-05T07:20:44.755400+0000","last_clean":"2024-02-05T07:20:44.755400+0000","last_became_active":"2024-02-05T07:18:55.474002+0000","last_became_peered":"2024-02-05T07:18:55.474002+0000","last_unstale":"2024-02-05T07:20:44.755400+0000","last_undegraded":"2024-02-05T07:20:44.755400+0000","last_fullsized":"2024-02-05T07:20:44.755400+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:31.252686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:31.252686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:37.020142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":66,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761076+0000","last_change":"2024-02-05T07:18:53.464788+0000","last_active":"2024-02-05T07:20:44.761076+0000","last_peered":"2024-02-05T07:20:44.761076+0000","last_clean":"2024-02-05T07:20:44.761076+0000","last_became_active":"2024-02-05T07:18:53.464520+0000","last_became_peered":"2024-02-05T07:18:53.464520+0000","last_unstale":"2024-02-05T07:20:44.761076+0000","last_undegraded":"2024-02-05T07:20:44.761076+0000","last_fullsized":"2024-02-05T07:20:44.761076+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:45.980339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761144+0000","last_change":"2024-02-05T07:19:59.782950+0000","last_active":"2024-02-05T07:20:44.761144+0000","last_peered":"2024-02-05T07:20:44.761144+0000","last_clean":"2024-02-05T07:20:44.761144+0000","last_became_active":"2024-02-05T07:18:49.435454+0000","last_became_peered":"2024-02-05T07:18:49.435454+0000","last_unstale":"2024-02-05T07:20:44.761144+0000","last_undegraded":"2024-02-05T07:20:44.761144+0000","last_fullsized":"2024-02-05T07:20:44.761144+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:59.782841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:59.782841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.410088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"89'13","reported_seq":98,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761788+0000","last_change":"2024-02-05T07:20:02.798288+0000","last_active":"2024-02-05T07:20:44.761788+0000","last_peered":"2024-02-05T07:20:44.761788+0000","last_clean":"2024-02-05T07:20:44.761788+0000","last_became_active":"2024-02-05T07:18:51.444592+0000","last_became_peered":"2024-02-05T07:18:51.444592+0000","last_unstale":"2024-02-05T07:20:44.761788+0000","last_undegraded":"2024-02-05T07:20:44.761788+0000","last_fullsized":"2024-02-05T07:20:44.761788+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.798180+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:11.151835+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.758943+0000","last_change":"2024-02-05T07:18:55.473667+0000","last_active":"2024-02-05T07:20:44.758943+0000","last_peered":"2024-02-05T07:20:44.758943+0000","last_clean":"2024-02-05T07:20:44.758943+0000","last_became_active":"2024-02-05T07:18:55.473368+0000","last_became_peered":"2024-02-05T07:18:55.473368+0000","last_unstale":"2024-02-05T07:20:44.758943+0000","last_undegraded":"2024-02-05T07:20:44.758943+0000","last_fullsized":"2024-02-05T07:20:44.758943+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:45.980283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755755+0000","last_change":"2024-02-05T07:20:23.756522+0000","last_active":"2024-02-05T07:20:44.755755+0000","last_peered":"2024-02-05T07:20:44.755755+0000","last_clean":"2024-02-05T07:20:44.755755+0000","last_became_active":"2024-02-05T07:18:53.456881+0000","last_became_peered":"2024-02-05T07:18:53.456881+0000","last_unstale":"2024-02-05T07:20:44.755755+0000","last_undegraded":"2024-02-05T07:20:44.755755+0000","last_fullsized":"2024-02-05T07:20:44.755755+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:23.756416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:23.756416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.140659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761166+0000","last_change":"2024-02-05T07:20:13.048248+0000","last_active":"2024-02-05T07:20:44.761166+0000","last_peered":"2024-02-05T07:20:44.761166+0000","last_clean":"2024-02-05T07:20:44.761166+0000","last_became_active":"2024-02-05T07:18:49.431220+0000","last_became_peered":"2024-02-05T07:18:49.431220+0000","last_unstale":"2024-02-05T07:20:44.761166+0000","last_undegraded":"2024-02-05T07:20:44.761166+0000","last_fullsized":"2024-02-05T07:20:44.761166+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:13.048126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:13.048126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:28.009618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761792+0000","last_change":"2024-02-05T07:18:51.448820+0000","last_active":"2024-02-05T07:20:44.761792+0000","last_peered":"2024-02-05T07:20:44.761792+0000","last_clean":"2024-02-05T07:20:44.761792+0000","last_became_active":"2024-02-05T07:18:51.448462+0000","last_became_peered":"2024-02-05T07:18:51.448462+0000","last_unstale":"2024-02-05T07:20:44.761792+0000","last_undegraded":"2024-02-05T07:20:44.761792+0000","last_fullsized":"2024-02-05T07:20:44.761792+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.925758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.762378+0000","last_change":"2024-02-05T07:20:00.760332+0000","last_active":"2024-02-05T07:20:44.762378+0000","last_peered":"2024-02-05T07:20:44.762378+0000","last_clean":"2024-02-05T07:20:44.762378+0000","last_became_active":"2024-02-05T07:18:55.505678+0000","last_became_peered":"2024-02-05T07:18:55.505678+0000","last_unstale":"2024-02-05T07:20:44.762378+0000","last_undegraded":"2024-02-05T07:20:44.762378+0000","last_fullsized":"2024-02-05T07:20:44.762378+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:00.760230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.760230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:19.923028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756731+0000","last_change":"2024-02-05T07:20:18.552686+0000","last_active":"2024-02-05T07:20:44.756731+0000","last_peered":"2024-02-05T07:20:44.756731+0000","last_clean":"2024-02-05T07:20:44.756731+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:20:44.756731+0000","last_undegraded":"2024-02-05T07:20:44.756731+0000","last_fullsized":"2024-02-05T07:20:44.756731+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:18.552608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.552608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:43.361846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756582+0000","last_change":"2024-02-05T07:18:49.432729+0000","last_active":"2024-02-05T07:20:44.756582+0000","last_peered":"2024-02-05T07:20:44.756582+0000","last_clean":"2024-02-05T07:20:44.756582+0000","last_became_active":"2024-02-05T07:18:49.432449+0000","last_became_peered":"2024-02-05T07:18:49.432449+0000","last_unstale":"2024-02-05T07:20:44.756582+0000","last_undegraded":"2024-02-05T07:20:44.756582+0000","last_fullsized":"2024-02-05T07:20:44.756582+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:47.259702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":103,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.762079+0000","last_change":"2024-02-05T07:19:59.624459+0000","last_active":"2024-02-05T07:20:44.762079+0000","last_peered":"2024-02-05T07:20:44.762079+0000","last_clean":"2024-02-05T07:20:44.762079+0000","last_became_active":"2024-02-05T07:18:51.447728+0000","last_became_peered":"2024-02-05T07:18:51.447728+0000","last_unstale":"2024-02-05T07:20:44.762079+0000","last_undegraded":"2024-02-05T07:20:44.762079+0000","last_fullsized":"2024-02-05T07:20:44.762079+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:19:59.624369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:59.624369+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:28.126457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.764527+0000","last_change":"2024-02-05T07:18:55.506354+0000","last_active":"2024-02-05T07:20:44.764527+0000","last_peered":"2024-02-05T07:20:44.764527+0000","last_clean":"2024-02-05T07:20:44.764527+0000","last_became_active":"2024-02-05T07:18:55.505375+0000","last_became_peered":"2024-02-05T07:18:55.505375+0000","last_unstale":"2024-02-05T07:20:44.764527+0000","last_undegraded":"2024-02-05T07:20:44.764527+0000","last_fullsized":"2024-02-05T07:20:44.764527+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.995465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.762102+0000","last_change":"2024-02-05T07:20:18.795431+0000","last_active":"2024-02-05T07:20:44.762102+0000","last_peered":"2024-02-05T07:20:44.762102+0000","last_clean":"2024-02-05T07:20:44.762102+0000","last_became_active":"2024-02-05T07:18:53.459077+0000","last_became_peered":"2024-02-05T07:18:53.459077+0000","last_unstale":"2024-02-05T07:20:44.762102+0000","last_undegraded":"2024-02-05T07:20:44.762102+0000","last_fullsized":"2024-02-05T07:20:44.762102+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:18.795341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.795341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:22.388371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761896+0000","last_change":"2024-02-05T07:20:39.898505+0000","last_active":"2024-02-05T07:20:44.761896+0000","last_peered":"2024-02-05T07:20:44.761896+0000","last_clean":"2024-02-05T07:20:44.761896+0000","last_became_active":"2024-02-05T07:18:49.427837+0000","last_became_peered":"2024-02-05T07:18:49.427837+0000","last_unstale":"2024-02-05T07:20:44.761896+0000","last_undegraded":"2024-02-05T07:20:44.761896+0000","last_fullsized":"2024-02-05T07:20:44.761896+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:39.898367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:39.898367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:00.742703+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":101,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763947+0000","last_change":"2024-02-05T07:20:01.841591+0000","last_active":"2024-02-05T07:20:44.763947+0000","last_peered":"2024-02-05T07:20:44.763947+0000","last_clean":"2024-02-05T07:20:44.763947+0000","last_became_active":"2024-02-05T07:18:51.447761+0000","last_became_peered":"2024-02-05T07:18:51.447761+0000","last_unstale":"2024-02-05T07:20:44.763947+0000","last_undegraded":"2024-02-05T07:20:44.763947+0000","last_fullsized":"2024-02-05T07:20:44.763947+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:01.841489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:01.841489+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:19.277134+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756105+0000","last_change":"2024-02-05T07:20:17.755329+0000","last_active":"2024-02-05T07:20:44.756105+0000","last_peered":"2024-02-05T07:20:44.756105+0000","last_clean":"2024-02-05T07:20:44.756105+0000","last_became_active":"2024-02-05T07:18:55.476071+0000","last_became_peered":"2024-02-05T07:18:55.476071+0000","last_unstale":"2024-02-05T07:20:44.756105+0000","last_undegraded":"2024-02-05T07:20:44.756105+0000","last_fullsized":"2024-02-05T07:20:44.756105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:20:17.755246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:17.755246+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:22.075178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":109,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.760511+0000","last_change":"2024-02-05T07:20:22.424747+0000","last_active":"2024-02-05T07:20:44.760511+0000","last_peered":"2024-02-05T07:20:44.760511+0000","last_clean":"2024-02-05T07:20:44.760511+0000","last_became_active":"2024-02-05T07:18:53.461011+0000","last_became_peered":"2024-02-05T07:18:53.461011+0000","last_unstale":"2024-02-05T07:20:44.760511+0000","last_undegraded":"2024-02-05T07:20:44.760511+0000","last_fullsized":"2024-02-05T07:20:44.760511+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:20:22.424647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:22.424647+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:22.759683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756392+0000","last_change":"2024-02-05T07:20:25.493793+0000","last_active":"2024-02-05T07:20:44.756392+0000","last_peered":"2024-02-05T07:20:44.756392+0000","last_clean":"2024-02-05T07:20:44.756392+0000","last_became_active":"2024-02-05T07:18:49.431031+0000","last_became_peered":"2024-02-05T07:18:49.431031+0000","last_unstale":"2024-02-05T07:20:44.756392+0000","last_undegraded":"2024-02-05T07:20:44.756392+0000","last_fullsized":"2024-02-05T07:20:44.756392+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:20:25.493710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:25.493710+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:39.620525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":93,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756271+0000","last_change":"2024-02-05T07:20:22.291474+0000","last_active":"2024-02-05T07:20:44.756271+0000","last_peered":"2024-02-05T07:20:44.756271+0000","last_clean":"2024-02-05T07:20:44.756271+0000","last_became_active":"2024-02-05T07:18:51.509918+0000","last_became_peered":"2024-02-05T07:18:51.509918+0000","last_unstale":"2024-02-05T07:20:44.756271+0000","last_undegraded":"2024-02-05T07:20:44.756271+0000","last_fullsized":"2024-02-05T07:20:44.756271+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:20:22.291371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:22.291371+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:50.152194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756875+0000","last_change":"2024-02-05T07:20:03.444274+0000","last_active":"2024-02-05T07:20:44.756875+0000","last_peered":"2024-02-05T07:20:44.756875+0000","last_clean":"2024-02-05T07:20:44.756875+0000","last_became_active":"2024-02-05T07:18:55.466164+0000","last_became_peered":"2024-02-05T07:18:55.466164+0000","last_unstale":"2024-02-05T07:20:44.756875+0000","last_undegraded":"2024-02-05T07:20:44.756875+0000","last_fullsized":"2024-02-05T07:20:44.756875+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:03.444153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:03.444153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:32.621342+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":66,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756444+0000","last_change":"2024-02-05T07:18:53.457928+0000","last_active":"2024-02-05T07:20:44.756444+0000","last_peered":"2024-02-05T07:20:44.756444+0000","last_clean":"2024-02-05T07:20:44.756444+0000","last_became_active":"2024-02-05T07:18:53.457587+0000","last_became_peered":"2024-02-05T07:18:53.457587+0000","last_unstale":"2024-02-05T07:20:44.756444+0000","last_undegraded":"2024-02-05T07:20:44.756444+0000","last_fullsized":"2024-02-05T07:20:44.756444+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:47.259669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763321+0000","last_change":"2024-02-05T07:19:49.788355+0000","last_active":"2024-02-05T07:20:44.763321+0000","last_peered":"2024-02-05T07:20:44.763321+0000","last_clean":"2024-02-05T07:20:44.763321+0000","last_became_active":"2024-02-05T07:18:49.430988+0000","last_became_peered":"2024-02-05T07:18:49.430988+0000","last_unstale":"2024-02-05T07:20:44.763321+0000","last_undegraded":"2024-02-05T07:20:44.763321+0000","last_fullsized":"2024-02-05T07:20:44.763321+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:49.788275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:49.788275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:50.100908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":97,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.760389+0000","last_change":"2024-02-05T07:20:06.366811+0000","last_active":"2024-02-05T07:20:44.760389+0000","last_peered":"2024-02-05T07:20:44.760389+0000","last_clean":"2024-02-05T07:20:44.760389+0000","last_became_active":"2024-02-05T07:18:51.446604+0000","last_became_peered":"2024-02-05T07:18:51.446604+0000","last_unstale":"2024-02-05T07:20:44.760389+0000","last_undegraded":"2024-02-05T07:20:44.760389+0000","last_fullsized":"2024-02-05T07:20:44.760389+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:20:06.366699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.366699+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.671090+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755754+0000","last_change":"2024-02-05T07:20:01.693975+0000","last_active":"2024-02-05T07:20:44.755754+0000","last_peered":"2024-02-05T07:20:44.755754+0000","last_clean":"2024-02-05T07:20:44.755754+0000","last_became_active":"2024-02-05T07:18:55.468086+0000","last_became_peered":"2024-02-05T07:18:55.468086+0000","last_unstale":"2024-02-05T07:20:44.755754+0000","last_undegraded":"2024-02-05T07:20:44.755754+0000","last_fullsized":"2024-02-05T07:20:44.755754+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:01.693871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:01.693871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:12.004988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.759371+0000","last_change":"2024-02-05T07:20:09.349490+0000","last_active":"2024-02-05T07:20:44.759371+0000","last_peered":"2024-02-05T07:20:44.759371+0000","last_clean":"2024-02-05T07:20:44.759371+0000","last_became_active":"2024-02-05T07:18:53.460307+0000","last_became_peered":"2024-02-05T07:18:53.460307+0000","last_unstale":"2024-02-05T07:20:44.759371+0000","last_undegraded":"2024-02-05T07:20:44.759371+0000","last_fullsized":"2024-02-05T07:20:44.759371+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.349342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.451852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756676+0000","last_change":"2024-02-05T07:20:19.326489+0000","last_active":"2024-02-05T07:20:44.756676+0000","last_peered":"2024-02-05T07:20:44.756676+0000","last_clean":"2024-02-05T07:20:44.756676+0000","last_became_active":"2024-02-05T07:18:49.434408+0000","last_became_peered":"2024-02-05T07:18:49.434408+0000","last_unstale":"2024-02-05T07:20:44.756676+0000","last_undegraded":"2024-02-05T07:20:44.756676+0000","last_fullsized":"2024-02-05T07:20:44.756676+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:19.326400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:19.326400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:45.594433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":76,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763387+0000","last_change":"2024-02-05T07:18:51.449489+0000","last_active":"2024-02-05T07:20:44.763387+0000","last_peered":"2024-02-05T07:20:44.763387+0000","last_clean":"2024-02-05T07:20:44.763387+0000","last_became_active":"2024-02-05T07:18:51.449083+0000","last_became_peered":"2024-02-05T07:18:51.449083+0000","last_unstale":"2024-02-05T07:20:44.763387+0000","last_undegraded":"2024-02-05T07:20:44.763387+0000","last_fullsized":"2024-02-05T07:20:44.763387+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.995377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.764269+0000","last_change":"2024-02-05T07:18:55.507549+0000","last_active":"2024-02-05T07:20:44.764269+0000","last_peered":"2024-02-05T07:20:44.764269+0000","last_clean":"2024-02-05T07:20:44.764269+0000","last_became_active":"2024-02-05T07:18:55.506968+0000","last_became_peered":"2024-02-05T07:18:55.506968+0000","last_unstale":"2024-02-05T07:20:44.764269+0000","last_undegraded":"2024-02-05T07:20:44.764269+0000","last_fullsized":"2024-02-05T07:20:44.764269+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.995404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755995+0000","last_change":"2024-02-05T07:20:05.674115+0000","last_active":"2024-02-05T07:20:44.755995+0000","last_peered":"2024-02-05T07:20:44.755995+0000","last_clean":"2024-02-05T07:20:44.755995+0000","last_became_active":"2024-02-05T07:18:53.460977+0000","last_became_peered":"2024-02-05T07:18:53.460977+0000","last_unstale":"2024-02-05T07:20:44.755995+0000","last_undegraded":"2024-02-05T07:20:44.755995+0000","last_fullsized":"2024-02-05T07:20:44.755995+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_clean_scrub_stamp":"2024-02-05T07:20:05.673983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:06.714441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761469+0000","last_change":"2024-02-05T07:19:48.944128+0000","last_active":"2024-02-05T07:20:44.761469+0000","last_peered":"2024-02-05T07:20:44.761469+0000","last_clean":"2024-02-05T07:20:44.761469+0000","last_became_active":"2024-02-05T07:18:49.434560+0000","last_became_peered":"2024-02-05T07:18:49.434560+0000","last_unstale":"2024-02-05T07:20:44.761469+0000","last_undegraded":"2024-02-05T07:20:44.761469+0000","last_fullsized":"2024-02-05T07:20:44.761469+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:48.944028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:48.944028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:56.156526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":89,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763768+0000","last_change":"2024-02-05T07:18:51.510217+0000","last_active":"2024-02-05T07:20:44.763768+0000","last_peered":"2024-02-05T07:20:44.763768+0000","last_clean":"2024-02-05T07:20:44.763768+0000","last_became_active":"2024-02-05T07:18:51.509911+0000","last_became_peered":"2024-02-05T07:18:51.509911+0000","last_unstale":"2024-02-05T07:20:44.763768+0000","last_undegraded":"2024-02-05T07:20:44.763768+0000","last_fullsized":"2024-02-05T07:20:44.763768+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.995340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755062+0000","last_change":"2024-02-05T07:20:27.243266+0000","last_active":"2024-02-05T07:20:44.755062+0000","last_peered":"2024-02-05T07:20:44.755062+0000","last_clean":"2024-02-05T07:20:44.755062+0000","last_became_active":"2024-02-05T07:18:55.477312+0000","last_became_peered":"2024-02-05T07:18:55.477312+0000","last_unstale":"2024-02-05T07:20:44.755062+0000","last_undegraded":"2024-02-05T07:20:44.755062+0000","last_fullsized":"2024-02-05T07:20:44.755062+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:27.243136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:27.243136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.798135+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":66,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763965+0000","last_change":"2024-02-05T07:18:53.468306+0000","last_active":"2024-02-05T07:20:44.763965+0000","last_peered":"2024-02-05T07:20:44.763965+0000","last_clean":"2024-02-05T07:20:44.763965+0000","last_became_active":"2024-02-05T07:18:53.467986+0000","last_became_peered":"2024-02-05T07:18:53.467986+0000","last_unstale":"2024-02-05T07:20:44.763965+0000","last_undegraded":"2024-02-05T07:20:44.763965+0000","last_fullsized":"2024-02-05T07:20:44.763965+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.995723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756126+0000","last_change":"2024-02-05T07:18:49.438840+0000","last_active":"2024-02-05T07:20:44.756126+0000","last_peered":"2024-02-05T07:20:44.756126+0000","last_clean":"2024-02-05T07:20:44.756126+0000","last_became_active":"2024-02-05T07:18:49.438565+0000","last_became_peered":"2024-02-05T07:18:49.438565+0000","last_unstale":"2024-02-05T07:20:44.756126+0000","last_undegraded":"2024-02-05T07:20:44.756126+0000","last_fullsized":"2024-02-05T07:20:44.756126+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:47.671239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":58,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.758833+0000","last_change":"2024-02-05T07:19:16.128305+0000","last_active":"2024-02-05T07:20:44.758833+0000","last_peered":"2024-02-05T07:20:44.758833+0000","last_clean":"2024-02-05T07:20:44.758833+0000","last_became_active":"2024-02-05T07:19:16.122059+0000","last_became_peered":"2024-02-05T07:19:16.122059+0000","last_unstale":"2024-02-05T07:20:44.758833+0000","last_undegraded":"2024-02-05T07:20:44.758833+0000","last_fullsized":"2024-02-05T07:20:44.758833+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:45.980306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763455+0000","last_change":"2024-02-05T07:20:27.853088+0000","last_active":"2024-02-05T07:20:44.763455+0000","last_peered":"2024-02-05T07:20:44.763455+0000","last_clean":"2024-02-05T07:20:44.763455+0000","last_became_active":"2024-02-05T07:18:55.478087+0000","last_became_peered":"2024-02-05T07:18:55.478087+0000","last_unstale":"2024-02-05T07:20:44.763455+0000","last_undegraded":"2024-02-05T07:20:44.763455+0000","last_fullsized":"2024-02-05T07:20:44.763455+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:27.852942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:27.852942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:43.443908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.758087+0000","last_change":"2024-02-05T07:20:21.504837+0000","last_active":"2024-02-05T07:20:44.758087+0000","last_peered":"2024-02-05T07:20:44.758087+0000","last_clean":"2024-02-05T07:20:44.758087+0000","last_became_active":"2024-02-05T07:18:53.463620+0000","last_became_peered":"2024-02-05T07:18:53.463620+0000","last_unstale":"2024-02-05T07:20:44.758087+0000","last_undegraded":"2024-02-05T07:20:44.758087+0000","last_fullsized":"2024-02-05T07:20:44.758087+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:21.504764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:21.504764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:43.888422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761791+0000","last_change":"2024-02-05T07:20:38.840775+0000","last_active":"2024-02-05T07:20:44.761791+0000","last_peered":"2024-02-05T07:20:44.761791+0000","last_clean":"2024-02-05T07:20:44.761791+0000","last_became_active":"2024-02-05T07:18:49.430766+0000","last_became_peered":"2024-02-05T07:18:49.430766+0000","last_unstale":"2024-02-05T07:20:44.761791+0000","last_undegraded":"2024-02-05T07:20:44.761791+0000","last_fullsized":"2024-02-05T07:20:44.761791+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:38.840701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:38.840701+0000","last_clean_scrub_stamp":"2024-02-05T07:20:38.840701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:50.837410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":115,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.762201+0000","last_change":"2024-02-05T07:20:06.870132+0000","last_active":"2024-02-05T07:20:44.762201+0000","last_peered":"2024-02-05T07:20:44.762201+0000","last_clean":"2024-02-05T07:20:44.762201+0000","last_became_active":"2024-02-05T07:19:16.121696+0000","last_became_peered":"2024-02-05T07:19:16.121696+0000","last_unstale":"2024-02-05T07:20:44.762201+0000","last_undegraded":"2024-02-05T07:20:44.762201+0000","last_fullsized":"2024-02-05T07:20:44.762201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:20:06.870006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.870006+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:12.414942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756059+0000","last_change":"2024-02-05T07:20:20.322398+0000","last_active":"2024-02-05T07:20:44.756059+0000","last_peered":"2024-02-05T07:20:44.756059+0000","last_clean":"2024-02-05T07:20:44.756059+0000","last_became_active":"2024-02-05T07:18:55.477896+0000","last_became_peered":"2024-02-05T07:18:55.477896+0000","last_unstale":"2024-02-05T07:20:44.756059+0000","last_undegraded":"2024-02-05T07:20:44.756059+0000","last_fullsized":"2024-02-05T07:20:44.756059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:20.322309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:20.322309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:20.952572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":106,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761241+0000","last_change":"2024-02-05T07:20:38.003311+0000","last_active":"2024-02-05T07:20:44.761241+0000","last_peered":"2024-02-05T07:20:44.761241+0000","last_clean":"2024-02-05T07:20:44.761241+0000","last_became_active":"2024-02-05T07:18:53.465287+0000","last_became_peered":"2024-02-05T07:18:53.465287+0000","last_unstale":"2024-02-05T07:20:44.761241+0000","last_undegraded":"2024-02-05T07:20:44.761241+0000","last_fullsized":"2024-02-05T07:20:44.761241+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:20:38.003219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:38.003219+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:03.547570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755964+0000","last_change":"2024-02-05T07:20:32.839729+0000","last_active":"2024-02-05T07:20:44.755964+0000","last_peered":"2024-02-05T07:20:44.755964+0000","last_clean":"2024-02-05T07:20:44.755964+0000","last_became_active":"2024-02-05T07:18:49.436131+0000","last_became_peered":"2024-02-05T07:18:49.436131+0000","last_unstale":"2024-02-05T07:20:44.755964+0000","last_undegraded":"2024-02-05T07:20:44.755964+0000","last_fullsized":"2024-02-05T07:20:44.755964+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:32.839614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:32.839614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:02.241396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":85,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763992+0000","last_change":"2024-02-05T07:18:51.507793+0000","last_active":"2024-02-05T07:20:44.763992+0000","last_peered":"2024-02-05T07:20:44.763992+0000","last_clean":"2024-02-05T07:20:44.763992+0000","last_became_active":"2024-02-05T07:18:51.507491+0000","last_became_peered":"2024-02-05T07:18:51.507491+0000","last_unstale":"2024-02-05T07:20:44.763992+0000","last_undegraded":"2024-02-05T07:20:44.763992+0000","last_fullsized":"2024-02-05T07:20:44.763992+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.995495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.762575+0000","last_change":"2024-02-05T07:20:33.860836+0000","last_active":"2024-02-05T07:20:44.762575+0000","last_peered":"2024-02-05T07:20:44.762575+0000","last_clean":"2024-02-05T07:20:44.762575+0000","last_became_active":"2024-02-05T07:18:55.467044+0000","last_became_peered":"2024-02-05T07:18:55.467044+0000","last_unstale":"2024-02-05T07:20:44.762575+0000","last_undegraded":"2024-02-05T07:20:44.762575+0000","last_fullsized":"2024-02-05T07:20:44.762575+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:33.860750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:33.860750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:01.300669+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":98,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763825+0000","last_change":"2024-02-05T07:18:53.465551+0000","last_active":"2024-02-05T07:20:44.763825+0000","last_peered":"2024-02-05T07:20:44.763825+0000","last_clean":"2024-02-05T07:20:44.763825+0000","last_became_active":"2024-02-05T07:18:53.465270+0000","last_became_peered":"2024-02-05T07:18:53.465270+0000","last_unstale":"2024-02-05T07:20:44.763825+0000","last_undegraded":"2024-02-05T07:20:44.763825+0000","last_fullsized":"2024-02-05T07:20:44.763825+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.995581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755825+0000","last_change":"2024-02-05T07:20:33.270501+0000","last_active":"2024-02-05T07:20:44.755825+0000","last_peered":"2024-02-05T07:20:44.755825+0000","last_clean":"2024-02-05T07:20:44.755825+0000","last_became_active":"2024-02-05T07:18:49.434402+0000","last_became_peered":"2024-02-05T07:18:49.434402+0000","last_unstale":"2024-02-05T07:20:44.755825+0000","last_undegraded":"2024-02-05T07:20:44.755825+0000","last_fullsized":"2024-02-05T07:20:44.755825+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:33.270421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:33.270421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:50.495185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":94,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761417+0000","last_change":"2024-02-05T07:20:06.852726+0000","last_active":"2024-02-05T07:20:44.761417+0000","last_peered":"2024-02-05T07:20:44.761417+0000","last_clean":"2024-02-05T07:20:44.761417+0000","last_became_active":"2024-02-05T07:18:51.508551+0000","last_became_peered":"2024-02-05T07:18:51.508551+0000","last_unstale":"2024-02-05T07:20:44.761417+0000","last_undegraded":"2024-02-05T07:20:44.761417+0000","last_fullsized":"2024-02-05T07:20:44.761417+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:20:06.852562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.852562+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:20.949048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.762288+0000","last_change":"2024-02-05T07:20:11.543111+0000","last_active":"2024-02-05T07:20:44.762288+0000","last_peered":"2024-02-05T07:20:44.762288+0000","last_clean":"2024-02-05T07:20:44.762288+0000","last_became_active":"2024-02-05T07:18:55.469044+0000","last_became_peered":"2024-02-05T07:18:55.469044+0000","last_unstale":"2024-02-05T07:20:44.762288+0000","last_undegraded":"2024-02-05T07:20:44.762288+0000","last_fullsized":"2024-02-05T07:20:44.762288+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:11.543036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:11.543036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:38.671608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.764443+0000","last_change":"2024-02-05T07:20:02.813432+0000","last_active":"2024-02-05T07:20:44.764443+0000","last_peered":"2024-02-05T07:20:44.764443+0000","last_clean":"2024-02-05T07:20:44.764443+0000","last_became_active":"2024-02-05T07:18:53.463607+0000","last_became_peered":"2024-02-05T07:18:53.463607+0000","last_unstale":"2024-02-05T07:20:44.764443+0000","last_undegraded":"2024-02-05T07:20:44.764443+0000","last_fullsized":"2024-02-05T07:20:44.764443+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.813344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.813344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:30.005634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755908+0000","last_change":"2024-02-05T07:20:02.673707+0000","last_active":"2024-02-05T07:20:44.755908+0000","last_peered":"2024-02-05T07:20:44.755908+0000","last_clean":"2024-02-05T07:20:44.755908+0000","last_became_active":"2024-02-05T07:18:49.437732+0000","last_became_peered":"2024-02-05T07:18:49.437732+0000","last_unstale":"2024-02-05T07:20:44.755908+0000","last_undegraded":"2024-02-05T07:20:44.755908+0000","last_fullsized":"2024-02-05T07:20:44.755908+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.673592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.665799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":86,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756339+0000","last_change":"2024-02-05T07:18:51.510254+0000","last_active":"2024-02-05T07:20:44.756339+0000","last_peered":"2024-02-05T07:20:44.756339+0000","last_clean":"2024-02-05T07:20:44.756339+0000","last_became_active":"2024-02-05T07:18:51.509917+0000","last_became_peered":"2024-02-05T07:18:51.509917+0000","last_unstale":"2024-02-05T07:20:44.756339+0000","last_undegraded":"2024-02-05T07:20:44.756339+0000","last_fullsized":"2024-02-05T07:20:44.756339+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:47.259719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756038+0000","last_change":"2024-02-05T07:20:19.732465+0000","last_active":"2024-02-05T07:20:44.756038+0000","last_peered":"2024-02-05T07:20:44.756038+0000","last_clean":"2024-02-05T07:20:44.756038+0000","last_became_active":"2024-02-05T07:18:55.507555+0000","last_became_peered":"2024-02-05T07:18:55.507555+0000","last_unstale":"2024-02-05T07:20:44.756038+0000","last_undegraded":"2024-02-05T07:20:44.756038+0000","last_fullsized":"2024-02-05T07:20:44.756038+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:19.732337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:19.732337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:28.044159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":66,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761475+0000","last_change":"2024-02-05T07:18:53.465851+0000","last_active":"2024-02-05T07:20:44.761475+0000","last_peered":"2024-02-05T07:20:44.761475+0000","last_clean":"2024-02-05T07:20:44.761475+0000","last_became_active":"2024-02-05T07:18:53.465591+0000","last_became_peered":"2024-02-05T07:18:53.465591+0000","last_unstale":"2024-02-05T07:20:44.761475+0000","last_undegraded":"2024-02-05T07:20:44.761475+0000","last_fullsized":"2024-02-05T07:20:44.761475+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:45.980322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":88,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.759382+0000","last_change":"2024-02-05T07:19:55.369939+0000","last_active":"2024-02-05T07:20:44.759382+0000","last_peered":"2024-02-05T07:20:44.759382+0000","last_clean":"2024-02-05T07:20:44.759382+0000","last_became_active":"2024-02-05T07:18:49.433121+0000","last_became_peered":"2024-02-05T07:18:49.433121+0000","last_unstale":"2024-02-05T07:20:44.759382+0000","last_undegraded":"2024-02-05T07:20:44.759382+0000","last_fullsized":"2024-02-05T07:20:44.759382+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:19:55.369800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:55.369800+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:04.984273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":79,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.764620+0000","last_change":"2024-02-05T07:18:51.508489+0000","last_active":"2024-02-05T07:20:44.764620+0000","last_peered":"2024-02-05T07:20:44.764620+0000","last_clean":"2024-02-05T07:20:44.764620+0000","last_became_active":"2024-02-05T07:18:51.507451+0000","last_became_peered":"2024-02-05T07:18:51.507451+0000","last_unstale":"2024-02-05T07:20:44.764620+0000","last_undegraded":"2024-02-05T07:20:44.764620+0000","last_fullsized":"2024-02-05T07:20:44.764620+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.995642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.757256+0000","last_change":"2024-02-05T07:20:44.757256+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.757256+0000","last_undegraded":"2024-02-05T07:20:44.757256+0000","last_fullsized":"2024-02-05T07:20:44.757256+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.760604+0000","last_change":"2024-02-05T07:20:37.032149+0000","last_active":"2024-02-05T07:20:44.760604+0000","last_peered":"2024-02-05T07:20:44.760604+0000","last_clean":"2024-02-05T07:20:44.760604+0000","last_became_active":"2024-02-05T07:18:55.505401+0000","last_became_peered":"2024-02-05T07:18:55.505401+0000","last_unstale":"2024-02-05T07:20:44.760604+0000","last_undegraded":"2024-02-05T07:20:44.760604+0000","last_fullsized":"2024-02-05T07:20:44.760604+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:37.032056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:37.032056+0000","last_clean_scrub_stamp":"2024-02-05T07:20:37.032056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:43.632882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":94,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756462+0000","last_change":"2024-02-05T07:20:13.701938+0000","last_active":"2024-02-05T07:20:44.756462+0000","last_peered":"2024-02-05T07:20:44.756462+0000","last_clean":"2024-02-05T07:20:44.756462+0000","last_became_active":"2024-02-05T07:18:49.437663+0000","last_became_peered":"2024-02-05T07:18:49.437663+0000","last_unstale":"2024-02-05T07:20:44.756462+0000","last_undegraded":"2024-02-05T07:20:44.756462+0000","last_fullsized":"2024-02-05T07:20:44.756462+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:20:13.701852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:13.701852+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:32.496072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":103,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.758898+0000","last_change":"2024-02-05T07:19:58.753612+0000","last_active":"2024-02-05T07:20:44.758898+0000","last_peered":"2024-02-05T07:20:44.758898+0000","last_clean":"2024-02-05T07:20:44.758898+0000","last_became_active":"2024-02-05T07:18:51.508097+0000","last_became_peered":"2024-02-05T07:18:51.508097+0000","last_unstale":"2024-02-05T07:20:44.758898+0000","last_undegraded":"2024-02-05T07:20:44.758898+0000","last_fullsized":"2024-02-05T07:20:44.758898+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:19:58.753472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:58.753472+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:26.081872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":107,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756190+0000","last_change":"2024-02-05T07:20:00.685371+0000","last_active":"2024-02-05T07:20:44.756190+0000","last_peered":"2024-02-05T07:20:44.756190+0000","last_clean":"2024-02-05T07:20:44.756190+0000","last_became_active":"2024-02-05T07:18:53.456287+0000","last_became_peered":"2024-02-05T07:18:53.456287+0000","last_unstale":"2024-02-05T07:20:44.756190+0000","last_undegraded":"2024-02-05T07:20:44.756190+0000","last_fullsized":"2024-02-05T07:20:44.756190+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.685282+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:11.368545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.762376+0000","last_change":"2024-02-05T07:20:44.762376+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.762376+0000","last_undegraded":"2024-02-05T07:20:44.762376+0000","last_fullsized":"2024-02-05T07:20:44.762376+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.764139+0000","last_change":"2024-02-05T07:20:28.882434+0000","last_active":"2024-02-05T07:20:44.764139+0000","last_peered":"2024-02-05T07:20:44.764139+0000","last_clean":"2024-02-05T07:20:44.764139+0000","last_became_active":"2024-02-05T07:18:55.474663+0000","last_became_peered":"2024-02-05T07:18:55.474663+0000","last_unstale":"2024-02-05T07:20:44.764139+0000","last_undegraded":"2024-02-05T07:20:44.764139+0000","last_fullsized":"2024-02-05T07:20:44.764139+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:28.882326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:28.882326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:37.233227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756319+0000","last_change":"2024-02-05T07:19:51.758261+0000","last_active":"2024-02-05T07:20:44.756319+0000","last_peered":"2024-02-05T07:20:44.756319+0000","last_clean":"2024-02-05T07:20:44.756319+0000","last_became_active":"2024-02-05T07:18:49.435621+0000","last_became_peered":"2024-02-05T07:18:49.435621+0000","last_unstale":"2024-02-05T07:20:44.756319+0000","last_undegraded":"2024-02-05T07:20:44.756319+0000","last_fullsized":"2024-02-05T07:20:44.756319+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:51.758163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.758163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:15.549152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":119,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.762097+0000","last_change":"2024-02-05T07:20:04.885890+0000","last_active":"2024-02-05T07:20:44.762097+0000","last_peered":"2024-02-05T07:20:44.762097+0000","last_clean":"2024-02-05T07:20:44.762097+0000","last_became_active":"2024-02-05T07:19:16.123650+0000","last_became_peered":"2024-02-05T07:19:16.123650+0000","last_unstale":"2024-02-05T07:20:44.762097+0000","last_undegraded":"2024-02-05T07:20:44.762097+0000","last_fullsized":"2024-02-05T07:20:44.762097+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:04.885778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:04.885778+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:07.176995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761525+0000","last_change":"2024-02-05T07:20:02.694603+0000","last_active":"2024-02-05T07:20:44.761525+0000","last_peered":"2024-02-05T07:20:44.761525+0000","last_clean":"2024-02-05T07:20:44.761525+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:20:44.761525+0000","last_undegraded":"2024-02-05T07:20:44.761525+0000","last_fullsized":"2024-02-05T07:20:44.761525+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.694506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.694506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.700377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.758742+0000","last_change":"2024-02-05T07:20:44.758742+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.758742+0000","last_undegraded":"2024-02-05T07:20:44.758742+0000","last_fullsized":"2024-02-05T07:20:44.758742+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755627+0000","last_change":"2024-02-05T07:20:38.715538+0000","last_active":"2024-02-05T07:20:44.755627+0000","last_peered":"2024-02-05T07:20:44.755627+0000","last_clean":"2024-02-05T07:20:44.755627+0000","last_became_active":"2024-02-05T07:18:55.473032+0000","last_became_peered":"2024-02-05T07:18:55.473032+0000","last_unstale":"2024-02-05T07:20:44.755627+0000","last_undegraded":"2024-02-05T07:20:44.755627+0000","last_fullsized":"2024-02-05T07:20:44.755627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:38.715447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:38.715447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:58.258241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761855+0000","last_change":"2024-02-05T07:20:16.476649+0000","last_active":"2024-02-05T07:20:44.761855+0000","last_peered":"2024-02-05T07:20:44.761855+0000","last_clean":"2024-02-05T07:20:44.761855+0000","last_became_active":"2024-02-05T07:18:53.464867+0000","last_became_peered":"2024-02-05T07:18:53.464867+0000","last_unstale":"2024-02-05T07:20:44.761855+0000","last_undegraded":"2024-02-05T07:20:44.761855+0000","last_fullsized":"2024-02-05T07:20:44.761855+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:16.476554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:16.476554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:31.284238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756319+0000","last_change":"2024-02-05T07:19:55.788134+0000","last_active":"2024-02-05T07:20:44.756319+0000","last_peered":"2024-02-05T07:20:44.756319+0000","last_clean":"2024-02-05T07:20:44.756319+0000","last_became_active":"2024-02-05T07:18:49.434451+0000","last_became_peered":"2024-02-05T07:18:49.434451+0000","last_unstale":"2024-02-05T07:20:44.756319+0000","last_undegraded":"2024-02-05T07:20:44.756319+0000","last_fullsized":"2024-02-05T07:20:44.756319+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:55.788005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:55.788005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.126482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":109,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.759737+0000","last_change":"2024-02-05T07:20:00.384918+0000","last_active":"2024-02-05T07:20:44.759737+0000","last_peered":"2024-02-05T07:20:44.759737+0000","last_clean":"2024-02-05T07:20:44.759737+0000","last_became_active":"2024-02-05T07:18:51.507672+0000","last_became_peered":"2024-02-05T07:18:51.507672+0000","last_unstale":"2024-02-05T07:20:44.759737+0000","last_undegraded":"2024-02-05T07:20:44.759737+0000","last_fullsized":"2024-02-05T07:20:44.759737+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:20:00.384813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.384813+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:12.230867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.760444+0000","last_change":"2024-02-05T07:20:44.760444+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.760444+0000","last_undegraded":"2024-02-05T07:20:44.760444+0000","last_fullsized":"2024-02-05T07:20:44.760444+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755326+0000","last_change":"2024-02-05T07:18:55.476043+0000","last_active":"2024-02-05T07:20:44.755326+0000","last_peered":"2024-02-05T07:20:44.755326+0000","last_clean":"2024-02-05T07:20:44.755326+0000","last_became_active":"2024-02-05T07:18:55.475702+0000","last_became_peered":"2024-02-05T07:18:55.475702+0000","last_unstale":"2024-02-05T07:20:44.755326+0000","last_undegraded":"2024-02-05T07:20:44.755326+0000","last_fullsized":"2024-02-05T07:20:44.755326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:47.259768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761514+0000","last_change":"2024-02-05T07:20:09.589044+0000","last_active":"2024-02-05T07:20:44.761514+0000","last_peered":"2024-02-05T07:20:44.761514+0000","last_clean":"2024-02-05T07:20:44.761514+0000","last_became_active":"2024-02-05T07:18:53.459560+0000","last_became_peered":"2024-02-05T07:18:53.459560+0000","last_unstale":"2024-02-05T07:20:44.761514+0000","last_undegraded":"2024-02-05T07:20:44.761514+0000","last_fullsized":"2024-02-05T07:20:44.761514+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:09.588893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.588893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:27.848596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.760988+0000","last_change":"2024-02-05T07:20:15.066894+0000","last_active":"2024-02-05T07:20:44.760988+0000","last_peered":"2024-02-05T07:20:44.760988+0000","last_clean":"2024-02-05T07:20:44.760988+0000","last_became_active":"2024-02-05T07:18:49.434238+0000","last_became_peered":"2024-02-05T07:18:49.434238+0000","last_unstale":"2024-02-05T07:20:44.760988+0000","last_undegraded":"2024-02-05T07:20:44.760988+0000","last_fullsized":"2024-02-05T07:20:44.760988+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:15.066768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:15.066768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:41.133131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":86,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756498+0000","last_change":"2024-02-05T07:18:51.441099+0000","last_active":"2024-02-05T07:20:44.756498+0000","last_peered":"2024-02-05T07:20:44.756498+0000","last_clean":"2024-02-05T07:20:44.756498+0000","last_became_active":"2024-02-05T07:18:51.440744+0000","last_became_peered":"2024-02-05T07:18:51.440744+0000","last_unstale":"2024-02-05T07:20:44.756498+0000","last_undegraded":"2024-02-05T07:20:44.756498+0000","last_fullsized":"2024-02-05T07:20:44.756498+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:47.259651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.758907+0000","last_change":"2024-02-05T07:20:44.758907+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.758907+0000","last_undegraded":"2024-02-05T07:20:44.758907+0000","last_fullsized":"2024-02-05T07:20:44.758907+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.764158+0000","last_change":"2024-02-05T07:18:55.476663+0000","last_active":"2024-02-05T07:20:44.764158+0000","last_peered":"2024-02-05T07:20:44.764158+0000","last_clean":"2024-02-05T07:20:44.764158+0000","last_became_active":"2024-02-05T07:18:55.476313+0000","last_became_peered":"2024-02-05T07:18:55.476313+0000","last_unstale":"2024-02-05T07:20:44.764158+0000","last_undegraded":"2024-02-05T07:20:44.764158+0000","last_fullsized":"2024-02-05T07:20:44.764158+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.995696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":94,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.760336+0000","last_change":"2024-02-05T07:20:38.993766+0000","last_active":"2024-02-05T07:20:44.760336+0000","last_peered":"2024-02-05T07:20:44.760336+0000","last_clean":"2024-02-05T07:20:44.760336+0000","last_became_active":"2024-02-05T07:18:48.423032+0000","last_became_peered":"2024-02-05T07:18:48.423032+0000","last_unstale":"2024-02-05T07:20:44.760336+0000","last_undegraded":"2024-02-05T07:20:44.760336+0000","last_fullsized":"2024-02-05T07:20:44.760336+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:20:38.993688+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:20:38.993688+0000","last_clean_scrub_stamp":"2024-02-05T07:20:38.993688+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:53.334904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761885+0000","last_change":"2024-02-05T07:20:32.870565+0000","last_active":"2024-02-05T07:20:44.761885+0000","last_peered":"2024-02-05T07:20:44.761885+0000","last_clean":"2024-02-05T07:20:44.761885+0000","last_became_active":"2024-02-05T07:18:49.430208+0000","last_became_peered":"2024-02-05T07:18:49.430208+0000","last_unstale":"2024-02-05T07:20:44.761885+0000","last_undegraded":"2024-02-05T07:20:44.761885+0000","last_fullsized":"2024-02-05T07:20:44.761885+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:32.870450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:32.870450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:47.599215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":151,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.757523+0000","last_change":"2024-02-05T07:20:24.521320+0000","last_active":"2024-02-05T07:20:44.757523+0000","last_peered":"2024-02-05T07:20:44.757523+0000","last_clean":"2024-02-05T07:20:44.757523+0000","last_became_active":"2024-02-05T07:18:51.442110+0000","last_became_peered":"2024-02-05T07:18:51.442110+0000","last_unstale":"2024-02-05T07:20:44.757523+0000","last_undegraded":"2024-02-05T07:20:44.757523+0000","last_fullsized":"2024-02-05T07:20:44.757523+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-02-05T07:20:24.521222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:24.521222+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:36.232296+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":66,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755983+0000","last_change":"2024-02-05T07:18:53.461030+0000","last_active":"2024-02-05T07:20:44.755983+0000","last_peered":"2024-02-05T07:20:44.755983+0000","last_clean":"2024-02-05T07:20:44.755983+0000","last_became_active":"2024-02-05T07:18:53.460715+0000","last_became_peered":"2024-02-05T07:18:53.460715+0000","last_unstale":"2024-02-05T07:20:44.755983+0000","last_undegraded":"2024-02-05T07:20:44.755983+0000","last_fullsized":"2024-02-05T07:20:44.755983+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:47.259735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.758946+0000","last_change":"2024-02-05T07:20:44.758946+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.758946+0000","last_undegraded":"2024-02-05T07:20:44.758946+0000","last_fullsized":"2024-02-05T07:20:44.758946+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756418+0000","last_change":"2024-02-05T07:20:06.673182+0000","last_active":"2024-02-05T07:20:44.756418+0000","last_peered":"2024-02-05T07:20:44.756418+0000","last_clean":"2024-02-05T07:20:44.756418+0000","last_became_active":"2024-02-05T07:18:55.506803+0000","last_became_peered":"2024-02-05T07:18:55.506803+0000","last_unstale":"2024-02-05T07:20:44.756418+0000","last_undegraded":"2024-02-05T07:20:44.756418+0000","last_fullsized":"2024-02-05T07:20:44.756418+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.673081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:21.893489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":89,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755861+0000","last_change":"2024-02-05T07:20:09.406817+0000","last_active":"2024-02-05T07:20:44.755861+0000","last_peered":"2024-02-05T07:20:44.755861+0000","last_clean":"2024-02-05T07:20:44.755861+0000","last_became_active":"2024-02-05T07:18:49.429651+0000","last_became_peered":"2024-02-05T07:18:49.429651+0000","last_unstale":"2024-02-05T07:20:44.755861+0000","last_undegraded":"2024-02-05T07:20:44.755861+0000","last_fullsized":"2024-02-05T07:20:44.755861+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:20:09.406718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.406718+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:33.441463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":91,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761201+0000","last_change":"2024-02-05T07:19:56.661601+0000","last_active":"2024-02-05T07:20:44.761201+0000","last_peered":"2024-02-05T07:20:44.761201+0000","last_clean":"2024-02-05T07:20:44.761201+0000","last_became_active":"2024-02-05T07:18:48.423639+0000","last_became_peered":"2024-02-05T07:18:48.423639+0000","last_unstale":"2024-02-05T07:20:44.761201+0000","last_undegraded":"2024-02-05T07:20:44.761201+0000","last_fullsized":"2024-02-05T07:20:44.761201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:56.661490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:19:56.661490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.585463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"96'18","reported_seq":111,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.757513+0000","last_change":"2024-02-05T07:20:28.409634+0000","last_active":"2024-02-05T07:20:44.757513+0000","last_peered":"2024-02-05T07:20:44.757513+0000","last_clean":"2024-02-05T07:20:44.757513+0000","last_became_active":"2024-02-05T07:18:51.436097+0000","last_became_peered":"2024-02-05T07:18:51.436097+0000","last_unstale":"2024-02-05T07:20:44.757513+0000","last_undegraded":"2024-02-05T07:20:44.757513+0000","last_fullsized":"2024-02-05T07:20:44.757513+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'18","last_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:20:28.409472+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:53.525418+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761933+0000","last_change":"2024-02-05T07:20:10.558042+0000","last_active":"2024-02-05T07:20:44.761933+0000","last_peered":"2024-02-05T07:20:44.761933+0000","last_clean":"2024-02-05T07:20:44.761933+0000","last_became_active":"2024-02-05T07:18:53.460771+0000","last_became_peered":"2024-02-05T07:18:53.460771+0000","last_unstale":"2024-02-05T07:20:44.761933+0000","last_undegraded":"2024-02-05T07:20:44.761933+0000","last_fullsized":"2024-02-05T07:20:44.761933+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:10.557948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:10.557948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:27.157314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.759040+0000","last_change":"2024-02-05T07:20:44.759040+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.759040+0000","last_undegraded":"2024-02-05T07:20:44.759040+0000","last_fullsized":"2024-02-05T07:20:44.759040+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761408+0000","last_change":"2024-02-05T07:20:35.085265+0000","last_active":"2024-02-05T07:20:44.761408+0000","last_peered":"2024-02-05T07:20:44.761408+0000","last_clean":"2024-02-05T07:20:44.761408+0000","last_became_active":"2024-02-05T07:18:55.473358+0000","last_became_peered":"2024-02-05T07:18:55.473358+0000","last_unstale":"2024-02-05T07:20:44.761408+0000","last_undegraded":"2024-02-05T07:20:44.761408+0000","last_fullsized":"2024-02-05T07:20:44.761408+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:35.085136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:35.085136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:43.102492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":340,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755768+0000","last_change":"2024-02-05T07:20:31.816346+0000","last_active":"2024-02-05T07:20:44.755768+0000","last_peered":"2024-02-05T07:20:44.755768+0000","last_clean":"2024-02-05T07:20:44.755768+0000","last_became_active":"2024-02-05T07:18:39.444898+0000","last_became_peered":"2024-02-05T07:18:39.444898+0000","last_unstale":"2024-02-05T07:20:44.755768+0000","last_undegraded":"2024-02-05T07:20:44.755768+0000","last_fullsized":"2024-02-05T07:20:44.755768+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:20:31.816160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:20:31.816160+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:40.393220+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763604+0000","last_change":"2024-02-05T07:20:29.851392+0000","last_active":"2024-02-05T07:20:44.763604+0000","last_peered":"2024-02-05T07:20:44.763604+0000","last_clean":"2024-02-05T07:20:44.763604+0000","last_became_active":"2024-02-05T07:18:49.439249+0000","last_became_peered":"2024-02-05T07:18:49.439249+0000","last_unstale":"2024-02-05T07:20:44.763604+0000","last_undegraded":"2024-02-05T07:20:44.763604+0000","last_fullsized":"2024-02-05T07:20:44.763604+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:29.851303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:29.851303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:33.322985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":124,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.760293+0000","last_change":"2024-02-05T07:20:25.394750+0000","last_active":"2024-02-05T07:20:44.760293+0000","last_peered":"2024-02-05T07:20:44.760293+0000","last_clean":"2024-02-05T07:20:44.760293+0000","last_became_active":"2024-02-05T07:18:51.507714+0000","last_became_peered":"2024-02-05T07:18:51.507714+0000","last_unstale":"2024-02-05T07:20:44.760293+0000","last_undegraded":"2024-02-05T07:20:44.760293+0000","last_fullsized":"2024-02-05T07:20:44.760293+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-02-05T07:20:25.394654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:25.394654+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:52.134692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":66,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756024+0000","last_change":"2024-02-05T07:18:53.460344+0000","last_active":"2024-02-05T07:20:44.756024+0000","last_peered":"2024-02-05T07:20:44.756024+0000","last_clean":"2024-02-05T07:20:44.756024+0000","last_became_active":"2024-02-05T07:18:53.459971+0000","last_became_peered":"2024-02-05T07:18:53.459971+0000","last_unstale":"2024-02-05T07:20:44.756024+0000","last_undegraded":"2024-02-05T07:20:44.756024+0000","last_fullsized":"2024-02-05T07:20:44.756024+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:47.671201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.755382+0000","last_change":"2024-02-05T07:20:44.755382+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.755382+0000","last_undegraded":"2024-02-05T07:20:44.755382+0000","last_fullsized":"2024-02-05T07:20:44.755382+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":72,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.757166+0000","last_change":"2024-02-05T07:20:37.499849+0000","last_active":"2024-02-05T07:20:44.757166+0000","last_peered":"2024-02-05T07:20:44.757166+0000","last_clean":"2024-02-05T07:20:44.757166+0000","last_became_active":"2024-02-05T07:18:55.473509+0000","last_became_peered":"2024-02-05T07:18:55.473509+0000","last_unstale":"2024-02-05T07:20:44.757166+0000","last_undegraded":"2024-02-05T07:20:44.757166+0000","last_fullsized":"2024-02-05T07:20:44.757166+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:20:37.499768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:37.499768+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:59.238955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755609+0000","last_change":"2024-02-05T07:20:21.318287+0000","last_active":"2024-02-05T07:20:44.755609+0000","last_peered":"2024-02-05T07:20:44.755609+0000","last_clean":"2024-02-05T07:20:44.755609+0000","last_became_active":"2024-02-05T07:18:49.434400+0000","last_became_peered":"2024-02-05T07:18:49.434400+0000","last_unstale":"2024-02-05T07:20:44.755609+0000","last_undegraded":"2024-02-05T07:20:44.755609+0000","last_fullsized":"2024-02-05T07:20:44.755609+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:21.318169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:21.318169+0000","last_clean_scrub_stamp":"2024-02-05T07:20:21.318169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.238151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":202,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755701+0000","last_change":"2024-02-05T07:20:34.765770+0000","last_active":"2024-02-05T07:20:44.755701+0000","last_peered":"2024-02-05T07:20:44.755701+0000","last_clean":"2024-02-05T07:20:44.755701+0000","last_became_active":"2024-02-05T07:18:48.424620+0000","last_became_peered":"2024-02-05T07:18:48.424620+0000","last_unstale":"2024-02-05T07:20:44.755701+0000","last_undegraded":"2024-02-05T07:20:44.755701+0000","last_fullsized":"2024-02-05T07:20:44.755701+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:20:34.765664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:20:34.765664+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:54.764530+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":108,"num_read_kb":103,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":96,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756150+0000","last_change":"2024-02-05T07:20:14.321610+0000","last_active":"2024-02-05T07:20:44.756150+0000","last_peered":"2024-02-05T07:20:44.756150+0000","last_clean":"2024-02-05T07:20:44.756150+0000","last_became_active":"2024-02-05T07:18:51.442184+0000","last_became_peered":"2024-02-05T07:18:51.442184+0000","last_unstale":"2024-02-05T07:20:44.756150+0000","last_undegraded":"2024-02-05T07:20:44.756150+0000","last_fullsized":"2024-02-05T07:20:44.756150+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:14.321523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:14.321523+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.745540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761749+0000","last_change":"2024-02-05T07:20:31.238357+0000","last_active":"2024-02-05T07:20:44.761749+0000","last_peered":"2024-02-05T07:20:44.761749+0000","last_clean":"2024-02-05T07:20:44.761749+0000","last_became_active":"2024-02-05T07:18:53.468512+0000","last_became_peered":"2024-02-05T07:18:53.468512+0000","last_unstale":"2024-02-05T07:20:44.761749+0000","last_undegraded":"2024-02-05T07:20:44.761749+0000","last_fullsized":"2024-02-05T07:20:44.761749+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:31.238251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:31.238251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:43.265894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.758554+0000","last_change":"2024-02-05T07:20:44.758554+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.758554+0000","last_undegraded":"2024-02-05T07:20:44.758554+0000","last_fullsized":"2024-02-05T07:20:44.758554+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756219+0000","last_change":"2024-02-05T07:20:20.526231+0000","last_active":"2024-02-05T07:20:44.756219+0000","last_peered":"2024-02-05T07:20:44.756219+0000","last_clean":"2024-02-05T07:20:44.756219+0000","last_became_active":"2024-02-05T07:18:55.505360+0000","last_became_peered":"2024-02-05T07:18:55.505360+0000","last_unstale":"2024-02-05T07:20:44.756219+0000","last_undegraded":"2024-02-05T07:20:44.756219+0000","last_fullsized":"2024-02-05T07:20:44.756219+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:20.526102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:20.526102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:49.916022+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.757426+0000","last_change":"2024-02-05T07:20:10.436628+0000","last_active":"2024-02-05T07:20:44.757426+0000","last_peered":"2024-02-05T07:20:44.757426+0000","last_clean":"2024-02-05T07:20:44.757426+0000","last_became_active":"2024-02-05T07:18:49.428470+0000","last_became_peered":"2024-02-05T07:18:49.428470+0000","last_unstale":"2024-02-05T07:20:44.757426+0000","last_undegraded":"2024-02-05T07:20:44.757426+0000","last_fullsized":"2024-02-05T07:20:44.757426+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:20:10.436535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:36.573911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":114,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756353+0000","last_change":"2024-02-05T07:20:12.309384+0000","last_active":"2024-02-05T07:20:44.756353+0000","last_peered":"2024-02-05T07:20:44.756353+0000","last_clean":"2024-02-05T07:20:44.756353+0000","last_became_active":"2024-02-05T07:18:51.507507+0000","last_became_peered":"2024-02-05T07:18:51.507507+0000","last_unstale":"2024-02-05T07:20:44.756353+0000","last_undegraded":"2024-02-05T07:20:44.756353+0000","last_fullsized":"2024-02-05T07:20:44.756353+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:20:12.309238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:12.309238+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:40.996099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.759833+0000","last_change":"2024-02-05T07:19:56.330173+0000","last_active":"2024-02-05T07:20:44.759833+0000","last_peered":"2024-02-05T07:20:44.759833+0000","last_clean":"2024-02-05T07:20:44.759833+0000","last_became_active":"2024-02-05T07:18:53.459474+0000","last_became_peered":"2024-02-05T07:18:53.459474+0000","last_unstale":"2024-02-05T07:20:44.759833+0000","last_undegraded":"2024-02-05T07:20:44.759833+0000","last_fullsized":"2024-02-05T07:20:44.759833+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:56.330068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:19:56.330068+0000","last_clean_scrub_stamp":"2024-02-05T07:19:56.330068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:22.066765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.760589+0000","last_change":"2024-02-05T07:20:44.760589+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.760589+0000","last_undegraded":"2024-02-05T07:20:44.760589+0000","last_fullsized":"2024-02-05T07:20:44.760589+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.755181+0000","last_change":"2024-02-05T07:18:55.474472+0000","last_active":"2024-02-05T07:20:44.755181+0000","last_peered":"2024-02-05T07:20:44.755181+0000","last_clean":"2024-02-05T07:20:44.755181+0000","last_became_active":"2024-02-05T07:18:55.474029+0000","last_became_peered":"2024-02-05T07:18:55.474029+0000","last_unstale":"2024-02-05T07:20:44.755181+0000","last_undegraded":"2024-02-05T07:20:44.755181+0000","last_fullsized":"2024-02-05T07:20:44.755181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:47.259686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761356+0000","last_change":"2024-02-05T07:20:32.197801+0000","last_active":"2024-02-05T07:20:44.761356+0000","last_peered":"2024-02-05T07:20:44.761356+0000","last_clean":"2024-02-05T07:20:44.761356+0000","last_became_active":"2024-02-05T07:18:49.435299+0000","last_became_peered":"2024-02-05T07:18:49.435299+0000","last_unstale":"2024-02-05T07:20:44.761356+0000","last_undegraded":"2024-02-05T07:20:44.761356+0000","last_fullsized":"2024-02-05T07:20:44.761356+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:20:32.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:41.939226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":94,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.758096+0000","last_change":"2024-02-05T07:20:22.461850+0000","last_active":"2024-02-05T07:20:44.758096+0000","last_peered":"2024-02-05T07:20:44.758096+0000","last_clean":"2024-02-05T07:20:44.758096+0000","last_became_active":"2024-02-05T07:18:51.448044+0000","last_became_peered":"2024-02-05T07:18:51.448044+0000","last_unstale":"2024-02-05T07:20:44.758096+0000","last_undegraded":"2024-02-05T07:20:44.758096+0000","last_fullsized":"2024-02-05T07:20:44.758096+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:20:22.461692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:22.461692+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:31.055528+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":98,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756150+0000","last_change":"2024-02-05T07:18:53.463629+0000","last_active":"2024-02-05T07:20:44.756150+0000","last_peered":"2024-02-05T07:20:44.756150+0000","last_clean":"2024-02-05T07:20:44.756150+0000","last_became_active":"2024-02-05T07:18:53.462890+0000","last_became_peered":"2024-02-05T07:18:53.462890+0000","last_unstale":"2024-02-05T07:20:44.756150+0000","last_undegraded":"2024-02-05T07:20:44.756150+0000","last_fullsized":"2024-02-05T07:20:44.756150+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:47.259752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.762751+0000","last_change":"2024-02-05T07:20:44.762751+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.762751+0000","last_undegraded":"2024-02-05T07:20:44.762751+0000","last_fullsized":"2024-02-05T07:20:44.762751+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":62,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756120+0000","last_change":"2024-02-05T07:18:55.508071+0000","last_active":"2024-02-05T07:20:44.756120+0000","last_peered":"2024-02-05T07:20:44.756120+0000","last_clean":"2024-02-05T07:20:44.756120+0000","last_became_active":"2024-02-05T07:18:55.507791+0000","last_became_peered":"2024-02-05T07:18:55.507791+0000","last_unstale":"2024-02-05T07:20:44.756120+0000","last_undegraded":"2024-02-05T07:20:44.756120+0000","last_fullsized":"2024-02-05T07:20:44.756120+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:47.671271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.756248+0000","last_change":"2024-02-05T07:19:51.364668+0000","last_active":"2024-02-05T07:20:44.756248+0000","last_peered":"2024-02-05T07:20:44.756248+0000","last_clean":"2024-02-05T07:20:44.756248+0000","last_became_active":"2024-02-05T07:18:49.430123+0000","last_became_peered":"2024-02-05T07:18:49.430123+0000","last_unstale":"2024-02-05T07:20:44.756248+0000","last_undegraded":"2024-02-05T07:20:44.756248+0000","last_fullsized":"2024-02-05T07:20:44.756248+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:19:51.364585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.364585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:57.656562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":91,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.762110+0000","last_change":"2024-02-05T07:18:51.438840+0000","last_active":"2024-02-05T07:20:44.762110+0000","last_peered":"2024-02-05T07:20:44.762110+0000","last_clean":"2024-02-05T07:20:44.762110+0000","last_became_active":"2024-02-05T07:18:51.438507+0000","last_became_peered":"2024-02-05T07:18:51.438507+0000","last_unstale":"2024-02-05T07:20:44.762110+0000","last_undegraded":"2024-02-05T07:20:44.762110+0000","last_fullsized":"2024-02-05T07:20:44.762110+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.925777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":98,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763533+0000","last_change":"2024-02-05T07:18:53.464137+0000","last_active":"2024-02-05T07:20:44.763533+0000","last_peered":"2024-02-05T07:20:44.763533+0000","last_clean":"2024-02-05T07:20:44.763533+0000","last_became_active":"2024-02-05T07:18:53.463651+0000","last_became_peered":"2024-02-05T07:18:53.463651+0000","last_unstale":"2024-02-05T07:20:44.763533+0000","last_undegraded":"2024-02-05T07:20:44.763533+0000","last_fullsized":"2024-02-05T07:20:44.763533+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.995611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.762332+0000","last_change":"2024-02-05T07:20:44.762332+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.762332+0000","last_undegraded":"2024-02-05T07:20:44.762332+0000","last_fullsized":"2024-02-05T07:20:44.762332+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761627+0000","last_change":"2024-02-05T07:20:30.893200+0000","last_active":"2024-02-05T07:20:44.761627+0000","last_peered":"2024-02-05T07:20:44.761627+0000","last_clean":"2024-02-05T07:20:44.761627+0000","last_became_active":"2024-02-05T07:18:55.474447+0000","last_became_peered":"2024-02-05T07:18:55.474447+0000","last_unstale":"2024-02-05T07:20:44.761627+0000","last_undegraded":"2024-02-05T07:20:44.761627+0000","last_fullsized":"2024-02-05T07:20:44.761627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:30.893106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:30.893106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:50.758530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.764614+0000","last_change":"2024-02-05T07:18:49.435071+0000","last_active":"2024-02-05T07:20:44.764614+0000","last_peered":"2024-02-05T07:20:44.764614+0000","last_clean":"2024-02-05T07:20:44.764614+0000","last_became_active":"2024-02-05T07:18:49.434719+0000","last_became_peered":"2024-02-05T07:18:49.434719+0000","last_unstale":"2024-02-05T07:20:44.764614+0000","last_undegraded":"2024-02-05T07:20:44.764614+0000","last_fullsized":"2024-02-05T07:20:44.764614+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.995550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":110,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763494+0000","last_change":"2024-02-05T07:20:00.797048+0000","last_active":"2024-02-05T07:20:44.763494+0000","last_peered":"2024-02-05T07:20:44.763494+0000","last_clean":"2024-02-05T07:20:44.763494+0000","last_became_active":"2024-02-05T07:18:51.507524+0000","last_became_peered":"2024-02-05T07:18:51.507524+0000","last_unstale":"2024-02-05T07:20:44.763494+0000","last_undegraded":"2024-02-05T07:20:44.763494+0000","last_fullsized":"2024-02-05T07:20:44.763494+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:20:00.796905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.796905+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:13.158158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761674+0000","last_change":"2024-02-05T07:20:24.864128+0000","last_active":"2024-02-05T07:20:44.761674+0000","last_peered":"2024-02-05T07:20:44.761674+0000","last_clean":"2024-02-05T07:20:44.761674+0000","last_became_active":"2024-02-05T07:18:53.461213+0000","last_became_peered":"2024-02-05T07:18:53.461213+0000","last_unstale":"2024-02-05T07:20:44.761674+0000","last_undegraded":"2024-02-05T07:20:44.761674+0000","last_fullsized":"2024-02-05T07:20:44.761674+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:24.864057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:24.864057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:40.694174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.758602+0000","last_change":"2024-02-05T07:20:44.758602+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.758602+0000","last_undegraded":"2024-02-05T07:20:44.758602+0000","last_fullsized":"2024-02-05T07:20:44.758602+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761042+0000","last_change":"2024-02-05T07:20:30.285373+0000","last_active":"2024-02-05T07:20:44.761042+0000","last_peered":"2024-02-05T07:20:44.761042+0000","last_clean":"2024-02-05T07:20:44.761042+0000","last_became_active":"2024-02-05T07:18:55.472172+0000","last_became_peered":"2024-02-05T07:18:55.472172+0000","last_unstale":"2024-02-05T07:20:44.761042+0000","last_undegraded":"2024-02-05T07:20:44.761042+0000","last_fullsized":"2024-02-05T07:20:44.761042+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:30.285296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:30.285296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:48.221271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":98,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.762441+0000","last_change":"2024-02-05T07:18:53.460159+0000","last_active":"2024-02-05T07:20:44.762441+0000","last_peered":"2024-02-05T07:20:44.762441+0000","last_clean":"2024-02-05T07:20:44.762441+0000","last_became_active":"2024-02-05T07:18:53.459832+0000","last_became_peered":"2024-02-05T07:18:53.459832+0000","last_unstale":"2024-02-05T07:20:44.762441+0000","last_undegraded":"2024-02-05T07:20:44.762441+0000","last_fullsized":"2024-02-05T07:20:44.762441+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.925738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763408+0000","last_change":"2024-02-05T07:18:49.431779+0000","last_active":"2024-02-05T07:20:44.763408+0000","last_peered":"2024-02-05T07:20:44.763408+0000","last_clean":"2024-02-05T07:20:44.763408+0000","last_became_active":"2024-02-05T07:18:49.431369+0000","last_became_peered":"2024-02-05T07:18:49.431369+0000","last_unstale":"2024-02-05T07:20:44.763408+0000","last_undegraded":"2024-02-05T07:20:44.763408+0000","last_fullsized":"2024-02-05T07:20:44.763408+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:48.995525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"98'22","reported_seq":111,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.757671+0000","last_change":"2024-02-05T07:20:36.467017+0000","last_active":"2024-02-05T07:20:44.757671+0000","last_peered":"2024-02-05T07:20:44.757671+0000","last_clean":"2024-02-05T07:20:44.757671+0000","last_became_active":"2024-02-05T07:18:51.447469+0000","last_became_peered":"2024-02-05T07:18:51.447469+0000","last_unstale":"2024-02-05T07:20:44.757671+0000","last_undegraded":"2024-02-05T07:20:44.757671+0000","last_fullsized":"2024-02-05T07:20:44.757671+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'22","last_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:20:36.466914+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:54.868534+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.758602+0000","last_change":"2024-02-05T07:20:44.758602+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.758602+0000","last_undegraded":"2024-02-05T07:20:44.758602+0000","last_fullsized":"2024-02-05T07:20:44.758602+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763722+0000","last_change":"2024-02-05T07:20:10.764040+0000","last_active":"2024-02-05T07:20:44.763722+0000","last_peered":"2024-02-05T07:20:44.763722+0000","last_clean":"2024-02-05T07:20:44.763722+0000","last_became_active":"2024-02-05T07:18:55.505340+0000","last_became_peered":"2024-02-05T07:18:55.505340+0000","last_unstale":"2024-02-05T07:20:44.763722+0000","last_undegraded":"2024-02-05T07:20:44.763722+0000","last_fullsized":"2024-02-05T07:20:44.763722+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:10.763957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:10.763957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:35.465090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":66,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761729+0000","last_change":"2024-02-05T07:18:53.462879+0000","last_active":"2024-02-05T07:20:44.761729+0000","last_peered":"2024-02-05T07:20:44.761729+0000","last_clean":"2024-02-05T07:20:44.761729+0000","last_became_active":"2024-02-05T07:18:53.462591+0000","last_became_peered":"2024-02-05T07:18:53.462591+0000","last_unstale":"2024-02-05T07:20:44.761729+0000","last_undegraded":"2024-02-05T07:20:44.761729+0000","last_fullsized":"2024-02-05T07:20:44.761729+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:45.980356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761697+0000","last_change":"2024-02-05T07:20:05.883941+0000","last_active":"2024-02-05T07:20:44.761697+0000","last_peered":"2024-02-05T07:20:44.761697+0000","last_clean":"2024-02-05T07:20:44.761697+0000","last_became_active":"2024-02-05T07:18:49.434074+0000","last_became_peered":"2024-02-05T07:18:49.434074+0000","last_unstale":"2024-02-05T07:20:44.761697+0000","last_undegraded":"2024-02-05T07:20:44.761697+0000","last_fullsized":"2024-02-05T07:20:44.761697+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:05.883840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:05.883840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:30.747182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"90'16","reported_seq":102,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.762359+0000","last_change":"2024-02-05T07:20:09.873512+0000","last_active":"2024-02-05T07:20:44.762359+0000","last_peered":"2024-02-05T07:20:44.762359+0000","last_clean":"2024-02-05T07:20:44.762359+0000","last_became_active":"2024-02-05T07:18:51.447922+0000","last_became_peered":"2024-02-05T07:18:51.447922+0000","last_unstale":"2024-02-05T07:20:44.762359+0000","last_undegraded":"2024-02-05T07:20:44.762359+0000","last_fullsized":"2024-02-05T07:20:44.762359+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.873388+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.334535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.760431+0000","last_change":"2024-02-05T07:20:44.760431+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.760431+0000","last_undegraded":"2024-02-05T07:20:44.760431+0000","last_fullsized":"2024-02-05T07:20:44.760431+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761810+0000","last_change":"2024-02-05T07:20:29.670971+0000","last_active":"2024-02-05T07:20:44.761810+0000","last_peered":"2024-02-05T07:20:44.761810+0000","last_clean":"2024-02-05T07:20:44.761810+0000","last_became_active":"2024-02-05T07:18:55.474992+0000","last_became_peered":"2024-02-05T07:18:55.474992+0000","last_unstale":"2024-02-05T07:20:44.761810+0000","last_undegraded":"2024-02-05T07:20:44.761810+0000","last_fullsized":"2024-02-05T07:20:44.761810+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:29.670866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:29.670866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:40.071312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.757677+0000","last_change":"2024-02-05T07:20:06.398771+0000","last_active":"2024-02-05T07:20:44.757677+0000","last_peered":"2024-02-05T07:20:44.757677+0000","last_clean":"2024-02-05T07:20:44.757677+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:20:44.757677+0000","last_undegraded":"2024-02-05T07:20:44.757677+0000","last_fullsized":"2024-02-05T07:20:44.757677+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:06.398697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.398697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:10.245750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.759567+0000","last_change":"2024-02-05T07:20:18.399191+0000","last_active":"2024-02-05T07:20:44.759567+0000","last_peered":"2024-02-05T07:20:44.759567+0000","last_clean":"2024-02-05T07:20:44.759567+0000","last_became_active":"2024-02-05T07:18:49.430518+0000","last_became_peered":"2024-02-05T07:18:49.430518+0000","last_unstale":"2024-02-05T07:20:44.759567+0000","last_undegraded":"2024-02-05T07:20:44.759567+0000","last_fullsized":"2024-02-05T07:20:44.759567+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:18.399075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.399075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:36.367477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":106,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.762234+0000","last_change":"2024-02-05T07:19:51.808315+0000","last_active":"2024-02-05T07:20:44.762234+0000","last_peered":"2024-02-05T07:20:44.762234+0000","last_clean":"2024-02-05T07:20:44.762234+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:20:44.762234+0000","last_undegraded":"2024-02-05T07:20:44.762234+0000","last_fullsized":"2024-02-05T07:20:44.762234+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:19:51.808218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.808218+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:20:55.279939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":100,"state":"creating+peering","last_fresh":"2024-02-05T07:20:44.758540+0000","last_change":"2024-02-05T07:20:44.758540+0000","last_active":"2024-02-05T07:20:44.752333+0000","last_peered":"2024-02-05T07:20:44.752333+0000","last_clean":"2024-02-05T07:20:44.752333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:20:44.758540+0000","last_undegraded":"2024-02-05T07:20:44.758540+0000","last_fullsized":"2024-02-05T07:20:44.758540+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":70,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.762474+0000","last_change":"2024-02-05T07:20:36.859642+0000","last_active":"2024-02-05T07:20:44.762474+0000","last_peered":"2024-02-05T07:20:44.762474+0000","last_clean":"2024-02-05T07:20:44.762474+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:20:44.762474+0000","last_undegraded":"2024-02-05T07:20:44.762474+0000","last_fullsized":"2024-02-05T07:20:44.762474+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:36.859545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:36.859545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:48.101578+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":109,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761000+0000","last_change":"2024-02-05T07:20:18.512034+0000","last_active":"2024-02-05T07:20:44.761000+0000","last_peered":"2024-02-05T07:20:44.761000+0000","last_clean":"2024-02-05T07:20:44.761000+0000","last_became_active":"2024-02-05T07:18:53.461852+0000","last_became_peered":"2024-02-05T07:18:53.461852+0000","last_unstale":"2024-02-05T07:20:44.761000+0000","last_undegraded":"2024-02-05T07:20:44.761000+0000","last_fullsized":"2024-02-05T07:20:44.761000+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:20:18.511957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.511957+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:23.253319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.763861+0000","last_change":"2024-02-05T07:20:04.827259+0000","last_active":"2024-02-05T07:20:44.763861+0000","last_peered":"2024-02-05T07:20:44.763861+0000","last_clean":"2024-02-05T07:20:44.763861+0000","last_became_active":"2024-02-05T07:18:49.435368+0000","last_became_peered":"2024-02-05T07:18:49.435368+0000","last_unstale":"2024-02-05T07:20:44.763861+0000","last_undegraded":"2024-02-05T07:20:44.763861+0000","last_fullsized":"2024-02-05T07:20:44.763861+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:04.827180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:04.827180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:17.606501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":93,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-02-05T07:20:44.761960+0000","last_change":"2024-02-05T07:20:13.843192+0000","last_active":"2024-02-05T07:20:44.761960+0000","last_peered":"2024-02-05T07:20:44.761960+0000","last_clean":"2024-02-05T07:20:44.761960+0000","last_became_active":"2024-02-05T07:18:51.442711+0000","last_became_peered":"2024-02-05T07:18:51.442711+0000","last_unstale":"2024-02-05T07:20:44.761960+0000","last_undegraded":"2024-02-05T07:20:44.761960+0000","last_fullsized":"2024-02-05T07:20:44.761960+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:20:13.843051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:13.843051+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:41.856009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":103,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037788,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28260,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716892,"statfs":{"total":95995035648,"available":95966097408,"internally_reserved":0,"allocated":1245184,"data_stored":893977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.436,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52100000000000002,"15min":0.51300000000000001},"min":{"1min":0.439,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.439}]},{"osd":1,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53300000000000003,"15min":0.52700000000000002},"min":{"1min":0.42699999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55700000000000005,"15min":0.55500000000000005},"min":{"1min":0.47299999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.69699999999999995,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.51100000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53200000000000003,"15min":0.52700000000000002},"min":{"1min":0.42799999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.69799999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.52900000000000003,"15min":0.52100000000000002},"min":{"1min":0.47399999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.67900000000000005,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.57899999999999996}]},{"osd":3,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.436,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.47199999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.47199999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.438,"15min":0.438},"min":{"1min":0.316,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.627,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.44900000000000001,"15min":0.441},"min":{"1min":0.33500000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68799999999999994,"5min":0.746,"15min":0.746},"last":0.54500000000000004}]},{"osd":5,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.35999999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.497,"15min":0.496},"min":{"1min":0.34899999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77000000000000002,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.56100000000000005}]},{"osd":6,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.502,"15min":0.498},"min":{"1min":0.38200000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67300000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.51800000000000002,"15min":0.51300000000000001},"min":{"1min":0.39800000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68799999999999994,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.59999999999999998}]}]},{"osd":6,"up_from":52,"seq":223338299426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32408,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712744,"statfs":{"total":95995035648,"available":95961849856,"internally_reserved":0,"allocated":1232896,"data_stored":894198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.35399999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70299999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.318},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.40000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.67100000000000004,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.35199999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68700000000000006,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38500000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.747,"5min":0.751,"15min":0.751},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.42199999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77000000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.375},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.53000000000000003,"15min":0.53100000000000003},"min":{"1min":0.36899999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.39500000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.41199999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68999999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38200000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.42299999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.41299999999999998,"15min":0.40500000000000003},"min":{"1min":0.28699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.45300000000000001},"min":{"1min":0.314,"5min":0.313,"15min":0.313},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.40699999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45200000000000001,"15min":0.44700000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.435},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.46899999999999997,"15min":0.45800000000000002},"min":{"1min":0.314,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.307}]},{"osd":7,"last update":"Mon Feb 5 07:20:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.44800000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31876,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713276,"statfs":{"total":95995035648,"available":95962394624,"internally_reserved":0,"allocated":688128,"data_stored":304790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.36199999999999999,"15min":0.33400000000000002},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.45200000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.376,"15min":0.33700000000000002},"min":{"1min":0.23000000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55100000000000005,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.40400000000000003}]},{"osd":2,"last update":"Mon Feb 5 07:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.42499999999999999,"15min":0.441},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.48899999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.38},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.71799999999999997,"15min":1.02},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.498,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.33200000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.45800000000000002,"15min":0.47499999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.63800000000000001,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.47999999999999998,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68100000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.42299999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:20:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.57899999999999996,"15min":0.58599999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.68200000000000005,"15min":0.73699999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88900000000000001,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Feb 5 07:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61699999999999999,"15min":0.622},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.75700000000000001,"15min":0.78100000000000003},"min":{"1min":0.46500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.04,"5min":5.0289999999999999,"15min":5.0289999999999999},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon Feb 5 07:20:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.66900000000000004,"15min":0.68100000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.97299999999999998,"5min":2.48,"15min":2.48},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.66200000000000003,"15min":0.67700000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0569999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":0.56499999999999995}]},{"osd":7,"last update":"Mon Feb 5 07:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.44600000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.98499999999999999,"5min":1.081,"15min":1.081},"last":0.77200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28268,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716884,"statfs":{"total":95995035648,"available":95966089216,"internally_reserved":0,"allocated":1253376,"data_stored":894118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.34300000000000003,"15min":0.31},"min":{"1min":0.23499999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.32500000000000001},"min":{"1min":0.26700000000000002,"5min":0.214,"15min":0.214},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.33200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.89600000000000002,"5min":1.262,"15min":1.262},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.42799999999999999,"15min":0.42099999999999999},"min":{"1min":0.28000000000000003,"5min":0.182,"15min":0.182},"max":{"1min":0.67900000000000005,"5min":1.149,"15min":1.149},"last":0.371}]},{"osd":3,"last update":"Mon Feb 5 07:20:00 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42899999999999999,"15min":0.43099999999999999},"min":{"1min":0.29899999999999999,"5min":0.246,"15min":0.246},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41399999999999998,"15min":0.41799999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.32500000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:20:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.36699999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.65000000000000002,"15min":0.58199999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":10.103,"5min":10.103,"15min":10.103},"last":0.53400000000000003}]},{"osd":5,"last update":"Mon Feb 5 07:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56000000000000005,"15min":0.55400000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.45200000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.55900000000000005}]},{"osd":7,"last update":"Mon Feb 5 07:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.63,"15min":0.63900000000000001},"min":{"1min":0.39800000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.91000000000000003,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.40699999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27672,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":643072,"data_stored":303876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.37,"15min":0.36599999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.47599999999999998,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.39400000000000002,"15min":0.38400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.51800000000000002,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.32200000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.40400000000000003,"15min":0.38900000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.56899999999999995,"5min":1.583,"15min":1.583},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.379,"5min":0.437,"15min":0.42399999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.59899999999999998,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.30499999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.46700000000000003,"15min":0.46899999999999997},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.56000000000000005,"5min":1.631,"15min":1.631},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46600000000000003,"15min":0.46400000000000002},"min":{"1min":0.32500000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.55000000000000004,"5min":1.391,"15min":1.391},"last":0.38}]},{"osd":4,"last update":"Mon Feb 5 07:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54300000000000004,"15min":0.53300000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70799999999999996,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57099999999999995,"15min":0.55700000000000005},"min":{"1min":0.42199999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78400000000000003,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.44900000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:19:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.61699999999999999,"15min":0.60699999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.46400000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.72799999999999998,"15min":0.75700000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86099999999999999,"5min":2.8319999999999999,"15min":2.8319999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.59799999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.72599999999999998,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.49399999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.66700000000000004,"15min":0.68799999999999994},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.75600000000000001,"5min":2.1309999999999998,"15min":2.1309999999999998},"last":0.438},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.628,"15min":0.63800000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.748,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.52400000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31860,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713292,"statfs":{"total":95995035648,"available":95962411008,"internally_reserved":0,"allocated":671744,"data_stored":303462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39300000000000002,"15min":0.40500000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.52500000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.44400000000000001,"15min":0.45700000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.20699999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45000000000000001,"15min":0.46800000000000003},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":0.63500000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48199999999999998,"15min":0.503},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57999999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.50700000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.45600000000000002,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.61899999999999999,"5min":0.75,"15min":0.75},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.47499999999999998,"15min":0.495},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.25900000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:20:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58799999999999997,"15min":0.59599999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.77000000000000002,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":0.39200000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79400000000000004,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":1.6539999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59699999999999998,"15min":0.59499999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58599999999999997,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.752,"5min":1.046,"15min":1.046},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.81100000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.53800000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:20:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":1.6339999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59199999999999997,"15min":0.59399999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69699999999999995,"5min":0.751,"15min":0.751},"last":0.55400000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953515,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31848,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713304,"statfs":{"total":95995035648,"available":95962423296,"internally_reserved":0,"allocated":659456,"data_stored":303163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:22 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51900000000000002,"15min":0.52700000000000002},"min":{"1min":0.307,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.754,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54000000000000004,"15min":0.55100000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.66400000000000003,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.443}]},{"osd":1,"last update":"Mon Feb 5 07:20:22 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.68799999999999994,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.53400000000000003,"15min":0.55000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.63300000000000001,"5min":0.754,"15min":0.754},"last":0.34000000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:20:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52200000000000002,"15min":0.53600000000000003},"min":{"1min":0.36099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.60099999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53600000000000003,"15min":0.53900000000000003},"min":{"1min":0.38600000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.55000000000000004}]},{"osd":3,"last update":"Mon Feb 5 07:20:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55000000000000004,"15min":0.55900000000000005},"min":{"1min":0.37,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73299999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55300000000000005,"15min":0.56000000000000005},"min":{"1min":0.38900000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.48799999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.376,"15min":0.36599999999999999},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.501},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.39000000000000001,"15min":0.378},"min":{"1min":0.309,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.57499999999999996,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.42899999999999999,"15min":0.41499999999999998},"min":{"1min":0.30599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.42699999999999999,"15min":0.40799999999999997},"min":{"1min":0.34999999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57099999999999995}]},{"osd":7,"last update":"Mon Feb 5 07:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.51200000000000001,"15min":0.499},"min":{"1min":0.38900000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.499,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":0.52900000000000003}]}]},{"osd":5,"up_from":41,"seq":176093659174,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":647168,"data_stored":303404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66800000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71199999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.63400000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.55400000000000005,"15min":0.56299999999999994},"min":{"1min":0.35399999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.69699999999999995,"5min":0.753,"15min":0.753},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.38200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.68200000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54400000000000004}]},{"osd":2,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68000000000000005,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.35099999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69999999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.64500000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64200000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71999999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.68700000000000006}]},{"osd":4,"last update":"Mon Feb 5 07:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.35999999999999999,"15min":0.34799999999999998},"min":{"1min":0.26800000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.38900000000000001,"15min":0.373},"min":{"1min":0.33200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.67000000000000004}]},{"osd":6,"last update":"Mon Feb 5 07:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.42099999999999999,"15min":0.40200000000000002},"min":{"1min":0.31900000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.47699999999999998,"15min":0.45300000000000001},"min":{"1min":0.41799999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.70399999999999996}]},{"osd":7,"last update":"Mon Feb 5 07:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51400000000000001,"15min":0.51200000000000001},"min":{"1min":0.32100000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.52000000000000002},"min":{"1min":0.40600000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.69699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:20:53.551 INFO:tasks.thrashosds.thrasher:Setting 3.18 to [3, 1, 7] 2024-02-05T07:20:53.551 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.18', '3', '1', '7'] 2024-02-05T07:20:53.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pg-upmap 3.18 3 1 7 2024-02-05T07:20:53.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:53.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:53.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:53.681 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:53.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:53.757 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:20:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:20:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:20:53.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:53.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:53.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:53.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:53.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:53.933 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:53.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:53.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:54.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:54.063 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:54.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:54.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:54.166 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:20:54.166 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:20:54.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:54.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:54.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:54.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:54.320 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:54.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:54.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:54.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:54.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:54 smithi007 ceph-mon[29098]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:20:54.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:54 smithi007 ceph-mon[37158]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:20:54.448 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:54.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:54.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:54.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:54 smithi090 ceph-mon[33213]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:20:54.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:54.578 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:54.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:54.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:54.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:54.707 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:54.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:54.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:54.836 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:54.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:54.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:54.940 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:54.966 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:54.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:55.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:55.095 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:55.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-02-05T07:20:55.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T07:20:55.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:55.225 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:55.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:55.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:55.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:55.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:55.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:55 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3367146593' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:20:55.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:55 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3367146593' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:20:55.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:55.483 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:55.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:55.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:55.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:55 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3367146593' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:20:55.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:55.613 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:55.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:55.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:55.715 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:55.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:55.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:55.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:55.869 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:55.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:55.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:55.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:55.995 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:55.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:56.097 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:56.120 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:56.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:56.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:56.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:56.246 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:56.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:56.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:56.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:56.368 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:56 smithi007 ceph-mon[29098]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:20:56.368 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:56 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3367146593' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:20:56.368 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:56 smithi007 ceph-mon[29098]: osdmap e103: 8 total, 8 up, 8 in 2024-02-05T07:20:56.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:56 smithi007 ceph-mon[37158]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:20:56.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:56 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3367146593' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:20:56.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:56 smithi007 ceph-mon[37158]: osdmap e103: 8 total, 8 up, 8 in 2024-02-05T07:20:56.372 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:56.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:56.426 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:56 smithi090 ceph-mon[33213]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:20:56.426 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:56 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3367146593' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:20:56.426 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:56 smithi090 ceph-mon[33213]: osdmap e103: 8 total, 8 up, 8 in 2024-02-05T07:20:56.475 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:56.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:56.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:56.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:56.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:56.627 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:56.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:56.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:56.752 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:56.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:56.813 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:56.819 DEBUG:teuthology.orchestra.run.smithi007:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-02-05T07:20:56.822 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T07:20:56.822 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T07:20:56.855 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:56.862 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:20:56.864 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:20:56.878 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:56.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:56.886 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:20:56.903 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:hints = 1 2024-02-05T07:20:56.904 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T07:20:56.904 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object prefix: benchmark_data_smithi007_77086 2024-02-05T07:20:56.904 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:20:56.904 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 0 0 0 0 0 0 - 0 2024-02-05T07:20:56.910 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:20:56.955 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:20:56.980 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:56.997 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:20:57.011 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:57.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:57.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:57.039 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:20:57.080 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:20:57.113 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:57.125 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:20:57.132 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:20:57.132 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:20:57.132 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:20:57.132 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:20:57.132 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:20:57.132 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:20:57.132 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:20:57.132 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:20:57.146 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:57.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:57.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:57.150 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:20:57.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:20:57.283 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:57.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:57.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:20:57.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:57.411 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:57.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:57.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:57 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3130258806' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:20:57.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:57 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3032861081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:20:57.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:57 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3130258806' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:20:57.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:57 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3032861081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:20:57.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:57 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3130258806' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:20:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:57 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3032861081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:20:57.554 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:57.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:57.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:57.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:57.698 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:57.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:57.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:57.832 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:57.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:57.904 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 1 15 3020 3005 187.8 187.812 0.00482799 0.00530256 2024-02-05T07:20:57.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:20:57.969 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:57.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:57.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:20:58.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:58.112 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:58.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:58.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:58 smithi007 ceph-mon[29098]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:20:58.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:58 smithi007 ceph-mon[29098]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:20:58.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:58 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:20:58.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:58 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3130258806' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:20:58.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:58 smithi007 ceph-mon[29098]: osdmap e104: 8 total, 8 up, 8 in 2024-02-05T07:20:58.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:58 smithi007 ceph-mon[37158]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:20:58.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:58 smithi007 ceph-mon[37158]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:20:58.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:58 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:20:58.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:58 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3130258806' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:20:58.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:58 smithi007 ceph-mon[37158]: osdmap e104: 8 total, 8 up, 8 in 2024-02-05T07:20:58.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:58.253 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:58.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:58.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:20:58.398 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:58.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:58.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:20:58.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:58.527 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:58 smithi090 ceph-mon[33213]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:20:58.528 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:58 smithi090 ceph-mon[33213]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:20:58.528 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:58 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:20:58.528 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:58 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3130258806' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:20:58.528 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:58 smithi090 ceph-mon[33213]: osdmap e104: 8 total, 8 up, 8 in 2024-02-05T07:20:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:58.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:58.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:58.640 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:58.669 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:58.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:58.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:58.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:20:58.803 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:58.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:58.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:20:58.905 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 2 16 6219 6203 193.781 199.875 0.00501608 0.00514966 2024-02-05T07:20:58.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:58.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:58.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:58.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:59.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:59.081 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:59.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:59.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:20:59 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:20:59.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:20:59 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:20:59.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:20:59.222 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:59.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:59.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:20:59.324 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:59.358 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:59.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:59.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:59.460 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:20:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:59.510 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:59.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:20:59.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:20:59 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:20:59.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:59.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:59.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:20:59.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:20:59.786 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:20:59.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:59.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:20:59.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:20:59.906 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 3 16 9193 9177 191.081 185.875 0.00663966 0.00522294 2024-02-05T07:20:59.930 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:20:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:20:59.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:00.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:00.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:00.078 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:00.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:00.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:00.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:00.228 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:00.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:00.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:21:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:21:00.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:00 smithi090 ceph-mon[33213]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:21:00.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:00.352 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:00 smithi007 ceph-mon[29098]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:21:00.352 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:00 smithi007 ceph-mon[37158]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:21:00.362 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:00.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:00.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:00.506 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:00.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:00.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:00.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:21:00.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:00.658 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:00.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:00.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:00.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:00.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:00.906 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 4 16 11974 11958 186.761 173.812 0.00765287 0.00534757 2024-02-05T07:21:00.941 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:00.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:00.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:01.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:01.072 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:01.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:01.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:01.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:01.215 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:01.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:01.317 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:01.358 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:01.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:01.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:01.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:01.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:01.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:01.494 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:01.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:01.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:01.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:01.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:01.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:01.620 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:01.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:01.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:01.755 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:01.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:01.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:01.795 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:21:01.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:01.887 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:01.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:01.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:01.907 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 5 16 14334 14318 178.864 147.5 0.00478853 0.00558098 2024-02-05T07:21:01.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:01.999 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:21:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:02.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:02.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:02.142 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:02.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:02.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:02.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:02.286 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:02.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:02.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:02.412 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:02.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:02.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:02 smithi007 ceph-mon[29098]: pgmap v68: 148 pgs: 148 active+clean; 578 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T07:21:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:02 smithi007 ceph-mon[37158]: pgmap v68: 148 pgs: 148 active+clean; 578 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T07:21:02.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:02.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:02 smithi090 ceph-mon[33213]: pgmap v68: 148 pgs: 148 active+clean; 578 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T07:21:02.552 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:02.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:02.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:02.685 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:02.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:02.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:02.820 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:02.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:02.907 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 6 16 16369 16353 170.253 127.188 0.00726695 0.00586665 2024-02-05T07:21:02.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:02.973 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:02.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:03.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:03.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:03.107 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:03.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:03.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:03.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:03.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:03.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:03.373 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:03.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:03.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:03.506 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:03.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:03.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:03.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:03.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:03.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:03.678 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:21:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:21:03.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:03.811 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:03.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:03.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:03.907 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 7 16 18848 18832 168.065 154.938 0.00552647 0.00594253 2024-02-05T07:21:03.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:03.943 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:03.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:04.045 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:04.074 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:04.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:04.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:04.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:04.185 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:21:04.185 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-02-05T07:21:04.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-02-05T07:21:04.185 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:21:04.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.185 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:13694 7.6 7:64bb3798:::benchmark_data_smithi007_77086_object13693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T07:21:04.185 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.559461+0000", 2024-02-05T07:21:04.185 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5721267499999998, 2024-02-05T07:21:04.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.114005255, 2024-02-05T07:21:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:21:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:21:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:21:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:21:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 13694 2024-02-05T07:21:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.559461+0000", 2024-02-05T07:21:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.559461+0000", 2024-02-05T07:21:04.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.559462+0000", 2024-02-05T07:21:04.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1800000000000005e-07 2024-02-05T07:21:04.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.559483+0000", 2024-02-05T07:21:04.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0902000000000001e-05 2024-02-05T07:21:04.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.559491+0000", 2024-02-05T07:21:04.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4290000000000001e-06 2024-02-05T07:21:04.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.559497+0000", 2024-02-05T07:21:04.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2110000000000003e-06 2024-02-05T07:21:04.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.562448+0000", 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002950795 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.562579+0000", 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000131237 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,7", 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.562706+0000", 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000127162 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:21:04.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673281+0000", 2024-02-05T07:21:04.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.110574778 2024-02-05T07:21:04.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673321+0000", 2024-02-05T07:21:04.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0067000000000001e-05 2024-02-05T07:21:04.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673376+0000", 2024-02-05T07:21:04.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4969000000000002e-05 2024-02-05T07:21:04.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.191 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:21:04.191 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673427+0000", 2024-02-05T07:21:04.191 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1128999999999998e-05 2024-02-05T07:21:04.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.191 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.191 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673466+0000", 2024-02-05T07:21:04.191 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8757999999999997e-05 2024-02-05T07:21:04.191 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.191 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.191 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:13698 7.6 7:652aca29:::benchmark_data_smithi007_77086_object13697:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T07:21:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.561732+0000", 2024-02-05T07:21:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5698554160000002, 2024-02-05T07:21:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.11203244, 2024-02-05T07:21:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:21:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:21:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:21:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:21:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 13698 2024-02-05T07:21:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.561732+0000", 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.561732+0000", 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.561734+0000", 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3999999999999999e-06 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.561755+0000", 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1455e-05 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.561766+0000", 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0710000000000001e-05 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.561780+0000", 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4110999999999999e-05 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563863+0000", 2024-02-05T07:21:04.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020835379999999998 2024-02-05T07:21:04.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564008+0000", 2024-02-05T07:21:04.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014498500000000001 2024-02-05T07:21:04.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,7", 2024-02-05T07:21:04.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564112+0000", 2024-02-05T07:21:04.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010349300000000001 2024-02-05T07:21:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:21:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673336+0000", 2024-02-05T07:21:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.109224485 2024-02-05T07:21:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673514+0000", 2024-02-05T07:21:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017784100000000001 2024-02-05T07:21:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673665+0000", 2024-02-05T07:21:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015064500000000001 2024-02-05T07:21:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:21:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673721+0000", 2024-02-05T07:21:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6449000000000001e-05 2024-02-05T07:21:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673765+0000", 2024-02-05T07:21:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3328000000000003e-05 2024-02-05T07:21:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:13701 7.6 7:6761e70a:::benchmark_data_smithi007_77086_object13700:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T07:21:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.562314+0000", 2024-02-05T07:21:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5692739410000001, 2024-02-05T07:21:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.111604788, 2024-02-05T07:21:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:21:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:21:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:21:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:21:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 13701 2024-02-05T07:21:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.562314+0000", 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.562314+0000", 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.562314+0000", 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9800000000000003e-07 2024-02-05T07:21:04.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.200 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.200 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.562339+0000", 2024-02-05T07:21:04.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4264999999999999e-05 2024-02-05T07:21:04.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.200 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.200 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.562349+0000", 2024-02-05T07:21:04.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9639999999999998e-06 2024-02-05T07:21:04.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.200 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.200 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.562358+0000", 2024-02-05T07:21:04.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2350000000000005e-06 2024-02-05T07:21:04.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.201 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.201 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564641+0000", 2024-02-05T07:21:04.201 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0022825979999999998 2024-02-05T07:21:04.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.201 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.201 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564782+0000", 2024-02-05T07:21:04.201 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000141044 2024-02-05T07:21:04.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.201 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,7", 2024-02-05T07:21:04.201 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564930+0000", 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014889100000000001 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673342+0000", 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.108411457 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673555+0000", 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00021262399999999999 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673820+0000", 2024-02-05T07:21:04.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026515399999999998 2024-02-05T07:21:04.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:21:04.203 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673874+0000", 2024-02-05T07:21:04.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4144999999999997e-05 2024-02-05T07:21:04.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.203 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673918+0000", 2024-02-05T07:21:04.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4613000000000001e-05 2024-02-05T07:21:04.203 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.203 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.203 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.204 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.24719.0:13694 7.6 e104/100 ondisk, result = 0)", 2024-02-05T07:21:04.204 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.563795+0000", 2024-02-05T07:21:04.204 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5677929829999999, 2024-02-05T07:21:04.204 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.109547911, 2024-02-05T07:21:04.204 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.204 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:21:04.204 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.204 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.204 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563795+0000", 2024-02-05T07:21:04.204 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563795+0000", 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563795+0000", 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.99e-07 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563799+0000", 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0999999999999997e-06 2024-02-05T07:21:04.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563809+0000", 2024-02-05T07:21:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3230000000000004e-06 2024-02-05T07:21:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563815+0000", 2024-02-05T07:21:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4810000000000003e-06 2024-02-05T07:21:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673310+0000", 2024-02-05T07:21:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10949446 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673316+0000", 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2890000000000003e-06 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673343+0000", 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6659e-05 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:13707 7.4 7:2eedff10:::benchmark_data_smithi007_77086_object13706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.563866+0000", 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5677212819999999, 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.11023775600000001, 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 13707 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.208 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563866+0000", 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563866+0000", 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563868+0000", 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.283e-06 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.209 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563894+0000", 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6398e-05 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563915+0000", 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1077000000000002e-05 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563923+0000", 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9340000000000006e-06 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564069+0000", 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014609200000000001 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564187+0000", 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000117929 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564323+0000", 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013551000000000001 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:21:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565735+0000", 2024-02-05T07:21:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0014123479999999999 2024-02-05T07:21:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566360+0000", 2024-02-05T07:21:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00062545600000000004 2024-02-05T07:21:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567605+0000", 2024-02-05T07:21:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012445030000000001 2024-02-05T07:21:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:21:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.674068+0000", 2024-02-05T07:21:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10646309900000001 2024-02-05T07:21:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.674104+0000", 2024-02-05T07:21:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6127000000000002e-05 2024-02-05T07:21:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.24719.0:13694 7.6 e104/100 ondisk, result = 0)", 2024-02-05T07:21:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.563951+0000", 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5676365830000001, 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.109526153, 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563951+0000", 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563951+0000", 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563952+0000", 2024-02-05T07:21:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5599999999999995e-07 2024-02-05T07:21:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563955+0000", 2024-02-05T07:21:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1300000000000001e-06 2024-02-05T07:21:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563963+0000", 2024-02-05T07:21:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8529999999999992e-06 2024-02-05T07:21:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.563968+0000", 2024-02-05T07:21:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6670000000000001e-06 2024-02-05T07:21:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673365+0000", 2024-02-05T07:21:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.109397038 2024-02-05T07:21:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673372+0000", 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5799999999999997e-06 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673477+0000", 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010532899999999999 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:13709 7.f 7:f83c0c28:::benchmark_data_smithi007_77086_object13708:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.564220+0000", 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5673674399999999, 2024-02-05T07:21:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10976625800000001, 2024-02-05T07:21:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:21:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:21:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:21:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:21:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 13709 2024-02-05T07:21:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564220+0000", 2024-02-05T07:21:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564220+0000", 2024-02-05T07:21:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564221+0000", 2024-02-05T07:21:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0750000000000001e-06 2024-02-05T07:21:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564244+0000", 2024-02-05T07:21:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3207999999999999e-05 2024-02-05T07:21:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564255+0000", 2024-02-05T07:21:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1026000000000001e-05 2024-02-05T07:21:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564264+0000", 2024-02-05T07:21:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1329999999999999e-06 2024-02-05T07:21:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565606+0000", 2024-02-05T07:21:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0013420179999999999 2024-02-05T07:21:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565724+0000", 2024-02-05T07:21:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011867 2024-02-05T07:21:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,7", 2024-02-05T07:21:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565920+0000", 2024-02-05T07:21:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019600299999999999 2024-02-05T07:21:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673276+0000", 2024-02-05T07:21:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10735559 2024-02-05T07:21:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:21:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673288+0000", 2024-02-05T07:21:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2082999999999999e-05 2024-02-05T07:21:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673880+0000", 2024-02-05T07:21:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00059245200000000004 2024-02-05T07:21:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673951+0000", 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0355e-05 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673986+0000", 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5645000000000002e-05 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:13711 7.f 7:f26f7ebd:::benchmark_data_smithi007_77086_object13710:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.564427+0000", 2024-02-05T07:21:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5671601869999998, 2024-02-05T07:21:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.115432442, 2024-02-05T07:21:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:21:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:21:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:21:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:21:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 13711 2024-02-05T07:21:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564427+0000", 2024-02-05T07:21:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564427+0000", 2024-02-05T07:21:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564428+0000", 2024-02-05T07:21:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1800000000000004e-07 2024-02-05T07:21:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564455+0000", 2024-02-05T07:21:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6625e-05 2024-02-05T07:21:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564466+0000", 2024-02-05T07:21:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0744000000000001e-05 2024-02-05T07:21:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.564479+0000", 2024-02-05T07:21:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3242e-05 2024-02-05T07:21:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566875+0000", 2024-02-05T07:21:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023960650000000002 2024-02-05T07:21:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566992+0000", 2024-02-05T07:21:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011671399999999999 2024-02-05T07:21:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,7", 2024-02-05T07:21:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567364+0000", 2024-02-05T07:21:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00037221200000000002 2024-02-05T07:21:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673608+0000", 2024-02-05T07:21:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.106244323 2024-02-05T07:21:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:21:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.675292+0000", 2024-02-05T07:21:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0016841040000000001 2024-02-05T07:21:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.679772+0000", 2024-02-05T07:21:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.004479767 2024-02-05T07:21:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:21:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.679822+0000", 2024-02-05T07:21:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0294999999999999e-05 2024-02-05T07:21:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.679860+0000", 2024-02-05T07:21:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7432999999999999e-05 2024-02-05T07:21:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.24719.0:13698 7.6 e104/100 ondisk, result = 0)", 2024-02-05T07:21:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.565139+0000", 2024-02-05T07:21:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5664490899999999, 2024-02-05T07:21:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10838231199999999, 2024-02-05T07:21:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:21:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565139+0000", 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565139+0000", 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565139+0000", 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4200000000000001e-07 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565142+0000", 2024-02-05T07:21:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8949999999999998e-06 2024-02-05T07:21:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565159+0000", 2024-02-05T07:21:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6925000000000001e-05 2024-02-05T07:21:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565164+0000", 2024-02-05T07:21:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3600000000000004e-06 2024-02-05T07:21:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673502+0000", 2024-02-05T07:21:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.108338305 2024-02-05T07:21:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673509+0000", 2024-02-05T07:21:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7399999999999998e-06 2024-02-05T07:21:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673521+0000", 2024-02-05T07:21:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1644999999999999e-05 2024-02-05T07:21:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.24719.0:13701 7.6 e104/100 ondisk, result = 0)", 2024-02-05T07:21:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.565946+0000", 2024-02-05T07:21:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.565641184, 2024-02-05T07:21:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.107615602, 2024-02-05T07:21:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:21:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565946+0000", 2024-02-05T07:21:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565946+0000", 2024-02-05T07:21:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565947+0000", 2024-02-05T07:21:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0999999999999998e-07 2024-02-05T07:21:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565952+0000", 2024-02-05T07:21:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4739999999999999e-06 2024-02-05T07:21:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565971+0000", 2024-02-05T07:21:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9721e-05 2024-02-05T07:21:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.565981+0000", 2024-02-05T07:21:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6409999999999997e-06 2024-02-05T07:21:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673543+0000", 2024-02-05T07:21:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.107562396 2024-02-05T07:21:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673549+0000", 2024-02-05T07:21:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1399999999999997e-06 2024-02-05T07:21:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673562+0000", 2024-02-05T07:21:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2619999999999999e-05 2024-02-05T07:21:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:13715 7.4 7:2022f8df:::benchmark_data_smithi007_77086_object13714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T07:21:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.566233+0000", 2024-02-05T07:21:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.565355115, 2024-02-05T07:21:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10966003000000001, 2024-02-05T07:21:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:21:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:21:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:21:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:21:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 13715 2024-02-05T07:21:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566233+0000", 2024-02-05T07:21:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566233+0000", 2024-02-05T07:21:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566234+0000", 2024-02-05T07:21:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4109999999999999e-06 2024-02-05T07:21:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566264+0000", 2024-02-05T07:21:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.0422000000000001e-05 2024-02-05T07:21:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566275+0000", 2024-02-05T07:21:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1075e-05 2024-02-05T07:21:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566284+0000", 2024-02-05T07:21:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1089999999999998e-06 2024-02-05T07:21:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566393+0000", 2024-02-05T07:21:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000109512 2024-02-05T07:21:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566781+0000", 2024-02-05T07:21:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00038824299999999998 2024-02-05T07:21:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:21:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566929+0000", 2024-02-05T07:21:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014784800000000001 2024-02-05T07:21:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.674156+0000", 2024-02-05T07:21:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10722712299999999 2024-02-05T07:21:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:21:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.675359+0000", 2024-02-05T07:21:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00120309 2024-02-05T07:21:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.675792+0000", 2024-02-05T07:21:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000432383 2024-02-05T07:21:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:21:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.675848+0000", 2024-02-05T07:21:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5825000000000002e-05 2024-02-05T07:21:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.675893+0000", 2024-02-05T07:21:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4988999999999999e-05 2024-02-05T07:21:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.24719.0:13707 7.4 e104/100 ondisk, result = 0)", 2024-02-05T07:21:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.566400+0000", 2024-02-05T07:21:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5651875459999998, 2024-02-05T07:21:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.107714456, 2024-02-05T07:21:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:21:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566400+0000", 2024-02-05T07:21:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566400+0000", 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566401+0000", 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5000000000000002e-07 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566405+0000", 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4780000000000002e-06 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566413+0000", 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0830000000000004e-06 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566420+0000", 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4479999999999999e-06 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567592+0000", 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0011726040000000001 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567599+0000", 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8920000000000003e-06 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.674115+0000", 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10651530100000001 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.24719.0:13713 7.d e104/100 7:b53189b2:::benchmark_data_smithi007_77086_object13712:head v 104'826, mlcod=104'823)", 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.566534+0000", 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5650539970000001, 2024-02-05T07:21:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.108523331, 2024-02-05T07:21:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:21:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566534+0000", 2024-02-05T07:21:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566534+0000", 2024-02-05T07:21:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566534+0000", 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1800000000000005e-07 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566598+0000", 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3536999999999998e-05 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566607+0000", 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9830000000000002e-06 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566614+0000", 2024-02-05T07:21:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9870000000000002e-06 2024-02-05T07:21:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566755+0000", 2024-02-05T07:21:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000141008 2024-02-05T07:21:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566773+0000", 2024-02-05T07:21:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7608000000000001e-05 2024-02-05T07:21:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:21:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.675014+0000", 2024-02-05T07:21:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.108241454 2024-02-05T07:21:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.675057+0000", 2024-02-05T07:21:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2935999999999997e-05 2024-02-05T07:21:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.24719.0:13714 7.7 e104/100 7:ea494e73:::benchmark_data_smithi007_77086_object13713:head v 104'868, mlcod=104'866)", 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.566633+0000", 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5649545499999999, 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10865077099999999, 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566633+0000", 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566633+0000", 2024-02-05T07:21:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566633+0000", 2024-02-05T07:21:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0999999999999999e-07 2024-02-05T07:21:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566714+0000", 2024-02-05T07:21:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0451000000000006e-05 2024-02-05T07:21:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566721+0000", 2024-02-05T07:21:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2089999999999999e-06 2024-02-05T07:21:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566728+0000", 2024-02-05T07:21:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4450000000000001e-06 2024-02-05T07:21:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566782+0000", 2024-02-05T07:21:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4185999999999999e-05 2024-02-05T07:21:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.566800+0000", 2024-02-05T07:21:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8513999999999999e-05 2024-02-05T07:21:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:21:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.675245+0000", 2024-02-05T07:21:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10844485500000001 2024-02-05T07:21:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.675284+0000", 2024-02-05T07:21:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8701000000000001e-05 2024-02-05T07:21:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.24719.0:13709 7.f e104/100 ondisk, result = 0)", 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.567304+0000", 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5642836089999999, 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10598471, 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567304+0000", 2024-02-05T07:21:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567304+0000", 2024-02-05T07:21:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567305+0000", 2024-02-05T07:21:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5700000000000002e-07 2024-02-05T07:21:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567310+0000", 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6790000000000002e-06 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567326+0000", 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5362999999999998e-05 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567333+0000", 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9110000000000004e-06 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673261+0000", 2024-02-05T07:21:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.105928276 2024-02-05T07:21:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673270+0000", 2024-02-05T07:21:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7409999999999998e-06 2024-02-05T07:21:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673289+0000", 2024-02-05T07:21:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9182999999999999e-05 2024-02-05T07:21:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:13717 7.9 7:9e7aa4ae:::benchmark_data_smithi007_77086_object13716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.567363+0000", 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5642251200000001, 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.109277155, 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 13717 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567363+0000", 2024-02-05T07:21:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567363+0000", 2024-02-05T07:21:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567364+0000", 2024-02-05T07:21:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1119999999999999e-06 2024-02-05T07:21:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567412+0000", 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8285000000000003e-05 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567422+0000", 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6229999999999992e-06 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567429+0000", 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.238e-06 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567498+0000", 2024-02-05T07:21:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9479000000000003e-05 2024-02-05T07:21:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567616+0000", 2024-02-05T07:21:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000117859 2024-02-05T07:21:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:21:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673357+0000", 2024-02-05T07:21:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.105740504 2024-02-05T07:21:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.675490+0000", 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021336599999999999 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.676220+0000", 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00072974800000000001 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.676402+0000", 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000181485 2024-02-05T07:21:04.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.676601+0000", 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000199693 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.676640+0000", 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8469e-05 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.270 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.24719.0:13698 7.6 e104/100 ondisk, result = 0)", 2024-02-05T07:21:04.271 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.567756+0000", 2024-02-05T07:21:04.271 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5638319740000002, 2024-02-05T07:21:04.271 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10601872499999999, 2024-02-05T07:21:04.271 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.271 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:21:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567756+0000", 2024-02-05T07:21:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567756+0000", 2024-02-05T07:21:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567756+0000", 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0500000000000002e-07 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.567761+0000", 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7389999999999999e-06 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673602+0000", 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10584110300000001 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673612+0000", 2024-02-05T07:21:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0008e-05 2024-02-05T07:21:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673654+0000", 2024-02-05T07:21:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1516e-05 2024-02-05T07:21:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673660+0000", 2024-02-05T07:21:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2600000000000002e-06 2024-02-05T07:21:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673774+0000", 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000114294 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.24719.0:13715 7.4 e104/100 ondisk, result = 0)", 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.568341+0000", 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.5632462929999997, 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10582237999999999, 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.275 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.568341+0000", 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.568341+0000", 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.568342+0000", 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9599999999999998e-07 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.276 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.568347+0000", 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5990000000000004e-06 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673312+0000", 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10496545 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673321+0000", 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.166e-06 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.674143+0000", 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00082139799999999998 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.674150+0000", 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7039999999999994e-06 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.674164+0000", 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3267e-05 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.24719.0:13711 7.f e104/100 ondisk, result = 0)", 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:01.568497+0000", 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.563091054, 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.105118398, 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.568497+0000", 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.568497+0000", 2024-02-05T07:21:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.568497+0000", 2024-02-05T07:21:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5499999999999998e-07 2024-02-05T07:21:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.568501+0000", 2024-02-05T07:21:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9720000000000003e-06 2024-02-05T07:21:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673404+0000", 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10490319100000001 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673410+0000", 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9819999999999996e-06 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673597+0000", 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018630099999999999 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673603+0000", 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5169999999999996e-06 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:01.673615+0000", 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1780000000000001e-05 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:15116 7.9 7:9e3f3073:::benchmark_data_smithi007_77086_object15115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:02.367670+0000", 2024-02-05T07:21:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 1.7639174290000001, 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.090999369999999996, 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 15116 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:02.367670+0000", 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:02.367670+0000", 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:02.367671+0000", 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2950000000000001e-06 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:02.367693+0000", 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1036e-05 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.285 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:21:04.285 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:02.367702+0000", 2024-02-05T07:21:04.285 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.516e-06 2024-02-05T07:21:04.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.285 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:21:04.285 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:02.367709+0000", 2024-02-05T07:21:04.285 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8850000000000001e-06 2024-02-05T07:21:04.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.285 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:21:04.285 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:02.367761+0000", 2024-02-05T07:21:04.285 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1990999999999997e-05 2024-02-05T07:21:04.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:02.368261+0000", 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00049969900000000004 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:02.368405+0000", 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014412 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:02.370536+0000", 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021310690000000002 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:21:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:02.370552+0000", 2024-02-05T07:21:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6283e-05 2024-02-05T07:21:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:21:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:02.458547+0000", 2024-02-05T07:21:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.087995240000000002 2024-02-05T07:21:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:21:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:02.458628+0000", 2024-02-05T07:21:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0727000000000001e-05 2024-02-05T07:21:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:21:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:21:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:21:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:02.458670+0000", 2024-02-05T07:21:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1508999999999998e-05 2024-02-05T07:21:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:21:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:21:04.288 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:21:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:04.289 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:04.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:04.290 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:04 smithi007 ceph-mon[29098]: pgmap v69: 148 pgs: 148 active+clean; 578 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T07:21:04.290 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:04 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:21:04.290 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:04 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:21:04.290 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:04 smithi007 ceph-mon[37158]: pgmap v69: 148 pgs: 148 active+clean; 578 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T07:21:04.290 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:04 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:21:04.290 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:04 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:21:04.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:04.424 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:04.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:04.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:04.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:04 smithi090 ceph-mon[33213]: pgmap v69: 148 pgs: 148 active+clean; 578 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T07:21:04.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:04 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:21:04.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:04 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:21:04.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:04.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:04.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:04.700 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:04.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:04.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:04.802 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:04.834 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:04.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:04.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:04.907 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 8 15 20824 20809 162.503 123.562 0.00807617 0.00614665 2024-02-05T07:21:04.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:04.967 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:04.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:04.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:05.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:05.100 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:05.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:05.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:05.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:05.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:05.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:05.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:05.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:05.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:05 smithi007 ceph-mon[29098]: osdmap e105: 8 total, 8 up, 8 in 2024-02-05T07:21:05.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:05 smithi007 ceph-mon[37158]: osdmap e105: 8 total, 8 up, 8 in 2024-02-05T07:21:05.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:05.504 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:05.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:05.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:05 smithi090 ceph-mon[33213]: osdmap e105: 8 total, 8 up, 8 in 2024-02-05T07:21:05.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:05.652 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:05.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:05.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:05.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:05.783 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:05.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:05.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:05.909 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 9 16 22915 22899 158.94 130.625 0.00685896 0.00628442 2024-02-05T07:21:05.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:05.941 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:05.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:06.042 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:06.094 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:06.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:06.195 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:06.203 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:21:06.203 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:21:06.223 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:06.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:06.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:06.325 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:06.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:06.377 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:06.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:06.454 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[29098]: pgmap v71: 148 pgs: 148 active+clean; 1.1 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.33k op/s 2024-02-05T07:21:06.454 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[29098]: osdmap e106: 8 total, 8 up, 8 in 2024-02-05T07:21:06.454 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:06.454 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:06.455 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2252519018' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [3, 1, 7]}]: dispatch 2024-02-05T07:21:06.455 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [3, 1, 7]}]: dispatch 2024-02-05T07:21:06.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:21:06.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:21:06.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[37158]: pgmap v71: 148 pgs: 148 active+clean; 1.1 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.33k op/s 2024-02-05T07:21:06.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[37158]: osdmap e106: 8 total, 8 up, 8 in 2024-02-05T07:21:06.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:06.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:06.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2252519018' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [3, 1, 7]}]: dispatch 2024-02-05T07:21:06.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:06 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [3, 1, 7]}]: dispatch 2024-02-05T07:21:06.456 INFO:teuthology.orchestra.run.smithi007.stderr:set 3.18 pg_upmap mapping to [3,1,7] 2024-02-05T07:21:06.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:06.524 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:06 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:21:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:06 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:21:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:06 smithi090 ceph-mon[33213]: pgmap v71: 148 pgs: 148 active+clean; 1.1 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.33k op/s 2024-02-05T07:21:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:06 smithi090 ceph-mon[33213]: osdmap e106: 8 total, 8 up, 8 in 2024-02-05T07:21:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:06 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:06 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:06 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2252519018' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [3, 1, 7]}]: dispatch 2024-02-05T07:21:06.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:06 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [3, 1, 7]}]: dispatch 2024-02-05T07:21:06.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:06.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:06.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:06.668 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:06.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:06.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:06.798 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:06.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:06.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T07:21:06.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:06.910 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 10 16 24644 24628 153.826 108.062 0.0127965 0.00649449 2024-02-05T07:21:06.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:06.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:07.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:07.059 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:07.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:07.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:07.190 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:07.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:07.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:07.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:07.319 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:07.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:07.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:07.421 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:07.452 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:07.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:07.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:07.553 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:07.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:07.597 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:07.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:07.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:07.727 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:07.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:07.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:07.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:07 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [3, 1, 7]}]': finished 2024-02-05T07:21:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:07 smithi090 ceph-mon[33213]: osdmap e107: 8 total, 8 up, 8 in 2024-02-05T07:21:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:21:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:21:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:07 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:07.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:07.872 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:07.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:07.911 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 11 16 26152 26136 148.412 94.25 0.00710438 0.00673173 2024-02-05T07:21:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:07 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [3, 1, 7]}]': finished 2024-02-05T07:21:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:07 smithi007 ceph-mon[29098]: osdmap e107: 8 total, 8 up, 8 in 2024-02-05T07:21:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:21:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:21:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:07 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:07.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:07 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [3, 1, 7]}]': finished 2024-02-05T07:21:07.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:07 smithi007 ceph-mon[37158]: osdmap e107: 8 total, 8 up, 8 in 2024-02-05T07:21:07.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:21:07.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:21:07.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:07 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:07.973 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:08.031 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:08.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:08.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:08.163 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:08.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:08.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:08.298 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:08.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:08.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:08.433 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:08.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:08.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:08.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:08.573 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:08.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:08.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:08.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:08.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:08.703 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:08.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:08 smithi090 ceph-mon[33213]: pgmap v74: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-02-05T07:21:08.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:08 smithi090 ceph-mon[33213]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-05T07:21:08.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:08 smithi090 ceph-mon[33213]: Reconfiguring daemon alertmanager.a on smithi007 2024-02-05T07:21:08.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:08 smithi090 ceph-mon[33213]: osdmap e108: 8 total, 8 up, 8 in 2024-02-05T07:21:08.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:08.834 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:08.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:08.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:08.911 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 12 16 27940 27924 145.357 111.75 0.00737887 0.00687234 2024-02-05T07:21:08.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:08 smithi007 ceph-mon[29098]: pgmap v74: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-02-05T07:21:08.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:08 smithi007 ceph-mon[29098]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-05T07:21:08.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:08 smithi007 ceph-mon[29098]: Reconfiguring daemon alertmanager.a on smithi007 2024-02-05T07:21:08.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:08 smithi007 ceph-mon[29098]: osdmap e108: 8 total, 8 up, 8 in 2024-02-05T07:21:08.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:08 smithi007 ceph-mon[37158]: pgmap v74: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-02-05T07:21:08.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:08 smithi007 ceph-mon[37158]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-05T07:21:08.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:08 smithi007 ceph-mon[37158]: Reconfiguring daemon alertmanager.a on smithi007 2024-02-05T07:21:08.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:08 smithi007 ceph-mon[37158]: osdmap e108: 8 total, 8 up, 8 in 2024-02-05T07:21:08.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:08.967 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:08.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:08.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:09.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:09.093 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:09.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:09.195 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:09.235 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:09.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:09.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:09.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:09.362 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:09.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:09.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:09.491 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:09.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:09.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:09.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:09.617 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:09.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:09.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:09.751 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:09.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:09.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:09.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:09.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:09.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:09.911 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 13 16 29157 29141 140.028 76.0625 0.00397893 0.00709952 2024-02-05T07:21:09.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:10.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:10.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:10.146 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:10.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:10.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:10.275 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:21:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:21:10.282 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:10.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:10.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:10.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:10.425 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:10.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:10.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:10.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:10.553 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:10.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:10.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:10.695 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:10.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:10.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:10 smithi090 ceph-mon[33213]: pgmap v76: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 153 MiB/s wr, 2.45k op/s 2024-02-05T07:21:10.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:10.807 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:10 smithi007 ceph-mon[29098]: pgmap v76: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 153 MiB/s wr, 2.45k op/s 2024-02-05T07:21:10.807 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:10 smithi007 ceph-mon[37158]: pgmap v76: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 153 MiB/s wr, 2.45k op/s 2024-02-05T07:21:10.825 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:10.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:10.911 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 14 16 30918 30902 137.888 110.062 0.0043505 0.0072462 2024-02-05T07:21:10.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:10.961 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:10.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:10.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:11.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:11.101 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:11.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:11.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:11.237 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:11.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:11.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:11.367 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:11.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:11.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:11.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:11.511 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:11.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:11.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:11.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:11.638 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:11.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:11.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:11.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:11.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:11 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:11.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:11.781 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:11.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:11.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:11.911 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 15 16 32618 32602 135.779 106.25 0.00614564 0.0073591 2024-02-05T07:21:11.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:11.922 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:11.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:11.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:11 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:11.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:11 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:12.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:12.056 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:12.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:12.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:12.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:12.190 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:12.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:12.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:12.322 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:12.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:12.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:12.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:12.472 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:12.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:12.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:12.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:12.620 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:12.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:12.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:12.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:12 smithi007 ceph-mon[29098]: pgmap v77: 148 pgs: 148 active+clean; 1.8 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-02-05T07:21:12.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:12 smithi007 ceph-mon[37158]: pgmap v77: 148 pgs: 148 active+clean; 1.8 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-02-05T07:21:12.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:12.770 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:12.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:12 smithi090 ceph-mon[33213]: pgmap v77: 148 pgs: 148 active+clean; 1.8 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-02-05T07:21:12.872 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:12.899 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:12.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:12.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:12.911 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 16 16 34368 34352 134.129 109.375 0.0040541 0.00744958 2024-02-05T07:21:13.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:13.042 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:13.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:13.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:13.187 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:13.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:13.289 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:13.321 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:13.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:13.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:13.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:13.437 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:21:13.452 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:13.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:13.489 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:21:13.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:13.605 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:13.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:13.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:13.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:13.739 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:13.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:13.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:13.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:13 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1345013266' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:21:13.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:13 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:21:13.841 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:13.884 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:13.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:13.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:13.911 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 17 16 36157 36141 132.815 111.812 0.00672574 0.00752256 2024-02-05T07:21:13.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:13 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1345013266' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:21:13.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:13 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:21:13.925 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:21:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:21:13.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:13 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1345013266' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:21:13.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:13 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:21:13.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:14.019 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:14.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:14.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:14.151 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:14.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:14.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:14.292 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:14.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:14.394 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:14.427 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:14.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:14.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:14.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:14.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:14.576 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:14.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:14.607 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:14 smithi007 ceph-mon[29098]: pgmap v78: 148 pgs: 148 active+clean; 1.8 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 662 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-05T07:21:14.607 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:14 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:21:14.607 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:14 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:21:14.607 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:14 smithi007 ceph-mon[29098]: osdmap e109: 8 total, 8 up, 8 in 2024-02-05T07:21:14.607 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:14 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:21:14.608 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:14 smithi007 ceph-mon[37158]: pgmap v78: 148 pgs: 148 active+clean; 1.8 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 662 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-05T07:21:14.608 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:14 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:21:14.608 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:14 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:21:14.608 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:14 smithi007 ceph-mon[37158]: osdmap e109: 8 total, 8 up, 8 in 2024-02-05T07:21:14.608 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:14 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:21:14.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:14.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:14 smithi090 ceph-mon[33213]: pgmap v78: 148 pgs: 148 active+clean; 1.8 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 662 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-05T07:21:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:14 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:21:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:14 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:21:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:14 smithi090 ceph-mon[33213]: osdmap e109: 8 total, 8 up, 8 in 2024-02-05T07:21:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:14 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:21:14.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:14.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:14.911 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 18 15 38054 38039 132.027 118.625 0.00513066 0.00756822 2024-02-05T07:21:14.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:14.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:14.995 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:14.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:15.096 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:15.128 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:15.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:15.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:15.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:15.261 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:15.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:15.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:15.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:15.442 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:15.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:15.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:15.544 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:15.578 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:15.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:15.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:15 smithi007 ceph-mon[29098]: 5.e scrub starts 2024-02-05T07:21:15.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:15 smithi007 ceph-mon[29098]: 5.e scrub ok 2024-02-05T07:21:15.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:15 smithi007 ceph-mon[29098]: 3.18 scrub starts 2024-02-05T07:21:15.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:15 smithi007 ceph-mon[29098]: 3.18 scrub ok 2024-02-05T07:21:15.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:15 smithi007 ceph-mon[37158]: 5.e scrub starts 2024-02-05T07:21:15.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:15 smithi007 ceph-mon[37158]: 5.e scrub ok 2024-02-05T07:21:15.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:15 smithi007 ceph-mon[37158]: 3.18 scrub starts 2024-02-05T07:21:15.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:15 smithi007 ceph-mon[37158]: 3.18 scrub ok 2024-02-05T07:21:15.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:15.710 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:15.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:15.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:15.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:15 smithi090 ceph-mon[33213]: 5.e scrub starts 2024-02-05T07:21:15.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:15 smithi090 ceph-mon[33213]: 5.e scrub ok 2024-02-05T07:21:15.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:15 smithi090 ceph-mon[33213]: 3.18 scrub starts 2024-02-05T07:21:15.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:15 smithi090 ceph-mon[33213]: 3.18 scrub ok 2024-02-05T07:21:15.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:15.852 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:15.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:15.911 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 19 16 39921 39905 131.216 116.625 0.00806868 0.00761528 2024-02-05T07:21:15.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:15.993 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:15.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:15.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:16.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:16.124 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:16.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:16.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:16.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:16.256 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:16.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:16.358 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:16.398 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:16.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:16.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:16.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:16.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:16.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:16.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:16.661 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:16.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:16.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:16.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:16.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:16 smithi090 ceph-mon[33213]: pgmap v80: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s 2024-02-05T07:21:16.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:16 smithi090 ceph-mon[33213]: 4.1b scrub starts 2024-02-05T07:21:16.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:16 smithi090 ceph-mon[33213]: 4.1b scrub ok 2024-02-05T07:21:16.801 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:16.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:16.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:16.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:16.911 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:21:16.910996+0000 min lat: 0.00125178 max lat: 0.180861 avg lat: 0.00766658 2024-02-05T07:21:16.911 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:21:16.912 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 20 15 41741 41726 130.346 113.812 0.0099295 0.00766658 2024-02-05T07:21:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:16 smithi007 ceph-mon[29098]: pgmap v80: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s 2024-02-05T07:21:16.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:16 smithi007 ceph-mon[29098]: 4.1b scrub starts 2024-02-05T07:21:16.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:16 smithi007 ceph-mon[29098]: 4.1b scrub ok 2024-02-05T07:21:16.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:16 smithi007 ceph-mon[37158]: pgmap v80: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s 2024-02-05T07:21:16.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:16 smithi007 ceph-mon[37158]: 4.1b scrub starts 2024-02-05T07:21:16.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:16 smithi007 ceph-mon[37158]: 4.1b scrub ok 2024-02-05T07:21:16.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:16.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:16.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:17.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:17.078 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:17.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:17.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:17.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:17.210 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:17.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:17.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:17.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:17.355 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:17.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:17.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:17.479 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T07:21:17.479 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:21:17.479 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-02-05T07:21:17.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:21:17.503 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:17.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:17.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:17.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:17.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:17 smithi007 ceph-mon[29098]: 5.4 scrub starts 2024-02-05T07:21:17.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:17 smithi007 ceph-mon[29098]: 5.4 scrub ok 2024-02-05T07:21:17.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:17 smithi007 ceph-mon[29098]: 6.19 scrub starts 2024-02-05T07:21:17.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:17 smithi007 ceph-mon[29098]: 6.19 scrub ok 2024-02-05T07:21:17.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:17 smithi007 ceph-mon[29098]: 4.12 scrub starts 2024-02-05T07:21:17.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:17 smithi007 ceph-mon[29098]: 4.12 scrub ok 2024-02-05T07:21:17.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:17 smithi007 ceph-mon[37158]: 5.4 scrub starts 2024-02-05T07:21:17.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:17 smithi007 ceph-mon[37158]: 5.4 scrub ok 2024-02-05T07:21:17.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:17 smithi007 ceph-mon[37158]: 6.19 scrub starts 2024-02-05T07:21:17.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:17 smithi007 ceph-mon[37158]: 6.19 scrub ok 2024-02-05T07:21:17.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:17 smithi007 ceph-mon[37158]: 4.12 scrub starts 2024-02-05T07:21:17.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:17 smithi007 ceph-mon[37158]: 4.12 scrub ok 2024-02-05T07:21:17.611 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:17 smithi007 systemd[1]: Stopping Ceph alertmanager.a for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:21:17.635 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:17.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:17.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:17.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:17 smithi090 ceph-mon[33213]: 5.4 scrub starts 2024-02-05T07:21:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:17 smithi090 ceph-mon[33213]: 5.4 scrub ok 2024-02-05T07:21:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:17 smithi090 ceph-mon[33213]: 6.19 scrub starts 2024-02-05T07:21:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:17 smithi090 ceph-mon[33213]: 6.19 scrub ok 2024-02-05T07:21:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:17 smithi090 ceph-mon[33213]: 4.12 scrub starts 2024-02-05T07:21:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:17 smithi090 ceph-mon[33213]: 4.12 scrub ok 2024-02-05T07:21:17.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:17.786 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:17.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:17.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:17.912 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 21 16 43374 43358 128.995 102 0.0117951 0.00774674 2024-02-05T07:21:17.931 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:17.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:17.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:17.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:21:17.974 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:21:18.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:18.059 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:18.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:18.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:18.236 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:18.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:18.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:18.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:18.366 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:18.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:18.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:18.509 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:18.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:18.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:18.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:18.642 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:18.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:18.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:18.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:18 smithi090 ceph-mon[33213]: pgmap v81: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-02-05T07:21:18.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:18 smithi090 ceph-mon[33213]: 5.c scrub starts 2024-02-05T07:21:18.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:18 smithi090 ceph-mon[33213]: 5.c scrub ok 2024-02-05T07:21:18.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:18 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:21:18.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:18 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:21:18.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:18 smithi090 ceph-mon[33213]: osdmap e110: 8 total, 8 up, 8 in 2024-02-05T07:21:18.776 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:18.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:18.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:18.878 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:18.912 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 22 16 44837 44821 127.288 91.4375 0.0108852 0.00785068 2024-02-05T07:21:18.913 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:18.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:18.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:18.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:18 smithi007 ceph-mon[29098]: pgmap v81: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-02-05T07:21:18.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:18 smithi007 ceph-mon[29098]: 5.c scrub starts 2024-02-05T07:21:18.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:18 smithi007 ceph-mon[29098]: 5.c scrub ok 2024-02-05T07:21:18.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:18 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:21:18.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:18 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:21:18.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:18 smithi007 ceph-mon[29098]: osdmap e110: 8 total, 8 up, 8 in 2024-02-05T07:21:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:18 smithi007 ceph-mon[37158]: pgmap v81: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-02-05T07:21:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:18 smithi007 ceph-mon[37158]: 5.c scrub starts 2024-02-05T07:21:18.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:18 smithi007 ceph-mon[37158]: 5.c scrub ok 2024-02-05T07:21:18.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:18 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:21:18.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:18 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:21:18.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:18 smithi007 ceph-mon[37158]: osdmap e110: 8 total, 8 up, 8 in 2024-02-05T07:21:19.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:19.056 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:19.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:19.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:19.219 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:19.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:19.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:19.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:19.350 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:19.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:19.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:19.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:19.485 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:19.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:19.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:19.521 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:21:19.572 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:21:19.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:19.613 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:21:19.638 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:19.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:19.663 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:21:19.707 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:21:19.740 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:19.748 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:21:19.769 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:19.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:19.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:19.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:19 smithi090 ceph-mon[33213]: 4.11 scrub starts 2024-02-05T07:21:19.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:19 smithi090 ceph-mon[33213]: 4.11 scrub ok 2024-02-05T07:21:19.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:19 smithi090 ceph-mon[33213]: 2.1 scrub starts 2024-02-05T07:21:19.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:19 smithi090 ceph-mon[33213]: 2.1 scrub ok 2024-02-05T07:21:19.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:19 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2600372532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:21:19.791 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:21:19.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:19 smithi007 ceph-mon[29098]: 4.11 scrub starts 2024-02-05T07:21:19.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:19 smithi007 ceph-mon[29098]: 4.11 scrub ok 2024-02-05T07:21:19.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:19 smithi007 ceph-mon[29098]: 2.1 scrub starts 2024-02-05T07:21:19.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:19 smithi007 ceph-mon[29098]: 2.1 scrub ok 2024-02-05T07:21:19.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:19 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2600372532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:21:19.793 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:19 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[71934]: ts=2024-02-05T07:21:19.505Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-05T07:21:19.793 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:19 smithi007 podman[79470]: 2024-02-05 07:21:19.58363933 +0000 UTC m=+0.419234375 container died 9e76a975cda0902668349e41e7862feeffe8f1e8ab75b39bcb73ac0fb40334e6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T07:21:19.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:19 smithi007 ceph-mon[37158]: 4.11 scrub starts 2024-02-05T07:21:19.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:19 smithi007 ceph-mon[37158]: 4.11 scrub ok 2024-02-05T07:21:19.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:19 smithi007 ceph-mon[37158]: 2.1 scrub starts 2024-02-05T07:21:19.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:19 smithi007 ceph-mon[37158]: 2.1 scrub ok 2024-02-05T07:21:19.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:19 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2600372532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:21:19.828 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:21:19.834 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:21:19.834 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:21:19.834 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:21:19.834 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:21:19.834 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:21:19.834 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:21:19.835 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:21:19.835 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:21:19.871 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:19.912 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 23 16 46369 46353 125.917 95.75 0.0230188 0.00793476 2024-02-05T07:21:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:19.918 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:19.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:20.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:20.052 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:20.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:20.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:20.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:20.185 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:20.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:20.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:20.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:21:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:21:20.288 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:20.318 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:20.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:20.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:20.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:20.461 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:20.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:20.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:20.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:20.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:20.587 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:20.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:20.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:20 smithi007 ceph-mon[29098]: 5.16 scrub starts 2024-02-05T07:21:20.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:20 smithi007 ceph-mon[29098]: 5.16 scrub ok 2024-02-05T07:21:20.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:20 smithi007 ceph-mon[29098]: pgmap v83: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-05T07:21:20.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:20 smithi007 ceph-mon[29098]: 5.1b scrub starts 2024-02-05T07:21:20.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:20 smithi007 ceph-mon[29098]: 5.1b scrub ok 2024-02-05T07:21:20.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:20 smithi007 ceph-mon[37158]: 5.16 scrub starts 2024-02-05T07:21:20.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:20 smithi007 ceph-mon[37158]: 5.16 scrub ok 2024-02-05T07:21:20.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:20 smithi007 ceph-mon[37158]: pgmap v83: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-05T07:21:20.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:20 smithi007 ceph-mon[37158]: 5.1b scrub starts 2024-02-05T07:21:20.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:20 smithi007 ceph-mon[37158]: 5.1b scrub ok 2024-02-05T07:21:20.688 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:20.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:20.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:20 smithi090 ceph-mon[33213]: 5.16 scrub starts 2024-02-05T07:21:20.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:20 smithi090 ceph-mon[33213]: 5.16 scrub ok 2024-02-05T07:21:20.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:20 smithi090 ceph-mon[33213]: pgmap v83: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-05T07:21:20.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:20 smithi090 ceph-mon[33213]: 5.1b scrub starts 2024-02-05T07:21:20.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:20 smithi090 ceph-mon[33213]: 5.1b scrub ok 2024-02-05T07:21:20.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:20.857 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:20.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:20.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:20.912 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 24 16 47970 47954 124.839 100.062 0.00737344 0.0080048 2024-02-05T07:21:20.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:20.984 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:20.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:21.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:21.117 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:21.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:21.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:21.249 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:21.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:21.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:21.390 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:21.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:21.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:21.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:21.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:21.524 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:21.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:21.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:21.667 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:21.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:21.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:21.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:21.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:21 smithi090 ceph-mon[33213]: 4.1 scrub starts 2024-02-05T07:21:21.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:21 smithi090 ceph-mon[33213]: 4.1 scrub ok 2024-02-05T07:21:21.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:21 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:21.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:21 smithi090 ceph-mon[33213]: 4.10 scrub starts 2024-02-05T07:21:21.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:21 smithi090 ceph-mon[33213]: 4.10 scrub ok 2024-02-05T07:21:21.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:21 smithi090 ceph-mon[33213]: 3.15 scrub starts 2024-02-05T07:21:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:21 smithi090 ceph-mon[33213]: 3.15 scrub ok 2024-02-05T07:21:21.794 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:21.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:21.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:21.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:21.912 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 25 15 49273 49258 123.105 81.5 0.0100225 0.00811791 2024-02-05T07:21:21.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:21 smithi007 ceph-mon[29098]: 4.1 scrub starts 2024-02-05T07:21:21.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:21 smithi007 ceph-mon[29098]: 4.1 scrub ok 2024-02-05T07:21:21.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:21 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:21.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:21 smithi007 ceph-mon[29098]: 4.10 scrub starts 2024-02-05T07:21:21.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:21 smithi007 ceph-mon[29098]: 4.10 scrub ok 2024-02-05T07:21:21.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:21 smithi007 ceph-mon[29098]: 3.15 scrub starts 2024-02-05T07:21:21.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:21 smithi007 ceph-mon[29098]: 3.15 scrub ok 2024-02-05T07:21:21.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:21 smithi007 ceph-mon[37158]: 4.1 scrub starts 2024-02-05T07:21:21.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:21 smithi007 ceph-mon[37158]: 4.1 scrub ok 2024-02-05T07:21:21.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:21 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:21.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:21 smithi007 ceph-mon[37158]: 4.10 scrub starts 2024-02-05T07:21:21.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:21 smithi007 ceph-mon[37158]: 4.10 scrub ok 2024-02-05T07:21:21.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:21 smithi007 ceph-mon[37158]: 3.15 scrub starts 2024-02-05T07:21:21.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:21 smithi007 ceph-mon[37158]: 3.15 scrub ok 2024-02-05T07:21:21.933 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:21.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:21.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:22.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:22.067 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:22.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:22.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:22.134 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:21:22.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:22.198 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:22.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:22.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:22.343 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:22.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:22.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:22.387 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:22 smithi007 podman[79470]: 2024-02-05 07:21:22.099794281 +0000 UTC m=+2.935389331 container cleanup 9e76a975cda0902668349e41e7862feeffe8f1e8ab75b39bcb73ac0fb40334e6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T07:21:22.387 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:22 smithi007 bash[79470]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a 2024-02-05T07:21:22.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:22.486 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:22.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:22.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:22.630 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:22.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:22 smithi007 ceph-mon[29098]: pgmap v84: 148 pgs: 148 active+clean; 2.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-02-05T07:21:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:22 smithi007 ceph-mon[29098]: 4.b scrub starts 2024-02-05T07:21:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:22 smithi007 ceph-mon[29098]: 4.b scrub ok 2024-02-05T07:21:22.674 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:22 smithi007 podman[79538]: 2024-02-05 07:21:22.385803186 +0000 UTC m=+2.830219182 container remove 9e76a975cda0902668349e41e7862feeffe8f1e8ab75b39bcb73ac0fb40334e6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T07:21:22.674 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:22 smithi007 podman[79538]: 2024-02-05 07:21:22.561741122 +0000 UTC m=+3.006157121 volume remove ecda5a5d6fc515d2203ceaa095a9ed2c44a35aa087b6cfa21c6664e1b4b37f90 2024-02-05T07:21:22.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:22 smithi007 ceph-mon[37158]: pgmap v84: 148 pgs: 148 active+clean; 2.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-02-05T07:21:22.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:22 smithi007 ceph-mon[37158]: 4.b scrub starts 2024-02-05T07:21:22.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:22 smithi007 ceph-mon[37158]: 4.b scrub ok 2024-02-05T07:21:22.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:22.762 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:22.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:22.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:22.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:22.889 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:22.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:22.912 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 26 16 50792 50776 122.019 94.875 0.00495559 0.00819057 2024-02-05T07:21:22.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:22 smithi090 ceph-mon[33213]: pgmap v84: 148 pgs: 148 active+clean; 2.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-02-05T07:21:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:22 smithi090 ceph-mon[33213]: 4.b scrub starts 2024-02-05T07:21:23.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:22 smithi090 ceph-mon[33213]: 4.b scrub ok 2024-02-05T07:21:23.033 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:23.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:23.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:23.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:23.176 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:23.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:23.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:23.307 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:23.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:23.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:23.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:23.438 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:23.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:23.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:23.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:23.570 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:23.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:23.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:23.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:23.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:23.817 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:23.858 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:23.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:23.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:23.912 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 27 16 52397 52381 121.215 100.312 0.0105049 0.00824442 2024-02-05T07:21:23.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:23 smithi007 ceph-mon[29098]: 6.13 scrub starts 2024-02-05T07:21:23.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:23 smithi007 ceph-mon[29098]: 6.13 scrub ok 2024-02-05T07:21:23.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:23 smithi007 ceph-mon[29098]: 6.3 scrub starts 2024-02-05T07:21:23.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:23 smithi007 ceph-mon[29098]: 6.3 scrub ok 2024-02-05T07:21:23.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:23 smithi007 ceph-mon[29098]: pgmap v85: 148 pgs: 148 active+clean; 2.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-02-05T07:21:23.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:21:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:21:23.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:23 smithi007 ceph-mon[37158]: 6.13 scrub starts 2024-02-05T07:21:23.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:23 smithi007 ceph-mon[37158]: 6.13 scrub ok 2024-02-05T07:21:23.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:23 smithi007 ceph-mon[37158]: 6.3 scrub starts 2024-02-05T07:21:23.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:23 smithi007 ceph-mon[37158]: 6.3 scrub ok 2024-02-05T07:21:23.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:23 smithi007 ceph-mon[37158]: pgmap v85: 148 pgs: 148 active+clean; 2.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-02-05T07:21:23.960 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:23.991 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:23.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:24.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:23 smithi090 ceph-mon[33213]: 6.13 scrub starts 2024-02-05T07:21:24.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:23 smithi090 ceph-mon[33213]: 6.13 scrub ok 2024-02-05T07:21:24.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:23 smithi090 ceph-mon[33213]: 6.3 scrub starts 2024-02-05T07:21:24.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:23 smithi090 ceph-mon[33213]: 6.3 scrub ok 2024-02-05T07:21:24.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:23 smithi090 ceph-mon[33213]: pgmap v85: 148 pgs: 148 active+clean; 2.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-02-05T07:21:24.093 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:24.123 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:24.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:24.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:24.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:24.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:24.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:24.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:24.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:24.406 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:24.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:24.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:24.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:24.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:24.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:24.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:24.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:24.699 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:24.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:24.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:24.850 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:24.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:24.913 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 28 16 53870 53854 120.174 92.0625 0.00990364 0.00831511 2024-02-05T07:21:24.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:25.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:25.001 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:25.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:25.102 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:25.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:25.136 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:25.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:25.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:25.267 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:25.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:25.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:25.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:25.400 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:25.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:25.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:25.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:25 smithi007 ceph-mon[29098]: 5.f scrub starts 2024-02-05T07:21:25.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:25 smithi007 ceph-mon[29098]: 5.f scrub ok 2024-02-05T07:21:25.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:25 smithi007 ceph-mon[29098]: 5.10 scrub starts 2024-02-05T07:21:25.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:25 smithi007 ceph-mon[29098]: 5.10 scrub ok 2024-02-05T07:21:25.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:25 smithi007 ceph-mon[37158]: 5.f scrub starts 2024-02-05T07:21:25.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:25 smithi007 ceph-mon[37158]: 5.f scrub ok 2024-02-05T07:21:25.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:25 smithi007 ceph-mon[37158]: 5.10 scrub starts 2024-02-05T07:21:25.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:25 smithi007 ceph-mon[37158]: 5.10 scrub ok 2024-02-05T07:21:25.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:25.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:25 smithi090 ceph-mon[33213]: 5.f scrub starts 2024-02-05T07:21:25.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:25 smithi090 ceph-mon[33213]: 5.f scrub ok 2024-02-05T07:21:25.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:25 smithi090 ceph-mon[33213]: 5.10 scrub starts 2024-02-05T07:21:25.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:25 smithi090 ceph-mon[33213]: 5.10 scrub ok 2024-02-05T07:21:25.531 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:25.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:25.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:25.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:25.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:25.674 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:25.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:25.738 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:21:25.775 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:25.803 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:25.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:25.904 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:25.913 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 29 16 55362 55346 119.246 93.25 0.0101291 0.00837974 2024-02-05T07:21:25.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:25.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:25.960 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:25 smithi007 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-05T07:21:25.960 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:25 smithi007 systemd[1]: Stopped Ceph alertmanager.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:21:26.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:26.081 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:26.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:26.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:26.227 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:26.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:26.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:26 smithi007 ceph-mon[29098]: pgmap v86: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-05T07:21:26.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:26 smithi007 ceph-mon[29098]: 3.10 scrub starts 2024-02-05T07:21:26.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:26 smithi007 ceph-mon[29098]: 3.10 scrub ok 2024-02-05T07:21:26.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:26 smithi007 ceph-mon[29098]: 3.11 scrub starts 2024-02-05T07:21:26.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:26 smithi007 ceph-mon[29098]: 3.11 scrub ok 2024-02-05T07:21:26.248 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:26 smithi007 ceph-mon[37158]: pgmap v86: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-05T07:21:26.248 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:26 smithi007 ceph-mon[37158]: 3.10 scrub starts 2024-02-05T07:21:26.249 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:26 smithi007 ceph-mon[37158]: 3.10 scrub ok 2024-02-05T07:21:26.249 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:26 smithi007 ceph-mon[37158]: 3.11 scrub starts 2024-02-05T07:21:26.249 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:26 smithi007 ceph-mon[37158]: 3.11 scrub ok 2024-02-05T07:21:26.249 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:25 smithi007 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@alertmanager.a.service: Consumed 1.202s CPU time. 2024-02-05T07:21:26.249 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:25 smithi007 systemd[1]: Starting Ceph alertmanager.a for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:21:26.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:26.360 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:26.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:26.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:26.425 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:26 smithi090 ceph-mon[33213]: pgmap v86: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-05T07:21:26.425 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:26 smithi090 ceph-mon[33213]: 3.10 scrub starts 2024-02-05T07:21:26.425 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:26 smithi090 ceph-mon[33213]: 3.10 scrub ok 2024-02-05T07:21:26.426 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:26 smithi090 ceph-mon[33213]: 3.11 scrub starts 2024-02-05T07:21:26.426 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:26 smithi090 ceph-mon[33213]: 3.11 scrub ok 2024-02-05T07:21:26.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:26.492 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:26.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:26.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:26.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:26.625 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:26.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:26.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:26.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:26.770 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:26.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:26.854 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:21:26.854 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:21:26.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:21:26.871 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:26.913 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 30 16 56614 56598 117.879 78.25 0.0146532 0.00847656 2024-02-05T07:21:26.916 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:26.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:26.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:27.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:27.062 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:27.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:27.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:27.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:27.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:27.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:27.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:27.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:27.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:27 smithi007 ceph-mon[29098]: 5.1e scrub starts 2024-02-05T07:21:27.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:27 smithi007 ceph-mon[29098]: 5.1e scrub ok 2024-02-05T07:21:27.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:27 smithi007 ceph-mon[29098]: 5.9 scrub starts 2024-02-05T07:21:27.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:27 smithi007 ceph-mon[29098]: 5.9 scrub ok 2024-02-05T07:21:27.406 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:27 smithi007 ceph-mon[37158]: 5.1e scrub starts 2024-02-05T07:21:27.407 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:27 smithi007 ceph-mon[37158]: 5.1e scrub ok 2024-02-05T07:21:27.407 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:27 smithi007 ceph-mon[37158]: 5.9 scrub starts 2024-02-05T07:21:27.407 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:27 smithi007 ceph-mon[37158]: 5.9 scrub ok 2024-02-05T07:21:27.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:27.493 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:27.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:27.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:27.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:27 smithi090 ceph-mon[33213]: 5.1e scrub starts 2024-02-05T07:21:27.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:27 smithi090 ceph-mon[33213]: 5.1e scrub ok 2024-02-05T07:21:27.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:27 smithi090 ceph-mon[33213]: 5.9 scrub starts 2024-02-05T07:21:27.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:27 smithi090 ceph-mon[33213]: 5.9 scrub ok 2024-02-05T07:21:27.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:27.638 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:27.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:27.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:27.783 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:27.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:21:27.841 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:21:27.886 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:27.913 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 31 16 57674 57658 116.214 66.25 0.0204276 0.0085991 2024-02-05T07:21:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:27.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:27.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:28.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:28.043 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:28.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:28.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:28.176 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:28.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:28.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:28 smithi090 ceph-mon[33213]: pgmap v87: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-02-05T07:21:28.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:28 smithi090 ceph-mon[33213]: 4.19 scrub starts 2024-02-05T07:21:28.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:28 smithi090 ceph-mon[33213]: 4.19 scrub ok 2024-02-05T07:21:28.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:28 smithi090 ceph-mon[33213]: 6.0 scrub starts 2024-02-05T07:21:28.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:28 smithi090 ceph-mon[33213]: 6.0 scrub ok 2024-02-05T07:21:28.277 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:28.315 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:28.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:28.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:28.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:28.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:28 smithi007 ceph-mon[29098]: pgmap v87: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-02-05T07:21:28.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:28 smithi007 ceph-mon[29098]: 4.19 scrub starts 2024-02-05T07:21:28.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:28 smithi007 ceph-mon[29098]: 4.19 scrub ok 2024-02-05T07:21:28.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:28 smithi007 ceph-mon[29098]: 6.0 scrub starts 2024-02-05T07:21:28.426 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:28 smithi007 ceph-mon[29098]: 6.0 scrub ok 2024-02-05T07:21:28.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:28 smithi007 ceph-mon[37158]: pgmap v87: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-02-05T07:21:28.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:28 smithi007 ceph-mon[37158]: 4.19 scrub starts 2024-02-05T07:21:28.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:28 smithi007 ceph-mon[37158]: 4.19 scrub ok 2024-02-05T07:21:28.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:28 smithi007 ceph-mon[37158]: 6.0 scrub starts 2024-02-05T07:21:28.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:28 smithi007 ceph-mon[37158]: 6.0 scrub ok 2024-02-05T07:21:28.468 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:28.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:28.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:28.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:28.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:28.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:28.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:28.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:28.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:28.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:28.889 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:28.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:28.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:28.913 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 32 16 59032 59016 115.235 84.875 0.0162368 0.00867231 2024-02-05T07:21:28.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:29.035 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:29.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:29.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:29.137 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:29.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:29.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:29.318 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:29.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:29.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:29.419 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:29.459 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:29.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:29.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:29 smithi090 ceph-mon[33213]: 5.d scrub starts 2024-02-05T07:21:29.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:29 smithi090 ceph-mon[33213]: 5.d scrub ok 2024-02-05T07:21:29.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:29 smithi090 ceph-mon[33213]: from='client.24737 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:21:29.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:21:29.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:29.585 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:29.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:29.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:29.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:29 smithi007 ceph-mon[29098]: 5.d scrub starts 2024-02-05T07:21:29.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:29 smithi007 ceph-mon[29098]: 5.d scrub ok 2024-02-05T07:21:29.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:29 smithi007 ceph-mon[29098]: from='client.24737 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:21:29.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:21:29.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:29 smithi007 ceph-mon[37158]: 5.d scrub starts 2024-02-05T07:21:29.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:29 smithi007 ceph-mon[37158]: 5.d scrub ok 2024-02-05T07:21:29.653 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:29 smithi007 ceph-mon[37158]: from='client.24737 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:21:29.653 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:21:29.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:29.716 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-02-05T07:21:26.943648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3533576540,"num_objects":54107,"num_object_clones":0,"num_object_copies":162321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54107,"num_whiteouts":0,"num_read":1082,"num_read_kb":928,"num_write":54524,"num_write_kb":3452637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54532,"ondisk_log_size":54532,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10545468,"kb_used_data":10191932,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":739415748,"statfs":{"total":767960285184,"available":757161725952,"internally_reserved":0,"allocated":10436538368,"data_stored":10433228596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":361942576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1036713984,"num_objects":15819,"num_object_clones":0,"num_object_copies":47457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15819,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":15819,"num_write_kb":1012416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004446"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.506631+0000","last_change":"2024-02-05T07:20:25.913055+0000","last_active":"2024-02-05T07:21:17.506631+0000","last_peered":"2024-02-05T07:21:17.506631+0000","last_clean":"2024-02-05T07:21:17.506631+0000","last_became_active":"2024-02-05T07:18:55.466001+0000","last_became_peered":"2024-02-05T07:18:55.466001+0000","last_unstale":"2024-02-05T07:21:17.506631+0000","last_undegraded":"2024-02-05T07:21:17.506631+0000","last_fullsized":"2024-02-05T07:21:17.506631+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:25.912938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:25.912938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:53.255881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":96,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.515252+0000","last_change":"2024-02-05T07:20:31.382191+0000","last_active":"2024-02-05T07:21:17.515252+0000","last_peered":"2024-02-05T07:21:17.515252+0000","last_clean":"2024-02-05T07:21:17.515252+0000","last_became_active":"2024-02-05T07:18:53.460820+0000","last_became_peered":"2024-02-05T07:18:53.460820+0000","last_unstale":"2024-02-05T07:21:17.515252+0000","last_undegraded":"2024-02-05T07:21:17.515252+0000","last_fullsized":"2024-02-05T07:21:17.515252+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:31.382106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:31.382106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:54.397321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.516370+0000","last_change":"2024-02-05T07:20:35.243411+0000","last_active":"2024-02-05T07:21:17.516370+0000","last_peered":"2024-02-05T07:21:17.516370+0000","last_clean":"2024-02-05T07:21:17.516370+0000","last_became_active":"2024-02-05T07:18:49.434456+0000","last_became_peered":"2024-02-05T07:18:49.434456+0000","last_unstale":"2024-02-05T07:21:17.516370+0000","last_undegraded":"2024-02-05T07:21:17.516370+0000","last_fullsized":"2024-02-05T07:21:17.516370+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:35.243281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:35.243281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:44.816513+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":115,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.506565+0000","last_change":"2024-02-05T07:20:23.852115+0000","last_active":"2024-02-05T07:21:17.506565+0000","last_peered":"2024-02-05T07:21:17.506565+0000","last_clean":"2024-02-05T07:21:17.506565+0000","last_became_active":"2024-02-05T07:18:51.448924+0000","last_became_peered":"2024-02-05T07:18:51.448924+0000","last_unstale":"2024-02-05T07:21:17.506565+0000","last_undegraded":"2024-02-05T07:21:17.506565+0000","last_fullsized":"2024-02-05T07:21:17.506565+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:20:23.851993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:23.851993+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:48.656007+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":83,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514102+0000","last_change":"2024-02-05T07:18:55.506169+0000","last_active":"2024-02-05T07:21:17.514102+0000","last_peered":"2024-02-05T07:21:17.514102+0000","last_clean":"2024-02-05T07:21:17.514102+0000","last_became_active":"2024-02-05T07:18:55.505718+0000","last_became_peered":"2024-02-05T07:18:55.505718+0000","last_unstale":"2024-02-05T07:21:17.514102+0000","last_undegraded":"2024-02-05T07:21:17.514102+0000","last_fullsized":"2024-02-05T07:21:17.514102+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.506601+0000","last_change":"2024-02-05T07:19:54.746304+0000","last_active":"2024-02-05T07:21:17.506601+0000","last_peered":"2024-02-05T07:21:17.506601+0000","last_clean":"2024-02-05T07:21:17.506601+0000","last_became_active":"2024-02-05T07:18:53.461506+0000","last_became_peered":"2024-02-05T07:18:53.461506+0000","last_unstale":"2024-02-05T07:21:17.506601+0000","last_undegraded":"2024-02-05T07:21:17.506601+0000","last_fullsized":"2024-02-05T07:21:17.506601+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:54.746206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:19:54.746206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514458+0000","last_change":"2024-02-05T07:20:39.926066+0000","last_active":"2024-02-05T07:21:17.514458+0000","last_peered":"2024-02-05T07:21:17.514458+0000","last_clean":"2024-02-05T07:21:17.514458+0000","last_became_active":"2024-02-05T07:18:49.436490+0000","last_became_peered":"2024-02-05T07:18:49.436490+0000","last_unstale":"2024-02-05T07:21:17.514458+0000","last_undegraded":"2024-02-05T07:21:17.514458+0000","last_fullsized":"2024-02-05T07:21:17.514458+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:39.925994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:39.925994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:57.408803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":116,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514892+0000","last_change":"2024-02-05T07:18:51.445160+0000","last_active":"2024-02-05T07:21:17.514892+0000","last_peered":"2024-02-05T07:21:17.514892+0000","last_clean":"2024-02-05T07:21:17.514892+0000","last_became_active":"2024-02-05T07:18:51.444783+0000","last_became_peered":"2024-02-05T07:18:51.444783+0000","last_unstale":"2024-02-05T07:21:17.514892+0000","last_undegraded":"2024-02-05T07:21:17.514892+0000","last_fullsized":"2024-02-05T07:21:17.514892+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.515347+0000","last_change":"2024-02-05T07:20:31.252764+0000","last_active":"2024-02-05T07:21:17.515347+0000","last_peered":"2024-02-05T07:21:17.515347+0000","last_clean":"2024-02-05T07:21:17.515347+0000","last_became_active":"2024-02-05T07:18:55.474002+0000","last_became_peered":"2024-02-05T07:18:55.474002+0000","last_unstale":"2024-02-05T07:21:17.515347+0000","last_undegraded":"2024-02-05T07:21:17.515347+0000","last_fullsized":"2024-02-05T07:21:17.515347+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:31.252686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:31.252686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:37.020142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:19.716500+0000","last_change":"2024-02-05T07:21:19.716500+0000","last_active":"2024-02-05T07:21:19.716500+0000","last_peered":"2024-02-05T07:21:19.716500+0000","last_clean":"2024-02-05T07:21:19.716500+0000","last_became_active":"2024-02-05T07:18:53.464520+0000","last_became_peered":"2024-02-05T07:18:53.464520+0000","last_unstale":"2024-02-05T07:21:19.716500+0000","last_undegraded":"2024-02-05T07:21:19.716500+0000","last_fullsized":"2024-02-05T07:21:19.716500+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:19.716375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:19.716375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:26.095649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511158+0000","last_change":"2024-02-05T07:19:59.782950+0000","last_active":"2024-02-05T07:21:17.511158+0000","last_peered":"2024-02-05T07:21:17.511158+0000","last_clean":"2024-02-05T07:21:17.511158+0000","last_became_active":"2024-02-05T07:18:49.435454+0000","last_became_peered":"2024-02-05T07:18:49.435454+0000","last_unstale":"2024-02-05T07:21:17.511158+0000","last_undegraded":"2024-02-05T07:21:17.511158+0000","last_fullsized":"2024-02-05T07:21:17.511158+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:59.782841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:59.782841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.410088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"89'13","reported_seq":119,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.506941+0000","last_change":"2024-02-05T07:20:02.798288+0000","last_active":"2024-02-05T07:21:17.506941+0000","last_peered":"2024-02-05T07:21:17.506941+0000","last_clean":"2024-02-05T07:21:17.506941+0000","last_became_active":"2024-02-05T07:18:51.444592+0000","last_became_peered":"2024-02-05T07:18:51.444592+0000","last_unstale":"2024-02-05T07:21:17.506941+0000","last_undegraded":"2024-02-05T07:21:17.506941+0000","last_fullsized":"2024-02-05T07:21:17.506941+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.798180+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.510944+0000","last_change":"2024-02-05T07:21:15.751184+0000","last_active":"2024-02-05T07:21:17.510944+0000","last_peered":"2024-02-05T07:21:17.510944+0000","last_clean":"2024-02-05T07:21:17.510944+0000","last_became_active":"2024-02-05T07:18:55.473368+0000","last_became_peered":"2024-02-05T07:18:55.473368+0000","last_unstale":"2024-02-05T07:21:17.510944+0000","last_undegraded":"2024-02-05T07:21:17.510944+0000","last_fullsized":"2024-02-05T07:21:17.510944+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:15.751084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:15.751084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:36.730506+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.515992+0000","last_change":"2024-02-05T07:20:23.756522+0000","last_active":"2024-02-05T07:21:17.515992+0000","last_peered":"2024-02-05T07:21:17.515992+0000","last_clean":"2024-02-05T07:21:17.515992+0000","last_became_active":"2024-02-05T07:18:53.456881+0000","last_became_peered":"2024-02-05T07:18:53.456881+0000","last_unstale":"2024-02-05T07:21:17.515992+0000","last_undegraded":"2024-02-05T07:21:17.515992+0000","last_fullsized":"2024-02-05T07:21:17.515992+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:23.756416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:23.756416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.140659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511281+0000","last_change":"2024-02-05T07:20:13.048248+0000","last_active":"2024-02-05T07:21:17.511281+0000","last_peered":"2024-02-05T07:21:17.511281+0000","last_clean":"2024-02-05T07:21:17.511281+0000","last_became_active":"2024-02-05T07:18:49.431220+0000","last_became_peered":"2024-02-05T07:18:49.431220+0000","last_unstale":"2024-02-05T07:21:17.511281+0000","last_undegraded":"2024-02-05T07:21:17.511281+0000","last_fullsized":"2024-02-05T07:21:17.511281+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:13.048126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:13.048126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:28.009618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":109,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.506708+0000","last_change":"2024-02-05T07:21:15.028676+0000","last_active":"2024-02-05T07:21:17.506708+0000","last_peered":"2024-02-05T07:21:17.506708+0000","last_clean":"2024-02-05T07:21:17.506708+0000","last_became_active":"2024-02-05T07:18:51.448462+0000","last_became_peered":"2024-02-05T07:18:51.448462+0000","last_unstale":"2024-02-05T07:21:17.506708+0000","last_undegraded":"2024-02-05T07:21:17.506708+0000","last_fullsized":"2024-02-05T07:21:17.506708+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:21:15.028573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:15.028573+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:34.229163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.507273+0000","last_change":"2024-02-05T07:20:00.760332+0000","last_active":"2024-02-05T07:21:17.507273+0000","last_peered":"2024-02-05T07:21:17.507273+0000","last_clean":"2024-02-05T07:21:17.507273+0000","last_became_active":"2024-02-05T07:18:55.505678+0000","last_became_peered":"2024-02-05T07:18:55.505678+0000","last_unstale":"2024-02-05T07:21:17.507273+0000","last_undegraded":"2024-02-05T07:21:17.507273+0000","last_fullsized":"2024-02-05T07:21:17.507273+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:00.760230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.760230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:19.923028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511669+0000","last_change":"2024-02-05T07:20:18.552686+0000","last_active":"2024-02-05T07:21:17.511669+0000","last_peered":"2024-02-05T07:21:17.511669+0000","last_clean":"2024-02-05T07:21:17.511669+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:21:17.511669+0000","last_undegraded":"2024-02-05T07:21:17.511669+0000","last_fullsized":"2024-02-05T07:21:17.511669+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:18.552608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.552608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:43.361846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.516232+0000","last_change":"2024-02-05T07:18:49.432729+0000","last_active":"2024-02-05T07:21:17.516232+0000","last_peered":"2024-02-05T07:21:17.516232+0000","last_clean":"2024-02-05T07:21:17.516232+0000","last_became_active":"2024-02-05T07:18:49.432449+0000","last_became_peered":"2024-02-05T07:18:49.432449+0000","last_unstale":"2024-02-05T07:21:17.516232+0000","last_undegraded":"2024-02-05T07:21:17.516232+0000","last_fullsized":"2024-02-05T07:21:17.516232+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":124,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.509333+0000","last_change":"2024-02-05T07:19:59.624459+0000","last_active":"2024-02-05T07:21:17.509333+0000","last_peered":"2024-02-05T07:21:17.509333+0000","last_clean":"2024-02-05T07:21:17.509333+0000","last_became_active":"2024-02-05T07:18:51.447728+0000","last_became_peered":"2024-02-05T07:18:51.447728+0000","last_unstale":"2024-02-05T07:21:17.509333+0000","last_undegraded":"2024-02-05T07:21:17.509333+0000","last_fullsized":"2024-02-05T07:21:17.509333+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:19:59.624369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:59.624369+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:28.126457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":83,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.519519+0000","last_change":"2024-02-05T07:18:55.506354+0000","last_active":"2024-02-05T07:21:17.519519+0000","last_peered":"2024-02-05T07:21:17.519519+0000","last_clean":"2024-02-05T07:21:17.519519+0000","last_became_active":"2024-02-05T07:18:55.505375+0000","last_became_peered":"2024-02-05T07:18:55.505375+0000","last_unstale":"2024-02-05T07:21:17.519519+0000","last_undegraded":"2024-02-05T07:21:17.519519+0000","last_fullsized":"2024-02-05T07:21:17.519519+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.507370+0000","last_change":"2024-02-05T07:20:18.795431+0000","last_active":"2024-02-05T07:21:17.507370+0000","last_peered":"2024-02-05T07:21:17.507370+0000","last_clean":"2024-02-05T07:21:17.507370+0000","last_became_active":"2024-02-05T07:18:53.459077+0000","last_became_peered":"2024-02-05T07:18:53.459077+0000","last_unstale":"2024-02-05T07:21:17.507370+0000","last_undegraded":"2024-02-05T07:21:17.507370+0000","last_fullsized":"2024-02-05T07:21:17.507370+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:18.795341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.795341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.506912+0000","last_change":"2024-02-05T07:20:39.898505+0000","last_active":"2024-02-05T07:21:17.506912+0000","last_peered":"2024-02-05T07:21:17.506912+0000","last_clean":"2024-02-05T07:21:17.506912+0000","last_became_active":"2024-02-05T07:18:49.427837+0000","last_became_peered":"2024-02-05T07:18:49.427837+0000","last_unstale":"2024-02-05T07:21:17.506912+0000","last_undegraded":"2024-02-05T07:21:17.506912+0000","last_fullsized":"2024-02-05T07:21:17.506912+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:39.898367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:39.898367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:00.742703+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.513699+0000","last_change":"2024-02-05T07:20:01.841591+0000","last_active":"2024-02-05T07:21:17.513699+0000","last_peered":"2024-02-05T07:21:17.513699+0000","last_clean":"2024-02-05T07:21:17.513699+0000","last_became_active":"2024-02-05T07:18:51.447761+0000","last_became_peered":"2024-02-05T07:18:51.447761+0000","last_unstale":"2024-02-05T07:21:17.513699+0000","last_undegraded":"2024-02-05T07:21:17.513699+0000","last_fullsized":"2024-02-05T07:21:17.513699+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:01.841489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:01.841489+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.516944+0000","last_change":"2024-02-05T07:20:17.755329+0000","last_active":"2024-02-05T07:21:17.516944+0000","last_peered":"2024-02-05T07:21:17.516944+0000","last_clean":"2024-02-05T07:21:17.516944+0000","last_became_active":"2024-02-05T07:18:55.476071+0000","last_became_peered":"2024-02-05T07:18:55.476071+0000","last_unstale":"2024-02-05T07:21:17.516944+0000","last_undegraded":"2024-02-05T07:21:17.516944+0000","last_fullsized":"2024-02-05T07:21:17.516944+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:20:17.755246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:17.755246+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":138,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:19.503641+0000","last_change":"2024-02-05T07:20:22.424747+0000","last_active":"2024-02-05T07:21:19.503641+0000","last_peered":"2024-02-05T07:21:19.503641+0000","last_clean":"2024-02-05T07:21:19.503641+0000","last_became_active":"2024-02-05T07:18:53.461011+0000","last_became_peered":"2024-02-05T07:18:53.461011+0000","last_unstale":"2024-02-05T07:21:19.503641+0000","last_undegraded":"2024-02-05T07:21:19.503641+0000","last_fullsized":"2024-02-05T07:21:19.503641+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:20:22.424647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:22.424647+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":105,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.515332+0000","last_change":"2024-02-05T07:20:25.493793+0000","last_active":"2024-02-05T07:21:17.515332+0000","last_peered":"2024-02-05T07:21:17.515332+0000","last_clean":"2024-02-05T07:21:17.515332+0000","last_became_active":"2024-02-05T07:18:49.431031+0000","last_became_peered":"2024-02-05T07:18:49.431031+0000","last_unstale":"2024-02-05T07:21:17.515332+0000","last_undegraded":"2024-02-05T07:21:17.515332+0000","last_fullsized":"2024-02-05T07:21:17.515332+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:20:25.493710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:25.493710+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:39.620525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.517068+0000","last_change":"2024-02-05T07:20:22.291474+0000","last_active":"2024-02-05T07:21:17.517068+0000","last_peered":"2024-02-05T07:21:17.517068+0000","last_clean":"2024-02-05T07:21:17.517068+0000","last_became_active":"2024-02-05T07:18:51.509918+0000","last_became_peered":"2024-02-05T07:18:51.509918+0000","last_unstale":"2024-02-05T07:21:17.517068+0000","last_undegraded":"2024-02-05T07:21:17.517068+0000","last_fullsized":"2024-02-05T07:21:17.517068+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:20:22.291371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:22.291371+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:50.152194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511817+0000","last_change":"2024-02-05T07:20:03.444274+0000","last_active":"2024-02-05T07:21:17.511817+0000","last_peered":"2024-02-05T07:21:17.511817+0000","last_clean":"2024-02-05T07:21:17.511817+0000","last_became_active":"2024-02-05T07:18:55.466164+0000","last_became_peered":"2024-02-05T07:18:55.466164+0000","last_unstale":"2024-02-05T07:21:17.511817+0000","last_undegraded":"2024-02-05T07:21:17.511817+0000","last_fullsized":"2024-02-05T07:21:17.511817+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:03.444153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:03.444153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:32.621342+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":87,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.518261+0000","last_change":"2024-02-05T07:18:53.457928+0000","last_active":"2024-02-05T07:21:17.518261+0000","last_peered":"2024-02-05T07:21:17.518261+0000","last_clean":"2024-02-05T07:21:17.518261+0000","last_became_active":"2024-02-05T07:18:53.457587+0000","last_became_peered":"2024-02-05T07:18:53.457587+0000","last_unstale":"2024-02-05T07:21:17.518261+0000","last_undegraded":"2024-02-05T07:21:17.518261+0000","last_fullsized":"2024-02-05T07:21:17.518261+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:18.127543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":117,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.513413+0000","last_change":"2024-02-05T07:21:14.255576+0000","last_active":"2024-02-05T07:21:17.513413+0000","last_peered":"2024-02-05T07:21:17.513413+0000","last_clean":"2024-02-05T07:21:17.513413+0000","last_became_active":"2024-02-05T07:21:07.472071+0000","last_became_peered":"2024-02-05T07:21:07.472071+0000","last_unstale":"2024-02-05T07:21:17.513413+0000","last_undegraded":"2024-02-05T07:21:17.513413+0000","last_fullsized":"2024-02-05T07:21:17.513413+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:14.255462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:14.255462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:27.519580+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":118,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514508+0000","last_change":"2024-02-05T07:20:06.366811+0000","last_active":"2024-02-05T07:21:17.514508+0000","last_peered":"2024-02-05T07:21:17.514508+0000","last_clean":"2024-02-05T07:21:17.514508+0000","last_became_active":"2024-02-05T07:18:51.446604+0000","last_became_peered":"2024-02-05T07:18:51.446604+0000","last_unstale":"2024-02-05T07:21:17.514508+0000","last_undegraded":"2024-02-05T07:21:17.514508+0000","last_fullsized":"2024-02-05T07:21:17.514508+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:20:06.366699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.366699+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.515867+0000","last_change":"2024-02-05T07:20:01.693975+0000","last_active":"2024-02-05T07:21:17.515867+0000","last_peered":"2024-02-05T07:21:17.515867+0000","last_clean":"2024-02-05T07:21:17.515867+0000","last_became_active":"2024-02-05T07:18:55.468086+0000","last_became_peered":"2024-02-05T07:18:55.468086+0000","last_unstale":"2024-02-05T07:21:17.515867+0000","last_undegraded":"2024-02-05T07:21:17.515867+0000","last_fullsized":"2024-02-05T07:21:17.515867+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:01.693871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:01.693871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514570+0000","last_change":"2024-02-05T07:20:09.349490+0000","last_active":"2024-02-05T07:21:17.514570+0000","last_peered":"2024-02-05T07:21:17.514570+0000","last_clean":"2024-02-05T07:21:17.514570+0000","last_became_active":"2024-02-05T07:18:53.460307+0000","last_became_peered":"2024-02-05T07:18:53.460307+0000","last_unstale":"2024-02-05T07:21:17.514570+0000","last_undegraded":"2024-02-05T07:21:17.514570+0000","last_fullsized":"2024-02-05T07:21:17.514570+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.349342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.451852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.517994+0000","last_change":"2024-02-05T07:20:19.326489+0000","last_active":"2024-02-05T07:21:17.517994+0000","last_peered":"2024-02-05T07:21:17.517994+0000","last_clean":"2024-02-05T07:21:17.517994+0000","last_became_active":"2024-02-05T07:18:49.434408+0000","last_became_peered":"2024-02-05T07:18:49.434408+0000","last_unstale":"2024-02-05T07:21:17.517994+0000","last_undegraded":"2024-02-05T07:21:17.517994+0000","last_fullsized":"2024-02-05T07:21:17.517994+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:19.326400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:19.326400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:45.594433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":105,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:20.184115+0000","last_change":"2024-02-05T07:21:20.184162+0000","last_active":"2024-02-05T07:21:20.184115+0000","last_peered":"2024-02-05T07:21:20.184115+0000","last_clean":"2024-02-05T07:21:20.184115+0000","last_became_active":"2024-02-05T07:18:51.449083+0000","last_became_peered":"2024-02-05T07:18:51.449083+0000","last_unstale":"2024-02-05T07:21:20.184115+0000","last_undegraded":"2024-02-05T07:21:20.184115+0000","last_fullsized":"2024-02-05T07:21:20.184115+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-02-05T07:21:20.184048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:20.184048+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:41.799065+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:22.138429+0000","last_change":"2024-02-05T07:21:22.138467+0000","last_active":"2024-02-05T07:21:22.138429+0000","last_peered":"2024-02-05T07:21:22.138429+0000","last_clean":"2024-02-05T07:21:22.138429+0000","last_became_active":"2024-02-05T07:18:55.506968+0000","last_became_peered":"2024-02-05T07:18:55.506968+0000","last_unstale":"2024-02-05T07:21:22.138429+0000","last_undegraded":"2024-02-05T07:21:22.138429+0000","last_fullsized":"2024-02-05T07:21:22.138429+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:22.138332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:22.138332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:48.600033+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.573731+0000","last_change":"2024-02-05T07:21:24.573753+0000","last_active":"2024-02-05T07:21:24.573731+0000","last_peered":"2024-02-05T07:21:24.573731+0000","last_clean":"2024-02-05T07:21:24.573731+0000","last_became_active":"2024-02-05T07:18:53.460977+0000","last_became_peered":"2024-02-05T07:18:53.460977+0000","last_unstale":"2024-02-05T07:21:24.573731+0000","last_undegraded":"2024-02-05T07:21:24.573731+0000","last_fullsized":"2024-02-05T07:21:24.573731+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:24.573655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_clean_scrub_stamp":"2024-02-05T07:21:24.573655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:34.653566+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511644+0000","last_change":"2024-02-05T07:19:48.944128+0000","last_active":"2024-02-05T07:21:17.511644+0000","last_peered":"2024-02-05T07:21:17.511644+0000","last_clean":"2024-02-05T07:21:17.511644+0000","last_became_active":"2024-02-05T07:18:49.434560+0000","last_became_peered":"2024-02-05T07:18:49.434560+0000","last_unstale":"2024-02-05T07:21:17.511644+0000","last_undegraded":"2024-02-05T07:21:17.511644+0000","last_fullsized":"2024-02-05T07:21:17.511644+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:48.944028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:48.944028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":118,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:18.095871+0000","last_change":"2024-02-05T07:21:18.095928+0000","last_active":"2024-02-05T07:21:18.095871+0000","last_peered":"2024-02-05T07:21:18.095871+0000","last_clean":"2024-02-05T07:21:18.095871+0000","last_became_active":"2024-02-05T07:18:51.509911+0000","last_became_peered":"2024-02-05T07:18:51.509911+0000","last_unstale":"2024-02-05T07:21:18.095871+0000","last_undegraded":"2024-02-05T07:21:18.095871+0000","last_fullsized":"2024-02-05T07:21:18.095871+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:21:18.095797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:18.095797+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:41.085815+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.513960+0000","last_change":"2024-02-05T07:20:27.243266+0000","last_active":"2024-02-05T07:21:17.513960+0000","last_peered":"2024-02-05T07:21:17.513960+0000","last_clean":"2024-02-05T07:21:17.513960+0000","last_became_active":"2024-02-05T07:18:55.477312+0000","last_became_peered":"2024-02-05T07:18:55.477312+0000","last_unstale":"2024-02-05T07:21:17.513960+0000","last_undegraded":"2024-02-05T07:21:17.513960+0000","last_fullsized":"2024-02-05T07:21:17.513960+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:27.243136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:27.243136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.798135+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":87,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514008+0000","last_change":"2024-02-05T07:18:53.468306+0000","last_active":"2024-02-05T07:21:17.514008+0000","last_peered":"2024-02-05T07:21:17.514008+0000","last_clean":"2024-02-05T07:21:17.514008+0000","last_became_active":"2024-02-05T07:18:53.467986+0000","last_became_peered":"2024-02-05T07:18:53.467986+0000","last_unstale":"2024-02-05T07:21:17.514008+0000","last_undegraded":"2024-02-05T07:21:17.514008+0000","last_fullsized":"2024-02-05T07:21:17.514008+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:20.600638+0000","last_change":"2024-02-05T07:21:20.600659+0000","last_active":"2024-02-05T07:21:20.600638+0000","last_peered":"2024-02-05T07:21:20.600638+0000","last_clean":"2024-02-05T07:21:20.600638+0000","last_became_active":"2024-02-05T07:18:49.438565+0000","last_became_peered":"2024-02-05T07:18:49.438565+0000","last_unstale":"2024-02-05T07:21:20.600638+0000","last_undegraded":"2024-02-05T07:21:20.600638+0000","last_fullsized":"2024-02-05T07:21:20.600638+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:20.600583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:20.600583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:25.878194+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511469+0000","last_change":"2024-02-05T07:21:16.721101+0000","last_active":"2024-02-05T07:21:17.511469+0000","last_peered":"2024-02-05T07:21:17.511469+0000","last_clean":"2024-02-05T07:21:17.511469+0000","last_became_active":"2024-02-05T07:19:16.122059+0000","last_became_peered":"2024-02-05T07:19:16.122059+0000","last_unstale":"2024-02-05T07:21:17.511469+0000","last_undegraded":"2024-02-05T07:21:17.511469+0000","last_fullsized":"2024-02-05T07:21:17.511469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:16.720264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:16.720264+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:45.462570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514418+0000","last_change":"2024-02-05T07:20:27.853088+0000","last_active":"2024-02-05T07:21:17.514418+0000","last_peered":"2024-02-05T07:21:17.514418+0000","last_clean":"2024-02-05T07:21:17.514418+0000","last_became_active":"2024-02-05T07:18:55.478087+0000","last_became_peered":"2024-02-05T07:18:55.478087+0000","last_unstale":"2024-02-05T07:21:17.514418+0000","last_undegraded":"2024-02-05T07:21:17.514418+0000","last_fullsized":"2024-02-05T07:21:17.514418+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:27.852942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:27.852942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:43.443908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":96,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511478+0000","last_change":"2024-02-05T07:20:21.504837+0000","last_active":"2024-02-05T07:21:17.511478+0000","last_peered":"2024-02-05T07:21:17.511478+0000","last_clean":"2024-02-05T07:21:17.511478+0000","last_became_active":"2024-02-05T07:18:53.463620+0000","last_became_peered":"2024-02-05T07:18:53.463620+0000","last_unstale":"2024-02-05T07:21:17.511478+0000","last_undegraded":"2024-02-05T07:21:17.511478+0000","last_fullsized":"2024-02-05T07:21:17.511478+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:21.504764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:21.504764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:43.888422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.506798+0000","last_change":"2024-02-05T07:20:38.840775+0000","last_active":"2024-02-05T07:21:17.506798+0000","last_peered":"2024-02-05T07:21:17.506798+0000","last_clean":"2024-02-05T07:21:17.506798+0000","last_became_active":"2024-02-05T07:18:49.430766+0000","last_became_peered":"2024-02-05T07:18:49.430766+0000","last_unstale":"2024-02-05T07:21:17.506798+0000","last_undegraded":"2024-02-05T07:21:17.506798+0000","last_fullsized":"2024-02-05T07:21:17.506798+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:38.840701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:38.840701+0000","last_clean_scrub_stamp":"2024-02-05T07:20:38.840701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:50.837410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":136,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.507501+0000","last_change":"2024-02-05T07:20:06.870132+0000","last_active":"2024-02-05T07:21:17.507501+0000","last_peered":"2024-02-05T07:21:17.507501+0000","last_clean":"2024-02-05T07:21:17.507501+0000","last_became_active":"2024-02-05T07:19:16.121696+0000","last_became_peered":"2024-02-05T07:19:16.121696+0000","last_unstale":"2024-02-05T07:21:17.507501+0000","last_undegraded":"2024-02-05T07:21:17.507501+0000","last_fullsized":"2024-02-05T07:21:17.507501+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:20:06.870006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.870006+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514661+0000","last_change":"2024-02-05T07:20:20.322398+0000","last_active":"2024-02-05T07:21:17.514661+0000","last_peered":"2024-02-05T07:21:17.514661+0000","last_clean":"2024-02-05T07:21:17.514661+0000","last_became_active":"2024-02-05T07:18:55.477896+0000","last_became_peered":"2024-02-05T07:18:55.477896+0000","last_unstale":"2024-02-05T07:21:17.514661+0000","last_undegraded":"2024-02-05T07:21:17.514661+0000","last_fullsized":"2024-02-05T07:21:17.514661+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:20.322309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:20.322309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:20.952572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":135,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:19.925393+0000","last_change":"2024-02-05T07:20:38.003311+0000","last_active":"2024-02-05T07:21:19.925393+0000","last_peered":"2024-02-05T07:21:19.925393+0000","last_clean":"2024-02-05T07:21:19.925393+0000","last_became_active":"2024-02-05T07:18:53.465287+0000","last_became_peered":"2024-02-05T07:18:53.465287+0000","last_unstale":"2024-02-05T07:21:19.925393+0000","last_undegraded":"2024-02-05T07:21:19.925393+0000","last_fullsized":"2024-02-05T07:21:19.925393+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:20:38.003219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:38.003219+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:03.547570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.515930+0000","last_change":"2024-02-05T07:20:32.839729+0000","last_active":"2024-02-05T07:21:17.515930+0000","last_peered":"2024-02-05T07:21:17.515930+0000","last_clean":"2024-02-05T07:21:17.515930+0000","last_became_active":"2024-02-05T07:18:49.436131+0000","last_became_peered":"2024-02-05T07:18:49.436131+0000","last_unstale":"2024-02-05T07:21:17.515930+0000","last_undegraded":"2024-02-05T07:21:17.515930+0000","last_fullsized":"2024-02-05T07:21:17.515930+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:32.839614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:32.839614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:02.241396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514877+0000","last_change":"2024-02-05T07:18:51.507793+0000","last_active":"2024-02-05T07:21:17.514877+0000","last_peered":"2024-02-05T07:21:17.514877+0000","last_clean":"2024-02-05T07:21:17.514877+0000","last_became_active":"2024-02-05T07:18:51.507491+0000","last_became_peered":"2024-02-05T07:18:51.507491+0000","last_unstale":"2024-02-05T07:21:17.514877+0000","last_undegraded":"2024-02-05T07:21:17.514877+0000","last_fullsized":"2024-02-05T07:21:17.514877+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.507357+0000","last_change":"2024-02-05T07:20:33.860836+0000","last_active":"2024-02-05T07:21:17.507357+0000","last_peered":"2024-02-05T07:21:17.507357+0000","last_clean":"2024-02-05T07:21:17.507357+0000","last_became_active":"2024-02-05T07:18:55.467044+0000","last_became_peered":"2024-02-05T07:18:55.467044+0000","last_unstale":"2024-02-05T07:21:17.507357+0000","last_undegraded":"2024-02-05T07:21:17.507357+0000","last_fullsized":"2024-02-05T07:21:17.507357+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:33.860750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:33.860750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:01.300669+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":127,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:19.956606+0000","last_change":"2024-02-05T07:18:53.465551+0000","last_active":"2024-02-05T07:21:19.956606+0000","last_peered":"2024-02-05T07:21:19.956606+0000","last_clean":"2024-02-05T07:21:19.956606+0000","last_became_active":"2024-02-05T07:18:53.465270+0000","last_became_peered":"2024-02-05T07:18:53.465270+0000","last_unstale":"2024-02-05T07:21:19.956606+0000","last_undegraded":"2024-02-05T07:21:19.956606+0000","last_fullsized":"2024-02-05T07:21:19.956606+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.515711+0000","last_change":"2024-02-05T07:20:33.270501+0000","last_active":"2024-02-05T07:21:17.515711+0000","last_peered":"2024-02-05T07:21:17.515711+0000","last_clean":"2024-02-05T07:21:17.515711+0000","last_became_active":"2024-02-05T07:18:49.434402+0000","last_became_peered":"2024-02-05T07:18:49.434402+0000","last_unstale":"2024-02-05T07:21:17.515711+0000","last_undegraded":"2024-02-05T07:21:17.515711+0000","last_fullsized":"2024-02-05T07:21:17.515711+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:33.270421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:33.270421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:50.495185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":115,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511341+0000","last_change":"2024-02-05T07:20:06.852726+0000","last_active":"2024-02-05T07:21:17.511341+0000","last_peered":"2024-02-05T07:21:17.511341+0000","last_clean":"2024-02-05T07:21:17.511341+0000","last_became_active":"2024-02-05T07:18:51.508551+0000","last_became_peered":"2024-02-05T07:18:51.508551+0000","last_unstale":"2024-02-05T07:21:17.511341+0000","last_undegraded":"2024-02-05T07:21:17.511341+0000","last_fullsized":"2024-02-05T07:21:17.511341+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:20:06.852562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.852562+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:20.949048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.509170+0000","last_change":"2024-02-05T07:20:11.543111+0000","last_active":"2024-02-05T07:21:17.509170+0000","last_peered":"2024-02-05T07:21:17.509170+0000","last_clean":"2024-02-05T07:21:17.509170+0000","last_became_active":"2024-02-05T07:18:55.469044+0000","last_became_peered":"2024-02-05T07:18:55.469044+0000","last_unstale":"2024-02-05T07:21:17.509170+0000","last_undegraded":"2024-02-05T07:21:17.509170+0000","last_fullsized":"2024-02-05T07:21:17.509170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:11.543036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:11.543036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:38.671608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.515510+0000","last_change":"2024-02-05T07:20:02.813432+0000","last_active":"2024-02-05T07:21:17.515510+0000","last_peered":"2024-02-05T07:21:17.515510+0000","last_clean":"2024-02-05T07:21:17.515510+0000","last_became_active":"2024-02-05T07:18:53.463607+0000","last_became_peered":"2024-02-05T07:18:53.463607+0000","last_unstale":"2024-02-05T07:21:17.515510+0000","last_undegraded":"2024-02-05T07:21:17.515510+0000","last_fullsized":"2024-02-05T07:21:17.515510+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.813344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.813344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:30.005634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.516087+0000","last_change":"2024-02-05T07:20:02.673707+0000","last_active":"2024-02-05T07:21:17.516087+0000","last_peered":"2024-02-05T07:21:17.516087+0000","last_clean":"2024-02-05T07:21:17.516087+0000","last_became_active":"2024-02-05T07:18:49.437732+0000","last_became_peered":"2024-02-05T07:18:49.437732+0000","last_unstale":"2024-02-05T07:21:17.516087+0000","last_undegraded":"2024-02-05T07:21:17.516087+0000","last_fullsized":"2024-02-05T07:21:17.516087+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.673592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":107,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.517168+0000","last_change":"2024-02-05T07:18:51.510254+0000","last_active":"2024-02-05T07:21:17.517168+0000","last_peered":"2024-02-05T07:21:17.517168+0000","last_clean":"2024-02-05T07:21:17.517168+0000","last_became_active":"2024-02-05T07:18:51.509917+0000","last_became_peered":"2024-02-05T07:18:51.509917+0000","last_unstale":"2024-02-05T07:21:17.517168+0000","last_undegraded":"2024-02-05T07:21:17.517168+0000","last_fullsized":"2024-02-05T07:21:17.517168+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:18.127629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.516748+0000","last_change":"2024-02-05T07:20:19.732465+0000","last_active":"2024-02-05T07:21:17.516748+0000","last_peered":"2024-02-05T07:21:17.516748+0000","last_clean":"2024-02-05T07:21:17.516748+0000","last_became_active":"2024-02-05T07:18:55.507555+0000","last_became_peered":"2024-02-05T07:18:55.507555+0000","last_unstale":"2024-02-05T07:21:17.516748+0000","last_undegraded":"2024-02-05T07:21:17.516748+0000","last_fullsized":"2024-02-05T07:21:17.516748+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:19.732337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:19.732337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:28.044159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:18.752232+0000","last_change":"2024-02-05T07:21:18.752232+0000","last_active":"2024-02-05T07:21:18.752232+0000","last_peered":"2024-02-05T07:21:18.752232+0000","last_clean":"2024-02-05T07:21:18.752232+0000","last_became_active":"2024-02-05T07:18:53.465591+0000","last_became_peered":"2024-02-05T07:18:53.465591+0000","last_unstale":"2024-02-05T07:21:18.752232+0000","last_undegraded":"2024-02-05T07:21:18.752232+0000","last_fullsized":"2024-02-05T07:21:18.752232+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:18.752135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:18.752135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:47.996163+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":109,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.513191+0000","last_change":"2024-02-05T07:19:55.369939+0000","last_active":"2024-02-05T07:21:17.513191+0000","last_peered":"2024-02-05T07:21:17.513191+0000","last_clean":"2024-02-05T07:21:17.513191+0000","last_became_active":"2024-02-05T07:18:49.433121+0000","last_became_peered":"2024-02-05T07:18:49.433121+0000","last_unstale":"2024-02-05T07:21:17.513191+0000","last_undegraded":"2024-02-05T07:21:17.513191+0000","last_fullsized":"2024-02-05T07:21:17.513191+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:19:55.369800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:55.369800+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":100,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.519342+0000","last_change":"2024-02-05T07:18:51.508489+0000","last_active":"2024-02-05T07:21:17.519342+0000","last_peered":"2024-02-05T07:21:17.519342+0000","last_clean":"2024-02-05T07:21:17.519342+0000","last_became_active":"2024-02-05T07:18:51.507451+0000","last_became_peered":"2024-02-05T07:18:51.507451+0000","last_unstale":"2024-02-05T07:21:17.519342+0000","last_undegraded":"2024-02-05T07:21:17.519342+0000","last_fullsized":"2024-02-05T07:21:17.519342+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"110'3347","reported_seq":3377,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.891138+0000","last_change":"2024-02-05T07:20:45.769630+0000","last_active":"2024-02-05T07:21:24.891138+0000","last_peered":"2024-02-05T07:21:24.891138+0000","last_clean":"2024-02-05T07:21:24.891138+0000","last_became_active":"2024-02-05T07:20:45.769333+0000","last_became_peered":"2024-02-05T07:20:45.769333+0000","last_unstale":"2024-02-05T07:21:24.891138+0000","last_undegraded":"2024-02-05T07:21:24.891138+0000","last_fullsized":"2024-02-05T07:21:24.891138+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3347,"log_dups_size":0,"ondisk_log_size":3347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:09.469881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219348992,"num_objects":3347,"num_object_clones":0,"num_object_copies":10041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3347,"num_write_kb":214208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511806+0000","last_change":"2024-02-05T07:20:37.032149+0000","last_active":"2024-02-05T07:21:17.511806+0000","last_peered":"2024-02-05T07:21:17.511806+0000","last_clean":"2024-02-05T07:21:17.511806+0000","last_became_active":"2024-02-05T07:18:55.505401+0000","last_became_peered":"2024-02-05T07:18:55.505401+0000","last_unstale":"2024-02-05T07:21:17.511806+0000","last_undegraded":"2024-02-05T07:21:17.511806+0000","last_fullsized":"2024-02-05T07:21:17.511806+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:37.032056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:37.032056+0000","last_clean_scrub_stamp":"2024-02-05T07:20:37.032056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:43.632882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":115,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.517108+0000","last_change":"2024-02-05T07:20:13.701938+0000","last_active":"2024-02-05T07:21:17.517108+0000","last_peered":"2024-02-05T07:21:17.517108+0000","last_clean":"2024-02-05T07:21:17.517108+0000","last_became_active":"2024-02-05T07:18:49.437663+0000","last_became_peered":"2024-02-05T07:18:49.437663+0000","last_unstale":"2024-02-05T07:21:17.517108+0000","last_undegraded":"2024-02-05T07:21:17.517108+0000","last_fullsized":"2024-02-05T07:21:17.517108+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:20:13.701852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:13.701852+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:32.496072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":124,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.510894+0000","last_change":"2024-02-05T07:19:58.753612+0000","last_active":"2024-02-05T07:21:17.510894+0000","last_peered":"2024-02-05T07:21:17.510894+0000","last_clean":"2024-02-05T07:21:17.510894+0000","last_became_active":"2024-02-05T07:18:51.508097+0000","last_became_peered":"2024-02-05T07:18:51.508097+0000","last_unstale":"2024-02-05T07:21:17.510894+0000","last_undegraded":"2024-02-05T07:21:17.510894+0000","last_fullsized":"2024-02-05T07:21:17.510894+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:19:58.753472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:58.753472+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:26.081872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":136,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:19.506575+0000","last_change":"2024-02-05T07:20:00.685371+0000","last_active":"2024-02-05T07:21:19.506575+0000","last_peered":"2024-02-05T07:21:19.506575+0000","last_clean":"2024-02-05T07:21:19.506575+0000","last_became_active":"2024-02-05T07:18:53.456287+0000","last_became_peered":"2024-02-05T07:18:53.456287+0000","last_unstale":"2024-02-05T07:21:19.506575+0000","last_undegraded":"2024-02-05T07:21:19.506575+0000","last_fullsized":"2024-02-05T07:21:19.506575+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.685282+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"110'3402","reported_seq":3432,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.913235+0000","last_change":"2024-02-05T07:20:45.772595+0000","last_active":"2024-02-05T07:21:24.913235+0000","last_peered":"2024-02-05T07:21:24.913235+0000","last_clean":"2024-02-05T07:21:24.913235+0000","last_became_active":"2024-02-05T07:20:45.771634+0000","last_became_peered":"2024-02-05T07:20:45.771634+0000","last_unstale":"2024-02-05T07:21:24.913235+0000","last_undegraded":"2024-02-05T07:21:24.913235+0000","last_fullsized":"2024-02-05T07:21:24.913235+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3402,"log_dups_size":0,"ondisk_log_size":3402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:14.707415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222953472,"num_objects":3402,"num_object_clones":0,"num_object_copies":10206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3402,"num_write_kb":217728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514151+0000","last_change":"2024-02-05T07:20:28.882434+0000","last_active":"2024-02-05T07:21:17.514151+0000","last_peered":"2024-02-05T07:21:17.514151+0000","last_clean":"2024-02-05T07:21:17.514151+0000","last_became_active":"2024-02-05T07:18:55.474663+0000","last_became_peered":"2024-02-05T07:18:55.474663+0000","last_unstale":"2024-02-05T07:21:17.514151+0000","last_undegraded":"2024-02-05T07:21:17.514151+0000","last_fullsized":"2024-02-05T07:21:17.514151+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:28.882326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:28.882326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:37.233227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.516561+0000","last_change":"2024-02-05T07:19:51.758261+0000","last_active":"2024-02-05T07:21:17.516561+0000","last_peered":"2024-02-05T07:21:17.516561+0000","last_clean":"2024-02-05T07:21:17.516561+0000","last_became_active":"2024-02-05T07:18:49.435621+0000","last_became_peered":"2024-02-05T07:18:49.435621+0000","last_unstale":"2024-02-05T07:21:17.516561+0000","last_undegraded":"2024-02-05T07:21:17.516561+0000","last_fullsized":"2024-02-05T07:21:17.516561+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:51.758163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.758163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":140,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.507164+0000","last_change":"2024-02-05T07:20:04.885890+0000","last_active":"2024-02-05T07:21:17.507164+0000","last_peered":"2024-02-05T07:21:17.507164+0000","last_clean":"2024-02-05T07:21:17.507164+0000","last_became_active":"2024-02-05T07:19:16.123650+0000","last_became_peered":"2024-02-05T07:19:16.123650+0000","last_unstale":"2024-02-05T07:21:17.507164+0000","last_undegraded":"2024-02-05T07:21:17.507164+0000","last_fullsized":"2024-02-05T07:21:17.507164+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:04.885778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:04.885778+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.509198+0000","last_change":"2024-02-05T07:20:02.694603+0000","last_active":"2024-02-05T07:21:17.509198+0000","last_peered":"2024-02-05T07:21:17.509198+0000","last_clean":"2024-02-05T07:21:17.509198+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:21:17.509198+0000","last_undegraded":"2024-02-05T07:21:17.509198+0000","last_fullsized":"2024-02-05T07:21:17.509198+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.694506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.694506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.700377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"110'3391","reported_seq":3421,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.941182+0000","last_change":"2024-02-05T07:20:45.771257+0000","last_active":"2024-02-05T07:21:24.941182+0000","last_peered":"2024-02-05T07:21:24.941182+0000","last_clean":"2024-02-05T07:21:24.941182+0000","last_became_active":"2024-02-05T07:20:45.770226+0000","last_became_peered":"2024-02-05T07:20:45.770226+0000","last_unstale":"2024-02-05T07:21:24.941182+0000","last_undegraded":"2024-02-05T07:21:24.941182+0000","last_fullsized":"2024-02-05T07:21:24.941182+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:47.503614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222232576,"num_objects":3391,"num_object_clones":0,"num_object_copies":10173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3391,"num_write_kb":217024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.516480+0000","last_change":"2024-02-05T07:20:38.715538+0000","last_active":"2024-02-05T07:21:17.516480+0000","last_peered":"2024-02-05T07:21:17.516480+0000","last_clean":"2024-02-05T07:21:17.516480+0000","last_became_active":"2024-02-05T07:18:55.473032+0000","last_became_peered":"2024-02-05T07:18:55.473032+0000","last_unstale":"2024-02-05T07:21:17.516480+0000","last_undegraded":"2024-02-05T07:21:17.516480+0000","last_fullsized":"2024-02-05T07:21:17.516480+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:38.715447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:38.715447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:58.258241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.509014+0000","last_change":"2024-02-05T07:20:16.476649+0000","last_active":"2024-02-05T07:21:17.509014+0000","last_peered":"2024-02-05T07:21:17.509014+0000","last_clean":"2024-02-05T07:21:17.509014+0000","last_became_active":"2024-02-05T07:18:53.464867+0000","last_became_peered":"2024-02-05T07:18:53.464867+0000","last_unstale":"2024-02-05T07:21:17.509014+0000","last_undegraded":"2024-02-05T07:21:17.509014+0000","last_fullsized":"2024-02-05T07:21:17.509014+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:16.476554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:16.476554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:31.284238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.516843+0000","last_change":"2024-02-05T07:19:55.788134+0000","last_active":"2024-02-05T07:21:17.516843+0000","last_peered":"2024-02-05T07:21:17.516843+0000","last_clean":"2024-02-05T07:21:17.516843+0000","last_became_active":"2024-02-05T07:18:49.434451+0000","last_became_peered":"2024-02-05T07:18:49.434451+0000","last_unstale":"2024-02-05T07:21:17.516843+0000","last_undegraded":"2024-02-05T07:21:17.516843+0000","last_fullsized":"2024-02-05T07:21:17.516843+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:55.788005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:55.788005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.126482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":130,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.513414+0000","last_change":"2024-02-05T07:20:00.384918+0000","last_active":"2024-02-05T07:21:17.513414+0000","last_peered":"2024-02-05T07:21:17.513414+0000","last_clean":"2024-02-05T07:21:17.513414+0000","last_became_active":"2024-02-05T07:18:51.507672+0000","last_became_peered":"2024-02-05T07:18:51.507672+0000","last_unstale":"2024-02-05T07:21:17.513414+0000","last_undegraded":"2024-02-05T07:21:17.513414+0000","last_fullsized":"2024-02-05T07:21:17.513414+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:20:00.384813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.384813+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"110'3311","reported_seq":3341,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.933801+0000","last_change":"2024-02-05T07:20:45.770387+0000","last_active":"2024-02-05T07:21:24.933801+0000","last_peered":"2024-02-05T07:21:24.933801+0000","last_clean":"2024-02-05T07:21:24.933801+0000","last_became_active":"2024-02-05T07:20:45.770103+0000","last_became_peered":"2024-02-05T07:20:45.770103+0000","last_unstale":"2024-02-05T07:21:24.933801+0000","last_undegraded":"2024-02-05T07:21:24.933801+0000","last_fullsized":"2024-02-05T07:21:24.933801+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3311,"log_dups_size":0,"ondisk_log_size":3311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:52.029323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216989696,"num_objects":3311,"num_object_clones":0,"num_object_copies":9933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3311,"num_write_kb":211904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":83,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514075+0000","last_change":"2024-02-05T07:18:55.476043+0000","last_active":"2024-02-05T07:21:17.514075+0000","last_peered":"2024-02-05T07:21:17.514075+0000","last_clean":"2024-02-05T07:21:17.514075+0000","last_became_active":"2024-02-05T07:18:55.475702+0000","last_became_peered":"2024-02-05T07:18:55.475702+0000","last_unstale":"2024-02-05T07:21:17.514075+0000","last_undegraded":"2024-02-05T07:21:17.514075+0000","last_fullsized":"2024-02-05T07:21:17.514075+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:18.127709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.508962+0000","last_change":"2024-02-05T07:20:09.589044+0000","last_active":"2024-02-05T07:21:17.508962+0000","last_peered":"2024-02-05T07:21:17.508962+0000","last_clean":"2024-02-05T07:21:17.508962+0000","last_became_active":"2024-02-05T07:18:53.459560+0000","last_became_peered":"2024-02-05T07:18:53.459560+0000","last_unstale":"2024-02-05T07:21:17.508962+0000","last_undegraded":"2024-02-05T07:21:17.508962+0000","last_fullsized":"2024-02-05T07:21:17.508962+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:09.588893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.588893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:27.848596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511095+0000","last_change":"2024-02-05T07:20:15.066894+0000","last_active":"2024-02-05T07:21:17.511095+0000","last_peered":"2024-02-05T07:21:17.511095+0000","last_clean":"2024-02-05T07:21:17.511095+0000","last_became_active":"2024-02-05T07:18:49.434238+0000","last_became_peered":"2024-02-05T07:18:49.434238+0000","last_unstale":"2024-02-05T07:21:17.511095+0000","last_undegraded":"2024-02-05T07:21:17.511095+0000","last_fullsized":"2024-02-05T07:21:17.511095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:15.066768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:15.066768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:41.133131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":115,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:21.184520+0000","last_change":"2024-02-05T07:21:21.184568+0000","last_active":"2024-02-05T07:21:21.184520+0000","last_peered":"2024-02-05T07:21:21.184520+0000","last_clean":"2024-02-05T07:21:21.184520+0000","last_became_active":"2024-02-05T07:18:51.440744+0000","last_became_peered":"2024-02-05T07:18:51.440744+0000","last_unstale":"2024-02-05T07:21:21.184520+0000","last_undegraded":"2024-02-05T07:21:21.184520+0000","last_fullsized":"2024-02-05T07:21:21.184520+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:21.184445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:21.184445+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:24.021168+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"110'3349","reported_seq":3379,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.904740+0000","last_change":"2024-02-05T07:20:45.769962+0000","last_active":"2024-02-05T07:21:24.904740+0000","last_peered":"2024-02-05T07:21:24.904740+0000","last_clean":"2024-02-05T07:21:24.904740+0000","last_became_active":"2024-02-05T07:20:45.769243+0000","last_became_peered":"2024-02-05T07:20:45.769243+0000","last_unstale":"2024-02-05T07:21:24.904740+0000","last_undegraded":"2024-02-05T07:21:24.904740+0000","last_fullsized":"2024-02-05T07:21:24.904740+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3349,"log_dups_size":0,"ondisk_log_size":3349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:48.274579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219480064,"num_objects":3349,"num_object_clones":0,"num_object_copies":10047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3349,"num_write_kb":214336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":83,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514642+0000","last_change":"2024-02-05T07:18:55.476663+0000","last_active":"2024-02-05T07:21:17.514642+0000","last_peered":"2024-02-05T07:21:17.514642+0000","last_clean":"2024-02-05T07:21:17.514642+0000","last_became_active":"2024-02-05T07:18:55.476313+0000","last_became_peered":"2024-02-05T07:18:55.476313+0000","last_unstale":"2024-02-05T07:21:17.514642+0000","last_undegraded":"2024-02-05T07:21:17.514642+0000","last_fullsized":"2024-02-05T07:21:17.514642+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":115,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511648+0000","last_change":"2024-02-05T07:20:38.993766+0000","last_active":"2024-02-05T07:21:17.511648+0000","last_peered":"2024-02-05T07:21:17.511648+0000","last_clean":"2024-02-05T07:21:17.511648+0000","last_became_active":"2024-02-05T07:18:48.423032+0000","last_became_peered":"2024-02-05T07:18:48.423032+0000","last_unstale":"2024-02-05T07:21:17.511648+0000","last_undegraded":"2024-02-05T07:21:17.511648+0000","last_fullsized":"2024-02-05T07:21:17.511648+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:20:38.993688+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:20:38.993688+0000","last_clean_scrub_stamp":"2024-02-05T07:20:38.993688+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:53.334904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.506776+0000","last_change":"2024-02-05T07:20:32.870565+0000","last_active":"2024-02-05T07:21:17.506776+0000","last_peered":"2024-02-05T07:21:17.506776+0000","last_clean":"2024-02-05T07:21:17.506776+0000","last_became_active":"2024-02-05T07:18:49.430208+0000","last_became_peered":"2024-02-05T07:18:49.430208+0000","last_unstale":"2024-02-05T07:21:17.506776+0000","last_undegraded":"2024-02-05T07:21:17.506776+0000","last_fullsized":"2024-02-05T07:21:17.506776+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:32.870450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:32.870450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:47.599215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":180,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:19.515078+0000","last_change":"2024-02-05T07:20:24.521320+0000","last_active":"2024-02-05T07:21:19.515078+0000","last_peered":"2024-02-05T07:21:19.515078+0000","last_clean":"2024-02-05T07:21:19.515078+0000","last_became_active":"2024-02-05T07:18:51.442110+0000","last_became_peered":"2024-02-05T07:18:51.442110+0000","last_unstale":"2024-02-05T07:21:19.515078+0000","last_undegraded":"2024-02-05T07:21:19.515078+0000","last_fullsized":"2024-02-05T07:21:19.515078+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-02-05T07:20:24.521222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:24.521222+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:36.232296+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":87,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.517759+0000","last_change":"2024-02-05T07:18:53.461030+0000","last_active":"2024-02-05T07:21:17.517759+0000","last_peered":"2024-02-05T07:21:17.517759+0000","last_clean":"2024-02-05T07:21:17.517759+0000","last_became_active":"2024-02-05T07:18:53.460715+0000","last_became_peered":"2024-02-05T07:18:53.460715+0000","last_unstale":"2024-02-05T07:21:17.517759+0000","last_undegraded":"2024-02-05T07:21:17.517759+0000","last_fullsized":"2024-02-05T07:21:17.517759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:18.127654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"110'3326","reported_seq":3356,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.930692+0000","last_change":"2024-02-05T07:20:45.769623+0000","last_active":"2024-02-05T07:21:24.930692+0000","last_peered":"2024-02-05T07:21:24.930692+0000","last_clean":"2024-02-05T07:21:24.930692+0000","last_became_active":"2024-02-05T07:20:45.769357+0000","last_became_peered":"2024-02-05T07:20:45.769357+0000","last_unstale":"2024-02-05T07:21:24.930692+0000","last_undegraded":"2024-02-05T07:21:24.930692+0000","last_fullsized":"2024-02-05T07:21:24.930692+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3326,"log_dups_size":0,"ondisk_log_size":3326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:11.070822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217972736,"num_objects":3326,"num_object_clones":0,"num_object_copies":9978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3326,"num_write_kb":212864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.516963+0000","last_change":"2024-02-05T07:20:06.673182+0000","last_active":"2024-02-05T07:21:17.516963+0000","last_peered":"2024-02-05T07:21:17.516963+0000","last_clean":"2024-02-05T07:21:17.516963+0000","last_became_active":"2024-02-05T07:18:55.506803+0000","last_became_peered":"2024-02-05T07:18:55.506803+0000","last_unstale":"2024-02-05T07:21:17.516963+0000","last_undegraded":"2024-02-05T07:21:17.516963+0000","last_fullsized":"2024-02-05T07:21:17.516963+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:20:06.673081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:21.893489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":110,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.510742+0000","last_change":"2024-02-05T07:20:09.406817+0000","last_active":"2024-02-05T07:21:17.510742+0000","last_peered":"2024-02-05T07:21:17.510742+0000","last_clean":"2024-02-05T07:21:17.510742+0000","last_became_active":"2024-02-05T07:18:49.429651+0000","last_became_peered":"2024-02-05T07:18:49.429651+0000","last_unstale":"2024-02-05T07:21:17.510742+0000","last_undegraded":"2024-02-05T07:21:17.510742+0000","last_fullsized":"2024-02-05T07:21:17.510742+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:20:09.406718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.406718+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:33.441463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":120,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:18.723210+0000","last_change":"2024-02-05T07:21:18.723248+0000","last_active":"2024-02-05T07:21:18.723210+0000","last_peered":"2024-02-05T07:21:18.723210+0000","last_clean":"2024-02-05T07:21:18.723210+0000","last_became_active":"2024-02-05T07:18:48.423639+0000","last_became_peered":"2024-02-05T07:18:48.423639+0000","last_unstale":"2024-02-05T07:21:18.723210+0000","last_undegraded":"2024-02-05T07:21:18.723210+0000","last_fullsized":"2024-02-05T07:21:18.723210+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:18.723118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:21:18.723118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:48.295265+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"96'18","reported_seq":132,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511085+0000","last_change":"2024-02-05T07:20:28.409634+0000","last_active":"2024-02-05T07:21:17.511085+0000","last_peered":"2024-02-05T07:21:17.511085+0000","last_clean":"2024-02-05T07:21:17.511085+0000","last_became_active":"2024-02-05T07:18:51.436097+0000","last_became_peered":"2024-02-05T07:18:51.436097+0000","last_unstale":"2024-02-05T07:21:17.511085+0000","last_undegraded":"2024-02-05T07:21:17.511085+0000","last_fullsized":"2024-02-05T07:21:17.511085+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'18","last_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:20:28.409472+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:53.525418+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.509507+0000","last_change":"2024-02-05T07:20:10.558042+0000","last_active":"2024-02-05T07:21:17.509507+0000","last_peered":"2024-02-05T07:21:17.509507+0000","last_clean":"2024-02-05T07:21:17.509507+0000","last_became_active":"2024-02-05T07:18:53.460771+0000","last_became_peered":"2024-02-05T07:18:53.460771+0000","last_unstale":"2024-02-05T07:21:17.509507+0000","last_undegraded":"2024-02-05T07:21:17.509507+0000","last_fullsized":"2024-02-05T07:21:17.509507+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:10.557948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:10.557948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:27.157314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"110'3403","reported_seq":3433,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.952745+0000","last_change":"2024-02-05T07:20:45.768490+0000","last_active":"2024-02-05T07:21:24.952745+0000","last_peered":"2024-02-05T07:21:24.952745+0000","last_clean":"2024-02-05T07:21:24.952745+0000","last_became_active":"2024-02-05T07:20:45.768210+0000","last_became_peered":"2024-02-05T07:20:45.768210+0000","last_unstale":"2024-02-05T07:21:24.952745+0000","last_undegraded":"2024-02-05T07:21:24.952745+0000","last_fullsized":"2024-02-05T07:21:24.952745+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3403,"log_dups_size":0,"ondisk_log_size":3403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:13.003579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223019008,"num_objects":3403,"num_object_clones":0,"num_object_copies":10209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3403,"num_write_kb":217792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511271+0000","last_change":"2024-02-05T07:20:35.085265+0000","last_active":"2024-02-05T07:21:17.511271+0000","last_peered":"2024-02-05T07:21:17.511271+0000","last_clean":"2024-02-05T07:21:17.511271+0000","last_became_active":"2024-02-05T07:18:55.473358+0000","last_became_peered":"2024-02-05T07:18:55.473358+0000","last_unstale":"2024-02-05T07:21:17.511271+0000","last_undegraded":"2024-02-05T07:21:17.511271+0000","last_fullsized":"2024-02-05T07:21:17.511271+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:35.085136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:35.085136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:43.102492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":360,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.516769+0000","last_change":"2024-02-05T07:20:31.816346+0000","last_active":"2024-02-05T07:21:17.516769+0000","last_peered":"2024-02-05T07:21:17.516769+0000","last_clean":"2024-02-05T07:21:17.516769+0000","last_became_active":"2024-02-05T07:18:39.444898+0000","last_became_peered":"2024-02-05T07:18:39.444898+0000","last_unstale":"2024-02-05T07:21:17.516769+0000","last_undegraded":"2024-02-05T07:21:17.516769+0000","last_fullsized":"2024-02-05T07:21:17.516769+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:20:31.816160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:20:31.816160+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:40.393220+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.516463+0000","last_change":"2024-02-05T07:20:29.851392+0000","last_active":"2024-02-05T07:21:17.516463+0000","last_peered":"2024-02-05T07:21:17.516463+0000","last_clean":"2024-02-05T07:21:17.516463+0000","last_became_active":"2024-02-05T07:18:49.439249+0000","last_became_peered":"2024-02-05T07:18:49.439249+0000","last_unstale":"2024-02-05T07:21:17.516463+0000","last_undegraded":"2024-02-05T07:21:17.516463+0000","last_fullsized":"2024-02-05T07:21:17.516463+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:29.851303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:29.851303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:33.322985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":153,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:19.503781+0000","last_change":"2024-02-05T07:20:25.394750+0000","last_active":"2024-02-05T07:21:19.503781+0000","last_peered":"2024-02-05T07:21:19.503781+0000","last_clean":"2024-02-05T07:21:19.503781+0000","last_became_active":"2024-02-05T07:18:51.507714+0000","last_became_peered":"2024-02-05T07:18:51.507714+0000","last_unstale":"2024-02-05T07:21:19.503781+0000","last_undegraded":"2024-02-05T07:21:19.503781+0000","last_fullsized":"2024-02-05T07:21:19.503781+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-02-05T07:20:25.394654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:25.394654+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:52.134692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.516985+0000","last_change":"2024-02-05T07:21:15.698783+0000","last_active":"2024-02-05T07:21:17.516985+0000","last_peered":"2024-02-05T07:21:17.516985+0000","last_clean":"2024-02-05T07:21:17.516985+0000","last_became_active":"2024-02-05T07:18:53.459971+0000","last_became_peered":"2024-02-05T07:18:53.459971+0000","last_unstale":"2024-02-05T07:21:17.516985+0000","last_undegraded":"2024-02-05T07:21:17.516985+0000","last_fullsized":"2024-02-05T07:21:17.516985+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:15.698665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:15.698665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:35.606774+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"110'3334","reported_seq":3364,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.898983+0000","last_change":"2024-02-05T07:20:45.773522+0000","last_active":"2024-02-05T07:21:24.898983+0000","last_peered":"2024-02-05T07:21:24.898983+0000","last_clean":"2024-02-05T07:21:24.898983+0000","last_became_active":"2024-02-05T07:20:45.773300+0000","last_became_peered":"2024-02-05T07:20:45.773300+0000","last_unstale":"2024-02-05T07:21:24.898983+0000","last_undegraded":"2024-02-05T07:21:24.898983+0000","last_fullsized":"2024-02-05T07:21:24.898983+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3334,"log_dups_size":0,"ondisk_log_size":3334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:13.438443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218497024,"num_objects":3334,"num_object_clones":0,"num_object_copies":10002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3334,"num_write_kb":213376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":93,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.512097+0000","last_change":"2024-02-05T07:20:37.499849+0000","last_active":"2024-02-05T07:21:17.512097+0000","last_peered":"2024-02-05T07:21:17.512097+0000","last_clean":"2024-02-05T07:21:17.512097+0000","last_became_active":"2024-02-05T07:18:55.473509+0000","last_became_peered":"2024-02-05T07:18:55.473509+0000","last_unstale":"2024-02-05T07:21:17.512097+0000","last_undegraded":"2024-02-05T07:21:17.512097+0000","last_fullsized":"2024-02-05T07:21:17.512097+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:20:37.499768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:37.499768+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:59.238955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514344+0000","last_change":"2024-02-05T07:20:21.318287+0000","last_active":"2024-02-05T07:21:17.514344+0000","last_peered":"2024-02-05T07:21:17.514344+0000","last_clean":"2024-02-05T07:21:17.514344+0000","last_became_active":"2024-02-05T07:18:49.434400+0000","last_became_peered":"2024-02-05T07:18:49.434400+0000","last_unstale":"2024-02-05T07:21:17.514344+0000","last_undegraded":"2024-02-05T07:21:17.514344+0000","last_fullsized":"2024-02-05T07:21:17.514344+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:21.318169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:21.318169+0000","last_clean_scrub_stamp":"2024-02-05T07:20:21.318169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:29.238151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":263,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:23.366985+0000","last_change":"2024-02-05T07:20:34.765770+0000","last_active":"2024-02-05T07:21:23.366985+0000","last_peered":"2024-02-05T07:21:23.366985+0000","last_clean":"2024-02-05T07:21:23.366985+0000","last_became_active":"2024-02-05T07:18:48.424620+0000","last_became_peered":"2024-02-05T07:18:48.424620+0000","last_unstale":"2024-02-05T07:21:23.366985+0000","last_undegraded":"2024-02-05T07:21:23.366985+0000","last_fullsized":"2024-02-05T07:21:23.366985+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:20:34.765664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:20:34.765664+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:54.764530+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":148,"num_read_kb":143,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":117,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514713+0000","last_change":"2024-02-05T07:20:14.321610+0000","last_active":"2024-02-05T07:21:17.514713+0000","last_peered":"2024-02-05T07:21:17.514713+0000","last_clean":"2024-02-05T07:21:17.514713+0000","last_became_active":"2024-02-05T07:18:51.442184+0000","last_became_peered":"2024-02-05T07:18:51.442184+0000","last_unstale":"2024-02-05T07:21:17.514713+0000","last_undegraded":"2024-02-05T07:21:17.514713+0000","last_fullsized":"2024-02-05T07:21:17.514713+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:14.321523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:14.321523+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.745540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511669+0000","last_change":"2024-02-05T07:20:31.238357+0000","last_active":"2024-02-05T07:21:17.511669+0000","last_peered":"2024-02-05T07:21:17.511669+0000","last_clean":"2024-02-05T07:21:17.511669+0000","last_became_active":"2024-02-05T07:18:53.468512+0000","last_became_peered":"2024-02-05T07:18:53.468512+0000","last_unstale":"2024-02-05T07:21:17.511669+0000","last_undegraded":"2024-02-05T07:21:17.511669+0000","last_fullsized":"2024-02-05T07:21:17.511669+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:31.238251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:31.238251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:43.265894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"110'3317","reported_seq":3345,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.908426+0000","last_change":"2024-02-05T07:20:45.772020+0000","last_active":"2024-02-05T07:21:24.908426+0000","last_peered":"2024-02-05T07:21:24.908426+0000","last_clean":"2024-02-05T07:21:24.908426+0000","last_became_active":"2024-02-05T07:20:45.771657+0000","last_became_peered":"2024-02-05T07:20:45.771657+0000","last_unstale":"2024-02-05T07:21:24.908426+0000","last_undegraded":"2024-02-05T07:21:24.908426+0000","last_fullsized":"2024-02-05T07:21:24.908426+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3317,"log_dups_size":0,"ondisk_log_size":3317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:58.590875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217382912,"num_objects":3317,"num_object_clones":0,"num_object_copies":9951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3317,"num_write_kb":212288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.515141+0000","last_change":"2024-02-05T07:20:20.526231+0000","last_active":"2024-02-05T07:21:17.515141+0000","last_peered":"2024-02-05T07:21:17.515141+0000","last_clean":"2024-02-05T07:21:17.515141+0000","last_became_active":"2024-02-05T07:18:55.505360+0000","last_became_peered":"2024-02-05T07:18:55.505360+0000","last_unstale":"2024-02-05T07:21:17.515141+0000","last_undegraded":"2024-02-05T07:21:17.515141+0000","last_fullsized":"2024-02-05T07:21:17.515141+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:20.526102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:20.526102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:49.916022+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.512310+0000","last_change":"2024-02-05T07:20:10.436628+0000","last_active":"2024-02-05T07:21:17.512310+0000","last_peered":"2024-02-05T07:21:17.512310+0000","last_clean":"2024-02-05T07:21:17.512310+0000","last_became_active":"2024-02-05T07:18:49.428470+0000","last_became_peered":"2024-02-05T07:18:49.428470+0000","last_unstale":"2024-02-05T07:21:17.512310+0000","last_undegraded":"2024-02-05T07:21:17.512310+0000","last_fullsized":"2024-02-05T07:21:17.512310+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:20:10.436535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:36.573911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":135,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.516279+0000","last_change":"2024-02-05T07:20:12.309384+0000","last_active":"2024-02-05T07:21:17.516279+0000","last_peered":"2024-02-05T07:21:17.516279+0000","last_clean":"2024-02-05T07:21:17.516279+0000","last_became_active":"2024-02-05T07:18:51.507507+0000","last_became_peered":"2024-02-05T07:18:51.507507+0000","last_unstale":"2024-02-05T07:21:17.516279+0000","last_undegraded":"2024-02-05T07:21:17.516279+0000","last_fullsized":"2024-02-05T07:21:17.516279+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:20:12.309238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:12.309238+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:40.996099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.513418+0000","last_change":"2024-02-05T07:19:56.330173+0000","last_active":"2024-02-05T07:21:17.513418+0000","last_peered":"2024-02-05T07:21:17.513418+0000","last_clean":"2024-02-05T07:21:17.513418+0000","last_became_active":"2024-02-05T07:18:53.459474+0000","last_became_peered":"2024-02-05T07:18:53.459474+0000","last_unstale":"2024-02-05T07:21:17.513418+0000","last_undegraded":"2024-02-05T07:21:17.513418+0000","last_fullsized":"2024-02-05T07:21:17.513418+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:19:56.330068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:19:56.330068+0000","last_clean_scrub_stamp":"2024-02-05T07:19:56.330068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:22.066765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"110'3420","reported_seq":3449,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.915634+0000","last_change":"2024-02-05T07:20:45.770489+0000","last_active":"2024-02-05T07:21:24.915634+0000","last_peered":"2024-02-05T07:21:24.915634+0000","last_clean":"2024-02-05T07:21:24.915634+0000","last_became_active":"2024-02-05T07:20:45.770157+0000","last_became_peered":"2024-02-05T07:20:45.770157+0000","last_unstale":"2024-02-05T07:21:24.915634+0000","last_undegraded":"2024-02-05T07:21:24.915634+0000","last_fullsized":"2024-02-05T07:21:24.915634+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3420,"log_dups_size":0,"ondisk_log_size":3420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:12.871064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224133120,"num_objects":3420,"num_object_clones":0,"num_object_copies":10260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3420,"num_write_kb":218880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":83,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.515141+0000","last_change":"2024-02-05T07:18:55.474472+0000","last_active":"2024-02-05T07:21:17.515141+0000","last_peered":"2024-02-05T07:21:17.515141+0000","last_clean":"2024-02-05T07:21:17.515141+0000","last_became_active":"2024-02-05T07:18:55.474029+0000","last_became_peered":"2024-02-05T07:18:55.474029+0000","last_unstale":"2024-02-05T07:21:17.515141+0000","last_undegraded":"2024-02-05T07:21:17.515141+0000","last_fullsized":"2024-02-05T07:21:17.515141+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:18.127574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511205+0000","last_change":"2024-02-05T07:20:32.197801+0000","last_active":"2024-02-05T07:21:17.511205+0000","last_peered":"2024-02-05T07:21:17.511205+0000","last_clean":"2024-02-05T07:21:17.511205+0000","last_became_active":"2024-02-05T07:18:49.435299+0000","last_became_peered":"2024-02-05T07:18:49.435299+0000","last_unstale":"2024-02-05T07:21:17.511205+0000","last_undegraded":"2024-02-05T07:21:17.511205+0000","last_fullsized":"2024-02-05T07:21:17.511205+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:20:32.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:41.939226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":115,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.515631+0000","last_change":"2024-02-05T07:20:22.461850+0000","last_active":"2024-02-05T07:21:17.515631+0000","last_peered":"2024-02-05T07:21:17.515631+0000","last_clean":"2024-02-05T07:21:17.515631+0000","last_became_active":"2024-02-05T07:18:51.448044+0000","last_became_peered":"2024-02-05T07:18:51.448044+0000","last_unstale":"2024-02-05T07:21:17.515631+0000","last_undegraded":"2024-02-05T07:21:17.515631+0000","last_fullsized":"2024-02-05T07:21:17.515631+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:20:22.461692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:22.461692+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:31.055528+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":127,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:19.504388+0000","last_change":"2024-02-05T07:18:53.463629+0000","last_active":"2024-02-05T07:21:19.504388+0000","last_peered":"2024-02-05T07:21:19.504388+0000","last_clean":"2024-02-05T07:21:19.504388+0000","last_became_active":"2024-02-05T07:18:53.462890+0000","last_became_peered":"2024-02-05T07:18:53.462890+0000","last_unstale":"2024-02-05T07:21:19.504388+0000","last_undegraded":"2024-02-05T07:21:19.504388+0000","last_fullsized":"2024-02-05T07:21:19.504388+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"110'3395","reported_seq":3425,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.948432+0000","last_change":"2024-02-05T07:20:45.771222+0000","last_active":"2024-02-05T07:21:24.948432+0000","last_peered":"2024-02-05T07:21:24.948432+0000","last_clean":"2024-02-05T07:21:24.948432+0000","last_became_active":"2024-02-05T07:20:45.770883+0000","last_became_peered":"2024-02-05T07:20:45.770883+0000","last_unstale":"2024-02-05T07:21:24.948432+0000","last_undegraded":"2024-02-05T07:21:24.948432+0000","last_fullsized":"2024-02-05T07:21:24.948432+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3395,"log_dups_size":0,"ondisk_log_size":3395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:44.850626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222494720,"num_objects":3395,"num_object_clones":0,"num_object_copies":10185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3395,"num_write_kb":217280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:22.584880+0000","last_change":"2024-02-05T07:21:22.584921+0000","last_active":"2024-02-05T07:21:22.584880+0000","last_peered":"2024-02-05T07:21:22.584880+0000","last_clean":"2024-02-05T07:21:22.584880+0000","last_became_active":"2024-02-05T07:18:55.507791+0000","last_became_peered":"2024-02-05T07:18:55.507791+0000","last_unstale":"2024-02-05T07:21:22.584880+0000","last_undegraded":"2024-02-05T07:21:22.584880+0000","last_fullsized":"2024-02-05T07:21:22.584880+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:22.584770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:22.584770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:22.752350+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.514805+0000","last_change":"2024-02-05T07:19:51.364668+0000","last_active":"2024-02-05T07:21:17.514805+0000","last_peered":"2024-02-05T07:21:17.514805+0000","last_clean":"2024-02-05T07:21:17.514805+0000","last_became_active":"2024-02-05T07:18:49.430123+0000","last_became_peered":"2024-02-05T07:18:49.430123+0000","last_unstale":"2024-02-05T07:21:17.514805+0000","last_undegraded":"2024-02-05T07:21:17.514805+0000","last_fullsized":"2024-02-05T07:21:17.514805+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:19:51.364585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.364585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:18.127738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":120,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:19.981410+0000","last_change":"2024-02-05T07:21:19.981440+0000","last_active":"2024-02-05T07:21:19.981410+0000","last_peered":"2024-02-05T07:21:19.981410+0000","last_clean":"2024-02-05T07:21:19.981410+0000","last_became_active":"2024-02-05T07:18:51.438507+0000","last_became_peered":"2024-02-05T07:18:51.438507+0000","last_unstale":"2024-02-05T07:21:19.981410+0000","last_undegraded":"2024-02-05T07:21:19.981410+0000","last_fullsized":"2024-02-05T07:21:19.981410+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-02-05T07:21:19.981349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:19.981349+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:40.260981+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":127,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:19.507707+0000","last_change":"2024-02-05T07:18:53.464137+0000","last_active":"2024-02-05T07:21:19.507707+0000","last_peered":"2024-02-05T07:21:19.507707+0000","last_clean":"2024-02-05T07:21:19.507707+0000","last_became_active":"2024-02-05T07:18:53.463651+0000","last_became_peered":"2024-02-05T07:18:53.463651+0000","last_unstale":"2024-02-05T07:21:19.507707+0000","last_undegraded":"2024-02-05T07:21:19.507707+0000","last_fullsized":"2024-02-05T07:21:19.507707+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"110'3365","reported_seq":3395,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.940290+0000","last_change":"2024-02-05T07:20:45.771988+0000","last_active":"2024-02-05T07:21:24.940290+0000","last_peered":"2024-02-05T07:21:24.940290+0000","last_clean":"2024-02-05T07:21:24.940290+0000","last_became_active":"2024-02-05T07:20:45.771721+0000","last_became_peered":"2024-02-05T07:20:45.771721+0000","last_unstale":"2024-02-05T07:21:24.940290+0000","last_undegraded":"2024-02-05T07:21:24.940290+0000","last_fullsized":"2024-02-05T07:21:24.940290+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3365,"log_dups_size":0,"ondisk_log_size":3365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:01.646020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220528640,"num_objects":3365,"num_object_clones":0,"num_object_copies":10095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3365,"num_write_kb":215360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.506680+0000","last_change":"2024-02-05T07:20:30.893200+0000","last_active":"2024-02-05T07:21:17.506680+0000","last_peered":"2024-02-05T07:21:17.506680+0000","last_clean":"2024-02-05T07:21:17.506680+0000","last_became_active":"2024-02-05T07:18:55.474447+0000","last_became_peered":"2024-02-05T07:18:55.474447+0000","last_unstale":"2024-02-05T07:21:17.506680+0000","last_undegraded":"2024-02-05T07:21:17.506680+0000","last_fullsized":"2024-02-05T07:21:17.506680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:30.893106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:30.893106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:50.758530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.515314+0000","last_change":"2024-02-05T07:18:49.435071+0000","last_active":"2024-02-05T07:21:17.515314+0000","last_peered":"2024-02-05T07:21:17.515314+0000","last_clean":"2024-02-05T07:21:17.515314+0000","last_became_active":"2024-02-05T07:18:49.434719+0000","last_became_peered":"2024-02-05T07:18:49.434719+0000","last_unstale":"2024-02-05T07:21:17.515314+0000","last_undegraded":"2024-02-05T07:21:17.515314+0000","last_fullsized":"2024-02-05T07:21:17.515314+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":131,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.513626+0000","last_change":"2024-02-05T07:20:00.797048+0000","last_active":"2024-02-05T07:21:17.513626+0000","last_peered":"2024-02-05T07:21:17.513626+0000","last_clean":"2024-02-05T07:21:17.513626+0000","last_became_active":"2024-02-05T07:18:51.507524+0000","last_became_peered":"2024-02-05T07:18:51.507524+0000","last_unstale":"2024-02-05T07:21:17.513626+0000","last_undegraded":"2024-02-05T07:21:17.513626+0000","last_fullsized":"2024-02-05T07:21:17.513626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:20:00.796905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.796905+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:18.207747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.506598+0000","last_change":"2024-02-05T07:20:24.864128+0000","last_active":"2024-02-05T07:21:17.506598+0000","last_peered":"2024-02-05T07:21:17.506598+0000","last_clean":"2024-02-05T07:21:17.506598+0000","last_became_active":"2024-02-05T07:18:53.461213+0000","last_became_peered":"2024-02-05T07:18:53.461213+0000","last_unstale":"2024-02-05T07:21:17.506598+0000","last_undegraded":"2024-02-05T07:21:17.506598+0000","last_fullsized":"2024-02-05T07:21:17.506598+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:24.864057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:24.864057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:40.694174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"110'3391","reported_seq":3421,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.877226+0000","last_change":"2024-02-05T07:20:45.769769+0000","last_active":"2024-02-05T07:21:24.877226+0000","last_peered":"2024-02-05T07:21:24.877226+0000","last_clean":"2024-02-05T07:21:24.877226+0000","last_became_active":"2024-02-05T07:20:45.769441+0000","last_became_peered":"2024-02-05T07:20:45.769441+0000","last_unstale":"2024-02-05T07:21:24.877226+0000","last_undegraded":"2024-02-05T07:21:24.877226+0000","last_fullsized":"2024-02-05T07:21:24.877226+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:00.119553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222232576,"num_objects":3391,"num_object_clones":0,"num_object_copies":10173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3391,"num_write_kb":217024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511070+0000","last_change":"2024-02-05T07:20:30.285373+0000","last_active":"2024-02-05T07:21:17.511070+0000","last_peered":"2024-02-05T07:21:17.511070+0000","last_clean":"2024-02-05T07:21:17.511070+0000","last_became_active":"2024-02-05T07:18:55.472172+0000","last_became_peered":"2024-02-05T07:18:55.472172+0000","last_unstale":"2024-02-05T07:21:17.511070+0000","last_undegraded":"2024-02-05T07:21:17.511070+0000","last_fullsized":"2024-02-05T07:21:17.511070+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:30.285296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:30.285296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:48.221271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":135,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:19.506632+0000","last_change":"2024-02-05T07:21:14.009133+0000","last_active":"2024-02-05T07:21:19.506632+0000","last_peered":"2024-02-05T07:21:19.506632+0000","last_clean":"2024-02-05T07:21:19.506632+0000","last_became_active":"2024-02-05T07:18:53.459832+0000","last_became_peered":"2024-02-05T07:18:53.459832+0000","last_unstale":"2024-02-05T07:21:19.506632+0000","last_undegraded":"2024-02-05T07:21:19.506632+0000","last_fullsized":"2024-02-05T07:21:19.506632+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:21:14.008994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:14.008994+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:24.575638+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.513542+0000","last_change":"2024-02-05T07:18:49.431779+0000","last_active":"2024-02-05T07:21:17.513542+0000","last_peered":"2024-02-05T07:21:17.513542+0000","last_clean":"2024-02-05T07:21:17.513542+0000","last_became_active":"2024-02-05T07:18:49.431369+0000","last_became_peered":"2024-02-05T07:18:49.431369+0000","last_unstale":"2024-02-05T07:21:17.513542+0000","last_undegraded":"2024-02-05T07:21:17.513542+0000","last_fullsized":"2024-02-05T07:21:17.513542+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:18:48.410364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"98'22","reported_seq":132,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.511234+0000","last_change":"2024-02-05T07:20:36.467017+0000","last_active":"2024-02-05T07:21:17.511234+0000","last_peered":"2024-02-05T07:21:17.511234+0000","last_clean":"2024-02-05T07:21:17.511234+0000","last_became_active":"2024-02-05T07:18:51.447469+0000","last_became_peered":"2024-02-05T07:18:51.447469+0000","last_unstale":"2024-02-05T07:21:17.511234+0000","last_undegraded":"2024-02-05T07:21:17.511234+0000","last_fullsized":"2024-02-05T07:21:17.511234+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'22","last_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:20:36.466914+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:54.868534+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"110'3417","reported_seq":3447,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.910758+0000","last_change":"2024-02-05T07:20:45.765039+0000","last_active":"2024-02-05T07:21:24.910758+0000","last_peered":"2024-02-05T07:21:24.910758+0000","last_clean":"2024-02-05T07:21:24.910758+0000","last_became_active":"2024-02-05T07:20:45.764732+0000","last_became_peered":"2024-02-05T07:20:45.764732+0000","last_unstale":"2024-02-05T07:21:24.910758+0000","last_undegraded":"2024-02-05T07:21:24.910758+0000","last_fullsized":"2024-02-05T07:21:24.910758+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:53.225984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223936512,"num_objects":3417,"num_object_clones":0,"num_object_copies":10251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3417,"num_write_kb":218688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.513908+0000","last_change":"2024-02-05T07:20:10.764040+0000","last_active":"2024-02-05T07:21:17.513908+0000","last_peered":"2024-02-05T07:21:17.513908+0000","last_clean":"2024-02-05T07:21:17.513908+0000","last_became_active":"2024-02-05T07:18:55.505340+0000","last_became_peered":"2024-02-05T07:18:55.505340+0000","last_unstale":"2024-02-05T07:21:17.513908+0000","last_undegraded":"2024-02-05T07:21:17.513908+0000","last_fullsized":"2024-02-05T07:21:17.513908+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:10.763957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:10.763957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:35.465090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:23.823890+0000","last_change":"2024-02-05T07:21:23.823912+0000","last_active":"2024-02-05T07:21:23.823890+0000","last_peered":"2024-02-05T07:21:23.823890+0000","last_clean":"2024-02-05T07:21:23.823890+0000","last_became_active":"2024-02-05T07:18:53.462591+0000","last_became_peered":"2024-02-05T07:18:53.462591+0000","last_unstale":"2024-02-05T07:21:23.823890+0000","last_undegraded":"2024-02-05T07:21:23.823890+0000","last_fullsized":"2024-02-05T07:21:23.823890+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:23.823835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:23.823835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:31.007968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.506612+0000","last_change":"2024-02-05T07:20:05.883941+0000","last_active":"2024-02-05T07:21:17.506612+0000","last_peered":"2024-02-05T07:21:17.506612+0000","last_clean":"2024-02-05T07:21:17.506612+0000","last_became_active":"2024-02-05T07:18:49.434074+0000","last_became_peered":"2024-02-05T07:18:49.434074+0000","last_unstale":"2024-02-05T07:21:17.506612+0000","last_undegraded":"2024-02-05T07:21:17.506612+0000","last_fullsized":"2024-02-05T07:21:17.506612+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:05.883840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:05.883840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:30.747182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"90'16","reported_seq":123,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.507073+0000","last_change":"2024-02-05T07:20:09.873512+0000","last_active":"2024-02-05T07:21:17.507073+0000","last_peered":"2024-02-05T07:21:17.507073+0000","last_clean":"2024-02-05T07:21:17.507073+0000","last_became_active":"2024-02-05T07:18:51.447922+0000","last_became_peered":"2024-02-05T07:18:51.447922+0000","last_unstale":"2024-02-05T07:21:17.507073+0000","last_undegraded":"2024-02-05T07:21:17.507073+0000","last_fullsized":"2024-02-05T07:21:17.507073+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.873388+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:25.334535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"110'3331","reported_seq":3361,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.928390+0000","last_change":"2024-02-05T07:20:45.771675+0000","last_active":"2024-02-05T07:21:24.928390+0000","last_peered":"2024-02-05T07:21:24.928390+0000","last_clean":"2024-02-05T07:21:24.928390+0000","last_became_active":"2024-02-05T07:20:45.771415+0000","last_became_peered":"2024-02-05T07:20:45.771415+0000","last_unstale":"2024-02-05T07:21:24.928390+0000","last_undegraded":"2024-02-05T07:21:24.928390+0000","last_fullsized":"2024-02-05T07:21:24.928390+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3331,"log_dups_size":0,"ondisk_log_size":3331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:59.667107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218300416,"num_objects":3331,"num_object_clones":0,"num_object_copies":9993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3331,"num_write_kb":213184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.509582+0000","last_change":"2024-02-05T07:20:29.670971+0000","last_active":"2024-02-05T07:21:17.509582+0000","last_peered":"2024-02-05T07:21:17.509582+0000","last_clean":"2024-02-05T07:21:17.509582+0000","last_became_active":"2024-02-05T07:18:55.474992+0000","last_became_peered":"2024-02-05T07:18:55.474992+0000","last_unstale":"2024-02-05T07:21:17.509582+0000","last_undegraded":"2024-02-05T07:21:17.509582+0000","last_fullsized":"2024-02-05T07:21:17.509582+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:29.670866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:29.670866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:40.071312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.512545+0000","last_change":"2024-02-05T07:21:17.402214+0000","last_active":"2024-02-05T07:21:17.512545+0000","last_peered":"2024-02-05T07:21:17.512545+0000","last_clean":"2024-02-05T07:21:17.512545+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:21:17.512545+0000","last_undegraded":"2024-02-05T07:21:17.512545+0000","last_fullsized":"2024-02-05T07:21:17.512545+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:17.401959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:17.401959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:22:44.473476+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.513324+0000","last_change":"2024-02-05T07:20:18.399191+0000","last_active":"2024-02-05T07:21:17.513324+0000","last_peered":"2024-02-05T07:21:17.513324+0000","last_clean":"2024-02-05T07:21:17.513324+0000","last_became_active":"2024-02-05T07:18:49.430518+0000","last_became_peered":"2024-02-05T07:18:49.430518+0000","last_unstale":"2024-02-05T07:21:17.513324+0000","last_undegraded":"2024-02-05T07:21:17.513324+0000","last_fullsized":"2024-02-05T07:21:17.513324+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:18.399075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.399075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:36.367477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":127,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.507065+0000","last_change":"2024-02-05T07:19:51.808315+0000","last_active":"2024-02-05T07:21:17.507065+0000","last_peered":"2024-02-05T07:21:17.507065+0000","last_clean":"2024-02-05T07:21:17.507065+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:21:17.507065+0000","last_undegraded":"2024-02-05T07:21:17.507065+0000","last_fullsized":"2024-02-05T07:21:17.507065+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:19:51.808218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:19:51.808218+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"110'3410","reported_seq":3440,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:24.937741+0000","last_change":"2024-02-05T07:20:45.769183+0000","last_active":"2024-02-05T07:21:24.937741+0000","last_peered":"2024-02-05T07:21:24.937741+0000","last_clean":"2024-02-05T07:21:24.937741+0000","last_became_active":"2024-02-05T07:20:45.768878+0000","last_became_peered":"2024-02-05T07:20:45.768878+0000","last_unstale":"2024-02-05T07:21:24.937741+0000","last_undegraded":"2024-02-05T07:21:24.937741+0000","last_fullsized":"2024-02-05T07:21:24.937741+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":3410,"log_dups_size":0,"ondisk_log_size":3410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:45.364438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223477760,"num_objects":3410,"num_object_clones":0,"num_object_copies":10230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3410,"num_write_kb":218240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.507276+0000","last_change":"2024-02-05T07:20:36.859642+0000","last_active":"2024-02-05T07:21:17.507276+0000","last_peered":"2024-02-05T07:21:17.507276+0000","last_clean":"2024-02-05T07:21:17.507276+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:21:17.507276+0000","last_undegraded":"2024-02-05T07:21:17.507276+0000","last_fullsized":"2024-02-05T07:21:17.507276+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:36.859545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:36.859545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:48.101578+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":138,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:19.506997+0000","last_change":"2024-02-05T07:20:18.512034+0000","last_active":"2024-02-05T07:21:19.506997+0000","last_peered":"2024-02-05T07:21:19.506997+0000","last_clean":"2024-02-05T07:21:19.506997+0000","last_became_active":"2024-02-05T07:18:53.461852+0000","last_became_peered":"2024-02-05T07:18:53.461852+0000","last_unstale":"2024-02-05T07:21:19.506997+0000","last_undegraded":"2024-02-05T07:21:19.506997+0000","last_fullsized":"2024-02-05T07:21:19.506997+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:20:18.511957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.511957+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.513880+0000","last_change":"2024-02-05T07:20:04.827259+0000","last_active":"2024-02-05T07:21:17.513880+0000","last_peered":"2024-02-05T07:21:17.513880+0000","last_clean":"2024-02-05T07:21:17.513880+0000","last_became_active":"2024-02-05T07:18:49.435368+0000","last_became_peered":"2024-02-05T07:18:49.435368+0000","last_unstale":"2024-02-05T07:21:17.513880+0000","last_undegraded":"2024-02-05T07:21:17.513880+0000","last_fullsized":"2024-02-05T07:21:17.513880+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:04.827180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:04.827180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:17.606501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-02-05T07:21:17.506982+0000","last_change":"2024-02-05T07:20:13.843192+0000","last_active":"2024-02-05T07:21:17.506982+0000","last_peered":"2024-02-05T07:21:17.506982+0000","last_clean":"2024-02-05T07:21:17.506982+0000","last_became_active":"2024-02-05T07:18:51.442711+0000","last_became_peered":"2024-02-05T07:18:51.442711+0000","last_unstale":"2024-02-05T07:21:17.506982+0000","last_undegraded":"2024-02-05T07:21:17.506982+0000","last_fullsized":"2024-02-05T07:21:17.506982+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:20:13.843051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:13.843051+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:21:41.856009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3532980224,"num_objects":53909,"num_object_clones":0,"num_object_copies":161727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53909,"num_write_kb":3450176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10599530496,"data_stored":10599530496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53909,"ondisk_log_size":53909,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":143,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037796,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1095372,"kb_used_data":1068356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92649780,"statfs":{"total":95995035648,"available":94873374720,"internally_reserved":0,"allocated":1093996544,"data_stored":1093586620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.436,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":2.4649999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52100000000000002,"15min":0.51300000000000001},"min":{"1min":0.439,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.32200000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53300000000000003,"15min":0.52700000000000002},"min":{"1min":0.42699999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":1.474},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55700000000000005,"15min":0.55500000000000005},"min":{"1min":0.47299999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.69699999999999995,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":1.8740000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53200000000000003,"15min":0.52700000000000002},"min":{"1min":0.42799999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.69799999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":1.9670000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.52900000000000003,"15min":0.52100000000000002},"min":{"1min":0.47399999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.67900000000000005,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.246}]},{"osd":3,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.436,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":1.718},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.47199999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":2.294}]},{"osd":4,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.438,"15min":0.438},"min":{"1min":0.316,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.627,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.44900000000000001,"15min":0.441},"min":{"1min":0.33500000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68799999999999994,"5min":0.746,"15min":0.746},"last":0.35399999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.35999999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.37},{"interface":"front","average":{"1min":0.502,"5min":0.497,"15min":0.496},"min":{"1min":0.34899999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77000000000000002,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.30399999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.502,"15min":0.498},"min":{"1min":0.38200000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67300000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.437},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.51800000000000002,"15min":0.51300000000000001},"min":{"1min":0.39800000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68799999999999994,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.59399999999999997}]}]},{"osd":6,"up_from":52,"seq":223338299434,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1117952,"kb_used_data":1068728,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92627200,"statfs":{"total":95995035648,"available":94850252800,"internally_reserved":0,"allocated":1094377472,"data_stored":1093980057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.56100000000000005,"15min":0.55300000000000005},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.60499999999999998,"15min":0.55400000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.6989999999999998,"5min":3.6989999999999998,"15min":3.6989999999999998},"last":0.65800000000000003}]},{"osd":1,"last update":"Mon Feb 5 07:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.6139999999999999,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":0.439},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.56499999999999995,"15min":0.55700000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.089,"5min":2.089,"15min":2.089},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.64300000000000002,"15min":0.60099999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":5.3479999999999999,"5min":5.3479999999999999,"15min":5.3479999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.54500000000000004,"15min":0.53600000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.3660000000000001,"5min":2.3660000000000001,"15min":2.3660000000000001},"last":0.53900000000000003}]},{"osd":3,"last update":"Mon Feb 5 07:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.58399999999999996,"15min":0.56799999999999995},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.3260000000000001,"5min":3.3260000000000001,"15min":3.3260000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.58699999999999997,"15min":0.57199999999999995},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.56499999999999995}]},{"osd":4,"last update":"Mon Feb 5 07:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42499999999999999,"15min":0.40899999999999997},"min":{"1min":0.28799999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46200000000000002,"15min":0.45400000000000001},"min":{"1min":0.314,"5min":0.313,"15min":0.313},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.46800000000000003}]},{"osd":5,"last update":"Mon Feb 5 07:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.46100000000000002,"15min":0.45000000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.47699999999999998,"15min":0.46100000000000002},"min":{"1min":0.307,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.5}]},{"osd":7,"last update":"Mon Feb 5 07:20:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.34100000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1324972,"kb_used_data":1275748,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92420180,"statfs":{"total":95995035648,"available":94638264320,"internally_reserved":0,"allocated":1306365952,"data_stored":1305923897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:21:21 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.502,"15min":0.38},"min":{"1min":0.26100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.3339999999999996,"5min":6.3339999999999996,"15min":6.3339999999999996},"last":0.628},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.46500000000000002,"15min":0.36699999999999999},"min":{"1min":0.253,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":6.0869999999999997,"5min":6.0869999999999997,"15min":6.0869999999999997},"last":0.253}]},{"osd":2,"last update":"Mon Feb 5 07:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.42499999999999999,"15min":0.441},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.48899999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":1.4450000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.71799999999999997,"15min":1.02},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.498,"5min":12.401999999999999,"15min":12.401999999999999},"last":1.9019999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.443,"15min":0.46999999999999997},"min":{"1min":0.27800000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":1.373},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47599999999999998,"15min":0.48599999999999999},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.377,"5min":1.377,"15min":1.377},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:20:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.57899999999999996,"15min":0.58599999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.68200000000000005,"15min":0.73699999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88900000000000001,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Feb 5 07:20:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86299999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.72299999999999998,"15min":0.77000000000000002},"min":{"1min":0.41899999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.94999999999999996,"5min":5.0289999999999999,"15min":5.0289999999999999},"last":1.3089999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:21:21 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.65600000000000003,"15min":0.67700000000000005},"min":{"1min":0.42799999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.399,"5min":2.48,"15min":2.48},"last":1.399},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66300000000000003,"15min":0.67700000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.8620000000000001,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":1.8620000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.44600000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":1.347},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.98499999999999999,"5min":1.081,"15min":1.081},"last":0.71799999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1740052,"kb_used_data":1690828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92005100,"statfs":{"total":95995035648,"available":94213222400,"internally_reserved":0,"allocated":1731407872,"data_stored":1730989897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.376,"15min":0.32100000000000001},"min":{"1min":0.20499999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.38400000000000001,"15min":0.33600000000000002},"min":{"1min":0.26800000000000002,"5min":0.214,"15min":0.214},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.89700000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.33200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.89600000000000002,"5min":1.262,"15min":1.262},"last":1.028},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.42799999999999999,"15min":0.42099999999999999},"min":{"1min":0.28000000000000003,"5min":0.182,"15min":0.182},"max":{"1min":0.67900000000000005,"5min":1.149,"15min":1.149},"last":0.74299999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:21:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.44700000000000001,"15min":0.437},"min":{"1min":0.28199999999999997,"5min":0.246,"15min":0.246},"max":{"1min":2.3319999999999999,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.41799999999999998,"15min":0.41999999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":1.968}]},{"osd":4,"last update":"Mon Feb 5 07:20:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.36699999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.65000000000000002,"15min":0.58199999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":10.103,"5min":10.103,"15min":10.103},"last":0.996}]},{"osd":5,"last update":"Mon Feb 5 07:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79900000000000004,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88600000000000001,"5min":1.296,"15min":1.296},"last":0.85799999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.67900000000000005,"15min":0.624},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.67600000000000005,"15min":0.65200000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.7709999999999999,"5min":2.7709999999999999,"15min":2.7709999999999999},"last":0.62}]},{"osd":7,"last update":"Mon Feb 5 07:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.63,"15min":0.63900000000000001},"min":{"1min":0.39800000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.91000000000000003,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.92800000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1329664,"kb_used_data":1280440,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92415488,"statfs":{"total":95995035648,"available":94633459712,"internally_reserved":0,"allocated":1311170560,"data_stored":1310772647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.37,"15min":0.36599999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.47599999999999998,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":1.1259999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.39400000000000002,"15min":0.38400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.51800000000000002,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Feb 5 07:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.40400000000000003,"15min":0.38900000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.56899999999999995,"5min":1.583,"15min":1.583},"last":1.4059999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.437,"15min":0.42399999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.59899999999999998,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":1.5609999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:21:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.501,"15min":0.47999999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":3.5089999999999999,"5min":3.5089999999999999,"15min":3.5089999999999999},"last":1.4590000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.48999999999999999,"15min":0.47199999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.657}]},{"osd":4,"last update":"Mon Feb 5 07:21:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54900000000000004,"15min":0.53500000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.1499999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.61699999999999999,"15min":0.57199999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":1.0209999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:20:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.60199999999999998,"15min":0.59599999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93400000000000005,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0229999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.55100000000000005}]},{"osd":6,"last update":"Mon Feb 5 07:21:18 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.74099999999999999,"15min":0.76100000000000001},"min":{"1min":0.40300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.6379999999999999,"5min":2.8319999999999999,"15min":2.8319999999999999},"last":0.627},{"interface":"front","average":{"1min":0.621,"5min":0.59899999999999998,"15min":0.60899999999999999},"min":{"1min":0.39500000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.51100000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.66700000000000004,"15min":0.68799999999999994},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.75600000000000001,"5min":2.1309999999999998,"15min":2.1309999999999998},"last":0.442},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.628,"15min":0.63800000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.748,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.57699999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1959644,"kb_used_data":1906260,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":91785508,"statfs":{"total":95995035648,"available":93988360192,"internally_reserved":0,"allocated":1952010240,"data_stored":1951583241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.39700000000000002,"15min":0.40600000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":2.5960000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.45900000000000002,"15min":0.46200000000000002},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":1.341}]},{"osd":1,"last update":"Mon Feb 5 07:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.46000000000000002,"15min":0.47199999999999998},"min":{"1min":0.29499999999999998,"5min":0.253,"15min":0.253},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.49199999999999999,"15min":0.50600000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":2.0059999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.47899999999999998,"15min":0.47599999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.4300000000000002,"5min":3.4300000000000002,"15min":3.4300000000000002},"last":2.0409999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.47999999999999998,"15min":0.497},"min":{"1min":0.25900000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.25900000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.61899999999999999,"15min":0.60599999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.1319999999999999,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":0.40000000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":2.5409999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.60099999999999998,"15min":0.59599999999999997},"min":{"1min":0.42499999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80200000000000005,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.34699999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:21:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.36099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82399999999999995,"5min":1.046,"15min":1.046},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.38200000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:20:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59199999999999997,"15min":0.59399999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69699999999999995,"5min":0.751,"15min":0.751},"last":2.5710000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953523,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1090384,"kb_used_data":1045320,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92654768,"statfs":{"total":95995035648,"available":94878482432,"internally_reserved":0,"allocated":1070407680,"data_stored":1069992670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:22 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51900000000000002,"15min":0.52700000000000002},"min":{"1min":0.307,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.754,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54000000000000004,"15min":0.55100000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.66400000000000003,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Feb 5 07:20:22 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.68799999999999994,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.309},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.53400000000000003,"15min":0.55000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.63300000000000001,"5min":0.754,"15min":0.754},"last":0.57499999999999996}]},{"osd":2,"last update":"Mon Feb 5 07:20:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52200000000000002,"15min":0.53600000000000003},"min":{"1min":0.36099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.60099999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53600000000000003,"15min":0.53900000000000003},"min":{"1min":0.38600000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.41399999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:20:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55000000000000004,"15min":0.55900000000000005},"min":{"1min":0.37,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73299999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55300000000000005,"15min":0.56000000000000005},"min":{"1min":0.38900000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.36699999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.41799999999999998,"15min":0.38},"min":{"1min":0.30499999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.41399999999999998,"15min":0.38600000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.39700000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44900000000000001,"15min":0.42199999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.96999999999999997,"5min":1.046,"15min":1.046},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.47299999999999998,"15min":0.42299999999999999},"min":{"1min":0.27600000000000002,"5min":0.248,"15min":0.248},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.34999999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.51200000000000001,"15min":0.499},"min":{"1min":0.38900000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.499,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":0.53300000000000003}]}]},{"osd":5,"up_from":41,"seq":176093659182,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887428,"kb_used_data":856252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92857724,"statfs":{"total":95995035648,"available":95086309376,"internally_reserved":0,"allocated":876802048,"data_stored":876399567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:20:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52300000000000002,"15min":0.53000000000000003},"min":{"1min":0.38600000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.78200000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":1.508},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52400000000000002,"15min":0.53200000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":3.3399999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:20:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54600000000000004,"15min":0.56000000000000005},"min":{"1min":0.36299999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.71599999999999997,"5min":0.753,"15min":0.753},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.53100000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.70399999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.496}]},{"osd":2,"last update":"Mon Feb 5 07:20:48 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52000000000000002,"15min":0.52700000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72999999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":1.74},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.36699999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.75900000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.35599999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:20:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73799999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.40899999999999997,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":1.613}]},{"osd":4,"last update":"Mon Feb 5 07:20:48 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.38500000000000001,"15min":0.35599999999999998},"min":{"1min":0.28699999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.41599999999999998,"15min":0.38200000000000001},"min":{"1min":0.25800000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.374}]},{"osd":6,"last update":"Mon Feb 5 07:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.45800000000000002,"15min":0.41399999999999998},"min":{"1min":0.34200000000000003,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.52300000000000002,"15min":0.46899999999999997},"min":{"1min":0.41699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":2.0179999999999998,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":0.32300000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51400000000000001,"15min":0.51200000000000001},"min":{"1min":0.32100000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.52000000000000002},"min":{"1min":0.40600000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1767768064,"data_stored":1767768064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1323106304,"data_stored":1323106304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1319895040,"data_stored":1319895040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1996554240,"data_stored":1996554240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1101266944,"data_stored":1101266944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":877920256,"data_stored":877920256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1109131264,"data_stored":1109131264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1103888384,"data_stored":1103888384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:21:29.723 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-05T07:21:29.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set unique_pool_0 pg_num 26 2024-02-05T07:21:29.741 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:29.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:29.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:29.843 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:29.888 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:29.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:29.913 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 33 16 60344 60328 114.227 82 0.0110428 0.00874943 2024-02-05T07:21:29.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:30.020 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:30.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:30.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:30.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:30.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:30.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T07:21:30.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:30.286 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:30.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:30.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:30.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:30.439 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:30.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:30.496 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:30 smithi007 ceph-mon[29098]: pgmap v88: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 895 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T07:21:30.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:30 smithi007 ceph-mon[37158]: pgmap v88: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 895 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T07:21:30.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:21:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:21:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:30 smithi090 ceph-mon[33213]: pgmap v88: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 895 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T07:21:30.540 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:30.588 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:30.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:30.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:30.690 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:30.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:30.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:30.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:30.914 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 34 16 61624 61608 113.215 80 0.00852158 0.00882651 2024-02-05T07:21:30.924 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:30 smithi007 podman[80610]: 2024-02-05 07:21:30.495695912 +0000 UTC m=+0.267667142 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-02-05T07:21:30.924 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:30 smithi007 podman[80610]: 2024-02-05 07:21:30.641714134 +0000 UTC m=+0.413685358 volume create cc89949747cd04ff9fcc0aaf67f9de1e2a560dd7a215618084c9692ed8cdd910 2024-02-05T07:21:30.924 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:30 smithi007 podman[80610]: 2024-02-05 07:21:30.742400628 +0000 UTC m=+0.514371852 container create c0a0ce033b5b8ff0ba54aba37e3075e41e7fb5e7198bf1b5e0d44f0eb4528bbe (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T07:21:30.955 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:30.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:30.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:31.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:31.088 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:31.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:31.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:31.203 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[29098]: 5.8 scrub starts 2024-02-05T07:21:31.203 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[29098]: 5.8 scrub ok 2024-02-05T07:21:31.203 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[29098]: 3.16 scrub starts 2024-02-05T07:21:31.204 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[29098]: 3.16 scrub ok 2024-02-05T07:21:31.204 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[29098]: 4.d scrub starts 2024-02-05T07:21:31.204 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[29098]: 4.d scrub ok 2024-02-05T07:21:31.204 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:31.204 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[29098]: 4.1f scrub starts 2024-02-05T07:21:31.204 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[29098]: 4.1f scrub ok 2024-02-05T07:21:31.205 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[29098]: 6.12 scrub starts 2024-02-05T07:21:31.205 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[29098]: 6.12 scrub ok 2024-02-05T07:21:31.205 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[37158]: 5.8 scrub starts 2024-02-05T07:21:31.206 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[37158]: 5.8 scrub ok 2024-02-05T07:21:31.206 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[37158]: 3.16 scrub starts 2024-02-05T07:21:31.206 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[37158]: 3.16 scrub ok 2024-02-05T07:21:31.206 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[37158]: 4.d scrub starts 2024-02-05T07:21:31.206 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[37158]: 4.d scrub ok 2024-02-05T07:21:31.206 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:31.206 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[37158]: 4.1f scrub starts 2024-02-05T07:21:31.207 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[37158]: 4.1f scrub ok 2024-02-05T07:21:31.207 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[37158]: 6.12 scrub starts 2024-02-05T07:21:31.207 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-mon[37158]: 6.12 scrub ok 2024-02-05T07:21:31.207 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:31 smithi007 podman[80610]: 2024-02-05 07:21:31.080327456 +0000 UTC m=+0.852298672 container init c0a0ce033b5b8ff0ba54aba37e3075e41e7fb5e7198bf1b5e0d44f0eb4528bbe (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T07:21:31.208 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:31 smithi007 podman[80610]: 2024-02-05 07:21:31.085827433 +0000 UTC m=+0.857798656 container start c0a0ce033b5b8ff0ba54aba37e3075e41e7fb5e7198bf1b5e0d44f0eb4528bbe (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T07:21:31.208 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[80747]: ts=2024-02-05T07:21:31.119Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-05T07:21:31.208 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[80747]: ts=2024-02-05T07:21:31.119Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-05T07:21:31.208 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[80747]: ts=2024-02-05T07:21:31.120Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.7 port=9094 2024-02-05T07:21:31.208 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[80747]: ts=2024-02-05T07:21:31.122Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-05T07:21:31.208 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[80747]: ts=2024-02-05T07:21:31.196Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-05T07:21:31.208 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[80747]: ts=2024-02-05T07:21:31.197Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-05T07:21:31.209 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[80747]: ts=2024-02-05T07:21:31.201Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-05T07:21:31.209 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:31 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[80747]: ts=2024-02-05T07:21:31.201Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-05T07:21:31.218 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:31.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:31.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:31.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:31.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:31.460 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:21:31.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:31.497 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:21:31.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:31 smithi090 ceph-mon[33213]: 5.8 scrub starts 2024-02-05T07:21:31.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:31 smithi090 ceph-mon[33213]: 5.8 scrub ok 2024-02-05T07:21:31.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:31 smithi090 ceph-mon[33213]: 3.16 scrub starts 2024-02-05T07:21:31.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:31 smithi090 ceph-mon[33213]: 3.16 scrub ok 2024-02-05T07:21:31.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:31 smithi090 ceph-mon[33213]: 4.d scrub starts 2024-02-05T07:21:31.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:31 smithi090 ceph-mon[33213]: 4.d scrub ok 2024-02-05T07:21:31.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:31 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:31.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:31 smithi090 ceph-mon[33213]: 4.1f scrub starts 2024-02-05T07:21:31.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:31 smithi090 ceph-mon[33213]: 4.1f scrub ok 2024-02-05T07:21:31.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:31 smithi090 ceph-mon[33213]: 6.12 scrub starts 2024-02-05T07:21:31.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:31 smithi090 ceph-mon[33213]: 6.12 scrub ok 2024-02-05T07:21:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:21:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:31.559 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:31.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:31.619 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:31 smithi007 bash[80610]: c0a0ce033b5b8ff0ba54aba37e3075e41e7fb5e7198bf1b5e0d44f0eb4528bbe 2024-02-05T07:21:31.619 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:31 smithi007 systemd[1]: Started Ceph alertmanager.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:21:31.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:31.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:31.707 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:31.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:31.808 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:31.851 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:31.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:31.915 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 35 16 62764 62748 112.017 71.25 0.0118883 0.00892184 2024-02-05T07:21:31.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:31.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:31.985 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:31.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:32.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:32.134 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:32.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:32.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:32.267 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:32.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:32.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:32.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:32.403 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:32.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:32.504 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:32.527 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:32.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:32.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:32.584 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:21:32.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:32.658 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:32.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:32.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:32.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: 4.15 scrub starts 2024-02-05T07:21:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: 4.15 scrub ok 2024-02-05T07:21:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: pgmap v89: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-02-05T07:21:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: 6.1f scrub starts 2024-02-05T07:21:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: 6.1f scrub ok 2024-02-05T07:21:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: 4.a scrub starts 2024-02-05T07:21:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: 4.a scrub ok 2024-02-05T07:21:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: 4.2 scrub starts 2024-02-05T07:21:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: 3.e scrub starts 2024-02-05T07:21:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: 3.e scrub ok 2024-02-05T07:21:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: 4.2 scrub ok 2024-02-05T07:21:32.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: 5.6 scrub starts 2024-02-05T07:21:32.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: 5.6 scrub ok 2024-02-05T07:21:32.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2796898691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:21:32.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:21:32.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: 5.18 scrub starts 2024-02-05T07:21:32.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:32 smithi090 ceph-mon[33213]: 5.18 scrub ok 2024-02-05T07:21:32.803 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:32.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:32.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:32.905 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:32.914 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 36 16 63882 63866 110.846 69.875 0.00313889 0.00900849 2024-02-05T07:21:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: 4.15 scrub starts 2024-02-05T07:21:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: 4.15 scrub ok 2024-02-05T07:21:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: pgmap v89: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-02-05T07:21:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: 6.1f scrub starts 2024-02-05T07:21:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: 6.1f scrub ok 2024-02-05T07:21:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: 4.a scrub starts 2024-02-05T07:21:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: 4.a scrub ok 2024-02-05T07:21:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: 4.2 scrub starts 2024-02-05T07:21:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: 3.e scrub starts 2024-02-05T07:21:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: 3.e scrub ok 2024-02-05T07:21:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: 4.2 scrub ok 2024-02-05T07:21:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: 5.6 scrub starts 2024-02-05T07:21:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: 5.6 scrub ok 2024-02-05T07:21:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2796898691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:21:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:21:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: 5.18 scrub starts 2024-02-05T07:21:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[37158]: 5.18 scrub ok 2024-02-05T07:21:32.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: 4.15 scrub starts 2024-02-05T07:21:32.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: 4.15 scrub ok 2024-02-05T07:21:32.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: pgmap v89: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-02-05T07:21:32.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: 6.1f scrub starts 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: 6.1f scrub ok 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: 4.a scrub starts 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: 4.a scrub ok 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: 4.2 scrub starts 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: 3.e scrub starts 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: 3.e scrub ok 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: 4.2 scrub ok 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: 5.6 scrub starts 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: 5.6 scrub ok 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2796898691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: 5.18 scrub starts 2024-02-05T07:21:32.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:32 smithi007 ceph-mon[29098]: 5.18 scrub ok 2024-02-05T07:21:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:32.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:32.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:33.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:33.071 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:33.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:33.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:33.206 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:33.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:33.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:33.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:33.339 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:33.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:33.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:33.424 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[80747]: ts=2024-02-05T07:21:33.123Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000784717s 2024-02-05T07:21:33.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:33.472 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:33.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:33.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:33 smithi090 ceph-mon[33213]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-05T07:21:33.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:33 smithi090 ceph-mon[33213]: Reconfiguring daemon prometheus.a on smithi090 2024-02-05T07:21:33.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:33 smithi090 ceph-mon[33213]: 3.1b scrub starts 2024-02-05T07:21:33.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:33 smithi090 ceph-mon[33213]: 3.1b scrub ok 2024-02-05T07:21:33.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:33 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:21:33.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:33 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:21:33.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:33 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:21:33.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:33 smithi090 ceph-mon[33213]: osdmap e111: 8 total, 8 up, 8 in 2024-02-05T07:21:33.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:33.606 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:33.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:33.708 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:33.750 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:33.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:33.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:21:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:21:33.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[29098]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-05T07:21:33.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[29098]: Reconfiguring daemon prometheus.a on smithi090 2024-02-05T07:21:33.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[29098]: 3.1b scrub starts 2024-02-05T07:21:33.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[29098]: 3.1b scrub ok 2024-02-05T07:21:33.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:21:33.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:21:33.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:21:33.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[29098]: osdmap e111: 8 total, 8 up, 8 in 2024-02-05T07:21:33.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[37158]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-05T07:21:33.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[37158]: Reconfiguring daemon prometheus.a on smithi090 2024-02-05T07:21:33.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[37158]: 3.1b scrub starts 2024-02-05T07:21:33.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[37158]: 3.1b scrub ok 2024-02-05T07:21:33.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:21:33.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:21:33.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:21:33.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:33 smithi007 ceph-mon[37158]: osdmap e111: 8 total, 8 up, 8 in 2024-02-05T07:21:33.852 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:33.892 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:33.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:33.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:33.915 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 37 16 65250 65234 110.161 85.5 0.0142075 0.00907237 2024-02-05T07:21:33.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:34.032 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:34.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:34.134 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:34.169 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:34.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:34.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:34.302 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:34.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:34.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:34.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:34.435 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:34.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:34.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:34.537 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:34.578 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:34.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:34.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:34.710 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:34.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:34 smithi090 ceph-mon[33213]: 4.8 scrub starts 2024-02-05T07:21:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:34 smithi090 ceph-mon[33213]: 4.8 scrub ok 2024-02-05T07:21:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:34 smithi090 ceph-mon[33213]: pgmap v91: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-02-05T07:21:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:34 smithi090 ceph-mon[33213]: 4.16 scrub starts 2024-02-05T07:21:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:34 smithi090 ceph-mon[33213]: 4.16 scrub ok 2024-02-05T07:21:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:34 smithi090 ceph-mon[33213]: 6.1d scrub starts 2024-02-05T07:21:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:34 smithi090 ceph-mon[33213]: 6.1d scrub ok 2024-02-05T07:21:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:34 smithi090 ceph-mon[33213]: osdmap e112: 8 total, 8 up, 8 in 2024-02-05T07:21:34.812 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:34.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:34.851 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:34.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:34.915 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 38 16 66612 66596 109.502 85.125 0.00945869 0.00912657 2024-02-05T07:21:34.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[29098]: 4.8 scrub starts 2024-02-05T07:21:34.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[29098]: 4.8 scrub ok 2024-02-05T07:21:34.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[29098]: pgmap v91: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-02-05T07:21:34.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[29098]: 4.16 scrub starts 2024-02-05T07:21:34.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[29098]: 4.16 scrub ok 2024-02-05T07:21:34.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[29098]: 6.1d scrub starts 2024-02-05T07:21:34.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[29098]: 6.1d scrub ok 2024-02-05T07:21:34.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[29098]: osdmap e112: 8 total, 8 up, 8 in 2024-02-05T07:21:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[37158]: 4.8 scrub starts 2024-02-05T07:21:34.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[37158]: 4.8 scrub ok 2024-02-05T07:21:34.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[37158]: pgmap v91: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-02-05T07:21:34.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[37158]: 4.16 scrub starts 2024-02-05T07:21:34.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[37158]: 4.16 scrub ok 2024-02-05T07:21:34.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[37158]: 6.1d scrub starts 2024-02-05T07:21:34.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[37158]: 6.1d scrub ok 2024-02-05T07:21:34.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:34 smithi007 ceph-mon[37158]: osdmap e112: 8 total, 8 up, 8 in 2024-02-05T07:21:34.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:34.997 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:34.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:35.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:35.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:35.148 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:35.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:35.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:35.283 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:35.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:35.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:35.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:35.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:35.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:35.556 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:35.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:35.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[29098]: 4.9 scrub starts 2024-02-05T07:21:35.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[29098]: 4.9 scrub ok 2024-02-05T07:21:35.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[29098]: 5.5 scrub starts 2024-02-05T07:21:35.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[29098]: 5.5 scrub ok 2024-02-05T07:21:35.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[29098]: 3.0 scrub starts 2024-02-05T07:21:35.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[29098]: 3.0 scrub ok 2024-02-05T07:21:35.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[29098]: 4.1a scrub starts 2024-02-05T07:21:35.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[29098]: 4.1a scrub ok 2024-02-05T07:21:35.643 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[37158]: 4.9 scrub starts 2024-02-05T07:21:35.643 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[37158]: 4.9 scrub ok 2024-02-05T07:21:35.643 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[37158]: 5.5 scrub starts 2024-02-05T07:21:35.643 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[37158]: 5.5 scrub ok 2024-02-05T07:21:35.643 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[37158]: 3.0 scrub starts 2024-02-05T07:21:35.643 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[37158]: 3.0 scrub ok 2024-02-05T07:21:35.643 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[37158]: 4.1a scrub starts 2024-02-05T07:21:35.643 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:35 smithi007 ceph-mon[37158]: 4.1a scrub ok 2024-02-05T07:21:35.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:35.691 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:35.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:35.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:35 smithi090 ceph-mon[33213]: 4.9 scrub starts 2024-02-05T07:21:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:35 smithi090 ceph-mon[33213]: 4.9 scrub ok 2024-02-05T07:21:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:35 smithi090 ceph-mon[33213]: 5.5 scrub starts 2024-02-05T07:21:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:35 smithi090 ceph-mon[33213]: 5.5 scrub ok 2024-02-05T07:21:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:35 smithi090 ceph-mon[33213]: 3.0 scrub starts 2024-02-05T07:21:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:35 smithi090 ceph-mon[33213]: 3.0 scrub ok 2024-02-05T07:21:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:35 smithi090 ceph-mon[33213]: 4.1a scrub starts 2024-02-05T07:21:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:35 smithi090 ceph-mon[33213]: 4.1a scrub ok 2024-02-05T07:21:35.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:35.822 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:35.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:35.916 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 39 16 67900 67884 108.758 80.5 0.0172857 0.00918926 2024-02-05T07:21:35.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:35.959 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:35.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:35.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:36.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:36.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:36.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:36.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:36.234 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:36.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:36.272 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:35 smithi090 systemd[1]: Stopping Ceph prometheus.a for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:21:36.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:36.388 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:36.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:36.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:36.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:36.519 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:36.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:36.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:21:36.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:21:36.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-mon[33213]: pgmap v93: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-02-05T07:21:36.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-mon[33213]: osdmap e113: 8 total, 8 up, 8 in 2024-02-05T07:21:36.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-mon[33213]: 5.a scrub starts 2024-02-05T07:21:36.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-mon[33213]: 5.a scrub ok 2024-02-05T07:21:36.525 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:21:36.269Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-05T07:21:36.525 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:21:36.269Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-05T07:21:36.525 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:21:36.269Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-05T07:21:36.525 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:21:36.269Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-05T07:21:36.525 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:21:36.269Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-05T07:21:36.525 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:21:36.269Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-05T07:21:36.525 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:21:36.269Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-05T07:21:36.525 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:21:36.269Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-05T07:21:36.525 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:21:36.270Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-05T07:21:36.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:21:36.271Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-05T07:21:36.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:21:36.271Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-05T07:21:36.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[61544]: ts=2024-02-05T07:21:36.271Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-05T07:21:36.526 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 podman[68525]: 2024-02-05 07:21:36.282399662 +0000 UTC m=+0.142023516 container died 7c2c4aabf01150ac95291ddf2e556238c03df157074d2c567298b9a4c8286c54 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T07:21:36.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:36 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:21:36.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:36 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:21:36.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:36 smithi007 ceph-mon[29098]: pgmap v93: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-02-05T07:21:36.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:36 smithi007 ceph-mon[29098]: osdmap e113: 8 total, 8 up, 8 in 2024-02-05T07:21:36.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:36 smithi007 ceph-mon[29098]: 5.a scrub starts 2024-02-05T07:21:36.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:36 smithi007 ceph-mon[29098]: 5.a scrub ok 2024-02-05T07:21:36.600 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:36 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:21:36.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:36 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:21:36.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:36 smithi007 ceph-mon[37158]: pgmap v93: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-02-05T07:21:36.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:36 smithi007 ceph-mon[37158]: osdmap e113: 8 total, 8 up, 8 in 2024-02-05T07:21:36.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:36 smithi007 ceph-mon[37158]: 5.a scrub starts 2024-02-05T07:21:36.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:36 smithi007 ceph-mon[37158]: 5.a scrub ok 2024-02-05T07:21:36.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:36.661 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:36.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:36.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:36.792 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:36.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:36.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:36.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:36.898 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 podman[68525]: 2024-02-05 07:21:36.695413148 +0000 UTC m=+0.555036997 container cleanup 7c2c4aabf01150ac95291ddf2e556238c03df157074d2c567298b9a4c8286c54 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T07:21:36.898 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 bash[68525]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a 2024-02-05T07:21:36.915 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:21:36.914641+0000 min lat: 0.00125178 max lat: 0.180861 avg lat: 0.00924413 2024-02-05T07:21:36.916 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:21:36.916 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 40 16 69228 69212 108.114 83 0.00755301 0.00924413 2024-02-05T07:21:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:36.940 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:36.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:37.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:37.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:37.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:37.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:37.190 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:36 smithi090 podman[68552]: 2024-02-05 07:21:36.897591593 +0000 UTC m=+0.616815745 container remove 7c2c4aabf01150ac95291ddf2e556238c03df157074d2c567298b9a4c8286c54 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T07:21:37.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:37.216 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:37.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:37.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:37.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:37.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:37.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:37.464 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-05T07:21:37.464 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 systemd[1]: Stopped Ceph prometheus.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:21:37.464 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@prometheus.a.service: Consumed 1.284s CPU time. 2024-02-05T07:21:37.464 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 systemd[1]: Starting Ceph prometheus.a for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:21:37.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:37.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:37.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:37.625 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:37.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:37.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:37 smithi007 ceph-mon[29098]: 5.3 scrub starts 2024-02-05T07:21:37.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:37 smithi007 ceph-mon[29098]: 5.3 scrub ok 2024-02-05T07:21:37.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:37 smithi007 ceph-mon[29098]: 4.14 deep-scrub starts 2024-02-05T07:21:37.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:37 smithi007 ceph-mon[29098]: 4.14 deep-scrub ok 2024-02-05T07:21:37.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:37 smithi007 ceph-mon[29098]: 4.1c scrub starts 2024-02-05T07:21:37.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:37 smithi007 ceph-mon[29098]: 4.1c scrub ok 2024-02-05T07:21:37.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:37 smithi007 ceph-mon[37158]: 5.3 scrub starts 2024-02-05T07:21:37.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:37 smithi007 ceph-mon[37158]: 5.3 scrub ok 2024-02-05T07:21:37.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:37 smithi007 ceph-mon[37158]: 4.14 deep-scrub starts 2024-02-05T07:21:37.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:37 smithi007 ceph-mon[37158]: 4.14 deep-scrub ok 2024-02-05T07:21:37.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:37 smithi007 ceph-mon[37158]: 4.1c scrub starts 2024-02-05T07:21:37.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:37 smithi007 ceph-mon[37158]: 4.1c scrub ok 2024-02-05T07:21:37.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:37.774 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 podman[68710]: 2024-02-05 07:21:37.369464964 +0000 UTC m=+0.023035101 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-02-05T07:21:37.775 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 podman[68710]: 2024-02-05 07:21:37.517234021 +0000 UTC m=+0.170804194 container create aca460d64221bfdc00e9cba9e129339e83cae80387660bf0959d7e283e48e4cb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T07:21:37.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-mon[33213]: 5.3 scrub starts 2024-02-05T07:21:37.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-mon[33213]: 5.3 scrub ok 2024-02-05T07:21:37.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-mon[33213]: 4.14 deep-scrub starts 2024-02-05T07:21:37.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-mon[33213]: 4.14 deep-scrub ok 2024-02-05T07:21:37.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-mon[33213]: 4.1c scrub starts 2024-02-05T07:21:37.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-mon[33213]: 4.1c scrub ok 2024-02-05T07:21:37.777 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:37.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:37.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:37.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:37.915 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 41 16 70477 70461 107.381 78.0625 0.00985439 0.00930796 2024-02-05T07:21:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:37.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:37.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:38.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:38.064 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:38.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:38.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:38.083 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 podman[68710]: 2024-02-05 07:21:37.779083133 +0000 UTC m=+0.432653262 container init aca460d64221bfdc00e9cba9e129339e83cae80387660bf0959d7e283e48e4cb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T07:21:38.083 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 podman[68710]: 2024-02-05 07:21:37.782656225 +0000 UTC m=+0.436226343 container start aca460d64221bfdc00e9cba9e129339e83cae80387660bf0959d7e283e48e4cb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T07:21:38.083 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.815Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-02-05T07:21:38.083 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.815Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-02-05T07:21:38.083 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.815Z caller=main.go:570 level=info host_details="(Linux 5.14.0-416.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jan 31 22:52:48 UTC 2024 x86_64 smithi090 (none))" 2024-02-05T07:21:38.083 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.815Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-05T07:21:38.083 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.815Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-05T07:21:38.083 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.817Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-02-05T07:21:38.083 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.818Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-02-05T07:21:38.083 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.821Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-02-05T07:21:38.083 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.821Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-02-05T07:21:38.083 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.821Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-05T07:21:38.084 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.822Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.659µs 2024-02-05T07:21:38.084 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.822Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-05T07:21:38.084 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.824Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-02-05T07:21:38.084 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.825Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-02-05T07:21:38.084 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.825Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=32.304µs wal_replay_duration=3.335107ms wbl_replay_duration=143ns total_replay_duration=3.417801ms 2024-02-05T07:21:38.084 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.826Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-02-05T07:21:38.084 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.826Z caller=main.go:1029 level=info msg="TSDB started" 2024-02-05T07:21:38.084 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.826Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-05T07:21:38.084 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.846Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.642285ms db_storage=961ns remote_storage=1.433µs web_handler=451ns query_engine=726ns scrape=508.613µs scrape_sd=132.025µs notify=14.974µs notify_sd=16.092µs rules=19.635667ms tracing=8.2µs 2024-02-05T07:21:38.084 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.847Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-02-05T07:21:38.084 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:21:37.847Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-02-05T07:21:38.084 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 bash[68710]: aca460d64221bfdc00e9cba9e129339e83cae80387660bf0959d7e283e48e4cb 2024-02-05T07:21:38.084 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:21:37 smithi090 systemd[1]: Started Ceph prometheus.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:21:38.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:38.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:38.228 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:38.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:38.266 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Bus STOPPING 2024-02-05T07:21:38.266 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T07:21:38.266 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Bus STOPPED 2024-02-05T07:21:38.266 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Bus STARTING 2024-02-05T07:21:38.266 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Serving on http://:::9283 2024-02-05T07:21:38.266 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Bus STARTED 2024-02-05T07:21:38.266 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Bus STOPPING 2024-02-05T07:21:38.266 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T07:21:38.267 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Bus STOPPED 2024-02-05T07:21:38.267 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Bus STARTING 2024-02-05T07:21:38.267 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Serving on http://:::9283 2024-02-05T07:21:38.267 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Bus STARTED 2024-02-05T07:21:38.267 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Bus STOPPING 2024-02-05T07:21:38.330 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:38.373 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:38.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:38.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:21:38.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:38.535 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:38.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:38.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: pgmap v95: 148 pgs: 148 active+clean; 4.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-05T07:21:38.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:21:38.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:21:38.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: 6.9 scrub starts 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: 6.9 scrub ok 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: 3.8 scrub starts 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: 3.8 scrub ok 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: 5.2 deep-scrub starts 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: 5.2 deep-scrub ok 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: osdmap e114: 8 total, 8 up, 8 in 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: 4.13 scrub starts 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: 4.13 scrub ok 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T07:21:38.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi090.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T07:21:38.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T07:21:38.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi090.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T07:21:38.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:21:38.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:21:38.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:21:38.554 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T07:21:38.554 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Bus STOPPED 2024-02-05T07:21:38.555 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Bus STARTING 2024-02-05T07:21:38.555 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Serving on http://:::9283 2024-02-05T07:21:38.555 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:21:38] ENGINE Bus STARTED 2024-02-05T07:21:38.555 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: pgmap v95: 148 pgs: 148 active+clean; 4.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-05T07:21:38.555 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:21:38.555 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:21:38.555 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: 6.9 scrub starts 2024-02-05T07:21:38.555 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: 6.9 scrub ok 2024-02-05T07:21:38.555 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: 3.8 scrub starts 2024-02-05T07:21:38.555 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: 3.8 scrub ok 2024-02-05T07:21:38.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: 5.2 deep-scrub starts 2024-02-05T07:21:38.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: 5.2 deep-scrub ok 2024-02-05T07:21:38.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: osdmap e114: 8 total, 8 up, 8 in 2024-02-05T07:21:38.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: 4.13 scrub starts 2024-02-05T07:21:38.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: 4.13 scrub ok 2024-02-05T07:21:38.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T07:21:38.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T07:21:38.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T07:21:38.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi090.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T07:21:38.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T07:21:38.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi090.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T07:21:38.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:21:38.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:21:38.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:38 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:21:38.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:38.674 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:38.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:38.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: pgmap v95: 148 pgs: 148 active+clean; 4.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-05T07:21:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:21:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:21:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: 6.9 scrub starts 2024-02-05T07:21:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: 6.9 scrub ok 2024-02-05T07:21:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: 3.8 scrub starts 2024-02-05T07:21:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: 3.8 scrub ok 2024-02-05T07:21:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: 5.2 deep-scrub starts 2024-02-05T07:21:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: 5.2 deep-scrub ok 2024-02-05T07:21:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: osdmap e114: 8 total, 8 up, 8 in 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: 4.13 scrub starts 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: 4.13 scrub ok 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi090.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi090.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:21:38.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:21:38.778 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:38 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:21:38.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:38.813 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:38.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:38.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:38.916 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 42 15 71441 71426 106.26 60.3125 0.0262219 0.0094052 2024-02-05T07:21:38.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:38.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:39.019 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:21:39.019 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:21:39.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:39.146 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:39.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:39.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:39.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:39.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:39.286 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:39.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:39.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:39.418 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:39.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:39.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:39.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:39.550 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:39.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:39.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:39.689 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:39.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:39.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:39.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T07:21:39.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T07:21:39.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T07:21:39.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi090.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T07:21:39.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T07:21:39.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi090.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T07:21:39.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: 5.b scrub starts 2024-02-05T07:21:39.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: 5.b scrub ok 2024-02-05T07:21:39.769 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: 6.5 scrub starts 2024-02-05T07:21:39.769 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: 6.5 scrub ok 2024-02-05T07:21:39.769 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: 4.4 scrub starts 2024-02-05T07:21:39.769 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: 4.4 scrub ok 2024-02-05T07:21:39.769 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: 6.1e scrub starts 2024-02-05T07:21:39.769 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: 6.1e scrub ok 2024-02-05T07:21:39.769 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: osdmap e115: 8 total, 8 up, 8 in 2024-02-05T07:21:39.769 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[29098]: osdmap e116: 8 total, 8 up, 8 in 2024-02-05T07:21:39.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T07:21:39.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T07:21:39.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T07:21:39.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi090.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T07:21:39.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T07:21:39.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi090.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T07:21:39.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: 5.b scrub starts 2024-02-05T07:21:39.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: 5.b scrub ok 2024-02-05T07:21:39.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: 6.5 scrub starts 2024-02-05T07:21:39.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: 6.5 scrub ok 2024-02-05T07:21:39.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: 4.4 scrub starts 2024-02-05T07:21:39.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: 4.4 scrub ok 2024-02-05T07:21:39.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: 6.1e scrub starts 2024-02-05T07:21:39.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: 6.1e scrub ok 2024-02-05T07:21:39.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: osdmap e115: 8 total, 8 up, 8 in 2024-02-05T07:21:39.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:39 smithi007 ceph-mon[37158]: osdmap e116: 8 total, 8 up, 8 in 2024-02-05T07:21:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T07:21:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T07:21:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T07:21:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi090.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T07:21:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T07:21:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi090.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T07:21:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: 5.b scrub starts 2024-02-05T07:21:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: 5.b scrub ok 2024-02-05T07:21:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: 6.5 scrub starts 2024-02-05T07:21:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: 6.5 scrub ok 2024-02-05T07:21:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: 4.4 scrub starts 2024-02-05T07:21:39.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: 4.4 scrub ok 2024-02-05T07:21:39.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: 6.1e scrub starts 2024-02-05T07:21:39.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: 6.1e scrub ok 2024-02-05T07:21:39.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: osdmap e115: 8 total, 8 up, 8 in 2024-02-05T07:21:39.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:39 smithi090 ceph-mon[33213]: osdmap e116: 8 total, 8 up, 8 in 2024-02-05T07:21:39.787 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:21:39.791 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:39.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:39.826 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:39.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:39.836 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:21:39.878 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:21:39.916 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 43 16 72479 72463 105.296 64.8125 0.0079076 0.00949167 2024-02-05T07:21:39.926 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:21:39.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:39.969 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:21:39.973 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:39.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:39.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:40.011 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:21:40.049 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:21:40.076 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:40.088 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:21:40.096 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:21:40.096 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:21:40.096 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:21:40.096 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:21:40.097 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:21:40.097 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:21:40.097 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:21:40.097 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:21:40.117 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:40.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:40.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:40.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:40.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:40.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:40.387 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:40.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:40.463 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:21:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:21:40.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:40.531 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:40.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:40.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:40.535 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 7 pg_num to 26 2024-02-05T07:21:40.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:40.663 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:40.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:40.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:40.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:40 smithi090 ceph-mon[33213]: pgmap v97: 148 pgs: 148 active+clean; 4.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-05T07:21:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:40 smithi090 ceph-mon[33213]: 3.6 scrub starts 2024-02-05T07:21:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:40 smithi090 ceph-mon[33213]: 3.6 scrub ok 2024-02-05T07:21:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:40 smithi090 ceph-mon[33213]: 5.1f scrub starts 2024-02-05T07:21:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:40 smithi090 ceph-mon[33213]: 5.1f scrub ok 2024-02-05T07:21:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:40 smithi090 ceph-mon[33213]: 6.1c deep-scrub starts 2024-02-05T07:21:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:40 smithi090 ceph-mon[33213]: 6.1c deep-scrub ok 2024-02-05T07:21:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:40 smithi090 ceph-mon[33213]: 6.14 scrub starts 2024-02-05T07:21:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:40 smithi090 ceph-mon[33213]: 6.14 scrub ok 2024-02-05T07:21:40.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:40 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1148905732' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:21:40.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:40 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1078707124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:21:40.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:40.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:40.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:40.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:40.916 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 44 16 73682 73666 104.611 75.1875 0.00787585 0.00955358 2024-02-05T07:21:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[29098]: pgmap v97: 148 pgs: 148 active+clean; 4.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-05T07:21:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[29098]: 3.6 scrub starts 2024-02-05T07:21:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[29098]: 3.6 scrub ok 2024-02-05T07:21:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[29098]: 5.1f scrub starts 2024-02-05T07:21:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[29098]: 5.1f scrub ok 2024-02-05T07:21:40.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[29098]: 6.1c deep-scrub starts 2024-02-05T07:21:40.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[29098]: 6.1c deep-scrub ok 2024-02-05T07:21:40.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[29098]: 6.14 scrub starts 2024-02-05T07:21:40.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[29098]: 6.14 scrub ok 2024-02-05T07:21:40.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1148905732' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:21:40.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1078707124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:21:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[37158]: pgmap v97: 148 pgs: 148 active+clean; 4.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-05T07:21:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[37158]: 3.6 scrub starts 2024-02-05T07:21:40.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[37158]: 3.6 scrub ok 2024-02-05T07:21:40.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[37158]: 5.1f scrub starts 2024-02-05T07:21:40.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[37158]: 5.1f scrub ok 2024-02-05T07:21:40.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[37158]: 6.1c deep-scrub starts 2024-02-05T07:21:40.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[37158]: 6.1c deep-scrub ok 2024-02-05T07:21:40.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[37158]: 6.14 scrub starts 2024-02-05T07:21:40.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[37158]: 6.14 scrub ok 2024-02-05T07:21:40.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1148905732' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:21:40.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:40 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1078707124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:21:40.949 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:40.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:40.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:41.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:41.088 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:41.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:41.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:41.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T07:21:41.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:41.219 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:41.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:41.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:41.321 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:41.368 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:41.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:41.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:41.385 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[80747]: ts=2024-02-05T07:21:41.125Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002972068s 2024-02-05T07:21:41.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:41.503 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:41.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:41.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:41.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:41.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:41.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:41.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[29098]: 3.1c scrub starts 2024-02-05T07:21:41.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[29098]: 3.1c scrub ok 2024-02-05T07:21:41.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:41.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1148905732' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-02-05T07:21:41.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[29098]: osdmap e117: 8 total, 8 up, 8 in 2024-02-05T07:21:41.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[29098]: 3.4 scrub starts 2024-02-05T07:21:41.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[29098]: 3.4 scrub ok 2024-02-05T07:21:41.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[29098]: pgmap v101: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-05T07:21:41.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:21:41.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:21:41.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[37158]: 3.1c scrub starts 2024-02-05T07:21:41.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[37158]: 3.1c scrub ok 2024-02-05T07:21:41.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:41.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1148905732' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-02-05T07:21:41.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[37158]: osdmap e117: 8 total, 8 up, 8 in 2024-02-05T07:21:41.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[37158]: 3.4 scrub starts 2024-02-05T07:21:41.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[37158]: 3.4 scrub ok 2024-02-05T07:21:41.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[37158]: pgmap v101: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-05T07:21:41.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:21:41.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:41 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:21:41.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:41.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:41.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:41.916 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 45 16 74440 74424 103.339 47.375 0.0259124 0.00960568 2024-02-05T07:21:41.919 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:41.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:41.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:42.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:41 smithi090 ceph-mon[33213]: 3.1c scrub starts 2024-02-05T07:21:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:41 smithi090 ceph-mon[33213]: 3.1c scrub ok 2024-02-05T07:21:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:41 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:41 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1148905732' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-02-05T07:21:42.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:41 smithi090 ceph-mon[33213]: osdmap e117: 8 total, 8 up, 8 in 2024-02-05T07:21:42.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:41 smithi090 ceph-mon[33213]: 3.4 scrub starts 2024-02-05T07:21:42.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:41 smithi090 ceph-mon[33213]: 3.4 scrub ok 2024-02-05T07:21:42.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:41 smithi090 ceph-mon[33213]: pgmap v101: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-05T07:21:42.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:41 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:21:42.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:41 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:21:42.044 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:42.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:42.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:42.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:42.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:42.170 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:42.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:42.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:42.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:42.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:42.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:42.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:42.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:42.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:42.548 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:42.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:42.615 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:21:42.649 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:42.675 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:42.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:42.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:42.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:42.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:42.832 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:42.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:42.873 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:42 smithi090 ceph-mon[33213]: 3.d scrub starts 2024-02-05T07:21:42.873 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:42 smithi090 ceph-mon[33213]: 3.d scrub ok 2024-02-05T07:21:42.873 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:42 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:21:42.873 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:42 smithi090 ceph-mon[33213]: osdmap e118: 8 total, 8 up, 8 in 2024-02-05T07:21:42.873 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:42 smithi090 ceph-mon[33213]: 5.1c scrub starts 2024-02-05T07:21:42.873 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:42 smithi090 ceph-mon[33213]: 5.1c scrub ok 2024-02-05T07:21:42.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:42 smithi007 ceph-mon[29098]: 3.d scrub starts 2024-02-05T07:21:42.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:42 smithi007 ceph-mon[29098]: 3.d scrub ok 2024-02-05T07:21:42.913 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:42 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:21:42.913 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:42 smithi007 ceph-mon[29098]: osdmap e118: 8 total, 8 up, 8 in 2024-02-05T07:21:42.913 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:42 smithi007 ceph-mon[29098]: 5.1c scrub starts 2024-02-05T07:21:42.913 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:42 smithi007 ceph-mon[29098]: 5.1c scrub ok 2024-02-05T07:21:42.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:42 smithi007 ceph-mon[37158]: 3.d scrub starts 2024-02-05T07:21:42.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:42 smithi007 ceph-mon[37158]: 3.d scrub ok 2024-02-05T07:21:42.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:42 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:21:42.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:42 smithi007 ceph-mon[37158]: osdmap e118: 8 total, 8 up, 8 in 2024-02-05T07:21:42.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:42 smithi007 ceph-mon[37158]: 5.1c scrub starts 2024-02-05T07:21:42.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:42 smithi007 ceph-mon[37158]: 5.1c scrub ok 2024-02-05T07:21:42.916 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 46 16 74881 74865 101.692 27.5625 0.00830434 0.00982839 2024-02-05T07:21:42.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:42.990 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:42.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:42.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:43.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:43.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:43.116 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:43.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:43.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:43.253 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:43.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:43.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:43.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:43.394 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:43.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:43.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:43.529 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:43.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:43.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:43.665 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:43.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:43.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:43.768 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:43.796 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:43.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:43.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:43.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:43.916 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 47 16 75638 75622 100.535 47.3125 0.00616932 0.00987787 2024-02-05T07:21:43.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:21:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:21:43.924 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[29098]: 3.7 scrub starts 2024-02-05T07:21:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[29098]: 3.7 scrub ok 2024-02-05T07:21:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[29098]: osdmap e119: 8 total, 8 up, 8 in 2024-02-05T07:21:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[29098]: 3.19 scrub starts 2024-02-05T07:21:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[29098]: 3.19 scrub ok 2024-02-05T07:21:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[29098]: 7.1 scrub starts 2024-02-05T07:21:43.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[29098]: pgmap v104: 158 pgs: 10 unknown, 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-02-05T07:21:43.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:21:43.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:21:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[37158]: 3.7 scrub starts 2024-02-05T07:21:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[37158]: 3.7 scrub ok 2024-02-05T07:21:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[37158]: osdmap e119: 8 total, 8 up, 8 in 2024-02-05T07:21:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[37158]: 3.19 scrub starts 2024-02-05T07:21:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[37158]: 3.19 scrub ok 2024-02-05T07:21:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[37158]: 7.1 scrub starts 2024-02-05T07:21:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[37158]: pgmap v104: 158 pgs: 10 unknown, 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-02-05T07:21:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:21:43.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:43 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:21:43.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:43.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:43.927 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:43 smithi090 ceph-mon[33213]: 3.7 scrub starts 2024-02-05T07:21:43.927 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:43 smithi090 ceph-mon[33213]: 3.7 scrub ok 2024-02-05T07:21:43.927 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:43 smithi090 ceph-mon[33213]: osdmap e119: 8 total, 8 up, 8 in 2024-02-05T07:21:43.927 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:43 smithi090 ceph-mon[33213]: 3.19 scrub starts 2024-02-05T07:21:43.927 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:43 smithi090 ceph-mon[33213]: 3.19 scrub ok 2024-02-05T07:21:43.927 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:43 smithi090 ceph-mon[33213]: 7.1 scrub starts 2024-02-05T07:21:43.927 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:43 smithi090 ceph-mon[33213]: pgmap v104: 158 pgs: 10 unknown, 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-02-05T07:21:43.927 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:43 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:21:43.927 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:43 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:21:44.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:44.050 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:44.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:44.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:44.177 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:44.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:44.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:44.303 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:44.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:44.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:44.433 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:44.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:44.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:44.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:44.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:44.560 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:44.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:44.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:21:44.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:44.685 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:44.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:44.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:44.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:44.811 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:44.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:44.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:44.916 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 48 16 75638 75622 98.4413 0 - 0.00987787 2024-02-05T07:21:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: 7.0 scrub starts 2024-02-05T07:21:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: 7.9 scrub starts 2024-02-05T07:21:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: 7.1 scrub ok 2024-02-05T07:21:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: 7.17 scrub starts 2024-02-05T07:21:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: 7.2 scrub starts 2024-02-05T07:21:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: 7.0 scrub ok 2024-02-05T07:21:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: 7.5 scrub starts 2024-02-05T07:21:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: 7.8 scrub starts 2024-02-05T07:21:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T07:21:44.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: osdmap e120: 8 total, 8 up, 8 in 2024-02-05T07:21:44.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: 4.e scrub starts 2024-02-05T07:21:44.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: 4.e scrub ok 2024-02-05T07:21:44.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: 7.8 scrub ok 2024-02-05T07:21:44.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: osdmap e121: 8 total, 8 up, 8 in 2024-02-05T07:21:44.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:44.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:21:44.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:44.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:44.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: 7.0 scrub starts 2024-02-05T07:21:44.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: 7.9 scrub starts 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: 7.1 scrub ok 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: 7.17 scrub starts 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: 7.2 scrub starts 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: 7.0 scrub ok 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: 7.5 scrub starts 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: 7.8 scrub starts 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: osdmap e120: 8 total, 8 up, 8 in 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: 4.e scrub starts 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: 4.e scrub ok 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: 7.8 scrub ok 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: osdmap e121: 8 total, 8 up, 8 in 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:44.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:44 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:44.938 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:44.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: 7.0 scrub starts 2024-02-05T07:21:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: 7.9 scrub starts 2024-02-05T07:21:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: 7.1 scrub ok 2024-02-05T07:21:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: 7.17 scrub starts 2024-02-05T07:21:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: 7.2 scrub starts 2024-02-05T07:21:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: 7.0 scrub ok 2024-02-05T07:21:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: 7.5 scrub starts 2024-02-05T07:21:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: 7.8 scrub starts 2024-02-05T07:21:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T07:21:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: osdmap e120: 8 total, 8 up, 8 in 2024-02-05T07:21:45.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: 4.e scrub starts 2024-02-05T07:21:45.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: 4.e scrub ok 2024-02-05T07:21:45.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: 7.8 scrub ok 2024-02-05T07:21:45.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: osdmap e121: 8 total, 8 up, 8 in 2024-02-05T07:21:45.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:45.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:21:45.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:45.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:44 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:21:45.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:45.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:45.065 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:45.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:45.118 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:21:45.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:45.194 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:45.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:45.296 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:45.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:45.347 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:45.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:45.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:45.472 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:45.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:45.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:45.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:45.598 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:45.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:45.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:45.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:45.729 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:45.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:45.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:45.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:45.880 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:45.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:45.916 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 49 16 75831 75815 96.6787 6.03125 0.0514878 0.0103361 2024-02-05T07:21:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[29098]: 7.5 scrub ok 2024-02-05T07:21:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[29098]: 7.9 scrub ok 2024-02-05T07:21:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[29098]: 7.2 scrub ok 2024-02-05T07:21:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[29098]: 7.4 scrub starts 2024-02-05T07:21:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[29098]: 7.7 deep-scrub starts 2024-02-05T07:21:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[29098]: 6.15 deep-scrub starts 2024-02-05T07:21:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[29098]: 6.15 deep-scrub ok 2024-02-05T07:21:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[29098]: 7.3 scrub starts 2024-02-05T07:21:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[29098]: 7.4 scrub ok 2024-02-05T07:21:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[29098]: 7.3 scrub ok 2024-02-05T07:21:45.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[29098]: pgmap v107: 158 pgs: 4 remapped+peering, 1 active+clean+scrubbing+deep, 6 unknown, 147 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-02-05T07:21:45.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[29098]: osdmap e122: 8 total, 8 up, 8 in 2024-02-05T07:21:45.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[37158]: 7.5 scrub ok 2024-02-05T07:21:45.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[37158]: 7.9 scrub ok 2024-02-05T07:21:45.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[37158]: 7.2 scrub ok 2024-02-05T07:21:45.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[37158]: 7.4 scrub starts 2024-02-05T07:21:45.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[37158]: 7.7 deep-scrub starts 2024-02-05T07:21:45.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[37158]: 6.15 deep-scrub starts 2024-02-05T07:21:45.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[37158]: 6.15 deep-scrub ok 2024-02-05T07:21:45.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[37158]: 7.3 scrub starts 2024-02-05T07:21:45.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[37158]: 7.4 scrub ok 2024-02-05T07:21:45.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[37158]: 7.3 scrub ok 2024-02-05T07:21:45.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[37158]: pgmap v107: 158 pgs: 4 remapped+peering, 1 active+clean+scrubbing+deep, 6 unknown, 147 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-02-05T07:21:45.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:45 smithi007 ceph-mon[37158]: osdmap e122: 8 total, 8 up, 8 in 2024-02-05T07:21:45.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:46.010 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:46.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:45 smithi090 ceph-mon[33213]: 7.5 scrub ok 2024-02-05T07:21:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:45 smithi090 ceph-mon[33213]: 7.9 scrub ok 2024-02-05T07:21:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:45 smithi090 ceph-mon[33213]: 7.2 scrub ok 2024-02-05T07:21:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:45 smithi090 ceph-mon[33213]: 7.4 scrub starts 2024-02-05T07:21:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:45 smithi090 ceph-mon[33213]: 7.7 deep-scrub starts 2024-02-05T07:21:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:45 smithi090 ceph-mon[33213]: 6.15 deep-scrub starts 2024-02-05T07:21:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:45 smithi090 ceph-mon[33213]: 6.15 deep-scrub ok 2024-02-05T07:21:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:45 smithi090 ceph-mon[33213]: 7.3 scrub starts 2024-02-05T07:21:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:45 smithi090 ceph-mon[33213]: 7.4 scrub ok 2024-02-05T07:21:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:45 smithi090 ceph-mon[33213]: 7.3 scrub ok 2024-02-05T07:21:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:45 smithi090 ceph-mon[33213]: pgmap v107: 158 pgs: 4 remapped+peering, 1 active+clean+scrubbing+deep, 6 unknown, 147 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-02-05T07:21:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:45 smithi090 ceph-mon[33213]: osdmap e122: 8 total, 8 up, 8 in 2024-02-05T07:21:46.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:46.145 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:21:46.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:46.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:46.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:46.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:46.319 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:46.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:46.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:46.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:46.448 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:46.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:46.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:46.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:46.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:46 smithi007 ceph-mon[37158]: 7.6 scrub starts 2024-02-05T07:21:46.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:46.618 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:46.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:46.719 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:46.769 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:46.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:46.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:46.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:46.901 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:46.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:46.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:46.916 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 50 16 76497 76481 95.5777 41.625 0.0224818 0.010457 2024-02-05T07:21:46.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:46 smithi007 ceph-mon[29098]: 7.6 scrub starts 2024-02-05T07:21:47.003 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:46 smithi090 ceph-mon[33213]: 7.6 scrub starts 2024-02-05T07:21:47.031 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:47.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:47.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:47.126 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T07:21:47.126 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:21:47.127 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-02-05T07:21:47.127 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-05T07:21:47.127 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-05T07:21:47.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-02-05T07:21:47.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:47.159 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:47.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:47.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:47.288 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:47.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:47.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:47.390 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:47.421 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:47.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:47.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:47.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:47.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:47.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:47.719 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:47.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:47.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:47.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:47.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:47.916 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:47 smithi007 ceph-mon[29098]: 7.6 scrub ok 2024-02-05T07:21:47.916 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:47 smithi007 ceph-mon[29098]: pgmap v109: 158 pgs: 10 remapped+peering, 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 44 KiB/s, 0 objects/s recovering 2024-02-05T07:21:47.916 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 51 16 77140 77124 94.4917 40.1875 0.0257827 0.0105756 2024-02-05T07:21:47.917 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:47 smithi007 ceph-mon[37158]: 7.6 scrub ok 2024-02-05T07:21:47.917 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:47 smithi007 ceph-mon[37158]: pgmap v109: 158 pgs: 10 remapped+peering, 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 44 KiB/s, 0 objects/s recovering 2024-02-05T07:21:47.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:47.995 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:47.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:47.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:47 smithi090 ceph-mon[33213]: 7.6 scrub ok 2024-02-05T07:21:48.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:47 smithi090 ceph-mon[33213]: pgmap v109: 158 pgs: 10 remapped+peering, 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 44 KiB/s, 0 objects/s recovering 2024-02-05T07:21:48.097 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:48.139 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:48.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:48.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:48.241 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:48.269 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:48.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:48.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:48.371 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:48.414 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:48.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:48.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:48.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:48.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:48.648 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:21:48.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:48.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:48.735 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:48.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:48.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:48.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:48.899 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:48.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:48.918 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 52 16 77799 77783 93.4667 41.1875 0.00951991 0.0106917 2024-02-05T07:21:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:48 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1802436138' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:21:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:48 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:21:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:48 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1802436138' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:21:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:48 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:21:49.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:48 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1802436138' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:21:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:48 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:21:49.029 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:49.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:49.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:49.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:49.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:49.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:49.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:49.294 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:49.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:49.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:49.426 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:49.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:49.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:49.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:49.574 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:49.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:49.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:49.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:49.723 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:49.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:49.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:49.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:49.854 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:49.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:49.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:49.917 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 53 16 78519 78503 92.5523 45 0.0147931 0.0107982 2024-02-05T07:21:49.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:49 smithi007 ceph-mon[29098]: 3.1d scrub starts 2024-02-05T07:21:49.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:49 smithi007 ceph-mon[29098]: 3.1d scrub ok 2024-02-05T07:21:49.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:49 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:21:49.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:49 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:21:49.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:49 smithi007 ceph-mon[29098]: osdmap e123: 8 total, 8 up, 8 in 2024-02-05T07:21:49.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:49 smithi007 ceph-mon[29098]: pgmap v111: 158 pgs: 10 remapped+peering, 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 957 B/s rd, 0 op/s; 36 KiB/s, 0 objects/s recovering 2024-02-05T07:21:49.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:49 smithi007 ceph-mon[37158]: 3.1d scrub starts 2024-02-05T07:21:49.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:49 smithi007 ceph-mon[37158]: 3.1d scrub ok 2024-02-05T07:21:49.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:49 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:21:49.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:49 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:21:49.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:49 smithi007 ceph-mon[37158]: osdmap e123: 8 total, 8 up, 8 in 2024-02-05T07:21:49.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:49 smithi007 ceph-mon[37158]: pgmap v111: 158 pgs: 10 remapped+peering, 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 957 B/s rd, 0 op/s; 36 KiB/s, 0 objects/s recovering 2024-02-05T07:21:49.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:49 smithi090 ceph-mon[33213]: 3.1d scrub starts 2024-02-05T07:21:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:49 smithi090 ceph-mon[33213]: 3.1d scrub ok 2024-02-05T07:21:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:49 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:21:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:49 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:21:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:49 smithi090 ceph-mon[33213]: osdmap e123: 8 total, 8 up, 8 in 2024-02-05T07:21:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:49 smithi090 ceph-mon[33213]: pgmap v111: 158 pgs: 10 remapped+peering, 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 957 B/s rd, 0 op/s; 36 KiB/s, 0 objects/s recovering 2024-02-05T07:21:50.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:50.043 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:50.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:50.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:50.177 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:50.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:50.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:50.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:50.332 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:50.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:50.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:50.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:50.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:50.464 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:50.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:50.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:21:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:21:50.565 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:50.592 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:50.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:50.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:50.602 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:21:50.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:50.769 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:50.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:50.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:50 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:21:50.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:50 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:21:50.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:50 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:50.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:50 smithi007 ceph-mon[29098]: osdmap e124: 8 total, 8 up, 8 in 2024-02-05T07:21:50.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:50 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:21:50.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:50 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:21:50.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:50 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:50.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:50 smithi007 ceph-mon[37158]: osdmap e124: 8 total, 8 up, 8 in 2024-02-05T07:21:50.871 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:50.917 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 54 16 79189 79173 91.6128 41.875 0.0522179 0.0109075 2024-02-05T07:21:50.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:50.932 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:50.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:51.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:50 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:21:51.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:50 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:21:51.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:50 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:21:51.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:50 smithi090 ceph-mon[33213]: osdmap e124: 8 total, 8 up, 8 in 2024-02-05T07:21:51.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:51.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:51.064 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:51.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:51.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:51.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:51.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:51.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:51.344 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:51.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:51.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:51.471 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:51.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:51.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:51.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:51.601 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:51.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:51.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:51.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:51.755 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:51.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:51.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:51.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:51.887 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:51.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:51.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:51.918 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 55 16 79818 79802 90.662 39.3125 0.0229557 0.0110232 2024-02-05T07:21:51.991 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:51 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:21:52.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:51 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:21:52.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:51 smithi090 ceph-mon[33213]: pgmap v113: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 16564/236139 objects degraded (7.015%); 26793/236139 objects misplaced (11.346%); 12 MiB/s, 184 objects/s recovering 2024-02-05T07:21:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:52.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:52.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:52.036 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:21:52.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:51 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:21:52.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:51 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:21:52.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:51 smithi007 ceph-mon[29098]: pgmap v113: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 16564/236139 objects degraded (7.015%); 26793/236139 objects misplaced (11.346%); 12 MiB/s, 184 objects/s recovering 2024-02-05T07:21:52.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:51 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:21:52.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:51 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:21:52.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:51 smithi007 ceph-mon[37158]: pgmap v113: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 16564/236139 objects degraded (7.015%); 26793/236139 objects misplaced (11.346%); 12 MiB/s, 184 objects/s recovering 2024-02-05T07:21:52.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:52.165 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:52.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:52.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:52.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:52.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:52.421 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:52.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:52.472 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:52.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:52.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:52.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:52.611 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:52.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:52.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:52.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:52.743 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:52.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:52.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:52.888 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:52.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:52.917 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 56 16 80543 80527 89.8522 45.3125 0.0108437 0.0111188 2024-02-05T07:21:52.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:53.020 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:53.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:52 smithi090 ceph-mon[33213]: 7.7 deep-scrub ok 2024-02-05T07:21:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:52 smithi090 ceph-mon[33213]: Health check failed: Degraded data redundancy: 16564/236139 objects degraded (7.015%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:21:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:52 smithi090 ceph-mon[33213]: osdmap e125: 8 total, 8 up, 8 in 2024-02-05T07:21:53.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:53.172 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:53.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:53.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:52 smithi007 ceph-mon[29098]: 7.7 deep-scrub ok 2024-02-05T07:21:53.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:52 smithi007 ceph-mon[29098]: Health check failed: Degraded data redundancy: 16564/236139 objects degraded (7.015%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:21:53.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:52 smithi007 ceph-mon[29098]: osdmap e125: 8 total, 8 up, 8 in 2024-02-05T07:21:53.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:52 smithi007 ceph-mon[37158]: 7.7 deep-scrub ok 2024-02-05T07:21:53.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:52 smithi007 ceph-mon[37158]: Health check failed: Degraded data redundancy: 16564/236139 objects degraded (7.015%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:21:53.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:52 smithi007 ceph-mon[37158]: osdmap e125: 8 total, 8 up, 8 in 2024-02-05T07:21:53.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:53.306 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:53.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:53.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:53.437 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:53.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:53.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:53.539 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:53.582 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:53.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:53.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:53.710 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:53.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:53.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:53.812 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:53.864 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:53.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:53.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:53.918 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 57 16 81241 81225 89.0413 43.625 0.0118523 0.0112254 2024-02-05T07:21:53.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:53 smithi007 ceph-mon[29098]: pgmap v115: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 5.2 MiB/s wr, 84 op/s; 16564/236139 objects degraded (7.015%); 26793/236139 objects misplaced (11.346%); 11 MiB/s, 183 objects/s recovering 2024-02-05T07:21:53.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:21:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:21:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:21:53.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:53 smithi007 ceph-mon[37158]: pgmap v115: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 5.2 MiB/s wr, 84 op/s; 16564/236139 objects degraded (7.015%); 26793/236139 objects misplaced (11.346%); 11 MiB/s, 183 objects/s recovering 2024-02-05T07:21:53.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:53.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:54.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:53 smithi090 ceph-mon[33213]: pgmap v115: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 5.2 MiB/s wr, 84 op/s; 16564/236139 objects degraded (7.015%); 26793/236139 objects misplaced (11.346%); 11 MiB/s, 183 objects/s recovering 2024-02-05T07:21:54.097 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:54.141 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:54.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:54.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:54.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:54.266 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:54.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:54.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:54.397 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:54.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:54.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:54.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:54.542 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:54.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:54.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:54.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:54.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:54.684 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:54.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:54.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:54.813 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:54.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:54.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:54.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:54.918 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 58 16 81954 81938 88.2745 44.5625 0.0130663 0.0113216 2024-02-05T07:21:54.943 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:54.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:54.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:55.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:55.091 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:55.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:55.193 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:55.222 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:55.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:55.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:55.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:55 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3122981822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:21:55.239 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:55 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3122981822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:21:55.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:55.366 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:55.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:55.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:55.492 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:55.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:55.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:55.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:55 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3122981822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:21:55.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:55.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:55.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:55.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:55.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:55.778 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:55.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:55.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:55.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:55.911 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:55.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:55.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:55.918 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 59 16 82624 82608 87.4881 41.875 0.0124984 0.0114244 2024-02-05T07:21:56.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:56.045 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:56.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:56.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:56.147 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:56.175 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:56.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:56.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:56.223 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:21:56.223 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:21:56.223 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:21:56.223 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:21:56.223 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:21:56.224 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:21:56.224 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:21:56.224 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:21:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:21:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:21:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:21:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:21:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:21:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:21:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:21:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:21:56.277 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:56.326 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:56.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:56.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:56.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:56 smithi007 ceph-mon[29098]: pgmap v116: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.6 KiB/s rd, 9.6 MiB/s wr, 155 op/s; 16564/237546 objects degraded (6.973%); 26428/237546 objects misplaced (11.125%); 15 MiB/s, 232 objects/s recovering 2024-02-05T07:21:56.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:56 smithi007 ceph-mon[37158]: pgmap v116: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.6 KiB/s rd, 9.6 MiB/s wr, 155 op/s; 16564/237546 objects degraded (6.973%); 26428/237546 objects misplaced (11.125%); 15 MiB/s, 232 objects/s recovering 2024-02-05T07:21:56.425 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:56 smithi090 ceph-mon[33213]: pgmap v116: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.6 KiB/s rd, 9.6 MiB/s wr, 155 op/s; 16564/237546 objects degraded (6.973%); 26428/237546 objects misplaced (11.125%); 15 MiB/s, 232 objects/s recovering 2024-02-05T07:21:56.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:56.474 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:56.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:56.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:56.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:56.621 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:56.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:56.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:56.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:56.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:56.883 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:56.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:56.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:56.918 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:21:56.917457+0000 min lat: 0.00125178 max lat: 2.10748 avg lat: 0.0115098 2024-02-05T07:21:56.918 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:21:56.918 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 60 16 83404 83388 86.8424 48.75 0.0132308 0.0115098 2024-02-05T07:21:56.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:57.013 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:57.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:57.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:57.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:57.165 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:57.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:57.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:57.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:57.289 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:57.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:57.390 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:57.420 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:57.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:57.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:57.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:57 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 16564/246504 objects degraded (6.720%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:21:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:57.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:57.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:57.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:57.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:57 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 16564/246504 objects degraded (6.720%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:21:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:57 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 16564/246504 objects degraded (6.720%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:21:57.679 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:57.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:57.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:57.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:57.805 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:57.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:57.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:57.918 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 61 16 84044 84028 86.0744 40 0.0170523 0.0116108 2024-02-05T07:21:57.952 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:57.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:58.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:58.094 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:58.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:58.195 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:58.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:58.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:58.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:58.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:21:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:58.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:58.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:21:58.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:21:58.499 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:58.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:21:58.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:21:58 smithi090 ceph-mon[33213]: pgmap v117: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 197 op/s; 16564/246504 objects degraded (6.720%); 25124/246504 objects misplaced (10.192%); 18 MiB/s, 288 objects/s recovering 2024-02-05T07:21:58.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:58.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:58.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:21:58 smithi007 ceph-mon[29098]: pgmap v117: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 197 op/s; 16564/246504 objects degraded (6.720%); 25124/246504 objects misplaced (10.192%); 18 MiB/s, 288 objects/s recovering 2024-02-05T07:21:58.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:21:58 smithi007 ceph-mon[37158]: pgmap v117: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 197 op/s; 16564/246504 objects degraded (6.720%); 25124/246504 objects misplaced (10.192%); 18 MiB/s, 288 objects/s recovering 2024-02-05T07:21:58.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:21:58.770 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:58.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:58.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:21:58.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:58.901 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:58.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:58.918 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 62 16 84764 84748 85.4119 45 0.0150832 0.0117018 2024-02-05T07:21:59.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:59.041 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:59.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:59.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:59.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:21:59.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:59.167 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:21:59.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:21:59.268 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:59.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:59.315 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:59.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:59.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:21:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:59.460 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:59.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:21:59.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:21:59.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:59.607 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:59.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:21:59.708 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:21:59.755 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:59.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:21:59.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:21:59.903 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:21:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:21:59.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:21:59.918 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 63 16 85482 85466 84.7685 44.875 0.0130035 0.0117907 2024-02-05T07:22:00.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:00.035 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:00.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:00.137 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:00.184 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:00.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:00.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:00.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:00.337 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:00.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:00.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:00.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:00 smithi007 ceph-mon[29098]: pgmap v118: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 580 B/s rd, 7.5 MiB/s wr, 121 op/s; 16564/246504 objects degraded (6.720%); 25124/246504 objects misplaced (10.192%); 8.5 MiB/s, 136 objects/s recovering 2024-02-05T07:22:00.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:00 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:22:00.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:00 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:22:00.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:00 smithi007 ceph-mon[37158]: pgmap v118: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 580 B/s rd, 7.5 MiB/s wr, 121 op/s; 16564/246504 objects degraded (6.720%); 25124/246504 objects misplaced (10.192%); 8.5 MiB/s, 136 objects/s recovering 2024-02-05T07:22:00.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:00 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:22:00.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:00 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:22:00.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:00.463 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:00.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:00.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:22:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:22:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:00 smithi090 ceph-mon[33213]: pgmap v118: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 580 B/s rd, 7.5 MiB/s wr, 121 op/s; 16564/246504 objects degraded (6.720%); 25124/246504 objects misplaced (10.192%); 8.5 MiB/s, 136 objects/s recovering 2024-02-05T07:22:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:00 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:22:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:00 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:22:00.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:00.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:00.591 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:00.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:00.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:00.720 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:00.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:00.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:00.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:00.854 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:00.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:00.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:00.918 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 64 16 86183 86167 84.1286 43.8125 0.0156602 0.0118804 2024-02-05T07:22:00.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:00.986 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:00.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:01.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:01.116 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:01.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:01.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:01.218 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:01.241 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:01.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:01.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:01.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:01.386 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:01.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:01.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:01.457 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:01.476 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:01.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:01.518 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:01.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:01.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:01.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:01.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:01.648 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:01.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:01.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:01.783 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:01.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:01.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:01.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:01.915 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:01.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:01.918 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 65 16 86878 86862 83.5026 43.4375 0.0232156 0.0119703 2024-02-05T07:22:02.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:02.044 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:02.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:02.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:02.147 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:02.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:02.173 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:02.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:02.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:02.303 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:02.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:02.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:02.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:02.449 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:02.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:02.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:02.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:02 smithi090 ceph-mon[33213]: pgmap v119: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 29 MiB/s wr, 464 op/s; 16564/257232 objects degraded (6.439%); 23455/257232 objects misplaced (9.118%); 15 MiB/s, 244 objects/s recovering 2024-02-05T07:22:02.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:02.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:02.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:02.649 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:02 smithi007 ceph-mon[29098]: pgmap v119: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 29 MiB/s wr, 464 op/s; 16564/257232 objects degraded (6.439%); 23455/257232 objects misplaced (9.118%); 15 MiB/s, 244 objects/s recovering 2024-02-05T07:22:02.650 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:02 smithi007 ceph-mon[37158]: pgmap v119: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 29 MiB/s wr, 464 op/s; 16564/257232 objects degraded (6.439%); 23455/257232 objects misplaced (9.118%); 15 MiB/s, 244 objects/s recovering 2024-02-05T07:22:02.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:02.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:02.740 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:02.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:02.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:02.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:02.870 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:02.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:02.919 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 66 16 87578 87562 82.9003 43.75 0.014002 0.012057 2024-02-05T07:22:02.970 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:03.011 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:03.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:03.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:03.113 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:03.157 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:03.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:03.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:03.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:03.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:03.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:03.455 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:03.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:03.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:03.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:03.588 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:03.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:03.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:03.690 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:03.734 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:03.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:03.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:03.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:03.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:03.871 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:03.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:03.919 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 67 16 88380 88364 82.4111 50.125 0.015465 0.0121281 2024-02-05T07:22:03.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:22:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:22:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:22:03.972 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:03.999 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:04.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:04.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:04.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:04.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:04.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:04.304 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:04.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:04.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:04.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:04 smithi007 ceph-mon[29098]: pgmap v120: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 907 B/s rd, 26 MiB/s wr, 412 op/s; 16564/257232 objects degraded (6.439%); 23455/257232 objects misplaced (9.118%); 14 MiB/s, 217 objects/s recovering 2024-02-05T07:22:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:04 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 16564/257232 objects degraded (6.439%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:04 smithi007 ceph-mon[37158]: pgmap v120: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 907 B/s rd, 26 MiB/s wr, 412 op/s; 16564/257232 objects degraded (6.439%); 23455/257232 objects misplaced (9.118%); 14 MiB/s, 217 objects/s recovering 2024-02-05T07:22:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:04 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 16564/257232 objects degraded (6.439%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:04.450 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:04.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:04.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:04.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:04 smithi090 ceph-mon[33213]: pgmap v120: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 907 B/s rd, 26 MiB/s wr, 412 op/s; 16564/257232 objects degraded (6.439%); 23455/257232 objects misplaced (9.118%); 14 MiB/s, 217 objects/s recovering 2024-02-05T07:22:04.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:04 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 16564/257232 objects degraded (6.439%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:04.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:04.581 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:04.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:04.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:04.713 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:04.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:04.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:04.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:04.842 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:04.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:04.919 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 68 16 88984 88968 81.7544 37.75 0.0266312 0.0122249 2024-02-05T07:22:04.943 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:04.990 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:04.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:04.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:05.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:05.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:05.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:05.240 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:05.267 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:05.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:05.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:05.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:05.406 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:05.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:05.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:05.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:05.539 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:05.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:05.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:05.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:05.691 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:05.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:05.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:05.859 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:05.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:05.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:05.919 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 69 16 89652 89636 81.1746 41.75 0.0202116 0.0123126 2024-02-05T07:22:05.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:05.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:06.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:06.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:06.142 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:06.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:06.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:06.277 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:06 smithi007 ceph-mon[29098]: pgmap v121: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 26 MiB/s wr, 424 op/s; 16564/258546 objects degraded (6.407%); 23024/258546 objects misplaced (8.905%); 15 MiB/s, 239 objects/s recovering 2024-02-05T07:22:06.278 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:06 smithi007 ceph-mon[37158]: pgmap v121: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 26 MiB/s wr, 424 op/s; 16564/258546 objects degraded (6.407%); 23024/258546 objects misplaced (8.905%); 15 MiB/s, 239 objects/s recovering 2024-02-05T07:22:06.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:06.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:06.399 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:06.427 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:06.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:06.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:06.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:06 smithi090 ceph-mon[33213]: pgmap v121: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 26 MiB/s wr, 424 op/s; 16564/258546 objects degraded (6.407%); 23024/258546 objects misplaced (8.905%); 15 MiB/s, 239 objects/s recovering 2024-02-05T07:22:06.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:06.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:06.561 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:06.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:06.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:06.692 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:06.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:06.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:06.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:06.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:06.922 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 70 16 90385 90369 80.6664 45.8125 0.0140303 0.0123906 2024-02-05T07:22:06.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:06.998 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:07.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:07.057 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:22:07.057 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:22:07.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:07.129 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:07.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:07.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:07.262 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:07.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:07.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:07.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:07.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:07.420 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:07.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:07.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:07.550 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:07.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:07.651 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:07.680 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:07.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:07.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:07.688 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:22:07.688 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-05T07:22:07.688 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:22:07.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:07.811 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:07.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:07.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:07.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:07.922 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 71 16 91138 91122 80.1931 47.0625 0.0112807 0.0124635 2024-02-05T07:22:07.942 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:07.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:07.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:08.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:08.104 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:08.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:08.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:08.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T07:22:08.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:08.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:08.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:08.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:08 smithi007 ceph-mon[29098]: pgmap v122: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 40 MiB/s wr, 639 op/s; 16564/267717 objects degraded (6.187%); 21704/267717 objects misplaced (8.107%); 18 MiB/s, 281 objects/s recovering 2024-02-05T07:22:08.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:08 smithi007 ceph-mon[37158]: pgmap v122: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 40 MiB/s wr, 639 op/s; 16564/267717 objects degraded (6.187%); 21704/267717 objects misplaced (8.107%); 18 MiB/s, 281 objects/s recovering 2024-02-05T07:22:08.432 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:08.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:08.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:08.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:08 smithi090 ceph-mon[33213]: pgmap v122: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 40 MiB/s wr, 639 op/s; 16564/267717 objects degraded (6.187%); 21704/267717 objects misplaced (8.107%); 18 MiB/s, 281 objects/s recovering 2024-02-05T07:22:08.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:08.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:08.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:08.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:08.732 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:08.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:08.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:08.810 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:22:08.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:08.859 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:08.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:08.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:08.922 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 72 16 91900 91884 79.7404 47.625 0.0232483 0.0125355 2024-02-05T07:22:08.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:08.992 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:08.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:08.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:09.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:09.137 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:09.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:09.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:09.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:09.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:09.345 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:22:09.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:09 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 16564/267717 objects degraded (6.187%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:09.396 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:09 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 16564/267717 objects degraded (6.187%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:09.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:09.466 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:09.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:09.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:09 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 16564/267717 objects degraded (6.187%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:09.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:09.620 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:09.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:09.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:09.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:09.746 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:09.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:09.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:09.879 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:09.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:09.922 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 73 16 92608 92592 79.2542 44.25 0.0129185 0.0126112 2024-02-05T07:22:09.980 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:10.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:10.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:10.163 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:10.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:10.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:10.295 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:10.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:10.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:10.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:10 smithi007 ceph-mon[29098]: pgmap v123: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 37 MiB/s wr, 589 op/s; 16564/267717 objects degraded (6.187%); 21704/267717 objects misplaced (8.107%); 13 MiB/s, 212 objects/s recovering 2024-02-05T07:22:10.303 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:10 smithi007 ceph-mon[37158]: pgmap v123: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 37 MiB/s wr, 589 op/s; 16564/267717 objects degraded (6.187%); 21704/267717 objects misplaced (8.107%); 13 MiB/s, 212 objects/s recovering 2024-02-05T07:22:10.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:10.440 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:10.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:10.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:10 smithi090 ceph-mon[33213]: pgmap v123: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 37 MiB/s wr, 589 op/s; 16564/267717 objects degraded (6.187%); 21704/267717 objects misplaced (8.107%); 13 MiB/s, 212 objects/s recovering 2024-02-05T07:22:10.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:22:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:22:10.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:10.566 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:10.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:10.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:10.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:22:10.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:10.736 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:10.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:10.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:10.911 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:10.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:10.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:10.922 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 74 16 93395 93379 78.8478 49.1875 0.0319877 0.0126759 2024-02-05T07:22:11.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:11.042 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:11.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:11.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:11.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:11.174 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:11.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:11.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:11.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:11.306 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:11.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:11.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:11.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:11 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:11.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:11 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:11.431 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:11.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:11.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:11.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:11 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:11.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:11.580 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:11.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:11.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:11.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:11.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:11.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:11.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:11.836 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:11.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:11.923 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 75 16 94143 94127 78.4193 46.75 0.0080959 0.0127462 2024-02-05T07:22:11.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:11.978 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:11.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:11.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:12.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:12.108 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:12.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:12.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:12.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:12.238 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:12.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:12.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:12.383 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:12.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:12.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:12.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:12.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:12 smithi090 ceph-mon[33213]: pgmap v124: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 891 op/s; 16564/278571 objects degraded (5.946%); 19758/278571 objects misplaced (7.093%); 21 MiB/s, 332 objects/s recovering 2024-02-05T07:22:12.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:12.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:12.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:12.664 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:12.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:12.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:12.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:12 smithi007 ceph-mon[29098]: pgmap v124: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 891 op/s; 16564/278571 objects degraded (5.946%); 19758/278571 objects misplaced (7.093%); 21 MiB/s, 332 objects/s recovering 2024-02-05T07:22:12.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:12 smithi007 ceph-mon[37158]: pgmap v124: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 891 op/s; 16564/278571 objects degraded (5.946%); 19758/278571 objects misplaced (7.093%); 21 MiB/s, 332 objects/s recovering 2024-02-05T07:22:12.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:12.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:12.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:12.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:12.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:12.923 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 76 16 94871 94855 77.9861 45.5 0.00341662 0.012809 2024-02-05T07:22:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:12.939 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:12.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:13.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:13.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:13.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:13.169 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:13.210 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:13.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:13.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:13.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:13.353 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:13.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:13.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:13.455 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:13.482 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:13.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:13.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:13.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:13.636 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:13.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:13.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:13.782 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:13.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:13.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:13.925 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 77 16 95712 95696 77.6558 52.5625 0.0188809 0.0128718 2024-02-05T07:22:13.926 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:22:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:22:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:22:13.929 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:13.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:13.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:14.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:14.104 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:14.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:14.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:14.257 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:14.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:14.277 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T07:22:14.277 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:22:14.277 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-05T07:22:14.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T07:22:14.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:14.417 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:14.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:14.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:14.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:14.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:14 smithi090 ceph-mon[33213]: pgmap v125: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 37 MiB/s wr, 593 op/s; 16564/278571 objects degraded (5.946%); 19758/278571 objects misplaced (7.093%); 14 MiB/s, 228 objects/s recovering 2024-02-05T07:22:14.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:14 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 16564/278571 objects degraded (5.946%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:14.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:14 smithi090 ceph-mon[33213]: osdmap e126: 8 total, 8 up, 8 in 2024-02-05T07:22:14.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:14 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:22:14.546 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:14.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:14.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:14.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:14.674 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:14.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:14.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:14 smithi007 ceph-mon[29098]: pgmap v125: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 37 MiB/s wr, 593 op/s; 16564/278571 objects degraded (5.946%); 19758/278571 objects misplaced (7.093%); 14 MiB/s, 228 objects/s recovering 2024-02-05T07:22:14.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:14 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 16564/278571 objects degraded (5.946%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:14.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:14 smithi007 ceph-mon[29098]: osdmap e126: 8 total, 8 up, 8 in 2024-02-05T07:22:14.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:14 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:22:14.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:14 smithi007 ceph-mon[37158]: pgmap v125: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 37 MiB/s wr, 593 op/s; 16564/278571 objects degraded (5.946%); 19758/278571 objects misplaced (7.093%); 14 MiB/s, 228 objects/s recovering 2024-02-05T07:22:14.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:14 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 16564/278571 objects degraded (5.946%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:14.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:14 smithi007 ceph-mon[37158]: osdmap e126: 8 total, 8 up, 8 in 2024-02-05T07:22:14.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:14 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:22:14.776 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:14.812 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:14.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:14.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:14.923 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 78 16 96262 96246 77.101 34.375 0.00506793 0.0128969 2024-02-05T07:22:14.949 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:14.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:14.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:15.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:15.076 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:15.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:15.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:15.178 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:15.207 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:15.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:15.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:15.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:15.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:15.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:15.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:15.483 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:15.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:15.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:15.614 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:15.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:15.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:15.715 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:15.744 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:15.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:15.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:15.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:15.872 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:15.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:15.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:15.924 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 79 16 96840 96824 76.5824 36.125 0.0211432 0.0130523 2024-02-05T07:22:15.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:16.018 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:16.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:16.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:16.153 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:16.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:16.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:16.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:16.307 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:16.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:16.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:16.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:16 smithi007 ceph-mon[29098]: pgmap v127: 158 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 148 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 48 MiB/s wr, 772 op/s; 16564/281703 objects degraded (5.880%); 19327/281703 objects misplaced (6.861%); 17 MiB/s, 273 objects/s recovering 2024-02-05T07:22:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:16 smithi007 ceph-mon[29098]: osdmap e127: 8 total, 8 up, 8 in 2024-02-05T07:22:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:16 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:22:16.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:16 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:22:16.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:16 smithi007 ceph-mon[37158]: pgmap v127: 158 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 148 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 48 MiB/s wr, 772 op/s; 16564/281703 objects degraded (5.880%); 19327/281703 objects misplaced (6.861%); 17 MiB/s, 273 objects/s recovering 2024-02-05T07:22:16.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:16 smithi007 ceph-mon[37158]: osdmap e127: 8 total, 8 up, 8 in 2024-02-05T07:22:16.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:16 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:22:16.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:16 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:22:16.436 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:16.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:16.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:16.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:16 smithi090 ceph-mon[33213]: pgmap v127: 158 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 148 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 48 MiB/s wr, 772 op/s; 16564/281703 objects degraded (5.880%); 19327/281703 objects misplaced (6.861%); 17 MiB/s, 273 objects/s recovering 2024-02-05T07:22:16.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:16 smithi090 ceph-mon[33213]: osdmap e127: 8 total, 8 up, 8 in 2024-02-05T07:22:16.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:16 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:22:16.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:16 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:22:16.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:16.567 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:16.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:16.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:16.669 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:16.711 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:16.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:16.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:16.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:16.866 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:16.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:16.924 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:22:16.923184+0000 min lat: 0.00125178 max lat: 2.10748 avg lat: 0.0131157 2024-02-05T07:22:16.924 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:22:16.924 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 80 16 97593 97577 76.2134 47.0625 0.00941691 0.0131157 2024-02-05T07:22:16.969 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:17.011 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:17.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:17.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:17.113 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:17.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:17.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:17.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:17.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:17.302 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:17.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:17.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:17.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:17 smithi007 ceph-mon[29098]: osdmap e128: 8 total, 8 up, 8 in 2024-02-05T07:22:17.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:17 smithi007 ceph-mon[37158]: osdmap e128: 8 total, 8 up, 8 in 2024-02-05T07:22:17.446 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:17.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:17.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:17 smithi090 ceph-mon[33213]: osdmap e128: 8 total, 8 up, 8 in 2024-02-05T07:22:17.548 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:17.577 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:17.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:17.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:17.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:17.704 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:17.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:17.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:17.836 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:17.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:17.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:17.924 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 81 16 98382 98366 75.8812 49.3125 0.0239734 0.0131732 2024-02-05T07:22:17.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:17.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:17.990 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:17.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:18.091 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:18.119 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:18.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:18.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:18.249 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:18.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:18.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:18.412 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:18.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:18.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:18.559 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:18.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:18.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:18.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:18 smithi007 ceph-mon[29098]: pgmap v130: 158 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 37 MiB/s wr, 598 op/s; 16564/289335 objects degraded (5.725%); 17302/289335 objects misplaced (5.980%); 19 MiB/s, 304 objects/s recovering 2024-02-05T07:22:18.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:18 smithi007 ceph-mon[37158]: pgmap v130: 158 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 37 MiB/s wr, 598 op/s; 16564/289335 objects degraded (5.725%); 17302/289335 objects misplaced (5.980%); 19 MiB/s, 304 objects/s recovering 2024-02-05T07:22:18.703 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:18.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:18.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:18.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:18 smithi090 ceph-mon[33213]: pgmap v130: 158 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 37 MiB/s wr, 598 op/s; 16564/289335 objects degraded (5.725%); 17302/289335 objects misplaced (5.980%); 19 MiB/s, 304 objects/s recovering 2024-02-05T07:22:18.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:18.831 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:18.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:18.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:18.924 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 82 16 99081 99065 75.4887 43.6875 0.00428953 0.0132081 2024-02-05T07:22:18.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:18.959 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:18.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:18.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:19.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:19.085 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:19.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:19.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:19.239 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:19.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:19.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:19.397 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:19.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:19.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:19 smithi007 ceph-mon[29098]: osdmap e129: 8 total, 8 up, 8 in 2024-02-05T07:22:19.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:19 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:22:19.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:19 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:22:19.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:19 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 16564/289335 objects degraded (5.725%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:19.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:19 smithi007 ceph-mon[29098]: osdmap e130: 8 total, 8 up, 8 in 2024-02-05T07:22:19.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:19 smithi007 ceph-mon[37158]: osdmap e129: 8 total, 8 up, 8 in 2024-02-05T07:22:19.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:19 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:22:19.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:19 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:22:19.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:19 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 16564/289335 objects degraded (5.725%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:19.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:19 smithi007 ceph-mon[37158]: osdmap e130: 8 total, 8 up, 8 in 2024-02-05T07:22:19.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:19.525 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:19.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:19.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:19.683 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:19.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:19.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:19 smithi090 ceph-mon[33213]: osdmap e129: 8 total, 8 up, 8 in 2024-02-05T07:22:19.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:19 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:22:19.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:19 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:22:19.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:19 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 16564/289335 objects degraded (5.725%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:19.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:19 smithi090 ceph-mon[33213]: osdmap e130: 8 total, 8 up, 8 in 2024-02-05T07:22:19.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:19.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:19.857 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:19.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:19.926 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 83 16 99596 99580 74.9657 32.1875 0.015092 0.0133331 2024-02-05T07:22:19.958 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:20.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:20.035 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:20.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:20.043 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:22:20.043 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:22:20.137 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:20.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:20.180 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:20.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:20.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:20.293 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:22:20.300 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:20 smithi007 ceph-mon[37158]: pgmap v132: 158 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 47 MiB/s wr, 745 op/s; 16564/289335 objects degraded (5.725%); 17302/289335 objects misplaced (5.980%); 24 MiB/s, 379 objects/s recovering 2024-02-05T07:22:20.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:20.362 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:20.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:20.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:20.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:20.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:20.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:20 smithi090 ceph-mon[33213]: pgmap v132: 158 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 47 MiB/s wr, 745 op/s; 16564/289335 objects degraded (5.725%); 17302/289335 objects misplaced (5.980%); 24 MiB/s, 379 objects/s recovering 2024-02-05T07:22:20.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:22:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:22:20.590 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:20.618 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:20.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:20.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:20.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:20 smithi007 ceph-mon[29098]: pgmap v132: 158 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 47 MiB/s wr, 745 op/s; 16564/289335 objects degraded (5.725%); 17302/289335 objects misplaced (5.980%); 24 MiB/s, 379 objects/s recovering 2024-02-05T07:22:20.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:20.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:20.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:20.892 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:20.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:20.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:20.925 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 84 16 100289 100273 74.5889 43.3125 0.00862006 0.0134006 2024-02-05T07:22:20.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:21.022 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:21.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:21.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:21.151 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:21.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:21.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:21.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:22:21.399 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:21.427 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:21.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:21.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:21.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:21.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:21.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:21 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:21 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/62763041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:22:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:21 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:22:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:21 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:22:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:21 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:22:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:21 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:21 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/62763041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:22:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:21 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:22:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:21 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:22:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:21 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:22:21.698 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:21.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:21 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:21 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/62763041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:22:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:21 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:22:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:21 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:22:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:21 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:22:21.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:21.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:21.926 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 85 16 100931 100915 74.1834 40.125 0.0105192 0.0134748 2024-02-05T07:22:21.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:21.968 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:21.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:22.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:22.099 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:22.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:22.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:22.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:22.230 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:22.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:22.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:22.332 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:22.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:22.380 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:22.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:22.481 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:22.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:22.527 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:22.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:22.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:22.657 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:22.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:22.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:22 smithi007 ceph-mon[29098]: pgmap v134: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 880 B/s rd, 43 MiB/s wr, 680 op/s; 14138/299505 objects degraded (4.720%); 17870/299505 objects misplaced (5.967%); 17 MiB/s, 279 objects/s recovering 2024-02-05T07:22:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:22 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:22:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:22 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/62763041' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:22:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:22 smithi007 ceph-mon[29098]: osdmap e131: 8 total, 8 up, 8 in 2024-02-05T07:22:22.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:22 smithi007 ceph-mon[37158]: pgmap v134: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 880 B/s rd, 43 MiB/s wr, 680 op/s; 14138/299505 objects degraded (4.720%); 17870/299505 objects misplaced (5.967%); 17 MiB/s, 279 objects/s recovering 2024-02-05T07:22:22.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:22 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:22:22.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:22 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/62763041' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:22:22.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:22 smithi007 ceph-mon[37158]: osdmap e131: 8 total, 8 up, 8 in 2024-02-05T07:22:22.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:22.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:22 smithi090 ceph-mon[33213]: pgmap v134: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 880 B/s rd, 43 MiB/s wr, 680 op/s; 14138/299505 objects degraded (4.720%); 17870/299505 objects misplaced (5.967%); 17 MiB/s, 279 objects/s recovering 2024-02-05T07:22:22.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:22 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:22:22.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:22 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/62763041' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:22:22.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:22 smithi090 ceph-mon[33213]: osdmap e131: 8 total, 8 up, 8 in 2024-02-05T07:22:22.789 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:22.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:22.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:22.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:22.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:22.925 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 86 16 101725 101709 73.8979 49.625 0.013148 0.0135269 2024-02-05T07:22:23.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:23.068 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:23.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:23.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:23.170 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:23.217 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:23.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:23.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:23.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:23.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:23.343 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:23.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:23.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:23.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:23.495 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:23.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:23.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:23.630 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:23.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:23.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:23.732 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:23.781 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:23.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:23.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:23.912 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:23.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:23.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:22:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:22:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:22:23.926 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 87 16 102480 102464 73.5909 47.1875 0.00747277 0.0135824 2024-02-05T07:22:24.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:24.063 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:24.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:24.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:24.212 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:24.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:24.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:24.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:24.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:24.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:24.468 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:24.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:24.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:24.617 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:24.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:24.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:24 smithi007 ceph-mon[29098]: pgmap v136: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 35 MiB/s wr, 565 op/s; 14138/299505 objects degraded (4.720%); 17870/299505 objects misplaced (5.967%); 13 MiB/s, 204 objects/s recovering 2024-02-05T07:22:24.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:24 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 14138/299505 objects degraded (4.720%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:24.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:24 smithi007 ceph-mon[37158]: pgmap v136: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 35 MiB/s wr, 565 op/s; 14138/299505 objects degraded (4.720%); 17870/299505 objects misplaced (5.967%); 13 MiB/s, 204 objects/s recovering 2024-02-05T07:22:24.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:24 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 14138/299505 objects degraded (4.720%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:24.719 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:24.761 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:24.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:24.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:24.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:24 smithi090 ceph-mon[33213]: pgmap v136: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 35 MiB/s wr, 565 op/s; 14138/299505 objects degraded (4.720%); 17870/299505 objects misplaced (5.967%); 13 MiB/s, 204 objects/s recovering 2024-02-05T07:22:24.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:24 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 14138/299505 objects degraded (4.720%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:24.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:24.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:24.897 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:24.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:24.926 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 88 16 103182 103166 73.2532 43.875 0.00788176 0.0136378 2024-02-05T07:22:24.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:25.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:25.051 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:25.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:25.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:25.180 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:25.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:25.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:25.311 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:25.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:25.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:25.463 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:25.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:25.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T07:22:25.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:22:25.617 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:25.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:25.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:25.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:25.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:25.747 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:25.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:25.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:25.906 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:25.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:25.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:25.926 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 89 16 103905 103889 72.9378 45.1875 0.0169105 0.013704 2024-02-05T07:22:26.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:26.035 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:26.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:26.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:26.168 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:26.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:26.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:26.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:26.303 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:26.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:26.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:26.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:26.450 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:26.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:26.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:26.576 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:26 smithi090 ceph-mon[33213]: pgmap v137: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 42 MiB/s wr, 676 op/s; 14138/302901 objects degraded (4.668%); 17870/302901 objects misplaced (5.900%); 11 MiB/s, 183 objects/s recovering 2024-02-05T07:22:26.592 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:26.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:26.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:26.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:26 smithi007 ceph-mon[29098]: pgmap v137: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 42 MiB/s wr, 676 op/s; 14138/302901 objects degraded (4.668%); 17870/302901 objects misplaced (5.900%); 11 MiB/s, 183 objects/s recovering 2024-02-05T07:22:26.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:26 smithi007 ceph-mon[37158]: pgmap v137: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 42 MiB/s wr, 676 op/s; 14138/302901 objects degraded (4.668%); 17870/302901 objects misplaced (5.900%); 11 MiB/s, 183 objects/s recovering 2024-02-05T07:22:26.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:26.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:26.721 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:26.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:26.822 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:26.889 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:26.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:26.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:26.926 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total time run: 90.0092 2024-02-05T07:22:26.926 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total writes made: 104592 2024-02-05T07:22:26.926 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Write size: 65536 2024-02-05T07:22:26.926 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object size: 65536 2024-02-05T07:22:26.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Bandwidth (MB/sec): 72.6259 2024-02-05T07:22:26.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Bandwidth: 40.0063 2024-02-05T07:22:26.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max bandwidth (MB/sec): 199.875 2024-02-05T07:22:26.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min bandwidth (MB/sec): 0 2024-02-05T07:22:26.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average IOPS: 1162 2024-02-05T07:22:26.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev IOPS: 640.11 2024-02-05T07:22:26.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max IOPS: 3198 2024-02-05T07:22:26.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min IOPS: 0 2024-02-05T07:22:26.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average Latency(s): 0.0137649 2024-02-05T07:22:26.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Latency(s): 0.0318278 2024-02-05T07:22:26.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max latency(s): 2.10748 2024-02-05T07:22:26.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min latency(s): 0.00125178 2024-02-05T07:22:26.976 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T07:22:26.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:27.021 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:27.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:27.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:27.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:27.147 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:27.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:27.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:27.278 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:27.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:27.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:27.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:27.419 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:27.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:27.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:27.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:27.548 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:27.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:27.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:27.690 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:27.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:27.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:27.830 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:27.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:27.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:27.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:27.962 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:27.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:28.064 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:28.103 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:28.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:28.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:28.231 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:28.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:28.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:28.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:28.380 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:28.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:28.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:28.511 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:28.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:28.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:28.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:28.641 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:28.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:28.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:28 smithi007 ceph-mon[29098]: pgmap v138: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 875 op/s; 14138/310323 objects degraded (4.556%); 16356/310323 objects misplaced (5.271%); 17 MiB/s, 269 objects/s recovering 2024-02-05T07:22:28.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:28 smithi007 ceph-mon[37158]: pgmap v138: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 875 op/s; 14138/310323 objects degraded (4.556%); 16356/310323 objects misplaced (5.271%); 17 MiB/s, 269 objects/s recovering 2024-02-05T07:22:28.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:28.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:28 smithi090 ceph-mon[33213]: pgmap v138: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 875 op/s; 14138/310323 objects degraded (4.556%); 16356/310323 objects misplaced (5.271%); 17 MiB/s, 269 objects/s recovering 2024-02-05T07:22:28.784 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:28.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:28.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:28.926 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:28.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:29.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:29.068 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:29.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:29.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:29.197 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:29.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:29.299 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:29.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:29.341 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:29.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:29.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:29.487 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:29.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:29.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:29.634 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:29.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:29.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:29.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:29 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:22:29.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:29 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:22:29.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:29 smithi007 ceph-mon[29098]: osdmap e132: 8 total, 8 up, 8 in 2024-02-05T07:22:29.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:29 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 14138/310323 objects degraded (4.556%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:29.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:29 smithi007 ceph-mon[29098]: osdmap e133: 8 total, 8 up, 8 in 2024-02-05T07:22:29.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:22:29.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:29 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:22:29.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:29 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:22:29.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:29 smithi007 ceph-mon[37158]: osdmap e132: 8 total, 8 up, 8 in 2024-02-05T07:22:29.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:29 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 14138/310323 objects degraded (4.556%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:29.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:29 smithi007 ceph-mon[37158]: osdmap e133: 8 total, 8 up, 8 in 2024-02-05T07:22:29.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:22:29.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:29.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:29 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:22:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:29 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:22:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:29 smithi090 ceph-mon[33213]: osdmap e132: 8 total, 8 up, 8 in 2024-02-05T07:22:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:29 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 14138/310323 objects degraded (4.556%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:29 smithi090 ceph-mon[33213]: osdmap e133: 8 total, 8 up, 8 in 2024-02-05T07:22:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:22:29.781 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:29.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:29.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:29.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:29.910 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:29.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:29.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:30.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:30.040 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:30.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:30.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:30.185 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:30.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:30.286 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:30.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:30.348 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:22:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:22:30.415 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:30.439 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:30.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:30.543 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:30.571 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:30.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:30.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:30.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:30.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:30 smithi007 ceph-mon[29098]: pgmap v140: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 28 MiB/s wr, 451 op/s; 14138/310323 objects degraded (4.556%); 16356/310323 objects misplaced (5.271%); 7.2 MiB/s, 115 objects/s recovering 2024-02-05T07:22:30.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:30 smithi007 ceph-mon[37158]: pgmap v140: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 28 MiB/s wr, 451 op/s; 14138/310323 objects degraded (4.556%); 16356/310323 objects misplaced (5.271%); 7.2 MiB/s, 115 objects/s recovering 2024-02-05T07:22:30.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:30.702 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:30.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:30 smithi090 ceph-mon[33213]: pgmap v140: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 28 MiB/s wr, 451 op/s; 14138/310323 objects degraded (4.556%); 16356/310323 objects misplaced (5.271%); 7.2 MiB/s, 115 objects/s recovering 2024-02-05T07:22:30.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:30.851 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:30.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:30.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:30.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:30.981 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:30.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:31.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:31.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:31.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:31.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:31.255 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:31.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:31.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:31.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:31.399 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:31.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:31.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:31.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:31.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:31.626 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:31.670 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:31.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:31.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:31.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:31 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:31.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:31 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:31.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:31.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:31 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:31.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:31.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:31.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:31.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:31.960 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:31.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:31.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:32.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:32.088 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:32.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:32.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:32.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:32.240 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:32.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:32.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:32.371 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:32.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:32.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:32.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:32 smithi007 ceph-mon[29098]: pgmap v142: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 38 MiB/s wr, 887 op/s; 13773/307806 objects degraded (4.475%); 14417/307806 objects misplaced (4.684%); 17 MiB/s, 274 objects/s recovering 2024-02-05T07:22:32.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:32 smithi007 ceph-mon[37158]: pgmap v142: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 38 MiB/s wr, 887 op/s; 13773/307806 objects degraded (4.475%); 14417/307806 objects misplaced (4.684%); 17 MiB/s, 274 objects/s recovering 2024-02-05T07:22:32.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:32.496 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:32.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:32.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:32.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:32.628 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:32.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:32.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:32.757 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:32.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:32.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:32 smithi090 ceph-mon[33213]: pgmap v142: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 38 MiB/s wr, 887 op/s; 13773/307806 objects degraded (4.475%); 14417/307806 objects misplaced (4.684%); 17 MiB/s, 274 objects/s recovering 2024-02-05T07:22:32.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:32.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:32.889 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:32.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:32.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:33.042 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:33.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:33.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:33.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:33.296 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:33.343 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:33.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:33.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:33.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:33.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:33.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:33.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:33.604 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:33.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:33.706 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:33.734 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:22:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:22:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:22:33.736 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:33.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:33.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:33.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:33.882 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:33.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:33.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:33.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:34.011 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:34.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:34.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:34.113 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:34.153 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:34.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:34.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:34.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:22:34.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:34.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:34.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:34.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:34 smithi007 ceph-mon[37158]: pgmap v143: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 767 B/s rd, 30 MiB/s wr, 745 op/s; 13773/307806 objects degraded (4.475%); 14417/307806 objects misplaced (4.684%); 17 MiB/s, 274 objects/s recovering 2024-02-05T07:22:34.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:34 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 13773/307806 objects degraded (4.475%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:34.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:34.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:34.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:34.594 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:34.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:34.646 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:34.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:34.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:34 smithi007 ceph-mon[29098]: pgmap v143: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 767 B/s rd, 30 MiB/s wr, 745 op/s; 13773/307806 objects degraded (4.475%); 14417/307806 objects misplaced (4.684%); 17 MiB/s, 274 objects/s recovering 2024-02-05T07:22:34.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:34 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 13773/307806 objects degraded (4.475%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:34.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:34.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:34 smithi090 ceph-mon[33213]: pgmap v143: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 767 B/s rd, 30 MiB/s wr, 745 op/s; 13773/307806 objects degraded (4.475%); 14417/307806 objects misplaced (4.684%); 17 MiB/s, 274 objects/s recovering 2024-02-05T07:22:34.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:34 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 13773/307806 objects degraded (4.475%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:34.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:34.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:34.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:34.931 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:34.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:34.941 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:22:34.985 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:22:35.033 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:35.036 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:22:35.060 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:35.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:35.086 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:22:35.118 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:22:35.128 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:22:35.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:35.178 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:22:35.219 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:22:35.259 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:22:35.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:35.260 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:35.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:35.268 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:22:35.268 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:22:35.268 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:22:35.268 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:22:35.268 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:22:35.268 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:22:35.268 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:22:35.268 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:22:35.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:35.400 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:35.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:35.478 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:35 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3779516813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:22:35.478 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:35 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3779516813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:22:35.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:35.530 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:35.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:35.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:35.661 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:35.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:35.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:35.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:35 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3779516813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:22:35.789 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:35.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:35.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:35.917 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:35.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:35.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:36.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:36.064 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:36.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:36.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:36.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:36.192 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:36.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:36.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:36.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:36.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:36.436 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:36.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:36.460 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:36.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:36.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:36.589 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:36.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:36.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:36 smithi007 ceph-mon[29098]: pgmap v144: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 10 MiB/s wr, 579 op/s; 13773/304374 objects degraded (4.525%); 14417/304374 objects misplaced (4.737%); 9.9 MiB/s, 158 objects/s recovering 2024-02-05T07:22:36.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:36 smithi007 ceph-mon[37158]: pgmap v144: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 10 MiB/s wr, 579 op/s; 13773/304374 objects degraded (4.525%); 14417/304374 objects misplaced (4.737%); 9.9 MiB/s, 158 objects/s recovering 2024-02-05T07:22:36.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:36.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:36.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:36.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:36.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:36 smithi090 ceph-mon[33213]: pgmap v144: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 10 MiB/s wr, 579 op/s; 13773/304374 objects degraded (4.525%); 14417/304374 objects misplaced (4.737%); 9.9 MiB/s, 158 objects/s recovering 2024-02-05T07:22:36.817 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:36.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:36.858 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:36.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:36.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:36.996 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:36.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:36.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:37.098 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:37.122 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:37.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:37.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:37.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:37.263 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:37.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:37.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:37.392 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:37.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:37.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:37.494 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:37.523 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:37.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:37.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:37.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:37.658 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:37.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:37.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:37.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:37 smithi007 ceph-mon[29098]: osdmap e134: 8 total, 8 up, 8 in 2024-02-05T07:22:37.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:37 smithi007 ceph-mon[37158]: osdmap e134: 8 total, 8 up, 8 in 2024-02-05T07:22:37.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:37.778 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:37 smithi090 ceph-mon[33213]: osdmap e134: 8 total, 8 up, 8 in 2024-02-05T07:22:37.814 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:37.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:37.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:37.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:37.946 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:37.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:38.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:38.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:38.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:38.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:38.228 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:38.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:38.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:38.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:38.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:38.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:38.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:38.486 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:38.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:38.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:38.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:38.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:38.647 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:38 smithi007 ceph-mon[29098]: pgmap v146: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 10 MiB/s wr, 889 op/s; 13094/296928 objects degraded (4.410%); 11973/296928 objects misplaced (4.032%); 22 MiB/s, 359 objects/s recovering 2024-02-05T07:22:38.647 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:38 smithi007 ceph-mon[29098]: osdmap e135: 8 total, 8 up, 8 in 2024-02-05T07:22:38.648 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:38 smithi007 ceph-mon[37158]: pgmap v146: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 10 MiB/s wr, 889 op/s; 13094/296928 objects degraded (4.410%); 11973/296928 objects misplaced (4.032%); 22 MiB/s, 359 objects/s recovering 2024-02-05T07:22:38.648 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:38 smithi007 ceph-mon[37158]: osdmap e135: 8 total, 8 up, 8 in 2024-02-05T07:22:38.717 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:38.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:38.748 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:38.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:38.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:38 smithi090 ceph-mon[33213]: pgmap v146: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 10 MiB/s wr, 889 op/s; 13094/296928 objects degraded (4.410%); 11973/296928 objects misplaced (4.032%); 22 MiB/s, 359 objects/s recovering 2024-02-05T07:22:38.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:38 smithi090 ceph-mon[33213]: osdmap e135: 8 total, 8 up, 8 in 2024-02-05T07:22:38.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:38.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:38.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:38.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:39.007 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:39.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:39.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:39.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:39.138 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:39.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:39.240 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:39.270 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:39.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:39.372 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:39.418 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:39.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:39.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:39.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:39.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:39.565 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:39.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:39.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:39.697 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:39.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:39.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:39 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:22:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:39 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:22:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:39 smithi090 ceph-mon[33213]: osdmap e136: 8 total, 8 up, 8 in 2024-02-05T07:22:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:39 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 13094/296928 objects degraded (4.410%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:39 smithi090 ceph-mon[33213]: osdmap e137: 8 total, 8 up, 8 in 2024-02-05T07:22:39.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:39.842 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:39.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:39.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:39 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:22:39.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:39 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:22:39.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:39 smithi007 ceph-mon[29098]: osdmap e136: 8 total, 8 up, 8 in 2024-02-05T07:22:39.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:39 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 13094/296928 objects degraded (4.410%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:39.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:39 smithi007 ceph-mon[29098]: osdmap e137: 8 total, 8 up, 8 in 2024-02-05T07:22:39.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:39 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:22:39.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:39 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:22:39.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:39 smithi007 ceph-mon[37158]: osdmap e136: 8 total, 8 up, 8 in 2024-02-05T07:22:39.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:39 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 13094/296928 objects degraded (4.410%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:39.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:39 smithi007 ceph-mon[37158]: osdmap e137: 8 total, 8 up, 8 in 2024-02-05T07:22:39.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:39.968 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:39.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:40.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:40.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:40.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:40.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:40.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:40.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:40.239 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:40.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:40.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:40.369 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:40.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:40.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:40.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:22:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:22:40.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:40.515 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:40.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:40.617 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:40.644 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:40.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:40.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:40.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:40.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:40 smithi090 ceph-mon[33213]: pgmap v149: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 604 op/s; 13094/296928 objects degraded (4.410%); 11973/296928 objects misplaced (4.032%); 17 MiB/s, 267 objects/s recovering 2024-02-05T07:22:40.776 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:40.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:40.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:40.923 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:40 smithi007 ceph-mon[29098]: pgmap v149: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 604 op/s; 13094/296928 objects degraded (4.410%); 11973/296928 objects misplaced (4.032%); 17 MiB/s, 267 objects/s recovering 2024-02-05T07:22:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:40.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:40 smithi007 ceph-mon[37158]: pgmap v149: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 604 op/s; 13094/296928 objects degraded (4.410%); 11973/296928 objects misplaced (4.032%); 17 MiB/s, 267 objects/s recovering 2024-02-05T07:22:41.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:41.055 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:41.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:41.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:41.157 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:41.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:41.203 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:41.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:41.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:41.352 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:41.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:41.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:41.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:41.484 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:41.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:41.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:41.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:41.717 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:41.750 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:41.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:41.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:41.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:41 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:41.789 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:41 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:41.790 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:41 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:41.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:41.882 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:41.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:41.983 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:42.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:42.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:42.168 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:42.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:42.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:42.270 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:42.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:42.321 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:42.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:42.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:42.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:42.449 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:42.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:42.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:42.579 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:42.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:42.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:42.632 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:42 smithi007 ceph-mon[29098]: pgmap v151: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 686 op/s; 12466/287520 objects degraded (4.336%); 9928/287520 objects misplaced (3.453%); 20 MiB/s, 315 objects/s recovering 2024-02-05T07:22:42.635 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:42 smithi007 ceph-mon[37158]: pgmap v151: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 686 op/s; 12466/287520 objects degraded (4.336%); 9928/287520 objects misplaced (3.453%); 20 MiB/s, 315 objects/s recovering 2024-02-05T07:22:42.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:42.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:42.730 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:42.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:42.758 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:22:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:22:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-02-05T07:22:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-02-05T07:22:42.758 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:22:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:42 smithi090 ceph-mon[33213]: pgmap v151: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 686 op/s; 12466/287520 objects degraded (4.336%); 9928/287520 objects misplaced (3.453%); 20 MiB/s, 315 objects/s recovering 2024-02-05T07:22:42.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:42.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:42.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:42.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:42.991 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:42.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:42.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:22:43.069 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:22:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 137, 2024-02-05T07:22:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "e3e7059e-c3f5-11ee-95b6-87774f69a715", 2024-02-05T07:22:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "created": "2024-02-05T07:13:14.603885+0000", 2024-02-05T07:22:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "modified": "2024-02-05T07:22:39.145419+0000", 2024-02-05T07:22:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "last_up_change": "2024-02-05T07:18:35.434647+0000", 2024-02-05T07:22:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "last_in_change": "2024-02-05T07:18:17.533340+0000", 2024-02-05T07:22:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T07:22:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_num": 5806080, 2024-02-05T07:22:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_set": [ 2024-02-05T07:22:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "nodeep-scrub", 2024-02-05T07:22:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "noscrub", 2024-02-05T07:22:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "pglog_hardlimit", 2024-02-05T07:22:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "purged_snapdirs", 2024-02-05T07:22:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_deletes", 2024-02-05T07:22:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "sortbitwise" 2024-02-05T07:22:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:22:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_version": 18, 2024-02-05T07:22:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T07:22:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T07:22:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T07:22:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_snapshot": "", 2024-02-05T07:22:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_max": 7, 2024-02-05T07:22:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd": 8, 2024-02-05T07:22:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "require_min_compat_client": "luminous", 2024-02-05T07:22:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "min_compat_client": "luminous", 2024-02-05T07:22:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "require_osd_release": "squid", 2024-02-05T07:22:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_crimson": false, 2024-02-05T07:22:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "pools": [ 2024-02-05T07:22:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 1, 2024-02-05T07:22:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".mgr", 2024-02-05T07:22:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:16:32.505567+0000", 2024-02-05T07:22:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:22:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:22:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:22:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:22:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:22:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:22:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:22:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:22:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:22:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:22:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:22:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:22:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 1, 2024-02-05T07:22:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 1, 2024-02-05T07:22:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 1, 2024-02-05T07:22:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 1, 2024-02-05T07:22:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 1, 2024-02-05T07:22:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:22:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:22:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:22:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:22:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:22:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:22:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:22:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "22", 2024-02-05T07:22:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:22:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:22:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:22:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:22:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:22:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:22:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:22:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:22:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:22:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:22:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:22:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:22:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:22:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:22:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:22:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:22:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:22:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:22:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:22:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:22:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:22:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:22:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:22:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:22:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:22:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:22:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:22:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:22:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:22:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:22:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:22:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:22:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:22:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:22:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:22:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:22:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:22:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-02-05T07:22:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_max": 32, 2024-02-05T07:22:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_min": 1 2024-02-05T07:22:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:22:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": {} 2024-02-05T07:22:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:22:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:22:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 7.8899998664855957, 2024-02-05T07:22:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 7.8899998664855957, 2024-02-05T07:22:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.37999999523162842, 2024-02-05T07:22:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 3, 2024-02-05T07:22:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 3, 2024-02-05T07:22:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 1, 2024-02-05T07:22:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 1, 2024-02-05T07:22:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 1 2024-02-05T07:22:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 2, 2024-02-05T07:22:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "datapool", 2024-02-05T07:22:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:47.259462+0000", 2024-02-05T07:22:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 8193, 2024-02-05T07:22:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T07:22:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:22:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:22:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:22:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:22:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:22:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:22:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:22:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:22:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:22:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:22:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 3, 2024-02-05T07:22:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 3, 2024-02-05T07:22:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 3, 2024-02-05T07:22:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 3, 2024-02-05T07:22:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 3, 2024-02-05T07:22:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:22:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:22:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:22:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:22:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:22:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:22:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:22:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "71", 2024-02-05T07:22:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:22:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:22:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:22:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:22:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:22:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 2, 2024-02-05T07:22:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 71, 2024-02-05T07:22:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:22:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:22:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:22:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:22:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:22:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:22:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:22:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:22:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:22:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:22:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:22:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:22:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:22:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:22:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:22:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:22:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:22:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:22:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:22:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:22:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:22:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:22:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:22:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:22:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:22:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:22:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:22:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:22:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:22:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:22:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:22:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:22:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd": {} 2024-02-05T07:22:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:22:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:22:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.6500000953674316, 2024-02-05T07:22:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.6500000953674316, 2024-02-05T07:22:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T07:22:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.3299999237060547, 2024-02-05T07:22:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.3299999237060547, 2024-02-05T07:22:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 1, 2024-02-05T07:22:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 1, 2024-02-05T07:22:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 1 2024-02-05T07:22:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 3, 2024-02-05T07:22:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".rgw.root", 2024-02-05T07:22:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:47.762336+0000", 2024-02-05T07:22:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:22:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:22:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:22:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:22:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:22:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:22:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:22:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:22:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:22:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:22:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:22:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:22:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:22:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:22:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:22:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:22:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:22:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:22:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:22:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:22:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:22:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:22:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:22:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:22:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "69", 2024-02-05T07:22:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:22:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:22:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:22:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:22:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:22:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:22:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:22:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:22:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:22:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:22:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:22:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:22:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:22:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:22:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:22:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:22:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:22:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:22:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:22:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:22:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:22:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:22:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:22:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:22:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:22:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:22:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:22:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:22:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:22:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:22:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:22:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:22:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:22:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:22:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:22:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:22:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:22:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:22:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:22:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:22:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:22:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:22:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.5, 2024-02-05T07:22:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.5, 2024-02-05T07:22:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-02-05T07:22:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.5, 2024-02-05T07:22:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.5, 2024-02-05T07:22:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 1, 2024-02-05T07:22:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 1, 2024-02-05T07:22:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 1 2024-02-05T07:22:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 4, 2024-02-05T07:22:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.log", 2024-02-05T07:22:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:49.736599+0000", 2024-02-05T07:22:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:22:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:22:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:22:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:22:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:22:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:22:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:22:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:22:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:22:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:22:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:22:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:22:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:22:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:22:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:22:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:22:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:22:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:22:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:22:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:22:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:22:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:22:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:22:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:22:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "71", 2024-02-05T07:22:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:22:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:22:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:22:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:22:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:22:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:22:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:22:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:22:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:22:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:22:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:22:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:22:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:22:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:22:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:22:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.25, 2024-02-05T07:22:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.25, 2024-02-05T07:22:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-02-05T07:22:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.25, 2024-02-05T07:22:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.25, 2024-02-05T07:22:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 1, 2024-02-05T07:22:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 1, 2024-02-05T07:22:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 1 2024-02-05T07:22:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 5, 2024-02-05T07:22:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.control", 2024-02-05T07:22:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:51.696948+0000", 2024-02-05T07:22:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:22:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:22:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:22:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:22:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:22:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "73", 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:22:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:22:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:22:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:22:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:22:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:22:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:22:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:22:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.25, 2024-02-05T07:22:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.25, 2024-02-05T07:22:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-02-05T07:22:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.25, 2024-02-05T07:22:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.25, 2024-02-05T07:22:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 1, 2024-02-05T07:22:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 1, 2024-02-05T07:22:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 1 2024-02-05T07:22:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 6, 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.meta", 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:53.768555+0000", 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:22:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:22:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:22:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:22:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:22:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:22:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:22:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:22:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:22:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:22:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:22:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:22:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:22:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:22:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:22:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "75", 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:22:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:22:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:22:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:22:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:22:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:22:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:22:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:22:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:22:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:22:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:22:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:22:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:22:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:22:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:22:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_bias": 4 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:22:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:22:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:22:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:22:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.75, 2024-02-05T07:22:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.75, 2024-02-05T07:22:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-02-05T07:22:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.75, 2024-02-05T07:22:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.75, 2024-02-05T07:22:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 1, 2024-02-05T07:22:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 1, 2024-02-05T07:22:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 1 2024-02-05T07:22:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 7, 2024-02-05T07:22:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "unique_pool_0", 2024-02-05T07:22:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:20:44.070155+0000", 2024-02-05T07:22:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:22:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:22:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:22:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:22:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:22:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:22:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:22:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:22:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:22:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 26, 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 26, 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 26, 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 26, 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 26, 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:22:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "120", 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "118", 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:22:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:22:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:22:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:22:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:22:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:22:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:22:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:22:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:22:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:22:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:22:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:22:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:22:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:22:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "rados": {} 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.4600000381469727, 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.8500000238418579, 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.4600000381469727, 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.8500000238418579, 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 1, 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 1, 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 1 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-02-05T07:22:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "703c3948-8003-4015-9dd8-21019cf0eeb8", 2024-02-05T07:22:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:22:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:22:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:22:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:22:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:22:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:22:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 8, 2024-02-05T07:22:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 121, 2024-02-05T07:22:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:22:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:22:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6802", 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:22:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:22:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6804", 2024-02-05T07:22:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:22:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:22:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6803", 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6801/2069016689", 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6802/2069016689", 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6804/2069016689", 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6803/2069016689", 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-02-05T07:22:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "5c3e7d1e-a61d-44b9-b21e-f91c99a096bc", 2024-02-05T07:22:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:22:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:22:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:22:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:22:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:22:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:22:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 13, 2024-02-05T07:22:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 133, 2024-02-05T07:22:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:22:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:22:43.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:22:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6805", 2024-02-05T07:22:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:22:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:22:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6806", 2024-02-05T07:22:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:22:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:22:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6808", 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6807", 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6805/865486532", 2024-02-05T07:22:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6806/865486532", 2024-02-05T07:22:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6808/865486532", 2024-02-05T07:22:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6807/865486532", 2024-02-05T07:22:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:22:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:22:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:22:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-02-05T07:22:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "f4fd930f-7bb2-46ca-9964-98235bd47b3a", 2024-02-05T07:22:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:22:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 18, 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 121, 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6809", 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:22:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6810", 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6812", 2024-02-05T07:22:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:22:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:22:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6811", 2024-02-05T07:22:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:22:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6809/904871370", 2024-02-05T07:22:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6810/904871370", 2024-02-05T07:22:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6812/904871370", 2024-02-05T07:22:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6811/904871370", 2024-02-05T07:22:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:22:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:22:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:22:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-02-05T07:22:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "149ed0d6-b049-4478-9819-cafc446e94d3", 2024-02-05T07:22:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:22:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 26, 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 128, 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6813", 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:22:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6814", 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6816", 2024-02-05T07:22:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:22:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:22:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6815", 2024-02-05T07:22:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:22:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6813/2758586242", 2024-02-05T07:22:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6814/2758586242", 2024-02-05T07:22:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6816/2758586242", 2024-02-05T07:22:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6815/2758586242", 2024-02-05T07:22:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:22:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:22:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:22:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-02-05T07:22:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d", 2024-02-05T07:22:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:22:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:22:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:22:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:22:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:22:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:22:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 32, 2024-02-05T07:22:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 120, 2024-02-05T07:22:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:22:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:22:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:22:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6800", 2024-02-05T07:22:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:22:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:22:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6801", 2024-02-05T07:22:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:22:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6803", 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6802", 2024-02-05T07:22:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:22:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6800/1154929470", 2024-02-05T07:22:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6801/1154929470", 2024-02-05T07:22:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6803/1154929470", 2024-02-05T07:22:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6802/1154929470", 2024-02-05T07:22:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:22:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:22:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:22:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "244a7930-c22c-413e-86e1-11e0749f7367", 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 41, 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 118, 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6804", 2024-02-05T07:22:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:22:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:22:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6805", 2024-02-05T07:22:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:22:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6807", 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6806", 2024-02-05T07:22:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:22:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6804/398113577", 2024-02-05T07:22:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6805/398113577", 2024-02-05T07:22:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6807/398113577", 2024-02-05T07:22:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6806/398113577", 2024-02-05T07:22:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:22:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:22:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:22:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7", 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 52, 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 134, 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6808", 2024-02-05T07:22:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2371506925 2024-02-05T07:22:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:22:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6809", 2024-02-05T07:22:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2371506925 2024-02-05T07:22:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6811", 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2371506925 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6810", 2024-02-05T07:22:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2371506925 2024-02-05T07:22:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6808/2371506925", 2024-02-05T07:22:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6809/2371506925", 2024-02-05T07:22:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6811/2371506925", 2024-02-05T07:22:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6810/2371506925", 2024-02-05T07:22:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:22:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:22:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:22:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-02-05T07:22:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9", 2024-02-05T07:22:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:22:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:22:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:22:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:22:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:22:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:22:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 60, 2024-02-05T07:22:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 129, 2024-02-05T07:22:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:22:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:22:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:22:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6812", 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6813", 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6815", 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:22:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6814", 2024-02-05T07:22:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:22:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6812/2983605980", 2024-02-05T07:22:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6813/2983605980", 2024-02-05T07:22:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2983605980", 2024-02-05T07:22:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6814/2983605980", 2024-02-05T07:22:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:22:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:22:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:22:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_xinfo": [ 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:21:34.870103+0000", 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:22:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:22:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:22:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:22:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:22:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:21:38.190770+0000", 2024-02-05T07:22:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:22:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-02-05T07:22:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:22:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:22:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:22:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:22:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:22:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:22:37.492367+0000", 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:21:49.329818+0000", 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-02-05T07:22:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:22:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:22:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:22:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:22:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:22:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:22:15.837361+0000", 2024-02-05T07:22:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:22:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-02-05T07:22:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:22:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:22:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:22:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:22:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:21:50.696968+0000", 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:21:37.206395+0000", 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-02-05T07:22:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:22:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:22:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:22:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:22:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:22:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:22:28.219176+0000", 2024-02-05T07:22:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:22:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:22:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap": [ 2024-02-05T07:22:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.18", 2024-02-05T07:22:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:22:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: 3, 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: 1, 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: 7 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_items": [ 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.9", 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.12", 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 5 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.13", 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:22:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_primaries": [], 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_temp": [ 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "7.10", 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: 3, 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: 1 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "7.13", 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: 2, 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: 0 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "7.14", 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: 0, 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: 2 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "7.16", 2024-02-05T07:22:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:22:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: 0, 2024-02-05T07:22:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: 7 2024-02-05T07:22:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "7.17", 2024-02-05T07:22:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:22:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: 0, 2024-02-05T07:22:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: 2 2024-02-05T07:22:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "7.18", 2024-02-05T07:22:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: 2, 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: 7 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "7.19", 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: 0, 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: 2 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_temp": [], 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklist": { 2024-02-05T07:22:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2180087772": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2066685086": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/4100020626": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1953351182": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3203189162": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3418781614": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1518592084": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/3380493069": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2208590106": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/223579766": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/2634895574": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1462812719": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2823976078": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1905520582": "2024-02-06T07:13:47.004765+0000" 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "range_blocklist": {}, 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profiles": { 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "default": { 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-failure-domain": "osd", 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "k": "2", 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "m": "1", 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin": "jerasure", 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "technique": "reed_sol_van" 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps_queue": [], 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "new_removed_snaps": [], 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "new_purged_snaps": [], 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_node_flags": {}, 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "device_class_flags": {}, 2024-02-05T07:22:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode": { 2024-02-05T07:22:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_enabled": false, 2024-02-05T07:22:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_bucket_count": 0, 2024-02-05T07:22:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "degraded_stretch_mode": 0, 2024-02-05T07:22:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "recovering_stretch_mode": 0, 2024-02-05T07:22:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_bucket": 0 2024-02-05T07:22:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:22:43.167 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:22:43.241 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:43.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:43.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:43.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:43.373 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:43.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:43.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:43.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:43.503 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:43.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:43.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:43.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:43.646 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:43.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:43.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:43.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:43 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2177152339' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:22:43.674 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:22:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:22:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:22:43.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:43 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2177152339' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:22:43.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:43.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:43 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2177152339' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:22:43.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:43.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:43.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:43.923 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:43.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:43.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:44.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:44.052 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:44.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:44.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:44.203 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:44.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:44.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:44.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:44.345 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:44.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:44.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:44.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:44.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:44.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:44.609 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:44.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:44.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:44.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:44 smithi007 ceph-mon[29098]: pgmap v152: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 523 op/s; 12466/287520 objects degraded (4.336%); 9928/287520 objects misplaced (3.453%); 15 MiB/s, 240 objects/s recovering 2024-02-05T07:22:44.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:44 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 12466/287520 objects degraded (4.336%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:44.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:44 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:22:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:44 smithi007 ceph-mon[37158]: pgmap v152: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 523 op/s; 12466/287520 objects degraded (4.336%); 9928/287520 objects misplaced (3.453%); 15 MiB/s, 240 objects/s recovering 2024-02-05T07:22:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:44 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 12466/287520 objects degraded (4.336%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:44 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:22:44.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:44.739 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:44.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:44.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:44.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:44 smithi090 ceph-mon[33213]: pgmap v152: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 523 op/s; 12466/287520 objects degraded (4.336%); 9928/287520 objects misplaced (3.453%); 15 MiB/s, 240 objects/s recovering 2024-02-05T07:22:44.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:44 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 12466/287520 objects degraded (4.336%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:22:44.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:44 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:22:44.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:44.868 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:44.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:44.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:44.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:45.001 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:45.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:45.103 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:45.149 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:45.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:45.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:45.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:22:45.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:45.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:45.332 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:45.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:45.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:45.461 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:45.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:45.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:45.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:45.592 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:45.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:45.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:45.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:45.712 INFO:tasks.thrashosds.thrasher:j is {'epoch': 137, 'fsid': 'e3e7059e-c3f5-11ee-95b6-87774f69a715', 'created': '2024-02-05T07:13:14.603885+0000', 'modified': '2024-02-05T07:22:39.145419+0000', 'last_up_change': '2024-02-05T07:18:35.434647+0000', 'last_in_change': '2024-02-05T07:18:17.533340+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T07:16:32.505567+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T07:18:47.259462+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T07:18:47.762336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T07:18:49.736599+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T07:18:51.696948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T07:18:53.768555+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-02-05T07:20:44.070155+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '118', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4600000381469727, 'score_stable': 1.850000023841858, 'optimal_score': 1, 'raw_score_acting': 2.4600000381469727, 'raw_score_stable': 1.850000023841858, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '703c3948-8003-4015-9dd8-21019cf0eeb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6801', 'nonce': 2069016689}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6802', 'nonce': 2069016689}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6804', 'nonce': 2069016689}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6803', 'nonce': 2069016689}]}, 'public_addr': '172.21.15.7:6801/2069016689', 'cluster_addr': '172.21.15.7:6802/2069016689', 'heartbeat_back_addr': '172.21.15.7:6804/2069016689', 'heartbeat_front_addr': '172.21.15.7:6803/2069016689', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '5c3e7d1e-a61d-44b9-b21e-f91c99a096bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 133, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6805', 'nonce': 865486532}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6806', 'nonce': 865486532}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6808', 'nonce': 865486532}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6807', 'nonce': 865486532}]}, 'public_addr': '172.21.15.7:6805/865486532', 'cluster_addr': '172.21.15.7:6806/865486532', 'heartbeat_back_addr': '172.21.15.7:6808/865486532', 'heartbeat_front_addr': '172.21.15.7:6807/865486532', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4fd930f-7bb2-46ca-9964-98235bd47b3a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6809', 'nonce': 904871370}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6810', 'nonce': 904871370}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6812', 'nonce': 904871370}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6811', 'nonce': 904871370}]}, 'public_addr': '172.21.15.7:6809/904871370', 'cluster_addr': '172.21.15.7:6810/904871370', 'heartbeat_back_addr': '172.21.15.7:6812/904871370', 'heartbeat_front_addr': '172.21.15.7:6811/904871370', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '149ed0d6-b049-4478-9819-cafc446e94d3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 128, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6813', 'nonce': 2758586242}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6814', 'nonce': 2758586242}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6816', 'nonce': 2758586242}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6815', 'nonce': 2758586242}]}, 'public_addr': '172.21.15.7:6813/2758586242', 'cluster_addr': '172.21.15.7:6814/2758586242', 'heartbeat_back_addr': '172.21.15.7:6816/2758586242', 'heartbeat_front_addr': '172.21.15.7:6815/2758586242', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 120, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 1154929470}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6801', 'nonce': 1154929470}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6803', 'nonce': 1154929470}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 1154929470}]}, 'public_addr': '172.21.15.90:6800/1154929470', 'cluster_addr': '172.21.15.90:6801/1154929470', 'heartbeat_back_addr': '172.21.15.90:6803/1154929470', 'heartbeat_front_addr': '172.21.15.90:6802/1154929470', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '244a7930-c22c-413e-86e1-11e0749f7367', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 118, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 398113577}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6805', 'nonce': 398113577}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6807', 'nonce': 398113577}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 398113577}]}, 'public_addr': '172.21.15.90:6804/398113577', 'cluster_addr': '172.21.15.90:6805/398113577', 'heartbeat_back_addr': '172.21.15.90:6807/398113577', 'heartbeat_front_addr': '172.21.15.90:6806/398113577', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0adbb562-3012-4f60-b5ec-02550a10b2c7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 134, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 2371506925}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 2371506925}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6811', 'nonce': 2371506925}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6810', 'nonce': 2371506925}]}, 'public_addr': '172.21.15.90:6808/2371506925', 'cluster_addr': '172.21.15.90:6809/2371506925', 'heartbeat_back_addr': '172.21.15.90:6811/2371506925', 'heartbeat_front_addr': '172.21.15.90:6810/2371506925', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dbc26c36-a0e9-4f21-9e98-739270cea6d9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 2983605980}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 2983605980}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6815', 'nonce': 2983605980}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6814', 'nonce': 2983605980}]}, 'public_addr': '172.21.15.90:6812/2983605980', 'cluster_addr': '172.21.15.90:6813/2983605980', 'heartbeat_back_addr': '172.21.15.90:6815/2983605980', 'heartbeat_front_addr': '172.21.15.90:6814/2983605980', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:21:34.870103+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:21:38.190770+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:22:37.492367+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:21:49.329818+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:22:15.837361+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:21:50.696968+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:21:37.206395+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:22:28.219176+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.18', 'osds': [3, 1, 7]}], 'pg_upmap_items': [{'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.10', 'osds': [3, 1]}, {'pgid': '7.13', 'osds': [2, 0]}, {'pgid': '7.14', 'osds': [0, 2]}, {'pgid': '7.16', 'osds': [0, 7]}, {'pgid': '7.17', 'osds': [0, 2]}, {'pgid': '7.18', 'osds': [2, 7]}, {'pgid': '7.19', 'osds': [0, 2]}], 'primary_temp': [], 'blocklist': {'172.21.15.7:0/2180087772': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/2066685086': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/4100020626': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/1953351182': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/3203189162': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/3418781614': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:6800/1518592084': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:6800/3380493069': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/2208590106': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:0/223579766': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:6800/2634895574': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/1462812719': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/2823976078': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/1905520582': '2024-02-06T07:13:47.004765+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T07:22:45.712 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.18 2024-02-05T07:22:45.712 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd rm-pg-upmap 3.18 2024-02-05T07:22:45.722 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:45.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:45.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:45.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:45.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:45.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:45.981 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:45.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:45.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:46.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:46.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:46.133 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:46.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:46.154 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:22:46.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T07:22:46.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:46.338 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:46.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:46.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:46.502 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:46.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:46.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:46.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:46.637 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:46.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:46.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:46.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:46.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:46.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:46 smithi090 ceph-mon[33213]: pgmap v153: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 604 op/s; 12466/283230 objects degraded (4.401%); 9928/283230 objects misplaced (3.505%); 12 MiB/s, 190 objects/s recovering 2024-02-05T07:22:46.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:46 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:22:46.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:46 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:22:46.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:46 smithi090 ceph-mon[33213]: osdmap e138: 8 total, 8 up, 8 in 2024-02-05T07:22:46.870 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:46.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:46.895 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:46.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:46.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:46 smithi007 ceph-mon[29098]: pgmap v153: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 604 op/s; 12466/283230 objects degraded (4.401%); 9928/283230 objects misplaced (3.505%); 12 MiB/s, 190 objects/s recovering 2024-02-05T07:22:46.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:46 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:22:46.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:46 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:22:46.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:46 smithi007 ceph-mon[29098]: osdmap e138: 8 total, 8 up, 8 in 2024-02-05T07:22:46.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:46 smithi007 ceph-mon[37158]: pgmap v153: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 604 op/s; 12466/283230 objects degraded (4.401%); 9928/283230 objects misplaced (3.505%); 12 MiB/s, 190 objects/s recovering 2024-02-05T07:22:46.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:46 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:22:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:46 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:22:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:46 smithi007 ceph-mon[37158]: osdmap e138: 8 total, 8 up, 8 in 2024-02-05T07:22:46.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:47.030 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:47.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:47.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:47.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:47.166 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:47.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:47.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:47.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:47.293 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:47.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:47.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:47.424 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:47.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:47.526 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:47.562 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:47.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:47.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:47.663 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:47.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:47 smithi007 ceph-mon[29098]: osdmap e139: 8 total, 8 up, 8 in 2024-02-05T07:22:47.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:47 smithi007 ceph-mon[37158]: osdmap e139: 8 total, 8 up, 8 in 2024-02-05T07:22:47.691 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:22:47.694 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:22:47.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:47.710 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:47.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:47.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:47 smithi090 ceph-mon[33213]: osdmap e139: 8 total, 8 up, 8 in 2024-02-05T07:22:47.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:47.848 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:47.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:47.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:47.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:47.979 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:47.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:48.081 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:48.109 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:48.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:48.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:48.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:48.238 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:48.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:48.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:48.370 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:48.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:48.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:48.497 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:48.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:48.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:48.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:48.633 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:48.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:48.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:48.735 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:48.778 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:48 smithi090 ceph-mon[33213]: pgmap v156: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 938 op/s; 11696/274920 objects degraded (4.254%); 7703/274920 objects misplaced (2.802%); 24 MiB/s, 389 objects/s recovering 2024-02-05T07:22:48.778 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:48 smithi090 ceph-mon[33213]: osdmap e140: 8 total, 8 up, 8 in 2024-02-05T07:22:48.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:48.779 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:48.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:48.810 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:48 smithi007 ceph-mon[29098]: pgmap v156: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 938 op/s; 11696/274920 objects degraded (4.254%); 7703/274920 objects misplaced (2.802%); 24 MiB/s, 389 objects/s recovering 2024-02-05T07:22:48.810 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:48 smithi007 ceph-mon[29098]: osdmap e140: 8 total, 8 up, 8 in 2024-02-05T07:22:48.810 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:48 smithi007 ceph-mon[37158]: pgmap v156: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 938 op/s; 11696/274920 objects degraded (4.254%); 7703/274920 objects misplaced (2.802%); 24 MiB/s, 389 objects/s recovering 2024-02-05T07:22:48.811 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:48 smithi007 ceph-mon[37158]: osdmap e140: 8 total, 8 up, 8 in 2024-02-05T07:22:48.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:48.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:48.910 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:48.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:49.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:49.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:49.072 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:49.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:49.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:49.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:49.202 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:49.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:49.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:49.345 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:49.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:49.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:49.476 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:49.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:49.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:49.578 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:49.606 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:49.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:49.708 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:49.750 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:49.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:49.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:49 smithi090 ceph-mon[33213]: osdmap e141: 8 total, 8 up, 8 in 2024-02-05T07:22:49.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:49 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 11696/274920 objects degraded (4.254%), 3 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-05T07:22:49.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:49 smithi007 ceph-mon[29098]: osdmap e141: 8 total, 8 up, 8 in 2024-02-05T07:22:49.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:49 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 11696/274920 objects degraded (4.254%), 3 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-05T07:22:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:49 smithi007 ceph-mon[37158]: osdmap e141: 8 total, 8 up, 8 in 2024-02-05T07:22:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:49 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 11696/274920 objects degraded (4.254%), 3 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-05T07:22:49.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:49.904 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:49.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:50.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:50.043 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:50.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:50.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:50.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:50.173 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:50.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:50.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:50.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:50.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:50.307 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:50.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:50.408 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:50.453 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:50.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:50.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:22:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:22:50.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:50.584 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:50.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:50.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:50.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:50.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:50.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:50.845 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:50.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:50.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:50.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:50 smithi007 ceph-mon[29098]: pgmap v159: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 11696/274920 objects degraded (4.254%); 7703/274920 objects misplaced (2.802%); 25 MiB/s, 400 objects/s recovering 2024-02-05T07:22:50.848 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:50 smithi007 ceph-mon[37158]: pgmap v159: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 11696/274920 objects degraded (4.254%); 7703/274920 objects misplaced (2.802%); 25 MiB/s, 400 objects/s recovering 2024-02-05T07:22:50.947 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:50.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:50.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:51.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:50 smithi090 ceph-mon[33213]: pgmap v159: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 11696/274920 objects degraded (4.254%); 7703/274920 objects misplaced (2.802%); 25 MiB/s, 400 objects/s recovering 2024-02-05T07:22:51.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:51.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:51.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:51.297 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:51.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:51.398 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:51.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:51.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:51.578 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:51.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:51.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:51.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:51.721 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:51.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:51.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:51.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:51.844 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:51 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:51.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:51 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:51.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:51.853 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:51.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:51.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:51.987 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:51.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:51.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:51 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:22:52.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:52.115 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:52.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:52.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:52.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:52.245 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:52.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:52.347 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:52.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:52.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:52.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:52.524 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:52.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:52.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:52.566 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:22:52.585 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:22:52.605 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:22:52.623 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:22:52.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:52.653 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:52.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:52.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:52.684 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:22:52.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:52.776 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:22:52.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:52 smithi007 ceph-mon[29098]: pgmap v160: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 931 B/s rd, 0 B/s wr, 591 op/s; 3758/265188 objects degraded (1.417%); 13125/265188 objects misplaced (4.949%); 14 MiB/s, 231 objects/s recovering 2024-02-05T07:22:52.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:52 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1359282499' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:22:52.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:52 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:22:52.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:52 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:22:52.777 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:52 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:22:52.777 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:52 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/224270396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:22:52.778 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:52 smithi007 ceph-mon[37158]: pgmap v160: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 931 B/s rd, 0 B/s wr, 591 op/s; 3758/265188 objects degraded (1.417%); 13125/265188 objects misplaced (4.949%); 14 MiB/s, 231 objects/s recovering 2024-02-05T07:22:52.778 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:52 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1359282499' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:22:52.778 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:52 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:22:52.778 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:52 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:22:52.778 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:52 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:22:52.778 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:52 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/224270396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:22:52.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:52.785 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:52.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:52.819 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:22:52.858 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:22:52.886 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:52.904 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:22:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:52.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:52.949 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:22:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:22:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:22:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:22:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:22:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:22:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:22:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:22:52.960 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:22:53.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:52 smithi090 ceph-mon[33213]: pgmap v160: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 931 B/s rd, 0 B/s wr, 591 op/s; 3758/265188 objects degraded (1.417%); 13125/265188 objects misplaced (4.949%); 14 MiB/s, 231 objects/s recovering 2024-02-05T07:22:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:52 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1359282499' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:22:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:52 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:22:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:52 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:22:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:52 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:22:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:52 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/224270396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:22:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:53.058 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:53.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:53.159 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:53.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:53.208 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:53.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:53.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:53.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:53.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:53.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:53.514 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:53.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:53.611 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:22:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:22:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:22:53.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:53 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:22:53.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:53 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:22:53.612 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:53 smithi007 ceph-mon[37158]: osdmap e142: 8 total, 8 up, 8 in 2024-02-05T07:22:53.612 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:53 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4188383336' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "3.18"}]: dispatch 2024-02-05T07:22:53.612 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:53 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "3.18"}]: dispatch 2024-02-05T07:22:53.612 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:53 smithi007 ceph-mon[37158]: pgmap v162: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 541 op/s; 3758/265188 objects degraded (1.417%); 13125/265188 objects misplaced (4.949%); 13 MiB/s, 212 objects/s recovering 2024-02-05T07:22:53.612 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:53 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:22:53.612 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:53 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:22:53.612 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:53 smithi007 ceph-mon[29098]: osdmap e142: 8 total, 8 up, 8 in 2024-02-05T07:22:53.612 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:53 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4188383336' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "3.18"}]: dispatch 2024-02-05T07:22:53.612 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:53 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "3.18"}]: dispatch 2024-02-05T07:22:53.612 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:53 smithi007 ceph-mon[29098]: pgmap v162: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 541 op/s; 3758/265188 objects degraded (1.417%); 13125/265188 objects misplaced (4.949%); 13 MiB/s, 212 objects/s recovering 2024-02-05T07:22:53.613 INFO:teuthology.orchestra.run.smithi007.stderr:clear 3.18 pg_upmap mapping 2024-02-05T07:22:53.616 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:53.661 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:53.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:53.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:53.796 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:53.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:53.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:53.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:54.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:53 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:22:54.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:53 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:22:54.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:53 smithi090 ceph-mon[33213]: osdmap e142: 8 total, 8 up, 8 in 2024-02-05T07:22:54.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:53 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4188383336' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "3.18"}]: dispatch 2024-02-05T07:22:54.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:53 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "3.18"}]: dispatch 2024-02-05T07:22:54.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:53 smithi090 ceph-mon[33213]: pgmap v162: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 541 op/s; 3758/265188 objects degraded (1.417%); 13125/265188 objects misplaced (4.949%); 13 MiB/s, 212 objects/s recovering 2024-02-05T07:22:54.042 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:54.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:54.093 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:54.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:54.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:54.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:54.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:54.371 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:54.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:54.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:54.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:54.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:54.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:54.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:22:54.782 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:54.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:54.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:22:54.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:54.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:54.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:54 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "3.18"}]': finished 2024-02-05T07:22:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:54 smithi007 ceph-mon[29098]: osdmap e143: 8 total, 8 up, 8 in 2024-02-05T07:22:54.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:54 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 3758/265188 objects degraded (1.417%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:22:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:54 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "3.18"}]': finished 2024-02-05T07:22:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:54 smithi007 ceph-mon[37158]: osdmap e143: 8 total, 8 up, 8 in 2024-02-05T07:22:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:54 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 3758/265188 objects degraded (1.417%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:22:55.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:55.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:54 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "3.18"}]': finished 2024-02-05T07:22:55.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:54 smithi090 ceph-mon[33213]: osdmap e143: 8 total, 8 up, 8 in 2024-02-05T07:22:55.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:54 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 3758/265188 objects degraded (1.417%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:22:55.056 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:55.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:55.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:55.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:55.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:55.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:55.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:55.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:55.328 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:55.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:55.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:55.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:55.458 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:55.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:55.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:55.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:55.592 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:55.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:55.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:55.733 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:55.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:55.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:55.835 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:55.876 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:55.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:55.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:55.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:55 smithi007 ceph-mon[29098]: 7.10 scrub starts 2024-02-05T07:22:55.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:55 smithi007 ceph-mon[29098]: osdmap e144: 8 total, 8 up, 8 in 2024-02-05T07:22:55.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:55 smithi007 ceph-mon[29098]: pgmap v165: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 786 op/s; 3758/260784 objects degraded (1.441%); 13125/260784 objects misplaced (5.033%); 13 MiB/s, 212 objects/s recovering 2024-02-05T07:22:55.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:55 smithi007 ceph-mon[37158]: 7.10 scrub starts 2024-02-05T07:22:55.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:55 smithi007 ceph-mon[37158]: osdmap e144: 8 total, 8 up, 8 in 2024-02-05T07:22:55.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:55 smithi007 ceph-mon[37158]: pgmap v165: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 786 op/s; 3758/260784 objects degraded (1.441%); 13125/260784 objects misplaced (5.033%); 13 MiB/s, 212 objects/s recovering 2024-02-05T07:22:55.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:56.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:56.002 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:56.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:56.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:55 smithi090 ceph-mon[33213]: 7.10 scrub starts 2024-02-05T07:22:56.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:55 smithi090 ceph-mon[33213]: osdmap e144: 8 total, 8 up, 8 in 2024-02-05T07:22:56.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:55 smithi090 ceph-mon[33213]: pgmap v165: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 786 op/s; 3758/260784 objects degraded (1.441%); 13125/260784 objects misplaced (5.033%); 13 MiB/s, 212 objects/s recovering 2024-02-05T07:22:56.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:56.134 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:56.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:56.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:56.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:56.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:56.264 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:56.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:56.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:56.406 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:56.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:56.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:22:56.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:22:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:56.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:22:56.649 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:56.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:56.679 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:56.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:56.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:56.837 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:56.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:56.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:56 smithi007 ceph-mon[29098]: 7.10 scrub ok 2024-02-05T07:22:56.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:56 smithi007 ceph-mon[29098]: 6.1d scrub starts 2024-02-05T07:22:56.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:56 smithi007 ceph-mon[29098]: 6.1d scrub ok 2024-02-05T07:22:56.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:56 smithi007 ceph-mon[37158]: 7.10 scrub ok 2024-02-05T07:22:56.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:56 smithi007 ceph-mon[37158]: 6.1d scrub starts 2024-02-05T07:22:56.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:56 smithi007 ceph-mon[37158]: 6.1d scrub ok 2024-02-05T07:22:56.925 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:56 smithi090 ceph-mon[33213]: 7.10 scrub ok 2024-02-05T07:22:56.926 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:56 smithi090 ceph-mon[33213]: 6.1d scrub starts 2024-02-05T07:22:56.926 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:56 smithi090 ceph-mon[33213]: 6.1d scrub ok 2024-02-05T07:22:56.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:56.966 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:56.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:56.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:57.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:57.139 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:57.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:57.241 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:57.270 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:57.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:57.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:57.401 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:57.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:57.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:57.533 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:57.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:57.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:57.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:57.663 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:57.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:57.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:57.794 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:57.794 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:22:57.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:57.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:57.827 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:22:57.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:57 smithi007 ceph-mon[29098]: pgmap v166: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 61 op/s; 3458/252552 objects degraded (1.369%); 11815/252552 objects misplaced (4.678%) 2024-02-05T07:22:57.876 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:57 smithi007 ceph-mon[37158]: pgmap v166: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 61 op/s; 3458/252552 objects degraded (1.369%); 11815/252552 objects misplaced (4.678%) 2024-02-05T07:22:57.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:57.953 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:57.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:58.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:57 smithi090 ceph-mon[33213]: pgmap v166: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 61 op/s; 3458/252552 objects degraded (1.369%); 11815/252552 objects misplaced (4.678%) 2024-02-05T07:22:58.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:58.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:58.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:58.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:58.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:58.258 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:58.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:58.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:58.387 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:58.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:58.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:58.516 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:58.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:58.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:58.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:58.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:58.647 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:58.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:58.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:22:58.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:58.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:58.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:22:58.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:22:58.940 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:22:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:58.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:22:59.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:59.071 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:59.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:59.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:59.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:59.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:58 smithi007 ceph-mon[29098]: 5.14 scrub starts 2024-02-05T07:22:59.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:22:58 smithi007 ceph-mon[29098]: 5.14 scrub ok 2024-02-05T07:22:59.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:58 smithi007 ceph-mon[37158]: 5.14 scrub starts 2024-02-05T07:22:59.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:22:58 smithi007 ceph-mon[37158]: 5.14 scrub ok 2024-02-05T07:22:59.182 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:22:59.201 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:59.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:59.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:58 smithi090 ceph-mon[33213]: 5.14 scrub starts 2024-02-05T07:22:59.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:22:58 smithi090 ceph-mon[33213]: 5.14 scrub ok 2024-02-05T07:22:59.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:22:59.332 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:59.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:59.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:22:59.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:22:59.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:59.464 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:59.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:22:59.531 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:22:59.531 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:22:59.565 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:22:59.594 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:59.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:59.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:22:59.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:59.724 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:59.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:59.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:22:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:22:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:22:59.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:22:59.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:00.005 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:00.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:00.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:00.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:00.145 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:00.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:00.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:00.311 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[29098]: 5.1d scrub starts 2024-02-05T07:23:00.311 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[29098]: 5.1d scrub ok 2024-02-05T07:23:00.311 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[29098]: pgmap v167: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 57 op/s; 3458/252552 objects degraded (1.369%); 11815/252552 objects misplaced (4.678%) 2024-02-05T07:23:00.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 3458/252552 objects degraded (1.369%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:23:00.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[29098]: osdmap e145: 8 total, 8 up, 8 in 2024-02-05T07:23:00.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:23:00.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:23:00.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:23:00.312 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[37158]: 5.1d scrub starts 2024-02-05T07:23:00.312 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[37158]: 5.1d scrub ok 2024-02-05T07:23:00.312 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[37158]: pgmap v167: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 57 op/s; 3458/252552 objects degraded (1.369%); 11815/252552 objects misplaced (4.678%) 2024-02-05T07:23:00.312 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 3458/252552 objects degraded (1.369%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:23:00.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[37158]: osdmap e145: 8 total, 8 up, 8 in 2024-02-05T07:23:00.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:23:00.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:23:00.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:00 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:23:00.318 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:00.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:00.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:00.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:00.463 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:00.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:00.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:23:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:23:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:00 smithi090 ceph-mon[33213]: 5.1d scrub starts 2024-02-05T07:23:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:00 smithi090 ceph-mon[33213]: 5.1d scrub ok 2024-02-05T07:23:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:00 smithi090 ceph-mon[33213]: pgmap v167: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 154 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 57 op/s; 3458/252552 objects degraded (1.369%); 11815/252552 objects misplaced (4.678%) 2024-02-05T07:23:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:00 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 3458/252552 objects degraded (1.369%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:23:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:00 smithi090 ceph-mon[33213]: osdmap e145: 8 total, 8 up, 8 in 2024-02-05T07:23:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:00 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:23:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:00 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:23:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:00 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:23:00.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:00.589 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:00.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:00.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:00.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:00.722 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:00.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:00.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:00.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:00.849 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:00.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:00.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:00.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:00.984 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:00.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:01.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:01.109 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:01.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:01.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T07:23:01.211 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:01.236 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:01.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:01.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:01.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:01.411 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:01.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:01.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:01.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:01.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:01 smithi090 ceph-mon[33213]: 5.12 scrub starts 2024-02-05T07:23:01.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:01 smithi090 ceph-mon[33213]: 5.12 scrub ok 2024-02-05T07:23:01.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:01.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:01 smithi090 ceph-mon[33213]: osdmap e146: 8 total, 8 up, 8 in 2024-02-05T07:23:01.544 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:01.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:01.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:01.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:01 smithi007 ceph-mon[29098]: 5.12 scrub starts 2024-02-05T07:23:01.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:01 smithi007 ceph-mon[29098]: 5.12 scrub ok 2024-02-05T07:23:01.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:01.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:01 smithi007 ceph-mon[29098]: osdmap e146: 8 total, 8 up, 8 in 2024-02-05T07:23:01.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:01 smithi007 ceph-mon[37158]: 5.12 scrub starts 2024-02-05T07:23:01.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:01 smithi007 ceph-mon[37158]: 5.12 scrub ok 2024-02-05T07:23:01.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:01.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:01 smithi007 ceph-mon[37158]: osdmap e146: 8 total, 8 up, 8 in 2024-02-05T07:23:01.679 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:01.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:01.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:01.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:01.811 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:01.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:01.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:01.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:01.945 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:01.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:01.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:02.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:02.079 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:02.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:02.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:02.085 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T07:23:02.085 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:23:02.085 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-05T07:23:02.085 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.736284 2024-02-05T07:23:02.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 5 0.7362844783032991 2024-02-05T07:23:02.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:02.194 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:23:02.238 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:02.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:02.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:02.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:02.370 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:02.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:02.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:02 smithi007 ceph-mon[29098]: pgmap v170: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 750 op/s; 14006/239337 objects misplaced (5.852%) 2024-02-05T07:23:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:02 smithi007 ceph-mon[29098]: osdmap e147: 8 total, 8 up, 8 in 2024-02-05T07:23:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:02 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1345146885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:23:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:02 smithi007 ceph-mon[29098]: 7.16 scrub starts 2024-02-05T07:23:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:02 smithi007 ceph-mon[37158]: pgmap v170: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 750 op/s; 14006/239337 objects misplaced (5.852%) 2024-02-05T07:23:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:02 smithi007 ceph-mon[37158]: osdmap e147: 8 total, 8 up, 8 in 2024-02-05T07:23:02.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:02 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1345146885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:23:02.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:02 smithi007 ceph-mon[37158]: 7.16 scrub starts 2024-02-05T07:23:02.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:02.496 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:02.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:02.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:02.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:02 smithi090 ceph-mon[33213]: pgmap v170: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 750 op/s; 14006/239337 objects misplaced (5.852%) 2024-02-05T07:23:02.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:02 smithi090 ceph-mon[33213]: osdmap e147: 8 total, 8 up, 8 in 2024-02-05T07:23:02.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:02 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1345146885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:23:02.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:02 smithi090 ceph-mon[33213]: 7.16 scrub starts 2024-02-05T07:23:02.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:02.636 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:02.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:02.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:02.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:02.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:02.862 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:02.892 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:02.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:02.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:02.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:03.024 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:03.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:03.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:03.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:03.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:03.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:03.300 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:03.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:03.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:03.430 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:03.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:03.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:03.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:03 smithi090 ceph-mon[33213]: 6.1 scrub starts 2024-02-05T07:23:03.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:03 smithi090 ceph-mon[33213]: 6.1 scrub ok 2024-02-05T07:23:03.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:03 smithi090 ceph-mon[33213]: 7.16 scrub ok 2024-02-05T07:23:03.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:03 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:23:03.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:03 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1345146885' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:23:03.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:03 smithi090 ceph-mon[33213]: osdmap e148: 8 total, 8 up, 8 in 2024-02-05T07:23:03.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:03 smithi090 ceph-mon[33213]: 3.9 deep-scrub starts 2024-02-05T07:23:03.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:03 smithi090 ceph-mon[33213]: 3.9 deep-scrub ok 2024-02-05T07:23:03.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:03.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[29098]: 6.1 scrub starts 2024-02-05T07:23:03.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[29098]: 6.1 scrub ok 2024-02-05T07:23:03.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[29098]: 7.16 scrub ok 2024-02-05T07:23:03.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:23:03.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1345146885' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:23:03.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[29098]: osdmap e148: 8 total, 8 up, 8 in 2024-02-05T07:23:03.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[29098]: 3.9 deep-scrub starts 2024-02-05T07:23:03.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[29098]: 3.9 deep-scrub ok 2024-02-05T07:23:03.536 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[37158]: 6.1 scrub starts 2024-02-05T07:23:03.536 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[37158]: 6.1 scrub ok 2024-02-05T07:23:03.536 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[37158]: 7.16 scrub ok 2024-02-05T07:23:03.536 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:23:03.537 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1345146885' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:23:03.537 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[37158]: osdmap e148: 8 total, 8 up, 8 in 2024-02-05T07:23:03.537 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[37158]: 3.9 deep-scrub starts 2024-02-05T07:23:03.537 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-mon[37158]: 3.9 deep-scrub ok 2024-02-05T07:23:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:03.557 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:03.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:03.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:03.760 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:03.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:03.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:03.906 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:03.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:03.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:03.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:23:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:23:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:23:04.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:04.056 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:04.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:04.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:04.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:04.189 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:04.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:04.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:04.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:04.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:04.421 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:04.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:04.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:04 smithi090 ceph-mon[33213]: pgmap v173: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 14006/239337 objects misplaced (5.852%); 6.6 MiB/s, 105 objects/s recovering 2024-02-05T07:23:04.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:04 smithi090 ceph-mon[33213]: 5.1 scrub starts 2024-02-05T07:23:04.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:04 smithi090 ceph-mon[33213]: 5.1 scrub ok 2024-02-05T07:23:04.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:04 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:23:04.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:04 smithi090 ceph-mon[33213]: osdmap e149: 8 total, 8 up, 8 in 2024-02-05T07:23:04.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:04.593 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:04.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:04.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:04 smithi007 ceph-mon[29098]: pgmap v173: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 14006/239337 objects misplaced (5.852%); 6.6 MiB/s, 105 objects/s recovering 2024-02-05T07:23:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:04 smithi007 ceph-mon[29098]: 5.1 scrub starts 2024-02-05T07:23:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:04 smithi007 ceph-mon[29098]: 5.1 scrub ok 2024-02-05T07:23:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:04 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:23:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:04 smithi007 ceph-mon[29098]: osdmap e149: 8 total, 8 up, 8 in 2024-02-05T07:23:04.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:04 smithi007 ceph-mon[37158]: pgmap v173: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 14006/239337 objects misplaced (5.852%); 6.6 MiB/s, 105 objects/s recovering 2024-02-05T07:23:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:04 smithi007 ceph-mon[37158]: 5.1 scrub starts 2024-02-05T07:23:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:04 smithi007 ceph-mon[37158]: 5.1 scrub ok 2024-02-05T07:23:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:04 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:23:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:04 smithi007 ceph-mon[37158]: osdmap e149: 8 total, 8 up, 8 in 2024-02-05T07:23:04.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:04.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:04.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:04.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:04.891 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:04.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:04.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:04.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:05.020 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:05.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:05.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:05.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:05.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:05.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:05.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:05.305 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:05.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:05.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:05.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:05.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:05 smithi007 ceph-mon[29098]: 4.c scrub starts 2024-02-05T07:23:05.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:05 smithi007 ceph-mon[29098]: 4.c scrub ok 2024-02-05T07:23:05.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:05 smithi007 ceph-mon[37158]: 4.c scrub starts 2024-02-05T07:23:05.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:05 smithi007 ceph-mon[37158]: 4.c scrub ok 2024-02-05T07:23:05.437 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:05.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:05.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:05 smithi090 ceph-mon[33213]: 4.c scrub starts 2024-02-05T07:23:05.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:05 smithi090 ceph-mon[33213]: 4.c scrub ok 2024-02-05T07:23:05.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:05.569 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:05.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:05.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:05.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:05.699 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:05.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:05.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:05.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:05.826 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:05.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:05.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:05.957 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:05.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:05.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:06.059 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:06.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:06.105 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:06.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:06.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:06.229 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:06.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:06.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:06.331 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:06.356 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:06.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:06.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:06 smithi007 ceph-mon[29098]: pgmap v175: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 13580/235002 objects misplaced (5.779%) 2024-02-05T07:23:06.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:06 smithi007 ceph-mon[37158]: pgmap v175: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 13580/235002 objects misplaced (5.779%) 2024-02-05T07:23:06.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:06.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:06.483 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:06.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:06.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:06 smithi090 ceph-mon[33213]: pgmap v175: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 13580/235002 objects misplaced (5.779%) 2024-02-05T07:23:06.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:06.617 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:06.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:06.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:06.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:06.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:06.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:06.883 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:06.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:06.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:06.985 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:07.026 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:07.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:07.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:07.128 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:07.166 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:07.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:07.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:07.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:07.296 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:07.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:07.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:07.422 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:07.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:07.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:07.524 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:07.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:07 smithi090 ceph-mon[33213]: 4.7 scrub starts 2024-02-05T07:23:07.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:07 smithi090 ceph-mon[33213]: 4.7 scrub ok 2024-02-05T07:23:07.567 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:07.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:07.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:07.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:07.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:07 smithi007 ceph-mon[29098]: 4.7 scrub starts 2024-02-05T07:23:07.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:07 smithi007 ceph-mon[29098]: 4.7 scrub ok 2024-02-05T07:23:07.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:07 smithi007 ceph-mon[37158]: 4.7 scrub starts 2024-02-05T07:23:07.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:07 smithi007 ceph-mon[37158]: 4.7 scrub ok 2024-02-05T07:23:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:07.715 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:07.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:07.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:07.846 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:07.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:07.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:07.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:07.993 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:07.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:08.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:08.137 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:08.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:08.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:08.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:08.267 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:08.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:08.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:08.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:08.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:08.397 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:08.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:08.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:08.538 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:08.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:08.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:08.640 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:08.668 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:08.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:08.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:08 smithi007 ceph-mon[29098]: pgmap v176: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 12147/228573 objects misplaced (5.314%) 2024-02-05T07:23:08.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:08 smithi007 ceph-mon[37158]: pgmap v176: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 12147/228573 objects misplaced (5.314%) 2024-02-05T07:23:08.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:08 smithi090 ceph-mon[33213]: pgmap v176: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 12147/228573 objects misplaced (5.314%) 2024-02-05T07:23:08.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:08.801 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:08.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:08.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:08.926 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:08.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:08.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:09.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:09.070 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:09.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:09.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:09.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:09.197 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:09.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:09.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:09.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:09.344 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:09.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:09.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:09.487 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:09.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:09.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:09.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:09.617 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:09.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:09.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:09 smithi007 ceph-mon[29098]: 5.19 scrub starts 2024-02-05T07:23:09.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:09 smithi007 ceph-mon[29098]: 5.19 scrub ok 2024-02-05T07:23:09.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:09 smithi007 ceph-mon[29098]: 3.3 scrub starts 2024-02-05T07:23:09.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:09 smithi007 ceph-mon[29098]: 3.3 scrub ok 2024-02-05T07:23:09.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:09 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-02-05T07:23:09.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:09 smithi007 ceph-mon[37158]: 5.19 scrub starts 2024-02-05T07:23:09.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:09 smithi007 ceph-mon[37158]: 5.19 scrub ok 2024-02-05T07:23:09.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:09 smithi007 ceph-mon[37158]: 3.3 scrub starts 2024-02-05T07:23:09.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:09 smithi007 ceph-mon[37158]: 3.3 scrub ok 2024-02-05T07:23:09.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:09 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-02-05T07:23:09.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:09.754 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:09.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:09 smithi090 ceph-mon[33213]: 5.19 scrub starts 2024-02-05T07:23:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:09 smithi090 ceph-mon[33213]: 5.19 scrub ok 2024-02-05T07:23:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:09 smithi090 ceph-mon[33213]: 3.3 scrub starts 2024-02-05T07:23:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:09 smithi090 ceph-mon[33213]: 3.3 scrub ok 2024-02-05T07:23:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:09 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-02-05T07:23:09.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:09.885 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:09.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:09.987 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:10.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:10.134 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:10.162 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:10.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:10.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:10.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:10.307 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:10.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:10.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:10.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:10.440 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:10.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:10.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:23:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:23:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:10 smithi090 ceph-mon[33213]: pgmap v177: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 655 B/s rd, 0 op/s; 12147/228573 objects misplaced (5.314%) 2024-02-05T07:23:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:10 smithi090 ceph-mon[33213]: 6.e scrub starts 2024-02-05T07:23:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:10 smithi090 ceph-mon[33213]: 6.e scrub ok 2024-02-05T07:23:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:10 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:23:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:10 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:23:10.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:10.571 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:10.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:10.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:10 smithi007 ceph-mon[29098]: pgmap v177: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 655 B/s rd, 0 op/s; 12147/228573 objects misplaced (5.314%) 2024-02-05T07:23:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:10 smithi007 ceph-mon[29098]: 6.e scrub starts 2024-02-05T07:23:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:10 smithi007 ceph-mon[29098]: 6.e scrub ok 2024-02-05T07:23:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:10 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:23:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:10 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:23:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:10 smithi007 ceph-mon[37158]: pgmap v177: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 655 B/s rd, 0 op/s; 12147/228573 objects misplaced (5.314%) 2024-02-05T07:23:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:10 smithi007 ceph-mon[37158]: 6.e scrub starts 2024-02-05T07:23:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:10 smithi007 ceph-mon[37158]: 6.e scrub ok 2024-02-05T07:23:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:10 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:23:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:10 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:23:10.720 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:10.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:10.822 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:10.848 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:10.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:10.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:10.977 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:10.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:11.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:11.111 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:11.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:11.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:11.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:11.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:11.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:11.402 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:11.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:11.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:11.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:11.545 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:11.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:11.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:11.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:11.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:11 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:11.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:11 smithi007 ceph-mon[29098]: osdmap e150: 8 total, 8 up, 8 in 2024-02-05T07:23:11.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:11 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:11.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:11 smithi007 ceph-mon[37158]: osdmap e150: 8 total, 8 up, 8 in 2024-02-05T07:23:11.691 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:11.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:11.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:11 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:11.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:11 smithi090 ceph-mon[33213]: osdmap e150: 8 total, 8 up, 8 in 2024-02-05T07:23:11.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:11.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:11.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:11.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:11.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:12.054 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:12.095 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:12.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:12.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:12.224 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:12.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:12.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:12.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:12.368 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:12.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:12.470 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:12.512 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:12.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:12.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:12.661 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:12.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:12.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:12.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:12 smithi007 ceph-mon[29098]: pgmap v179: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 458 op/s; 11032/216027 objects misplaced (5.107%) 2024-02-05T07:23:12.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:12 smithi007 ceph-mon[37158]: pgmap v179: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 458 op/s; 11032/216027 objects misplaced (5.107%) 2024-02-05T07:23:12.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:12 smithi090 ceph-mon[33213]: pgmap v179: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 458 op/s; 11032/216027 objects misplaced (5.107%) 2024-02-05T07:23:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:12.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:12.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:12.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:12.919 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:12.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:13.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:13.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:13.046 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:13.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:13.147 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:13.177 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:13.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:13.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:13.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:13.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:13.309 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:13.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:13.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:13.440 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:13.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:13.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:13.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:13.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:13.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:13.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:13.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:13.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:13.843 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:13.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:13.847 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:23:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:23:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:23:13.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:14.017 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:14.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:14.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:14.053 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:23:14.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:14.135 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:23:14.149 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:14.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:14.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:14.251 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:14.279 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:14.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:14.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:14.381 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:14.412 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:14.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:14.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:14.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:14 smithi007 ceph-mon[29098]: pgmap v180: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 415 op/s; 11032/216027 objects misplaced (5.107%) 2024-02-05T07:23:14.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:14 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:14.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:14 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:23:14.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:14 smithi007 ceph-mon[37158]: pgmap v180: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 415 op/s; 11032/216027 objects misplaced (5.107%) 2024-02-05T07:23:14.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:14 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:14.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:14 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:23:14.514 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:14.545 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:14.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:14.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:14.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:14.677 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:14.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:14.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:14.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:14 smithi090 ceph-mon[33213]: pgmap v180: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 415 op/s; 11032/216027 objects misplaced (5.107%) 2024-02-05T07:23:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:14 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:14 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:23:14.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:14.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:14.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:14.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:23:14.950 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:14.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:14.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:15.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:15.079 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:15.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:15.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:15.181 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:15.209 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:15.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:15.311 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:15.369 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:15.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:15.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:15.498 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:15.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:15.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:15.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:15.630 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:15.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:15.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:15.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:15.760 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:15.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:15.862 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:15.891 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:15.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:15.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:15.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:16.038 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:16.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:16.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:16.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:16.169 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:16.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:16.270 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:16.318 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:16.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:16.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:16.447 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:16.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:16.534 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:16 smithi007 ceph-mon[37158]: pgmap v181: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 793 op/s; 11032/211203 objects misplaced (5.223%); 5.3 MiB/s, 84 objects/s recovering 2024-02-05T07:23:16.534 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:16 smithi007 ceph-mon[37158]: 4.18 scrub starts 2024-02-05T07:23:16.534 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:16 smithi007 ceph-mon[37158]: 4.18 scrub ok 2024-02-05T07:23:16.534 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:16 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2116451545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:23:16.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:16 smithi007 ceph-mon[29098]: pgmap v181: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 793 op/s; 11032/211203 objects misplaced (5.223%); 5.3 MiB/s, 84 objects/s recovering 2024-02-05T07:23:16.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:16 smithi007 ceph-mon[29098]: 4.18 scrub starts 2024-02-05T07:23:16.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:16 smithi007 ceph-mon[29098]: 4.18 scrub ok 2024-02-05T07:23:16.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:16 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2116451545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:23:16.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:16.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:16.582 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:16.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:16.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:16.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:16.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:16.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:16 smithi090 ceph-mon[33213]: pgmap v181: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 793 op/s; 11032/211203 objects misplaced (5.223%); 5.3 MiB/s, 84 objects/s recovering 2024-02-05T07:23:16.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:16 smithi090 ceph-mon[33213]: 4.18 scrub starts 2024-02-05T07:23:16.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:16 smithi090 ceph-mon[33213]: 4.18 scrub ok 2024-02-05T07:23:16.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:16 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2116451545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:23:16.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:16.842 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:16.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:16.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:16.988 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:16.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:17.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:17.126 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:17.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:17.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:17.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:17.257 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:17.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:17.358 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:17.409 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:17.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:17.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:17.515 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:23:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:23:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:23:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:23:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:23:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:23:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:23:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:23:17.516 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:23:17.516 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:23:17.516 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:23:17.517 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:23:17.517 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:23:17.517 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:23:17.517 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:23:17.517 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:23:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:17.551 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:17.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:17.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:17 smithi007 ceph-mon[29098]: 6.2 deep-scrub starts 2024-02-05T07:23:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:17 smithi007 ceph-mon[29098]: 6.2 deep-scrub ok 2024-02-05T07:23:17.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:17 smithi007 ceph-mon[37158]: 6.2 deep-scrub starts 2024-02-05T07:23:17.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:17 smithi007 ceph-mon[37158]: 6.2 deep-scrub ok 2024-02-05T07:23:17.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:17.694 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:17.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:17.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:17 smithi090 ceph-mon[33213]: 6.2 deep-scrub starts 2024-02-05T07:23:17.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:17 smithi090 ceph-mon[33213]: 6.2 deep-scrub ok 2024-02-05T07:23:17.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:17.822 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:17.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:17.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:17.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:17.953 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:17.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:17.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:18.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:18.086 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:18.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:18.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:18.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:18.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:18.364 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:18.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:18.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:18.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:18.509 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:18.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:18.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:18.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:18.638 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:18 smithi007 ceph-mon[29098]: pgmap v182: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 851 op/s; 9702/203031 objects misplaced (4.779%); 5.1 MiB/s, 81 objects/s recovering 2024-02-05T07:23:18.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:18 smithi007 ceph-mon[37158]: pgmap v182: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 851 op/s; 9702/203031 objects misplaced (4.779%); 5.1 MiB/s, 81 objects/s recovering 2024-02-05T07:23:18.656 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:18.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:18.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:18.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:18.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:18 smithi090 ceph-mon[33213]: pgmap v182: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 851 op/s; 9702/203031 objects misplaced (4.779%); 5.1 MiB/s, 81 objects/s recovering 2024-02-05T07:23:18.790 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:18.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:18.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:18.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:18.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:19.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:19.061 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:19.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:19.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:19.101 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:23:19.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:19.213 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:19.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:19.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:19.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:19.346 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:19.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:19.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:19.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:19.478 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:19.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:19.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:19.607 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:19.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:19.708 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:19.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:19.762 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:19.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:19.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:19.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:19.916 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:19.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:20.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:20.059 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:20.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:20.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:23:20.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:20.252 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:20.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:20.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:20.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:20 smithi007 ceph-mon[37158]: pgmap v183: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 851 op/s; 9702/203031 objects misplaced (4.779%); 5.1 MiB/s, 81 objects/s recovering 2024-02-05T07:23:20.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:20 smithi007 ceph-mon[37158]: 4.f scrub starts 2024-02-05T07:23:20.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:20 smithi007 ceph-mon[37158]: 4.f scrub ok 2024-02-05T07:23:20.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:20 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:23:20.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:20 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:23:20.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:20 smithi007 ceph-mon[37158]: osdmap e151: 8 total, 8 up, 8 in 2024-02-05T07:23:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:20.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:20.508 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:20.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:23:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:23:20.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:20 smithi090 ceph-mon[33213]: pgmap v183: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 851 op/s; 9702/203031 objects misplaced (4.779%); 5.1 MiB/s, 81 objects/s recovering 2024-02-05T07:23:20.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:20 smithi090 ceph-mon[33213]: 4.f scrub starts 2024-02-05T07:23:20.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:20 smithi090 ceph-mon[33213]: 4.f scrub ok 2024-02-05T07:23:20.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:20 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:23:20.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:20 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:23:20.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:20 smithi090 ceph-mon[33213]: osdmap e151: 8 total, 8 up, 8 in 2024-02-05T07:23:20.542 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:20.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:20.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:20.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:20.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:20 smithi007 ceph-mon[29098]: pgmap v183: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 851 op/s; 9702/203031 objects misplaced (4.779%); 5.1 MiB/s, 81 objects/s recovering 2024-02-05T07:23:20.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:20 smithi007 ceph-mon[29098]: 4.f scrub starts 2024-02-05T07:23:20.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:20 smithi007 ceph-mon[29098]: 4.f scrub ok 2024-02-05T07:23:20.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:20 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:23:20.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:20 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:23:20.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:20 smithi007 ceph-mon[29098]: osdmap e151: 8 total, 8 up, 8 in 2024-02-05T07:23:20.695 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:20.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:20.797 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:20.844 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:20.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:20.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:20.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:20.971 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:20.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:21.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:21.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:21.122 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:21.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:21.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:21.256 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:21.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:21.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:21.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:21.413 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:21.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:21.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:21.554 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:21.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:21.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:21 smithi007 ceph-mon[29098]: 7.b deep-scrub starts 2024-02-05T07:23:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:21 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:21 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:23:21.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:21 smithi007 ceph-mon[37158]: 7.b deep-scrub starts 2024-02-05T07:23:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:21 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:21 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:23:21.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:21.689 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:21.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:21.719 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:21 smithi090 ceph-mon[33213]: 7.b deep-scrub starts 2024-02-05T07:23:21.719 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:21 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:21.719 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:21 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:23:21.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:21.818 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:21.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:21.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:21.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:21.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:21.952 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:21.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:22.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:22.094 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:22.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:22.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:22.234 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:22.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:22.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:22.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:22.362 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:22.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:22.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:22.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:22.512 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:22.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:22.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:22.644 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:22.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:22.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:22 smithi007 ceph-mon[29098]: pgmap v185: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 880 op/s; 8261/189633 objects misplaced (4.356%); 5.8 MiB/s, 92 objects/s recovering 2024-02-05T07:23:22.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:22 smithi007 ceph-mon[37158]: pgmap v185: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 880 op/s; 8261/189633 objects misplaced (4.356%); 5.8 MiB/s, 92 objects/s recovering 2024-02-05T07:23:22.715 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:22 smithi090 ceph-mon[33213]: pgmap v185: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 880 op/s; 8261/189633 objects misplaced (4.356%); 5.8 MiB/s, 92 objects/s recovering 2024-02-05T07:23:22.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:22.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:22.786 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:22.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:22.887 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:22.923 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:22.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:23.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:23.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:23.079 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:23.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:23.180 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:23.232 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:23.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:23.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:23.373 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:23.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:23.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:23.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:23.511 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:23.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:23.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:23.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:23.649 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:23.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:23.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:23.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:23 smithi007 ceph-mon[29098]: 6.4 scrub starts 2024-02-05T07:23:23.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:23 smithi007 ceph-mon[29098]: 6.4 scrub ok 2024-02-05T07:23:23.674 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:23:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:23:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:23:23.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:23 smithi007 ceph-mon[37158]: 6.4 scrub starts 2024-02-05T07:23:23.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:23 smithi007 ceph-mon[37158]: 6.4 scrub ok 2024-02-05T07:23:23.732 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:23 smithi090 ceph-mon[33213]: 6.4 scrub starts 2024-02-05T07:23:23.732 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:23 smithi090 ceph-mon[33213]: 6.4 scrub ok 2024-02-05T07:23:23.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:23.783 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:23.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:23.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:23.928 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:23.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:24.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:24.060 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:24.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:24.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:24.222 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:24.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:24.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:24.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:24.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:24.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:24.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:24.520 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:24.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:24.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:24.669 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:24.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:24.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:24 smithi007 ceph-mon[29098]: pgmap v186: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 880 op/s; 8261/189633 objects misplaced (4.356%); 5.8 MiB/s, 92 objects/s recovering 2024-02-05T07:23:24.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:24 smithi007 ceph-mon[29098]: 3.14 deep-scrub starts 2024-02-05T07:23:24.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:24 smithi007 ceph-mon[29098]: 3.14 deep-scrub ok 2024-02-05T07:23:24.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:24 smithi007 ceph-mon[37158]: pgmap v186: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 880 op/s; 8261/189633 objects misplaced (4.356%); 5.8 MiB/s, 92 objects/s recovering 2024-02-05T07:23:24.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:24 smithi007 ceph-mon[37158]: 3.14 deep-scrub starts 2024-02-05T07:23:24.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:24 smithi007 ceph-mon[37158]: 3.14 deep-scrub ok 2024-02-05T07:23:24.744 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:24 smithi090 ceph-mon[33213]: pgmap v186: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 880 op/s; 8261/189633 objects misplaced (4.356%); 5.8 MiB/s, 92 objects/s recovering 2024-02-05T07:23:24.744 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:24 smithi090 ceph-mon[33213]: 3.14 deep-scrub starts 2024-02-05T07:23:24.745 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:24 smithi090 ceph-mon[33213]: 3.14 deep-scrub ok 2024-02-05T07:23:24.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:24.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:24.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:24.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:24.875 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:23:24.875 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:23:24.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.875 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.24719.0:149060 7.7 e151/118 7:e3224076:::benchmark_data_smithi007_77086_object44465:head v 151'8258, mlcod=151'8255)", 2024-02-05T07:23:24.875 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:24.193397+0000", 2024-02-05T07:23:24.875 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0083187959999999998, 2024-02-05T07:23:24.875 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0083592300000000005, 2024-02-05T07:23:24.875 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:23:24.876 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:23:24.876 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:23:24.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.877 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:23:24.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.193397+0000", 2024-02-05T07:23:24.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:23:24.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.877 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:23:24.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.193397+0000", 2024-02-05T07:23:24.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:23:24.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:23:24.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.193398+0000", 2024-02-05T07:23:24.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9800000000000002e-07 2024-02-05T07:23:24.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:23:24.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.193406+0000", 2024-02-05T07:23:24.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5630000000000003e-06 2024-02-05T07:23:24.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:23:24.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.193424+0000", 2024-02-05T07:23:24.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8674000000000001e-05 2024-02-05T07:23:24.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:23:24.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.193434+0000", 2024-02-05T07:23:24.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9499999999999996e-06 2024-02-05T07:23:24.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:23:24.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.193622+0000", 2024-02-05T07:23:24.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018792799999999999 2024-02-05T07:23:24.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.880 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:23:24.880 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.193673+0000", 2024-02-05T07:23:24.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1034e-05 2024-02-05T07:23:24.880 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:23:24.880 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:23:24.880 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:23:24.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.881 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.24719.0:149059 7.f e151/100 7:f3a7773e:::benchmark_data_smithi007_77086_object44464:head v 151'9424, mlcod=151'9422)", 2024-02-05T07:23:24.881 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:24.194296+0000", 2024-02-05T07:23:24.881 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0074196679999999999, 2024-02-05T07:23:24.881 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0075340449999999996, 2024-02-05T07:23:24.881 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:23:24.881 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:23:24.881 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:23:24.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:23:24.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.194296+0000", 2024-02-05T07:23:24.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:23:24.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:23:24.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.194296+0000", 2024-02-05T07:23:24.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:23:24.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:23:24.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.194297+0000", 2024-02-05T07:23:24.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4499999999999997e-07 2024-02-05T07:23:24.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:23:24.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.194303+0000", 2024-02-05T07:23:24.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.28e-06 2024-02-05T07:23:24.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:23:24.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.194315+0000", 2024-02-05T07:23:24.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1532000000000001e-05 2024-02-05T07:23:24.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:23:24.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.194322+0000", 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9179999999999997e-06 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.194427+0000", 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000105618 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.194457+0000", 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9496e-05 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:23:24.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.885 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:149058 7.a 7:5478bee6:::benchmark_data_smithi007_77086_object44463:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-05T07:23:24.885 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:24.194552+0000", 2024-02-05T07:23:24.885 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0071640710000000002, 2024-02-05T07:23:24.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0073574440000000003, 2024-02-05T07:23:24.885 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:23:24.885 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:23:24.885 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:23:24.885 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:23:24.885 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:23:24.885 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 149058 2024-02-05T07:23:24.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.886 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:23:24.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:23:24.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.194552+0000", 2024-02-05T07:23:24.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:23:24.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:23:24.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.194552+0000", 2024-02-05T07:23:24.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:23:24.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.887 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:23:24.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.194553+0000", 2024-02-05T07:23:24.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.297e-06 2024-02-05T07:23:24.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:23:24.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.194558+0000", 2024-02-05T07:23:24.888 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.293e-06 2024-02-05T07:23:24.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:23:24.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.194572+0000", 2024-02-05T07:23:24.888 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3383999999999999e-05 2024-02-05T07:23:24.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:23:24.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.194579+0000", 2024-02-05T07:23:24.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4120000000000004e-06 2024-02-05T07:23:24.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:23:24.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.201549+0000", 2024-02-05T07:23:24.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0069696769999999996 2024-02-05T07:23:24.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:23:24.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.201661+0000", 2024-02-05T07:23:24.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000111728 2024-02-05T07:23:24.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-02-05T07:23:24.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.201769+0000", 2024-02-05T07:23:24.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108208 2024-02-05T07:23:24.890 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:23:24.890 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:23:24.890 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:23:24.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.890 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.24719.0:149061 7.7 e151/118 7:e7c9f742:::benchmark_data_smithi007_77086_object44466:head v 151'8259, mlcod=151'8255)", 2024-02-05T07:23:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:24.195486+0000", 2024-02-05T07:23:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0062298249999999996, 2024-02-05T07:23:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.006502936, 2024-02-05T07:23:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:23:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:23:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:23:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:23:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.195486+0000", 2024-02-05T07:23:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:23:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:23:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.195486+0000", 2024-02-05T07:23:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:23:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:23:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.195487+0000", 2024-02-05T07:23:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3699999999999999e-07 2024-02-05T07:23:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:23:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.195493+0000", 2024-02-05T07:23:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2009999999999996e-06 2024-02-05T07:23:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:23:24.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.195506+0000", 2024-02-05T07:23:24.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3063e-05 2024-02-05T07:23:24.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:23:24.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.195513+0000", 2024-02-05T07:23:24.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7549999999999997e-06 2024-02-05T07:23:24.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:23:24.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.199932+0000", 2024-02-05T07:23:24.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044190970000000003 2024-02-05T07:23:24.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:23:24.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.199953+0000", 2024-02-05T07:23:24.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0893000000000001e-05 2024-02-05T07:23:24.894 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:23:24.894 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:23:24.894 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:23:24.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.895 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:149065 7.3 7:c0068d37:::benchmark_data_smithi007_77086_object44470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-05T07:23:24.895 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:24.201300+0000", 2024-02-05T07:23:24.895 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00041577900000000002, 2024-02-05T07:23:24.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00076124299999999999, 2024-02-05T07:23:24.895 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:23:24.895 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:23:24.895 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:23:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:23:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:23:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 149065 2024-02-05T07:23:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:23:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:23:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.201300+0000", 2024-02-05T07:23:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:23:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:23:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.201300+0000", 2024-02-05T07:23:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:23:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:23:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.201302+0000", 2024-02-05T07:23:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5999999999999999e-06 2024-02-05T07:23:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:23:24.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.201309+0000", 2024-02-05T07:23:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0149999999999998e-06 2024-02-05T07:23:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:23:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.201327+0000", 2024-02-05T07:23:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7893999999999998e-05 2024-02-05T07:23:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:23:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.201337+0000", 2024-02-05T07:23:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.084e-05 2024-02-05T07:23:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:23:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.201407+0000", 2024-02-05T07:23:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9691999999999996e-05 2024-02-05T07:23:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:23:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.201561+0000", 2024-02-05T07:23:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015357500000000001 2024-02-05T07:23:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,4", 2024-02-05T07:23:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.201967+0000", 2024-02-05T07:23:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000406275 2024-02-05T07:23:24.899 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:23:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:23:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:23:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:149048 7.a 7:5b0dd911:::benchmark_data_smithi007_77086_object44453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-05T07:23:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:24.171245+0000", 2024-02-05T07:23:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.030471081000000001, 2024-02-05T07:23:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.030892492000000001, 2024-02-05T07:23:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:23:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:23:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:23:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:23:24.900 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:23:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 149048 2024-02-05T07:23:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:23:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:23:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.171245+0000", 2024-02-05T07:23:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:23:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:23:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.171245+0000", 2024-02-05T07:23:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:23:24.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.902 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:23:24.902 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.171246+0000", 2024-02-05T07:23:24.902 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3740000000000001e-06 2024-02-05T07:23:24.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.902 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:23:24.902 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.171251+0000", 2024-02-05T07:23:24.902 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4569999999999998e-06 2024-02-05T07:23:24.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.902 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:23:24.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.171261+0000", 2024-02-05T07:23:24.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0703000000000001e-05 2024-02-05T07:23:24.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.903 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:23:24.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.171268+0000", 2024-02-05T07:23:24.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.058e-06 2024-02-05T07:23:24.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.903 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:23:24.904 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.173834+0000", 2024-02-05T07:23:24.904 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002565716 2024-02-05T07:23:24.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.904 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:23:24.904 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.173968+0000", 2024-02-05T07:23:24.904 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013442799999999999 2024-02-05T07:23:24.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.904 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-02-05T07:23:24.905 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.174092+0000", 2024-02-05T07:23:24.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012360499999999999 2024-02-05T07:23:24.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.905 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:23:24.906 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.201428+0000", 2024-02-05T07:23:24.906 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.027335768999999999 2024-02-05T07:23:24.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:23:24.906 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:23:24.906 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:23:24.906 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:24.201516+0000", 2024-02-05T07:23:24.906 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7850999999999996e-05 2024-02-05T07:23:24.906 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:23:24.906 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:23:24.906 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:23:24.906 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:23:24.906 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:23:24.907 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 6 2024-02-05T07:23:24.907 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:23:24.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:24.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:24.965 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:24.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:25.066 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:25.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:25.115 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:25.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:25.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:25.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:25.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:25.371 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:25.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:25.418 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:25.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:25.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:25.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:25.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:25.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:25.687 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:25.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:25.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:25.821 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:25.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:25.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:25.962 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:25.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:25.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:26.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:26.092 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:26.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:26.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:26.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:26.237 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:26.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:26.339 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:26.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T07:23:26.406 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:26.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:26.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:26.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:26.545 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:26.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:26.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:26.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:26.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:26 smithi007 ceph-mon[29098]: pgmap v187: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 852 op/s; 8261/185640 objects misplaced (4.450%); 5.8 MiB/s, 92 objects/s recovering 2024-02-05T07:23:26.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:26 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1339061287' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7362844783032991}]: dispatch 2024-02-05T07:23:26.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:26 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7362844783032991}]: dispatch 2024-02-05T07:23:26.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:26 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:26.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:26 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:26.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:26 smithi007 ceph-mon[37158]: pgmap v187: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 852 op/s; 8261/185640 objects misplaced (4.450%); 5.8 MiB/s, 92 objects/s recovering 2024-02-05T07:23:26.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:26 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1339061287' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7362844783032991}]: dispatch 2024-02-05T07:23:26.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:26 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7362844783032991}]: dispatch 2024-02-05T07:23:26.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:26 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:26.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:26 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:26.677 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:26 smithi090 ceph-mon[33213]: pgmap v187: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 852 op/s; 8261/185640 objects misplaced (4.450%); 5.8 MiB/s, 92 objects/s recovering 2024-02-05T07:23:26.677 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:26 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1339061287' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7362844783032991}]: dispatch 2024-02-05T07:23:26.677 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:26 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7362844783032991}]: dispatch 2024-02-05T07:23:26.677 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:26 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:26.677 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:26 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:26.679 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:26.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:26.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:26.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:26.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:26.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:26.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:26.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.5 primary-affinity to 0.736284 (8482532) 2024-02-05T07:23:27.084 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:27.128 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:27.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:27.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:27.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:27.265 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:27.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:27.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:27.418 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:27.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:27.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:27.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:27.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:27.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:27.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:27 smithi007 ceph-mon[29098]: 4.2 deep-scrub starts 2024-02-05T07:23:27.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:27 smithi007 ceph-mon[29098]: 4.2 deep-scrub ok 2024-02-05T07:23:27.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:27 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7362844783032991}]': finished 2024-02-05T07:23:27.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:27 smithi007 ceph-mon[29098]: osdmap e152: 8 total, 8 up, 8 in 2024-02-05T07:23:27.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:27 smithi007 ceph-mon[29098]: osdmap e153: 8 total, 8 up, 8 in 2024-02-05T07:23:27.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:27 smithi007 ceph-mon[37158]: 4.2 deep-scrub starts 2024-02-05T07:23:27.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:27 smithi007 ceph-mon[37158]: 4.2 deep-scrub ok 2024-02-05T07:23:27.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:27 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7362844783032991}]': finished 2024-02-05T07:23:27.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:27 smithi007 ceph-mon[37158]: osdmap e152: 8 total, 8 up, 8 in 2024-02-05T07:23:27.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:27 smithi007 ceph-mon[37158]: osdmap e153: 8 total, 8 up, 8 in 2024-02-05T07:23:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:27.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:27.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:27.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:27 smithi090 ceph-mon[33213]: 4.2 deep-scrub starts 2024-02-05T07:23:27.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:27 smithi090 ceph-mon[33213]: 4.2 deep-scrub ok 2024-02-05T07:23:27.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:27 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7362844783032991}]': finished 2024-02-05T07:23:27.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:27 smithi090 ceph-mon[33213]: osdmap e152: 8 total, 8 up, 8 in 2024-02-05T07:23:27.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:27 smithi090 ceph-mon[33213]: osdmap e153: 8 total, 8 up, 8 in 2024-02-05T07:23:27.795 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:27.847 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:27.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:27.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:27.994 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:27.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:27.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:28.096 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:28.126 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:28.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:28.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:28.262 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:28.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:28.331 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:23:28.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:28.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:28.393 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:28.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:28.421 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:23:28.494 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:28.565 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:28.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:28.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:28.695 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:28.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:28.703 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:28 smithi007 ceph-mon[37158]: pgmap v189: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 6780/178803 objects misplaced (3.792%); 7.6 MiB/s, 121 objects/s recovering 2024-02-05T07:23:28.703 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:28 smithi007 ceph-mon[37158]: 4.4 scrub starts 2024-02-05T07:23:28.703 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:28 smithi007 ceph-mon[37158]: 4.4 scrub ok 2024-02-05T07:23:28.703 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:28 smithi007 ceph-mon[29098]: pgmap v189: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 6780/178803 objects misplaced (3.792%); 7.6 MiB/s, 121 objects/s recovering 2024-02-05T07:23:28.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:28 smithi007 ceph-mon[29098]: 4.4 scrub starts 2024-02-05T07:23:28.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:28 smithi007 ceph-mon[29098]: 4.4 scrub ok 2024-02-05T07:23:28.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:28 smithi090 ceph-mon[33213]: pgmap v189: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 6780/178803 objects misplaced (3.792%); 7.6 MiB/s, 121 objects/s recovering 2024-02-05T07:23:28.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:28 smithi090 ceph-mon[33213]: 4.4 scrub starts 2024-02-05T07:23:28.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:28 smithi090 ceph-mon[33213]: 4.4 scrub ok 2024-02-05T07:23:28.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:28.827 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:28.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:28.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:28.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:28.974 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:28.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:28.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:29.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:29.107 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:29.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:29.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:29.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:29.241 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:29.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:29.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:29.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:29.385 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:29.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:29.449 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:23:29.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:29.534 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:29.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:29.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:29.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:29.670 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:29.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:29.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:29.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:29 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/92263038' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:23:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:29 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:23:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:23:29.812 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:29.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:29.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:29 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/92263038' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:23:29.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:29 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:23:29.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:23:29.904 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:29 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/92263038' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:23:29.904 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:29 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:23:29.904 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:23:29.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:29.944 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:29.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:30.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:30.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:30.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:30.200 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:30.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:30.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:30.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:30.335 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:30.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:30.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:30.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:30.451 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:23:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:23:30.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:30.479 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:30.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:30.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:30.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:30.615 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:30.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:30.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:30.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:30.741 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:30.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:30 smithi090 ceph-mon[33213]: pgmap v191: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 451 op/s; 6780/178803 objects misplaced (3.792%); 3.9 MiB/s, 61 objects/s recovering 2024-02-05T07:23:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:30 smithi090 ceph-mon[33213]: 2.2 scrub starts 2024-02-05T07:23:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:30 smithi090 ceph-mon[33213]: 2.2 scrub ok 2024-02-05T07:23:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:30 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:23:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:30 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:23:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:30 smithi090 ceph-mon[33213]: osdmap e154: 8 total, 8 up, 8 in 2024-02-05T07:23:30.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:30.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:30.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:30 smithi007 ceph-mon[29098]: pgmap v191: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 451 op/s; 6780/178803 objects misplaced (3.792%); 3.9 MiB/s, 61 objects/s recovering 2024-02-05T07:23:30.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:30 smithi007 ceph-mon[29098]: 2.2 scrub starts 2024-02-05T07:23:30.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:30 smithi007 ceph-mon[29098]: 2.2 scrub ok 2024-02-05T07:23:30.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:30 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:23:30.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:30 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:23:30.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:30 smithi007 ceph-mon[29098]: osdmap e154: 8 total, 8 up, 8 in 2024-02-05T07:23:30.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:30 smithi007 ceph-mon[37158]: pgmap v191: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 451 op/s; 6780/178803 objects misplaced (3.792%); 3.9 MiB/s, 61 objects/s recovering 2024-02-05T07:23:30.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:30 smithi007 ceph-mon[37158]: 2.2 scrub starts 2024-02-05T07:23:30.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:30 smithi007 ceph-mon[37158]: 2.2 scrub ok 2024-02-05T07:23:30.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:30 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:23:30.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:30 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:23:30.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:30 smithi007 ceph-mon[37158]: osdmap e154: 8 total, 8 up, 8 in 2024-02-05T07:23:30.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:31.010 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:31.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:31.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:31.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:31.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:31.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:31.279 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:31.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:31.381 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:31.420 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:31.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:31.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:31.526 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:31.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:31.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:31.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:31.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:31 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:31.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:31 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:31.696 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:31.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:31.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:31 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:31.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:31.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:31.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:31.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:31.962 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:31.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:31.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:32.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:32.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:32.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:32.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:32.232 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:32.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:32.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:32.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:32.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:32.369 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:32.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:32.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:32.500 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:32.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:32.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:32.602 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:32.632 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:32.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:32.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:32.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:32 smithi007 ceph-mon[29098]: pgmap v193: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 4965/167328 objects misplaced (2.967%); 11 MiB/s, 183 objects/s recovering 2024-02-05T07:23:32.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:32 smithi007 ceph-mon[37158]: pgmap v193: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 4965/167328 objects misplaced (2.967%); 11 MiB/s, 183 objects/s recovering 2024-02-05T07:23:32.733 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:32.753 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T07:23:32.753 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:23:32.753 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T07:23:32.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd out 3 2024-02-05T07:23:32.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:32.760 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:32.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:32.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:32 smithi090 ceph-mon[33213]: pgmap v193: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 4965/167328 objects misplaced (2.967%); 11 MiB/s, 183 objects/s recovering 2024-02-05T07:23:32.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:32.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:32.917 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:32.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:33.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:33.076 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:33.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:33.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:33.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:33.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:33.330 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:33.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:33.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:33.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:33.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:33.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:33.900 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:23:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:23:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:23:33.921 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:33.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:33.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:34.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:34.070 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:34.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:34.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:34.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:34.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:34.225 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:34.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:34.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:34.361 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:34.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:34.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:34.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:34.524 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:34.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:34.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:34.652 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:34.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:34.753 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:34 smithi090 ceph-mon[33213]: pgmap v194: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 775 B/s rd, 0 B/s wr, 580 op/s; 4965/167328 objects misplaced (2.967%); 5.7 MiB/s, 91 objects/s recovering 2024-02-05T07:23:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:34.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:34.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:34.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:34 smithi007 ceph-mon[29098]: pgmap v194: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 775 B/s rd, 0 B/s wr, 580 op/s; 4965/167328 objects misplaced (2.967%); 5.7 MiB/s, 91 objects/s recovering 2024-02-05T07:23:34.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:34 smithi007 ceph-mon[37158]: pgmap v194: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 775 B/s rd, 0 B/s wr, 580 op/s; 4965/167328 objects misplaced (2.967%); 5.7 MiB/s, 91 objects/s recovering 2024-02-05T07:23:34.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:34.928 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:34.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:35.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:35.059 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:35.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:35.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:35.202 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:35.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:35.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:35.334 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:35.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:35.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:35.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:35.514 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:35.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:35.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:35.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:35.645 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:35.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:35.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:35.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:35.791 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:35.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:35.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:35.945 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:35.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:35.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:23:36.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:36.087 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:36.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:36.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:36.167 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:23:36.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:36.216 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:23:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:36.259 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:36.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:36.274 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:23:36.319 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:23:36.358 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:23:36.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:36.390 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:36.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:36.404 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:23:36.447 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:23:36.488 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:23:36.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:36.504 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:23:36.504 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:23:36.504 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:23:36.504 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:23:36.504 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:23:36.505 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:23:36.505 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:23:36.505 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:23:36.518 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:36.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:36.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:36.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:36.654 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:36.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:36.752 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:36 smithi007 ceph-mon[29098]: pgmap v195: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 658 op/s; 4965/163020 objects misplaced (3.046%); 4.7 MiB/s, 75 objects/s recovering 2024-02-05T07:23:36.753 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:36 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3301104907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:23:36.753 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:36 smithi007 ceph-mon[37158]: pgmap v195: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 658 op/s; 4965/163020 objects misplaced (3.046%); 4.7 MiB/s, 75 objects/s recovering 2024-02-05T07:23:36.753 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:36 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3301104907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:23:36.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:36.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:36 smithi090 ceph-mon[33213]: pgmap v195: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 658 op/s; 4965/163020 objects misplaced (3.046%); 4.7 MiB/s, 75 objects/s recovering 2024-02-05T07:23:36.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:36 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3301104907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:23:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:36.830 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:36.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:36.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:36.963 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:36.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:37.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:37.125 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:37.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:37.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:37.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:37.261 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:37.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:37.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:37.400 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:37.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:37.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:37.548 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:37.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:37.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:37.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:37.681 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:37.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:37.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:37.809 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:37.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:37.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:37.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:37.942 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:37.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:37.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:38.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:38.074 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:38.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:38.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:38.162 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:23:38.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:38.196 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:23:38.197 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:23:38.231 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:38.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:38.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:38.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:38.365 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:38.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:38.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:38.467 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:38.500 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:38.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:38.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:38.602 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:38.636 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:38.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:38.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:38 smithi007 ceph-mon[29098]: pgmap v196: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 791 op/s; 3090/156069 objects misplaced (1.980%); 8.0 MiB/s, 128 objects/s recovering 2024-02-05T07:23:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:38 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:23:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:38 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:23:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:38 smithi007 ceph-mon[29098]: osdmap e155: 8 total, 8 up, 8 in 2024-02-05T07:23:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:38 smithi007 ceph-mon[37158]: pgmap v196: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 791 op/s; 3090/156069 objects misplaced (1.980%); 8.0 MiB/s, 128 objects/s recovering 2024-02-05T07:23:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:38 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:23:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:38 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:23:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:38 smithi007 ceph-mon[37158]: osdmap e155: 8 total, 8 up, 8 in 2024-02-05T07:23:38.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:38.770 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:38.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:38.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:38.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:38 smithi090 ceph-mon[33213]: pgmap v196: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 791 op/s; 3090/156069 objects misplaced (1.980%); 8.0 MiB/s, 128 objects/s recovering 2024-02-05T07:23:38.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:38 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:23:38.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:38 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:23:38.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:38 smithi090 ceph-mon[33213]: osdmap e155: 8 total, 8 up, 8 in 2024-02-05T07:23:38.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:38.912 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:38.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:39.014 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:39.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:39.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:39.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:39.185 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:39.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:39.286 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:39.320 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:39.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:39.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:39.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:39.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:39.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:39.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:39.580 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:39.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:39.665 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:23:39.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:39.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:39.757 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:39.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:39.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:39 smithi090 ceph-mon[33213]: osdmap e156: 8 total, 8 up, 8 in 2024-02-05T07:23:39.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:39.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:39.891 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:39.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:39.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:39 smithi007 ceph-mon[37158]: osdmap e156: 8 total, 8 up, 8 in 2024-02-05T07:23:39.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:39 smithi007 ceph-mon[29098]: osdmap e156: 8 total, 8 up, 8 in 2024-02-05T07:23:39.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:40.042 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:40.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:40.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:40.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:40.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:40.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:40.269 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:40.312 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:40.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:40.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:40.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:40.447 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:40.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:40.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:23:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:23:40.544 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:23:40.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:40.577 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:40.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:40.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:40.702 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:40.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:40.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:40.835 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:40.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:40.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:40 smithi007 ceph-mon[29098]: pgmap v199: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 469 op/s; 3090/156069 objects misplaced (1.980%); 4.9 MiB/s, 78 objects/s recovering 2024-02-05T07:23:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:40 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:23:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:40 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:23:40.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:40 smithi007 ceph-mon[37158]: pgmap v199: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 469 op/s; 3090/156069 objects misplaced (1.980%); 4.9 MiB/s, 78 objects/s recovering 2024-02-05T07:23:40.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:40 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:23:40.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:40 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:23:40.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:40.969 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:40.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:40 smithi090 ceph-mon[33213]: pgmap v199: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 154 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 469 op/s; 3090/156069 objects misplaced (1.980%); 4.9 MiB/s, 78 objects/s recovering 2024-02-05T07:23:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:40 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:23:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:40 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:23:41.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:41.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:41.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:41.218 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:41.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:41.259 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:41.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:41.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:41.391 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:41.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:41.493 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:41.536 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:41.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:41.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:41.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:41.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:41.668 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:41.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:41.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:41 smithi007 ceph-mon[29098]: 7.b deep-scrub ok 2024-02-05T07:23:41.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:41 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:41.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:41 smithi007 ceph-mon[29098]: osdmap e157: 8 total, 8 up, 8 in 2024-02-05T07:23:41.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:41 smithi007 ceph-mon[37158]: 7.b deep-scrub ok 2024-02-05T07:23:41.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:41 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:41.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:41 smithi007 ceph-mon[37158]: osdmap e157: 8 total, 8 up, 8 in 2024-02-05T07:23:41.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:41.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:41.805 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:41.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:41.875 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T07:23:41.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:41.937 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:41.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:42.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:41 smithi090 ceph-mon[33213]: 7.b deep-scrub ok 2024-02-05T07:23:42.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:41 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:42.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:41 smithi090 ceph-mon[33213]: osdmap e157: 8 total, 8 up, 8 in 2024-02-05T07:23:42.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:42.068 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:42.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:42.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:42.170 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:42.240 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:42.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:42.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:42.344 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:42.385 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:42.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:42.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:42.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:42.519 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:42.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:42.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:42.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:42.640 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:42 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:23:42.640 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:42 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:23:42.640 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:42 smithi007 ceph-mon[37158]: pgmap v201: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 1189/144804 objects misplaced (0.821%); 13 MiB/s, 209 objects/s recovering 2024-02-05T07:23:42.640 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:42 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:23:42.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:42 smithi007 ceph-mon[37158]: osdmap e158: 8 total, 8 up, 8 in 2024-02-05T07:23:42.641 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:42 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:23:42.641 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:42 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:23:42.641 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:42 smithi007 ceph-mon[29098]: pgmap v201: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 1189/144804 objects misplaced (0.821%); 13 MiB/s, 209 objects/s recovering 2024-02-05T07:23:42.641 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:42 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:23:42.641 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:42 smithi007 ceph-mon[29098]: osdmap e158: 8 total, 8 up, 8 in 2024-02-05T07:23:42.652 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:42.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:42.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:42.799 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:42.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:42.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:42.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:42.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:42.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:42.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:42 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:23:43.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:42 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:23:43.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:42 smithi090 ceph-mon[33213]: pgmap v201: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 1189/144804 objects misplaced (0.821%); 13 MiB/s, 209 objects/s recovering 2024-02-05T07:23:43.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:42 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:23:43.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:42 smithi090 ceph-mon[33213]: osdmap e158: 8 total, 8 up, 8 in 2024-02-05T07:23:43.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:43.072 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:43.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:43.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:43.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:43.204 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:43.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:43.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:43.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:43.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:43.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:43.454 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:43.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:43.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:43.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:43.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:43.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:43.640 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:43.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:43.741 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:43.783 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:43.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:43.841 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:23:43.842 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:23:43.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:43.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:23:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:23:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:23:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:43.947 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:43.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:44.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:44.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:44.076 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:44.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:44.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:44.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:44.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:44.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:44.331 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:44.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:44.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:44.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:44.456 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:44.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:44.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:44.558 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:44.583 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:44.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:44.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:44.684 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:44.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:44.727 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:44.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:44.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:44.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:44.856 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:44.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:44.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:44 smithi007 ceph-mon[29098]: pgmap v203: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 930 B/s rd, 0 B/s wr, 683 op/s; 1189/144804 objects misplaced (0.821%); 7.2 MiB/s, 115 objects/s recovering 2024-02-05T07:23:44.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:44 smithi007 ceph-mon[29098]: osdmap e159: 8 total, 8 up, 8 in 2024-02-05T07:23:44.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:44 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:44.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:44 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:23:44.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:44 smithi007 ceph-mon[37158]: pgmap v203: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 930 B/s rd, 0 B/s wr, 683 op/s; 1189/144804 objects misplaced (0.821%); 7.2 MiB/s, 115 objects/s recovering 2024-02-05T07:23:44.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:44 smithi007 ceph-mon[37158]: osdmap e159: 8 total, 8 up, 8 in 2024-02-05T07:23:44.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:44 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:44.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:44 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:23:44.957 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:44.983 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:44.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:44 smithi090 ceph-mon[33213]: pgmap v203: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 930 B/s rd, 0 B/s wr, 683 op/s; 1189/144804 objects misplaced (0.821%); 7.2 MiB/s, 115 objects/s recovering 2024-02-05T07:23:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:44 smithi090 ceph-mon[33213]: osdmap e159: 8 total, 8 up, 8 in 2024-02-05T07:23:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:44 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:44 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:23:45.084 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:45.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:45.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:45.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:45.234 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:45.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:45.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:45.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:45.361 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:45.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:45.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:45.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:45.487 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:45.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:45.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:45.622 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:45.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:45.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:45.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:45.755 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:45.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:45.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:45.907 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:45.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:45.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:45 smithi007 ceph-mon[29098]: osdmap e160: 8 total, 8 up, 8 in 2024-02-05T07:23:45.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:45 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a[29094]: 2024-02-05T07:23:45.566+0000 7fe3564d3700 -1 mon.a@0(leader).osd e161 definitely_dead 0 2024-02-05T07:23:45.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:45 smithi007 ceph-mon[37158]: osdmap e160: 8 total, 8 up, 8 in 2024-02-05T07:23:46.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:45 smithi090 ceph-mon[33213]: osdmap e160: 8 total, 8 up, 8 in 2024-02-05T07:23:46.039 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:46.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:46.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:46.141 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:46.182 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:46.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:46.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:46.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:46.312 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:46.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:46.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:46.414 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:46.461 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:46.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:46.589 INFO:teuthology.orchestra.run.smithi007.stderr:marked out osd.3. 2024-02-05T07:23:46.598 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:46.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:46.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:46.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:46.725 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:46.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:46.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:46.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:46.850 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:46.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:46.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:46 smithi007 ceph-mon[29098]: pgmap v206: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 681 op/s; 790/138987 objects misplaced (0.568%); 9.2 MiB/s, 146 objects/s recovering 2024-02-05T07:23:46.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:46 smithi007 ceph-mon[29098]: osdmap e161: 8 total, 8 up, 8 in 2024-02-05T07:23:46.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:46 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/789547544' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-05T07:23:46.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:46 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-05T07:23:46.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:46 smithi007 ceph-mon[29098]: Client client.admin marked osd.3 out, while it was still marked up 2024-02-05T07:23:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:46 smithi007 ceph-mon[37158]: pgmap v206: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 681 op/s; 790/138987 objects misplaced (0.568%); 9.2 MiB/s, 146 objects/s recovering 2024-02-05T07:23:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:46 smithi007 ceph-mon[37158]: osdmap e161: 8 total, 8 up, 8 in 2024-02-05T07:23:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:46 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/789547544' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-05T07:23:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:46 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-05T07:23:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:46 smithi007 ceph-mon[37158]: Client client.admin marked osd.3 out, while it was still marked up 2024-02-05T07:23:46.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:46.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:46.975 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:46.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:47.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:46 smithi090 ceph-mon[33213]: pgmap v206: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 681 op/s; 790/138987 objects misplaced (0.568%); 9.2 MiB/s, 146 objects/s recovering 2024-02-05T07:23:47.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:46 smithi090 ceph-mon[33213]: osdmap e161: 8 total, 8 up, 8 in 2024-02-05T07:23:47.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:46 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/789547544' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-05T07:23:47.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:46 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-05T07:23:47.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:46 smithi090 ceph-mon[33213]: Client client.admin marked osd.3 out, while it was still marked up 2024-02-05T07:23:47.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:47.103 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:47.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:47.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:47.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:47.232 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:47.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:47.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:47.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:47.360 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:47.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:47.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:47.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:47.487 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:47.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:47.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:47.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:47.634 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:47.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:47.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:47 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-05T07:23:47.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:47 smithi007 ceph-mon[29098]: osdmap e162: 8 total, 8 up, 7 in 2024-02-05T07:23:47.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:47 smithi007 ceph-mon[29098]: pgmap v209: 158 pgs: 1 creating+peering, 157 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 6.2 MiB/s, 99 objects/s recovering 2024-02-05T07:23:47.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:47 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-05T07:23:47.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:47 smithi007 ceph-mon[37158]: osdmap e162: 8 total, 8 up, 7 in 2024-02-05T07:23:47.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:47 smithi007 ceph-mon[37158]: pgmap v209: 158 pgs: 1 creating+peering, 157 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 6.2 MiB/s, 99 objects/s recovering 2024-02-05T07:23:47.735 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:47.759 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:47.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:47.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:47.885 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:47.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:47.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:48.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:48.011 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:48.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:47 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-05T07:23:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:47 smithi090 ceph-mon[33213]: osdmap e162: 8 total, 8 up, 7 in 2024-02-05T07:23:48.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:47 smithi090 ceph-mon[33213]: pgmap v209: 158 pgs: 1 creating+peering, 157 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 6.2 MiB/s, 99 objects/s recovering 2024-02-05T07:23:48.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:48.136 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:48.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:48.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:48.266 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:48.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:48.368 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:48.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:48.393 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:48.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:48.494 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:48.519 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:48.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:48.549 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:23:48.549 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:23:48.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:48.664 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:48.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:48 smithi007 ceph-mon[29098]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-02-05T07:23:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:48 smithi007 ceph-mon[29098]: osdmap e163: 8 total, 8 up, 7 in 2024-02-05T07:23:48.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:48 smithi007 ceph-mon[37158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-02-05T07:23:48.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:48 smithi007 ceph-mon[37158]: osdmap e163: 8 total, 8 up, 7 in 2024-02-05T07:23:48.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:48.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:48.810 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:48.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:48.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:48.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:48 smithi090 ceph-mon[33213]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-02-05T07:23:49.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:48 smithi090 ceph-mon[33213]: osdmap e163: 8 total, 8 up, 7 in 2024-02-05T07:23:49.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:49.079 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:49.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:49.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:49.181 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:49.210 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:49.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:49.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:49.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:49.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:49.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:49.470 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:49.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:49.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:49.622 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:49.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:49.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:49.751 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:49.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:49.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:49.850 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[29098]: 7.10 starting backfill to osd.7 from (0'0,0'0] MAX to 161'7438 2024-02-05T07:23:49.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[29098]: 7.a starting backfill to osd.1 from (0'0,0'0] MAX to 161'10563 2024-02-05T07:23:49.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[29098]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 161'7601 2024-02-05T07:23:49.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[29098]: 7.14 starting backfill to osd.0 from (104'856,159'7532] MIN to 161'7598 2024-02-05T07:23:49.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[29098]: osdmap e164: 8 total, 8 up, 7 in 2024-02-05T07:23:49.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[29098]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 161'10516 2024-02-05T07:23:49.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[29098]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 161'8812 2024-02-05T07:23:49.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[29098]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 161'7731 2024-02-05T07:23:49.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[29098]: 7.11 starting backfill to osd.1 from (0'0,0'0] MAX to 161'7509 2024-02-05T07:23:49.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[29098]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 161'10398 2024-02-05T07:23:49.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[29098]: 7.3 starting backfill to osd.6 from (0'0,0'0] MAX to 161'7580 2024-02-05T07:23:49.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[29098]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 161'7611 2024-02-05T07:23:49.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[29098]: pgmap v212: 158 pgs: 1 creating+peering, 157 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 614 op/s; 4.1 MiB/s, 65 objects/s recovering 2024-02-05T07:23:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[37158]: 7.10 starting backfill to osd.7 from (0'0,0'0] MAX to 161'7438 2024-02-05T07:23:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[37158]: 7.a starting backfill to osd.1 from (0'0,0'0] MAX to 161'10563 2024-02-05T07:23:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[37158]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 161'7601 2024-02-05T07:23:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[37158]: 7.14 starting backfill to osd.0 from (104'856,159'7532] MIN to 161'7598 2024-02-05T07:23:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[37158]: osdmap e164: 8 total, 8 up, 7 in 2024-02-05T07:23:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[37158]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 161'10516 2024-02-05T07:23:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[37158]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 161'8812 2024-02-05T07:23:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[37158]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 161'7731 2024-02-05T07:23:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[37158]: 7.11 starting backfill to osd.1 from (0'0,0'0] MAX to 161'7509 2024-02-05T07:23:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[37158]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 161'10398 2024-02-05T07:23:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[37158]: 7.3 starting backfill to osd.6 from (0'0,0'0] MAX to 161'7580 2024-02-05T07:23:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[37158]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 161'7611 2024-02-05T07:23:49.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:49 smithi007 ceph-mon[37158]: pgmap v212: 158 pgs: 1 creating+peering, 157 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 614 op/s; 4.1 MiB/s, 65 objects/s recovering 2024-02-05T07:23:49.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:49.878 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:49.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:49.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:49.993 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:23:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:49 smithi090 ceph-mon[33213]: 7.10 starting backfill to osd.7 from (0'0,0'0] MAX to 161'7438 2024-02-05T07:23:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:49 smithi090 ceph-mon[33213]: 7.a starting backfill to osd.1 from (0'0,0'0] MAX to 161'10563 2024-02-05T07:23:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:49 smithi090 ceph-mon[33213]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 161'7601 2024-02-05T07:23:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:49 smithi090 ceph-mon[33213]: 7.14 starting backfill to osd.0 from (104'856,159'7532] MIN to 161'7598 2024-02-05T07:23:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:49 smithi090 ceph-mon[33213]: osdmap e164: 8 total, 8 up, 7 in 2024-02-05T07:23:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:49 smithi090 ceph-mon[33213]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 161'10516 2024-02-05T07:23:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:49 smithi090 ceph-mon[33213]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 161'8812 2024-02-05T07:23:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:49 smithi090 ceph-mon[33213]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 161'7731 2024-02-05T07:23:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:49 smithi090 ceph-mon[33213]: 7.11 starting backfill to osd.1 from (0'0,0'0] MAX to 161'7509 2024-02-05T07:23:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:49 smithi090 ceph-mon[33213]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 161'10398 2024-02-05T07:23:50.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:49 smithi090 ceph-mon[33213]: 7.3 starting backfill to osd.6 from (0'0,0'0] MAX to 161'7580 2024-02-05T07:23:50.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:49 smithi090 ceph-mon[33213]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 161'7611 2024-02-05T07:23:50.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:49 smithi090 ceph-mon[33213]: pgmap v212: 158 pgs: 1 creating+peering, 157 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 614 op/s; 4.1 MiB/s, 65 objects/s recovering 2024-02-05T07:23:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:50.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:50.043 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:23:50.085 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:23:50.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:50.136 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:23:50.167 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:50.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:50.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:50.178 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:23:50.220 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:23:50.260 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:23:50.263 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T07:23:50.268 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:50.299 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:23:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:23:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:23:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:23:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:23:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:23:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:23:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:23:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:23:50.339 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:50.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:50.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:50.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:50.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:50.477 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:50.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:50.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:23:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:23:50.578 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:50.612 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:23:50.613 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:50.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:50.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:23:50.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:50.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:50.750 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:50.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:50.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:50.886 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:50.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:50.905 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:50 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2453732654' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:23:50.905 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:50 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1195197267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:23:50.905 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:50 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:50.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:50 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2453732654' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:23:50.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:50 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1195197267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:23:50.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:50 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:50.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:50 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2453732654' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:23:50.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:50 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1195197267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:23:50.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:50 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:23:50.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:51.026 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:51.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:51.128 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:51.165 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:51.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:51.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:51.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:51.299 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:51.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:51.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:51.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:51.433 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:51.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:51.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:51.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:51.566 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:51.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:51.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:51.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:51.700 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:51.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:51.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:51.802 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:51.838 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:51.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:51.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:51.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:51 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:23:51.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:51 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1195197267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:23:51.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:51 smithi007 ceph-mon[29098]: osdmap e165: 8 total, 8 up, 7 in 2024-02-05T07:23:51.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:51 smithi007 ceph-mon[29098]: pgmap v214: 158 pgs: 7 active+remapped+backfilling, 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.5 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 455 op/s; 2/125550 objects degraded (0.002%); 17553/125550 objects misplaced (13.981%); 17 MiB/s, 300 objects/s recovering 2024-02-05T07:23:51.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:51 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:23:51.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:51 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1195197267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:23:51.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:51 smithi007 ceph-mon[37158]: osdmap e165: 8 total, 8 up, 7 in 2024-02-05T07:23:51.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:51 smithi007 ceph-mon[37158]: pgmap v214: 158 pgs: 7 active+remapped+backfilling, 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.5 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 455 op/s; 2/125550 objects degraded (0.002%); 17553/125550 objects misplaced (13.981%); 17 MiB/s, 300 objects/s recovering 2024-02-05T07:23:51.940 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:51.997 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:51.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:52.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:51 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:23:52.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:51 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1195197267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:23:52.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:51 smithi090 ceph-mon[33213]: osdmap e165: 8 total, 8 up, 7 in 2024-02-05T07:23:52.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:51 smithi090 ceph-mon[33213]: pgmap v214: 158 pgs: 7 active+remapped+backfilling, 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.5 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 455 op/s; 2/125550 objects degraded (0.002%); 17553/125550 objects misplaced (13.981%); 17 MiB/s, 300 objects/s recovering 2024-02-05T07:23:52.098 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:52.151 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:52.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:52.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:52.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:52.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:52.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:52.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:52.432 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:52.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:52.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:52.561 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:52.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:52.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:52.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:52.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:52.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:52.807 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:52.852 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:52.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:52.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:52 smithi007 ceph-mon[29098]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:23:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:52 smithi007 ceph-mon[29098]: Health check failed: Degraded data redundancy: 2/125550 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:23:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:52 smithi007 ceph-mon[29098]: osdmap e166: 8 total, 8 up, 7 in 2024-02-05T07:23:52.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:52 smithi007 ceph-mon[37158]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:23:52.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:52 smithi007 ceph-mon[37158]: Health check failed: Degraded data redundancy: 2/125550 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:23:52.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:52 smithi007 ceph-mon[37158]: osdmap e166: 8 total, 8 up, 7 in 2024-02-05T07:23:52.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:52.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:52.991 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:52.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:52 smithi090 ceph-mon[33213]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:23:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:52 smithi090 ceph-mon[33213]: Health check failed: Degraded data redundancy: 2/125550 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:23:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:52 smithi090 ceph-mon[33213]: osdmap e166: 8 total, 8 up, 7 in 2024-02-05T07:23:53.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:53.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:53.147 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:53.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:53.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:53.292 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:53.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:53.306 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:23:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:23:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:169688 7.16 7:6f9351ff:::benchmark_data_smithi007_77086_object65093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-02-05T07:23:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:23:53.293505+0000", 2024-02-05T07:23:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0097877490000000001, 2024-02-05T07:23:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098209790000000005, 2024-02-05T07:23:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:23:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:23:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:23:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:23:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:23:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169688 2024-02-05T07:23:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:23:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:23:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.293505+0000", 2024-02-05T07:23:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:23:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:23:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.293505+0000", 2024-02-05T07:23:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:23:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:23:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.293506+0000", 2024-02-05T07:23:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.57e-06 2024-02-05T07:23:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:23:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.293511+0000", 2024-02-05T07:23:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1059999999999999e-06 2024-02-05T07:23:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:23:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.293527+0000", 2024-02-05T07:23:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5764e-05 2024-02-05T07:23:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:23:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.293536+0000", 2024-02-05T07:23:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0489999999999993e-06 2024-02-05T07:23:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:23:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.295125+0000", 2024-02-05T07:23:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001588915 2024-02-05T07:23:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:23:53.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.295264+0000", 2024-02-05T07:23:53.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139204 2024-02-05T07:23:53.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:23:53.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.295611+0000", 2024-02-05T07:23:53.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034630899999999999 2024-02-05T07:23:53.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:23:53.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.299752+0000", 2024-02-05T07:23:53.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041418189999999997 2024-02-05T07:23:53.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:23:53.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.301118+0000", 2024-02-05T07:23:53.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013661120000000001 2024-02-05T07:23:53.312 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:23:53.312 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:169693 7.1 7:8126f653:::benchmark_data_smithi007_77086_object65098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:23:53.297391+0000", 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0059016800000000003, 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0060284479999999996, 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169693 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.313 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.297391+0000", 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.297391+0000", 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.297393+0000", 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8780000000000001e-06 2024-02-05T07:23:53.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.297399+0000", 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0750000000000003e-06 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.297412+0000", 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3474e-05 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.297422+0000", 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0074e-05 2024-02-05T07:23:53.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:23:53.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.297503+0000", 2024-02-05T07:23:53.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0981000000000003e-05 2024-02-05T07:23:53.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:23:53.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.297644+0000", 2024-02-05T07:23:53.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014101700000000001 2024-02-05T07:23:53.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3,6", 2024-02-05T07:23:53.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.298072+0000", 2024-02-05T07:23:53.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042808000000000001 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.300866+0000", 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027939509999999998 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:53.302361+0000", 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00149475 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:23:53.317 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-02-05T07:23:53.318 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 2 2024-02-05T07:23:53.318 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:23:53.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:53.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:53.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:53.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:53.593 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:53.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:53.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:53.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:53.739 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:53.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:53.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:53.841 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:53.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:53 smithi007 ceph-mon[29098]: pgmap v216: 158 pgs: 7 active+remapped+backfilling, 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.5 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 371 op/s; 2/125550 objects degraded (0.002%); 17553/125550 objects misplaced (13.981%); 14 MiB/s, 245 objects/s recovering 2024-02-05T07:23:53.869 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:23:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:23:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:23:53.869 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:53 smithi007 ceph-mon[37158]: pgmap v216: 158 pgs: 7 active+remapped+backfilling, 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.5 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 371 op/s; 2/125550 objects degraded (0.002%); 17553/125550 objects misplaced (13.981%); 14 MiB/s, 245 objects/s recovering 2024-02-05T07:23:53.882 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:53.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:53.915 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:53 smithi090 ceph-mon[33213]: pgmap v216: 158 pgs: 7 active+remapped+backfilling, 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.5 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 371 op/s; 2/125550 objects degraded (0.002%); 17553/125550 objects misplaced (13.981%); 14 MiB/s, 245 objects/s recovering 2024-02-05T07:23:53.983 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:54.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:54.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:54.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T07:23:54.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:54.203 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:54.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:54.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:54.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:54.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:54.460 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:54.514 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:54.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:54.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:23:54.569 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:23:54.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:54.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:54.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:54.793 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:54.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:54.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:54.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:54.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:54.969 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:54.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:55.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:23:55.068 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:23:55.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:55.151 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:55.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:55.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:55 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:55.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:55 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:55.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:55.326 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:55.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:55.386 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:55 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:55.386 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:55 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:55.388 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:55 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:55.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:55 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:55.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:55.484 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:55.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:55.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:55.631 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:55.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:55.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:55.733 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:55.774 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:55.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:55.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:55.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:55.920 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:55.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:56.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:56.081 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:56.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:56.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:56.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:56.217 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:56.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:56.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:56.252 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:56 smithi090 ceph-mon[33213]: pgmap v217: 158 pgs: 7 active+remapped+backfilling, 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.4 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 637 B/s rd, 0 B/s wr, 574 op/s; 2/120546 objects degraded (0.002%); 15251/120546 objects misplaced (12.652%); 30 MiB/s, 489 objects/s recovering 2024-02-05T07:23:56.252 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:56 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:23:56.252 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:56 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:23:56.252 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:56 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:56.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:56.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:56.357 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:56.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:56.390 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T07:23:56.391 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:23:56.391 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-02-05T07:23:56.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:23:56.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:56.492 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:56.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:56.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:56.608 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:23:56.608 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:23:56.608 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:23:56.609 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:23:56.609 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:23:56.625 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:56.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:56.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:56.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:56 smithi007 ceph-mon[29098]: pgmap v217: 158 pgs: 7 active+remapped+backfilling, 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.4 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 637 B/s rd, 0 B/s wr, 574 op/s; 2/120546 objects degraded (0.002%); 15251/120546 objects misplaced (12.652%); 30 MiB/s, 489 objects/s recovering 2024-02-05T07:23:56.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:56 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:23:56.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:56 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:23:56.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:56 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:56.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:56 smithi007 ceph-mon[37158]: pgmap v217: 158 pgs: 7 active+remapped+backfilling, 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.4 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 637 B/s rd, 0 B/s wr, 574 op/s; 2/120546 objects degraded (0.002%); 15251/120546 objects misplaced (12.652%); 30 MiB/s, 489 objects/s recovering 2024-02-05T07:23:56.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:56 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:23:56.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:56 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:23:56.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:56 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:23:56.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:56.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:56.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:56.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:56.891 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:56.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:56.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:56.993 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:57.024 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:57.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:57.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:57.155 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:57.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:57.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:57.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:57.286 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:57.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:57.296 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:23:57.303 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:23:57.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:57.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T07:23:57.420 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:57.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:57.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:57 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:23:57.506 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:57 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:23:57.506 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:57 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:23:57.506 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:57 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:23:57.506 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:57 smithi007 ceph-mon[29098]: osdmap e167: 8 total, 8 up, 7 in 2024-02-05T07:23:57.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:57 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:23:57.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:57 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:23:57.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:57 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:23:57.507 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:57 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:23:57.507 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:57 smithi007 ceph-mon[37158]: osdmap e167: 8 total, 8 up, 7 in 2024-02-05T07:23:57.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:57.584 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:57.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:57.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:57.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:23:57.721 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:57.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:23:57.763 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:23:57.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:57 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:23:57.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:57 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:23:57.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:57 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:23:57.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:57 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:23:57.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:57 smithi090 ceph-mon[33213]: osdmap e167: 8 total, 8 up, 7 in 2024-02-05T07:23:57.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:57.850 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:57.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:57.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:57.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:57.978 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:57.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:57.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:58.080 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:58.111 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:58.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:58.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:58.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:58.245 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:58.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:58.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:58.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:58.378 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:58.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:58.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:58.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:58.518 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:58.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:58.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:58.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:58 smithi090 ceph-mon[33213]: pgmap v219: 158 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.3 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 802 B/s rd, 0 B/s wr, 673 op/s; 2/113994 objects degraded (0.002%); 10029/113994 objects misplaced (8.798%); 63 MiB/s, 1.01k objects/s recovering 2024-02-05T07:23:58.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:58 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 2/113994 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:23:58.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:58 smithi090 ceph-mon[33213]: osdmap e168: 8 total, 8 up, 7 in 2024-02-05T07:23:58.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:23:58.647 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:58.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:58.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:23:58.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:58 smithi007 ceph-mon[29098]: pgmap v219: 158 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.3 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 802 B/s rd, 0 B/s wr, 673 op/s; 2/113994 objects degraded (0.002%); 10029/113994 objects misplaced (8.798%); 63 MiB/s, 1.01k objects/s recovering 2024-02-05T07:23:58.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:58 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 2/113994 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:23:58.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:58 smithi007 ceph-mon[29098]: osdmap e168: 8 total, 8 up, 7 in 2024-02-05T07:23:58.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:58 smithi007 ceph-mon[37158]: pgmap v219: 158 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.3 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 802 B/s rd, 0 B/s wr, 673 op/s; 2/113994 objects degraded (0.002%); 10029/113994 objects misplaced (8.798%); 63 MiB/s, 1.01k objects/s recovering 2024-02-05T07:23:58.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:58 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 2/113994 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:23:58.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:58 smithi007 ceph-mon[37158]: osdmap e168: 8 total, 8 up, 7 in 2024-02-05T07:23:58.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:58.783 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:58.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:58.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:58.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:58.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:58.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:59.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:23:59.054 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:59.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:59.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:23:59.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:23:59.193 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:59.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:23:59.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:23:59.322 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:59.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:23:59.363 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:23:59.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:23:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:59.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:23:59.513 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:59 smithi007 ceph-mon[29098]: osdmap e169: 8 total, 8 up, 7 in 2024-02-05T07:23:59.513 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:23:59 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3835973096' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:23:59.514 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:59 smithi007 ceph-mon[37158]: osdmap e169: 8 total, 8 up, 7 in 2024-02-05T07:23:59.514 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:23:59 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3835973096' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:23:59.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:23:59.586 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:23:59.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:59.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:23:59.688 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:59.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:59.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:59.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:59 smithi090 ceph-mon[33213]: osdmap e169: 8 total, 8 up, 7 in 2024-02-05T07:23:59.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:23:59 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3835973096' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:23:59.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:23:59.876 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:23:59.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:23:59.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:23:59.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:00.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:00.014 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:00.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:00.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:00.154 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:00.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:00.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:00.288 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:00.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:00.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:00.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:00.417 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:00.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:00.503 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:00 smithi007 ceph-mon[29098]: pgmap v222: 158 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.3 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 642 op/s; 2/113994 objects degraded (0.002%); 10029/113994 objects misplaced (8.798%); 63 MiB/s, 1.01k objects/s recovering 2024-02-05T07:24:00.503 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:00 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:24:00.503 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:00 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:00.503 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:00 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3835973096' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:24:00.503 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:00 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:24:00.503 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:00 smithi007 ceph-mon[29098]: osdmap e170: 8 total, 8 up, 7 in 2024-02-05T07:24:00.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:00 smithi007 ceph-mon[37158]: pgmap v222: 158 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.3 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 642 op/s; 2/113994 objects degraded (0.002%); 10029/113994 objects misplaced (8.798%); 63 MiB/s, 1.01k objects/s recovering 2024-02-05T07:24:00.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:00 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:24:00.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:00 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:00.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:00 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3835973096' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:24:00.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:00 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:24:00.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:00 smithi007 ceph-mon[37158]: osdmap e170: 8 total, 8 up, 7 in 2024-02-05T07:24:00.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:00.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:24:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:24:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:00 smithi090 ceph-mon[33213]: pgmap v222: 158 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.3 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 642 op/s; 2/113994 objects degraded (0.002%); 10029/113994 objects misplaced (8.798%); 63 MiB/s, 1.01k objects/s recovering 2024-02-05T07:24:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:00 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:24:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:00 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:00 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3835973096' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:24:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:00 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:24:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:00 smithi090 ceph-mon[33213]: osdmap e170: 8 total, 8 up, 7 in 2024-02-05T07:24:00.544 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:00.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:00.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:00.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:00.669 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:00.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:00.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:00.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:00.796 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:00.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:00.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:00.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:00.923 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:00.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:00.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:01.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:01.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:01.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:01.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:01.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:01.176 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:01.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:01.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:01.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:01.301 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:01.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:01.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:01.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:01.428 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:01.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:01.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:01.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:01.571 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:01.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:01.672 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:01.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:01.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:01 smithi007 ceph-mon[29098]: osdmap e171: 8 total, 8 up, 7 in 2024-02-05T07:24:01.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:01 smithi007 ceph-mon[29098]: osdmap e172: 8 total, 8 up, 7 in 2024-02-05T07:24:01.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:01.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:01 smithi007 ceph-mon[37158]: osdmap e171: 8 total, 8 up, 7 in 2024-02-05T07:24:01.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:01 smithi007 ceph-mon[37158]: osdmap e172: 8 total, 8 up, 7 in 2024-02-05T07:24:01.696 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:01.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:01.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:01 smithi090 ceph-mon[33213]: osdmap e171: 8 total, 8 up, 7 in 2024-02-05T07:24:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:01 smithi090 ceph-mon[33213]: osdmap e172: 8 total, 8 up, 7 in 2024-02-05T07:24:01.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:01.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:01.925 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:01.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:01.950 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:01.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:02.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:02.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:02.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:02.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:02.201 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:02.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:02.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:02.327 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:02.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:02.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:02.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:02.452 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:02.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:02.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:02.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:02.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:02.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:02.665 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:02 smithi090 ceph-mon[33213]: pgmap v225: 158 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 152 active+clean; 2.1 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 748 op/s; 2/105027 objects degraded (0.002%); 4353/105027 objects misplaced (4.145%); 80 MiB/s, 1.27k objects/s recovering 2024-02-05T07:24:02.665 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:02 smithi090 ceph-mon[33213]: 5.13 scrub starts 2024-02-05T07:24:02.665 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:02 smithi090 ceph-mon[33213]: 5.13 scrub ok 2024-02-05T07:24:02.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:02 smithi007 ceph-mon[29098]: pgmap v225: 158 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 152 active+clean; 2.1 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 748 op/s; 2/105027 objects degraded (0.002%); 4353/105027 objects misplaced (4.145%); 80 MiB/s, 1.27k objects/s recovering 2024-02-05T07:24:02.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:02 smithi007 ceph-mon[29098]: 5.13 scrub starts 2024-02-05T07:24:02.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:02 smithi007 ceph-mon[29098]: 5.13 scrub ok 2024-02-05T07:24:02.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:02 smithi007 ceph-mon[37158]: pgmap v225: 158 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 152 active+clean; 2.1 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 748 op/s; 2/105027 objects degraded (0.002%); 4353/105027 objects misplaced (4.145%); 80 MiB/s, 1.27k objects/s recovering 2024-02-05T07:24:02.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:02 smithi007 ceph-mon[37158]: 5.13 scrub starts 2024-02-05T07:24:02.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:02 smithi007 ceph-mon[37158]: 5.13 scrub ok 2024-02-05T07:24:02.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:02.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:02.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:02.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:02.843 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:02.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:02.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:02.983 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:02.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:03.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:03.108 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:03.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:03.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:03.249 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:03.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:03.351 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:03.375 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:03.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:03.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:03.504 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:03.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:03.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:03.637 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:03.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:03.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:03.651 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:24:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:24:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:24:03.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:03 smithi007 ceph-mon[29098]: 7.19 scrub starts 2024-02-05T07:24:03.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:03 smithi007 ceph-mon[29098]: 7.15 scrub starts 2024-02-05T07:24:03.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:03 smithi007 ceph-mon[29098]: 7.19 scrub ok 2024-02-05T07:24:03.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:03 smithi007 ceph-mon[29098]: 7.15 scrub ok 2024-02-05T07:24:03.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:03 smithi007 ceph-mon[29098]: osdmap e173: 8 total, 8 up, 7 in 2024-02-05T07:24:03.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:03 smithi007 ceph-mon[37158]: 7.19 scrub starts 2024-02-05T07:24:03.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:03 smithi007 ceph-mon[37158]: 7.15 scrub starts 2024-02-05T07:24:03.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:03 smithi007 ceph-mon[37158]: 7.19 scrub ok 2024-02-05T07:24:03.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:03 smithi007 ceph-mon[37158]: 7.15 scrub ok 2024-02-05T07:24:03.653 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:03 smithi007 ceph-mon[37158]: osdmap e173: 8 total, 8 up, 7 in 2024-02-05T07:24:03.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:03.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:03.767 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:03.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:03.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:03 smithi090 ceph-mon[33213]: 7.19 scrub starts 2024-02-05T07:24:03.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:03 smithi090 ceph-mon[33213]: 7.15 scrub starts 2024-02-05T07:24:03.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:03 smithi090 ceph-mon[33213]: 7.19 scrub ok 2024-02-05T07:24:03.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:03 smithi090 ceph-mon[33213]: 7.15 scrub ok 2024-02-05T07:24:03.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:03 smithi090 ceph-mon[33213]: osdmap e173: 8 total, 8 up, 7 in 2024-02-05T07:24:03.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:03.901 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:03.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:03.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:04.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:04.038 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:04.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:04.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:04.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:04.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:04.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:04.619 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:04.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:04 smithi007 ceph-mon[29098]: pgmap v228: 158 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 152 active+clean; 2.1 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 748 op/s; 2/105027 objects degraded (0.002%); 4353/105027 objects misplaced (4.145%); 80 MiB/s, 1.27k objects/s recovering 2024-02-05T07:24:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:04 smithi007 ceph-mon[29098]: 7.17 scrub starts 2024-02-05T07:24:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:04 smithi007 ceph-mon[29098]: 7.12 scrub starts 2024-02-05T07:24:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:04 smithi007 ceph-mon[29098]: 7.17 scrub ok 2024-02-05T07:24:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:04 smithi007 ceph-mon[29098]: 7.12 scrub ok 2024-02-05T07:24:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:04 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 2/105027 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:24:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:04 smithi007 ceph-mon[37158]: pgmap v228: 158 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 152 active+clean; 2.1 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 748 op/s; 2/105027 objects degraded (0.002%); 4353/105027 objects misplaced (4.145%); 80 MiB/s, 1.27k objects/s recovering 2024-02-05T07:24:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:04 smithi007 ceph-mon[37158]: 7.17 scrub starts 2024-02-05T07:24:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:04 smithi007 ceph-mon[37158]: 7.12 scrub starts 2024-02-05T07:24:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:04 smithi007 ceph-mon[37158]: 7.17 scrub ok 2024-02-05T07:24:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:04 smithi007 ceph-mon[37158]: 7.12 scrub ok 2024-02-05T07:24:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:04 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 2/105027 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:24:04.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:04.746 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:04.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:04.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:04 smithi090 ceph-mon[33213]: pgmap v228: 158 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 152 active+clean; 2.1 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 748 op/s; 2/105027 objects degraded (0.002%); 4353/105027 objects misplaced (4.145%); 80 MiB/s, 1.27k objects/s recovering 2024-02-05T07:24:04.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:04 smithi090 ceph-mon[33213]: 7.17 scrub starts 2024-02-05T07:24:04.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:04 smithi090 ceph-mon[33213]: 7.12 scrub starts 2024-02-05T07:24:04.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:04 smithi090 ceph-mon[33213]: 7.17 scrub ok 2024-02-05T07:24:04.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:04 smithi090 ceph-mon[33213]: 7.12 scrub ok 2024-02-05T07:24:04.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:04 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 2/105027 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:24:04.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:04.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:04.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:05.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:05.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:05.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:05.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:05.145 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:05.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:05.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:05.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:05.283 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:05.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:05.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:05.421 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:05.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:05.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:05.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:05.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:05.565 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:05.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:05.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:05.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:05 smithi007 ceph-mon[29098]: 7.11 scrub starts 2024-02-05T07:24:05.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:05 smithi007 ceph-mon[29098]: 7.11 scrub ok 2024-02-05T07:24:05.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:05 smithi007 ceph-mon[29098]: 7.18 scrub starts 2024-02-05T07:24:05.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:05 smithi007 ceph-mon[37158]: 7.11 scrub starts 2024-02-05T07:24:05.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:05 smithi007 ceph-mon[37158]: 7.11 scrub ok 2024-02-05T07:24:05.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:05 smithi007 ceph-mon[37158]: 7.18 scrub starts 2024-02-05T07:24:05.705 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:05.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:05.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:05.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:05 smithi090 ceph-mon[33213]: 7.11 scrub starts 2024-02-05T07:24:05.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:05 smithi090 ceph-mon[33213]: 7.11 scrub ok 2024-02-05T07:24:05.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:05 smithi090 ceph-mon[33213]: 7.18 scrub starts 2024-02-05T07:24:05.807 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:05.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:05.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:05.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:05.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:05.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:06.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:06.135 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:06.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:06.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:06.273 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:06.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:06.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:06.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:06.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:06.468 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:06 smithi007 ceph-mon[29098]: pgmap v229: 158 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 152 active+clean; 2.1 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s; 2/103932 objects degraded (0.002%); 4353/103932 objects misplaced (4.188%) 2024-02-05T07:24:06.469 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:06 smithi007 ceph-mon[29098]: 5.11 scrub starts 2024-02-05T07:24:06.469 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:06 smithi007 ceph-mon[29098]: 5.11 scrub ok 2024-02-05T07:24:06.469 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:06 smithi007 ceph-mon[29098]: 6.6 scrub starts 2024-02-05T07:24:06.469 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:06 smithi007 ceph-mon[29098]: 6.6 scrub ok 2024-02-05T07:24:06.469 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:06 smithi007 ceph-mon[29098]: 7.18 scrub ok 2024-02-05T07:24:06.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:06 smithi007 ceph-mon[37158]: pgmap v229: 158 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 152 active+clean; 2.1 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s; 2/103932 objects degraded (0.002%); 4353/103932 objects misplaced (4.188%) 2024-02-05T07:24:06.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:06 smithi007 ceph-mon[37158]: 5.11 scrub starts 2024-02-05T07:24:06.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:06 smithi007 ceph-mon[37158]: 5.11 scrub ok 2024-02-05T07:24:06.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:06 smithi007 ceph-mon[37158]: 6.6 scrub starts 2024-02-05T07:24:06.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:06 smithi007 ceph-mon[37158]: 6.6 scrub ok 2024-02-05T07:24:06.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:06 smithi007 ceph-mon[37158]: 7.18 scrub ok 2024-02-05T07:24:06.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:06.526 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:06.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:06.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:06.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:06.663 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:06.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:06.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:06.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:06 smithi090 ceph-mon[33213]: pgmap v229: 158 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 152 active+clean; 2.1 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s; 2/103932 objects degraded (0.002%); 4353/103932 objects misplaced (4.188%) 2024-02-05T07:24:06.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:06 smithi090 ceph-mon[33213]: 5.11 scrub starts 2024-02-05T07:24:06.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:06 smithi090 ceph-mon[33213]: 5.11 scrub ok 2024-02-05T07:24:06.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:06 smithi090 ceph-mon[33213]: 6.6 scrub starts 2024-02-05T07:24:06.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:06 smithi090 ceph-mon[33213]: 6.6 scrub ok 2024-02-05T07:24:06.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:06 smithi090 ceph-mon[33213]: 7.18 scrub ok 2024-02-05T07:24:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:06.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:06.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:06.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:06.915 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:06.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:06.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:07.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:07.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:07.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:07.220 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:07.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:07.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:07.322 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:07.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:07.346 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:07.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:07.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:07.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:07.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:07.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:07.505 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:24:07.549 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:24:07.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:07.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:07 smithi007 ceph-mon[29098]: 7.13 scrub starts 2024-02-05T07:24:07.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:07 smithi007 ceph-mon[29098]: 6.1a scrub starts 2024-02-05T07:24:07.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:07 smithi007 ceph-mon[29098]: 6.1a scrub ok 2024-02-05T07:24:07.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:07 smithi007 ceph-mon[29098]: 7.13 scrub ok 2024-02-05T07:24:07.601 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:24:07.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:07 smithi007 ceph-mon[37158]: 7.13 scrub starts 2024-02-05T07:24:07.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:07 smithi007 ceph-mon[37158]: 6.1a scrub starts 2024-02-05T07:24:07.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:07 smithi007 ceph-mon[37158]: 6.1a scrub ok 2024-02-05T07:24:07.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:07 smithi007 ceph-mon[37158]: 7.13 scrub ok 2024-02-05T07:24:07.626 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:07.627 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:24:07.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:07.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:07.641 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:24:07.648 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:24:07.686 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:24:07.728 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:24:07.728 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:07.770 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:07.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:07.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:07 smithi090 ceph-mon[33213]: 7.13 scrub starts 2024-02-05T07:24:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:07 smithi090 ceph-mon[33213]: 6.1a scrub starts 2024-02-05T07:24:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:07 smithi090 ceph-mon[33213]: 6.1a scrub ok 2024-02-05T07:24:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:07 smithi090 ceph-mon[33213]: 7.13 scrub ok 2024-02-05T07:24:07.778 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:24:07.824 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:24:07.828 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:24:07.828 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:24:07.829 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:24:07.829 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:24:07.829 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:24:07.829 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:24:07.829 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:24:07.829 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:24:07.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:07.914 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:07.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:07.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:08.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:08.052 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:08.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:08.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:08.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:08.199 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:08.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:08.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:08.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:08.341 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:08.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:08.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:08.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:08 smithi007 ceph-mon[37158]: pgmap v230: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.0 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 769 B/s rd, 0 op/s; 2/96891 objects degraded (0.002%); 3225/96891 objects misplaced (3.328%) 2024-02-05T07:24:08.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:08 smithi007 ceph-mon[37158]: 4.17 scrub starts 2024-02-05T07:24:08.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:08 smithi007 ceph-mon[37158]: 4.17 scrub ok 2024-02-05T07:24:08.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:08 smithi007 ceph-mon[37158]: 3.c scrub starts 2024-02-05T07:24:08.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:08 smithi007 ceph-mon[37158]: 3.c scrub ok 2024-02-05T07:24:08.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:08 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1239978019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:08.469 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:08 smithi007 ceph-mon[29098]: pgmap v230: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.0 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 769 B/s rd, 0 op/s; 2/96891 objects degraded (0.002%); 3225/96891 objects misplaced (3.328%) 2024-02-05T07:24:08.469 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:08 smithi007 ceph-mon[29098]: 4.17 scrub starts 2024-02-05T07:24:08.469 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:08 smithi007 ceph-mon[29098]: 4.17 scrub ok 2024-02-05T07:24:08.469 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:08 smithi007 ceph-mon[29098]: 3.c scrub starts 2024-02-05T07:24:08.469 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:08 smithi007 ceph-mon[29098]: 3.c scrub ok 2024-02-05T07:24:08.469 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:08 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1239978019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:08.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:08.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:08.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:08.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:08.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:08.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:08.715 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:08.749 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:08.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:08.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:08 smithi090 ceph-mon[33213]: pgmap v230: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.0 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 769 B/s rd, 0 op/s; 2/96891 objects degraded (0.002%); 3225/96891 objects misplaced (3.328%) 2024-02-05T07:24:08.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:08 smithi090 ceph-mon[33213]: 4.17 scrub starts 2024-02-05T07:24:08.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:08 smithi090 ceph-mon[33213]: 4.17 scrub ok 2024-02-05T07:24:08.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:08 smithi090 ceph-mon[33213]: 3.c scrub starts 2024-02-05T07:24:08.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:08 smithi090 ceph-mon[33213]: 3.c scrub ok 2024-02-05T07:24:08.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:08 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1239978019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:08.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:24:08.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:08.875 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:08.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:08.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:08.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:09.002 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:09.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:09.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:09.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:09.149 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:09.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:09.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:09.292 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:09.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:09.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:09.394 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:09.423 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:09.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:09.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:09.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:09.549 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:09.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:09.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:09.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:09 smithi007 ceph-mon[29098]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:24:09.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:09 smithi007 ceph-mon[29098]: 6.c scrub starts 2024-02-05T07:24:09.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:09 smithi007 ceph-mon[29098]: 6.c scrub ok 2024-02-05T07:24:09.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:09 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:24:09.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:09 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:24:09.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:09 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 2/96891 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:24:09.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:09 smithi007 ceph-mon[37158]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:24:09.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:09 smithi007 ceph-mon[37158]: 6.c scrub starts 2024-02-05T07:24:09.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:09 smithi007 ceph-mon[37158]: 6.c scrub ok 2024-02-05T07:24:09.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:09 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:24:09.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:09 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:24:09.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:09 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 2/96891 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:24:09.686 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:09.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:09.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:09 smithi090 ceph-mon[33213]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:24:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:09 smithi090 ceph-mon[33213]: 6.c scrub starts 2024-02-05T07:24:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:09 smithi090 ceph-mon[33213]: 6.c scrub ok 2024-02-05T07:24:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:09 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:24:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:09 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:24:09.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:09 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 2/96891 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:24:09.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:09.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:09.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:09.943 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-02-05T07:24:07.014151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2104170868,"num_objects":32297,"num_object_clones":0,"num_object_copies":96891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":3225,"num_objects_unfound":0,"num_objects_dirty":32297,"num_whiteouts":0,"num_read":1237,"num_read_kb":1083,"num_write":113201,"num_write_kb":3511966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16913,"num_bytes_recovered":1102516157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138799,"ondisk_log_size":138799,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":482,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7768228,"kb_used_data":6560044,"kb_used_omap":10,"kb_used_meta":1207989,"kb_avail":648447836,"statfs":{"total":671965249536,"available":664010584064,"internally_reserved":0,"allocated":6717485056,"data_stored":6714886774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11159,"internal_metadata":1236980841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,2,2,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-268173312,"num_objects":-4093,"num_object_clones":0,"num_object_copies":-12279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3631,"num_objects_unfound":0,"num_objects_dirty":-4093,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-28773,"num_write_kb":-1252608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-12226,"num_bytes_recovered":-801243136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.652081"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":218,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.547965+0000","last_change":"2024-02-05T07:20:25.913055+0000","last_active":"2024-02-05T07:24:02.547965+0000","last_peered":"2024-02-05T07:24:02.547965+0000","last_clean":"2024-02-05T07:24:02.547965+0000","last_became_active":"2024-02-05T07:18:55.466001+0000","last_became_peered":"2024-02-05T07:18:55.466001+0000","last_unstale":"2024-02-05T07:24:02.547965+0000","last_undegraded":"2024-02-05T07:24:02.547965+0000","last_fullsized":"2024-02-05T07:24:02.547965+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:25.912938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:25.912938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":231,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.382090+0000","last_change":"2024-02-05T07:23:07.907068+0000","last_active":"2024-02-05T07:24:02.382090+0000","last_peered":"2024-02-05T07:24:02.382090+0000","last_clean":"2024-02-05T07:24:02.382090+0000","last_became_active":"2024-02-05T07:18:53.460820+0000","last_became_peered":"2024-02-05T07:18:53.460820+0000","last_unstale":"2024-02-05T07:24:02.382090+0000","last_undegraded":"2024-02-05T07:24:02.382090+0000","last_fullsized":"2024-02-05T07:24:02.382090+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:07.906963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:23:07.906963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:31.542740+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.549235+0000","last_change":"2024-02-05T07:20:35.243411+0000","last_active":"2024-02-05T07:24:02.549235+0000","last_peered":"2024-02-05T07:24:02.549235+0000","last_clean":"2024-02-05T07:24:02.549235+0000","last_became_active":"2024-02-05T07:18:49.434456+0000","last_became_peered":"2024-02-05T07:18:49.434456+0000","last_unstale":"2024-02-05T07:24:02.549235+0000","last_undegraded":"2024-02-05T07:24:02.549235+0000","last_fullsized":"2024-02-05T07:24:02.549235+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:35.243281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:35.243281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":250,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.547681+0000","last_change":"2024-02-05T07:23:15.477701+0000","last_active":"2024-02-05T07:24:02.547681+0000","last_peered":"2024-02-05T07:24:02.547681+0000","last_clean":"2024-02-05T07:24:02.547681+0000","last_became_active":"2024-02-05T07:18:51.448924+0000","last_became_peered":"2024-02-05T07:18:51.448924+0000","last_unstale":"2024-02-05T07:24:02.547681+0000","last_undegraded":"2024-02-05T07:24:02.547681+0000","last_fullsized":"2024-02-05T07:24:02.547681+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:23:15.477555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:15.477555+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:41.604865+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":32,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.382067+0000","last_change":"2024-02-05T07:23:47.610351+0000","last_active":"2024-02-05T07:24:02.382067+0000","last_peered":"2024-02-05T07:24:02.382067+0000","last_clean":"2024-02-05T07:24:02.382067+0000","last_became_active":"2024-02-05T07:23:47.609992+0000","last_became_peered":"2024-02-05T07:23:47.609992+0000","last_unstale":"2024-02-05T07:24:02.382067+0000","last_undegraded":"2024-02-05T07:24:02.382067+0000","last_fullsized":"2024-02-05T07:24:02.382067+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.547477+0000","last_change":"2024-02-05T07:21:31.904690+0000","last_active":"2024-02-05T07:24:02.547477+0000","last_peered":"2024-02-05T07:24:02.547477+0000","last_clean":"2024-02-05T07:24:02.547477+0000","last_became_active":"2024-02-05T07:18:53.461506+0000","last_became_peered":"2024-02-05T07:18:53.461506+0000","last_unstale":"2024-02-05T07:24:02.547477+0000","last_undegraded":"2024-02-05T07:24:02.547477+0000","last_fullsized":"2024-02-05T07:24:02.547477+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:31.904568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:31.904568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":32,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.514361+0000","last_change":"2024-02-05T07:23:47.611056+0000","last_active":"2024-02-05T07:24:02.514361+0000","last_peered":"2024-02-05T07:24:02.514361+0000","last_clean":"2024-02-05T07:24:02.514361+0000","last_became_active":"2024-02-05T07:23:47.610737+0000","last_became_peered":"2024-02-05T07:23:47.610737+0000","last_unstale":"2024-02-05T07:24:02.514361+0000","last_undegraded":"2024-02-05T07:24:02.514361+0000","last_fullsized":"2024-02-05T07:24:02.514361+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:39.925994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:39.925994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":92,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.381657+0000","last_change":"2024-02-05T07:23:47.645884+0000","last_active":"2024-02-05T07:24:02.381657+0000","last_peered":"2024-02-05T07:24:02.381657+0000","last_clean":"2024-02-05T07:24:02.381657+0000","last_became_active":"2024-02-05T07:23:47.621081+0000","last_became_peered":"2024-02-05T07:23:47.621081+0000","last_unstale":"2024-02-05T07:24:02.381657+0000","last_undegraded":"2024-02-05T07:24:02.381657+0000","last_fullsized":"2024-02-05T07:24:02.381657+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:21:27.141800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:27.141800+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.19","version":"173'7758","reported_seq":17225,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:04.939932+0000","last_change":"2024-02-05T07:24:02.315647+0000","last_active":"2024-02-05T07:24:04.939932+0000","last_peered":"2024-02-05T07:24:04.939932+0000","last_clean":"2024-02-05T07:24:04.939932+0000","last_became_active":"2024-02-05T07:23:38.523311+0000","last_became_peered":"2024-02-05T07:23:38.523311+0000","last_unstale":"2024-02-05T07:24:04.939932+0000","last_undegraded":"2024-02-05T07:24:04.939932+0000","last_fullsized":"2024-02-05T07:24:04.939932+0000","mapping_epoch":155,"log_start":"108'1891","ondisk_log_start":"108'1891","created":118,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"172'7705","last_scrub_stamp":"2024-02-05T07:24:02.315408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:02.315408+0000","objects_scrubbed":1054,"log_size":5867,"log_dups_size":0,"ondisk_log_size":5867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:05.625286+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65601536,"num_objects":1001,"num_object_clones":0,"num_object_copies":3003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":218,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.547739+0000","last_change":"2024-02-05T07:20:31.252764+0000","last_active":"2024-02-05T07:24:02.547739+0000","last_peered":"2024-02-05T07:24:02.547739+0000","last_clean":"2024-02-05T07:24:02.547739+0000","last_became_active":"2024-02-05T07:18:55.474002+0000","last_became_peered":"2024-02-05T07:18:55.474002+0000","last_unstale":"2024-02-05T07:24:02.547739+0000","last_undegraded":"2024-02-05T07:24:02.547739+0000","last_fullsized":"2024-02-05T07:24:02.547739+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:31.252686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:31.252686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":222,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513609+0000","last_change":"2024-02-05T07:21:19.716500+0000","last_active":"2024-02-05T07:24:02.513609+0000","last_peered":"2024-02-05T07:24:02.513609+0000","last_clean":"2024-02-05T07:24:02.513609+0000","last_became_active":"2024-02-05T07:18:53.464520+0000","last_became_peered":"2024-02-05T07:18:53.464520+0000","last_unstale":"2024-02-05T07:24:02.513609+0000","last_undegraded":"2024-02-05T07:24:02.513609+0000","last_fullsized":"2024-02-05T07:24:02.513609+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:19.716375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:19.716375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":52,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.549161+0000","last_change":"2024-02-05T07:23:27.418016+0000","last_active":"2024-02-05T07:24:02.549161+0000","last_peered":"2024-02-05T07:24:02.549161+0000","last_clean":"2024-02-05T07:24:02.549161+0000","last_became_active":"2024-02-05T07:23:27.417717+0000","last_became_peered":"2024-02-05T07:23:27.417717+0000","last_unstale":"2024-02-05T07:24:02.549161+0000","last_undegraded":"2024-02-05T07:24:02.549161+0000","last_fullsized":"2024-02-05T07:24:02.549161+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:47.776379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:47.776379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"89'13","reported_seq":270,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548187+0000","last_change":"2024-02-05T07:23:47.643672+0000","last_active":"2024-02-05T07:24:02.548187+0000","last_peered":"2024-02-05T07:24:02.548187+0000","last_clean":"2024-02-05T07:24:02.548187+0000","last_became_active":"2024-02-05T07:23:47.618422+0000","last_became_peered":"2024-02-05T07:23:47.618422+0000","last_unstale":"2024-02-05T07:24:02.548187+0000","last_undegraded":"2024-02-05T07:24:02.548187+0000","last_fullsized":"2024-02-05T07:24:02.548187+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-05T07:21:34.888266+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:21:34.888266+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"173'7773","reported_seq":14988,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:05.618381+0000","last_change":"2024-02-05T07:24:05.618381+0000","last_active":"2024-02-05T07:24:05.618381+0000","last_peered":"2024-02-05T07:24:05.618381+0000","last_clean":"2024-02-05T07:24:05.618381+0000","last_became_active":"2024-02-05T07:22:48.518160+0000","last_became_peered":"2024-02-05T07:22:48.518160+0000","last_unstale":"2024-02-05T07:24:05.618381+0000","last_undegraded":"2024-02-05T07:24:05.618381+0000","last_fullsized":"2024-02-05T07:24:05.618381+0000","mapping_epoch":140,"log_start":"108'2083","ondisk_log_start":"108'2083","created":118,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":5,"last_scrub":"173'7772","last_scrub_stamp":"2024-02-05T07:24:05.618199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:05.618199+0000","objects_scrubbed":977,"log_size":5690,"log_dups_size":0,"ondisk_log_size":5690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:08.952813+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63373312,"num_objects":967,"num_object_clones":0,"num_object_copies":2901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":224,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384219+0000","last_change":"2024-02-05T07:23:47.597763+0000","last_active":"2024-02-05T07:24:02.384219+0000","last_peered":"2024-02-05T07:24:02.384219+0000","last_clean":"2024-02-05T07:24:02.384219+0000","last_became_active":"2024-02-05T07:23:47.596885+0000","last_became_peered":"2024-02-05T07:23:47.596885+0000","last_unstale":"2024-02-05T07:24:02.384219+0000","last_undegraded":"2024-02-05T07:24:02.384219+0000","last_fullsized":"2024-02-05T07:24:02.384219+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:15.751084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:15.751084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":222,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.382045+0000","last_change":"2024-02-05T07:20:23.756522+0000","last_active":"2024-02-05T07:24:02.382045+0000","last_peered":"2024-02-05T07:24:02.382045+0000","last_clean":"2024-02-05T07:24:02.382045+0000","last_became_active":"2024-02-05T07:18:53.456881+0000","last_became_peered":"2024-02-05T07:18:53.456881+0000","last_unstale":"2024-02-05T07:24:02.382045+0000","last_undegraded":"2024-02-05T07:24:02.382045+0000","last_fullsized":"2024-02-05T07:24:02.382045+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:23.756416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:23.756416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":52,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548870+0000","last_change":"2024-02-05T07:23:27.438105+0000","last_active":"2024-02-05T07:24:02.548870+0000","last_peered":"2024-02-05T07:24:02.548870+0000","last_clean":"2024-02-05T07:24:02.548870+0000","last_became_active":"2024-02-05T07:23:27.437776+0000","last_became_peered":"2024-02-05T07:23:27.437776+0000","last_unstale":"2024-02-05T07:24:02.548870+0000","last_undegraded":"2024-02-05T07:24:02.548870+0000","last_fullsized":"2024-02-05T07:24:02.548870+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:39.936643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.936643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":248,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548531+0000","last_change":"2024-02-05T07:23:47.616444+0000","last_active":"2024-02-05T07:24:02.548531+0000","last_peered":"2024-02-05T07:24:02.548531+0000","last_clean":"2024-02-05T07:24:02.548531+0000","last_became_active":"2024-02-05T07:23:47.603918+0000","last_became_peered":"2024-02-05T07:23:47.603918+0000","last_unstale":"2024-02-05T07:24:02.548531+0000","last_undegraded":"2024-02-05T07:24:02.548531+0000","last_fullsized":"2024-02-05T07:24:02.548531+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:21:15.028573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:15.028573+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":226,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.549034+0000","last_change":"2024-02-05T07:21:38.839453+0000","last_active":"2024-02-05T07:24:02.549034+0000","last_peered":"2024-02-05T07:24:02.549034+0000","last_clean":"2024-02-05T07:24:02.549034+0000","last_became_active":"2024-02-05T07:18:55.505678+0000","last_became_peered":"2024-02-05T07:18:55.505678+0000","last_unstale":"2024-02-05T07:24:02.549034+0000","last_undegraded":"2024-02-05T07:24:02.549034+0000","last_fullsized":"2024-02-05T07:24:02.549034+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:38.839336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:38.839336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":231,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384446+0000","last_change":"2024-02-05T07:22:58.849105+0000","last_active":"2024-02-05T07:24:02.384446+0000","last_peered":"2024-02-05T07:24:02.384446+0000","last_clean":"2024-02-05T07:24:02.384446+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:24:02.384446+0000","last_undegraded":"2024-02-05T07:24:02.384446+0000","last_fullsized":"2024-02-05T07:24:02.384446+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:22:58.846677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:22:58.846677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:16.633924+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548294+0000","last_change":"2024-02-05T07:21:32.192559+0000","last_active":"2024-02-05T07:24:02.548294+0000","last_peered":"2024-02-05T07:24:02.548294+0000","last_clean":"2024-02-05T07:24:02.548294+0000","last_became_active":"2024-02-05T07:18:49.432449+0000","last_became_peered":"2024-02-05T07:18:49.432449+0000","last_unstale":"2024-02-05T07:24:02.548294+0000","last_undegraded":"2024-02-05T07:24:02.548294+0000","last_fullsized":"2024-02-05T07:24:02.548294+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:32.192372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:32.192372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":281,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.383410+0000","last_change":"2024-02-05T07:23:47.657951+0000","last_active":"2024-02-05T07:24:02.383410+0000","last_peered":"2024-02-05T07:24:02.383410+0000","last_clean":"2024-02-05T07:24:02.383410+0000","last_became_active":"2024-02-05T07:23:47.619716+0000","last_became_peered":"2024-02-05T07:23:47.619716+0000","last_unstale":"2024-02-05T07:24:02.383410+0000","last_undegraded":"2024-02-05T07:24:02.383410+0000","last_fullsized":"2024-02-05T07:24:02.383410+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:21:36.561503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:36.561503+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":32,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.514742+0000","last_change":"2024-02-05T07:23:47.610492+0000","last_active":"2024-02-05T07:24:02.514742+0000","last_peered":"2024-02-05T07:24:02.514742+0000","last_clean":"2024-02-05T07:24:02.514742+0000","last_became_active":"2024-02-05T07:23:47.609917+0000","last_became_peered":"2024-02-05T07:23:47.609917+0000","last_unstale":"2024-02-05T07:24:02.514742+0000","last_undegraded":"2024-02-05T07:24:02.514742+0000","last_fullsized":"2024-02-05T07:24:02.514742+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:31.211338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:31.211338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":236,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.549173+0000","last_change":"2024-02-05T07:23:47.606077+0000","last_active":"2024-02-05T07:24:02.549173+0000","last_peered":"2024-02-05T07:24:02.549173+0000","last_clean":"2024-02-05T07:24:02.549173+0000","last_became_active":"2024-02-05T07:23:47.605523+0000","last_became_peered":"2024-02-05T07:23:47.605523+0000","last_unstale":"2024-02-05T07:24:02.549173+0000","last_undegraded":"2024-02-05T07:24:02.549173+0000","last_fullsized":"2024-02-05T07:24:02.549173+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:41.800067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:41.800067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548273+0000","last_change":"2024-02-05T07:20:39.898505+0000","last_active":"2024-02-05T07:24:02.548273+0000","last_peered":"2024-02-05T07:24:02.548273+0000","last_clean":"2024-02-05T07:24:02.548273+0000","last_became_active":"2024-02-05T07:18:49.427837+0000","last_became_peered":"2024-02-05T07:18:49.427837+0000","last_unstale":"2024-02-05T07:24:02.548273+0000","last_undegraded":"2024-02-05T07:24:02.548273+0000","last_fullsized":"2024-02-05T07:24:02.548273+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:39.898367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:39.898367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":86,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384380+0000","last_change":"2024-02-05T07:23:47.648580+0000","last_active":"2024-02-05T07:24:02.384380+0000","last_peered":"2024-02-05T07:24:02.384380+0000","last_clean":"2024-02-05T07:24:02.384380+0000","last_became_active":"2024-02-05T07:23:47.610303+0000","last_became_peered":"2024-02-05T07:23:47.610303+0000","last_unstale":"2024-02-05T07:24:02.384380+0000","last_undegraded":"2024-02-05T07:24:02.384380+0000","last_fullsized":"2024-02-05T07:24:02.384380+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:01.841489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:01.841489+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":228,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513331+0000","last_change":"2024-02-05T07:21:39.490602+0000","last_active":"2024-02-05T07:24:02.513331+0000","last_peered":"2024-02-05T07:24:02.513331+0000","last_clean":"2024-02-05T07:24:02.513331+0000","last_became_active":"2024-02-05T07:18:55.476071+0000","last_became_peered":"2024-02-05T07:18:55.476071+0000","last_unstale":"2024-02-05T07:24:02.513331+0000","last_undegraded":"2024-02-05T07:24:02.513331+0000","last_fullsized":"2024-02-05T07:24:02.513331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.490479+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081510+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":305,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.514547+0000","last_change":"2024-02-05T07:21:39.441651+0000","last_active":"2024-02-05T07:24:02.514547+0000","last_peered":"2024-02-05T07:24:02.514547+0000","last_clean":"2024-02-05T07:24:02.514547+0000","last_became_active":"2024-02-05T07:18:53.461011+0000","last_became_peered":"2024-02-05T07:18:53.461011+0000","last_unstale":"2024-02-05T07:24:02.514547+0000","last_undegraded":"2024-02-05T07:24:02.514547+0000","last_fullsized":"2024-02-05T07:24:02.514547+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:21:39.441520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.441520+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":250,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.381990+0000","last_change":"2024-02-05T07:23:47.646441+0000","last_active":"2024-02-05T07:24:02.381990+0000","last_peered":"2024-02-05T07:24:02.381990+0000","last_clean":"2024-02-05T07:24:02.381990+0000","last_became_active":"2024-02-05T07:23:47.610436+0000","last_became_peered":"2024-02-05T07:23:47.610436+0000","last_unstale":"2024-02-05T07:24:02.381990+0000","last_undegraded":"2024-02-05T07:24:02.381990+0000","last_fullsized":"2024-02-05T07:24:02.381990+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:21:42.616129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:42.616129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":259,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.549086+0000","last_change":"2024-02-05T07:23:47.654786+0000","last_active":"2024-02-05T07:24:02.549086+0000","last_peered":"2024-02-05T07:24:02.549086+0000","last_clean":"2024-02-05T07:24:02.549086+0000","last_became_active":"2024-02-05T07:23:47.635619+0000","last_became_peered":"2024-02-05T07:23:47.635619+0000","last_unstale":"2024-02-05T07:24:02.549086+0000","last_undegraded":"2024-02-05T07:24:02.549086+0000","last_fullsized":"2024-02-05T07:24:02.549086+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:20:22.291371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:22.291371+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":235,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384601+0000","last_change":"2024-02-05T07:22:55.794256+0000","last_active":"2024-02-05T07:24:02.384601+0000","last_peered":"2024-02-05T07:24:02.384601+0000","last_clean":"2024-02-05T07:24:02.384601+0000","last_became_active":"2024-02-05T07:18:55.466164+0000","last_became_peered":"2024-02-05T07:18:55.466164+0000","last_unstale":"2024-02-05T07:24:02.384601+0000","last_undegraded":"2024-02-05T07:24:02.384601+0000","last_fullsized":"2024-02-05T07:24:02.384601+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:22:55.794149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:22:55.794149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":222,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.549278+0000","last_change":"2024-02-05T07:21:26.202262+0000","last_active":"2024-02-05T07:24:02.549278+0000","last_peered":"2024-02-05T07:24:02.549278+0000","last_clean":"2024-02-05T07:24:02.549278+0000","last_became_active":"2024-02-05T07:18:53.457587+0000","last_became_peered":"2024-02-05T07:18:53.457587+0000","last_unstale":"2024-02-05T07:24:02.549278+0000","last_undegraded":"2024-02-05T07:24:02.549278+0000","last_fullsized":"2024-02-05T07:24:02.549278+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:26.202131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:26.202131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":214,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.547420+0000","last_change":"2024-02-05T07:23:47.634669+0000","last_active":"2024-02-05T07:24:02.547420+0000","last_peered":"2024-02-05T07:24:02.547420+0000","last_clean":"2024-02-05T07:24:02.547420+0000","last_became_active":"2024-02-05T07:23:47.634347+0000","last_became_peered":"2024-02-05T07:23:47.634347+0000","last_unstale":"2024-02-05T07:24:02.547420+0000","last_undegraded":"2024-02-05T07:24:02.547420+0000","last_fullsized":"2024-02-05T07:24:02.547420+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:14.255462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:14.255462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":253,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.514597+0000","last_change":"2024-02-05T07:21:30.382672+0000","last_active":"2024-02-05T07:24:02.514597+0000","last_peered":"2024-02-05T07:24:02.514597+0000","last_clean":"2024-02-05T07:24:02.514597+0000","last_became_active":"2024-02-05T07:18:51.446604+0000","last_became_peered":"2024-02-05T07:18:51.446604+0000","last_unstale":"2024-02-05T07:24:02.514597+0000","last_undegraded":"2024-02-05T07:24:02.514597+0000","last_fullsized":"2024-02-05T07:24:02.514597+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:21:30.382584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:30.382584+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.13","version":"173'7924","reported_seq":15221,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:05.674808+0000","last_change":"2024-02-05T07:22:48.526749+0000","last_active":"2024-02-05T07:24:05.674808+0000","last_peered":"2024-02-05T07:24:05.674808+0000","last_clean":"2024-02-05T07:24:05.674808+0000","last_became_active":"2024-02-05T07:22:48.517185+0000","last_became_peered":"2024-02-05T07:22:48.517185+0000","last_unstale":"2024-02-05T07:24:05.674808+0000","last_undegraded":"2024-02-05T07:24:05.674808+0000","last_fullsized":"2024-02-05T07:24:05.674808+0000","mapping_epoch":140,"log_start":"109'2391","ondisk_log_start":"109'2391","created":118,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":5533,"log_dups_size":0,"ondisk_log_size":5533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67174400,"num_objects":1025,"num_object_clones":0,"num_object_copies":3075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5593,"num_write_kb":211776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2375,"num_bytes_recovered":155648000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":226,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.382051+0000","last_change":"2024-02-05T07:21:30.473743+0000","last_active":"2024-02-05T07:24:02.382051+0000","last_peered":"2024-02-05T07:24:02.382051+0000","last_clean":"2024-02-05T07:24:02.382051+0000","last_became_active":"2024-02-05T07:18:55.468086+0000","last_became_peered":"2024-02-05T07:18:55.468086+0000","last_unstale":"2024-02-05T07:24:02.382051+0000","last_undegraded":"2024-02-05T07:24:02.382051+0000","last_fullsized":"2024-02-05T07:24:02.382051+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:30.473582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:30.473582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081598+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":222,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384783+0000","last_change":"2024-02-05T07:20:09.349490+0000","last_active":"2024-02-05T07:24:02.384783+0000","last_peered":"2024-02-05T07:24:02.384783+0000","last_clean":"2024-02-05T07:24:02.384783+0000","last_became_active":"2024-02-05T07:18:53.460307+0000","last_became_peered":"2024-02-05T07:18:53.460307+0000","last_unstale":"2024-02-05T07:24:02.384783+0000","last_undegraded":"2024-02-05T07:24:02.384783+0000","last_fullsized":"2024-02-05T07:24:02.384783+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:20:09.349342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":236,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.549423+0000","last_change":"2024-02-05T07:23:47.627596+0000","last_active":"2024-02-05T07:24:02.549423+0000","last_peered":"2024-02-05T07:24:02.549423+0000","last_clean":"2024-02-05T07:24:02.549423+0000","last_became_active":"2024-02-05T07:23:47.625599+0000","last_became_peered":"2024-02-05T07:23:47.625599+0000","last_unstale":"2024-02-05T07:24:02.549423+0000","last_undegraded":"2024-02-05T07:24:02.549423+0000","last_fullsized":"2024-02-05T07:24:02.549423+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:19.326400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:19.326400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":63,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.382300+0000","last_change":"2024-02-05T07:23:47.645600+0000","last_active":"2024-02-05T07:24:02.382300+0000","last_peered":"2024-02-05T07:24:02.382300+0000","last_clean":"2024-02-05T07:24:02.382300+0000","last_became_active":"2024-02-05T07:23:47.610110+0000","last_became_peered":"2024-02-05T07:23:47.610110+0000","last_unstale":"2024-02-05T07:24:02.382300+0000","last_undegraded":"2024-02-05T07:24:02.382300+0000","last_fullsized":"2024-02-05T07:24:02.382300+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-02-05T07:21:20.184048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:20.184048+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.12","version":"173'7886","reported_seq":15207,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:04.872939+0000","last_change":"2024-02-05T07:24:03.700312+0000","last_active":"2024-02-05T07:24:04.872939+0000","last_peered":"2024-02-05T07:24:04.872939+0000","last_clean":"2024-02-05T07:24:04.872939+0000","last_became_active":"2024-02-05T07:22:37.417198+0000","last_became_peered":"2024-02-05T07:22:37.417198+0000","last_unstale":"2024-02-05T07:24:04.872939+0000","last_undegraded":"2024-02-05T07:24:04.872939+0000","last_fullsized":"2024-02-05T07:24:04.872939+0000","mapping_epoch":134,"log_start":"108'2207","ondisk_log_start":"108'2207","created":118,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"173'7850","last_scrub_stamp":"2024-02-05T07:24:03.700068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:03.700068+0000","objects_scrubbed":1049,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:23.547229+0000","scrub_duration":517,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65929216,"num_objects":1006,"num_object_clones":0,"num_object_copies":3018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":32,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.547732+0000","last_change":"2024-02-05T07:23:47.618375+0000","last_active":"2024-02-05T07:24:02.547732+0000","last_peered":"2024-02-05T07:24:02.547732+0000","last_clean":"2024-02-05T07:24:02.547732+0000","last_became_active":"2024-02-05T07:23:47.618046+0000","last_became_peered":"2024-02-05T07:23:47.618046+0000","last_unstale":"2024-02-05T07:24:02.547732+0000","last_undegraded":"2024-02-05T07:24:02.547732+0000","last_fullsized":"2024-02-05T07:24:02.547732+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:22.138332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:22.138332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.382128+0000","last_change":"2024-02-05T07:21:24.573753+0000","last_active":"2024-02-05T07:24:02.382128+0000","last_peered":"2024-02-05T07:24:02.382128+0000","last_clean":"2024-02-05T07:24:02.382128+0000","last_became_active":"2024-02-05T07:18:53.460977+0000","last_became_peered":"2024-02-05T07:18:53.460977+0000","last_unstale":"2024-02-05T07:24:02.382128+0000","last_undegraded":"2024-02-05T07:24:02.382128+0000","last_fullsized":"2024-02-05T07:24:02.382128+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:24.573655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_clean_scrub_stamp":"2024-02-05T07:21:24.573655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":238,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513777+0000","last_change":"2024-02-05T07:21:29.818030+0000","last_active":"2024-02-05T07:24:02.513777+0000","last_peered":"2024-02-05T07:24:02.513777+0000","last_clean":"2024-02-05T07:24:02.513777+0000","last_became_active":"2024-02-05T07:18:49.434560+0000","last_became_peered":"2024-02-05T07:18:49.434560+0000","last_unstale":"2024-02-05T07:24:02.513777+0000","last_undegraded":"2024-02-05T07:24:02.513777+0000","last_fullsized":"2024-02-05T07:24:02.513777+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:29.817921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:29.817921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":82,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513549+0000","last_change":"2024-02-05T07:23:47.638791+0000","last_active":"2024-02-05T07:24:02.513549+0000","last_peered":"2024-02-05T07:24:02.513549+0000","last_clean":"2024-02-05T07:24:02.513549+0000","last_became_active":"2024-02-05T07:23:47.613111+0000","last_became_peered":"2024-02-05T07:23:47.613111+0000","last_unstale":"2024-02-05T07:24:02.513549+0000","last_undegraded":"2024-02-05T07:24:02.513549+0000","last_fullsized":"2024-02-05T07:24:02.513549+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:21:18.095797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:18.095797+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.11","version":"173'7811","reported_seq":15111,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:04.929293+0000","last_change":"2024-02-05T07:24:04.484761+0000","last_active":"2024-02-05T07:24:04.929293+0000","last_peered":"2024-02-05T07:24:04.929293+0000","last_clean":"2024-02-05T07:24:04.929293+0000","last_became_active":"2024-02-05T07:23:58.360798+0000","last_became_peered":"2024-02-05T07:23:58.360798+0000","last_unstale":"2024-02-05T07:24:04.929293+0000","last_undegraded":"2024-02-05T07:24:04.929293+0000","last_fullsized":"2024-02-05T07:24:04.929293+0000","mapping_epoch":168,"log_start":"109'2519","ondisk_log_start":"109'2519","created":118,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":5,"last_scrub":"173'7795","last_scrub_stamp":"2024-02-05T07:24:04.484558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:04.484558+0000","objects_scrubbed":1044,"log_size":5292,"log_dups_size":0,"ondisk_log_size":5292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:17.884783+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67174400,"num_objects":1025,"num_object_clones":0,"num_object_copies":3075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":218,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.547566+0000","last_change":"2024-02-05T07:20:27.243266+0000","last_active":"2024-02-05T07:24:02.547566+0000","last_peered":"2024-02-05T07:24:02.547566+0000","last_clean":"2024-02-05T07:24:02.547566+0000","last_became_active":"2024-02-05T07:18:55.477312+0000","last_became_peered":"2024-02-05T07:18:55.477312+0000","last_unstale":"2024-02-05T07:24:02.547566+0000","last_undegraded":"2024-02-05T07:24:02.547566+0000","last_fullsized":"2024-02-05T07:24:02.547566+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:27.243136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:27.243136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":40,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548044+0000","last_change":"2024-02-05T07:24:02.051866+0000","last_active":"2024-02-05T07:24:02.548044+0000","last_peered":"2024-02-05T07:24:02.548044+0000","last_clean":"2024-02-05T07:24:02.548044+0000","last_became_active":"2024-02-05T07:23:47.622888+0000","last_became_peered":"2024-02-05T07:23:47.622888+0000","last_unstale":"2024-02-05T07:24:02.548044+0000","last_undegraded":"2024-02-05T07:24:02.548044+0000","last_fullsized":"2024-02-05T07:24:02.548044+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:02.051772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:02.051772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:22.676039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":236,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.514942+0000","last_change":"2024-02-05T07:23:47.616365+0000","last_active":"2024-02-05T07:24:02.514942+0000","last_peered":"2024-02-05T07:24:02.514942+0000","last_clean":"2024-02-05T07:24:02.514942+0000","last_became_active":"2024-02-05T07:23:47.616029+0000","last_became_peered":"2024-02-05T07:23:47.616029+0000","last_unstale":"2024-02-05T07:24:02.514942+0000","last_undegraded":"2024-02-05T07:24:02.514942+0000","last_fullsized":"2024-02-05T07:24:02.514942+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:20.600583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:20.600583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081297+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":214,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384149+0000","last_change":"2024-02-05T07:21:16.721101+0000","last_active":"2024-02-05T07:24:02.384149+0000","last_peered":"2024-02-05T07:24:02.384149+0000","last_clean":"2024-02-05T07:24:02.384149+0000","last_became_active":"2024-02-05T07:19:16.122059+0000","last_became_peered":"2024-02-05T07:19:16.122059+0000","last_unstale":"2024-02-05T07:24:02.384149+0000","last_undegraded":"2024-02-05T07:24:02.384149+0000","last_fullsized":"2024-02-05T07:24:02.384149+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:16.720264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:16.720264+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.10","version":"173'7740","reported_seq":14996,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:05.446972+0000","last_change":"2024-02-05T07:23:58.379360+0000","last_active":"2024-02-05T07:24:05.446972+0000","last_peered":"2024-02-05T07:24:05.446972+0000","last_clean":"2024-02-05T07:24:05.446972+0000","last_became_active":"2024-02-05T07:23:58.357839+0000","last_became_peered":"2024-02-05T07:23:58.357839+0000","last_unstale":"2024-02-05T07:24:05.446972+0000","last_undegraded":"2024-02-05T07:24:05.446972+0000","last_fullsized":"2024-02-05T07:24:05.446972+0000","mapping_epoch":168,"log_start":"107'1514","ondisk_log_start":"107'1514","created":118,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":5,"last_scrub":"144'6154","last_scrub_stamp":"2024-02-05T07:22:55.148513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:22:55.148513+0000","objects_scrubbed":2599,"log_size":6226,"log_dups_size":0,"ondisk_log_size":6226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1080,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65208320,"num_objects":995,"num_object_clones":0,"num_object_copies":2985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1604,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1183,"num_bytes_recovered":77529088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":32,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.547741+0000","last_change":"2024-02-05T07:23:47.632859+0000","last_active":"2024-02-05T07:24:02.547741+0000","last_peered":"2024-02-05T07:24:02.547741+0000","last_clean":"2024-02-05T07:24:02.547741+0000","last_became_active":"2024-02-05T07:23:47.632438+0000","last_became_peered":"2024-02-05T07:23:47.632438+0000","last_unstale":"2024-02-05T07:24:02.547741+0000","last_undegraded":"2024-02-05T07:24:02.547741+0000","last_fullsized":"2024-02-05T07:24:02.547741+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:27.852942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:27.852942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":237,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.383373+0000","last_change":"2024-02-05T07:23:47.594349+0000","last_active":"2024-02-05T07:24:02.383373+0000","last_peered":"2024-02-05T07:24:02.383373+0000","last_clean":"2024-02-05T07:24:02.383373+0000","last_became_active":"2024-02-05T07:23:47.594030+0000","last_became_peered":"2024-02-05T07:23:47.594030+0000","last_unstale":"2024-02-05T07:24:02.383373+0000","last_undegraded":"2024-02-05T07:24:02.383373+0000","last_fullsized":"2024-02-05T07:24:02.383373+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:22:59.851438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:22:59.851438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:04.946335+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":238,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548953+0000","last_change":"2024-02-05T07:23:23.435405+0000","last_active":"2024-02-05T07:24:02.548953+0000","last_peered":"2024-02-05T07:24:02.548953+0000","last_clean":"2024-02-05T07:24:02.548953+0000","last_became_active":"2024-02-05T07:18:49.430766+0000","last_became_peered":"2024-02-05T07:18:49.430766+0000","last_unstale":"2024-02-05T07:24:02.548953+0000","last_undegraded":"2024-02-05T07:24:02.548953+0000","last_fullsized":"2024-02-05T07:24:02.548953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:23:23.435135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.864941+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":271,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548722+0000","last_change":"2024-02-05T07:21:37.876638+0000","last_active":"2024-02-05T07:24:02.548722+0000","last_peered":"2024-02-05T07:24:02.548722+0000","last_clean":"2024-02-05T07:24:02.548722+0000","last_became_active":"2024-02-05T07:19:16.121696+0000","last_became_peered":"2024-02-05T07:19:16.121696+0000","last_unstale":"2024-02-05T07:24:02.548722+0000","last_undegraded":"2024-02-05T07:24:02.548722+0000","last_fullsized":"2024-02-05T07:24:02.548722+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:21:37.876511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.876511+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"173'7789","reported_seq":17385,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:04.801057+0000","last_change":"2024-02-05T07:24:03.599184+0000","last_active":"2024-02-05T07:24:04.801057+0000","last_peered":"2024-02-05T07:24:04.801057+0000","last_clean":"2024-02-05T07:24:04.801057+0000","last_became_active":"2024-02-05T07:23:44.564445+0000","last_became_peered":"2024-02-05T07:23:44.564445+0000","last_unstale":"2024-02-05T07:24:04.801057+0000","last_undegraded":"2024-02-05T07:24:04.801057+0000","last_fullsized":"2024-02-05T07:24:04.801057+0000","mapping_epoch":159,"log_start":"109'2502","ondisk_log_start":"109'2502","created":118,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"173'7768","last_scrub_stamp":"2024-02-05T07:24:03.599036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:03.599036+0000","objects_scrubbed":1041,"log_size":5287,"log_dups_size":0,"ondisk_log_size":5287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:20.337171+0000","scrub_duration":468,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66584576,"num_objects":1016,"num_object_clones":0,"num_object_copies":3048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":224,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548761+0000","last_change":"2024-02-05T07:23:47.635013+0000","last_active":"2024-02-05T07:24:02.548761+0000","last_peered":"2024-02-05T07:24:02.548761+0000","last_clean":"2024-02-05T07:24:02.548761+0000","last_became_active":"2024-02-05T07:23:47.634690+0000","last_became_peered":"2024-02-05T07:23:47.634690+0000","last_unstale":"2024-02-05T07:24:02.548761+0000","last_undegraded":"2024-02-05T07:24:02.548761+0000","last_fullsized":"2024-02-05T07:24:02.548761+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:20.322309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:20.322309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":294,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513594+0000","last_change":"2024-02-05T07:20:38.003311+0000","last_active":"2024-02-05T07:24:02.513594+0000","last_peered":"2024-02-05T07:24:02.513594+0000","last_clean":"2024-02-05T07:24:02.513594+0000","last_became_active":"2024-02-05T07:18:53.465287+0000","last_became_peered":"2024-02-05T07:18:53.465287+0000","last_unstale":"2024-02-05T07:24:02.513594+0000","last_undegraded":"2024-02-05T07:24:02.513594+0000","last_fullsized":"2024-02-05T07:24:02.513594+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:20:38.003219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:38.003219+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.382175+0000","last_change":"2024-02-05T07:20:32.839729+0000","last_active":"2024-02-05T07:24:02.382175+0000","last_peered":"2024-02-05T07:24:02.382175+0000","last_clean":"2024-02-05T07:24:02.382175+0000","last_became_active":"2024-02-05T07:18:49.436131+0000","last_became_peered":"2024-02-05T07:18:49.436131+0000","last_unstale":"2024-02-05T07:24:02.382175+0000","last_undegraded":"2024-02-05T07:24:02.382175+0000","last_fullsized":"2024-02-05T07:24:02.382175+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:32.839614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:32.839614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"113'15","reported_seq":159,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384242+0000","last_change":"2024-02-05T07:23:47.640930+0000","last_active":"2024-02-05T07:24:02.384242+0000","last_peered":"2024-02-05T07:24:02.384242+0000","last_clean":"2024-02-05T07:24:02.384242+0000","last_became_active":"2024-02-05T07:23:47.608158+0000","last_became_peered":"2024-02-05T07:23:47.608158+0000","last_unstale":"2024-02-05T07:24:02.384242+0000","last_undegraded":"2024-02-05T07:24:02.384242+0000","last_fullsized":"2024-02-05T07:24:02.384242+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'15","last_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:21:36.359546+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"173'7996","reported_seq":13060,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:04.845611+0000","last_change":"2024-02-05T07:23:02.140812+0000","last_active":"2024-02-05T07:24:04.845611+0000","last_peered":"2024-02-05T07:24:04.845611+0000","last_clean":"2024-02-05T07:24:04.845611+0000","last_became_active":"2024-02-05T07:23:01.185149+0000","last_became_peered":"2024-02-05T07:23:01.185149+0000","last_unstale":"2024-02-05T07:24:04.845611+0000","last_undegraded":"2024-02-05T07:24:04.845611+0000","last_fullsized":"2024-02-05T07:24:04.845611+0000","mapping_epoch":146,"log_start":"109'2579","ondisk_log_start":"109'2579","created":118,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'6509","last_scrub_stamp":"2024-02-05T07:23:02.140598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:23:02.140598+0000","objects_scrubbed":2503,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:06.315047+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65863680,"num_objects":1005,"num_object_clones":0,"num_object_copies":3015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1498,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":218,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.549347+0000","last_change":"2024-02-05T07:20:33.860836+0000","last_active":"2024-02-05T07:24:02.549347+0000","last_peered":"2024-02-05T07:24:02.549347+0000","last_clean":"2024-02-05T07:24:02.549347+0000","last_became_active":"2024-02-05T07:18:55.467044+0000","last_became_peered":"2024-02-05T07:18:55.467044+0000","last_unstale":"2024-02-05T07:24:02.549347+0000","last_undegraded":"2024-02-05T07:24:02.549347+0000","last_fullsized":"2024-02-05T07:24:02.549347+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:33.860750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:33.860750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"164'9","reported_seq":65,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513282+0000","last_change":"2024-02-05T07:23:47.620866+0000","last_active":"2024-02-05T07:24:02.513282+0000","last_peered":"2024-02-05T07:24:02.513282+0000","last_clean":"2024-02-05T07:24:02.513282+0000","last_became_active":"2024-02-05T07:23:47.605296+0000","last_became_peered":"2024-02-05T07:23:47.605296+0000","last_unstale":"2024-02-05T07:24:02.513282+0000","last_undegraded":"2024-02-05T07:24:02.513282+0000","last_fullsized":"2024-02-05T07:24:02.513282+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:22:58.051782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:22:58.051782+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:18.433963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":236,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.547854+0000","last_change":"2024-02-05T07:23:47.623344+0000","last_active":"2024-02-05T07:24:02.547854+0000","last_peered":"2024-02-05T07:24:02.547854+0000","last_clean":"2024-02-05T07:24:02.547854+0000","last_became_active":"2024-02-05T07:23:47.623024+0000","last_became_peered":"2024-02-05T07:23:47.623024+0000","last_unstale":"2024-02-05T07:24:02.547854+0000","last_undegraded":"2024-02-05T07:24:02.547854+0000","last_fullsized":"2024-02-05T07:24:02.547854+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:33.270421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:33.270421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":266,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513671+0000","last_change":"2024-02-05T07:23:47.619736+0000","last_active":"2024-02-05T07:24:02.513671+0000","last_peered":"2024-02-05T07:24:02.513671+0000","last_clean":"2024-02-05T07:24:02.513671+0000","last_became_active":"2024-02-05T07:23:47.602628+0000","last_became_peered":"2024-02-05T07:23:47.602628+0000","last_unstale":"2024-02-05T07:24:02.513671+0000","last_undegraded":"2024-02-05T07:24:02.513671+0000","last_fullsized":"2024-02-05T07:24:02.513671+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:30.859017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:30.859017+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.15","version":"173'7958","reported_seq":12976,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:04.942141+0000","last_change":"2024-02-05T07:24:02.322881+0000","last_active":"2024-02-05T07:24:04.942141+0000","last_peered":"2024-02-05T07:24:04.942141+0000","last_clean":"2024-02-05T07:24:04.942141+0000","last_became_active":"2024-02-05T07:22:15.165819+0000","last_became_peered":"2024-02-05T07:22:15.165819+0000","last_unstale":"2024-02-05T07:24:04.942141+0000","last_undegraded":"2024-02-05T07:24:04.942141+0000","last_fullsized":"2024-02-05T07:24:04.942141+0000","mapping_epoch":126,"log_start":"110'2835","ondisk_log_start":"110'2835","created":118,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"172'7884","last_scrub_stamp":"2024-02-05T07:24:02.322788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:02.322788+0000","objects_scrubbed":1115,"log_size":5123,"log_dups_size":0,"ondisk_log_size":5123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:02.928527+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68222976,"num_objects":1041,"num_object_clones":0,"num_object_copies":3123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":226,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.383619+0000","last_change":"2024-02-05T07:21:39.515473+0000","last_active":"2024-02-05T07:24:02.383619+0000","last_peered":"2024-02-05T07:24:02.383619+0000","last_clean":"2024-02-05T07:24:02.383619+0000","last_became_active":"2024-02-05T07:18:55.469044+0000","last_became_peered":"2024-02-05T07:18:55.469044+0000","last_unstale":"2024-02-05T07:24:02.383619+0000","last_undegraded":"2024-02-05T07:24:02.383619+0000","last_fullsized":"2024-02-05T07:24:02.383619+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:39.515332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.515332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":32,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.383151+0000","last_change":"2024-02-05T07:23:47.606649+0000","last_active":"2024-02-05T07:24:02.383151+0000","last_peered":"2024-02-05T07:24:02.383151+0000","last_clean":"2024-02-05T07:24:02.383151+0000","last_became_active":"2024-02-05T07:23:47.606266+0000","last_became_peered":"2024-02-05T07:23:47.606266+0000","last_unstale":"2024-02-05T07:24:02.383151+0000","last_undegraded":"2024-02-05T07:24:02.383151+0000","last_fullsized":"2024-02-05T07:24:02.383151+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.813344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.813344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":238,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.382070+0000","last_change":"2024-02-05T07:21:25.529255+0000","last_active":"2024-02-05T07:24:02.382070+0000","last_peered":"2024-02-05T07:24:02.382070+0000","last_clean":"2024-02-05T07:24:02.382070+0000","last_became_active":"2024-02-05T07:18:49.437732+0000","last_became_peered":"2024-02-05T07:18:49.437732+0000","last_unstale":"2024-02-05T07:24:02.382070+0000","last_undegraded":"2024-02-05T07:24:02.382070+0000","last_fullsized":"2024-02-05T07:24:02.382070+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:25.529174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:21:25.529174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":258,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.549191+0000","last_change":"2024-02-05T07:23:47.644385+0000","last_active":"2024-02-05T07:24:02.549191+0000","last_peered":"2024-02-05T07:24:02.549191+0000","last_clean":"2024-02-05T07:24:02.549191+0000","last_became_active":"2024-02-05T07:23:47.633285+0000","last_became_peered":"2024-02-05T07:23:47.633285+0000","last_unstale":"2024-02-05T07:24:02.549191+0000","last_undegraded":"2024-02-05T07:24:02.549191+0000","last_fullsized":"2024-02-05T07:24:02.549191+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:33.244655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:33.244655+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"173'7940","reported_seq":17794,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:05.265761+0000","last_change":"2024-02-05T07:23:51.671341+0000","last_active":"2024-02-05T07:24:05.265761+0000","last_peered":"2024-02-05T07:24:05.265761+0000","last_clean":"2024-02-05T07:24:05.265761+0000","last_became_active":"2024-02-05T07:23:51.653282+0000","last_became_peered":"2024-02-05T07:23:51.653282+0000","last_unstale":"2024-02-05T07:24:05.265761+0000","last_undegraded":"2024-02-05T07:24:05.265761+0000","last_fullsized":"2024-02-05T07:24:05.265761+0000","mapping_epoch":165,"log_start":"108'1931","ondisk_log_start":"108'1931","created":118,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:07.863257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66125824,"num_objects":1009,"num_object_clones":0,"num_object_copies":3027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5617,"num_write_kb":212032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2428,"num_bytes_recovered":159121408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":226,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.514591+0000","last_change":"2024-02-05T07:21:44.435932+0000","last_active":"2024-02-05T07:24:02.514591+0000","last_peered":"2024-02-05T07:24:02.514591+0000","last_clean":"2024-02-05T07:24:02.514591+0000","last_became_active":"2024-02-05T07:18:55.507555+0000","last_became_peered":"2024-02-05T07:18:55.507555+0000","last_unstale":"2024-02-05T07:24:02.514591+0000","last_undegraded":"2024-02-05T07:24:02.514591+0000","last_fullsized":"2024-02-05T07:24:02.514591+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:21:44.435826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":228,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513688+0000","last_change":"2024-02-05T07:23:47.601964+0000","last_active":"2024-02-05T07:24:02.513688+0000","last_peered":"2024-02-05T07:24:02.513688+0000","last_clean":"2024-02-05T07:24:02.513688+0000","last_became_active":"2024-02-05T07:23:47.601556+0000","last_became_peered":"2024-02-05T07:23:47.601556+0000","last_unstale":"2024-02-05T07:24:02.513688+0000","last_undegraded":"2024-02-05T07:24:02.513688+0000","last_fullsized":"2024-02-05T07:24:02.513688+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:18.752135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:18.752135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":244,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384295+0000","last_change":"2024-02-05T07:21:25.389276+0000","last_active":"2024-02-05T07:24:02.384295+0000","last_peered":"2024-02-05T07:24:02.384295+0000","last_clean":"2024-02-05T07:24:02.384295+0000","last_became_active":"2024-02-05T07:18:49.433121+0000","last_became_peered":"2024-02-05T07:18:49.433121+0000","last_unstale":"2024-02-05T07:24:02.384295+0000","last_undegraded":"2024-02-05T07:24:02.384295+0000","last_fullsized":"2024-02-05T07:24:02.384295+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:21:25.389138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:25.389138+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":68,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.515160+0000","last_change":"2024-02-05T07:23:47.630558+0000","last_active":"2024-02-05T07:24:02.515160+0000","last_peered":"2024-02-05T07:24:02.515160+0000","last_clean":"2024-02-05T07:24:02.515160+0000","last_became_active":"2024-02-05T07:23:47.607589+0000","last_became_peered":"2024-02-05T07:23:47.607589+0000","last_unstale":"2024-02-05T07:24:02.515160+0000","last_undegraded":"2024-02-05T07:24:02.515160+0000","last_fullsized":"2024-02-05T07:24:02.515160+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:18:50.421025+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"173'14265","reported_seq":14573,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:05.071732+0000","last_change":"2024-02-05T07:23:40.012966+0000","last_active":"2024-02-05T07:24:05.071732+0000","last_peered":"2024-02-05T07:24:05.071732+0000","last_clean":"2024-02-05T07:24:05.071732+0000","last_became_active":"2024-02-05T07:20:45.769333+0000","last_became_peered":"2024-02-05T07:20:45.769333+0000","last_unstale":"2024-02-05T07:24:05.071732+0000","last_undegraded":"2024-02-05T07:24:05.071732+0000","last_fullsized":"2024-02-05T07:24:05.071732+0000","mapping_epoch":100,"log_start":"152'10464","ondisk_log_start":"152'10464","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'13339","last_scrub_stamp":"2024-02-05T07:23:40.012704+0000","last_deep_scrub":"156'13339","last_deep_scrub_stamp":"2024-02-05T07:23:40.012704+0000","last_clean_scrub_stamp":"2024-02-05T07:23:40.012704+0000","objects_scrubbed":3400,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":20,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:52.266462+0000","scrub_duration":19992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133627904,"num_objects":2039,"num_object_clones":0,"num_object_copies":6117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10865,"num_write_kb":412928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":53,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384721+0000","last_change":"2024-02-05T07:23:27.420162+0000","last_active":"2024-02-05T07:24:02.384721+0000","last_peered":"2024-02-05T07:24:02.384721+0000","last_clean":"2024-02-05T07:24:02.384721+0000","last_became_active":"2024-02-05T07:23:27.419855+0000","last_became_peered":"2024-02-05T07:23:27.419855+0000","last_unstale":"2024-02-05T07:24:02.384721+0000","last_undegraded":"2024-02-05T07:24:02.384721+0000","last_fullsized":"2024-02-05T07:24:02.384721+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:37.032056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:37.032056+0000","last_clean_scrub_stamp":"2024-02-05T07:20:37.032056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":242,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.514992+0000","last_change":"2024-02-05T07:20:13.701938+0000","last_active":"2024-02-05T07:24:02.514992+0000","last_peered":"2024-02-05T07:24:02.514992+0000","last_clean":"2024-02-05T07:24:02.514992+0000","last_became_active":"2024-02-05T07:18:49.437663+0000","last_became_peered":"2024-02-05T07:18:49.437663+0000","last_unstale":"2024-02-05T07:24:02.514992+0000","last_undegraded":"2024-02-05T07:24:02.514992+0000","last_fullsized":"2024-02-05T07:24:02.514992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:20:13.701852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:13.701852+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":259,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.512921+0000","last_change":"2024-02-05T07:21:32.875499+0000","last_active":"2024-02-05T07:24:02.512921+0000","last_peered":"2024-02-05T07:24:02.512921+0000","last_clean":"2024-02-05T07:24:02.512921+0000","last_became_active":"2024-02-05T07:18:51.508097+0000","last_became_peered":"2024-02-05T07:18:51.508097+0000","last_unstale":"2024-02-05T07:24:02.512921+0000","last_undegraded":"2024-02-05T07:24:02.512921+0000","last_fullsized":"2024-02-05T07:24:02.512921+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:21:32.875346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:32.875346+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":303,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513692+0000","last_change":"2024-02-05T07:21:26.505275+0000","last_active":"2024-02-05T07:24:02.513692+0000","last_peered":"2024-02-05T07:24:02.513692+0000","last_clean":"2024-02-05T07:24:02.513692+0000","last_became_active":"2024-02-05T07:18:53.456287+0000","last_became_peered":"2024-02-05T07:18:53.456287+0000","last_unstale":"2024-02-05T07:24:02.513692+0000","last_undegraded":"2024-02-05T07:24:02.513692+0000","last_fullsized":"2024-02-05T07:24:02.513692+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-02-05T07:21:26.505161+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:21:26.505161+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"173'11274","reported_seq":16230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:05.447150+0000","last_change":"2024-02-05T07:24:01.504189+0000","last_active":"2024-02-05T07:24:05.447150+0000","last_peered":"2024-02-05T07:24:05.447150+0000","last_clean":"2024-02-05T07:24:05.447150+0000","last_became_active":"2024-02-05T07:24:01.475156+0000","last_became_peered":"2024-02-05T07:24:01.475156+0000","last_unstale":"2024-02-05T07:24:05.447150+0000","last_undegraded":"2024-02-05T07:24:05.447150+0000","last_fullsized":"2024-02-05T07:24:05.447150+0000","mapping_epoch":171,"log_start":"131'6569","ondisk_log_start":"131'6569","created":100,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132382720,"num_objects":2020,"num_object_clones":0,"num_object_copies":6060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11274,"num_write_kb":425408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2391,"num_bytes_recovered":156696576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":32,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513134+0000","last_change":"2024-02-05T07:23:47.610635+0000","last_active":"2024-02-05T07:24:02.513134+0000","last_peered":"2024-02-05T07:24:02.513134+0000","last_clean":"2024-02-05T07:24:02.513134+0000","last_became_active":"2024-02-05T07:23:47.610289+0000","last_became_peered":"2024-02-05T07:23:47.610289+0000","last_unstale":"2024-02-05T07:24:02.513134+0000","last_undegraded":"2024-02-05T07:24:02.513134+0000","last_fullsized":"2024-02-05T07:24:02.513134+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:28.882326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:28.882326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":238,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.514871+0000","last_change":"2024-02-05T07:21:31.527396+0000","last_active":"2024-02-05T07:24:02.514871+0000","last_peered":"2024-02-05T07:24:02.514871+0000","last_clean":"2024-02-05T07:24:02.514871+0000","last_became_active":"2024-02-05T07:18:49.435621+0000","last_became_peered":"2024-02-05T07:18:49.435621+0000","last_unstale":"2024-02-05T07:24:02.514871+0000","last_undegraded":"2024-02-05T07:24:02.514871+0000","last_fullsized":"2024-02-05T07:24:02.514871+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:31.527325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:31.527325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":293,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548492+0000","last_change":"2024-02-05T07:23:47.625879+0000","last_active":"2024-02-05T07:24:02.548492+0000","last_peered":"2024-02-05T07:24:02.548492+0000","last_clean":"2024-02-05T07:24:02.548492+0000","last_became_active":"2024-02-05T07:23:47.607181+0000","last_became_peered":"2024-02-05T07:23:47.607181+0000","last_unstale":"2024-02-05T07:24:02.548492+0000","last_undegraded":"2024-02-05T07:24:02.548492+0000","last_fullsized":"2024-02-05T07:24:02.548492+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:21:33.928419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:33.928419+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.381444+0000","last_change":"2024-02-05T07:21:29.559984+0000","last_active":"2024-02-05T07:24:02.381444+0000","last_peered":"2024-02-05T07:24:02.381444+0000","last_clean":"2024-02-05T07:24:02.381444+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:24:02.381444+0000","last_undegraded":"2024-02-05T07:24:02.381444+0000","last_fullsized":"2024-02-05T07:24:02.381444+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:29.559910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:29.559910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"173'7839","reported_seq":8033,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:05.613772+0000","last_change":"2024-02-05T07:21:43.853670+0000","last_active":"2024-02-05T07:24:05.613772+0000","last_peered":"2024-02-05T07:24:05.613772+0000","last_clean":"2024-02-05T07:24:05.613772+0000","last_became_active":"2024-02-05T07:21:42.561220+0000","last_became_peered":"2024-02-05T07:21:42.561220+0000","last_unstale":"2024-02-05T07:24:05.613772+0000","last_undegraded":"2024-02-05T07:24:05.613772+0000","last_fullsized":"2024-02-05T07:24:05.613772+0000","mapping_epoch":100,"log_start":"109'2262","ondisk_log_start":"109'2262","created":100,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'4670","last_scrub_stamp":"2024-02-05T07:21:43.853520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.853520+0000","objects_scrubbed":2392,"log_size":5577,"log_dups_size":0,"ondisk_log_size":5577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62324736,"num_objects":951,"num_object_clones":0,"num_object_copies":2853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3171,"num_write_kb":55360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":224,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.381873+0000","last_change":"2024-02-05T07:23:47.611420+0000","last_active":"2024-02-05T07:24:02.381873+0000","last_peered":"2024-02-05T07:24:02.381873+0000","last_clean":"2024-02-05T07:24:02.381873+0000","last_became_active":"2024-02-05T07:23:47.611066+0000","last_became_peered":"2024-02-05T07:23:47.611066+0000","last_unstale":"2024-02-05T07:24:02.381873+0000","last_undegraded":"2024-02-05T07:24:02.381873+0000","last_fullsized":"2024-02-05T07:24:02.381873+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:38.715447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:38.715447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.382769+0000","last_change":"2024-02-05T07:21:38.468561+0000","last_active":"2024-02-05T07:24:02.382769+0000","last_peered":"2024-02-05T07:24:02.382769+0000","last_clean":"2024-02-05T07:24:02.382769+0000","last_became_active":"2024-02-05T07:18:53.464867+0000","last_became_peered":"2024-02-05T07:18:53.464867+0000","last_unstale":"2024-02-05T07:24:02.382769+0000","last_undegraded":"2024-02-05T07:24:02.382769+0000","last_fullsized":"2024-02-05T07:24:02.382769+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:38.468456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:38.468456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":238,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.514936+0000","last_change":"2024-02-05T07:21:41.444299+0000","last_active":"2024-02-05T07:24:02.514936+0000","last_peered":"2024-02-05T07:24:02.514936+0000","last_clean":"2024-02-05T07:24:02.514936+0000","last_became_active":"2024-02-05T07:18:49.434451+0000","last_became_peered":"2024-02-05T07:18:49.434451+0000","last_unstale":"2024-02-05T07:24:02.514936+0000","last_undegraded":"2024-02-05T07:24:02.514936+0000","last_fullsized":"2024-02-05T07:24:02.514936+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:41.444193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:41.444193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":265,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.512488+0000","last_change":"2024-02-05T07:21:31.340373+0000","last_active":"2024-02-05T07:24:02.512488+0000","last_peered":"2024-02-05T07:24:02.512488+0000","last_clean":"2024-02-05T07:24:02.512488+0000","last_became_active":"2024-02-05T07:18:51.507672+0000","last_became_peered":"2024-02-05T07:18:51.507672+0000","last_unstale":"2024-02-05T07:24:02.512488+0000","last_undegraded":"2024-02-05T07:24:02.512488+0000","last_fullsized":"2024-02-05T07:24:02.512488+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:21:31.340234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:31.340234+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"173'7748","reported_seq":7945,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:05.440449+0000","last_change":"2024-02-05T07:21:43.922152+0000","last_active":"2024-02-05T07:24:05.440449+0000","last_peered":"2024-02-05T07:24:05.440449+0000","last_clean":"2024-02-05T07:24:05.440449+0000","last_became_active":"2024-02-05T07:21:42.569592+0000","last_became_peered":"2024-02-05T07:21:42.569592+0000","last_unstale":"2024-02-05T07:24:05.440449+0000","last_undegraded":"2024-02-05T07:24:05.440449+0000","last_fullsized":"2024-02-05T07:24:05.440449+0000","mapping_epoch":100,"log_start":"107'1537","ondisk_log_start":"107'1537","created":100,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'4643","last_scrub_stamp":"2024-02-05T07:21:43.922034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.922034+0000","objects_scrubbed":2331,"log_size":6211,"log_dups_size":0,"ondisk_log_size":6211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":428,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62521344,"num_objects":954,"num_object_clones":0,"num_object_copies":2862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3105,"num_write_kb":55296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":218,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.547972+0000","last_change":"2024-02-05T07:21:37.210122+0000","last_active":"2024-02-05T07:24:02.547972+0000","last_peered":"2024-02-05T07:24:02.547972+0000","last_clean":"2024-02-05T07:24:02.547972+0000","last_became_active":"2024-02-05T07:18:55.475702+0000","last_became_peered":"2024-02-05T07:18:55.475702+0000","last_unstale":"2024-02-05T07:24:02.547972+0000","last_undegraded":"2024-02-05T07:24:02.547972+0000","last_fullsized":"2024-02-05T07:24:02.547972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:37.209967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.209967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":236,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384485+0000","last_change":"2024-02-05T07:23:47.620787+0000","last_active":"2024-02-05T07:24:02.384485+0000","last_peered":"2024-02-05T07:24:02.384485+0000","last_clean":"2024-02-05T07:24:02.384485+0000","last_became_active":"2024-02-05T07:23:47.619951+0000","last_became_peered":"2024-02-05T07:23:47.619951+0000","last_unstale":"2024-02-05T07:24:02.384485+0000","last_undegraded":"2024-02-05T07:24:02.384485+0000","last_fullsized":"2024-02-05T07:24:02.384485+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:35.587887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:35.587887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":34,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.512557+0000","last_change":"2024-02-05T07:23:47.611344+0000","last_active":"2024-02-05T07:24:02.512557+0000","last_peered":"2024-02-05T07:24:02.512557+0000","last_clean":"2024-02-05T07:24:02.512557+0000","last_became_active":"2024-02-05T07:23:47.610698+0000","last_became_peered":"2024-02-05T07:23:47.610698+0000","last_unstale":"2024-02-05T07:24:02.512557+0000","last_undegraded":"2024-02-05T07:24:02.512557+0000","last_fullsized":"2024-02-05T07:24:02.512557+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:15.066768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:15.066768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":242,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548452+0000","last_change":"2024-02-05T07:21:21.184568+0000","last_active":"2024-02-05T07:24:02.548452+0000","last_peered":"2024-02-05T07:24:02.548452+0000","last_clean":"2024-02-05T07:24:02.548452+0000","last_became_active":"2024-02-05T07:18:51.440744+0000","last_became_peered":"2024-02-05T07:18:51.440744+0000","last_unstale":"2024-02-05T07:24:02.548452+0000","last_undegraded":"2024-02-05T07:24:02.548452+0000","last_fullsized":"2024-02-05T07:24:02.548452+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:21.184445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:21.184445+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"173'9138","reported_seq":11702,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:04.921711+0000","last_change":"2024-02-05T07:24:01.487722+0000","last_active":"2024-02-05T07:24:04.921711+0000","last_peered":"2024-02-05T07:24:04.921711+0000","last_clean":"2024-02-05T07:24:04.921711+0000","last_became_active":"2024-02-05T07:24:01.468725+0000","last_became_peered":"2024-02-05T07:24:01.468725+0000","last_unstale":"2024-02-05T07:24:04.921711+0000","last_undegraded":"2024-02-05T07:24:04.921711+0000","last_fullsized":"2024-02-05T07:24:04.921711+0000","mapping_epoch":171,"log_start":"117'4561","ondisk_log_start":"117'4561","created":100,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'6081","last_scrub_stamp":"2024-02-05T07:21:51.270929+0000","last_deep_scrub":"124'6081","last_deep_scrub_stamp":"2024-02-05T07:21:51.270929+0000","last_clean_scrub_stamp":"2024-02-05T07:21:51.270929+0000","objects_scrubbed":2391,"log_size":4577,"log_dups_size":0,"ondisk_log_size":4577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"queued for scrub","scrub_duration":6883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66584576,"num_objects":1016,"num_object_clones":0,"num_object_copies":3048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3137,"num_write_kb":56384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":74448896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":32,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548773+0000","last_change":"2024-02-05T07:23:47.603603+0000","last_active":"2024-02-05T07:24:02.548773+0000","last_peered":"2024-02-05T07:24:02.548773+0000","last_clean":"2024-02-05T07:24:02.548773+0000","last_became_active":"2024-02-05T07:23:47.603091+0000","last_became_peered":"2024-02-05T07:23:47.603091+0000","last_unstale":"2024-02-05T07:24:02.548773+0000","last_undegraded":"2024-02-05T07:24:02.548773+0000","last_fullsized":"2024-02-05T07:24:02.548773+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:18:54.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":144,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.383058+0000","last_change":"2024-02-05T07:23:29.341025+0000","last_active":"2024-02-05T07:24:02.383058+0000","last_peered":"2024-02-05T07:24:02.383058+0000","last_clean":"2024-02-05T07:24:02.383058+0000","last_became_active":"2024-02-05T07:23:27.439333+0000","last_became_peered":"2024-02-05T07:23:27.439333+0000","last_unstale":"2024-02-05T07:24:02.383058+0000","last_undegraded":"2024-02-05T07:24:02.383058+0000","last_fullsized":"2024-02-05T07:24:02.383058+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:23:29.340876+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:20:38.993688+0000","last_clean_scrub_stamp":"2024-02-05T07:23:29.340876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:29.494825+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":238,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548597+0000","last_change":"2024-02-05T07:23:08.606203+0000","last_active":"2024-02-05T07:24:02.548597+0000","last_peered":"2024-02-05T07:24:02.548597+0000","last_clean":"2024-02-05T07:24:02.548597+0000","last_became_active":"2024-02-05T07:18:49.430208+0000","last_became_peered":"2024-02-05T07:18:49.430208+0000","last_unstale":"2024-02-05T07:24:02.548597+0000","last_undegraded":"2024-02-05T07:24:02.548597+0000","last_fullsized":"2024-02-05T07:24:02.548597+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:08.606061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:23:08.606061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:33.306988+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"164'29","reported_seq":383,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384184+0000","last_change":"2024-02-05T07:23:47.660633+0000","last_active":"2024-02-05T07:24:02.384184+0000","last_peered":"2024-02-05T07:24:02.384184+0000","last_clean":"2024-02-05T07:24:02.384184+0000","last_became_active":"2024-02-05T07:23:47.608917+0000","last_became_peered":"2024-02-05T07:23:47.608917+0000","last_unstale":"2024-02-05T07:24:02.384184+0000","last_undegraded":"2024-02-05T07:24:02.384184+0000","last_fullsized":"2024-02-05T07:24:02.384184+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-02-05T07:23:27.310935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:27.310935+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:44.481733+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":222,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548697+0000","last_change":"2024-02-05T07:21:34.202945+0000","last_active":"2024-02-05T07:24:02.548697+0000","last_peered":"2024-02-05T07:24:02.548697+0000","last_clean":"2024-02-05T07:24:02.548697+0000","last_became_active":"2024-02-05T07:18:53.460715+0000","last_became_peered":"2024-02-05T07:18:53.460715+0000","last_unstale":"2024-02-05T07:24:02.548697+0000","last_undegraded":"2024-02-05T07:24:02.548697+0000","last_fullsized":"2024-02-05T07:24:02.548697+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:34.202812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:34.202812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"173'7815","reported_seq":8012,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:05.594806+0000","last_change":"2024-02-05T07:21:44.496740+0000","last_active":"2024-02-05T07:24:05.594806+0000","last_peered":"2024-02-05T07:24:05.594806+0000","last_clean":"2024-02-05T07:24:05.594806+0000","last_became_active":"2024-02-05T07:21:42.560423+0000","last_became_peered":"2024-02-05T07:21:42.560423+0000","last_unstale":"2024-02-05T07:24:05.594806+0000","last_undegraded":"2024-02-05T07:24:05.594806+0000","last_fullsized":"2024-02-05T07:24:05.594806+0000","mapping_epoch":100,"log_start":"108'2091","ondisk_log_start":"108'2091","created":100,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'4680","last_scrub_stamp":"2024-02-05T07:21:44.496596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:44.496596+0000","objects_scrubbed":2317,"log_size":5724,"log_dups_size":0,"ondisk_log_size":5724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63832064,"num_objects":974,"num_object_clones":0,"num_object_copies":2922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3135,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":232,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.515098+0000","last_change":"2024-02-05T07:23:47.611214+0000","last_active":"2024-02-05T07:24:02.515098+0000","last_peered":"2024-02-05T07:24:02.515098+0000","last_clean":"2024-02-05T07:24:02.515098+0000","last_became_active":"2024-02-05T07:23:47.610875+0000","last_became_peered":"2024-02-05T07:23:47.610875+0000","last_unstale":"2024-02-05T07:24:02.515098+0000","last_undegraded":"2024-02-05T07:24:02.515098+0000","last_fullsized":"2024-02-05T07:24:02.515098+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:38.531265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:21:38.531265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081581+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":245,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.382220+0000","last_change":"2024-02-05T07:21:34.523665+0000","last_active":"2024-02-05T07:24:02.382220+0000","last_peered":"2024-02-05T07:24:02.382220+0000","last_clean":"2024-02-05T07:24:02.382220+0000","last_became_active":"2024-02-05T07:18:49.429651+0000","last_became_peered":"2024-02-05T07:18:49.429651+0000","last_unstale":"2024-02-05T07:24:02.382220+0000","last_undegraded":"2024-02-05T07:24:02.382220+0000","last_fullsized":"2024-02-05T07:24:02.382220+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:21:34.523215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:34.523215+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":253,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.381570+0000","last_change":"2024-02-05T07:23:47.622111+0000","last_active":"2024-02-05T07:24:02.381570+0000","last_peered":"2024-02-05T07:24:02.381570+0000","last_clean":"2024-02-05T07:24:02.381570+0000","last_became_active":"2024-02-05T07:23:47.621133+0000","last_became_peered":"2024-02-05T07:23:47.621133+0000","last_unstale":"2024-02-05T07:24:02.381570+0000","last_undegraded":"2024-02-05T07:24:02.381570+0000","last_fullsized":"2024-02-05T07:24:02.381570+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:18.723118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:21:18.723118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"96'18","reported_seq":267,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.383236+0000","last_change":"2024-02-05T07:23:05.927322+0000","last_active":"2024-02-05T07:24:02.383236+0000","last_peered":"2024-02-05T07:24:02.383236+0000","last_clean":"2024-02-05T07:24:02.383236+0000","last_became_active":"2024-02-05T07:18:51.436097+0000","last_became_peered":"2024-02-05T07:18:51.436097+0000","last_unstale":"2024-02-05T07:24:02.383236+0000","last_undegraded":"2024-02-05T07:24:02.383236+0000","last_fullsized":"2024-02-05T07:24:02.383236+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'18","last_scrub_stamp":"2024-02-05T07:23:05.927169+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:23:05.927169+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:17.987664+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384342+0000","last_change":"2024-02-05T07:21:31.546003+0000","last_active":"2024-02-05T07:24:02.384342+0000","last_peered":"2024-02-05T07:24:02.384342+0000","last_clean":"2024-02-05T07:24:02.384342+0000","last_became_active":"2024-02-05T07:18:53.460771+0000","last_became_peered":"2024-02-05T07:18:53.460771+0000","last_unstale":"2024-02-05T07:24:02.384342+0000","last_undegraded":"2024-02-05T07:24:02.384342+0000","last_fullsized":"2024-02-05T07:24:02.384342+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:31.545467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:31.545467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"173'7953","reported_seq":10794,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:05.620239+0000","last_change":"2024-02-05T07:23:57.365741+0000","last_active":"2024-02-05T07:24:05.620239+0000","last_peered":"2024-02-05T07:24:05.620239+0000","last_clean":"2024-02-05T07:24:05.620239+0000","last_became_active":"2024-02-05T07:23:57.348018+0000","last_became_peered":"2024-02-05T07:23:57.348018+0000","last_unstale":"2024-02-05T07:24:05.620239+0000","last_undegraded":"2024-02-05T07:24:05.620239+0000","last_fullsized":"2024-02-05T07:24:05.620239+0000","mapping_epoch":167,"log_start":"109'2369","ondisk_log_start":"109'2369","created":100,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'4786","last_scrub_stamp":"2024-02-05T07:21:46.577376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:46.577376+0000","objects_scrubbed":2382,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":1024,"num_object_clones":0,"num_object_copies":3072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":58304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":85917696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":224,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513949+0000","last_change":"2024-02-05T07:23:47.592633+0000","last_active":"2024-02-05T07:24:02.513949+0000","last_peered":"2024-02-05T07:24:02.513949+0000","last_clean":"2024-02-05T07:24:02.513949+0000","last_became_active":"2024-02-05T07:23:47.592290+0000","last_became_peered":"2024-02-05T07:23:47.592290+0000","last_unstale":"2024-02-05T07:24:02.513949+0000","last_undegraded":"2024-02-05T07:24:02.513949+0000","last_fullsized":"2024-02-05T07:24:02.513949+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:35.085136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:35.085136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":487,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.381956+0000","last_change":"2024-02-05T07:20:31.816346+0000","last_active":"2024-02-05T07:24:02.381956+0000","last_peered":"2024-02-05T07:24:02.381956+0000","last_clean":"2024-02-05T07:24:02.381956+0000","last_became_active":"2024-02-05T07:18:39.444898+0000","last_became_peered":"2024-02-05T07:18:39.444898+0000","last_unstale":"2024-02-05T07:24:02.381956+0000","last_undegraded":"2024-02-05T07:24:02.381956+0000","last_fullsized":"2024-02-05T07:24:02.381956+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:20:31.816160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:20:31.816160+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081120+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":32,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384254+0000","last_change":"2024-02-05T07:23:47.609950+0000","last_active":"2024-02-05T07:24:02.384254+0000","last_peered":"2024-02-05T07:24:02.384254+0000","last_clean":"2024-02-05T07:24:02.384254+0000","last_became_active":"2024-02-05T07:23:47.609497+0000","last_became_peered":"2024-02-05T07:23:47.609497+0000","last_unstale":"2024-02-05T07:24:02.384254+0000","last_undegraded":"2024-02-05T07:24:02.384254+0000","last_fullsized":"2024-02-05T07:24:02.384254+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:29.851303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:29.851303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":312,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.514565+0000","last_change":"2024-02-05T07:20:25.394750+0000","last_active":"2024-02-05T07:24:02.514565+0000","last_peered":"2024-02-05T07:24:02.514565+0000","last_clean":"2024-02-05T07:24:02.514565+0000","last_became_active":"2024-02-05T07:18:51.507714+0000","last_became_peered":"2024-02-05T07:18:51.507714+0000","last_unstale":"2024-02-05T07:24:02.514565+0000","last_undegraded":"2024-02-05T07:24:02.514565+0000","last_fullsized":"2024-02-05T07:24:02.514565+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-02-05T07:20:25.394654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:25.394654+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":222,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513417+0000","last_change":"2024-02-05T07:21:15.698783+0000","last_active":"2024-02-05T07:24:02.513417+0000","last_peered":"2024-02-05T07:24:02.513417+0000","last_clean":"2024-02-05T07:24:02.513417+0000","last_became_active":"2024-02-05T07:18:53.459971+0000","last_became_peered":"2024-02-05T07:18:53.459971+0000","last_unstale":"2024-02-05T07:24:02.513417+0000","last_undegraded":"2024-02-05T07:24:02.513417+0000","last_fullsized":"2024-02-05T07:24:02.513417+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:15.698665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:15.698665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081280+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"173'7854","reported_seq":8051,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:04.852272+0000","last_change":"2024-02-05T07:21:43.836751+0000","last_active":"2024-02-05T07:24:04.852272+0000","last_peered":"2024-02-05T07:24:04.852272+0000","last_clean":"2024-02-05T07:24:04.852272+0000","last_became_active":"2024-02-05T07:21:42.573038+0000","last_became_peered":"2024-02-05T07:21:42.573038+0000","last_unstale":"2024-02-05T07:24:04.852272+0000","last_undegraded":"2024-02-05T07:24:04.852272+0000","last_fullsized":"2024-02-05T07:24:04.852272+0000","mapping_epoch":100,"log_start":"109'2516","ondisk_log_start":"109'2516","created":100,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'4736","last_scrub_stamp":"2024-02-05T07:21:43.836623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.836623+0000","objects_scrubbed":2356,"log_size":5338,"log_dups_size":0,"ondisk_log_size":5338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66387968,"num_objects":1013,"num_object_clones":0,"num_object_copies":3039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3119,"num_write_kb":56832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":238,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.383772+0000","last_change":"2024-02-05T07:23:47.618520+0000","last_active":"2024-02-05T07:24:02.383772+0000","last_peered":"2024-02-05T07:24:02.383772+0000","last_clean":"2024-02-05T07:24:02.383772+0000","last_became_active":"2024-02-05T07:23:47.600441+0000","last_became_peered":"2024-02-05T07:23:47.600441+0000","last_unstale":"2024-02-05T07:24:02.383772+0000","last_undegraded":"2024-02-05T07:24:02.383772+0000","last_fullsized":"2024-02-05T07:24:02.383772+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:23:22.071923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:22.071923+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:25.764893+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":236,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.547653+0000","last_change":"2024-02-05T07:23:47.627064+0000","last_active":"2024-02-05T07:24:02.547653+0000","last_peered":"2024-02-05T07:24:02.547653+0000","last_clean":"2024-02-05T07:24:02.547653+0000","last_became_active":"2024-02-05T07:23:47.626143+0000","last_became_peered":"2024-02-05T07:23:47.626143+0000","last_unstale":"2024-02-05T07:24:02.547653+0000","last_undegraded":"2024-02-05T07:24:02.547653+0000","last_fullsized":"2024-02-05T07:24:02.547653+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:21.318169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:21.318169+0000","last_clean_scrub_stamp":"2024-02-05T07:20:21.318169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":550,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:03.540853+0000","last_change":"2024-02-05T07:20:34.765770+0000","last_active":"2024-02-05T07:24:03.540853+0000","last_peered":"2024-02-05T07:24:03.540853+0000","last_clean":"2024-02-05T07:24:03.540853+0000","last_became_active":"2024-02-05T07:18:48.424620+0000","last_became_peered":"2024-02-05T07:18:48.424620+0000","last_unstale":"2024-02-05T07:24:03.540853+0000","last_undegraded":"2024-02-05T07:24:03.540853+0000","last_fullsized":"2024-02-05T07:24:03.540853+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:20:34.765664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:20:34.765664+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":308,"num_read_kb":303,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":244,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548860+0000","last_change":"2024-02-05T07:20:14.321610+0000","last_active":"2024-02-05T07:24:02.548860+0000","last_peered":"2024-02-05T07:24:02.548860+0000","last_clean":"2024-02-05T07:24:02.548860+0000","last_became_active":"2024-02-05T07:18:51.442184+0000","last_became_peered":"2024-02-05T07:18:51.442184+0000","last_unstale":"2024-02-05T07:24:02.548860+0000","last_undegraded":"2024-02-05T07:24:02.548860+0000","last_fullsized":"2024-02-05T07:24:02.548860+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:14.321523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:14.321523+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":222,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.514320+0000","last_change":"2024-02-05T07:20:31.238357+0000","last_active":"2024-02-05T07:24:02.514320+0000","last_peered":"2024-02-05T07:24:02.514320+0000","last_clean":"2024-02-05T07:24:02.514320+0000","last_became_active":"2024-02-05T07:18:53.468512+0000","last_became_peered":"2024-02-05T07:18:53.468512+0000","last_unstale":"2024-02-05T07:24:02.514320+0000","last_undegraded":"2024-02-05T07:24:02.514320+0000","last_fullsized":"2024-02-05T07:24:02.514320+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:31.238251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:31.238251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"173'7655","reported_seq":10076,"reported_epoch":173,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-05T07:24:04.900637+0000","last_change":"2024-02-05T07:23:48.916169+0000","last_active":"2024-02-05T07:24:04.900637+0000","last_peered":"2024-02-05T07:24:04.900637+0000","last_clean":"2024-02-05T07:23:46.558465+0000","last_became_active":"2024-02-05T07:23:48.879960+0000","last_became_peered":"2024-02-05T07:23:48.879960+0000","last_unstale":"2024-02-05T07:24:04.900637+0000","last_undegraded":"2024-02-05T07:23:48.611563+0000","last_fullsized":"2024-02-05T07:23:48.584747+0000","mapping_epoch":163,"log_start":"104'1060","ondisk_log_start":"104'1060","created":100,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'4604","last_scrub_stamp":"2024-02-05T07:21:43.479351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.479351+0000","objects_scrubbed":2300,"log_size":6595,"log_dups_size":0,"ondisk_log_size":6595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64552960,"num_objects":985,"num_object_clones":0,"num_object_copies":2955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":178,"num_objects_unfound":0,"num_objects_dirty":985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3063,"num_write_kb":55936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1110,"num_bytes_recovered":72744960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":178},{"shards":"1,5","objects":805}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":227,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.382173+0000","last_change":"2024-02-05T07:23:01.849413+0000","last_active":"2024-02-05T07:24:02.382173+0000","last_peered":"2024-02-05T07:24:02.382173+0000","last_clean":"2024-02-05T07:24:02.382173+0000","last_became_active":"2024-02-05T07:18:55.505360+0000","last_became_peered":"2024-02-05T07:18:55.505360+0000","last_unstale":"2024-02-05T07:24:02.382173+0000","last_undegraded":"2024-02-05T07:24:02.382173+0000","last_fullsized":"2024-02-05T07:24:02.382173+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:01.849346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:01.849346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:23.072077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":239,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.383995+0000","last_change":"2024-02-05T07:21:40.542969+0000","last_active":"2024-02-05T07:24:02.383995+0000","last_peered":"2024-02-05T07:24:02.383995+0000","last_clean":"2024-02-05T07:24:02.383995+0000","last_became_active":"2024-02-05T07:18:49.428470+0000","last_became_peered":"2024-02-05T07:18:49.428470+0000","last_unstale":"2024-02-05T07:24:02.383995+0000","last_undegraded":"2024-02-05T07:24:02.383995+0000","last_fullsized":"2024-02-05T07:24:02.383995+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:40.542479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:21:40.542479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":262,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548333+0000","last_change":"2024-02-05T07:20:12.309384+0000","last_active":"2024-02-05T07:24:02.548333+0000","last_peered":"2024-02-05T07:24:02.548333+0000","last_clean":"2024-02-05T07:24:02.548333+0000","last_became_active":"2024-02-05T07:18:51.507507+0000","last_became_peered":"2024-02-05T07:18:51.507507+0000","last_unstale":"2024-02-05T07:24:02.548333+0000","last_undegraded":"2024-02-05T07:24:02.548333+0000","last_fullsized":"2024-02-05T07:24:02.548333+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:20:12.309238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:12.309238+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.512596+0000","last_change":"2024-02-05T07:21:37.374835+0000","last_active":"2024-02-05T07:24:02.512596+0000","last_peered":"2024-02-05T07:24:02.512596+0000","last_clean":"2024-02-05T07:24:02.512596+0000","last_became_active":"2024-02-05T07:18:53.459474+0000","last_became_peered":"2024-02-05T07:18:53.459474+0000","last_unstale":"2024-02-05T07:24:02.512596+0000","last_undegraded":"2024-02-05T07:24:02.512596+0000","last_fullsized":"2024-02-05T07:24:02.512596+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.374703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"173'8015","reported_seq":8220,"reported_epoch":173,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:24:04.933722+0000","last_change":"2024-02-05T07:23:48.786606+0000","last_active":"2024-02-05T07:24:04.933722+0000","last_peered":"2024-02-05T07:24:04.933722+0000","last_clean":"2024-02-05T07:23:46.544320+0000","last_became_active":"2024-02-05T07:23:48.768689+0000","last_became_peered":"2024-02-05T07:23:48.768689+0000","last_unstale":"2024-02-05T07:24:04.933722+0000","last_undegraded":"2024-02-05T07:24:04.933722+0000","last_fullsized":"2024-02-05T07:24:04.933722+0000","mapping_epoch":163,"log_start":"109'2625","ondisk_log_start":"109'2625","created":100,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'4712","last_scrub_stamp":"2024-02-05T07:21:43.408403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.408403+0000","objects_scrubbed":2431,"log_size":5390,"log_dups_size":0,"ondisk_log_size":5390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":616,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69533696,"num_objects":1061,"num_object_clones":0,"num_object_copies":3183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1061,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3314,"num_write_kb":62208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,3],"avail_no_missing":["4","3","6"],"object_location_counts":[{"shards":"3,4,6","objects":1061}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":224,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.547586+0000","last_change":"2024-02-05T07:23:47.622723+0000","last_active":"2024-02-05T07:24:02.547586+0000","last_peered":"2024-02-05T07:24:02.547586+0000","last_clean":"2024-02-05T07:24:02.547586+0000","last_became_active":"2024-02-05T07:23:47.617848+0000","last_became_peered":"2024-02-05T07:23:47.617848+0000","last_unstale":"2024-02-05T07:24:02.547586+0000","last_undegraded":"2024-02-05T07:24:02.547586+0000","last_fullsized":"2024-02-05T07:24:02.547586+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:27.196824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:27.196824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":236,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513546+0000","last_change":"2024-02-05T07:23:47.605453+0000","last_active":"2024-02-05T07:24:02.513546+0000","last_peered":"2024-02-05T07:24:02.513546+0000","last_clean":"2024-02-05T07:24:02.513546+0000","last_became_active":"2024-02-05T07:23:47.604997+0000","last_became_peered":"2024-02-05T07:23:47.604997+0000","last_unstale":"2024-02-05T07:24:02.513546+0000","last_undegraded":"2024-02-05T07:24:02.513546+0000","last_fullsized":"2024-02-05T07:24:02.513546+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:20:32.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"151'15","reported_seq":263,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.383462+0000","last_change":"2024-02-05T07:23:26.247211+0000","last_active":"2024-02-05T07:24:02.383462+0000","last_peered":"2024-02-05T07:24:02.383462+0000","last_clean":"2024-02-05T07:24:02.383462+0000","last_became_active":"2024-02-05T07:18:51.448044+0000","last_became_peered":"2024-02-05T07:18:51.448044+0000","last_unstale":"2024-02-05T07:24:02.383462+0000","last_undegraded":"2024-02-05T07:24:02.383462+0000","last_fullsized":"2024-02-05T07:24:02.383462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:23:26.247047+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:30.727049+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":294,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548280+0000","last_change":"2024-02-05T07:21:36.187571+0000","last_active":"2024-02-05T07:24:02.548280+0000","last_peered":"2024-02-05T07:24:02.548280+0000","last_clean":"2024-02-05T07:24:02.548280+0000","last_became_active":"2024-02-05T07:18:53.462890+0000","last_became_peered":"2024-02-05T07:18:53.462890+0000","last_unstale":"2024-02-05T07:24:02.548280+0000","last_undegraded":"2024-02-05T07:24:02.548280+0000","last_fullsized":"2024-02-05T07:24:02.548280+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:21:36.187453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:36.187453+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"173'7917","reported_seq":10719,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:05.070988+0000","last_change":"2024-02-05T07:23:57.368069+0000","last_active":"2024-02-05T07:24:05.070988+0000","last_peered":"2024-02-05T07:24:05.070988+0000","last_clean":"2024-02-05T07:24:05.070988+0000","last_became_active":"2024-02-05T07:23:57.347714+0000","last_became_peered":"2024-02-05T07:23:57.347714+0000","last_unstale":"2024-02-05T07:24:05.070988+0000","last_undegraded":"2024-02-05T07:24:05.070988+0000","last_fullsized":"2024-02-05T07:24:05.070988+0000","mapping_epoch":167,"log_start":"110'3772","ondisk_log_start":"110'3772","created":100,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'4711","last_scrub_stamp":"2024-02-05T07:21:43.941589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.941589+0000","objects_scrubbed":2370,"log_size":4145,"log_dups_size":0,"ondisk_log_size":4145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":475,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67895296,"num_objects":1036,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3206,"num_write_kb":59904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1295,"num_bytes_recovered":84869120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":218,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513040+0000","last_change":"2024-02-05T07:21:22.584921+0000","last_active":"2024-02-05T07:24:02.513040+0000","last_peered":"2024-02-05T07:24:02.513040+0000","last_clean":"2024-02-05T07:24:02.513040+0000","last_became_active":"2024-02-05T07:18:55.507791+0000","last_became_peered":"2024-02-05T07:18:55.507791+0000","last_unstale":"2024-02-05T07:24:02.513040+0000","last_undegraded":"2024-02-05T07:24:02.513040+0000","last_fullsized":"2024-02-05T07:24:02.513040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:22.584770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:22.584770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081227+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":239,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548983+0000","last_change":"2024-02-05T07:21:39.233919+0000","last_active":"2024-02-05T07:24:02.548983+0000","last_peered":"2024-02-05T07:24:02.548983+0000","last_clean":"2024-02-05T07:24:02.548983+0000","last_became_active":"2024-02-05T07:18:49.430123+0000","last_became_peered":"2024-02-05T07:18:49.430123+0000","last_unstale":"2024-02-05T07:24:02.548983+0000","last_undegraded":"2024-02-05T07:24:02.548983+0000","last_fullsized":"2024-02-05T07:24:02.548983+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:21:39.233796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.233796+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":247,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548332+0000","last_change":"2024-02-05T07:21:19.981440+0000","last_active":"2024-02-05T07:24:02.548332+0000","last_peered":"2024-02-05T07:24:02.548332+0000","last_clean":"2024-02-05T07:24:02.548332+0000","last_became_active":"2024-02-05T07:18:51.438507+0000","last_became_peered":"2024-02-05T07:18:51.438507+0000","last_unstale":"2024-02-05T07:24:02.548332+0000","last_undegraded":"2024-02-05T07:24:02.548332+0000","last_fullsized":"2024-02-05T07:24:02.548332+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-02-05T07:21:19.981349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:19.981349+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"164'9","reported_seq":65,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.381923+0000","last_change":"2024-02-05T07:23:47.650997+0000","last_active":"2024-02-05T07:24:02.381923+0000","last_peered":"2024-02-05T07:24:02.381923+0000","last_clean":"2024-02-05T07:24:02.381923+0000","last_became_active":"2024-02-05T07:23:47.626082+0000","last_became_peered":"2024-02-05T07:23:47.626082+0000","last_unstale":"2024-02-05T07:24:02.381923+0000","last_undegraded":"2024-02-05T07:24:02.381923+0000","last_fullsized":"2024-02-05T07:24:02.381923+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:18:52.436288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"173'7901","reported_seq":10670,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:05.425142+0000","last_change":"2024-02-05T07:23:57.370127+0000","last_active":"2024-02-05T07:24:05.425142+0000","last_peered":"2024-02-05T07:24:05.425142+0000","last_clean":"2024-02-05T07:24:05.425142+0000","last_became_active":"2024-02-05T07:23:57.350366+0000","last_became_peered":"2024-02-05T07:23:57.350366+0000","last_unstale":"2024-02-05T07:24:05.425142+0000","last_undegraded":"2024-02-05T07:24:05.425142+0000","last_fullsized":"2024-02-05T07:24:05.425142+0000","mapping_epoch":167,"log_start":"108'1749","ondisk_log_start":"108'1749","created":100,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'4694","last_scrub_stamp":"2024-02-05T07:21:44.743455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:44.743455+0000","objects_scrubbed":2359,"log_size":6152,"log_dups_size":0,"ondisk_log_size":6152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67371008,"num_objects":1028,"num_object_clones":0,"num_object_copies":3084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3207,"num_write_kb":60032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1277,"num_bytes_recovered":83623936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":232,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548048+0000","last_change":"2024-02-05T07:23:47.603895+0000","last_active":"2024-02-05T07:24:02.548048+0000","last_peered":"2024-02-05T07:24:02.548048+0000","last_clean":"2024-02-05T07:24:02.548048+0000","last_became_active":"2024-02-05T07:23:47.603326+0000","last_became_peered":"2024-02-05T07:23:47.603326+0000","last_unstale":"2024-02-05T07:24:02.548048+0000","last_undegraded":"2024-02-05T07:24:02.548048+0000","last_fullsized":"2024-02-05T07:24:02.548048+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:16.499591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:16.499591+0000","last_clean_scrub_stamp":"2024-02-05T07:23:16.499591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:36.394227+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":32,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.514823+0000","last_change":"2024-02-05T07:23:47.611708+0000","last_active":"2024-02-05T07:24:02.514823+0000","last_peered":"2024-02-05T07:24:02.514823+0000","last_clean":"2024-02-05T07:24:02.514823+0000","last_became_active":"2024-02-05T07:23:47.609054+0000","last_became_peered":"2024-02-05T07:23:47.609054+0000","last_unstale":"2024-02-05T07:24:02.514823+0000","last_undegraded":"2024-02-05T07:24:02.514823+0000","last_fullsized":"2024-02-05T07:24:02.514823+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:42.464894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:42.464894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":101,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.382079+0000","last_change":"2024-02-05T07:23:47.649411+0000","last_active":"2024-02-05T07:24:02.382079+0000","last_peered":"2024-02-05T07:24:02.382079+0000","last_clean":"2024-02-05T07:24:02.382079+0000","last_became_active":"2024-02-05T07:23:47.618052+0000","last_became_peered":"2024-02-05T07:23:47.618052+0000","last_unstale":"2024-02-05T07:24:02.382079+0000","last_undegraded":"2024-02-05T07:24:02.382079+0000","last_fullsized":"2024-02-05T07:24:02.382079+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:20:00.796905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:00.796905+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:03.081067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":236,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548126+0000","last_change":"2024-02-05T07:23:47.626460+0000","last_active":"2024-02-05T07:24:02.548126+0000","last_peered":"2024-02-05T07:24:02.548126+0000","last_clean":"2024-02-05T07:24:02.548126+0000","last_became_active":"2024-02-05T07:23:47.626072+0000","last_became_peered":"2024-02-05T07:23:47.626072+0000","last_unstale":"2024-02-05T07:24:02.548126+0000","last_undegraded":"2024-02-05T07:24:02.548126+0000","last_fullsized":"2024-02-05T07:24:02.548126+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:03.505163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:23:03.505163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:32.119390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"173'11041","reported_seq":15805,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:04.932863+0000","last_change":"2024-02-05T07:24:02.542365+0000","last_active":"2024-02-05T07:24:04.932863+0000","last_peered":"2024-02-05T07:24:04.932863+0000","last_clean":"2024-02-05T07:24:04.932863+0000","last_became_active":"2024-02-05T07:24:02.515998+0000","last_became_peered":"2024-02-05T07:24:02.515998+0000","last_unstale":"2024-02-05T07:24:04.932863+0000","last_undegraded":"2024-02-05T07:24:04.932863+0000","last_fullsized":"2024-02-05T07:24:04.932863+0000","mapping_epoch":172,"log_start":"131'6521","ondisk_log_start":"131'6521","created":100,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":4520,"log_dups_size":0,"ondisk_log_size":4520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129957888,"num_objects":1983,"num_object_clones":0,"num_object_copies":5949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11041,"num_write_kb":416768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2293,"num_bytes_recovered":150142976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":218,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.513378+0000","last_change":"2024-02-05T07:20:30.285373+0000","last_active":"2024-02-05T07:24:02.513378+0000","last_peered":"2024-02-05T07:24:02.513378+0000","last_clean":"2024-02-05T07:24:02.513378+0000","last_became_active":"2024-02-05T07:18:55.472172+0000","last_became_peered":"2024-02-05T07:18:55.472172+0000","last_unstale":"2024-02-05T07:24:02.513378+0000","last_undegraded":"2024-02-05T07:24:02.513378+0000","last_fullsized":"2024-02-05T07:24:02.513378+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:30.285296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:30.285296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":294,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.549155+0000","last_change":"2024-02-05T07:21:14.009133+0000","last_active":"2024-02-05T07:24:02.549155+0000","last_peered":"2024-02-05T07:24:02.549155+0000","last_clean":"2024-02-05T07:24:02.549155+0000","last_became_active":"2024-02-05T07:18:53.459832+0000","last_became_peered":"2024-02-05T07:18:53.459832+0000","last_unstale":"2024-02-05T07:24:02.549155+0000","last_undegraded":"2024-02-05T07:24:02.549155+0000","last_fullsized":"2024-02-05T07:24:02.549155+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:21:14.008994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:14.008994+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":32,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.382378+0000","last_change":"2024-02-05T07:23:47.600893+0000","last_active":"2024-02-05T07:24:02.382378+0000","last_peered":"2024-02-05T07:24:02.382378+0000","last_clean":"2024-02-05T07:24:02.382378+0000","last_became_active":"2024-02-05T07:23:47.600273+0000","last_became_peered":"2024-02-05T07:23:47.600273+0000","last_unstale":"2024-02-05T07:24:02.382378+0000","last_undegraded":"2024-02-05T07:24:02.382378+0000","last_fullsized":"2024-02-05T07:24:02.382378+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:37.367711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.367711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"98'22","reported_seq":289,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.383004+0000","last_change":"2024-02-05T07:23:47.637874+0000","last_active":"2024-02-05T07:24:02.383004+0000","last_peered":"2024-02-05T07:24:02.383004+0000","last_clean":"2024-02-05T07:24:02.383004+0000","last_became_active":"2024-02-05T07:23:47.606483+0000","last_became_peered":"2024-02-05T07:23:47.606483+0000","last_unstale":"2024-02-05T07:24:02.383004+0000","last_undegraded":"2024-02-05T07:24:02.383004+0000","last_fullsized":"2024-02-05T07:24:02.383004+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'22","last_scrub_stamp":"2024-02-05T07:23:19.000981+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:23:19.000981+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:30.391160+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"173'11163","reported_seq":11324,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:04.929543+0000","last_change":"2024-02-05T07:20:45.765039+0000","last_active":"2024-02-05T07:24:04.929543+0000","last_peered":"2024-02-05T07:24:04.929543+0000","last_clean":"2024-02-05T07:24:04.929543+0000","last_became_active":"2024-02-05T07:20:45.764732+0000","last_became_peered":"2024-02-05T07:20:45.764732+0000","last_unstale":"2024-02-05T07:24:04.929543+0000","last_undegraded":"2024-02-05T07:24:04.929543+0000","last_fullsized":"2024-02-05T07:24:04.929543+0000","mapping_epoch":100,"log_start":"131'6624","ondisk_log_start":"131'6624","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":4539,"log_dups_size":0,"ondisk_log_size":4539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128581656,"num_objects":1963,"num_object_clones":0,"num_object_copies":5889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1963,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11163,"num_write_kb":419969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":32,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.512740+0000","last_change":"2024-02-05T07:23:47.612589+0000","last_active":"2024-02-05T07:24:02.512740+0000","last_peered":"2024-02-05T07:24:02.512740+0000","last_clean":"2024-02-05T07:24:02.512740+0000","last_became_active":"2024-02-05T07:23:47.611030+0000","last_became_peered":"2024-02-05T07:23:47.611030+0000","last_unstale":"2024-02-05T07:24:02.512740+0000","last_undegraded":"2024-02-05T07:24:02.512740+0000","last_fullsized":"2024-02-05T07:24:02.512740+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:10.763957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:10.763957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":52,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.549255+0000","last_change":"2024-02-05T07:23:27.417952+0000","last_active":"2024-02-05T07:24:02.549255+0000","last_peered":"2024-02-05T07:24:02.549255+0000","last_clean":"2024-02-05T07:24:02.549255+0000","last_became_active":"2024-02-05T07:23:27.417675+0000","last_became_peered":"2024-02-05T07:23:27.417675+0000","last_unstale":"2024-02-05T07:24:02.549255+0000","last_undegraded":"2024-02-05T07:24:02.549255+0000","last_fullsized":"2024-02-05T07:24:02.549255+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:23.823835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:23.823835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":238,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548202+0000","last_change":"2024-02-05T07:23:02.519644+0000","last_active":"2024-02-05T07:24:02.548202+0000","last_peered":"2024-02-05T07:24:02.548202+0000","last_clean":"2024-02-05T07:24:02.548202+0000","last_became_active":"2024-02-05T07:18:49.434074+0000","last_became_peered":"2024-02-05T07:18:49.434074+0000","last_unstale":"2024-02-05T07:24:02.548202+0000","last_undegraded":"2024-02-05T07:24:02.548202+0000","last_fullsized":"2024-02-05T07:24:02.548202+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:02.519566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:02.519566+0000","last_clean_scrub_stamp":"2024-02-05T07:23:02.519566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:31.730136+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"90'16","reported_seq":258,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548941+0000","last_change":"2024-02-05T07:21:43.811840+0000","last_active":"2024-02-05T07:24:02.548941+0000","last_peered":"2024-02-05T07:24:02.548941+0000","last_clean":"2024-02-05T07:24:02.548941+0000","last_became_active":"2024-02-05T07:18:51.447922+0000","last_became_peered":"2024-02-05T07:18:51.447922+0000","last_unstale":"2024-02-05T07:24:02.548941+0000","last_undegraded":"2024-02-05T07:24:02.548941+0000","last_fullsized":"2024-02-05T07:24:02.548941+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-02-05T07:21:43.811596+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.811596+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"173'10884","reported_seq":11044,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:05.422415+0000","last_change":"2024-02-05T07:20:45.771675+0000","last_active":"2024-02-05T07:24:05.422415+0000","last_peered":"2024-02-05T07:24:05.422415+0000","last_clean":"2024-02-05T07:24:05.422415+0000","last_became_active":"2024-02-05T07:20:45.771415+0000","last_became_peered":"2024-02-05T07:20:45.771415+0000","last_unstale":"2024-02-05T07:24:05.422415+0000","last_undegraded":"2024-02-05T07:24:05.422415+0000","last_fullsized":"2024-02-05T07:24:05.422415+0000","mapping_epoch":100,"log_start":"131'6184","ondisk_log_start":"131'6184","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129499136,"num_objects":1976,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10884,"num_write_kb":411520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":224,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.383989+0000","last_change":"2024-02-05T07:23:47.609782+0000","last_active":"2024-02-05T07:24:02.383989+0000","last_peered":"2024-02-05T07:24:02.383989+0000","last_clean":"2024-02-05T07:24:02.383989+0000","last_became_active":"2024-02-05T07:23:47.609458+0000","last_became_peered":"2024-02-05T07:23:47.609458+0000","last_unstale":"2024-02-05T07:24:02.383989+0000","last_undegraded":"2024-02-05T07:24:02.383989+0000","last_fullsized":"2024-02-05T07:24:02.383989+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:29.670866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:29.670866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":231,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384337+0000","last_change":"2024-02-05T07:21:17.402214+0000","last_active":"2024-02-05T07:24:02.384337+0000","last_peered":"2024-02-05T07:24:02.384337+0000","last_clean":"2024-02-05T07:24:02.384337+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:24:02.384337+0000","last_undegraded":"2024-02-05T07:24:02.384337+0000","last_fullsized":"2024-02-05T07:24:02.384337+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:17.401959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:17.401959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.384810+0000","last_change":"2024-02-05T07:20:18.399191+0000","last_active":"2024-02-05T07:24:02.384810+0000","last_peered":"2024-02-05T07:24:02.384810+0000","last_clean":"2024-02-05T07:24:02.384810+0000","last_became_active":"2024-02-05T07:18:49.430518+0000","last_became_peered":"2024-02-05T07:18:49.430518+0000","last_unstale":"2024-02-05T07:24:02.384810+0000","last_undegraded":"2024-02-05T07:24:02.384810+0000","last_fullsized":"2024-02-05T07:24:02.384810+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:18.399075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:18.399075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":262,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.549251+0000","last_change":"2024-02-05T07:21:29.926437+0000","last_active":"2024-02-05T07:24:02.549251+0000","last_peered":"2024-02-05T07:24:02.549251+0000","last_clean":"2024-02-05T07:24:02.549251+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:24:02.549251+0000","last_undegraded":"2024-02-05T07:24:02.549251+0000","last_fullsized":"2024-02-05T07:24:02.549251+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:21:29.926339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:29.926339+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"173'11186","reported_seq":11353,"reported_epoch":173,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:24:05.667367+0000","last_change":"2024-02-05T07:23:48.776967+0000","last_active":"2024-02-05T07:24:05.667367+0000","last_peered":"2024-02-05T07:24:05.667367+0000","last_clean":"2024-02-05T07:23:46.587406+0000","last_became_active":"2024-02-05T07:23:48.681890+0000","last_became_peered":"2024-02-05T07:23:48.681890+0000","last_unstale":"2024-02-05T07:24:05.667367+0000","last_undegraded":"2024-02-05T07:24:05.667367+0000","last_fullsized":"2024-02-05T07:24:05.667367+0000","mapping_epoch":163,"log_start":"133'6705","ondisk_log_start":"133'6705","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130154496,"num_objects":1986,"num_object_clones":0,"num_object_copies":5958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1986,"num_objects_unfound":0,"num_objects_dirty":1986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11186,"num_write_kb":421504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":1986}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":226,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.549199+0000","last_change":"2024-02-05T07:23:09.542975+0000","last_active":"2024-02-05T07:24:02.549199+0000","last_peered":"2024-02-05T07:24:02.549199+0000","last_clean":"2024-02-05T07:24:02.549199+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:24:02.549199+0000","last_undegraded":"2024-02-05T07:24:02.549199+0000","last_fullsized":"2024-02-05T07:24:02.549199+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:09.542903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:09.542903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:11.586742+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":305,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.382447+0000","last_change":"2024-02-05T07:21:27.645530+0000","last_active":"2024-02-05T07:24:02.382447+0000","last_peered":"2024-02-05T07:24:02.382447+0000","last_clean":"2024-02-05T07:24:02.382447+0000","last_became_active":"2024-02-05T07:18:53.461852+0000","last_became_peered":"2024-02-05T07:18:53.461852+0000","last_unstale":"2024-02-05T07:24:02.382447+0000","last_undegraded":"2024-02-05T07:24:02.382447+0000","last_fullsized":"2024-02-05T07:24:02.382447+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:21:27.645263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:27.645263+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":32,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.548382+0000","last_change":"2024-02-05T07:23:47.607003+0000","last_active":"2024-02-05T07:24:02.548382+0000","last_peered":"2024-02-05T07:24:02.548382+0000","last_clean":"2024-02-05T07:24:02.548382+0000","last_became_active":"2024-02-05T07:23:47.606691+0000","last_became_peered":"2024-02-05T07:23:47.606691+0000","last_unstale":"2024-02-05T07:24:02.548382+0000","last_undegraded":"2024-02-05T07:24:02.548382+0000","last_fullsized":"2024-02-05T07:24:02.548382+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:04.827180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:04.827180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":267,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-05T07:24:02.549082+0000","last_change":"2024-02-05T07:23:47.629122+0000","last_active":"2024-02-05T07:24:02.549082+0000","last_peered":"2024-02-05T07:24:02.549082+0000","last_clean":"2024-02-05T07:24:02.549082+0000","last_became_active":"2024-02-05T07:23:47.597686+0000","last_became_peered":"2024-02-05T07:23:47.597686+0000","last_unstale":"2024-02-05T07:24:02.549082+0000","last_undegraded":"2024-02-05T07:24:02.549082+0000","last_fullsized":"2024-02-05T07:24:02.549082+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:23:04.538922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:04.538922+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:15.661571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":2103574552,"num_objects":32099,"num_object_clones":0,"num_object_copies":96297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":3225,"num_objects_unfound":0,"num_objects_dirty":32099,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112586,"num_write_kb":3509505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16801,"num_bytes_recovered":1100742656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6902591488,"data_stored":6902579272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138162,"ondisk_log_size":138162,"up":78,"acting":77,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":303,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":692,"num_read_kb":449,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":2338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037828,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1257832,"kb_used_data":1055840,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92487320,"statfs":{"total":95995035648,"available":94707015680,"internally_reserved":0,"allocated":1081180160,"data_stored":1080820060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":206764490},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":6.8929999999999998,"5min":2.9369999999999998,"15min":1.3089999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":63.039000000000001,"5min":63.039000000000001,"15min":63.039000000000001},"last":10.901},{"interface":"front","average":{"1min":6.6950000000000003,"5min":2.8119999999999998,"15min":1.2769999999999999},"min":{"1min":0.28699999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":65.296000000000006,"5min":65.296000000000006,"15min":65.296000000000006},"last":10.989000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":3.8210000000000002,"5min":1.4079999999999999,"15min":0.81899999999999995},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":32.127000000000002,"5min":32.127000000000002,"15min":32.127000000000002},"last":5.3200000000000003},{"interface":"front","average":{"1min":4.6710000000000003,"5min":1.5329999999999999,"15min":0.88},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":32.173999999999999,"5min":32.173999999999999,"15min":32.173999999999999},"last":5.1529999999999996}]},{"osd":2,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":2.6499999999999999,"5min":1.8169999999999999,"15min":0.95499999999999996},"min":{"1min":0.26300000000000001,"5min":0.249,"15min":0.249},"max":{"1min":32.262999999999998,"5min":51.127000000000002,"15min":51.127000000000002},"last":1.8},{"interface":"front","average":{"1min":3.3010000000000002,"5min":2.1680000000000001,"15min":1.0669999999999999},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":32.296999999999997,"5min":51.113999999999997,"15min":51.113999999999997},"last":2.3799999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.7769999999999999,"15min":0.94799999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.1349999999999998,"5min":58.341999999999999,"15min":58.341999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":1.073,"5min":1.6100000000000001,"15min":0.90600000000000003},"min":{"1min":0.311,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.8730000000000002,"5min":57.765000000000001,"15min":57.765000000000001},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.45900000000000002,"15min":0.44500000000000001},"min":{"1min":0.27000000000000002,"5min":0.25,"15min":0.25},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.308},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.48999999999999999,"15min":0.45500000000000002},"min":{"1min":0.28899999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":0.41999999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.49199999999999999,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.3940000000000001,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.56299999999999994,"15min":0.51800000000000002},"min":{"1min":0.34899999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.8679999999999999,"5min":2.8679999999999999,"15min":2.8679999999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.56000000000000005,"15min":0.51700000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.372},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.55100000000000005,"15min":0.52400000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.97,"5min":1.97,"15min":1.97},"last":0.45400000000000001}]}]},{"osd":6,"up_from":52,"seq":223338299466,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1034504,"kb_used_data":854784,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92710648,"statfs":{"total":95995035648,"available":94935703552,"internally_reserved":0,"allocated":875298816,"data_stored":874907050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":184023494},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:23:16 2024","interfaces":[{"interface":"back","average":{"1min":8.9429999999999996,"5min":2.536,"15min":1.212},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":57.289999999999999,"5min":57.289999999999999,"15min":57.289999999999999},"last":4.1260000000000003},{"interface":"front","average":{"1min":8.4649999999999999,"5min":2.4049999999999998,"15min":1.1539999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":56.302,"5min":56.302,"15min":56.302},"last":1.327}]},{"osd":1,"last update":"Mon Feb 5 07:23:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.71799999999999997,"15min":0.60599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.0890000000000004,"5min":4.9560000000000004,"15min":4.9560000000000004},"last":1.2150000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.71899999999999997,"15min":0.60899999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.1669999999999998,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":1.587}]},{"osd":2,"last update":"Mon Feb 5 07:23:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6800000000000002,"5min":1.427,"15min":0.86199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":32.509999999999998,"5min":32.509999999999998,"15min":32.509999999999998},"last":2.7160000000000002},{"interface":"front","average":{"1min":4.3120000000000003,"5min":1.532,"15min":0.86499999999999999},"min":{"1min":0.36399999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"last":1.407}]},{"osd":3,"last update":"Mon Feb 5 07:23:16 2024","interfaces":[{"interface":"back","average":{"1min":4.6449999999999996,"5min":1.6419999999999999,"15min":0.92000000000000004},"min":{"1min":0.33500000000000002,"5min":0.22,"15min":0.22},"max":{"1min":36.976999999999997,"5min":36.976999999999997,"15min":36.976999999999997},"last":1.2869999999999999},{"interface":"front","average":{"1min":4.0250000000000004,"5min":1.536,"15min":0.88800000000000001},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":34.222999999999999,"5min":34.222999999999999,"15min":34.222999999999999},"last":1.6439999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:23:16 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.45700000000000002,"15min":0.41899999999999998},"min":{"1min":0.28199999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.84199999999999997,"5min":1.621,"15min":1.621},"last":1.351},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.46200000000000002,"15min":0.45400000000000001},"min":{"1min":0.32600000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.754,"5min":1.254,"15min":1.254},"last":1.3109999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:23:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47199999999999998,"15min":0.45400000000000001},"min":{"1min":0.33400000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76600000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":1.508},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.46800000000000003},"min":{"1min":0.33900000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82799999999999996,"5min":1.393,"15min":1.393},"last":2.351}]},{"osd":7,"last update":"Mon Feb 5 07:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.35399999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.6200000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54500000000000004,"15min":0.54200000000000004},"min":{"1min":0.36499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.88500000000000001,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":2.798}]}]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1337552,"kb_used_data":1171720,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92407600,"statfs":{"total":95995035648,"available":94625382400,"internally_reserved":0,"allocated":1199841280,"data_stored":1199440991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:23:28 2024","interfaces":[{"interface":"back","average":{"1min":7.952,"5min":3.4119999999999999,"15min":1.359},"min":{"1min":0.69599999999999995,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":30.536999999999999,"5min":45.393000000000001,"15min":45.393000000000001},"last":0.223},{"interface":"front","average":{"1min":8.6069999999999993,"5min":3.407,"15min":1.355},"min":{"1min":0.26100000000000001,"5min":0.23000000000000001,"15min":0.22800000000000001},"max":{"1min":31.853999999999999,"5min":45.031999999999996,"15min":45.031999999999996},"last":0.246}]},{"osd":2,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":2.04,"15min":0.97899999999999998},"min":{"1min":0.22700000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.3380000000000001,"5min":39.259999999999998,"15min":39.259999999999998},"last":0.29999999999999999},{"interface":"front","average":{"1min":2.4119999999999999,"5min":1.911,"15min":1.3720000000000001},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.7160000000000002,"5min":40.972000000000001,"15min":40.972000000000001},"last":0.35799999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:23:07 2024","interfaces":[{"interface":"back","average":{"1min":4.9349999999999996,"5min":2.0960000000000001,"15min":1.0209999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":20.446000000000002,"5min":43.823,"15min":43.823},"last":0.28799999999999998},{"interface":"front","average":{"1min":4.0590000000000002,"5min":1.901,"15min":0.96099999999999997},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":21.228000000000002,"5min":39.314999999999998,"15min":39.314999999999998},"last":0.32200000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:23:28 2024","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.0069999999999999,"15min":0.72899999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":5.9640000000000004,"5min":5.9640000000000004,"15min":5.9640000000000004},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.204,"5min":1.081,"15min":0.87},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":5.2110000000000003,"5min":17.960999999999999,"15min":17.960999999999999},"last":0.33600000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.7549999999999999,"5min":1.131,"15min":0.79300000000000004},"min":{"1min":0.33600000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":4.9199999999999999,"5min":5.9329999999999998,"15min":5.9329999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":2.2029999999999998,"5min":1.2869999999999999,"15min":0.95799999999999996},"min":{"1min":0.38100000000000001,"5min":0.307,"15min":0.307},"max":{"1min":12.353999999999999,"5min":18.425999999999998,"15min":18.425999999999998},"last":0.502}]},{"osd":6,"last update":"Mon Feb 5 07:23:28 2024","interfaces":[{"interface":"back","average":{"1min":2.7069999999999999,"5min":1.21,"15min":0.86199999999999999},"min":{"1min":0.36899999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":18.456,"5min":18.456,"15min":18.456},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.569,"5min":1.0549999999999999,"15min":0.80800000000000005},"min":{"1min":0.39100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.9930000000000003,"5min":5.9930000000000003,"15min":5.9930000000000003},"last":0.42899999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":2.1219999999999999,"5min":1.296,"15min":0.81599999999999995},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":12.305999999999999,"5min":18.513000000000002,"15min":18.513000000000002},"last":0.371},{"interface":"front","average":{"1min":1.7989999999999999,"5min":1.2130000000000001,"15min":0.79900000000000004},"min":{"1min":0.35599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":5.165,"5min":18.117999999999999,"15min":18.117999999999999},"last":0.41599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1428276,"kb_used_data":1226284,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92316876,"statfs":{"total":95995035648,"available":94532481024,"internally_reserved":0,"allocated":1255714816,"data_stored":1255342593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:23:16 2024","interfaces":[{"interface":"back","average":{"1min":2.4569999999999999,"5min":1.4059999999999999,"15min":0.66900000000000004},"min":{"1min":0.247,"5min":0.20499999999999999,"15min":0.17599999999999999},"max":{"1min":9.7309999999999999,"5min":15.731999999999999,"15min":15.731999999999999},"last":0.16900000000000001},{"interface":"front","average":{"1min":2.1080000000000001,"5min":1.601,"15min":0.746},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.214},"max":{"1min":8.0139999999999993,"5min":32.396000000000001,"15min":32.396000000000001},"last":0.186}]},{"osd":2,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":2.8639999999999999,"5min":1.6890000000000001,"15min":0.84699999999999998},"min":{"1min":0.28399999999999997,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":9.1440000000000001,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":65.903999999999996},{"interface":"front","average":{"1min":2.2519999999999998,"5min":1.583,"15min":0.80800000000000005},"min":{"1min":0.19700000000000001,"5min":0.182,"15min":0.182},"max":{"1min":9.5150000000000006,"5min":10.75,"15min":10.75},"last":65.694000000000003}]},{"osd":3,"last update":"Mon Feb 5 07:23:04 2024","interfaces":[{"interface":"back","average":{"1min":3.117,"5min":1.8540000000000001,"15min":0.90600000000000003},"min":{"1min":0.26500000000000001,"5min":0.246,"15min":0.246},"max":{"1min":10.257999999999999,"5min":25.137,"15min":25.137},"last":65.155000000000001},{"interface":"front","average":{"1min":3.9830000000000001,"5min":2.1280000000000001,"15min":0.98999999999999999},"min":{"1min":0.58099999999999996,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":15.896000000000001,"5min":22.949000000000002,"15min":22.949000000000002},"last":65.114000000000004}]},{"osd":4,"last update":"Mon Feb 5 07:23:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.28,"15min":0.79400000000000004},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":5.181,"5min":8.5709999999999997,"15min":8.5709999999999997},"last":65.188000000000002},{"interface":"front","average":{"1min":1.843,"5min":1.387,"15min":0.82799999999999996},"min":{"1min":0.433,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":8.0679999999999996,"5min":10.103,"15min":10.103},"last":64.873000000000005}]},{"osd":5,"last update":"Mon Feb 5 07:24:03 2024","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":1.8959999999999999,"15min":1},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":65.555999999999997,"5min":65.555999999999997,"15min":65.555999999999997},"last":65.555999999999997},{"interface":"front","average":{"1min":5.2930000000000001,"5min":2.0830000000000002,"15min":1.085},"min":{"1min":0.436,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":64.936999999999998,"5min":64.936999999999998,"15min":64.936999999999998},"last":64.936999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:23:22 2024","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":1.214,"15min":0.80200000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":11.888999999999999,"5min":11.888999999999999,"15min":11.888999999999999},"last":65.215000000000003},{"interface":"front","average":{"1min":1.8340000000000001,"5min":1.337,"15min":0.873},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":5.2080000000000002,"5min":8.6329999999999991,"15min":8.6329999999999991},"last":65.230000000000004}]},{"osd":7,"last update":"Mon Feb 5 07:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.45,"15min":0.88800000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":5.3730000000000002,"5min":8.6679999999999993,"15min":8.6679999999999993},"last":65.263000000000005},{"interface":"front","average":{"1min":1.8620000000000001,"5min":1.3999999999999999,"15min":0.89600000000000002},"min":{"1min":0.46999999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":8.1940000000000008,"5min":8.6829999999999998,"15min":8.6829999999999998},"last":64.906000000000006}]}]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1398180,"kb_used_data":1186524,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92346972,"statfs":{"total":95995035648,"available":94563299328,"internally_reserved":0,"allocated":1215000576,"data_stored":1214644515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:23:45 2024","interfaces":[{"interface":"back","average":{"1min":4.9640000000000004,"5min":2.1760000000000002,"15min":0.97099999999999997},"min":{"1min":0.19500000000000001,"5min":0.192,"15min":0.192},"max":{"1min":67.418999999999997,"5min":67.418999999999997,"15min":67.418999999999997},"last":1.804},{"interface":"front","average":{"1min":6.7030000000000003,"5min":2.4660000000000002,"15min":1.0780000000000001},"min":{"1min":0.34200000000000003,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":68.171000000000006,"5min":68.171000000000006,"15min":68.171000000000006},"last":0.29999999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.2230000000000001,"15min":0.66600000000000004},"min":{"1min":0.20699999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.9500000000000002,"5min":12.609,"15min":12.609},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.635,"5min":1.3320000000000001,"15min":0.72699999999999998},"min":{"1min":0.26800000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":5.984,"5min":9.641,"15min":9.641},"last":1.764}]},{"osd":3,"last update":"Mon Feb 5 07:23:06 2024","interfaces":[{"interface":"back","average":{"1min":2.4249999999999998,"5min":1.3680000000000001,"15min":0.76900000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":9.6790000000000003,"5min":14.756,"15min":14.756},"last":0.318},{"interface":"front","average":{"1min":2.7250000000000001,"5min":1.4710000000000001,"15min":0.79900000000000004},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":8.5370000000000008,"5min":9.6669999999999998,"15min":9.6669999999999998},"last":1.8360000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:23:30 2024","interfaces":[{"interface":"back","average":{"1min":1.8400000000000001,"5min":1.077,"15min":0.71099999999999997},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.29,"5min":7.29,"15min":7.29},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.115,"15min":0.73799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":5.3019999999999996,"5min":8.5199999999999996,"15min":8.5199999999999996},"last":1.9410000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.1779999999999999,"15min":0.78800000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.1019999999999999,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":1.972},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.1040000000000001,"15min":0.77000000000000002},"min":{"1min":0.30599999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.9029999999999996,"5min":7.7279999999999998,"15min":7.7279999999999998},"last":0.55300000000000005}]},{"osd":6,"last update":"Mon Feb 5 07:23:19 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.1080000000000001,"15min":0.88400000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.8929999999999998,"5min":7.6829999999999998,"15min":7.6829999999999998},"last":0.623},{"interface":"front","average":{"1min":1.9970000000000001,"5min":1.179,"15min":0.80200000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":7.3150000000000004,"5min":7.3150000000000004,"15min":7.3150000000000004},"last":0.60899999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:23:45 2024","interfaces":[{"interface":"back","average":{"1min":2.3330000000000002,"5min":1.2529999999999999,"15min":0.88300000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":7.2590000000000003,"5min":7.2590000000000003,"15min":7.2590000000000003},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.115,"5min":1.089,"15min":0.79100000000000004},"min":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.8039999999999998,"5min":7.7619999999999996,"15min":7.7619999999999996},"last":0.59499999999999997}]}]},{"osd":3,"up_from":0,"seq":111669149784,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953555,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":499752,"kb_used_data":392224,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93245400,"statfs":{"total":95995035648,"available":95483289600,"internally_reserved":0,"allocated":401637376,"data_stored":401276176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:23:28 2024","interfaces":[{"interface":"back","average":{"1min":4.7300000000000004,"5min":1.5720000000000001,"15min":0.878},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":67.719999999999999,"5min":67.719999999999999,"15min":67.719999999999999},"last":0.311},{"interface":"front","average":{"1min":5.6180000000000003,"5min":1.8109999999999999,"15min":0.97499999999999998},"min":{"1min":0.40799999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":66.882000000000005,"5min":66.882000000000005,"15min":66.882000000000005},"last":0.252}]},{"osd":1,"last update":"Mon Feb 5 07:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.81200000000000006,"15min":0.61799999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.69699999999999995,"15min":0.60399999999999998},"min":{"1min":0.28799999999999998,"5min":0.254,"15min":0.254},"max":{"1min":3.4609999999999999,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":0.38700000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:23:28 2024","interfaces":[{"interface":"back","average":{"1min":4.5439999999999996,"5min":1.4530000000000001,"15min":0.84699999999999998},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":67.144999999999996,"5min":67.144999999999996,"15min":67.144999999999996},"last":0.27500000000000002},{"interface":"front","average":{"1min":4.6299999999999999,"5min":1.5169999999999999,"15min":0.86599999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":66.959999999999994,"5min":66.959999999999994,"15min":66.959999999999994},"last":0.48799999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.85299999999999998,"15min":0.66000000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":2.6259999999999999,"5min":8.8019999999999996,"15min":8.8019999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.94599999999999995,"15min":0.69099999999999995},"min":{"1min":0.28499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.8779999999999999,"5min":10.106,"15min":10.106},"last":0.46400000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.45200000000000001,"15min":0.39100000000000001},"min":{"1min":0.28199999999999997,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.4930000000000001,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.5,"15min":0.41499999999999998},"min":{"1min":0.318,"5min":0.254,"15min":0.254},"max":{"1min":2.1859999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":0.40799999999999997}]},{"osd":6,"last update":"Mon Feb 5 07:23:22 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50700000000000001,"15min":0.441},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.57399999999999995,"15min":0.45700000000000002},"min":{"1min":0.311,"5min":0.248,"15min":0.248},"max":{"1min":2.9209999999999998,"5min":2.9209999999999998,"15min":2.9209999999999998},"last":0.52600000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.57799999999999996,"15min":0.52100000000000002},"min":{"1min":0.38200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.4089999999999998,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.52400000000000002,"15min":0.496},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.581,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":0.44900000000000001}]}]},{"osd":5,"up_from":41,"seq":176093659214,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":812132,"kb_used_data":672668,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92933020,"statfs":{"total":95995035648,"available":95163412480,"internally_reserved":0,"allocated":688812032,"data_stored":688455389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:23:51 2024","interfaces":[{"interface":"back","average":{"1min":3.5550000000000002,"5min":3.2189999999999999,"15min":1.429},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":45.073,"5min":69.855999999999995,"15min":69.855999999999995},"last":5.2039999999999997},{"interface":"front","average":{"1min":3.7370000000000001,"5min":3.0249999999999999,"15min":1.365},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":47.031999999999996,"5min":71.513999999999996,"15min":71.513999999999996},"last":0.63800000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:23:51 2024","interfaces":[{"interface":"back","average":{"1min":1.7909999999999999,"5min":1.0900000000000001,"15min":0.74199999999999999},"min":{"1min":0.26100000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":11.839,"5min":11.839,"15min":11.839},"last":5.1219999999999999},{"interface":"front","average":{"1min":1.2809999999999999,"5min":0.77900000000000003,"15min":0.61699999999999999},"min":{"1min":0.27400000000000002,"5min":0.253,"15min":0.253},"max":{"1min":9.5950000000000006,"5min":9.5950000000000006,"15min":9.5950000000000006},"last":0.66600000000000004}]},{"osd":2,"last update":"Mon Feb 5 07:23:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.887,"15min":0.98299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":9.5419999999999998,"5min":70.108000000000004,"15min":70.108000000000004},"last":0.379},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.9610000000000001,"15min":1.0209999999999999},"min":{"1min":0.315,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.9340000000000002,"5min":72.817999999999998,"15min":72.817999999999998},"last":5.1740000000000004}]},{"osd":3,"last update":"Mon Feb 5 07:23:51 2024","interfaces":[{"interface":"back","average":{"1min":3.7480000000000002,"5min":2.0859999999999999,"15min":1.0589999999999999},"min":{"1min":0.313,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":47.408000000000001,"5min":69.793000000000006,"15min":69.793000000000006},"last":0.34899999999999998},{"interface":"front","average":{"1min":3.9609999999999999,"5min":2.3250000000000002,"15min":1.143},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":45.140999999999998,"5min":69.820999999999998,"15min":69.820999999999998},"last":5.2320000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.443,"15min":0.375},"min":{"1min":0.254,"5min":0.24099999999999999,"15min":0.23599999999999999},"max":{"1min":1.367,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.48799999999999999,"15min":0.40600000000000003},"min":{"1min":0.26200000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.518,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":0.309}]},{"osd":6,"last update":"Mon Feb 5 07:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":1.0549999999999999,"15min":0.61299999999999999},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.0939999999999999,"5min":51.566000000000003,"15min":51.566000000000003},"last":2.0790000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":1.087,"15min":0.65700000000000003},"min":{"1min":0.318,"5min":0.313,"15min":0.313},"max":{"1min":1.956,"5min":51.616,"15min":51.616},"last":0.41099999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":2.3860000000000001,"15min":1.1359999999999999},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.8260000000000001,"5min":182.87799999999999,"15min":182.87799999999999},"last":5.2610000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":2.3690000000000002,"15min":1.135},"min":{"1min":0.33600000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.3999999999999999,"5min":183.02199999999999,"15min":183.02199999999999},"last":5.3070000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1236668416,"data_stored":1236664344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1177554944,"data_stored":1177550872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1200750592,"data_stored":1200750592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285409280,"data_stored":285409280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":394985472,"data_stored":394985472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":687079424,"data_stored":687079424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":859770880,"data_stored":859766808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1060372480,"data_stored":1060372480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:24:09.951 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-05T07:24:09.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set unique_pool_0 pg_num 36 2024-02-05T07:24:09.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:09.952 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:09.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:10.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:10.078 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:10.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:10.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:10.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:10.212 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:10.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:10.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:10.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:10.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:10.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:10.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:10.483 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:10.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:10.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:10.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:24:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:10 smithi090 ceph-mon[33213]: pgmap v231: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.0 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 2/96891 objects degraded (0.002%); 3225/96891 objects misplaced (3.328%) 2024-02-05T07:24:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:10 smithi090 ceph-mon[33213]: 6.1b scrub starts 2024-02-05T07:24:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:10 smithi090 ceph-mon[33213]: 6.1b scrub ok 2024-02-05T07:24:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:10 smithi090 ceph-mon[33213]: osdmap e174: 8 total, 8 up, 7 in 2024-02-05T07:24:10.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:24:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:24:10.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:10.642 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:10.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:10.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:10.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:10 smithi007 ceph-mon[37158]: pgmap v231: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.0 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 2/96891 objects degraded (0.002%); 3225/96891 objects misplaced (3.328%) 2024-02-05T07:24:10.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:10 smithi007 ceph-mon[37158]: 6.1b scrub starts 2024-02-05T07:24:10.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:10 smithi007 ceph-mon[37158]: 6.1b scrub ok 2024-02-05T07:24:10.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:10 smithi007 ceph-mon[37158]: osdmap e174: 8 total, 8 up, 7 in 2024-02-05T07:24:10.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:10 smithi007 ceph-mon[29098]: pgmap v231: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.0 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 2/96891 objects degraded (0.002%); 3225/96891 objects misplaced (3.328%) 2024-02-05T07:24:10.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:10 smithi007 ceph-mon[29098]: 6.1b scrub starts 2024-02-05T07:24:10.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:10 smithi007 ceph-mon[29098]: 6.1b scrub ok 2024-02-05T07:24:10.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:10 smithi007 ceph-mon[29098]: osdmap e174: 8 total, 8 up, 7 in 2024-02-05T07:24:10.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:10.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:10.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:10.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:10.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:10.903 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:10.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:10.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:11.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:11.042 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:11.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:11.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:11.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:11.167 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:11.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:11.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:11.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:11.293 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:11.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:11.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:11.394 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:11.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:11.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:11.537 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:11.560 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:11.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:11.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:11.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:11 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:11.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:11 smithi007 ceph-mon[29098]: 3.1a scrub starts 2024-02-05T07:24:11.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:11 smithi007 ceph-mon[29098]: 3.1a scrub ok 2024-02-05T07:24:11.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:11 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:11.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:11 smithi007 ceph-mon[37158]: 3.1a scrub starts 2024-02-05T07:24:11.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:11 smithi007 ceph-mon[37158]: 3.1a scrub ok 2024-02-05T07:24:11.698 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:11.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:11.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:11.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:11 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:11.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:11 smithi090 ceph-mon[33213]: 3.1a scrub starts 2024-02-05T07:24:11.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:11 smithi090 ceph-mon[33213]: 3.1a scrub ok 2024-02-05T07:24:11.795 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:11.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:11.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:11.833 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:11.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:11.834 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:11.934 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:11.978 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:11.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:12.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:12.103 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:12.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:12.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:12.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:12.228 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:12.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:12.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:12.330 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:12.371 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:12.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:12.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:12.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:12.509 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:12.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:12.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:12.636 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:12 smithi007 ceph-mon[29098]: pgmap v233: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.7 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/83544 objects degraded (0.002%); 2620/83544 objects misplaced (3.136%) 2024-02-05T07:24:12.636 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:12 smithi007 ceph-mon[37158]: pgmap v233: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.7 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/83544 objects degraded (0.002%); 2620/83544 objects misplaced (3.136%) 2024-02-05T07:24:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:12.647 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:12.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:12.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:12 smithi090 ceph-mon[33213]: pgmap v233: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.7 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2/83544 objects degraded (0.002%); 2620/83544 objects misplaced (3.136%) 2024-02-05T07:24:12.788 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:12.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:12.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:12.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:12.938 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:12.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:13.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:13.084 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:13.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:13.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:13.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:13.225 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:13.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:13.290 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:24:13.290 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:24:13.325 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:13.366 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:13.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:13.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:13.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:13.501 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:13.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:13.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:13.539 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:13 smithi007 ceph-mon[29098]: 3.b scrub starts 2024-02-05T07:24:13.539 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:13 smithi007 ceph-mon[37158]: 3.b scrub starts 2024-02-05T07:24:13.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:13.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:13.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:13.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:13.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:13.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:13 smithi090 ceph-mon[33213]: 3.b scrub starts 2024-02-05T07:24:13.783 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:13.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:13.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:13.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:13.918 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:13.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:13.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:24:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:24:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:24:14.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:14.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:14.056 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:14.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:14.157 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:14.194 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:14.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:14.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:14.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T07:24:14.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:14.334 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:14.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:14.436 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:14.471 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:24:14.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:14.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:14.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:14.602 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:24:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:14.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:14.649 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:14 smithi007 ceph-mon[29098]: pgmap v234: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.7 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 2/83544 objects degraded (0.002%); 2620/83544 objects misplaced (3.136%) 2024-02-05T07:24:14.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:14 smithi007 ceph-mon[29098]: 3.b scrub ok 2024-02-05T07:24:14.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:14 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2262346718' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:24:14.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:14 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 2/83544 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:24:14.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:14 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:24:14.650 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:14 smithi007 ceph-mon[37158]: pgmap v234: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.7 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 2/83544 objects degraded (0.002%); 2620/83544 objects misplaced (3.136%) 2024-02-05T07:24:14.650 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:14 smithi007 ceph-mon[37158]: 3.b scrub ok 2024-02-05T07:24:14.650 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:14 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2262346718' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:24:14.650 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:14 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 2/83544 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:24:14.650 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:14 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:24:14.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:14.750 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:14 smithi090 ceph-mon[33213]: pgmap v234: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.7 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 2/83544 objects degraded (0.002%); 2620/83544 objects misplaced (3.136%) 2024-02-05T07:24:14.750 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:14 smithi090 ceph-mon[33213]: 3.b scrub ok 2024-02-05T07:24:14.750 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:14 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2262346718' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:24:14.750 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:14 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 2/83544 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:24:14.750 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:14 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:24:14.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:14.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:14.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:14.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:14.904 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:14.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:15.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:15.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:15.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:15.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:15.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:15.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:15.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:15.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:15.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:15.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:15.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:15.470 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:15.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:15.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:15.557 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[29098]: 3.a scrub starts 2024-02-05T07:24:15.557 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[29098]: 3.a scrub ok 2024-02-05T07:24:15.557 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[29098]: 4.0 scrub starts 2024-02-05T07:24:15.557 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[29098]: 4.0 scrub ok 2024-02-05T07:24:15.557 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:24:15.557 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2262346718' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:24:15.557 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[29098]: osdmap e175: 8 total, 8 up, 7 in 2024-02-05T07:24:15.557 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[29098]: 6.17 scrub starts 2024-02-05T07:24:15.558 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[29098]: 6.17 scrub ok 2024-02-05T07:24:15.558 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[29098]: 4.6 scrub starts 2024-02-05T07:24:15.558 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[29098]: 4.6 scrub ok 2024-02-05T07:24:15.558 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[37158]: 3.a scrub starts 2024-02-05T07:24:15.558 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[37158]: 3.a scrub ok 2024-02-05T07:24:15.558 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[37158]: 4.0 scrub starts 2024-02-05T07:24:15.558 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[37158]: 4.0 scrub ok 2024-02-05T07:24:15.558 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:24:15.558 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2262346718' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:24:15.558 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[37158]: osdmap e175: 8 total, 8 up, 7 in 2024-02-05T07:24:15.558 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[37158]: 6.17 scrub starts 2024-02-05T07:24:15.558 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[37158]: 6.17 scrub ok 2024-02-05T07:24:15.558 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[37158]: 4.6 scrub starts 2024-02-05T07:24:15.559 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:15 smithi007 ceph-mon[37158]: 4.6 scrub ok 2024-02-05T07:24:15.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:15.616 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:15.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:15.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:15.761 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:15.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:15.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:15 smithi090 ceph-mon[33213]: 3.a scrub starts 2024-02-05T07:24:15.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:15 smithi090 ceph-mon[33213]: 3.a scrub ok 2024-02-05T07:24:15.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:15 smithi090 ceph-mon[33213]: 4.0 scrub starts 2024-02-05T07:24:15.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:15 smithi090 ceph-mon[33213]: 4.0 scrub ok 2024-02-05T07:24:15.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:15 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:24:15.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:15 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2262346718' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:24:15.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:15 smithi090 ceph-mon[33213]: osdmap e175: 8 total, 8 up, 7 in 2024-02-05T07:24:15.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:15 smithi090 ceph-mon[33213]: 6.17 scrub starts 2024-02-05T07:24:15.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:15 smithi090 ceph-mon[33213]: 6.17 scrub ok 2024-02-05T07:24:15.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:15 smithi090 ceph-mon[33213]: 4.6 scrub starts 2024-02-05T07:24:15.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:15 smithi090 ceph-mon[33213]: 4.6 scrub ok 2024-02-05T07:24:15.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:15.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:15.908 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:15.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:16.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:16.044 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:16.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:16.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:16.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:16.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:16.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:16.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:16.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:16.347 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:16.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:16.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:16.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:16.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:16 smithi090 ceph-mon[33213]: pgmap v236: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.7 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 2/83544 objects degraded (0.002%); 2620/83544 objects misplaced (3.136%) 2024-02-05T07:24:16.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:16 smithi090 ceph-mon[33213]: 3.1e scrub starts 2024-02-05T07:24:16.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:16 smithi090 ceph-mon[33213]: 3.1e scrub ok 2024-02-05T07:24:16.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:16 smithi090 ceph-mon[33213]: 6.16 scrub starts 2024-02-05T07:24:16.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:16 smithi090 ceph-mon[33213]: 6.16 scrub ok 2024-02-05T07:24:16.539 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:24:16.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:16.592 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:24:16.633 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:16.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:16.643 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:24:16.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:16 smithi007 ceph-mon[29098]: pgmap v236: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.7 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 2/83544 objects degraded (0.002%); 2620/83544 objects misplaced (3.136%) 2024-02-05T07:24:16.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:16 smithi007 ceph-mon[29098]: 3.1e scrub starts 2024-02-05T07:24:16.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:16 smithi007 ceph-mon[29098]: 3.1e scrub ok 2024-02-05T07:24:16.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:16 smithi007 ceph-mon[29098]: 6.16 scrub starts 2024-02-05T07:24:16.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:16 smithi007 ceph-mon[29098]: 6.16 scrub ok 2024-02-05T07:24:16.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:16 smithi007 ceph-mon[37158]: pgmap v236: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.7 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 2/83544 objects degraded (0.002%); 2620/83544 objects misplaced (3.136%) 2024-02-05T07:24:16.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:16 smithi007 ceph-mon[37158]: 3.1e scrub starts 2024-02-05T07:24:16.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:16 smithi007 ceph-mon[37158]: 3.1e scrub ok 2024-02-05T07:24:16.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:16 smithi007 ceph-mon[37158]: 6.16 scrub starts 2024-02-05T07:24:16.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:16 smithi007 ceph-mon[37158]: 6.16 scrub ok 2024-02-05T07:24:16.693 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:24:16.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:16.742 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:24:16.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:16.767 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:16.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:16.795 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:24:16.842 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:24:16.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:16.888 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:24:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:24:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:24:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:24:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:24:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:24:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:24:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:24:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:24:16.910 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:16.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:17.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:17.048 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:17.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:17.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:17.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:17.192 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:17.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:17.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:17.340 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:17.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:17.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:17.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:17.486 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:17.487 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:24:17.487 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:24:17.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:17.487 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:190027 7.0 7:03097dd6:::benchmark_data_smithi007_77086_object85432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-02-05T07:24:17.487 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:17.477232+0000", 2024-02-05T07:24:17.487 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0078139420000000008, 2024-02-05T07:24:17.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0078466409999999997, 2024-02-05T07:24:17.487 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:17.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:24:17.487 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:24:17.488 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:24:17.488 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:24:17.488 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 190027 2024-02-05T07:24:17.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:17.488 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:17.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:17.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:17.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:17.477232+0000", 2024-02-05T07:24:17.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:17.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:17.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:17.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:17.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:17.477232+0000", 2024-02-05T07:24:17.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:17.477234+0000", 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8610000000000001e-06 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:17.477240+0000", 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3110000000000002e-06 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:17.477254+0000", 2024-02-05T07:24:17.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3548e-05 2024-02-05T07:24:17.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:17.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:17.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:17.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:17.477264+0000", 2024-02-05T07:24:17.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0012e-05 2024-02-05T07:24:17.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:17.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:17.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:17.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:17.477328+0000", 2024-02-05T07:24:17.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4614000000000002e-05 2024-02-05T07:24:17.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:17.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:17.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:17.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:17.477463+0000", 2024-02-05T07:24:17.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013451899999999999 2024-02-05T07:24:17.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:17.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:17.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:24:17.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:17.477879+0000", 2024-02-05T07:24:17.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000416377 2024-02-05T07:24:17.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:17.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:17.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:24:17.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:17.480044+0000", 2024-02-05T07:24:17.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002164717 2024-02-05T07:24:17.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:17.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:17.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:17.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:17.483961+0000", 2024-02-05T07:24:17.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039165900000000002 2024-02-05T07:24:17.493 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:17.493 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:17.493 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:17.493 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:17.493 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-02-05T07:24:17.493 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2024-02-05T07:24:17.493 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:24:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:17.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:17.495 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 7 pg_num to 36 2024-02-05T07:24:17.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:17.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:24:17.629 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:17.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:17.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:17.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:17 smithi007 ceph-mon[29098]: 5.1a scrub starts 2024-02-05T07:24:17.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:17 smithi007 ceph-mon[29098]: 5.1a scrub ok 2024-02-05T07:24:17.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:17 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/739697437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:17.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:17 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2920075636' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2024-02-05T07:24:17.646 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:17 smithi007 ceph-mon[29098]: 3.17 scrub starts 2024-02-05T07:24:17.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:17 smithi007 ceph-mon[37158]: 5.1a scrub starts 2024-02-05T07:24:17.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:17 smithi007 ceph-mon[37158]: 5.1a scrub ok 2024-02-05T07:24:17.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:17 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/739697437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:17.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:17 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2920075636' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2024-02-05T07:24:17.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:17 smithi007 ceph-mon[37158]: 3.17 scrub starts 2024-02-05T07:24:17.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:17.765 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:17.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:17.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:17 smithi090 ceph-mon[33213]: 5.1a scrub starts 2024-02-05T07:24:17.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:17 smithi090 ceph-mon[33213]: 5.1a scrub ok 2024-02-05T07:24:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:17 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/739697437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:17 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2920075636' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2024-02-05T07:24:17.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:17 smithi090 ceph-mon[33213]: 3.17 scrub starts 2024-02-05T07:24:17.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:17.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:18.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:18.041 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:18.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:18.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:18.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:18.206 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:18.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:18.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:18.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T07:24:18.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:18.350 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:18.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:18.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:18.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:18.510 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:18.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:18.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:18 smithi007 ceph-mon[29098]: pgmap v237: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.3 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 578 op/s; 2/66219 objects degraded (0.003%); 2099/66219 objects misplaced (3.170%) 2024-02-05T07:24:18.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:18 smithi007 ceph-mon[29098]: 3.17 scrub ok 2024-02-05T07:24:18.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:18 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2920075636' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-02-05T07:24:18.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:18 smithi007 ceph-mon[29098]: osdmap e176: 8 total, 8 up, 7 in 2024-02-05T07:24:18.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:18 smithi007 ceph-mon[37158]: pgmap v237: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.3 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 578 op/s; 2/66219 objects degraded (0.003%); 2099/66219 objects misplaced (3.170%) 2024-02-05T07:24:18.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:18 smithi007 ceph-mon[37158]: 3.17 scrub ok 2024-02-05T07:24:18.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:18 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2920075636' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-02-05T07:24:18.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:18 smithi007 ceph-mon[37158]: osdmap e176: 8 total, 8 up, 7 in 2024-02-05T07:24:18.609 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:18.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:18.627 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:24:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:18.661 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:18.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:18.762 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:18 smithi090 ceph-mon[33213]: pgmap v237: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.3 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 578 op/s; 2/66219 objects degraded (0.003%); 2099/66219 objects misplaced (3.170%) 2024-02-05T07:24:18.762 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:18 smithi090 ceph-mon[33213]: 3.17 scrub ok 2024-02-05T07:24:18.762 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:18 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2920075636' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-02-05T07:24:18.762 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:18 smithi090 ceph-mon[33213]: osdmap e176: 8 total, 8 up, 7 in 2024-02-05T07:24:18.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:18.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:18.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:18.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:18.947 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:18.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:19.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:19.089 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:19.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:19.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:19.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:19.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:19.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:19.377 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:19.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:19.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:19.522 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:19.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:19.623 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:19.676 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:19.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:19.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:19.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:19 smithi090 ceph-mon[33213]: 7.c deep-scrub starts 2024-02-05T07:24:19.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:19 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-05T07:24:19.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:19 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-05T07:24:19.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:19 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 2/66219 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:24:19.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:19.802 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:19.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:19.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:19 smithi007 ceph-mon[29098]: 7.c deep-scrub starts 2024-02-05T07:24:19.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:19 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-05T07:24:19.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:19 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-05T07:24:19.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:19 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 2/66219 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:24:19.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:19 smithi007 ceph-mon[37158]: 7.c deep-scrub starts 2024-02-05T07:24:19.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:19 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-05T07:24:19.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:19 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-05T07:24:19.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:19 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 2/66219 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:24:19.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:24:19.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:19.927 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:19.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:20.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:20.052 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:20.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:20.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:20.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:20.178 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:20.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:20.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:20.280 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:20.303 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:20.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:20.388 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:24:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:24:20.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:20.419 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:20.428 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:20.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:20.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:20.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:20.574 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:20.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:20.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:20.709 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:20.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:20.716 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:24:20.716 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:24:20.716 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:24:20.716 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:24:20.716 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:24:20.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:20 smithi090 ceph-mon[33213]: pgmap v239: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.3 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 722 op/s; 2/66219 objects degraded (0.003%); 2099/66219 objects misplaced (3.170%); 0 B/s, 42 objects/s recovering 2024-02-05T07:24:20.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:20 smithi090 ceph-mon[33213]: 6.a scrub starts 2024-02-05T07:24:20.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:20 smithi090 ceph-mon[33213]: 6.a scrub ok 2024-02-05T07:24:20.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:20 smithi090 ceph-mon[33213]: 3.f scrub starts 2024-02-05T07:24:20.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:20 smithi090 ceph-mon[33213]: 3.f scrub ok 2024-02-05T07:24:20.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:20 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-02-05T07:24:20.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:20 smithi090 ceph-mon[33213]: osdmap e177: 8 total, 8 up, 7 in 2024-02-05T07:24:20.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:20 smithi090 ceph-mon[33213]: 5.e scrub starts 2024-02-05T07:24:20.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:20 smithi090 ceph-mon[33213]: 5.e scrub ok 2024-02-05T07:24:20.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:20 smithi090 ceph-mon[33213]: 6.10 deep-scrub starts 2024-02-05T07:24:20.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[29098]: pgmap v239: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.3 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 722 op/s; 2/66219 objects degraded (0.003%); 2099/66219 objects misplaced (3.170%); 0 B/s, 42 objects/s recovering 2024-02-05T07:24:20.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[29098]: 6.a scrub starts 2024-02-05T07:24:20.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[29098]: 6.a scrub ok 2024-02-05T07:24:20.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[29098]: 3.f scrub starts 2024-02-05T07:24:20.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[29098]: 3.f scrub ok 2024-02-05T07:24:20.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-02-05T07:24:20.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[29098]: osdmap e177: 8 total, 8 up, 7 in 2024-02-05T07:24:20.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[29098]: 5.e scrub starts 2024-02-05T07:24:20.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[29098]: 5.e scrub ok 2024-02-05T07:24:20.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[29098]: 6.10 deep-scrub starts 2024-02-05T07:24:20.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[37158]: pgmap v239: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 1.3 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 722 op/s; 2/66219 objects degraded (0.003%); 2099/66219 objects misplaced (3.170%); 0 B/s, 42 objects/s recovering 2024-02-05T07:24:20.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[37158]: 6.a scrub starts 2024-02-05T07:24:20.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[37158]: 6.a scrub ok 2024-02-05T07:24:20.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[37158]: 3.f scrub starts 2024-02-05T07:24:20.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[37158]: 3.f scrub ok 2024-02-05T07:24:20.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-02-05T07:24:20.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[37158]: osdmap e177: 8 total, 8 up, 7 in 2024-02-05T07:24:20.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[37158]: 5.e scrub starts 2024-02-05T07:24:20.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[37158]: 5.e scrub ok 2024-02-05T07:24:20.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:20 smithi007 ceph-mon[37158]: 6.10 deep-scrub starts 2024-02-05T07:24:20.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:20.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:20.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:20.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:20.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:20.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:21.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:21.094 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:21.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:21.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:21.225 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:21.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:21.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:21.356 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:21.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:21.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:21.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T07:24:21.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:21.483 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:21.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:21.520 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:24:21.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:21.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:21.611 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:21.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:21.671 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:24:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[29098]: 7.14 scrub starts 2024-02-05T07:24:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[29098]: 7.14 scrub ok 2024-02-05T07:24:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[29098]: 6.10 deep-scrub ok 2024-02-05T07:24:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[29098]: 4.5 scrub starts 2024-02-05T07:24:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[29098]: 4.5 scrub ok 2024-02-05T07:24:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[29098]: 5.0 scrub starts 2024-02-05T07:24:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[29098]: 5.0 scrub ok 2024-02-05T07:24:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[29098]: osdmap e178: 8 total, 8 up, 7 in 2024-02-05T07:24:21.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[29098]: 7.1 continuing backfill to osd.1 from (110'3615,176'8518] MIN to 176'8518 2024-02-05T07:24:21.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[29098]: 7.f starting backfill to osd.4 from (141'7605,176'12099] MIN to 176'12100 2024-02-05T07:24:21.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2069156021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:24:21.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-02-05T07:24:21.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-02-05T07:24:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[37158]: 7.14 scrub starts 2024-02-05T07:24:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[37158]: 7.14 scrub ok 2024-02-05T07:24:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[37158]: 6.10 deep-scrub ok 2024-02-05T07:24:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[37158]: 4.5 scrub starts 2024-02-05T07:24:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[37158]: 4.5 scrub ok 2024-02-05T07:24:21.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[37158]: 5.0 scrub starts 2024-02-05T07:24:21.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[37158]: 5.0 scrub ok 2024-02-05T07:24:21.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[37158]: osdmap e178: 8 total, 8 up, 7 in 2024-02-05T07:24:21.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[37158]: 7.1 continuing backfill to osd.1 from (110'3615,176'8518] MIN to 176'8518 2024-02-05T07:24:21.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[37158]: 7.f starting backfill to osd.4 from (141'7605,176'12099] MIN to 176'12100 2024-02-05T07:24:21.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2069156021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:24:21.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-02-05T07:24:21.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:21 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-02-05T07:24:21.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:21.736 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:21.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:21.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:21 smithi090 ceph-mon[33213]: 7.14 scrub starts 2024-02-05T07:24:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:21 smithi090 ceph-mon[33213]: 7.14 scrub ok 2024-02-05T07:24:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:21 smithi090 ceph-mon[33213]: 6.10 deep-scrub ok 2024-02-05T07:24:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:21 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:21 smithi090 ceph-mon[33213]: 4.5 scrub starts 2024-02-05T07:24:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:21 smithi090 ceph-mon[33213]: 4.5 scrub ok 2024-02-05T07:24:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:21 smithi090 ceph-mon[33213]: 5.0 scrub starts 2024-02-05T07:24:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:21 smithi090 ceph-mon[33213]: 5.0 scrub ok 2024-02-05T07:24:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:21 smithi090 ceph-mon[33213]: osdmap e178: 8 total, 8 up, 7 in 2024-02-05T07:24:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:21 smithi090 ceph-mon[33213]: 7.1 continuing backfill to osd.1 from (110'3615,176'8518] MIN to 176'8518 2024-02-05T07:24:21.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:21 smithi090 ceph-mon[33213]: 7.f starting backfill to osd.4 from (141'7605,176'12099] MIN to 176'12100 2024-02-05T07:24:21.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:21 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2069156021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:24:21.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:21 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-02-05T07:24:21.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:21 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-02-05T07:24:21.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:21.864 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:21.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:21.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:21.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:21.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:21.991 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:21.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:22.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:22.117 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:22.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:22.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:22.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:22.244 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:22.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:22.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:22.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:22.369 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:22.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:22.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:22.470 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:22.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:22.495 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:22.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:22.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:22.630 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:22.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:22.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:22.758 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:22.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:22.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:22.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:22 smithi090 ceph-mon[33213]: pgmap v242: 168 pgs: 7 peering, 2 remapped+peering, 1 active+remapped+backfilling, 10 unknown, 148 active+clean; 1.0 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 825 op/s; 503/50496 objects misplaced (0.996%) 2024-02-05T07:24:22.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:22 smithi090 ceph-mon[33213]: 1.0 scrub starts 2024-02-05T07:24:22.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:22 smithi090 ceph-mon[33213]: 7.d scrub starts 2024-02-05T07:24:22.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:22 smithi090 ceph-mon[33213]: 1.0 scrub ok 2024-02-05T07:24:22.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:22 smithi090 ceph-mon[33213]: 7.d scrub ok 2024-02-05T07:24:22.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:22 smithi090 ceph-mon[33213]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-02-05T07:24:22.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:22 smithi090 ceph-mon[33213]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/66219 objects degraded (0.003%), 1 pg degraded) 2024-02-05T07:24:22.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:22 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:24:22.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:22 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:24:22.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:22 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2069156021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:24:22.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:22 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-02-05T07:24:22.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:22 smithi090 ceph-mon[33213]: osdmap e179: 8 total, 8 up, 7 in 2024-02-05T07:24:22.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[29098]: pgmap v242: 168 pgs: 7 peering, 2 remapped+peering, 1 active+remapped+backfilling, 10 unknown, 148 active+clean; 1.0 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 825 op/s; 503/50496 objects misplaced (0.996%) 2024-02-05T07:24:22.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[29098]: 1.0 scrub starts 2024-02-05T07:24:22.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[29098]: 7.d scrub starts 2024-02-05T07:24:22.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[29098]: 1.0 scrub ok 2024-02-05T07:24:22.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[29098]: 7.d scrub ok 2024-02-05T07:24:22.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[29098]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-02-05T07:24:22.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[29098]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/66219 objects degraded (0.003%), 1 pg degraded) 2024-02-05T07:24:22.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:24:22.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:24:22.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2069156021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:24:22.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-02-05T07:24:22.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[29098]: osdmap e179: 8 total, 8 up, 7 in 2024-02-05T07:24:22.856 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[37158]: pgmap v242: 168 pgs: 7 peering, 2 remapped+peering, 1 active+remapped+backfilling, 10 unknown, 148 active+clean; 1.0 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 825 op/s; 503/50496 objects misplaced (0.996%) 2024-02-05T07:24:22.856 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[37158]: 1.0 scrub starts 2024-02-05T07:24:22.856 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[37158]: 7.d scrub starts 2024-02-05T07:24:22.856 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[37158]: 1.0 scrub ok 2024-02-05T07:24:22.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[37158]: 7.d scrub ok 2024-02-05T07:24:22.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[37158]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-02-05T07:24:22.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[37158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/66219 objects degraded (0.003%), 1 pg degraded) 2024-02-05T07:24:22.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:24:22.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:24:22.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2069156021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:24:22.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-02-05T07:24:22.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:22 smithi007 ceph-mon[37158]: osdmap e179: 8 total, 8 up, 7 in 2024-02-05T07:24:22.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:22.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:22.888 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:22.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:22.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:23.016 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:23.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:23.118 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:23.145 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:23.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:23.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:23.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:23.275 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:23.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:23.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:23.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:23.404 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:23.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:23.425 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:24:23.446 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:24:23.465 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:24:23.466 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-02-05T07:24:23.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-02-05T07:24:23.466 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:24:23.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.466 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24719.0:19429 7.a e104/100 7:5bea2309:::benchmark_data_smithi007_77086_object19428:head v 104'1270, mlcod=104'1267)", 2024-02-05T07:24:23.466 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:21:04.194825+0000", 2024-02-05T07:24:23.466 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 199.26686089200001, 2024-02-05T07:24:23.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.154896319, 2024-02-05T07:24:23.466 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.466 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:24:23.466 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.194825+0000", 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.194825+0000", 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.194826+0000", 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4600000000000004e-07 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.194855+0000", 2024-02-05T07:24:23.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9595999999999999e-05 2024-02-05T07:24:23.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.194863+0000", 2024-02-05T07:24:23.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9179999999999994e-06 2024-02-05T07:24:23.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.194911+0000", 2024-02-05T07:24:23.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8041999999999998e-05 2024-02-05T07:24:23.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.195050+0000", 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138486 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.195089+0000", 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9283999999999999e-05 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.349664+0000", 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15457462999999999 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.349721+0000", 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7717000000000003e-05 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24719.0:19437 7.b e105/100 7:d0c1b1ad:::benchmark_data_smithi007_77086_object19436:head v 105'1245, mlcod=104'1243)", 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:21:04.201325+0000", 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 199.26036103600001, 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14875788000000001, 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.201325+0000", 2024-02-05T07:24:23.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.201325+0000", 2024-02-05T07:24:23.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.201325+0000", 2024-02-05T07:24:23.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0800000000000001e-07 2024-02-05T07:24:23.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.201363+0000", 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8299e-05 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.347594+0000", 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14623012099999999 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.347603+0000", 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1439999999999992e-06 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.348236+0000", 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063316499999999999 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.348260+0000", 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4125e-05 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.350046+0000", 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001786253 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.350083+0000", 2024-02-05T07:24:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6565e-05 2024-02-05T07:24:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:19442 7.0 7:0f33aa44:::benchmark_data_smithi007_77086_object19441:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-02-05T07:24:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:21:04.204638+0000", 2024-02-05T07:24:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 199.25704747899999, 2024-02-05T07:24:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14630232100000001, 2024-02-05T07:24:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19442 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.204638+0000", 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.204638+0000", 2024-02-05T07:24:23.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.204640+0000", 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8190000000000001e-06 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.204689+0000", 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8726000000000001e-05 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.347669+0000", 2024-02-05T07:24:23.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.142979945 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.347680+0000", 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0626e-05 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.347846+0000", 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000166219 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.347982+0000", 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013597699999999999 2024-02-05T07:24:23.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:24:23.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.348135+0000", 2024-02-05T07:24:23.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015316599999999999 2024-02-05T07:24:23.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:24:23.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.350048+0000", 2024-02-05T07:24:23.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001913117 2024-02-05T07:24:23.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.350214+0000", 2024-02-05T07:24:23.479 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:24:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000166372 2024-02-05T07:24:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.350845+0000", 2024-02-05T07:24:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063041300000000002 2024-02-05T07:24:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:24:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.350903+0000", 2024-02-05T07:24:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8505000000000003e-05 2024-02-05T07:24:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:04.350941+0000", 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7435999999999999e-05 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:74430 7.10 7:0c0995b9:::benchmark_data_smithi007_77086_object74429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e118)", 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:21:41.581567+0000", 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 161.88011895299999, 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0644127990000001, 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:24:23.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 74430 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.581567+0000", 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.581567+0000", 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.581569+0000", 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.272e-06 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.581614+0000", 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4505999999999997e-05 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.581638+0000", 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3935e-05 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.581650+0000", 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2055000000000001e-05 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.653767+0000", 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072117143999999994 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.653776+0000", 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2299999999999997e-06 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.637518+0000", 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98374208600000002 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.638180+0000", 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066153800000000003 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.645888+0000", 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0077084679999999996 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.645980+0000", 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1564999999999995e-05 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24719.0:173212 7.a e169/163 7:524131d3:::benchmark_data_smithi007_77086_object68617:head v 169'11017, mlcod=169'11015)", 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:23:58.619484+0000", 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.842201684999999, 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.1392996, 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.619484+0000", 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.619484+0000", 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.619485+0000", 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7899999999999999e-07 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.619489+0000", 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.878e-06 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.751694+0000", 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13220579800000001 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.751708+0000", 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3473e-05 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.751949+0000", 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024158200000000001 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.751973+0000", 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4023999999999999e-05 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.758667+0000", 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0066935750000000002 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.758784+0000", 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116891 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:173890 7.0 7:07ac86d9:::benchmark_data_smithi007_77086_object69295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:23:59.479295+0000", 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.982390886000001, 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.139151099, 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:24:23.490 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 173890 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.479295+0000", 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.479295+0000", 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.479297+0000", 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5969999999999999e-06 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.479303+0000", 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0129999999999999e-06 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.479318+0000", 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5248e-05 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.479327+0000", 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4380000000000001e-06 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.479387+0000", 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9287999999999999e-05 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.479550+0000", 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016343 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.479977+0000", 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042694199999999998 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.481848+0000", 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018710039999999999 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.489323+0000", 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074752409999999997 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.617697+0000", 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12837363800000001 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.617771+0000", 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4293000000000002e-05 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.618446+0000", 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067496699999999995 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:173894 7.0 7:04b43b0b:::benchmark_data_smithi007_77086_object69299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:23:59.483170+0000", 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.978515951999999, 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13569863800000001, 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 173894 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.483170+0000", 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.483170+0000", 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.483172+0000", 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5740000000000001e-06 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.483176+0000", 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6859999999999997e-06 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.483189+0000", 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2473999999999999e-05 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.483196+0000", 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4510000000000003e-06 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.483252+0000", 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5618999999999997e-05 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.483405+0000", 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015280099999999999 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.483776+0000", 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000371092 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:24:23.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.485596+0000", 2024-02-05T07:24:23.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001820162 2024-02-05T07:24:23.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.490201+0000", 2024-02-05T07:24:23.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046051160000000002 2024-02-05T07:24:23.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.618504+0000", 2024-02-05T07:24:23.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12830353799999999 2024-02-05T07:24:23.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.618556+0000", 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2008000000000001e-05 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:59.618869+0000", 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031211699999999999 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:176345 7.0 7:027c7abd:::benchmark_data_smithi007_77086_object71750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:04.263576+0000", 2024-02-05T07:24:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.198109884000001, 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.138624211, 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 176345 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.263576+0000", 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.263576+0000", 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.263577+0000", 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3289999999999999e-06 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.263582+0000", 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5669999999999996e-06 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.263592+0000", 2024-02-05T07:24:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5820000000000008e-06 2024-02-05T07:24:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.263597+0000", 2024-02-05T07:24:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6319999999999996e-06 2024-02-05T07:24:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.264136+0000", 2024-02-05T07:24:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053851099999999998 2024-02-05T07:24:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.264239+0000", 2024-02-05T07:24:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103669 2024-02-05T07:24:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:24:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.264678+0000", 2024-02-05T07:24:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043837100000000001 2024-02-05T07:24:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.387015+0000", 2024-02-05T07:24:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.122337038 2024-02-05T07:24:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.387055+0000", 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0173000000000003e-05 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.401732+0000", 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014676896 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.401791+0000", 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9175000000000002e-05 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.402200+0000", 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040926800000000001 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPushReply(7.0 173/163 [PushReplyOp(7:07e96472:::benchmark_data_smithi007_77086_object98916:head)])", 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:04.900394+0000", 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.561292212000001, 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.143832549, 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:24:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.900394+0000", 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.900394+0000", 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.900395+0000", 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.477e-06 2024-02-05T07:24:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.900405+0000", 2024-02-05T07:24:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6199999999999994e-06 2024-02-05T07:24:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.900427+0000", 2024-02-05T07:24:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2189999999999999e-05 2024-02-05T07:24:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.900442+0000", 2024-02-05T07:24:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4679e-05 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.900618+0000", 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017612800000000001 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.044226+0000", 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.143608455 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24719.0:176825 7.a e173/171 7:52985966:::benchmark_data_smithi007_77086_object72230:head v 173'11252, mlcod=173'11250)", 2024-02-05T07:24:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:04.922856+0000", 2024-02-05T07:24:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.538830379, 2024-02-05T07:24:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13581578, 2024-02-05T07:24:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:24:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.922856+0000", 2024-02-05T07:24:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.922856+0000", 2024-02-05T07:24:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.922857+0000", 2024-02-05T07:24:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3820000000000001e-06 2024-02-05T07:24:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:04.922866+0000", 2024-02-05T07:24:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1819999999999995e-06 2024-02-05T07:24:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.042902+0000", 2024-02-05T07:24:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120035737 2024-02-05T07:24:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.042909+0000", 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4919999999999996e-06 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.043711+0000", 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080182799999999998 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.043730+0000", 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8932000000000001e-05 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.058597+0000", 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014866570000000001 2024-02-05T07:24:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.058671+0000", 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4657e-05 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:177449 7.0 7:046c7cf6:::benchmark_data_smithi007_77086_object72854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:05.710528+0000", 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.751157710000001, 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.186820809, 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:24:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:24:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:24:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:24:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 177449 2024-02-05T07:24:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.710528+0000", 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.710528+0000", 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.710530+0000", 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.313e-06 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.710534+0000", 2024-02-05T07:24:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0579999999999999e-06 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.710542+0000", 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6289999999999999e-06 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.710548+0000", 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.682e-06 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.710584+0000", 2024-02-05T07:24:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5756000000000001e-05 2024-02-05T07:24:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:23.520 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:24:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.859488+0000", 2024-02-05T07:24:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14890389400000001 2024-02-05T07:24:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:24:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.859967+0000", 2024-02-05T07:24:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047977099999999998 2024-02-05T07:24:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:24:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.862586+0000", 2024-02-05T07:24:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026184540000000001 2024-02-05T07:24:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.873872+0000", 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011286288 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.896935+0000", 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023062446 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.897008+0000", 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3549999999999999e-05 2024-02-05T07:24:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.897349+0000", 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034096799999999998 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPushReply(7.0 173/163 [PushReplyOp(7:0057f2ab:::benchmark_data_smithi007_77086_object100320:head)])", 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:05.711374+0000", 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.750311872000001, 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.151199058, 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:24:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.711374+0000", 2024-02-05T07:24:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.711374+0000", 2024-02-05T07:24:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.711376+0000", 2024-02-05T07:24:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.457e-06 2024-02-05T07:24:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.711384+0000", 2024-02-05T07:24:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9430000000000006e-06 2024-02-05T07:24:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.859053+0000", 2024-02-05T07:24:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14766810899999999 2024-02-05T07:24:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.859066+0000", 2024-02-05T07:24:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3793e-05 2024-02-05T07:24:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.861950+0000", 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028841040000000002 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.862573+0000", 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062265200000000001 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPushReply(7.0 173/163 [PushReplyOp(7:07dd9c0d:::benchmark_data_smithi007_77086_object102778:head)])", 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:07.318321+0000", 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.143364861999999, 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.142646194, 2024-02-05T07:24:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:24:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.318321+0000", 2024-02-05T07:24:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.318321+0000", 2024-02-05T07:24:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.318322+0000", 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.82e-07 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.318327+0000", 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0180000000000001e-06 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.460213+0000", 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.141886026 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.460236+0000", 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3754999999999999e-05 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.460366+0000", 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129944 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.460967+0000", 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060086899999999997 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192527 7.0 7:06760b4b:::benchmark_data_smithi007_77086_object87932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:19.506098+0000", 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.955588396, 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.162741418, 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192527 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.506098+0000", 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.506098+0000", 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.506100+0000", 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9810000000000002e-06 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.506107+0000", 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8860000000000004e-06 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.506129+0000", 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1892999999999999e-05 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.506140+0000", 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0829000000000001e-05 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.506207+0000", 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7199000000000005e-05 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.506357+0000", 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149517 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.506844+0000", 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048720099999999999 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.509555+0000", 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027106270000000002 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.668824+0000", 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15926927099999999 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.668839+0000", 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5014000000000001e-05 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192527 7.20 7:06760b4b:::benchmark_data_smithi007_77086_object87932:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e177)", 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:19.508948+0000", 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9527377279999998, 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.074903948, 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192527 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.508948+0000", 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.508948+0000", 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.508950+0000", 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9089999999999998e-06 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.508957+0000", 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9399999999999996e-06 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.508978+0000", 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0649000000000002e-05 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.508988+0000", 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0084e-05 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.669209+0000", 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.160221062 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.669218+0000", 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1570000000000006e-06 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.577149+0000", 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.90793126800000001 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-02-05T07:24:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.577279+0000", 2024-02-05T07:24:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013019400000000001 2024-02-05T07:24:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583760+0000", 2024-02-05T07:24:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064800639999999998 2024-02-05T07:24:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583852+0000", 2024-02-05T07:24:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2621000000000006e-05 2024-02-05T07:24:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24719.0:192526 7.6 e177/167 7:638418e0:::benchmark_data_smithi007_77086_object87931:head v 177'8477, mlcod=176'8474)", 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:19.517743+0000", 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9439431850000002, 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14755467, 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.517743+0000", 2024-02-05T07:24:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.517743+0000", 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.517744+0000", 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4160000000000001e-06 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.517753+0000", 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6519999999999995e-06 2024-02-05T07:24:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.517767+0000", 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.434e-05 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.517776+0000", 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2969999999999992e-06 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.521613+0000", 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038361659999999998 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.521638+0000", 2024-02-05T07:24:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5769e-05 2024-02-05T07:24:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:24:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.665259+0000", 2024-02-05T07:24:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14362011699999999 2024-02-05T07:24:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.665297+0000", 2024-02-05T07:24:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8912999999999998e-05 2024-02-05T07:24:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192545 7.0 7:000d1bd5:::benchmark_data_smithi007_77086_object87950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:24:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:20.577543+0000", 2024-02-05T07:24:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.8841426559999999, 2024-02-05T07:24:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.96322299199999994, 2024-02-05T07:24:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:24:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:24:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:24:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:24:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192545 2024-02-05T07:24:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.577543+0000", 2024-02-05T07:24:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.577543+0000", 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.577544+0000", 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0979999999999999e-06 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.577549+0000", 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9350000000000002e-06 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.577561+0000", 2024-02-05T07:24:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1455e-05 2024-02-05T07:24:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.577582+0000", 2024-02-05T07:24:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1501999999999998e-05 2024-02-05T07:24:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.577642+0000", 2024-02-05T07:24:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9741000000000003e-05 2024-02-05T07:24:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.577652+0000", 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7580000000000004e-06 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.534928+0000", 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95727587999999997 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.535035+0000", 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107645 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:24:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.535211+0000", 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017521999999999999 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.540581+0000", 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053700520000000002 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.540590+0000", 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9549999999999998e-06 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.540646+0000", 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6873999999999998e-05 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.540680+0000", 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3460999999999997e-05 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.540766+0000", 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6415999999999994e-05 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192685 7.0 7:01fb0478:::benchmark_data_smithi007_77086_object88090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:20.699958+0000", 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.7617278490000001, 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.84692497099999997, 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192685 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.699958+0000", 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.699958+0000", 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.699959+0000", 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.139e-06 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.699964+0000", 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0590000000000002e-06 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.699975+0000", 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1035e-05 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.699981+0000", 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9689999999999999e-06 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.700037+0000", 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6202999999999997e-05 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.700050+0000", 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2797e-05 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.540786+0000", 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.84073531300000004 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.540868+0000", 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1930999999999999e-05 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.541060+0000", 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019201299999999999 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.542838+0000", 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001778906 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.544028+0000", 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001189664 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.546681+0000", 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002652518 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.546735+0000", 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4079000000000002e-05 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.546883+0000", 2024-02-05T07:24:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148345 2024-02-05T07:24:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192690 7.0 7:020e83e6:::benchmark_data_smithi007_77086_object88095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:24:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:20.706299+0000", 2024-02-05T07:24:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.7553868850000001, 2024-02-05T07:24:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.84075392699999996, 2024-02-05T07:24:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:24:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192690 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.706299+0000", 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.706299+0000", 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.706300+0000", 2024-02-05T07:24:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1689999999999999e-06 2024-02-05T07:24:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.706305+0000", 2024-02-05T07:24:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4299999999999999e-06 2024-02-05T07:24:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.706318+0000", 2024-02-05T07:24:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2999999999999999e-05 2024-02-05T07:24:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.706325+0000", 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0330000000000003e-06 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.706381+0000", 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6181000000000003e-05 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.706392+0000", 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1354e-05 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.542051+0000", 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.83565910300000001 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.542124+0000", 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2831999999999997e-05 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.542250+0000", 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012628499999999999 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.543645+0000", 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013941839999999999 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.545755+0000", 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021105730000000001 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.546931+0000", 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001175497 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.546961+0000", 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0664000000000003e-05 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.547053+0000", 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1621999999999998e-05 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192715 7.0 7:003a935f:::benchmark_data_smithi007_77086_object88120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:24:23.564 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:20.739430+0000", 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.7222563700000002, 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.80795375000000003, 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192715 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.739430+0000", 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:24:23.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.739430+0000", 2024-02-05T07:24:23.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:24:23.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:24:23.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.739431+0000", 2024-02-05T07:24:23.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2410000000000001e-06 2024-02-05T07:24:23.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:24:23.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.739435+0000", 2024-02-05T07:24:23.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6539999999999998e-06 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.739446+0000", 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.739452+0000", 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8619999999999999e-06 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.739508+0000", 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5858e-05 2024-02-05T07:24:23.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:24:23.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.739516+0000", 2024-02-05T07:24:23.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-02-05T07:24:23.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:24:23.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.542859+0000", 2024-02-05T07:24:23.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.80334251400000001 2024-02-05T07:24:23.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:24:23.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.542928+0000", 2024-02-05T07:24:23.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9108000000000003e-05 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.543045+0000", 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116903 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.543849+0000", 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080448499999999997 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.545986+0000", 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021368699999999999 2024-02-05T07:24:23.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.547248+0000", 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012616940000000001 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.547296+0000", 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7957999999999997e-05 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.547383+0000", 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7579999999999998e-05 2024-02-05T07:24:23.570 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.571 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.571 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.571 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:24:23.571 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:24:23.571 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:24:23.572 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:24:23.572 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:24:23.596 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T07:24:23.596 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:24:23.596 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-05T07:24:23.597 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-02-05T07:24:23.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 0 0 2024-02-05T07:24:23.602 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:23.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:23.611 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:24:23.652 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:24:23.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[29098]: 7.21 starting backfill to osd.0 from (0'0,0'0] MAX to 176'8518 2024-02-05T07:24:23.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[29098]: osdmap e180: 8 total, 8 up, 7 in 2024-02-05T07:24:23.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[29098]: 7.1f starting backfill to osd.4 from (141'7605,176'12099] MIN to 176'12100 2024-02-05T07:24:23.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[29098]: 7.21 starting backfill to osd.5 from (0'0,0'0] MAX to 176'8518 2024-02-05T07:24:23.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[29098]: 7.21 starting backfill to osd.7 from (0'0,0'0] MAX to 176'8518 2024-02-05T07:24:23.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[29098]: 7.1f starting backfill to osd.5 from (0'0,0'0] MAX to 176'12100 2024-02-05T07:24:23.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[29098]: 7.1f starting backfill to osd.6 from (0'0,0'0] MAX to 176'12100 2024-02-05T07:24:23.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3567452090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:23.678 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:24:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:24:23.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[37158]: 7.21 starting backfill to osd.0 from (0'0,0'0] MAX to 176'8518 2024-02-05T07:24:23.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[37158]: osdmap e180: 8 total, 8 up, 7 in 2024-02-05T07:24:23.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[37158]: 7.1f starting backfill to osd.4 from (141'7605,176'12099] MIN to 176'12100 2024-02-05T07:24:23.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[37158]: 7.21 starting backfill to osd.5 from (0'0,0'0] MAX to 176'8518 2024-02-05T07:24:23.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[37158]: 7.21 starting backfill to osd.7 from (0'0,0'0] MAX to 176'8518 2024-02-05T07:24:23.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[37158]: 7.1f starting backfill to osd.5 from (0'0,0'0] MAX to 176'12100 2024-02-05T07:24:23.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[37158]: 7.1f starting backfill to osd.6 from (0'0,0'0] MAX to 176'12100 2024-02-05T07:24:23.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:23 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3567452090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:23.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:23.739 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:23.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:23.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:23.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:23 smithi090 ceph-mon[33213]: 7.21 starting backfill to osd.0 from (0'0,0'0] MAX to 176'8518 2024-02-05T07:24:23.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:23 smithi090 ceph-mon[33213]: osdmap e180: 8 total, 8 up, 7 in 2024-02-05T07:24:23.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:23 smithi090 ceph-mon[33213]: 7.1f starting backfill to osd.4 from (141'7605,176'12099] MIN to 176'12100 2024-02-05T07:24:23.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:23 smithi090 ceph-mon[33213]: 7.21 starting backfill to osd.5 from (0'0,0'0] MAX to 176'8518 2024-02-05T07:24:23.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:23 smithi090 ceph-mon[33213]: 7.21 starting backfill to osd.7 from (0'0,0'0] MAX to 176'8518 2024-02-05T07:24:23.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:23 smithi090 ceph-mon[33213]: 7.1f starting backfill to osd.5 from (0'0,0'0] MAX to 176'12100 2024-02-05T07:24:23.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:23 smithi090 ceph-mon[33213]: 7.1f starting backfill to osd.6 from (0'0,0'0] MAX to 176'12100 2024-02-05T07:24:23.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:23 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3567452090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:23.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:23.878 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:23.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:23.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:23.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:24.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:24.040 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:24.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:24.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:24.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:24.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:24.324 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:24.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:24.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:24.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:24.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:24.502 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:24.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T07:24:24.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:24.641 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:24.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:24.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:24.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[29098]: pgmap v245: 168 pgs: 7 peering, 2 remapped+peering, 1 active+remapped+backfilling, 10 unknown, 148 active+clean; 1.0 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 503/50496 objects misplaced (0.996%) 2024-02-05T07:24:24.643 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[29098]: 7.1b starting backfill to osd.0 from (0'0,0'0] MAX to 178'15281 2024-02-05T07:24:24.643 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[29098]: 7.1e starting backfill to osd.1 from (0'0,0'0] MAX to 178'11816 2024-02-05T07:24:24.643 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[29098]: osdmap e181: 8 total, 8 up, 7 in 2024-02-05T07:24:24.643 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[29098]: 7.1c starting backfill to osd.1 from (0'0,0'0] MAX to 178'12233 2024-02-05T07:24:24.643 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[29098]: 7.20 starting backfill to osd.6 from (0'0,0'0] MAX to 178'8145 2024-02-05T07:24:24.643 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[29098]: 7.23 starting backfill to osd.1 from (0'0,0'0] MAX to 178'8416 2024-02-05T07:24:24.643 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[29098]: 7.1e starting backfill to osd.4 from (0'0,0'0] MAX to 178'11816 2024-02-05T07:24:24.643 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[29098]: 7.1a starting backfill to osd.4 from (0'0,0'0] MAX to 178'12231 2024-02-05T07:24:24.643 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[29098]: 7.22 starting backfill to osd.4 from (0'0,0'0] MAX to 178'8430 2024-02-05T07:24:24.643 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[29098]: 7.1b starting backfill to osd.2 from (0'0,0'0] MAX to 178'15281 2024-02-05T07:24:24.643 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[29098]: 7.23 starting backfill to osd.5 from (0'0,0'0] MAX to 178'8416 2024-02-05T07:24:24.643 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[29098]: 7.1c starting backfill to osd.5 from (0'0,0'0] MAX to 178'12233 2024-02-05T07:24:24.643 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[29098]: 7.1a starting backfill to osd.6 from (0'0,0'0] MAX to 178'12231 2024-02-05T07:24:24.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[37158]: pgmap v245: 168 pgs: 7 peering, 2 remapped+peering, 1 active+remapped+backfilling, 10 unknown, 148 active+clean; 1.0 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 503/50496 objects misplaced (0.996%) 2024-02-05T07:24:24.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[37158]: 7.1b starting backfill to osd.0 from (0'0,0'0] MAX to 178'15281 2024-02-05T07:24:24.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[37158]: 7.1e starting backfill to osd.1 from (0'0,0'0] MAX to 178'11816 2024-02-05T07:24:24.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[37158]: osdmap e181: 8 total, 8 up, 7 in 2024-02-05T07:24:24.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[37158]: 7.1c starting backfill to osd.1 from (0'0,0'0] MAX to 178'12233 2024-02-05T07:24:24.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[37158]: 7.20 starting backfill to osd.6 from (0'0,0'0] MAX to 178'8145 2024-02-05T07:24:24.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[37158]: 7.23 starting backfill to osd.1 from (0'0,0'0] MAX to 178'8416 2024-02-05T07:24:24.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[37158]: 7.1e starting backfill to osd.4 from (0'0,0'0] MAX to 178'11816 2024-02-05T07:24:24.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[37158]: 7.1a starting backfill to osd.4 from (0'0,0'0] MAX to 178'12231 2024-02-05T07:24:24.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[37158]: 7.22 starting backfill to osd.4 from (0'0,0'0] MAX to 178'8430 2024-02-05T07:24:24.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[37158]: 7.1b starting backfill to osd.2 from (0'0,0'0] MAX to 178'15281 2024-02-05T07:24:24.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[37158]: 7.23 starting backfill to osd.5 from (0'0,0'0] MAX to 178'8416 2024-02-05T07:24:24.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[37158]: 7.1c starting backfill to osd.5 from (0'0,0'0] MAX to 178'12233 2024-02-05T07:24:24.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:24 smithi007 ceph-mon[37158]: 7.1a starting backfill to osd.6 from (0'0,0'0] MAX to 178'12231 2024-02-05T07:24:24.742 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:24.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:24 smithi090 ceph-mon[33213]: pgmap v245: 168 pgs: 7 peering, 2 remapped+peering, 1 active+remapped+backfilling, 10 unknown, 148 active+clean; 1.0 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 503/50496 objects misplaced (0.996%) 2024-02-05T07:24:24.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:24 smithi090 ceph-mon[33213]: 7.1b starting backfill to osd.0 from (0'0,0'0] MAX to 178'15281 2024-02-05T07:24:24.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:24 smithi090 ceph-mon[33213]: 7.1e starting backfill to osd.1 from (0'0,0'0] MAX to 178'11816 2024-02-05T07:24:24.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:24 smithi090 ceph-mon[33213]: osdmap e181: 8 total, 8 up, 7 in 2024-02-05T07:24:24.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:24 smithi090 ceph-mon[33213]: 7.1c starting backfill to osd.1 from (0'0,0'0] MAX to 178'12233 2024-02-05T07:24:24.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:24 smithi090 ceph-mon[33213]: 7.20 starting backfill to osd.6 from (0'0,0'0] MAX to 178'8145 2024-02-05T07:24:24.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:24 smithi090 ceph-mon[33213]: 7.23 starting backfill to osd.1 from (0'0,0'0] MAX to 178'8416 2024-02-05T07:24:24.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:24 smithi090 ceph-mon[33213]: 7.1e starting backfill to osd.4 from (0'0,0'0] MAX to 178'11816 2024-02-05T07:24:24.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:24 smithi090 ceph-mon[33213]: 7.1a starting backfill to osd.4 from (0'0,0'0] MAX to 178'12231 2024-02-05T07:24:24.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:24 smithi090 ceph-mon[33213]: 7.22 starting backfill to osd.4 from (0'0,0'0] MAX to 178'8430 2024-02-05T07:24:24.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:24 smithi090 ceph-mon[33213]: 7.1b starting backfill to osd.2 from (0'0,0'0] MAX to 178'15281 2024-02-05T07:24:24.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:24 smithi090 ceph-mon[33213]: 7.23 starting backfill to osd.5 from (0'0,0'0] MAX to 178'8416 2024-02-05T07:24:24.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:24 smithi090 ceph-mon[33213]: 7.1c starting backfill to osd.5 from (0'0,0'0] MAX to 178'12233 2024-02-05T07:24:24.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:24 smithi090 ceph-mon[33213]: 7.1a starting backfill to osd.6 from (0'0,0'0] MAX to 178'12231 2024-02-05T07:24:24.778 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:24.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:24.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:24.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:24.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:24:24.918 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:24.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:24.974 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:24:25.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:25.061 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:25.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:25.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:25.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:25.219 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:25.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:25.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:25.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:25.357 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:25.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:25.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:25.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:25.495 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:25.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:25.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:25.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:25.631 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:25.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:25.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:25.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:25.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:25.763 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:25.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:25.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:25.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:25.896 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:25.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:25.944 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:25.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:26.025 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:26.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:26.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:26.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:26.151 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:26.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:26.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:26.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:26 smithi090 ceph-mon[33213]: pgmap v247: 168 pgs: 7 peering, 2 remapped+peering, 1 active+remapped+backfilling, 10 unknown, 148 active+clean; 1.0 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 244 op/s; 503/50496 objects misplaced (0.996%); 411 KiB/s, 6 objects/s recovering 2024-02-05T07:24:26.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:26 smithi090 ceph-mon[33213]: osdmap e182: 8 total, 8 up, 7 in 2024-02-05T07:24:26.292 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:26.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:26.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:26.419 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:26.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:26.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:26 smithi007 ceph-mon[29098]: pgmap v247: 168 pgs: 7 peering, 2 remapped+peering, 1 active+remapped+backfilling, 10 unknown, 148 active+clean; 1.0 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 244 op/s; 503/50496 objects misplaced (0.996%); 411 KiB/s, 6 objects/s recovering 2024-02-05T07:24:26.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:26 smithi007 ceph-mon[29098]: osdmap e182: 8 total, 8 up, 7 in 2024-02-05T07:24:26.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:26 smithi007 ceph-mon[37158]: pgmap v247: 168 pgs: 7 peering, 2 remapped+peering, 1 active+remapped+backfilling, 10 unknown, 148 active+clean; 1.0 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 244 op/s; 503/50496 objects misplaced (0.996%); 411 KiB/s, 6 objects/s recovering 2024-02-05T07:24:26.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:26 smithi007 ceph-mon[37158]: osdmap e182: 8 total, 8 up, 7 in 2024-02-05T07:24:26.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:24:26.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:26.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:26.546 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:26.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:26.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:26.670 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:26.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:26.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:26.788 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:24:26.789 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:24:26.796 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:26.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:26.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:26.921 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:26.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:26.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:27.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:27.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:27.047 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:27.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:27.149 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:27.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:27.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:27.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:27.312 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:27.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:27.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:27.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:27.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:27 smithi007 ceph-mon[29098]: osdmap e183: 8 total, 8 up, 7 in 2024-02-05T07:24:27.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:27 smithi007 ceph-mon[29098]: osdmap e184: 8 total, 8 up, 7 in 2024-02-05T07:24:27.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:27 smithi007 ceph-mon[37158]: osdmap e183: 8 total, 8 up, 7 in 2024-02-05T07:24:27.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:27 smithi007 ceph-mon[37158]: osdmap e184: 8 total, 8 up, 7 in 2024-02-05T07:24:27.437 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:27.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:27.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:27.445 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:27 smithi090 ceph-mon[33213]: osdmap e183: 8 total, 8 up, 7 in 2024-02-05T07:24:27.445 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:27 smithi090 ceph-mon[33213]: osdmap e184: 8 total, 8 up, 7 in 2024-02-05T07:24:27.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T07:24:27.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:27.576 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:27.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:27.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:27.721 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:27.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:27.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:27.805 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:24:27.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:27.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:27.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:27.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:27.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:28.007 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:28.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:28.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:28.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:28.146 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:28.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:28.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:28.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:28.272 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:28.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:28.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:28.409 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:28.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:28.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:28.511 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:28 smithi090 ceph-mon[33213]: pgmap v250: 168 pgs: 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 4 active+remapped+backfilling, 1 unknown, 158 active+clean; 897 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 4089/43614 objects misplaced (9.375%); 20 MiB/s, 440 objects/s recovering 2024-02-05T07:24:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:28 smithi090 ceph-mon[33213]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:24:28.549 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:28.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:28.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:28.651 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:28.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:28 smithi007 ceph-mon[29098]: pgmap v250: 168 pgs: 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 4 active+remapped+backfilling, 1 unknown, 158 active+clean; 897 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 4089/43614 objects misplaced (9.375%); 20 MiB/s, 440 objects/s recovering 2024-02-05T07:24:28.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:28 smithi007 ceph-mon[29098]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:24:28.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:28 smithi007 ceph-mon[37158]: pgmap v250: 168 pgs: 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 4 active+remapped+backfilling, 1 unknown, 158 active+clean; 897 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 4089/43614 objects misplaced (9.375%); 20 MiB/s, 440 objects/s recovering 2024-02-05T07:24:28.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:28 smithi007 ceph-mon[37158]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:24:28.688 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:28.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:28.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:28.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:28.824 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:28.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:28.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:28.966 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:28.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:29.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:29.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:29.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:29.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:29.245 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:29.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:29.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:29.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:29.382 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:29.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:29.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:29.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:29.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:29.508 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:29.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:29.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:29.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:29.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:29.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:29.784 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:29.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:29.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:29.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:29.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:29.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:30.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:30.058 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:30.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:30.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:30.080 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:24:30.080 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:24:30.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:30.194 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:30.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:30.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:30 smithi090 ceph-mon[33213]: pgmap v252: 168 pgs: 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 4 active+remapped+backfilling, 1 unknown, 158 active+clean; 897 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 930 B/s rd, 0 op/s; 4089/43614 objects misplaced (9.375%); 16 MiB/s, 360 objects/s recovering 2024-02-05T07:24:30.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:30 smithi090 ceph-mon[33213]: osdmap e185: 8 total, 8 up, 7 in 2024-02-05T07:24:30.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:30.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:24:30.275 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:24:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:24:30.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:30.332 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:30.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:30.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:30 smithi007 ceph-mon[29098]: pgmap v252: 168 pgs: 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 4 active+remapped+backfilling, 1 unknown, 158 active+clean; 897 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 930 B/s rd, 0 op/s; 4089/43614 objects misplaced (9.375%); 16 MiB/s, 360 objects/s recovering 2024-02-05T07:24:30.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:30 smithi007 ceph-mon[29098]: osdmap e185: 8 total, 8 up, 7 in 2024-02-05T07:24:30.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:30.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:24:30.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:30 smithi007 ceph-mon[37158]: pgmap v252: 168 pgs: 4 active+remapped+backfill_toofull, 1 active+clean+remapped, 4 active+remapped+backfilling, 1 unknown, 158 active+clean; 897 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 930 B/s rd, 0 op/s; 4089/43614 objects misplaced (9.375%); 16 MiB/s, 360 objects/s recovering 2024-02-05T07:24:30.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:30 smithi007 ceph-mon[37158]: osdmap e185: 8 total, 8 up, 7 in 2024-02-05T07:24:30.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:30.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:24:30.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:30.469 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:30.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:30.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:30.606 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:30.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:30.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:30.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:30.744 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:30.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:30.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:30.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:30.876 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:30.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:30.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:31.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:31.026 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:31.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:31.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T07:24:31.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:31.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:31.169 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:31.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:31.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:31 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:31.223 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:31 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:31.270 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:31.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:31 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:31.310 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:31.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:31.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:31.412 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:31.432 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:24:31.448 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:31.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:31.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:31.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:31.588 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:31.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:31.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:31.690 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:31.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:31.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:31.827 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:31.867 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:31.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:31.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:31.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:32.005 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:32.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:32.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:32.144 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:32.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:32.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:32.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:32.268 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:32.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:32.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:32.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:32.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:32.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:32.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:32 smithi090 ceph-mon[33213]: pgmap v254: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 576 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 880 B/s rd, 0 B/s wr, 1.08k op/s; 1733/28200 objects misplaced (6.145%); 22 MiB/s, 358 objects/s recovering 2024-02-05T07:24:32.542 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:32.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:32.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:32 smithi007 ceph-mon[29098]: pgmap v254: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 576 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 880 B/s rd, 0 B/s wr, 1.08k op/s; 1733/28200 objects misplaced (6.145%); 22 MiB/s, 358 objects/s recovering 2024-02-05T07:24:32.559 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:32 smithi007 ceph-mon[37158]: pgmap v254: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 576 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 880 B/s rd, 0 B/s wr, 1.08k op/s; 1733/28200 objects misplaced (6.145%); 22 MiB/s, 358 objects/s recovering 2024-02-05T07:24:32.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:32.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:32.689 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:32.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:32.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:32.833 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:32.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:32.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:32.935 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:32.971 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:32.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:32.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:33.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:33.112 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:33.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:33.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:33.213 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.0 primary-affinity to 0 (802) 2024-02-05T07:24:33.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:33.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:33.248 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:33.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:33 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3860137962' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-05T07:24:33.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:33 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:24:33.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:33 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:24:33.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:33.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:33.374 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:33.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:33.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:33.499 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:33.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:33.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:33 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3860137962' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-05T07:24:33.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:33 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:24:33.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:33 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:24:33.536 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:33 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3860137962' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-05T07:24:33.537 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:33 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:24:33.537 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:33 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:24:33.564 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:24:33.564 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:24:33.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:33.626 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:33.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:33.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:33.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:33.751 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:33.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:33.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:33.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:33.878 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:33.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:33.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:33.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:24:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:24:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:24:33.979 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:34.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:34.004 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:34.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:34.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:34.129 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:34.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:24:34.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:34.264 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:34.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:34.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:34.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:34 smithi007 ceph-mon[29098]: pgmap v255: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 576 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 751 B/s rd, 0 B/s wr, 754 op/s; 1733/28200 objects misplaced (6.145%); 11 MiB/s, 168 objects/s recovering 2024-02-05T07:24:34.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:34 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3860137962' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-02-05T07:24:34.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:34 smithi007 ceph-mon[29098]: osdmap e186: 8 total, 8 up, 7 in 2024-02-05T07:24:34.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:34 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1602370797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:24:34.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:34 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:24:34.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:34 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1602370797' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:24:34.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:34 smithi007 ceph-mon[29098]: osdmap e187: 8 total, 8 up, 7 in 2024-02-05T07:24:34.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:34 smithi007 ceph-mon[37158]: pgmap v255: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 576 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 751 B/s rd, 0 B/s wr, 754 op/s; 1733/28200 objects misplaced (6.145%); 11 MiB/s, 168 objects/s recovering 2024-02-05T07:24:34.402 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:34 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3860137962' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-02-05T07:24:34.402 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:34 smithi007 ceph-mon[37158]: osdmap e186: 8 total, 8 up, 7 in 2024-02-05T07:24:34.402 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:34 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1602370797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:24:34.402 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:34 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:24:34.402 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:34 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1602370797' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:24:34.402 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:34 smithi007 ceph-mon[37158]: osdmap e187: 8 total, 8 up, 7 in 2024-02-05T07:24:34.404 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:34.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:34.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:34.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:34 smithi090 ceph-mon[33213]: pgmap v255: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 576 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 751 B/s rd, 0 B/s wr, 754 op/s; 1733/28200 objects misplaced (6.145%); 11 MiB/s, 168 objects/s recovering 2024-02-05T07:24:34.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:34 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3860137962' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-02-05T07:24:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:34 smithi090 ceph-mon[33213]: osdmap e186: 8 total, 8 up, 7 in 2024-02-05T07:24:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:34 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1602370797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:24:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:34 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:24:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:34 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1602370797' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:24:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:34 smithi090 ceph-mon[33213]: osdmap e187: 8 total, 8 up, 7 in 2024-02-05T07:24:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:34.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:34.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:34.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:34.667 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:34.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:34.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:34.712 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:34.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:34.804 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:34.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:34.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:34.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:34.929 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:34.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:35.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:35.528 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:35 smithi090 ceph-mon[33213]: 7.1b continuing backfill to osd.0 from (154'11564,185'15653] MIN to 185'15653 2024-02-05T07:24:35.528 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:35 smithi090 ceph-mon[33213]: 7.1b continuing backfill to osd.2 from (154'11564,185'15653] MIN to 185'15653 2024-02-05T07:24:35.530 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:35 smithi007 ceph-mon[29098]: 7.1b continuing backfill to osd.0 from (154'11564,185'15653] MIN to 185'15653 2024-02-05T07:24:35.530 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:35 smithi007 ceph-mon[29098]: 7.1b continuing backfill to osd.2 from (154'11564,185'15653] MIN to 185'15653 2024-02-05T07:24:35.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:35 smithi007 ceph-mon[37158]: 7.1b continuing backfill to osd.0 from (154'11564,185'15653] MIN to 185'15653 2024-02-05T07:24:35.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:35 smithi007 ceph-mon[37158]: 7.1b continuing backfill to osd.2 from (154'11564,185'15653] MIN to 185'15653 2024-02-05T07:24:35.559 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:35.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:35.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:35.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:35.691 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:35.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:35.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:35.818 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:35.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:35.920 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:35.945 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:35.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:36.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:36.070 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:36.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:36.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:36.195 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:36.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:36.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:36.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:36.327 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:36.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:36.363 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:36 smithi007 ceph-mon[29098]: pgmap v258: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 576 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 856 op/s; 1733/28200 objects misplaced (6.145%); 12 MiB/s, 191 objects/s recovering 2024-02-05T07:24:36.363 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:36 smithi007 ceph-mon[29098]: osdmap e188: 8 total, 8 up, 7 in 2024-02-05T07:24:36.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:36 smithi007 ceph-mon[37158]: pgmap v258: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 576 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 856 op/s; 1733/28200 objects misplaced (6.145%); 12 MiB/s, 191 objects/s recovering 2024-02-05T07:24:36.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:36 smithi007 ceph-mon[37158]: osdmap e188: 8 total, 8 up, 7 in 2024-02-05T07:24:36.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:36.474 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:36.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:36.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:36 smithi090 ceph-mon[33213]: pgmap v258: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 576 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 856 op/s; 1733/28200 objects misplaced (6.145%); 12 MiB/s, 191 objects/s recovering 2024-02-05T07:24:36.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:36 smithi090 ceph-mon[33213]: osdmap e188: 8 total, 8 up, 7 in 2024-02-05T07:24:36.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:36.598 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:36.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:36.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:36.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:36.733 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:36.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:36.835 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:36.866 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:36.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:36.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:37.005 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:37.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:37.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:37.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:37.149 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:37.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:37.177 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:24:37.211 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:24:37.230 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:24:37.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:37.264 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:24:37.279 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:37.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:37.381 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:37.382 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:24:37.416 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:37.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:37.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:37.421 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:24:37.464 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:24:37.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:37 smithi007 ceph-mon[29098]: osdmap e189: 8 total, 8 up, 7 in 2024-02-05T07:24:37.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:37 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/999026860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:37.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:37 smithi007 ceph-mon[37158]: osdmap e189: 8 total, 8 up, 7 in 2024-02-05T07:24:37.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:37 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/999026860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:37.502 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:24:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:24:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:24:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:24:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:24:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:24:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:24:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:24:37.506 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:24:37.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:37.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:37 smithi090 ceph-mon[33213]: osdmap e189: 8 total, 8 up, 7 in 2024-02-05T07:24:37.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:37 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/999026860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:37.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:24:37.551 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:37.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:37.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:37.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:37.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:37.682 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:37.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:37.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:37.818 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:37.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:37.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:37.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:37.958 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:37.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:37.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:38.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:38.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:38.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:38.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:38.220 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:38.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:38.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:38.231 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Removed 104592 objects 2024-02-05T07:24:38.232 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Clean up completed and total clean up time :131.255 2024-02-05T07:24:38.295 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:38.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:38.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:38.346 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:38.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:38.438 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:38 smithi007 ceph-mon[29098]: pgmap v261: 168 pgs: 1 peering, 3 active+remapped+backfill_toofull, 164 active+clean; 205 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 1.48k op/s; 409/10395 objects misplaced (3.935%); 2.2 MiB/s, 33 objects/s recovering 2024-02-05T07:24:38.438 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:38 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:24:38.438 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:38 smithi007 ceph-mon[37158]: pgmap v261: 168 pgs: 1 peering, 3 active+remapped+backfill_toofull, 164 active+clean; 205 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 1.48k op/s; 409/10395 objects misplaced (3.935%); 2.2 MiB/s, 33 objects/s recovering 2024-02-05T07:24:38.438 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:38 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:24:38.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:38.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:38.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:38 smithi090 ceph-mon[33213]: pgmap v261: 168 pgs: 1 peering, 3 active+remapped+backfill_toofull, 164 active+clean; 205 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 1.48k op/s; 409/10395 objects misplaced (3.935%); 2.2 MiB/s, 33 objects/s recovering 2024-02-05T07:24:38.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:38 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:24:38.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:38.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:38.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:38.699 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:38.722 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:38.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:38.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:38.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:38.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:38.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:38.953 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:38.977 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:38.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:38.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:39.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:39.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:39.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:39.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:39.233 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:39.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:39.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:39.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:39.358 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:39.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:39.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:39.451 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T07:24:39.451 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:24:39.451 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-02-05T07:24:39.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:24:39.460 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:39.467 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T07:24:39.519 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:39.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:39.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:39.646 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:39.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:39.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:39.717 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:24:39.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:39.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:39.774 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:39.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:39.875 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:39.900 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:39.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:40.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:40.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:40.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:40.056 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:40.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:40.158 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:40.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:40.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:24:40.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:40.300 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:40.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:40.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:40.426 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:40.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:40.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:40 smithi007 ceph-mon[29098]: pgmap v262: 168 pgs: 1 peering, 3 active+remapped+backfill_toofull, 164 active+clean; 205 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 409/10395 objects misplaced (3.935%); 1.5 MiB/s, 23 objects/s recovering 2024-02-05T07:24:40.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:40 smithi007 ceph-mon[29098]: osdmap e190: 8 total, 8 up, 7 in 2024-02-05T07:24:40.445 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:40 smithi007 ceph-mon[37158]: pgmap v262: 168 pgs: 1 peering, 3 active+remapped+backfill_toofull, 164 active+clean; 205 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 409/10395 objects misplaced (3.935%); 1.5 MiB/s, 23 objects/s recovering 2024-02-05T07:24:40.445 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:40 smithi007 ceph-mon[37158]: osdmap e190: 8 total, 8 up, 7 in 2024-02-05T07:24:40.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:24:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:24:40.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:40 smithi090 ceph-mon[33213]: pgmap v262: 168 pgs: 1 peering, 3 active+remapped+backfill_toofull, 164 active+clean; 205 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 409/10395 objects misplaced (3.935%); 1.5 MiB/s, 23 objects/s recovering 2024-02-05T07:24:40.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:40 smithi090 ceph-mon[33213]: osdmap e190: 8 total, 8 up, 7 in 2024-02-05T07:24:40.528 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:40.551 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:40.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:40.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:40.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:40.678 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:40.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:40.779 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:40.807 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:40.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:40.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:40.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:40.933 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:40.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:40.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:41.036 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:41.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:41.062 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:41.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:41.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:41.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:41.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:24:41.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:41.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:41.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:41.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:41.444 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:41.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:41.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:41.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:41 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:41.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:41 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3500473523' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:24:41.446 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:41 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:41.446 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:41 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3500473523' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:24:41.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:41 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:41.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:41 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3500473523' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:24:41.546 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:41.573 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:41.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:41.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:41.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:41.701 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:41.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:41.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:41.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:41.830 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:41.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:41.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:41.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:41.956 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:41.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:41.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:42.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:42.082 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:42.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:42.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:42.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:42.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:42.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:42.340 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:42.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:42.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:42 smithi007 ceph-mon[37158]: pgmap v264: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 8.8 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.53k op/s; 77/984 objects misplaced (7.825%); 1.5 MiB/s, 22 objects/s recovering 2024-02-05T07:24:42.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:42 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:24:42.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:42 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3500473523' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:24:42.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:42 smithi007 ceph-mon[37158]: osdmap e191: 8 total, 8 up, 7 in 2024-02-05T07:24:42.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:42 smithi007 ceph-mon[29098]: pgmap v264: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 8.8 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.53k op/s; 77/984 objects misplaced (7.825%); 1.5 MiB/s, 22 objects/s recovering 2024-02-05T07:24:42.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:42 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:24:42.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:42 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3500473523' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:24:42.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:42 smithi007 ceph-mon[29098]: osdmap e191: 8 total, 8 up, 7 in 2024-02-05T07:24:42.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:42.469 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:42.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:42.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:42.499 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:42 smithi090 ceph-mon[33213]: pgmap v264: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 8.8 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.53k op/s; 77/984 objects misplaced (7.825%); 1.5 MiB/s, 22 objects/s recovering 2024-02-05T07:24:42.499 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:42 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:24:42.499 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:42 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3500473523' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:24:42.499 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:42 smithi090 ceph-mon[33213]: osdmap e191: 8 total, 8 up, 7 in 2024-02-05T07:24:42.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:42.595 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:42.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:42.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:42.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:42.722 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:42.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:42.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:42.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:42.852 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:42.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:42.953 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:42.977 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:42.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:43.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:43.102 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:43.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:43.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:43.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:43.231 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:43.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:43.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:43.245 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:43 smithi090 ceph-mon[33213]: 7.1b scrub starts 2024-02-05T07:24:43.245 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:43 smithi090 ceph-mon[33213]: 7.1b scrub ok 2024-02-05T07:24:43.245 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:43 smithi090 ceph-mon[33213]: 6.b scrub starts 2024-02-05T07:24:43.245 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:43 smithi090 ceph-mon[33213]: 6.b scrub ok 2024-02-05T07:24:43.245 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:43 smithi090 ceph-mon[33213]: 7.1a scrub starts 2024-02-05T07:24:43.245 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:43 smithi090 ceph-mon[33213]: 7.1a scrub ok 2024-02-05T07:24:43.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:43.358 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:43.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:43.460 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:43.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:43 smithi007 ceph-mon[29098]: 7.1b scrub starts 2024-02-05T07:24:43.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:43 smithi007 ceph-mon[29098]: 7.1b scrub ok 2024-02-05T07:24:43.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:43 smithi007 ceph-mon[29098]: 6.b scrub starts 2024-02-05T07:24:43.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:43 smithi007 ceph-mon[29098]: 6.b scrub ok 2024-02-05T07:24:43.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:43 smithi007 ceph-mon[29098]: 7.1a scrub starts 2024-02-05T07:24:43.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:43 smithi007 ceph-mon[29098]: 7.1a scrub ok 2024-02-05T07:24:43.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:43 smithi007 ceph-mon[37158]: 7.1b scrub starts 2024-02-05T07:24:43.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:43 smithi007 ceph-mon[37158]: 7.1b scrub ok 2024-02-05T07:24:43.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:43 smithi007 ceph-mon[37158]: 6.b scrub starts 2024-02-05T07:24:43.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:43 smithi007 ceph-mon[37158]: 6.b scrub ok 2024-02-05T07:24:43.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:43 smithi007 ceph-mon[37158]: 7.1a scrub starts 2024-02-05T07:24:43.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:43 smithi007 ceph-mon[37158]: 7.1a scrub ok 2024-02-05T07:24:43.487 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:43.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:43.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:43.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:43.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:43.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:24:43.625 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:24:43.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:43.740 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:43.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:43.808 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:24:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:24:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:24:43.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-02-05T07:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:24:43.864 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:43.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:43.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:43.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:43.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:44.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:44.114 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:24:44.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:44.118 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:44.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:44.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:44.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:44.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:44.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:44.344 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:44.369 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:44.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:44.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:44.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[37158]: 7.b scrub starts 2024-02-05T07:24:44.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[37158]: 7.b scrub ok 2024-02-05T07:24:44.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[37158]: 7.e scrub starts 2024-02-05T07:24:44.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[37158]: 7.e scrub ok 2024-02-05T07:24:44.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[37158]: pgmap v266: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 8.8 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 753 B/s rd, 0 B/s wr, 481 op/s; 77/984 objects misplaced (7.825%) 2024-02-05T07:24:44.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[37158]: 7.22 scrub starts 2024-02-05T07:24:44.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[37158]: 7.22 scrub ok 2024-02-05T07:24:44.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[37158]: osdmap e192: 8 total, 8 up, 7 in 2024-02-05T07:24:44.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[29098]: 7.b scrub starts 2024-02-05T07:24:44.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[29098]: 7.b scrub ok 2024-02-05T07:24:44.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[29098]: 7.e scrub starts 2024-02-05T07:24:44.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[29098]: 7.e scrub ok 2024-02-05T07:24:44.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[29098]: pgmap v266: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 8.8 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 753 B/s rd, 0 B/s wr, 481 op/s; 77/984 objects misplaced (7.825%) 2024-02-05T07:24:44.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[29098]: 7.22 scrub starts 2024-02-05T07:24:44.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[29098]: 7.22 scrub ok 2024-02-05T07:24:44.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:44 smithi007 ceph-mon[29098]: osdmap e192: 8 total, 8 up, 7 in 2024-02-05T07:24:44.458 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:44 smithi090 ceph-mon[33213]: 7.b scrub starts 2024-02-05T07:24:44.458 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:44 smithi090 ceph-mon[33213]: 7.b scrub ok 2024-02-05T07:24:44.458 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:44 smithi090 ceph-mon[33213]: 7.e scrub starts 2024-02-05T07:24:44.458 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:44 smithi090 ceph-mon[33213]: 7.e scrub ok 2024-02-05T07:24:44.458 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:44 smithi090 ceph-mon[33213]: pgmap v266: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 8.8 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 753 B/s rd, 0 B/s wr, 481 op/s; 77/984 objects misplaced (7.825%) 2024-02-05T07:24:44.458 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:44 smithi090 ceph-mon[33213]: 7.22 scrub starts 2024-02-05T07:24:44.458 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:44 smithi090 ceph-mon[33213]: 7.22 scrub ok 2024-02-05T07:24:44.458 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:44 smithi090 ceph-mon[33213]: osdmap e192: 8 total, 8 up, 7 in 2024-02-05T07:24:44.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:44.474 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-02-05T07:24:43.024794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9181520,"num_objects":328,"num_object_clones":0,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":77,"num_objects_unfound":0,"num_objects_dirty":328,"num_whiteouts":0,"num_read":1267,"num_read_kb":1113,"num_write":127574,"num_write_kb":2878174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16487,"num_bytes_recovered":986910543,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":169715,"ondisk_log_size":169715,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":519,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1521960,"kb_used_data":150044,"kb_used_omap":10,"kb_used_meta":1371829,"kb_avail":654694104,"statfs":{"total":671965249536,"available":670406762496,"internally_reserved":0,"allocated":153645056,"data_stored":150453719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11159,"internal_metadata":1404753001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-205488152,"num_objects":-3137,"num_object_clones":0,"num_object_copies":-9411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-332,"num_objects_unfound":0,"num_objects_dirty":-3137,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3267,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.797196"},"pg_stats":[{"pgid":"7.23","version":"189'8678","reported_seq":12234,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666494+0000","last_change":"2024-02-05T07:24:27.263897+0000","last_active":"2024-02-05T07:24:39.666494+0000","last_peered":"2024-02-05T07:24:39.666494+0000","last_clean":"2024-02-05T07:24:39.666494+0000","last_became_active":"2024-02-05T07:24:27.248368+0000","last_became_peered":"2024-02-05T07:24:27.248368+0000","last_unstale":"2024-02-05T07:24:39.666494+0000","last_undegraded":"2024-02-05T07:24:39.666494+0000","last_fullsized":"2024-02-05T07:24:39.666494+0000","mapping_epoch":183,"log_start":"110'2776","ondisk_log_start":"110'2776","created":177,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":6,"last_scrub":"119'4694","last_scrub_stamp":"2024-02-05T07:21:44.743455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:44.743455+0000","objects_scrubbed":2359,"log_size":5902,"log_dups_size":0,"ondisk_log_size":5902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.357580+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2124,"num_write_kb":30016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":57737216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"189'8683","reported_seq":12001,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193856+0000","last_change":"2024-02-05T07:24:27.263895+0000","last_active":"2024-02-05T07:24:39.193856+0000","last_peered":"2024-02-05T07:24:39.193856+0000","last_clean":"2024-02-05T07:24:39.193856+0000","last_became_active":"2024-02-05T07:24:27.251006+0000","last_became_peered":"2024-02-05T07:24:27.251006+0000","last_unstale":"2024-02-05T07:24:39.193856+0000","last_undegraded":"2024-02-05T07:24:39.193856+0000","last_fullsized":"2024-02-05T07:24:39.193856+0000","mapping_epoch":183,"log_start":"122'4716","ondisk_log_start":"122'4716","created":177,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":6,"last_scrub":"120'4711","last_scrub_stamp":"2024-02-05T07:21:43.941589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.941589+0000","objects_scrubbed":2370,"log_size":3967,"log_dups_size":0,"ondisk_log_size":3967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.712124+0000","scrub_duration":475,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2115,"num_write_kb":29952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":878,"num_bytes_recovered":57507840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":301,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.373051+0000","last_change":"2024-02-05T07:23:47.629122+0000","last_active":"2024-02-05T07:24:39.373051+0000","last_peered":"2024-02-05T07:24:39.373051+0000","last_clean":"2024-02-05T07:24:39.373051+0000","last_became_active":"2024-02-05T07:23:47.597686+0000","last_became_peered":"2024-02-05T07:23:47.597686+0000","last_unstale":"2024-02-05T07:24:39.373051+0000","last_undegraded":"2024-02-05T07:24:39.373051+0000","last_fullsized":"2024-02-05T07:24:39.373051+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:23:04.538922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:04.538922+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.372572+0000","last_change":"2024-02-05T07:24:34.251845+0000","last_active":"2024-02-05T07:24:39.372572+0000","last_peered":"2024-02-05T07:24:39.372572+0000","last_clean":"2024-02-05T07:24:39.372572+0000","last_became_active":"2024-02-05T07:24:34.251554+0000","last_became_peered":"2024-02-05T07:24:34.251554+0000","last_unstale":"2024-02-05T07:24:39.372572+0000","last_undegraded":"2024-02-05T07:24:39.372572+0000","last_fullsized":"2024-02-05T07:24:39.372572+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:13.198115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:13.198115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:13.263707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":346,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194728+0000","last_change":"2024-02-05T07:21:27.645530+0000","last_active":"2024-02-05T07:24:39.194728+0000","last_peered":"2024-02-05T07:24:39.194728+0000","last_clean":"2024-02-05T07:24:39.194728+0000","last_became_active":"2024-02-05T07:18:53.461852+0000","last_became_peered":"2024-02-05T07:18:53.461852+0000","last_unstale":"2024-02-05T07:24:39.194728+0000","last_undegraded":"2024-02-05T07:24:39.194728+0000","last_fullsized":"2024-02-05T07:24:39.194728+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:21:27.645263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:27.645263+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741826+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":260,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.373680+0000","last_change":"2024-02-05T07:23:09.542975+0000","last_active":"2024-02-05T07:24:39.373680+0000","last_peered":"2024-02-05T07:24:39.373680+0000","last_clean":"2024-02-05T07:24:39.373680+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:24:39.373680+0000","last_undegraded":"2024-02-05T07:24:39.373680+0000","last_fullsized":"2024-02-05T07:24:39.373680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:09.542903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:09.542903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"189'12633","reported_seq":13923,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.670542+0000","last_change":"2024-02-05T07:24:34.286889+0000","last_active":"2024-02-05T07:24:39.670542+0000","last_peered":"2024-02-05T07:24:39.670542+0000","last_clean":"2024-02-05T07:24:39.670542+0000","last_became_active":"2024-02-05T07:24:34.246368+0000","last_became_peered":"2024-02-05T07:24:34.246368+0000","last_unstale":"2024-02-05T07:24:39.670542+0000","last_undegraded":"2024-02-05T07:24:39.670542+0000","last_fullsized":"2024-02-05T07:24:39.670542+0000","mapping_epoch":186,"log_start":"141'7605","ondisk_log_start":"141'7605","created":100,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":5028,"log_dups_size":0,"ondisk_log_size":5028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.244914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6583,"num_write_kb":210752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":34668544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":296,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.372954+0000","last_change":"2024-02-05T07:21:29.926437+0000","last_active":"2024-02-05T07:24:39.372954+0000","last_peered":"2024-02-05T07:24:39.372954+0000","last_clean":"2024-02-05T07:24:39.372954+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:24:39.372954+0000","last_undegraded":"2024-02-05T07:24:39.372954+0000","last_fullsized":"2024-02-05T07:24:39.372954+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:21:29.926339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:29.926339+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":272,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668707+0000","last_change":"2024-02-05T07:24:14.277115+0000","last_active":"2024-02-05T07:24:39.668707+0000","last_peered":"2024-02-05T07:24:39.668707+0000","last_clean":"2024-02-05T07:24:39.668707+0000","last_became_active":"2024-02-05T07:18:49.430518+0000","last_became_peered":"2024-02-05T07:18:49.430518+0000","last_unstale":"2024-02-05T07:24:39.668707+0000","last_undegraded":"2024-02-05T07:24:39.668707+0000","last_fullsized":"2024-02-05T07:24:39.668707+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:14.277005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:14.277005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:34.125024+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":265,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.196430+0000","last_change":"2024-02-05T07:21:17.402214+0000","last_active":"2024-02-05T07:24:39.196430+0000","last_peered":"2024-02-05T07:24:39.196430+0000","last_clean":"2024-02-05T07:24:39.196430+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:24:39.196430+0000","last_undegraded":"2024-02-05T07:24:39.196430+0000","last_fullsized":"2024-02-05T07:24:39.196430+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:17.401959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:17.401959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670845+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":258,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194784+0000","last_change":"2024-02-05T07:23:47.609782+0000","last_active":"2024-02-05T07:24:39.194784+0000","last_peered":"2024-02-05T07:24:39.194784+0000","last_clean":"2024-02-05T07:24:39.194784+0000","last_became_active":"2024-02-05T07:23:47.609458+0000","last_became_peered":"2024-02-05T07:23:47.609458+0000","last_unstale":"2024-02-05T07:24:39.194784+0000","last_undegraded":"2024-02-05T07:24:39.194784+0000","last_fullsized":"2024-02-05T07:24:39.194784+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:29.670866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:29.670866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"189'12361","reported_seq":12563,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.195214+0000","last_change":"2024-02-05T07:24:20.541301+0000","last_active":"2024-02-05T07:24:39.195214+0000","last_peered":"2024-02-05T07:24:39.195214+0000","last_clean":"2024-02-05T07:24:39.195214+0000","last_became_active":"2024-02-05T07:24:20.518205+0000","last_became_peered":"2024-02-05T07:24:20.518205+0000","last_unstale":"2024-02-05T07:24:39.195214+0000","last_undegraded":"2024-02-05T07:24:39.195214+0000","last_fullsized":"2024-02-05T07:24:39.195214+0000","mapping_epoch":100,"log_start":"137'7084","ondisk_log_start":"137'7084","created":100,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6455,"num_write_kb":205760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"90'16","reported_seq":292,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.373438+0000","last_change":"2024-02-05T07:21:43.811840+0000","last_active":"2024-02-05T07:24:39.373438+0000","last_peered":"2024-02-05T07:24:39.373438+0000","last_clean":"2024-02-05T07:24:39.373438+0000","last_became_active":"2024-02-05T07:18:51.447922+0000","last_became_peered":"2024-02-05T07:18:51.447922+0000","last_unstale":"2024-02-05T07:24:39.373438+0000","last_undegraded":"2024-02-05T07:24:39.373438+0000","last_fullsized":"2024-02-05T07:24:39.373438+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-02-05T07:21:43.811596+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.811596+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":272,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194077+0000","last_change":"2024-02-05T07:23:02.519644+0000","last_active":"2024-02-05T07:24:39.194077+0000","last_peered":"2024-02-05T07:24:39.194077+0000","last_clean":"2024-02-05T07:24:39.194077+0000","last_became_active":"2024-02-05T07:18:49.434074+0000","last_became_peered":"2024-02-05T07:18:49.434074+0000","last_unstale":"2024-02-05T07:24:39.194077+0000","last_undegraded":"2024-02-05T07:24:39.194077+0000","last_fullsized":"2024-02-05T07:24:39.194077+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:02.519566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:02.519566+0000","last_clean_scrub_stamp":"2024-02-05T07:23:02.519566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":86,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.373658+0000","last_change":"2024-02-05T07:23:27.417952+0000","last_active":"2024-02-05T07:24:39.373658+0000","last_peered":"2024-02-05T07:24:39.373658+0000","last_clean":"2024-02-05T07:24:39.373658+0000","last_became_active":"2024-02-05T07:23:27.417675+0000","last_became_peered":"2024-02-05T07:23:27.417675+0000","last_unstale":"2024-02-05T07:24:39.373658+0000","last_undegraded":"2024-02-05T07:24:39.373658+0000","last_fullsized":"2024-02-05T07:24:39.373658+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:23.823835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:23.823835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":74,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668027+0000","last_change":"2024-02-05T07:24:08.261218+0000","last_active":"2024-02-05T07:24:39.668027+0000","last_peered":"2024-02-05T07:24:39.668027+0000","last_clean":"2024-02-05T07:24:39.668027+0000","last_became_active":"2024-02-05T07:23:47.611030+0000","last_became_peered":"2024-02-05T07:23:47.611030+0000","last_unstale":"2024-02-05T07:24:39.668027+0000","last_undegraded":"2024-02-05T07:24:39.668027+0000","last_fullsized":"2024-02-05T07:24:39.668027+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:08.261123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:08.261123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:37.699207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.d","version":"189'12608","reported_seq":12825,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.669683+0000","last_change":"2024-02-05T07:24:21.354782+0000","last_active":"2024-02-05T07:24:39.669683+0000","last_peered":"2024-02-05T07:24:39.669683+0000","last_clean":"2024-02-05T07:24:39.669683+0000","last_became_active":"2024-02-05T07:24:20.518342+0000","last_became_peered":"2024-02-05T07:24:20.518342+0000","last_unstale":"2024-02-05T07:24:39.669683+0000","last_undegraded":"2024-02-05T07:24:39.669683+0000","last_fullsized":"2024-02-05T07:24:39.669683+0000","mapping_epoch":100,"log_start":"141'7524","ondisk_log_start":"141'7524","created":100,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'12096","last_scrub_stamp":"2024-02-05T07:24:21.354684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.354684+0000","objects_scrubbed":512,"log_size":5084,"log_dups_size":0,"ondisk_log_size":5084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:38.401974+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":512,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"98'22","reported_seq":323,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.195928+0000","last_change":"2024-02-05T07:23:47.637874+0000","last_active":"2024-02-05T07:24:39.195928+0000","last_peered":"2024-02-05T07:24:39.195928+0000","last_clean":"2024-02-05T07:24:39.195928+0000","last_became_active":"2024-02-05T07:23:47.606483+0000","last_became_peered":"2024-02-05T07:23:47.606483+0000","last_unstale":"2024-02-05T07:24:39.195928+0000","last_undegraded":"2024-02-05T07:24:39.195928+0000","last_fullsized":"2024-02-05T07:24:39.195928+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'22","last_scrub_stamp":"2024-02-05T07:23:19.000981+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:23:19.000981+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.671110+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":66,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194010+0000","last_change":"2024-02-05T07:23:47.600893+0000","last_active":"2024-02-05T07:24:39.194010+0000","last_peered":"2024-02-05T07:24:39.194010+0000","last_clean":"2024-02-05T07:24:39.194010+0000","last_became_active":"2024-02-05T07:23:47.600273+0000","last_became_peered":"2024-02-05T07:23:47.600273+0000","last_unstale":"2024-02-05T07:24:39.194010+0000","last_undegraded":"2024-02-05T07:24:39.194010+0000","last_fullsized":"2024-02-05T07:24:39.194010+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:37.367711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.367711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":343,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.373629+0000","last_change":"2024-02-05T07:24:19.650665+0000","last_active":"2024-02-05T07:24:39.373629+0000","last_peered":"2024-02-05T07:24:39.373629+0000","last_clean":"2024-02-05T07:24:39.373629+0000","last_became_active":"2024-02-05T07:18:53.459832+0000","last_became_peered":"2024-02-05T07:18:53.459832+0000","last_unstale":"2024-02-05T07:24:39.373629+0000","last_undegraded":"2024-02-05T07:24:39.373629+0000","last_fullsized":"2024-02-05T07:24:39.373629+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:24:19.650521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:19.650521+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:23.514384+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":252,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666396+0000","last_change":"2024-02-05T07:20:30.285373+0000","last_active":"2024-02-05T07:24:39.666396+0000","last_peered":"2024-02-05T07:24:39.666396+0000","last_clean":"2024-02-05T07:24:39.666396+0000","last_became_active":"2024-02-05T07:18:55.472172+0000","last_became_peered":"2024-02-05T07:18:55.472172+0000","last_unstale":"2024-02-05T07:24:39.666396+0000","last_undegraded":"2024-02-05T07:24:39.666396+0000","last_fullsized":"2024-02-05T07:24:39.666396+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:30.285296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:30.285296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.864073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"189'12758","reported_seq":17571,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668119+0000","last_change":"2024-02-05T07:24:20.556987+0000","last_active":"2024-02-05T07:24:39.668119+0000","last_peered":"2024-02-05T07:24:39.668119+0000","last_clean":"2024-02-05T07:24:39.668119+0000","last_became_active":"2024-02-05T07:24:20.539178+0000","last_became_peered":"2024-02-05T07:24:20.539178+0000","last_unstale":"2024-02-05T07:24:39.668119+0000","last_undegraded":"2024-02-05T07:24:39.668119+0000","last_fullsized":"2024-02-05T07:24:39.668119+0000","mapping_epoch":172,"log_start":"142'7721","ondisk_log_start":"142'7721","created":100,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":224,"log_size":5037,"log_dups_size":0,"ondisk_log_size":5037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.357447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6534,"num_write_kb":208384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1146,"num_bytes_recovered":75071488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":270,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193955+0000","last_change":"2024-02-05T07:23:47.626460+0000","last_active":"2024-02-05T07:24:39.193955+0000","last_peered":"2024-02-05T07:24:39.193955+0000","last_clean":"2024-02-05T07:24:39.193955+0000","last_became_active":"2024-02-05T07:23:47.626072+0000","last_became_peered":"2024-02-05T07:23:47.626072+0000","last_unstale":"2024-02-05T07:24:39.193955+0000","last_undegraded":"2024-02-05T07:24:39.193955+0000","last_fullsized":"2024-02-05T07:24:39.193955+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:03.505163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:23:03.505163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":143,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668119+0000","last_change":"2024-02-05T07:24:14.360014+0000","last_active":"2024-02-05T07:24:39.668119+0000","last_peered":"2024-02-05T07:24:39.668119+0000","last_clean":"2024-02-05T07:24:39.668119+0000","last_became_active":"2024-02-05T07:23:47.618052+0000","last_became_peered":"2024-02-05T07:23:47.618052+0000","last_unstale":"2024-02-05T07:24:39.668119+0000","last_undegraded":"2024-02-05T07:24:39.668119+0000","last_fullsized":"2024-02-05T07:24:39.668119+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:24:14.359892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:14.359892+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:34.362411+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":66,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.670321+0000","last_change":"2024-02-05T07:23:47.611708+0000","last_active":"2024-02-05T07:24:39.670321+0000","last_peered":"2024-02-05T07:24:39.670321+0000","last_clean":"2024-02-05T07:24:39.670321+0000","last_became_active":"2024-02-05T07:23:47.609054+0000","last_became_peered":"2024-02-05T07:23:47.609054+0000","last_unstale":"2024-02-05T07:24:39.670321+0000","last_undegraded":"2024-02-05T07:24:39.670321+0000","last_fullsized":"2024-02-05T07:24:39.670321+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:42.464894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:42.464894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":266,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.372563+0000","last_change":"2024-02-05T07:23:47.603895+0000","last_active":"2024-02-05T07:24:39.372563+0000","last_peered":"2024-02-05T07:24:39.372563+0000","last_clean":"2024-02-05T07:24:39.372563+0000","last_became_active":"2024-02-05T07:23:47.603326+0000","last_became_peered":"2024-02-05T07:23:47.603326+0000","last_unstale":"2024-02-05T07:24:39.372563+0000","last_undegraded":"2024-02-05T07:24:39.372563+0000","last_fullsized":"2024-02-05T07:24:39.372563+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:16.499591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:16.499591+0000","last_clean_scrub_stamp":"2024-02-05T07:23:16.499591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"189'8665","reported_seq":11477,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194242+0000","last_change":"2024-02-05T07:24:20.565531+0000","last_active":"2024-02-05T07:24:39.194242+0000","last_peered":"2024-02-05T07:24:39.194242+0000","last_clean":"2024-02-05T07:24:39.194242+0000","last_became_active":"2024-02-05T07:24:20.538846+0000","last_became_peered":"2024-02-05T07:24:20.538846+0000","last_unstale":"2024-02-05T07:24:39.194242+0000","last_undegraded":"2024-02-05T07:24:39.194242+0000","last_fullsized":"2024-02-05T07:24:39.194242+0000","mapping_epoch":167,"log_start":"110'2776","ondisk_log_start":"110'2776","created":100,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'4694","last_scrub_stamp":"2024-02-05T07:21:44.743455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:44.743455+0000","objects_scrubbed":2359,"log_size":5889,"log_dups_size":0,"ondisk_log_size":5889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741675+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425984,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2111,"num_write_kb":30016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":638,"num_bytes_recovered":41811968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"164'9","reported_seq":114,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193416+0000","last_change":"2024-02-05T07:24:20.453876+0000","last_active":"2024-02-05T07:24:39.193416+0000","last_peered":"2024-02-05T07:24:39.193416+0000","last_clean":"2024-02-05T07:24:39.193416+0000","last_became_active":"2024-02-05T07:23:47.626082+0000","last_became_peered":"2024-02-05T07:23:47.626082+0000","last_unstale":"2024-02-05T07:24:39.193416+0000","last_undegraded":"2024-02-05T07:24:39.193416+0000","last_fullsized":"2024-02-05T07:24:39.193416+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:24:20.453746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:20.453746+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:32.518804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":281,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.372790+0000","last_change":"2024-02-05T07:21:19.981440+0000","last_active":"2024-02-05T07:24:39.372790+0000","last_peered":"2024-02-05T07:24:39.372790+0000","last_clean":"2024-02-05T07:24:39.372790+0000","last_became_active":"2024-02-05T07:18:51.438507+0000","last_became_peered":"2024-02-05T07:18:51.438507+0000","last_unstale":"2024-02-05T07:24:39.372790+0000","last_undegraded":"2024-02-05T07:24:39.372790+0000","last_fullsized":"2024-02-05T07:24:39.372790+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-02-05T07:21:19.981349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:19.981349+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":26,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194693+0000","last_change":"2024-02-05T07:24:34.238802+0000","last_active":"2024-02-05T07:24:39.194693+0000","last_peered":"2024-02-05T07:24:39.194693+0000","last_clean":"2024-02-05T07:24:39.194693+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:24:39.194693+0000","last_undegraded":"2024-02-05T07:24:39.194693+0000","last_fullsized":"2024-02-05T07:24:39.194693+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:21:39.233796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.233796+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":252,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666274+0000","last_change":"2024-02-05T07:21:22.584921+0000","last_active":"2024-02-05T07:24:39.666274+0000","last_peered":"2024-02-05T07:24:39.666274+0000","last_clean":"2024-02-05T07:24:39.666274+0000","last_became_active":"2024-02-05T07:18:55.507791+0000","last_became_peered":"2024-02-05T07:18:55.507791+0000","last_unstale":"2024-02-05T07:24:39.666274+0000","last_undegraded":"2024-02-05T07:24:39.666274+0000","last_fullsized":"2024-02-05T07:24:39.666274+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:22.584770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:22.584770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"189'8695","reported_seq":11540,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194694+0000","last_change":"2024-02-05T07:24:20.541863+0000","last_active":"2024-02-05T07:24:39.194694+0000","last_peered":"2024-02-05T07:24:39.194694+0000","last_clean":"2024-02-05T07:24:39.194694+0000","last_became_active":"2024-02-05T07:24:20.517520+0000","last_became_peered":"2024-02-05T07:24:20.517520+0000","last_unstale":"2024-02-05T07:24:39.194694+0000","last_undegraded":"2024-02-05T07:24:39.194694+0000","last_fullsized":"2024-02-05T07:24:39.194694+0000","mapping_epoch":167,"log_start":"122'4716","ondisk_log_start":"122'4716","created":100,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'4711","last_scrub_stamp":"2024-02-05T07:21:43.941589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.941589+0000","objects_scrubbed":2370,"log_size":3979,"log_dups_size":0,"ondisk_log_size":3979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.688122+0000","scrub_duration":475,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2127,"num_write_kb":29952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":647,"num_bytes_recovered":42434560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"187'9","reported_seq":45,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667211+0000","last_change":"2024-02-05T07:24:34.230867+0000","last_active":"2024-02-05T07:24:39.667211+0000","last_peered":"2024-02-05T07:24:39.667211+0000","last_clean":"2024-02-05T07:24:39.667211+0000","last_became_active":"2024-02-05T07:24:34.230336+0000","last_became_peered":"2024-02-05T07:24:34.230336+0000","last_unstale":"2024-02-05T07:24:39.667211+0000","last_undegraded":"2024-02-05T07:24:39.667211+0000","last_fullsized":"2024-02-05T07:24:39.667211+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:21:36.187453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:36.187453+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.367767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"151'15","reported_seq":297,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.196172+0000","last_change":"2024-02-05T07:23:26.247211+0000","last_active":"2024-02-05T07:24:39.196172+0000","last_peered":"2024-02-05T07:24:39.196172+0000","last_clean":"2024-02-05T07:24:39.196172+0000","last_became_active":"2024-02-05T07:18:51.448044+0000","last_became_peered":"2024-02-05T07:18:51.448044+0000","last_unstale":"2024-02-05T07:24:39.196172+0000","last_undegraded":"2024-02-05T07:24:39.196172+0000","last_fullsized":"2024-02-05T07:24:39.196172+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:23:26.247047+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.671128+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":270,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666764+0000","last_change":"2024-02-05T07:23:47.605453+0000","last_active":"2024-02-05T07:24:39.666764+0000","last_peered":"2024-02-05T07:24:39.666764+0000","last_clean":"2024-02-05T07:24:39.666764+0000","last_became_active":"2024-02-05T07:23:47.604997+0000","last_became_peered":"2024-02-05T07:23:47.604997+0000","last_unstale":"2024-02-05T07:24:39.666764+0000","last_undegraded":"2024-02-05T07:24:39.666764+0000","last_fullsized":"2024-02-05T07:24:39.666764+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:20:32.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.864103+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":213,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193179+0000","last_change":"2024-02-05T07:24:34.231420+0000","last_active":"2024-02-05T07:24:39.193179+0000","last_peered":"2024-02-05T07:24:39.193179+0000","last_clean":"2024-02-05T07:24:39.193179+0000","last_became_active":"2024-02-05T07:24:34.229242+0000","last_became_peered":"2024-02-05T07:24:34.229242+0000","last_unstale":"2024-02-05T07:24:39.193179+0000","last_undegraded":"2024-02-05T07:24:39.193179+0000","last_fullsized":"2024-02-05T07:24:39.193179+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:27.196824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:27.196824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741855+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"189'8806","reported_seq":9295,"reported_epoch":190,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:24:39.193620+0000","last_change":"2024-02-05T07:24:20.539501+0000","last_active":"2024-02-05T07:24:39.193620+0000","last_peered":"2024-02-05T07:24:39.193620+0000","last_clean":"2024-02-05T07:23:46.544320+0000","last_became_active":"2024-02-05T07:24:20.515728+0000","last_became_peered":"2024-02-05T07:24:20.515728+0000","last_unstale":"2024-02-05T07:24:39.193620+0000","last_undegraded":"2024-02-05T07:24:39.193620+0000","last_fullsized":"2024-02-05T07:24:39.193620+0000","mapping_epoch":163,"log_start":"110'3615","ondisk_log_start":"110'3615","created":100,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'4712","last_scrub_stamp":"2024-02-05T07:21:43.408403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.408403+0000","objects_scrubbed":2431,"log_size":5191,"log_dups_size":0,"ondisk_log_size":5191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":616,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2196,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,3],"avail_no_missing":["4","1","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":264,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668894+0000","last_change":"2024-02-05T07:21:37.374835+0000","last_active":"2024-02-05T07:24:39.668894+0000","last_peered":"2024-02-05T07:24:39.668894+0000","last_clean":"2024-02-05T07:24:39.668894+0000","last_became_active":"2024-02-05T07:18:53.459474+0000","last_became_peered":"2024-02-05T07:18:53.459474+0000","last_unstale":"2024-02-05T07:24:39.668894+0000","last_undegraded":"2024-02-05T07:24:39.668894+0000","last_fullsized":"2024-02-05T07:24:39.668894+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.374703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.367799+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":73,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668764+0000","last_change":"2024-02-05T07:24:34.229757+0000","last_active":"2024-02-05T07:24:39.668764+0000","last_peered":"2024-02-05T07:24:39.668764+0000","last_clean":"2024-02-05T07:24:39.668764+0000","last_became_active":"2024-02-05T07:24:34.229372+0000","last_became_peered":"2024-02-05T07:24:34.229372+0000","last_unstale":"2024-02-05T07:24:39.668764+0000","last_undegraded":"2024-02-05T07:24:39.668764+0000","last_fullsized":"2024-02-05T07:24:39.668764+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:20:12.309238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:12.309238+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.864014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":273,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.196281+0000","last_change":"2024-02-05T07:21:40.542969+0000","last_active":"2024-02-05T07:24:39.196281+0000","last_peered":"2024-02-05T07:24:39.196281+0000","last_clean":"2024-02-05T07:24:39.196281+0000","last_became_active":"2024-02-05T07:18:49.428470+0000","last_became_peered":"2024-02-05T07:18:49.428470+0000","last_unstale":"2024-02-05T07:24:39.196281+0000","last_undegraded":"2024-02-05T07:24:39.196281+0000","last_fullsized":"2024-02-05T07:24:39.196281+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:40.542479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:21:40.542479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670976+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":261,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194172+0000","last_change":"2024-02-05T07:23:01.849413+0000","last_active":"2024-02-05T07:24:39.194172+0000","last_peered":"2024-02-05T07:24:39.194172+0000","last_clean":"2024-02-05T07:24:39.194172+0000","last_became_active":"2024-02-05T07:18:55.505360+0000","last_became_peered":"2024-02-05T07:18:55.505360+0000","last_unstale":"2024-02-05T07:24:39.194172+0000","last_undegraded":"2024-02-05T07:24:39.194172+0000","last_fullsized":"2024-02-05T07:24:39.194172+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:01.849346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:01.849346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.671056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"189'8396","reported_seq":13873,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666769+0000","last_change":"2024-02-05T07:24:22.312146+0000","last_active":"2024-02-05T07:24:39.666769+0000","last_peered":"2024-02-05T07:24:39.666769+0000","last_clean":"2024-02-05T07:24:39.666769+0000","last_became_active":"2024-02-05T07:24:21.528361+0000","last_became_peered":"2024-02-05T07:24:21.528361+0000","last_unstale":"2024-02-05T07:24:39.666769+0000","last_undegraded":"2024-02-05T07:24:39.666769+0000","last_fullsized":"2024-02-05T07:24:39.666769+0000","mapping_epoch":178,"log_start":"108'1895","ondisk_log_start":"108'1895","created":100,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'4604","last_scrub_stamp":"2024-02-05T07:21:43.479351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.479351+0000","objects_scrubbed":2300,"log_size":6501,"log_dups_size":0,"ondisk_log_size":6501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:39.907538+0000","scrub_duration":618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2029,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1579,"num_bytes_recovered":42205184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":256,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668950+0000","last_change":"2024-02-05T07:20:31.238357+0000","last_active":"2024-02-05T07:24:39.668950+0000","last_peered":"2024-02-05T07:24:39.668950+0000","last_clean":"2024-02-05T07:24:39.668950+0000","last_became_active":"2024-02-05T07:18:53.468512+0000","last_became_peered":"2024-02-05T07:18:53.468512+0000","last_unstale":"2024-02-05T07:24:39.668950+0000","last_undegraded":"2024-02-05T07:24:39.668950+0000","last_fullsized":"2024-02-05T07:24:39.668950+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:31.238251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:31.238251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.864045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":55,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.195502+0000","last_change":"2024-02-05T07:24:34.210115+0000","last_active":"2024-02-05T07:24:39.195502+0000","last_peered":"2024-02-05T07:24:39.195502+0000","last_clean":"2024-02-05T07:24:39.195502+0000","last_became_active":"2024-02-05T07:24:34.209716+0000","last_became_peered":"2024-02-05T07:24:34.209716+0000","last_unstale":"2024-02-05T07:24:39.195502+0000","last_undegraded":"2024-02-05T07:24:39.195502+0000","last_fullsized":"2024-02-05T07:24:39.195502+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:24:15.127119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:15.127119+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:22.751406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":619,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667869+0000","last_change":"2024-02-05T07:20:34.765770+0000","last_active":"2024-02-05T07:24:39.667869+0000","last_peered":"2024-02-05T07:24:39.667869+0000","last_clean":"2024-02-05T07:24:39.667869+0000","last_became_active":"2024-02-05T07:18:48.424620+0000","last_became_peered":"2024-02-05T07:18:48.424620+0000","last_unstale":"2024-02-05T07:24:39.667869+0000","last_undegraded":"2024-02-05T07:24:39.667869+0000","last_fullsized":"2024-02-05T07:24:39.667869+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:20:34.765664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:20:34.765664+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":343,"num_read_kb":338,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":225,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.373105+0000","last_change":"2024-02-05T07:24:34.242361+0000","last_active":"2024-02-05T07:24:39.373105+0000","last_peered":"2024-02-05T07:24:39.373105+0000","last_clean":"2024-02-05T07:24:39.373105+0000","last_became_active":"2024-02-05T07:24:34.241978+0000","last_became_peered":"2024-02-05T07:24:34.241978+0000","last_unstale":"2024-02-05T07:24:39.373105+0000","last_undegraded":"2024-02-05T07:24:39.373105+0000","last_fullsized":"2024-02-05T07:24:39.373105+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:21.318169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:21.318169+0000","last_clean_scrub_stamp":"2024-02-05T07:20:21.318169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":272,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.196226+0000","last_change":"2024-02-05T07:23:47.618520+0000","last_active":"2024-02-05T07:24:39.196226+0000","last_peered":"2024-02-05T07:24:39.196226+0000","last_clean":"2024-02-05T07:24:39.196226+0000","last_became_active":"2024-02-05T07:23:47.600441+0000","last_became_peered":"2024-02-05T07:23:47.600441+0000","last_unstale":"2024-02-05T07:24:39.196226+0000","last_undegraded":"2024-02-05T07:24:39.196226+0000","last_fullsized":"2024-02-05T07:24:39.196226+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:23:22.071923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:22.071923+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.671073+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"189'8867","reported_seq":9101,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667396+0000","last_change":"2024-02-05T07:21:43.836751+0000","last_active":"2024-02-05T07:24:39.667396+0000","last_peered":"2024-02-05T07:24:39.667396+0000","last_clean":"2024-02-05T07:24:39.667396+0000","last_became_active":"2024-02-05T07:21:42.573038+0000","last_became_peered":"2024-02-05T07:21:42.573038+0000","last_unstale":"2024-02-05T07:24:39.667396+0000","last_undegraded":"2024-02-05T07:24:39.667396+0000","last_fullsized":"2024-02-05T07:24:39.667396+0000","mapping_epoch":100,"log_start":"123'4830","ondisk_log_start":"123'4830","created":100,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'4736","last_scrub_stamp":"2024-02-05T07:21:43.836623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.836623+0000","objects_scrubbed":2356,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4132,"num_write_kb":56832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":256,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.670342+0000","last_change":"2024-02-05T07:21:15.698783+0000","last_active":"2024-02-05T07:24:39.670342+0000","last_peered":"2024-02-05T07:24:39.670342+0000","last_clean":"2024-02-05T07:24:39.670342+0000","last_became_active":"2024-02-05T07:18:53.459971+0000","last_became_peered":"2024-02-05T07:18:53.459971+0000","last_unstale":"2024-02-05T07:24:39.670342+0000","last_undegraded":"2024-02-05T07:24:39.670342+0000","last_fullsized":"2024-02-05T07:24:39.670342+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:15.698665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:15.698665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":361,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.669508+0000","last_change":"2024-02-05T07:24:20.278989+0000","last_active":"2024-02-05T07:24:39.669508+0000","last_peered":"2024-02-05T07:24:39.669508+0000","last_clean":"2024-02-05T07:24:39.669508+0000","last_became_active":"2024-02-05T07:18:51.507714+0000","last_became_peered":"2024-02-05T07:18:51.507714+0000","last_unstale":"2024-02-05T07:24:39.669508+0000","last_undegraded":"2024-02-05T07:24:39.669508+0000","last_fullsized":"2024-02-05T07:24:39.669508+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-02-05T07:24:20.278897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:20.278897+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:36.052175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":66,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667466+0000","last_change":"2024-02-05T07:23:47.609950+0000","last_active":"2024-02-05T07:24:39.667466+0000","last_peered":"2024-02-05T07:24:39.667466+0000","last_clean":"2024-02-05T07:24:39.667466+0000","last_became_active":"2024-02-05T07:23:47.609497+0000","last_became_peered":"2024-02-05T07:23:47.609497+0000","last_unstale":"2024-02-05T07:24:39.667466+0000","last_undegraded":"2024-02-05T07:24:39.667466+0000","last_fullsized":"2024-02-05T07:24:39.667466+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:29.851303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:29.851303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.863975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":529,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667917+0000","last_change":"2024-02-05T07:24:21.294644+0000","last_active":"2024-02-05T07:24:39.667917+0000","last_peered":"2024-02-05T07:24:39.667917+0000","last_clean":"2024-02-05T07:24:39.667917+0000","last_became_active":"2024-02-05T07:18:39.444898+0000","last_became_peered":"2024-02-05T07:18:39.444898+0000","last_unstale":"2024-02-05T07:24:39.667917+0000","last_undegraded":"2024-02-05T07:24:39.667917+0000","last_fullsized":"2024-02-05T07:24:39.667917+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:30.591394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":258,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668371+0000","last_change":"2024-02-05T07:23:47.592633+0000","last_active":"2024-02-05T07:24:39.668371+0000","last_peered":"2024-02-05T07:24:39.668371+0000","last_clean":"2024-02-05T07:24:39.668371+0000","last_became_active":"2024-02-05T07:23:47.592290+0000","last_became_peered":"2024-02-05T07:23:47.592290+0000","last_unstale":"2024-02-05T07:24:39.668371+0000","last_undegraded":"2024-02-05T07:24:39.668371+0000","last_fullsized":"2024-02-05T07:24:39.668371+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:35.085136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:35.085136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.864131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"189'8977","reported_seq":11861,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.669843+0000","last_change":"2024-02-05T07:24:34.275155+0000","last_active":"2024-02-05T07:24:39.669843+0000","last_peered":"2024-02-05T07:24:39.669843+0000","last_clean":"2024-02-05T07:24:39.669843+0000","last_became_active":"2024-02-05T07:24:34.240386+0000","last_became_peered":"2024-02-05T07:24:34.240386+0000","last_unstale":"2024-02-05T07:24:39.669843+0000","last_undegraded":"2024-02-05T07:24:39.669843+0000","last_fullsized":"2024-02-05T07:24:39.669843+0000","mapping_epoch":186,"log_start":"125'4924","ondisk_log_start":"125'4924","created":100,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'4786","last_scrub_stamp":"2024-02-05T07:21:46.577376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:46.577376+0000","objects_scrubbed":2382,"log_size":4053,"log_dups_size":0,"ondisk_log_size":4053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.239586+0000","scrub_duration":956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4204,"num_write_kb":58304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":85917696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":264,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.195386+0000","last_change":"2024-02-05T07:21:31.546003+0000","last_active":"2024-02-05T07:24:39.195386+0000","last_peered":"2024-02-05T07:24:39.195386+0000","last_clean":"2024-02-05T07:24:39.195386+0000","last_became_active":"2024-02-05T07:18:53.460771+0000","last_became_peered":"2024-02-05T07:18:53.460771+0000","last_unstale":"2024-02-05T07:24:39.195386+0000","last_undegraded":"2024-02-05T07:24:39.195386+0000","last_fullsized":"2024-02-05T07:24:39.195386+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:31.545467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:31.545467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741796+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"96'18","reported_seq":301,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.196050+0000","last_change":"2024-02-05T07:23:05.927322+0000","last_active":"2024-02-05T07:24:39.196050+0000","last_peered":"2024-02-05T07:24:39.196050+0000","last_clean":"2024-02-05T07:24:39.196050+0000","last_became_active":"2024-02-05T07:18:51.436097+0000","last_became_peered":"2024-02-05T07:18:51.436097+0000","last_unstale":"2024-02-05T07:24:39.196050+0000","last_undegraded":"2024-02-05T07:24:39.196050+0000","last_fullsized":"2024-02-05T07:24:39.196050+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'18","last_scrub_stamp":"2024-02-05T07:23:05.927169+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:23:05.927169+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.671041+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":287,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193152+0000","last_change":"2024-02-05T07:23:47.622111+0000","last_active":"2024-02-05T07:24:39.193152+0000","last_peered":"2024-02-05T07:24:39.193152+0000","last_clean":"2024-02-05T07:24:39.193152+0000","last_became_active":"2024-02-05T07:23:47.621133+0000","last_became_peered":"2024-02-05T07:23:47.621133+0000","last_unstale":"2024-02-05T07:24:39.193152+0000","last_undegraded":"2024-02-05T07:24:39.193152+0000","last_fullsized":"2024-02-05T07:24:39.193152+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:18.723118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:21:18.723118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741765+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":279,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193778+0000","last_change":"2024-02-05T07:21:34.523665+0000","last_active":"2024-02-05T07:24:39.193778+0000","last_peered":"2024-02-05T07:24:39.193778+0000","last_clean":"2024-02-05T07:24:39.193778+0000","last_became_active":"2024-02-05T07:18:49.429651+0000","last_became_peered":"2024-02-05T07:18:49.429651+0000","last_unstale":"2024-02-05T07:24:39.193778+0000","last_undegraded":"2024-02-05T07:24:39.193778+0000","last_fullsized":"2024-02-05T07:24:39.193778+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:21:34.523215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:34.523215+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670904+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":266,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.669602+0000","last_change":"2024-02-05T07:23:47.611214+0000","last_active":"2024-02-05T07:24:39.669602+0000","last_peered":"2024-02-05T07:24:39.669602+0000","last_clean":"2024-02-05T07:24:39.669602+0000","last_became_active":"2024-02-05T07:23:47.610875+0000","last_became_peered":"2024-02-05T07:23:47.610875+0000","last_unstale":"2024-02-05T07:24:39.669602+0000","last_undegraded":"2024-02-05T07:24:39.669602+0000","last_fullsized":"2024-02-05T07:24:39.669602+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:38.531265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:21:38.531265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"189'8789","reported_seq":9033,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193969+0000","last_change":"2024-02-05T07:24:34.284471+0000","last_active":"2024-02-05T07:24:39.193969+0000","last_peered":"2024-02-05T07:24:39.193969+0000","last_clean":"2024-02-05T07:24:39.193969+0000","last_became_active":"2024-02-05T07:24:34.247168+0000","last_became_peered":"2024-02-05T07:24:34.247168+0000","last_unstale":"2024-02-05T07:24:39.193969+0000","last_undegraded":"2024-02-05T07:24:39.193969+0000","last_fullsized":"2024-02-05T07:24:39.193969+0000","mapping_epoch":186,"log_start":"122'4714","ondisk_log_start":"122'4714","created":100,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'4680","last_scrub_stamp":"2024-02-05T07:21:44.496596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:44.496596+0000","objects_scrubbed":2317,"log_size":4075,"log_dups_size":0,"ondisk_log_size":4075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4109,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":18,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.195050+0000","last_change":"2024-02-05T07:24:34.234702+0000","last_active":"2024-02-05T07:24:39.195050+0000","last_peered":"2024-02-05T07:24:39.195050+0000","last_clean":"2024-02-05T07:24:39.195050+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:24:39.195050+0000","last_undegraded":"2024-02-05T07:24:39.195050+0000","last_fullsized":"2024-02-05T07:24:39.195050+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:34.202812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:34.202812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"164'29","reported_seq":424,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.196380+0000","last_change":"2024-02-05T07:23:47.660633+0000","last_active":"2024-02-05T07:24:39.196380+0000","last_peered":"2024-02-05T07:24:39.196380+0000","last_clean":"2024-02-05T07:24:39.196380+0000","last_became_active":"2024-02-05T07:23:47.608917+0000","last_became_peered":"2024-02-05T07:23:47.608917+0000","last_unstale":"2024-02-05T07:24:39.196380+0000","last_undegraded":"2024-02-05T07:24:39.196380+0000","last_fullsized":"2024-02-05T07:24:39.196380+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-02-05T07:23:27.310935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:27.310935+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:44.481733+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":272,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193943+0000","last_change":"2024-02-05T07:23:08.606203+0000","last_active":"2024-02-05T07:24:39.193943+0000","last_peered":"2024-02-05T07:24:39.193943+0000","last_clean":"2024-02-05T07:24:39.193943+0000","last_became_active":"2024-02-05T07:18:49.430208+0000","last_became_peered":"2024-02-05T07:18:49.430208+0000","last_unstale":"2024-02-05T07:24:39.193943+0000","last_undegraded":"2024-02-05T07:24:39.193943+0000","last_fullsized":"2024-02-05T07:24:39.193943+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:08.606061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:23:08.606061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":178,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.195171+0000","last_change":"2024-02-05T07:23:29.341025+0000","last_active":"2024-02-05T07:24:39.195171+0000","last_peered":"2024-02-05T07:24:39.195171+0000","last_clean":"2024-02-05T07:24:39.195171+0000","last_became_active":"2024-02-05T07:23:27.439333+0000","last_became_peered":"2024-02-05T07:23:27.439333+0000","last_unstale":"2024-02-05T07:24:39.195171+0000","last_undegraded":"2024-02-05T07:24:39.195171+0000","last_fullsized":"2024-02-05T07:24:39.195171+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:23:29.340876+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:20:38.993688+0000","last_clean_scrub_stamp":"2024-02-05T07:23:29.340876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.671092+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":74,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.373330+0000","last_change":"2024-02-05T07:24:05.412794+0000","last_active":"2024-02-05T07:24:39.373330+0000","last_peered":"2024-02-05T07:24:39.373330+0000","last_clean":"2024-02-05T07:24:39.373330+0000","last_became_active":"2024-02-05T07:23:47.603091+0000","last_became_peered":"2024-02-05T07:23:47.603091+0000","last_unstale":"2024-02-05T07:24:39.373330+0000","last_undegraded":"2024-02-05T07:24:39.373330+0000","last_fullsized":"2024-02-05T07:24:39.373330+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:05.412661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:05.412661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:15.571463+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.7","version":"189'10154","reported_seq":12755,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667833+0000","last_change":"2024-02-05T07:24:01.487722+0000","last_active":"2024-02-05T07:24:39.667833+0000","last_peered":"2024-02-05T07:24:39.667833+0000","last_clean":"2024-02-05T07:24:39.667833+0000","last_became_active":"2024-02-05T07:24:01.468725+0000","last_became_peered":"2024-02-05T07:24:01.468725+0000","last_unstale":"2024-02-05T07:24:39.667833+0000","last_undegraded":"2024-02-05T07:24:39.667833+0000","last_fullsized":"2024-02-05T07:24:39.667833+0000","mapping_epoch":171,"log_start":"124'6039","ondisk_log_start":"124'6039","created":100,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'6081","last_scrub_stamp":"2024-02-05T07:21:51.270929+0000","last_deep_scrub":"124'6081","last_deep_scrub_stamp":"2024-02-05T07:21:51.270929+0000","last_clean_scrub_stamp":"2024-02-05T07:21:51.270929+0000","objects_scrubbed":2391,"log_size":4115,"log_dups_size":0,"ondisk_log_size":4115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.367871+0000","scrub_duration":6883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4153,"num_write_kb":56384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":74448896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":53,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194238+0000","last_change":"2024-02-05T07:24:34.250380+0000","last_active":"2024-02-05T07:24:39.194238+0000","last_peered":"2024-02-05T07:24:39.194238+0000","last_clean":"2024-02-05T07:24:39.194238+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:24:39.194238+0000","last_undegraded":"2024-02-05T07:24:39.194238+0000","last_fullsized":"2024-02-05T07:24:39.194238+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:21.184445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:21.184445+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":76,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667826+0000","last_change":"2024-02-05T07:24:07.278812+0000","last_active":"2024-02-05T07:24:39.667826+0000","last_peered":"2024-02-05T07:24:39.667826+0000","last_clean":"2024-02-05T07:24:39.667826+0000","last_became_active":"2024-02-05T07:23:47.610698+0000","last_became_peered":"2024-02-05T07:23:47.610698+0000","last_unstale":"2024-02-05T07:24:39.667826+0000","last_undegraded":"2024-02-05T07:24:39.667826+0000","last_fullsized":"2024-02-05T07:24:39.667826+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:07.278696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:07.278696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:35.119282+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":270,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193485+0000","last_change":"2024-02-05T07:23:47.620787+0000","last_active":"2024-02-05T07:24:39.193485+0000","last_peered":"2024-02-05T07:24:39.193485+0000","last_clean":"2024-02-05T07:24:39.193485+0000","last_became_active":"2024-02-05T07:23:47.619951+0000","last_became_peered":"2024-02-05T07:23:47.619951+0000","last_unstale":"2024-02-05T07:24:39.193485+0000","last_undegraded":"2024-02-05T07:24:39.193485+0000","last_fullsized":"2024-02-05T07:24:39.193485+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:35.587887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:35.587887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741810+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":18,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.669754+0000","last_change":"2024-02-05T07:24:34.236399+0000","last_active":"2024-02-05T07:24:39.669754+0000","last_peered":"2024-02-05T07:24:39.669754+0000","last_clean":"2024-02-05T07:24:39.669754+0000","last_became_active":"2024-02-05T07:24:34.235899+0000","last_became_peered":"2024-02-05T07:24:34.235899+0000","last_unstale":"2024-02-05T07:24:39.669754+0000","last_undegraded":"2024-02-05T07:24:39.669754+0000","last_fullsized":"2024-02-05T07:24:39.669754+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:37.209967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.209967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.239712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"189'8702","reported_seq":8936,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193386+0000","last_change":"2024-02-05T07:21:43.922152+0000","last_active":"2024-02-05T07:24:39.193386+0000","last_peered":"2024-02-05T07:24:39.193386+0000","last_clean":"2024-02-05T07:24:39.193386+0000","last_became_active":"2024-02-05T07:21:42.569592+0000","last_became_peered":"2024-02-05T07:21:42.569592+0000","last_unstale":"2024-02-05T07:24:39.193386+0000","last_undegraded":"2024-02-05T07:24:39.193386+0000","last_fullsized":"2024-02-05T07:24:39.193386+0000","mapping_epoch":100,"log_start":"111'4064","ondisk_log_start":"111'4064","created":100,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'4643","last_scrub_stamp":"2024-02-05T07:21:43.922034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.922034+0000","objects_scrubbed":2331,"log_size":4638,"log_dups_size":0,"ondisk_log_size":4638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741938+0000","scrub_duration":428,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4059,"num_write_kb":55296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":299,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668807+0000","last_change":"2024-02-05T07:21:31.340373+0000","last_active":"2024-02-05T07:24:39.668807+0000","last_peered":"2024-02-05T07:24:39.668807+0000","last_clean":"2024-02-05T07:24:39.668807+0000","last_became_active":"2024-02-05T07:18:51.507672+0000","last_became_peered":"2024-02-05T07:18:51.507672+0000","last_unstale":"2024-02-05T07:24:39.668807+0000","last_undegraded":"2024-02-05T07:24:39.668807+0000","last_fullsized":"2024-02-05T07:24:39.668807+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:21:31.340234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:31.340234+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.367704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":272,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.669501+0000","last_change":"2024-02-05T07:21:41.444299+0000","last_active":"2024-02-05T07:24:39.669501+0000","last_peered":"2024-02-05T07:24:39.669501+0000","last_clean":"2024-02-05T07:24:39.669501+0000","last_became_active":"2024-02-05T07:18:49.434451+0000","last_became_peered":"2024-02-05T07:18:49.434451+0000","last_unstale":"2024-02-05T07:24:39.669501+0000","last_undegraded":"2024-02-05T07:24:39.669501+0000","last_fullsized":"2024-02-05T07:24:39.669501+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:41.444193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:41.444193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":264,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193325+0000","last_change":"2024-02-05T07:21:38.468561+0000","last_active":"2024-02-05T07:24:39.193325+0000","last_peered":"2024-02-05T07:24:39.193325+0000","last_clean":"2024-02-05T07:24:39.193325+0000","last_became_active":"2024-02-05T07:18:53.464867+0000","last_became_peered":"2024-02-05T07:18:53.464867+0000","last_unstale":"2024-02-05T07:24:39.193325+0000","last_undegraded":"2024-02-05T07:24:39.193325+0000","last_fullsized":"2024-02-05T07:24:39.193325+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:38.468456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:38.468456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741870+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":258,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667742+0000","last_change":"2024-02-05T07:23:47.611420+0000","last_active":"2024-02-05T07:24:39.667742+0000","last_peered":"2024-02-05T07:24:39.667742+0000","last_clean":"2024-02-05T07:24:39.667742+0000","last_became_active":"2024-02-05T07:23:47.611066+0000","last_became_peered":"2024-02-05T07:23:47.611066+0000","last_unstale":"2024-02-05T07:24:39.667742+0000","last_undegraded":"2024-02-05T07:24:39.667742+0000","last_fullsized":"2024-02-05T07:24:39.667742+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:38.715447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:38.715447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"189'8790","reported_seq":9033,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194304+0000","last_change":"2024-02-05T07:24:34.283279+0000","last_active":"2024-02-05T07:24:39.194304+0000","last_peered":"2024-02-05T07:24:39.194304+0000","last_clean":"2024-02-05T07:24:39.194304+0000","last_became_active":"2024-02-05T07:24:34.243473+0000","last_became_peered":"2024-02-05T07:24:34.243473+0000","last_unstale":"2024-02-05T07:24:39.194304+0000","last_undegraded":"2024-02-05T07:24:39.194304+0000","last_fullsized":"2024-02-05T07:24:39.194304+0000","mapping_epoch":186,"log_start":"130'5392","ondisk_log_start":"130'5392","created":100,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'4670","last_scrub_stamp":"2024-02-05T07:21:43.853520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.853520+0000","objects_scrubbed":2392,"log_size":3398,"log_dups_size":0,"ondisk_log_size":3398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670943+0000","scrub_duration":656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4122,"num_write_kb":55360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":264,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.192998+0000","last_change":"2024-02-05T07:21:29.559984+0000","last_active":"2024-02-05T07:24:39.192998+0000","last_peered":"2024-02-05T07:24:39.192998+0000","last_clean":"2024-02-05T07:24:39.192998+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:24:39.192998+0000","last_undegraded":"2024-02-05T07:24:39.192998+0000","last_fullsized":"2024-02-05T07:24:39.192998+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:29.559910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:29.559910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":327,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.373290+0000","last_change":"2024-02-05T07:23:47.625879+0000","last_active":"2024-02-05T07:24:39.373290+0000","last_peered":"2024-02-05T07:24:39.373290+0000","last_clean":"2024-02-05T07:24:39.373290+0000","last_became_active":"2024-02-05T07:23:47.607181+0000","last_became_peered":"2024-02-05T07:23:47.607181+0000","last_unstale":"2024-02-05T07:24:39.373290+0000","last_undegraded":"2024-02-05T07:24:39.373290+0000","last_fullsized":"2024-02-05T07:24:39.373290+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:21:33.928419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:33.928419+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":272,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.669290+0000","last_change":"2024-02-05T07:21:31.527396+0000","last_active":"2024-02-05T07:24:39.669290+0000","last_peered":"2024-02-05T07:24:39.669290+0000","last_clean":"2024-02-05T07:24:39.669290+0000","last_became_active":"2024-02-05T07:18:49.435621+0000","last_became_peered":"2024-02-05T07:18:49.435621+0000","last_unstale":"2024-02-05T07:24:39.669290+0000","last_undegraded":"2024-02-05T07:24:39.669290+0000","last_fullsized":"2024-02-05T07:24:39.669290+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:31.527325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:31.527325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":66,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667239+0000","last_change":"2024-02-05T07:23:47.610635+0000","last_active":"2024-02-05T07:24:39.667239+0000","last_peered":"2024-02-05T07:24:39.667239+0000","last_clean":"2024-02-05T07:24:39.667239+0000","last_became_active":"2024-02-05T07:23:47.610289+0000","last_became_peered":"2024-02-05T07:23:47.610289+0000","last_unstale":"2024-02-05T07:24:39.667239+0000","last_undegraded":"2024-02-05T07:24:39.667239+0000","last_fullsized":"2024-02-05T07:24:39.667239+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:28.882326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:28.882326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"189'12795","reported_seq":17794,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193226+0000","last_change":"2024-02-05T07:24:20.540583+0000","last_active":"2024-02-05T07:24:39.193226+0000","last_peered":"2024-02-05T07:24:39.193226+0000","last_clean":"2024-02-05T07:24:39.193226+0000","last_became_active":"2024-02-05T07:24:20.517637+0000","last_became_peered":"2024-02-05T07:24:20.517637+0000","last_unstale":"2024-02-05T07:24:39.193226+0000","last_undegraded":"2024-02-05T07:24:39.193226+0000","last_fullsized":"2024-02-05T07:24:39.193226+0000","mapping_epoch":171,"log_start":"137'7469","ondisk_log_start":"137'7469","created":100,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":5326,"log_dups_size":0,"ondisk_log_size":5326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6683,"num_write_kb":212704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":78348288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":344,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.670375+0000","last_change":"2024-02-05T07:21:26.505275+0000","last_active":"2024-02-05T07:24:39.670375+0000","last_peered":"2024-02-05T07:24:39.670375+0000","last_clean":"2024-02-05T07:24:39.670375+0000","last_became_active":"2024-02-05T07:18:53.456287+0000","last_became_peered":"2024-02-05T07:18:53.456287+0000","last_unstale":"2024-02-05T07:24:39.670375+0000","last_undegraded":"2024-02-05T07:24:39.670375+0000","last_fullsized":"2024-02-05T07:24:39.670375+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-02-05T07:21:26.505161+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:21:26.505161+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":293,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666985+0000","last_change":"2024-02-05T07:21:32.875499+0000","last_active":"2024-02-05T07:24:39.666985+0000","last_peered":"2024-02-05T07:24:39.666985+0000","last_clean":"2024-02-05T07:24:39.666985+0000","last_became_active":"2024-02-05T07:18:51.508097+0000","last_became_peered":"2024-02-05T07:18:51.508097+0000","last_unstale":"2024-02-05T07:24:39.666985+0000","last_undegraded":"2024-02-05T07:24:39.666985+0000","last_fullsized":"2024-02-05T07:24:39.666985+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:21:32.875346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:32.875346+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.864381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":284,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.670482+0000","last_change":"2024-02-05T07:24:19.277756+0000","last_active":"2024-02-05T07:24:39.670482+0000","last_peered":"2024-02-05T07:24:39.670482+0000","last_clean":"2024-02-05T07:24:39.670482+0000","last_became_active":"2024-02-05T07:18:49.437663+0000","last_became_peered":"2024-02-05T07:18:49.437663+0000","last_unstale":"2024-02-05T07:24:39.670482+0000","last_undegraded":"2024-02-05T07:24:39.670482+0000","last_fullsized":"2024-02-05T07:24:39.670482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:24:19.277618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:19.277618+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:22.979043+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":95,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668619+0000","last_change":"2024-02-05T07:24:19.263465+0000","last_active":"2024-02-05T07:24:39.668619+0000","last_peered":"2024-02-05T07:24:39.668619+0000","last_clean":"2024-02-05T07:24:39.668619+0000","last_became_active":"2024-02-05T07:23:27.419855+0000","last_became_peered":"2024-02-05T07:23:27.419855+0000","last_unstale":"2024-02-05T07:24:39.668619+0000","last_undegraded":"2024-02-05T07:24:39.668619+0000","last_fullsized":"2024-02-05T07:24:39.668619+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:19.263379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:37.032056+0000","last_clean_scrub_stamp":"2024-02-05T07:24:19.263379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:32.556946+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"189'15772","reported_seq":16123,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.195294+0000","last_change":"2024-02-05T07:24:20.539957+0000","last_active":"2024-02-05T07:24:39.195294+0000","last_peered":"2024-02-05T07:24:39.195294+0000","last_clean":"2024-02-05T07:24:39.195294+0000","last_became_active":"2024-02-05T07:24:20.516986+0000","last_became_peered":"2024-02-05T07:24:20.516986+0000","last_unstale":"2024-02-05T07:24:39.195294+0000","last_undegraded":"2024-02-05T07:24:39.195294+0000","last_fullsized":"2024-02-05T07:24:39.195294+0000","mapping_epoch":100,"log_start":"154'11564","ondisk_log_start":"154'11564","created":100,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'13339","last_scrub_stamp":"2024-02-05T07:23:40.012704+0000","last_deep_scrub":"156'13339","last_deep_scrub_stamp":"2024-02-05T07:23:40.012704+0000","last_clean_scrub_stamp":"2024-02-05T07:23:40.012704+0000","objects_scrubbed":3400,"log_size":4208,"log_dups_size":0,"ondisk_log_size":4208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":20,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.688087+0000","scrub_duration":19992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1179648,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6434,"num_write_kb":206464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":110,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.670470+0000","last_change":"2024-02-05T07:24:07.263004+0000","last_active":"2024-02-05T07:24:39.670470+0000","last_peered":"2024-02-05T07:24:39.670470+0000","last_clean":"2024-02-05T07:24:39.670470+0000","last_became_active":"2024-02-05T07:23:47.607589+0000","last_became_peered":"2024-02-05T07:23:47.607589+0000","last_unstale":"2024-02-05T07:24:39.670470+0000","last_undegraded":"2024-02-05T07:24:39.670470+0000","last_fullsized":"2024-02-05T07:24:39.670470+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-02-05T07:24:07.262821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:07.262821+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:07.828893+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":278,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666777+0000","last_change":"2024-02-05T07:21:25.389276+0000","last_active":"2024-02-05T07:24:39.666777+0000","last_peered":"2024-02-05T07:24:39.666777+0000","last_clean":"2024-02-05T07:24:39.666777+0000","last_became_active":"2024-02-05T07:18:49.433121+0000","last_became_peered":"2024-02-05T07:18:49.433121+0000","last_unstale":"2024-02-05T07:24:39.666777+0000","last_undegraded":"2024-02-05T07:24:39.666777+0000","last_fullsized":"2024-02-05T07:24:39.666777+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:21:25.389138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:25.389138+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.367661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":262,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668843+0000","last_change":"2024-02-05T07:23:47.601964+0000","last_active":"2024-02-05T07:24:39.668843+0000","last_peered":"2024-02-05T07:24:39.668843+0000","last_clean":"2024-02-05T07:24:39.668843+0000","last_became_active":"2024-02-05T07:23:47.601556+0000","last_became_peered":"2024-02-05T07:23:47.601556+0000","last_unstale":"2024-02-05T07:24:39.668843+0000","last_undegraded":"2024-02-05T07:24:39.668843+0000","last_fullsized":"2024-02-05T07:24:39.668843+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:18.752135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:18.752135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.864193+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":260,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666175+0000","last_change":"2024-02-05T07:21:44.435932+0000","last_active":"2024-02-05T07:24:39.666175+0000","last_peered":"2024-02-05T07:24:39.666175+0000","last_clean":"2024-02-05T07:24:39.666175+0000","last_became_active":"2024-02-05T07:18:55.507555+0000","last_became_peered":"2024-02-05T07:18:55.507555+0000","last_unstale":"2024-02-05T07:24:39.666175+0000","last_undegraded":"2024-02-05T07:24:39.666175+0000","last_fullsized":"2024-02-05T07:24:39.666175+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:21:44.435826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"189'8989","reported_seq":18893,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.195082+0000","last_change":"2024-02-05T07:24:19.799744+0000","last_active":"2024-02-05T07:24:39.195082+0000","last_peered":"2024-02-05T07:24:39.195082+0000","last_clean":"2024-02-05T07:24:39.195082+0000","last_became_active":"2024-02-05T07:23:51.653282+0000","last_became_peered":"2024-02-05T07:23:51.653282+0000","last_unstale":"2024-02-05T07:24:39.195082+0000","last_undegraded":"2024-02-05T07:24:39.195082+0000","last_fullsized":"2024-02-05T07:24:39.195082+0000","mapping_epoch":165,"log_start":"116'4575","ondisk_log_start":"116'4575","created":118,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"176'8433","last_scrub_stamp":"2024-02-05T07:24:19.799617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:19.799617+0000","objects_scrubbed":556,"log_size":4414,"log_dups_size":0,"ondisk_log_size":4414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:48.076339+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":556,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":53,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.669943+0000","last_change":"2024-02-05T07:24:34.274818+0000","last_active":"2024-02-05T07:24:39.669943+0000","last_peered":"2024-02-05T07:24:39.669943+0000","last_clean":"2024-02-05T07:24:39.669943+0000","last_became_active":"2024-02-05T07:24:34.241945+0000","last_became_peered":"2024-02-05T07:24:34.241945+0000","last_unstale":"2024-02-05T07:24:39.669943+0000","last_undegraded":"2024-02-05T07:24:39.669943+0000","last_fullsized":"2024-02-05T07:24:39.669943+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:33.244655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:33.244655+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.239616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":272,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.670046+0000","last_change":"2024-02-05T07:21:25.529255+0000","last_active":"2024-02-05T07:24:39.670046+0000","last_peered":"2024-02-05T07:24:39.670046+0000","last_clean":"2024-02-05T07:24:39.670046+0000","last_became_active":"2024-02-05T07:18:49.437732+0000","last_became_peered":"2024-02-05T07:18:49.437732+0000","last_unstale":"2024-02-05T07:24:39.670046+0000","last_undegraded":"2024-02-05T07:24:39.670046+0000","last_fullsized":"2024-02-05T07:24:39.670046+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:25.529174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:21:25.529174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":66,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.196002+0000","last_change":"2024-02-05T07:23:47.606649+0000","last_active":"2024-02-05T07:24:39.196002+0000","last_peered":"2024-02-05T07:24:39.196002+0000","last_clean":"2024-02-05T07:24:39.196002+0000","last_became_active":"2024-02-05T07:23:47.606266+0000","last_became_peered":"2024-02-05T07:23:47.606266+0000","last_unstale":"2024-02-05T07:24:39.196002+0000","last_undegraded":"2024-02-05T07:24:39.196002+0000","last_fullsized":"2024-02-05T07:24:39.196002+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:02.813344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:02.813344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":260,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.195581+0000","last_change":"2024-02-05T07:21:39.515473+0000","last_active":"2024-02-05T07:24:39.195581+0000","last_peered":"2024-02-05T07:24:39.195581+0000","last_clean":"2024-02-05T07:24:39.195581+0000","last_became_active":"2024-02-05T07:18:55.469044+0000","last_became_peered":"2024-02-05T07:18:55.469044+0000","last_unstale":"2024-02-05T07:24:39.195581+0000","last_undegraded":"2024-02-05T07:24:39.195581+0000","last_fullsized":"2024-02-05T07:24:39.195581+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:39.515332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.515332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741841+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"189'8999","reported_seq":14055,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.669371+0000","last_change":"2024-02-05T07:24:02.322881+0000","last_active":"2024-02-05T07:24:39.669371+0000","last_peered":"2024-02-05T07:24:39.669371+0000","last_clean":"2024-02-05T07:24:39.669371+0000","last_became_active":"2024-02-05T07:22:15.165819+0000","last_became_peered":"2024-02-05T07:22:15.165819+0000","last_unstale":"2024-02-05T07:24:39.669371+0000","last_undegraded":"2024-02-05T07:24:39.669371+0000","last_fullsized":"2024-02-05T07:24:39.669371+0000","mapping_epoch":126,"log_start":"125'4967","ondisk_log_start":"125'4967","created":118,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"172'7884","last_scrub_stamp":"2024-02-05T07:24:02.322788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:02.322788+0000","objects_scrubbed":1115,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.467497+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1115,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":300,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666901+0000","last_change":"2024-02-05T07:23:47.619736+0000","last_active":"2024-02-05T07:24:39.666901+0000","last_peered":"2024-02-05T07:24:39.666901+0000","last_clean":"2024-02-05T07:24:39.666901+0000","last_became_active":"2024-02-05T07:23:47.602628+0000","last_became_peered":"2024-02-05T07:23:47.602628+0000","last_unstale":"2024-02-05T07:24:39.666901+0000","last_undegraded":"2024-02-05T07:24:39.666901+0000","last_fullsized":"2024-02-05T07:24:39.666901+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:30.859017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:30.859017+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.864287+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":225,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666404+0000","last_change":"2024-02-05T07:24:34.231657+0000","last_active":"2024-02-05T07:24:39.666404+0000","last_peered":"2024-02-05T07:24:39.666404+0000","last_clean":"2024-02-05T07:24:39.666404+0000","last_became_active":"2024-02-05T07:24:34.230825+0000","last_became_peered":"2024-02-05T07:24:34.230825+0000","last_unstale":"2024-02-05T07:24:39.666404+0000","last_undegraded":"2024-02-05T07:24:39.666404+0000","last_fullsized":"2024-02-05T07:24:39.666404+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:33.270421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:33.270421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.239529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"164'9","reported_seq":106,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.670226+0000","last_change":"2024-02-05T07:23:47.620866+0000","last_active":"2024-02-05T07:24:39.670226+0000","last_peered":"2024-02-05T07:24:39.670226+0000","last_clean":"2024-02-05T07:24:39.670226+0000","last_became_active":"2024-02-05T07:23:47.605296+0000","last_became_peered":"2024-02-05T07:23:47.605296+0000","last_unstale":"2024-02-05T07:24:39.670226+0000","last_undegraded":"2024-02-05T07:24:39.670226+0000","last_fullsized":"2024-02-05T07:24:39.670226+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:22:58.051782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:22:58.051782+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":260,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.373666+0000","last_change":"2024-02-05T07:24:14.511046+0000","last_active":"2024-02-05T07:24:39.373666+0000","last_peered":"2024-02-05T07:24:39.373666+0000","last_clean":"2024-02-05T07:24:39.373666+0000","last_became_active":"2024-02-05T07:18:55.467044+0000","last_became_peered":"2024-02-05T07:18:55.467044+0000","last_unstale":"2024-02-05T07:24:39.373666+0000","last_undegraded":"2024-02-05T07:24:39.373666+0000","last_fullsized":"2024-02-05T07:24:39.373666+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:14.510933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:14.510933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:37.853613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"189'9001","reported_seq":14102,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.373058+0000","last_change":"2024-02-05T07:23:02.140812+0000","last_active":"2024-02-05T07:24:39.373058+0000","last_peered":"2024-02-05T07:24:39.373058+0000","last_clean":"2024-02-05T07:24:39.373058+0000","last_became_active":"2024-02-05T07:23:01.185149+0000","last_became_peered":"2024-02-05T07:23:01.185149+0000","last_unstale":"2024-02-05T07:24:39.373058+0000","last_undegraded":"2024-02-05T07:24:39.373058+0000","last_fullsized":"2024-02-05T07:24:39.373058+0000","mapping_epoch":146,"log_start":"125'4898","ondisk_log_start":"125'4898","created":118,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'6509","last_scrub_stamp":"2024-02-05T07:23:02.140598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:23:02.140598+0000","objects_scrubbed":2503,"log_size":4103,"log_dups_size":0,"ondisk_log_size":4103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2503,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"113'15","reported_seq":193,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.196389+0000","last_change":"2024-02-05T07:23:47.640930+0000","last_active":"2024-02-05T07:24:39.196389+0000","last_peered":"2024-02-05T07:24:39.196389+0000","last_clean":"2024-02-05T07:24:39.196389+0000","last_became_active":"2024-02-05T07:23:47.608158+0000","last_became_peered":"2024-02-05T07:23:47.608158+0000","last_unstale":"2024-02-05T07:24:39.196389+0000","last_undegraded":"2024-02-05T07:24:39.196389+0000","last_fullsized":"2024-02-05T07:24:39.196389+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'15","last_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:21:36.359546+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":264,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667034+0000","last_change":"2024-02-05T07:20:32.839729+0000","last_active":"2024-02-05T07:24:39.667034+0000","last_peered":"2024-02-05T07:24:39.667034+0000","last_clean":"2024-02-05T07:24:39.667034+0000","last_became_active":"2024-02-05T07:18:49.436131+0000","last_became_peered":"2024-02-05T07:18:49.436131+0000","last_unstale":"2024-02-05T07:24:39.667034+0000","last_undegraded":"2024-02-05T07:24:39.667034+0000","last_fullsized":"2024-02-05T07:24:39.667034+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:32.839614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:32.839614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":335,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666667+0000","last_change":"2024-02-05T07:20:38.003311+0000","last_active":"2024-02-05T07:24:39.666667+0000","last_peered":"2024-02-05T07:24:39.666667+0000","last_clean":"2024-02-05T07:24:39.666667+0000","last_became_active":"2024-02-05T07:18:53.465287+0000","last_became_peered":"2024-02-05T07:18:53.465287+0000","last_unstale":"2024-02-05T07:24:39.666667+0000","last_undegraded":"2024-02-05T07:24:39.666667+0000","last_fullsized":"2024-02-05T07:24:39.666667+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:20:38.003219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:38.003219+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.864165+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":213,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666947+0000","last_change":"2024-02-05T07:24:34.242269+0000","last_active":"2024-02-05T07:24:39.666947+0000","last_peered":"2024-02-05T07:24:39.666947+0000","last_clean":"2024-02-05T07:24:39.666947+0000","last_became_active":"2024-02-05T07:24:34.238479+0000","last_became_peered":"2024-02-05T07:24:34.238479+0000","last_unstale":"2024-02-05T07:24:39.666947+0000","last_undegraded":"2024-02-05T07:24:39.666947+0000","last_fullsized":"2024-02-05T07:24:39.666947+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:16.156709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:16.156709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:30.913274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"189'8805","reported_seq":18438,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.670309+0000","last_change":"2024-02-05T07:24:03.599184+0000","last_active":"2024-02-05T07:24:39.670309+0000","last_peered":"2024-02-05T07:24:39.670309+0000","last_clean":"2024-02-05T07:24:39.670309+0000","last_became_active":"2024-02-05T07:23:44.564445+0000","last_became_peered":"2024-02-05T07:23:44.564445+0000","last_unstale":"2024-02-05T07:24:39.670309+0000","last_undegraded":"2024-02-05T07:24:39.670309+0000","last_fullsized":"2024-02-05T07:24:39.670309+0000","mapping_epoch":159,"log_start":"123'4737","ondisk_log_start":"123'4737","created":118,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"173'7768","last_scrub_stamp":"2024-02-05T07:24:03.599036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:03.599036+0000","objects_scrubbed":1041,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:20.490002+0000","scrub_duration":468,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1041,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":305,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.372639+0000","last_change":"2024-02-05T07:21:37.876638+0000","last_active":"2024-02-05T07:24:39.372639+0000","last_peered":"2024-02-05T07:24:39.372639+0000","last_clean":"2024-02-05T07:24:39.372639+0000","last_became_active":"2024-02-05T07:19:16.121696+0000","last_became_peered":"2024-02-05T07:19:16.121696+0000","last_unstale":"2024-02-05T07:24:39.372639+0000","last_undegraded":"2024-02-05T07:24:39.372639+0000","last_fullsized":"2024-02-05T07:24:39.372639+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:21:37.876511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.876511+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":272,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.372827+0000","last_change":"2024-02-05T07:23:23.435405+0000","last_active":"2024-02-05T07:24:39.372827+0000","last_peered":"2024-02-05T07:24:39.372827+0000","last_clean":"2024-02-05T07:24:39.372827+0000","last_became_active":"2024-02-05T07:18:49.430766+0000","last_became_peered":"2024-02-05T07:18:49.430766+0000","last_unstale":"2024-02-05T07:24:39.372827+0000","last_undegraded":"2024-02-05T07:24:39.372827+0000","last_fullsized":"2024-02-05T07:24:39.372827+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:23:23.435135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.864941+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":271,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.195561+0000","last_change":"2024-02-05T07:23:47.594349+0000","last_active":"2024-02-05T07:24:39.195561+0000","last_peered":"2024-02-05T07:24:39.195561+0000","last_clean":"2024-02-05T07:24:39.195561+0000","last_became_active":"2024-02-05T07:23:47.594030+0000","last_became_peered":"2024-02-05T07:23:47.594030+0000","last_unstale":"2024-02-05T07:24:39.195561+0000","last_undegraded":"2024-02-05T07:24:39.195561+0000","last_fullsized":"2024-02-05T07:24:39.195561+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:22:59.851438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:22:59.851438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.671007+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":20,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.192928+0000","last_change":"2024-02-05T07:24:34.229646+0000","last_active":"2024-02-05T07:24:39.192928+0000","last_peered":"2024-02-05T07:24:39.192928+0000","last_clean":"2024-02-05T07:24:39.192928+0000","last_became_active":"2024-02-05T07:24:34.229238+0000","last_became_peered":"2024-02-05T07:24:34.229238+0000","last_unstale":"2024-02-05T07:24:39.192928+0000","last_undegraded":"2024-02-05T07:24:39.192928+0000","last_fullsized":"2024-02-05T07:24:39.192928+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:27.852942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:27.852942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"189'8735","reported_seq":16028,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193086+0000","last_change":"2024-02-05T07:23:58.379360+0000","last_active":"2024-02-05T07:24:39.193086+0000","last_peered":"2024-02-05T07:24:39.193086+0000","last_clean":"2024-02-05T07:24:39.193086+0000","last_became_active":"2024-02-05T07:23:58.357839+0000","last_became_peered":"2024-02-05T07:23:58.357839+0000","last_unstale":"2024-02-05T07:24:39.193086+0000","last_undegraded":"2024-02-05T07:24:39.193086+0000","last_fullsized":"2024-02-05T07:24:39.193086+0000","mapping_epoch":168,"log_start":"112'4090","ondisk_log_start":"112'4090","created":118,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":5,"last_scrub":"144'6154","last_scrub_stamp":"2024-02-05T07:22:55.148513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:22:55.148513+0000","objects_scrubbed":2599,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741971+0000","scrub_duration":1080,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2599,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1183,"num_bytes_recovered":77529088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":248,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667353+0000","last_change":"2024-02-05T07:21:16.721101+0000","last_active":"2024-02-05T07:24:39.667353+0000","last_peered":"2024-02-05T07:24:39.667353+0000","last_clean":"2024-02-05T07:24:39.667353+0000","last_became_active":"2024-02-05T07:19:16.122059+0000","last_became_peered":"2024-02-05T07:19:16.122059+0000","last_unstale":"2024-02-05T07:24:39.667353+0000","last_undegraded":"2024-02-05T07:24:39.667353+0000","last_fullsized":"2024-02-05T07:24:39.667353+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:16.720264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:16.720264+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.864320+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":270,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.669344+0000","last_change":"2024-02-05T07:23:47.616365+0000","last_active":"2024-02-05T07:24:39.669344+0000","last_peered":"2024-02-05T07:24:39.669344+0000","last_clean":"2024-02-05T07:24:39.669344+0000","last_became_active":"2024-02-05T07:23:47.616029+0000","last_became_peered":"2024-02-05T07:23:47.616029+0000","last_unstale":"2024-02-05T07:24:39.669344+0000","last_undegraded":"2024-02-05T07:24:39.669344+0000","last_fullsized":"2024-02-05T07:24:39.669344+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:20.600583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:20.600583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":20,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.195008+0000","last_change":"2024-02-05T07:24:34.249659+0000","last_active":"2024-02-05T07:24:39.195008+0000","last_peered":"2024-02-05T07:24:39.195008+0000","last_clean":"2024-02-05T07:24:39.195008+0000","last_became_active":"2024-02-05T07:24:34.231409+0000","last_became_peered":"2024-02-05T07:24:34.231409+0000","last_unstale":"2024-02-05T07:24:39.195008+0000","last_undegraded":"2024-02-05T07:24:39.195008+0000","last_fullsized":"2024-02-05T07:24:39.195008+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:02.051772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:02.051772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:19.652909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":18,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193284+0000","last_change":"2024-02-05T07:24:34.231186+0000","last_active":"2024-02-05T07:24:39.193284+0000","last_peered":"2024-02-05T07:24:39.193284+0000","last_clean":"2024-02-05T07:24:39.193284+0000","last_became_active":"2024-02-05T07:24:34.230006+0000","last_became_peered":"2024-02-05T07:24:34.230006+0000","last_unstale":"2024-02-05T07:24:39.193284+0000","last_undegraded":"2024-02-05T07:24:39.193284+0000","last_fullsized":"2024-02-05T07:24:39.193284+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:24:20.215062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:42.873758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"189'8836","reported_seq":16174,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.669693+0000","last_change":"2024-02-05T07:24:04.484761+0000","last_active":"2024-02-05T07:24:39.669693+0000","last_peered":"2024-02-05T07:24:39.669693+0000","last_clean":"2024-02-05T07:24:39.669693+0000","last_became_active":"2024-02-05T07:23:58.360798+0000","last_became_peered":"2024-02-05T07:23:58.360798+0000","last_unstale":"2024-02-05T07:24:39.669693+0000","last_undegraded":"2024-02-05T07:24:39.669693+0000","last_fullsized":"2024-02-05T07:24:39.669693+0000","mapping_epoch":168,"log_start":"124'4818","ondisk_log_start":"124'4818","created":118,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":5,"last_scrub":"173'7795","last_scrub_stamp":"2024-02-05T07:24:04.484558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:04.484558+0000","objects_scrubbed":1044,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:23.274007+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":116,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.670260+0000","last_change":"2024-02-05T07:23:47.638791+0000","last_active":"2024-02-05T07:24:39.670260+0000","last_peered":"2024-02-05T07:24:39.670260+0000","last_clean":"2024-02-05T07:24:39.670260+0000","last_became_active":"2024-02-05T07:23:47.613111+0000","last_became_peered":"2024-02-05T07:23:47.613111+0000","last_unstale":"2024-02-05T07:24:39.670260+0000","last_undegraded":"2024-02-05T07:24:39.670260+0000","last_fullsized":"2024-02-05T07:24:39.670260+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:21:18.095797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:18.095797+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":272,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668923+0000","last_change":"2024-02-05T07:21:29.818030+0000","last_active":"2024-02-05T07:24:39.668923+0000","last_peered":"2024-02-05T07:24:39.668923+0000","last_clean":"2024-02-05T07:24:39.668923+0000","last_became_active":"2024-02-05T07:18:49.434560+0000","last_became_peered":"2024-02-05T07:18:49.434560+0000","last_unstale":"2024-02-05T07:24:39.668923+0000","last_undegraded":"2024-02-05T07:24:39.668923+0000","last_fullsized":"2024-02-05T07:24:39.668923+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:29.817921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:29.817921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.864348+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":264,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668210+0000","last_change":"2024-02-05T07:21:24.573753+0000","last_active":"2024-02-05T07:24:39.668210+0000","last_peered":"2024-02-05T07:24:39.668210+0000","last_clean":"2024-02-05T07:24:39.668210+0000","last_became_active":"2024-02-05T07:18:53.460977+0000","last_became_peered":"2024-02-05T07:18:53.460977+0000","last_unstale":"2024-02-05T07:24:39.668210+0000","last_undegraded":"2024-02-05T07:24:39.668210+0000","last_fullsized":"2024-02-05T07:24:39.668210+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:24.573655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_clean_scrub_stamp":"2024-02-05T07:21:24.573655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":20,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667059+0000","last_change":"2024-02-05T07:24:34.226706+0000","last_active":"2024-02-05T07:24:39.667059+0000","last_peered":"2024-02-05T07:24:39.667059+0000","last_clean":"2024-02-05T07:24:39.667059+0000","last_became_active":"2024-02-05T07:24:34.225863+0000","last_became_peered":"2024-02-05T07:24:34.225863+0000","last_unstale":"2024-02-05T07:24:39.667059+0000","last_undegraded":"2024-02-05T07:24:39.667059+0000","last_fullsized":"2024-02-05T07:24:39.667059+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:22.138332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:22.138332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.367621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.12","version":"189'8892","reported_seq":16250,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668454+0000","last_change":"2024-02-05T07:24:03.700312+0000","last_active":"2024-02-05T07:24:39.668454+0000","last_peered":"2024-02-05T07:24:39.668454+0000","last_clean":"2024-02-05T07:24:39.668454+0000","last_became_active":"2024-02-05T07:22:37.417198+0000","last_became_peered":"2024-02-05T07:22:37.417198+0000","last_unstale":"2024-02-05T07:24:39.668454+0000","last_undegraded":"2024-02-05T07:24:39.668454+0000","last_fullsized":"2024-02-05T07:24:39.668454+0000","mapping_epoch":134,"log_start":"122'4720","ondisk_log_start":"122'4720","created":118,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"173'7850","last_scrub_stamp":"2024-02-05T07:24:03.700068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:03.700068+0000","objects_scrubbed":1049,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:31.909201+0000","scrub_duration":517,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1049,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":97,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193967+0000","last_change":"2024-02-05T07:23:47.645600+0000","last_active":"2024-02-05T07:24:39.193967+0000","last_peered":"2024-02-05T07:24:39.193967+0000","last_clean":"2024-02-05T07:24:39.193967+0000","last_became_active":"2024-02-05T07:23:47.610110+0000","last_became_peered":"2024-02-05T07:23:47.610110+0000","last_unstale":"2024-02-05T07:24:39.193967+0000","last_undegraded":"2024-02-05T07:24:39.193967+0000","last_fullsized":"2024-02-05T07:24:39.193967+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-02-05T07:21:20.184048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:20.184048+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":225,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668520+0000","last_change":"2024-02-05T07:24:34.229700+0000","last_active":"2024-02-05T07:24:39.668520+0000","last_peered":"2024-02-05T07:24:39.668520+0000","last_clean":"2024-02-05T07:24:39.668520+0000","last_became_active":"2024-02-05T07:24:34.229229+0000","last_became_peered":"2024-02-05T07:24:34.229229+0000","last_unstale":"2024-02-05T07:24:39.668520+0000","last_undegraded":"2024-02-05T07:24:39.668520+0000","last_fullsized":"2024-02-05T07:24:39.668520+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:17.202540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:17.202540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:24.251692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":264,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668708+0000","last_change":"2024-02-05T07:24:05.236956+0000","last_active":"2024-02-05T07:24:39.668708+0000","last_peered":"2024-02-05T07:24:39.668708+0000","last_clean":"2024-02-05T07:24:39.668708+0000","last_became_active":"2024-02-05T07:18:53.460307+0000","last_became_peered":"2024-02-05T07:18:53.460307+0000","last_unstale":"2024-02-05T07:24:39.668708+0000","last_undegraded":"2024-02-05T07:24:39.668708+0000","last_fullsized":"2024-02-05T07:24:39.668708+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:05.236812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:24:05.236812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:11.357567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":260,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666131+0000","last_change":"2024-02-05T07:21:30.473743+0000","last_active":"2024-02-05T07:24:39.666131+0000","last_peered":"2024-02-05T07:24:39.666131+0000","last_clean":"2024-02-05T07:24:39.666131+0000","last_became_active":"2024-02-05T07:18:55.468086+0000","last_became_peered":"2024-02-05T07:18:55.468086+0000","last_unstale":"2024-02-05T07:24:39.666131+0000","last_undegraded":"2024-02-05T07:24:39.666131+0000","last_fullsized":"2024-02-05T07:24:39.666131+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:30.473582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:30.473582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"189'8953","reported_seq":16310,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666377+0000","last_change":"2024-02-05T07:24:34.267533+0000","last_active":"2024-02-05T07:24:39.666377+0000","last_peered":"2024-02-05T07:24:39.666377+0000","last_clean":"2024-02-05T07:24:39.666377+0000","last_became_active":"2024-02-05T07:24:34.235391+0000","last_became_peered":"2024-02-05T07:24:34.235391+0000","last_unstale":"2024-02-05T07:24:39.666377+0000","last_undegraded":"2024-02-05T07:24:39.666377+0000","last_fullsized":"2024-02-05T07:24:39.666377+0000","mapping_epoch":186,"log_start":"125'4908","ondisk_log_start":"125'4908","created":118,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"173'7943","last_scrub_stamp":"2024-02-05T07:24:06.490687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:06.490687+0000","objects_scrubbed":1013,"log_size":4045,"log_dups_size":0,"ondisk_log_size":4045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:31.561677+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1013,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":287,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667637+0000","last_change":"2024-02-05T07:21:30.382672+0000","last_active":"2024-02-05T07:24:39.667637+0000","last_peered":"2024-02-05T07:24:39.667637+0000","last_clean":"2024-02-05T07:24:39.667637+0000","last_became_active":"2024-02-05T07:18:51.446604+0000","last_became_peered":"2024-02-05T07:18:51.446604+0000","last_unstale":"2024-02-05T07:24:39.667637+0000","last_undegraded":"2024-02-05T07:24:39.667637+0000","last_fullsized":"2024-02-05T07:24:39.667637+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:21:30.382584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:30.382584+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.367680+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":202,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193905+0000","last_change":"2024-02-05T07:24:34.210161+0000","last_active":"2024-02-05T07:24:39.193905+0000","last_peered":"2024-02-05T07:24:39.193905+0000","last_clean":"2024-02-05T07:24:39.193905+0000","last_became_active":"2024-02-05T07:24:34.209580+0000","last_became_peered":"2024-02-05T07:24:34.209580+0000","last_unstale":"2024-02-05T07:24:39.193905+0000","last_undegraded":"2024-02-05T07:24:39.193905+0000","last_fullsized":"2024-02-05T07:24:39.193905+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:14.255462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:14.255462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670777+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":18,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.669751+0000","last_change":"2024-02-05T07:24:34.242587+0000","last_active":"2024-02-05T07:24:39.669751+0000","last_peered":"2024-02-05T07:24:39.669751+0000","last_clean":"2024-02-05T07:24:39.669751+0000","last_became_active":"2024-02-05T07:24:34.240115+0000","last_became_peered":"2024-02-05T07:24:34.240115+0000","last_unstale":"2024-02-05T07:24:39.669751+0000","last_undegraded":"2024-02-05T07:24:39.669751+0000","last_fullsized":"2024-02-05T07:24:39.669751+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:26.202131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:26.202131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.239556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":269,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.195165+0000","last_change":"2024-02-05T07:22:55.794256+0000","last_active":"2024-02-05T07:24:39.195165+0000","last_peered":"2024-02-05T07:24:39.195165+0000","last_clean":"2024-02-05T07:24:39.195165+0000","last_became_active":"2024-02-05T07:18:55.466164+0000","last_became_peered":"2024-02-05T07:18:55.466164+0000","last_unstale":"2024-02-05T07:24:39.195165+0000","last_undegraded":"2024-02-05T07:24:39.195165+0000","last_fullsized":"2024-02-05T07:24:39.195165+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:22:55.794149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:22:55.794149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670991+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"189'12714","reported_seq":18811,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.196045+0000","last_change":"2024-02-05T07:24:27.272509+0000","last_active":"2024-02-05T07:24:39.196045+0000","last_peered":"2024-02-05T07:24:39.196045+0000","last_clean":"2024-02-05T07:24:39.196045+0000","last_became_active":"2024-02-05T07:24:27.247408+0000","last_became_peered":"2024-02-05T07:24:27.247408+0000","last_unstale":"2024-02-05T07:24:39.196045+0000","last_undegraded":"2024-02-05T07:24:39.196045+0000","last_fullsized":"2024-02-05T07:24:39.196045+0000","mapping_epoch":183,"log_start":"142'7721","ondisk_log_start":"142'7721","created":177,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":224,"log_size":4993,"log_dups_size":0,"ondisk_log_size":4993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.688157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6490,"num_write_kb":208384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1571,"num_bytes_recovered":102858752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":52,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.670039+0000","last_change":"2024-02-05T07:24:34.241141+0000","last_active":"2024-02-05T07:24:39.670039+0000","last_peered":"2024-02-05T07:24:39.670039+0000","last_clean":"2024-02-05T07:24:39.670039+0000","last_became_active":"2024-02-05T07:24:34.237056+0000","last_became_peered":"2024-02-05T07:24:34.237056+0000","last_unstale":"2024-02-05T07:24:39.670039+0000","last_undegraded":"2024-02-05T07:24:39.670039+0000","last_fullsized":"2024-02-05T07:24:39.670039+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:20:22.291371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:22.291371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.239498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":284,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194438+0000","last_change":"2024-02-05T07:23:47.646441+0000","last_active":"2024-02-05T07:24:39.194438+0000","last_peered":"2024-02-05T07:24:39.194438+0000","last_clean":"2024-02-05T07:24:39.194438+0000","last_became_active":"2024-02-05T07:23:47.610436+0000","last_became_peered":"2024-02-05T07:23:47.610436+0000","last_unstale":"2024-02-05T07:24:39.194438+0000","last_undegraded":"2024-02-05T07:24:39.194438+0000","last_fullsized":"2024-02-05T07:24:39.194438+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:21:42.616129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:42.616129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670863+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":346,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667545+0000","last_change":"2024-02-05T07:21:39.441651+0000","last_active":"2024-02-05T07:24:39.667545+0000","last_peered":"2024-02-05T07:24:39.667545+0000","last_clean":"2024-02-05T07:24:39.667545+0000","last_became_active":"2024-02-05T07:18:53.461011+0000","last_became_peered":"2024-02-05T07:18:53.461011+0000","last_unstale":"2024-02-05T07:24:39.667545+0000","last_undegraded":"2024-02-05T07:24:39.667545+0000","last_fullsized":"2024-02-05T07:24:39.667545+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:21:39.441520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.441520+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.367834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":262,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.669756+0000","last_change":"2024-02-05T07:21:39.490602+0000","last_active":"2024-02-05T07:24:39.669756+0000","last_peered":"2024-02-05T07:24:39.669756+0000","last_clean":"2024-02-05T07:24:39.669756+0000","last_became_active":"2024-02-05T07:18:55.476071+0000","last_became_peered":"2024-02-05T07:18:55.476071+0000","last_unstale":"2024-02-05T07:24:39.669756+0000","last_undegraded":"2024-02-05T07:24:39.669756+0000","last_fullsized":"2024-02-05T07:24:39.669756+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.490479+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"189'12608","reported_seq":12817,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194666+0000","last_change":"2024-02-05T07:24:22.549844+0000","last_active":"2024-02-05T07:24:39.194666+0000","last_peered":"2024-02-05T07:24:39.194666+0000","last_clean":"2024-02-05T07:24:39.194666+0000","last_became_active":"2024-02-05T07:24:22.524807+0000","last_became_peered":"2024-02-05T07:24:22.524807+0000","last_unstale":"2024-02-05T07:24:39.194666+0000","last_undegraded":"2024-02-05T07:24:39.194666+0000","last_fullsized":"2024-02-05T07:24:39.194666+0000","mapping_epoch":179,"log_start":"141'7524","ondisk_log_start":"141'7524","created":177,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":5084,"log_dups_size":0,"ondisk_log_size":5084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32756,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6563,"num_write_kb":209985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":120,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194906+0000","last_change":"2024-02-05T07:23:47.648580+0000","last_active":"2024-02-05T07:24:39.194906+0000","last_peered":"2024-02-05T07:24:39.194906+0000","last_clean":"2024-02-05T07:24:39.194906+0000","last_became_active":"2024-02-05T07:23:47.610303+0000","last_became_peered":"2024-02-05T07:23:47.610303+0000","last_unstale":"2024-02-05T07:24:39.194906+0000","last_undegraded":"2024-02-05T07:24:39.194906+0000","last_fullsized":"2024-02-05T07:24:39.194906+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:20:01.841489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:20:01.841489+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":272,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.372892+0000","last_change":"2024-02-05T07:24:10.630154+0000","last_active":"2024-02-05T07:24:39.372892+0000","last_peered":"2024-02-05T07:24:39.372892+0000","last_clean":"2024-02-05T07:24:39.372892+0000","last_became_active":"2024-02-05T07:18:49.427837+0000","last_became_peered":"2024-02-05T07:18:49.427837+0000","last_unstale":"2024-02-05T07:24:39.372892+0000","last_undegraded":"2024-02-05T07:24:39.372892+0000","last_fullsized":"2024-02-05T07:24:39.372892+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:10.630020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:10.630020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:13.542894+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":270,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.373072+0000","last_change":"2024-02-05T07:23:47.606077+0000","last_active":"2024-02-05T07:24:39.373072+0000","last_peered":"2024-02-05T07:24:39.373072+0000","last_clean":"2024-02-05T07:24:39.373072+0000","last_became_active":"2024-02-05T07:23:47.605523+0000","last_became_peered":"2024-02-05T07:23:47.605523+0000","last_unstale":"2024-02-05T07:24:39.373072+0000","last_undegraded":"2024-02-05T07:24:39.373072+0000","last_fullsized":"2024-02-05T07:24:39.373072+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:41.800067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:41.800067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":66,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667741+0000","last_change":"2024-02-05T07:23:47.610492+0000","last_active":"2024-02-05T07:24:39.667741+0000","last_peered":"2024-02-05T07:24:39.667741+0000","last_clean":"2024-02-05T07:24:39.667741+0000","last_became_active":"2024-02-05T07:23:47.609917+0000","last_became_peered":"2024-02-05T07:23:47.609917+0000","last_unstale":"2024-02-05T07:24:39.667741+0000","last_undegraded":"2024-02-05T07:24:39.667741+0000","last_fullsized":"2024-02-05T07:24:39.667741+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:31.211338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:31.211338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.367734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1e","version":"189'12311","reported_seq":12519,"reported_epoch":190,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:24:39.665802+0000","last_change":"2024-02-05T07:24:23.631683+0000","last_active":"2024-02-05T07:24:39.665802+0000","last_peered":"2024-02-05T07:24:39.665802+0000","last_clean":"2024-02-05T07:24:20.735206+0000","last_became_active":"2024-02-05T07:24:23.585766+0000","last_became_peered":"2024-02-05T07:24:23.585766+0000","last_unstale":"2024-02-05T07:24:39.665802+0000","last_undegraded":"2024-02-05T07:24:39.665802+0000","last_fullsized":"2024-02-05T07:24:39.665802+0000","mapping_epoch":180,"log_start":"137'7084","ondisk_log_start":"137'7084","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":50,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6405,"num_write_kb":205760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[0,5,2],"avail_no_missing":["0","2","5"],"object_location_counts":[{"shards":"0,2,5","objects":25}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":315,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.195323+0000","last_change":"2024-02-05T07:23:47.657951+0000","last_active":"2024-02-05T07:24:39.195323+0000","last_peered":"2024-02-05T07:24:39.195323+0000","last_clean":"2024-02-05T07:24:39.195323+0000","last_became_active":"2024-02-05T07:23:47.619716+0000","last_became_peered":"2024-02-05T07:23:47.619716+0000","last_unstale":"2024-02-05T07:24:39.195323+0000","last_undegraded":"2024-02-05T07:24:39.195323+0000","last_fullsized":"2024-02-05T07:24:39.195323+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:21:36.561503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:36.561503+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741894+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193973+0000","last_change":"2024-02-05T07:24:34.234686+0000","last_active":"2024-02-05T07:24:39.193973+0000","last_peered":"2024-02-05T07:24:39.193973+0000","last_clean":"2024-02-05T07:24:39.193973+0000","last_became_active":"2024-02-05T07:24:34.234199+0000","last_became_peered":"2024-02-05T07:24:34.234199+0000","last_unstale":"2024-02-05T07:24:39.193973+0000","last_undegraded":"2024-02-05T07:24:39.193973+0000","last_fullsized":"2024-02-05T07:24:39.193973+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:32.192372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:32.192372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":265,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.195005+0000","last_change":"2024-02-05T07:22:58.849105+0000","last_active":"2024-02-05T07:24:39.195005+0000","last_peered":"2024-02-05T07:24:39.195005+0000","last_clean":"2024-02-05T07:24:39.195005+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:24:39.195005+0000","last_undegraded":"2024-02-05T07:24:39.195005+0000","last_fullsized":"2024-02-05T07:24:39.195005+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:22:58.846677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:22:58.846677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.671023+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":260,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.373448+0000","last_change":"2024-02-05T07:21:38.839453+0000","last_active":"2024-02-05T07:24:39.373448+0000","last_peered":"2024-02-05T07:24:39.373448+0000","last_clean":"2024-02-05T07:24:39.373448+0000","last_became_active":"2024-02-05T07:18:55.505678+0000","last_became_peered":"2024-02-05T07:18:55.505678+0000","last_unstale":"2024-02-05T07:24:39.373448+0000","last_undegraded":"2024-02-05T07:24:39.373448+0000","last_fullsized":"2024-02-05T07:24:39.373448+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:38.839336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:38.839336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"189'12639","reported_seq":14960,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667329+0000","last_change":"2024-02-05T07:24:26.234249+0000","last_active":"2024-02-05T07:24:39.667329+0000","last_peered":"2024-02-05T07:24:39.667329+0000","last_clean":"2024-02-05T07:24:39.667329+0000","last_became_active":"2024-02-05T07:24:26.219215+0000","last_became_peered":"2024-02-05T07:24:26.219215+0000","last_unstale":"2024-02-05T07:24:39.667329+0000","last_undegraded":"2024-02-05T07:24:39.667329+0000","last_fullsized":"2024-02-05T07:24:39.667329+0000","mapping_epoch":182,"log_start":"141'7605","ondisk_log_start":"141'7605","created":177,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":5034,"log_dups_size":0,"ondisk_log_size":5034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.357530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6589,"num_write_kb":210752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":525,"num_bytes_recovered":34406400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":282,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193859+0000","last_change":"2024-02-05T07:23:47.616444+0000","last_active":"2024-02-05T07:24:39.193859+0000","last_peered":"2024-02-05T07:24:39.193859+0000","last_clean":"2024-02-05T07:24:39.193859+0000","last_became_active":"2024-02-05T07:23:47.603918+0000","last_became_peered":"2024-02-05T07:23:47.603918+0000","last_unstale":"2024-02-05T07:24:39.193859+0000","last_undegraded":"2024-02-05T07:24:39.193859+0000","last_fullsized":"2024-02-05T07:24:39.193859+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:21:15.028573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:15.028573+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":86,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.372690+0000","last_change":"2024-02-05T07:23:27.438105+0000","last_active":"2024-02-05T07:24:39.372690+0000","last_peered":"2024-02-05T07:24:39.372690+0000","last_clean":"2024-02-05T07:24:39.372690+0000","last_became_active":"2024-02-05T07:23:27.437776+0000","last_became_peered":"2024-02-05T07:23:27.437776+0000","last_unstale":"2024-02-05T07:24:39.372690+0000","last_undegraded":"2024-02-05T07:24:39.372690+0000","last_fullsized":"2024-02-05T07:24:39.372690+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:39.936643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.936643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":264,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666555+0000","last_change":"2024-02-05T07:24:16.310748+0000","last_active":"2024-02-05T07:24:39.666555+0000","last_peered":"2024-02-05T07:24:39.666555+0000","last_clean":"2024-02-05T07:24:39.666555+0000","last_became_active":"2024-02-05T07:18:53.456881+0000","last_became_peered":"2024-02-05T07:18:53.456881+0000","last_unstale":"2024-02-05T07:24:39.666555+0000","last_undegraded":"2024-02-05T07:24:39.666555+0000","last_fullsized":"2024-02-05T07:24:39.666555+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:16.310666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:16.310666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:30.390505+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":258,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.667505+0000","last_change":"2024-02-05T07:23:47.597763+0000","last_active":"2024-02-05T07:24:39.667505+0000","last_peered":"2024-02-05T07:24:39.667505+0000","last_clean":"2024-02-05T07:24:39.667505+0000","last_became_active":"2024-02-05T07:23:47.596885+0000","last_became_peered":"2024-02-05T07:23:47.596885+0000","last_unstale":"2024-02-05T07:24:39.667505+0000","last_undegraded":"2024-02-05T07:24:39.667505+0000","last_fullsized":"2024-02-05T07:24:39.667505+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:15.751084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:15.751084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.864250+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"189'8740","reported_seq":15998,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666566+0000","last_change":"2024-02-05T07:24:34.257325+0000","last_active":"2024-02-05T07:24:39.666566+0000","last_peered":"2024-02-05T07:24:39.666566+0000","last_clean":"2024-02-05T07:24:39.666566+0000","last_became_active":"2024-02-05T07:24:34.234000+0000","last_became_peered":"2024-02-05T07:24:34.234000+0000","last_unstale":"2024-02-05T07:24:39.666566+0000","last_undegraded":"2024-02-05T07:24:39.666566+0000","last_fullsized":"2024-02-05T07:24:39.666566+0000","mapping_epoch":186,"log_start":"117'4570","ondisk_log_start":"117'4570","created":118,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"173'7772","last_scrub_stamp":"2024-02-05T07:24:05.618199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:05.618199+0000","objects_scrubbed":977,"log_size":4170,"log_dups_size":0,"ondisk_log_size":4170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:07.682718+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":977,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"89'13","reported_seq":304,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.372899+0000","last_change":"2024-02-05T07:23:47.643672+0000","last_active":"2024-02-05T07:24:39.372899+0000","last_peered":"2024-02-05T07:24:39.372899+0000","last_clean":"2024-02-05T07:24:39.372899+0000","last_became_active":"2024-02-05T07:23:47.618422+0000","last_became_peered":"2024-02-05T07:23:47.618422+0000","last_unstale":"2024-02-05T07:24:39.372899+0000","last_undegraded":"2024-02-05T07:24:39.372899+0000","last_fullsized":"2024-02-05T07:24:39.372899+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-05T07:21:34.888266+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:21:34.888266+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":86,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.372878+0000","last_change":"2024-02-05T07:23:27.418016+0000","last_active":"2024-02-05T07:24:39.372878+0000","last_peered":"2024-02-05T07:24:39.372878+0000","last_clean":"2024-02-05T07:24:39.372878+0000","last_became_active":"2024-02-05T07:23:27.417717+0000","last_became_peered":"2024-02-05T07:23:27.417717+0000","last_unstale":"2024-02-05T07:24:39.372878+0000","last_undegraded":"2024-02-05T07:24:39.372878+0000","last_fullsized":"2024-02-05T07:24:39.372878+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:47.776379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:47.776379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":256,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668677+0000","last_change":"2024-02-05T07:21:19.716500+0000","last_active":"2024-02-05T07:24:39.668677+0000","last_peered":"2024-02-05T07:24:39.668677+0000","last_clean":"2024-02-05T07:24:39.668677+0000","last_became_active":"2024-02-05T07:18:53.464520+0000","last_became_peered":"2024-02-05T07:18:53.464520+0000","last_unstale":"2024-02-05T07:24:39.668677+0000","last_undegraded":"2024-02-05T07:24:39.668677+0000","last_fullsized":"2024-02-05T07:24:39.668677+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:19.716375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:19.716375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:40.864221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":18,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193463+0000","last_change":"2024-02-05T07:24:34.230834+0000","last_active":"2024-02-05T07:24:39.193463+0000","last_peered":"2024-02-05T07:24:39.193463+0000","last_clean":"2024-02-05T07:24:39.193463+0000","last_became_active":"2024-02-05T07:24:34.230537+0000","last_became_peered":"2024-02-05T07:24:34.230537+0000","last_unstale":"2024-02-05T07:24:39.193463+0000","last_undegraded":"2024-02-05T07:24:39.193463+0000","last_fullsized":"2024-02-05T07:24:39.193463+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:31.252686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:20:31.252686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.670761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.19","version":"189'8759","reported_seq":18263,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.668160+0000","last_change":"2024-02-05T07:24:02.315647+0000","last_active":"2024-02-05T07:24:39.668160+0000","last_peered":"2024-02-05T07:24:39.668160+0000","last_clean":"2024-02-05T07:24:39.668160+0000","last_became_active":"2024-02-05T07:23:38.523311+0000","last_became_peered":"2024-02-05T07:23:38.523311+0000","last_unstale":"2024-02-05T07:24:39.668160+0000","last_undegraded":"2024-02-05T07:24:39.668160+0000","last_fullsized":"2024-02-05T07:24:39.668160+0000","mapping_epoch":155,"log_start":"117'4554","ondisk_log_start":"117'4554","created":118,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"172'7705","last_scrub_stamp":"2024-02-05T07:24:02.315408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:24:02.315408+0000","objects_scrubbed":1054,"log_size":4205,"log_dups_size":0,"ondisk_log_size":4205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:21.398564+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1054,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":126,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193249+0000","last_change":"2024-02-05T07:23:47.645884+0000","last_active":"2024-02-05T07:24:39.193249+0000","last_peered":"2024-02-05T07:24:39.193249+0000","last_clean":"2024-02-05T07:24:39.193249+0000","last_became_active":"2024-02-05T07:23:47.621081+0000","last_became_peered":"2024-02-05T07:23:47.621081+0000","last_unstale":"2024-02-05T07:24:39.193249+0000","last_undegraded":"2024-02-05T07:24:39.193249+0000","last_fullsized":"2024-02-05T07:24:39.193249+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:21:27.141800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:27.141800+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":74,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.670150+0000","last_change":"2024-02-05T07:24:15.235343+0000","last_active":"2024-02-05T07:24:39.670150+0000","last_peered":"2024-02-05T07:24:39.670150+0000","last_clean":"2024-02-05T07:24:39.670150+0000","last_became_active":"2024-02-05T07:23:47.610737+0000","last_became_peered":"2024-02-05T07:23:47.610737+0000","last_unstale":"2024-02-05T07:24:39.670150+0000","last_undegraded":"2024-02-05T07:24:39.670150+0000","last_fullsized":"2024-02-05T07:24:39.670150+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:15.235232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:15.235232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:43.769610+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":264,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193512+0000","last_change":"2024-02-05T07:21:31.904690+0000","last_active":"2024-02-05T07:24:39.193512+0000","last_peered":"2024-02-05T07:24:39.193512+0000","last_clean":"2024-02-05T07:24:39.193512+0000","last_became_active":"2024-02-05T07:18:53.461506+0000","last_became_peered":"2024-02-05T07:18:53.461506+0000","last_unstale":"2024-02-05T07:24:39.193512+0000","last_undegraded":"2024-02-05T07:24:39.193512+0000","last_fullsized":"2024-02-05T07:24:39.193512+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:31.904568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:31.904568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":74,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.666942+0000","last_change":"2024-02-05T07:24:09.290070+0000","last_active":"2024-02-05T07:24:39.666942+0000","last_peered":"2024-02-05T07:24:39.666942+0000","last_clean":"2024-02-05T07:24:39.666942+0000","last_became_active":"2024-02-05T07:23:47.609992+0000","last_became_peered":"2024-02-05T07:23:47.609992+0000","last_unstale":"2024-02-05T07:24:39.666942+0000","last_undegraded":"2024-02-05T07:24:39.666942+0000","last_fullsized":"2024-02-05T07:24:39.666942+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:09.289989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:09.289989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:17.419052+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"189'12723","reported_seq":19006,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193693+0000","last_change":"2024-02-05T07:24:27.230992+0000","last_active":"2024-02-05T07:24:39.193693+0000","last_peered":"2024-02-05T07:24:39.193693+0000","last_clean":"2024-02-05T07:24:39.193693+0000","last_became_active":"2024-02-05T07:24:27.214954+0000","last_became_peered":"2024-02-05T07:24:27.214954+0000","last_unstale":"2024-02-05T07:24:39.193693+0000","last_undegraded":"2024-02-05T07:24:39.193693+0000","last_fullsized":"2024-02-05T07:24:39.193693+0000","mapping_epoch":183,"log_start":"137'7469","ondisk_log_start":"137'7469","created":177,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:20:44.752333+0000","objects_scrubbed":0,"log_size":5254,"log_dups_size":0,"ondisk_log_size":5254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.712091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6611,"num_write_kb":212704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1620,"num_bytes_recovered":106135552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":284,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193734+0000","last_change":"2024-02-05T07:23:15.477701+0000","last_active":"2024-02-05T07:24:39.193734+0000","last_peered":"2024-02-05T07:24:39.193734+0000","last_clean":"2024-02-05T07:24:39.193734+0000","last_became_active":"2024-02-05T07:18:51.448924+0000","last_became_peered":"2024-02-05T07:18:51.448924+0000","last_unstale":"2024-02-05T07:24:39.193734+0000","last_undegraded":"2024-02-05T07:24:39.193734+0000","last_fullsized":"2024-02-05T07:24:39.193734+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:23:15.477555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:15.477555+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:41.604865+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194882+0000","last_change":"2024-02-05T07:24:34.229611+0000","last_active":"2024-02-05T07:24:39.194882+0000","last_peered":"2024-02-05T07:24:39.194882+0000","last_clean":"2024-02-05T07:24:39.194882+0000","last_became_active":"2024-02-05T07:24:34.229294+0000","last_became_peered":"2024-02-05T07:24:34.229294+0000","last_unstale":"2024-02-05T07:24:39.194882+0000","last_undegraded":"2024-02-05T07:24:39.194882+0000","last_fullsized":"2024-02-05T07:24:39.194882+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:20:35.243281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:20:35.243281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:42.741736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":265,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194030+0000","last_change":"2024-02-05T07:23:07.907068+0000","last_active":"2024-02-05T07:24:39.194030+0000","last_peered":"2024-02-05T07:24:39.194030+0000","last_clean":"2024-02-05T07:24:39.194030+0000","last_became_active":"2024-02-05T07:18:53.460820+0000","last_became_peered":"2024-02-05T07:18:53.460820+0000","last_unstale":"2024-02-05T07:24:39.194030+0000","last_undegraded":"2024-02-05T07:24:39.194030+0000","last_fullsized":"2024-02-05T07:24:39.194030+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:07.906963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:23:07.906963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.671143+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":260,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.194086+0000","last_change":"2024-02-05T07:24:06.379008+0000","last_active":"2024-02-05T07:24:39.194086+0000","last_peered":"2024-02-05T07:24:39.194086+0000","last_clean":"2024-02-05T07:24:39.194086+0000","last_became_active":"2024-02-05T07:18:55.466001+0000","last_became_peered":"2024-02-05T07:18:55.466001+0000","last_unstale":"2024-02-05T07:24:39.194086+0000","last_undegraded":"2024-02-05T07:24:39.194086+0000","last_fullsized":"2024-02-05T07:24:39.194086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:06.378880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:06.378880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:22.043257+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"189'15808","reported_seq":16602,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.193161+0000","last_change":"2024-02-05T07:24:36.256508+0000","last_active":"2024-02-05T07:24:39.193161+0000","last_peered":"2024-02-05T07:24:39.193161+0000","last_clean":"2024-02-05T07:24:39.193161+0000","last_became_active":"2024-02-05T07:24:36.236463+0000","last_became_peered":"2024-02-05T07:24:36.236463+0000","last_unstale":"2024-02-05T07:24:39.193161+0000","last_undegraded":"2024-02-05T07:24:39.193161+0000","last_fullsized":"2024-02-05T07:24:39.193161+0000","mapping_epoch":188,"log_start":"154'11564","ondisk_log_start":"154'11564","created":177,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"156'13339","last_scrub_stamp":"2024-02-05T07:23:40.012704+0000","last_deep_scrub":"156'13339","last_deep_scrub_stamp":"2024-02-05T07:23:40.012704+0000","last_clean_scrub_stamp":"2024-02-05T07:23:40.012704+0000","objects_scrubbed":3400,"log_size":4244,"log_dups_size":0,"ondisk_log_size":4244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":20,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:41.712296+0000","scrub_duration":19992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6470,"num_write_kb":206464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":9175040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.20","version":"189'8386","reported_seq":13283,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-02-05T07:24:39.192854+0000","last_change":"2024-02-05T07:24:27.266405+0000","last_active":"2024-02-05T07:24:39.192854+0000","last_peered":"2024-02-05T07:24:39.192854+0000","last_clean":"2024-02-05T07:24:39.192854+0000","last_became_active":"2024-02-05T07:24:27.249856+0000","last_became_peered":"2024-02-05T07:24:27.249856+0000","last_unstale":"2024-02-05T07:24:39.192854+0000","last_undegraded":"2024-02-05T07:24:39.192854+0000","last_fullsized":"2024-02-05T07:24:39.192854+0000","mapping_epoch":183,"log_start":"108'1895","ondisk_log_start":"108'1895","created":177,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":6,"last_scrub":"119'4604","last_scrub_stamp":"2024-02-05T07:21:43.479351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.479351+0000","objects_scrubbed":2300,"log_size":6491,"log_dups_size":0,"ondisk_log_size":6491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:24:43.776352+0000","scrub_duration":618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2019,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1281,"num_bytes_recovered":57147392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.21","version":"189'8788","reported_seq":9276,"reported_epoch":190,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:24:39.193824+0000","last_change":"2024-02-05T07:24:22.575857+0000","last_active":"2024-02-05T07:24:39.193824+0000","last_peered":"2024-02-05T07:24:39.193824+0000","last_clean":"2024-02-05T07:23:46.544320+0000","last_became_active":"2024-02-05T07:24:22.558108+0000","last_became_peered":"2024-02-05T07:24:22.558108+0000","last_unstale":"2024-02-05T07:24:39.193824+0000","last_undegraded":"2024-02-05T07:24:39.193824+0000","last_fullsized":"2024-02-05T07:24:39.193824+0000","mapping_epoch":179,"log_start":"110'3615","ondisk_log_start":"110'3615","created":177,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":6,"last_scrub":"119'4712","last_scrub_stamp":"2024-02-05T07:21:43.408403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:44.752333+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.408403+0000","objects_scrubbed":2431,"log_size":5173,"log_dups_size":0,"ondisk_log_size":5173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":616,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2179,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[4,6,3],"avail_no_missing":["4","1","3","6"],"object_location_counts":[{"shards":"3,4,6","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":8585204,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":77,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126959,"num_write_kb":2875713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16384,"num_bytes_recovered":985137152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":132644864,"data_stored":132644864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":169077,"ondisk_log_size":169077,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":343,"num_read_kb":338,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":687,"num_read_kb":444,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037835,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":222120,"kb_used_data":6304,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93523032,"statfs":{"total":95995035648,"available":95767584768,"internally_reserved":0,"allocated":6455296,"data_stored":6012538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":220985802},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":6.8929999999999998,"5min":2.9369999999999998,"15min":1.3089999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":63.039000000000001,"5min":63.039000000000001,"15min":63.039000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":6.6950000000000003,"5min":2.8119999999999998,"15min":1.2769999999999999},"min":{"1min":0.28699999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":65.296000000000006,"5min":65.296000000000006,"15min":65.296000000000006},"last":2.2069999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":3.8210000000000002,"5min":1.4079999999999999,"15min":0.81899999999999995},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":32.127000000000002,"5min":32.127000000000002,"15min":32.127000000000002},"last":2.1579999999999999},{"interface":"front","average":{"1min":4.6710000000000003,"5min":1.5329999999999999,"15min":0.88},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":32.173999999999999,"5min":32.173999999999999,"15min":32.173999999999999},"last":0.34399999999999997}]},{"osd":2,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":2.6499999999999999,"5min":1.8169999999999999,"15min":0.95499999999999996},"min":{"1min":0.26300000000000001,"5min":0.249,"15min":0.249},"max":{"1min":32.262999999999998,"5min":51.127000000000002,"15min":51.127000000000002},"last":0.374},{"interface":"front","average":{"1min":3.3010000000000002,"5min":2.1680000000000001,"15min":1.0669999999999999},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":32.296999999999997,"5min":51.113999999999997,"15min":51.113999999999997},"last":1.8400000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.7769999999999999,"15min":0.94799999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.1349999999999998,"5min":58.341999999999999,"15min":58.341999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.6100000000000001,"15min":0.90600000000000003},"min":{"1min":0.311,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.8730000000000002,"5min":57.765000000000001,"15min":57.765000000000001},"last":2.0710000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.45900000000000002,"15min":0.44500000000000001},"min":{"1min":0.27000000000000002,"5min":0.25,"15min":0.25},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.48999999999999999,"15min":0.45500000000000002},"min":{"1min":0.28899999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":0.84499999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.49199999999999999,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.3940000000000001,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":2.1269999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.56299999999999994,"15min":0.51800000000000002},"min":{"1min":0.34899999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.8679999999999999,"5min":2.8679999999999999,"15min":2.8679999999999999},"last":0.64900000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.56000000000000005,"15min":0.51700000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.55100000000000005,"15min":0.52400000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.97,"5min":1.97,"15min":1.97},"last":0.98599999999999999}]}]},{"osd":6,"up_from":52,"seq":223338299473,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":195140,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93550012,"statfs":{"total":95995035648,"available":95795212288,"internally_reserved":0,"allocated":1568768,"data_stored":1098416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":198244806},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:24:17 2024","interfaces":[{"interface":"back","average":{"1min":3.141,"5min":3.0550000000000002,"15min":1.385},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":22.024000000000001,"5min":57.289999999999999,"15min":57.289999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":3.0019999999999998,"5min":2.8999999999999999,"15min":1.319},"min":{"1min":0.28999999999999998,"5min":0.22,"15min":0.22},"max":{"1min":22.161000000000001,"5min":56.302,"15min":56.302},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:24:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":0.874,"15min":0.65800000000000003},"min":{"1min":0.26300000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":6.2439999999999998,"5min":6.2439999999999998,"15min":6.2439999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.117,"5min":0.83099999999999996,"15min":0.64600000000000002},"min":{"1min":0.30499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":6.915,"5min":6.915,"15min":6.915},"last":0.443}]},{"osd":2,"last update":"Mon Feb 5 07:24:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.5529999999999999,"15min":0.90400000000000003},"min":{"1min":0.27500000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.423,"5min":32.509999999999998,"15min":32.509999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.8380000000000001,"5min":1.7929999999999999,"15min":0.95199999999999996},"min":{"1min":0.33400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":8.9979999999999993,"5min":32.536000000000001,"15min":32.536000000000001},"last":0.45900000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:24:17 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":2.0179999999999998,"15min":1.046},"min":{"1min":0.40600000000000003,"5min":0.22,"15min":0.22},"max":{"1min":20.503,"5min":36.976999999999997,"15min":36.976999999999997},"last":0.55100000000000005},{"interface":"front","average":{"1min":2.1000000000000001,"5min":1.843,"15min":0.99099999999999999},"min":{"1min":0.33400000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":20.491,"5min":34.222999999999999,"15min":34.222999999999999},"last":0.52500000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.55700000000000005,"15min":0.45300000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.5609999999999999,"5min":4.5609999999999999,"15min":4.5609999999999999},"last":4.173},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.55800000000000005,"15min":0.48599999999999999},"min":{"1min":0.31900000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.0060000000000002,"5min":5.0060000000000002,"15min":5.0060000000000002},"last":7.7960000000000003}]},{"osd":5,"last update":"Mon Feb 5 07:24:17 2024","interfaces":[{"interface":"back","average":{"1min":2.3540000000000001,"5min":0.85299999999999998,"15min":0.58099999999999996},"min":{"1min":0.32100000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":27.021999999999998,"5min":27.021999999999998,"15min":27.021999999999998},"last":4.1369999999999996},{"interface":"front","average":{"1min":2.472,"5min":0.90400000000000003,"15min":0.60299999999999998},"min":{"1min":0.35299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":23.611999999999998,"5min":23.611999999999998,"15min":23.611999999999998},"last":5.0629999999999997}]},{"osd":7,"last update":"Mon Feb 5 07:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.35399999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":4.2130000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54500000000000004,"15min":0.54200000000000004},"min":{"1min":0.36499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.88500000000000001,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":5.0190000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":296732,"kb_used_data":98960,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93448420,"statfs":{"total":95995035648,"available":95691182080,"internally_reserved":0,"allocated":101335040,"data_stored":100850832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":1.9970000000000001,"5min":3.7330000000000001,"15min":1.4710000000000001},"min":{"1min":0.223,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":6.1299999999999999,"5min":45.393000000000001,"15min":45.393000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.863,"5min":3.6960000000000002,"15min":1.458},"min":{"1min":0.246,"5min":0.23000000000000001,"15min":0.22800000000000001},"max":{"1min":8.7330000000000005,"5min":45.031999999999996,"15min":45.031999999999996},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":2.04,"15min":0.97899999999999998},"min":{"1min":0.22700000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.3380000000000001,"5min":39.259999999999998,"15min":39.259999999999998},"last":0.376},{"interface":"front","average":{"1min":2.4119999999999999,"5min":1.911,"15min":1.3720000000000001},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.7160000000000002,"5min":40.972000000000001,"15min":40.972000000000001},"last":0.55100000000000005}]},{"osd":3,"last update":"Mon Feb 5 07:24:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":2.2770000000000001,"15min":1.0780000000000001},"min":{"1min":0.254,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.1699999999999999,"5min":43.823,"15min":43.823},"last":0.627},{"interface":"front","average":{"1min":3.1070000000000002,"5min":2.4199999999999999,"15min":1.1359999999999999},"min":{"1min":0.32200000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":7.617,"5min":39.314999999999998,"15min":39.314999999999998},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":5.327,"5min":1.9530000000000001,"15min":1.044},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":61.334000000000003,"5min":61.334000000000003,"15min":61.334000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":5.5339999999999998,"5min":2.0699999999999998,"15min":1.1879999999999999},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":61.786000000000001,"5min":61.786000000000001,"15min":61.786000000000001},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Feb 5 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.7549999999999999,"5min":1.131,"15min":0.79300000000000004},"min":{"1min":0.33600000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":4.9199999999999999,"5min":5.9329999999999998,"15min":5.9329999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":2.2029999999999998,"5min":1.2869999999999999,"15min":0.95799999999999996},"min":{"1min":0.38100000000000001,"5min":0.307,"15min":0.307},"max":{"1min":12.353999999999999,"5min":18.425999999999998,"15min":18.425999999999998},"last":0.72799999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.357,"15min":0.91100000000000003},"min":{"1min":0.34699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":5.2930000000000001,"5min":18.456,"15min":18.456},"last":0.83999999999999997},{"interface":"front","average":{"1min":2.0920000000000001,"5min":1.337,"15min":0.90200000000000002},"min":{"1min":0.376,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.9969999999999999,"5min":5.9930000000000003,"15min":5.9930000000000003},"last":0.77300000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":2.1219999999999999,"5min":1.296,"15min":0.81599999999999995},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":12.305999999999999,"5min":18.513000000000002,"15min":18.513000000000002},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.7989999999999999,"5min":1.2130000000000001,"15min":0.79900000000000004},"min":{"1min":0.35599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":5.165,"5min":18.117999999999999,"15min":18.117999999999999},"last":0.66100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738478,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":267584,"kb_used_data":37876,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93477568,"statfs":{"total":95995035648,"available":95721029632,"internally_reserved":0,"allocated":38785024,"data_stored":38328882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:24:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.645,"15min":0.755},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.349,"5min":15.731999999999999,"15min":15.731999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":1.8740000000000001,"5min":1.903,"15min":0.84999999999999998},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":9.3469999999999995,"5min":32.396000000000001,"15min":32.396000000000001},"last":0.371}]},{"osd":2,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":2.8639999999999999,"5min":1.6890000000000001,"15min":0.84699999999999998},"min":{"1min":0.28399999999999997,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":9.1440000000000001,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":2.2519999999999998,"5min":1.583,"15min":0.80800000000000005},"min":{"1min":0.19700000000000001,"5min":0.182,"15min":0.182},"max":{"1min":9.5150000000000006,"5min":10.75,"15min":10.75},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:24:06 2024","interfaces":[{"interface":"back","average":{"1min":5.5800000000000001,"5min":2.8849999999999998,"15min":1.2490000000000001},"min":{"1min":0.311,"5min":0.26500000000000001,"15min":0.246},"max":{"1min":65.155000000000001,"5min":65.155000000000001,"15min":65.155000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":5.2229999999999999,"5min":3.0899999999999999,"15min":1.3100000000000001},"min":{"1min":0.35499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":65.114000000000004,"5min":65.114000000000004,"15min":65.114000000000004},"last":0.39700000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:24:29 2024","interfaces":[{"interface":"back","average":{"1min":4.2270000000000003,"5min":2.0129999999999999,"15min":1.0389999999999999},"min":{"1min":0.35099999999999998,"5min":0.31,"15min":0.31},"max":{"1min":65.188000000000002,"5min":65.188000000000002,"15min":65.188000000000002},"last":0.435},{"interface":"front","average":{"1min":4.2530000000000001,"5min":2.1269999999999998,"15min":1.075},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":64.873000000000005,"5min":64.873000000000005,"15min":64.873000000000005},"last":0.48599999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:24:03 2024","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":1.8959999999999999,"15min":1},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":65.555999999999997,"5min":65.555999999999997,"15min":65.555999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":5.2930000000000001,"5min":2.0830000000000002,"15min":1.085},"min":{"1min":0.436,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":64.936999999999998,"5min":64.936999999999998,"15min":64.936999999999998},"last":0.58299999999999996}]},{"osd":6,"last update":"Mon Feb 5 07:24:24 2024","interfaces":[{"interface":"back","average":{"1min":4.4370000000000003,"5min":1.982,"15min":1.0580000000000001},"min":{"1min":0.34799999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":65.215000000000003,"5min":65.215000000000003,"15min":65.215000000000003},"last":0.59799999999999998},{"interface":"front","average":{"1min":4.5730000000000004,"5min":2.1230000000000002,"15min":1.135},"min":{"1min":0.376,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":65.230000000000004,"5min":65.230000000000004,"15min":65.230000000000004},"last":0.5}]},{"osd":7,"last update":"Mon Feb 5 07:23:39 2024","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.45,"15min":0.88800000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":5.3730000000000002,"5min":8.6679999999999993,"15min":8.6679999999999993},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.8620000000000001,"5min":1.3999999999999999,"15min":0.89600000000000002},"min":{"1min":0.46999999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":8.1940000000000008,"5min":8.6829999999999998,"15min":8.6829999999999998},"last":0.61799999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411428,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":259572,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93485580,"statfs":{"total":95995035648,"available":95729233920,"internally_reserved":0,"allocated":3579904,"data_stored":3135852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:23:45 2024","interfaces":[{"interface":"back","average":{"1min":4.9640000000000004,"5min":2.1760000000000002,"15min":0.97099999999999997},"min":{"1min":0.19500000000000001,"5min":0.192,"15min":0.192},"max":{"1min":67.418999999999997,"5min":67.418999999999997,"15min":67.418999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":6.7030000000000003,"5min":2.4660000000000002,"15min":1.0780000000000001},"min":{"1min":0.34200000000000003,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":68.171000000000006,"5min":68.171000000000006,"15min":68.171000000000006},"last":0.27400000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:23:45 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.2230000000000001,"15min":0.66600000000000004},"min":{"1min":0.20699999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.9500000000000002,"5min":12.609,"15min":12.609},"last":0.313},{"interface":"front","average":{"1min":1.635,"5min":1.3320000000000001,"15min":0.72699999999999998},"min":{"1min":0.26800000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":5.984,"5min":9.641,"15min":9.641},"last":0.28499999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:24:10 2024","interfaces":[{"interface":"back","average":{"1min":2.4169999999999998,"5min":1.7450000000000001,"15min":0.89900000000000002},"min":{"1min":0.249,"5min":0.214,"15min":0.214},"max":{"1min":7.3929999999999998,"5min":14.756,"15min":14.756},"last":0.36899999999999999},{"interface":"front","average":{"1min":2.2639999999999998,"5min":1.821,"15min":0.91900000000000004},"min":{"1min":0.28699999999999998,"5min":0.193,"15min":0.193},"max":{"1min":7.8099999999999996,"5min":9.6669999999999998,"15min":9.6669999999999998},"last":0.378}]},{"osd":4,"last update":"Mon Feb 5 07:24:32 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.2050000000000001,"15min":0.75800000000000001},"min":{"1min":0.33500000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":4.2130000000000001,"5min":7.29,"15min":7.29},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.236,"15min":0.78500000000000003},"min":{"1min":0.32200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":5.3499999999999996,"5min":8.5199999999999996,"15min":8.5199999999999996},"last":0.40799999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.1779999999999999,"15min":0.78800000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.1019999999999999,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.1040000000000001,"15min":0.77000000000000002},"min":{"1min":0.30599999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.9029999999999996,"5min":7.7279999999999998,"15min":7.7279999999999998},"last":0.53700000000000003}]},{"osd":6,"last update":"Mon Feb 5 07:24:21 2024","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.286,"15min":0.94299999999999995},"min":{"1min":0.376,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.9119999999999999,"5min":7.6829999999999998,"15min":7.6829999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.464,"5min":1.349,"15min":0.85899999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":4.1769999999999996,"5min":7.3150000000000004,"15min":7.3150000000000004},"last":0.41699999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:23:45 2024","interfaces":[{"interface":"back","average":{"1min":2.3330000000000002,"5min":1.2529999999999999,"15min":0.88300000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":7.2590000000000003,"5min":7.2590000000000003,"15min":7.2590000000000003},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.115,"5min":1.089,"15min":0.79100000000000004},"min":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.8039999999999998,"5min":7.7619999999999996,"15min":7.7619999999999996},"last":0.48699999999999999}]}]},{"osd":3,"up_from":0,"seq":111669149791,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953562,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122360,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622792,"statfs":{"total":95995035648,"available":95869739008,"internally_reserved":0,"allocated":962560,"data_stored":513369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:24:31 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.6759999999999999,"15min":0.91100000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.7280000000000002,"5min":67.719999999999999,"15min":67.719999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.9410000000000001,"15min":1.0149999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.3980000000000001,"5min":66.882000000000005,"15min":66.882000000000005},"last":0.52300000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.85799999999999998,"15min":0.63400000000000001},"min":{"1min":0.247,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":3.464,"5min":5.5709999999999997,"15min":5.5709999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.78100000000000003,"15min":0.629},"min":{"1min":0.32300000000000001,"5min":0.254,"15min":0.254},"max":{"1min":2.3740000000000001,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":0.53200000000000003}]},{"osd":2,"last update":"Mon Feb 5 07:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.5529999999999999,"15min":0.877},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.052,"5min":67.144999999999996,"15min":67.144999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.583,"15min":0.88700000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.9369999999999998,"5min":66.959999999999994,"15min":66.959999999999994},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Feb 5 07:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.86199999999999999,"15min":0.66100000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.7709999999999999,"5min":8.8019999999999996,"15min":8.8019999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.137,"5min":1.071,"15min":0.72899999999999998},"min":{"1min":0.29699999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":5.3449999999999998,"5min":10.106,"15min":10.106},"last":0.503}]},{"osd":5,"last update":"Mon Feb 5 07:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.45200000000000001,"15min":0.39100000000000001},"min":{"1min":0.28199999999999997,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.4930000000000001,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.432},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.5,"15min":0.41499999999999998},"min":{"1min":0.318,"5min":0.254,"15min":0.254},"max":{"1min":2.1859999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":0.443}]},{"osd":6,"last update":"Mon Feb 5 07:24:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":0.68999999999999995,"15min":0.502},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":6.8940000000000001,"5min":6.8940000000000001,"15min":6.8940000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.63500000000000001,"15min":0.47699999999999998},"min":{"1min":0.28999999999999998,"5min":0.27600000000000002,"15min":0.248},"max":{"1min":1.839,"5min":2.9209999999999998,"15min":2.9209999999999998},"last":0.38200000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.57799999999999996,"15min":0.52100000000000002},"min":{"1min":0.38200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.4089999999999998,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.52400000000000002,"15min":0.496},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.581,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":0.41299999999999998}]}]},{"osd":5,"up_from":41,"seq":176093659221,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":158452,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93586700,"statfs":{"total":95995035648,"available":95832780800,"internally_reserved":0,"allocated":958464,"data_stored":513830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:23:51 2024","interfaces":[{"interface":"back","average":{"1min":3.5550000000000002,"5min":3.2189999999999999,"15min":1.429},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":45.073,"5min":69.855999999999995,"15min":69.855999999999995},"last":0.51400000000000001},{"interface":"front","average":{"1min":3.7370000000000001,"5min":3.0249999999999999,"15min":1.365},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":47.031999999999996,"5min":71.513999999999996,"15min":71.513999999999996},"last":0.46500000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:23:51 2024","interfaces":[{"interface":"back","average":{"1min":1.7909999999999999,"5min":1.0900000000000001,"15min":0.74199999999999999},"min":{"1min":0.26100000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":11.839,"5min":11.839,"15min":11.839},"last":0.436},{"interface":"front","average":{"1min":1.2809999999999999,"5min":0.77900000000000003,"15min":0.61699999999999999},"min":{"1min":0.27400000000000002,"5min":0.253,"15min":0.253},"max":{"1min":9.5950000000000006,"5min":9.5950000000000006,"15min":9.5950000000000006},"last":0.58699999999999997}]},{"osd":2,"last update":"Mon Feb 5 07:23:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.887,"15min":0.98299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":9.5419999999999998,"5min":70.108000000000004,"15min":70.108000000000004},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.9610000000000001,"15min":1.0209999999999999},"min":{"1min":0.315,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.9340000000000002,"5min":72.817999999999998,"15min":72.817999999999998},"last":0.63100000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:23:51 2024","interfaces":[{"interface":"back","average":{"1min":3.7480000000000002,"5min":2.0859999999999999,"15min":1.0589999999999999},"min":{"1min":0.313,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":47.408000000000001,"5min":69.793000000000006,"15min":69.793000000000006},"last":0.64400000000000002},{"interface":"front","average":{"1min":3.9609999999999999,"5min":2.3250000000000002,"15min":1.143},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":45.140999999999998,"5min":69.820999999999998,"15min":69.820999999999998},"last":0.56899999999999995}]},{"osd":4,"last update":"Mon Feb 5 07:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.443,"15min":0.375},"min":{"1min":0.254,"5min":0.24099999999999999,"15min":0.23599999999999999},"max":{"1min":1.367,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.48799999999999999,"15min":0.40600000000000003},"min":{"1min":0.26200000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.518,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":0.60499999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:24:26 2024","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":1.8149999999999999,"15min":0.86699999999999999},"min":{"1min":0.309,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":76.668999999999997,"5min":76.668999999999997,"15min":76.668999999999997},"last":0.53800000000000003},{"interface":"front","average":{"1min":4.056,"5min":1.8100000000000001,"15min":0.89800000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":76.698999999999998,"5min":76.698999999999998,"15min":76.698999999999998},"last":0.68000000000000005}]},{"osd":7,"last update":"Mon Feb 5 07:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":2.3860000000000001,"15min":1.1359999999999999},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.8260000000000001,"5min":182.87799999999999,"15min":182.87799999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":2.3690000000000002,"15min":1.135},"min":{"1min":0.33600000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.3999999999999999,"5min":183.02199999999999,"15min":183.02199999999999},"last":0.62}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33292288,"data_stored":33292288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96403456,"data_stored":96403456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2949120,"data_stored":2949120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:24:44.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-02-05T07:24:44.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T07:24:44.495 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:44.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:44.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:44.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:44.621 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:44.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:44.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:44.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:44.725 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:24:44.748 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:44.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:44.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:44.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:44.873 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:44.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:44.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:44.881 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:44.975 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:44.999 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:45.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:45.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:45.126 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:45.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:45.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:45.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:45.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:45.254 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:45.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:45.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:45.378 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:45.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:45.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[29098]: 7.c scrub starts 2024-02-05T07:24:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[29098]: 7.c scrub ok 2024-02-05T07:24:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[29098]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:24:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[29098]: 7.2 scrub starts 2024-02-05T07:24:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[29098]: 7.2 scrub ok 2024-02-05T07:24:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[29098]: 7.3 scrub starts 2024-02-05T07:24:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[29098]: 7.3 scrub ok 2024-02-05T07:24:45.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:24:45.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[37158]: 7.c scrub starts 2024-02-05T07:24:45.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[37158]: 7.c scrub ok 2024-02-05T07:24:45.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[37158]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:24:45.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[37158]: 7.2 scrub starts 2024-02-05T07:24:45.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[37158]: 7.2 scrub ok 2024-02-05T07:24:45.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[37158]: 7.3 scrub starts 2024-02-05T07:24:45.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[37158]: 7.3 scrub ok 2024-02-05T07:24:45.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:45 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:24:45.480 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:45.503 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:45.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:45.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:45 smithi090 ceph-mon[33213]: 7.c scrub starts 2024-02-05T07:24:45.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:45 smithi090 ceph-mon[33213]: 7.c scrub ok 2024-02-05T07:24:45.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:45 smithi090 ceph-mon[33213]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:24:45.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:45 smithi090 ceph-mon[33213]: 7.2 scrub starts 2024-02-05T07:24:45.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:45 smithi090 ceph-mon[33213]: 7.2 scrub ok 2024-02-05T07:24:45.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:45 smithi090 ceph-mon[33213]: 7.3 scrub starts 2024-02-05T07:24:45.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:45 smithi090 ceph-mon[33213]: 7.3 scrub ok 2024-02-05T07:24:45.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:45 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:24:45.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:45.630 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:45.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:45.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:45.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:45.750 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:45.755 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:45.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:45.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:45.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:45.880 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:45.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:45.982 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:46.005 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:46.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:46.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:46.106 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:46.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:46.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:46.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:46.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:46.257 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:46.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:46.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:46.382 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:46.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:46.383 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[29098]: 7.1f scrub starts 2024-02-05T07:24:46.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[29098]: 7.1f scrub ok 2024-02-05T07:24:46.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[29098]: 7.1d scrub starts 2024-02-05T07:24:46.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[29098]: 7.1d scrub ok 2024-02-05T07:24:46.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[29098]: 7.a scrub starts 2024-02-05T07:24:46.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[29098]: 7.a scrub ok 2024-02-05T07:24:46.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[29098]: pgmap v268: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 8.8 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 545 op/s; 77/984 objects misplaced (7.825%) 2024-02-05T07:24:46.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:24:46.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:24:46.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[29098]: 3.2 scrub starts 2024-02-05T07:24:46.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[29098]: 3.2 scrub ok 2024-02-05T07:24:46.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[37158]: 7.1f scrub starts 2024-02-05T07:24:46.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[37158]: 7.1f scrub ok 2024-02-05T07:24:46.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[37158]: 7.1d scrub starts 2024-02-05T07:24:46.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[37158]: 7.1d scrub ok 2024-02-05T07:24:46.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[37158]: 7.a scrub starts 2024-02-05T07:24:46.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[37158]: 7.a scrub ok 2024-02-05T07:24:46.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[37158]: pgmap v268: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 8.8 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 545 op/s; 77/984 objects misplaced (7.825%) 2024-02-05T07:24:46.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:24:46.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:24:46.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[37158]: 3.2 scrub starts 2024-02-05T07:24:46.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:46 smithi007 ceph-mon[37158]: 3.2 scrub ok 2024-02-05T07:24:46.473 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:46 smithi090 ceph-mon[33213]: 7.1f scrub starts 2024-02-05T07:24:46.473 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:46 smithi090 ceph-mon[33213]: 7.1f scrub ok 2024-02-05T07:24:46.473 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:46 smithi090 ceph-mon[33213]: 7.1d scrub starts 2024-02-05T07:24:46.473 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:46 smithi090 ceph-mon[33213]: 7.1d scrub ok 2024-02-05T07:24:46.473 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:46 smithi090 ceph-mon[33213]: 7.a scrub starts 2024-02-05T07:24:46.474 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:46 smithi090 ceph-mon[33213]: 7.a scrub ok 2024-02-05T07:24:46.474 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:46 smithi090 ceph-mon[33213]: pgmap v268: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 8.8 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 545 op/s; 77/984 objects misplaced (7.825%) 2024-02-05T07:24:46.474 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:46 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:24:46.474 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:46 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:24:46.474 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:46 smithi090 ceph-mon[33213]: 3.2 scrub starts 2024-02-05T07:24:46.474 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:46 smithi090 ceph-mon[33213]: 3.2 scrub ok 2024-02-05T07:24:46.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:46.506 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:46.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:46.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:46.598 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:24:46.599 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:24:46.599 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:24:46.599 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:24:46.599 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:24:46.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:46.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:46.658 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:46.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:46.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:46.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:46.783 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:46.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:46.790 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:24:46.810 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:24:46.830 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:24:46.849 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:24:46.870 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:24:46.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:46.890 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:24:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:46.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:46.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:46.912 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:24:46.932 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:24:46.933 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:24:46.934 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:24:46.934 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:24:46.934 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:24:46.934 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:24:46.934 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:24:46.934 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:24:46.934 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:24:47.011 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:47.035 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:47.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:47.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:47.160 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:47.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:47.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:47.285 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:47.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:47.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[29098]: 7.23 scrub starts 2024-02-05T07:24:47.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[29098]: 7.23 scrub ok 2024-02-05T07:24:47.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[29098]: 7.1c scrub starts 2024-02-05T07:24:47.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[29098]: 7.1c scrub ok 2024-02-05T07:24:47.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[29098]: 7.20 scrub starts 2024-02-05T07:24:47.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[29098]: 7.20 scrub ok 2024-02-05T07:24:47.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[29098]: 4.1e scrub starts 2024-02-05T07:24:47.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[29098]: 4.1e scrub ok 2024-02-05T07:24:47.346 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[29098]: osdmap e193: 8 total, 8 up, 7 in 2024-02-05T07:24:47.346 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/11393038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:47.346 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[29098]: 7.0 scrub starts 2024-02-05T07:24:47.346 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[29098]: 7.0 scrub ok 2024-02-05T07:24:47.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[37158]: 7.23 scrub starts 2024-02-05T07:24:47.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[37158]: 7.23 scrub ok 2024-02-05T07:24:47.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[37158]: 7.1c scrub starts 2024-02-05T07:24:47.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[37158]: 7.1c scrub ok 2024-02-05T07:24:47.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[37158]: 7.20 scrub starts 2024-02-05T07:24:47.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[37158]: 7.20 scrub ok 2024-02-05T07:24:47.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[37158]: 4.1e scrub starts 2024-02-05T07:24:47.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[37158]: 4.1e scrub ok 2024-02-05T07:24:47.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[37158]: osdmap e193: 8 total, 8 up, 7 in 2024-02-05T07:24:47.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/11393038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:47.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[37158]: 7.0 scrub starts 2024-02-05T07:24:47.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:47 smithi007 ceph-mon[37158]: 7.0 scrub ok 2024-02-05T07:24:47.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:47.390 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T07:24:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:47.412 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:47.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:47.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:47.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:47 smithi090 ceph-mon[33213]: 7.23 scrub starts 2024-02-05T07:24:47.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:47 smithi090 ceph-mon[33213]: 7.23 scrub ok 2024-02-05T07:24:47.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:47 smithi090 ceph-mon[33213]: 7.1c scrub starts 2024-02-05T07:24:47.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:47 smithi090 ceph-mon[33213]: 7.1c scrub ok 2024-02-05T07:24:47.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:47 smithi090 ceph-mon[33213]: 7.20 scrub starts 2024-02-05T07:24:47.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:47 smithi090 ceph-mon[33213]: 7.20 scrub ok 2024-02-05T07:24:47.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:47 smithi090 ceph-mon[33213]: 4.1e scrub starts 2024-02-05T07:24:47.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:47 smithi090 ceph-mon[33213]: 4.1e scrub ok 2024-02-05T07:24:47.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:47 smithi090 ceph-mon[33213]: osdmap e193: 8 total, 8 up, 7 in 2024-02-05T07:24:47.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:47 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/11393038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:47.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:47 smithi090 ceph-mon[33213]: 7.0 scrub starts 2024-02-05T07:24:47.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:47 smithi090 ceph-mon[33213]: 7.0 scrub ok 2024-02-05T07:24:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:47.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:47.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:47.652 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:24:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:47.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:47.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:47.793 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:47.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:47.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:47.918 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:47.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:48.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:48.045 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:48.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:48.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:48.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:48.229 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:24:48.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:48.296 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:48.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:48.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:48.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[29098]: 6.13 scrub starts 2024-02-05T07:24:48.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[29098]: 6.13 scrub ok 2024-02-05T07:24:48.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[29098]: 4.1d scrub starts 2024-02-05T07:24:48.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[29098]: 4.1d scrub ok 2024-02-05T07:24:48.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[29098]: 7.f scrub starts 2024-02-05T07:24:48.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[29098]: 7.f scrub ok 2024-02-05T07:24:48.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[29098]: pgmap v270: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 77/786 objects misplaced (9.796%) 2024-02-05T07:24:48.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2796625803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:24:48.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:24:48.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[37158]: 6.13 scrub starts 2024-02-05T07:24:48.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[37158]: 6.13 scrub ok 2024-02-05T07:24:48.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[37158]: 4.1d scrub starts 2024-02-05T07:24:48.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[37158]: 4.1d scrub ok 2024-02-05T07:24:48.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[37158]: 7.f scrub starts 2024-02-05T07:24:48.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[37158]: 7.f scrub ok 2024-02-05T07:24:48.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[37158]: pgmap v270: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 77/786 objects misplaced (9.796%) 2024-02-05T07:24:48.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2796625803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:24:48.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:48 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:24:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:48.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:48.464 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:48 smithi090 ceph-mon[33213]: 6.13 scrub starts 2024-02-05T07:24:48.464 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:48 smithi090 ceph-mon[33213]: 6.13 scrub ok 2024-02-05T07:24:48.464 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:48 smithi090 ceph-mon[33213]: 4.1d scrub starts 2024-02-05T07:24:48.464 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:48 smithi090 ceph-mon[33213]: 4.1d scrub ok 2024-02-05T07:24:48.464 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:48 smithi090 ceph-mon[33213]: 7.f scrub starts 2024-02-05T07:24:48.464 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:48 smithi090 ceph-mon[33213]: 7.f scrub ok 2024-02-05T07:24:48.465 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:48 smithi090 ceph-mon[33213]: pgmap v270: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 77/786 objects misplaced (9.796%) 2024-02-05T07:24:48.465 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:48 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2796625803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:24:48.465 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:48 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:24:48.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:48.547 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:48.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:48.649 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:48.672 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:48.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:48.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:48.774 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:48.798 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:48.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:48.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:48.924 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:48.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:49.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:49.051 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:49.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:49.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:49.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:49.176 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:49.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:49.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:49.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:49.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:49.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:49.352 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:24:49.353 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:24:49.353 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:49 smithi090 ceph-mon[33213]: 5.17 scrub starts 2024-02-05T07:24:49.353 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:49 smithi090 ceph-mon[33213]: 5.17 scrub ok 2024-02-05T07:24:49.353 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:49 smithi090 ceph-mon[33213]: 6.8 scrub starts 2024-02-05T07:24:49.353 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:49 smithi090 ceph-mon[33213]: 6.8 scrub ok 2024-02-05T07:24:49.353 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:49 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:24:49.353 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:49 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:24:49.353 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:49 smithi090 ceph-mon[33213]: osdmap e194: 8 total, 8 up, 7 in 2024-02-05T07:24:49.353 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:49 smithi090 ceph-mon[33213]: 3.1 deep-scrub starts 2024-02-05T07:24:49.354 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:49 smithi090 ceph-mon[33213]: 3.1 deep-scrub ok 2024-02-05T07:24:49.404 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[29098]: 5.17 scrub starts 2024-02-05T07:24:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[29098]: 5.17 scrub ok 2024-02-05T07:24:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[29098]: 6.8 scrub starts 2024-02-05T07:24:49.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[29098]: 6.8 scrub ok 2024-02-05T07:24:49.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:24:49.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:24:49.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[29098]: osdmap e194: 8 total, 8 up, 7 in 2024-02-05T07:24:49.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[29098]: 3.1 deep-scrub starts 2024-02-05T07:24:49.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[29098]: 3.1 deep-scrub ok 2024-02-05T07:24:49.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[37158]: 5.17 scrub starts 2024-02-05T07:24:49.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[37158]: 5.17 scrub ok 2024-02-05T07:24:49.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[37158]: 6.8 scrub starts 2024-02-05T07:24:49.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[37158]: 6.8 scrub ok 2024-02-05T07:24:49.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:24:49.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:24:49.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[37158]: osdmap e194: 8 total, 8 up, 7 in 2024-02-05T07:24:49.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[37158]: 3.1 deep-scrub starts 2024-02-05T07:24:49.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:49 smithi007 ceph-mon[37158]: 3.1 deep-scrub ok 2024-02-05T07:24:49.429 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:49.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:49.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:49.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:49.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:49.556 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:49.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:49.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:49.682 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:49.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:49.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:49.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:49.809 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:49.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:49.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:49.934 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:49.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:49.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:49.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T07:24:50.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:50.059 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:50.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:50.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:50.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:50.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:50.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:50.221 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:50.234 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pgp_num to 1 2024-02-05T07:24:50.286 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:50.310 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:50.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:50.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:50.412 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:50.436 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:50.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:50.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:24:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:24:50.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:50 smithi090 ceph-mon[33213]: pgmap v272: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 77/786 objects misplaced (9.796%) 2024-02-05T07:24:50.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:50 smithi090 ceph-mon[33213]: 3.12 scrub starts 2024-02-05T07:24:50.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:50 smithi090 ceph-mon[33213]: 3.12 scrub ok 2024-02-05T07:24:50.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:50 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2531747022' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-05T07:24:50.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:50 smithi090 ceph-mon[33213]: 6.18 scrub starts 2024-02-05T07:24:50.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:50 smithi090 ceph-mon[33213]: 6.18 scrub ok 2024-02-05T07:24:50.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:50 smithi090 ceph-mon[33213]: 5.f scrub starts 2024-02-05T07:24:50.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:50 smithi090 ceph-mon[33213]: 5.f scrub ok 2024-02-05T07:24:50.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:50 smithi090 ceph-mon[33213]: 4.3 scrub starts 2024-02-05T07:24:50.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:50 smithi090 ceph-mon[33213]: 4.3 scrub ok 2024-02-05T07:24:50.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:50.562 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:50.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:50.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[29098]: pgmap v272: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 77/786 objects misplaced (9.796%) 2024-02-05T07:24:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[29098]: 3.12 scrub starts 2024-02-05T07:24:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[29098]: 3.12 scrub ok 2024-02-05T07:24:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2531747022' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-05T07:24:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[29098]: 6.18 scrub starts 2024-02-05T07:24:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[29098]: 6.18 scrub ok 2024-02-05T07:24:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[29098]: 5.f scrub starts 2024-02-05T07:24:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[29098]: 5.f scrub ok 2024-02-05T07:24:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[29098]: 4.3 scrub starts 2024-02-05T07:24:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[29098]: 4.3 scrub ok 2024-02-05T07:24:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[37158]: pgmap v272: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 77/786 objects misplaced (9.796%) 2024-02-05T07:24:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[37158]: 3.12 scrub starts 2024-02-05T07:24:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[37158]: 3.12 scrub ok 2024-02-05T07:24:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2531747022' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-05T07:24:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[37158]: 6.18 scrub starts 2024-02-05T07:24:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[37158]: 6.18 scrub ok 2024-02-05T07:24:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[37158]: 5.f scrub starts 2024-02-05T07:24:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[37158]: 5.f scrub ok 2024-02-05T07:24:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[37158]: 4.3 scrub starts 2024-02-05T07:24:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:50 smithi007 ceph-mon[37158]: 4.3 scrub ok 2024-02-05T07:24:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:50.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:50.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:50.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:50.817 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:50.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:50.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:50.941 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:50.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:50.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:51.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:51.069 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:51.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:51.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:51.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:51.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:51.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[29098]: 3.10 scrub starts 2024-02-05T07:24:51.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[29098]: 3.10 scrub ok 2024-02-05T07:24:51.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:51.322 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:51.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:51.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:51.451 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:51.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:51.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:51 smithi090 ceph-mon[33213]: 3.10 scrub starts 2024-02-05T07:24:51.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:51 smithi090 ceph-mon[33213]: 3.10 scrub ok 2024-02-05T07:24:51.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:51 smithi090 ceph-mon[33213]: 2.0 scrub starts 2024-02-05T07:24:51.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:51 smithi090 ceph-mon[33213]: 2.0 scrub ok 2024-02-05T07:24:51.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:51 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2531747022' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-02-05T07:24:51.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:51 smithi090 ceph-mon[33213]: osdmap e195: 8 total, 8 up, 7 in 2024-02-05T07:24:51.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:51 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:51.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:51 smithi090 ceph-mon[33213]: 5.7 scrub starts 2024-02-05T07:24:51.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:51 smithi090 ceph-mon[33213]: 5.7 scrub ok 2024-02-05T07:24:51.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:51 smithi090 ceph-mon[33213]: 7.1 scrub starts 2024-02-05T07:24:51.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:51 smithi090 ceph-mon[33213]: 7.1 scrub ok 2024-02-05T07:24:51.553 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:51.577 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:51.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:51.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:51.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:24:51.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[29098]: 2.0 scrub starts 2024-02-05T07:24:51.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[29098]: 2.0 scrub ok 2024-02-05T07:24:51.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2531747022' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-02-05T07:24:51.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[29098]: osdmap e195: 8 total, 8 up, 7 in 2024-02-05T07:24:51.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:51.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[29098]: 5.7 scrub starts 2024-02-05T07:24:51.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[29098]: 5.7 scrub ok 2024-02-05T07:24:51.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[29098]: 7.1 scrub starts 2024-02-05T07:24:51.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[29098]: 7.1 scrub ok 2024-02-05T07:24:51.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[37158]: 3.10 scrub starts 2024-02-05T07:24:51.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[37158]: 3.10 scrub ok 2024-02-05T07:24:51.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[37158]: 2.0 scrub starts 2024-02-05T07:24:51.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[37158]: 2.0 scrub ok 2024-02-05T07:24:51.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2531747022' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-02-05T07:24:51.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[37158]: osdmap e195: 8 total, 8 up, 7 in 2024-02-05T07:24:51.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:24:51.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[37158]: 5.7 scrub starts 2024-02-05T07:24:51.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[37158]: 5.7 scrub ok 2024-02-05T07:24:51.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[37158]: 7.1 scrub starts 2024-02-05T07:24:51.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:51 smithi007 ceph-mon[37158]: 7.1 scrub ok 2024-02-05T07:24:51.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:51.703 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:51.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:51.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:51.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:51.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:51.830 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:51.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:51.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:51.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:24:51.955 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:51.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:52.020 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-02-05T07:24:52.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-02-05T07:24:52.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:52.087 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:52.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:52.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:52.213 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:52.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:52.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:52.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:52.338 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:52.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:52.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:52.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:52.465 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:52.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:52.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:52.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:52 smithi090 ceph-mon[33213]: pgmap v274: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.7 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 77/696 objects misplaced (11.063%) 2024-02-05T07:24:52.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:52 smithi090 ceph-mon[33213]: osdmap e196: 8 total, 8 up, 7 in 2024-02-05T07:24:52.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:52 smithi090 ceph-mon[33213]: 5.18 scrub starts 2024-02-05T07:24:52.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:52 smithi090 ceph-mon[33213]: 5.18 scrub ok 2024-02-05T07:24:52.528 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:52.542 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:52.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:52.591 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:52.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:52.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:52.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:52 smithi007 ceph-mon[29098]: pgmap v274: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.7 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 77/696 objects misplaced (11.063%) 2024-02-05T07:24:52.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:52 smithi007 ceph-mon[29098]: osdmap e196: 8 total, 8 up, 7 in 2024-02-05T07:24:52.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:52 smithi007 ceph-mon[29098]: 5.18 scrub starts 2024-02-05T07:24:52.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:52 smithi007 ceph-mon[29098]: 5.18 scrub ok 2024-02-05T07:24:52.634 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:52 smithi007 ceph-mon[37158]: pgmap v274: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.7 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 77/696 objects misplaced (11.063%) 2024-02-05T07:24:52.634 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:52 smithi007 ceph-mon[37158]: osdmap e196: 8 total, 8 up, 7 in 2024-02-05T07:24:52.634 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:52 smithi007 ceph-mon[37158]: 5.18 scrub starts 2024-02-05T07:24:52.634 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:52 smithi007 ceph-mon[37158]: 5.18 scrub ok 2024-02-05T07:24:52.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:52.717 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:52.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:52.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:52.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:52.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:52.844 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:52.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:52.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:52.969 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:52.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:52.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:53.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:53.094 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:53.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:53.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:53.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:53.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:53.221 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:53.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:53.271 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:24:53.271 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:24:53.322 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:53.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:53.347 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:53.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:53.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:53.472 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:53.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:53.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:53.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:53 smithi007 ceph-mon[37158]: osdmap e197: 8 total, 8 up, 7 in 2024-02-05T07:24:53.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:53 smithi007 ceph-mon[37158]: 6.d scrub starts 2024-02-05T07:24:53.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:53 smithi007 ceph-mon[37158]: 6.d scrub ok 2024-02-05T07:24:53.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:53 smithi007 ceph-mon[37158]: 7.21 deep-scrub starts 2024-02-05T07:24:53.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:53 smithi007 ceph-mon[37158]: 7.21 deep-scrub ok 2024-02-05T07:24:53.498 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:53 smithi007 ceph-mon[29098]: osdmap e197: 8 total, 8 up, 7 in 2024-02-05T07:24:53.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:53 smithi007 ceph-mon[29098]: 6.d scrub starts 2024-02-05T07:24:53.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:53 smithi007 ceph-mon[29098]: 6.d scrub ok 2024-02-05T07:24:53.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:53 smithi007 ceph-mon[29098]: 7.21 deep-scrub starts 2024-02-05T07:24:53.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:53 smithi007 ceph-mon[29098]: 7.21 deep-scrub ok 2024-02-05T07:24:53.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:53 smithi090 ceph-mon[33213]: osdmap e197: 8 total, 8 up, 7 in 2024-02-05T07:24:53.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:53 smithi090 ceph-mon[33213]: 6.d scrub starts 2024-02-05T07:24:53.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:53 smithi090 ceph-mon[33213]: 6.d scrub ok 2024-02-05T07:24:53.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:53 smithi090 ceph-mon[33213]: 7.21 deep-scrub starts 2024-02-05T07:24:53.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:53 smithi090 ceph-mon[33213]: 7.21 deep-scrub ok 2024-02-05T07:24:53.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:53.599 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:53.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:53.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:53.724 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:53.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:53.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:53.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:53.841 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:24:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:24:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:24:53.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:53.852 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:53.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:53.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:53.980 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:53.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:54.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:54.109 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:54.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:54.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:54.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:54.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:54.364 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:54.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:54.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:54.491 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:54.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:54.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:54.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:54 smithi090 ceph-mon[33213]: pgmap v277: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.7 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 77/696 objects misplaced (11.063%) 2024-02-05T07:24:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:54 smithi090 ceph-mon[33213]: 3.13 scrub starts 2024-02-05T07:24:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:54 smithi090 ceph-mon[33213]: 3.13 scrub ok 2024-02-05T07:24:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:54 smithi090 ceph-mon[33213]: osdmap e198: 8 total, 8 up, 7 in 2024-02-05T07:24:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:54 smithi090 ceph-mon[33213]: 4.1b scrub starts 2024-02-05T07:24:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:54 smithi090 ceph-mon[33213]: 4.1b scrub ok 2024-02-05T07:24:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:54 smithi090 ceph-mon[33213]: 7.1e deep-scrub starts 2024-02-05T07:24:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:54 smithi090 ceph-mon[33213]: 7.1e deep-scrub ok 2024-02-05T07:24:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:54 smithi090 ceph-mon[33213]: osdmap e199: 8 total, 8 up, 7 in 2024-02-05T07:24:54.575 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[29098]: pgmap v277: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.7 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 77/696 objects misplaced (11.063%) 2024-02-05T07:24:54.575 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[29098]: 3.13 scrub starts 2024-02-05T07:24:54.575 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[29098]: 3.13 scrub ok 2024-02-05T07:24:54.575 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[29098]: osdmap e198: 8 total, 8 up, 7 in 2024-02-05T07:24:54.576 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[29098]: 4.1b scrub starts 2024-02-05T07:24:54.576 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[29098]: 4.1b scrub ok 2024-02-05T07:24:54.576 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[29098]: 7.1e deep-scrub starts 2024-02-05T07:24:54.576 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[29098]: 7.1e deep-scrub ok 2024-02-05T07:24:54.576 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[29098]: osdmap e199: 8 total, 8 up, 7 in 2024-02-05T07:24:54.576 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[37158]: pgmap v277: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.7 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 77/696 objects misplaced (11.063%) 2024-02-05T07:24:54.576 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[37158]: 3.13 scrub starts 2024-02-05T07:24:54.576 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[37158]: 3.13 scrub ok 2024-02-05T07:24:54.576 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[37158]: osdmap e198: 8 total, 8 up, 7 in 2024-02-05T07:24:54.576 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[37158]: 4.1b scrub starts 2024-02-05T07:24:54.576 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[37158]: 4.1b scrub ok 2024-02-05T07:24:54.577 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[37158]: 7.1e deep-scrub starts 2024-02-05T07:24:54.577 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[37158]: 7.1e deep-scrub ok 2024-02-05T07:24:54.577 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:54 smithi007 ceph-mon[37158]: osdmap e199: 8 total, 8 up, 7 in 2024-02-05T07:24:54.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:54.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:54.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:54.743 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:54.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:54.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:54.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:54.868 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:54.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:54.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:54.995 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:54.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:54.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:55.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:55.123 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:55.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:55.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T07:24:55.224 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:55.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:55.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[29098]: 4.1f scrub starts 2024-02-05T07:24:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[29098]: 4.1f scrub ok 2024-02-05T07:24:55.249 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[37158]: 4.1f scrub starts 2024-02-05T07:24:55.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:55.376 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:55.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:55.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:55.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:55.502 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:55.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:55.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:55 smithi090 ceph-mon[33213]: 4.1f scrub starts 2024-02-05T07:24:55.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:55 smithi090 ceph-mon[33213]: 4.1f scrub ok 2024-02-05T07:24:55.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:55 smithi090 ceph-mon[33213]: 3.18 scrub starts 2024-02-05T07:24:55.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:55 smithi090 ceph-mon[33213]: 3.18 scrub ok 2024-02-05T07:24:55.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:55 smithi090 ceph-mon[33213]: 7.21 scrub starts 2024-02-05T07:24:55.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:55 smithi090 ceph-mon[33213]: 7.21 scrub ok 2024-02-05T07:24:55.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:55 smithi090 ceph-mon[33213]: 4.d scrub starts 2024-02-05T07:24:55.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:55 smithi090 ceph-mon[33213]: 4.d scrub ok 2024-02-05T07:24:55.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:55 smithi090 ceph-mon[33213]: osdmap e200: 8 total, 8 up, 7 in 2024-02-05T07:24:55.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[29098]: 3.18 scrub starts 2024-02-05T07:24:55.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[29098]: 3.18 scrub ok 2024-02-05T07:24:55.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[29098]: 7.21 scrub starts 2024-02-05T07:24:55.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[29098]: 7.21 scrub ok 2024-02-05T07:24:55.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[29098]: 4.d scrub starts 2024-02-05T07:24:55.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[29098]: 4.d scrub ok 2024-02-05T07:24:55.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[29098]: osdmap e200: 8 total, 8 up, 7 in 2024-02-05T07:24:55.578 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[37158]: 4.1f scrub ok 2024-02-05T07:24:55.578 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[37158]: 3.18 scrub starts 2024-02-05T07:24:55.578 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[37158]: 3.18 scrub ok 2024-02-05T07:24:55.579 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[37158]: 7.21 scrub starts 2024-02-05T07:24:55.579 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[37158]: 7.21 scrub ok 2024-02-05T07:24:55.579 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[37158]: 4.d scrub starts 2024-02-05T07:24:55.579 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[37158]: 4.d scrub ok 2024-02-05T07:24:55.579 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:55 smithi007 ceph-mon[37158]: osdmap e200: 8 total, 8 up, 7 in 2024-02-05T07:24:55.584 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:24:55.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:55.604 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:24:55.623 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:55.626 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:24:55.627 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:55.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:55.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:55.654 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:24:55.674 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:24:55.696 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:24:55.718 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:24:55.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:55.737 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:24:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:24:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:24:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:24:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:24:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:24:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:24:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:24:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:24:55.752 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:55.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:55.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:55.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:55.895 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:55.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:55.996 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:56.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:56.022 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:56.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:56.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:56.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:56.149 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:56.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:56.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:56.284 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:56.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:56.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:56.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:56.414 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:56.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:56.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:56.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:56 smithi090 ceph-mon[33213]: 4.a scrub starts 2024-02-05T07:24:56.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:56 smithi090 ceph-mon[33213]: 4.a scrub ok 2024-02-05T07:24:56.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:56 smithi090 ceph-mon[33213]: 6.f scrub starts 2024-02-05T07:24:56.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:56 smithi090 ceph-mon[33213]: 6.f scrub ok 2024-02-05T07:24:56.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:56 smithi090 ceph-mon[33213]: pgmap v280: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.7 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 77/696 objects misplaced (11.063%) 2024-02-05T07:24:56.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:56 smithi090 ceph-mon[33213]: 7.1e scrub starts 2024-02-05T07:24:56.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:56 smithi090 ceph-mon[33213]: 7.1e scrub ok 2024-02-05T07:24:56.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:56 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:24:56.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:56 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/26429972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:56.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:56 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:56.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:56 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:56.540 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:56.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:56.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:56.665 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:56.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:56.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:56.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[29098]: 4.a scrub starts 2024-02-05T07:24:56.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[29098]: 4.a scrub ok 2024-02-05T07:24:56.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[29098]: 6.f scrub starts 2024-02-05T07:24:56.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[29098]: 6.f scrub ok 2024-02-05T07:24:56.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[29098]: pgmap v280: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.7 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 77/696 objects misplaced (11.063%) 2024-02-05T07:24:56.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[29098]: 7.1e scrub starts 2024-02-05T07:24:56.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[29098]: 7.1e scrub ok 2024-02-05T07:24:56.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:24:56.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/26429972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:56.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:56.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:56.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[37158]: 4.a scrub starts 2024-02-05T07:24:56.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[37158]: 4.a scrub ok 2024-02-05T07:24:56.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[37158]: 6.f scrub starts 2024-02-05T07:24:56.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[37158]: 6.f scrub ok 2024-02-05T07:24:56.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[37158]: pgmap v280: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.7 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 77/696 objects misplaced (11.063%) 2024-02-05T07:24:56.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[37158]: 7.1e scrub starts 2024-02-05T07:24:56.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[37158]: 7.1e scrub ok 2024-02-05T07:24:56.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:24:56.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/26429972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:24:56.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:56.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:56 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:56.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:56.791 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:56.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:56.893 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:56.916 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:56.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:56.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:24:57.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:57.021 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T07:24:57.021 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:24:57.021 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-05T07:24:57.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T07:24:57.043 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:57.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:57.147 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:57.171 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:57.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:57.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:57.272 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:57.296 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:57.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:57.398 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:57.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:57.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:57.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:57 smithi090 ceph-mon[33213]: 6.11 scrub starts 2024-02-05T07:24:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:57 smithi090 ceph-mon[33213]: 6.11 scrub ok 2024-02-05T07:24:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:57 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:57.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:57 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:57.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:57.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:57.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:57 smithi007 ceph-mon[29098]: 6.11 scrub starts 2024-02-05T07:24:57.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:57 smithi007 ceph-mon[29098]: 6.11 scrub ok 2024-02-05T07:24:57.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:57 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:57.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:57 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:57.550 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:57 smithi007 ceph-mon[37158]: 6.11 scrub starts 2024-02-05T07:24:57.550 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:57 smithi007 ceph-mon[37158]: 6.11 scrub ok 2024-02-05T07:24:57.550 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:57 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:57.550 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:57 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:57.649 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:57.672 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:57.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:57.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:57.799 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:57.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:57.899 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:57.923 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:57.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:57.988 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:24:58.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:58.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:58.049 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:58.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:58.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:58.173 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:58.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:58.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:58.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:58.301 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:58.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[29098]: 3.1f scrub starts 2024-02-05T07:24:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[29098]: 3.1f scrub ok 2024-02-05T07:24:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[29098]: pgmap v282: 168 pgs: 1 peering, 167 active+clean; 2.1 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:24:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:24:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:24:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[29098]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-02-05T07:24:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:24:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[29098]: 3.5 scrub starts 2024-02-05T07:24:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[29098]: 3.5 scrub ok 2024-02-05T07:24:58.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[37158]: 3.1f scrub starts 2024-02-05T07:24:58.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[37158]: 3.1f scrub ok 2024-02-05T07:24:58.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[37158]: pgmap v282: 168 pgs: 1 peering, 167 active+clean; 2.1 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:24:58.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:24:58.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:24:58.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:58.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[37158]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-02-05T07:24:58.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:24:58.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[37158]: 3.5 scrub starts 2024-02-05T07:24:58.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:58 smithi007 ceph-mon[37158]: 3.5 scrub ok 2024-02-05T07:24:58.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:58.427 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:58.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:58.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:58 smithi090 ceph-mon[33213]: 3.1f scrub starts 2024-02-05T07:24:58.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:58 smithi090 ceph-mon[33213]: 3.1f scrub ok 2024-02-05T07:24:58.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:58 smithi090 ceph-mon[33213]: pgmap v282: 168 pgs: 1 peering, 167 active+clean; 2.1 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:24:58.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:58 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:24:58.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:58 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:24:58.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:58 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:24:58.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:58 smithi090 ceph-mon[33213]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-02-05T07:24:58.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:58 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:24:58.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:58 smithi090 ceph-mon[33213]: 3.5 scrub starts 2024-02-05T07:24:58.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:58 smithi090 ceph-mon[33213]: 3.5 scrub ok 2024-02-05T07:24:58.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:58.551 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:58.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:58.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:58.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:58.676 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:58.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:58.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:58.803 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:58.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:58.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:58.905 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:24:58.928 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:58.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:58.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:24:59.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:24:59.053 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:59.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:59.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:24:59.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:24:59.179 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:59.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:24:59.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:24:59.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:24:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:59.306 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:59.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:24:59.406 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[29098]: 5.8 scrub starts 2024-02-05T07:24:59.406 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[29098]: 5.8 scrub ok 2024-02-05T07:24:59.406 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:24:59.407 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:24:59.407 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[29098]: osdmap e201: 8 total, 8 up, 7 in 2024-02-05T07:24:59.407 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[29098]: 6.7 scrub starts 2024-02-05T07:24:59.407 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[29098]: 6.7 scrub ok 2024-02-05T07:24:59.407 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[29098]: osdmap e202: 8 total, 8 up, 7 in 2024-02-05T07:24:59.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:24:59.408 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[37158]: 5.8 scrub starts 2024-02-05T07:24:59.408 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[37158]: 5.8 scrub ok 2024-02-05T07:24:59.408 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:24:59.408 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:24:59.408 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[37158]: osdmap e201: 8 total, 8 up, 7 in 2024-02-05T07:24:59.408 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[37158]: 6.7 scrub starts 2024-02-05T07:24:59.408 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[37158]: 6.7 scrub ok 2024-02-05T07:24:59.408 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:24:59 smithi007 ceph-mon[37158]: osdmap e202: 8 total, 8 up, 7 in 2024-02-05T07:24:59.433 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:24:59.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:59.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:24:59.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:59 smithi090 ceph-mon[33213]: 5.8 scrub starts 2024-02-05T07:24:59.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:59 smithi090 ceph-mon[33213]: 5.8 scrub ok 2024-02-05T07:24:59.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:59 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:24:59.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:59 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:24:59.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:59 smithi090 ceph-mon[33213]: osdmap e201: 8 total, 8 up, 7 in 2024-02-05T07:24:59.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:59 smithi090 ceph-mon[33213]: 6.7 scrub starts 2024-02-05T07:24:59.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:59 smithi090 ceph-mon[33213]: 6.7 scrub ok 2024-02-05T07:24:59.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:24:59 smithi090 ceph-mon[33213]: osdmap e202: 8 total, 8 up, 7 in 2024-02-05T07:24:59.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:59.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:59.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:24:59.685 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:59.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:59.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:24:59.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:24:59.810 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:59.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:24:59.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:24:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:24:59.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:24:59.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:00.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:00.061 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:00.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:00.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:00.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:00.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:00.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:00.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:25:00.291 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:00.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:00.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:00.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:00.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:00.441 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:00.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:00.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:25:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:25:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:00 smithi090 ceph-mon[33213]: 6.1f scrub starts 2024-02-05T07:25:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:00 smithi090 ceph-mon[33213]: 6.1f scrub ok 2024-02-05T07:25:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:00 smithi090 ceph-mon[33213]: pgmap v284: 168 pgs: 1 peering, 167 active+clean; 2.1 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 885 B/s rd, 0 op/s 2024-02-05T07:25:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:00 smithi090 ceph-mon[33213]: 6.3 scrub starts 2024-02-05T07:25:00.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:00 smithi090 ceph-mon[33213]: 6.3 scrub ok 2024-02-05T07:25:00.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:00 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2374119645' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:25:00.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:00 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:25:00.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:00 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:00.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:00 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:25:00.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:00 smithi090 ceph-mon[33213]: 4.1 deep-scrub starts 2024-02-05T07:25:00.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:00 smithi090 ceph-mon[33213]: 4.1 deep-scrub ok 2024-02-05T07:25:00.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:00.566 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:00.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:00.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[29098]: 6.1f scrub starts 2024-02-05T07:25:00.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[29098]: 6.1f scrub ok 2024-02-05T07:25:00.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[29098]: pgmap v284: 168 pgs: 1 peering, 167 active+clean; 2.1 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 885 B/s rd, 0 op/s 2024-02-05T07:25:00.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[29098]: 6.3 scrub starts 2024-02-05T07:25:00.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[29098]: 6.3 scrub ok 2024-02-05T07:25:00.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2374119645' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:25:00.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:25:00.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:00.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:25:00.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[29098]: 4.1 deep-scrub starts 2024-02-05T07:25:00.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[29098]: 4.1 deep-scrub ok 2024-02-05T07:25:00.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[37158]: 6.1f scrub starts 2024-02-05T07:25:00.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[37158]: 6.1f scrub ok 2024-02-05T07:25:00.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[37158]: pgmap v284: 168 pgs: 1 peering, 167 active+clean; 2.1 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 885 B/s rd, 0 op/s 2024-02-05T07:25:00.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[37158]: 6.3 scrub starts 2024-02-05T07:25:00.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[37158]: 6.3 scrub ok 2024-02-05T07:25:00.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2374119645' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:25:00.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:25:00.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:00.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:25:00.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[37158]: 4.1 deep-scrub starts 2024-02-05T07:25:00.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:00 smithi007 ceph-mon[37158]: 4.1 deep-scrub ok 2024-02-05T07:25:00.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:00.693 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:00.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:00.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:00.705 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:00.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:00.820 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:00.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:00.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:00.946 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:00.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:00.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:01.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:01.071 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:01.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:01.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:01.173 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:01.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:01.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:01.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:01.322 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:01.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:01.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:01.346 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[29098]: 5.3 scrub starts 2024-02-05T07:25:01.346 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[29098]: 5.3 scrub ok 2024-02-05T07:25:01.346 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:01.346 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:25:01.346 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:25:01.346 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[29098]: osdmap e203: 8 total, 8 up, 7 in 2024-02-05T07:25:01.347 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:25:01.347 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:25:01.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[37158]: 5.3 scrub starts 2024-02-05T07:25:01.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[37158]: 5.3 scrub ok 2024-02-05T07:25:01.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:01.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:25:01.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:25:01.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[37158]: osdmap e203: 8 total, 8 up, 7 in 2024-02-05T07:25:01.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:25:01.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:01 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:25:01.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:01.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:01.448 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:01.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:01.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:01 smithi090 ceph-mon[33213]: 5.3 scrub starts 2024-02-05T07:25:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:01 smithi090 ceph-mon[33213]: 5.3 scrub ok 2024-02-05T07:25:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:01 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:25:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:01 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:25:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:01 smithi090 ceph-mon[33213]: osdmap e203: 8 total, 8 up, 7 in 2024-02-05T07:25:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:01 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:25:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:01 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:25:01.548 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:25:01.549 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:25:01.549 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-02-05T07:25:01.549 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-02-05T07:25:01.549 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:25:01.549 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:01.574 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:01.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:01.676 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:01.700 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:01.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:01.802 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:01.825 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:01.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:01.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:01.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:01.950 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:01.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:02.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:02.077 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:02.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:02.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:02.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:02.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:02.203 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:02.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:02.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:02.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:02.328 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:02.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:02.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:02.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:02.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:02.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:02 smithi090 ceph-mon[33213]: pgmap v287: 168 pgs: 168 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 878 B/s rd, 0 op/s 2024-02-05T07:25:02.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:02 smithi090 ceph-mon[33213]: osdmap e204: 8 total, 8 up, 7 in 2024-02-05T07:25:02.554 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:02.569 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:02.578 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:02.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:02.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:02.655 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:02 smithi007 ceph-mon[29098]: pgmap v287: 168 pgs: 168 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 878 B/s rd, 0 op/s 2024-02-05T07:25:02.655 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:02 smithi007 ceph-mon[29098]: osdmap e204: 8 total, 8 up, 7 in 2024-02-05T07:25:02.655 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:02 smithi007 ceph-mon[37158]: pgmap v287: 168 pgs: 168 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 878 B/s rd, 0 op/s 2024-02-05T07:25:02.655 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:02 smithi007 ceph-mon[37158]: osdmap e204: 8 total, 8 up, 7 in 2024-02-05T07:25:02.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:02.703 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:02.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:02.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:02.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:02.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:02.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:02.955 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:02.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:02.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:03.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:03.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:03.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:03.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:25:03.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T07:25:03.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:03.217 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:03.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:03.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:03.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:03.342 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:03.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:03.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:03.470 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:03.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:03.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:03.596 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:03.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:03.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:03.674 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:25:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:25:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:25:03.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:03.724 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:03.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:03.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:03.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:03.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:03.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:03.953 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:03.977 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:03.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:04.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:04.105 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:04.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:04.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:04.207 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:25:04.212 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:25:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 204, 2024-02-05T07:25:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "e3e7059e-c3f5-11ee-95b6-87774f69a715", 2024-02-05T07:25:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "created": "2024-02-05T07:13:14.603885+0000", 2024-02-05T07:25:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "modified": "2024-02-05T07:25:01.256363+0000", 2024-02-05T07:25:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "last_up_change": "2024-02-05T07:18:35.434647+0000", 2024-02-05T07:25:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "last_in_change": "2024-02-05T07:23:46.563809+0000", 2024-02-05T07:25:04.212 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T07:25:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_num": 5801984, 2024-02-05T07:25:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_set": [ 2024-02-05T07:25:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "noscrub", 2024-02-05T07:25:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "pglog_hardlimit", 2024-02-05T07:25:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "purged_snapdirs", 2024-02-05T07:25:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_deletes", 2024-02-05T07:25:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "sortbitwise" 2024-02-05T07:25:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:25:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_version": 18, 2024-02-05T07:25:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T07:25:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T07:25:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T07:25:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_snapshot": "", 2024-02-05T07:25:04.213 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_max": 7, 2024-02-05T07:25:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd": 8, 2024-02-05T07:25:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "require_min_compat_client": "luminous", 2024-02-05T07:25:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "min_compat_client": "luminous", 2024-02-05T07:25:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "require_osd_release": "squid", 2024-02-05T07:25:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_crimson": false, 2024-02-05T07:25:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "pools": [ 2024-02-05T07:25:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 1, 2024-02-05T07:25:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".mgr", 2024-02-05T07:25:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:16:32.505567+0000", 2024-02-05T07:25:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:25:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:25:04.214 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:25:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:25:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:25:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:25:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:25:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:25:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:25:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:25:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:25:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:25:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 1, 2024-02-05T07:25:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 1, 2024-02-05T07:25:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 1, 2024-02-05T07:25:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 1, 2024-02-05T07:25:04.215 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 1, 2024-02-05T07:25:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:25:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:25:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:25:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:25:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:25:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:25:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:25:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "195", 2024-02-05T07:25:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:25:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:25:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:25:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:25:04.216 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:25:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:25:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:25:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:25:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:25:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:25:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:25:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:25:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:25:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:25:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:25:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:25:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:25:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:25:04.217 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:25:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:25:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:25:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:25:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:25:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:25:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-02-05T07:25:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_max": 32, 2024-02-05T07:25:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_min": 1 2024-02-05T07:25:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:25:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": {} 2024-02-05T07:25:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:25:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:25:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 6.9800000190734863, 2024-02-05T07:25:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 6.9800000190734863, 2024-02-05T07:25:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.43000000715255737, 2024-02-05T07:25:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 3, 2024-02-05T07:25:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 3, 2024-02-05T07:25:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:25:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:25:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:25:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 2, 2024-02-05T07:25:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "datapool", 2024-02-05T07:25:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:47.259462+0000", 2024-02-05T07:25:04.220 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 8193, 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 3, 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 3, 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 3, 2024-02-05T07:25:04.221 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 3, 2024-02-05T07:25:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 3, 2024-02-05T07:25:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:25:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:25:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:25:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:25:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:25:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:25:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:25:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "71", 2024-02-05T07:25:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:25:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:25:04.222 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 2, 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 71, 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:25:04.223 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:25:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:25:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:25:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:25:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:25:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:25:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:25:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:25:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:25:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:25:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:25:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:25:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:25:04.224 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:25:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:25:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:25:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:25:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:25:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:25:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:25:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:25:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:25:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd": {} 2024-02-05T07:25:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:25:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:25:04.225 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.3299999237060547, 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.3299999237060547, 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2, 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2, 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 3, 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".rgw.root", 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:47.762336+0000", 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:25:04.226 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:25:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:25:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:25:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:25:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:25:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:25:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:25:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:25:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:25:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:25:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:25:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:25:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:25:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:25:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:25:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:25:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:25:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:25:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:25:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:25:04.228 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:25:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:25:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:25:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "69", 2024-02-05T07:25:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:25:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:25:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:25:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:25:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:25:04.229 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:25:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:25:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:25:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:25:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:25:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:25:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:25:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:25:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:25:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:25:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:25:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:25:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:25:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:25:04.230 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:25:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:25:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:25:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:25:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:25:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:25:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:25:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:25:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:25:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:25:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:25:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:25:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:25:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:25:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:25:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:25:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:25:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:25:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:25:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:25:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:25:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:25:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:25:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9700000286102295, 2024-02-05T07:25:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9700000286102295, 2024-02-05T07:25:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-02-05T07:25:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.9700000286102295, 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.9700000286102295, 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 4, 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.log", 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:49.736599+0000", 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:25:04.233 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:25:04.234 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:25:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:25:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:25:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:25:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:25:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "71", 2024-02-05T07:25:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:25:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:25:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:25:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:25:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:25:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:25:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:25:04.235 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:25:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:25:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:25:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:25:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:25:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:25:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:25:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:25:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:25:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:25:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:25:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:25:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:25:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:25:04.236 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:25:04.237 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:25:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:25:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:25:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:25:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:25:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:25:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:25:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9700000286102295, 2024-02-05T07:25:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9700000286102295, 2024-02-05T07:25:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-02-05T07:25:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.9700000286102295, 2024-02-05T07:25:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.9700000286102295, 2024-02-05T07:25:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:25:04.238 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 5, 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.control", 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:51.696948+0000", 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:25:04.239 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:25:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:25:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:25:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:25:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:25:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:25:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:25:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:25:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:25:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:25:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:25:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:25:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:25:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:25:04.240 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "73", 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:25:04.241 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:25:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:25:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:25:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:25:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:25:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:25:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:25:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:25:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:25:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:25:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:25:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:25:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:25:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:25:04.242 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:25:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:25:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:25:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:25:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:25:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:25:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:25:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:25:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:25:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:25:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:25:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:25:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:25:04.243 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:25:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:25:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:25:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:25:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:25:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.75, 2024-02-05T07:25:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.75, 2024-02-05T07:25:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-02-05T07:25:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.75, 2024-02-05T07:25:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.75, 2024-02-05T07:25:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:25:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:25:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:25:04.244 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 6, 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.meta", 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:53.768555+0000", 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:25:04.245 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.246 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "75", 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:25:04.247 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:25:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:25:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:25:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:25:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:25:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:25:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:25:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:25:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:25:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:25:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:25:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:25:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:25:04.248 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_bias": 4 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:25:04.249 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:25:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:25:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.190000057220459, 2024-02-05T07:25:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.190000057220459, 2024-02-05T07:25:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-02-05T07:25:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.190000057220459, 2024-02-05T07:25:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.190000057220459, 2024-02-05T07:25:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:25:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:25:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:25:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 7, 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "unique_pool_0", 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:20:44.070155+0000", 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:25:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:25:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 36, 2024-02-05T07:25:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 36, 2024-02-05T07:25:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 36, 2024-02-05T07:25:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 36, 2024-02-05T07:25:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 36, 2024-02-05T07:25:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:25:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:25:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:25:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:25:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:25:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:25:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:25:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "179", 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "177", 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:25:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:25:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "rados": {} 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.559999942779541, 2024-02-05T07:25:04.255 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.559999942779541, 2024-02-05T07:25:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-02-05T07:25:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.559999942779541, 2024-02-05T07:25:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.559999942779541, 2024-02-05T07:25:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:25:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:25:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:25:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:25:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:25:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-02-05T07:25:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "703c3948-8003-4015-9dd8-21019cf0eeb8", 2024-02-05T07:25:04.256 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0, 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 8, 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 198, 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-02-05T07:25:04.257 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6802", 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6804", 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6803", 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6801/2069016689", 2024-02-05T07:25:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6802/2069016689", 2024-02-05T07:25:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6804/2069016689", 2024-02-05T07:25:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6803/2069016689", 2024-02-05T07:25:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:25:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:25:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:25:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-02-05T07:25:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "5c3e7d1e-a61d-44b9-b21e-f91c99a096bc", 2024-02-05T07:25:04.260 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 13, 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 199, 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6805", 2024-02-05T07:25:04.261 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6806", 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6808", 2024-02-05T07:25:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:25:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:25:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6807", 2024-02-05T07:25:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:25:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.263 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6805/865486532", 2024-02-05T07:25:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6806/865486532", 2024-02-05T07:25:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6808/865486532", 2024-02-05T07:25:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6807/865486532", 2024-02-05T07:25:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:25:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:25:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:25:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-02-05T07:25:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "f4fd930f-7bb2-46ca-9964-98235bd47b3a", 2024-02-05T07:25:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:25:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:25:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:25:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:25:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:25:04.265 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:25:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 18, 2024-02-05T07:25:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 188, 2024-02-05T07:25:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:25:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:25:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:25:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6809", 2024-02-05T07:25:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:25:04.266 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:25:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6810", 2024-02-05T07:25:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:25:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:25:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6812", 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6811", 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.271 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6809/904871370", 2024-02-05T07:25:04.271 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6810/904871370", 2024-02-05T07:25:04.271 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6812/904871370", 2024-02-05T07:25:04.271 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6811/904871370", 2024-02-05T07:25:04.271 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:25:04.271 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:25:04.271 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:25:04.271 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-02-05T07:25:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "149ed0d6-b049-4478-9819-cafc446e94d3", 2024-02-05T07:25:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:25:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-02-05T07:25:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-02-05T07:25:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:25:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:25:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:25:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 26, 2024-02-05T07:25:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 160, 2024-02-05T07:25:04.272 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:25:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:25:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:25:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6813", 2024-02-05T07:25:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:25:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.273 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:25:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6814", 2024-02-05T07:25:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:25:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:25:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.274 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6816", 2024-02-05T07:25:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:25:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:25:04.277 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6815", 2024-02-05T07:25:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:25:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6813/2758586242", 2024-02-05T07:25:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6814/2758586242", 2024-02-05T07:25:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6816/2758586242", 2024-02-05T07:25:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6815/2758586242", 2024-02-05T07:25:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:25:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:25:04.278 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d", 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 32, 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 197, 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:25:04.279 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:25:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:25:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6800", 2024-02-05T07:25:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:25:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:25:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.280 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6801", 2024-02-05T07:25:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:25:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:25:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6803", 2024-02-05T07:25:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:25:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.281 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:25:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6802", 2024-02-05T07:25:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:25:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.282 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6800/1154929470", 2024-02-05T07:25:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6801/1154929470", 2024-02-05T07:25:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6803/1154929470", 2024-02-05T07:25:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6802/1154929470", 2024-02-05T07:25:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:25:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:25:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:25:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-02-05T07:25:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "244a7930-c22c-413e-86e1-11e0749f7367", 2024-02-05T07:25:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:25:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:25:04.283 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:25:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0.7362823486328125, 2024-02-05T07:25:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:25:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:25:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 41, 2024-02-05T07:25:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 198, 2024-02-05T07:25:04.284 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:25:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:25:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:25:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6804", 2024-02-05T07:25:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:25:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.287 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:25:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.288 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6805", 2024-02-05T07:25:04.289 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:25:04.289 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.289 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6807", 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6806", 2024-02-05T07:25:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:25:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6804/398113577", 2024-02-05T07:25:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6805/398113577", 2024-02-05T07:25:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6807/398113577", 2024-02-05T07:25:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6806/398113577", 2024-02-05T07:25:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:25:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:25:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:25:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-02-05T07:25:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7", 2024-02-05T07:25:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:25:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:25:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:25:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:25:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:25:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:25:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 52, 2024-02-05T07:25:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 186, 2024-02-05T07:25:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:25:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:25:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:25:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6808", 2024-02-05T07:25:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2371506925 2024-02-05T07:25:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:25:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6809", 2024-02-05T07:25:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2371506925 2024-02-05T07:25:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.293 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.294 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:25:04.294 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.294 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.294 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6811", 2024-02-05T07:25:04.294 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2371506925 2024-02-05T07:25:04.294 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.294 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.294 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:25:04.294 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.294 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6810", 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2371506925 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6808/2371506925", 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6809/2371506925", 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6811/2371506925", 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6810/2371506925", 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.296 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-02-05T07:25:04.296 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9", 2024-02-05T07:25:04.296 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:25:04.296 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:25:04.296 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:25:04.296 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:25:04.296 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:25:04.296 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:25:04.296 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 60, 2024-02-05T07:25:04.296 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 186, 2024-02-05T07:25:04.296 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:25:04.296 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:25:04.296 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:25:04.296 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6812", 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6813", 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6815", 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6814", 2024-02-05T07:25:04.298 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:25:04.299 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.299 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.299 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.299 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6812/2983605980", 2024-02-05T07:25:04.299 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6813/2983605980", 2024-02-05T07:25:04.299 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2983605980", 2024-02-05T07:25:04.299 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6814/2983605980", 2024-02-05T07:25:04.299 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:25:04.299 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:25:04.299 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:25:04.299 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.299 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.299 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:25:04.299 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_xinfo": [ 2024-02-05T07:25:04.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.300 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-02-05T07:25:04.300 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:25:04.300 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:25:04.300 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:25:04.300 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:25:04.300 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:25:04.300 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:23:56.257753+0000", 2024-02-05T07:25:04.300 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:25:04.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.300 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-02-05T07:25:04.300 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:25:04.300 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:25:04.301 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:25:04.301 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:25:04.301 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:25:04.301 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:23:55.923130+0000", 2024-02-05T07:25:04.301 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:25:04.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.301 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-02-05T07:25:04.301 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:25:04.301 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:25:04.301 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:25:04.301 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:25:04.301 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:25:04.301 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:23:40.814320+0000", 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:24:45.570840+0000", 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-02-05T07:25:04.302 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:25:00.861196+0000", 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:25:04.303 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:24:32.792654+0000", 2024-02-05T07:25:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:25:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-02-05T07:25:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:25:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:25:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:25:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:25:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:25:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:24:09.097695+0000", 2024-02-05T07:25:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:25:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:24:57.888062+0000", 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap": [], 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_items": [ 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.9", 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:25:04.305 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.12", 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 5 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.306 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.13", 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_primaries": [], 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_temp": [], 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_temp": [], 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklist": { 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2180087772": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2066685086": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:25:04.307 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/4100020626": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:25:04.308 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1953351182": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:25:04.308 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3203189162": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:25:04.308 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3418781614": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:25:04.308 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1518592084": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:25:04.308 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/3380493069": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:25:04.308 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2208590106": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:25:04.308 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/223579766": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:25:04.308 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/2634895574": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:25:04.308 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1462812719": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:25:04.308 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2823976078": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:25:04.308 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1905520582": "2024-02-06T07:13:47.004765+0000" 2024-02-05T07:25:04.308 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.308 INFO:teuthology.orchestra.run.smithi007.stdout: "range_blocklist": {}, 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profiles": { 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: "default": { 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-failure-domain": "osd", 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: "k": "2", 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: "m": "1", 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin": "jerasure", 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: "technique": "reed_sol_van" 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps_queue": [], 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: "new_removed_snaps": [], 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: "new_purged_snaps": [], 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_node_flags": {}, 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: "device_class_flags": {}, 2024-02-05T07:25:04.309 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode": { 2024-02-05T07:25:04.310 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_enabled": false, 2024-02-05T07:25:04.310 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_bucket_count": 0, 2024-02-05T07:25:04.310 INFO:teuthology.orchestra.run.smithi007.stdout: "degraded_stretch_mode": 0, 2024-02-05T07:25:04.310 INFO:teuthology.orchestra.run.smithi007.stdout: "recovering_stretch_mode": 0, 2024-02-05T07:25:04.310 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_bucket": 0 2024-02-05T07:25:04.310 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:04.310 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:25:04.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:04.310 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:04.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:04.411 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:04 smithi007 ceph-mon[29098]: pgmap v289: 168 pgs: 168 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:25:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:04 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3226920731' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:25:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:04 smithi007 ceph-mon[37158]: pgmap v289: 168 pgs: 168 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:25:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:04 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3226920731' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:25:04.436 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:04.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:04.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:04.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:04 smithi090 ceph-mon[33213]: pgmap v289: 168 pgs: 168 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:25:04.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:04 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3226920731' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:25:04.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:04.566 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:04.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:04.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:04.696 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:04.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:04.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:04.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:04.826 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:04.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:04.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:04.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:04.955 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:04.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:05.056 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:05.084 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:05.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:05.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:05.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:05.196 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:05.215 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:05.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:05.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:05.289 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_0' removed 2024-02-05T07:25:05.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:05.342 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:05.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:05.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:05.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:05.485 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:05.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:05.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:05.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:05 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1040638707' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:25:05.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:05 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:25:05.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:05 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1040638707' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:25:05.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:05 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:25:05.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:05 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1040638707' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:25:05.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:05 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:25:05.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:05.611 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:05.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:05.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:05.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:05.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:05.737 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:05.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:05.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:05.861 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:05.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:05.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:05.955 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T07:25:05.956 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T07:25:05.956 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-02-05T07:25:05.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-02-05T07:25:05.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:05.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:05.987 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:05.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:06.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:06.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:06.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:06.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:06.239 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:06.359 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:06.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:06.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:06.484 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:06.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:06.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:06 smithi090 ceph-mon[33213]: pgmap v290: 168 pgs: 168 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:25:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:06 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:25:06.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:06 smithi090 ceph-mon[33213]: osdmap e205: 8 total, 8 up, 7 in 2024-02-05T07:25:06.538 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:06 smithi007 ceph-mon[29098]: pgmap v290: 168 pgs: 168 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:25:06.539 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:06 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:25:06.539 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:06 smithi007 ceph-mon[29098]: osdmap e205: 8 total, 8 up, 7 in 2024-02-05T07:25:06.539 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:06 smithi007 ceph-mon[37158]: pgmap v290: 168 pgs: 168 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:25:06.539 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:06 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:25:06.539 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:06 smithi007 ceph-mon[37158]: osdmap e205: 8 total, 8 up, 7 in 2024-02-05T07:25:06.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:06.613 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:06.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:06.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:06.649 INFO:tasks.thrashosds.thrasher:j is {'epoch': 204, 'fsid': 'e3e7059e-c3f5-11ee-95b6-87774f69a715', 'created': '2024-02-05T07:13:14.603885+0000', 'modified': '2024-02-05T07:25:01.256363+0000', 'last_up_change': '2024-02-05T07:18:35.434647+0000', 'last_in_change': '2024-02-05T07:23:46.563809+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T07:16:32.505567+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.980000019073486, 'score_stable': 6.980000019073486, 'optimal_score': 0.4300000071525574, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T07:18:47.259462+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T07:18:47.762336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T07:18:49.736599+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T07:18:51.696948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T07:18:53.768555+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-02-05T07:20:44.070155+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 36, 'pg_placement_num': 36, 'pg_placement_num_target': 36, 'pg_num_target': 36, 'pg_num_pending': 36, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '177', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.559999942779541, 'score_stable': 1.559999942779541, 'optimal_score': 1, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}], 'osds': [{'osd': 0, 'uuid': '703c3948-8003-4015-9dd8-21019cf0eeb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6801', 'nonce': 2069016689}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6802', 'nonce': 2069016689}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6804', 'nonce': 2069016689}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6803', 'nonce': 2069016689}]}, 'public_addr': '172.21.15.7:6801/2069016689', 'cluster_addr': '172.21.15.7:6802/2069016689', 'heartbeat_back_addr': '172.21.15.7:6804/2069016689', 'heartbeat_front_addr': '172.21.15.7:6803/2069016689', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '5c3e7d1e-a61d-44b9-b21e-f91c99a096bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 199, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6805', 'nonce': 865486532}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6806', 'nonce': 865486532}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6808', 'nonce': 865486532}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6807', 'nonce': 865486532}]}, 'public_addr': '172.21.15.7:6805/865486532', 'cluster_addr': '172.21.15.7:6806/865486532', 'heartbeat_back_addr': '172.21.15.7:6808/865486532', 'heartbeat_front_addr': '172.21.15.7:6807/865486532', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4fd930f-7bb2-46ca-9964-98235bd47b3a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 188, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6809', 'nonce': 904871370}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6810', 'nonce': 904871370}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6812', 'nonce': 904871370}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6811', 'nonce': 904871370}]}, 'public_addr': '172.21.15.7:6809/904871370', 'cluster_addr': '172.21.15.7:6810/904871370', 'heartbeat_back_addr': '172.21.15.7:6812/904871370', 'heartbeat_front_addr': '172.21.15.7:6811/904871370', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '149ed0d6-b049-4478-9819-cafc446e94d3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6813', 'nonce': 2758586242}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6814', 'nonce': 2758586242}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6816', 'nonce': 2758586242}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6815', 'nonce': 2758586242}]}, 'public_addr': '172.21.15.7:6813/2758586242', 'cluster_addr': '172.21.15.7:6814/2758586242', 'heartbeat_back_addr': '172.21.15.7:6816/2758586242', 'heartbeat_front_addr': '172.21.15.7:6815/2758586242', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 197, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 1154929470}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6801', 'nonce': 1154929470}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6803', 'nonce': 1154929470}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 1154929470}]}, 'public_addr': '172.21.15.90:6800/1154929470', 'cluster_addr': '172.21.15.90:6801/1154929470', 'heartbeat_back_addr': '172.21.15.90:6803/1154929470', 'heartbeat_front_addr': '172.21.15.90:6802/1154929470', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '244a7930-c22c-413e-86e1-11e0749f7367', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7362823486328125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 398113577}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6805', 'nonce': 398113577}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6807', 'nonce': 398113577}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 398113577}]}, 'public_addr': '172.21.15.90:6804/398113577', 'cluster_addr': '172.21.15.90:6805/398113577', 'heartbeat_back_addr': '172.21.15.90:6807/398113577', 'heartbeat_front_addr': '172.21.15.90:6806/398113577', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0adbb562-3012-4f60-b5ec-02550a10b2c7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 186, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 2371506925}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 2371506925}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6811', 'nonce': 2371506925}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6810', 'nonce': 2371506925}]}, 'public_addr': '172.21.15.90:6808/2371506925', 'cluster_addr': '172.21.15.90:6809/2371506925', 'heartbeat_back_addr': '172.21.15.90:6811/2371506925', 'heartbeat_front_addr': '172.21.15.90:6810/2371506925', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dbc26c36-a0e9-4f21-9e98-739270cea6d9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 186, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 2983605980}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 2983605980}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6815', 'nonce': 2983605980}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6814', 'nonce': 2983605980}]}, 'public_addr': '172.21.15.90:6812/2983605980', 'cluster_addr': '172.21.15.90:6813/2983605980', 'heartbeat_back_addr': '172.21.15.90:6815/2983605980', 'heartbeat_front_addr': '172.21.15.90:6814/2983605980', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:23:56.257753+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:23:55.923130+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:23:40.814320+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T07:24:45.570840+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:25:00.861196+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:24:32.792654+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:24:09.097695+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:24:57.888062+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.7:0/2180087772': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/2066685086': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/4100020626': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/1953351182': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/3203189162': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/3418781614': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:6800/1518592084': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:6800/3380493069': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/2208590106': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:0/223579766': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:6800/2634895574': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/1462812719': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/2823976078': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/1905520582': '2024-02-06T07:13:47.004765+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T07:25:06.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:25:06.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:06.739 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:06.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:06.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:06.868 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:06.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:06.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:06.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:06.995 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:06.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:06.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:07.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:07.126 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:07.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:07.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:07.252 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:07.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:07.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:07.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:07.380 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:07.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:07.431 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:25:07.431 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:25:07.431 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:25:07.431 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:25:07.432 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:25:07.432 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:25:07.452 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:25:07.472 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:25:07.474 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:25:07.474 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:25:07.474 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:25:07.474 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:25:07.474 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:25:07.474 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:25:07.474 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:25:07.474 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:25:07.481 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:07.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:07.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:07.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:07.634 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:07.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:07.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:07.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:07.761 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:07.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:07.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:07.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:07.887 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:07.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:07.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:08.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:08.014 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:08.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:08.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:08.141 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:08.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:08.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:08.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:08.268 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:08.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:08.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:08.396 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:08.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:08.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:08.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:08 smithi007 ceph-mon[29098]: pgmap v292: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 755 B/s rd, 0 op/s 2024-02-05T07:25:08.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:08 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1012754653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:08.414 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:08 smithi007 ceph-mon[37158]: pgmap v292: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 755 B/s rd, 0 op/s 2024-02-05T07:25:08.414 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:08 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1012754653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:08.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:08.522 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:08.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:08.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:08.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:08 smithi090 ceph-mon[33213]: pgmap v292: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 755 B/s rd, 0 op/s 2024-02-05T07:25:08.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:08 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1012754653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:08.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:08.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:08.649 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:08.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:08.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:08.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:08.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:08.907 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:08.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:09.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:09.036 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:09.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:09.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:09.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:09.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:09.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:09.291 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:09.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:09.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:09.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:09.417 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:09.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:09.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:09.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:09.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:09.544 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:09.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:09.590 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:09.595 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:09.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:09.674 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:09.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:09.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:09.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:09.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:09.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:09.929 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:09.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:10.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:10.059 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:10.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:10.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:10.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:10.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:10.277 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:10 smithi007 ceph-mon[29098]: pgmap v293: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:25:10.277 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:10 smithi007 ceph-mon[29098]: osdmap e206: 8 total, 8 up, 7 in 2024-02-05T07:25:10.278 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:10 smithi007 ceph-mon[37158]: pgmap v293: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:25:10.278 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:10 smithi007 ceph-mon[37158]: osdmap e206: 8 total, 8 up, 7 in 2024-02-05T07:25:10.291 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:10.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:10.319 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:10.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:10.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:10.444 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:10.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:10.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:10 smithi090 ceph-mon[33213]: pgmap v293: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:25:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:10 smithi090 ceph-mon[33213]: osdmap e206: 8 total, 8 up, 7 in 2024-02-05T07:25:10.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:25:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:25:10.546 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:10.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:10.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:10.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:10.699 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:10.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:10.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:10.758 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:25:10.758 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:25:10.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:10.829 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:10.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:10.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:10.957 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:10.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:10.993 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:25:11.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:11.087 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:11.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:11.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:11.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:11.217 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:11.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:11.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:11.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:11.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:11.344 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:11.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:11.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:11.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:11.470 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:11.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:11.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:11 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:11.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:11.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:11.599 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:11.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:11.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:11 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:11.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:11 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:11.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:11.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:11.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:11.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:11.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:11.861 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:11.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:11.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:11.991 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:11.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:12.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:12.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:12.122 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:12.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:12.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:12.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:12.252 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:12.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:12.343 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:12 smithi007 ceph-mon[29098]: pgmap v295: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:25:12.343 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:12 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:25:12.343 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:12 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:25:12.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:12 smithi007 ceph-mon[37158]: pgmap v295: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:25:12.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:12 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:25:12.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:12 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:25:12.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:12.381 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:12.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:12.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:12.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:12.507 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:12.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:12.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:12 smithi090 ceph-mon[33213]: pgmap v295: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:25:12.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:12 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:25:12.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:12 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:25:12.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:12.638 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:12.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:12.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:12.770 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:12.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:12.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:12.897 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:12.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:12.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:12.999 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:13.027 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:13.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:13.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:13.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:13.153 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:13.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:13.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:13.283 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:13.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:13.358 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:25:13.359 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:13 smithi007 ceph-mon[29098]: osdmap e207: 8 total, 8 up, 7 in 2024-02-05T07:25:13.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:13 smithi007 ceph-mon[37158]: osdmap e207: 8 total, 8 up, 7 in 2024-02-05T07:25:13.363 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:25:13.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:13.413 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:13.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:13.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:13.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:13 smithi090 ceph-mon[33213]: osdmap e207: 8 total, 8 up, 7 in 2024-02-05T07:25:13.537 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:13.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:13.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:13.639 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:13.660 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:25:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:25:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:25:13.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:13.665 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:13.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:13.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:13.794 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:13.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:13.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:13.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:13.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:13.921 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:13.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:14.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:14.049 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:14.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:14.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:14.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:14.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:14.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:14.208 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:25:14.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:14.304 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:14.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:14.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:14.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:14.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:14.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:14 smithi090 ceph-mon[33213]: pgmap v297: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 659 B/s rd, 0 op/s 2024-02-05T07:25:14.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:14 smithi090 ceph-mon[33213]: from='client.24932 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:25:14.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:14 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/465324375' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:25:14.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:14 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:25:14.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:14 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/465324375' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:25:14.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:14 smithi090 ceph-mon[33213]: osdmap e208: 8 total, 8 up, 7 in 2024-02-05T07:25:14.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:14.558 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:14.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:14.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:14 smithi007 ceph-mon[29098]: pgmap v297: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 659 B/s rd, 0 op/s 2024-02-05T07:25:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:14 smithi007 ceph-mon[29098]: from='client.24932 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:25:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:14 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/465324375' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:25:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:14 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:25:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:14 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/465324375' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:25:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:14 smithi007 ceph-mon[29098]: osdmap e208: 8 total, 8 up, 7 in 2024-02-05T07:25:14.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:14 smithi007 ceph-mon[37158]: pgmap v297: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 659 B/s rd, 0 op/s 2024-02-05T07:25:14.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:14 smithi007 ceph-mon[37158]: from='client.24932 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:25:14.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:14 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/465324375' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:25:14.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:14 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:25:14.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:14 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/465324375' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:25:14.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:14 smithi007 ceph-mon[37158]: osdmap e208: 8 total, 8 up, 7 in 2024-02-05T07:25:14.684 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:14.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:14.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:14.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:14.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T07:25:14.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:14.810 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:14.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:14.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:14.939 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:14.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:14.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:15.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:15.065 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:15.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:15.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:15.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:15.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:15.320 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:15.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:15.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:15.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:15.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:15.450 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:15.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:15.480 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_1' created 2024-02-05T07:25:15.525 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:15 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:15.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:25:15.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:15 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3176576720' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T07:25:15.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:15 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T07:25:15.526 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:15 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:15.526 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:25:15.526 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:15 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3176576720' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T07:25:15.526 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:15 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T07:25:15.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:15.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:15.676 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:15.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:15.700 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:15.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:15.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:15 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:15.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:25:15.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:15 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3176576720' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T07:25:15.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:15 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T07:25:15.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:15.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:15.827 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:15.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:15.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:15.955 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:15.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:15.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:15.960 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-02-05T07:25:13.031040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1296,"num_read_kb":1142,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":1773391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":645,"ondisk_log_size":645,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":397,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1422348,"kb_used_data":10168,"kb_used_omap":10,"kb_used_meta":1412085,"kb_avail":654793716,"statfs":{"total":671965249536,"available":670508765184,"internally_reserved":0,"allocated":10412032,"data_stored":6591105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11159,"internal_metadata":1445975145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":76,"apply_latency_ms":76,"commit_latency_ns":76000000,"apply_latency_ns":76000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.761704"},"pg_stats":[{"pgid":"4.c","version":"75'10","reported_seq":333,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856918+0000","last_change":"2024-02-05T07:23:47.629122+0000","last_active":"2024-02-05T07:25:09.856918+0000","last_peered":"2024-02-05T07:25:09.856918+0000","last_clean":"2024-02-05T07:25:09.856918+0000","last_became_active":"2024-02-05T07:23:47.597686+0000","last_became_peered":"2024-02-05T07:23:47.597686+0000","last_unstale":"2024-02-05T07:25:09.856918+0000","last_undegraded":"2024-02-05T07:25:09.856918+0000","last_fullsized":"2024-02-05T07:25:09.856918+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:23:04.538922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:04.538922+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":52,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857312+0000","last_change":"2024-02-05T07:24:34.251845+0000","last_active":"2024-02-05T07:25:09.857312+0000","last_peered":"2024-02-05T07:25:09.857312+0000","last_clean":"2024-02-05T07:25:09.857312+0000","last_became_active":"2024-02-05T07:24:34.251554+0000","last_became_peered":"2024-02-05T07:24:34.251554+0000","last_unstale":"2024-02-05T07:25:09.857312+0000","last_undegraded":"2024-02-05T07:25:09.857312+0000","last_fullsized":"2024-02-05T07:25:09.857312+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:13.198115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:13.198115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:13.263707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":384,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.685241+0000","last_change":"2024-02-05T07:21:27.645530+0000","last_active":"2024-02-05T07:25:09.685241+0000","last_peered":"2024-02-05T07:25:09.685241+0000","last_clean":"2024-02-05T07:25:09.685241+0000","last_became_active":"2024-02-05T07:18:53.461852+0000","last_became_peered":"2024-02-05T07:18:53.461852+0000","last_unstale":"2024-02-05T07:25:09.685241+0000","last_undegraded":"2024-02-05T07:25:09.685241+0000","last_fullsized":"2024-02-05T07:25:09.685241+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:21:27.645263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:27.645263+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:11.721072+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":292,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858405+0000","last_change":"2024-02-05T07:23:09.542975+0000","last_active":"2024-02-05T07:25:09.858405+0000","last_peered":"2024-02-05T07:25:09.858405+0000","last_clean":"2024-02-05T07:25:09.858405+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:25:09.858405+0000","last_undegraded":"2024-02-05T07:25:09.858405+0000","last_fullsized":"2024-02-05T07:25:09.858405+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:09.542903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:09.542903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019602+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":336,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857945+0000","last_change":"2024-02-05T07:24:54.875478+0000","last_active":"2024-02-05T07:25:09.857945+0000","last_peered":"2024-02-05T07:25:09.857945+0000","last_clean":"2024-02-05T07:25:09.857945+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:25:09.857945+0000","last_undegraded":"2024-02-05T07:25:09.857945+0000","last_fullsized":"2024-02-05T07:25:09.857945+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:24:54.875336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:54.875336+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:21.497738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.545526+0000","last_change":"2024-02-05T07:24:14.277115+0000","last_active":"2024-02-05T07:25:09.545526+0000","last_peered":"2024-02-05T07:25:09.545526+0000","last_clean":"2024-02-05T07:25:09.545526+0000","last_became_active":"2024-02-05T07:18:49.430518+0000","last_became_peered":"2024-02-05T07:18:49.430518+0000","last_unstale":"2024-02-05T07:25:09.545526+0000","last_undegraded":"2024-02-05T07:25:09.545526+0000","last_fullsized":"2024-02-05T07:25:09.545526+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:14.277005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:14.277005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:34.125024+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":297,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204582+0000","last_change":"2024-02-05T07:21:17.402214+0000","last_active":"2024-02-05T07:25:09.204582+0000","last_peered":"2024-02-05T07:25:09.204582+0000","last_clean":"2024-02-05T07:25:09.204582+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:25:09.204582+0000","last_undegraded":"2024-02-05T07:25:09.204582+0000","last_fullsized":"2024-02-05T07:25:09.204582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:17.401959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:17.401959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":298,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.686008+0000","last_change":"2024-02-05T07:24:54.743577+0000","last_active":"2024-02-05T07:25:09.686008+0000","last_peered":"2024-02-05T07:25:09.686008+0000","last_clean":"2024-02-05T07:25:09.686008+0000","last_became_active":"2024-02-05T07:23:47.609458+0000","last_became_peered":"2024-02-05T07:23:47.609458+0000","last_unstale":"2024-02-05T07:25:09.686008+0000","last_undegraded":"2024-02-05T07:25:09.686008+0000","last_fullsized":"2024-02-05T07:25:09.686008+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:54.743499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:54.743499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:04.040973+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"90'16","reported_seq":324,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858329+0000","last_change":"2024-02-05T07:21:43.811840+0000","last_active":"2024-02-05T07:25:09.858329+0000","last_peered":"2024-02-05T07:25:09.858329+0000","last_clean":"2024-02-05T07:25:09.858329+0000","last_became_active":"2024-02-05T07:18:51.447922+0000","last_became_peered":"2024-02-05T07:18:51.447922+0000","last_unstale":"2024-02-05T07:25:09.858329+0000","last_undegraded":"2024-02-05T07:25:09.858329+0000","last_fullsized":"2024-02-05T07:25:09.858329+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-02-05T07:21:43.811596+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:21:43.811596+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019468+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856875+0000","last_change":"2024-02-05T07:23:02.519644+0000","last_active":"2024-02-05T07:25:09.856875+0000","last_peered":"2024-02-05T07:25:09.856875+0000","last_clean":"2024-02-05T07:25:09.856875+0000","last_became_active":"2024-02-05T07:18:49.434074+0000","last_became_peered":"2024-02-05T07:18:49.434074+0000","last_unstale":"2024-02-05T07:25:09.856875+0000","last_undegraded":"2024-02-05T07:25:09.856875+0000","last_fullsized":"2024-02-05T07:25:09.856875+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:02.519566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:02.519566+0000","last_clean_scrub_stamp":"2024-02-05T07:23:02.519566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":126,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858171+0000","last_change":"2024-02-05T07:24:49.760637+0000","last_active":"2024-02-05T07:25:09.858171+0000","last_peered":"2024-02-05T07:25:09.858171+0000","last_clean":"2024-02-05T07:25:09.858171+0000","last_became_active":"2024-02-05T07:23:27.417675+0000","last_became_peered":"2024-02-05T07:23:27.417675+0000","last_unstale":"2024-02-05T07:25:09.858171+0000","last_undegraded":"2024-02-05T07:25:09.858171+0000","last_fullsized":"2024-02-05T07:25:09.858171+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:49.760543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.760543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:03.923284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":106,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.545976+0000","last_change":"2024-02-05T07:24:08.261218+0000","last_active":"2024-02-05T07:25:09.545976+0000","last_peered":"2024-02-05T07:25:09.545976+0000","last_clean":"2024-02-05T07:25:09.545976+0000","last_became_active":"2024-02-05T07:23:47.611030+0000","last_became_peered":"2024-02-05T07:23:47.611030+0000","last_unstale":"2024-02-05T07:25:09.545976+0000","last_undegraded":"2024-02-05T07:25:09.545976+0000","last_fullsized":"2024-02-05T07:25:09.545976+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:08.261123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:08.261123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:37.699207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"98'22","reported_seq":355,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.203270+0000","last_change":"2024-02-05T07:23:47.637874+0000","last_active":"2024-02-05T07:25:09.203270+0000","last_peered":"2024-02-05T07:25:09.203270+0000","last_clean":"2024-02-05T07:25:09.203270+0000","last_became_active":"2024-02-05T07:23:47.606483+0000","last_became_peered":"2024-02-05T07:23:47.606483+0000","last_unstale":"2024-02-05T07:25:09.203270+0000","last_undegraded":"2024-02-05T07:25:09.203270+0000","last_fullsized":"2024-02-05T07:25:09.203270+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'22","last_scrub_stamp":"2024-02-05T07:23:19.000981+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:23:19.000981+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444762+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":98,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.202392+0000","last_change":"2024-02-05T07:23:47.600893+0000","last_active":"2024-02-05T07:25:09.202392+0000","last_peered":"2024-02-05T07:25:09.202392+0000","last_clean":"2024-02-05T07:25:09.202392+0000","last_became_active":"2024-02-05T07:23:47.600273+0000","last_became_peered":"2024-02-05T07:23:47.600273+0000","last_unstale":"2024-02-05T07:25:09.202392+0000","last_undegraded":"2024-02-05T07:25:09.202392+0000","last_fullsized":"2024-02-05T07:25:09.202392+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:37.367711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.367711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":381,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856212+0000","last_change":"2024-02-05T07:24:19.650665+0000","last_active":"2024-02-05T07:25:09.856212+0000","last_peered":"2024-02-05T07:25:09.856212+0000","last_clean":"2024-02-05T07:25:09.856212+0000","last_became_active":"2024-02-05T07:18:53.459832+0000","last_became_peered":"2024-02-05T07:18:53.459832+0000","last_unstale":"2024-02-05T07:25:09.856212+0000","last_undegraded":"2024-02-05T07:25:09.856212+0000","last_fullsized":"2024-02-05T07:25:09.856212+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:24:19.650521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:19.650521+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:23.514384+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":292,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204481+0000","last_change":"2024-02-05T07:24:52.767947+0000","last_active":"2024-02-05T07:25:09.204481+0000","last_peered":"2024-02-05T07:25:09.204481+0000","last_clean":"2024-02-05T07:25:09.204481+0000","last_became_active":"2024-02-05T07:18:55.472172+0000","last_became_peered":"2024-02-05T07:18:55.472172+0000","last_unstale":"2024-02-05T07:25:09.204481+0000","last_undegraded":"2024-02-05T07:25:09.204481+0000","last_fullsized":"2024-02-05T07:25:09.204481+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:52.767872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:52.767872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:00.263931+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":303,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856604+0000","last_change":"2024-02-05T07:23:47.626460+0000","last_active":"2024-02-05T07:25:09.856604+0000","last_peered":"2024-02-05T07:25:09.856604+0000","last_clean":"2024-02-05T07:25:09.856604+0000","last_became_active":"2024-02-05T07:23:47.626072+0000","last_became_peered":"2024-02-05T07:23:47.626072+0000","last_unstale":"2024-02-05T07:25:09.856604+0000","last_undegraded":"2024-02-05T07:25:09.856604+0000","last_fullsized":"2024-02-05T07:25:09.856604+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:03.505163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:23:03.505163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":175,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856925+0000","last_change":"2024-02-05T07:24:14.360014+0000","last_active":"2024-02-05T07:25:09.856925+0000","last_peered":"2024-02-05T07:25:09.856925+0000","last_clean":"2024-02-05T07:25:09.856925+0000","last_became_active":"2024-02-05T07:23:47.618052+0000","last_became_peered":"2024-02-05T07:23:47.618052+0000","last_unstale":"2024-02-05T07:25:09.856925+0000","last_undegraded":"2024-02-05T07:25:09.856925+0000","last_fullsized":"2024-02-05T07:25:09.856925+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:24:14.359892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:14.359892+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:34.362411+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":98,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857941+0000","last_change":"2024-02-05T07:23:47.611708+0000","last_active":"2024-02-05T07:25:09.857941+0000","last_peered":"2024-02-05T07:25:09.857941+0000","last_clean":"2024-02-05T07:25:09.857941+0000","last_became_active":"2024-02-05T07:23:47.609054+0000","last_became_peered":"2024-02-05T07:23:47.609054+0000","last_unstale":"2024-02-05T07:25:09.857941+0000","last_undegraded":"2024-02-05T07:25:09.857941+0000","last_fullsized":"2024-02-05T07:25:09.857941+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:42.464894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:42.464894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":298,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856612+0000","last_change":"2024-02-05T07:23:47.603895+0000","last_active":"2024-02-05T07:25:09.856612+0000","last_peered":"2024-02-05T07:25:09.856612+0000","last_clean":"2024-02-05T07:25:09.856612+0000","last_became_active":"2024-02-05T07:23:47.603326+0000","last_became_peered":"2024-02-05T07:23:47.603326+0000","last_unstale":"2024-02-05T07:25:09.856612+0000","last_undegraded":"2024-02-05T07:25:09.856612+0000","last_fullsized":"2024-02-05T07:25:09.856612+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:16.499591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:16.499591+0000","last_clean_scrub_stamp":"2024-02-05T07:23:16.499591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019688+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"164'9","reported_seq":152,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.201951+0000","last_change":"2024-02-05T07:24:20.453876+0000","last_active":"2024-02-05T07:25:09.201951+0000","last_peered":"2024-02-05T07:25:09.201951+0000","last_clean":"2024-02-05T07:25:09.201951+0000","last_became_active":"2024-02-05T07:23:47.626082+0000","last_became_peered":"2024-02-05T07:23:47.626082+0000","last_unstale":"2024-02-05T07:25:09.201951+0000","last_undegraded":"2024-02-05T07:25:09.201951+0000","last_fullsized":"2024-02-05T07:25:09.201951+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:24:20.453746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:20.453746+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:32.518804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"202'21","reported_seq":329,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857055+0000","last_change":"2024-02-05T07:24:59.870626+0000","last_active":"2024-02-05T07:25:09.857055+0000","last_peered":"2024-02-05T07:25:09.857055+0000","last_clean":"2024-02-05T07:25:09.857055+0000","last_became_active":"2024-02-05T07:18:51.438507+0000","last_became_peered":"2024-02-05T07:18:51.438507+0000","last_unstale":"2024-02-05T07:25:09.857055+0000","last_undegraded":"2024-02-05T07:25:09.857055+0000","last_fullsized":"2024-02-05T07:25:09.857055+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'21","last_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:24:59.870542+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:09.890255+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":58,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.202756+0000","last_change":"2024-02-05T07:24:34.238802+0000","last_active":"2024-02-05T07:25:09.202756+0000","last_peered":"2024-02-05T07:25:09.202756+0000","last_clean":"2024-02-05T07:25:09.202756+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:25:09.202756+0000","last_undegraded":"2024-02-05T07:25:09.202756+0000","last_fullsized":"2024-02-05T07:25:09.202756+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:21:39.233796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.233796+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":293,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857379+0000","last_change":"2024-02-05T07:24:59.124223+0000","last_active":"2024-02-05T07:25:09.857379+0000","last_peered":"2024-02-05T07:25:09.857379+0000","last_clean":"2024-02-05T07:25:09.857379+0000","last_became_active":"2024-02-05T07:18:55.507791+0000","last_became_peered":"2024-02-05T07:18:55.507791+0000","last_unstale":"2024-02-05T07:25:09.857379+0000","last_undegraded":"2024-02-05T07:25:09.857379+0000","last_fullsized":"2024-02-05T07:25:09.857379+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:59.124133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:59.124133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:16.719629+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"187'9","reported_seq":91,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.545985+0000","last_change":"2024-02-05T07:24:59.438290+0000","last_active":"2024-02-05T07:25:09.545985+0000","last_peered":"2024-02-05T07:25:09.545985+0000","last_clean":"2024-02-05T07:25:09.545985+0000","last_became_active":"2024-02-05T07:24:34.230336+0000","last_became_peered":"2024-02-05T07:24:34.230336+0000","last_unstale":"2024-02-05T07:25:09.545985+0000","last_undegraded":"2024-02-05T07:25:09.545985+0000","last_fullsized":"2024-02-05T07:25:09.545985+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'9","last_scrub_stamp":"2024-02-05T07:24:59.438123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:59.438123+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:25.060163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"151'15","reported_seq":330,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.202960+0000","last_change":"2024-02-05T07:23:26.247211+0000","last_active":"2024-02-05T07:25:09.202960+0000","last_peered":"2024-02-05T07:25:09.202960+0000","last_clean":"2024-02-05T07:25:09.202960+0000","last_became_active":"2024-02-05T07:18:51.448044+0000","last_became_peered":"2024-02-05T07:18:51.448044+0000","last_unstale":"2024-02-05T07:25:09.202960+0000","last_undegraded":"2024-02-05T07:25:09.202960+0000","last_fullsized":"2024-02-05T07:25:09.202960+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:23:26.247047+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444780+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":311,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204604+0000","last_change":"2024-02-05T07:24:57.766610+0000","last_active":"2024-02-05T07:25:09.204604+0000","last_peered":"2024-02-05T07:25:09.204604+0000","last_clean":"2024-02-05T07:25:09.204604+0000","last_became_active":"2024-02-05T07:23:47.604997+0000","last_became_peered":"2024-02-05T07:23:47.604997+0000","last_unstale":"2024-02-05T07:25:09.204604+0000","last_undegraded":"2024-02-05T07:25:09.204604+0000","last_fullsized":"2024-02-05T07:25:09.204604+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:57.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:24:57.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:02.316661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":245,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.685422+0000","last_change":"2024-02-05T07:24:34.231420+0000","last_active":"2024-02-05T07:25:09.685422+0000","last_peered":"2024-02-05T07:25:09.685422+0000","last_clean":"2024-02-05T07:25:09.685422+0000","last_became_active":"2024-02-05T07:24:34.229242+0000","last_became_peered":"2024-02-05T07:24:34.229242+0000","last_unstale":"2024-02-05T07:25:09.685422+0000","last_undegraded":"2024-02-05T07:25:09.685422+0000","last_fullsized":"2024-02-05T07:25:09.685422+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:27.196824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:27.196824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:11.721143+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":297,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.545187+0000","last_change":"2024-02-05T07:21:37.374835+0000","last_active":"2024-02-05T07:25:09.545187+0000","last_peered":"2024-02-05T07:25:09.545187+0000","last_clean":"2024-02-05T07:25:09.545187+0000","last_became_active":"2024-02-05T07:18:53.459474+0000","last_became_peered":"2024-02-05T07:18:53.459474+0000","last_unstale":"2024-02-05T07:25:09.545187+0000","last_undegraded":"2024-02-05T07:25:09.545187+0000","last_fullsized":"2024-02-05T07:25:09.545187+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.374703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":113,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204355+0000","last_change":"2024-02-05T07:24:49.809811+0000","last_active":"2024-02-05T07:25:09.204355+0000","last_peered":"2024-02-05T07:25:09.204355+0000","last_clean":"2024-02-05T07:25:09.204355+0000","last_became_active":"2024-02-05T07:24:34.229372+0000","last_became_peered":"2024-02-05T07:24:34.229372+0000","last_unstale":"2024-02-05T07:25:09.204355+0000","last_undegraded":"2024-02-05T07:25:09.204355+0000","last_fullsized":"2024-02-05T07:25:09.204355+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:24:49.809687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.809687+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:00.732863+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":306,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204247+0000","last_change":"2024-02-05T07:21:40.542969+0000","last_active":"2024-02-05T07:25:09.204247+0000","last_peered":"2024-02-05T07:25:09.204247+0000","last_clean":"2024-02-05T07:25:09.204247+0000","last_became_active":"2024-02-05T07:18:49.428470+0000","last_became_peered":"2024-02-05T07:18:49.428470+0000","last_unstale":"2024-02-05T07:25:09.204247+0000","last_undegraded":"2024-02-05T07:25:09.204247+0000","last_fullsized":"2024-02-05T07:25:09.204247+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:40.542479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:21:40.542479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444630+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":294,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.202086+0000","last_change":"2024-02-05T07:23:01.849413+0000","last_active":"2024-02-05T07:25:09.202086+0000","last_peered":"2024-02-05T07:25:09.202086+0000","last_clean":"2024-02-05T07:25:09.202086+0000","last_became_active":"2024-02-05T07:18:55.505360+0000","last_became_peered":"2024-02-05T07:18:55.505360+0000","last_unstale":"2024-02-05T07:25:09.202086+0000","last_undegraded":"2024-02-05T07:25:09.202086+0000","last_fullsized":"2024-02-05T07:25:09.202086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:01.849346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:01.849346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":296,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204996+0000","last_change":"2024-02-05T07:24:50.782870+0000","last_active":"2024-02-05T07:25:09.204996+0000","last_peered":"2024-02-05T07:25:09.204996+0000","last_clean":"2024-02-05T07:25:09.204996+0000","last_became_active":"2024-02-05T07:18:53.468512+0000","last_became_peered":"2024-02-05T07:18:53.468512+0000","last_unstale":"2024-02-05T07:25:09.204996+0000","last_undegraded":"2024-02-05T07:25:09.204996+0000","last_fullsized":"2024-02-05T07:25:09.204996+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:50.782805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:50.782805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:08.806209+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":87,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.203030+0000","last_change":"2024-02-05T07:24:34.210115+0000","last_active":"2024-02-05T07:25:09.203030+0000","last_peered":"2024-02-05T07:25:09.203030+0000","last_clean":"2024-02-05T07:25:09.203030+0000","last_became_active":"2024-02-05T07:24:34.209716+0000","last_became_peered":"2024-02-05T07:24:34.209716+0000","last_unstale":"2024-02-05T07:25:09.203030+0000","last_undegraded":"2024-02-05T07:25:09.203030+0000","last_fullsized":"2024-02-05T07:25:09.203030+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:24:15.127119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:15.127119+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:22.751406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":690,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856547+0000","last_change":"2024-02-05T07:24:50.043491+0000","last_active":"2024-02-05T07:25:09.856547+0000","last_peered":"2024-02-05T07:25:09.856547+0000","last_clean":"2024-02-05T07:25:09.856547+0000","last_became_active":"2024-02-05T07:18:48.424620+0000","last_became_peered":"2024-02-05T07:18:48.424620+0000","last_unstale":"2024-02-05T07:25:09.856547+0000","last_undegraded":"2024-02-05T07:25:09.856547+0000","last_fullsized":"2024-02-05T07:25:09.856547+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:24:50.043397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:24:50.043397+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:19.735031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":373,"num_read_kb":368,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":265,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857342+0000","last_change":"2024-02-05T07:24:48.775742+0000","last_active":"2024-02-05T07:25:09.857342+0000","last_peered":"2024-02-05T07:25:09.857342+0000","last_clean":"2024-02-05T07:25:09.857342+0000","last_became_active":"2024-02-05T07:24:34.241978+0000","last_became_peered":"2024-02-05T07:24:34.241978+0000","last_unstale":"2024-02-05T07:25:09.857342+0000","last_undegraded":"2024-02-05T07:25:09.857342+0000","last_fullsized":"2024-02-05T07:25:09.857342+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:24:48.775665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:56.120076+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.203981+0000","last_change":"2024-02-05T07:23:47.618520+0000","last_active":"2024-02-05T07:25:09.203981+0000","last_peered":"2024-02-05T07:25:09.203981+0000","last_clean":"2024-02-05T07:25:09.203981+0000","last_became_active":"2024-02-05T07:23:47.600441+0000","last_became_peered":"2024-02-05T07:23:47.600441+0000","last_unstale":"2024-02-05T07:25:09.203981+0000","last_undegraded":"2024-02-05T07:25:09.203981+0000","last_fullsized":"2024-02-05T07:25:09.203981+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:23:22.071923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:22.071923+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444728+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":289,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856342+0000","last_change":"2024-02-05T07:21:15.698783+0000","last_active":"2024-02-05T07:25:09.856342+0000","last_peered":"2024-02-05T07:25:09.856342+0000","last_clean":"2024-02-05T07:25:09.856342+0000","last_became_active":"2024-02-05T07:18:53.459971+0000","last_became_peered":"2024-02-05T07:18:53.459971+0000","last_unstale":"2024-02-05T07:25:09.856342+0000","last_undegraded":"2024-02-05T07:25:09.856342+0000","last_fullsized":"2024-02-05T07:25:09.856342+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:15.698665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:15.698665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048750+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":400,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.549781+0000","last_change":"2024-02-05T07:24:20.278989+0000","last_active":"2024-02-05T07:25:09.549781+0000","last_peered":"2024-02-05T07:25:09.549781+0000","last_clean":"2024-02-05T07:25:09.549781+0000","last_became_active":"2024-02-05T07:18:51.507714+0000","last_became_peered":"2024-02-05T07:18:51.507714+0000","last_unstale":"2024-02-05T07:25:09.549781+0000","last_undegraded":"2024-02-05T07:25:09.549781+0000","last_fullsized":"2024-02-05T07:25:09.549781+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-02-05T07:24:20.278897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:20.278897+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:36.052175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":106,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204389+0000","last_change":"2024-02-05T07:24:45.761682+0000","last_active":"2024-02-05T07:25:09.204389+0000","last_peered":"2024-02-05T07:25:09.204389+0000","last_clean":"2024-02-05T07:25:09.204389+0000","last_became_active":"2024-02-05T07:23:47.609497+0000","last_became_peered":"2024-02-05T07:23:47.609497+0000","last_unstale":"2024-02-05T07:25:09.204389+0000","last_undegraded":"2024-02-05T07:25:09.204389+0000","last_fullsized":"2024-02-05T07:25:09.204389+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:45.761597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:45.761597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:05.259798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":563,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856707+0000","last_change":"2024-02-05T07:24:21.294644+0000","last_active":"2024-02-05T07:25:09.856707+0000","last_peered":"2024-02-05T07:25:09.856707+0000","last_clean":"2024-02-05T07:25:09.856707+0000","last_became_active":"2024-02-05T07:18:39.444898+0000","last_became_peered":"2024-02-05T07:18:39.444898+0000","last_unstale":"2024-02-05T07:25:09.856707+0000","last_undegraded":"2024-02-05T07:25:09.856707+0000","last_fullsized":"2024-02-05T07:25:09.856707+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:48.367585+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":298,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204767+0000","last_change":"2024-02-05T07:24:58.774799+0000","last_active":"2024-02-05T07:25:09.204767+0000","last_peered":"2024-02-05T07:25:09.204767+0000","last_clean":"2024-02-05T07:25:09.204767+0000","last_became_active":"2024-02-05T07:23:47.592290+0000","last_became_peered":"2024-02-05T07:23:47.592290+0000","last_unstale":"2024-02-05T07:25:09.204767+0000","last_undegraded":"2024-02-05T07:25:09.204767+0000","last_fullsized":"2024-02-05T07:25:09.204767+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:58.774721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:58.774721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:15.475417+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":296,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.685258+0000","last_change":"2024-02-05T07:21:31.546003+0000","last_active":"2024-02-05T07:25:09.685258+0000","last_peered":"2024-02-05T07:25:09.685258+0000","last_clean":"2024-02-05T07:25:09.685258+0000","last_became_active":"2024-02-05T07:18:53.460771+0000","last_became_peered":"2024-02-05T07:18:53.460771+0000","last_unstale":"2024-02-05T07:25:09.685258+0000","last_undegraded":"2024-02-05T07:25:09.685258+0000","last_fullsized":"2024-02-05T07:25:09.685258+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:31.545467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:31.545467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:11.721009+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"96'18","reported_seq":333,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.203513+0000","last_change":"2024-02-05T07:23:05.927322+0000","last_active":"2024-02-05T07:25:09.203513+0000","last_peered":"2024-02-05T07:25:09.203513+0000","last_clean":"2024-02-05T07:25:09.203513+0000","last_became_active":"2024-02-05T07:18:51.436097+0000","last_became_peered":"2024-02-05T07:18:51.436097+0000","last_unstale":"2024-02-05T07:25:09.203513+0000","last_undegraded":"2024-02-05T07:25:09.203513+0000","last_fullsized":"2024-02-05T07:25:09.203513+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'18","last_scrub_stamp":"2024-02-05T07:23:05.927169+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:23:05.927169+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444694+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":320,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.685470+0000","last_change":"2024-02-05T07:23:47.622111+0000","last_active":"2024-02-05T07:25:09.685470+0000","last_peered":"2024-02-05T07:25:09.685470+0000","last_clean":"2024-02-05T07:25:09.685470+0000","last_became_active":"2024-02-05T07:23:47.621133+0000","last_became_peered":"2024-02-05T07:23:47.621133+0000","last_unstale":"2024-02-05T07:25:09.685470+0000","last_undegraded":"2024-02-05T07:25:09.685470+0000","last_fullsized":"2024-02-05T07:25:09.685470+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:18.723118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:21:18.723118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:11.720936+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":312,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.202290+0000","last_change":"2024-02-05T07:21:34.523665+0000","last_active":"2024-02-05T07:25:09.202290+0000","last_peered":"2024-02-05T07:25:09.202290+0000","last_clean":"2024-02-05T07:25:09.202290+0000","last_became_active":"2024-02-05T07:18:49.429651+0000","last_became_peered":"2024-02-05T07:18:49.429651+0000","last_unstale":"2024-02-05T07:25:09.202290+0000","last_undegraded":"2024-02-05T07:25:09.202290+0000","last_fullsized":"2024-02-05T07:25:09.202290+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:21:34.523215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:34.523215+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444581+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":298,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858438+0000","last_change":"2024-02-05T07:23:47.611214+0000","last_active":"2024-02-05T07:25:09.858438+0000","last_peered":"2024-02-05T07:25:09.858438+0000","last_clean":"2024-02-05T07:25:09.858438+0000","last_became_active":"2024-02-05T07:23:47.610875+0000","last_became_peered":"2024-02-05T07:23:47.610875+0000","last_unstale":"2024-02-05T07:25:09.858438+0000","last_undegraded":"2024-02-05T07:25:09.858438+0000","last_fullsized":"2024-02-05T07:25:09.858438+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:38.531265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:21:38.531265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048888+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":50,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204872+0000","last_change":"2024-02-05T07:24:34.234702+0000","last_active":"2024-02-05T07:25:09.204872+0000","last_peered":"2024-02-05T07:25:09.204872+0000","last_clean":"2024-02-05T07:25:09.204872+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:25:09.204872+0000","last_undegraded":"2024-02-05T07:25:09.204872+0000","last_fullsized":"2024-02-05T07:25:09.204872+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:34.202812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:34.202812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"164'29","reported_seq":463,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.549658+0000","last_change":"2024-02-05T07:23:47.660633+0000","last_active":"2024-02-05T07:25:09.549658+0000","last_peered":"2024-02-05T07:25:09.549658+0000","last_clean":"2024-02-05T07:25:09.549658+0000","last_became_active":"2024-02-05T07:23:47.608917+0000","last_became_peered":"2024-02-05T07:23:47.608917+0000","last_unstale":"2024-02-05T07:25:09.549658+0000","last_undegraded":"2024-02-05T07:25:09.549658+0000","last_fullsized":"2024-02-05T07:25:09.549658+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-02-05T07:23:27.310935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:27.310935+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444816+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":305,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857148+0000","last_change":"2024-02-05T07:23:08.606203+0000","last_active":"2024-02-05T07:25:09.857148+0000","last_peered":"2024-02-05T07:25:09.857148+0000","last_clean":"2024-02-05T07:25:09.857148+0000","last_became_active":"2024-02-05T07:18:49.430208+0000","last_became_peered":"2024-02-05T07:18:49.430208+0000","last_unstale":"2024-02-05T07:25:09.857148+0000","last_undegraded":"2024-02-05T07:25:09.857148+0000","last_fullsized":"2024-02-05T07:25:09.857148+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:08.606061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:23:08.606061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019672+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":210,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.202570+0000","last_change":"2024-02-05T07:23:29.341025+0000","last_active":"2024-02-05T07:25:09.202570+0000","last_peered":"2024-02-05T07:25:09.202570+0000","last_clean":"2024-02-05T07:25:09.202570+0000","last_became_active":"2024-02-05T07:23:27.439333+0000","last_became_peered":"2024-02-05T07:23:27.439333+0000","last_unstale":"2024-02-05T07:25:09.202570+0000","last_undegraded":"2024-02-05T07:25:09.202570+0000","last_fullsized":"2024-02-05T07:25:09.202570+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:23:29.340876+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:20:38.993688+0000","last_clean_scrub_stamp":"2024-02-05T07:23:29.340876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444745+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":106,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858030+0000","last_change":"2024-02-05T07:24:05.412794+0000","last_active":"2024-02-05T07:25:09.858030+0000","last_peered":"2024-02-05T07:25:09.858030+0000","last_clean":"2024-02-05T07:25:09.858030+0000","last_became_active":"2024-02-05T07:23:47.603091+0000","last_became_peered":"2024-02-05T07:23:47.603091+0000","last_unstale":"2024-02-05T07:25:09.858030+0000","last_undegraded":"2024-02-05T07:25:09.858030+0000","last_fullsized":"2024-02-05T07:25:09.858030+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:05.412661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:05.412661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:15.571463+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":85,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.202132+0000","last_change":"2024-02-05T07:24:34.250380+0000","last_active":"2024-02-05T07:25:09.202132+0000","last_peered":"2024-02-05T07:25:09.202132+0000","last_clean":"2024-02-05T07:25:09.202132+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:25:09.202132+0000","last_undegraded":"2024-02-05T07:25:09.202132+0000","last_fullsized":"2024-02-05T07:25:09.202132+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:21.184445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:21.184445+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":108,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.545821+0000","last_change":"2024-02-05T07:24:07.278812+0000","last_active":"2024-02-05T07:25:09.545821+0000","last_peered":"2024-02-05T07:25:09.545821+0000","last_clean":"2024-02-05T07:25:09.545821+0000","last_became_active":"2024-02-05T07:23:47.610698+0000","last_became_peered":"2024-02-05T07:23:47.610698+0000","last_unstale":"2024-02-05T07:25:09.545821+0000","last_undegraded":"2024-02-05T07:25:09.545821+0000","last_fullsized":"2024-02-05T07:25:09.545821+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:07.278696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:07.278696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:35.119282+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":302,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.685533+0000","last_change":"2024-02-05T07:23:47.620787+0000","last_active":"2024-02-05T07:25:09.685533+0000","last_peered":"2024-02-05T07:25:09.685533+0000","last_clean":"2024-02-05T07:25:09.685533+0000","last_became_active":"2024-02-05T07:23:47.619951+0000","last_became_peered":"2024-02-05T07:23:47.619951+0000","last_unstale":"2024-02-05T07:25:09.685533+0000","last_undegraded":"2024-02-05T07:25:09.685533+0000","last_fullsized":"2024-02-05T07:25:09.685533+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:35.587887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:35.587887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:11.721039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":50,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857152+0000","last_change":"2024-02-05T07:24:34.236399+0000","last_active":"2024-02-05T07:25:09.857152+0000","last_peered":"2024-02-05T07:25:09.857152+0000","last_clean":"2024-02-05T07:25:09.857152+0000","last_became_active":"2024-02-05T07:24:34.235899+0000","last_became_peered":"2024-02-05T07:24:34.235899+0000","last_unstale":"2024-02-05T07:25:09.857152+0000","last_undegraded":"2024-02-05T07:25:09.857152+0000","last_fullsized":"2024-02-05T07:25:09.857152+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:37.209967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.209967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":339,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.545635+0000","last_change":"2024-02-05T07:24:54.413619+0000","last_active":"2024-02-05T07:25:09.545635+0000","last_peered":"2024-02-05T07:25:09.545635+0000","last_clean":"2024-02-05T07:25:09.545635+0000","last_became_active":"2024-02-05T07:18:51.507672+0000","last_became_peered":"2024-02-05T07:18:51.507672+0000","last_unstale":"2024-02-05T07:25:09.545635+0000","last_undegraded":"2024-02-05T07:25:09.545635+0000","last_fullsized":"2024-02-05T07:25:09.545635+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:24:54.413486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:54.413486+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:14.005875+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858346+0000","last_change":"2024-02-05T07:21:41.444299+0000","last_active":"2024-02-05T07:25:09.858346+0000","last_peered":"2024-02-05T07:25:09.858346+0000","last_clean":"2024-02-05T07:25:09.858346+0000","last_became_active":"2024-02-05T07:18:49.434451+0000","last_became_peered":"2024-02-05T07:18:49.434451+0000","last_unstale":"2024-02-05T07:25:09.858346+0000","last_undegraded":"2024-02-05T07:25:09.858346+0000","last_fullsized":"2024-02-05T07:25:09.858346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:41.444193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:41.444193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":296,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.685235+0000","last_change":"2024-02-05T07:21:38.468561+0000","last_active":"2024-02-05T07:25:09.685235+0000","last_peered":"2024-02-05T07:25:09.685235+0000","last_clean":"2024-02-05T07:25:09.685235+0000","last_became_active":"2024-02-05T07:18:53.464867+0000","last_became_peered":"2024-02-05T07:18:53.464867+0000","last_unstale":"2024-02-05T07:25:09.685235+0000","last_undegraded":"2024-02-05T07:25:09.685235+0000","last_fullsized":"2024-02-05T07:25:09.685235+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:38.468456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:38.468456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:11.721172+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":298,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856432+0000","last_change":"2024-02-05T07:24:48.036108+0000","last_active":"2024-02-05T07:25:09.856432+0000","last_peered":"2024-02-05T07:25:09.856432+0000","last_clean":"2024-02-05T07:25:09.856432+0000","last_became_active":"2024-02-05T07:23:47.611066+0000","last_became_peered":"2024-02-05T07:23:47.611066+0000","last_unstale":"2024-02-05T07:25:09.856432+0000","last_undegraded":"2024-02-05T07:25:09.856432+0000","last_fullsized":"2024-02-05T07:25:09.856432+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:48.036033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:48.036033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:55.002158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.685194+0000","last_change":"2024-02-05T07:24:57.734948+0000","last_active":"2024-02-05T07:25:09.685194+0000","last_peered":"2024-02-05T07:25:09.685194+0000","last_clean":"2024-02-05T07:25:09.685194+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:25:09.685194+0000","last_undegraded":"2024-02-05T07:25:09.685194+0000","last_fullsized":"2024-02-05T07:25:09.685194+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:57.734858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:57.734858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:58.093175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":359,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857427+0000","last_change":"2024-02-05T07:23:47.625879+0000","last_active":"2024-02-05T07:25:09.857427+0000","last_peered":"2024-02-05T07:25:09.857427+0000","last_clean":"2024-02-05T07:25:09.857427+0000","last_became_active":"2024-02-05T07:23:47.607181+0000","last_became_peered":"2024-02-05T07:23:47.607181+0000","last_unstale":"2024-02-05T07:25:09.857427+0000","last_undegraded":"2024-02-05T07:25:09.857427+0000","last_fullsized":"2024-02-05T07:25:09.857427+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:21:33.928419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:33.928419+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019503+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858083+0000","last_change":"2024-02-05T07:21:31.527396+0000","last_active":"2024-02-05T07:25:09.858083+0000","last_peered":"2024-02-05T07:25:09.858083+0000","last_clean":"2024-02-05T07:25:09.858083+0000","last_became_active":"2024-02-05T07:18:49.435621+0000","last_became_peered":"2024-02-05T07:18:49.435621+0000","last_unstale":"2024-02-05T07:25:09.858083+0000","last_undegraded":"2024-02-05T07:25:09.858083+0000","last_fullsized":"2024-02-05T07:25:09.858083+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:31.527325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:31.527325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048870+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":106,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857429+0000","last_change":"2024-02-05T07:24:42.108185+0000","last_active":"2024-02-05T07:25:09.857429+0000","last_peered":"2024-02-05T07:25:09.857429+0000","last_clean":"2024-02-05T07:25:09.857429+0000","last_became_active":"2024-02-05T07:23:47.610289+0000","last_became_peered":"2024-02-05T07:23:47.610289+0000","last_unstale":"2024-02-05T07:25:09.857429+0000","last_undegraded":"2024-02-05T07:25:09.857429+0000","last_fullsized":"2024-02-05T07:25:09.857429+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:42.108043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:42.108043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:50.123540+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":382,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857625+0000","last_change":"2024-02-05T07:21:26.505275+0000","last_active":"2024-02-05T07:25:09.857625+0000","last_peered":"2024-02-05T07:25:09.857625+0000","last_clean":"2024-02-05T07:25:09.857625+0000","last_became_active":"2024-02-05T07:18:53.456287+0000","last_became_peered":"2024-02-05T07:18:53.456287+0000","last_unstale":"2024-02-05T07:25:09.857625+0000","last_undegraded":"2024-02-05T07:25:09.857625+0000","last_fullsized":"2024-02-05T07:25:09.857625+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-02-05T07:21:26.505161+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:21:26.505161+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048832+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":325,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204176+0000","last_change":"2024-02-05T07:21:32.875499+0000","last_active":"2024-02-05T07:25:09.204176+0000","last_peered":"2024-02-05T07:25:09.204176+0000","last_clean":"2024-02-05T07:25:09.204176+0000","last_became_active":"2024-02-05T07:18:51.508097+0000","last_became_peered":"2024-02-05T07:18:51.508097+0000","last_unstale":"2024-02-05T07:25:09.204176+0000","last_undegraded":"2024-02-05T07:25:09.204176+0000","last_fullsized":"2024-02-05T07:25:09.204176+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:21:32.875346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:32.875346+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:09.768219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":316,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858084+0000","last_change":"2024-02-05T07:24:19.277756+0000","last_active":"2024-02-05T07:25:09.858084+0000","last_peered":"2024-02-05T07:25:09.858084+0000","last_clean":"2024-02-05T07:25:09.858084+0000","last_became_active":"2024-02-05T07:18:49.437663+0000","last_became_peered":"2024-02-05T07:18:49.437663+0000","last_unstale":"2024-02-05T07:25:09.858084+0000","last_undegraded":"2024-02-05T07:25:09.858084+0000","last_fullsized":"2024-02-05T07:25:09.858084+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:24:19.277618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:19.277618+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:22.979043+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":127,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.545448+0000","last_change":"2024-02-05T07:24:19.263465+0000","last_active":"2024-02-05T07:25:09.545448+0000","last_peered":"2024-02-05T07:25:09.545448+0000","last_clean":"2024-02-05T07:25:09.545448+0000","last_became_active":"2024-02-05T07:23:27.419855+0000","last_became_peered":"2024-02-05T07:23:27.419855+0000","last_unstale":"2024-02-05T07:25:09.545448+0000","last_undegraded":"2024-02-05T07:25:09.545448+0000","last_fullsized":"2024-02-05T07:25:09.545448+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:19.263379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:37.032056+0000","last_clean_scrub_stamp":"2024-02-05T07:24:19.263379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:32.556946+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":142,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858209+0000","last_change":"2024-02-05T07:24:07.263004+0000","last_active":"2024-02-05T07:25:09.858209+0000","last_peered":"2024-02-05T07:25:09.858209+0000","last_clean":"2024-02-05T07:25:09.858209+0000","last_became_active":"2024-02-05T07:23:47.607589+0000","last_became_peered":"2024-02-05T07:23:47.607589+0000","last_unstale":"2024-02-05T07:25:09.858209+0000","last_undegraded":"2024-02-05T07:25:09.858209+0000","last_fullsized":"2024-02-05T07:25:09.858209+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-02-05T07:24:07.262821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:07.262821+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":318,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.545368+0000","last_change":"2024-02-05T07:24:49.359644+0000","last_active":"2024-02-05T07:25:09.545368+0000","last_peered":"2024-02-05T07:25:09.545368+0000","last_clean":"2024-02-05T07:25:09.545368+0000","last_became_active":"2024-02-05T07:18:49.433121+0000","last_became_peered":"2024-02-05T07:18:49.433121+0000","last_unstale":"2024-02-05T07:25:09.545368+0000","last_undegraded":"2024-02-05T07:25:09.545368+0000","last_fullsized":"2024-02-05T07:25:09.545368+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:24:49.359534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.359534+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:55.921947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":294,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204868+0000","last_change":"2024-02-05T07:23:47.601964+0000","last_active":"2024-02-05T07:25:09.204868+0000","last_peered":"2024-02-05T07:25:09.204868+0000","last_clean":"2024-02-05T07:25:09.204868+0000","last_became_active":"2024-02-05T07:23:47.601556+0000","last_became_peered":"2024-02-05T07:23:47.601556+0000","last_unstale":"2024-02-05T07:25:09.204868+0000","last_undegraded":"2024-02-05T07:25:09.204868+0000","last_fullsized":"2024-02-05T07:25:09.204868+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:18.752135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:18.752135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:09.768083+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":292,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857672+0000","last_change":"2024-02-05T07:21:44.435932+0000","last_active":"2024-02-05T07:25:09.857672+0000","last_peered":"2024-02-05T07:25:09.857672+0000","last_clean":"2024-02-05T07:25:09.857672+0000","last_became_active":"2024-02-05T07:18:55.507555+0000","last_became_peered":"2024-02-05T07:18:55.507555+0000","last_unstale":"2024-02-05T07:25:09.857672+0000","last_undegraded":"2024-02-05T07:25:09.857672+0000","last_fullsized":"2024-02-05T07:25:09.857672+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:21:44.435826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":85,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858332+0000","last_change":"2024-02-05T07:24:34.274818+0000","last_active":"2024-02-05T07:25:09.858332+0000","last_peered":"2024-02-05T07:25:09.858332+0000","last_clean":"2024-02-05T07:25:09.858332+0000","last_became_active":"2024-02-05T07:24:34.241945+0000","last_became_peered":"2024-02-05T07:24:34.241945+0000","last_unstale":"2024-02-05T07:25:09.858332+0000","last_undegraded":"2024-02-05T07:25:09.858332+0000","last_fullsized":"2024-02-05T07:25:09.858332+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:33.244655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:33.244655+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857378+0000","last_change":"2024-02-05T07:21:25.529255+0000","last_active":"2024-02-05T07:25:09.857378+0000","last_peered":"2024-02-05T07:25:09.857378+0000","last_clean":"2024-02-05T07:25:09.857378+0000","last_became_active":"2024-02-05T07:18:49.437732+0000","last_became_peered":"2024-02-05T07:18:49.437732+0000","last_unstale":"2024-02-05T07:25:09.857378+0000","last_undegraded":"2024-02-05T07:25:09.857378+0000","last_fullsized":"2024-02-05T07:25:09.857378+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:25.529174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:21:25.529174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":106,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.203428+0000","last_change":"2024-02-05T07:24:47.676565+0000","last_active":"2024-02-05T07:25:09.203428+0000","last_peered":"2024-02-05T07:25:09.203428+0000","last_clean":"2024-02-05T07:25:09.203428+0000","last_became_active":"2024-02-05T07:23:47.606266+0000","last_became_peered":"2024-02-05T07:23:47.606266+0000","last_unstale":"2024-02-05T07:25:09.203428+0000","last_undegraded":"2024-02-05T07:25:09.203428+0000","last_fullsized":"2024-02-05T07:25:09.203428+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:47.676484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:47.676484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:56.595972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":292,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.685885+0000","last_change":"2024-02-05T07:21:39.515473+0000","last_active":"2024-02-05T07:25:09.685885+0000","last_peered":"2024-02-05T07:25:09.685885+0000","last_clean":"2024-02-05T07:25:09.685885+0000","last_became_active":"2024-02-05T07:18:55.469044+0000","last_became_peered":"2024-02-05T07:18:55.469044+0000","last_unstale":"2024-02-05T07:25:09.685885+0000","last_undegraded":"2024-02-05T07:25:09.685885+0000","last_fullsized":"2024-02-05T07:25:09.685885+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:39.515332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.515332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:11.721101+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":332,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204802+0000","last_change":"2024-02-05T07:23:47.619736+0000","last_active":"2024-02-05T07:25:09.204802+0000","last_peered":"2024-02-05T07:25:09.204802+0000","last_clean":"2024-02-05T07:25:09.204802+0000","last_became_active":"2024-02-05T07:23:47.602628+0000","last_became_peered":"2024-02-05T07:23:47.602628+0000","last_unstale":"2024-02-05T07:25:09.204802+0000","last_undegraded":"2024-02-05T07:25:09.204802+0000","last_fullsized":"2024-02-05T07:25:09.204802+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:30.859017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:30.859017+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:09.768142+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":265,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856764+0000","last_change":"2024-02-05T07:24:49.035482+0000","last_active":"2024-02-05T07:25:09.856764+0000","last_peered":"2024-02-05T07:25:09.856764+0000","last_clean":"2024-02-05T07:25:09.856764+0000","last_became_active":"2024-02-05T07:24:34.230825+0000","last_became_peered":"2024-02-05T07:24:34.230825+0000","last_unstale":"2024-02-05T07:25:09.856764+0000","last_undegraded":"2024-02-05T07:25:09.856764+0000","last_fullsized":"2024-02-05T07:25:09.856764+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:49.035385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.035385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:59.566665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"164'9","reported_seq":144,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857879+0000","last_change":"2024-02-05T07:23:47.620866+0000","last_active":"2024-02-05T07:25:09.857879+0000","last_peered":"2024-02-05T07:25:09.857879+0000","last_clean":"2024-02-05T07:25:09.857879+0000","last_became_active":"2024-02-05T07:23:47.605296+0000","last_became_peered":"2024-02-05T07:23:47.605296+0000","last_unstale":"2024-02-05T07:25:09.857879+0000","last_undegraded":"2024-02-05T07:25:09.857879+0000","last_fullsized":"2024-02-05T07:25:09.857879+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:22:58.051782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:22:58.051782+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.049019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":292,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858230+0000","last_change":"2024-02-05T07:24:14.511046+0000","last_active":"2024-02-05T07:25:09.858230+0000","last_peered":"2024-02-05T07:25:09.858230+0000","last_clean":"2024-02-05T07:25:09.858230+0000","last_became_active":"2024-02-05T07:18:55.467044+0000","last_became_peered":"2024-02-05T07:18:55.467044+0000","last_unstale":"2024-02-05T07:25:09.858230+0000","last_undegraded":"2024-02-05T07:25:09.858230+0000","last_fullsized":"2024-02-05T07:25:09.858230+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:14.510933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:14.510933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:37.853613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"113'15","reported_seq":225,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204489+0000","last_change":"2024-02-05T07:23:47.640930+0000","last_active":"2024-02-05T07:25:09.204489+0000","last_peered":"2024-02-05T07:25:09.204489+0000","last_clean":"2024-02-05T07:25:09.204489+0000","last_became_active":"2024-02-05T07:23:47.608158+0000","last_became_peered":"2024-02-05T07:23:47.608158+0000","last_unstale":"2024-02-05T07:25:09.204489+0000","last_undegraded":"2024-02-05T07:25:09.204489+0000","last_fullsized":"2024-02-05T07:25:09.204489+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'15","last_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:21:36.359546+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857134+0000","last_change":"2024-02-05T07:24:53.108878+0000","last_active":"2024-02-05T07:25:09.857134+0000","last_peered":"2024-02-05T07:25:09.857134+0000","last_clean":"2024-02-05T07:25:09.857134+0000","last_became_active":"2024-02-05T07:18:49.436131+0000","last_became_peered":"2024-02-05T07:18:49.436131+0000","last_unstale":"2024-02-05T07:25:09.857134+0000","last_undegraded":"2024-02-05T07:25:09.857134+0000","last_fullsized":"2024-02-05T07:25:09.857134+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:53.108801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:53.108801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:10.506459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":373,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204629+0000","last_change":"2024-02-05T07:20:38.003311+0000","last_active":"2024-02-05T07:25:09.204629+0000","last_peered":"2024-02-05T07:25:09.204629+0000","last_clean":"2024-02-05T07:25:09.204629+0000","last_became_active":"2024-02-05T07:18:53.465287+0000","last_became_peered":"2024-02-05T07:18:53.465287+0000","last_unstale":"2024-02-05T07:25:09.204629+0000","last_undegraded":"2024-02-05T07:25:09.204629+0000","last_fullsized":"2024-02-05T07:25:09.204629+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:20:38.003219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:20:38.003219+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:09.768060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":245,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857874+0000","last_change":"2024-02-05T07:24:34.242269+0000","last_active":"2024-02-05T07:25:09.857874+0000","last_peered":"2024-02-05T07:25:09.857874+0000","last_clean":"2024-02-05T07:25:09.857874+0000","last_became_active":"2024-02-05T07:24:34.238479+0000","last_became_peered":"2024-02-05T07:24:34.238479+0000","last_unstale":"2024-02-05T07:25:09.857874+0000","last_undegraded":"2024-02-05T07:25:09.857874+0000","last_fullsized":"2024-02-05T07:25:09.857874+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:16.156709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:16.156709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:30.913274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":337,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857305+0000","last_change":"2024-02-05T07:21:37.876638+0000","last_active":"2024-02-05T07:25:09.857305+0000","last_peered":"2024-02-05T07:25:09.857305+0000","last_clean":"2024-02-05T07:25:09.857305+0000","last_became_active":"2024-02-05T07:19:16.121696+0000","last_became_peered":"2024-02-05T07:19:16.121696+0000","last_unstale":"2024-02-05T07:25:09.857305+0000","last_undegraded":"2024-02-05T07:25:09.857305+0000","last_fullsized":"2024-02-05T07:25:09.857305+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:21:37.876511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.876511+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856766+0000","last_change":"2024-02-05T07:23:23.435405+0000","last_active":"2024-02-05T07:25:09.856766+0000","last_peered":"2024-02-05T07:25:09.856766+0000","last_clean":"2024-02-05T07:25:09.856766+0000","last_became_active":"2024-02-05T07:18:49.430766+0000","last_became_peered":"2024-02-05T07:18:49.430766+0000","last_unstale":"2024-02-05T07:25:09.856766+0000","last_undegraded":"2024-02-05T07:25:09.856766+0000","last_fullsized":"2024-02-05T07:25:09.856766+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:23:23.435135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019722+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":303,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.202871+0000","last_change":"2024-02-05T07:23:47.594349+0000","last_active":"2024-02-05T07:25:09.202871+0000","last_peered":"2024-02-05T07:25:09.202871+0000","last_clean":"2024-02-05T07:25:09.202871+0000","last_became_active":"2024-02-05T07:23:47.594030+0000","last_became_peered":"2024-02-05T07:23:47.594030+0000","last_unstale":"2024-02-05T07:25:09.202871+0000","last_undegraded":"2024-02-05T07:25:09.202871+0000","last_fullsized":"2024-02-05T07:25:09.202871+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:22:59.851438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:22:59.851438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444662+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":60,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.685350+0000","last_change":"2024-02-05T07:24:55.749908+0000","last_active":"2024-02-05T07:25:09.685350+0000","last_peered":"2024-02-05T07:25:09.685350+0000","last_clean":"2024-02-05T07:25:09.685350+0000","last_became_active":"2024-02-05T07:24:34.229238+0000","last_became_peered":"2024-02-05T07:24:34.229238+0000","last_unstale":"2024-02-05T07:25:09.685350+0000","last_undegraded":"2024-02-05T07:25:09.685350+0000","last_fullsized":"2024-02-05T07:25:09.685350+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:55.749767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:55.749767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:07.800010+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":280,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204200+0000","last_change":"2024-02-05T07:21:16.721101+0000","last_active":"2024-02-05T07:25:09.204200+0000","last_peered":"2024-02-05T07:25:09.204200+0000","last_clean":"2024-02-05T07:25:09.204200+0000","last_became_active":"2024-02-05T07:19:16.122059+0000","last_became_peered":"2024-02-05T07:19:16.122059+0000","last_unstale":"2024-02-05T07:25:09.204200+0000","last_undegraded":"2024-02-05T07:25:09.204200+0000","last_fullsized":"2024-02-05T07:25:09.204200+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:16.720264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:16.720264+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:09.768161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":302,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858286+0000","last_change":"2024-02-05T07:23:47.616365+0000","last_active":"2024-02-05T07:25:09.858286+0000","last_peered":"2024-02-05T07:25:09.858286+0000","last_clean":"2024-02-05T07:25:09.858286+0000","last_became_active":"2024-02-05T07:23:47.616029+0000","last_became_peered":"2024-02-05T07:23:47.616029+0000","last_unstale":"2024-02-05T07:25:09.858286+0000","last_undegraded":"2024-02-05T07:25:09.858286+0000","last_fullsized":"2024-02-05T07:25:09.858286+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:20.600583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:20.600583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048771+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":52,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.202253+0000","last_change":"2024-02-05T07:24:34.249659+0000","last_active":"2024-02-05T07:25:09.202253+0000","last_peered":"2024-02-05T07:25:09.202253+0000","last_clean":"2024-02-05T07:25:09.202253+0000","last_became_active":"2024-02-05T07:24:34.231409+0000","last_became_peered":"2024-02-05T07:24:34.231409+0000","last_unstale":"2024-02-05T07:25:09.202253+0000","last_undegraded":"2024-02-05T07:25:09.202253+0000","last_fullsized":"2024-02-05T07:25:09.202253+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:02.051772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:02.051772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:19.652909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":50,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.201768+0000","last_change":"2024-02-05T07:24:34.231186+0000","last_active":"2024-02-05T07:25:09.201768+0000","last_peered":"2024-02-05T07:25:09.201768+0000","last_clean":"2024-02-05T07:25:09.201768+0000","last_became_active":"2024-02-05T07:24:34.230006+0000","last_became_peered":"2024-02-05T07:24:34.230006+0000","last_unstale":"2024-02-05T07:25:09.201768+0000","last_undegraded":"2024-02-05T07:25:09.201768+0000","last_fullsized":"2024-02-05T07:25:09.201768+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:24:20.215062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:42.873758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":148,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857524+0000","last_change":"2024-02-05T07:23:47.638791+0000","last_active":"2024-02-05T07:25:09.857524+0000","last_peered":"2024-02-05T07:25:09.857524+0000","last_clean":"2024-02-05T07:25:09.857524+0000","last_became_active":"2024-02-05T07:23:47.613111+0000","last_became_peered":"2024-02-05T07:23:47.613111+0000","last_unstale":"2024-02-05T07:25:09.857524+0000","last_undegraded":"2024-02-05T07:25:09.857524+0000","last_fullsized":"2024-02-05T07:25:09.857524+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:21:18.095797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:18.095797+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204824+0000","last_change":"2024-02-05T07:21:29.818030+0000","last_active":"2024-02-05T07:25:09.204824+0000","last_peered":"2024-02-05T07:25:09.204824+0000","last_clean":"2024-02-05T07:25:09.204824+0000","last_became_active":"2024-02-05T07:18:49.434560+0000","last_became_peered":"2024-02-05T07:18:49.434560+0000","last_unstale":"2024-02-05T07:25:09.204824+0000","last_undegraded":"2024-02-05T07:25:09.204824+0000","last_fullsized":"2024-02-05T07:25:09.204824+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:29.817921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:29.817921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:09.768198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":296,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856975+0000","last_change":"2024-02-05T07:21:24.573753+0000","last_active":"2024-02-05T07:25:09.856975+0000","last_peered":"2024-02-05T07:25:09.856975+0000","last_clean":"2024-02-05T07:25:09.856975+0000","last_became_active":"2024-02-05T07:18:53.460977+0000","last_became_peered":"2024-02-05T07:18:53.460977+0000","last_unstale":"2024-02-05T07:25:09.856975+0000","last_undegraded":"2024-02-05T07:25:09.856975+0000","last_fullsized":"2024-02-05T07:25:09.856975+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:24.573655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_clean_scrub_stamp":"2024-02-05T07:21:24.573655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048714+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":60,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.545722+0000","last_change":"2024-02-05T07:24:46.423838+0000","last_active":"2024-02-05T07:25:09.545722+0000","last_peered":"2024-02-05T07:25:09.545722+0000","last_clean":"2024-02-05T07:25:09.545722+0000","last_became_active":"2024-02-05T07:24:34.225863+0000","last_became_peered":"2024-02-05T07:24:34.225863+0000","last_unstale":"2024-02-05T07:25:09.545722+0000","last_undegraded":"2024-02-05T07:25:09.545722+0000","last_fullsized":"2024-02-05T07:25:09.545722+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:46.423760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:46.423760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:58.661226+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":129,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.202349+0000","last_change":"2024-02-05T07:23:47.645600+0000","last_active":"2024-02-05T07:25:09.202349+0000","last_peered":"2024-02-05T07:25:09.202349+0000","last_clean":"2024-02-05T07:25:09.202349+0000","last_became_active":"2024-02-05T07:23:47.610110+0000","last_became_peered":"2024-02-05T07:23:47.610110+0000","last_unstale":"2024-02-05T07:25:09.202349+0000","last_undegraded":"2024-02-05T07:25:09.202349+0000","last_fullsized":"2024-02-05T07:25:09.202349+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-02-05T07:21:20.184048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:20.184048+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":257,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204834+0000","last_change":"2024-02-05T07:24:34.229700+0000","last_active":"2024-02-05T07:25:09.204834+0000","last_peered":"2024-02-05T07:25:09.204834+0000","last_clean":"2024-02-05T07:25:09.204834+0000","last_became_active":"2024-02-05T07:24:34.229229+0000","last_became_peered":"2024-02-05T07:24:34.229229+0000","last_unstale":"2024-02-05T07:25:09.204834+0000","last_undegraded":"2024-02-05T07:25:09.204834+0000","last_fullsized":"2024-02-05T07:25:09.204834+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:17.202540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:17.202540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:24.251692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":296,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.545628+0000","last_change":"2024-02-05T07:24:05.236956+0000","last_active":"2024-02-05T07:25:09.545628+0000","last_peered":"2024-02-05T07:25:09.545628+0000","last_clean":"2024-02-05T07:25:09.545628+0000","last_became_active":"2024-02-05T07:18:53.460307+0000","last_became_peered":"2024-02-05T07:18:53.460307+0000","last_unstale":"2024-02-05T07:25:09.545628+0000","last_undegraded":"2024-02-05T07:25:09.545628+0000","last_fullsized":"2024-02-05T07:25:09.545628+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:05.236812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:24:05.236812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:11.357567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":292,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857183+0000","last_change":"2024-02-05T07:21:30.473743+0000","last_active":"2024-02-05T07:25:09.857183+0000","last_peered":"2024-02-05T07:25:09.857183+0000","last_clean":"2024-02-05T07:25:09.857183+0000","last_became_active":"2024-02-05T07:18:55.468086+0000","last_became_peered":"2024-02-05T07:18:55.468086+0000","last_unstale":"2024-02-05T07:25:09.857183+0000","last_undegraded":"2024-02-05T07:25:09.857183+0000","last_fullsized":"2024-02-05T07:25:09.857183+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:30.473582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:30.473582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":328,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.546217+0000","last_change":"2024-02-05T07:24:53.457250+0000","last_active":"2024-02-05T07:25:09.546217+0000","last_peered":"2024-02-05T07:25:09.546217+0000","last_clean":"2024-02-05T07:25:09.546217+0000","last_became_active":"2024-02-05T07:18:51.446604+0000","last_became_peered":"2024-02-05T07:18:51.446604+0000","last_unstale":"2024-02-05T07:25:09.546217+0000","last_undegraded":"2024-02-05T07:25:09.546217+0000","last_fullsized":"2024-02-05T07:25:09.546217+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:24:53.457167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:53.457167+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:08.339750+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":242,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.202256+0000","last_change":"2024-02-05T07:24:54.528959+0000","last_active":"2024-02-05T07:25:09.202256+0000","last_peered":"2024-02-05T07:25:09.202256+0000","last_clean":"2024-02-05T07:25:09.202256+0000","last_became_active":"2024-02-05T07:24:34.209580+0000","last_became_peered":"2024-02-05T07:24:34.209580+0000","last_unstale":"2024-02-05T07:25:09.202256+0000","last_undegraded":"2024-02-05T07:25:09.202256+0000","last_fullsized":"2024-02-05T07:25:09.202256+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:54.528848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:54.528848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:13.804773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":50,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858141+0000","last_change":"2024-02-05T07:24:34.242587+0000","last_active":"2024-02-05T07:25:09.858141+0000","last_peered":"2024-02-05T07:25:09.858141+0000","last_clean":"2024-02-05T07:25:09.858141+0000","last_became_active":"2024-02-05T07:24:34.240115+0000","last_became_peered":"2024-02-05T07:24:34.240115+0000","last_unstale":"2024-02-05T07:25:09.858141+0000","last_undegraded":"2024-02-05T07:25:09.858141+0000","last_fullsized":"2024-02-05T07:25:09.858141+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:26.202131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:26.202131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":301,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204891+0000","last_change":"2024-02-05T07:22:55.794256+0000","last_active":"2024-02-05T07:25:09.204891+0000","last_peered":"2024-02-05T07:25:09.204891+0000","last_clean":"2024-02-05T07:25:09.204891+0000","last_became_active":"2024-02-05T07:18:55.466164+0000","last_became_peered":"2024-02-05T07:18:55.466164+0000","last_unstale":"2024-02-05T07:25:09.204891+0000","last_undegraded":"2024-02-05T07:25:09.204891+0000","last_fullsized":"2024-02-05T07:25:09.204891+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:22:55.794149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:22:55.794149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444645+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":92,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858482+0000","last_change":"2024-02-05T07:24:46.018438+0000","last_active":"2024-02-05T07:25:09.858482+0000","last_peered":"2024-02-05T07:25:09.858482+0000","last_clean":"2024-02-05T07:25:09.858482+0000","last_became_active":"2024-02-05T07:24:34.237056+0000","last_became_peered":"2024-02-05T07:24:34.237056+0000","last_unstale":"2024-02-05T07:25:09.858482+0000","last_undegraded":"2024-02-05T07:25:09.858482+0000","last_fullsized":"2024-02-05T07:25:09.858482+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:24:46.018340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:46.018340+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:12.154764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":316,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.201830+0000","last_change":"2024-02-05T07:23:47.646441+0000","last_active":"2024-02-05T07:25:09.201830+0000","last_peered":"2024-02-05T07:25:09.201830+0000","last_clean":"2024-02-05T07:25:09.201830+0000","last_became_active":"2024-02-05T07:23:47.610436+0000","last_became_peered":"2024-02-05T07:23:47.610436+0000","last_unstale":"2024-02-05T07:25:09.201830+0000","last_undegraded":"2024-02-05T07:25:09.201830+0000","last_fullsized":"2024-02-05T07:25:09.201830+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:21:42.616129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:42.616129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444546+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":385,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.549919+0000","last_change":"2024-02-05T07:21:39.441651+0000","last_active":"2024-02-05T07:25:09.549919+0000","last_peered":"2024-02-05T07:25:09.549919+0000","last_clean":"2024-02-05T07:25:09.549919+0000","last_became_active":"2024-02-05T07:18:53.461011+0000","last_became_peered":"2024-02-05T07:18:53.461011+0000","last_unstale":"2024-02-05T07:25:09.549919+0000","last_undegraded":"2024-02-05T07:25:09.549919+0000","last_fullsized":"2024-02-05T07:25:09.549919+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:21:39.441520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.441520+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":294,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857997+0000","last_change":"2024-02-05T07:21:39.490602+0000","last_active":"2024-02-05T07:25:09.857997+0000","last_peered":"2024-02-05T07:25:09.857997+0000","last_clean":"2024-02-05T07:25:09.857997+0000","last_became_active":"2024-02-05T07:18:55.476071+0000","last_became_peered":"2024-02-05T07:18:55.476071+0000","last_unstale":"2024-02-05T07:25:09.857997+0000","last_undegraded":"2024-02-05T07:25:09.857997+0000","last_fullsized":"2024-02-05T07:25:09.857997+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.490479+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.048853+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":160,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204774+0000","last_change":"2024-02-05T07:24:46.667850+0000","last_active":"2024-02-05T07:25:09.204774+0000","last_peered":"2024-02-05T07:25:09.204774+0000","last_clean":"2024-02-05T07:25:09.204774+0000","last_became_active":"2024-02-05T07:23:47.610303+0000","last_became_peered":"2024-02-05T07:23:47.610303+0000","last_unstale":"2024-02-05T07:25:09.204774+0000","last_undegraded":"2024-02-05T07:25:09.204774+0000","last_fullsized":"2024-02-05T07:25:09.204774+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:24:46.667757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:46.667757+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:49.251381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856889+0000","last_change":"2024-02-05T07:24:10.630154+0000","last_active":"2024-02-05T07:25:09.856889+0000","last_peered":"2024-02-05T07:25:09.856889+0000","last_clean":"2024-02-05T07:25:09.856889+0000","last_became_active":"2024-02-05T07:18:49.427837+0000","last_became_peered":"2024-02-05T07:18:49.427837+0000","last_unstale":"2024-02-05T07:25:09.856889+0000","last_undegraded":"2024-02-05T07:25:09.856889+0000","last_fullsized":"2024-02-05T07:25:09.856889+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:10.630020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:10.630020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:13.542894+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":302,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857119+0000","last_change":"2024-02-05T07:23:47.606077+0000","last_active":"2024-02-05T07:25:09.857119+0000","last_peered":"2024-02-05T07:25:09.857119+0000","last_clean":"2024-02-05T07:25:09.857119+0000","last_became_active":"2024-02-05T07:23:47.605523+0000","last_became_peered":"2024-02-05T07:23:47.605523+0000","last_unstale":"2024-02-05T07:25:09.857119+0000","last_undegraded":"2024-02-05T07:25:09.857119+0000","last_fullsized":"2024-02-05T07:25:09.857119+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:41.800067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:41.800067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019537+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":106,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.546307+0000","last_change":"2024-02-05T07:24:58.410795+0000","last_active":"2024-02-05T07:25:09.546307+0000","last_peered":"2024-02-05T07:25:09.546307+0000","last_clean":"2024-02-05T07:25:09.546307+0000","last_became_active":"2024-02-05T07:23:47.609917+0000","last_became_peered":"2024-02-05T07:23:47.609917+0000","last_unstale":"2024-02-05T07:25:09.546307+0000","last_undegraded":"2024-02-05T07:25:09.546307+0000","last_fullsized":"2024-02-05T07:25:09.546307+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:58.410667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:58.410667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:28.003988+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":347,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.685754+0000","last_change":"2024-02-05T07:23:47.657951+0000","last_active":"2024-02-05T07:25:09.685754+0000","last_peered":"2024-02-05T07:25:09.685754+0000","last_clean":"2024-02-05T07:25:09.685754+0000","last_became_active":"2024-02-05T07:23:47.619716+0000","last_became_peered":"2024-02-05T07:23:47.619716+0000","last_unstale":"2024-02-05T07:25:09.685754+0000","last_undegraded":"2024-02-05T07:25:09.685754+0000","last_fullsized":"2024-02-05T07:25:09.685754+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:21:36.561503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:36.561503+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:11.721207+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":50,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857202+0000","last_change":"2024-02-05T07:24:34.234686+0000","last_active":"2024-02-05T07:25:09.857202+0000","last_peered":"2024-02-05T07:25:09.857202+0000","last_clean":"2024-02-05T07:25:09.857202+0000","last_became_active":"2024-02-05T07:24:34.234199+0000","last_became_peered":"2024-02-05T07:24:34.234199+0000","last_unstale":"2024-02-05T07:25:09.857202+0000","last_undegraded":"2024-02-05T07:25:09.857202+0000","last_fullsized":"2024-02-05T07:25:09.857202+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:32.192372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:32.192372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":297,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204802+0000","last_change":"2024-02-05T07:22:58.849105+0000","last_active":"2024-02-05T07:25:09.204802+0000","last_peered":"2024-02-05T07:25:09.204802+0000","last_clean":"2024-02-05T07:25:09.204802+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:25:09.204802+0000","last_undegraded":"2024-02-05T07:25:09.204802+0000","last_fullsized":"2024-02-05T07:25:09.204802+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:22:58.846677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:22:58.846677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444677+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":293,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.858387+0000","last_change":"2024-02-05T07:21:38.839453+0000","last_active":"2024-02-05T07:25:09.858387+0000","last_peered":"2024-02-05T07:25:09.858387+0000","last_clean":"2024-02-05T07:25:09.858387+0000","last_became_active":"2024-02-05T07:18:55.505678+0000","last_became_peered":"2024-02-05T07:18:55.505678+0000","last_unstale":"2024-02-05T07:25:09.858387+0000","last_undegraded":"2024-02-05T07:25:09.858387+0000","last_fullsized":"2024-02-05T07:25:09.858387+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:38.839336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:38.839336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":322,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856508+0000","last_change":"2024-02-05T07:24:53.834758+0000","last_active":"2024-02-05T07:25:09.856508+0000","last_peered":"2024-02-05T07:25:09.856508+0000","last_clean":"2024-02-05T07:25:09.856508+0000","last_became_active":"2024-02-05T07:23:47.603918+0000","last_became_peered":"2024-02-05T07:23:47.603918+0000","last_unstale":"2024-02-05T07:25:09.856508+0000","last_undegraded":"2024-02-05T07:25:09.856508+0000","last_fullsized":"2024-02-05T07:25:09.856508+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:24:53.834622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:53.834622+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:57.247218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":118,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856735+0000","last_change":"2024-02-05T07:23:27.438105+0000","last_active":"2024-02-05T07:25:09.856735+0000","last_peered":"2024-02-05T07:25:09.856735+0000","last_clean":"2024-02-05T07:25:09.856735+0000","last_became_active":"2024-02-05T07:23:27.437776+0000","last_became_peered":"2024-02-05T07:23:27.437776+0000","last_unstale":"2024-02-05T07:25:09.856735+0000","last_undegraded":"2024-02-05T07:25:09.856735+0000","last_fullsized":"2024-02-05T07:25:09.856735+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:39.936643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.936643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":296,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857065+0000","last_change":"2024-02-05T07:24:16.310748+0000","last_active":"2024-02-05T07:25:09.857065+0000","last_peered":"2024-02-05T07:25:09.857065+0000","last_clean":"2024-02-05T07:25:09.857065+0000","last_became_active":"2024-02-05T07:18:53.456881+0000","last_became_peered":"2024-02-05T07:18:53.456881+0000","last_unstale":"2024-02-05T07:25:09.857065+0000","last_undegraded":"2024-02-05T07:25:09.857065+0000","last_fullsized":"2024-02-05T07:25:09.857065+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:16.310666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:16.310666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:30.390505+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":290,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204298+0000","last_change":"2024-02-05T07:23:47.597763+0000","last_active":"2024-02-05T07:25:09.204298+0000","last_peered":"2024-02-05T07:25:09.204298+0000","last_clean":"2024-02-05T07:25:09.204298+0000","last_became_active":"2024-02-05T07:23:47.596885+0000","last_became_peered":"2024-02-05T07:23:47.596885+0000","last_unstale":"2024-02-05T07:25:09.204298+0000","last_undegraded":"2024-02-05T07:25:09.204298+0000","last_fullsized":"2024-02-05T07:25:09.204298+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:15.751084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:15.751084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:09.768122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"89'13","reported_seq":336,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856762+0000","last_change":"2024-02-05T07:23:47.643672+0000","last_active":"2024-02-05T07:25:09.856762+0000","last_peered":"2024-02-05T07:25:09.856762+0000","last_clean":"2024-02-05T07:25:09.856762+0000","last_became_active":"2024-02-05T07:23:47.618422+0000","last_became_peered":"2024-02-05T07:23:47.618422+0000","last_unstale":"2024-02-05T07:25:09.856762+0000","last_undegraded":"2024-02-05T07:25:09.856762+0000","last_fullsized":"2024-02-05T07:25:09.856762+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-05T07:21:34.888266+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:21:34.888266+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019438+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":118,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857913+0000","last_change":"2024-02-05T07:23:27.418016+0000","last_active":"2024-02-05T07:25:09.857913+0000","last_peered":"2024-02-05T07:25:09.857913+0000","last_clean":"2024-02-05T07:25:09.857913+0000","last_became_active":"2024-02-05T07:23:27.417717+0000","last_became_peered":"2024-02-05T07:23:27.417717+0000","last_unstale":"2024-02-05T07:25:09.857913+0000","last_undegraded":"2024-02-05T07:25:09.857913+0000","last_fullsized":"2024-02-05T07:25:09.857913+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:47.776379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:47.776379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":288,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.204255+0000","last_change":"2024-02-05T07:21:19.716500+0000","last_active":"2024-02-05T07:25:09.204255+0000","last_peered":"2024-02-05T07:25:09.204255+0000","last_clean":"2024-02-05T07:25:09.204255+0000","last_became_active":"2024-02-05T07:18:53.464520+0000","last_became_peered":"2024-02-05T07:18:53.464520+0000","last_unstale":"2024-02-05T07:25:09.204255+0000","last_undegraded":"2024-02-05T07:25:09.204255+0000","last_fullsized":"2024-02-05T07:25:09.204255+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:19.716375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:19.716375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:09.768104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":58,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.202015+0000","last_change":"2024-02-05T07:24:49.612194+0000","last_active":"2024-02-05T07:25:09.202015+0000","last_peered":"2024-02-05T07:25:09.202015+0000","last_clean":"2024-02-05T07:25:09.202015+0000","last_became_active":"2024-02-05T07:24:34.230537+0000","last_became_peered":"2024-02-05T07:24:34.230537+0000","last_unstale":"2024-02-05T07:25:09.202015+0000","last_undegraded":"2024-02-05T07:25:09.202015+0000","last_fullsized":"2024-02-05T07:25:09.202015+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:49.612116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.612116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:07.615104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":158,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.685495+0000","last_change":"2024-02-05T07:23:47.645884+0000","last_active":"2024-02-05T07:25:09.685495+0000","last_peered":"2024-02-05T07:25:09.685495+0000","last_clean":"2024-02-05T07:25:09.685495+0000","last_became_active":"2024-02-05T07:23:47.621081+0000","last_became_peered":"2024-02-05T07:23:47.621081+0000","last_unstale":"2024-02-05T07:25:09.685495+0000","last_undegraded":"2024-02-05T07:25:09.685495+0000","last_fullsized":"2024-02-05T07:25:09.685495+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:21:27.141800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:27.141800+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:11.720979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":106,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.546070+0000","last_change":"2024-02-05T07:24:15.235343+0000","last_active":"2024-02-05T07:25:09.546070+0000","last_peered":"2024-02-05T07:25:09.546070+0000","last_clean":"2024-02-05T07:25:09.546070+0000","last_became_active":"2024-02-05T07:23:47.610737+0000","last_became_peered":"2024-02-05T07:23:47.610737+0000","last_unstale":"2024-02-05T07:25:09.546070+0000","last_undegraded":"2024-02-05T07:25:09.546070+0000","last_fullsized":"2024-02-05T07:25:09.546070+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:15.235232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:15.235232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:43.769610+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856105+0000","last_change":"2024-02-05T07:24:51.823741+0000","last_active":"2024-02-05T07:25:09.856105+0000","last_peered":"2024-02-05T07:25:09.856105+0000","last_clean":"2024-02-05T07:25:09.856105+0000","last_became_active":"2024-02-05T07:18:53.461506+0000","last_became_peered":"2024-02-05T07:18:53.461506+0000","last_unstale":"2024-02-05T07:25:09.856105+0000","last_undegraded":"2024-02-05T07:25:09.856105+0000","last_fullsized":"2024-02-05T07:25:09.856105+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:51.823640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:51.823640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:18.329269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":106,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.857026+0000","last_change":"2024-02-05T07:24:09.290070+0000","last_active":"2024-02-05T07:25:09.857026+0000","last_peered":"2024-02-05T07:25:09.857026+0000","last_clean":"2024-02-05T07:25:09.857026+0000","last_became_active":"2024-02-05T07:23:47.609992+0000","last_became_peered":"2024-02-05T07:23:47.609992+0000","last_unstale":"2024-02-05T07:25:09.857026+0000","last_undegraded":"2024-02-05T07:25:09.857026+0000","last_fullsized":"2024-02-05T07:25:09.857026+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:09.289989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:09.289989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:17.419052+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":316,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856590+0000","last_change":"2024-02-05T07:23:15.477701+0000","last_active":"2024-02-05T07:25:09.856590+0000","last_peered":"2024-02-05T07:25:09.856590+0000","last_clean":"2024-02-05T07:25:09.856590+0000","last_became_active":"2024-02-05T07:18:51.448924+0000","last_became_peered":"2024-02-05T07:18:51.448924+0000","last_unstale":"2024-02-05T07:25:09.856590+0000","last_undegraded":"2024-02-05T07:25:09.856590+0000","last_fullsized":"2024-02-05T07:25:09.856590+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:23:15.477555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:15.477555+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:14.019706+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":58,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.686013+0000","last_change":"2024-02-05T07:24:56.703719+0000","last_active":"2024-02-05T07:25:09.686013+0000","last_peered":"2024-02-05T07:25:09.686013+0000","last_clean":"2024-02-05T07:25:09.686013+0000","last_became_active":"2024-02-05T07:24:34.229294+0000","last_became_peered":"2024-02-05T07:24:34.229294+0000","last_unstale":"2024-02-05T07:25:09.686013+0000","last_undegraded":"2024-02-05T07:25:09.686013+0000","last_fullsized":"2024-02-05T07:25:09.686013+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:56.703642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:56.703642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:13.419627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":297,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.201955+0000","last_change":"2024-02-05T07:23:07.907068+0000","last_active":"2024-02-05T07:25:09.201955+0000","last_peered":"2024-02-05T07:25:09.201955+0000","last_clean":"2024-02-05T07:25:09.201955+0000","last_became_active":"2024-02-05T07:18:53.460820+0000","last_became_peered":"2024-02-05T07:18:53.460820+0000","last_unstale":"2024-02-05T07:25:09.201955+0000","last_undegraded":"2024-02-05T07:25:09.201955+0000","last_fullsized":"2024-02-05T07:25:09.201955+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:07.906963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:23:07.906963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:12.444797+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":292,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-02-05T07:25:09.856565+0000","last_change":"2024-02-05T07:24:06.379008+0000","last_active":"2024-02-05T07:25:09.856565+0000","last_peered":"2024-02-05T07:25:09.856565+0000","last_clean":"2024-02-05T07:25:09.856565+0000","last_became_active":"2024-02-05T07:18:55.466001+0000","last_became_peered":"2024-02-05T07:18:55.466001+0000","last_unstale":"2024-02-05T07:25:09.856565+0000","last_undegraded":"2024-02-05T07:25:09.856565+0000","last_fullsized":"2024-02-05T07:25:09.856565+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:06.378880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:06.378880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:22.043257+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":373,"num_read_kb":368,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":687,"num_read_kb":444,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":432,"ondisk_log_size":432,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037841,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":217532,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93527620,"statfs":{"total":95995035648,"available":95772282880,"internally_reserved":0,"allocated":1753088,"data_stored":1214923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":220985802},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1789999999999998,"5min":3.4740000000000002,"15min":1.488},"min":{"1min":0.40400000000000003,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":24.856999999999999,"5min":63.039000000000001,"15min":63.039000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":3.6379999999999999,"5min":3.4380000000000002,"15min":1.486},"min":{"1min":0.33000000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":24.794,"5min":65.296000000000006,"15min":65.296000000000006},"last":0.496}]},{"osd":1,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":3.298,"5min":1.9630000000000001,"15min":1.004},"min":{"1min":0.38400000000000001,"5min":0.222,"15min":0.222},"max":{"1min":25.774999999999999,"5min":32.127000000000002,"15min":32.127000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":3.1219999999999999,"5min":2.0470000000000002,"15min":1.0509999999999999},"min":{"1min":0.32500000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":26.222999999999999,"5min":32.173999999999999,"15min":32.173999999999999},"last":0.46899999999999997}]},{"osd":2,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":2.7010000000000001,"5min":2.2519999999999998,"15min":1.1000000000000001},"min":{"1min":0.34899999999999998,"5min":0.249,"15min":0.249},"max":{"1min":25.805,"5min":51.127000000000002,"15min":51.127000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":2.6139999999999999,"5min":2.5870000000000002,"15min":1.2070000000000001},"min":{"1min":0.30199999999999999,"5min":0.246,"15min":0.246},"max":{"1min":24.579999999999998,"5min":51.113999999999997,"15min":51.113999999999997},"last":0.61899999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.843,"15min":0.96999999999999997},"min":{"1min":0.33600000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.6399999999999997,"5min":58.341999999999999,"15min":58.341999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.351,"5min":1.7689999999999999,"15min":0.95899999999999996},"min":{"1min":0.35499999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":6.5880000000000001,"5min":57.765000000000001,"15min":57.765000000000001},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.56999999999999995,"15min":0.48199999999999998},"min":{"1min":0.28199999999999997,"5min":0.25,"15min":0.25},"max":{"1min":4.8659999999999997,"5min":4.8659999999999997,"15min":4.8659999999999997},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.56699999999999995,"15min":0.48099999999999998},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.577,"5min":4.577,"15min":4.577},"last":0.59399999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":0.67300000000000004,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":6.9020000000000001,"5min":6.9020000000000001,"15min":6.9020000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.2,"5min":0.70299999999999996,"15min":0.56499999999999995},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":6.3449999999999998,"5min":6.3449999999999998,"15min":6.3449999999999998},"last":0.57799999999999996}]},{"osd":6,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":0.67600000000000005,"15min":0.55600000000000005},"min":{"1min":0.33300000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.9630000000000001,"5min":4.9630000000000001,"15min":4.9630000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":1,"5min":0.64900000000000002,"15min":0.55700000000000005},"min":{"1min":0.35699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":5.5179999999999998,"5min":5.5179999999999998,"15min":5.5179999999999998},"last":0.52400000000000002}]}]},{"osd":6,"up_from":52,"seq":223338299479,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":195340,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93549812,"statfs":{"total":95995035648,"available":95795007488,"internally_reserved":0,"allocated":1769472,"data_stored":1209510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":198244806},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:24:17 2024","interfaces":[{"interface":"back","average":{"1min":3.141,"5min":3.0550000000000002,"15min":1.385},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":22.024000000000001,"5min":57.289999999999999,"15min":57.289999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":3.0019999999999998,"5min":2.8999999999999999,"15min":1.319},"min":{"1min":0.28999999999999998,"5min":0.22,"15min":0.22},"max":{"1min":22.161000000000001,"5min":56.302,"15min":56.302},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Feb 5 07:24:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":0.874,"15min":0.65800000000000003},"min":{"1min":0.26300000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":6.2439999999999998,"5min":6.2439999999999998,"15min":6.2439999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.117,"5min":0.83099999999999996,"15min":0.64600000000000002},"min":{"1min":0.30499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":6.915,"5min":6.915,"15min":6.915},"last":0.51800000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:24:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.5529999999999999,"15min":0.90400000000000003},"min":{"1min":0.27500000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.423,"5min":32.509999999999998,"15min":32.509999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.8380000000000001,"5min":1.7929999999999999,"15min":0.95199999999999996},"min":{"1min":0.33400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":8.9979999999999993,"5min":32.536000000000001,"15min":32.536000000000001},"last":0.50800000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:24:17 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":2.0179999999999998,"15min":1.046},"min":{"1min":0.40600000000000003,"5min":0.22,"15min":0.22},"max":{"1min":20.503,"5min":36.976999999999997,"15min":36.976999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":2.1000000000000001,"5min":1.843,"15min":0.99099999999999999},"min":{"1min":0.33400000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":20.491,"5min":34.222999999999999,"15min":34.222999999999999},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.55700000000000005,"15min":0.45300000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.5609999999999999,"5min":4.5609999999999999,"15min":4.5609999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.55800000000000005,"15min":0.48599999999999999},"min":{"1min":0.31900000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.0060000000000002,"5min":5.0060000000000002,"15min":5.0060000000000002},"last":0.44400000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:24:17 2024","interfaces":[{"interface":"back","average":{"1min":2.3540000000000001,"5min":0.85299999999999998,"15min":0.58099999999999996},"min":{"1min":0.32100000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":27.021999999999998,"5min":27.021999999999998,"15min":27.021999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":2.472,"5min":0.90400000000000003,"15min":0.60299999999999998},"min":{"1min":0.35299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":23.611999999999998,"5min":23.611999999999998,"15min":23.611999999999998},"last":0.52800000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.61899999999999999,"15min":0.53900000000000003},"min":{"1min":0.29599999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.2130000000000001,"5min":4.2130000000000001,"15min":4.2130000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":2.5550000000000002,"5min":0.94799999999999995,"15min":0.67600000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":25.207000000000001,"5min":25.207000000000001,"15min":25.207000000000001},"last":0.38500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":199392,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93545760,"statfs":{"total":95995035648,"available":95790858240,"internally_reserved":0,"allocated":1658880,"data_stored":1084598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":1.9970000000000001,"5min":3.7330000000000001,"15min":1.4710000000000001},"min":{"1min":0.223,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":6.1299999999999999,"5min":45.393000000000001,"15min":45.393000000000001},"last":0.372},{"interface":"front","average":{"1min":1.863,"5min":3.6960000000000002,"15min":1.458},"min":{"1min":0.246,"5min":0.23000000000000001,"15min":0.22800000000000001},"max":{"1min":8.7330000000000005,"5min":45.031999999999996,"15min":45.031999999999996},"last":0.42399999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:24:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":2.2440000000000002,"15min":1.0449999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.0030000000000001,"5min":39.259999999999998,"15min":39.259999999999998},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":2.093,"15min":1.381},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.532,"5min":40.972000000000001,"15min":40.972000000000001},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:24:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":2.2770000000000001,"15min":1.0780000000000001},"min":{"1min":0.254,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.1699999999999999,"5min":43.823,"15min":43.823},"last":0.504},{"interface":"front","average":{"1min":3.1070000000000002,"5min":2.4199999999999999,"15min":1.1359999999999999},"min":{"1min":0.32200000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":7.617,"5min":39.314999999999998,"15min":39.314999999999998},"last":0.34300000000000003}]},{"osd":4,"last update":"Mon Feb 5 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":5.327,"5min":1.9530000000000001,"15min":1.044},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":61.334000000000003,"5min":61.334000000000003,"15min":61.334000000000003},"last":0.93400000000000005},{"interface":"front","average":{"1min":5.5339999999999998,"5min":2.0699999999999998,"15min":1.1879999999999999},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":61.786000000000001,"5min":61.786000000000001,"15min":61.786000000000001},"last":0.751}]},{"osd":5,"last update":"Mon Feb 5 07:24:55 2024","interfaces":[{"interface":"back","average":{"1min":4.4900000000000002,"5min":1.9119999999999999,"15min":1.0509999999999999},"min":{"1min":0.40500000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":66.078000000000003,"5min":66.078000000000003,"15min":66.078000000000003},"last":0.89500000000000002},{"interface":"front","average":{"1min":4.4480000000000004,"5min":2.0539999999999998,"15min":1.202},"min":{"1min":0.40799999999999997,"5min":0.307,"15min":0.307},"max":{"1min":66.194000000000003,"5min":66.194000000000003,"15min":66.194000000000003},"last":0.54500000000000004}]},{"osd":6,"last update":"Mon Feb 5 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.357,"15min":0.91100000000000003},"min":{"1min":0.34699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":5.2930000000000001,"5min":18.456,"15min":18.456},"last":1.663},{"interface":"front","average":{"1min":2.0920000000000001,"5min":1.337,"15min":0.90200000000000002},"min":{"1min":0.376,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.9969999999999999,"5min":5.9930000000000003,"15min":5.9930000000000003},"last":0.81000000000000005}]},{"osd":7,"last update":"Mon Feb 5 07:24:49 2024","interfaces":[{"interface":"back","average":{"1min":4.8159999999999998,"5min":2.1440000000000001,"15min":1.0980000000000001},"min":{"1min":0.371,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":66.703999999999994,"5min":66.703999999999994,"15min":66.703999999999994},"last":1.7190000000000001},{"interface":"front","average":{"1min":4.931,"5min":2.0800000000000001,"15min":1.0880000000000001},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":61.594999999999999,"5min":61.594999999999999,"15min":61.594999999999999},"last":0.70999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":231428,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93513724,"statfs":{"total":95995035648,"available":95758053376,"internally_reserved":0,"allocated":1761280,"data_stored":1215064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:24:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.645,"15min":0.755},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.349,"5min":15.731999999999999,"15min":15.731999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.8740000000000001,"5min":1.903,"15min":0.84999999999999998},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":9.3469999999999995,"5min":32.396000000000001,"15min":32.396000000000001},"last":0.44600000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":5.3499999999999996,"5min":2.665,"15min":1.175},"min":{"1min":0.28899999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":65.903999999999996,"5min":65.903999999999996,"15min":65.903999999999996},"last":1.121},{"interface":"front","average":{"1min":5.5010000000000003,"5min":2.5950000000000002,"15min":1.147},"min":{"1min":0.186,"5min":0.186,"15min":0.182},"max":{"1min":65.694000000000003,"5min":65.694000000000003,"15min":65.694000000000003},"last":1.0960000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:25:06 2024","interfaces":[{"interface":"back","average":{"1min":1.9530000000000001,"5min":3.1899999999999999,"15min":1.3500000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":25.795000000000002,"5min":65.155000000000001,"15min":65.155000000000001},"last":1.147},{"interface":"front","average":{"1min":2.0600000000000001,"5min":3.423,"15min":1.419},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":28.658000000000001,"5min":65.114000000000004,"15min":65.114000000000004},"last":0.48199999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:24:29 2024","interfaces":[{"interface":"back","average":{"1min":4.2270000000000003,"5min":2.0129999999999999,"15min":1.0389999999999999},"min":{"1min":0.35099999999999998,"5min":0.31,"15min":0.31},"max":{"1min":65.188000000000002,"5min":65.188000000000002,"15min":65.188000000000002},"last":0.73599999999999999},{"interface":"front","average":{"1min":4.2530000000000001,"5min":2.1269999999999998,"15min":1.075},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":64.873000000000005,"5min":64.873000000000005,"15min":64.873000000000005},"last":1.171}]},{"osd":5,"last update":"Mon Feb 5 07:25:06 2024","interfaces":[{"interface":"back","average":{"1min":1.9399999999999999,"5min":2.1659999999999999,"15min":1.0920000000000001},"min":{"1min":0.32200000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":25.747,"5min":65.555999999999997,"15min":65.555999999999997},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.786,"5min":2.3109999999999999,"15min":1.165},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":25.567,"5min":64.936999999999998,"15min":64.936999999999998},"last":1.228}]},{"osd":6,"last update":"Mon Feb 5 07:24:24 2024","interfaces":[{"interface":"back","average":{"1min":4.4370000000000003,"5min":1.982,"15min":1.0580000000000001},"min":{"1min":0.34799999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":65.215000000000003,"5min":65.215000000000003,"15min":65.215000000000003},"last":1.2130000000000001},{"interface":"front","average":{"1min":4.5730000000000004,"5min":2.1230000000000002,"15min":1.135},"min":{"1min":0.376,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":65.230000000000004,"5min":65.230000000000004,"15min":65.230000000000004},"last":0.80800000000000005}]},{"osd":7,"last update":"Mon Feb 5 07:24:40 2024","interfaces":[{"interface":"back","average":{"1min":5.3840000000000003,"5min":2.4049999999999998,"15min":1.206},"min":{"1min":0.44400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":65.263000000000005,"5min":65.263000000000005,"15min":65.263000000000005},"last":0.77600000000000002},{"interface":"front","average":{"1min":5.4509999999999996,"5min":2.3610000000000002,"15min":1.216},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":64.906000000000006,"5min":64.906000000000006,"15min":64.906000000000006},"last":1.2010000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411434,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":275244,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93469908,"statfs":{"total":95995035648,"available":95713185792,"internally_reserved":0,"allocated":1146880,"data_stored":618551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:24:49 2024","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":2.4220000000000002,"15min":1.0549999999999999},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":8.9139999999999997,"5min":67.418999999999997,"15min":67.418999999999997},"last":0.376},{"interface":"front","average":{"1min":1.7430000000000001,"5min":2.7280000000000002,"15min":1.169},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.782,"5min":68.171000000000006,"15min":68.171000000000006},"last":0.30399999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:24:49 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.337,"15min":0.70999999999999996},"min":{"1min":0.19800000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":4.9420000000000002,"5min":12.609,"15min":12.609},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.276,"5min":1.49,"15min":0.78400000000000003},"min":{"1min":0.23100000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":5.4580000000000002,"5min":9.641,"15min":9.641},"last":0.42799999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:24:10 2024","interfaces":[{"interface":"back","average":{"1min":2.4169999999999998,"5min":1.7450000000000001,"15min":0.89900000000000002},"min":{"1min":0.249,"5min":0.214,"15min":0.214},"max":{"1min":7.3929999999999998,"5min":14.756,"15min":14.756},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.2639999999999998,"5min":1.821,"15min":0.91900000000000004},"min":{"1min":0.28699999999999998,"5min":0.193,"15min":0.193},"max":{"1min":7.8099999999999996,"5min":9.6669999999999998,"15min":9.6669999999999998},"last":0.46200000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:24:32 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.2050000000000001,"15min":0.75800000000000001},"min":{"1min":0.33500000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":4.2130000000000001,"5min":7.29,"15min":7.29},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.236,"15min":0.78500000000000003},"min":{"1min":0.32200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":5.3499999999999996,"5min":8.5199999999999996,"15min":8.5199999999999996},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Feb 5 07:24:53 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.2969999999999999,"15min":0.83099999999999996},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":5.4100000000000001,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.62},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.167,"15min":0.79600000000000004},"min":{"1min":0.42399999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":5.1710000000000003,"5min":7.7279999999999998,"15min":7.7279999999999998},"last":0.60399999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:24:21 2024","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.286,"15min":0.94299999999999995},"min":{"1min":0.376,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.9119999999999999,"5min":7.6829999999999998,"15min":7.6829999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.464,"5min":1.349,"15min":0.85899999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":4.1769999999999996,"5min":7.3150000000000004,"15min":7.3150000000000004},"last":0.58099999999999996}]},{"osd":7,"last update":"Mon Feb 5 07:24:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.3260000000000001,"15min":0.90800000000000003},"min":{"1min":0.375,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.1130000000000004,"5min":7.2590000000000003,"15min":7.2590000000000003},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.2569999999999999,"15min":0.84699999999999998},"min":{"1min":0.40899999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":5.9450000000000003,"5min":7.7619999999999996,"15min":7.7619999999999996},"last":0.51200000000000001}]}]},{"osd":3,"up_from":0,"seq":111669149797,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953568,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":140604,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93604548,"statfs":{"total":95995035648,"available":95851057152,"internally_reserved":0,"allocated":1163264,"data_stored":623999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:24:31 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.6759999999999999,"15min":0.91100000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.7280000000000002,"5min":67.719999999999999,"15min":67.719999999999999},"last":0.497},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.9410000000000001,"15min":1.0149999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.3980000000000001,"5min":66.882000000000005,"15min":66.882000000000005},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.85799999999999998,"15min":0.63400000000000001},"min":{"1min":0.247,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":3.464,"5min":5.5709999999999997,"15min":5.5709999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.78100000000000003,"15min":0.629},"min":{"1min":0.32300000000000001,"5min":0.254,"15min":0.254},"max":{"1min":2.3740000000000001,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":0.67100000000000004}]},{"osd":2,"last update":"Mon Feb 5 07:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.5529999999999999,"15min":0.877},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.052,"5min":67.144999999999996,"15min":67.144999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.583,"15min":0.88700000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.9369999999999998,"5min":66.959999999999994,"15min":66.959999999999994},"last":0.33800000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.86199999999999999,"15min":0.66100000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.7709999999999999,"5min":8.8019999999999996,"15min":8.8019999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.137,"5min":1.071,"15min":0.72899999999999998},"min":{"1min":0.29699999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":5.3449999999999998,"5min":10.106,"15min":10.106},"last":0.40699999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.53400000000000003,"15min":0.42399999999999999},"min":{"1min":0.29199999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.8420000000000001,"5min":2.8420000000000001,"15min":2.8420000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.58999999999999997,"15min":0.45100000000000001},"min":{"1min":0.307,"5min":0.254,"15min":0.254},"max":{"1min":3.4409999999999998,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":0.47899999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:24:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":0.68999999999999995,"15min":0.502},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":6.8940000000000001,"5min":6.8940000000000001,"15min":6.8940000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.63500000000000001,"15min":0.47699999999999998},"min":{"1min":0.28999999999999998,"5min":0.27600000000000002,"15min":0.248},"max":{"1min":1.839,"5min":2.9209999999999998,"15min":2.9209999999999998},"last":0.57199999999999995}]},{"osd":7,"last update":"Mon Feb 5 07:24:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":0.69599999999999995,"15min":0.56000000000000005},"min":{"1min":0.33800000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.0640000000000001,"5min":6.0640000000000001,"15min":6.0640000000000001},"last":0.44},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.60599999999999998,"15min":0.52400000000000002},"min":{"1min":0.41299999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":0.63200000000000001}]}]},{"osd":5,"up_from":41,"seq":176093659227,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":162808,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93582344,"statfs":{"total":95995035648,"available":95828320256,"internally_reserved":0,"allocated":1159168,"data_stored":624460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:24:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":3.3580000000000001,"15min":1.472},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":5.2039999999999997,"5min":69.855999999999995,"15min":69.855999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":3.0590000000000002,"15min":1.3740000000000001},"min":{"1min":0.24299999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.9079999999999999,"5min":71.513999999999996,"15min":71.513999999999996},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Feb 5 07:24:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.2689999999999999,"15min":0.79700000000000004},"min":{"1min":0.36399999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.8680000000000003,"5min":11.839,"15min":11.839},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.008,"5min":0.88,"15min":0.64800000000000002},"min":{"1min":0.30499999999999999,"5min":0.253,"15min":0.253},"max":{"1min":6.8079999999999998,"5min":9.5950000000000006,"15min":9.5950000000000006},"last":0.42099999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:24:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":2.0350000000000001,"15min":1.03},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.8529999999999998,"5min":70.108000000000004,"15min":70.108000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.5780000000000001,"5min":2.1720000000000002,"15min":1.0900000000000001},"min":{"1min":0.40600000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.8390000000000004,"5min":72.817999999999998,"15min":72.817999999999998},"last":0.59699999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:24:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":2.2010000000000001,"15min":1.097},"min":{"1min":0.34899999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":6.8250000000000002,"5min":69.793000000000006,"15min":69.793000000000006},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.034,"5min":2.427,"15min":1.175},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.2320000000000002,"5min":69.820999999999998,"15min":69.820999999999998},"last":0.56799999999999995}]},{"osd":4,"last update":"Mon Feb 5 07:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.47299999999999998,"15min":0.39200000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.53800000000000003,"15min":0.43099999999999999},"min":{"1min":0.25,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.6360000000000001,"5min":3.6360000000000001,"15min":3.6360000000000001},"last":0.36499999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:24:26 2024","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":1.8149999999999999,"15min":0.86699999999999999},"min":{"1min":0.309,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":76.668999999999997,"5min":76.668999999999997,"15min":76.668999999999997},"last":0.32200000000000001},{"interface":"front","average":{"1min":4.056,"5min":1.8100000000000001,"15min":0.89800000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":76.698999999999998,"5min":76.698999999999998,"15min":76.698999999999998},"last":0.39900000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:24:43 2024","interfaces":[{"interface":"back","average":{"1min":6.742,"5min":3.6320000000000001,"15min":1.552},"min":{"1min":0.34999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":108.7,"5min":182.87799999999999,"15min":182.87799999999999},"last":0.496},{"interface":"front","average":{"1min":6.7080000000000002,"5min":3.6070000000000002,"15min":1.5469999999999999},"min":{"1min":0.33700000000000002,"5min":0.32200000000000001,"15min":0.30199999999999999},"max":{"1min":108.89400000000001,"5min":183.02199999999999,"15min":183.02199999999999},"last":0.38600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:25:15.966 INFO:tasks.thrashosds.thrasher:Setting 5.15 to [2, 0, 1] 2024-02-05T07:25:15.966 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.15', '2', '0', '1'] 2024-02-05T07:25:15.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pg-upmap 5.15 2 0 1 2024-02-05T07:25:16.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:16.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:16.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:16.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:16.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:16.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:16.211 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:16.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:16.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:16.336 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:16.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:16.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:16.461 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:16.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:16.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:16.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:16.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:16.717 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:16.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:16.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:16.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:16 smithi090 ceph-mon[33213]: pgmap v299: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:25:16.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:16 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-05T07:25:16.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:16 smithi090 ceph-mon[33213]: osdmap e209: 8 total, 8 up, 7 in 2024-02-05T07:25:16.798 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:16.798 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:16.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:16.847 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:16.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:16.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:16.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:16 smithi007 ceph-mon[29098]: pgmap v299: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:25:16.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:16 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-05T07:25:16.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:16 smithi007 ceph-mon[29098]: osdmap e209: 8 total, 8 up, 7 in 2024-02-05T07:25:16.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:16 smithi007 ceph-mon[37158]: pgmap v299: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:25:16.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:16 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-05T07:25:16.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:16 smithi007 ceph-mon[37158]: osdmap e209: 8 total, 8 up, 7 in 2024-02-05T07:25:16.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:16.976 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:16.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:17.078 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:17.103 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:17.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:17.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-02-05T07:25:17.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:17.231 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:17.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:17.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:17.272 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:17.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:17.356 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:17.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:17.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:17.482 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:17.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:17.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:17.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:17.607 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:17.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:17.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:17.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:17.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:17 smithi007 ceph-mon[29098]: osdmap e210: 8 total, 8 up, 7 in 2024-02-05T07:25:17.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:17 smithi007 ceph-mon[37158]: osdmap e210: 8 total, 8 up, 7 in 2024-02-05T07:25:17.737 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:17.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:17.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:17.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:17 smithi090 ceph-mon[33213]: osdmap e210: 8 total, 8 up, 7 in 2024-02-05T07:25:17.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:17.866 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:17.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:17.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:17.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:17.995 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:17.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:17.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:18.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:18.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:18.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:18.250 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:18.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:18.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:18.376 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:18.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:18.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:18.502 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:18.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:18.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:25:18.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:18.627 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:18.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:18.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:18.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:18.734 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:18 smithi007 ceph-mon[29098]: pgmap v302: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:25:18.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:18 smithi007 ceph-mon[29098]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:25:18.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:18 smithi007 ceph-mon[29098]: osdmap e211: 8 total, 8 up, 7 in 2024-02-05T07:25:18.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:18 smithi007 ceph-mon[37158]: pgmap v302: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:25:18.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:18 smithi007 ceph-mon[37158]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:25:18.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:18 smithi007 ceph-mon[37158]: osdmap e211: 8 total, 8 up, 7 in 2024-02-05T07:25:18.757 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:18.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:18.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:18.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:18 smithi090 ceph-mon[33213]: pgmap v302: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:25:18.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:18 smithi090 ceph-mon[33213]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:25:18.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:18 smithi090 ceph-mon[33213]: osdmap e211: 8 total, 8 up, 7 in 2024-02-05T07:25:18.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:18.887 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:18.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:18.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:19.013 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:19.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:19.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:19.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:19.139 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:19.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:19.241 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:19.268 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:19.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:19.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:19.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:19.398 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:19.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:19.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:19.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:19.524 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:19.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:19.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:19.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:19.653 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:19.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:19.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:19.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:19.779 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:19.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:19.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:19.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:19.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:19.906 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:19.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:20.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:20.009 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:20.035 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:20.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:20.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:20.137 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:20.161 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:20.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:20.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:20.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:20.299 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:20.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:20.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:20.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:20 smithi007 ceph-mon[29098]: pgmap v304: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:25:20.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:20 smithi007 ceph-mon[29098]: osdmap e212: 8 total, 8 up, 7 in 2024-02-05T07:25:20.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:20 smithi007 ceph-mon[37158]: pgmap v304: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:25:20.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:20 smithi007 ceph-mon[37158]: osdmap e212: 8 total, 8 up, 7 in 2024-02-05T07:25:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:20.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:20.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:20.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:20 smithi090 ceph-mon[33213]: pgmap v304: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:25:20.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:20 smithi090 ceph-mon[33213]: osdmap e212: 8 total, 8 up, 7 in 2024-02-05T07:25:20.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:25:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:25:20.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:20.558 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:20.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:20.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:20.586 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:25:20.605 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:25:20.624 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:25:20.644 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:25:20.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:20.665 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:25:20.684 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:20.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:20.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:20.685 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:25:20.706 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:25:20.726 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:25:20.728 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:25:20.728 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:25:20.728 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:25:20.728 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:25:20.728 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:25:20.728 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:25:20.728 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:25:20.728 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:25:20.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:20.814 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:20.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:20.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:20.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:20.940 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:20.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:21.041 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:21.069 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:21.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:21.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:21.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:21.198 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:21.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:21.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:21.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:21.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:21.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:21.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:21 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:21.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:21 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/661805516' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:21.416 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:21 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:21.416 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:21 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/661805516' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:21.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:21.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:21.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:21.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:21 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:21.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:21 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/661805516' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:21.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:21.579 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:21.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:21.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:21.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:21.707 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:21.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:21.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:21.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:21.835 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:21.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:21.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:21.962 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:21.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:21.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:22.064 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:22.088 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:22.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:22.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:22.214 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:22.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:22.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:22.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:22.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:22.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:22.470 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:22.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:22.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:22 smithi090 ceph-mon[33213]: pgmap v306: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 920 B/s rd, 0 op/s 2024-02-05T07:25:22.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:22 smithi007 ceph-mon[29098]: pgmap v306: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 920 B/s rd, 0 op/s 2024-02-05T07:25:22.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:22 smithi007 ceph-mon[37158]: pgmap v306: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 920 B/s rd, 0 op/s 2024-02-05T07:25:22.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:22.597 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:22.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:22.675 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:25:22.676 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:25:22.676 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 0 2024-02-05T07:25:22.676 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:25:22.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:22.721 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:22.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:22.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:22.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:22.850 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:22.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:22.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:22.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:22.977 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:22.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:23.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:23.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:23.107 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:23.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:23.208 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:23.233 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:23.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:23.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:23.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:23.359 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:23.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:23.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:25:23.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:23.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T07:25:23.488 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:23.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:23.590 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:23.618 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:23.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:23.709 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:23.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:23.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:23.745 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:23.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:23.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:23.871 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:25:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:25:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:25:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:23.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:23.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:23.973 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:23.999 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:23.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:24.080 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:24.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:24.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:24.126 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:24.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:24.222 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:25:24.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:24.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:24.253 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:24.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:24.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:24.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:24.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:24.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:24 smithi007 ceph-mon[29098]: pgmap v307: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 780 B/s rd, 0 op/s 2024-02-05T07:25:24.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:24 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3763123769' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:25:24.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:24 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:25:24.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:24 smithi007 ceph-mon[37158]: pgmap v307: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 780 B/s rd, 0 op/s 2024-02-05T07:25:24.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:24 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3763123769' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:25:24.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:24 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:25:24.481 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:24.506 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:24.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:24.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:24 smithi090 ceph-mon[33213]: pgmap v307: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 780 B/s rd, 0 op/s 2024-02-05T07:25:24.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:24 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3763123769' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:25:24.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:24 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:25:24.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:24.633 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:24.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:24.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:24.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:24.760 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:24.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:24.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:24.862 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:24.887 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:24.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:24.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:24.989 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:25.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:25.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:25.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:25.144 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:25.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:25.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:25.270 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:25.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:25.372 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:25.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:25.398 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:25.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:25.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:25.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:25.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:25 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:25:25.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:25 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:25:25.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:25 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:25:25.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:25 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:25:25.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:25 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:25:25.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:25 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:25:25.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:25 smithi090 ceph-mon[33213]: osdmap e213: 8 total, 8 up, 7 in 2024-02-05T07:25:25.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:25 smithi090 ceph-mon[33213]: 4.6 scrub starts 2024-02-05T07:25:25.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:25 smithi090 ceph-mon[33213]: 4.6 scrub ok 2024-02-05T07:25:25.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:25.649 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:25:25.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:25:25.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:25:25.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:25:25.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:25:25.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:25:25.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[29098]: osdmap e213: 8 total, 8 up, 7 in 2024-02-05T07:25:25.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[29098]: 4.6 scrub starts 2024-02-05T07:25:25.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[29098]: 4.6 scrub ok 2024-02-05T07:25:25.650 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:25:25.650 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:25:25.650 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:25:25.650 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:25:25.651 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:25:25.651 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:25:25.651 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[37158]: osdmap e213: 8 total, 8 up, 7 in 2024-02-05T07:25:25.651 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[37158]: 4.6 scrub starts 2024-02-05T07:25:25.651 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:25 smithi007 ceph-mon[37158]: 4.6 scrub ok 2024-02-05T07:25:25.651 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:25.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:25.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:25.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:25.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:25.781 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:25.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:25.882 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:25.906 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:25.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:25.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:26.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:26.036 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:26.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:26.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:26.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:26.164 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:26.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:26.243 INFO:teuthology.orchestra.run.smithi007.stderr:set 5.15 pg_upmap mapping to [2,0,1] 2024-02-05T07:25:26.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:26.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:26.290 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:26.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:26.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:26.419 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:26.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:26.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:26.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:26.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:26 smithi090 ceph-mon[33213]: pgmap v309: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 678 B/s rd, 0 op/s 2024-02-05T07:25:26.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:26 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/115259553' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.15", "id": [2, 0, 1]}]: dispatch 2024-02-05T07:25:26.546 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:26.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:26.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:26.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:26.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:26 smithi007 ceph-mon[29098]: pgmap v309: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 678 B/s rd, 0 op/s 2024-02-05T07:25:26.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:26 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/115259553' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.15", "id": [2, 0, 1]}]: dispatch 2024-02-05T07:25:26.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:26 smithi007 ceph-mon[37158]: pgmap v309: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 678 B/s rd, 0 op/s 2024-02-05T07:25:26.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:26 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/115259553' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.15", "id": [2, 0, 1]}]: dispatch 2024-02-05T07:25:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:26.675 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:26.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:26.776 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:26.800 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:26.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:26.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:26.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:26.929 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:26.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:26.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:26.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:25:27.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:27.056 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:27.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:27.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:27.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:27.183 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:27.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:27.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:27.308 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:27.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:27.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:27.337 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:27.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:27.433 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:27.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:27.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:27.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:27 smithi090 ceph-mon[33213]: 4.1a scrub starts 2024-02-05T07:25:27.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:27 smithi090 ceph-mon[33213]: 4.1a scrub ok 2024-02-05T07:25:27.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:27 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/115259553' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.15", "id": [2, 0, 1]}]': finished 2024-02-05T07:25:27.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:27 smithi090 ceph-mon[33213]: osdmap e214: 8 total, 8 up, 7 in 2024-02-05T07:25:27.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:27.560 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:27.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:27.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:27.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:27 smithi007 ceph-mon[29098]: 4.1a scrub starts 2024-02-05T07:25:27.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:27 smithi007 ceph-mon[29098]: 4.1a scrub ok 2024-02-05T07:25:27.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:27 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/115259553' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.15", "id": [2, 0, 1]}]': finished 2024-02-05T07:25:27.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:27 smithi007 ceph-mon[29098]: osdmap e214: 8 total, 8 up, 7 in 2024-02-05T07:25:27.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:27 smithi007 ceph-mon[37158]: 4.1a scrub starts 2024-02-05T07:25:27.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:27 smithi007 ceph-mon[37158]: 4.1a scrub ok 2024-02-05T07:25:27.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:27 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/115259553' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.15", "id": [2, 0, 1]}]': finished 2024-02-05T07:25:27.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:27 smithi007 ceph-mon[37158]: osdmap e214: 8 total, 8 up, 7 in 2024-02-05T07:25:27.684 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:27.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:27.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:27.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:27.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:27.812 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:27.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:27.913 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:27.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:27.937 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:27.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:28.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:28.064 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:28.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:28.165 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:28.191 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:28.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:28.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:28.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:28.317 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:28.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:28.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:28.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:28.444 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:28.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:28.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:28.478 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:28 smithi007 ceph-mon[29098]: pgmap v311: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:25:28.478 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:28 smithi007 ceph-mon[29098]: 4.e scrub starts 2024-02-05T07:25:28.479 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:28 smithi007 ceph-mon[29098]: 4.e scrub ok 2024-02-05T07:25:28.479 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:28 smithi007 ceph-mon[29098]: osdmap e215: 8 total, 8 up, 7 in 2024-02-05T07:25:28.479 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:28 smithi007 ceph-mon[37158]: pgmap v311: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:25:28.479 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:28 smithi007 ceph-mon[37158]: 4.e scrub starts 2024-02-05T07:25:28.479 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:28 smithi007 ceph-mon[37158]: 4.e scrub ok 2024-02-05T07:25:28.479 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:28 smithi007 ceph-mon[37158]: osdmap e215: 8 total, 8 up, 7 in 2024-02-05T07:25:28.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:28 smithi090 ceph-mon[33213]: pgmap v311: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:25:28.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:28 smithi090 ceph-mon[33213]: 4.e scrub starts 2024-02-05T07:25:28.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:28 smithi090 ceph-mon[33213]: 4.e scrub ok 2024-02-05T07:25:28.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:28 smithi090 ceph-mon[33213]: osdmap e215: 8 total, 8 up, 7 in 2024-02-05T07:25:28.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:28.573 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:28.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:28.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:28.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:28.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:28.700 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:28.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:28.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:28.826 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:28.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:28.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:28.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:28.953 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:28.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:29.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:29.082 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:29.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:29.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:29.208 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:29.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:29.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:29.262 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-02-05T07:25:29.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:29.333 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:29.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:29.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:25:29.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:25:29.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[29098]: 3.1d scrub starts 2024-02-05T07:25:29.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[29098]: 3.1d scrub ok 2024-02-05T07:25:29.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[29098]: osdmap e216: 8 total, 8 up, 7 in 2024-02-05T07:25:29.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[29098]: 5.16 scrub starts 2024-02-05T07:25:29.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[29098]: 5.16 scrub ok 2024-02-05T07:25:29.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [4, 0]}]: dispatch 2024-02-05T07:25:29.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-02-05T07:25:29.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]: dispatch 2024-02-05T07:25:29.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [4, 0]}]: dispatch 2024-02-05T07:25:29.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-02-05T07:25:29.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]: dispatch 2024-02-05T07:25:29.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2846549325' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:25:29.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:25:29.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:25:29.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[37158]: 3.1d scrub starts 2024-02-05T07:25:29.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[37158]: 3.1d scrub ok 2024-02-05T07:25:29.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[37158]: osdmap e216: 8 total, 8 up, 7 in 2024-02-05T07:25:29.395 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[37158]: 5.16 scrub starts 2024-02-05T07:25:29.395 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[37158]: 5.16 scrub ok 2024-02-05T07:25:29.395 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [4, 0]}]: dispatch 2024-02-05T07:25:29.395 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-02-05T07:25:29.395 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]: dispatch 2024-02-05T07:25:29.395 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [4, 0]}]: dispatch 2024-02-05T07:25:29.395 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-02-05T07:25:29.395 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]: dispatch 2024-02-05T07:25:29.395 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:29 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2846549325' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:25:29.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:29.465 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:29.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:29.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:29.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:29 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:25:29.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:29 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:25:29.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:29 smithi090 ceph-mon[33213]: 3.1d scrub starts 2024-02-05T07:25:29.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:29 smithi090 ceph-mon[33213]: 3.1d scrub ok 2024-02-05T07:25:29.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:29 smithi090 ceph-mon[33213]: osdmap e216: 8 total, 8 up, 7 in 2024-02-05T07:25:29.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:29 smithi090 ceph-mon[33213]: 5.16 scrub starts 2024-02-05T07:25:29.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:29 smithi090 ceph-mon[33213]: 5.16 scrub ok 2024-02-05T07:25:29.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [4, 0]}]: dispatch 2024-02-05T07:25:29.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-02-05T07:25:29.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]: dispatch 2024-02-05T07:25:29.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:29 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [4, 0]}]: dispatch 2024-02-05T07:25:29.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:29 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-02-05T07:25:29.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:29 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]: dispatch 2024-02-05T07:25:29.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:29 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2846549325' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:25:29.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:29.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:29.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:29.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:29.598 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:29.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:29.722 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:29.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:29.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:29.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:29.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:29.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:29.981 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:29.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:29.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:30.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:30.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:30.107 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:30.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:30.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:30.241 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:30.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:30.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:30.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:30.369 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:30.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:30.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:30.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:30.497 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:30.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:30.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:25:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:25:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:30 smithi090 ceph-mon[33213]: pgmap v314: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:25:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [4, 0]}]': finished 2024-02-05T07:25:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]': finished 2024-02-05T07:25:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]': finished 2024-02-05T07:25:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:30 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2846549325' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:25:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:30 smithi090 ceph-mon[33213]: osdmap e217: 8 total, 8 up, 7 in 2024-02-05T07:25:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:30.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:25:30.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:30.623 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:30.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:30.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[29098]: pgmap v314: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:25:30.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [4, 0]}]': finished 2024-02-05T07:25:30.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]': finished 2024-02-05T07:25:30.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]': finished 2024-02-05T07:25:30.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2846549325' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:25:30.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[29098]: osdmap e217: 8 total, 8 up, 7 in 2024-02-05T07:25:30.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:30.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:25:30.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[37158]: pgmap v314: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:25:30.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [4, 0]}]': finished 2024-02-05T07:25:30.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]': finished 2024-02-05T07:25:30.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]': finished 2024-02-05T07:25:30.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2846549325' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:25:30.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[37158]: osdmap e217: 8 total, 8 up, 7 in 2024-02-05T07:25:30.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:30.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:25:30.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:30.747 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:30.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:30.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:30.872 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:30.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:30.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:30.968 DEBUG:teuthology.orchestra.run.smithi007:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-02-05T07:25:30.971 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T07:25:30.971 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T07:25:30.975 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:30.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:31.019 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:hints = 1 2024-02-05T07:25:31.019 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T07:25:31.019 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object prefix: benchmark_data_smithi007_105755 2024-02-05T07:25:31.019 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:25:31.020 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 0 0 0 0 0 0 - 0 2024-02-05T07:25:31.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:31.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:31.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:31.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:31.278 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:31.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:31.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:31.414 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:31.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:31.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:31.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:31 smithi090 ceph-mon[33213]: osdmap e218: 8 total, 8 up, 7 in 2024-02-05T07:25:31.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:31 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:31.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:31 smithi090 ceph-mon[33213]: 5.5 scrub starts 2024-02-05T07:25:31.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:31 smithi090 ceph-mon[33213]: 5.5 scrub ok 2024-02-05T07:25:31.554 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:31.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:31.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:31.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:31.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:31 smithi007 ceph-mon[29098]: osdmap e218: 8 total, 8 up, 7 in 2024-02-05T07:25:31.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:31 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:31.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:31 smithi007 ceph-mon[29098]: 5.5 scrub starts 2024-02-05T07:25:31.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:31 smithi007 ceph-mon[29098]: 5.5 scrub ok 2024-02-05T07:25:31.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:31 smithi007 ceph-mon[37158]: osdmap e218: 8 total, 8 up, 7 in 2024-02-05T07:25:31.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:31 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:31.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:31 smithi007 ceph-mon[37158]: 5.5 scrub starts 2024-02-05T07:25:31.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:31 smithi007 ceph-mon[37158]: 5.5 scrub ok 2024-02-05T07:25:31.691 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:31.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:31.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:31.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:31.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:31.841 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:31.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:31.943 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:31.987 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:31.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:31.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:32.019 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 1 15 3828 3813 238.3 238.312 0.00666017 0.00418361 2024-02-05T07:25:32.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:32.143 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:32.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:32.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:32.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:32.282 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:32.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:32.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:32.368 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[29098]: 5.2 scrub starts 2024-02-05T07:25:32.368 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[29098]: 5.2 scrub ok 2024-02-05T07:25:32.368 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[29098]: 5.1e scrub starts 2024-02-05T07:25:32.368 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[29098]: 5.1e scrub ok 2024-02-05T07:25:32.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[29098]: pgmap v317: 148 pgs: 2 peering, 146 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:32.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[29098]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:25:32.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[29098]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:25:32.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[29098]: 4.10 scrub starts 2024-02-05T07:25:32.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[29098]: 4.10 scrub ok 2024-02-05T07:25:32.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[37158]: 5.2 scrub starts 2024-02-05T07:25:32.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[37158]: 5.2 scrub ok 2024-02-05T07:25:32.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[37158]: 5.1e scrub starts 2024-02-05T07:25:32.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[37158]: 5.1e scrub ok 2024-02-05T07:25:32.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[37158]: pgmap v317: 148 pgs: 2 peering, 146 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:32.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[37158]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:25:32.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[37158]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:25:32.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[37158]: 4.10 scrub starts 2024-02-05T07:25:32.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:32 smithi007 ceph-mon[37158]: 4.10 scrub ok 2024-02-05T07:25:32.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:32.427 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:32.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:32.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:32.502 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:25:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:32 smithi090 ceph-mon[33213]: 5.2 scrub starts 2024-02-05T07:25:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:32 smithi090 ceph-mon[33213]: 5.2 scrub ok 2024-02-05T07:25:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:32 smithi090 ceph-mon[33213]: 5.1e scrub starts 2024-02-05T07:25:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:32 smithi090 ceph-mon[33213]: 5.1e scrub ok 2024-02-05T07:25:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:32 smithi090 ceph-mon[33213]: pgmap v317: 148 pgs: 2 peering, 146 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:32 smithi090 ceph-mon[33213]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:25:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:32 smithi090 ceph-mon[33213]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:25:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:32 smithi090 ceph-mon[33213]: 4.10 scrub starts 2024-02-05T07:25:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:32 smithi090 ceph-mon[33213]: 4.10 scrub ok 2024-02-05T07:25:32.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:32.542 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:25:32.566 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:32.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:32.585 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:25:32.625 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:25:32.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:32.669 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:25:32.705 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:32.705 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:25:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:32.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:32.749 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:25:32.791 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:25:32.799 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:25:32.799 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:25:32.799 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:25:32.799 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:25:32.799 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:25:32.799 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:25:32.799 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:25:32.799 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:25:32.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:32.842 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:32.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:32.847 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T07:25:32.847 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:25:32.847 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-02-05T07:25:32.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:25:32.934 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:25:32.934 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:25:32.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:32.985 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:32.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:32.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:33.019 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 2 15 7092 7077 221.142 204 0.00410157 0.00451598 2024-02-05T07:25:33.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:33.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:33.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:33.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:33.264 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:33.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:33.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:33.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:33.405 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:33.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:33.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:33.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:33 smithi090 ceph-mon[33213]: 5.1f scrub starts 2024-02-05T07:25:33.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:33 smithi090 ceph-mon[33213]: 5.1f scrub ok 2024-02-05T07:25:33.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:33 smithi090 ceph-mon[33213]: 5.15 scrub starts 2024-02-05T07:25:33.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:33 smithi090 ceph-mon[33213]: 5.15 scrub ok 2024-02-05T07:25:33.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:33 smithi090 ceph-mon[33213]: 4.9 scrub starts 2024-02-05T07:25:33.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:33 smithi090 ceph-mon[33213]: 4.9 scrub ok 2024-02-05T07:25:33.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:33 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3123730335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:33.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-mon[29098]: 5.1f scrub starts 2024-02-05T07:25:33.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-mon[29098]: 5.1f scrub ok 2024-02-05T07:25:33.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-mon[29098]: 5.15 scrub starts 2024-02-05T07:25:33.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-mon[29098]: 5.15 scrub ok 2024-02-05T07:25:33.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-mon[29098]: 4.9 scrub starts 2024-02-05T07:25:33.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-mon[29098]: 4.9 scrub ok 2024-02-05T07:25:33.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3123730335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:33.537 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-mon[37158]: 5.1f scrub starts 2024-02-05T07:25:33.537 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-mon[37158]: 5.1f scrub ok 2024-02-05T07:25:33.537 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-mon[37158]: 5.15 scrub starts 2024-02-05T07:25:33.537 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-mon[37158]: 5.15 scrub ok 2024-02-05T07:25:33.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-mon[37158]: 4.9 scrub starts 2024-02-05T07:25:33.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-mon[37158]: 4.9 scrub ok 2024-02-05T07:25:33.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3123730335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:33.540 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:33.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:33.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:33.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:33.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:33.679 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:33.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:33.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:33.805 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:33.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:33.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:33.841 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:25:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:25:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:25:33.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:33.938 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:33.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:33.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:34.019 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 3 16 9716 9700 202.069 163.938 0.00393706 0.0049415 2024-02-05T07:25:34.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:34.076 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:34.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:34.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:34.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:34.216 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:34.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:34.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:34.277 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T07:25:34.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:34.352 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:34.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:34.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:34.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:34 smithi007 ceph-mon[29098]: 5.11 scrub starts 2024-02-05T07:25:34.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:34 smithi007 ceph-mon[29098]: 5.11 scrub ok 2024-02-05T07:25:34.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:34 smithi007 ceph-mon[29098]: pgmap v318: 148 pgs: 2 peering, 146 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 706 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:34.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:34 smithi007 ceph-mon[29098]: 3.1b scrub starts 2024-02-05T07:25:34.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:34 smithi007 ceph-mon[29098]: 3.1b scrub ok 2024-02-05T07:25:34.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:34 smithi007 ceph-mon[29098]: 3.6 scrub starts 2024-02-05T07:25:34.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:34 smithi007 ceph-mon[29098]: 3.6 scrub ok 2024-02-05T07:25:34.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:34 smithi007 ceph-mon[37158]: 5.11 scrub starts 2024-02-05T07:25:34.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:34 smithi007 ceph-mon[37158]: 5.11 scrub ok 2024-02-05T07:25:34.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:34 smithi007 ceph-mon[37158]: pgmap v318: 148 pgs: 2 peering, 146 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 706 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:34.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:34 smithi007 ceph-mon[37158]: 3.1b scrub starts 2024-02-05T07:25:34.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:34 smithi007 ceph-mon[37158]: 3.1b scrub ok 2024-02-05T07:25:34.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:34 smithi007 ceph-mon[37158]: 3.6 scrub starts 2024-02-05T07:25:34.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:34 smithi007 ceph-mon[37158]: 3.6 scrub ok 2024-02-05T07:25:34.484 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:34.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:34.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:34 smithi090 ceph-mon[33213]: 5.11 scrub starts 2024-02-05T07:25:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:34 smithi090 ceph-mon[33213]: 5.11 scrub ok 2024-02-05T07:25:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:34 smithi090 ceph-mon[33213]: pgmap v318: 148 pgs: 2 peering, 146 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 706 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:34 smithi090 ceph-mon[33213]: 3.1b scrub starts 2024-02-05T07:25:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:34 smithi090 ceph-mon[33213]: 3.1b scrub ok 2024-02-05T07:25:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:34 smithi090 ceph-mon[33213]: 3.6 scrub starts 2024-02-05T07:25:34.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:34 smithi090 ceph-mon[33213]: 3.6 scrub ok 2024-02-05T07:25:34.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:34.627 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:34.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:34.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:34.769 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:34.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:34.870 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:34.907 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:34.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:35.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:35.019 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 4 16 12413 12397 193.689 168.562 0.00422071 0.00515579 2024-02-05T07:25:35.044 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:35.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:35.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:35.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:35.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:35.304 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:25:35.331 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:35.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:35.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:35.470 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:35.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:35.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:35 smithi007 ceph-mon[29098]: 5.1c scrub starts 2024-02-05T07:25:35.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:35 smithi007 ceph-mon[29098]: 5.1c scrub ok 2024-02-05T07:25:35.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:35 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1132224973' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:25:35.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:35 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:25:35.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:35 smithi007 ceph-mon[37158]: 5.1c scrub starts 2024-02-05T07:25:35.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:35 smithi007 ceph-mon[37158]: 5.1c scrub ok 2024-02-05T07:25:35.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:35 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1132224973' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:25:35.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:35 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:25:35.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:35.612 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:35.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:35.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:35.751 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:35.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:35 smithi090 ceph-mon[33213]: 5.1c scrub starts 2024-02-05T07:25:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:35 smithi090 ceph-mon[33213]: 5.1c scrub ok 2024-02-05T07:25:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:35 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1132224973' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:25:35.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:35 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:25:35.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:35.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:35.893 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:35.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:35.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:36.020 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 5 16 15295 15279 190.974 180.125 0.00513952 0.00523012 2024-02-05T07:25:36.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:36.032 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:36.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:36.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:36.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:36.169 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:36.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:36.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:36.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:36.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:36.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[29098]: 6.a scrub starts 2024-02-05T07:25:36.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[29098]: 6.a scrub ok 2024-02-05T07:25:36.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[29098]: 5.10 scrub starts 2024-02-05T07:25:36.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[29098]: 5.10 scrub ok 2024-02-05T07:25:36.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[29098]: pgmap v319: 148 pgs: 2 peering, 146 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 603 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:36.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[29098]: 3.1c scrub starts 2024-02-05T07:25:36.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[29098]: 3.1c scrub ok 2024-02-05T07:25:36.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:25:36.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:25:36.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[29098]: osdmap e219: 8 total, 8 up, 7 in 2024-02-05T07:25:36.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[37158]: 6.a scrub starts 2024-02-05T07:25:36.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[37158]: 6.a scrub ok 2024-02-05T07:25:36.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[37158]: 5.10 scrub starts 2024-02-05T07:25:36.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[37158]: 5.10 scrub ok 2024-02-05T07:25:36.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[37158]: pgmap v319: 148 pgs: 2 peering, 146 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 603 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:36.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[37158]: 3.1c scrub starts 2024-02-05T07:25:36.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[37158]: 3.1c scrub ok 2024-02-05T07:25:36.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:25:36.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:25:36.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:36 smithi007 ceph-mon[37158]: osdmap e219: 8 total, 8 up, 7 in 2024-02-05T07:25:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:36.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:36.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:36.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:36.585 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:36.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:36.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:36.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:36.723 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:36.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:36 smithi090 ceph-mon[33213]: 6.a scrub starts 2024-02-05T07:25:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:36 smithi090 ceph-mon[33213]: 6.a scrub ok 2024-02-05T07:25:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:36 smithi090 ceph-mon[33213]: 5.10 scrub starts 2024-02-05T07:25:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:36 smithi090 ceph-mon[33213]: 5.10 scrub ok 2024-02-05T07:25:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:36 smithi090 ceph-mon[33213]: pgmap v319: 148 pgs: 2 peering, 146 active+clean; 582 KiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 603 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:36 smithi090 ceph-mon[33213]: 3.1c scrub starts 2024-02-05T07:25:36.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:36 smithi090 ceph-mon[33213]: 3.1c scrub ok 2024-02-05T07:25:36.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:36 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:25:36.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:36 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:25:36.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:36 smithi090 ceph-mon[33213]: osdmap e219: 8 total, 8 up, 7 in 2024-02-05T07:25:36.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:36.860 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:36.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:36.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:37.002 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:37.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:37.019 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 6 16 18265 18249 190.08 185.625 0.00366924 0.00525503 2024-02-05T07:25:37.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:25:37.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:37.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:37.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:37.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:37.341 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:37.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:37.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:37.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:37 smithi007 ceph-mon[29098]: 3.a scrub starts 2024-02-05T07:25:37.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:37 smithi007 ceph-mon[29098]: 3.a scrub ok 2024-02-05T07:25:37.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:37 smithi007 ceph-mon[29098]: 2.1 scrub starts 2024-02-05T07:25:37.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:37 smithi007 ceph-mon[29098]: 2.1 scrub ok 2024-02-05T07:25:37.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:37 smithi007 ceph-mon[29098]: 5.4 scrub starts 2024-02-05T07:25:37.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:37 smithi007 ceph-mon[29098]: 5.4 scrub ok 2024-02-05T07:25:37.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:37 smithi007 ceph-mon[37158]: 3.a scrub starts 2024-02-05T07:25:37.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:37 smithi007 ceph-mon[37158]: 3.a scrub ok 2024-02-05T07:25:37.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:37 smithi007 ceph-mon[37158]: 2.1 scrub starts 2024-02-05T07:25:37.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:37 smithi007 ceph-mon[37158]: 2.1 scrub ok 2024-02-05T07:25:37.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:37 smithi007 ceph-mon[37158]: 5.4 scrub starts 2024-02-05T07:25:37.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:37 smithi007 ceph-mon[37158]: 5.4 scrub ok 2024-02-05T07:25:37.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:37.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:37 smithi090 ceph-mon[33213]: 3.a scrub starts 2024-02-05T07:25:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:37 smithi090 ceph-mon[33213]: 3.a scrub ok 2024-02-05T07:25:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:37 smithi090 ceph-mon[33213]: 2.1 scrub starts 2024-02-05T07:25:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:37 smithi090 ceph-mon[33213]: 2.1 scrub ok 2024-02-05T07:25:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:37 smithi090 ceph-mon[33213]: 5.4 scrub starts 2024-02-05T07:25:37.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:37 smithi090 ceph-mon[33213]: 5.4 scrub ok 2024-02-05T07:25:37.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:37.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:37.606 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:37.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:37.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:37.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:37.744 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:37.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:37.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:37.884 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:37.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:37.986 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:38.020 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 7 16 20792 20776 185.487 157.938 0.00492298 0.00538446 2024-02-05T07:25:38.021 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:38.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:38.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:38.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:38.158 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:38.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:38.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:38.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:38.301 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:38.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:38.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:38.446 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:38.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:38.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:38.587 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:38.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:38 smithi007 ceph-mon[29098]: 3.15 scrub starts 2024-02-05T07:25:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:38 smithi007 ceph-mon[29098]: 3.15 scrub ok 2024-02-05T07:25:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:38 smithi007 ceph-mon[29098]: pgmap v321: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 800 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:38 smithi007 ceph-mon[29098]: 6.1e scrub starts 2024-02-05T07:25:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:38 smithi007 ceph-mon[29098]: 6.1e scrub ok 2024-02-05T07:25:38.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:38 smithi007 ceph-mon[29098]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-05T07:25:38.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:38 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:25:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:38 smithi007 ceph-mon[37158]: 3.15 scrub starts 2024-02-05T07:25:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:38 smithi007 ceph-mon[37158]: 3.15 scrub ok 2024-02-05T07:25:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:38 smithi007 ceph-mon[37158]: pgmap v321: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 800 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:38 smithi007 ceph-mon[37158]: 6.1e scrub starts 2024-02-05T07:25:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:38 smithi007 ceph-mon[37158]: 6.1e scrub ok 2024-02-05T07:25:38.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:38 smithi007 ceph-mon[37158]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-05T07:25:38.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:38 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:25:38.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:38.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:38.725 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:38.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:38 smithi090 ceph-mon[33213]: 3.15 scrub starts 2024-02-05T07:25:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:38 smithi090 ceph-mon[33213]: 3.15 scrub ok 2024-02-05T07:25:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:38 smithi090 ceph-mon[33213]: pgmap v321: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 800 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:38 smithi090 ceph-mon[33213]: 6.1e scrub starts 2024-02-05T07:25:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:38 smithi090 ceph-mon[33213]: 6.1e scrub ok 2024-02-05T07:25:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:38 smithi090 ceph-mon[33213]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-05T07:25:38.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:38 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:25:38.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:38.862 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:38.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:38.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:39.001 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:39.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:39.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:39.020 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 8 16 23223 23207 181.292 151.938 0.00157102 0.00549976 2024-02-05T07:25:39.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:39.143 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:39.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:39.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:39.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:39.270 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:39.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:39.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:39.406 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:39.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:39.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:39.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:39.546 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:39.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:39.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:39.682 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:39.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:39.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:39.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:39.813 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:39.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:39.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:39.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:39.938 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:39.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:40.020 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 9 15 25231 25216 175.098 125.562 0.00420074 0.00570543 2024-02-05T07:25:40.039 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:40.076 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:40.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:40.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:40.097 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:25:40.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:40.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:40.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:40.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:40.381 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:40.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:40.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:40.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[29098]: 3.c scrub starts 2024-02-05T07:25:40.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[29098]: 3.c scrub ok 2024-02-05T07:25:40.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[29098]: 4.19 scrub starts 2024-02-05T07:25:40.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[29098]: 4.19 scrub ok 2024-02-05T07:25:40.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[29098]: pgmap v322: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 800 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 466 B/s rd, 91 MiB/s wr, 1.46k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:40.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[29098]: osdmap e220: 8 total, 8 up, 7 in 2024-02-05T07:25:40.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[29098]: 5.c scrub starts 2024-02-05T07:25:40.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[29098]: 5.c scrub ok 2024-02-05T07:25:40.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[37158]: 3.c scrub starts 2024-02-05T07:25:40.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[37158]: 3.c scrub ok 2024-02-05T07:25:40.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[37158]: 4.19 scrub starts 2024-02-05T07:25:40.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[37158]: 4.19 scrub ok 2024-02-05T07:25:40.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[37158]: pgmap v322: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 800 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 466 B/s rd, 91 MiB/s wr, 1.46k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:40.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[37158]: osdmap e220: 8 total, 8 up, 7 in 2024-02-05T07:25:40.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[37158]: 5.c scrub starts 2024-02-05T07:25:40.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:40 smithi007 ceph-mon[37158]: 5.c scrub ok 2024-02-05T07:25:40.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:40.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:40.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:25:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:25:40.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:40 smithi090 ceph-mon[33213]: 3.c scrub starts 2024-02-05T07:25:40.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:40 smithi090 ceph-mon[33213]: 3.c scrub ok 2024-02-05T07:25:40.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:40 smithi090 ceph-mon[33213]: 4.19 scrub starts 2024-02-05T07:25:40.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:40 smithi090 ceph-mon[33213]: 4.19 scrub ok 2024-02-05T07:25:40.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:40 smithi090 ceph-mon[33213]: pgmap v322: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 800 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 466 B/s rd, 91 MiB/s wr, 1.46k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:40.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:40 smithi090 ceph-mon[33213]: osdmap e220: 8 total, 8 up, 7 in 2024-02-05T07:25:40.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:40 smithi090 ceph-mon[33213]: 5.c scrub starts 2024-02-05T07:25:40.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:40 smithi090 ceph-mon[33213]: 5.c scrub ok 2024-02-05T07:25:40.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:40.619 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:25:40.623 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:25:40.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:40.651 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:40.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:40.753 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:40.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:40.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:40.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:40.948 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:40.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:41.020 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 10 16 27297 27281 170.494 129.062 0.00619236 0.0058605 2024-02-05T07:25:41.050 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:41.085 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:41.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:41.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:41.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:41.224 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:41.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:41.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:41.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:41.365 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:41.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:41.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:41 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:41.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:41 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:41.467 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:41.484 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-02-05T07:25:39.207956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":839326044,"num_objects":12996,"num_object_clones":0,"num_object_copies":38988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12996,"num_whiteouts":0,"num_read":1320,"num_read_kb":1166,"num_write":13413,"num_write_kb":821533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":1773391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13445,"ondisk_log_size":13445,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3708024,"kb_used_data":2275072,"kb_used_omap":10,"kb_used_meta":1432885,"kb_avail":652508040,"statfs":{"total":671965249536,"available":668168232960,"internally_reserved":0,"allocated":2329673728,"data_stored":2325391527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11159,"internal_metadata":1467274345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":838729728,"num_objects":12798,"num_object_clones":0,"num_object_copies":38394,"num_objects_missing_on_primary":0,"num_objects_missing":-5,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12798,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12798,"num_write_kb":819072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.173234"},"pg_stats":[{"pgid":"8.0","version":"218'832","reported_seq":860,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.091096+0000","last_change":"2024-02-05T07:25:16.485753+0000","last_active":"2024-02-05T07:25:35.091096+0000","last_peered":"2024-02-05T07:25:35.091096+0000","last_clean":"2024-02-05T07:25:35.091096+0000","last_became_active":"2024-02-05T07:25:16.485417+0000","last_became_peered":"2024-02-05T07:25:16.485417+0000","last_unstale":"2024-02-05T07:25:35.091096+0000","last_undegraded":"2024-02-05T07:25:35.091096+0000","last_fullsized":"2024-02-05T07:25:35.091096+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":832,"log_dups_size":0,"ondisk_log_size":832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:34.780965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":832,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":832,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":357,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255719+0000","last_change":"2024-02-05T07:23:47.629122+0000","last_active":"2024-02-05T07:25:30.255719+0000","last_peered":"2024-02-05T07:25:30.255719+0000","last_clean":"2024-02-05T07:25:30.255719+0000","last_became_active":"2024-02-05T07:23:47.597686+0000","last_became_peered":"2024-02-05T07:23:47.597686+0000","last_unstale":"2024-02-05T07:25:30.255719+0000","last_undegraded":"2024-02-05T07:25:30.255719+0000","last_fullsized":"2024-02-05T07:25:30.255719+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:23:04.538922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:04.538922+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":76,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255722+0000","last_change":"2024-02-05T07:24:34.251845+0000","last_active":"2024-02-05T07:25:30.255722+0000","last_peered":"2024-02-05T07:25:30.255722+0000","last_clean":"2024-02-05T07:25:30.255722+0000","last_became_active":"2024-02-05T07:24:34.251554+0000","last_became_peered":"2024-02-05T07:24:34.251554+0000","last_unstale":"2024-02-05T07:25:30.255722+0000","last_undegraded":"2024-02-05T07:25:30.255722+0000","last_fullsized":"2024-02-05T07:25:30.255722+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:13.198115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:13.198115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":415,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.296458+0000","last_change":"2024-02-05T07:21:27.645530+0000","last_active":"2024-02-05T07:25:35.296458+0000","last_peered":"2024-02-05T07:25:35.296458+0000","last_clean":"2024-02-05T07:25:35.296458+0000","last_became_active":"2024-02-05T07:18:53.461852+0000","last_became_peered":"2024-02-05T07:18:53.461852+0000","last_unstale":"2024-02-05T07:25:35.296458+0000","last_undegraded":"2024-02-05T07:25:35.296458+0000","last_fullsized":"2024-02-05T07:25:35.296458+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:21:27.645263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:27.645263+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":316,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256399+0000","last_change":"2024-02-05T07:23:09.542975+0000","last_active":"2024-02-05T07:25:30.256399+0000","last_peered":"2024-02-05T07:25:30.256399+0000","last_clean":"2024-02-05T07:25:30.256399+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:25:30.256399+0000","last_undegraded":"2024-02-05T07:25:30.256399+0000","last_fullsized":"2024-02-05T07:25:30.256399+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:09.542903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:09.542903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"218'757","reported_seq":785,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.178303+0000","last_change":"2024-02-05T07:25:16.685080+0000","last_active":"2024-02-05T07:25:35.178303+0000","last_peered":"2024-02-05T07:25:35.178303+0000","last_clean":"2024-02-05T07:25:35.178303+0000","last_became_active":"2024-02-05T07:25:16.684739+0000","last_became_peered":"2024-02-05T07:25:16.684739+0000","last_unstale":"2024-02-05T07:25:35.178303+0000","last_undegraded":"2024-02-05T07:25:35.178303+0000","last_fullsized":"2024-02-05T07:25:35.178303+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":757,"log_dups_size":0,"ondisk_log_size":757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:21.952495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49610752,"num_objects":757,"num_object_clones":0,"num_object_copies":2271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":757,"num_write_kb":48448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":360,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256016+0000","last_change":"2024-02-05T07:24:54.875478+0000","last_active":"2024-02-05T07:25:30.256016+0000","last_peered":"2024-02-05T07:25:30.256016+0000","last_clean":"2024-02-05T07:25:30.256016+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:25:30.256016+0000","last_undegraded":"2024-02-05T07:25:30.256016+0000","last_fullsized":"2024-02-05T07:25:30.256016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:24:54.875336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:54.875336+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:21.497738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":329,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.263286+0000","last_change":"2024-02-05T07:24:14.277115+0000","last_active":"2024-02-05T07:25:30.263286+0000","last_peered":"2024-02-05T07:25:30.263286+0000","last_clean":"2024-02-05T07:25:30.263286+0000","last_became_active":"2024-02-05T07:18:49.430518+0000","last_became_peered":"2024-02-05T07:18:49.430518+0000","last_unstale":"2024-02-05T07:25:30.263286+0000","last_undegraded":"2024-02-05T07:25:30.263286+0000","last_fullsized":"2024-02-05T07:25:30.263286+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:14.277005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:14.277005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:34.125024+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":321,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.257396+0000","last_change":"2024-02-05T07:21:17.402214+0000","last_active":"2024-02-05T07:25:30.257396+0000","last_peered":"2024-02-05T07:25:30.257396+0000","last_clean":"2024-02-05T07:25:30.257396+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:25:30.257396+0000","last_undegraded":"2024-02-05T07:25:30.257396+0000","last_fullsized":"2024-02-05T07:25:30.257396+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:17.401959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:17.401959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":324,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.296877+0000","last_change":"2024-02-05T07:24:54.743577+0000","last_active":"2024-02-05T07:25:35.296877+0000","last_peered":"2024-02-05T07:25:35.296877+0000","last_clean":"2024-02-05T07:25:35.296877+0000","last_became_active":"2024-02-05T07:23:47.609458+0000","last_became_peered":"2024-02-05T07:23:47.609458+0000","last_unstale":"2024-02-05T07:25:35.296877+0000","last_undegraded":"2024-02-05T07:25:35.296877+0000","last_fullsized":"2024-02-05T07:25:35.296877+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:54.743499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:54.743499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:04.040973+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"218'764","reported_seq":792,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.045689+0000","last_change":"2024-02-05T07:25:16.684635+0000","last_active":"2024-02-05T07:25:35.045689+0000","last_peered":"2024-02-05T07:25:35.045689+0000","last_clean":"2024-02-05T07:25:35.045689+0000","last_became_active":"2024-02-05T07:25:16.684316+0000","last_became_peered":"2024-02-05T07:25:16.684316+0000","last_unstale":"2024-02-05T07:25:35.045689+0000","last_undegraded":"2024-02-05T07:25:35.045689+0000","last_fullsized":"2024-02-05T07:25:35.045689+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":764,"log_dups_size":0,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:21.892273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50069504,"num_objects":764,"num_object_clones":0,"num_object_copies":2292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":764,"num_write_kb":48896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"90'16","reported_seq":356,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256255+0000","last_change":"2024-02-05T07:25:27.129692+0000","last_active":"2024-02-05T07:25:30.256255+0000","last_peered":"2024-02-05T07:25:30.256255+0000","last_clean":"2024-02-05T07:25:30.256255+0000","last_became_active":"2024-02-05T07:18:51.447922+0000","last_became_peered":"2024-02-05T07:18:51.447922+0000","last_unstale":"2024-02-05T07:25:30.256255+0000","last_undegraded":"2024-02-05T07:25:30.256255+0000","last_fullsized":"2024-02-05T07:25:30.256255+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-02-05T07:25:27.129600+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:25:27.129600+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:34.339679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":329,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255029+0000","last_change":"2024-02-05T07:23:02.519644+0000","last_active":"2024-02-05T07:25:30.255029+0000","last_peered":"2024-02-05T07:25:30.255029+0000","last_clean":"2024-02-05T07:25:30.255029+0000","last_became_active":"2024-02-05T07:18:49.434074+0000","last_became_peered":"2024-02-05T07:18:49.434074+0000","last_unstale":"2024-02-05T07:25:30.255029+0000","last_undegraded":"2024-02-05T07:25:30.255029+0000","last_fullsized":"2024-02-05T07:25:30.255029+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:02.519566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:02.519566+0000","last_clean_scrub_stamp":"2024-02-05T07:23:02.519566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":150,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256244+0000","last_change":"2024-02-05T07:24:49.760637+0000","last_active":"2024-02-05T07:25:30.256244+0000","last_peered":"2024-02-05T07:25:30.256244+0000","last_clean":"2024-02-05T07:25:30.256244+0000","last_became_active":"2024-02-05T07:23:27.417675+0000","last_became_peered":"2024-02-05T07:23:27.417675+0000","last_unstale":"2024-02-05T07:25:30.256244+0000","last_undegraded":"2024-02-05T07:25:30.256244+0000","last_fullsized":"2024-02-05T07:25:30.256244+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:49.760543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.760543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:03.923284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":130,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.263896+0000","last_change":"2024-02-05T07:24:08.261218+0000","last_active":"2024-02-05T07:25:30.263896+0000","last_peered":"2024-02-05T07:25:30.263896+0000","last_clean":"2024-02-05T07:25:30.263896+0000","last_became_active":"2024-02-05T07:23:47.611030+0000","last_became_peered":"2024-02-05T07:23:47.611030+0000","last_unstale":"2024-02-05T07:25:30.263896+0000","last_undegraded":"2024-02-05T07:25:30.263896+0000","last_fullsized":"2024-02-05T07:25:30.263896+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:08.261123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:08.261123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:37.699207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"218'755","reported_seq":783,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.164601+0000","last_change":"2024-02-05T07:25:16.487219+0000","last_active":"2024-02-05T07:25:35.164601+0000","last_peered":"2024-02-05T07:25:35.164601+0000","last_clean":"2024-02-05T07:25:35.164601+0000","last_became_active":"2024-02-05T07:25:16.486922+0000","last_became_peered":"2024-02-05T07:25:16.486922+0000","last_unstale":"2024-02-05T07:25:35.164601+0000","last_undegraded":"2024-02-05T07:25:35.164601+0000","last_fullsized":"2024-02-05T07:25:35.164601+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":755,"log_dups_size":0,"ondisk_log_size":755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:15.590732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49479680,"num_objects":755,"num_object_clones":0,"num_object_copies":2265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":755,"num_write_kb":48320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"98'22","reported_seq":379,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256134+0000","last_change":"2024-02-05T07:23:47.637874+0000","last_active":"2024-02-05T07:25:30.256134+0000","last_peered":"2024-02-05T07:25:30.256134+0000","last_clean":"2024-02-05T07:25:30.256134+0000","last_became_active":"2024-02-05T07:23:47.606483+0000","last_became_peered":"2024-02-05T07:23:47.606483+0000","last_unstale":"2024-02-05T07:25:30.256134+0000","last_undegraded":"2024-02-05T07:25:30.256134+0000","last_fullsized":"2024-02-05T07:25:30.256134+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'22","last_scrub_stamp":"2024-02-05T07:23:19.000981+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:23:19.000981+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":122,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255887+0000","last_change":"2024-02-05T07:23:47.600893+0000","last_active":"2024-02-05T07:25:30.255887+0000","last_peered":"2024-02-05T07:25:30.255887+0000","last_clean":"2024-02-05T07:25:30.255887+0000","last_became_active":"2024-02-05T07:23:47.600273+0000","last_became_peered":"2024-02-05T07:23:47.600273+0000","last_unstale":"2024-02-05T07:25:30.255887+0000","last_undegraded":"2024-02-05T07:25:30.255887+0000","last_fullsized":"2024-02-05T07:25:30.255887+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:37.367711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.367711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":410,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256418+0000","last_change":"2024-02-05T07:24:19.650665+0000","last_active":"2024-02-05T07:25:30.256418+0000","last_peered":"2024-02-05T07:25:30.256418+0000","last_clean":"2024-02-05T07:25:30.256418+0000","last_became_active":"2024-02-05T07:18:53.459832+0000","last_became_peered":"2024-02-05T07:18:53.459832+0000","last_unstale":"2024-02-05T07:25:30.256418+0000","last_undegraded":"2024-02-05T07:25:30.256418+0000","last_fullsized":"2024-02-05T07:25:30.256418+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:24:19.650521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:19.650521+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":316,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.259841+0000","last_change":"2024-02-05T07:24:52.767947+0000","last_active":"2024-02-05T07:25:30.259841+0000","last_peered":"2024-02-05T07:25:30.259841+0000","last_clean":"2024-02-05T07:25:30.259841+0000","last_became_active":"2024-02-05T07:18:55.472172+0000","last_became_peered":"2024-02-05T07:18:55.472172+0000","last_unstale":"2024-02-05T07:25:30.259841+0000","last_undegraded":"2024-02-05T07:25:30.259841+0000","last_fullsized":"2024-02-05T07:25:30.259841+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:52.767872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:52.767872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:00.263931+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"218'737","reported_seq":765,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.112064+0000","last_change":"2024-02-05T07:25:16.684780+0000","last_active":"2024-02-05T07:25:35.112064+0000","last_peered":"2024-02-05T07:25:35.112064+0000","last_clean":"2024-02-05T07:25:35.112064+0000","last_became_active":"2024-02-05T07:25:16.684374+0000","last_became_peered":"2024-02-05T07:25:16.684374+0000","last_unstale":"2024-02-05T07:25:35.112064+0000","last_undegraded":"2024-02-05T07:25:35.112064+0000","last_fullsized":"2024-02-05T07:25:35.112064+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":737,"log_dups_size":0,"ondisk_log_size":737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:39.488642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48300032,"num_objects":737,"num_object_clones":0,"num_object_copies":2211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":737,"num_write_kb":47168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":327,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.254982+0000","last_change":"2024-02-05T07:23:47.626460+0000","last_active":"2024-02-05T07:25:30.254982+0000","last_peered":"2024-02-05T07:25:30.254982+0000","last_clean":"2024-02-05T07:25:30.254982+0000","last_became_active":"2024-02-05T07:23:47.626072+0000","last_became_peered":"2024-02-05T07:23:47.626072+0000","last_unstale":"2024-02-05T07:25:30.254982+0000","last_undegraded":"2024-02-05T07:25:30.254982+0000","last_fullsized":"2024-02-05T07:25:30.254982+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:03.505163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:23:03.505163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":199,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261279+0000","last_change":"2024-02-05T07:24:14.360014+0000","last_active":"2024-02-05T07:25:30.261279+0000","last_peered":"2024-02-05T07:25:30.261279+0000","last_clean":"2024-02-05T07:25:30.261279+0000","last_became_active":"2024-02-05T07:23:47.618052+0000","last_became_peered":"2024-02-05T07:23:47.618052+0000","last_unstale":"2024-02-05T07:25:30.261279+0000","last_undegraded":"2024-02-05T07:25:30.261279+0000","last_fullsized":"2024-02-05T07:25:30.261279+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:24:14.359892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:14.359892+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:34.362411+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":122,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261985+0000","last_change":"2024-02-05T07:23:47.611708+0000","last_active":"2024-02-05T07:25:30.261985+0000","last_peered":"2024-02-05T07:25:30.261985+0000","last_clean":"2024-02-05T07:25:30.261985+0000","last_became_active":"2024-02-05T07:23:47.609054+0000","last_became_peered":"2024-02-05T07:23:47.609054+0000","last_unstale":"2024-02-05T07:25:30.261985+0000","last_undegraded":"2024-02-05T07:25:30.261985+0000","last_fullsized":"2024-02-05T07:25:30.261985+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:42.464894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:42.464894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":323,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255259+0000","last_change":"2024-02-05T07:23:47.603895+0000","last_active":"2024-02-05T07:25:30.255259+0000","last_peered":"2024-02-05T07:25:30.255259+0000","last_clean":"2024-02-05T07:25:30.255259+0000","last_became_active":"2024-02-05T07:23:47.603326+0000","last_became_peered":"2024-02-05T07:23:47.603326+0000","last_unstale":"2024-02-05T07:25:30.255259+0000","last_undegraded":"2024-02-05T07:25:30.255259+0000","last_fullsized":"2024-02-05T07:25:30.255259+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:16.499591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:16.499591+0000","last_clean_scrub_stamp":"2024-02-05T07:23:16.499591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"218'788","reported_seq":816,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.121698+0000","last_change":"2024-02-05T07:25:16.486195+0000","last_active":"2024-02-05T07:25:35.121698+0000","last_peered":"2024-02-05T07:25:35.121698+0000","last_clean":"2024-02-05T07:25:35.121698+0000","last_became_active":"2024-02-05T07:25:16.485881+0000","last_became_peered":"2024-02-05T07:25:16.485881+0000","last_unstale":"2024-02-05T07:25:35.121698+0000","last_undegraded":"2024-02-05T07:25:35.121698+0000","last_fullsized":"2024-02-05T07:25:35.121698+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.672798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51642368,"num_objects":788,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":788,"num_write_kb":50432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"164'9","reported_seq":181,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255179+0000","last_change":"2024-02-05T07:24:20.453876+0000","last_active":"2024-02-05T07:25:30.255179+0000","last_peered":"2024-02-05T07:25:30.255179+0000","last_clean":"2024-02-05T07:25:30.255179+0000","last_became_active":"2024-02-05T07:23:47.626082+0000","last_became_peered":"2024-02-05T07:23:47.626082+0000","last_unstale":"2024-02-05T07:25:30.255179+0000","last_undegraded":"2024-02-05T07:25:30.255179+0000","last_fullsized":"2024-02-05T07:25:30.255179+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:24:20.453746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:20.453746+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"202'21","reported_seq":353,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255174+0000","last_change":"2024-02-05T07:24:59.870626+0000","last_active":"2024-02-05T07:25:30.255174+0000","last_peered":"2024-02-05T07:25:30.255174+0000","last_clean":"2024-02-05T07:25:30.255174+0000","last_became_active":"2024-02-05T07:18:51.438507+0000","last_became_peered":"2024-02-05T07:18:51.438507+0000","last_unstale":"2024-02-05T07:25:30.255174+0000","last_undegraded":"2024-02-05T07:25:30.255174+0000","last_fullsized":"2024-02-05T07:25:30.255174+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'21","last_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:24:59.870542+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:09.890255+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":90,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:33.462672+0000","last_change":"2024-02-05T07:25:33.462720+0000","last_active":"2024-02-05T07:25:33.462672+0000","last_peered":"2024-02-05T07:25:33.462672+0000","last_clean":"2024-02-05T07:25:33.462672+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:25:33.462672+0000","last_undegraded":"2024-02-05T07:25:33.462672+0000","last_fullsized":"2024-02-05T07:25:33.462672+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:25:33.462588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:33.462588+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:44.148228+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":317,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261096+0000","last_change":"2024-02-05T07:24:59.124223+0000","last_active":"2024-02-05T07:25:30.261096+0000","last_peered":"2024-02-05T07:25:30.261096+0000","last_clean":"2024-02-05T07:25:30.261096+0000","last_became_active":"2024-02-05T07:18:55.507791+0000","last_became_peered":"2024-02-05T07:18:55.507791+0000","last_unstale":"2024-02-05T07:25:30.261096+0000","last_undegraded":"2024-02-05T07:25:30.261096+0000","last_fullsized":"2024-02-05T07:25:30.261096+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:59.124133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:59.124133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:16.719629+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"219'837","reported_seq":867,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.561651+0000","last_change":"2024-02-05T07:25:16.686731+0000","last_active":"2024-02-05T07:25:35.561651+0000","last_peered":"2024-02-05T07:25:35.561651+0000","last_clean":"2024-02-05T07:25:35.561651+0000","last_became_active":"2024-02-05T07:25:16.685782+0000","last_became_peered":"2024-02-05T07:25:16.685782+0000","last_unstale":"2024-02-05T07:25:35.561651+0000","last_undegraded":"2024-02-05T07:25:35.561651+0000","last_fullsized":"2024-02-05T07:25:35.561651+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:16.424707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54853632,"num_objects":837,"num_object_clones":0,"num_object_copies":2511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":837,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"187'9","reported_seq":120,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.263627+0000","last_change":"2024-02-05T07:24:59.438290+0000","last_active":"2024-02-05T07:25:30.263627+0000","last_peered":"2024-02-05T07:25:30.263627+0000","last_clean":"2024-02-05T07:25:30.263627+0000","last_became_active":"2024-02-05T07:24:34.230336+0000","last_became_peered":"2024-02-05T07:24:34.230336+0000","last_unstale":"2024-02-05T07:25:30.263627+0000","last_undegraded":"2024-02-05T07:25:30.263627+0000","last_fullsized":"2024-02-05T07:25:30.263627+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'9","last_scrub_stamp":"2024-02-05T07:24:59.438123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:59.438123+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:25.060163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"151'15","reported_seq":354,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.257552+0000","last_change":"2024-02-05T07:23:26.247211+0000","last_active":"2024-02-05T07:25:30.257552+0000","last_peered":"2024-02-05T07:25:30.257552+0000","last_clean":"2024-02-05T07:25:30.257552+0000","last_became_active":"2024-02-05T07:18:51.448044+0000","last_became_peered":"2024-02-05T07:18:51.448044+0000","last_unstale":"2024-02-05T07:25:30.257552+0000","last_undegraded":"2024-02-05T07:25:30.257552+0000","last_fullsized":"2024-02-05T07:25:30.257552+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:23:26.247047+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":335,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.259926+0000","last_change":"2024-02-05T07:24:57.766610+0000","last_active":"2024-02-05T07:25:30.259926+0000","last_peered":"2024-02-05T07:25:30.259926+0000","last_clean":"2024-02-05T07:25:30.259926+0000","last_became_active":"2024-02-05T07:23:47.604997+0000","last_became_peered":"2024-02-05T07:23:47.604997+0000","last_unstale":"2024-02-05T07:25:30.259926+0000","last_undegraded":"2024-02-05T07:25:30.259926+0000","last_fullsized":"2024-02-05T07:25:30.259926+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:57.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:24:57.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:02.316661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":271,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.291042+0000","last_change":"2024-02-05T07:24:34.231420+0000","last_active":"2024-02-05T07:25:35.291042+0000","last_peered":"2024-02-05T07:25:35.291042+0000","last_clean":"2024-02-05T07:25:35.291042+0000","last_became_active":"2024-02-05T07:24:34.229242+0000","last_became_peered":"2024-02-05T07:24:34.229242+0000","last_unstale":"2024-02-05T07:25:35.291042+0000","last_undegraded":"2024-02-05T07:25:35.291042+0000","last_fullsized":"2024-02-05T07:25:35.291042+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:27.196824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:27.196824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"219'894","reported_seq":924,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.562238+0000","last_change":"2024-02-05T07:25:16.482780+0000","last_active":"2024-02-05T07:25:35.562238+0000","last_peered":"2024-02-05T07:25:35.562238+0000","last_clean":"2024-02-05T07:25:35.562238+0000","last_became_active":"2024-02-05T07:25:16.482501+0000","last_became_peered":"2024-02-05T07:25:16.482501+0000","last_unstale":"2024-02-05T07:25:35.562238+0000","last_undegraded":"2024-02-05T07:25:35.562238+0000","last_fullsized":"2024-02-05T07:25:35.562238+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:36.711342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58589184,"num_objects":894,"num_object_clones":0,"num_object_copies":2682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":894,"num_write_kb":57216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":329,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.536051+0000","last_change":"2024-02-05T07:25:30.536088+0000","last_active":"2024-02-05T07:25:30.536051+0000","last_peered":"2024-02-05T07:25:30.536051+0000","last_clean":"2024-02-05T07:25:30.536051+0000","last_became_active":"2024-02-05T07:18:53.459474+0000","last_became_peered":"2024-02-05T07:18:53.459474+0000","last_unstale":"2024-02-05T07:25:30.536051+0000","last_undegraded":"2024-02-05T07:25:30.536051+0000","last_fullsized":"2024-02-05T07:25:30.536051+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:30.535958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:25:30.535958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:57.414648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":137,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.259333+0000","last_change":"2024-02-05T07:24:49.809811+0000","last_active":"2024-02-05T07:25:30.259333+0000","last_peered":"2024-02-05T07:25:30.259333+0000","last_clean":"2024-02-05T07:25:30.259333+0000","last_became_active":"2024-02-05T07:24:34.229372+0000","last_became_peered":"2024-02-05T07:24:34.229372+0000","last_unstale":"2024-02-05T07:25:30.259333+0000","last_undegraded":"2024-02-05T07:25:30.259333+0000","last_fullsized":"2024-02-05T07:25:30.259333+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:24:49.809687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.809687+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:00.732863+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":330,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.257008+0000","last_change":"2024-02-05T07:21:40.542969+0000","last_active":"2024-02-05T07:25:30.257008+0000","last_peered":"2024-02-05T07:25:30.257008+0000","last_clean":"2024-02-05T07:25:30.257008+0000","last_became_active":"2024-02-05T07:18:49.428470+0000","last_became_peered":"2024-02-05T07:18:49.428470+0000","last_unstale":"2024-02-05T07:25:30.257008+0000","last_undegraded":"2024-02-05T07:25:30.257008+0000","last_fullsized":"2024-02-05T07:25:30.257008+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:40.542479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:21:40.542479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":318,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255592+0000","last_change":"2024-02-05T07:23:01.849413+0000","last_active":"2024-02-05T07:25:30.255592+0000","last_peered":"2024-02-05T07:25:30.255592+0000","last_clean":"2024-02-05T07:25:30.255592+0000","last_became_active":"2024-02-05T07:18:55.505360+0000","last_became_peered":"2024-02-05T07:18:55.505360+0000","last_unstale":"2024-02-05T07:25:30.255592+0000","last_undegraded":"2024-02-05T07:25:30.255592+0000","last_fullsized":"2024-02-05T07:25:30.255592+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:01.849346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:01.849346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.a","version":"218'841","reported_seq":869,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.178125+0000","last_change":"2024-02-05T07:25:16.485301+0000","last_active":"2024-02-05T07:25:35.178125+0000","last_peered":"2024-02-05T07:25:35.178125+0000","last_clean":"2024-02-05T07:25:35.178125+0000","last_became_active":"2024-02-05T07:25:16.485029+0000","last_became_peered":"2024-02-05T07:25:16.485029+0000","last_unstale":"2024-02-05T07:25:35.178125+0000","last_undegraded":"2024-02-05T07:25:35.178125+0000","last_fullsized":"2024-02-05T07:25:35.178125+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:18.881410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55115776,"num_objects":841,"num_object_clones":0,"num_object_copies":2523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":841,"num_write_kb":53824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":321,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.260368+0000","last_change":"2024-02-05T07:24:50.782870+0000","last_active":"2024-02-05T07:25:30.260368+0000","last_peered":"2024-02-05T07:25:30.260368+0000","last_clean":"2024-02-05T07:25:30.260368+0000","last_became_active":"2024-02-05T07:18:53.468512+0000","last_became_peered":"2024-02-05T07:18:53.468512+0000","last_unstale":"2024-02-05T07:25:30.260368+0000","last_undegraded":"2024-02-05T07:25:30.260368+0000","last_fullsized":"2024-02-05T07:25:30.260368+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:50.782805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:50.782805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:08.806209+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":119,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256491+0000","last_change":"2024-02-05T07:25:24.528053+0000","last_active":"2024-02-05T07:25:30.256491+0000","last_peered":"2024-02-05T07:25:30.256491+0000","last_clean":"2024-02-05T07:25:30.256491+0000","last_became_active":"2024-02-05T07:24:34.209716+0000","last_became_peered":"2024-02-05T07:24:34.209716+0000","last_unstale":"2024-02-05T07:25:30.256491+0000","last_undegraded":"2024-02-05T07:25:30.256491+0000","last_fullsized":"2024-02-05T07:25:30.256491+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:25:24.527960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:24.527960+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.343172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":739,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:33.766512+0000","last_change":"2024-02-05T07:24:50.043491+0000","last_active":"2024-02-05T07:25:33.766512+0000","last_peered":"2024-02-05T07:25:33.766512+0000","last_clean":"2024-02-05T07:25:33.766512+0000","last_became_active":"2024-02-05T07:18:48.424620+0000","last_became_peered":"2024-02-05T07:18:48.424620+0000","last_unstale":"2024-02-05T07:25:33.766512+0000","last_undegraded":"2024-02-05T07:25:33.766512+0000","last_fullsized":"2024-02-05T07:25:33.766512+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:24:50.043397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:24:50.043397+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:19.735031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":398,"num_read_kb":393,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":289,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255297+0000","last_change":"2024-02-05T07:24:48.775742+0000","last_active":"2024-02-05T07:25:30.255297+0000","last_peered":"2024-02-05T07:25:30.255297+0000","last_clean":"2024-02-05T07:25:30.255297+0000","last_became_active":"2024-02-05T07:24:34.241978+0000","last_became_peered":"2024-02-05T07:24:34.241978+0000","last_unstale":"2024-02-05T07:25:30.255297+0000","last_undegraded":"2024-02-05T07:25:30.255297+0000","last_fullsized":"2024-02-05T07:25:30.255297+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:24:48.775665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:56.120076+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":329,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256787+0000","last_change":"2024-02-05T07:23:47.618520+0000","last_active":"2024-02-05T07:25:30.256787+0000","last_peered":"2024-02-05T07:25:30.256787+0000","last_clean":"2024-02-05T07:25:30.256787+0000","last_became_active":"2024-02-05T07:23:47.600441+0000","last_became_peered":"2024-02-05T07:23:47.600441+0000","last_unstale":"2024-02-05T07:25:30.256787+0000","last_undegraded":"2024-02-05T07:25:30.256787+0000","last_fullsized":"2024-02-05T07:25:30.256787+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:23:22.071923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:22.071923+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"218'800","reported_seq":828,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.080933+0000","last_change":"2024-02-05T07:25:16.685615+0000","last_active":"2024-02-05T07:25:35.080933+0000","last_peered":"2024-02-05T07:25:35.080933+0000","last_clean":"2024-02-05T07:25:35.080933+0000","last_became_active":"2024-02-05T07:25:16.685286+0000","last_became_peered":"2024-02-05T07:25:16.685286+0000","last_unstale":"2024-02-05T07:25:35.080933+0000","last_undegraded":"2024-02-05T07:25:35.080933+0000","last_fullsized":"2024-02-05T07:25:35.080933+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":800,"log_dups_size":0,"ondisk_log_size":800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:25.747779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52428800,"num_objects":800,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":800,"num_write_kb":51200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":313,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.262030+0000","last_change":"2024-02-05T07:21:15.698783+0000","last_active":"2024-02-05T07:25:30.262030+0000","last_peered":"2024-02-05T07:25:30.262030+0000","last_clean":"2024-02-05T07:25:30.262030+0000","last_became_active":"2024-02-05T07:18:53.459971+0000","last_became_peered":"2024-02-05T07:18:53.459971+0000","last_unstale":"2024-02-05T07:25:30.262030+0000","last_undegraded":"2024-02-05T07:25:30.262030+0000","last_fullsized":"2024-02-05T07:25:30.262030+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:15.698665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:15.698665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":429,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.264045+0000","last_change":"2024-02-05T07:24:20.278989+0000","last_active":"2024-02-05T07:25:30.264045+0000","last_peered":"2024-02-05T07:25:30.264045+0000","last_clean":"2024-02-05T07:25:30.264045+0000","last_became_active":"2024-02-05T07:18:51.507714+0000","last_became_peered":"2024-02-05T07:18:51.507714+0000","last_unstale":"2024-02-05T07:25:30.264045+0000","last_undegraded":"2024-02-05T07:25:30.264045+0000","last_fullsized":"2024-02-05T07:25:30.264045+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-02-05T07:24:20.278897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:20.278897+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:36.052175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":130,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.259581+0000","last_change":"2024-02-05T07:24:45.761682+0000","last_active":"2024-02-05T07:25:30.259581+0000","last_peered":"2024-02-05T07:25:30.259581+0000","last_clean":"2024-02-05T07:25:30.259581+0000","last_became_active":"2024-02-05T07:23:47.609497+0000","last_became_peered":"2024-02-05T07:23:47.609497+0000","last_unstale":"2024-02-05T07:25:30.259581+0000","last_undegraded":"2024-02-05T07:25:30.259581+0000","last_fullsized":"2024-02-05T07:25:30.259581+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:45.761597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:45.761597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:05.259798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":587,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261063+0000","last_change":"2024-02-05T07:24:21.294644+0000","last_active":"2024-02-05T07:25:30.261063+0000","last_peered":"2024-02-05T07:25:30.261063+0000","last_clean":"2024-02-05T07:25:30.261063+0000","last_became_active":"2024-02-05T07:18:39.444898+0000","last_became_peered":"2024-02-05T07:18:39.444898+0000","last_unstale":"2024-02-05T07:25:30.261063+0000","last_undegraded":"2024-02-05T07:25:30.261063+0000","last_fullsized":"2024-02-05T07:25:30.261063+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:48.367585+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":323,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.259941+0000","last_change":"2024-02-05T07:24:58.774799+0000","last_active":"2024-02-05T07:25:30.259941+0000","last_peered":"2024-02-05T07:25:30.259941+0000","last_clean":"2024-02-05T07:25:30.259941+0000","last_became_active":"2024-02-05T07:23:47.592290+0000","last_became_peered":"2024-02-05T07:23:47.592290+0000","last_unstale":"2024-02-05T07:25:30.259941+0000","last_undegraded":"2024-02-05T07:25:30.259941+0000","last_fullsized":"2024-02-05T07:25:30.259941+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:58.774721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:58.774721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:15.475417+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"218'809","reported_seq":836,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.038935+0000","last_change":"2024-02-05T07:25:16.482190+0000","last_active":"2024-02-05T07:25:35.038935+0000","last_peered":"2024-02-05T07:25:35.038935+0000","last_clean":"2024-02-05T07:25:35.038935+0000","last_became_active":"2024-02-05T07:25:16.481838+0000","last_became_peered":"2024-02-05T07:25:16.481838+0000","last_unstale":"2024-02-05T07:25:35.038935+0000","last_undegraded":"2024-02-05T07:25:35.038935+0000","last_fullsized":"2024-02-05T07:25:35.038935+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:28.163149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53018624,"num_objects":809,"num_object_clones":0,"num_object_copies":2427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":809,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":323,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.297121+0000","last_change":"2024-02-05T07:21:31.546003+0000","last_active":"2024-02-05T07:25:35.297121+0000","last_peered":"2024-02-05T07:25:35.297121+0000","last_clean":"2024-02-05T07:25:35.297121+0000","last_became_active":"2024-02-05T07:18:53.460771+0000","last_became_peered":"2024-02-05T07:18:53.460771+0000","last_unstale":"2024-02-05T07:25:35.297121+0000","last_undegraded":"2024-02-05T07:25:35.297121+0000","last_fullsized":"2024-02-05T07:25:35.297121+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:31.545467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:31.545467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"96'18","reported_seq":358,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256393+0000","last_change":"2024-02-05T07:23:05.927322+0000","last_active":"2024-02-05T07:25:30.256393+0000","last_peered":"2024-02-05T07:25:30.256393+0000","last_clean":"2024-02-05T07:25:30.256393+0000","last_became_active":"2024-02-05T07:18:51.436097+0000","last_became_peered":"2024-02-05T07:18:51.436097+0000","last_unstale":"2024-02-05T07:25:30.256393+0000","last_undegraded":"2024-02-05T07:25:30.256393+0000","last_fullsized":"2024-02-05T07:25:30.256393+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'18","last_scrub_stamp":"2024-02-05T07:23:05.927169+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:23:05.927169+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":346,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.290802+0000","last_change":"2024-02-05T07:23:47.622111+0000","last_active":"2024-02-05T07:25:35.290802+0000","last_peered":"2024-02-05T07:25:35.290802+0000","last_clean":"2024-02-05T07:25:35.290802+0000","last_became_active":"2024-02-05T07:23:47.621133+0000","last_became_peered":"2024-02-05T07:23:47.621133+0000","last_unstale":"2024-02-05T07:25:35.290802+0000","last_undegraded":"2024-02-05T07:25:35.290802+0000","last_fullsized":"2024-02-05T07:25:35.290802+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:18.723118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:21:18.723118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":336,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255706+0000","last_change":"2024-02-05T07:21:34.523665+0000","last_active":"2024-02-05T07:25:30.255706+0000","last_peered":"2024-02-05T07:25:30.255706+0000","last_clean":"2024-02-05T07:25:30.255706+0000","last_became_active":"2024-02-05T07:18:49.429651+0000","last_became_peered":"2024-02-05T07:18:49.429651+0000","last_unstale":"2024-02-05T07:25:30.255706+0000","last_undegraded":"2024-02-05T07:25:30.255706+0000","last_fullsized":"2024-02-05T07:25:30.255706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:21:34.523215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:34.523215+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":323,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.262190+0000","last_change":"2024-02-05T07:23:47.611214+0000","last_active":"2024-02-05T07:25:30.262190+0000","last_peered":"2024-02-05T07:25:30.262190+0000","last_clean":"2024-02-05T07:25:30.262190+0000","last_became_active":"2024-02-05T07:23:47.610875+0000","last_became_peered":"2024-02-05T07:23:47.610875+0000","last_unstale":"2024-02-05T07:25:30.262190+0000","last_undegraded":"2024-02-05T07:25:30.262190+0000","last_fullsized":"2024-02-05T07:25:30.262190+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:38.531265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:21:38.531265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.8","version":"218'755","reported_seq":783,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.085830+0000","last_change":"2024-02-05T07:25:16.685108+0000","last_active":"2024-02-05T07:25:35.085830+0000","last_peered":"2024-02-05T07:25:35.085830+0000","last_clean":"2024-02-05T07:25:35.085830+0000","last_became_active":"2024-02-05T07:25:16.684816+0000","last_became_peered":"2024-02-05T07:25:16.684816+0000","last_unstale":"2024-02-05T07:25:35.085830+0000","last_undegraded":"2024-02-05T07:25:35.085830+0000","last_fullsized":"2024-02-05T07:25:35.085830+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":755,"log_dups_size":0,"ondisk_log_size":755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:19.711093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49479680,"num_objects":755,"num_object_clones":0,"num_object_copies":2265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":755,"num_write_kb":48320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":82,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.477201+0000","last_change":"2024-02-05T07:25:30.477223+0000","last_active":"2024-02-05T07:25:30.477201+0000","last_peered":"2024-02-05T07:25:30.477201+0000","last_clean":"2024-02-05T07:25:30.477201+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:25:30.477201+0000","last_undegraded":"2024-02-05T07:25:30.477201+0000","last_fullsized":"2024-02-05T07:25:30.477201+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:30.477135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:30.477135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:54.617889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"218'30","reported_seq":107,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.295973+0000","last_change":"2024-02-05T07:25:30.273272+0000","last_active":"2024-02-05T07:25:35.295973+0000","last_peered":"2024-02-05T07:25:35.295973+0000","last_clean":"2024-02-05T07:25:35.295973+0000","last_became_active":"2024-02-05T07:25:30.264607+0000","last_became_peered":"2024-02-05T07:25:30.264607+0000","last_unstale":"2024-02-05T07:25:35.295973+0000","last_undegraded":"2024-02-05T07:25:35.295973+0000","last_fullsized":"2024-02-05T07:25:35.295973+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-02-05T07:23:27.310935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:27.310935+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":329,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255233+0000","last_change":"2024-02-05T07:23:08.606203+0000","last_active":"2024-02-05T07:25:30.255233+0000","last_peered":"2024-02-05T07:25:30.255233+0000","last_clean":"2024-02-05T07:25:30.255233+0000","last_became_active":"2024-02-05T07:18:49.430208+0000","last_became_peered":"2024-02-05T07:18:49.430208+0000","last_unstale":"2024-02-05T07:25:30.255233+0000","last_undegraded":"2024-02-05T07:25:30.255233+0000","last_fullsized":"2024-02-05T07:25:30.255233+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:08.606061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:23:08.606061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":235,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256967+0000","last_change":"2024-02-05T07:23:29.341025+0000","last_active":"2024-02-05T07:25:30.256967+0000","last_peered":"2024-02-05T07:25:30.256967+0000","last_clean":"2024-02-05T07:25:30.256967+0000","last_became_active":"2024-02-05T07:23:27.439333+0000","last_became_peered":"2024-02-05T07:23:27.439333+0000","last_unstale":"2024-02-05T07:25:30.256967+0000","last_undegraded":"2024-02-05T07:25:30.256967+0000","last_fullsized":"2024-02-05T07:25:30.256967+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:23:29.340876+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:20:38.993688+0000","last_clean_scrub_stamp":"2024-02-05T07:23:29.340876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":130,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256089+0000","last_change":"2024-02-05T07:24:05.412794+0000","last_active":"2024-02-05T07:25:30.256089+0000","last_peered":"2024-02-05T07:25:30.256089+0000","last_clean":"2024-02-05T07:25:30.256089+0000","last_became_active":"2024-02-05T07:23:47.603091+0000","last_became_peered":"2024-02-05T07:23:47.603091+0000","last_unstale":"2024-02-05T07:25:30.256089+0000","last_undegraded":"2024-02-05T07:25:30.256089+0000","last_fullsized":"2024-02-05T07:25:30.256089+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:05.412661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:05.412661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"218'820","reported_seq":848,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.178560+0000","last_change":"2024-02-05T07:25:16.487417+0000","last_active":"2024-02-05T07:25:35.178560+0000","last_peered":"2024-02-05T07:25:35.178560+0000","last_clean":"2024-02-05T07:25:35.178560+0000","last_became_active":"2024-02-05T07:25:16.486865+0000","last_became_peered":"2024-02-05T07:25:16.486865+0000","last_unstale":"2024-02-05T07:25:35.178560+0000","last_undegraded":"2024-02-05T07:25:35.178560+0000","last_fullsized":"2024-02-05T07:25:35.178560+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:45.019182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53739520,"num_objects":820,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":820,"num_write_kb":52480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":109,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255359+0000","last_change":"2024-02-05T07:24:34.250380+0000","last_active":"2024-02-05T07:25:30.255359+0000","last_peered":"2024-02-05T07:25:30.255359+0000","last_clean":"2024-02-05T07:25:30.255359+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:25:30.255359+0000","last_undegraded":"2024-02-05T07:25:30.255359+0000","last_fullsized":"2024-02-05T07:25:30.255359+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:21.184445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:21.184445+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":132,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.263632+0000","last_change":"2024-02-05T07:24:07.278812+0000","last_active":"2024-02-05T07:25:30.263632+0000","last_peered":"2024-02-05T07:25:30.263632+0000","last_clean":"2024-02-05T07:25:30.263632+0000","last_became_active":"2024-02-05T07:23:47.610698+0000","last_became_peered":"2024-02-05T07:23:47.610698+0000","last_unstale":"2024-02-05T07:25:30.263632+0000","last_undegraded":"2024-02-05T07:25:30.263632+0000","last_fullsized":"2024-02-05T07:25:30.263632+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:07.278696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:07.278696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:35.119282+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":329,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.290360+0000","last_change":"2024-02-05T07:23:47.620787+0000","last_active":"2024-02-05T07:25:35.290360+0000","last_peered":"2024-02-05T07:25:35.290360+0000","last_clean":"2024-02-05T07:25:35.290360+0000","last_became_active":"2024-02-05T07:23:47.619951+0000","last_became_peered":"2024-02-05T07:23:47.619951+0000","last_unstale":"2024-02-05T07:25:35.290360+0000","last_undegraded":"2024-02-05T07:25:35.290360+0000","last_fullsized":"2024-02-05T07:25:35.290360+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:35.587887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:35.587887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":74,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.260579+0000","last_change":"2024-02-05T07:24:34.236399+0000","last_active":"2024-02-05T07:25:30.260579+0000","last_peered":"2024-02-05T07:25:30.260579+0000","last_clean":"2024-02-05T07:25:30.260579+0000","last_became_active":"2024-02-05T07:24:34.235899+0000","last_became_peered":"2024-02-05T07:24:34.235899+0000","last_unstale":"2024-02-05T07:25:30.260579+0000","last_undegraded":"2024-02-05T07:25:30.260579+0000","last_fullsized":"2024-02-05T07:25:30.260579+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:37.209967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.209967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"218'785","reported_seq":813,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.105810+0000","last_change":"2024-02-05T07:25:16.684639+0000","last_active":"2024-02-05T07:25:35.105810+0000","last_peered":"2024-02-05T07:25:35.105810+0000","last_clean":"2024-02-05T07:25:35.105810+0000","last_became_active":"2024-02-05T07:25:16.684246+0000","last_became_peered":"2024-02-05T07:25:16.684246+0000","last_unstale":"2024-02-05T07:25:35.105810+0000","last_undegraded":"2024-02-05T07:25:35.105810+0000","last_fullsized":"2024-02-05T07:25:35.105810+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":785,"log_dups_size":0,"ondisk_log_size":785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:17.415334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51445760,"num_objects":785,"num_object_clones":0,"num_object_copies":2355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":785,"num_write_kb":50240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":364,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.263405+0000","last_change":"2024-02-05T07:24:54.413619+0000","last_active":"2024-02-05T07:25:30.263405+0000","last_peered":"2024-02-05T07:25:30.263405+0000","last_clean":"2024-02-05T07:25:30.263405+0000","last_became_active":"2024-02-05T07:18:51.507672+0000","last_became_peered":"2024-02-05T07:18:51.507672+0000","last_unstale":"2024-02-05T07:25:30.263405+0000","last_undegraded":"2024-02-05T07:25:30.263405+0000","last_fullsized":"2024-02-05T07:25:30.263405+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:24:54.413486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:54.413486+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:14.005875+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":329,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261993+0000","last_change":"2024-02-05T07:21:41.444299+0000","last_active":"2024-02-05T07:25:30.261993+0000","last_peered":"2024-02-05T07:25:30.261993+0000","last_clean":"2024-02-05T07:25:30.261993+0000","last_became_active":"2024-02-05T07:18:49.434451+0000","last_became_peered":"2024-02-05T07:18:49.434451+0000","last_unstale":"2024-02-05T07:25:30.261993+0000","last_undegraded":"2024-02-05T07:25:30.261993+0000","last_fullsized":"2024-02-05T07:25:30.261993+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:41.444193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:41.444193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":322,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.291227+0000","last_change":"2024-02-05T07:21:38.468561+0000","last_active":"2024-02-05T07:25:35.291227+0000","last_peered":"2024-02-05T07:25:35.291227+0000","last_clean":"2024-02-05T07:25:35.291227+0000","last_became_active":"2024-02-05T07:18:53.464867+0000","last_became_peered":"2024-02-05T07:18:53.464867+0000","last_unstale":"2024-02-05T07:25:35.291227+0000","last_undegraded":"2024-02-05T07:25:35.291227+0000","last_fullsized":"2024-02-05T07:25:35.291227+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:38.468456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:38.468456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":323,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.260601+0000","last_change":"2024-02-05T07:24:48.036108+0000","last_active":"2024-02-05T07:25:30.260601+0000","last_peered":"2024-02-05T07:25:30.260601+0000","last_clean":"2024-02-05T07:25:30.260601+0000","last_became_active":"2024-02-05T07:23:47.611066+0000","last_became_peered":"2024-02-05T07:23:47.611066+0000","last_unstale":"2024-02-05T07:25:30.260601+0000","last_undegraded":"2024-02-05T07:25:30.260601+0000","last_fullsized":"2024-02-05T07:25:30.260601+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:48.036033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:48.036033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:55.002158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"218'841","reported_seq":869,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.116286+0000","last_change":"2024-02-05T07:25:16.484041+0000","last_active":"2024-02-05T07:25:35.116286+0000","last_peered":"2024-02-05T07:25:35.116286+0000","last_clean":"2024-02-05T07:25:35.116286+0000","last_became_active":"2024-02-05T07:25:16.483737+0000","last_became_peered":"2024-02-05T07:25:16.483737+0000","last_unstale":"2024-02-05T07:25:35.116286+0000","last_undegraded":"2024-02-05T07:25:35.116286+0000","last_fullsized":"2024-02-05T07:25:35.116286+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:15.929102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55115776,"num_objects":841,"num_object_clones":0,"num_object_copies":2523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":841,"num_write_kb":53824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":331,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.290947+0000","last_change":"2024-02-05T07:24:57.734948+0000","last_active":"2024-02-05T07:25:35.290947+0000","last_peered":"2024-02-05T07:25:35.290947+0000","last_clean":"2024-02-05T07:25:35.290947+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:25:35.290947+0000","last_undegraded":"2024-02-05T07:25:35.290947+0000","last_fullsized":"2024-02-05T07:25:35.290947+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:57.734858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:57.734858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:58.093175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":392,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:32.094078+0000","last_change":"2024-02-05T07:25:32.094129+0000","last_active":"2024-02-05T07:25:32.094078+0000","last_peered":"2024-02-05T07:25:32.094078+0000","last_clean":"2024-02-05T07:25:32.094078+0000","last_became_active":"2024-02-05T07:23:47.607181+0000","last_became_peered":"2024-02-05T07:23:47.607181+0000","last_unstale":"2024-02-05T07:25:32.094078+0000","last_undegraded":"2024-02-05T07:25:32.094078+0000","last_fullsized":"2024-02-05T07:25:32.094078+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:25:32.094000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:32.094000+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:44.366738+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":329,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261966+0000","last_change":"2024-02-05T07:21:31.527396+0000","last_active":"2024-02-05T07:25:30.261966+0000","last_peered":"2024-02-05T07:25:30.261966+0000","last_clean":"2024-02-05T07:25:30.261966+0000","last_became_active":"2024-02-05T07:18:49.435621+0000","last_became_peered":"2024-02-05T07:18:49.435621+0000","last_unstale":"2024-02-05T07:25:30.261966+0000","last_undegraded":"2024-02-05T07:25:30.261966+0000","last_fullsized":"2024-02-05T07:25:30.261966+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:31.527325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:31.527325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":130,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261193+0000","last_change":"2024-02-05T07:24:42.108185+0000","last_active":"2024-02-05T07:25:30.261193+0000","last_peered":"2024-02-05T07:25:30.261193+0000","last_clean":"2024-02-05T07:25:30.261193+0000","last_became_active":"2024-02-05T07:23:47.610289+0000","last_became_peered":"2024-02-05T07:23:47.610289+0000","last_unstale":"2024-02-05T07:25:30.261193+0000","last_undegraded":"2024-02-05T07:25:30.261193+0000","last_fullsized":"2024-02-05T07:25:30.261193+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:42.108043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:42.108043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:50.123540+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"218'783","reported_seq":811,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.047194+0000","last_change":"2024-02-05T07:25:16.486080+0000","last_active":"2024-02-05T07:25:35.047194+0000","last_peered":"2024-02-05T07:25:35.047194+0000","last_clean":"2024-02-05T07:25:35.047194+0000","last_became_active":"2024-02-05T07:25:16.485792+0000","last_became_peered":"2024-02-05T07:25:16.485792+0000","last_unstale":"2024-02-05T07:25:35.047194+0000","last_undegraded":"2024-02-05T07:25:35.047194+0000","last_fullsized":"2024-02-05T07:25:35.047194+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":783,"log_dups_size":0,"ondisk_log_size":783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:27.193089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51314688,"num_objects":783,"num_object_clones":0,"num_object_copies":2349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":783,"num_write_kb":50112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"89'9","reported_seq":412,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261378+0000","last_change":"2024-02-05T07:21:26.505275+0000","last_active":"2024-02-05T07:25:30.261378+0000","last_peered":"2024-02-05T07:25:30.261378+0000","last_clean":"2024-02-05T07:25:30.261378+0000","last_became_active":"2024-02-05T07:18:53.456287+0000","last_became_peered":"2024-02-05T07:18:53.456287+0000","last_unstale":"2024-02-05T07:25:30.261378+0000","last_undegraded":"2024-02-05T07:25:30.261378+0000","last_fullsized":"2024-02-05T07:25:30.261378+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-02-05T07:21:26.505161+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:21:26.505161+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":350,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.258764+0000","last_change":"2024-02-05T07:21:32.875499+0000","last_active":"2024-02-05T07:25:30.258764+0000","last_peered":"2024-02-05T07:25:30.258764+0000","last_clean":"2024-02-05T07:25:30.258764+0000","last_became_active":"2024-02-05T07:18:51.508097+0000","last_became_peered":"2024-02-05T07:18:51.508097+0000","last_unstale":"2024-02-05T07:25:30.258764+0000","last_undegraded":"2024-02-05T07:25:30.258764+0000","last_fullsized":"2024-02-05T07:25:30.258764+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:21:32.875346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:32.875346+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":340,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.262051+0000","last_change":"2024-02-05T07:24:19.277756+0000","last_active":"2024-02-05T07:25:30.262051+0000","last_peered":"2024-02-05T07:25:30.262051+0000","last_clean":"2024-02-05T07:25:30.262051+0000","last_became_active":"2024-02-05T07:18:49.437663+0000","last_became_peered":"2024-02-05T07:18:49.437663+0000","last_unstale":"2024-02-05T07:25:30.262051+0000","last_undegraded":"2024-02-05T07:25:30.262051+0000","last_fullsized":"2024-02-05T07:25:30.262051+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:24:19.277618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:19.277618+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":159,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:34.561348+0000","last_change":"2024-02-05T07:25:34.561389+0000","last_active":"2024-02-05T07:25:34.561348+0000","last_peered":"2024-02-05T07:25:34.561348+0000","last_clean":"2024-02-05T07:25:34.561348+0000","last_became_active":"2024-02-05T07:23:27.419855+0000","last_became_peered":"2024-02-05T07:23:27.419855+0000","last_unstale":"2024-02-05T07:25:34.561348+0000","last_undegraded":"2024-02-05T07:25:34.561348+0000","last_fullsized":"2024-02-05T07:25:34.561348+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:34.561259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:37.032056+0000","last_clean_scrub_stamp":"2024-02-05T07:25:34.561259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:52.049460+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":166,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.262086+0000","last_change":"2024-02-05T07:24:07.263004+0000","last_active":"2024-02-05T07:25:30.262086+0000","last_peered":"2024-02-05T07:25:30.262086+0000","last_clean":"2024-02-05T07:25:30.262086+0000","last_became_active":"2024-02-05T07:23:47.607589+0000","last_became_peered":"2024-02-05T07:23:47.607589+0000","last_unstale":"2024-02-05T07:25:30.262086+0000","last_undegraded":"2024-02-05T07:25:30.262086+0000","last_fullsized":"2024-02-05T07:25:30.262086+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-02-05T07:24:07.262821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:07.262821+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":342,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.263624+0000","last_change":"2024-02-05T07:24:49.359644+0000","last_active":"2024-02-05T07:25:30.263624+0000","last_peered":"2024-02-05T07:25:30.263624+0000","last_clean":"2024-02-05T07:25:30.263624+0000","last_became_active":"2024-02-05T07:18:49.433121+0000","last_became_peered":"2024-02-05T07:18:49.433121+0000","last_unstale":"2024-02-05T07:25:30.263624+0000","last_undegraded":"2024-02-05T07:25:30.263624+0000","last_fullsized":"2024-02-05T07:25:30.263624+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:24:49.359534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.359534+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:55.921947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":326,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.259630+0000","last_change":"2024-02-05T07:25:28.567920+0000","last_active":"2024-02-05T07:25:30.259630+0000","last_peered":"2024-02-05T07:25:30.259630+0000","last_clean":"2024-02-05T07:25:30.259630+0000","last_became_active":"2024-02-05T07:23:47.601556+0000","last_became_peered":"2024-02-05T07:23:47.601556+0000","last_unstale":"2024-02-05T07:25:30.259630+0000","last_undegraded":"2024-02-05T07:25:30.259630+0000","last_fullsized":"2024-02-05T07:25:30.259630+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:28.567844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:28.567844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.349006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":316,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261561+0000","last_change":"2024-02-05T07:21:44.435932+0000","last_active":"2024-02-05T07:25:30.261561+0000","last_peered":"2024-02-05T07:25:30.261561+0000","last_clean":"2024-02-05T07:25:30.261561+0000","last_became_active":"2024-02-05T07:18:55.507555+0000","last_became_peered":"2024-02-05T07:18:55.507555+0000","last_unstale":"2024-02-05T07:25:30.261561+0000","last_undegraded":"2024-02-05T07:25:30.261561+0000","last_fullsized":"2024-02-05T07:25:30.261561+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:21:44.435826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":109,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.262157+0000","last_change":"2024-02-05T07:24:34.274818+0000","last_active":"2024-02-05T07:25:30.262157+0000","last_peered":"2024-02-05T07:25:30.262157+0000","last_clean":"2024-02-05T07:25:30.262157+0000","last_became_active":"2024-02-05T07:24:34.241945+0000","last_became_peered":"2024-02-05T07:24:34.241945+0000","last_unstale":"2024-02-05T07:25:30.262157+0000","last_undegraded":"2024-02-05T07:25:30.262157+0000","last_fullsized":"2024-02-05T07:25:30.262157+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:33.244655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:33.244655+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":328,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.260861+0000","last_change":"2024-02-05T07:21:25.529255+0000","last_active":"2024-02-05T07:25:30.260861+0000","last_peered":"2024-02-05T07:25:30.260861+0000","last_clean":"2024-02-05T07:25:30.260861+0000","last_became_active":"2024-02-05T07:18:49.437732+0000","last_became_peered":"2024-02-05T07:18:49.437732+0000","last_unstale":"2024-02-05T07:25:30.260861+0000","last_undegraded":"2024-02-05T07:25:30.260861+0000","last_fullsized":"2024-02-05T07:25:30.260861+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:25.529174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:21:25.529174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":130,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256315+0000","last_change":"2024-02-05T07:24:47.676565+0000","last_active":"2024-02-05T07:25:30.256315+0000","last_peered":"2024-02-05T07:25:30.256315+0000","last_clean":"2024-02-05T07:25:30.256315+0000","last_became_active":"2024-02-05T07:23:47.606266+0000","last_became_peered":"2024-02-05T07:23:47.606266+0000","last_unstale":"2024-02-05T07:25:30.256315+0000","last_undegraded":"2024-02-05T07:25:30.256315+0000","last_fullsized":"2024-02-05T07:25:30.256315+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:47.676484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:47.676484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:56.595972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":318,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.296255+0000","last_change":"2024-02-05T07:21:39.515473+0000","last_active":"2024-02-05T07:25:35.296255+0000","last_peered":"2024-02-05T07:25:35.296255+0000","last_clean":"2024-02-05T07:25:35.296255+0000","last_became_active":"2024-02-05T07:18:55.469044+0000","last_became_peered":"2024-02-05T07:18:55.469044+0000","last_unstale":"2024-02-05T07:25:35.296255+0000","last_undegraded":"2024-02-05T07:25:35.296255+0000","last_fullsized":"2024-02-05T07:25:35.296255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:39.515332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.515332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":356,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.260076+0000","last_change":"2024-02-05T07:23:47.619736+0000","last_active":"2024-02-05T07:25:30.260076+0000","last_peered":"2024-02-05T07:25:30.260076+0000","last_clean":"2024-02-05T07:25:30.260076+0000","last_became_active":"2024-02-05T07:23:47.602628+0000","last_became_peered":"2024-02-05T07:23:47.602628+0000","last_unstale":"2024-02-05T07:25:30.260076+0000","last_undegraded":"2024-02-05T07:25:30.260076+0000","last_fullsized":"2024-02-05T07:25:30.260076+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:30.859017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:30.859017+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":289,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.260712+0000","last_change":"2024-02-05T07:24:49.035482+0000","last_active":"2024-02-05T07:25:30.260712+0000","last_peered":"2024-02-05T07:25:30.260712+0000","last_clean":"2024-02-05T07:25:30.260712+0000","last_became_active":"2024-02-05T07:24:34.230825+0000","last_became_peered":"2024-02-05T07:24:34.230825+0000","last_unstale":"2024-02-05T07:25:30.260712+0000","last_undegraded":"2024-02-05T07:25:30.260712+0000","last_fullsized":"2024-02-05T07:25:30.260712+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:49.035385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.035385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:59.566665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"164'9","reported_seq":173,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261896+0000","last_change":"2024-02-05T07:23:47.620866+0000","last_active":"2024-02-05T07:25:30.261896+0000","last_peered":"2024-02-05T07:25:30.261896+0000","last_clean":"2024-02-05T07:25:30.261896+0000","last_became_active":"2024-02-05T07:23:47.605296+0000","last_became_peered":"2024-02-05T07:23:47.605296+0000","last_unstale":"2024-02-05T07:25:30.261896+0000","last_undegraded":"2024-02-05T07:25:30.261896+0000","last_fullsized":"2024-02-05T07:25:30.261896+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:22:58.051782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:22:58.051782+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":316,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256318+0000","last_change":"2024-02-05T07:24:14.511046+0000","last_active":"2024-02-05T07:25:30.256318+0000","last_peered":"2024-02-05T07:25:30.256318+0000","last_clean":"2024-02-05T07:25:30.256318+0000","last_became_active":"2024-02-05T07:18:55.467044+0000","last_became_peered":"2024-02-05T07:18:55.467044+0000","last_unstale":"2024-02-05T07:25:30.256318+0000","last_undegraded":"2024-02-05T07:25:30.256318+0000","last_fullsized":"2024-02-05T07:25:30.256318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:14.510933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:14.510933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:37.853613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"113'15","reported_seq":23,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.270864+0000","last_change":"2024-02-05T07:25:30.270864+0000","last_active":"2024-02-05T07:25:30.270864+0000","last_peered":"2024-02-05T07:25:30.270864+0000","last_clean":"2024-02-05T07:25:30.270864+0000","last_became_active":"2024-02-05T07:25:30.262689+0000","last_became_peered":"2024-02-05T07:25:30.262689+0000","last_unstale":"2024-02-05T07:25:30.270864+0000","last_undegraded":"2024-02-05T07:25:30.270864+0000","last_fullsized":"2024-02-05T07:25:30.270864+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'15","last_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:21:36.359546+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":328,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.260921+0000","last_change":"2024-02-05T07:24:53.108878+0000","last_active":"2024-02-05T07:25:30.260921+0000","last_peered":"2024-02-05T07:25:30.260921+0000","last_clean":"2024-02-05T07:25:30.260921+0000","last_became_active":"2024-02-05T07:18:49.436131+0000","last_became_peered":"2024-02-05T07:18:49.436131+0000","last_unstale":"2024-02-05T07:25:30.260921+0000","last_undegraded":"2024-02-05T07:25:30.260921+0000","last_fullsized":"2024-02-05T07:25:30.260921+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:53.108801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:53.108801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:10.506459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"217'9","reported_seq":33,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.296551+0000","last_change":"2024-02-05T07:25:31.727799+0000","last_active":"2024-02-05T07:25:35.296551+0000","last_peered":"2024-02-05T07:25:35.296551+0000","last_clean":"2024-02-05T07:25:35.296551+0000","last_became_active":"2024-02-05T07:25:27.246616+0000","last_became_peered":"2024-02-05T07:25:27.246616+0000","last_unstale":"2024-02-05T07:25:35.296551+0000","last_undegraded":"2024-02-05T07:25:35.296551+0000","last_fullsized":"2024-02-05T07:25:35.296551+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9","last_scrub_stamp":"2024-02-05T07:25:31.727658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:31.727658+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:44.449404+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":269,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261746+0000","last_change":"2024-02-05T07:24:34.242269+0000","last_active":"2024-02-05T07:25:30.261746+0000","last_peered":"2024-02-05T07:25:30.261746+0000","last_clean":"2024-02-05T07:25:30.261746+0000","last_became_active":"2024-02-05T07:24:34.238479+0000","last_became_peered":"2024-02-05T07:24:34.238479+0000","last_unstale":"2024-02-05T07:25:30.261746+0000","last_undegraded":"2024-02-05T07:25:30.261746+0000","last_fullsized":"2024-02-05T07:25:30.261746+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:16.156709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:16.156709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:30.913274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":361,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255618+0000","last_change":"2024-02-05T07:21:37.876638+0000","last_active":"2024-02-05T07:25:30.255618+0000","last_peered":"2024-02-05T07:25:30.255618+0000","last_clean":"2024-02-05T07:25:30.255618+0000","last_became_active":"2024-02-05T07:19:16.121696+0000","last_became_peered":"2024-02-05T07:19:16.121696+0000","last_unstale":"2024-02-05T07:25:30.255618+0000","last_undegraded":"2024-02-05T07:25:30.255618+0000","last_fullsized":"2024-02-05T07:25:30.255618+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:21:37.876511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:37.876511+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":328,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255513+0000","last_change":"2024-02-05T07:23:23.435405+0000","last_active":"2024-02-05T07:25:30.255513+0000","last_peered":"2024-02-05T07:25:30.255513+0000","last_clean":"2024-02-05T07:25:30.255513+0000","last_became_active":"2024-02-05T07:18:49.430766+0000","last_became_peered":"2024-02-05T07:18:49.430766+0000","last_unstale":"2024-02-05T07:25:30.255513+0000","last_undegraded":"2024-02-05T07:25:30.255513+0000","last_fullsized":"2024-02-05T07:25:30.255513+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:23:23.435135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":327,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.257460+0000","last_change":"2024-02-05T07:23:47.594349+0000","last_active":"2024-02-05T07:25:30.257460+0000","last_peered":"2024-02-05T07:25:30.257460+0000","last_clean":"2024-02-05T07:25:30.257460+0000","last_became_active":"2024-02-05T07:23:47.594030+0000","last_became_peered":"2024-02-05T07:23:47.594030+0000","last_unstale":"2024-02-05T07:25:30.257460+0000","last_undegraded":"2024-02-05T07:25:30.257460+0000","last_fullsized":"2024-02-05T07:25:30.257460+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:22:59.851438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:22:59.851438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":86,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.290691+0000","last_change":"2024-02-05T07:24:55.749908+0000","last_active":"2024-02-05T07:25:35.290691+0000","last_peered":"2024-02-05T07:25:35.290691+0000","last_clean":"2024-02-05T07:25:35.290691+0000","last_became_active":"2024-02-05T07:24:34.229238+0000","last_became_peered":"2024-02-05T07:24:34.229238+0000","last_unstale":"2024-02-05T07:25:35.290691+0000","last_undegraded":"2024-02-05T07:25:35.290691+0000","last_fullsized":"2024-02-05T07:25:35.290691+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:55.749767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:55.749767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:07.800010+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":304,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.259199+0000","last_change":"2024-02-05T07:21:16.721101+0000","last_active":"2024-02-05T07:25:30.259199+0000","last_peered":"2024-02-05T07:25:30.259199+0000","last_clean":"2024-02-05T07:25:30.259199+0000","last_became_active":"2024-02-05T07:19:16.122059+0000","last_became_peered":"2024-02-05T07:19:16.122059+0000","last_unstale":"2024-02-05T07:25:30.259199+0000","last_undegraded":"2024-02-05T07:25:30.259199+0000","last_fullsized":"2024-02-05T07:25:30.259199+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:21:16.720264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:16.720264+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":326,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261710+0000","last_change":"2024-02-05T07:23:47.616365+0000","last_active":"2024-02-05T07:25:30.261710+0000","last_peered":"2024-02-05T07:25:30.261710+0000","last_clean":"2024-02-05T07:25:30.261710+0000","last_became_active":"2024-02-05T07:23:47.616029+0000","last_became_peered":"2024-02-05T07:23:47.616029+0000","last_unstale":"2024-02-05T07:25:30.261710+0000","last_undegraded":"2024-02-05T07:25:30.261710+0000","last_fullsized":"2024-02-05T07:25:30.261710+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:20.600583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:20.600583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":76,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255456+0000","last_change":"2024-02-05T07:24:34.249659+0000","last_active":"2024-02-05T07:25:30.255456+0000","last_peered":"2024-02-05T07:25:30.255456+0000","last_clean":"2024-02-05T07:25:30.255456+0000","last_became_active":"2024-02-05T07:24:34.231409+0000","last_became_peered":"2024-02-05T07:24:34.231409+0000","last_unstale":"2024-02-05T07:25:30.255456+0000","last_undegraded":"2024-02-05T07:25:30.255456+0000","last_fullsized":"2024-02-05T07:25:30.255456+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:02.051772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:02.051772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":74,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255041+0000","last_change":"2024-02-05T07:24:34.231186+0000","last_active":"2024-02-05T07:25:30.255041+0000","last_peered":"2024-02-05T07:25:30.255041+0000","last_clean":"2024-02-05T07:25:30.255041+0000","last_became_active":"2024-02-05T07:24:34.230006+0000","last_became_peered":"2024-02-05T07:24:34.230006+0000","last_unstale":"2024-02-05T07:25:30.255041+0000","last_undegraded":"2024-02-05T07:25:30.255041+0000","last_fullsized":"2024-02-05T07:25:30.255041+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:24:20.215062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:42.873758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":172,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261235+0000","last_change":"2024-02-05T07:23:47.638791+0000","last_active":"2024-02-05T07:25:30.261235+0000","last_peered":"2024-02-05T07:25:30.261235+0000","last_clean":"2024-02-05T07:25:30.261235+0000","last_became_active":"2024-02-05T07:23:47.613111+0000","last_became_peered":"2024-02-05T07:23:47.613111+0000","last_unstale":"2024-02-05T07:25:30.261235+0000","last_undegraded":"2024-02-05T07:25:30.261235+0000","last_fullsized":"2024-02-05T07:25:30.261235+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:21:18.095797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:18.095797+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":328,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.259730+0000","last_change":"2024-02-05T07:21:29.818030+0000","last_active":"2024-02-05T07:25:30.259730+0000","last_peered":"2024-02-05T07:25:30.259730+0000","last_clean":"2024-02-05T07:25:30.259730+0000","last_became_active":"2024-02-05T07:18:49.434560+0000","last_became_peered":"2024-02-05T07:18:49.434560+0000","last_unstale":"2024-02-05T07:25:30.259730+0000","last_undegraded":"2024-02-05T07:25:30.259730+0000","last_fullsized":"2024-02-05T07:25:30.259730+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:29.817921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:29.817921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":328,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:34.862202+0000","last_change":"2024-02-05T07:25:34.862235+0000","last_active":"2024-02-05T07:25:34.862202+0000","last_peered":"2024-02-05T07:25:34.862202+0000","last_clean":"2024-02-05T07:25:34.862202+0000","last_became_active":"2024-02-05T07:18:53.460977+0000","last_became_peered":"2024-02-05T07:18:53.460977+0000","last_unstale":"2024-02-05T07:25:34.862202+0000","last_undegraded":"2024-02-05T07:25:34.862202+0000","last_fullsized":"2024-02-05T07:25:34.862202+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:34.862131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_clean_scrub_stamp":"2024-02-05T07:25:34.862131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:53.060690+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.263349+0000","last_change":"2024-02-05T07:24:46.423838+0000","last_active":"2024-02-05T07:25:30.263349+0000","last_peered":"2024-02-05T07:25:30.263349+0000","last_clean":"2024-02-05T07:25:30.263349+0000","last_became_active":"2024-02-05T07:24:34.225863+0000","last_became_peered":"2024-02-05T07:24:34.225863+0000","last_unstale":"2024-02-05T07:25:30.263349+0000","last_undegraded":"2024-02-05T07:25:30.263349+0000","last_fullsized":"2024-02-05T07:25:30.263349+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:46.423760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:46.423760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:58.661226+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":161,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:31.462854+0000","last_change":"2024-02-05T07:25:31.462890+0000","last_active":"2024-02-05T07:25:31.462854+0000","last_peered":"2024-02-05T07:25:31.462854+0000","last_clean":"2024-02-05T07:25:31.462854+0000","last_became_active":"2024-02-05T07:23:47.610110+0000","last_became_peered":"2024-02-05T07:23:47.610110+0000","last_unstale":"2024-02-05T07:25:31.462854+0000","last_undegraded":"2024-02-05T07:25:31.462854+0000","last_fullsized":"2024-02-05T07:25:31.462854+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-02-05T07:25:31.462787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:31.462787+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:50.246334+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":281,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.260107+0000","last_change":"2024-02-05T07:24:34.229700+0000","last_active":"2024-02-05T07:25:30.260107+0000","last_peered":"2024-02-05T07:25:30.260107+0000","last_clean":"2024-02-05T07:25:30.260107+0000","last_became_active":"2024-02-05T07:24:34.229229+0000","last_became_peered":"2024-02-05T07:24:34.229229+0000","last_unstale":"2024-02-05T07:25:30.260107+0000","last_undegraded":"2024-02-05T07:25:30.260107+0000","last_fullsized":"2024-02-05T07:25:30.260107+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:17.202540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:17.202540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":328,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:32.554704+0000","last_change":"2024-02-05T07:25:32.554739+0000","last_active":"2024-02-05T07:25:32.554704+0000","last_peered":"2024-02-05T07:25:32.554704+0000","last_clean":"2024-02-05T07:25:32.554704+0000","last_became_active":"2024-02-05T07:18:53.460307+0000","last_became_peered":"2024-02-05T07:18:53.460307+0000","last_unstale":"2024-02-05T07:25:32.554704+0000","last_undegraded":"2024-02-05T07:25:32.554704+0000","last_fullsized":"2024-02-05T07:25:32.554704+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:32.554622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:25:32.554622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:41.257524+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":316,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.260979+0000","last_change":"2024-02-05T07:21:30.473743+0000","last_active":"2024-02-05T07:25:30.260979+0000","last_peered":"2024-02-05T07:25:30.260979+0000","last_clean":"2024-02-05T07:25:30.260979+0000","last_became_active":"2024-02-05T07:18:55.468086+0000","last_became_peered":"2024-02-05T07:18:55.468086+0000","last_unstale":"2024-02-05T07:25:30.260979+0000","last_undegraded":"2024-02-05T07:25:30.260979+0000","last_fullsized":"2024-02-05T07:25:30.260979+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:30.473582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:30.473582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":352,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.264510+0000","last_change":"2024-02-05T07:24:53.457250+0000","last_active":"2024-02-05T07:25:30.264510+0000","last_peered":"2024-02-05T07:25:30.264510+0000","last_clean":"2024-02-05T07:25:30.264510+0000","last_became_active":"2024-02-05T07:18:51.446604+0000","last_became_peered":"2024-02-05T07:18:51.446604+0000","last_unstale":"2024-02-05T07:25:30.264510+0000","last_undegraded":"2024-02-05T07:25:30.264510+0000","last_fullsized":"2024-02-05T07:25:30.264510+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:24:53.457167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:53.457167+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:08.339750+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":266,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255628+0000","last_change":"2024-02-05T07:24:54.528959+0000","last_active":"2024-02-05T07:25:30.255628+0000","last_peered":"2024-02-05T07:25:30.255628+0000","last_clean":"2024-02-05T07:25:30.255628+0000","last_became_active":"2024-02-05T07:24:34.209580+0000","last_became_peered":"2024-02-05T07:24:34.209580+0000","last_unstale":"2024-02-05T07:25:30.255628+0000","last_undegraded":"2024-02-05T07:25:30.255628+0000","last_fullsized":"2024-02-05T07:25:30.255628+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:54.528848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:54.528848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:13.804773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":82,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.943013+0000","last_change":"2024-02-05T07:25:30.943037+0000","last_active":"2024-02-05T07:25:30.943013+0000","last_peered":"2024-02-05T07:25:30.943013+0000","last_clean":"2024-02-05T07:25:30.943013+0000","last_became_active":"2024-02-05T07:24:34.240115+0000","last_became_peered":"2024-02-05T07:24:34.240115+0000","last_unstale":"2024-02-05T07:25:30.943013+0000","last_undegraded":"2024-02-05T07:25:30.943013+0000","last_fullsized":"2024-02-05T07:25:30.943013+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:30.942942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:30.942942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:51.102432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":325,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256232+0000","last_change":"2024-02-05T07:22:55.794256+0000","last_active":"2024-02-05T07:25:30.256232+0000","last_peered":"2024-02-05T07:25:30.256232+0000","last_clean":"2024-02-05T07:25:30.256232+0000","last_became_active":"2024-02-05T07:18:55.466164+0000","last_became_peered":"2024-02-05T07:18:55.466164+0000","last_unstale":"2024-02-05T07:25:30.256232+0000","last_undegraded":"2024-02-05T07:25:30.256232+0000","last_fullsized":"2024-02-05T07:25:30.256232+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:22:55.794149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:22:55.794149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":116,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.262256+0000","last_change":"2024-02-05T07:24:46.018438+0000","last_active":"2024-02-05T07:25:30.262256+0000","last_peered":"2024-02-05T07:25:30.262256+0000","last_clean":"2024-02-05T07:25:30.262256+0000","last_became_active":"2024-02-05T07:24:34.237056+0000","last_became_peered":"2024-02-05T07:24:34.237056+0000","last_unstale":"2024-02-05T07:25:30.262256+0000","last_undegraded":"2024-02-05T07:25:30.262256+0000","last_fullsized":"2024-02-05T07:25:30.262256+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:24:46.018340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:46.018340+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:12.154764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":340,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255417+0000","last_change":"2024-02-05T07:23:47.646441+0000","last_active":"2024-02-05T07:25:30.255417+0000","last_peered":"2024-02-05T07:25:30.255417+0000","last_clean":"2024-02-05T07:25:30.255417+0000","last_became_active":"2024-02-05T07:23:47.610436+0000","last_became_peered":"2024-02-05T07:23:47.610436+0000","last_unstale":"2024-02-05T07:25:30.255417+0000","last_undegraded":"2024-02-05T07:25:30.255417+0000","last_fullsized":"2024-02-05T07:25:30.255417+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:21:42.616129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:42.616129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":422,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:31.524626+0000","last_change":"2024-02-05T07:25:31.524626+0000","last_active":"2024-02-05T07:25:31.524626+0000","last_peered":"2024-02-05T07:25:31.524626+0000","last_clean":"2024-02-05T07:25:31.524626+0000","last_became_active":"2024-02-05T07:18:53.461011+0000","last_became_peered":"2024-02-05T07:18:53.461011+0000","last_unstale":"2024-02-05T07:25:31.524626+0000","last_undegraded":"2024-02-05T07:25:31.524626+0000","last_fullsized":"2024-02-05T07:25:31.524626+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:25:31.524507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:31.524507+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:48.356847+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":318,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261770+0000","last_change":"2024-02-05T07:21:39.490602+0000","last_active":"2024-02-05T07:25:30.261770+0000","last_peered":"2024-02-05T07:25:30.261770+0000","last_clean":"2024-02-05T07:25:30.261770+0000","last_became_active":"2024-02-05T07:18:55.476071+0000","last_became_peered":"2024-02-05T07:18:55.476071+0000","last_unstale":"2024-02-05T07:25:30.261770+0000","last_undegraded":"2024-02-05T07:25:30.261770+0000","last_fullsized":"2024-02-05T07:25:30.261770+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.490479+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":184,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255978+0000","last_change":"2024-02-05T07:24:46.667850+0000","last_active":"2024-02-05T07:25:30.255978+0000","last_peered":"2024-02-05T07:25:30.255978+0000","last_clean":"2024-02-05T07:25:30.255978+0000","last_became_active":"2024-02-05T07:23:47.610303+0000","last_became_peered":"2024-02-05T07:23:47.610303+0000","last_unstale":"2024-02-05T07:25:30.255978+0000","last_undegraded":"2024-02-05T07:25:30.255978+0000","last_fullsized":"2024-02-05T07:25:30.255978+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:24:46.667757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:46.667757+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:49.251381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.261406+0000","last_change":"2024-02-05T07:25:30.261406+0000","last_active":"2024-02-05T07:25:30.261406+0000","last_peered":"2024-02-05T07:25:30.261406+0000","last_clean":"2024-02-05T07:25:30.261406+0000","last_became_active":"2024-02-05T07:25:30.261037+0000","last_became_peered":"2024-02-05T07:25:30.261037+0000","last_unstale":"2024-02-05T07:25:30.261406+0000","last_undegraded":"2024-02-05T07:25:30.261406+0000","last_fullsized":"2024-02-05T07:25:30.261406+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:10.630020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:10.630020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":334,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:34.068405+0000","last_change":"2024-02-05T07:25:34.068446+0000","last_active":"2024-02-05T07:25:34.068405+0000","last_peered":"2024-02-05T07:25:34.068405+0000","last_clean":"2024-02-05T07:25:34.068405+0000","last_became_active":"2024-02-05T07:23:47.605523+0000","last_became_peered":"2024-02-05T07:23:47.605523+0000","last_unstale":"2024-02-05T07:25:34.068405+0000","last_undegraded":"2024-02-05T07:25:34.068405+0000","last_fullsized":"2024-02-05T07:25:34.068405+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:34.068335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:34.068335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:42.963475+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":130,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.264653+0000","last_change":"2024-02-05T07:24:58.410795+0000","last_active":"2024-02-05T07:25:30.264653+0000","last_peered":"2024-02-05T07:25:30.264653+0000","last_clean":"2024-02-05T07:25:30.264653+0000","last_became_active":"2024-02-05T07:23:47.609917+0000","last_became_peered":"2024-02-05T07:23:47.609917+0000","last_unstale":"2024-02-05T07:25:30.264653+0000","last_undegraded":"2024-02-05T07:25:30.264653+0000","last_fullsized":"2024-02-05T07:25:30.264653+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:58.410667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:58.410667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:28.003988+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":373,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.295858+0000","last_change":"2024-02-05T07:23:47.657951+0000","last_active":"2024-02-05T07:25:35.295858+0000","last_peered":"2024-02-05T07:25:35.295858+0000","last_clean":"2024-02-05T07:25:35.295858+0000","last_became_active":"2024-02-05T07:23:47.619716+0000","last_became_peered":"2024-02-05T07:23:47.619716+0000","last_unstale":"2024-02-05T07:25:35.295858+0000","last_undegraded":"2024-02-05T07:25:35.295858+0000","last_fullsized":"2024-02-05T07:25:35.295858+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:21:36.561503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:36.561503+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":82,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:33.059199+0000","last_change":"2024-02-05T07:25:33.059237+0000","last_active":"2024-02-05T07:25:33.059199+0000","last_peered":"2024-02-05T07:25:33.059199+0000","last_clean":"2024-02-05T07:25:33.059199+0000","last_became_active":"2024-02-05T07:24:34.234199+0000","last_became_peered":"2024-02-05T07:24:34.234199+0000","last_unstale":"2024-02-05T07:25:33.059199+0000","last_undegraded":"2024-02-05T07:25:33.059199+0000","last_fullsized":"2024-02-05T07:25:33.059199+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:33.059121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:33.059121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:49.169289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":321,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256070+0000","last_change":"2024-02-05T07:22:58.849105+0000","last_active":"2024-02-05T07:25:30.256070+0000","last_peered":"2024-02-05T07:25:30.256070+0000","last_clean":"2024-02-05T07:25:30.256070+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:25:30.256070+0000","last_undegraded":"2024-02-05T07:25:30.256070+0000","last_fullsized":"2024-02-05T07:25:30.256070+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:22:58.846677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:22:58.846677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":317,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.256280+0000","last_change":"2024-02-05T07:21:38.839453+0000","last_active":"2024-02-05T07:25:30.256280+0000","last_peered":"2024-02-05T07:25:30.256280+0000","last_clean":"2024-02-05T07:25:30.256280+0000","last_became_active":"2024-02-05T07:18:55.505678+0000","last_became_peered":"2024-02-05T07:18:55.505678+0000","last_unstale":"2024-02-05T07:25:30.256280+0000","last_undegraded":"2024-02-05T07:25:30.256280+0000","last_fullsized":"2024-02-05T07:25:30.256280+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:38.839336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:38.839336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":346,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255433+0000","last_change":"2024-02-05T07:24:53.834758+0000","last_active":"2024-02-05T07:25:30.255433+0000","last_peered":"2024-02-05T07:25:30.255433+0000","last_clean":"2024-02-05T07:25:30.255433+0000","last_became_active":"2024-02-05T07:23:47.603918+0000","last_became_peered":"2024-02-05T07:23:47.603918+0000","last_unstale":"2024-02-05T07:25:30.255433+0000","last_undegraded":"2024-02-05T07:25:30.255433+0000","last_fullsized":"2024-02-05T07:25:30.255433+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:24:53.834622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:53.834622+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:57.247218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":143,"reported_epoch":218,"state":"active+clean+scrubbing","last_fresh":"2024-02-05T07:25:30.255615+0000","last_change":"2024-02-05T07:25:35.055661+0000","last_active":"2024-02-05T07:25:30.255615+0000","last_peered":"2024-02-05T07:25:30.255615+0000","last_clean":"2024-02-05T07:25:30.255615+0000","last_became_active":"2024-02-05T07:23:27.437776+0000","last_became_peered":"2024-02-05T07:23:27.437776+0000","last_unstale":"2024-02-05T07:25:30.255615+0000","last_undegraded":"2024-02-05T07:25:30.255615+0000","last_fullsized":"2024-02-05T07:25:30.255615+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:39.936643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:39.936643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":320,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.260878+0000","last_change":"2024-02-05T07:24:16.310748+0000","last_active":"2024-02-05T07:25:30.260878+0000","last_peered":"2024-02-05T07:25:30.260878+0000","last_clean":"2024-02-05T07:25:30.260878+0000","last_became_active":"2024-02-05T07:18:53.456881+0000","last_became_peered":"2024-02-05T07:18:53.456881+0000","last_unstale":"2024-02-05T07:25:30.260878+0000","last_undegraded":"2024-02-05T07:25:30.260878+0000","last_fullsized":"2024-02-05T07:25:30.260878+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:16.310666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:16.310666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:30.390505+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":314,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.258934+0000","last_change":"2024-02-05T07:23:47.597763+0000","last_active":"2024-02-05T07:25:30.258934+0000","last_peered":"2024-02-05T07:25:30.258934+0000","last_clean":"2024-02-05T07:25:30.258934+0000","last_became_active":"2024-02-05T07:23:47.596885+0000","last_became_peered":"2024-02-05T07:23:47.596885+0000","last_unstale":"2024-02-05T07:25:30.258934+0000","last_undegraded":"2024-02-05T07:25:30.258934+0000","last_fullsized":"2024-02-05T07:25:30.258934+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:15.751084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:21:15.751084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"89'13","reported_seq":368,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255431+0000","last_change":"2024-02-05T07:25:26.134223+0000","last_active":"2024-02-05T07:25:30.255431+0000","last_peered":"2024-02-05T07:25:30.255431+0000","last_clean":"2024-02-05T07:25:30.255431+0000","last_became_active":"2024-02-05T07:23:47.618422+0000","last_became_peered":"2024-02-05T07:23:47.618422+0000","last_unstale":"2024-02-05T07:25:30.255431+0000","last_undegraded":"2024-02-05T07:25:30.255431+0000","last_fullsized":"2024-02-05T07:25:30.255431+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-05T07:25:26.134110+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:25:26.134110+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:48.192200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":150,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255941+0000","last_change":"2024-02-05T07:25:28.169453+0000","last_active":"2024-02-05T07:25:30.255941+0000","last_peered":"2024-02-05T07:25:30.255941+0000","last_clean":"2024-02-05T07:25:30.255941+0000","last_became_active":"2024-02-05T07:23:27.417717+0000","last_became_peered":"2024-02-05T07:23:27.417717+0000","last_unstale":"2024-02-05T07:25:30.255941+0000","last_undegraded":"2024-02-05T07:25:30.255941+0000","last_fullsized":"2024-02-05T07:25:30.255941+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:28.169335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:28.169335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:28.615234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":312,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.259298+0000","last_change":"2024-02-05T07:21:19.716500+0000","last_active":"2024-02-05T07:25:30.259298+0000","last_peered":"2024-02-05T07:25:30.259298+0000","last_clean":"2024-02-05T07:25:30.259298+0000","last_became_active":"2024-02-05T07:18:53.464520+0000","last_became_peered":"2024-02-05T07:18:53.464520+0000","last_unstale":"2024-02-05T07:25:30.259298+0000","last_undegraded":"2024-02-05T07:25:30.259298+0000","last_fullsized":"2024-02-05T07:25:30.259298+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:19.716375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:21:19.716375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":82,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255392+0000","last_change":"2024-02-05T07:24:49.612194+0000","last_active":"2024-02-05T07:25:30.255392+0000","last_peered":"2024-02-05T07:25:30.255392+0000","last_clean":"2024-02-05T07:25:30.255392+0000","last_became_active":"2024-02-05T07:24:34.230537+0000","last_became_peered":"2024-02-05T07:24:34.230537+0000","last_unstale":"2024-02-05T07:25:30.255392+0000","last_undegraded":"2024-02-05T07:25:30.255392+0000","last_fullsized":"2024-02-05T07:25:30.255392+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:49.612116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.612116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:07.615104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":184,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.290912+0000","last_change":"2024-02-05T07:23:47.645884+0000","last_active":"2024-02-05T07:25:35.290912+0000","last_peered":"2024-02-05T07:25:35.290912+0000","last_clean":"2024-02-05T07:25:35.290912+0000","last_became_active":"2024-02-05T07:23:47.621081+0000","last_became_peered":"2024-02-05T07:23:47.621081+0000","last_unstale":"2024-02-05T07:25:35.290912+0000","last_undegraded":"2024-02-05T07:25:35.290912+0000","last_fullsized":"2024-02-05T07:25:35.290912+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:21:27.141800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:27.141800+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":130,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.264014+0000","last_change":"2024-02-05T07:24:15.235343+0000","last_active":"2024-02-05T07:25:30.264014+0000","last_peered":"2024-02-05T07:25:30.264014+0000","last_clean":"2024-02-05T07:25:30.264014+0000","last_became_active":"2024-02-05T07:23:47.610737+0000","last_became_peered":"2024-02-05T07:23:47.610737+0000","last_unstale":"2024-02-05T07:25:30.264014+0000","last_undegraded":"2024-02-05T07:25:30.264014+0000","last_fullsized":"2024-02-05T07:25:30.264014+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:15.235232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:15.235232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:25:43.769610+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":328,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.254928+0000","last_change":"2024-02-05T07:24:51.823741+0000","last_active":"2024-02-05T07:25:30.254928+0000","last_peered":"2024-02-05T07:25:30.254928+0000","last_clean":"2024-02-05T07:25:30.254928+0000","last_became_active":"2024-02-05T07:18:53.461506+0000","last_became_peered":"2024-02-05T07:18:53.461506+0000","last_unstale":"2024-02-05T07:25:30.254928+0000","last_undegraded":"2024-02-05T07:25:30.254928+0000","last_fullsized":"2024-02-05T07:25:30.254928+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:51.823640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:51.823640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:18.329269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":130,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.260809+0000","last_change":"2024-02-05T07:24:09.290070+0000","last_active":"2024-02-05T07:25:30.260809+0000","last_peered":"2024-02-05T07:25:30.260809+0000","last_clean":"2024-02-05T07:25:30.260809+0000","last_became_active":"2024-02-05T07:23:47.609992+0000","last_became_peered":"2024-02-05T07:23:47.609992+0000","last_unstale":"2024-02-05T07:25:30.260809+0000","last_undegraded":"2024-02-05T07:25:30.260809+0000","last_fullsized":"2024-02-05T07:25:30.260809+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:09.289989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:09.289989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":340,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255130+0000","last_change":"2024-02-05T07:23:15.477701+0000","last_active":"2024-02-05T07:25:30.255130+0000","last_peered":"2024-02-05T07:25:30.255130+0000","last_clean":"2024-02-05T07:25:30.255130+0000","last_became_active":"2024-02-05T07:18:51.448924+0000","last_became_peered":"2024-02-05T07:18:51.448924+0000","last_unstale":"2024-02-05T07:25:30.255130+0000","last_undegraded":"2024-02-05T07:25:30.255130+0000","last_fullsized":"2024-02-05T07:25:30.255130+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:23:15.477555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:15.477555+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":84,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-02-05T07:25:35.297095+0000","last_change":"2024-02-05T07:24:56.703719+0000","last_active":"2024-02-05T07:25:35.297095+0000","last_peered":"2024-02-05T07:25:35.297095+0000","last_clean":"2024-02-05T07:25:35.297095+0000","last_became_active":"2024-02-05T07:24:34.229294+0000","last_became_peered":"2024-02-05T07:24:34.229294+0000","last_unstale":"2024-02-05T07:25:35.297095+0000","last_undegraded":"2024-02-05T07:25:35.297095+0000","last_fullsized":"2024-02-05T07:25:35.297095+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:56.703642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:56.703642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:13.419627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":321,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255512+0000","last_change":"2024-02-05T07:23:07.907068+0000","last_active":"2024-02-05T07:25:30.255512+0000","last_peered":"2024-02-05T07:25:30.255512+0000","last_clean":"2024-02-05T07:25:30.255512+0000","last_became_active":"2024-02-05T07:18:53.460820+0000","last_became_peered":"2024-02-05T07:18:53.460820+0000","last_unstale":"2024-02-05T07:25:30.255512+0000","last_undegraded":"2024-02-05T07:25:30.255512+0000","last_fullsized":"2024-02-05T07:25:30.255512+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:07.906963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:23:07.906963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":316,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-02-05T07:25:30.255229+0000","last_change":"2024-02-05T07:24:06.379008+0000","last_active":"2024-02-05T07:25:30.255229+0000","last_peered":"2024-02-05T07:25:30.255229+0000","last_clean":"2024-02-05T07:25:30.255229+0000","last_became_active":"2024-02-05T07:18:55.466001+0000","last_became_peered":"2024-02-05T07:18:55.466001+0000","last_unstale":"2024-02-05T07:25:30.255229+0000","last_undegraded":"2024-02-05T07:25:30.255229+0000","last_fullsized":"2024-02-05T07:25:30.255229+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:06.378880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:06.378880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":838729728,"num_objects":12798,"num_object_clones":0,"num_object_copies":38394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12798,"num_write_kb":819072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2574319616,"data_stored":2574319616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12798,"ondisk_log_size":12798,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":398,"num_read_kb":393,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":686,"num_read_kb":443,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":433,"ondisk_log_size":433,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037846,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":449800,"kb_used_data":238144,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93295352,"statfs":{"total":95995035648,"available":95534440448,"internally_reserved":0,"allocated":243859456,"data_stored":243257051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":216725962},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":3.1789999999999998,"5min":3.4740000000000002,"15min":1.488},"min":{"1min":0.40400000000000003,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":24.856999999999999,"5min":63.039000000000001,"15min":63.039000000000001},"last":0.223},{"interface":"front","average":{"1min":3.6379999999999999,"5min":3.4380000000000002,"15min":1.486},"min":{"1min":0.33000000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":24.794,"5min":65.296000000000006,"15min":65.296000000000006},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":3.298,"5min":1.9630000000000001,"15min":1.004},"min":{"1min":0.38400000000000001,"5min":0.222,"15min":0.222},"max":{"1min":25.774999999999999,"5min":32.127000000000002,"15min":32.127000000000002},"last":2.472},{"interface":"front","average":{"1min":3.1219999999999999,"5min":2.0470000000000002,"15min":1.0509999999999999},"min":{"1min":0.32500000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":26.222999999999999,"5min":32.173999999999999,"15min":32.173999999999999},"last":0.82199999999999995}]},{"osd":2,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":2.7010000000000001,"5min":2.2519999999999998,"15min":1.1000000000000001},"min":{"1min":0.34899999999999998,"5min":0.249,"15min":0.249},"max":{"1min":25.805,"5min":51.127000000000002,"15min":51.127000000000002},"last":0.78600000000000003},{"interface":"front","average":{"1min":2.6139999999999999,"5min":2.5870000000000002,"15min":1.2070000000000001},"min":{"1min":0.30199999999999999,"5min":0.246,"15min":0.246},"max":{"1min":24.579999999999998,"5min":51.113999999999997,"15min":51.113999999999997},"last":0.28599999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.843,"15min":0.96999999999999997},"min":{"1min":0.33600000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.6399999999999997,"5min":58.341999999999999,"15min":58.341999999999999},"last":0.377},{"interface":"front","average":{"1min":1.351,"5min":1.7689999999999999,"15min":0.95899999999999996},"min":{"1min":0.35499999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":6.5880000000000001,"5min":57.765000000000001,"15min":57.765000000000001},"last":0.43099999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.56999999999999995,"15min":0.48199999999999998},"min":{"1min":0.28199999999999997,"5min":0.25,"15min":0.25},"max":{"1min":4.8659999999999997,"5min":4.8659999999999997,"15min":4.8659999999999997},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.56699999999999995,"15min":0.48099999999999998},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.577,"5min":4.577,"15min":4.577},"last":0.25900000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":0.67300000000000004,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":6.9020000000000001,"5min":6.9020000000000001,"15min":6.9020000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.2,"5min":0.70299999999999996,"15min":0.56499999999999995},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":6.3449999999999998,"5min":6.3449999999999998,"15min":6.3449999999999998},"last":0.57499999999999996}]},{"osd":6,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":0.67600000000000005,"15min":0.55600000000000005},"min":{"1min":0.33300000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.9630000000000001,"5min":4.9630000000000001,"15min":4.9630000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":1,"5min":0.64900000000000002,"15min":0.55700000000000005},"min":{"1min":0.35699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":5.5179999999999998,"5min":5.5179999999999998,"15min":5.5179999999999998},"last":0.70399999999999996}]}]},{"osd":6,"up_from":52,"seq":223338299484,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":431524,"kb_used_data":225436,"kb_used_omap":1,"kb_used_meta":206078,"kb_avail":93313628,"statfs":{"total":95995035648,"available":95553155072,"internally_reserved":0,"allocated":230846464,"data_stored":230216465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":211024326},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":6.6749999999999998,"5min":4.2910000000000004,"15min":1.7929999999999999},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":111.81699999999999,"5min":111.81699999999999,"15min":111.81699999999999},"last":1.6519999999999999},{"interface":"front","average":{"1min":6.8140000000000001,"5min":4.1550000000000002,"15min":1.738},"min":{"1min":0.34000000000000002,"5min":0.22,"15min":0.22},"max":{"1min":111.09699999999999,"5min":111.09699999999999,"15min":111.09699999999999},"last":1.0669999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":2.387,"5min":1.254,"15min":0.78100000000000003},"min":{"1min":0.26700000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":33.984999999999999,"5min":33.984999999999999,"15min":33.984999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":2.3220000000000001,"5min":1.1859999999999999,"15min":0.76400000000000001},"min":{"1min":0.374,"5min":0.249,"15min":0.249},"max":{"1min":32.134,"5min":32.134,"15min":32.134},"last":1.72}]},{"osd":2,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":1.9019999999999999,"15min":1.0169999999999999},"min":{"1min":0.36399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":31.878,"5min":32.509999999999998,"15min":32.509999999999998},"last":2.6970000000000001},{"interface":"front","average":{"1min":2.214,"5min":2.1309999999999998,"15min":1.0640000000000001},"min":{"1min":0.36699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":31.507000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"last":2.2749999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":2.0299999999999998,"15min":1.0469999999999999},"min":{"1min":0.376,"5min":0.22,"15min":0.22},"max":{"1min":1.4470000000000001,"5min":36.976999999999997,"15min":36.976999999999997},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.835,"15min":0.98799999999999999},"min":{"1min":0.34899999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72099999999999997,"5min":34.222999999999999,"15min":34.222999999999999},"last":1.6899999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.59999999999999998,"15min":0.47099999999999997},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.173,"5min":4.5609999999999999,"15min":4.5609999999999999},"last":1.127},{"interface":"front","average":{"1min":0.876,"5min":0.63400000000000001,"15min":0.51500000000000001},"min":{"1min":0.28599999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":7.7960000000000003,"5min":7.7960000000000003,"15min":7.7960000000000003},"last":1.405}]},{"osd":5,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.90000000000000002,"15min":0.59899999999999998},"min":{"1min":0.307,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.1369999999999996,"5min":27.021999999999998,"15min":27.021999999999998},"last":1.097},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.94899999999999995,"15min":0.623},"min":{"1min":0.34300000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.0629999999999997,"5min":23.611999999999998,"15min":23.611999999999998},"last":2.5800000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.61899999999999999,"15min":0.53900000000000003},"min":{"1min":0.29599999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.2130000000000001,"5min":4.2130000000000001,"15min":4.2130000000000001},"last":1.7569999999999999},{"interface":"front","average":{"1min":2.5550000000000002,"5min":0.94799999999999995,"15min":0.67600000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":25.207000000000001,"5min":25.207000000000001,"15min":25.207000000000001},"last":0.68500000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":511260,"kb_used_data":313492,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93233892,"statfs":{"total":95995035648,"available":95471505408,"internally_reserved":0,"allocated":321015808,"data_stored":320393312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":3.778,"15min":1.4870000000000001},"min":{"1min":0.28000000000000003,"5min":0.223,"15min":0.21199999999999999},"max":{"1min":2.8650000000000002,"5min":45.393000000000001,"15min":45.393000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":3.7269999999999999,"15min":1.4730000000000001},"min":{"1min":0.251,"5min":0.246,"15min":0.22800000000000001},"max":{"1min":2.8439999999999999,"5min":45.031999999999996,"15min":45.031999999999996},"last":0.78500000000000003}]},{"osd":2,"last update":"Mon Feb 5 07:24:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":2.2440000000000002,"15min":1.0449999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":6.0030000000000001,"5min":39.259999999999998,"15min":39.259999999999998},"last":0.878},{"interface":"front","average":{"1min":1.3049999999999999,"5min":2.093,"15min":1.381},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.532,"5min":40.972000000000001,"15min":40.972000000000001},"last":3.0870000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:25:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":2.3180000000000001,"15min":1.0860000000000001},"min":{"1min":0.316,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.734,"5min":43.823,"15min":43.823},"last":0.627},{"interface":"front","average":{"1min":0.71499999999999997,"5min":2.48,"15min":1.1499999999999999},"min":{"1min":0.29999999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.9409999999999998,"5min":39.314999999999998,"15min":39.314999999999998},"last":2.2610000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.9970000000000001,"15min":1.0549999999999999},"min":{"1min":0.375,"5min":0.312,"15min":0.312},"max":{"1min":1.78,"5min":61.334000000000003,"15min":61.334000000000003},"last":2.2210000000000001},{"interface":"front","average":{"1min":0.92300000000000004,"5min":2.149,"15min":1.1990000000000001},"min":{"1min":0.35899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.7160000000000002,"5min":61.786000000000001,"15min":61.786000000000001},"last":0.97899999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:24:55 2024","interfaces":[{"interface":"back","average":{"1min":4.4900000000000002,"5min":1.9119999999999999,"15min":1.0509999999999999},"min":{"1min":0.40500000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":66.078000000000003,"5min":66.078000000000003,"15min":66.078000000000003},"last":2.1760000000000002},{"interface":"front","average":{"1min":4.4480000000000004,"5min":2.0539999999999998,"15min":1.202},"min":{"1min":0.40799999999999997,"5min":0.307,"15min":0.307},"max":{"1min":66.194000000000003,"5min":66.194000000000003,"15min":66.194000000000003},"last":0.71999999999999997}]},{"osd":6,"last update":"Mon Feb 5 07:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.4330000000000001,"15min":0.93000000000000005},"min":{"1min":0.42499999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.5739999999999998,"5min":18.456,"15min":18.456},"last":0.754},{"interface":"front","average":{"1min":0.86499999999999999,"5min":1.395,"15min":0.91400000000000003},"min":{"1min":0.46300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.4340000000000002,"5min":5.9930000000000003,"15min":5.9930000000000003},"last":2.129}]},{"osd":7,"last update":"Mon Feb 5 07:24:49 2024","interfaces":[{"interface":"back","average":{"1min":4.8159999999999998,"5min":2.1440000000000001,"15min":1.0980000000000001},"min":{"1min":0.371,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":66.703999999999994,"5min":66.703999999999994,"15min":66.703999999999994},"last":0.66500000000000004},{"interface":"front","average":{"1min":4.931,"5min":2.0800000000000001,"15min":1.0880000000000001},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":61.594999999999999,"5min":61.594999999999999,"15min":61.594999999999999},"last":0.94199999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738489,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":591008,"kb_used_data":357144,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":93154144,"statfs":{"total":95995035648,"available":95389843456,"internally_reserved":0,"allocated":365715456,"data_stored":365088974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:25:24 2024","interfaces":[{"interface":"back","average":{"1min":2.3900000000000001,"5min":2.0470000000000002,"15min":0.89500000000000002},"min":{"1min":0.20300000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":37.564,"5min":37.564,"15min":37.564},"last":1.488},{"interface":"front","average":{"1min":2.202,"5min":2.2629999999999999,"15min":0.97599999999999998},"min":{"1min":0.221,"5min":0.186,"15min":0.186},"max":{"1min":35.415999999999997,"5min":35.415999999999997,"15min":35.415999999999997},"last":3.3090000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:24:50 2024","interfaces":[{"interface":"back","average":{"1min":5.3499999999999996,"5min":2.665,"15min":1.175},"min":{"1min":0.28899999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":65.903999999999996,"5min":65.903999999999996,"15min":65.903999999999996},"last":2.9289999999999998},{"interface":"front","average":{"1min":5.5010000000000003,"5min":2.5950000000000002,"15min":1.147},"min":{"1min":0.186,"5min":0.186,"15min":0.182},"max":{"1min":65.694000000000003,"5min":65.694000000000003,"15min":65.694000000000003},"last":3.1779999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:25:06 2024","interfaces":[{"interface":"back","average":{"1min":1.9530000000000001,"5min":3.1899999999999999,"15min":1.3500000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":25.795000000000002,"5min":65.155000000000001,"15min":65.155000000000001},"last":1.5409999999999999},{"interface":"front","average":{"1min":2.0600000000000001,"5min":3.423,"15min":1.419},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":28.658000000000001,"5min":65.114000000000004,"15min":65.114000000000004},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:25:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":2.3220000000000001,"15min":1.1439999999999999},"min":{"1min":0.32300000000000001,"5min":0.31,"15min":0.31},"max":{"1min":31.584,"5min":65.188000000000002,"15min":65.188000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.865,"5min":2.29,"15min":1.163},"min":{"1min":0.44400000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":25.625,"5min":64.873000000000005,"15min":64.873000000000005},"last":1.0389999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:25:06 2024","interfaces":[{"interface":"back","average":{"1min":1.9399999999999999,"5min":2.1659999999999999,"15min":1.0920000000000001},"min":{"1min":0.32200000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":25.747,"5min":65.555999999999997,"15min":65.555999999999997},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.786,"5min":2.3109999999999999,"15min":1.165},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":25.567,"5min":64.936999999999998,"15min":64.936999999999998},"last":1.1719999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:25:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":2.2610000000000001,"15min":1.155},"min":{"1min":0.45600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":25.681999999999999,"5min":65.215000000000003,"15min":65.215000000000003},"last":1.3620000000000001},{"interface":"front","average":{"1min":2.2530000000000001,"5min":2.4489999999999998,"15min":1.242},"min":{"1min":0.41899999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":31.524000000000001,"5min":65.230000000000004,"15min":65.230000000000004},"last":1.1419999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:24:40 2024","interfaces":[{"interface":"back","average":{"1min":5.3840000000000003,"5min":2.4049999999999998,"15min":1.206},"min":{"1min":0.44400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":65.263000000000005,"5min":65.263000000000005,"15min":65.263000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":5.4509999999999996,"5min":2.3610000000000002,"15min":1.216},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":64.906000000000006,"5min":64.906000000000006,"15min":64.906000000000006},"last":0.61299999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411439,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654724,"kb_used_data":376444,"kb_used_omap":1,"kb_used_meta":278270,"kb_avail":93090428,"statfs":{"total":95995035648,"available":95324598272,"internally_reserved":0,"allocated":385478656,"data_stored":384880290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":284948933},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:24:49 2024","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":2.4220000000000002,"15min":1.0549999999999999},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":8.9139999999999997,"5min":67.418999999999997,"15min":67.418999999999997},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.7430000000000001,"5min":2.7280000000000002,"15min":1.169},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.782,"5min":68.171000000000006,"15min":68.171000000000006},"last":1.603}]},{"osd":1,"last update":"Mon Feb 5 07:24:49 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.337,"15min":0.70999999999999996},"min":{"1min":0.19800000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":4.9420000000000002,"5min":12.609,"15min":12.609},"last":1.823},{"interface":"front","average":{"1min":1.276,"5min":1.49,"15min":0.78400000000000003},"min":{"1min":0.23100000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":5.4580000000000002,"5min":9.641,"15min":9.641},"last":1.5129999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":1.776,"15min":0.90400000000000003},"min":{"1min":0.28499999999999998,"5min":0.214,"15min":0.214},"max":{"1min":2.3439999999999999,"5min":14.756,"15min":14.756},"last":1.361},{"interface":"front","average":{"1min":0.47399999999999998,"5min":1.8300000000000001,"15min":0.92000000000000004},"min":{"1min":0.32400000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.75600000000000001,"5min":9.6669999999999998,"15min":9.6669999999999998},"last":1.4259999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:24:32 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.2050000000000001,"15min":0.75800000000000001},"min":{"1min":0.33500000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":4.2130000000000001,"5min":7.29,"15min":7.29},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.236,"15min":0.78500000000000003},"min":{"1min":0.32200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":5.3499999999999996,"5min":8.5199999999999996,"15min":8.5199999999999996},"last":2.1139999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:24:53 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.2969999999999999,"15min":0.83099999999999996},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":5.4100000000000001,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":2.2069999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.167,"15min":0.79600000000000004},"min":{"1min":0.42399999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":5.1710000000000003,"5min":7.7279999999999998,"15min":7.7279999999999998},"last":1.8680000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":1.3109999999999999,"15min":0.93700000000000006},"min":{"1min":0.42799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.3069999999999999,"5min":7.6829999999999998,"15min":7.6829999999999998},"last":1.4970000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.381,"15min":0.86399999999999999},"min":{"1min":0.41699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":2.0489999999999999,"5min":7.3150000000000004,"15min":7.3150000000000004},"last":1.2929999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:24:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.3260000000000001,"15min":0.90800000000000003},"min":{"1min":0.375,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.1130000000000004,"5min":7.2590000000000003,"15min":7.2590000000000003},"last":3.371},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.2569999999999999,"15min":0.84699999999999998},"min":{"1min":0.40899999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":5.9450000000000003,"5min":7.7619999999999996,"15min":7.7619999999999996},"last":2.2290000000000001}]}]},{"osd":3,"up_from":0,"seq":111669149802,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953573,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501896,"kb_used_data":358272,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93243256,"statfs":{"total":95995035648,"available":95481094144,"internally_reserved":0,"allocated":366870528,"data_stored":366267023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":1.6950000000000001,"15min":0.91500000000000004},"min":{"1min":0.38800000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.244,"5min":67.719999999999999,"15min":67.719999999999999},"last":1.244},{"interface":"front","average":{"1min":0.75700000000000001,"5min":1.9870000000000001,"15min":1.028},"min":{"1min":0.39000000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8999999999999999,"5min":66.882000000000005,"15min":66.882000000000005},"last":1.877}]},{"osd":1,"last update":"Mon Feb 5 07:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.86699999999999999,"15min":0.63600000000000001},"min":{"1min":0.39000000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.73799999999999999,"5min":5.5709999999999997,"15min":5.5709999999999997},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.4350000000000001,"5min":0.96999999999999997,"15min":0.68700000000000006},"min":{"1min":0.35999999999999999,"5min":0.254,"15min":0.254},"max":{"1min":17.065999999999999,"5min":17.065999999999999,"15min":17.065999999999999},"last":1.2090000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:25:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.7430000000000001,"15min":0.93600000000000005},"min":{"1min":0.42499999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":17.013000000000002,"5min":67.144999999999996,"15min":67.144999999999996},"last":1.2709999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":1.6359999999999999,"15min":0.90400000000000003},"min":{"1min":0.33800000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":3.7709999999999999,"5min":66.959999999999994,"15min":66.959999999999994},"last":1.9630000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.90900000000000003,"15min":0.67400000000000004},"min":{"1min":0.40400000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.8740000000000001,"5min":8.8019999999999996,"15min":8.8019999999999996},"last":1.835},{"interface":"front","average":{"1min":0.53300000000000003,"5min":1.0649999999999999,"15min":0.72699999999999998},"min":{"1min":0.374,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65800000000000003,"5min":10.106,"15min":10.106},"last":0.52100000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.53400000000000003,"15min":0.42399999999999999},"min":{"1min":0.29199999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.8420000000000001,"5min":2.8420000000000001,"15min":2.8420000000000001},"last":2.0419999999999998},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.58999999999999997,"15min":0.45100000000000001},"min":{"1min":0.307,"5min":0.254,"15min":0.254},"max":{"1min":3.4409999999999998,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon Feb 5 07:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.70999999999999996,"15min":0.51600000000000001},"min":{"1min":0.36199999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.29,"5min":6.8940000000000001,"15min":6.8940000000000001},"last":0.442},{"interface":"front","average":{"1min":0.754,"5min":0.67800000000000005,"15min":0.501},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.248},"max":{"1min":2.9260000000000002,"5min":2.9260000000000002,"15min":2.9260000000000002},"last":2.1909999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:24:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":0.69599999999999995,"15min":0.56000000000000005},"min":{"1min":0.33800000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.0640000000000001,"5min":6.0640000000000001,"15min":6.0640000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.60599999999999998,"15min":0.52400000000000002},"min":{"1min":0.41299999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.1659999999999999}]}]},{"osd":5,"up_from":41,"seq":176093659232,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":567812,"kb_used_data":406140,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93177340,"statfs":{"total":95995035648,"available":95413596160,"internally_reserved":0,"allocated":415887360,"data_stored":415288412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:24:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":3.3580000000000001,"15min":1.472},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":5.2039999999999997,"5min":69.855999999999995,"15min":69.855999999999995},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":3.0590000000000002,"15min":1.3740000000000001},"min":{"1min":0.24299999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.9079999999999999,"5min":71.513999999999996,"15min":71.513999999999996},"last":1.2909999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:24:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.2689999999999999,"15min":0.79700000000000004},"min":{"1min":0.36399999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.8680000000000003,"5min":11.839,"15min":11.839},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.008,"5min":0.88,"15min":0.64800000000000002},"min":{"1min":0.30499999999999999,"5min":0.253,"15min":0.253},"max":{"1min":6.8079999999999998,"5min":9.5950000000000006,"15min":9.5950000000000006},"last":1.361}]},{"osd":2,"last update":"Mon Feb 5 07:24:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":2.0350000000000001,"15min":1.03},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.8529999999999998,"5min":70.108000000000004,"15min":70.108000000000004},"last":1.083},{"interface":"front","average":{"1min":1.5780000000000001,"5min":2.1720000000000002,"15min":1.0900000000000001},"min":{"1min":0.40600000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.8390000000000004,"5min":72.817999999999998,"15min":72.817999999999998},"last":0.35499999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:24:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":2.2010000000000001,"15min":1.097},"min":{"1min":0.34899999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":6.8250000000000002,"5min":69.793000000000006,"15min":69.793000000000006},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.034,"5min":2.427,"15min":1.175},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.2320000000000002,"5min":69.820999999999998,"15min":69.820999999999998},"last":0.42499999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.47299999999999998,"15min":0.39200000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.4119999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.53800000000000003,"15min":0.43099999999999999},"min":{"1min":0.25,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.6360000000000001,"5min":3.6360000000000001,"15min":3.6360000000000001},"last":2.6619999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:25:26 2024","interfaces":[{"interface":"back","average":{"1min":5.6280000000000001,"5min":2.8340000000000001,"15min":1.216},"min":{"1min":0.30199999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":109.09399999999999,"5min":109.09399999999999,"15min":109.09399999999999},"last":1.0549999999999999},{"interface":"front","average":{"1min":5.6710000000000003,"5min":2.8210000000000002,"15min":1.246},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":108.965,"5min":108.965,"15min":108.965},"last":1.9910000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:24:43 2024","interfaces":[{"interface":"back","average":{"1min":6.742,"5min":3.6320000000000001,"15min":1.552},"min":{"1min":0.34999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":108.7,"5min":182.87799999999999,"15min":182.87799999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":6.7080000000000002,"5min":3.6070000000000002,"15min":1.5469999999999999},"min":{"1min":0.33700000000000002,"5min":0.32200000000000001,"15min":0.30199999999999999},"max":{"1min":108.89400000000001,"5min":183.02199999999999,"15min":183.02199999999999},"last":1.3280000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":400556032,"data_stored":400556032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":371982336,"data_stored":371982336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":457244672,"data_stored":457244672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":365690880,"data_stored":365690880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":461438976,"data_stored":461438976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":260898816,"data_stored":260898816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":256507904,"data_stored":256507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:25:41.491 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-05T07:25:41.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-02-05T07:25:41.502 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:41.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:41.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:41.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:41 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:41.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:41.639 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:41.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:41.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:41.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:41.777 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:41.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:41.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:41.879 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:41.919 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:41.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:42.020 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 11 16 29649 29633 168.357 147 0.00569819 0.00593403 2024-02-05T07:25:42.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:42.058 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:42.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:42.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:42.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:42.200 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:42.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:42.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:42.347 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:42.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:42.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:42.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:42 smithi007 ceph-mon[29098]: from='client.24886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:25:42.625 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:42 smithi007 ceph-mon[29098]: pgmap v324: 148 pgs: 148 active+clean; 1.6 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 200 MiB/s wr, 3.19k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:42.625 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:42 smithi007 ceph-mon[37158]: from='client.24886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:25:42.625 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:42 smithi007 ceph-mon[37158]: pgmap v324: 148 pgs: 148 active+clean; 1.6 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 200 MiB/s wr, 3.19k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:42.626 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:42.626 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:42 smithi090 ceph-mon[33213]: from='client.24886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:25:42.626 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:42 smithi090 ceph-mon[33213]: pgmap v324: 148 pgs: 148 active+clean; 1.6 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 200 MiB/s wr, 3.19k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:42.668 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:42.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:42.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:42.807 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:42.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:42.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:42.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:42.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:43.020 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 12 16 31898 31882 166.04 140.562 0.0053881 0.00601667 2024-02-05T07:25:43.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:43.085 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:43.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:43.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:43.188 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:43.211 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:43.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:43.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:43.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:43.353 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:43.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:43.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[29098]: 4.13 scrub starts 2024-02-05T07:25:43.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[29098]: 4.13 scrub ok 2024-02-05T07:25:43.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[29098]: 3.19 scrub starts 2024-02-05T07:25:43.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[29098]: 3.19 scrub ok 2024-02-05T07:25:43.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[29098]: 5.1b scrub starts 2024-02-05T07:25:43.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[29098]: 5.1b scrub ok 2024-02-05T07:25:43.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:25:43.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:25:43.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[37158]: 4.13 scrub starts 2024-02-05T07:25:43.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[37158]: 4.13 scrub ok 2024-02-05T07:25:43.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[37158]: 3.19 scrub starts 2024-02-05T07:25:43.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[37158]: 3.19 scrub ok 2024-02-05T07:25:43.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[37158]: 5.1b scrub starts 2024-02-05T07:25:43.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[37158]: 5.1b scrub ok 2024-02-05T07:25:43.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:25:43.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:25:43.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:43.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:43.478 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:43.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:43.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:43 smithi090 ceph-mon[33213]: 4.13 scrub starts 2024-02-05T07:25:43.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:43 smithi090 ceph-mon[33213]: 4.13 scrub ok 2024-02-05T07:25:43.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:43 smithi090 ceph-mon[33213]: 3.19 scrub starts 2024-02-05T07:25:43.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:43 smithi090 ceph-mon[33213]: 3.19 scrub ok 2024-02-05T07:25:43.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:43 smithi090 ceph-mon[33213]: 5.1b scrub starts 2024-02-05T07:25:43.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:43 smithi090 ceph-mon[33213]: 5.1b scrub ok 2024-02-05T07:25:43.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:43 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:25:43.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:43 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:25:43.580 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:43.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:43.619 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:43.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:43.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:43.757 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:43.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:43.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:43.859 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:43.894 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:43.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:43.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:43.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:25:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:25:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:25:43.996 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:44.020 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 13 16 33495 33479 160.945 99.8125 0.013227 0.00620794 2024-02-05T07:25:44.034 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:44.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:44.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:44.038 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:25:44.038 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:25:44.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:44.172 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:44.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:44.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:44.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:44.309 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:44.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:44.411 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:44.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[29098]: 4.11 scrub starts 2024-02-05T07:25:44.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[29098]: 4.11 scrub ok 2024-02-05T07:25:44.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[29098]: 5.6 scrub starts 2024-02-05T07:25:44.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[29098]: 5.6 scrub ok 2024-02-05T07:25:44.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[29098]: pgmap v325: 148 pgs: 148 active+clean; 1.6 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 200 MiB/s wr, 3.19k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:44.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[29098]: osdmap e221: 8 total, 8 up, 7 in 2024-02-05T07:25:44.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[29098]: 6.e deep-scrub starts 2024-02-05T07:25:44.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[29098]: 6.e deep-scrub ok 2024-02-05T07:25:44.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[29098]: 6.19 scrub starts 2024-02-05T07:25:44.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[29098]: 6.19 scrub ok 2024-02-05T07:25:44.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[29098]: osdmap e222: 8 total, 8 up, 7 in 2024-02-05T07:25:44.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[37158]: 4.11 scrub starts 2024-02-05T07:25:44.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[37158]: 4.11 scrub ok 2024-02-05T07:25:44.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[37158]: 5.6 scrub starts 2024-02-05T07:25:44.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[37158]: 5.6 scrub ok 2024-02-05T07:25:44.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[37158]: pgmap v325: 148 pgs: 148 active+clean; 1.6 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 200 MiB/s wr, 3.19k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:44.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[37158]: osdmap e221: 8 total, 8 up, 7 in 2024-02-05T07:25:44.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[37158]: 6.e deep-scrub starts 2024-02-05T07:25:44.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[37158]: 6.e deep-scrub ok 2024-02-05T07:25:44.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[37158]: 6.19 scrub starts 2024-02-05T07:25:44.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[37158]: 6.19 scrub ok 2024-02-05T07:25:44.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:44 smithi007 ceph-mon[37158]: osdmap e222: 8 total, 8 up, 7 in 2024-02-05T07:25:44.449 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:44.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:44.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:44.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:44 smithi090 ceph-mon[33213]: 4.11 scrub starts 2024-02-05T07:25:44.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:44 smithi090 ceph-mon[33213]: 4.11 scrub ok 2024-02-05T07:25:44.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:44 smithi090 ceph-mon[33213]: 5.6 scrub starts 2024-02-05T07:25:44.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:44 smithi090 ceph-mon[33213]: 5.6 scrub ok 2024-02-05T07:25:44.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:44 smithi090 ceph-mon[33213]: pgmap v325: 148 pgs: 148 active+clean; 1.6 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 200 MiB/s wr, 3.19k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:25:44.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:44 smithi090 ceph-mon[33213]: osdmap e221: 8 total, 8 up, 7 in 2024-02-05T07:25:44.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:44 smithi090 ceph-mon[33213]: 6.e deep-scrub starts 2024-02-05T07:25:44.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:44 smithi090 ceph-mon[33213]: 6.e deep-scrub ok 2024-02-05T07:25:44.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:44 smithi090 ceph-mon[33213]: 6.19 scrub starts 2024-02-05T07:25:44.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:44 smithi090 ceph-mon[33213]: 6.19 scrub ok 2024-02-05T07:25:44.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:44 smithi090 ceph-mon[33213]: osdmap e222: 8 total, 8 up, 7 in 2024-02-05T07:25:44.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:44.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:44.588 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:44.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:44.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:44.726 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:44.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:44.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:44.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:44.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:45.005 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:45.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:45.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:45.020 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 14 16 35585 35569 158.779 130.625 0.00864086 0.00629251 2024-02-05T07:25:45.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:45.124 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:45.145 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:45.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:45.151 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:45.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:45.285 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:45.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:45.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:45.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:45.426 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:45.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:45.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:45.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:45.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:45.667 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:45.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T07:25:45.703 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:45.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:45.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:45.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:45 smithi090 ceph-mon[33213]: 5.a scrub starts 2024-02-05T07:25:45.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:45 smithi090 ceph-mon[33213]: 5.a scrub ok 2024-02-05T07:25:45.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:45 smithi090 ceph-mon[33213]: 4.c deep-scrub starts 2024-02-05T07:25:45.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:45 smithi090 ceph-mon[33213]: 4.c deep-scrub ok 2024-02-05T07:25:45.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:45 smithi090 ceph-mon[33213]: 4.b scrub starts 2024-02-05T07:25:45.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:45 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:45.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:45 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:25:45.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:45 smithi090 ceph-mon[33213]: 4.b scrub ok 2024-02-05T07:25:45.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:45.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[29098]: 5.a scrub starts 2024-02-05T07:25:45.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[29098]: 5.a scrub ok 2024-02-05T07:25:45.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[29098]: 4.c deep-scrub starts 2024-02-05T07:25:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[29098]: 4.c deep-scrub ok 2024-02-05T07:25:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[29098]: 4.b scrub starts 2024-02-05T07:25:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:25:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[29098]: 4.b scrub ok 2024-02-05T07:25:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[37158]: 5.a scrub starts 2024-02-05T07:25:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[37158]: 5.a scrub ok 2024-02-05T07:25:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[37158]: 4.c deep-scrub starts 2024-02-05T07:25:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[37158]: 4.c deep-scrub ok 2024-02-05T07:25:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[37158]: 4.b scrub starts 2024-02-05T07:25:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:25:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:45 smithi007 ceph-mon[37158]: 4.b scrub ok 2024-02-05T07:25:45.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:45.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:45.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:45.997 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:45.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:45.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:46.020 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 15 16 37615 37599 156.651 126.875 0.00701818 0.00637776 2024-02-05T07:25:46.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:46.136 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:46.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:46.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:46.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:46.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:46.378 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:46.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:46.414 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:46.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:46.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:46.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:46.540 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:46.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:46.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:46.666 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:46.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:46.692 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:25:46.711 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:25:46.731 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:25:46.750 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:25:46.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:46.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:46 smithi090 ceph-mon[33213]: pgmap v328: 148 pgs: 148 active+clean; 1.6 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 133 MiB/s wr, 2.13k op/s 2024-02-05T07:25:46.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:46.791 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:46.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:46.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:46 smithi007 ceph-mon[29098]: pgmap v328: 148 pgs: 148 active+clean; 1.6 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 133 MiB/s wr, 2.13k op/s 2024-02-05T07:25:46.803 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:46 smithi007 ceph-mon[37158]: pgmap v328: 148 pgs: 148 active+clean; 1.6 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 133 MiB/s wr, 2.13k op/s 2024-02-05T07:25:46.845 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:25:46.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:46.918 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:25:46.930 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:46.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:46.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:46.961 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:25:47.004 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:25:47.009 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:25:47.010 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:25:47.010 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:25:47.010 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:25:47.010 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:25:47.010 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:25:47.010 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:25:47.010 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:25:47.020 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 16 15 38824 38809 151.586 75.625 0.00824662 0.00659222 2024-02-05T07:25:47.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:47.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:47.067 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:47.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:47.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:47.205 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:47.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:47.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:47.307 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:47.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:47.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:47.481 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:47.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:47.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:47.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:47.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:47.625 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:47.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:47.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:47.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:47.769 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:47.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:47.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:47 smithi090 ceph-mon[33213]: 5.d scrub starts 2024-02-05T07:25:47.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:47 smithi090 ceph-mon[33213]: 5.d scrub ok 2024-02-05T07:25:47.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:47 smithi090 ceph-mon[33213]: 3.9 deep-scrub starts 2024-02-05T07:25:47.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:47 smithi090 ceph-mon[33213]: 3.9 deep-scrub ok 2024-02-05T07:25:47.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:47 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3916407102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:47.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:47 smithi090 ceph-mon[33213]: 4.5 scrub starts 2024-02-05T07:25:47.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:47 smithi090 ceph-mon[33213]: 4.5 scrub ok 2024-02-05T07:25:47.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:47 smithi090 ceph-mon[33213]: 4.15 deep-scrub starts 2024-02-05T07:25:47.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:47 smithi090 ceph-mon[33213]: 4.15 deep-scrub ok 2024-02-05T07:25:47.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[29098]: 5.d scrub starts 2024-02-05T07:25:47.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[29098]: 5.d scrub ok 2024-02-05T07:25:47.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[29098]: 3.9 deep-scrub starts 2024-02-05T07:25:47.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[29098]: 3.9 deep-scrub ok 2024-02-05T07:25:47.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3916407102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:47.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[29098]: 4.5 scrub starts 2024-02-05T07:25:47.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[29098]: 4.5 scrub ok 2024-02-05T07:25:47.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[29098]: 4.15 deep-scrub starts 2024-02-05T07:25:47.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[29098]: 4.15 deep-scrub ok 2024-02-05T07:25:47.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[37158]: 5.d scrub starts 2024-02-05T07:25:47.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[37158]: 5.d scrub ok 2024-02-05T07:25:47.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[37158]: 3.9 deep-scrub starts 2024-02-05T07:25:47.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[37158]: 3.9 deep-scrub ok 2024-02-05T07:25:47.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3916407102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:47.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[37158]: 4.5 scrub starts 2024-02-05T07:25:47.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[37158]: 4.5 scrub ok 2024-02-05T07:25:47.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[37158]: 4.15 deep-scrub starts 2024-02-05T07:25:47.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:47 smithi007 ceph-mon[37158]: 4.15 deep-scrub ok 2024-02-05T07:25:47.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:47.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:47.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:48.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:48.021 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 17 16 40833 40817 150.051 125.5 0.00832337 0.00665929 2024-02-05T07:25:48.054 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:48.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:48.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:48.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:48.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:48.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:48.344 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:48.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:48.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:48.482 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:48.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:48.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:48.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:48.625 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:48.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:48.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:48 smithi007 ceph-mon[37158]: pgmap v329: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 185 MiB/s wr, 2.95k op/s 2024-02-05T07:25:48.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:48 smithi007 ceph-mon[37158]: 3.0 scrub starts 2024-02-05T07:25:48.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:48 smithi007 ceph-mon[37158]: 3.0 scrub ok 2024-02-05T07:25:48.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:48 smithi007 ceph-mon[37158]: 6.c scrub starts 2024-02-05T07:25:48.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:48 smithi007 ceph-mon[37158]: 6.c scrub ok 2024-02-05T07:25:48.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:48 smithi007 ceph-mon[37158]: 4.14 scrub starts 2024-02-05T07:25:48.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:48 smithi007 ceph-mon[37158]: 4.14 scrub ok 2024-02-05T07:25:48.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:48 smithi007 ceph-mon[29098]: pgmap v329: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 185 MiB/s wr, 2.95k op/s 2024-02-05T07:25:48.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:48 smithi007 ceph-mon[29098]: 3.0 scrub starts 2024-02-05T07:25:48.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:48 smithi007 ceph-mon[29098]: 3.0 scrub ok 2024-02-05T07:25:48.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:48 smithi007 ceph-mon[29098]: 6.c scrub starts 2024-02-05T07:25:48.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:48 smithi007 ceph-mon[29098]: 6.c scrub ok 2024-02-05T07:25:48.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:48 smithi007 ceph-mon[29098]: 4.14 scrub starts 2024-02-05T07:25:48.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:48 smithi007 ceph-mon[29098]: 4.14 scrub ok 2024-02-05T07:25:48.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:48.756 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:48.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:48.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:48.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:48 smithi090 ceph-mon[33213]: pgmap v329: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 185 MiB/s wr, 2.95k op/s 2024-02-05T07:25:48.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:48 smithi090 ceph-mon[33213]: 3.0 scrub starts 2024-02-05T07:25:48.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:48 smithi090 ceph-mon[33213]: 3.0 scrub ok 2024-02-05T07:25:48.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:48 smithi090 ceph-mon[33213]: 6.c scrub starts 2024-02-05T07:25:48.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:48 smithi090 ceph-mon[33213]: 6.c scrub ok 2024-02-05T07:25:48.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:48 smithi090 ceph-mon[33213]: 4.14 scrub starts 2024-02-05T07:25:48.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:48 smithi090 ceph-mon[33213]: 4.14 scrub ok 2024-02-05T07:25:48.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:48.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:48.898 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:48.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:48.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:49.022 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 18 16 42559 42543 147.697 107.875 0.0106281 0.00676534 2024-02-05T07:25:49.038 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:49.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:49.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:49.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:49.181 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:49.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:49.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:49.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:49.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:49.321 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:49.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:49.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:49.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:49.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:49.485 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:49.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:49.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:49.608 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:49.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:49.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:49.732 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:49.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:49.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:49.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:49 smithi007 ceph-mon[29098]: 6.14 deep-scrub starts 2024-02-05T07:25:49.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:49 smithi007 ceph-mon[29098]: 6.14 deep-scrub ok 2024-02-05T07:25:49.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:49 smithi007 ceph-mon[29098]: 3.8 scrub starts 2024-02-05T07:25:49.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:49 smithi007 ceph-mon[29098]: 3.8 scrub ok 2024-02-05T07:25:49.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:49 smithi007 ceph-mon[37158]: 6.14 deep-scrub starts 2024-02-05T07:25:49.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:49 smithi007 ceph-mon[37158]: 6.14 deep-scrub ok 2024-02-05T07:25:49.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:49 smithi007 ceph-mon[37158]: 3.8 scrub starts 2024-02-05T07:25:49.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:49 smithi007 ceph-mon[37158]: 3.8 scrub ok 2024-02-05T07:25:49.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:49.877 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:49.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:49.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:50.014 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:50.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:50.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:50.022 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 19 16 44069 44053 144.89 94.375 0.00392295 0.00689627 2024-02-05T07:25:50.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:49 smithi090 ceph-mon[33213]: 6.14 deep-scrub starts 2024-02-05T07:25:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:49 smithi090 ceph-mon[33213]: 6.14 deep-scrub ok 2024-02-05T07:25:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:49 smithi090 ceph-mon[33213]: 3.8 scrub starts 2024-02-05T07:25:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:49 smithi090 ceph-mon[33213]: 3.8 scrub ok 2024-02-05T07:25:50.116 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:50.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:50.154 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:50.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:50.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:50.290 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:50.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:50.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:50.392 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:50.430 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:50.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:50.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:50.539 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pg_num to 11 2024-02-05T07:25:50.566 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:50.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:50.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:50.706 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:50.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:50.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:50.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:25:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:25:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:50 smithi090 ceph-mon[33213]: pgmap v330: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 638 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T07:25:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:50 smithi090 ceph-mon[33213]: 3.4 scrub starts 2024-02-05T07:25:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:50 smithi090 ceph-mon[33213]: 3.4 scrub ok 2024-02-05T07:25:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:50 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2191054625' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T07:25:50.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:25:50.808 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:50.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:50 smithi007 ceph-mon[29098]: pgmap v330: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 638 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T07:25:50.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:50 smithi007 ceph-mon[29098]: 3.4 scrub starts 2024-02-05T07:25:50.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:50 smithi007 ceph-mon[29098]: 3.4 scrub ok 2024-02-05T07:25:50.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:50 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2191054625' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T07:25:50.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:50 smithi007 ceph-mon[37158]: pgmap v330: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 638 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T07:25:50.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:50 smithi007 ceph-mon[37158]: 3.4 scrub starts 2024-02-05T07:25:50.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:50 smithi007 ceph-mon[37158]: 3.4 scrub ok 2024-02-05T07:25:50.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:50 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2191054625' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T07:25:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:50.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:50.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:51.009 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:51.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:51.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:51.022 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:25:51.021438+0000 min lat: 0.00128428 max lat: 0.385573 avg lat: 0.00696451 2024-02-05T07:25:51.022 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:25:51.022 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 20 16 45929 45913 143.458 116.25 0.00577761 0.00696451 2024-02-05T07:25:51.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:51.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:51.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:51.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:51.297 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:51.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:51.399 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:51.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:51.436 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:51.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:51.537 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:51.573 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:51.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:51.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:51.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2191054625' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-05T07:25:51.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[29098]: osdmap e223: 8 total, 8 up, 7 in 2024-02-05T07:25:51.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[29098]: 3.1e scrub starts 2024-02-05T07:25:51.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[29098]: 3.1e scrub ok 2024-02-05T07:25:51.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[29098]: 4.12 scrub starts 2024-02-05T07:25:51.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[29098]: 4.12 scrub ok 2024-02-05T07:25:51.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T07:25:51.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T07:25:51.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:51.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2191054625' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-05T07:25:51.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[37158]: osdmap e223: 8 total, 8 up, 7 in 2024-02-05T07:25:51.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[37158]: 3.1e scrub starts 2024-02-05T07:25:51.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[37158]: 3.1e scrub ok 2024-02-05T07:25:51.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[37158]: 4.12 scrub starts 2024-02-05T07:25:51.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[37158]: 4.12 scrub ok 2024-02-05T07:25:51.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T07:25:51.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:51 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T07:25:51.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:51.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:51.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:51.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:51.856 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:51.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:51.958 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:51.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:51.984 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:51.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:52.022 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 21 16 47727 47711 141.977 112.375 0.00810362 0.00703798 2024-02-05T07:25:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:51 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:25:52.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:51 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2191054625' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-05T07:25:52.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:51 smithi090 ceph-mon[33213]: osdmap e223: 8 total, 8 up, 7 in 2024-02-05T07:25:52.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:51 smithi090 ceph-mon[33213]: 3.1e scrub starts 2024-02-05T07:25:52.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:51 smithi090 ceph-mon[33213]: 3.1e scrub ok 2024-02-05T07:25:52.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:51 smithi090 ceph-mon[33213]: 4.12 scrub starts 2024-02-05T07:25:52.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:51 smithi090 ceph-mon[33213]: 4.12 scrub ok 2024-02-05T07:25:52.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:51 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T07:25:52.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:51 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T07:25:52.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:52.126 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:52.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:52.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:52.228 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:52.265 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:52.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:52.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:52.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:52.403 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:52.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:52.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:52.540 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:52.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:52.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:52.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:52.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:52.700 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:52.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:52.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:52.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[29098]: pgmap v332: 148 pgs: 148 active+clean; 2.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 150 MiB/s wr, 2.40k op/s 2024-02-05T07:25:52.807 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[29098]: 5.1 scrub starts 2024-02-05T07:25:52.807 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[29098]: 5.1 scrub ok 2024-02-05T07:25:52.807 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[29098]: 6.1d scrub starts 2024-02-05T07:25:52.807 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[29098]: 6.1d scrub ok 2024-02-05T07:25:52.807 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-05T07:25:52.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[29098]: osdmap e224: 8 total, 8 up, 7 in 2024-02-05T07:25:52.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:25:52.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:25:52.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[29098]: 3.11 scrub starts 2024-02-05T07:25:52.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[29098]: 3.11 scrub ok 2024-02-05T07:25:52.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[37158]: pgmap v332: 148 pgs: 148 active+clean; 2.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 150 MiB/s wr, 2.40k op/s 2024-02-05T07:25:52.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[37158]: 5.1 scrub starts 2024-02-05T07:25:52.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[37158]: 5.1 scrub ok 2024-02-05T07:25:52.810 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[37158]: 6.1d scrub starts 2024-02-05T07:25:52.810 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[37158]: 6.1d scrub ok 2024-02-05T07:25:52.810 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-05T07:25:52.810 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[37158]: osdmap e224: 8 total, 8 up, 7 in 2024-02-05T07:25:52.810 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:25:52.810 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:25:52.811 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[37158]: 3.11 scrub starts 2024-02-05T07:25:52.811 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:52 smithi007 ceph-mon[37158]: 3.11 scrub ok 2024-02-05T07:25:52.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:52.843 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:52.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:52.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:52.979 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:52.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:53.022 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 22 16 49501 49485 140.563 110.875 0.00656889 0.00710804 2024-02-05T07:25:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:52 smithi090 ceph-mon[33213]: pgmap v332: 148 pgs: 148 active+clean; 2.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 150 MiB/s wr, 2.40k op/s 2024-02-05T07:25:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:52 smithi090 ceph-mon[33213]: 5.1 scrub starts 2024-02-05T07:25:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:52 smithi090 ceph-mon[33213]: 5.1 scrub ok 2024-02-05T07:25:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:52 smithi090 ceph-mon[33213]: 6.1d scrub starts 2024-02-05T07:25:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:52 smithi090 ceph-mon[33213]: 6.1d scrub ok 2024-02-05T07:25:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:52 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-05T07:25:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:52 smithi090 ceph-mon[33213]: osdmap e224: 8 total, 8 up, 7 in 2024-02-05T07:25:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:52 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:25:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:52 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:25:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:52 smithi090 ceph-mon[33213]: 3.11 scrub starts 2024-02-05T07:25:53.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:52 smithi090 ceph-mon[33213]: 3.11 scrub ok 2024-02-05T07:25:53.081 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:53.140 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:53.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:53.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:53.242 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:53.298 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:53.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:53.342 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:53.400 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:53.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:53.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:53.581 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:53.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:53.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:53.584 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:25:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:25:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:25:53.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:53.730 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:53.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:53.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:53.866 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:53.867 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:53 smithi007 ceph-mon[29098]: osdmap e225: 8 total, 8 up, 7 in 2024-02-05T07:25:53.867 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:53 smithi007 ceph-mon[29098]: 1.0 scrub starts 2024-02-05T07:25:53.867 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:53 smithi007 ceph-mon[29098]: 1.0 scrub ok 2024-02-05T07:25:53.867 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:53 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T07:25:53.867 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:53 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T07:25:53.868 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:53 smithi007 ceph-mon[37158]: osdmap e225: 8 total, 8 up, 7 in 2024-02-05T07:25:53.868 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:53 smithi007 ceph-mon[37158]: 1.0 scrub starts 2024-02-05T07:25:53.868 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:53 smithi007 ceph-mon[37158]: 1.0 scrub ok 2024-02-05T07:25:53.868 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:53 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T07:25:53.868 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:53 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T07:25:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:53.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:53.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:54.006 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:54.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:54.022 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 23 15 50953 50938 138.4 90.8125 0.00600152 0.00721984 2024-02-05T07:25:54.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:53 smithi090 ceph-mon[33213]: osdmap e225: 8 total, 8 up, 7 in 2024-02-05T07:25:54.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:53 smithi090 ceph-mon[33213]: 1.0 scrub starts 2024-02-05T07:25:54.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:53 smithi090 ceph-mon[33213]: 1.0 scrub ok 2024-02-05T07:25:54.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:53 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T07:25:54.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:53 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T07:25:54.093 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:25:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:25:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:51030 8.f 8:f4ad9fec:::benchmark_data_smithi007_105755_object51029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-02-05T07:25:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:25:54.077174+0000", 2024-02-05T07:25:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.012225421, 2024-02-05T07:25:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012261114, 2024-02-05T07:25:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:25:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:25:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:25:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:25:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:25:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 51030 2024-02-05T07:25:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.094 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:25:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:25:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.077174+0000", 2024-02-05T07:25:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:25:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:25:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.077174+0000", 2024-02-05T07:25:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:25:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:25:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.077175+0000", 2024-02-05T07:25:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3430000000000001e-06 2024-02-05T07:25:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:25:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.077259+0000", 2024-02-05T07:25:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3335000000000001e-05 2024-02-05T07:25:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:25:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.077271+0000", 2024-02-05T07:25:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2298e-05 2024-02-05T07:25:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:25:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.077280+0000", 2024-02-05T07:25:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3700000000000001e-06 2024-02-05T07:25:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:25:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.079023+0000", 2024-02-05T07:25:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017426989999999999 2024-02-05T07:25:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:25:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.079147+0000", 2024-02-05T07:25:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000124158 2024-02-05T07:25:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:25:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.079286+0000", 2024-02-05T07:25:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000138436 2024-02-05T07:25:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:25:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.082105+0000", 2024-02-05T07:25:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0028192099999999999 2024-02-05T07:25:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:25:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.082156+0000", 2024-02-05T07:25:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1615e-05 2024-02-05T07:25:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.24977.0:51031 8.7 e226/209 8:eb99edf1:::benchmark_data_smithi007_105755_object51030:head v 226'3179, mlcod=226'3176)", 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:25:54.079856+0000", 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0095436859999999991, 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0096761839999999991, 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.079856+0000", 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:25:54.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:25:54.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.079856+0000", 2024-02-05T07:25:54.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:25:54.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:25:54.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.079857+0000", 2024-02-05T07:25:54.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9899999999999999e-07 2024-02-05T07:25:54.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:25:54.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.079880+0000", 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2979000000000001e-05 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.079889+0000", 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7389999999999994e-06 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.079896+0000", 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8129999999999998e-06 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:25:54.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.082383+0000", 2024-02-05T07:25:54.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024872219999999999 2024-02-05T07:25:54.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.102 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:25:54.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.082401+0000", 2024-02-05T07:25:54.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7612999999999999e-05 2024-02-05T07:25:54.102 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.102 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:54.102 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.102 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:51039 8.f 8:f0be723f:::benchmark_data_smithi007_105755_object51038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-02-05T07:25:54.102 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:25:54.082438+0000", 2024-02-05T07:25:54.102 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0069618639999999999, 2024-02-05T07:25:54.103 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0071692819999999999, 2024-02-05T07:25:54.103 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:25:54.103 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:25:54.103 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:25:54.103 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:25:54.103 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:25:54.103 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 51039 2024-02-05T07:25:54.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.103 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:25:54.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.103 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:25:54.103 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.082438+0000", 2024-02-05T07:25:54.103 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:25:54.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.082438+0000", 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.082440+0000", 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9470000000000002e-06 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.082460+0000", 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0755e-05 2024-02-05T07:25:54.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:25:54.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.082472+0000", 2024-02-05T07:25:54.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1396999999999999e-05 2024-02-05T07:25:54.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:25:54.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.082479+0000", 2024-02-05T07:25:54.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5229999999999998e-06 2024-02-05T07:25:54.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:25:54.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.082775+0000", 2024-02-05T07:25:54.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029533400000000002 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.085247+0000", 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024724529999999999 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.085390+0000", 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000142714 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:51041 8.f 8:f84844fd:::benchmark_data_smithi007_105755_object51040:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:25:54.085026+0000", 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0043739180000000001, 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0047301870000000003, 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 51041 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.085026+0000", 2024-02-05T07:25:54.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.085026+0000", 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.085027+0000", 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8169999999999999e-06 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.085049+0000", 2024-02-05T07:25:54.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1365e-05 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.085060+0000", 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1515000000000001e-05 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.085068+0000", 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6680000000000001e-06 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087617+0000", 2024-02-05T07:25:54.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002549595 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087734+0000", 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000116641 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087868+0000", 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013395900000000001 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.24977.0:51030 8.f e226/209)", 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:25:54.087088+0000", 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.002311787, 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0027456849999999999, 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087088+0000", 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087088+0000", 2024-02-05T07:25:54.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:25:54.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:25:54.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087088+0000", 2024-02-05T07:25:54.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8299999999999996e-07 2024-02-05T07:25:54.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:25:54.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087093+0000", 2024-02-05T07:25:54.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2490000000000003e-06 2024-02-05T07:25:54.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:25:54.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087106+0000", 2024-02-05T07:25:54.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2855e-05 2024-02-05T07:25:54.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:25:54.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087113+0000", 2024-02-05T07:25:54.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.289e-06 2024-02-05T07:25:54.113 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.113 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:54.113 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.113 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.24977.0:51047 8.4 e226/209 8:2e22a7e9:::benchmark_data_smithi007_105755_object51046:head v 226'3183, mlcod=226'3181)", 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:25:54.087574+0000", 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0018256450000000001, 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023117200000000002, 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087574+0000", 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087574+0000", 2024-02-05T07:25:54.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087575+0000", 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0400000000000005e-07 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087608+0000", 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3510000000000003e-05 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087619+0000", 2024-02-05T07:25:54.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1313e-05 2024-02-05T07:25:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:25:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087627+0000", 2024-02-05T07:25:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6909999999999997e-06 2024-02-05T07:25:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.116 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:25:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087715+0000", 2024-02-05T07:25:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7661e-05 2024-02-05T07:25:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:25:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.087736+0000", 2024-02-05T07:25:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0752000000000001e-05 2024-02-05T07:25:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.24977.0:51039 8.f e226/209)", 2024-02-05T07:25:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:25:54.088323+0000", 2024-02-05T07:25:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0010760450000000001, 2024-02-05T07:25:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0016270989999999999, 2024-02-05T07:25:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:25:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-02-05T07:25:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:25:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:25:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.088323+0000", 2024-02-05T07:25:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:25:54.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:25:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.088323+0000", 2024-02-05T07:25:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:25:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:25:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.088324+0000", 2024-02-05T07:25:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5799999999999999e-07 2024-02-05T07:25:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:25:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.088329+0000", 2024-02-05T07:25:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2859999999999999e-06 2024-02-05T07:25:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:25:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.088338+0000", 2024-02-05T07:25:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3129999999999999e-06 2024-02-05T07:25:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:25:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:25:54.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:25:54.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:25:54.088345+0000", 2024-02-05T07:25:54.121 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0639999999999996e-06 2024-02-05T07:25:54.121 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.121 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:25:54.121 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.121 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:25:54.121 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:25:54.121 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 7 2024-02-05T07:25:54.121 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:25:54.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:54.161 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:54.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:25:54.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:54.307 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:54.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:54.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:54.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:54.459 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:54.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:54.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:54.593 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:54.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:54.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:54.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[29098]: 6.0 scrub starts 2024-02-05T07:25:54.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[29098]: 6.0 scrub ok 2024-02-05T07:25:54.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[29098]: pgmap v335: 158 pgs: 10 unknown, 148 active+clean; 2.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 850 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-02-05T07:25:54.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[29098]: 3.3 scrub starts 2024-02-05T07:25:54.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[29098]: 3.3 scrub ok 2024-02-05T07:25:54.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-05T07:25:54.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[29098]: osdmap e226: 8 total, 8 up, 7 in 2024-02-05T07:25:54.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2779531996' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:25:54.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:25:54.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:25:54.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:25:54.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[29098]: osdmap e227: 8 total, 8 up, 7 in 2024-02-05T07:25:54.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[37158]: 6.0 scrub starts 2024-02-05T07:25:54.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[37158]: 6.0 scrub ok 2024-02-05T07:25:54.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[37158]: pgmap v335: 158 pgs: 10 unknown, 148 active+clean; 2.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 850 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-02-05T07:25:54.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[37158]: 3.3 scrub starts 2024-02-05T07:25:54.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[37158]: 3.3 scrub ok 2024-02-05T07:25:54.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-05T07:25:54.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[37158]: osdmap e226: 8 total, 8 up, 7 in 2024-02-05T07:25:54.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2779531996' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:25:54.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:25:54.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:25:54.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:25:54.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:54 smithi007 ceph-mon[37158]: osdmap e227: 8 total, 8 up, 7 in 2024-02-05T07:25:54.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:54.731 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:54.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:54.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:54.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:54.872 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:54.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:54.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:55.011 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:55.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:55.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:55.022 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 24 16 52518 52502 136.706 97.75 0.00680541 0.00730952 2024-02-05T07:25:55.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:54 smithi090 ceph-mon[33213]: 6.0 scrub starts 2024-02-05T07:25:55.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:54 smithi090 ceph-mon[33213]: 6.0 scrub ok 2024-02-05T07:25:55.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:54 smithi090 ceph-mon[33213]: pgmap v335: 158 pgs: 10 unknown, 148 active+clean; 2.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 850 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-02-05T07:25:55.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:54 smithi090 ceph-mon[33213]: 3.3 scrub starts 2024-02-05T07:25:55.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:54 smithi090 ceph-mon[33213]: 3.3 scrub ok 2024-02-05T07:25:55.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:54 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-05T07:25:55.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:54 smithi090 ceph-mon[33213]: osdmap e226: 8 total, 8 up, 7 in 2024-02-05T07:25:55.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:54 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2779531996' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:25:55.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:54 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:25:55.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:54 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:25:55.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:54 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:25:55.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:54 smithi090 ceph-mon[33213]: osdmap e227: 8 total, 8 up, 7 in 2024-02-05T07:25:55.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:55.150 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:55.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:55.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:55.289 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:55.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:55.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:55.429 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:55.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:55.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:55.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:55.566 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:55.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:55.667 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:55.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:55.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:55.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:55.840 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:55.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:55.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:55.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:56.007 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:56.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:56.023 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 25 16 54166 54150 135.357 103 0.00468985 0.00738277 2024-02-05T07:25:56.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:25:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:56.143 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:56.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:25:56.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:56.282 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:56.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:56.377 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T07:25:56.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:56.462 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:56.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:56.469 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:56 smithi007 ceph-mon[29098]: pgmap v338: 158 pgs: 10 unknown, 148 active+clean; 2.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail 2024-02-05T07:25:56.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:56 smithi007 ceph-mon[29098]: osdmap e228: 8 total, 8 up, 7 in 2024-02-05T07:25:56.471 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:56 smithi007 ceph-mon[37158]: pgmap v338: 158 pgs: 10 unknown, 148 active+clean; 2.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail 2024-02-05T07:25:56.471 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:56 smithi007 ceph-mon[37158]: osdmap e228: 8 total, 8 up, 7 in 2024-02-05T07:25:56.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:56 smithi090 ceph-mon[33213]: pgmap v338: 158 pgs: 10 unknown, 148 active+clean; 2.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail 2024-02-05T07:25:56.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:56 smithi090 ceph-mon[33213]: osdmap e228: 8 total, 8 up, 7 in 2024-02-05T07:25:56.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:56.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:56.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:56.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:56.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:56.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:56.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:56.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:25:56.868 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:56.882 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:56.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:25:56.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:57.020 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:57.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:57.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:57.022 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 26 16 55777 55761 134.023 100.688 0.0101579 0.00745648 2024-02-05T07:25:57.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:57.163 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:57.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:57.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:57.305 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:57.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:57.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:57 smithi007 ceph-mon[29098]: osdmap e229: 8 total, 8 up, 7 in 2024-02-05T07:25:57.396 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:57 smithi007 ceph-mon[37158]: osdmap e229: 8 total, 8 up, 7 in 2024-02-05T07:25:57.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:57.442 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:57.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:57.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:57.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:57 smithi090 ceph-mon[33213]: osdmap e229: 8 total, 8 up, 7 in 2024-02-05T07:25:57.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:57.597 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:57.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:57.630 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:25:57.672 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:25:57.676 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T07:25:57.677 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:25:57.677 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T07:25:57.677 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-05T07:25:57.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.6 2024-02-05T07:25:57.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:57.716 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:25:57.759 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:25:57.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:57.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:57.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:57.802 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:25:57.838 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:25:57.872 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:25:57.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:25:57.910 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:25:57.912 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:25:57.912 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:25:57.912 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:25:57.912 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:25:57.912 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:25:57.912 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:25:57.912 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:25:57.912 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:25:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:57.932 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:57.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:25:57.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:25:58.023 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 27 16 57200 57184 132.352 88.9375 0.00961171 0.00755024 2024-02-05T07:25:58.024 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:25:57 smithi090 systemd[1]: Stopping Ceph osd.6 for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:25:58.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:58.085 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:58.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:58.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:58.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:58.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:58.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:58.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:58.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:58 smithi007 ceph-mon[37158]: pgmap v341: 158 pgs: 1 active+remapped, 4 remapped+peering, 5 active+undersized+remapped, 148 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 132 MiB/s wr, 2.10k op/s; 105 KiB/s, 0 objects/s recovering 2024-02-05T07:25:58.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:58 smithi007 ceph-mon[37158]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-02-05T07:25:58.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:58 smithi007 ceph-mon[37158]: osdmap e230: 8 total, 8 up, 7 in 2024-02-05T07:25:58.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:58 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:25:58.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:58.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:58.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:58.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:58.494 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:58.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:58.525 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:25:58 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[49515]: 2024-02-05T07:25:58.188+0000 7f9b05d35700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:25:58.525 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:25:58 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[49515]: 2024-02-05T07:25:58.188+0000 7f9b05d35700 -1 osd.6 230 *** Got signal Terminated *** 2024-02-05T07:25:58.525 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:25:58 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[49515]: 2024-02-05T07:25:58.188+0000 7f9b05d35700 -1 osd.6 230 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T07:25:58.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:58 smithi090 ceph-mon[33213]: pgmap v341: 158 pgs: 1 active+remapped, 4 remapped+peering, 5 active+undersized+remapped, 148 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 132 MiB/s wr, 2.10k op/s; 105 KiB/s, 0 objects/s recovering 2024-02-05T07:25:58.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:58 smithi090 ceph-mon[33213]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-02-05T07:25:58.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:58 smithi090 ceph-mon[33213]: osdmap e230: 8 total, 8 up, 7 in 2024-02-05T07:25:58.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:58 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:25:58.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:58 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2447824449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:58.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:58 smithi090 ceph-mon[33213]: osd.6 marked itself down and dead 2024-02-05T07:25:58.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:58.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:58.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:58 smithi007 ceph-mon[29098]: pgmap v341: 158 pgs: 1 active+remapped, 4 remapped+peering, 5 active+undersized+remapped, 148 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 132 MiB/s wr, 2.10k op/s; 105 KiB/s, 0 objects/s recovering 2024-02-05T07:25:58.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:58 smithi007 ceph-mon[29098]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-02-05T07:25:58.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:58 smithi007 ceph-mon[29098]: osdmap e230: 8 total, 8 up, 7 in 2024-02-05T07:25:58.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:58 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:25:58.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:58 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2447824449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:58.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:58 smithi007 ceph-mon[29098]: osd.6 marked itself down and dead 2024-02-05T07:25:58.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:58 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2447824449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:25:58.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:58 smithi007 ceph-mon[37158]: osd.6 marked itself down and dead 2024-02-05T07:25:58.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:58.746 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:58.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:58.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:25:58.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:58.874 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:58.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:25:58.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:25:58.998 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:58.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:58.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:25:59.023 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 28 16 57541 57525 128.386 21.3125 0.00325866 0.00758085 2024-02-05T07:25:59.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:59.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:59.124 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:59.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:59.146 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:25:58 smithi090 podman[88344]: 2024-02-05 07:25:58.856281551 +0000 UTC m=+0.816765543 container died df3176c4cf90198e2aaad985c40746ddbecdab6a0191448f312a0ad657c9ba8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-02-05T07:25:59.225 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:59.267 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:59.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:59.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:59.333 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:25:59.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:59.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:59.417 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:59.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:25:59.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:59 smithi007 ceph-mon[29098]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-05T07:25:59.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:59 smithi007 ceph-mon[29098]: osdmap e231: 8 total, 7 up, 7 in 2024-02-05T07:25:59.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:59 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:25:59.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:59 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:25:59.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:59 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:59.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:25:59 smithi007 ceph-mon[29098]: osdmap e232: 8 total, 7 up, 7 in 2024-02-05T07:25:59.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:59 smithi007 ceph-mon[37158]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-05T07:25:59.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:59 smithi007 ceph-mon[37158]: osdmap e231: 8 total, 7 up, 7 in 2024-02-05T07:25:59.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:59 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:25:59.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:59 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:25:59.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:59 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:59.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:25:59 smithi007 ceph-mon[37158]: osdmap e232: 8 total, 7 up, 7 in 2024-02-05T07:25:59.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:59.525 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:25:59 smithi090 podman[88344]: 2024-02-05 07:25:59.336135261 +0000 UTC m=+1.296619246 container cleanup df3176c4cf90198e2aaad985c40746ddbecdab6a0191448f312a0ad657c9ba8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-02-05T07:25:59.525 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:25:59 smithi090 bash[88344]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6 2024-02-05T07:25:59.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:59 smithi090 ceph-mon[33213]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-05T07:25:59.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:59 smithi090 ceph-mon[33213]: osdmap e231: 8 total, 7 up, 7 in 2024-02-05T07:25:59.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:59 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:25:59.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:59 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:25:59.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:59 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:25:59.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:25:59 smithi090 ceph-mon[33213]: osdmap e232: 8 total, 7 up, 7 in 2024-02-05T07:25:59.559 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:59.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:59.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:59.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:25:59.701 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:59.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:59.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:25:59.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:25:59.840 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:25:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:59.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:25:59.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:25:59.984 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:25:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:25:59.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:00.023 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 29 16 58799 58783 126.67 78.625 0.00598891 0.00788983 2024-02-05T07:26:00.025 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:25:59 smithi090 podman[88424]: 2024-02-05 07:25:59.621824603 +0000 UTC m=+0.761374818 container remove df3176c4cf90198e2aaad985c40746ddbecdab6a0191448f312a0ad657c9ba8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-05T07:26:00.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:00.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:00.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:00.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:00.228 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:00.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:00.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:00.399 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:00.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:00.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:00.411 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:26:00 smithi090 podman[88571]: 2024-02-05 07:26:00.007434407 +0000 UTC m=+0.027647365 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T07:26:00.411 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:26:00 smithi090 podman[88571]: 2024-02-05 07:26:00.13158133 +0000 UTC m=+0.151794282 container create f1be03630815d6383ef1383200353d470a0ab41d5b5946a8d719bc2b7bb1377e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-05T07:26:00.411 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:26:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:26:00.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:00.546 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:00.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:00.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:00.685 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:00.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:00.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:00.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:26:00.776 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:26:00 smithi090 podman[88571]: 2024-02-05 07:26:00.410037242 +0000 UTC m=+0.430250199 container init f1be03630815d6383ef1383200353d470a0ab41d5b5946a8d719bc2b7bb1377e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-05T07:26:00.776 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:26:00 smithi090 podman[88571]: 2024-02-05 07:26:00.41527928 +0000 UTC m=+0.435492228 container start f1be03630815d6383ef1383200353d470a0ab41d5b5946a8d719bc2b7bb1377e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T07:26:00.776 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:26:00 smithi090 podman[88571]: 2024-02-05 07:26:00.51154135 +0000 UTC m=+0.531754301 container attach f1be03630815d6383ef1383200353d470a0ab41d5b5946a8d719bc2b7bb1377e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T07:26:00.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:00 smithi090 ceph-mon[33213]: pgmap v344: 158 pgs: 18 stale+active+clean, 1 active+remapped, 2 remapped+peering, 5 active+undersized+remapped, 2 stale+remapped+peering, 130 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 132 MiB/s wr, 2.10k op/s; 105 KiB/s, 0 objects/s recovering 2024-02-05T07:26:00.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:00 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:26:00.777 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:00 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:26:00.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:00.810 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:00 smithi007 ceph-mon[29098]: pgmap v344: 158 pgs: 18 stale+active+clean, 1 active+remapped, 2 remapped+peering, 5 active+undersized+remapped, 2 stale+remapped+peering, 130 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 132 MiB/s wr, 2.10k op/s; 105 KiB/s, 0 objects/s recovering 2024-02-05T07:26:00.810 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:00 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:26:00.811 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:00 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:26:00.811 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:00 smithi007 ceph-mon[37158]: pgmap v344: 158 pgs: 18 stale+active+clean, 1 active+remapped, 2 remapped+peering, 5 active+undersized+remapped, 2 stale+remapped+peering, 130 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 132 MiB/s wr, 2.10k op/s; 105 KiB/s, 0 objects/s recovering 2024-02-05T07:26:00.811 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:00 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:26:00.811 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:00 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:26:00.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:00.856 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:00.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:00.888 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:26:00.888 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:26:00.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:00.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:00.993 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:00.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:01.023 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 30 16 60316 60300 125.608 94.8125 0.0328362 0.00795167 2024-02-05T07:26:01.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:01.134 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:01.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:01.200 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:26:00 smithi090 podman[88571]: 2024-02-05 07:26:00.871227777 +0000 UTC m=+0.891440726 container died f1be03630815d6383ef1383200353d470a0ab41d5b5946a8d719bc2b7bb1377e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-02-05T07:26:01.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:01.265 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:01.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:01.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:01.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:01.405 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:01.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:01.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:01.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:01.531 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:01.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:01.632 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:01.667 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:01.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:01.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:01.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:01.804 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:01.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:01.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:01.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:01.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:01.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:01.944 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:01.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:01.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:02.023 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 31 16 61711 61695 124.368 87.1875 0.0063112 0.00803569 2024-02-05T07:26:02.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:02.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:02.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.6.service' 2024-02-05T07:26:02.083 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:02.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:02.130 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:26:02.130 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-05T07:26:02.130 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.6 2024-02-05T07:26:02.130 INFO:tasks.thrashosds.thrasher:remote for osd 6 is ubuntu@smithi090.front.sepia.ceph.com 2024-02-05T07:26:02.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -n osd.6 --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2024-02-05T07:26:02.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:02.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:02.223 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:02.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:02.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:02.349 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:02.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:02.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:02.391 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/osd.6/config 2024-02-05T07:26:02.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:02.484 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:02.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:02.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:02.620 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:02.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:02.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:02.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:02.763 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:02.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:02.827 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:02 smithi090 ceph-mon[33213]: pgmap v346: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s; 18602/177831 objects degraded (10.460%); 13 KiB/s, 0 objects/s recovering 2024-02-05T07:26:02.827 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:02 smithi090 ceph-mon[33213]: Health check failed: Degraded data redundancy: 18602/177831 objects degraded (10.460%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:02.827 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:02 smithi090 ceph-mon[33213]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-02-05T07:26:02.865 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:02.891 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:02.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:02.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:02 smithi007 ceph-mon[29098]: pgmap v346: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s; 18602/177831 objects degraded (10.460%); 13 KiB/s, 0 objects/s recovering 2024-02-05T07:26:02.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:02 smithi007 ceph-mon[29098]: Health check failed: Degraded data redundancy: 18602/177831 objects degraded (10.460%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:02.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:02 smithi007 ceph-mon[29098]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-02-05T07:26:02.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:02 smithi007 ceph-mon[37158]: pgmap v346: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s; 18602/177831 objects degraded (10.460%); 13 KiB/s, 0 objects/s recovering 2024-02-05T07:26:02.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:02 smithi007 ceph-mon[37158]: Health check failed: Degraded data redundancy: 18602/177831 objects degraded (10.460%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:02.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:02 smithi007 ceph-mon[37158]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-02-05T07:26:02.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:03.023 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 32 16 63162 63146 123.315 90.6875 0.0167417 0.00810455 2024-02-05T07:26:03.031 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:03.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:03.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:03.167 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:03.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:03.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:03.269 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:03.305 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:03.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:03.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:03.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:03.447 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:03.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:03.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:03.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:03.582 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:03.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:03.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-05T07:26:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:03.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:03.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:03.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:03.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:03.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:03.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:26:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:26:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:26:03.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:04.008 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:04.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:04.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:04.024 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 33 16 64777 64761 122.637 100.938 0.00940967 0.00814893 2024-02-05T07:26:04.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:04.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:04.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:04.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:04.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:04.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:04.432 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:04.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:04.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:04.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:04.569 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:04.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:04.670 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:04.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:04.713 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:04.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:04.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:04.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:04.853 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:04.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:04.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:04 smithi007 ceph-mon[29098]: pgmap v347: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 65 MiB/s wr, 1.05k op/s; 18602/177831 objects degraded (10.460%); 11 KiB/s, 0 objects/s recovering 2024-02-05T07:26:04.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:04 smithi007 ceph-mon[37158]: pgmap v347: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 65 MiB/s wr, 1.05k op/s; 18602/177831 objects degraded (10.460%); 11 KiB/s, 0 objects/s recovering 2024-02-05T07:26:04.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:04.993 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:04.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:04.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:05.024 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 34 16 66340 66324 121.903 97.6875 0.00793655 0.00819796 2024-02-05T07:26:05.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:04 smithi090 ceph-mon[33213]: pgmap v347: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 65 MiB/s wr, 1.05k op/s; 18602/177831 objects degraded (10.460%); 11 KiB/s, 0 objects/s recovering 2024-02-05T07:26:05.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:05.132 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:05.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:05.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:05.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:05.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:05.267 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:05.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:05.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:05.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:05.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:05.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:05.553 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:05.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:05.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:05.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T07:26:05.696 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:05.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:05.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:05.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:05.831 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:05.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:05.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:05.977 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:05.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:05.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:06.025 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 35 16 67690 67674 120.826 84.375 0.0125082 0.00827157 2024-02-05T07:26:06.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:06.112 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:06.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:06.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:06.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:06.247 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:06.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:06.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:06.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:06.407 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:06.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:06.508 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:06.544 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:06.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:06.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:06.630 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:06 smithi007 ceph-mon[29098]: pgmap v348: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 50 MiB/s wr, 802 op/s; 18602/177831 objects degraded (10.460%); 8.2 KiB/s, 0 objects/s recovering 2024-02-05T07:26:06.630 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:06 smithi007 ceph-mon[37158]: pgmap v348: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 50 MiB/s wr, 802 op/s; 18602/177831 objects degraded (10.460%); 8.2 KiB/s, 0 objects/s recovering 2024-02-05T07:26:06.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:06.675 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:06.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:06.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:06.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:06.822 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:06.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:06.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:06.926 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:06.972 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:06.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:06.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:07.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:06 smithi090 ceph-mon[33213]: pgmap v348: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 50 MiB/s wr, 802 op/s; 18602/177831 objects degraded (10.460%); 8.2 KiB/s, 0 objects/s recovering 2024-02-05T07:26:07.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 36 16 68957 68941 119.668 79.1875 0.0075295 0.00835134 2024-02-05T07:26:07.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:07.099 INFO:teuthology.orchestra.run.smithi090.stdout:8.9 2024-02-05T07:26:07.100 INFO:teuthology.orchestra.run.smithi090.stdout:8.7 2024-02-05T07:26:07.100 INFO:teuthology.orchestra.run.smithi090.stdout:8.5 2024-02-05T07:26:07.100 INFO:teuthology.orchestra.run.smithi090.stdout:6.8 2024-02-05T07:26:07.100 INFO:teuthology.orchestra.run.smithi090.stdout:8.6 2024-02-05T07:26:07.100 INFO:teuthology.orchestra.run.smithi090.stdout:6.6 2024-02-05T07:26:07.100 INFO:teuthology.orchestra.run.smithi090.stdout:6.5 2024-02-05T07:26:07.100 INFO:teuthology.orchestra.run.smithi090.stdout:6.c 2024-02-05T07:26:07.100 INFO:teuthology.orchestra.run.smithi090.stdout:6.4 2024-02-05T07:26:07.100 INFO:teuthology.orchestra.run.smithi090.stdout:6.1f 2024-02-05T07:26:07.100 INFO:teuthology.orchestra.run.smithi090.stdout:6.1e 2024-02-05T07:26:07.100 INFO:teuthology.orchestra.run.smithi090.stdout:5.f 2024-02-05T07:26:07.100 INFO:teuthology.orchestra.run.smithi090.stdout:5.3 2024-02-05T07:26:07.100 INFO:teuthology.orchestra.run.smithi090.stdout:6.f 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:6.a 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:6.2 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:5.2 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:6.3 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:5.18 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:5.16 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:5.11 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:4.e 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:5.10 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:4.c 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:1.3 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:6.15 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:4.a 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:3.17 2024-02-05T07:26:07.101 INFO:teuthology.orchestra.run.smithi090.stdout:3.14 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:5.9 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:3.10 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:3.0 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:2.2 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:3.a 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:3.11 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:1.7 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:3.1e 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:1.0 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:6.1 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:1.5 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:4.1d 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:1.4 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:3.3 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:4.1f 2024-02-05T07:26:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:8.3 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:4.4 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:1.6 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:6.b 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:3.2 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:6.13 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:1.a 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:3.5 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:6.1b 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:3.19 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:1.8 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:3.c 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:4.8 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:3.1d 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:4.10 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:4.11 2024-02-05T07:26:07.103 INFO:teuthology.orchestra.run.smithi090.stdout:3.d 2024-02-05T07:26:07.104 INFO:teuthology.orchestra.run.smithi090.stdout:4.9 2024-02-05T07:26:07.104 INFO:teuthology.orchestra.run.smithi090.stdout:4.12 2024-02-05T07:26:07.104 INFO:teuthology.orchestra.run.smithi090.stdout:4.13 2024-02-05T07:26:07.104 INFO:teuthology.orchestra.run.smithi090.stdout:4.15 2024-02-05T07:26:07.104 INFO:teuthology.orchestra.run.smithi090.stdout:4.18 2024-02-05T07:26:07.104 INFO:teuthology.orchestra.run.smithi090.stdout:4.1 2024-02-05T07:26:07.104 INFO:teuthology.orchestra.run.smithi090.stdout:5.1f 2024-02-05T07:26:07.104 INFO:teuthology.orchestra.run.smithi090.stdout:4.5 2024-02-05T07:26:07.115 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:07.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:07.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:07.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:07.257 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:07.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:07.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:07.359 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:07.401 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:07.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:07.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:07.538 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:07.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:07.578 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:26:07.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:07 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/864384165' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:07.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:07 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:07.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:07 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/864384165' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:07.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:07 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:07.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:07.671 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:07.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:07.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:07.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:07.798 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:07.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:07.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:07.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:07.938 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:07.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:07.966 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:07 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/864384165' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:07.966 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:07 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:08.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 37 16 70414 70398 118.895 91.0625 0.00654246 0.00840634 2024-02-05T07:26:08.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:08.077 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:08.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:08.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:08.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:08.215 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:08.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:08.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -n osd.6 --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op export --pgid 3.0 --file /var/log/ceph/exp.3.0.6 2024-02-05T07:26:08.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:08.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:08.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:08.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:08.520 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:08.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:08.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:08.597 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/osd.6/config 2024-02-05T07:26:08.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:08.660 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:08.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:08.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:08.796 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:08.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:08.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:08.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:08 smithi007 ceph-mon[29098]: pgmap v349: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 116 B/s rd, 98 MiB/s wr, 1.57k op/s; 20968/200364 objects degraded (10.465%); 7.3 KiB/s, 0 objects/s recovering 2024-02-05T07:26:08.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:08 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 20968/200364 objects degraded (10.465%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:08.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:08 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:08.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:08 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:26:08.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:08 smithi007 ceph-mon[29098]: osdmap e233: 8 total, 7 up, 7 in 2024-02-05T07:26:08.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:08 smithi007 ceph-mon[37158]: pgmap v349: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 116 B/s rd, 98 MiB/s wr, 1.57k op/s; 20968/200364 objects degraded (10.465%); 7.3 KiB/s, 0 objects/s recovering 2024-02-05T07:26:08.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:08 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 20968/200364 objects degraded (10.465%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:08.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:08 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:08.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:08 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:26:08.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:08 smithi007 ceph-mon[37158]: osdmap e233: 8 total, 7 up, 7 in 2024-02-05T07:26:08.875 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:08 smithi090 ceph-mon[33213]: pgmap v349: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 116 B/s rd, 98 MiB/s wr, 1.57k op/s; 20968/200364 objects degraded (10.465%); 7.3 KiB/s, 0 objects/s recovering 2024-02-05T07:26:08.875 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:08 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 20968/200364 objects degraded (10.465%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:08.875 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:08 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:08.875 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:08 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:26:08.875 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:08 smithi090 ceph-mon[33213]: osdmap e233: 8 total, 7 up, 7 in 2024-02-05T07:26:08.899 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:08.937 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:08.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:08.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:09.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 38 16 71926 71910 118.253 94.5 0.00721451 0.00845102 2024-02-05T07:26:09.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:09.078 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:09.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:09.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:09.180 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:09.218 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:09.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:09.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:09.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:09.473 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:09.508 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:09.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:09.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:09.561 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:26:09.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:09.611 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:26:09.652 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:26:09.653 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:09.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:09.694 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:26:09.734 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:26:09.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:09.775 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:26:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:09.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:09.815 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:26:09.815 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:26:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:26:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:26:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:26:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:26:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:26:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-02-05T07:26:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:26:09.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:09.923 INFO:teuthology.orchestra.run.smithi090.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-05T07:26:09.939 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:09.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:10.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 39 16 73414 73398 117.605 93 0.00757819 0.0084983 2024-02-05T07:26:10.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:10.088 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:10.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:10.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:10.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:10.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:10.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:10.377 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:10.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:10.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:10 smithi007 ceph-mon[29098]: pgmap v351: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 103 B/s rd, 88 MiB/s wr, 1.40k op/s; 20968/200364 objects degraded (10.465%); 6.5 KiB/s, 0 objects/s recovering 2024-02-05T07:26:10.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:10 smithi007 ceph-mon[29098]: osdmap e234: 8 total, 7 up, 7 in 2024-02-05T07:26:10.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:10 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2151698052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:10.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:10 smithi007 ceph-mon[37158]: pgmap v351: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 103 B/s rd, 88 MiB/s wr, 1.40k op/s; 20968/200364 objects degraded (10.465%); 6.5 KiB/s, 0 objects/s recovering 2024-02-05T07:26:10.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:10 smithi007 ceph-mon[37158]: osdmap e234: 8 total, 7 up, 7 in 2024-02-05T07:26:10.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:10 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2151698052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:10.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:10.515 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:26:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:26:10.515 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:10 smithi090 ceph-mon[33213]: pgmap v351: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 103 B/s rd, 88 MiB/s wr, 1.40k op/s; 20968/200364 objects degraded (10.465%); 6.5 KiB/s, 0 objects/s recovering 2024-02-05T07:26:10.515 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:10 smithi090 ceph-mon[33213]: osdmap e234: 8 total, 7 up, 7 in 2024-02-05T07:26:10.515 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:10 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2151698052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:10.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:10.517 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:10.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:10.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:10.652 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:10.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:10.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:10.757 INFO:teuthology.orchestra.run.smithi090.stderr:Exporting 3.0 info 3.0( v 68'1 (0'0,68'1] local-lis/les=67/68 n=1 ec=67/67 lis/c=67/67 les/c/f=68/68/0 sis=67) 2024-02-05T07:26:10.768 INFO:teuthology.orchestra.run.smithi090.stderr:Read #3:008372bf:::zone_info.1547a86d-f9f1-4768-bef5-8d2d69f94d5a:head# 2024-02-05T07:26:10.768 INFO:teuthology.orchestra.run.smithi090.stderr:Export successful 2024-02-05T07:26:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:10.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:10.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:10.925 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:10.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:10.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:11.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:26:11.025213+0000 min lat: 0.00128428 max lat: 0.975632 avg lat: 0.00853832 2024-02-05T07:26:11.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:26:11.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 40 16 74939 74923 117.048 95.3125 0.00857788 0.00853832 2024-02-05T07:26:11.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:11.065 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:11.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:11.154 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:26:11.154 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-02-05T07:26:11.154 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75617 7.13 7:cf0c022b:::benchmark_data_smithi007_77086_object75616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:44.133001+0000", 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 267.01266847900001, 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3484787149999999, 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75617 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133001+0000", 2024-02-05T07:26:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133001+0000", 2024-02-05T07:26:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133003+0000", 2024-02-05T07:26:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7689999999999999e-06 2024-02-05T07:26:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133028+0000", 2024-02-05T07:26:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5148999999999999e-05 2024-02-05T07:26:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133044+0000", 2024-02-05T07:26:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6588000000000001e-05 2024-02-05T07:26:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133054+0000", 2024-02-05T07:26:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0030000000000001e-05 2024-02-05T07:26:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135655+0000", 2024-02-05T07:26:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002600406 2024-02-05T07:26:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135664+0000", 2024-02-05T07:26:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0839999999999998e-06 2024-02-05T07:26:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.441044+0000", 2024-02-05T07:26:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3053797330000001 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.441211+0000", 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016769699999999999 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.444465+0000", 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032534159999999999 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.449312+0000", 2024-02-05T07:26:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0048477559999999999 2024-02-05T07:26:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.456573+0000", 2024-02-05T07:26:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0072603210000000001 2024-02-05T07:26:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.479561+0000", 2024-02-05T07:26:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.022988044999999999 2024-02-05T07:26:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.479628+0000", 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6933000000000004e-05 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.481480+0000", 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001851788 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75621 7.13 7:ce4ef747:::benchmark_data_smithi007_77086_object75620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:44.133075+0000", 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 267.01259475400002, 2024-02-05T07:26:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.350304054, 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75621 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133075+0000", 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133075+0000", 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133075+0000", 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4700000000000001e-07 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133096+0000", 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0849999999999999e-05 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133103+0000", 2024-02-05T07:26:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.289e-06 2024-02-05T07:26:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133110+0000", 2024-02-05T07:26:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2559999999999999e-06 2024-02-05T07:26:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135691+0000", 2024-02-05T07:26:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025816580000000001 2024-02-05T07:26:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135699+0000", 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5569999999999998e-06 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.447693+0000", 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.311994023 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.447870+0000", 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017759699999999999 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.448090+0000", 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000219566 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.453587+0000", 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054966609999999999 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.456653+0000", 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030662329999999998 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.481496+0000", 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.024843124000000001 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.481553+0000", 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6889999999999999e-05 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.483379+0000", 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001825803 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:167698 7.14 7:29493818:::benchmark_data_smithi007_77086_object63103:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e164)", 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:50.618760+0000", 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 140.52690884899999, 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.052304715, 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 167698 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.618760+0000", 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.618760+0000", 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.618761+0000", 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9599999999999998e-07 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.618765+0000", 2024-02-05T07:26:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.189e-06 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.618776+0000", 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0902e-05 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.618785+0000", 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1710000000000002e-06 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.619268+0000", 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00048305199999999998 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.619387+0000", 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011914899999999999 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,3,5", 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.619751+0000", 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00036416399999999999 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.622694+0000", 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0029429730000000002 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.629424+0000", 2024-02-05T07:26:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00673056 2024-02-05T07:26:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.634303+0000", 2024-02-05T07:26:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0048785130000000001 2024-02-05T07:26:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.652101+0000", 2024-02-05T07:26:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.017798293999999999 2024-02-05T07:26:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.652116+0000", 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4260999999999999e-05 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.670979+0000", 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.01886342 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.671065+0000", 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6271000000000001e-05 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:167774 7.14 7:2c569e8c:::benchmark_data_smithi007_77086_object63179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:50.714711+0000", 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 140.430958809, 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99667040399999995, 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 167774 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714711+0000", 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714711+0000", 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714711+0000", 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9400000000000004e-07 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714715+0000", 2024-02-05T07:26:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8029999999999998e-06 2024-02-05T07:26:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714724+0000", 2024-02-05T07:26:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7360000000000007e-06 2024-02-05T07:26:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714731+0000", 2024-02-05T07:26:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8360000000000003e-06 2024-02-05T07:26:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714778+0000", 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6859e-05 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714785+0000", 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.255e-06 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.674406+0000", 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.95962156499999995 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.674516+0000", 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000109478 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.674803+0000", 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028674499999999998 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.679654+0000", 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0048510940000000002 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.679715+0000", 2024-02-05T07:26:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1455e-05 2024-02-05T07:26:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.711096+0000", 2024-02-05T07:26:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.031380907999999999 2024-02-05T07:26:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.711335+0000", 2024-02-05T07:26:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000239344 2024-02-05T07:26:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.711381+0000", 2024-02-05T07:26:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5531999999999997e-05 2024-02-05T07:26:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:167790 7.14 7:2e0a6b7a:::benchmark_data_smithi007_77086_object63195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T07:26:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:50.728786+0000", 2024-02-05T07:26:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 140.41688315799999, 2024-02-05T07:26:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98310929400000002, 2024-02-05T07:26:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 167790 2024-02-05T07:26:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.728786+0000", 2024-02-05T07:26:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.728786+0000", 2024-02-05T07:26:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.728787+0000", 2024-02-05T07:26:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4300000000000001e-07 2024-02-05T07:26:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.728792+0000", 2024-02-05T07:26:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.544e-06 2024-02-05T07:26:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.728802+0000", 2024-02-05T07:26:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0753e-05 2024-02-05T07:26:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.728811+0000", 2024-02-05T07:26:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0989999999999999e-06 2024-02-05T07:26:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.729151+0000", 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00034059900000000002 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.729160+0000", 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2699999999999993e-06 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.676673+0000", 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.94751278999999999 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.676786+0000", 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011234300000000001 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.677150+0000", 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00036407799999999999 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.683604+0000", 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0064540830000000002 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.690849+0000", 2024-02-05T07:26:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0072456550000000002 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.711389+0000", 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.020539458999999999 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.711857+0000", 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00046811200000000002 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.711895+0000", 2024-02-05T07:26:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8566000000000003e-05 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:172539 7.3 7:c1cc9372:::benchmark_data_smithi007_77086_object67944:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:56.366961+0000", 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 134.77870783700001, 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0132783910000001, 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 172539 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.366961+0000", 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.366961+0000", 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.366964+0000", 2024-02-05T07:26:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0540000000000002e-06 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.366968+0000", 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8180000000000003e-06 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.366989+0000", 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0443000000000001e-05 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.366998+0000", 2024-02-05T07:26:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0429999999999996e-06 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.367057+0000", 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9054000000000001e-05 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.367066+0000", 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8720000000000008e-06 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.367124+0000", 2024-02-05T07:26:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0000582739999999 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.367240+0000", 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011589199999999999 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6", 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.367712+0000", 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00047228400000000001 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.376730+0000", 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0090178670000000006 2024-02-05T07:26:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.376768+0000", 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7564999999999997e-05 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.380151+0000", 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033831460000000001 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.380196+0000", 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5605999999999997e-05 2024-02-05T07:26:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.380240+0000", 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3473000000000002e-05 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:172576 7.3 7:c184961a:::benchmark_data_smithi007_77086_object67981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:56.399380+0000", 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 134.746289007, 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98101083600000005, 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 172576 2024-02-05T07:26:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399380+0000", 2024-02-05T07:26:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399380+0000", 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399382+0000", 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5489999999999999e-06 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399388+0000", 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9610000000000002e-06 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399400+0000", 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1773999999999999e-05 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399408+0000", 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.089e-06 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399465+0000", 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7015999999999998e-05 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399474+0000", 2024-02-05T07:26:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8449999999999992e-06 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.370277+0000", 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97080362399999998 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.370383+0000", 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010588 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6", 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.370860+0000", 2024-02-05T07:26:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00047653399999999997 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.376807+0000", 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059478839999999996 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.376849+0000", 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1213e-05 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.380314+0000", 2024-02-05T07:26:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0034657580000000002 2024-02-05T07:26:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.380356+0000", 2024-02-05T07:26:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1798000000000002e-05 2024-02-05T07:26:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.380391+0000", 2024-02-05T07:26:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4910999999999998e-05 2024-02-05T07:26:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:172590 7.3 7:c2af45be:::benchmark_data_smithi007_77086_object67995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-02-05T07:26:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:56.414416+0000", 2024-02-05T07:26:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 134.73125382399999, 2024-02-05T07:26:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96900304500000001, 2024-02-05T07:26:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 172590 2024-02-05T07:26:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414416+0000", 2024-02-05T07:26:11.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414416+0000", 2024-02-05T07:26:11.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414417+0000", 2024-02-05T07:26:11.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5680000000000001e-06 2024-02-05T07:26:11.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.203 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414422+0000", 2024-02-05T07:26:11.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7090000000000001e-06 2024-02-05T07:26:11.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.203 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414434+0000", 2024-02-05T07:26:11.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1782e-05 2024-02-05T07:26:11.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.203 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414442+0000", 2024-02-05T07:26:11.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1440000000000003e-06 2024-02-05T07:26:11.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.204 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.204 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414568+0000", 2024-02-05T07:26:11.204 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012621400000000001 2024-02-05T07:26:11.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.204 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.204 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414578+0000", 2024-02-05T07:26:11.204 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6120000000000005e-06 2024-02-05T07:26:11.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.204 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.204 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.373764+0000", 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.95918598799999999 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.373873+0000", 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000109548 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6", 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.374423+0000", 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00054968200000000001 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.380397+0000", 2024-02-05T07:26:11.206 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059747280000000003 2024-02-05T07:26:11.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.206 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.381204+0000", 2024-02-05T07:26:11.206 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00080686199999999999 2024-02-05T07:26:11.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.383267+0000", 2024-02-05T07:26:11.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020622879999999998 2024-02-05T07:26:11.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.207 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.383377+0000", 2024-02-05T07:26:11.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011037 2024-02-05T07:26:11.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.207 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.383419+0000", 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.155e-05 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174548 7.a 7:51469907:::benchmark_data_smithi007_77086_object69953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.374698+0000", 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 130.77097173600001, 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1298535379999999, 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.208 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174548 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.374698+0000", 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.374698+0000", 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.374699+0000", 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.747e-06 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.374707+0000", 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2269999999999996e-06 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.374720+0000", 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3045e-05 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.374729+0000", 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8390000000000002e-06 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.438715+0000", 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.063985095000000006 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.438727+0000", 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2291e-05 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.501248+0000", 2024-02-05T07:26:11.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.062520916 2024-02-05T07:26:11.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.212 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.212 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.501389+0000", 2024-02-05T07:26:11.212 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014126500000000001 2024-02-05T07:26:11.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.212 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:26:11.212 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.501459+0000", 2024-02-05T07:26:11.212 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9876000000000007e-05 2024-02-05T07:26:11.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.212 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.212 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504052+0000", 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025928729999999999 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504402+0000", 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00035052899999999998 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504465+0000", 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2397999999999994e-05 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504516+0000", 2024-02-05T07:26:11.214 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1242000000000002e-05 2024-02-05T07:26:11.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.214 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.214 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504551+0000", 2024-02-05T07:26:11.214 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5194999999999998e-05 2024-02-05T07:26:11.214 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.214 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.214 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.214 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174560 7.a 7:52e66e99:::benchmark_data_smithi007_77086_object69965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:26:11.214 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.439432+0000", 2024-02-05T07:26:11.214 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 130.706237034, 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.073606013, 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174560 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.439432+0000", 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.215 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.439432+0000", 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.439433+0000", 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7800000000000002e-07 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.439435+0000", 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6589999999999999e-06 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.439442+0000", 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2339999999999999e-06 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.439447+0000", 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.677e-06 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.441161+0000", 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017137260000000001 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.441169+0000", 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7300000000000005e-06 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504603+0000", 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0634339829999999 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504696+0000", 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3459000000000001e-05 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.218 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:26:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504776+0000", 2024-02-05T07:26:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0060000000000003e-05 2024-02-05T07:26:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.512888+0000", 2024-02-05T07:26:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0081121189999999992 2024-02-05T07:26:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.512907+0000", 2024-02-05T07:26:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8950999999999999e-05 2024-02-05T07:26:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.219 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.512943+0000", 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5565999999999998e-05 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.512999+0000", 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6399e-05 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.513038+0000", 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8971999999999999e-05 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.220 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174570 7.a 7:56c3dbcb:::benchmark_data_smithi007_77086_object69975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.442757+0000", 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 130.70291266199999, 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0731317869999999, 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174570 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442757+0000", 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442757+0000", 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442758+0000", 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.564e-06 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442764+0000", 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6779999999999997e-06 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442773+0000", 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3810000000000005e-06 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442781+0000", 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0390000000000005e-06 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.223 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442954+0000", 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017238600000000001 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442962+0000", 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4090000000000003e-06 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.507306+0000", 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0643434119999999 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.224 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.507414+0000", 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108436 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.507527+0000", 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011256699999999999 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.513048+0000", 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0055218790000000004 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.513083+0000", 2024-02-05T07:26:11.225 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4686e-05 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.515804+0000", 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002721135 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.515862+0000", 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7716000000000001e-05 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.515888+0000", 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6499000000000002e-05 2024-02-05T07:26:11.226 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.227 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.227 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.227 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.227 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.227 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174574 7.a 7:57454b6e:::benchmark_data_smithi007_77086_object69979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:26:11.227 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.444932+0000", 2024-02-05T07:26:11.227 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 130.70073727400001, 2024-02-05T07:26:11.227 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.071142161, 2024-02-05T07:26:11.227 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.227 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.227 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.227 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.227 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.227 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174574 2024-02-05T07:26:11.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.228 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.228 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.228 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.444932+0000", 2024-02-05T07:26:11.228 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.228 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.228 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.444932+0000", 2024-02-05T07:26:11.228 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.228 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.444934+0000", 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2699999999999999e-06 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.444940+0000", 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3929999999999999e-06 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.444952+0000", 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.187e-05 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.229 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.444960+0000", 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8910000000000001e-06 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445018+0000", 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7571999999999997e-05 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445028+0000", 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0023e-05 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.230 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.510169+0000", 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0651404339999999 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.510300+0000", 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013160900000000001 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.510424+0000", 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000124313 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.231 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.515907+0000", 2024-02-05T07:26:11.232 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.005482748 2024-02-05T07:26:11.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.232 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.232 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.232 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.515947+0000", 2024-02-05T07:26:11.233 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9425000000000003e-05 2024-02-05T07:26:11.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.233 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.233 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.233 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.515986+0000", 2024-02-05T07:26:11.233 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9833000000000003e-05 2024-02-05T07:26:11.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.233 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.233 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.233 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516038+0000", 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1842000000000003e-05 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516074+0000", 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5938e-05 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174575 7.a 7:546d0937:::benchmark_data_smithi007_77086_object69980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.445882+0000", 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 130.699786923, 2024-02-05T07:26:11.234 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0703599480000001, 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174575 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445882+0000", 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.235 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445882+0000", 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445883+0000", 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0800000000000003e-07 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445887+0000", 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.388e-06 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.236 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445895+0000", 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9950000000000005e-06 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445912+0000", 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7648000000000001e-05 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445991+0000", 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8555999999999998e-05 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445999+0000", 2024-02-05T07:26:11.237 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1759999999999993e-06 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.513115+0000", 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0671161650000001 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.513221+0000", 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000105461 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.513337+0000", 2024-02-05T07:26:11.238 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115876 2024-02-05T07:26:11.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.239 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.239 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516083+0000", 2024-02-05T07:26:11.239 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0027464239999999999 2024-02-05T07:26:11.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.239 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.239 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516119+0000", 2024-02-05T07:26:11.239 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6038000000000003e-05 2024-02-05T07:26:11.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.239 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.239 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516161+0000", 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1631999999999997e-05 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516212+0000", 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1264999999999998e-05 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516242+0000", 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.0416000000000002e-05 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174577 7.a 7:5537fb92:::benchmark_data_smithi007_77086_object69982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.448795+0000", 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 130.696874767, 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0775387089999999, 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174577 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448795+0000", 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448795+0000", 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448796+0000", 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1909999999999999e-06 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.242 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448800+0000", 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8449999999999996e-06 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448808+0000", 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5529999999999993e-06 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448815+0000", 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4710000000000004e-06 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.243 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448956+0000", 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000141304 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448968+0000", 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1703e-05 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516273+0000", 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0673056679999999 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516386+0000", 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000112255 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516498+0000", 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011268300000000001 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.522764+0000", 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0062659279999999996 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.245 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.246 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.522779+0000", 2024-02-05T07:26:11.246 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4786e-05 2024-02-05T07:26:11.246 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.246 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.246 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.246 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526239+0000", 2024-02-05T07:26:11.246 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003459873 2024-02-05T07:26:11.246 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.246 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.246 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.246 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526293+0000", 2024-02-05T07:26:11.246 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3841000000000002e-05 2024-02-05T07:26:11.246 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.246 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.247 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.247 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526333+0000", 2024-02-05T07:26:11.247 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0608000000000002e-05 2024-02-05T07:26:11.247 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.247 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.247 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.247 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.247 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.247 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174585 7.a 7:5d56743a:::benchmark_data_smithi007_77086_object69990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:26:11.247 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.455082+0000", 2024-02-05T07:26:11.247 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 130.69058727800001, 2024-02-05T07:26:11.247 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0802006580000001, 2024-02-05T07:26:11.247 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.247 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.248 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.248 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.248 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.248 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174585 2024-02-05T07:26:11.248 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.248 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.248 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.248 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.248 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455082+0000", 2024-02-05T07:26:11.248 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.248 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.248 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.248 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.248 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455082+0000", 2024-02-05T07:26:11.249 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.249 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.249 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.249 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.249 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455084+0000", 2024-02-05T07:26:11.249 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.497e-06 2024-02-05T07:26:11.249 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.249 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.249 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.249 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455088+0000", 2024-02-05T07:26:11.249 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1010000000000002e-06 2024-02-05T07:26:11.249 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.249 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.249 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455097+0000", 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8950000000000004e-06 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455104+0000", 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1420000000000004e-06 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455217+0000", 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113678 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.250 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455226+0000", 2024-02-05T07:26:11.251 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8729999999999996e-06 2024-02-05T07:26:11.251 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.251 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.251 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.251 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.519622+0000", 2024-02-05T07:26:11.251 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0643955549999999 2024-02-05T07:26:11.251 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.251 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.251 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.251 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.519736+0000", 2024-02-05T07:26:11.251 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000114598 2024-02-05T07:26:11.251 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.251 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.251 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:26:11.252 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.519855+0000", 2024-02-05T07:26:11.252 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011813 2024-02-05T07:26:11.252 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.252 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.252 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.252 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.522879+0000", 2024-02-05T07:26:11.252 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030242789999999999 2024-02-05T07:26:11.252 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.252 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.252 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.252 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526372+0000", 2024-02-05T07:26:11.252 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0034927719999999999 2024-02-05T07:26:11.252 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.252 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535191+0000", 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0088197870000000008 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535240+0000", 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8751999999999997e-05 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535283+0000", 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2599000000000003e-05 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.253 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174598 7.a 7:5c33d55a:::benchmark_data_smithi007_77086_object70003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.466123+0000", 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 130.67954594599999, 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0693828480000001, 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174598 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.254 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.255 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.255 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.255 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466123+0000", 2024-02-05T07:26:11.255 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.255 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.255 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.255 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466123+0000", 2024-02-05T07:26:11.255 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.255 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.255 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.255 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466124+0000", 2024-02-05T07:26:11.255 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7199999999999998e-07 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466128+0000", 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2629999999999997e-06 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466136+0000", 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8970000000000008e-06 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466143+0000", 2024-02-05T07:26:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3430000000000001e-06 2024-02-05T07:26:11.257 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.257 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.257 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.257 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466266+0000", 2024-02-05T07:26:11.257 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000123196 2024-02-05T07:26:11.257 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.257 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.257 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.257 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466274+0000", 2024-02-05T07:26:11.257 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8380000000000002e-06 2024-02-05T07:26:11.257 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.257 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.257 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.257 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.522923+0000", 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.056649795 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.523050+0000", 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012692199999999999 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.523166+0000", 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011522 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526407+0000", 2024-02-05T07:26:11.258 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032419609999999998 2024-02-05T07:26:11.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.259 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.259 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.259 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535326+0000", 2024-02-05T07:26:11.259 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0089187580000000006 2024-02-05T07:26:11.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.259 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.259 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.259 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535424+0000", 2024-02-05T07:26:11.260 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7769000000000005e-05 2024-02-05T07:26:11.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.260 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.260 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.260 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535470+0000", 2024-02-05T07:26:11.260 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6415000000000002e-05 2024-02-05T07:26:11.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.260 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.260 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.260 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535506+0000", 2024-02-05T07:26:11.260 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5799000000000001e-05 2024-02-05T07:26:11.260 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.260 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174602 7.a 7:574a2af3:::benchmark_data_smithi007_77086_object70007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.470147+0000", 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 130.675522218, 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.065611767, 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174602 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470147+0000", 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470147+0000", 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470148+0000", 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2139999999999999e-06 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.262 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470151+0000", 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1559999999999999e-06 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470159+0000", 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9540000000000004e-06 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470166+0000", 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4409999999999998e-06 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470215+0000", 2024-02-05T07:26:11.263 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9268999999999999e-05 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470223+0000", 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3159999999999997e-06 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526550+0000", 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0563263060000001 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526752+0000", 2024-02-05T07:26:11.264 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020180799999999999 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526904+0000", 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000152776 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535587+0000", 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0086831110000000003 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535636+0000", 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8826999999999999e-05 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535676+0000", 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0061999999999997e-05 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535723+0000", 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7150000000000001e-05 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535759+0000", 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5376999999999997e-05 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.266 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174607 7.a 7:59b090e0:::benchmark_data_smithi007_77086_object70012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.516807+0000", 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 130.62886194699999, 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.025446978, 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174607 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516807+0000", 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516807+0000", 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516809+0000", 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4780000000000001e-06 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516815+0000", 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.772e-06 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516826+0000", 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1617000000000001e-05 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516835+0000", 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5210000000000002e-06 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.269 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516892+0000", 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7663e-05 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516913+0000", 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0279e-05 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.531495+0000", 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0145823300000001 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.531602+0000", 2024-02-05T07:26:11.270 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010724499999999999 2024-02-05T07:26:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:26:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.531682+0000", 2024-02-05T07:26:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9888000000000005e-05 2024-02-05T07:26:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.271 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535596+0000", 2024-02-05T07:26:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:11.272 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:11.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039139190000000001 2024-02-05T07:26:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535808+0000", 2024-02-05T07:26:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00021158799999999999 2024-02-05T07:26:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.542052+0000", 2024-02-05T07:26:11.273 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0062446610000000003 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.542200+0000", 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014800800000000001 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.542254+0000", 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4008999999999997e-05 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:192513 7.e 7:72d2a106:::benchmark_data_smithi007_77086_object87918:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e177)", 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:19.509516+0000", 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 111.636153433, 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0316103679999999, 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 192513 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.275 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.509516+0000", 2024-02-05T07:26:11.276 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.276 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.276 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.276 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.509516+0000", 2024-02-05T07:26:11.276 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.276 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.276 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.276 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.509517+0000", 2024-02-05T07:26:11.276 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4810000000000001e-06 2024-02-05T07:26:11.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.276 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.276 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.509521+0000", 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3450000000000002e-06 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.509530+0000", 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0850000000000003e-06 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.509536+0000", 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2859999999999996e-06 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.277 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.278 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.512281+0000", 2024-02-05T07:26:11.278 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002744486 2024-02-05T07:26:11.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.278 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-02-05T07:26:11.278 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.512287+0000", 2024-02-05T07:26:11.278 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2960000000000004e-06 2024-02-05T07:26:11.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.278 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.278 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.722402+0000", 2024-02-05T07:26:11.278 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.210115154 2024-02-05T07:26:11.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.722412+0000", 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0029e-05 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.541052+0000", 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.81864034900000004 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.541126+0000", 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3856999999999994e-05 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.279 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:192531 7.1a 7:5f4bc8c6:::benchmark_data_smithi007_77086_object87936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:19.636512+0000", 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 111.50915693899999, 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99201874700000003, 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 192531 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.280 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.636512+0000", 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.636512+0000", 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.636514+0000", 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.392e-06 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.636518+0000", 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.121e-06 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.636529+0000", 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1238e-05 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.636538+0000", 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9800000000000004e-06 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.282 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.736108+0000", 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.099569629000000007 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.736119+0000", 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1232e-05 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.620471+0000", 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.88435169400000002 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:11.283 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.620582+0000", 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000111731 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.620822+0000", 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023922 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.622275+0000", 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001453712 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.626649+0000", 2024-02-05T07:26:11.284 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043739740000000001 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.628445+0000", 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001795439 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.628494+0000", 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9283999999999998e-05 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.628531+0000", 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7101000000000003e-05 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.286 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.286 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:11.286 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:11.286 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:26:11.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:11.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:11.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:11.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:11.472 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:11 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:11.472 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:11 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:11.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:11.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:11 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T07:26:11.527 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T07:26:11.549 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:11.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:11.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:11.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:11.692 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:11.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:11.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:11.829 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:11.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:11.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:11.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:11.972 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:11.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:11.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:12.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 41 16 76375 76359 116.382 89.75 0.00467809 0.00858744 2024-02-05T07:26:12.075 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:12.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:26:12.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:12.119 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:12.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:12.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:12.253 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:12.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:12.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:12.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:12.388 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:12.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:12.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:12.497 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:12 smithi007 ceph-mon[29098]: pgmap v353: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s; 23141/221667 objects degraded (10.440%) 2024-02-05T07:26:12.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:12 smithi007 ceph-mon[37158]: pgmap v353: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s; 23141/221667 objects degraded (10.440%) 2024-02-05T07:26:12.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:12.513 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:12.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:12.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:12 smithi090 ceph-mon[33213]: pgmap v353: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s; 23141/221667 objects degraded (10.440%) 2024-02-05T07:26:12.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T07:26:12.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:12.628 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:12.682 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:12.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:12.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:12.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:12.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:12.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:12.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:12.962 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:12.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:13.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 42 16 77484 77468 115.261 69.3125 0.0111677 0.00867066 2024-02-05T07:26:13.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:13.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:13.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:13.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:13.225 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:13.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:13.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:13.407 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:13.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:13.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:13.528 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:26:13.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:13.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:13.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:13.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:13.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:13.680 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:13.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:13.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:13.817 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:13.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:13.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:13.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:26:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:26:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:26:13.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:13.943 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:13.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:14.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 43 16 78491 78475 114.044 62.9375 0.0186738 0.00876326 2024-02-05T07:26:14.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:14.077 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:14.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:14.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:14.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:14.214 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:14.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:14.316 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:14.352 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:14.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:14.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:14.480 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:14.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:14.498 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:14.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:14 smithi090 ceph-mon[33213]: pgmap v354: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s; 23141/221667 objects degraded (10.440%) 2024-02-05T07:26:14.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:14 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 23141/221667 objects degraded (10.440%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:14.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:14.609 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:14 smithi007 ceph-mon[29098]: pgmap v354: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s; 23141/221667 objects degraded (10.440%) 2024-02-05T07:26:14.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:14 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 23141/221667 objects degraded (10.440%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:14.610 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:14 smithi007 ceph-mon[37158]: pgmap v354: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s; 23141/221667 objects degraded (10.440%) 2024-02-05T07:26:14.610 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:14 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 23141/221667 objects degraded (10.440%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:14.612 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:14.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:14.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:14.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:14.756 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:14.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:14.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:14.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:14.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:14.905 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:14.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:15.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:15.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 44 15 79852 79837 113.387 85.125 0.00455775 0.0088149 2024-02-05T07:26:15.042 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:15.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:15.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:15.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:15.176 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:15.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:15.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:15.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:15.311 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:15.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:15.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:15.449 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:15.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:15.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:15.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:15 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:26:15.588 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:15.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:15.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:15.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:15.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:15.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:15.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:15.874 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:15.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:15.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:15 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:26:15.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:26:15.919 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:26:15.975 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:16.018 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:16.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:15 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:26:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:26:16.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 45 16 81136 81120 112.649 80.1875 0.0101198 0.00887249 2024-02-05T07:26:16.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:16.152 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:16.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:16.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:16.233 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -n osd.6 --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --force --op remove --pgid 3.0 2024-02-05T07:26:16.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:16.324 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:16.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:16.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:16.451 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:16.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:16.492 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/osd.6/config 2024-02-05T07:26:16.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:16.580 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:26:16.598 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:16.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:16.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:16.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:16.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:16.746 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:16.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:16.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:16.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:16.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:16.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:16 smithi007 ceph-mon[29098]: pgmap v355: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 55 MiB/s wr, 887 op/s; 23141/221667 objects degraded (10.440%) 2024-02-05T07:26:16.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:16 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3354289213' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:26:16.896 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:16 smithi007 ceph-mon[37158]: pgmap v355: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 55 MiB/s wr, 887 op/s; 23141/221667 objects degraded (10.440%) 2024-02-05T07:26:16.897 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:16 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3354289213' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:26:16.972 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:16.985 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:16 smithi090 ceph-mon[33213]: pgmap v355: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 55 MiB/s wr, 887 op/s; 23141/221667 objects degraded (10.440%) 2024-02-05T07:26:16.985 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:16 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3354289213' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:26:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:17.012 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:17.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:17.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 46 16 82298 82282 111.779 72.625 0.00966696 0.00894161 2024-02-05T07:26:17.113 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:17.150 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:17.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:17.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:17.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:17.283 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:17.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:17.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:17.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:17.427 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:17.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:17.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:17.528 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:17.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:17.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:17.655 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:17 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:17 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3354289213' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:26:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:17 smithi007 ceph-mon[29098]: osdmap e235: 8 total, 7 up, 7 in 2024-02-05T07:26:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:17 smithi007 ceph-mon[29098]: 3.16 scrub starts 2024-02-05T07:26:17.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:17 smithi007 ceph-mon[29098]: 3.16 scrub ok 2024-02-05T07:26:17.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:17 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:17.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:17 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3354289213' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:26:17.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:17 smithi007 ceph-mon[37158]: osdmap e235: 8 total, 7 up, 7 in 2024-02-05T07:26:17.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:17 smithi007 ceph-mon[37158]: 3.16 scrub starts 2024-02-05T07:26:17.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:17 smithi007 ceph-mon[37158]: 3.16 scrub ok 2024-02-05T07:26:17.695 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:17.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:17.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:17.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:17.830 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:17.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:17.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:17.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:17.968 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:17.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:17.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:17.990 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:17 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:17.990 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:17 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3354289213' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:26:17.990 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:17 smithi090 ceph-mon[33213]: osdmap e235: 8 total, 7 up, 7 in 2024-02-05T07:26:17.990 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:17 smithi090 ceph-mon[33213]: 3.16 scrub starts 2024-02-05T07:26:17.990 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:17 smithi090 ceph-mon[33213]: 3.16 scrub ok 2024-02-05T07:26:18.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 47 16 83669 83653 111.223 85.6875 0.00671658 0.00898569 2024-02-05T07:26:18.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:18.108 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:18.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:18.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:18.243 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:18.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:18.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:18.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:18.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:18.386 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:18.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:18.487 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:18.522 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:18.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:18.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:18.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:18.646 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:18 smithi090 ceph-mon[33213]: pgmap v357: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 25122/240675 objects degraded (10.438%) 2024-02-05T07:26:18.647 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:18 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:26:18.647 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:18 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:26:18.648 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:18 smithi090 ceph-mon[33213]: 6.1c scrub starts 2024-02-05T07:26:18.648 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:18 smithi090 ceph-mon[33213]: 6.1c scrub ok 2024-02-05T07:26:18.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:18.685 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:18.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:18.709 INFO:teuthology.orchestra.run.smithi090.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-05T07:26:18.786 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:18.787 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:18 smithi007 ceph-mon[29098]: pgmap v357: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 25122/240675 objects degraded (10.438%) 2024-02-05T07:26:18.787 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:18 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:26:18.788 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:18 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:26:18.788 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:18 smithi007 ceph-mon[29098]: 6.1c scrub starts 2024-02-05T07:26:18.788 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:18 smithi007 ceph-mon[29098]: 6.1c scrub ok 2024-02-05T07:26:18.789 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:18 smithi007 ceph-mon[37158]: pgmap v357: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 25122/240675 objects degraded (10.438%) 2024-02-05T07:26:18.789 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:18 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:26:18.789 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:18 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:26:18.790 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:18 smithi007 ceph-mon[37158]: 6.1c scrub starts 2024-02-05T07:26:18.790 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:18 smithi007 ceph-mon[37158]: 6.1c scrub ok 2024-02-05T07:26:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:18.841 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:18.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:18.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:18.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:18.978 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:18.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:19.027 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 48 16 85092 85076 110.759 88.9375 0.0215142 0.00902392 2024-02-05T07:26:19.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:19.124 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:19.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:19.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:19.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:19.257 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:19.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:19.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:19.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:19.390 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:19.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:19.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:19.455 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:26:19.455 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:26:19.455 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 0 2024-02-05T07:26:19.455 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:26:19.478 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:26:19.478 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:26:19.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: marking collection for removal 2024-02-05T07:26:19.502 INFO:teuthology.orchestra.run.smithi090.stdout:setting '_remove' omap key 2024-02-05T07:26:19.502 INFO:teuthology.orchestra.run.smithi090.stdout:finish_remove_pgs 3.0_head removing 3.0 2024-02-05T07:26:19.503 INFO:teuthology.orchestra.run.smithi090.stdout:Remove successful 2024-02-05T07:26:19.504 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:26:19.525 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:19.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:19.549 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:26:19.591 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:26:19.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:19.634 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:26:19.662 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:19.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:19.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:19.677 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:26:19.677 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:26:19.677 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:26:19.677 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:26:19.677 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:26:19.677 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:26:19.677 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:26:19.678 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-02-05T07:26:19.678 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:26:19.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:19.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:26:19.805 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:19.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:19.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:19.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[29098]: 8.e scrub starts 2024-02-05T07:26:19.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[29098]: 3.b scrub starts 2024-02-05T07:26:19.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[29098]: 3.b scrub ok 2024-02-05T07:26:19.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[29098]: 5.12 scrub starts 2024-02-05T07:26:19.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[29098]: 5.12 scrub ok 2024-02-05T07:26:19.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[29098]: osdmap e236: 8 total, 7 up, 7 in 2024-02-05T07:26:19.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[29098]: 6.d scrub starts 2024-02-05T07:26:19.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[29098]: 6.d scrub ok 2024-02-05T07:26:19.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[29098]: 1.9 scrub starts 2024-02-05T07:26:19.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[29098]: 1.9 scrub ok 2024-02-05T07:26:19.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[29098]: pgmap v359: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 49 MiB/s wr, 792 op/s; 25122/240675 objects degraded (10.438%) 2024-02-05T07:26:19.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 25122/240675 objects degraded (10.438%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:19.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1822087785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:19.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[37158]: 8.e scrub starts 2024-02-05T07:26:19.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[37158]: 3.b scrub starts 2024-02-05T07:26:19.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[37158]: 3.b scrub ok 2024-02-05T07:26:19.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[37158]: 5.12 scrub starts 2024-02-05T07:26:19.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[37158]: 5.12 scrub ok 2024-02-05T07:26:19.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[37158]: osdmap e236: 8 total, 7 up, 7 in 2024-02-05T07:26:19.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[37158]: 6.d scrub starts 2024-02-05T07:26:19.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[37158]: 6.d scrub ok 2024-02-05T07:26:19.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[37158]: 1.9 scrub starts 2024-02-05T07:26:19.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[37158]: 1.9 scrub ok 2024-02-05T07:26:19.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[37158]: pgmap v359: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 49 MiB/s wr, 792 op/s; 25122/240675 objects degraded (10.438%) 2024-02-05T07:26:19.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 25122/240675 objects degraded (10.438%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:19.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:19 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1822087785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:19.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:19.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:20.017 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:19 smithi090 ceph-mon[33213]: 8.e scrub starts 2024-02-05T07:26:20.017 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:19 smithi090 ceph-mon[33213]: 3.b scrub starts 2024-02-05T07:26:20.017 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:19 smithi090 ceph-mon[33213]: 3.b scrub ok 2024-02-05T07:26:20.017 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:19 smithi090 ceph-mon[33213]: 5.12 scrub starts 2024-02-05T07:26:20.017 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:19 smithi090 ceph-mon[33213]: 5.12 scrub ok 2024-02-05T07:26:20.017 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:19 smithi090 ceph-mon[33213]: osdmap e236: 8 total, 7 up, 7 in 2024-02-05T07:26:20.017 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:19 smithi090 ceph-mon[33213]: 6.d scrub starts 2024-02-05T07:26:20.017 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:19 smithi090 ceph-mon[33213]: 6.d scrub ok 2024-02-05T07:26:20.018 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:19 smithi090 ceph-mon[33213]: 1.9 scrub starts 2024-02-05T07:26:20.018 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:19 smithi090 ceph-mon[33213]: 1.9 scrub ok 2024-02-05T07:26:20.018 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:19 smithi090 ceph-mon[33213]: pgmap v359: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 49 MiB/s wr, 792 op/s; 25122/240675 objects degraded (10.438%) 2024-02-05T07:26:20.018 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:19 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 25122/240675 objects degraded (10.438%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:20.018 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:19 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1822087785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:20.026 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 49 16 86375 86359 110.135 80.1875 0.0110896 0.00907459 2024-02-05T07:26:20.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:20.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:20.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:20.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:20.216 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:20.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:20.317 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:20.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:20.355 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:20.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:20.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:20.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:20.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:20.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:20.615 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:26:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:26:20.630 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:20.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:20.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:20.732 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:20.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -n osd.6 --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.0.6 2024-02-05T07:26:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:20.772 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:20.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:20.821 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T07:26:20.821 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T07:26:20.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:20.909 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:20.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:20.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:20.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:20 smithi007 ceph-mon[29098]: 1.1 scrub starts 2024-02-05T07:26:20.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:20 smithi007 ceph-mon[29098]: 1.1 scrub ok 2024-02-05T07:26:20.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:20 smithi007 ceph-mon[29098]: 1.2 scrub starts 2024-02-05T07:26:20.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:20 smithi007 ceph-mon[29098]: 1.2 scrub ok 2024-02-05T07:26:20.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:20 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:20 smithi007 ceph-mon[37158]: 1.1 scrub starts 2024-02-05T07:26:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:20 smithi007 ceph-mon[37158]: 1.1 scrub ok 2024-02-05T07:26:20.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:20 smithi007 ceph-mon[37158]: 1.2 scrub starts 2024-02-05T07:26:20.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:20 smithi007 ceph-mon[37158]: 1.2 scrub ok 2024-02-05T07:26:20.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:20 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:21.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:21.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:20 smithi090 ceph-mon[33213]: 1.1 scrub starts 2024-02-05T07:26:21.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:20 smithi090 ceph-mon[33213]: 1.1 scrub ok 2024-02-05T07:26:21.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:20 smithi090 ceph-mon[33213]: 1.2 scrub starts 2024-02-05T07:26:21.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:20 smithi090 ceph-mon[33213]: 1.2 scrub ok 2024-02-05T07:26:21.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:20 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:21.027 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 50 16 87819 87803 109.736 90.25 0.0066485 0.00910784 2024-02-05T07:26:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/osd.6/config 2024-02-05T07:26:21.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:21.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:21.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:21.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:21.185 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:21.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:21.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:21.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:21.326 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:21.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:21.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:21.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T07:26:21.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:21.486 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:21.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:21.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:21.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:21.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:21.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:21.756 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:21.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:21.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:21 smithi090 ceph-mon[33213]: 8.e scrub ok 2024-02-05T07:26:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:21 smithi090 ceph-mon[33213]: 4.3 scrub starts 2024-02-05T07:26:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:21 smithi090 ceph-mon[33213]: 4.3 scrub ok 2024-02-05T07:26:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:21 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:26:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:21 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:26:21.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:21 smithi090 ceph-mon[33213]: pgmap v360: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 101 MiB/s wr, 1.61k op/s; 27205/260583 objects degraded (10.440%) 2024-02-05T07:26:21.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:21.863 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:21 smithi007 ceph-mon[29098]: 8.e scrub ok 2024-02-05T07:26:21.863 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:21 smithi007 ceph-mon[29098]: 4.3 scrub starts 2024-02-05T07:26:21.863 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:21 smithi007 ceph-mon[29098]: 4.3 scrub ok 2024-02-05T07:26:21.863 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:21 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:26:21.863 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:21 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:26:21.863 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:21 smithi007 ceph-mon[29098]: pgmap v360: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 101 MiB/s wr, 1.61k op/s; 27205/260583 objects degraded (10.440%) 2024-02-05T07:26:21.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:21 smithi007 ceph-mon[37158]: 8.e scrub ok 2024-02-05T07:26:21.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:21 smithi007 ceph-mon[37158]: 4.3 scrub starts 2024-02-05T07:26:21.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:21 smithi007 ceph-mon[37158]: 4.3 scrub ok 2024-02-05T07:26:21.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:21 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:26:21.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:21 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:26:21.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:21 smithi007 ceph-mon[37158]: pgmap v360: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 101 MiB/s wr, 1.61k op/s; 27205/260583 objects degraded (10.440%) 2024-02-05T07:26:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:21.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:21.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:22.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:22.027 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:22.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:22.028 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 51 16 89226 89210 109.309 87.9375 0.0090703 0.00914356 2024-02-05T07:26:22.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:22.168 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:22.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:22.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:22.270 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:22.301 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:22.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:22.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:22.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:22.438 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:22.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:22.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:22.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:22.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:22.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:22.709 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:22.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:22.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:22.710 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:22.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:22.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:26:22.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:22.860 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:22.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:22.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:22 smithi007 ceph-mon[29098]: 6.1a scrub starts 2024-02-05T07:26:22.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:22 smithi007 ceph-mon[29098]: 6.1a scrub ok 2024-02-05T07:26:22.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:22 smithi007 ceph-mon[29098]: 5.1d scrub starts 2024-02-05T07:26:22.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:22 smithi007 ceph-mon[29098]: 5.1d scrub ok 2024-02-05T07:26:22.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:22 smithi007 ceph-mon[29098]: osdmap e237: 8 total, 7 up, 7 in 2024-02-05T07:26:22.897 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:22 smithi007 ceph-mon[37158]: 6.1a scrub starts 2024-02-05T07:26:22.897 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:22 smithi007 ceph-mon[37158]: 6.1a scrub ok 2024-02-05T07:26:22.897 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:22 smithi007 ceph-mon[37158]: 5.1d scrub starts 2024-02-05T07:26:22.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:22 smithi007 ceph-mon[37158]: 5.1d scrub ok 2024-02-05T07:26:22.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:22 smithi007 ceph-mon[37158]: osdmap e237: 8 total, 7 up, 7 in 2024-02-05T07:26:22.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:22.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:22.986 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:22.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:23.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:22 smithi090 ceph-mon[33213]: 6.1a scrub starts 2024-02-05T07:26:23.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:22 smithi090 ceph-mon[33213]: 6.1a scrub ok 2024-02-05T07:26:23.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:22 smithi090 ceph-mon[33213]: 5.1d scrub starts 2024-02-05T07:26:23.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:22 smithi090 ceph-mon[33213]: 5.1d scrub ok 2024-02-05T07:26:23.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:22 smithi090 ceph-mon[33213]: osdmap e237: 8 total, 7 up, 7 in 2024-02-05T07:26:23.027 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 52 15 90583 90568 108.839 84.875 0.00494836 0.0091831 2024-02-05T07:26:23.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:23.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:23.145 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:23.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:23.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:23.280 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:23.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:23.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:23.382 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:23.422 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:23.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:23.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:23.561 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:23.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:23.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:23.663 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:23.694 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:23.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:23.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:23.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:23.835 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:23.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:23.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:23 smithi007 ceph-mon[29098]: 5.b scrub starts 2024-02-05T07:26:23.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:23 smithi007 ceph-mon[29098]: 5.b scrub ok 2024-02-05T07:26:23.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:23 smithi007 ceph-mon[29098]: 5.e scrub starts 2024-02-05T07:26:23.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:23 smithi007 ceph-mon[29098]: 5.e scrub ok 2024-02-05T07:26:23.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:23 smithi007 ceph-mon[29098]: pgmap v362: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 64 MiB/s wr, 1.01k op/s; 27205/260583 objects degraded (10.440%) 2024-02-05T07:26:23.925 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:26:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:26:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:26:23.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:23 smithi007 ceph-mon[37158]: 5.b scrub starts 2024-02-05T07:26:23.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:23 smithi007 ceph-mon[37158]: 5.b scrub ok 2024-02-05T07:26:23.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:23 smithi007 ceph-mon[37158]: 5.e scrub starts 2024-02-05T07:26:23.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:23 smithi007 ceph-mon[37158]: 5.e scrub ok 2024-02-05T07:26:23.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:23 smithi007 ceph-mon[37158]: pgmap v362: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 64 MiB/s wr, 1.01k op/s; 27205/260583 objects degraded (10.440%) 2024-02-05T07:26:23.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:23.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:23.977 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:23.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:24.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:23 smithi090 ceph-mon[33213]: 5.b scrub starts 2024-02-05T07:26:24.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:23 smithi090 ceph-mon[33213]: 5.b scrub ok 2024-02-05T07:26:24.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:23 smithi090 ceph-mon[33213]: 5.e scrub starts 2024-02-05T07:26:24.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:23 smithi090 ceph-mon[33213]: 5.e scrub ok 2024-02-05T07:26:24.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:23 smithi090 ceph-mon[33213]: pgmap v362: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 64 MiB/s wr, 1.01k op/s; 27205/260583 objects degraded (10.440%) 2024-02-05T07:26:24.027 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 53 15 91808 91793 108.23 76.5625 0.00606831 0.0092353 2024-02-05T07:26:24.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:24.113 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:24.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:24.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:24.243 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:24.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:24.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:24.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:24.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:24.373 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:24.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:24.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:24.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:24.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:24.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:24.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:24.635 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:24.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:24.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:24.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:24.874 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:24.912 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:24.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:24.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:25.014 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:25.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:24 smithi090 ceph-mon[33213]: 4.1c scrub starts 2024-02-05T07:26:25.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:24 smithi090 ceph-mon[33213]: 4.1c scrub ok 2024-02-05T07:26:25.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:24 smithi090 ceph-mon[33213]: 3.e scrub starts 2024-02-05T07:26:25.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:24 smithi090 ceph-mon[33213]: 3.e scrub ok 2024-02-05T07:26:25.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:24 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 27205/260583 objects degraded (10.440%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:25.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:24 smithi090 ceph-mon[33213]: osdmap e238: 8 total, 7 up, 7 in 2024-02-05T07:26:25.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:24 smithi090 ceph-mon[33213]: 4.7 scrub starts 2024-02-05T07:26:25.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:24 smithi090 ceph-mon[33213]: 4.7 scrub ok 2024-02-05T07:26:25.027 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 54 15 92972 92957 107.572 72.75 0.00821171 0.00929157 2024-02-05T07:26:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:25.057 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:25.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:25.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:25.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[37158]: 4.1c scrub starts 2024-02-05T07:26:25.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[37158]: 4.1c scrub ok 2024-02-05T07:26:25.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[37158]: 3.e scrub starts 2024-02-05T07:26:25.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[37158]: 3.e scrub ok 2024-02-05T07:26:25.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 27205/260583 objects degraded (10.440%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:25.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[37158]: osdmap e238: 8 total, 7 up, 7 in 2024-02-05T07:26:25.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[37158]: 4.7 scrub starts 2024-02-05T07:26:25.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[37158]: 4.7 scrub ok 2024-02-05T07:26:25.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[29098]: 4.1c scrub starts 2024-02-05T07:26:25.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[29098]: 4.1c scrub ok 2024-02-05T07:26:25.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[29098]: 3.e scrub starts 2024-02-05T07:26:25.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[29098]: 3.e scrub ok 2024-02-05T07:26:25.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 27205/260583 objects degraded (10.440%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:25.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[29098]: osdmap e238: 8 total, 7 up, 7 in 2024-02-05T07:26:25.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[29098]: 4.7 scrub starts 2024-02-05T07:26:25.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:24 smithi007 ceph-mon[29098]: 4.7 scrub ok 2024-02-05T07:26:25.181 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:25.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:25.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:25.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:25.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:25.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:25.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-05T07:26:25.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:25.482 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:25.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:25.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:25.615 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:25.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:25.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:25.747 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:25.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:25.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:25.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:25.887 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:25.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:25.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:25.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:26.013 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:26.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:26.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:25 smithi090 ceph-mon[33213]: pgmap v364: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 27205/260583 objects degraded (10.440%) 2024-02-05T07:26:26.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:25 smithi090 ceph-mon[33213]: 4.f scrub starts 2024-02-05T07:26:26.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:25 smithi090 ceph-mon[33213]: 4.f scrub ok 2024-02-05T07:26:26.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:25 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:26:26.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:25 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:26:26.027 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 55 16 93973 93957 106.753 62.5 0.00699041 0.00936136 2024-02-05T07:26:26.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:26.154 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:26.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:26.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:26.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:25 smithi007 ceph-mon[29098]: pgmap v364: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 27205/260583 objects degraded (10.440%) 2024-02-05T07:26:26.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:25 smithi007 ceph-mon[29098]: 4.f scrub starts 2024-02-05T07:26:26.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:25 smithi007 ceph-mon[29098]: 4.f scrub ok 2024-02-05T07:26:26.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:25 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:26:26.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:25 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:26:26.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:25 smithi007 ceph-mon[37158]: pgmap v364: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 27205/260583 objects degraded (10.440%) 2024-02-05T07:26:26.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:25 smithi007 ceph-mon[37158]: 4.f scrub starts 2024-02-05T07:26:26.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:25 smithi007 ceph-mon[37158]: 4.f scrub ok 2024-02-05T07:26:26.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:25 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:26:26.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:25 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:26:26.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:26.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:26.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:26.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:26.420 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:26.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:26.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:26.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:26.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:26.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:26.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:26.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:26.843 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:26.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:26.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:26.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:26.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:27.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:26 smithi090 ceph-mon[33213]: 6.17 scrub starts 2024-02-05T07:26:27.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:26 smithi090 ceph-mon[33213]: 6.17 scrub ok 2024-02-05T07:26:27.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:26 smithi090 ceph-mon[33213]: osdmap e239: 8 total, 7 up, 7 in 2024-02-05T07:26:27.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:26 smithi090 ceph-mon[33213]: 5.8 scrub starts 2024-02-05T07:26:27.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:26 smithi090 ceph-mon[33213]: 5.8 scrub ok 2024-02-05T07:26:27.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:26 smithi090 ceph-mon[33213]: 4.2 scrub starts 2024-02-05T07:26:27.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:26 smithi090 ceph-mon[33213]: 4.2 scrub ok 2024-02-05T07:26:27.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:26 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:26:27.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:26 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:26:27.028 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 56 16 95190 95174 106.205 76.0625 0.00921563 0.00941076 2024-02-05T07:26:27.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[29098]: 6.17 scrub starts 2024-02-05T07:26:27.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[29098]: 6.17 scrub ok 2024-02-05T07:26:27.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[29098]: osdmap e239: 8 total, 7 up, 7 in 2024-02-05T07:26:27.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[29098]: 5.8 scrub starts 2024-02-05T07:26:27.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[29098]: 5.8 scrub ok 2024-02-05T07:26:27.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[29098]: 4.2 scrub starts 2024-02-05T07:26:27.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[29098]: 4.2 scrub ok 2024-02-05T07:26:27.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:26:27.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:26:27.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[37158]: 6.17 scrub starts 2024-02-05T07:26:27.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[37158]: 6.17 scrub ok 2024-02-05T07:26:27.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[37158]: osdmap e239: 8 total, 7 up, 7 in 2024-02-05T07:26:27.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[37158]: 5.8 scrub starts 2024-02-05T07:26:27.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[37158]: 5.8 scrub ok 2024-02-05T07:26:27.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[37158]: 4.2 scrub starts 2024-02-05T07:26:27.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[37158]: 4.2 scrub ok 2024-02-05T07:26:27.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:26:27.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:26 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:26:27.084 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:27.131 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:27.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:27.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:27.183 INFO:teuthology.orchestra.run.smithi090.stdout:get_pg_num_history pg_num_history pg_num_history(e230 pg_nums {1={20=1,224=11},2={66=3},3={67=32},4={69=32},5={71=32},6={73=32},7={118=26,177=36},8={209=16}} deleted_pools 205,7) 2024-02-05T07:26:27.183 INFO:teuthology.orchestra.run.smithi090.stdout:Importing pgid 3.0 2024-02-05T07:26:27.183 INFO:teuthology.orchestra.run.smithi090.stdout:Write #3:008372bf:::zone_info.1547a86d-f9f1-4768-bef5-8d2d69f94d5a:head# 2024-02-05T07:26:27.183 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-02-05T07:26:27.184 INFO:teuthology.orchestra.run.smithi090.stdout:write_pg epoch 222 info 3.0( v 68'1 (0'0,68'1] local-lis/les=67/68 n=1 ec=67/67 lis/c=67/67 les/c/f=68/68/0 sis=67) 2024-02-05T07:26:27.186 INFO:teuthology.orchestra.run.smithi090.stdout:Import successful 2024-02-05T07:26:27.233 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:27.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:27.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:27.412 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:27.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:27.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:27.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:27.552 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:27.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:27.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:27.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:27.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:27.750 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:27 smithi007 ceph-mon[29098]: osdmap e240: 8 total, 7 up, 7 in 2024-02-05T07:26:27.750 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:27 smithi007 ceph-mon[29098]: 6.12 scrub starts 2024-02-05T07:26:27.750 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:27 smithi007 ceph-mon[29098]: 6.12 scrub ok 2024-02-05T07:26:27.750 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:27 smithi007 ceph-mon[29098]: pgmap v367: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 940 B/s rd, 75 MiB/s wr, 1.19k op/s; 29250/280074 objects degraded (10.444%) 2024-02-05T07:26:27.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:27 smithi007 ceph-mon[37158]: osdmap e240: 8 total, 7 up, 7 in 2024-02-05T07:26:27.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:27 smithi007 ceph-mon[37158]: 6.12 scrub starts 2024-02-05T07:26:27.751 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:27 smithi007 ceph-mon[37158]: 6.12 scrub ok 2024-02-05T07:26:27.751 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:27 smithi007 ceph-mon[37158]: pgmap v367: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 940 B/s rd, 75 MiB/s wr, 1.19k op/s; 29250/280074 objects degraded (10.444%) 2024-02-05T07:26:27.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:27.838 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:27.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:27.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:27.940 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:27.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:27.987 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:27.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:27 smithi090 ceph-mon[33213]: osdmap e240: 8 total, 7 up, 7 in 2024-02-05T07:26:28.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:27 smithi090 ceph-mon[33213]: 6.12 scrub starts 2024-02-05T07:26:28.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:27 smithi090 ceph-mon[33213]: 6.12 scrub ok 2024-02-05T07:26:28.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:27 smithi090 ceph-mon[33213]: pgmap v367: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 940 B/s rd, 75 MiB/s wr, 1.19k op/s; 29250/280074 objects degraded (10.444%) 2024-02-05T07:26:28.028 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 57 16 96329 96313 105.59 71.1875 0.00540063 0.00946581 2024-02-05T07:26:28.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:28.129 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:28.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:28.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:28.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:28.256 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:28.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:28.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:28.388 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:28.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:28.431 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:28.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:28.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:28.566 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:28.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:28.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:28.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:28.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:28.704 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:28.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:28.767 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:26:28.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:28.841 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:28.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:28.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:28.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:28 smithi007 ceph-mon[29098]: 4.16 scrub starts 2024-02-05T07:26:28.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:28 smithi007 ceph-mon[29098]: 4.16 scrub ok 2024-02-05T07:26:28.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:28 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2935844965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:28.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:28 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:28.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:28 smithi007 ceph-mon[29098]: 5.19 scrub starts 2024-02-05T07:26:28.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:28 smithi007 ceph-mon[29098]: 5.19 scrub ok 2024-02-05T07:26:28.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:28 smithi007 ceph-mon[37158]: 4.16 scrub starts 2024-02-05T07:26:28.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:28 smithi007 ceph-mon[37158]: 4.16 scrub ok 2024-02-05T07:26:28.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:28 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2935844965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:28.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:28 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:28.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:28 smithi007 ceph-mon[37158]: 5.19 scrub starts 2024-02-05T07:26:28.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:28 smithi007 ceph-mon[37158]: 5.19 scrub ok 2024-02-05T07:26:28.924 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:28 smithi090 ceph-mon[33213]: 4.16 scrub starts 2024-02-05T07:26:28.924 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:28 smithi090 ceph-mon[33213]: 4.16 scrub ok 2024-02-05T07:26:28.924 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:28 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2935844965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:28.924 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:28 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:28.924 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:28 smithi090 ceph-mon[33213]: 5.19 scrub starts 2024-02-05T07:26:28.924 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:28 smithi090 ceph-mon[33213]: 5.19 scrub ok 2024-02-05T07:26:28.943 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:28.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/exp.3.0.6 2024-02-05T07:26:28.985 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:28.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:28.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:29.028 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 58 16 97425 97409 104.951 68.5 0.00583293 0.00952359 2024-02-05T07:26:29.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:29.131 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:29.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:29.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:29.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:29.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:29.279 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:29.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:29.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:29.415 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:29.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:29.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:29.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:29.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:29.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:29.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:29.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:29.695 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:29.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:29.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:29.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:29.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:29.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:29.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:29.980 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:29.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:30.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:29 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:26:30.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:29 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:26:30.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:29 smithi090 ceph-mon[33213]: osdmap e241: 8 total, 7 up, 7 in 2024-02-05T07:26:30.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:29 smithi090 ceph-mon[33213]: pgmap v369: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 29250/280074 objects degraded (10.444%) 2024-02-05T07:26:30.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:29 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 29250/280074 objects degraded (10.444%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:30.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:29 smithi090 ceph-mon[33213]: osdmap e242: 8 total, 7 up, 7 in 2024-02-05T07:26:30.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:29 smithi090 ceph-mon[33213]: 3.1a scrub starts 2024-02-05T07:26:30.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:29 smithi090 ceph-mon[33213]: 3.1a scrub ok 2024-02-05T07:26:30.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:29 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:26:30.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:26:30.028 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 59 16 98965 98949 104.803 96.25 0.00472217 0.00953724 2024-02-05T07:26:30.047 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:26:30.047 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:26:30.048 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[29098]: osdmap e241: 8 total, 7 up, 7 in 2024-02-05T07:26:30.048 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[29098]: pgmap v369: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 29250/280074 objects degraded (10.444%) 2024-02-05T07:26:30.048 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 29250/280074 objects degraded (10.444%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:30.048 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[29098]: osdmap e242: 8 total, 7 up, 7 in 2024-02-05T07:26:30.048 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[29098]: 3.1a scrub starts 2024-02-05T07:26:30.048 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[29098]: 3.1a scrub ok 2024-02-05T07:26:30.048 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:26:30.048 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:26:30.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:26:30.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:26:30.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[37158]: osdmap e241: 8 total, 7 up, 7 in 2024-02-05T07:26:30.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[37158]: pgmap v369: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 29250/280074 objects degraded (10.444%) 2024-02-05T07:26:30.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 29250/280074 objects degraded (10.444%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:30.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[37158]: osdmap e242: 8 total, 7 up, 7 in 2024-02-05T07:26:30.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[37158]: 3.1a scrub starts 2024-02-05T07:26:30.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[37158]: 3.1a scrub ok 2024-02-05T07:26:30.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:26:30.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:26:30.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:30.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:30.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:30.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:30.228 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:30.267 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:30.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:30.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:30.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:30.406 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:30.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:30.506 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:30.550 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:30.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:30.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:30.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:30.691 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:30.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:30.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:30.745 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:26:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:26:30.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:30.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:30.827 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:30.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:30.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:30.968 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:30.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:30.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:31.005 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:26:31.005 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:26:31.007 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:30 smithi007 ceph-mon[29098]: 6.9 scrub starts 2024-02-05T07:26:31.007 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:30 smithi007 ceph-mon[29098]: 6.9 scrub ok 2024-02-05T07:26:31.007 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:30 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:31.007 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:30 smithi007 ceph-mon[37158]: 6.9 scrub starts 2024-02-05T07:26:31.007 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:30 smithi007 ceph-mon[37158]: 6.9 scrub ok 2024-02-05T07:26:31.007 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:30 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:31.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:30 smithi090 ceph-mon[33213]: 6.9 scrub starts 2024-02-05T07:26:31.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:30 smithi090 ceph-mon[33213]: 6.9 scrub ok 2024-02-05T07:26:31.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:30 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:31.028 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:26:31.028086+0000 min lat: 0.00128428 max lat: 0.975632 avg lat: 0.00955443 2024-02-05T07:26:31.029 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:26:31.029 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 60 16 100465 100449 104.618 93.75 0.00495717 0.00955443 2024-02-05T07:26:31.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:31.107 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:31.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:31.208 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:31.244 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:31.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:31.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:31.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:31.383 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:31.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:31.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:31.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:31.522 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:31.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:31.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:31.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:31.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T07:26:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:31.672 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:31.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:31.774 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:31.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:26:31.819 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:31.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:31.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:31 smithi007 ceph-mon[29098]: 3.1 scrub starts 2024-02-05T07:26:31.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:31 smithi007 ceph-mon[29098]: 3.1 scrub ok 2024-02-05T07:26:31.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:31 smithi007 ceph-mon[29098]: pgmap v371: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 951 B/s rd, 71 MiB/s wr, 1.14k op/s; 31142/298518 objects degraded (10.432%) 2024-02-05T07:26:31.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:31 smithi007 ceph-mon[29098]: 5.13 deep-scrub starts 2024-02-05T07:26:31.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:31 smithi007 ceph-mon[29098]: 5.13 deep-scrub ok 2024-02-05T07:26:31.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:31 smithi007 ceph-mon[37158]: 3.1 scrub starts 2024-02-05T07:26:31.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:31 smithi007 ceph-mon[37158]: 3.1 scrub ok 2024-02-05T07:26:31.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:31 smithi007 ceph-mon[37158]: pgmap v371: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 951 B/s rd, 71 MiB/s wr, 1.14k op/s; 31142/298518 objects degraded (10.432%) 2024-02-05T07:26:31.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:31 smithi007 ceph-mon[37158]: 5.13 deep-scrub starts 2024-02-05T07:26:31.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:31 smithi007 ceph-mon[37158]: 5.13 deep-scrub ok 2024-02-05T07:26:31.921 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:31.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:31.952 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:31.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:32.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:31 smithi090 ceph-mon[33213]: 3.1 scrub starts 2024-02-05T07:26:32.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:31 smithi090 ceph-mon[33213]: 3.1 scrub ok 2024-02-05T07:26:32.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:31 smithi090 ceph-mon[33213]: pgmap v371: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 951 B/s rd, 71 MiB/s wr, 1.14k op/s; 31142/298518 objects degraded (10.432%) 2024-02-05T07:26:32.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:31 smithi090 ceph-mon[33213]: 5.13 deep-scrub starts 2024-02-05T07:26:32.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:31 smithi090 ceph-mon[33213]: 5.13 deep-scrub ok 2024-02-05T07:26:32.029 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 61 16 101931 101915 104.405 91.625 0.00343592 0.0095735 2024-02-05T07:26:32.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:32.111 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:32.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:32.212 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:32.239 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:32.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:32.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:32.287 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:26:32.302 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:32.312 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:32.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:32.345 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:26:32.386 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:26:32.386 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:32.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:32.426 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:26:32.467 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:26:32.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:32.507 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:26:32.521 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:32.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:32.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:32.551 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:26:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:26:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:26:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:26:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:26:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:26:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:26:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-02-05T07:26:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:26:32.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:32.657 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:32.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:32.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:32.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:32.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:32.800 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:32.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:32.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:32.937 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:32.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:32.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:32 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3032050982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:33.029 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 62 16 103331 103315 104.132 87.5 0.0104589 0.00959891 2024-02-05T07:26:33.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:33.073 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:33.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:33.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:33.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:32 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3032050982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:33.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:32 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3032050982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:33.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:33.216 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:33.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:33.318 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:33.359 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T07:26:33.360 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T07:26:33.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:33.366 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:33.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:33.467 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:33.498 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:33.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:33.565 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:26:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:26:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:26:33.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:33.634 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:33.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:33.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:33.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:33.770 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:33.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:33.872 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:33.920 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:33.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:33.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:33 smithi007 ceph-mon[29098]: 5.7 scrub starts 2024-02-05T07:26:33.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:33 smithi007 ceph-mon[29098]: 5.7 scrub ok 2024-02-05T07:26:33.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:33 smithi007 ceph-mon[29098]: pgmap v372: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 803 B/s rd, 60 MiB/s wr, 965 op/s; 31142/298518 objects degraded (10.432%) 2024-02-05T07:26:33.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:33 smithi007 ceph-mon[29098]: 3.7 scrub starts 2024-02-05T07:26:33.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:33 smithi007 ceph-mon[29098]: 3.7 scrub ok 2024-02-05T07:26:33.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:33 smithi007 ceph-mon[37158]: 5.7 scrub starts 2024-02-05T07:26:33.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:33 smithi007 ceph-mon[37158]: 5.7 scrub ok 2024-02-05T07:26:33.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:33 smithi007 ceph-mon[37158]: pgmap v372: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 803 B/s rd, 60 MiB/s wr, 965 op/s; 31142/298518 objects degraded (10.432%) 2024-02-05T07:26:33.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:33 smithi007 ceph-mon[37158]: 3.7 scrub starts 2024-02-05T07:26:33.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:33 smithi007 ceph-mon[37158]: 3.7 scrub ok 2024-02-05T07:26:34.006 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-02-05T07:26:34.006 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:26:34.007 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-02-05T07:26:34.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:26:34.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:34.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:33 smithi090 ceph-mon[33213]: 5.7 scrub starts 2024-02-05T07:26:34.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:33 smithi090 ceph-mon[33213]: 5.7 scrub ok 2024-02-05T07:26:34.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:33 smithi090 ceph-mon[33213]: pgmap v372: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 803 B/s rd, 60 MiB/s wr, 965 op/s; 31142/298518 objects degraded (10.432%) 2024-02-05T07:26:34.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:33 smithi090 ceph-mon[33213]: 3.7 scrub starts 2024-02-05T07:26:34.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:33 smithi090 ceph-mon[33213]: 3.7 scrub ok 2024-02-05T07:26:34.030 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 63 16 104887 104871 104.023 97.25 0.0176836 0.00960833 2024-02-05T07:26:34.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:34.057 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:34.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:34.159 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:34.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:34.214 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:34.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:34.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:34.360 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:34.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:34.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:34.496 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:34.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:34.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:34.632 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:34.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:34.734 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:34.770 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:34.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:34.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:34.872 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:34.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:34.909 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:34.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:35.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:35.029 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 64 16 106371 106355 103.846 92.75 0.00926628 0.00962526 2024-02-05T07:26:35.042 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:35.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:35.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:35.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:35.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:34 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 31142/298518 objects degraded (10.432%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:35.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:34 smithi007 ceph-mon[29098]: 5.0 scrub starts 2024-02-05T07:26:35.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:34 smithi007 ceph-mon[29098]: 5.0 scrub ok 2024-02-05T07:26:35.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:34 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 31142/298518 objects degraded (10.432%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:35.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:34 smithi007 ceph-mon[37158]: 5.0 scrub starts 2024-02-05T07:26:35.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:34 smithi007 ceph-mon[37158]: 5.0 scrub ok 2024-02-05T07:26:35.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:35.185 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:35.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:34 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 31142/298518 objects degraded (10.432%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:35.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:34 smithi090 ceph-mon[33213]: 5.0 scrub starts 2024-02-05T07:26:35.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:34 smithi090 ceph-mon[33213]: 5.0 scrub ok 2024-02-05T07:26:35.286 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:35.319 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:35.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:35.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:35.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:26:35.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:35.452 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:35.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:35.554 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:35.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:35.588 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:35.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:35.655 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:35.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:35.731 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:35.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:35.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:35.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:35.871 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:35.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:35.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:35.973 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:35.977 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:35 smithi007 ceph-mon[29098]: pgmap v373: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 768 op/s; 31142/298518 objects degraded (10.432%) 2024-02-05T07:26:35.978 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:35 smithi007 ceph-mon[29098]: 5.14 scrub starts 2024-02-05T07:26:35.978 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:35 smithi007 ceph-mon[29098]: 5.14 scrub ok 2024-02-05T07:26:35.978 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:35 smithi007 ceph-mon[29098]: 6.11 deep-scrub starts 2024-02-05T07:26:35.978 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:35 smithi007 ceph-mon[29098]: 6.11 deep-scrub ok 2024-02-05T07:26:35.978 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:35 smithi007 ceph-mon[37158]: pgmap v373: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 768 op/s; 31142/298518 objects degraded (10.432%) 2024-02-05T07:26:35.978 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:35 smithi007 ceph-mon[37158]: 5.14 scrub starts 2024-02-05T07:26:35.978 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:35 smithi007 ceph-mon[37158]: 5.14 scrub ok 2024-02-05T07:26:35.978 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:35 smithi007 ceph-mon[37158]: 6.11 deep-scrub starts 2024-02-05T07:26:35.979 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:35 smithi007 ceph-mon[37158]: 6.11 deep-scrub ok 2024-02-05T07:26:36.012 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:36.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:36.029 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 65 16 107803 107787 103.625 89.5 0.00983063 0.00964528 2024-02-05T07:26:36.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:36.148 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:36.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:36.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:36.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:35 smithi090 ceph-mon[33213]: pgmap v373: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 768 op/s; 31142/298518 objects degraded (10.432%) 2024-02-05T07:26:36.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:35 smithi090 ceph-mon[33213]: 5.14 scrub starts 2024-02-05T07:26:36.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:35 smithi090 ceph-mon[33213]: 5.14 scrub ok 2024-02-05T07:26:36.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:35 smithi090 ceph-mon[33213]: 6.11 deep-scrub starts 2024-02-05T07:26:36.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:35 smithi090 ceph-mon[33213]: 6.11 deep-scrub ok 2024-02-05T07:26:36.291 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:36.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:36.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:36.423 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:36.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:36.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:36.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:36.573 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:36.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:36.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:36.715 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:36.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:36.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:36.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:36.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:36.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:36.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:36.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:36.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:36.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:37.029 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 66 16 109249 109233 103.425 90.375 0.0127613 0.00966422 2024-02-05T07:26:37.084 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:36 smithi090 ceph-mon[33213]: 4.1b scrub starts 2024-02-05T07:26:37.084 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:36 smithi090 ceph-mon[33213]: 4.1b scrub ok 2024-02-05T07:26:37.084 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:36 smithi090 ceph-mon[33213]: 4.17 scrub starts 2024-02-05T07:26:37.084 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:36 smithi090 ceph-mon[33213]: 4.17 scrub ok 2024-02-05T07:26:37.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:37.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:36 smithi007 ceph-mon[29098]: 4.1b scrub starts 2024-02-05T07:26:37.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:36 smithi007 ceph-mon[29098]: 4.1b scrub ok 2024-02-05T07:26:37.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:36 smithi007 ceph-mon[29098]: 4.17 scrub starts 2024-02-05T07:26:37.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:36 smithi007 ceph-mon[29098]: 4.17 scrub ok 2024-02-05T07:26:37.096 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:36 smithi007 ceph-mon[37158]: 4.1b scrub starts 2024-02-05T07:26:37.096 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:36 smithi007 ceph-mon[37158]: 4.1b scrub ok 2024-02-05T07:26:37.096 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:36 smithi007 ceph-mon[37158]: 4.17 scrub starts 2024-02-05T07:26:37.096 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:36 smithi007 ceph-mon[37158]: 4.17 scrub ok 2024-02-05T07:26:37.145 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:37.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:37.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:37.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:37.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:37.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:37.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:37.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:37.424 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:37.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:37.526 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:37.564 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:37.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:37.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:37.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:37.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:37.801 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:37.819 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:26:37.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:37.845 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:37.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:37.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[29098]: 4.d scrub starts 2024-02-05T07:26:37.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[29098]: 4.d scrub ok 2024-02-05T07:26:37.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[29098]: 6.7 scrub starts 2024-02-05T07:26:37.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[29098]: 6.7 scrub ok 2024-02-05T07:26:37.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2818136249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:26:37.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:26:37.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[29098]: 3.f scrub starts 2024-02-05T07:26:37.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[29098]: 3.f scrub ok 2024-02-05T07:26:37.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[29098]: pgmap v374: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 33469/320847 objects degraded (10.431%) 2024-02-05T07:26:37.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[29098]: 6.10 scrub starts 2024-02-05T07:26:37.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[29098]: 6.10 scrub ok 2024-02-05T07:26:37.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[37158]: 4.d scrub starts 2024-02-05T07:26:37.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[37158]: 4.d scrub ok 2024-02-05T07:26:37.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[37158]: 6.7 scrub starts 2024-02-05T07:26:37.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[37158]: 6.7 scrub ok 2024-02-05T07:26:37.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2818136249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:26:37.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:26:37.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[37158]: 3.f scrub starts 2024-02-05T07:26:37.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[37158]: 3.f scrub ok 2024-02-05T07:26:37.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[37158]: pgmap v374: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 33469/320847 objects degraded (10.431%) 2024-02-05T07:26:37.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[37158]: 6.10 scrub starts 2024-02-05T07:26:37.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:37 smithi007 ceph-mon[37158]: 6.10 scrub ok 2024-02-05T07:26:37.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:37.982 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:37.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:37.993 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:26:37.993 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-02-05T07:26:37.993 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-02-05T07:26:37.993 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:26:37.993 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.993 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:74431 7.13 7:ce63305d:::benchmark_data_smithi007_77086_object74430:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e118)", 2024-02-05T07:26:37.993 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:41.581545+0000", 2024-02-05T07:26:37.993 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 296.406453849, 2024-02-05T07:26:37.993 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0557121, 2024-02-05T07:26:37.993 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:37.993 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 74431 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.581545+0000", 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.581545+0000", 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:37.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.995 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:37.995 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.581547+0000", 2024-02-05T07:26:37.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.984e-06 2024-02-05T07:26:37.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.995 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:37.995 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.581573+0000", 2024-02-05T07:26:37.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5371000000000002e-05 2024-02-05T07:26:37.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.995 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:37.995 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.581587+0000", 2024-02-05T07:26:37.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3935e-05 2024-02-05T07:26:37.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.996 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:37.996 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.581597+0000", 2024-02-05T07:26:37.996 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9879999999999999e-06 2024-02-05T07:26:37.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.996 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:37.996 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.801714+0000", 2024-02-05T07:26:37.996 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.22011765699999999 2024-02-05T07:26:37.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.996 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:37.996 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.801724+0000", 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1479999999999995e-06 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:42.633098+0000", 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.83137443600000005 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:42.633493+0000", 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00039521199999999998 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:37.997 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:42.637164+0000", 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0036706849999999999 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:42.637258+0000", 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3684000000000006e-05 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:74434 7.12 7:4fc5a4a1:::benchmark_data_smithi007_77086_object74433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e118)", 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:41.581616+0000", 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 296.40638310200001, 2024-02-05T07:26:37.998 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.072719744, 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 74434 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.581616+0000", 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:37.999 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.581616+0000", 2024-02-05T07:26:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.581617+0000", 2024-02-05T07:26:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8999999999999997e-07 2024-02-05T07:26:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.581637+0000", 2024-02-05T07:26:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0721999999999999e-05 2024-02-05T07:26:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.581644+0000", 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9079999999999998e-06 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.581650+0000", 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3879999999999999e-06 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.791282+0000", 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.209632183 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:41.791372+0000", 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9976999999999996e-05 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:42.640695+0000", 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.84932345600000003 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:42.640818+0000", 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012269099999999999 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:42.641108+0000", 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029032299999999999 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:42.647113+0000", 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0060047859999999998 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:42.649570+0000", 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024570379999999999 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.003 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:42.654244+0000", 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.004673417 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:42.654295+0000", 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1025e-05 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:42.654336+0000", 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1340000000000001e-05 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.004 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75610 7.14 7.bfe121f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:43.610331+0000", 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 294.37766812400002, 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.53071111400000004, 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75610 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.007 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.007 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.610331+0000", 2024-02-05T07:26:38.007 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.007 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.007 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.007 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.610331+0000", 2024-02-05T07:26:38.007 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.007 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.007 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.610333+0000", 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5939999999999999e-06 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.610357+0000", 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4553999999999999e-05 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.610369+0000", 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1542999999999999e-05 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.008 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.009 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.610381+0000", 2024-02-05T07:26:38.009 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2086000000000001e-05 2024-02-05T07:26:38.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.009 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.009 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.618849+0000", 2024-02-05T07:26:38.009 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0084680759999999997 2024-02-05T07:26:38.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.009 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.618856+0000", 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7120000000000003e-06 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.141033+0000", 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.52217726399999997 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.141042+0000", 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.285e-06 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.010 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75612 7.10 7:08aa0df0:::benchmark_data_smithi007_77086_object75611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:43.612973+0000", 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 294.37502652299997, 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8790344320000001, 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75612 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.012 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.012 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.612973+0000", 2024-02-05T07:26:38.012 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.012 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.012 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.012 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.612973+0000", 2024-02-05T07:26:38.012 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.012 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.012 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.012 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.612974+0000", 2024-02-05T07:26:38.012 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3400000000000001e-06 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.612995+0000", 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1154000000000001e-05 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.613007+0000", 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1647e-05 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.613016+0000", 2024-02-05T07:26:38.013 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6039999999999993e-06 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.613078+0000", 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2063999999999994e-05 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.613084+0000", 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.922e-06 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.617430+0000", 2024-02-05T07:26:38.014 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043459809999999996 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.617442+0000", 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1806e-05 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.140594+0000", 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.52315191500000002 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.140607+0000", 2024-02-05T07:26:38.015 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2422000000000001e-05 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.467165+0000", 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3265580729999999 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.467325+0000", 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016011399999999999 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.467659+0000", 2024-02-05T07:26:38.016 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000334328 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.475070+0000", 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0074112980000000002 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.476322+0000", 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012519530000000001 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.489799+0000", 2024-02-05T07:26:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013476676999999999 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.490051+0000", 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00025161699999999999 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.492007+0000", 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0019565170000000001 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75625 7.10 7:0903b300:::benchmark_data_smithi007_77086_object75624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:43.630274+0000", 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 294.35772526900001, 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8716550380000001, 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75625 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.019 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.630274+0000", 2024-02-05T07:26:38.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.020 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.020 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.630274+0000", 2024-02-05T07:26:38.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.020 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.020 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.630276+0000", 2024-02-05T07:26:38.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.782e-06 2024-02-05T07:26:38.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.020 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.021 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.630305+0000", 2024-02-05T07:26:38.021 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9496999999999999e-05 2024-02-05T07:26:38.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.021 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.021 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.630315+0000", 2024-02-05T07:26:38.021 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9329999999999996e-06 2024-02-05T07:26:38.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.021 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.021 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.630323+0000", 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0539999999999995e-06 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.630452+0000", 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012835099999999999 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.630463+0000", 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1131000000000001e-05 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.022 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.140633+0000", 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.51016999799999996 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.140642+0000", 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6700000000000006e-06 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.470694+0000", 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.330051697 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.023 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.470868+0000", 2024-02-05T07:26:38.024 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017340500000000001 2024-02-05T07:26:38.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.024 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.024 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.024 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.471213+0000", 2024-02-05T07:26:38.024 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00034575600000000001 2024-02-05T07:26:38.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.024 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.024 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.024 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.475078+0000", 2024-02-05T07:26:38.024 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038652080000000002 2024-02-05T07:26:38.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.024 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.024 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.476365+0000", 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012860129999999999 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.500301+0000", 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.023936483000000001 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.500362+0000", 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0992999999999997e-05 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.501929+0000", 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0015670669999999999 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75634 7.10 7:0a57221a:::benchmark_data_smithi007_77086_object75633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:43.654072+0000", 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 294.33392704900001, 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8495972410000001, 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.027 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.027 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75634 2024-02-05T07:26:38.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.027 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.027 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.027 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.654072+0000", 2024-02-05T07:26:38.027 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.027 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.654072+0000", 2024-02-05T07:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.654074+0000", 2024-02-05T07:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.663e-06 2024-02-05T07:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.654099+0000", 2024-02-05T07:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5182999999999999e-05 2024-02-05T07:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.654112+0000", 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2523e-05 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.654121+0000", 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1609999999999992e-06 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.654192+0000", 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1647e-05 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.029 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:43.654204+0000", 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1610000000000001e-05 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.140667+0000", 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.48646297999999999 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.140675+0000", 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.841e-06 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.472533+0000", 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.331857751 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.472690+0000", 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000157085 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.473038+0000", 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00034878600000000002 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.475085+0000", 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020463980000000001 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.476402+0000", 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001317592 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.502026+0000", 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.025623566 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.033 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.033 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.502078+0000", 2024-02-05T07:26:38.033 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2438999999999997e-05 2024-02-05T07:26:38.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.033 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.033 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.503669+0000", 2024-02-05T07:26:38.033 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001591016 2024-02-05T07:26:38.034 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.034 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.034 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.034 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75626 7.11 7:89200899:::benchmark_data_smithi007_77086_object75625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T07:26:38.034 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:44.135751+0000", 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 293.85224817099999, 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1397510310000001, 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75626 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135751+0000", 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135751+0000", 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135753+0000", 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6449999999999999e-06 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135781+0000", 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7951000000000001e-05 2024-02-05T07:26:38.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135793+0000", 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2112e-05 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135802+0000", 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5580000000000006e-06 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.137926+0000", 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021234719999999999 2024-02-05T07:26:38.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.137937+0000", 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0869e-05 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.265855+0000", 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1279184579999999 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.266035+0000", 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000180135 2024-02-05T07:26:38.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.039 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.039 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6,7", 2024-02-05T07:26:38.039 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.266359+0000", 2024-02-05T07:26:38.039 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032418200000000001 2024-02-05T07:26:38.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.039 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.039 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.039 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.267782+0000", 2024-02-05T07:26:38.039 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0014222239999999999 2024-02-05T07:26:38.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.039 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.039 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.039 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.268639+0000", 2024-02-05T07:26:38.039 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00085747999999999998 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.272603+0000", 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039641090000000004 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.272722+0000", 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000118508 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.275502+0000", 2024-02-05T07:26:38.040 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0027803279999999999 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:118812 7.10 7:0d273f1a:::benchmark_data_smithi007_77086_object14217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:22:46.496890+0000", 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 231.49110954400001, 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.032906788, 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 118812 2024-02-05T07:26:38.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.042 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.042 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.042 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.496890+0000", 2024-02-05T07:26:38.042 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.042 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.042 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.496890+0000", 2024-02-05T07:26:38.042 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.043 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.043 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.496891+0000", 2024-02-05T07:26:38.043 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.33e-06 2024-02-05T07:26:38.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.496896+0000", 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6949999999999999e-06 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.496912+0000", 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.594e-05 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.496920+0000", 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8150000000000006e-06 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.496976+0000", 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6889999999999999e-05 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.497107+0000", 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000130708 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.497471+0000", 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00036432800000000003 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.502490+0000", 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0050189299999999996 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.502508+0000", 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7535000000000001e-05 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.523562+0000", 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.021053984000000001 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.523574+0000", 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1646e-05 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.529719+0000", 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.006145807 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.529797+0000", 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7180000000000003e-05 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.047 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.048 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.048 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:118816 7.10 7:0d5449d0:::benchmark_data_smithi007_77086_object14221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-02-05T07:26:38.048 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:22:46.507725+0000", 2024-02-05T07:26:38.048 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 231.48027386999999, 2024-02-05T07:26:38.048 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0350707020000001, 2024-02-05T07:26:38.048 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.048 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:26:38.048 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.048 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.048 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.048 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 118816 2024-02-05T07:26:38.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.048 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.049 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.049 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.049 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.507725+0000", 2024-02-05T07:26:38.049 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.049 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.050 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.050 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.507725+0000", 2024-02-05T07:26:38.050 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.050 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.050 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.507727+0000", 2024-02-05T07:26:38.050 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.767e-06 2024-02-05T07:26:38.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.050 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.050 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.507731+0000", 2024-02-05T07:26:38.050 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2400000000000001e-06 2024-02-05T07:26:38.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.507742+0000", 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0257e-05 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.507749+0000", 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.311e-06 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.508733+0000", 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00098419499999999999 2024-02-05T07:26:38.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.508878+0000", 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014485100000000001 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.509273+0000", 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00039517799999999999 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.527384+0000", 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.018110292 2024-02-05T07:26:38.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.527392+0000", 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9409999999999996e-06 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.534665+0000", 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.007272395 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for missing object", 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.534750+0000", 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5487000000000003e-05 2024-02-05T07:26:38.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.542717+0000", 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0079664220000000008 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.542796+0000", 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9365999999999999e-05 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.054 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:118837 7.10 7:0901777a:::benchmark_data_smithi007_77086_object14242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:22:46.550824+0000", 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 231.43717532100001, 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99218854700000003, 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 118837 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.550824+0000", 2024-02-05T07:26:38.055 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.550824+0000", 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.550826+0000", 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.81e-06 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.550833+0000", 2024-02-05T07:26:38.056 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.153e-06 2024-02-05T07:26:38.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.057 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.057 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.550855+0000", 2024-02-05T07:26:38.057 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2464e-05 2024-02-05T07:26:38.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.057 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.057 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.550866+0000", 2024-02-05T07:26:38.057 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0679999999999999e-05 2024-02-05T07:26:38.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.550956+0000", 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9815000000000007e-05 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.550970+0000", 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3903999999999999e-05 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.534842+0000", 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.983871779 2024-02-05T07:26:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.534977+0000", 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000135797 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.535334+0000", 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00035629500000000002 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.542426+0000", 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.007092292 2024-02-05T07:26:38.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.542439+0000", 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3380999999999999e-05 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.542867+0000", 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00042802200000000002 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.542921+0000", 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3742000000000001e-05 2024-02-05T07:26:38.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.061 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.061 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.543013+0000", 2024-02-05T07:26:38.061 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1413e-05 2024-02-05T07:26:38.061 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.061 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.061 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.061 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:118888 7.10 7:09f0d148:::benchmark_data_smithi007_77086_object14293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T07:26:38.062 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 67 16 110692 110676 103.227 90.1875 0.00478737 0.00968258 2024-02-05T07:26:38.062 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:22:46.611838+0000", 2024-02-05T07:26:38.062 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 231.37616157100001, 2024-02-05T07:26:38.062 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.93752835099999998, 2024-02-05T07:26:38.062 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.062 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.062 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.062 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.062 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 118888 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.611838+0000", 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.611838+0000", 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.063 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.064 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.611839+0000", 2024-02-05T07:26:38.064 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0550000000000001e-06 2024-02-05T07:26:38.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.064 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.064 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.611842+0000", 2024-02-05T07:26:38.064 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.05e-06 2024-02-05T07:26:38.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.064 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.064 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.611848+0000", 2024-02-05T07:26:38.064 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0270000000000001e-06 2024-02-05T07:26:38.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.611852+0000", 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0450000000000002e-06 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.611921+0000", 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9567999999999995e-05 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.611934+0000", 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2595999999999999e-05 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.065 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.066 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.537168+0000", 2024-02-05T07:26:38.066 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.925233949 2024-02-05T07:26:38.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.066 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.066 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.537307+0000", 2024-02-05T07:26:38.066 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013942800000000001 2024-02-05T07:26:38.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.066 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.066 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.537617+0000", 2024-02-05T07:26:38.066 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030918900000000002 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.542445+0000", 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0048282400000000001 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.542831+0000", 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000386072 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.548893+0000", 2024-02-05T07:26:38.067 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0060624160000000002 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.548957+0000", 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4012999999999999e-05 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.549366+0000", 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00040870300000000001 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:118889 7.10 7:094fea3b:::benchmark_data_smithi007_77086_object14294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:22:46.614427+0000", 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 231.37357272099999, 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.93597349500000004, 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 118889 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.614427+0000", 2024-02-05T07:26:38.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.614427+0000", 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.614428+0000", 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2699999999999999e-06 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.614432+0000", 2024-02-05T07:26:38.070 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4800000000000003e-06 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.614441+0000", 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1209999999999996e-06 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.614447+0000", 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7899999999999996e-06 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.614504+0000", 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6852999999999999e-05 2024-02-05T07:26:38.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.614515+0000", 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1382999999999999e-05 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.539847+0000", 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.92533200699999996 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.539974+0000", 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012681899999999999 2024-02-05T07:26:38.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.540287+0000", 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00031303800000000002 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.548850+0000", 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0085625990000000006 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.549419+0000", 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00056893999999999998 2024-02-05T07:26:38.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.550252+0000", 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00083277600000000002 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.550311+0000", 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9114999999999999e-05 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.550400+0000", 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9303999999999997e-05 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.074 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:118910 7.10 7:0c825d1c:::benchmark_data_smithi007_77086_object14315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:22:46.636458+0000", 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 231.35154150599999, 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.914142907, 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 118910 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.075 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.636458+0000", 2024-02-05T07:26:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.636458+0000", 2024-02-05T07:26:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.076 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.636460+0000", 2024-02-05T07:26:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7940000000000001e-06 2024-02-05T07:26:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.636465+0000", 2024-02-05T07:26:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6400000000000002e-06 2024-02-05T07:26:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.636477+0000", 2024-02-05T07:26:38.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1892000000000001e-05 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.636486+0000", 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6889999999999993e-06 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.636557+0000", 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1608999999999998e-05 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.636568+0000", 2024-02-05T07:26:38.078 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0908e-05 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.543143+0000", 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.90657486300000001 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.543269+0000", 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000125504 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.543607+0000", 2024-02-05T07:26:38.079 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00033851699999999998 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.549379+0000", 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0057722269999999996 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.550437+0000", 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001057559 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.550467+0000", 2024-02-05T07:26:38.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9601e-05 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.550509+0000", 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.206e-05 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.550601+0000", 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2044000000000002e-05 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.081 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:118923 7.10 7:0eb55389:::benchmark_data_smithi007_77086_object14328:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:22:46.649041+0000", 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 231.338958676, 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.90590387999999999, 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 118923 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.649041+0000", 2024-02-05T07:26:38.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.649041+0000", 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.649042+0000", 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8100000000000001e-07 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.649045+0000", 2024-02-05T07:26:38.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8500000000000004e-06 2024-02-05T07:26:38.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.084 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.084 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.649052+0000", 2024-02-05T07:26:38.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9979999999999997e-06 2024-02-05T07:26:38.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.084 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.084 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.649058+0000", 2024-02-05T07:26:38.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0760000000000002e-06 2024-02-05T07:26:38.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.649116+0000", 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8406999999999998e-05 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.649126+0000", 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0336000000000001e-05 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.545944+0000", 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.89681783500000001 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.546808+0000", 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00086408800000000005 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.547135+0000", 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032632599999999999 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.549431+0000", 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0022961470000000001 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.550639+0000", 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001208087 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.554612+0000", 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039734000000000002 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.554813+0000", 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020055100000000001 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.088 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.554944+0000", 2024-02-05T07:26:38.088 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000131798 2024-02-05T07:26:38.088 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.088 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.088 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:38.089 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:118955 7.10 7:096879d6:::benchmark_data_smithi007_77086_object14360:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T07:26:38.090 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:22:46.751498+0000", 2024-02-05T07:26:38.090 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 231.23650150700001, 2024-02-05T07:26:38.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.81624261200000003, 2024-02-05T07:26:38.090 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.090 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.090 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.090 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.090 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.090 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 118955 2024-02-05T07:26:38.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.090 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.090 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.090 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.751498+0000", 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.751498+0000", 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.751500+0000", 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9630000000000001e-06 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.091 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.751508+0000", 2024-02-05T07:26:38.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1510000000000004e-06 2024-02-05T07:26:38.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.751531+0000", 2024-02-05T07:26:38.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3184e-05 2024-02-05T07:26:38.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.751543+0000", 2024-02-05T07:26:38.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2006e-05 2024-02-05T07:26:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.751626+0000", 2024-02-05T07:26:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2917000000000006e-05 2024-02-05T07:26:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.751639+0000", 2024-02-05T07:26:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2921999999999999e-05 2024-02-05T07:26:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.550667+0000", 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.79902812499999998 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.550774+0000", 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010656 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.551129+0000", 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00035551799999999998 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567305+0000", 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016176202000000001 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567353+0000", 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7301e-05 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567587+0000", 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023395600000000001 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567647+0000", 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.012e-05 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567740+0000", 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3686999999999999e-05 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:118992 7.10 7:0de7edbd:::benchmark_data_smithi007_77086_object14397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:22:46.795489+0000", 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 231.19250989299999, 2024-02-05T07:26:38.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.77242103299999998, 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 118992 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.795489+0000", 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.795489+0000", 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.795491+0000", 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2550000000000001e-06 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.795495+0000", 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4800000000000003e-06 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.795506+0000", 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1158000000000001e-05 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.795514+0000", 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1030000000000002e-06 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.795597+0000", 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2127999999999997e-05 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.795619+0000", 2024-02-05T07:26:38.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2410999999999999e-05 2024-02-05T07:26:38.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.552705+0000", 2024-02-05T07:26:38.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.75708571400000002 2024-02-05T07:26:38.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.552829+0000", 2024-02-05T07:26:38.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000124306 2024-02-05T07:26:38.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.553109+0000", 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000280418 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567406+0000", 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014296402999999999 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567457+0000", 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0847000000000003e-05 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567781+0000", 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032469399999999999 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567811+0000", 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9779000000000001e-05 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567910+0000", 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9337000000000001e-05 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.102 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.103 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.103 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:118993 7.10 7:085a37c5:::benchmark_data_smithi007_77086_object14398:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T07:26:38.103 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:22:46.796212+0000", 2024-02-05T07:26:38.103 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 231.191787449, 2024-02-05T07:26:38.103 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.77187101000000002, 2024-02-05T07:26:38.103 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.103 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.103 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.103 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.103 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.103 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 118993 2024-02-05T07:26:38.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.104 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.796212+0000", 2024-02-05T07:26:38.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.796212+0000", 2024-02-05T07:26:38.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.796213+0000", 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5300000000000002e-07 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.796217+0000", 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6090000000000003e-06 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.796236+0000", 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.897e-05 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.796244+0000", 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7209999999999994e-06 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.796305+0000", 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0435e-05 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.796318+0000", 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3447e-05 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.554984+0000", 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.75866587399999996 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.555093+0000", 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000109634 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.555341+0000", 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00024789199999999999 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567410+0000", 2024-02-05T07:26:38.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012068834000000001 2024-02-05T07:26:38.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567501+0000", 2024-02-05T07:26:38.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0446999999999996e-05 2024-02-05T07:26:38.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567953+0000", 2024-02-05T07:26:38.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00045217800000000002 2024-02-05T07:26:38.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.568000+0000", 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7274000000000002e-05 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.568083+0000", 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2742000000000002e-05 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:119047 7.10 7:0a0ef017:::benchmark_data_smithi007_77086_object14452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:22:46.918663+0000", 2024-02-05T07:26:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 231.069336427, 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.64958196300000004, 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 119047 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.918663+0000", 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.918663+0000", 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.918664+0000", 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2389999999999999e-06 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.918669+0000", 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8629999999999999e-06 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.918684+0000", 2024-02-05T07:26:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4810999999999999e-05 2024-02-05T07:26:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.918692+0000", 2024-02-05T07:26:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2290000000000003e-06 2024-02-05T07:26:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.918757+0000", 2024-02-05T07:26:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4573999999999995e-05 2024-02-05T07:26:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.918766+0000", 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.815e-06 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.557156+0000", 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.63838924100000005 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.557275+0000", 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011900699999999999 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.557534+0000", 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00025925699999999999 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567415+0000", 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0098815399999999994 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567549+0000", 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013400599999999999 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.568108+0000", 2024-02-05T07:26:38.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00055823299999999999 2024-02-05T07:26:38.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.568159+0000", 2024-02-05T07:26:38.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1221999999999998e-05 2024-02-05T07:26:38.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.116 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.568245+0000", 2024-02-05T07:26:38.116 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5926000000000004e-05 2024-02-05T07:26:38.116 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.116 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.116 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.116 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:119069 7.10 7:087c949d:::benchmark_data_smithi007_77086_object14474:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T07:26:38.116 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:22:46.949055+0000", 2024-02-05T07:26:38.116 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 231.03894435699999, 2024-02-05T07:26:38.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.61963992899999998, 2024-02-05T07:26:38.117 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.117 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.117 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.117 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.117 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.117 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 119069 2024-02-05T07:26:38.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.117 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.117 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.949055+0000", 2024-02-05T07:26:38.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.949055+0000", 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.949057+0000", 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6559999999999999e-06 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.949062+0000", 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6919999999999999e-06 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.949076+0000", 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3718e-05 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.949085+0000", 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6829999999999997e-06 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.949154+0000", 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9467000000000004e-05 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.949162+0000", 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7829999999999999e-06 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.559656+0000", 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.61049427199999995 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.559784+0000", 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012824099999999999 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.560448+0000", 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00066359800000000005 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567421+0000", 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0069731250000000002 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.568289+0000", 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00086763200000000002 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.568331+0000", 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2698999999999999e-05 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.568598+0000", 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026663199999999999 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.568695+0000", 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6731000000000006e-05 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.122 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:119071 7.10 7:0a9ee9ec:::benchmark_data_smithi007_77086_object14476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:22:46.950744+0000", 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 231.037255415, 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.61851896900000003, 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 119071 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.123 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:26:38.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.124 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:26:38.124 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.950744+0000", 2024-02-05T07:26:38.124 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.124 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:26:38.124 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.950744+0000", 2024-02-05T07:26:38.124 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:26:38.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.124 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:26:38.124 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.950745+0000", 2024-02-05T07:26:38.124 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.531e-06 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.950751+0000", 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2610000000000001e-06 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.950767+0000", 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6430999999999999e-05 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.950776+0000", 2024-02-05T07:26:38.125 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7639999999999994e-06 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.950849+0000", 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2991999999999995e-05 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:46.950863+0000", 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3671e-05 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.562690+0000", 2024-02-05T07:26:38.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.611827012 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.563162+0000", 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000471991 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.563482+0000", 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032031900000000001 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.567513+0000", 2024-02-05T07:26:38.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0040308030000000003 2024-02-05T07:26:38.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.128 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.128 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.568740+0000", 2024-02-05T07:26:38.128 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012271350000000001 2024-02-05T07:26:38.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.128 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:38.128 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.568786+0000", 2024-02-05T07:26:38.128 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6412999999999998e-05 2024-02-05T07:26:38.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.128 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:26:38.128 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.569103+0000", 2024-02-05T07:26:38.129 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00031713100000000001 2024-02-05T07:26:38.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:26:38.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:26:38.129 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:26:38.129 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:22:47.569263+0000", 2024-02-05T07:26:38.129 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015951499999999999 2024-02-05T07:26:38.129 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.129 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.129 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.129 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:26:38.129 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:26:38.129 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:26:38.157 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:38.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:38.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:37 smithi090 ceph-mon[33213]: 4.d scrub starts 2024-02-05T07:26:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:37 smithi090 ceph-mon[33213]: 4.d scrub ok 2024-02-05T07:26:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:37 smithi090 ceph-mon[33213]: 6.7 scrub starts 2024-02-05T07:26:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:37 smithi090 ceph-mon[33213]: 6.7 scrub ok 2024-02-05T07:26:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:37 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2818136249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:26:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:37 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:26:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:37 smithi090 ceph-mon[33213]: 3.f scrub starts 2024-02-05T07:26:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:37 smithi090 ceph-mon[33213]: 3.f scrub ok 2024-02-05T07:26:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:37 smithi090 ceph-mon[33213]: pgmap v374: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 33469/320847 objects degraded (10.431%) 2024-02-05T07:26:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:37 smithi090 ceph-mon[33213]: 6.10 scrub starts 2024-02-05T07:26:38.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:37 smithi090 ceph-mon[33213]: 6.10 scrub ok 2024-02-05T07:26:38.296 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:38.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:38.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:38.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:38.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:38.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:38.569 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:38.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:38.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:38.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:38.703 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:38.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:38.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:38.835 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:38.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:38.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:38.977 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:38.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:38.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:39.029 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 68 16 112154 112138 103.053 91.375 0.00529296 0.00969911 2024-02-05T07:26:39.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:39.116 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:39.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:39.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:38 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:39.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:38 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:26:39.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:38 smithi007 ceph-mon[29098]: osdmap e243: 8 total, 7 up, 7 in 2024-02-05T07:26:39.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:38 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:39.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:38 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:26:39.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:38 smithi007 ceph-mon[37158]: osdmap e243: 8 total, 7 up, 7 in 2024-02-05T07:26:39.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:39.255 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:39.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:38 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:38 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:26:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:38 smithi090 ceph-mon[33213]: osdmap e243: 8 total, 7 up, 7 in 2024-02-05T07:26:39.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:39.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:39.400 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:39.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:39.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:39.539 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:39.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:39.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:39.680 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:39.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:39.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:39.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:39.815 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:39.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:39.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:39.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:39.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:40.029 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 69 16 113697 113681 102.957 96.4375 0.0119777 0.00970825 2024-02-05T07:26:40.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:40.091 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:40.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:40.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:40.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:39 smithi007 ceph-mon[29098]: pgmap v376: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.0 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 33469/320847 objects degraded (10.431%) 2024-02-05T07:26:40.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:39 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 33469/320847 objects degraded (10.431%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:40.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:39 smithi007 ceph-mon[29098]: osdmap e244: 8 total, 7 up, 7 in 2024-02-05T07:26:40.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:39 smithi007 ceph-mon[37158]: pgmap v376: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.0 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 33469/320847 objects degraded (10.431%) 2024-02-05T07:26:40.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:39 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 33469/320847 objects degraded (10.431%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:40.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:39 smithi007 ceph-mon[37158]: osdmap e244: 8 total, 7 up, 7 in 2024-02-05T07:26:40.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:40.241 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:40.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:40.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:39 smithi090 ceph-mon[33213]: pgmap v376: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 6.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.0 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 33469/320847 objects degraded (10.431%) 2024-02-05T07:26:40.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:39 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 33469/320847 objects degraded (10.431%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:40.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:39 smithi090 ceph-mon[33213]: osdmap e244: 8 total, 7 up, 7 in 2024-02-05T07:26:40.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:40.377 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:40.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:40.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:40.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:40.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:40.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:40.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:40.664 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:40.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:40.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:40.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:26:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:26:40.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:40.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:40.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:40.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:40.941 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:40.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:41.029 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 70 16 115184 115168 102.813 92.9375 0.024114 0.009722 2024-02-05T07:26:41.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:41.070 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:41.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:41.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:41.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:40 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:41.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:40 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:41.203 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:41.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:41.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:41.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:40 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:41.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:41.339 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:41.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:41.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:41.480 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:41.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:41.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:41.624 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:41.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:41.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:41.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:41.762 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:41.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:41.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:41.907 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:41.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:42.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:42.029 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 71 15 116500 116485 102.525 82.3125 0.0064952 0.00974909 2024-02-05T07:26:42.042 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:42.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:42.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:41 smithi007 ceph-mon[29098]: pgmap v378: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.85k op/s; 35726/342861 objects degraded (10.420%) 2024-02-05T07:26:42.102 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:41 smithi007 ceph-mon[37158]: pgmap v378: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.85k op/s; 35726/342861 objects degraded (10.420%) 2024-02-05T07:26:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:26:42.111 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:26:42.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:42.175 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:42.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:42.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:42.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T07:26:42.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:26:42.252 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:41 smithi090 ceph-mon[33213]: pgmap v378: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.85k op/s; 35726/342861 objects degraded (10.420%) 2024-02-05T07:26:42.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:42.313 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:42.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:42.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:42.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:42.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:42.465 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:42.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:42.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:42.587 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:26:42.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:42.604 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:42.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:42.705 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:42.748 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:42.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:42.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:42.873 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:42.889 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:42.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:42.915 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-02-05T07:26:41.078326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7477449413,"num_objects":114287,"num_object_clones":0,"num_object_copies":342861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114287,"num_whiteouts":0,"num_read":1290,"num_read_kb":1104,"num_write":114640,"num_write_kb":7303407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":1785257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72332,"ondisk_log_size":72332,"up":405,"acting":405,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":477,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":22697448,"kb_used_data":20604988,"kb_used_omap":10,"kb_used_meta":2092341,"kb_avail":633518616,"statfs":{"total":671965249536,"available":648723062784,"internally_reserved":0,"allocated":21099507712,"data_stored":21094376195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11159,"internal_metadata":2142557289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,9,13,5,8],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":968687616,"num_objects":14781,"num_object_clones":0,"num_object_copies":44343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14781,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":14781,"num_write_kb":945984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002991"},"pg_stats":[{"pgid":"1.9","version":"86'128","reported_seq":664,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.242889+0000","last_change":"2024-02-05T07:26:18.994035+0000","last_active":"2024-02-05T07:26:39.242889+0000","last_peered":"2024-02-05T07:26:39.242889+0000","last_clean":"2024-02-05T07:26:39.242889+0000","last_became_active":"2024-02-05T07:25:57.243685+0000","last_became_peered":"2024-02-05T07:25:57.243685+0000","last_unstale":"2024-02-05T07:26:39.242889+0000","last_undegraded":"2024-02-05T07:26:39.242889+0000","last_fullsized":"2024-02-05T07:26:39.242889+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:26:18.993912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.993912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:21.690246+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"244'7130","reported_seq":7210,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:40.193651+0000","last_change":"2024-02-05T07:25:16.485753+0000","last_active":"2024-02-05T07:26:40.193651+0000","last_peered":"2024-02-05T07:26:40.193651+0000","last_clean":"2024-02-05T07:26:40.193651+0000","last_became_active":"2024-02-05T07:25:16.485417+0000","last_became_peered":"2024-02-05T07:25:16.485417+0000","last_unstale":"2024-02-05T07:26:40.193651+0000","last_undegraded":"2024-02-05T07:26:40.193651+0000","last_fullsized":"2024-02-05T07:26:40.193651+0000","mapping_epoch":209,"log_start":"222'2700","ondisk_log_start":"222'2700","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4430,"log_dups_size":0,"ondisk_log_size":4430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467271680,"num_objects":7130,"num_object_clones":0,"num_object_copies":21390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7130,"num_write_kb":456320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"221'15","reported_seq":426,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.240566+0000","last_change":"2024-02-05T07:25:59.232703+0000","last_active":"2024-02-05T07:26:39.240566+0000","last_peered":"2024-02-05T07:26:39.240566+0000","last_clean":"2024-02-05T07:25:57.230157+0000","last_became_active":"2024-02-05T07:25:59.232703+0000","last_became_peered":"2024-02-05T07:25:59.232703+0000","last_unstale":"2024-02-05T07:26:39.240566+0000","last_undegraded":"2024-02-05T07:25:59.217986+0000","last_fullsized":"2024-02-05T07:25:59.217840+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'15","last_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:25:44.205475+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:46.274798+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":136,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.238057+0000","last_change":"2024-02-05T07:26:18.258738+0000","last_active":"2024-02-05T07:26:39.238057+0000","last_peered":"2024-02-05T07:26:39.238057+0000","last_clean":"2024-02-05T07:26:39.238057+0000","last_became_active":"2024-02-05T07:24:34.251554+0000","last_became_peered":"2024-02-05T07:24:34.251554+0000","last_unstale":"2024-02-05T07:26:39.238057+0000","last_undegraded":"2024-02-05T07:26:39.238057+0000","last_fullsized":"2024-02-05T07:26:39.238057+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.258469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.258469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:45.164674+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":487,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.246725+0000","last_change":"2024-02-05T07:25:45.823562+0000","last_active":"2024-02-05T07:26:39.246725+0000","last_peered":"2024-02-05T07:26:39.246725+0000","last_clean":"2024-02-05T07:26:39.246725+0000","last_became_active":"2024-02-05T07:18:53.461852+0000","last_became_peered":"2024-02-05T07:18:53.461852+0000","last_unstale":"2024-02-05T07:26:39.246725+0000","last_undegraded":"2024-02-05T07:26:39.246725+0000","last_fullsized":"2024-02-05T07:26:39.246725+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:25:45.823432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:45.823432+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:07.835198+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":377,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.241139+0000","last_change":"2024-02-05T07:25:43.273140+0000","last_active":"2024-02-05T07:26:39.241139+0000","last_peered":"2024-02-05T07:26:39.241139+0000","last_clean":"2024-02-05T07:26:39.241139+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:26:39.241139+0000","last_undegraded":"2024-02-05T07:26:39.241139+0000","last_fullsized":"2024-02-05T07:26:39.241139+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:25:43.273019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:09.905005+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":347,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.245176+0000","last_change":"2024-02-05T07:25:59.237390+0000","last_active":"2024-02-05T07:26:39.245176+0000","last_peered":"2024-02-05T07:26:39.245176+0000","last_clean":"2024-02-05T07:25:57.254058+0000","last_became_active":"2024-02-05T07:25:59.237390+0000","last_became_peered":"2024-02-05T07:25:59.237390+0000","last_unstale":"2024-02-05T07:26:39.245176+0000","last_undegraded":"2024-02-05T07:26:39.245176+0000","last_fullsized":"2024-02-05T07:25:59.222228+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:42.098342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":423697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"244'7185","reported_seq":7265,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:40.292513+0000","last_change":"2024-02-05T07:25:16.685080+0000","last_active":"2024-02-05T07:26:40.292513+0000","last_peered":"2024-02-05T07:26:40.292513+0000","last_clean":"2024-02-05T07:26:40.292513+0000","last_became_active":"2024-02-05T07:25:16.684739+0000","last_became_peered":"2024-02-05T07:25:16.684739+0000","last_unstale":"2024-02-05T07:26:40.292513+0000","last_undegraded":"2024-02-05T07:26:40.292513+0000","last_fullsized":"2024-02-05T07:26:40.292513+0000","mapping_epoch":209,"log_start":"224'3000","ondisk_log_start":"224'3000","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470876160,"num_objects":7185,"num_object_clones":0,"num_object_copies":21555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7185,"num_write_kb":459840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":421,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.240929+0000","last_change":"2024-02-05T07:26:36.275060+0000","last_active":"2024-02-05T07:26:39.240929+0000","last_peered":"2024-02-05T07:26:39.240929+0000","last_clean":"2024-02-05T07:26:39.240929+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:26:39.240929+0000","last_undegraded":"2024-02-05T07:26:39.240929+0000","last_fullsized":"2024-02-05T07:26:39.240929+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:26:36.274939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.274939+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:45.343791+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.235704+0000","last_change":"2024-02-05T07:25:59.233701+0000","last_active":"2024-02-05T07:26:39.235704+0000","last_peered":"2024-02-05T07:26:39.235704+0000","last_clean":"2024-02-05T07:18:48.410364+0000","last_became_active":"2024-02-05T07:25:59.233701+0000","last_became_peered":"2024-02-05T07:25:59.233701+0000","last_unstale":"2024-02-05T07:26:39.235704+0000","last_undegraded":"2024-02-05T07:26:39.235704+0000","last_fullsized":"2024-02-05T07:25:59.218441+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:35.518301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:35.518301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:58.329970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":382,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.235559+0000","last_change":"2024-02-05T07:25:39.466584+0000","last_active":"2024-02-05T07:26:39.235559+0000","last_peered":"2024-02-05T07:26:39.235559+0000","last_clean":"2024-02-05T07:26:39.235559+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:26:39.235559+0000","last_undegraded":"2024-02-05T07:26:39.235559+0000","last_fullsized":"2024-02-05T07:26:39.235559+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:39.466451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:39.466451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:55.949640+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":378,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.248574+0000","last_change":"2024-02-05T07:25:59.233048+0000","last_active":"2024-02-05T07:26:39.248574+0000","last_peered":"2024-02-05T07:26:39.248574+0000","last_clean":"2024-02-05T07:25:57.240349+0000","last_became_active":"2024-02-05T07:25:59.233048+0000","last_became_peered":"2024-02-05T07:25:59.233048+0000","last_unstale":"2024-02-05T07:26:39.248574+0000","last_undegraded":"2024-02-05T07:26:39.248574+0000","last_fullsized":"2024-02-05T07:25:59.224482+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:54.743499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:54.743499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"244'7091","reported_seq":7171,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:40.175347+0000","last_change":"2024-02-05T07:25:16.684635+0000","last_active":"2024-02-05T07:26:40.175347+0000","last_peered":"2024-02-05T07:26:40.175347+0000","last_clean":"2024-02-05T07:26:40.175347+0000","last_became_active":"2024-02-05T07:25:16.684316+0000","last_became_peered":"2024-02-05T07:25:16.684316+0000","last_unstale":"2024-02-05T07:26:40.175347+0000","last_undegraded":"2024-02-05T07:26:40.175347+0000","last_fullsized":"2024-02-05T07:26:40.175347+0000","mapping_epoch":209,"log_start":"223'2900","ondisk_log_start":"223'2900","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4191,"log_dups_size":0,"ondisk_log_size":4191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464715776,"num_objects":7091,"num_object_clones":0,"num_object_copies":21273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7091,"num_write_kb":453824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"90'16","reported_seq":412,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.240582+0000","last_change":"2024-02-05T07:25:59.232148+0000","last_active":"2024-02-05T07:26:39.240582+0000","last_peered":"2024-02-05T07:26:39.240582+0000","last_clean":"2024-02-05T07:25:57.230984+0000","last_became_active":"2024-02-05T07:25:59.232148+0000","last_became_peered":"2024-02-05T07:25:59.232148+0000","last_unstale":"2024-02-05T07:26:39.240582+0000","last_undegraded":"2024-02-05T07:25:59.222004+0000","last_fullsized":"2024-02-05T07:25:59.221213+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-02-05T07:25:27.129600+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:25:27.129600+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:52.803796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":389,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.236237+0000","last_change":"2024-02-05T07:25:46.149566+0000","last_active":"2024-02-05T07:26:39.236237+0000","last_peered":"2024-02-05T07:26:39.236237+0000","last_clean":"2024-02-05T07:26:39.236237+0000","last_became_active":"2024-02-05T07:18:49.434074+0000","last_became_peered":"2024-02-05T07:18:49.434074+0000","last_unstale":"2024-02-05T07:26:39.236237+0000","last_undegraded":"2024-02-05T07:26:39.236237+0000","last_fullsized":"2024-02-05T07:26:39.236237+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:25:46.149431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:09.295357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":206,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.241556+0000","last_change":"2024-02-05T07:25:59.239892+0000","last_active":"2024-02-05T07:26:39.241556+0000","last_peered":"2024-02-05T07:26:39.241556+0000","last_clean":"2024-02-05T07:25:57.231478+0000","last_became_active":"2024-02-05T07:25:59.239892+0000","last_became_peered":"2024-02-05T07:25:59.239892+0000","last_unstale":"2024-02-05T07:26:39.241556+0000","last_undegraded":"2024-02-05T07:26:39.241556+0000","last_fullsized":"2024-02-05T07:25:59.218622+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:49.760543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.760543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:41.264541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.239914+0000","last_change":"2024-02-05T07:25:59.242623+0000","last_active":"2024-02-05T07:26:39.239914+0000","last_peered":"2024-02-05T07:26:39.239914+0000","last_clean":"2024-02-05T07:18:54.446999+0000","last_became_active":"2024-02-05T07:25:59.242623+0000","last_became_peered":"2024-02-05T07:25:59.242623+0000","last_unstale":"2024-02-05T07:26:39.239914+0000","last_undegraded":"2024-02-05T07:26:39.239914+0000","last_fullsized":"2024-02-05T07:25:59.220676+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:47.885294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.885294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:12.618000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":347,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.244717+0000","last_change":"2024-02-05T07:25:59.238680+0000","last_active":"2024-02-05T07:26:39.244717+0000","last_peered":"2024-02-05T07:26:39.244717+0000","last_clean":"2024-02-05T07:25:57.244953+0000","last_became_active":"2024-02-05T07:25:59.238680+0000","last_became_peered":"2024-02-05T07:25:59.238680+0000","last_unstale":"2024-02-05T07:26:39.244717+0000","last_undegraded":"2024-02-05T07:26:39.244717+0000","last_fullsized":"2024-02-05T07:25:59.222657+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:42.098359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3","version":"244'6950","reported_seq":7033,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:40.306549+0000","last_change":"2024-02-05T07:25:59.231802+0000","last_active":"2024-02-05T07:26:40.306549+0000","last_peered":"2024-02-05T07:26:40.306549+0000","last_clean":"2024-02-05T07:25:58.269991+0000","last_became_active":"2024-02-05T07:25:59.231802+0000","last_became_peered":"2024-02-05T07:25:59.231802+0000","last_unstale":"2024-02-05T07:26:40.306549+0000","last_undegraded":"2024-02-05T07:25:59.219344+0000","last_fullsized":"2024-02-05T07:25:59.219214+0000","mapping_epoch":231,"log_start":"222'2700","ondisk_log_start":"222'2700","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455475200,"num_objects":6950,"num_object_clones":0,"num_object_copies":20850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6950,"num_write_kb":444800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":6950}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"98'22","reported_seq":440,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.241376+0000","last_change":"2024-02-05T07:26:25.602299+0000","last_active":"2024-02-05T07:26:39.241376+0000","last_peered":"2024-02-05T07:26:39.241376+0000","last_clean":"2024-02-05T07:26:39.241376+0000","last_became_active":"2024-02-05T07:23:47.606483+0000","last_became_peered":"2024-02-05T07:23:47.606483+0000","last_unstale":"2024-02-05T07:26:39.241376+0000","last_undegraded":"2024-02-05T07:26:39.241376+0000","last_fullsized":"2024-02-05T07:26:39.241376+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'22","last_scrub_stamp":"2024-02-05T07:26:25.602159+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:26:25.602159+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:45.632920+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":183,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.239714+0000","last_change":"2024-02-05T07:25:48.513731+0000","last_active":"2024-02-05T07:26:39.239714+0000","last_peered":"2024-02-05T07:26:39.239714+0000","last_clean":"2024-02-05T07:26:39.239714+0000","last_became_active":"2024-02-05T07:23:47.600273+0000","last_became_peered":"2024-02-05T07:23:47.600273+0000","last_unstale":"2024-02-05T07:26:39.239714+0000","last_undegraded":"2024-02-05T07:26:39.239714+0000","last_fullsized":"2024-02-05T07:26:39.239714+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:48.513614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:48.513614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:14.712372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":484,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.241085+0000","last_change":"2024-02-05T07:26:22.295885+0000","last_active":"2024-02-05T07:26:39.241085+0000","last_peered":"2024-02-05T07:26:39.241085+0000","last_clean":"2024-02-05T07:26:39.241085+0000","last_became_active":"2024-02-05T07:18:53.459832+0000","last_became_peered":"2024-02-05T07:18:53.459832+0000","last_unstale":"2024-02-05T07:26:39.241085+0000","last_undegraded":"2024-02-05T07:26:39.241085+0000","last_fullsized":"2024-02-05T07:26:39.241085+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:26:22.295732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:22.295732+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:49.510901+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":377,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.238630+0000","last_change":"2024-02-05T07:26:18.725054+0000","last_active":"2024-02-05T07:26:39.238630+0000","last_peered":"2024-02-05T07:26:39.238630+0000","last_clean":"2024-02-05T07:26:39.238630+0000","last_became_active":"2024-02-05T07:18:55.472172+0000","last_became_peered":"2024-02-05T07:18:55.472172+0000","last_unstale":"2024-02-05T07:26:39.238630+0000","last_undegraded":"2024-02-05T07:26:39.238630+0000","last_fullsized":"2024-02-05T07:26:39.238630+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.724944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.724944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:48.112137+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":347,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.240076+0000","last_change":"2024-02-05T07:25:59.232121+0000","last_active":"2024-02-05T07:26:39.240076+0000","last_peered":"2024-02-05T07:26:39.240076+0000","last_clean":"2024-02-05T07:25:57.249010+0000","last_became_active":"2024-02-05T07:25:59.232121+0000","last_became_peered":"2024-02-05T07:25:59.232121+0000","last_unstale":"2024-02-05T07:26:39.240076+0000","last_undegraded":"2024-02-05T07:26:39.240076+0000","last_fullsized":"2024-02-05T07:25:59.217731+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.186799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"244'6908","reported_seq":6988,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:40.196538+0000","last_change":"2024-02-05T07:25:16.684780+0000","last_active":"2024-02-05T07:26:40.196538+0000","last_peered":"2024-02-05T07:26:40.196538+0000","last_clean":"2024-02-05T07:26:40.196538+0000","last_became_active":"2024-02-05T07:25:16.684374+0000","last_became_peered":"2024-02-05T07:25:16.684374+0000","last_unstale":"2024-02-05T07:26:40.196538+0000","last_undegraded":"2024-02-05T07:26:40.196538+0000","last_fullsized":"2024-02-05T07:26:40.196538+0000","mapping_epoch":209,"log_start":"222'2300","ondisk_log_start":"222'2300","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452722688,"num_objects":6908,"num_object_clones":0,"num_object_copies":20724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":442112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":387,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.236163+0000","last_change":"2024-02-05T07:25:51.211993+0000","last_active":"2024-02-05T07:26:39.236163+0000","last_peered":"2024-02-05T07:26:39.236163+0000","last_clean":"2024-02-05T07:26:39.236163+0000","last_became_active":"2024-02-05T07:23:47.626072+0000","last_became_peered":"2024-02-05T07:23:47.626072+0000","last_unstale":"2024-02-05T07:26:39.236163+0000","last_undegraded":"2024-02-05T07:26:39.236163+0000","last_fullsized":"2024-02-05T07:26:39.236163+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:51.211877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:51.211877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:08.209334+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":252,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.244944+0000","last_change":"2024-02-05T07:24:14.360014+0000","last_active":"2024-02-05T07:26:39.244944+0000","last_peered":"2024-02-05T07:26:39.244944+0000","last_clean":"2024-02-05T07:26:39.244944+0000","last_became_active":"2024-02-05T07:23:47.618052+0000","last_became_peered":"2024-02-05T07:23:47.618052+0000","last_unstale":"2024-02-05T07:26:39.244944+0000","last_undegraded":"2024-02-05T07:26:39.244944+0000","last_fullsized":"2024-02-05T07:26:39.244944+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:24:14.359892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:14.359892+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.059059+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":183,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.244717+0000","last_change":"2024-02-05T07:26:33.084149+0000","last_active":"2024-02-05T07:26:39.244717+0000","last_peered":"2024-02-05T07:26:39.244717+0000","last_clean":"2024-02-05T07:26:39.244717+0000","last_became_active":"2024-02-05T07:23:47.609054+0000","last_became_peered":"2024-02-05T07:23:47.609054+0000","last_unstale":"2024-02-05T07:26:39.244717+0000","last_undegraded":"2024-02-05T07:26:39.244717+0000","last_fullsized":"2024-02-05T07:26:39.244717+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:33.084016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:33.084016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:56.291496+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":378,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.236804+0000","last_change":"2024-02-05T07:25:59.233964+0000","last_active":"2024-02-05T07:26:39.236804+0000","last_peered":"2024-02-05T07:26:39.236804+0000","last_clean":"2024-02-05T07:25:57.241137+0000","last_became_active":"2024-02-05T07:25:59.233964+0000","last_became_peered":"2024-02-05T07:25:59.233964+0000","last_unstale":"2024-02-05T07:26:39.236804+0000","last_undegraded":"2024-02-05T07:26:39.236804+0000","last_fullsized":"2024-02-05T07:25:59.217450+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:16.499591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:16.499591+0000","last_clean_scrub_stamp":"2024-02-05T07:23:16.499591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:41.264450+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":347,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.238845+0000","last_change":"2024-02-05T07:25:59.242800+0000","last_active":"2024-02-05T07:26:39.238845+0000","last_peered":"2024-02-05T07:26:39.238845+0000","last_clean":"2024-02-05T07:25:57.248422+0000","last_became_active":"2024-02-05T07:25:59.242800+0000","last_became_peered":"2024-02-05T07:25:59.242800+0000","last_unstale":"2024-02-05T07:26:39.238845+0000","last_undegraded":"2024-02-05T07:26:39.238845+0000","last_fullsized":"2024-02-05T07:25:59.219651+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.739711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":488690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.d","version":"244'7188","reported_seq":7267,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:40.215537+0000","last_change":"2024-02-05T07:25:16.486195+0000","last_active":"2024-02-05T07:26:40.215537+0000","last_peered":"2024-02-05T07:26:40.215537+0000","last_clean":"2024-02-05T07:26:40.215537+0000","last_became_active":"2024-02-05T07:25:16.485881+0000","last_became_peered":"2024-02-05T07:25:16.485881+0000","last_unstale":"2024-02-05T07:26:40.215537+0000","last_undegraded":"2024-02-05T07:26:40.215537+0000","last_fullsized":"2024-02-05T07:26:40.215537+0000","mapping_epoch":209,"log_start":"222'2600","ondisk_log_start":"222'2600","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4588,"log_dups_size":0,"ondisk_log_size":4588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.672798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471072768,"num_objects":7188,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7188,"num_write_kb":460032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"164'9","reported_seq":255,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.238856+0000","last_change":"2024-02-05T07:26:34.536185+0000","last_active":"2024-02-05T07:26:39.238856+0000","last_peered":"2024-02-05T07:26:39.238856+0000","last_clean":"2024-02-05T07:26:39.238856+0000","last_became_active":"2024-02-05T07:23:47.626082+0000","last_became_peered":"2024-02-05T07:23:47.626082+0000","last_unstale":"2024-02-05T07:26:39.238856+0000","last_undegraded":"2024-02-05T07:26:39.238856+0000","last_fullsized":"2024-02-05T07:26:39.238856+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:26:34.535447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:34.535447+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:28:00.354122+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"202'21","reported_seq":408,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.236367+0000","last_change":"2024-02-05T07:25:59.229588+0000","last_active":"2024-02-05T07:26:39.236367+0000","last_peered":"2024-02-05T07:26:39.236367+0000","last_clean":"2024-02-05T07:25:57.230607+0000","last_became_active":"2024-02-05T07:25:59.229588+0000","last_became_peered":"2024-02-05T07:25:59.229588+0000","last_unstale":"2024-02-05T07:26:39.236367+0000","last_undegraded":"2024-02-05T07:25:59.217880+0000","last_fullsized":"2024-02-05T07:25:59.217228+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'21","last_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:24:59.870542+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:41.264568+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":142,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.236990+0000","last_change":"2024-02-05T07:25:33.462720+0000","last_active":"2024-02-05T07:26:39.236990+0000","last_peered":"2024-02-05T07:26:39.236990+0000","last_clean":"2024-02-05T07:26:39.236990+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:26:39.236990+0000","last_undegraded":"2024-02-05T07:26:39.236990+0000","last_fullsized":"2024-02-05T07:26:39.236990+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:25:33.462588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:33.462588+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:44.148228+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":372,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.243774+0000","last_change":"2024-02-05T07:25:59.226701+0000","last_active":"2024-02-05T07:26:39.243774+0000","last_peered":"2024-02-05T07:26:39.243774+0000","last_clean":"2024-02-05T07:25:57.239687+0000","last_became_active":"2024-02-05T07:25:59.226701+0000","last_became_peered":"2024-02-05T07:25:59.226701+0000","last_unstale":"2024-02-05T07:26:39.243774+0000","last_undegraded":"2024-02-05T07:26:39.243774+0000","last_fullsized":"2024-02-05T07:25:59.222593+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:59.124133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:59.124133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.059185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":640,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.242517+0000","last_change":"2024-02-05T07:25:59.238608+0000","last_active":"2024-02-05T07:26:39.242517+0000","last_peered":"2024-02-05T07:26:39.242517+0000","last_clean":"2024-02-05T07:25:52.604094+0000","last_became_active":"2024-02-05T07:25:59.238608+0000","last_became_peered":"2024-02-05T07:25:59.238608+0000","last_unstale":"2024-02-05T07:26:39.242517+0000","last_undegraded":"2024-02-05T07:26:39.242517+0000","last_fullsized":"2024-02-05T07:25:59.225763+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:44.059873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"244'7048","reported_seq":7189,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:40.655327+0000","last_change":"2024-02-05T07:26:19.695559+0000","last_active":"2024-02-05T07:26:40.655327+0000","last_peered":"2024-02-05T07:26:40.655327+0000","last_clean":"2024-02-05T07:26:40.655327+0000","last_became_active":"2024-02-05T07:25:16.685782+0000","last_became_peered":"2024-02-05T07:25:16.685782+0000","last_unstale":"2024-02-05T07:26:40.655327+0000","last_undegraded":"2024-02-05T07:26:40.655327+0000","last_fullsized":"2024-02-05T07:26:40.655327+0000","mapping_epoch":209,"log_start":"222'2200","ondisk_log_start":"222'2200","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'5314","last_scrub_stamp":"2024-02-05T07:26:19.695409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:26:19.695409+0000","objects_scrubbed":5255,"log_size":4848,"log_dups_size":0,"ondisk_log_size":4848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:39.411249+0000","scrub_duration":1692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461897728,"num_objects":7048,"num_object_clones":0,"num_object_copies":21144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7048,"num_write_kb":451072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"232'10","reported_seq":68,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.238810+0000","last_change":"2024-02-05T07:25:59.225675+0000","last_active":"2024-02-05T07:26:39.238810+0000","last_peered":"2024-02-05T07:26:39.238810+0000","last_clean":"2024-02-05T07:18:56.639791+0000","last_became_active":"2024-02-05T07:25:59.225675+0000","last_became_peered":"2024-02-05T07:25:59.225675+0000","last_unstale":"2024-02-05T07:26:39.238810+0000","last_undegraded":"2024-02-05T07:25:59.219060+0000","last_fullsized":"2024-02-05T07:25:59.218929+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'9","last_scrub_stamp":"2024-02-05T07:24:59.438123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:59.438123+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:41.707357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"151'15","reported_seq":414,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.244663+0000","last_change":"2024-02-05T07:26:26.616319+0000","last_active":"2024-02-05T07:26:39.244663+0000","last_peered":"2024-02-05T07:26:39.244663+0000","last_clean":"2024-02-05T07:26:39.244663+0000","last_became_active":"2024-02-05T07:18:51.448044+0000","last_became_peered":"2024-02-05T07:18:51.448044+0000","last_unstale":"2024-02-05T07:26:39.244663+0000","last_undegraded":"2024-02-05T07:26:39.244663+0000","last_fullsized":"2024-02-05T07:26:39.244663+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-02-05T07:26:26.616180+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:26:26.616180+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:45.683238+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":390,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.238375+0000","last_change":"2024-02-05T07:25:59.232188+0000","last_active":"2024-02-05T07:26:39.238375+0000","last_peered":"2024-02-05T07:26:39.238375+0000","last_clean":"2024-02-05T07:25:57.236865+0000","last_became_active":"2024-02-05T07:25:59.232188+0000","last_became_peered":"2024-02-05T07:25:59.232188+0000","last_unstale":"2024-02-05T07:26:39.238375+0000","last_undegraded":"2024-02-05T07:26:39.238375+0000","last_fullsized":"2024-02-05T07:25:59.220037+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:57.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:24:57.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":330,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.245698+0000","last_change":"2024-02-05T07:25:52.848138+0000","last_active":"2024-02-05T07:26:39.245698+0000","last_peered":"2024-02-05T07:26:39.245698+0000","last_clean":"2024-02-05T07:26:39.245698+0000","last_became_active":"2024-02-05T07:24:34.229242+0000","last_became_peered":"2024-02-05T07:24:34.229242+0000","last_unstale":"2024-02-05T07:26:39.245698+0000","last_undegraded":"2024-02-05T07:26:39.245698+0000","last_fullsized":"2024-02-05T07:26:39.245698+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:52.847946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:52.847946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:10.227784+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":338,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.240791+0000","last_change":"2024-02-05T07:25:59.230750+0000","last_active":"2024-02-05T07:26:39.240791+0000","last_peered":"2024-02-05T07:26:39.240791+0000","last_clean":"2024-02-05T07:19:30.999323+0000","last_became_active":"2024-02-05T07:25:59.230750+0000","last_became_peered":"2024-02-05T07:25:59.230750+0000","last_unstale":"2024-02-05T07:26:39.240791+0000","last_undegraded":"2024-02-05T07:26:39.240791+0000","last_fullsized":"2024-02-05T07:25:59.217016+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.186767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"244'7288","reported_seq":7366,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:40.619514+0000","last_change":"2024-02-05T07:25:16.482780+0000","last_active":"2024-02-05T07:26:40.619514+0000","last_peered":"2024-02-05T07:26:40.619514+0000","last_clean":"2024-02-05T07:26:40.619514+0000","last_became_active":"2024-02-05T07:25:16.482501+0000","last_became_peered":"2024-02-05T07:25:16.482501+0000","last_unstale":"2024-02-05T07:26:40.619514+0000","last_undegraded":"2024-02-05T07:26:40.619514+0000","last_fullsized":"2024-02-05T07:26:40.619514+0000","mapping_epoch":209,"log_start":"222'2400","ondisk_log_start":"222'2400","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477626368,"num_objects":7288,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7288,"num_write_kb":466432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":33,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.240908+0000","last_change":"2024-02-05T07:25:59.237506+0000","last_active":"2024-02-05T07:26:39.240908+0000","last_peered":"2024-02-05T07:26:39.240908+0000","last_clean":"2024-02-05T07:18:52.436288+0000","last_became_active":"2024-02-05T07:25:59.237506+0000","last_became_peered":"2024-02-05T07:25:59.237506+0000","last_unstale":"2024-02-05T07:26:39.240908+0000","last_undegraded":"2024-02-05T07:26:39.240908+0000","last_fullsized":"2024-02-05T07:25:59.222666+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:30.535958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:25:30.535958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:54.621161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":198,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.238689+0000","last_change":"2024-02-05T07:26:20.705484+0000","last_active":"2024-02-05T07:26:39.238689+0000","last_peered":"2024-02-05T07:26:39.238689+0000","last_clean":"2024-02-05T07:26:39.238689+0000","last_became_active":"2024-02-05T07:24:34.229372+0000","last_became_peered":"2024-02-05T07:24:34.229372+0000","last_unstale":"2024-02-05T07:26:39.238689+0000","last_undegraded":"2024-02-05T07:26:39.238689+0000","last_fullsized":"2024-02-05T07:26:39.238689+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:26:20.705363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:20.705363+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:49.840358+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":390,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.235425+0000","last_change":"2024-02-05T07:25:49.513525+0000","last_active":"2024-02-05T07:26:39.235425+0000","last_peered":"2024-02-05T07:26:39.235425+0000","last_clean":"2024-02-05T07:26:39.235425+0000","last_became_active":"2024-02-05T07:18:49.428470+0000","last_became_peered":"2024-02-05T07:18:49.428470+0000","last_unstale":"2024-02-05T07:26:39.235425+0000","last_undegraded":"2024-02-05T07:26:39.235425+0000","last_fullsized":"2024-02-05T07:26:39.235425+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:49.513404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:25:49.513404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:11.112788+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":373,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.235484+0000","last_change":"2024-02-05T07:25:59.224828+0000","last_active":"2024-02-05T07:26:39.235484+0000","last_peered":"2024-02-05T07:26:39.235484+0000","last_clean":"2024-02-05T07:25:57.237073+0000","last_became_active":"2024-02-05T07:25:59.224828+0000","last_became_peered":"2024-02-05T07:25:59.224828+0000","last_unstale":"2024-02-05T07:26:39.235484+0000","last_undegraded":"2024-02-05T07:26:39.235484+0000","last_fullsized":"2024-02-05T07:25:59.217054+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:01.849346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:01.849346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:42.576352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":347,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.235518+0000","last_change":"2024-02-05T07:25:59.232229+0000","last_active":"2024-02-05T07:26:39.235518+0000","last_peered":"2024-02-05T07:26:39.235518+0000","last_clean":"2024-02-05T07:25:57.253533+0000","last_became_active":"2024-02-05T07:25:59.232229+0000","last_became_peered":"2024-02-05T07:25:59.232229+0000","last_unstale":"2024-02-05T07:26:39.235518+0000","last_undegraded":"2024-02-05T07:26:39.235518+0000","last_fullsized":"2024-02-05T07:25:59.220509+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:41.600515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.a","version":"244'7315","reported_seq":7395,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:40.304820+0000","last_change":"2024-02-05T07:25:16.485301+0000","last_active":"2024-02-05T07:26:40.304820+0000","last_peered":"2024-02-05T07:26:40.304820+0000","last_clean":"2024-02-05T07:26:40.304820+0000","last_became_active":"2024-02-05T07:25:16.485029+0000","last_became_peered":"2024-02-05T07:25:16.485029+0000","last_unstale":"2024-02-05T07:26:40.304820+0000","last_undegraded":"2024-02-05T07:26:40.304820+0000","last_fullsized":"2024-02-05T07:26:40.304820+0000","mapping_epoch":209,"log_start":"224'3100","ondisk_log_start":"224'3100","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4215,"log_dups_size":0,"ondisk_log_size":4215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479395840,"num_objects":7315,"num_object_clones":0,"num_object_copies":21945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7315,"num_write_kb":468160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":381,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.239279+0000","last_change":"2024-02-05T07:26:32.652647+0000","last_active":"2024-02-05T07:26:39.239279+0000","last_peered":"2024-02-05T07:26:39.239279+0000","last_clean":"2024-02-05T07:26:39.239279+0000","last_became_active":"2024-02-05T07:18:53.468512+0000","last_became_peered":"2024-02-05T07:18:53.468512+0000","last_unstale":"2024-02-05T07:26:39.239279+0000","last_undegraded":"2024-02-05T07:26:39.239279+0000","last_fullsized":"2024-02-05T07:26:39.239279+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:32.652537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:32.652537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:56.130640+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":171,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.237246+0000","last_change":"2024-02-05T07:25:24.528053+0000","last_active":"2024-02-05T07:26:39.237246+0000","last_peered":"2024-02-05T07:26:39.237246+0000","last_clean":"2024-02-05T07:26:39.237246+0000","last_became_active":"2024-02-05T07:24:34.209716+0000","last_became_peered":"2024-02-05T07:24:34.209716+0000","last_unstale":"2024-02-05T07:26:39.237246+0000","last_undegraded":"2024-02-05T07:26:39.237246+0000","last_fullsized":"2024-02-05T07:26:39.237246+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:25:24.527960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:24.527960+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.343172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":856,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.244615+0000","last_change":"2024-02-05T07:24:50.043491+0000","last_active":"2024-02-05T07:26:39.244615+0000","last_peered":"2024-02-05T07:26:39.244615+0000","last_clean":"2024-02-05T07:26:39.244615+0000","last_became_active":"2024-02-05T07:18:48.424620+0000","last_became_peered":"2024-02-05T07:18:48.424620+0000","last_unstale":"2024-02-05T07:26:39.244615+0000","last_undegraded":"2024-02-05T07:26:39.244615+0000","last_fullsized":"2024-02-05T07:26:39.244615+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:24:50.043397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:24:50.043397+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.059213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":463,"num_read_kb":458,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":350,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.236838+0000","last_change":"2024-02-05T07:26:30.351769+0000","last_active":"2024-02-05T07:26:39.236838+0000","last_peered":"2024-02-05T07:26:39.236838+0000","last_clean":"2024-02-05T07:26:39.236838+0000","last_became_active":"2024-02-05T07:24:34.241978+0000","last_became_peered":"2024-02-05T07:24:34.241978+0000","last_unstale":"2024-02-05T07:26:39.236838+0000","last_undegraded":"2024-02-05T07:26:39.236838+0000","last_fullsized":"2024-02-05T07:26:39.236838+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:30.351655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:26:30.351655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:41.877335+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":384,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.235122+0000","last_change":"2024-02-05T07:25:59.244209+0000","last_active":"2024-02-05T07:26:39.235122+0000","last_peered":"2024-02-05T07:26:39.235122+0000","last_clean":"2024-02-05T07:25:57.235814+0000","last_became_active":"2024-02-05T07:25:59.244209+0000","last_became_peered":"2024-02-05T07:25:59.244209+0000","last_unstale":"2024-02-05T07:26:39.235122+0000","last_undegraded":"2024-02-05T07:25:59.219470+0000","last_fullsized":"2024-02-05T07:25:59.219348+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:23:22.071923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:22.071923+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:42.576334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"244'7082","reported_seq":7164,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:40.205508+0000","last_change":"2024-02-05T07:25:59.222591+0000","last_active":"2024-02-05T07:26:40.205508+0000","last_peered":"2024-02-05T07:26:40.205508+0000","last_clean":"2024-02-05T07:25:58.264786+0000","last_became_active":"2024-02-05T07:25:59.222591+0000","last_became_peered":"2024-02-05T07:25:59.222591+0000","last_unstale":"2024-02-05T07:26:40.205508+0000","last_undegraded":"2024-02-05T07:25:59.218928+0000","last_fullsized":"2024-02-05T07:25:59.218814+0000","mapping_epoch":231,"log_start":"222'2700","ondisk_log_start":"222'2700","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464125952,"num_objects":7082,"num_object_clones":0,"num_object_copies":21246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7082,"num_write_kb":453248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":7082}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":373,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.243536+0000","last_change":"2024-02-05T07:25:35.850887+0000","last_active":"2024-02-05T07:26:39.243536+0000","last_peered":"2024-02-05T07:26:39.243536+0000","last_clean":"2024-02-05T07:26:39.243536+0000","last_became_active":"2024-02-05T07:18:53.459971+0000","last_became_peered":"2024-02-05T07:18:53.459971+0000","last_unstale":"2024-02-05T07:26:39.243536+0000","last_undegraded":"2024-02-05T07:26:39.243536+0000","last_fullsized":"2024-02-05T07:26:39.243536+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:35.850761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:35.850761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:47.149079+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"232'17","reported_seq":86,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.240411+0000","last_change":"2024-02-05T07:25:59.239638+0000","last_active":"2024-02-05T07:26:39.240411+0000","last_peered":"2024-02-05T07:26:39.240411+0000","last_clean":"2024-02-05T07:18:56.805155+0000","last_became_active":"2024-02-05T07:25:59.239638+0000","last_became_peered":"2024-02-05T07:25:59.239638+0000","last_unstale":"2024-02-05T07:26:39.240411+0000","last_undegraded":"2024-02-05T07:25:59.221950+0000","last_fullsized":"2024-02-05T07:25:59.221844+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-02-05T07:25:46.826417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:46.826417+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:59.698915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":186,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.239840+0000","last_change":"2024-02-05T07:25:59.236510+0000","last_active":"2024-02-05T07:26:39.239840+0000","last_peered":"2024-02-05T07:26:39.239840+0000","last_clean":"2024-02-05T07:25:57.236146+0000","last_became_active":"2024-02-05T07:25:59.236510+0000","last_became_peered":"2024-02-05T07:25:59.236510+0000","last_unstale":"2024-02-05T07:26:39.239840+0000","last_undegraded":"2024-02-05T07:26:39.239840+0000","last_fullsized":"2024-02-05T07:25:59.219151+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:45.761597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:45.761597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":659,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.244667+0000","last_change":"2024-02-05T07:25:59.226600+0000","last_active":"2024-02-05T07:26:39.244667+0000","last_peered":"2024-02-05T07:26:39.244667+0000","last_clean":"2024-02-05T07:25:57.239270+0000","last_became_active":"2024-02-05T07:25:59.226600+0000","last_became_peered":"2024-02-05T07:25:59.226600+0000","last_unstale":"2024-02-05T07:26:39.244667+0000","last_undegraded":"2024-02-05T07:26:39.244667+0000","last_fullsized":"2024-02-05T07:25:59.224267+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:25:52.772049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:25:52.772049+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:18.116719+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":383,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.238914+0000","last_change":"2024-02-05T07:26:36.753182+0000","last_active":"2024-02-05T07:26:39.238914+0000","last_peered":"2024-02-05T07:26:39.238914+0000","last_clean":"2024-02-05T07:26:39.238914+0000","last_became_active":"2024-02-05T07:23:47.592290+0000","last_became_peered":"2024-02-05T07:23:47.592290+0000","last_unstale":"2024-02-05T07:26:39.238914+0000","last_undegraded":"2024-02-05T07:26:39.238914+0000","last_fullsized":"2024-02-05T07:26:39.238914+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:36.753098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.753098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:53.714599+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":664,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.243190+0000","last_change":"2024-02-05T07:26:19.993252+0000","last_active":"2024-02-05T07:26:39.243190+0000","last_peered":"2024-02-05T07:26:39.243190+0000","last_clean":"2024-02-05T07:26:39.243190+0000","last_became_active":"2024-02-05T07:25:57.252020+0000","last_became_peered":"2024-02-05T07:25:57.252020+0000","last_unstale":"2024-02-05T07:26:39.243190+0000","last_undegraded":"2024-02-05T07:26:39.243190+0000","last_fullsized":"2024-02-05T07:26:39.243190+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:26:19.993124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:26:19.993124+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:37.760808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"244'7097","reported_seq":7177,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:40.172890+0000","last_change":"2024-02-05T07:25:16.482190+0000","last_active":"2024-02-05T07:26:40.172890+0000","last_peered":"2024-02-05T07:26:40.172890+0000","last_clean":"2024-02-05T07:26:40.172890+0000","last_became_active":"2024-02-05T07:25:16.481838+0000","last_became_peered":"2024-02-05T07:25:16.481838+0000","last_unstale":"2024-02-05T07:26:40.172890+0000","last_undegraded":"2024-02-05T07:26:40.172890+0000","last_fullsized":"2024-02-05T07:26:40.172890+0000","mapping_epoch":209,"log_start":"223'2900","ondisk_log_start":"223'2900","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465108992,"num_objects":7097,"num_object_clones":0,"num_object_copies":21291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7097,"num_write_kb":454208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":381,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.249875+0000","last_change":"2024-02-05T07:25:42.863763+0000","last_active":"2024-02-05T07:26:39.249875+0000","last_peered":"2024-02-05T07:26:39.249875+0000","last_clean":"2024-02-05T07:26:39.249875+0000","last_became_active":"2024-02-05T07:18:53.460771+0000","last_became_peered":"2024-02-05T07:18:53.460771+0000","last_unstale":"2024-02-05T07:26:39.249875+0000","last_undegraded":"2024-02-05T07:26:39.249875+0000","last_fullsized":"2024-02-05T07:26:39.249875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:42.863654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.863654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:02.227531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"96'18","reported_seq":418,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.241617+0000","last_change":"2024-02-05T07:26:24.578495+0000","last_active":"2024-02-05T07:26:39.241617+0000","last_peered":"2024-02-05T07:26:39.241617+0000","last_clean":"2024-02-05T07:26:39.241617+0000","last_became_active":"2024-02-05T07:18:51.436097+0000","last_became_peered":"2024-02-05T07:18:51.436097+0000","last_unstale":"2024-02-05T07:26:39.241617+0000","last_undegraded":"2024-02-05T07:26:39.241617+0000","last_fullsized":"2024-02-05T07:26:39.241617+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'18","last_scrub_stamp":"2024-02-05T07:26:24.578301+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.578301+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:54.125229+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":404,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.246105+0000","last_change":"2024-02-05T07:25:35.774077+0000","last_active":"2024-02-05T07:26:39.246105+0000","last_peered":"2024-02-05T07:26:39.246105+0000","last_clean":"2024-02-05T07:26:39.246105+0000","last_became_active":"2024-02-05T07:23:47.621133+0000","last_became_peered":"2024-02-05T07:23:47.621133+0000","last_unstale":"2024-02-05T07:26:39.246105+0000","last_undegraded":"2024-02-05T07:26:39.246105+0000","last_fullsized":"2024-02-05T07:26:39.246105+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:35.773959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:25:35.773959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:58.214145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":399,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.239344+0000","last_change":"2024-02-05T07:25:59.223287+0000","last_active":"2024-02-05T07:26:39.239344+0000","last_peered":"2024-02-05T07:26:39.239344+0000","last_clean":"2024-02-05T07:25:57.238772+0000","last_became_active":"2024-02-05T07:25:59.223287+0000","last_became_peered":"2024-02-05T07:25:59.223287+0000","last_unstale":"2024-02-05T07:26:39.239344+0000","last_undegraded":"2024-02-05T07:25:59.217360+0000","last_fullsized":"2024-02-05T07:25:59.217254+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:25:47.537014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.537014+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:49.815776+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":378,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.244965+0000","last_change":"2024-02-05T07:25:59.236953+0000","last_active":"2024-02-05T07:26:39.244965+0000","last_peered":"2024-02-05T07:26:39.244965+0000","last_clean":"2024-02-05T07:25:57.240231+0000","last_became_active":"2024-02-05T07:25:59.236953+0000","last_became_peered":"2024-02-05T07:25:59.236953+0000","last_unstale":"2024-02-05T07:26:39.244965+0000","last_undegraded":"2024-02-05T07:26:39.244965+0000","last_fullsized":"2024-02-05T07:25:59.228615+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:38.531265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:21:38.531265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:40.079905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":351,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.245470+0000","last_change":"2024-02-05T07:26:19.017643+0000","last_active":"2024-02-05T07:26:39.245470+0000","last_peered":"2024-02-05T07:26:39.245470+0000","last_clean":"2024-02-05T07:26:39.245470+0000","last_became_active":"2024-02-05T07:25:57.244248+0000","last_became_peered":"2024-02-05T07:25:57.244248+0000","last_unstale":"2024-02-05T07:26:39.245470+0000","last_undegraded":"2024-02-05T07:26:39.245470+0000","last_fullsized":"2024-02-05T07:26:39.245470+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:26:19.017521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:26:19.017521+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:25.540812+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"244'7068","reported_seq":7148,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:40.239441+0000","last_change":"2024-02-05T07:25:16.685108+0000","last_active":"2024-02-05T07:26:40.239441+0000","last_peered":"2024-02-05T07:26:40.239441+0000","last_clean":"2024-02-05T07:26:40.239441+0000","last_became_active":"2024-02-05T07:25:16.684816+0000","last_became_peered":"2024-02-05T07:25:16.684816+0000","last_unstale":"2024-02-05T07:26:40.239441+0000","last_undegraded":"2024-02-05T07:26:40.239441+0000","last_fullsized":"2024-02-05T07:26:40.239441+0000","mapping_epoch":209,"log_start":"223'2700","ondisk_log_start":"223'2700","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4368,"log_dups_size":0,"ondisk_log_size":4368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463208448,"num_objects":7068,"num_object_clones":0,"num_object_copies":21204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7068,"num_write_kb":452352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":135,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.237723+0000","last_change":"2024-02-05T07:25:30.477223+0000","last_active":"2024-02-05T07:26:39.237723+0000","last_peered":"2024-02-05T07:26:39.237723+0000","last_clean":"2024-02-05T07:26:39.237723+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:26:39.237723+0000","last_undegraded":"2024-02-05T07:26:39.237723+0000","last_fullsized":"2024-02-05T07:26:39.237723+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:30.477135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:30.477135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:54.617889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"232'31","reported_seq":174,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.246979+0000","last_change":"2024-02-05T07:25:59.225828+0000","last_active":"2024-02-05T07:26:39.246979+0000","last_peered":"2024-02-05T07:26:39.246979+0000","last_clean":"2024-02-05T07:25:57.239446+0000","last_became_active":"2024-02-05T07:25:59.225828+0000","last_became_peered":"2024-02-05T07:25:59.225828+0000","last_unstale":"2024-02-05T07:26:39.246979+0000","last_undegraded":"2024-02-05T07:25:59.222965+0000","last_fullsized":"2024-02-05T07:25:59.222842+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-02-05T07:23:27.310935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:27.310935+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:40.095474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":392,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.236449+0000","last_change":"2024-02-05T07:25:59.229812+0000","last_active":"2024-02-05T07:26:39.236449+0000","last_peered":"2024-02-05T07:26:39.236449+0000","last_clean":"2024-02-05T07:25:57.231952+0000","last_became_active":"2024-02-05T07:25:59.229812+0000","last_became_peered":"2024-02-05T07:25:59.229812+0000","last_unstale":"2024-02-05T07:26:39.236449+0000","last_undegraded":"2024-02-05T07:26:39.236449+0000","last_fullsized":"2024-02-05T07:25:59.217783+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:53.185503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:53.185503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:06.879660+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":290,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.244020+0000","last_change":"2024-02-05T07:25:59.243163+0000","last_active":"2024-02-05T07:26:39.244020+0000","last_peered":"2024-02-05T07:26:39.244020+0000","last_clean":"2024-02-05T07:25:57.243065+0000","last_became_active":"2024-02-05T07:25:59.243163+0000","last_became_peered":"2024-02-05T07:25:59.243163+0000","last_unstale":"2024-02-05T07:26:39.244020+0000","last_undegraded":"2024-02-05T07:25:59.219001+0000","last_fullsized":"2024-02-05T07:25:59.218871+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:23:29.340876+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:20:38.993688+0000","last_clean_scrub_stamp":"2024-02-05T07:23:29.340876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:42.576368+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":186,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.236924+0000","last_change":"2024-02-05T07:25:59.236059+0000","last_active":"2024-02-05T07:26:39.236924+0000","last_peered":"2024-02-05T07:26:39.236924+0000","last_clean":"2024-02-05T07:25:57.230357+0000","last_became_active":"2024-02-05T07:25:59.236059+0000","last_became_peered":"2024-02-05T07:25:59.236059+0000","last_unstale":"2024-02-05T07:26:39.236924+0000","last_undegraded":"2024-02-05T07:26:39.236924+0000","last_fullsized":"2024-02-05T07:25:59.220810+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:05.412661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:05.412661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:41.264527+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"244'7186","reported_seq":7269,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:40.284884+0000","last_change":"2024-02-05T07:25:59.226709+0000","last_active":"2024-02-05T07:26:40.284884+0000","last_peered":"2024-02-05T07:26:40.284884+0000","last_clean":"2024-02-05T07:25:58.271170+0000","last_became_active":"2024-02-05T07:25:59.226709+0000","last_became_peered":"2024-02-05T07:25:59.226709+0000","last_unstale":"2024-02-05T07:26:40.284884+0000","last_undegraded":"2024-02-05T07:25:59.219937+0000","last_fullsized":"2024-02-05T07:25:59.219797+0000","mapping_epoch":231,"log_start":"224'3000","ondisk_log_start":"224'3000","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470941696,"num_objects":7186,"num_object_clones":0,"num_object_copies":21558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7186,"num_write_kb":459904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":7186}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":169,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.237011+0000","last_change":"2024-02-05T07:25:44.515426+0000","last_active":"2024-02-05T07:26:39.237011+0000","last_peered":"2024-02-05T07:26:39.237011+0000","last_clean":"2024-02-05T07:26:39.237011+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:26:39.237011+0000","last_undegraded":"2024-02-05T07:26:39.237011+0000","last_fullsized":"2024-02-05T07:26:39.237011+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:25:44.515297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:44.515297+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:03.223479+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":37,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.234868+0000","last_change":"2024-02-05T07:25:59.243659+0000","last_active":"2024-02-05T07:26:39.234868+0000","last_peered":"2024-02-05T07:26:39.234868+0000","last_clean":"2024-02-05T07:18:48.410364+0000","last_became_active":"2024-02-05T07:25:59.243659+0000","last_became_peered":"2024-02-05T07:25:59.243659+0000","last_unstale":"2024-02-05T07:26:39.234868+0000","last_undegraded":"2024-02-05T07:26:39.234868+0000","last_fullsized":"2024-02-05T07:25:59.219085+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:38.539777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:38.539777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:02.139064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":387,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.252394+0000","last_change":"2024-02-05T07:25:43.842981+0000","last_active":"2024-02-05T07:26:39.252394+0000","last_peered":"2024-02-05T07:26:39.252394+0000","last_clean":"2024-02-05T07:26:39.252394+0000","last_became_active":"2024-02-05T07:23:47.619951+0000","last_became_peered":"2024-02-05T07:23:47.619951+0000","last_unstale":"2024-02-05T07:26:39.252394+0000","last_undegraded":"2024-02-05T07:26:39.252394+0000","last_fullsized":"2024-02-05T07:26:39.252394+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:43.842850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:43.842850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:47.618148+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":134,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.242890+0000","last_change":"2024-02-05T07:26:30.076326+0000","last_active":"2024-02-05T07:26:39.242890+0000","last_peered":"2024-02-05T07:26:39.242890+0000","last_clean":"2024-02-05T07:26:39.242890+0000","last_became_active":"2024-02-05T07:24:34.235899+0000","last_became_peered":"2024-02-05T07:24:34.235899+0000","last_unstale":"2024-02-05T07:26:39.242890+0000","last_undegraded":"2024-02-05T07:26:39.242890+0000","last_fullsized":"2024-02-05T07:26:39.242890+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:30.076226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:30.076226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:31.423632+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"244'7127","reported_seq":7214,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:40.182402+0000","last_change":"2024-02-05T07:25:59.241351+0000","last_active":"2024-02-05T07:26:40.182402+0000","last_peered":"2024-02-05T07:26:40.182402+0000","last_clean":"2024-02-05T07:25:58.252183+0000","last_became_active":"2024-02-05T07:25:59.227518+0000","last_became_peered":"2024-02-05T07:25:59.227518+0000","last_unstale":"2024-02-05T07:26:40.182402+0000","last_undegraded":"2024-02-05T07:25:59.225223+0000","last_fullsized":"2024-02-05T07:25:59.224711+0000","mapping_epoch":231,"log_start":"222'2500","ondisk_log_start":"222'2500","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.059228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467075072,"num_objects":7127,"num_object_clones":0,"num_object_copies":21381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7127,"num_write_kb":456128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":7127}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":83,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.244278+0000","last_change":"2024-02-05T07:25:59.228711+0000","last_active":"2024-02-05T07:26:39.244278+0000","last_peered":"2024-02-05T07:26:39.244278+0000","last_clean":"2024-02-05T07:18:56.828300+0000","last_became_active":"2024-02-05T07:25:59.228711+0000","last_became_peered":"2024-02-05T07:25:59.228711+0000","last_unstale":"2024-02-05T07:26:39.244278+0000","last_undegraded":"2024-02-05T07:25:59.219065+0000","last_fullsized":"2024-02-05T07:25:59.218949+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:24:54.413486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:54.413486+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":384,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.244619+0000","last_change":"2024-02-05T07:25:59.238635+0000","last_active":"2024-02-05T07:26:39.244619+0000","last_peered":"2024-02-05T07:26:39.244619+0000","last_clean":"2024-02-05T07:25:57.240156+0000","last_became_active":"2024-02-05T07:25:59.238635+0000","last_became_peered":"2024-02-05T07:25:59.238635+0000","last_unstale":"2024-02-05T07:26:39.244619+0000","last_undegraded":"2024-02-05T07:26:39.244619+0000","last_fullsized":"2024-02-05T07:25:59.228211+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:41.444193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:41.444193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":381,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.249348+0000","last_change":"2024-02-05T07:26:22.018694+0000","last_active":"2024-02-05T07:26:39.249348+0000","last_peered":"2024-02-05T07:26:39.249348+0000","last_clean":"2024-02-05T07:26:39.249348+0000","last_became_active":"2024-02-05T07:18:53.464867+0000","last_became_peered":"2024-02-05T07:18:53.464867+0000","last_unstale":"2024-02-05T07:26:39.249348+0000","last_undegraded":"2024-02-05T07:26:39.249348+0000","last_fullsized":"2024-02-05T07:26:39.249348+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:22.018572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:22.018572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:43.229426+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":378,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.244373+0000","last_change":"2024-02-05T07:25:59.227643+0000","last_active":"2024-02-05T07:26:39.244373+0000","last_peered":"2024-02-05T07:26:39.244373+0000","last_clean":"2024-02-05T07:25:57.239079+0000","last_became_active":"2024-02-05T07:25:59.227643+0000","last_became_peered":"2024-02-05T07:25:59.227643+0000","last_unstale":"2024-02-05T07:26:39.244373+0000","last_undegraded":"2024-02-05T07:26:39.244373+0000","last_fullsized":"2024-02-05T07:25:59.224797+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:48.036033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:48.036033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.059121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"244'7283","reported_seq":7365,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:40.178791+0000","last_change":"2024-02-05T07:25:59.239758+0000","last_active":"2024-02-05T07:26:40.178791+0000","last_peered":"2024-02-05T07:26:40.178791+0000","last_clean":"2024-02-05T07:25:58.207147+0000","last_became_active":"2024-02-05T07:25:59.239758+0000","last_became_peered":"2024-02-05T07:25:59.239758+0000","last_unstale":"2024-02-05T07:26:40.178791+0000","last_undegraded":"2024-02-05T07:25:59.230463+0000","last_fullsized":"2024-02-05T07:25:59.230347+0000","mapping_epoch":231,"log_start":"224'3100","ondisk_log_start":"224'3100","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477298688,"num_objects":7283,"num_object_clones":0,"num_object_copies":21849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7283,"num_write_kb":466112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":7283}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":389,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.245282+0000","last_change":"2024-02-05T07:26:26.018740+0000","last_active":"2024-02-05T07:26:39.245282+0000","last_peered":"2024-02-05T07:26:39.245282+0000","last_clean":"2024-02-05T07:26:39.245282+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:26:39.245282+0000","last_undegraded":"2024-02-05T07:26:39.245282+0000","last_fullsized":"2024-02-05T07:26:39.245282+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:26.018629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:26.018629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:49.532442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":447,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.236938+0000","last_change":"2024-02-05T07:25:59.232068+0000","last_active":"2024-02-05T07:26:39.236938+0000","last_peered":"2024-02-05T07:26:39.236938+0000","last_clean":"2024-02-05T07:25:57.230804+0000","last_became_active":"2024-02-05T07:25:59.232068+0000","last_became_peered":"2024-02-05T07:25:59.232068+0000","last_unstale":"2024-02-05T07:26:39.236938+0000","last_undegraded":"2024-02-05T07:25:59.217794+0000","last_fullsized":"2024-02-05T07:25:59.217666+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:25:32.094000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:32.094000+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:01.898325+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":389,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.245013+0000","last_change":"2024-02-05T07:26:24.102911+0000","last_active":"2024-02-05T07:26:39.245013+0000","last_peered":"2024-02-05T07:26:39.245013+0000","last_clean":"2024-02-05T07:26:39.245013+0000","last_became_active":"2024-02-05T07:18:49.435621+0000","last_became_peered":"2024-02-05T07:18:49.435621+0000","last_unstale":"2024-02-05T07:26:39.245013+0000","last_undegraded":"2024-02-05T07:26:39.245013+0000","last_fullsized":"2024-02-05T07:26:39.245013+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:24.102597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.102597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:53.808064+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":186,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.243945+0000","last_change":"2024-02-05T07:25:59.229405+0000","last_active":"2024-02-05T07:26:39.243945+0000","last_peered":"2024-02-05T07:26:39.243945+0000","last_clean":"2024-02-05T07:25:57.239776+0000","last_became_active":"2024-02-05T07:25:59.229405+0000","last_became_peered":"2024-02-05T07:25:59.229405+0000","last_unstale":"2024-02-05T07:26:39.243945+0000","last_undegraded":"2024-02-05T07:26:39.243945+0000","last_fullsized":"2024-02-05T07:25:59.222087+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:42.108043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:42.108043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.059084+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"244'7145","reported_seq":7225,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:40.160404+0000","last_change":"2024-02-05T07:25:16.486080+0000","last_active":"2024-02-05T07:26:40.160404+0000","last_peered":"2024-02-05T07:26:40.160404+0000","last_clean":"2024-02-05T07:26:40.160404+0000","last_became_active":"2024-02-05T07:25:16.485792+0000","last_became_peered":"2024-02-05T07:25:16.485792+0000","last_unstale":"2024-02-05T07:26:40.160404+0000","last_undegraded":"2024-02-05T07:26:40.160404+0000","last_fullsized":"2024-02-05T07:26:40.160404+0000","mapping_epoch":209,"log_start":"223'2900","ondisk_log_start":"223'2900","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4245,"log_dups_size":0,"ondisk_log_size":4245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468254720,"num_objects":7145,"num_object_clones":0,"num_object_copies":21435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7145,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"232'10","reported_seq":481,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.243521+0000","last_change":"2024-02-05T07:25:59.233152+0000","last_active":"2024-02-05T07:26:39.243521+0000","last_peered":"2024-02-05T07:26:39.243521+0000","last_clean":"2024-02-05T07:25:57.241390+0000","last_became_active":"2024-02-05T07:25:59.233152+0000","last_became_peered":"2024-02-05T07:25:59.233152+0000","last_unstale":"2024-02-05T07:26:39.243521+0000","last_undegraded":"2024-02-05T07:25:59.221022+0000","last_fullsized":"2024-02-05T07:25:59.220901+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-02-05T07:21:26.505161+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:21:26.505161+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:40.079886+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":405,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.239215+0000","last_change":"2024-02-05T07:25:59.236665+0000","last_active":"2024-02-05T07:26:39.239215+0000","last_peered":"2024-02-05T07:26:39.239215+0000","last_clean":"2024-02-05T07:25:57.238771+0000","last_became_active":"2024-02-05T07:25:59.236665+0000","last_became_peered":"2024-02-05T07:25:59.236665+0000","last_unstale":"2024-02-05T07:26:39.239215+0000","last_undegraded":"2024-02-05T07:25:59.220638+0000","last_fullsized":"2024-02-05T07:25:59.220531+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:21:32.875346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:32.875346+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":401,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.244855+0000","last_change":"2024-02-05T07:26:37.022248+0000","last_active":"2024-02-05T07:26:39.244855+0000","last_peered":"2024-02-05T07:26:39.244855+0000","last_clean":"2024-02-05T07:26:39.244855+0000","last_became_active":"2024-02-05T07:18:49.437663+0000","last_became_peered":"2024-02-05T07:18:49.437663+0000","last_unstale":"2024-02-05T07:26:39.244855+0000","last_undegraded":"2024-02-05T07:26:39.244855+0000","last_fullsized":"2024-02-05T07:26:39.244855+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:26:37.022129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:37.022129+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:46.252824+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":206,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.243171+0000","last_change":"2024-02-05T07:25:59.237760+0000","last_active":"2024-02-05T07:26:39.243171+0000","last_peered":"2024-02-05T07:26:39.243171+0000","last_clean":"2024-02-05T07:18:54.446999+0000","last_became_active":"2024-02-05T07:25:59.237760+0000","last_became_peered":"2024-02-05T07:25:59.237760+0000","last_unstale":"2024-02-05T07:26:39.243171+0000","last_undegraded":"2024-02-05T07:26:39.243171+0000","last_fullsized":"2024-02-05T07:25:59.222589+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:34.561259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:37.032056+0000","last_clean_scrub_stamp":"2024-02-05T07:25:34.561259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:39.304228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":227,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.244951+0000","last_change":"2024-02-05T07:26:36.068660+0000","last_active":"2024-02-05T07:26:39.244951+0000","last_peered":"2024-02-05T07:26:39.244951+0000","last_clean":"2024-02-05T07:26:39.244951+0000","last_became_active":"2024-02-05T07:23:47.607589+0000","last_became_peered":"2024-02-05T07:23:47.607589+0000","last_unstale":"2024-02-05T07:26:39.244951+0000","last_undegraded":"2024-02-05T07:26:39.244951+0000","last_fullsized":"2024-02-05T07:26:39.244951+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-02-05T07:26:36.068507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.068507+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:40.474967+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":41,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.239540+0000","last_change":"2024-02-05T07:25:59.224180+0000","last_active":"2024-02-05T07:26:39.239540+0000","last_peered":"2024-02-05T07:26:39.239540+0000","last_clean":"2024-02-05T07:18:49.436250+0000","last_became_active":"2024-02-05T07:25:59.224180+0000","last_became_peered":"2024-02-05T07:25:59.224180+0000","last_unstale":"2024-02-05T07:26:39.239540+0000","last_undegraded":"2024-02-05T07:25:59.217123+0000","last_fullsized":"2024-02-05T07:25:59.217010+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:24:49.359534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.359534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:44.129284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":382,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.239320+0000","last_change":"2024-02-05T07:25:59.221968+0000","last_active":"2024-02-05T07:26:39.239320+0000","last_peered":"2024-02-05T07:26:39.239320+0000","last_clean":"2024-02-05T07:25:57.234986+0000","last_became_active":"2024-02-05T07:25:59.221968+0000","last_became_peered":"2024-02-05T07:25:59.221968+0000","last_unstale":"2024-02-05T07:26:39.239320+0000","last_undegraded":"2024-02-05T07:26:39.239320+0000","last_fullsized":"2024-02-05T07:25:59.219603+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:28.567844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:28.567844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:39.854931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":372,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.242669+0000","last_change":"2024-02-05T07:25:59.235975+0000","last_active":"2024-02-05T07:26:39.242669+0000","last_peered":"2024-02-05T07:26:39.242669+0000","last_clean":"2024-02-05T07:25:57.239964+0000","last_became_active":"2024-02-05T07:25:59.235975+0000","last_became_peered":"2024-02-05T07:25:59.235975+0000","last_unstale":"2024-02-05T07:26:39.242669+0000","last_undegraded":"2024-02-05T07:26:39.242669+0000","last_fullsized":"2024-02-05T07:25:59.226429+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:21:44.435826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":169,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.245182+0000","last_change":"2024-02-05T07:26:28.038064+0000","last_active":"2024-02-05T07:26:39.245182+0000","last_peered":"2024-02-05T07:26:39.245182+0000","last_clean":"2024-02-05T07:26:39.245182+0000","last_became_active":"2024-02-05T07:24:34.241945+0000","last_became_peered":"2024-02-05T07:24:34.241945+0000","last_unstale":"2024-02-05T07:26:39.245182+0000","last_undegraded":"2024-02-05T07:26:39.245182+0000","last_fullsized":"2024-02-05T07:26:39.245182+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:26:28.037945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:28.037945+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:51.994543+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":392,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.242308+0000","last_change":"2024-02-05T07:25:59.234625+0000","last_active":"2024-02-05T07:26:39.242308+0000","last_peered":"2024-02-05T07:26:39.242308+0000","last_clean":"2024-02-05T07:25:57.241166+0000","last_became_active":"2024-02-05T07:25:59.234625+0000","last_became_peered":"2024-02-05T07:25:59.234625+0000","last_unstale":"2024-02-05T07:26:39.242308+0000","last_undegraded":"2024-02-05T07:26:39.242308+0000","last_fullsized":"2024-02-05T07:25:59.221239+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:51.811993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:25:51.811993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:18.549406+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":183,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.241532+0000","last_change":"2024-02-05T07:24:47.676565+0000","last_active":"2024-02-05T07:26:39.241532+0000","last_peered":"2024-02-05T07:26:39.241532+0000","last_clean":"2024-02-05T07:26:39.241532+0000","last_became_active":"2024-02-05T07:23:47.606266+0000","last_became_peered":"2024-02-05T07:23:47.606266+0000","last_unstale":"2024-02-05T07:26:39.241532+0000","last_undegraded":"2024-02-05T07:26:39.241532+0000","last_fullsized":"2024-02-05T07:26:39.241532+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:47.676484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:47.676484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":377,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.246983+0000","last_change":"2024-02-05T07:25:47.862699+0000","last_active":"2024-02-05T07:26:39.246983+0000","last_peered":"2024-02-05T07:26:39.246983+0000","last_clean":"2024-02-05T07:26:39.246983+0000","last_became_active":"2024-02-05T07:18:55.469044+0000","last_became_peered":"2024-02-05T07:18:55.469044+0000","last_unstale":"2024-02-05T07:26:39.246983+0000","last_undegraded":"2024-02-05T07:26:39.246983+0000","last_fullsized":"2024-02-05T07:26:39.246983+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.862362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:08.622645+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"222'13","reported_seq":424,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.238831+0000","last_change":"2024-02-05T07:25:59.233945+0000","last_active":"2024-02-05T07:26:39.238831+0000","last_peered":"2024-02-05T07:26:39.238831+0000","last_clean":"2024-02-05T07:25:57.237017+0000","last_became_active":"2024-02-05T07:25:59.233945+0000","last_became_peered":"2024-02-05T07:25:59.233945+0000","last_unstale":"2024-02-05T07:26:39.238831+0000","last_undegraded":"2024-02-05T07:25:59.220506+0000","last_fullsized":"2024-02-05T07:25:59.220409+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'13","last_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:25:46.891183+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:57.894604+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":341,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.242971+0000","last_change":"2024-02-05T07:24:49.035482+0000","last_active":"2024-02-05T07:26:39.242971+0000","last_peered":"2024-02-05T07:26:39.242971+0000","last_clean":"2024-02-05T07:26:39.242971+0000","last_became_active":"2024-02-05T07:24:34.230825+0000","last_became_peered":"2024-02-05T07:24:34.230825+0000","last_unstale":"2024-02-05T07:26:39.242971+0000","last_undegraded":"2024-02-05T07:26:39.242971+0000","last_fullsized":"2024-02-05T07:26:39.242971+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:49.035385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.035385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.059109+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"164'9","reported_seq":247,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.243377+0000","last_change":"2024-02-05T07:26:35.092187+0000","last_active":"2024-02-05T07:26:39.243377+0000","last_peered":"2024-02-05T07:26:39.243377+0000","last_clean":"2024-02-05T07:26:39.243377+0000","last_became_active":"2024-02-05T07:23:47.605296+0000","last_became_peered":"2024-02-05T07:23:47.605296+0000","last_unstale":"2024-02-05T07:26:39.243377+0000","last_undegraded":"2024-02-05T07:26:39.243377+0000","last_fullsized":"2024-02-05T07:26:39.243377+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:26:35.091894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.091894+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:37.256393+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":377,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.241716+0000","last_change":"2024-02-05T07:26:25.323633+0000","last_active":"2024-02-05T07:26:39.241716+0000","last_peered":"2024-02-05T07:26:39.241716+0000","last_clean":"2024-02-05T07:26:39.241716+0000","last_became_active":"2024-02-05T07:18:55.467044+0000","last_became_peered":"2024-02-05T07:18:55.467044+0000","last_unstale":"2024-02-05T07:26:39.241716+0000","last_undegraded":"2024-02-05T07:26:39.241716+0000","last_fullsized":"2024-02-05T07:26:39.241716+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:25.323384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:25.323384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:54.639966+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"113'15","reported_seq":83,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.239654+0000","last_change":"2024-02-05T07:25:47.928406+0000","last_active":"2024-02-05T07:26:39.239654+0000","last_peered":"2024-02-05T07:26:39.239654+0000","last_clean":"2024-02-05T07:26:39.239654+0000","last_became_active":"2024-02-05T07:25:30.262689+0000","last_became_peered":"2024-02-05T07:25:30.262689+0000","last_unstale":"2024-02-05T07:26:39.239654+0000","last_undegraded":"2024-02-05T07:26:39.239654+0000","last_fullsized":"2024-02-05T07:26:39.239654+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'15","last_scrub_stamp":"2024-02-05T07:25:47.927943+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.927943+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:53.815841+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":381,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.243593+0000","last_change":"2024-02-05T07:24:53.108878+0000","last_active":"2024-02-05T07:26:39.243593+0000","last_peered":"2024-02-05T07:26:39.243593+0000","last_clean":"2024-02-05T07:26:39.243593+0000","last_became_active":"2024-02-05T07:18:49.436131+0000","last_became_peered":"2024-02-05T07:18:49.436131+0000","last_unstale":"2024-02-05T07:26:39.243593+0000","last_undegraded":"2024-02-05T07:26:39.243593+0000","last_fullsized":"2024-02-05T07:26:39.243593+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:53.108801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:53.108801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.059145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"217'9","reported_seq":96,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.246566+0000","last_change":"2024-02-05T07:25:31.727799+0000","last_active":"2024-02-05T07:26:39.246566+0000","last_peered":"2024-02-05T07:26:39.246566+0000","last_clean":"2024-02-05T07:26:39.246566+0000","last_became_active":"2024-02-05T07:25:27.246616+0000","last_became_peered":"2024-02-05T07:25:27.246616+0000","last_unstale":"2024-02-05T07:26:39.246566+0000","last_undegraded":"2024-02-05T07:26:39.246566+0000","last_fullsized":"2024-02-05T07:26:39.246566+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9","last_scrub_stamp":"2024-02-05T07:25:31.727658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:31.727658+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:44.449404+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":321,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.244856+0000","last_change":"2024-02-05T07:24:34.242269+0000","last_active":"2024-02-05T07:26:39.244856+0000","last_peered":"2024-02-05T07:26:39.244856+0000","last_clean":"2024-02-05T07:26:39.244856+0000","last_became_active":"2024-02-05T07:24:34.238479+0000","last_became_peered":"2024-02-05T07:24:34.238479+0000","last_unstale":"2024-02-05T07:26:39.244856+0000","last_undegraded":"2024-02-05T07:26:39.244856+0000","last_fullsized":"2024-02-05T07:26:39.244856+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:16.156709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:16.156709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.059029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":425,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.237864+0000","last_change":"2024-02-05T07:25:59.232617+0000","last_active":"2024-02-05T07:26:39.237864+0000","last_peered":"2024-02-05T07:26:39.237864+0000","last_clean":"2024-02-05T07:25:57.232059+0000","last_became_active":"2024-02-05T07:25:59.232617+0000","last_became_peered":"2024-02-05T07:25:59.232617+0000","last_unstale":"2024-02-05T07:26:39.237864+0000","last_undegraded":"2024-02-05T07:25:59.218091+0000","last_fullsized":"2024-02-05T07:25:59.217973+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:25:42.192387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.192387+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:09.534064+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":384,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.237146+0000","last_change":"2024-02-05T07:25:59.236021+0000","last_active":"2024-02-05T07:26:39.237146+0000","last_peered":"2024-02-05T07:26:39.237146+0000","last_clean":"2024-02-05T07:25:57.230023+0000","last_became_active":"2024-02-05T07:25:59.236021+0000","last_became_peered":"2024-02-05T07:25:59.236021+0000","last_unstale":"2024-02-05T07:26:39.237146+0000","last_undegraded":"2024-02-05T07:26:39.237146+0000","last_fullsized":"2024-02-05T07:25:59.217540+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:23:23.435135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:41.264481+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":388,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.244588+0000","last_change":"2024-02-05T07:26:18.555960+0000","last_active":"2024-02-05T07:26:39.244588+0000","last_peered":"2024-02-05T07:26:39.244588+0000","last_clean":"2024-02-05T07:26:39.244588+0000","last_became_active":"2024-02-05T07:23:47.594030+0000","last_became_peered":"2024-02-05T07:23:47.594030+0000","last_unstale":"2024-02-05T07:26:39.244588+0000","last_undegraded":"2024-02-05T07:26:39.244588+0000","last_fullsized":"2024-02-05T07:26:39.244588+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.555824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.555824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:40.860535+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":144,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.245850+0000","last_change":"2024-02-05T07:26:35.120630+0000","last_active":"2024-02-05T07:26:39.245850+0000","last_peered":"2024-02-05T07:26:39.245850+0000","last_clean":"2024-02-05T07:26:39.245850+0000","last_became_active":"2024-02-05T07:24:34.229238+0000","last_became_peered":"2024-02-05T07:24:34.229238+0000","last_unstale":"2024-02-05T07:26:39.245850+0000","last_undegraded":"2024-02-05T07:26:39.245850+0000","last_fullsized":"2024-02-05T07:26:39.245850+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.120512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:54.865770+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":368,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.239267+0000","last_change":"2024-02-05T07:25:59.237717+0000","last_active":"2024-02-05T07:26:39.239267+0000","last_peered":"2024-02-05T07:26:39.239267+0000","last_clean":"2024-02-05T07:25:57.235913+0000","last_became_active":"2024-02-05T07:25:59.237717+0000","last_became_peered":"2024-02-05T07:25:59.237717+0000","last_unstale":"2024-02-05T07:26:39.239267+0000","last_undegraded":"2024-02-05T07:25:59.223130+0000","last_fullsized":"2024-02-05T07:25:59.223035+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:25:50.880674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:50.880674+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:08.841868+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":387,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.244448+0000","last_change":"2024-02-05T07:25:36.822948+0000","last_active":"2024-02-05T07:26:39.244448+0000","last_peered":"2024-02-05T07:26:39.244448+0000","last_clean":"2024-02-05T07:26:39.244448+0000","last_became_active":"2024-02-05T07:23:47.616029+0000","last_became_peered":"2024-02-05T07:23:47.616029+0000","last_unstale":"2024-02-05T07:26:39.244448+0000","last_undegraded":"2024-02-05T07:26:39.244448+0000","last_fullsized":"2024-02-05T07:26:39.244448+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:36.822835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:36.822835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:01.214849+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":136,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.237123+0000","last_change":"2024-02-05T07:26:31.506870+0000","last_active":"2024-02-05T07:26:39.237123+0000","last_peered":"2024-02-05T07:26:39.237123+0000","last_clean":"2024-02-05T07:26:39.237123+0000","last_became_active":"2024-02-05T07:24:34.231409+0000","last_became_peered":"2024-02-05T07:24:34.231409+0000","last_unstale":"2024-02-05T07:26:39.237123+0000","last_undegraded":"2024-02-05T07:26:39.237123+0000","last_fullsized":"2024-02-05T07:26:39.237123+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:26:31.506641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:37.736043+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":134,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.238726+0000","last_change":"2024-02-05T07:26:37.616855+0000","last_active":"2024-02-05T07:26:39.238726+0000","last_peered":"2024-02-05T07:26:39.238726+0000","last_clean":"2024-02-05T07:26:39.238726+0000","last_became_active":"2024-02-05T07:24:34.230006+0000","last_became_peered":"2024-02-05T07:24:34.230006+0000","last_unstale":"2024-02-05T07:26:39.238726+0000","last_undegraded":"2024-02-05T07:26:39.238726+0000","last_fullsized":"2024-02-05T07:26:39.238726+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:37.616737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:26:37.616737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:28:06.054974+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":236,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.243213+0000","last_change":"2024-02-05T07:25:59.243954+0000","last_active":"2024-02-05T07:26:39.243213+0000","last_peered":"2024-02-05T07:26:39.243213+0000","last_clean":"2024-02-05T07:25:57.241307+0000","last_became_active":"2024-02-05T07:25:59.243954+0000","last_became_peered":"2024-02-05T07:25:59.243954+0000","last_unstale":"2024-02-05T07:26:39.243213+0000","last_undegraded":"2024-02-05T07:25:59.224127+0000","last_fullsized":"2024-02-05T07:25:59.224002+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:25:42.761130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.761130+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:12.192141+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":389,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.239389+0000","last_change":"2024-02-05T07:26:16.644982+0000","last_active":"2024-02-05T07:26:39.239389+0000","last_peered":"2024-02-05T07:26:39.239389+0000","last_clean":"2024-02-05T07:26:39.239389+0000","last_became_active":"2024-02-05T07:18:49.434560+0000","last_became_peered":"2024-02-05T07:18:49.434560+0000","last_unstale":"2024-02-05T07:26:39.239389+0000","last_undegraded":"2024-02-05T07:26:39.239389+0000","last_fullsized":"2024-02-05T07:26:39.239389+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:16.644911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:16.644911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:31.878581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":384,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.242283+0000","last_change":"2024-02-05T07:25:59.235562+0000","last_active":"2024-02-05T07:26:39.242283+0000","last_peered":"2024-02-05T07:26:39.242283+0000","last_clean":"2024-02-05T07:25:57.239615+0000","last_became_active":"2024-02-05T07:25:59.235562+0000","last_became_peered":"2024-02-05T07:25:59.235562+0000","last_unstale":"2024-02-05T07:26:39.242283+0000","last_undegraded":"2024-02-05T07:26:39.242283+0000","last_fullsized":"2024-02-05T07:25:59.225564+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:34.862131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_clean_scrub_stamp":"2024-02-05T07:25:34.862131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:46.095982+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":37,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.246756+0000","last_change":"2024-02-05T07:25:59.225826+0000","last_active":"2024-02-05T07:26:39.246756+0000","last_peered":"2024-02-05T07:26:39.246756+0000","last_clean":"2024-02-05T07:18:54.446999+0000","last_became_active":"2024-02-05T07:25:59.225826+0000","last_became_peered":"2024-02-05T07:25:59.225826+0000","last_unstale":"2024-02-05T07:26:39.246756+0000","last_undegraded":"2024-02-05T07:26:39.246756+0000","last_fullsized":"2024-02-05T07:25:59.223172+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:46.423760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:46.423760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:40.095491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":217,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.239541+0000","last_change":"2024-02-05T07:25:59.232083+0000","last_active":"2024-02-05T07:26:39.239541+0000","last_peered":"2024-02-05T07:26:39.239541+0000","last_clean":"2024-02-05T07:25:57.238847+0000","last_became_active":"2024-02-05T07:25:59.232083+0000","last_became_peered":"2024-02-05T07:25:59.232083+0000","last_unstale":"2024-02-05T07:26:39.239541+0000","last_undegraded":"2024-02-05T07:25:59.216538+0000","last_fullsized":"2024-02-05T07:25:59.216428+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-02-05T07:25:31.462787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:31.462787+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:41.282878+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":336,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.239028+0000","last_change":"2024-02-05T07:25:59.221623+0000","last_active":"2024-02-05T07:26:39.239028+0000","last_peered":"2024-02-05T07:26:39.239028+0000","last_clean":"2024-02-05T07:25:57.235863+0000","last_became_active":"2024-02-05T07:25:59.221623+0000","last_became_peered":"2024-02-05T07:25:59.221623+0000","last_unstale":"2024-02-05T07:26:39.239028+0000","last_undegraded":"2024-02-05T07:26:39.239028+0000","last_fullsized":"2024-02-05T07:25:59.218920+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:17.202540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:17.202540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":33,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.235799+0000","last_change":"2024-02-05T07:25:59.229147+0000","last_active":"2024-02-05T07:26:39.235799+0000","last_peered":"2024-02-05T07:26:39.235799+0000","last_clean":"2024-02-05T07:18:52.436288+0000","last_became_active":"2024-02-05T07:25:59.229147+0000","last_became_peered":"2024-02-05T07:25:59.229147+0000","last_unstale":"2024-02-05T07:26:39.235799+0000","last_undegraded":"2024-02-05T07:26:39.235799+0000","last_fullsized":"2024-02-05T07:25:59.216730+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:32.554622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:25:32.554622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:51.563019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":377,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.243271+0000","last_change":"2024-02-05T07:26:27.014044+0000","last_active":"2024-02-05T07:26:39.243271+0000","last_peered":"2024-02-05T07:26:39.243271+0000","last_clean":"2024-02-05T07:26:39.243271+0000","last_became_active":"2024-02-05T07:18:55.468086+0000","last_became_peered":"2024-02-05T07:18:55.468086+0000","last_unstale":"2024-02-05T07:26:39.243271+0000","last_undegraded":"2024-02-05T07:26:39.243271+0000","last_fullsized":"2024-02-05T07:26:39.243271+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:27.013942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:27.013942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:38.382354+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":71,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.241715+0000","last_change":"2024-02-05T07:25:59.237913+0000","last_active":"2024-02-05T07:26:39.241715+0000","last_peered":"2024-02-05T07:26:39.241715+0000","last_clean":"2024-02-05T07:18:56.817839+0000","last_became_active":"2024-02-05T07:25:59.237913+0000","last_became_peered":"2024-02-05T07:25:59.237913+0000","last_unstale":"2024-02-05T07:26:39.241715+0000","last_undegraded":"2024-02-05T07:25:59.220649+0000","last_fullsized":"2024-02-05T07:25:59.220537+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:24:53.457167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:53.457167+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":318,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.239164+0000","last_change":"2024-02-05T07:24:54.528959+0000","last_active":"2024-02-05T07:26:39.239164+0000","last_peered":"2024-02-05T07:26:39.239164+0000","last_clean":"2024-02-05T07:26:39.239164+0000","last_became_active":"2024-02-05T07:24:34.209580+0000","last_became_peered":"2024-02-05T07:24:34.209580+0000","last_unstale":"2024-02-05T07:26:39.239164+0000","last_undegraded":"2024-02-05T07:26:39.239164+0000","last_fullsized":"2024-02-05T07:26:39.239164+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:54.528848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:54.528848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":135,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.245093+0000","last_change":"2024-02-05T07:25:30.943037+0000","last_active":"2024-02-05T07:26:39.245093+0000","last_peered":"2024-02-05T07:26:39.245093+0000","last_clean":"2024-02-05T07:26:39.245093+0000","last_became_active":"2024-02-05T07:24:34.240115+0000","last_became_peered":"2024-02-05T07:24:34.240115+0000","last_unstale":"2024-02-05T07:26:39.245093+0000","last_undegraded":"2024-02-05T07:26:39.245093+0000","last_fullsized":"2024-02-05T07:26:39.245093+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:30.942942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:30.942942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:51.102432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":386,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.237768+0000","last_change":"2024-02-05T07:25:51.548947+0000","last_active":"2024-02-05T07:26:39.237768+0000","last_peered":"2024-02-05T07:26:39.237768+0000","last_clean":"2024-02-05T07:26:39.237768+0000","last_became_active":"2024-02-05T07:18:55.466164+0000","last_became_peered":"2024-02-05T07:18:55.466164+0000","last_unstale":"2024-02-05T07:26:39.237768+0000","last_undegraded":"2024-02-05T07:26:39.237768+0000","last_fullsized":"2024-02-05T07:26:39.237768+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:51.546973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:51.546973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:15.013007+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":169,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.245285+0000","last_change":"2024-02-05T07:24:46.018438+0000","last_active":"2024-02-05T07:26:39.245285+0000","last_peered":"2024-02-05T07:26:39.245285+0000","last_clean":"2024-02-05T07:26:39.245285+0000","last_became_active":"2024-02-05T07:24:34.237056+0000","last_became_peered":"2024-02-05T07:24:34.237056+0000","last_unstale":"2024-02-05T07:26:39.245285+0000","last_undegraded":"2024-02-05T07:26:39.245285+0000","last_fullsized":"2024-02-05T07:26:39.245285+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:24:46.018340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:46.018340+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.059174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":404,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.235191+0000","last_change":"2024-02-05T07:25:59.229605+0000","last_active":"2024-02-05T07:26:39.235191+0000","last_peered":"2024-02-05T07:26:39.235191+0000","last_clean":"2024-02-05T07:25:57.236963+0000","last_became_active":"2024-02-05T07:25:59.229605+0000","last_became_peered":"2024-02-05T07:25:59.229605+0000","last_unstale":"2024-02-05T07:26:39.235191+0000","last_undegraded":"2024-02-05T07:25:59.216317+0000","last_fullsized":"2024-02-05T07:25:59.216195+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:25:42.504403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.504403+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:06.455804+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"232'9","reported_seq":71,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.237292+0000","last_change":"2024-02-05T07:25:59.236281+0000","last_active":"2024-02-05T07:26:39.237292+0000","last_peered":"2024-02-05T07:26:39.237292+0000","last_clean":"2024-02-05T07:18:56.648590+0000","last_became_active":"2024-02-05T07:25:59.236281+0000","last_became_peered":"2024-02-05T07:25:59.236281+0000","last_unstale":"2024-02-05T07:26:39.237292+0000","last_undegraded":"2024-02-05T07:25:59.219241+0000","last_fullsized":"2024-02-05T07:25:59.218638+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:25:31.524507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:31.524507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:47.765018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":379,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.243433+0000","last_change":"2024-02-05T07:26:18.020075+0000","last_active":"2024-02-05T07:26:39.243433+0000","last_peered":"2024-02-05T07:26:39.243433+0000","last_clean":"2024-02-05T07:26:39.243433+0000","last_became_active":"2024-02-05T07:18:55.476071+0000","last_became_peered":"2024-02-05T07:18:55.476071+0000","last_unstale":"2024-02-05T07:26:39.243433+0000","last_undegraded":"2024-02-05T07:26:39.243433+0000","last_fullsized":"2024-02-05T07:26:39.243433+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:26:18.019957+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.019957+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:33.613062+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":240,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.237411+0000","last_change":"2024-02-05T07:25:59.233133+0000","last_active":"2024-02-05T07:26:39.237411+0000","last_peered":"2024-02-05T07:26:39.237411+0000","last_clean":"2024-02-05T07:25:57.236423+0000","last_became_active":"2024-02-05T07:25:59.233133+0000","last_became_peered":"2024-02-05T07:25:59.233133+0000","last_unstale":"2024-02-05T07:26:39.237411+0000","last_undegraded":"2024-02-05T07:25:59.219550+0000","last_fullsized":"2024-02-05T07:25:59.219433+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:24:46.667757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:46.667757+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":72,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.244464+0000","last_change":"2024-02-05T07:26:29.558558+0000","last_active":"2024-02-05T07:26:39.244464+0000","last_peered":"2024-02-05T07:26:39.244464+0000","last_clean":"2024-02-05T07:26:39.244464+0000","last_became_active":"2024-02-05T07:25:30.261037+0000","last_became_peered":"2024-02-05T07:25:30.261037+0000","last_unstale":"2024-02-05T07:26:39.244464+0000","last_undegraded":"2024-02-05T07:26:39.244464+0000","last_fullsized":"2024-02-05T07:26:39.244464+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:29.558353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:29.558353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:40.224760+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":387,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.240640+0000","last_change":"2024-02-05T07:25:34.068446+0000","last_active":"2024-02-05T07:26:39.240640+0000","last_peered":"2024-02-05T07:26:39.240640+0000","last_clean":"2024-02-05T07:26:39.240640+0000","last_became_active":"2024-02-05T07:23:47.605523+0000","last_became_peered":"2024-02-05T07:23:47.605523+0000","last_unstale":"2024-02-05T07:26:39.240640+0000","last_undegraded":"2024-02-05T07:26:39.240640+0000","last_fullsized":"2024-02-05T07:26:39.240640+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:34.068335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:34.068335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:42.963475+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":35,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.238678+0000","last_change":"2024-02-05T07:25:59.234097+0000","last_active":"2024-02-05T07:26:39.238678+0000","last_peered":"2024-02-05T07:26:39.238678+0000","last_clean":"2024-02-05T07:18:54.446999+0000","last_became_active":"2024-02-05T07:25:59.234097+0000","last_became_peered":"2024-02-05T07:25:59.234097+0000","last_unstale":"2024-02-05T07:26:39.238678+0000","last_undegraded":"2024-02-05T07:26:39.238678+0000","last_fullsized":"2024-02-05T07:25:59.220930+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:58.410667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:58.410667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":432,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.246756+0000","last_change":"2024-02-05T07:26:24.030463+0000","last_active":"2024-02-05T07:26:39.246756+0000","last_peered":"2024-02-05T07:26:39.246756+0000","last_clean":"2024-02-05T07:26:39.246756+0000","last_became_active":"2024-02-05T07:23:47.619716+0000","last_became_peered":"2024-02-05T07:23:47.619716+0000","last_unstale":"2024-02-05T07:26:39.246756+0000","last_undegraded":"2024-02-05T07:26:39.246756+0000","last_fullsized":"2024-02-05T07:26:39.246756+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:26:24.030302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.030302+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:38.381589+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":134,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.237745+0000","last_change":"2024-02-05T07:25:33.059237+0000","last_active":"2024-02-05T07:26:39.237745+0000","last_peered":"2024-02-05T07:26:39.237745+0000","last_clean":"2024-02-05T07:26:39.237745+0000","last_became_active":"2024-02-05T07:24:34.234199+0000","last_became_peered":"2024-02-05T07:24:34.234199+0000","last_unstale":"2024-02-05T07:26:39.237745+0000","last_undegraded":"2024-02-05T07:26:39.237745+0000","last_fullsized":"2024-02-05T07:26:39.237745+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:33.059121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:33.059121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:49.169289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":382,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.237603+0000","last_change":"2024-02-05T07:26:21.548894+0000","last_active":"2024-02-05T07:26:39.237603+0000","last_peered":"2024-02-05T07:26:39.237603+0000","last_clean":"2024-02-05T07:26:39.237603+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:26:39.237603+0000","last_undegraded":"2024-02-05T07:26:39.237603+0000","last_fullsized":"2024-02-05T07:26:39.237603+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:21.548785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:21.548785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.889253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":380,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.236719+0000","last_change":"2024-02-05T07:25:59.234649+0000","last_active":"2024-02-05T07:26:39.236719+0000","last_peered":"2024-02-05T07:26:39.236719+0000","last_clean":"2024-02-05T07:25:57.231294+0000","last_became_active":"2024-02-05T07:25:59.234649+0000","last_became_peered":"2024-02-05T07:25:59.234649+0000","last_unstale":"2024-02-05T07:26:39.236719+0000","last_undegraded":"2024-02-05T07:26:39.236719+0000","last_fullsized":"2024-02-05T07:25:59.222220+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:37.074827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:37.074827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:51.428861+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":407,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.237654+0000","last_change":"2024-02-05T07:26:35.253640+0000","last_active":"2024-02-05T07:26:39.237654+0000","last_peered":"2024-02-05T07:26:39.237654+0000","last_clean":"2024-02-05T07:26:39.237654+0000","last_became_active":"2024-02-05T07:23:47.603918+0000","last_became_peered":"2024-02-05T07:23:47.603918+0000","last_unstale":"2024-02-05T07:26:39.237654+0000","last_undegraded":"2024-02-05T07:26:39.237654+0000","last_fullsized":"2024-02-05T07:26:39.237654+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:26:35.253452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.253452+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.453327+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":203,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.240546+0000","last_change":"2024-02-05T07:25:35.059055+0000","last_active":"2024-02-05T07:26:39.240546+0000","last_peered":"2024-02-05T07:26:39.240546+0000","last_clean":"2024-02-05T07:26:39.240546+0000","last_became_active":"2024-02-05T07:23:27.437776+0000","last_became_peered":"2024-02-05T07:23:27.437776+0000","last_unstale":"2024-02-05T07:26:39.240546+0000","last_undegraded":"2024-02-05T07:26:39.240546+0000","last_fullsized":"2024-02-05T07:26:39.240546+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:35.058925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:35.058925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":373,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.243113+0000","last_change":"2024-02-05T07:24:16.310748+0000","last_active":"2024-02-05T07:26:39.243113+0000","last_peered":"2024-02-05T07:26:39.243113+0000","last_clean":"2024-02-05T07:26:39.243113+0000","last_became_active":"2024-02-05T07:18:53.456881+0000","last_became_peered":"2024-02-05T07:18:53.456881+0000","last_unstale":"2024-02-05T07:26:39.243113+0000","last_undegraded":"2024-02-05T07:26:39.243113+0000","last_fullsized":"2024-02-05T07:26:39.243113+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:16.310666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:16.310666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.059001+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":375,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.239852+0000","last_change":"2024-02-05T07:25:43.606335+0000","last_active":"2024-02-05T07:26:39.239852+0000","last_peered":"2024-02-05T07:26:39.239852+0000","last_clean":"2024-02-05T07:26:39.239852+0000","last_became_active":"2024-02-05T07:23:47.596885+0000","last_became_peered":"2024-02-05T07:23:47.596885+0000","last_unstale":"2024-02-05T07:26:39.239852+0000","last_undegraded":"2024-02-05T07:26:39.239852+0000","last_fullsized":"2024-02-05T07:26:39.239852+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:43.606209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:43.606209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:50.857608+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"89'13","reported_seq":421,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.238706+0000","last_change":"2024-02-05T07:25:26.134223+0000","last_active":"2024-02-05T07:26:39.238706+0000","last_peered":"2024-02-05T07:26:39.238706+0000","last_clean":"2024-02-05T07:26:39.238706+0000","last_became_active":"2024-02-05T07:23:47.618422+0000","last_became_peered":"2024-02-05T07:23:47.618422+0000","last_unstale":"2024-02-05T07:26:39.238706+0000","last_undegraded":"2024-02-05T07:26:39.238706+0000","last_fullsized":"2024-02-05T07:26:39.238706+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-05T07:25:26.134110+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:25:26.134110+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:48.192200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":206,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.240750+0000","last_change":"2024-02-05T07:25:59.239400+0000","last_active":"2024-02-05T07:26:39.240750+0000","last_peered":"2024-02-05T07:26:39.240750+0000","last_clean":"2024-02-05T07:25:57.231962+0000","last_became_active":"2024-02-05T07:25:59.239400+0000","last_became_peered":"2024-02-05T07:25:59.239400+0000","last_unstale":"2024-02-05T07:26:39.240750+0000","last_undegraded":"2024-02-05T07:26:39.240750+0000","last_fullsized":"2024-02-05T07:25:59.230663+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:28.169335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:28.169335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":373,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.238553+0000","last_change":"2024-02-05T07:25:42.577644+0000","last_active":"2024-02-05T07:26:39.238553+0000","last_peered":"2024-02-05T07:26:39.238553+0000","last_clean":"2024-02-05T07:26:39.238553+0000","last_became_active":"2024-02-05T07:18:53.464520+0000","last_became_peered":"2024-02-05T07:18:53.464520+0000","last_unstale":"2024-02-05T07:26:39.238553+0000","last_undegraded":"2024-02-05T07:26:39.238553+0000","last_fullsized":"2024-02-05T07:26:39.238553+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:42.577533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.577533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:55.315443+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":134,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.238916+0000","last_change":"2024-02-05T07:24:49.612194+0000","last_active":"2024-02-05T07:26:39.238916+0000","last_peered":"2024-02-05T07:26:39.238916+0000","last_clean":"2024-02-05T07:26:39.238916+0000","last_became_active":"2024-02-05T07:24:34.230537+0000","last_became_peered":"2024-02-05T07:24:34.230537+0000","last_unstale":"2024-02-05T07:26:39.238916+0000","last_undegraded":"2024-02-05T07:26:39.238916+0000","last_fullsized":"2024-02-05T07:26:39.238916+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:49.612116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.612116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":243,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.246163+0000","last_change":"2024-02-05T07:25:38.772967+0000","last_active":"2024-02-05T07:26:39.246163+0000","last_peered":"2024-02-05T07:26:39.246163+0000","last_clean":"2024-02-05T07:26:39.246163+0000","last_became_active":"2024-02-05T07:23:47.621081+0000","last_became_peered":"2024-02-05T07:23:47.621081+0000","last_unstale":"2024-02-05T07:26:39.246163+0000","last_undegraded":"2024-02-05T07:26:39.246163+0000","last_fullsized":"2024-02-05T07:26:39.246163+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:25:38.772812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:38.772812+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:52.652739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":35,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.248652+0000","last_change":"2024-02-05T07:25:59.225883+0000","last_active":"2024-02-05T07:26:39.248652+0000","last_peered":"2024-02-05T07:26:39.248652+0000","last_clean":"2024-02-05T07:18:48.410364+0000","last_became_active":"2024-02-05T07:25:59.225883+0000","last_became_peered":"2024-02-05T07:25:59.225883+0000","last_unstale":"2024-02-05T07:26:39.248652+0000","last_undegraded":"2024-02-05T07:26:39.248652+0000","last_fullsized":"2024-02-05T07:25:59.221878+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:50.859511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:50.859511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:54.855321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":384,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.235817+0000","last_change":"2024-02-05T07:25:59.223932+0000","last_active":"2024-02-05T07:26:39.235817+0000","last_peered":"2024-02-05T07:26:39.235817+0000","last_clean":"2024-02-05T07:25:57.229914+0000","last_became_active":"2024-02-05T07:25:59.223932+0000","last_became_peered":"2024-02-05T07:25:59.223932+0000","last_unstale":"2024-02-05T07:26:39.235817+0000","last_undegraded":"2024-02-05T07:26:39.235817+0000","last_fullsized":"2024-02-05T07:25:59.216590+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:51.823640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:51.823640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:41.264553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":186,"reported_epoch":244,"state":"active+undersized","last_fresh":"2024-02-05T07:26:39.243419+0000","last_change":"2024-02-05T07:25:59.229105+0000","last_active":"2024-02-05T07:26:39.243419+0000","last_peered":"2024-02-05T07:26:39.243419+0000","last_clean":"2024-02-05T07:25:57.239378+0000","last_became_active":"2024-02-05T07:25:59.229105+0000","last_became_peered":"2024-02-05T07:25:59.229105+0000","last_unstale":"2024-02-05T07:26:39.243419+0000","last_undegraded":"2024-02-05T07:26:39.243419+0000","last_fullsized":"2024-02-05T07:25:59.221777+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:09.289989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:09.289989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:43.059072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":396,"reported_epoch":244,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:26:39.236135+0000","last_change":"2024-02-05T07:25:59.232137+0000","last_active":"2024-02-05T07:26:39.236135+0000","last_peered":"2024-02-05T07:26:39.236135+0000","last_clean":"2024-02-05T07:25:57.230142+0000","last_became_active":"2024-02-05T07:25:59.232137+0000","last_became_peered":"2024-02-05T07:25:59.232137+0000","last_unstale":"2024-02-05T07:26:39.236135+0000","last_undegraded":"2024-02-05T07:25:59.223036+0000","last_fullsized":"2024-02-05T07:25:59.222942+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:23:15.477555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:15.477555+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:26:41.264512+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":135,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.249317+0000","last_change":"2024-02-05T07:24:56.703719+0000","last_active":"2024-02-05T07:26:39.249317+0000","last_peered":"2024-02-05T07:26:39.249317+0000","last_clean":"2024-02-05T07:26:39.249317+0000","last_became_active":"2024-02-05T07:24:34.229294+0000","last_became_peered":"2024-02-05T07:24:34.229294+0000","last_unstale":"2024-02-05T07:26:39.249317+0000","last_undegraded":"2024-02-05T07:26:39.249317+0000","last_fullsized":"2024-02-05T07:26:39.249317+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:56.703642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:56.703642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":382,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.235392+0000","last_change":"2024-02-05T07:26:28.576091+0000","last_active":"2024-02-05T07:26:39.235392+0000","last_peered":"2024-02-05T07:26:39.235392+0000","last_clean":"2024-02-05T07:26:39.235392+0000","last_became_active":"2024-02-05T07:18:53.460820+0000","last_became_peered":"2024-02-05T07:18:53.460820+0000","last_unstale":"2024-02-05T07:26:39.235392+0000","last_undegraded":"2024-02-05T07:26:39.235392+0000","last_fullsized":"2024-02-05T07:26:39.235392+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:28.575859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:28.575859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:52.998374+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":377,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-05T07:26:39.238589+0000","last_change":"2024-02-05T07:26:21.317529+0000","last_active":"2024-02-05T07:26:39.238589+0000","last_peered":"2024-02-05T07:26:39.238589+0000","last_clean":"2024-02-05T07:26:39.238589+0000","last_became_active":"2024-02-05T07:18:55.466001+0000","last_became_peered":"2024-02-05T07:18:55.466001+0000","last_unstale":"2024-02-05T07:26:39.238589+0000","last_undegraded":"2024-02-05T07:26:39.238589+0000","last_fullsized":"2024-02-05T07:26:39.238589+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:21.317417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:21.317417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:50.152412+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":7477067776,"num_objects":114091,"num_object_clones":0,"num_object_copies":342273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35628,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114091,"num_write_kb":7301824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21211447296,"data_stored":21211447296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70391,"ondisk_log_size":70391,"up":43,"acting":43,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":463,"num_read_kb":458,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":375689,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":127,"num_read_kb":189,"num_write":113,"num_write_kb":1538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1717434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":25,"acting":25,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":82,"acting":82,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":683,"num_read_kb":440,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":444,"ondisk_log_size":444,"up":80,"acting":80,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":80,"acting":80,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037859,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2561412,"kb_used_data":2273396,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91183740,"statfs":{"total":95995035648,"available":93372149760,"internally_reserved":0,"allocated":2327957504,"data_stored":2327224756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":294910410},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":3.476,"15min":1.4930000000000001},"min":{"1min":0.223,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":1.161,"5min":63.039000000000001,"15min":63.039000000000001},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":3.4649999999999999,"15min":1.498},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.9119999999999999,"5min":65.296000000000006,"15min":65.296000000000006},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Feb 5 07:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":2.0310000000000001,"15min":1.0289999999999999},"min":{"1min":0.42899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":2.472,"5min":32.127000000000002,"15min":32.127000000000002},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.79400000000000004,"5min":2.0910000000000002,"15min":1.0669999999999999},"min":{"1min":0.313,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":4.5179999999999998,"5min":32.173999999999999,"15min":32.173999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Mon Feb 5 07:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":2.3180000000000001,"15min":1.125},"min":{"1min":0.39900000000000002,"5min":0.249,"15min":0.249},"max":{"1min":4.5659999999999998,"5min":51.127000000000002,"15min":51.127000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.754,"5min":2.6230000000000002,"15min":1.2230000000000001},"min":{"1min":0.28599999999999998,"5min":0.246,"15min":0.246},"max":{"1min":2.1120000000000001,"5min":51.113999999999997,"15min":51.113999999999997},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.8460000000000001,"15min":0.97599999999999998},"min":{"1min":0.377,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.1160000000000001,"5min":58.341999999999999,"15min":58.341999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.753,"5min":1.806,"15min":0.97299999999999998},"min":{"1min":0.38400000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.869,"5min":57.765000000000001,"15min":57.765000000000001},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.64600000000000002,"15min":0.50700000000000001},"min":{"1min":0.36199999999999999,"5min":0.25,"15min":0.25},"max":{"1min":4.5999999999999996,"5min":4.8659999999999997,"15min":4.8659999999999997},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.60799999999999998,"15min":0.498},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.4940000000000002,"5min":4.577,"15min":4.577},"last":0.36499999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.73799999999999999,"15min":0.52900000000000003},"min":{"1min":0.38300000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.7200000000000002,"5min":6.9020000000000001,"15min":6.9020000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.167,"5min":0.83599999999999997,"15min":0.60999999999999999},"min":{"1min":0.44800000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.9829999999999997,"5min":6.3449999999999998,"15min":6.3449999999999998},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Feb 5 07:25:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.77600000000000002,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.6310000000000002,"5min":4.9630000000000001,"15min":4.9630000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.872,"5min":0.71299999999999997,"15min":0.58099999999999996},"min":{"1min":0.36699999999999999,"5min":0.318,"15min":0.318},"max":{"1min":3.5150000000000001,"5min":5.5179999999999998,"15min":5.5179999999999998},"last":1.956}]}]},{"osd":6,"up_from":52,"seq":223338299488,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1295576,"kb_used_data":1057544,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92449576,"statfs":{"total":95995035648,"available":94668365824,"internally_reserved":0,"allocated":1082925056,"data_stored":1082249111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":243726790},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":6.6749999999999998,"5min":4.2910000000000004,"15min":1.7929999999999999},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":111.81699999999999,"5min":111.81699999999999,"15min":111.81699999999999},"last":2.2709999999999999},{"interface":"front","average":{"1min":6.8140000000000001,"5min":4.1550000000000002,"15min":1.738},"min":{"1min":0.34000000000000002,"5min":0.22,"15min":0.22},"max":{"1min":111.09699999999999,"5min":111.09699999999999,"15min":111.09699999999999},"last":1.772}]},{"osd":1,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":2.387,"5min":1.254,"15min":0.78100000000000003},"min":{"1min":0.26700000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":33.984999999999999,"5min":33.984999999999999,"15min":33.984999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":2.3220000000000001,"5min":1.1859999999999999,"15min":0.76400000000000001},"min":{"1min":0.374,"5min":0.249,"15min":0.249},"max":{"1min":32.134,"5min":32.134,"15min":32.134},"last":0.94499999999999995}]},{"osd":2,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":1.9019999999999999,"15min":1.0169999999999999},"min":{"1min":0.36399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":31.878,"5min":32.509999999999998,"15min":32.509999999999998},"last":0.79900000000000004},{"interface":"front","average":{"1min":2.214,"5min":2.1309999999999998,"15min":1.0640000000000001},"min":{"1min":0.36699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":31.507000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"last":2.524}]},{"osd":3,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":2.0299999999999998,"15min":1.0469999999999999},"min":{"1min":0.376,"5min":0.22,"15min":0.22},"max":{"1min":1.4470000000000001,"5min":36.976999999999997,"15min":36.976999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.835,"15min":0.98799999999999999},"min":{"1min":0.34899999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72099999999999997,"5min":34.222999999999999,"15min":34.222999999999999},"last":0.83399999999999996}]},{"osd":4,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.59999999999999998,"15min":0.47099999999999997},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.173,"5min":4.5609999999999999,"15min":4.5609999999999999},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.876,"5min":0.63400000000000001,"15min":0.51500000000000001},"min":{"1min":0.28599999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":7.7960000000000003,"5min":7.7960000000000003,"15min":7.7960000000000003},"last":0.871}]},{"osd":5,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.90000000000000002,"15min":0.59899999999999998},"min":{"1min":0.307,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.1369999999999996,"5min":27.021999999999998,"15min":27.021999999999998},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.94899999999999995,"15min":0.623},"min":{"1min":0.34300000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.0629999999999997,"5min":23.611999999999998,"15min":23.611999999999998},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Feb 5 07:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.66600000000000004,"15min":0.55800000000000005},"min":{"1min":0.29399999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.044,"5min":4.2130000000000001,"15min":4.2130000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.98999999999999999,"15min":0.69299999999999995},"min":{"1min":0.33300000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.9740000000000002,"5min":25.207000000000001,"15min":25.207000000000001},"last":0.69599999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574977,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3492572,"kb_used_data":3178188,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":90252580,"statfs":{"total":95995035648,"available":92418641920,"internally_reserved":0,"allocated":3254464512,"data_stored":3253708121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":321911237},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,5,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":3.7679999999999998,"15min":1.53},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":4.3689999999999998,"5min":45.393000000000001,"15min":45.393000000000001},"last":2.476},{"interface":"front","average":{"1min":0.875,"5min":3.7490000000000001,"15min":1.51},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.27,"5min":45.031999999999996,"15min":45.031999999999996},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":2.3460000000000001,"15min":1.073},"min":{"1min":0.27100000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.194,"5min":39.259999999999998,"15min":39.259999999999998},"last":1.796},{"interface":"front","average":{"1min":0.93899999999999995,"5min":2.2080000000000002,"15min":1.3660000000000001},"min":{"1min":0.315,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8570000000000002,"5min":40.972000000000001,"15min":40.972000000000001},"last":1.9139999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:26:13 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":2.387,"15min":1.105},"min":{"1min":0.27500000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5099999999999998,"5min":43.823,"15min":43.823},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.0149999999999999,"5min":2.589,"15min":1.1850000000000001},"min":{"1min":0.315,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.754,"5min":39.314999999999998,"15min":39.314999999999998},"last":1.9530000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:26:34 2024","interfaces":[{"interface":"back","average":{"1min":5.3319999999999999,"5min":2.927,"15min":1.371},"min":{"1min":0.42099999999999999,"5min":0.312,"15min":0.312},"max":{"1min":61.722000000000001,"5min":61.722000000000001,"15min":61.722000000000001},"last":1.988},{"interface":"front","average":{"1min":4.5650000000000004,"5min":2.9199999999999999,"15min":1.452},"min":{"1min":0.34100000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":61.622999999999998,"5min":61.786000000000001,"15min":61.786000000000001},"last":1.893}]},{"osd":5,"last update":"Mon Feb 5 07:25:57 2024","interfaces":[{"interface":"back","average":{"1min":6.2199999999999998,"5min":3.0390000000000001,"15min":1.4239999999999999},"min":{"1min":0.49399999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":106.672,"5min":106.672,"15min":106.672},"last":2.016},{"interface":"front","average":{"1min":6.1230000000000002,"5min":3.1539999999999999,"15min":1.5569999999999999},"min":{"1min":0.45800000000000002,"5min":0.307,"15min":0.307},"max":{"1min":107.348,"5min":107.348,"15min":107.348},"last":1.0980000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.4330000000000001,"15min":0.93000000000000005},"min":{"1min":0.42499999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.5739999999999998,"5min":18.456,"15min":18.456},"last":1.016},{"interface":"front","average":{"1min":0.86499999999999999,"5min":1.395,"15min":0.91400000000000003},"min":{"1min":0.46300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.4340000000000002,"5min":5.9930000000000003,"15min":5.9930000000000003},"last":1.5740000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":2.2149999999999999,"15min":1.123},"min":{"1min":0.39300000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.2999999999999998,"5min":66.703999999999994,"15min":66.703999999999994},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.96099999999999997,"5min":2.1499999999999999,"15min":1.113},"min":{"1min":0.42699999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":4.484,"5min":61.594999999999999,"15min":61.594999999999999},"last":1.841}]}]},{"osd":0,"up_from":8,"seq":34359738502,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3516504,"kb_used_data":3164620,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":90228648,"statfs":{"total":95995035648,"available":92394135552,"internally_reserved":0,"allocated":3240570880,"data_stored":3239814055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:26:26 2024","interfaces":[{"interface":"back","average":{"1min":2.8809999999999998,"5min":2.532,"15min":1.0669999999999999},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":45.372,"5min":45.372,"15min":45.372},"last":2.9670000000000001},{"interface":"front","average":{"1min":3.1240000000000001,"5min":2.7949999999999999,"15min":1.163},"min":{"1min":0.217,"5min":0.186,"15min":0.186},"max":{"1min":45.231999999999999,"5min":45.231999999999999,"15min":45.231999999999999},"last":1.7809999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":2.7149999999999999,"15min":1.194},"min":{"1min":0.246,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.9289999999999998,"5min":65.903999999999996,"15min":65.903999999999996},"last":1.857},{"interface":"front","average":{"1min":0.68700000000000006,"5min":2.649,"15min":1.165},"min":{"1min":0.308,"5min":0.186,"15min":0.182},"max":{"1min":3.1779999999999999,"5min":65.694000000000003,"15min":65.694000000000003},"last":1.639}]},{"osd":3,"last update":"Mon Feb 5 07:26:07 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":3.2200000000000002,"15min":1.3660000000000001},"min":{"1min":0.28699999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.8400000000000001,"5min":65.155000000000001,"15min":65.155000000000001},"last":1.589},{"interface":"front","average":{"1min":0.66400000000000003,"5min":3.4670000000000001,"15min":1.4350000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":2.4470000000000001,"5min":65.114000000000004,"15min":65.114000000000004},"last":3.0579999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:26:31 2024","interfaces":[{"interface":"back","average":{"1min":3.359,"5min":2.859,"15min":1.331},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":50.545000000000002,"5min":65.188000000000002,"15min":65.188000000000002},"last":3.0070000000000001},{"interface":"front","average":{"1min":3.5009999999999999,"5min":2.8170000000000002,"15min":1.3600000000000001},"min":{"1min":0.33900000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":50.310000000000002,"5min":64.873000000000005,"15min":64.873000000000005},"last":3.9060000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:26:07 2024","interfaces":[{"interface":"back","average":{"1min":5.5599999999999996,"5min":3.1640000000000001,"15min":1.4259999999999999},"min":{"1min":0.39800000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":95.772999999999996,"5min":95.772999999999996,"15min":95.772999999999996},"last":1.671},{"interface":"front","average":{"1min":5.6340000000000003,"5min":3.3210000000000002,"15min":1.5009999999999999},"min":{"1min":0.32300000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":96.028000000000006,"5min":96.028000000000006,"15min":96.028000000000006},"last":1.8169999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:25:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":2.2610000000000001,"15min":1.155},"min":{"1min":0.45600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":25.681999999999999,"5min":65.215000000000003,"15min":65.215000000000003},"last":0.63500000000000001},{"interface":"front","average":{"1min":2.2530000000000001,"5min":2.4489999999999998,"15min":1.242},"min":{"1min":0.41899999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":31.524000000000001,"5min":65.230000000000004,"15min":65.230000000000004},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":2.4220000000000002,"15min":1.2110000000000001},"min":{"1min":0.38700000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":2.133,"5min":65.263000000000005,"15min":65.263000000000005},"last":3.032},{"interface":"front","average":{"1min":0.75600000000000001,"5min":2.3969999999999998,"15min":1.224},"min":{"1min":0.44500000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.974,"5min":64.906000000000006,"15min":64.906000000000006},"last":1.889}]}]},{"osd":2,"up_from":18,"seq":77309411452,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4066824,"kb_used_data":3666360,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":89678328,"statfs":{"total":95995035648,"available":91830607872,"internally_reserved":0,"allocated":3754352640,"data_stored":3753616827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,4,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":2.456,"15min":1.0629999999999999},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":1.2729999999999999,"5min":67.418999999999997,"15min":67.418999999999997},"last":1.915},{"interface":"front","average":{"1min":0.63100000000000001,"5min":2.7850000000000001,"15min":1.1859999999999999},"min":{"1min":0.23899999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.879,"5min":68.171000000000006,"15min":68.171000000000006},"last":0.82599999999999996}]},{"osd":1,"last update":"Mon Feb 5 07:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.409,"15min":0.73199999999999998},"min":{"1min":0.20999999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":4.1900000000000004,"5min":12.609,"15min":12.609},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.623,"5min":1.5389999999999999,"15min":0.79800000000000004},"min":{"1min":0.30399999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.093,"5min":9.641,"15min":9.641},"last":0.22700000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:26:13 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":1.8280000000000001,"15min":0.93300000000000005},"min":{"1min":0.311,"5min":0.214,"15min":0.214},"max":{"1min":3.7810000000000001,"5min":14.756,"15min":14.756},"last":1.3320000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.8480000000000001,"15min":0.93300000000000005},"min":{"1min":0.29199999999999998,"5min":0.193,"15min":0.193},"max":{"1min":2.4049999999999998,"5min":9.6669999999999998,"15min":9.6669999999999998},"last":1.9410000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:26:38 2024","interfaces":[{"interface":"back","average":{"1min":2.8919999999999999,"5min":1.6930000000000001,"15min":0.92300000000000004},"min":{"1min":0.318,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":37.479999999999997,"5min":37.479999999999997,"15min":37.479999999999997},"last":0.58099999999999996},{"interface":"front","average":{"1min":3.0190000000000001,"5min":1.704,"15min":0.95699999999999996},"min":{"1min":0.41799999999999998,"5min":0.317,"15min":0.317},"max":{"1min":38.853999999999999,"5min":38.853999999999999,"15min":38.853999999999999},"last":1.879}]},{"osd":5,"last update":"Mon Feb 5 07:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.389,"15min":0.86099999999999999},"min":{"1min":0.42099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.2749999999999999,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.268,"15min":0.82599999999999996},"min":{"1min":0.503,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.056,"5min":7.7279999999999998,"15min":7.7279999999999998},"last":0.56299999999999994}]},{"osd":6,"last update":"Mon Feb 5 07:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":1.3109999999999999,"15min":0.93700000000000006},"min":{"1min":0.42799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.3069999999999999,"5min":7.6829999999999998,"15min":7.6829999999999998},"last":2.7400000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.381,"15min":0.86399999999999999},"min":{"1min":0.41699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":2.0489999999999999,"5min":7.3150000000000004,"15min":7.3150000000000004},"last":1.9830000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.405,"15min":0.92400000000000004},"min":{"1min":0.44400000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.371,"5min":7.2590000000000003,"15min":7.2590000000000003},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.3080000000000001,"15min":0.85899999999999999},"min":{"1min":0.38500000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.2290000000000001,"5min":7.7619999999999996,"15min":7.7619999999999996},"last":0.51200000000000001}]}]},{"osd":3,"up_from":0,"seq":111669149815,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953586,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3412040,"kb_used_data":3200376,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":90333112,"statfs":{"total":95995035648,"available":92501106688,"internally_reserved":0,"allocated":3277185024,"data_stored":3276447624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:26:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.8,"15min":0.95399999999999996},"min":{"1min":0.23499999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.504,"5min":67.719999999999999,"15min":67.719999999999999},"last":1.742},{"interface":"front","average":{"1min":1.1719999999999999,"5min":2.0720000000000001,"15min":1.069},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.8399999999999999,"5min":66.882000000000005,"15min":66.882000000000005},"last":0.39800000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.93100000000000005,"15min":0.66400000000000003},"min":{"1min":0.35199999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":3.6080000000000001,"5min":5.5709999999999997,"15min":5.5709999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.0980000000000001,"15min":0.73099999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.593,"5min":17.065999999999999,"15min":17.065999999999999},"last":1.802}]},{"osd":2,"last update":"Mon Feb 5 07:26:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.829,"15min":0.97899999999999998},"min":{"1min":0.27900000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.851,"5min":67.144999999999996,"15min":67.144999999999996},"last":1.778},{"interface":"front","average":{"1min":1.496,"5min":1.78,"15min":0.96799999999999997},"min":{"1min":0.35199999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":11.055,"5min":66.959999999999994,"15min":66.959999999999994},"last":0.50600000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:26:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.93500000000000005,"15min":0.71099999999999997},"min":{"1min":0.379,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.698,"5min":3.8359999999999999,"15min":8.8019999999999996},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.97499999999999998,"15min":0.748},"min":{"1min":0.32500000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.0620000000000003,"5min":5.3449999999999998,"15min":10.106},"last":0.33000000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.57399999999999995,"15min":0.45100000000000001},"min":{"1min":0.27400000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.157,"5min":2.8420000000000001,"15min":2.8420000000000001},"last":2.5790000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.61899999999999999,"15min":0.46899999999999997},"min":{"1min":0.29299999999999998,"5min":0.255,"15min":0.254},"max":{"1min":1.6379999999999999,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.159}]},{"osd":6,"last update":"Mon Feb 5 07:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.70999999999999996,"15min":0.51600000000000001},"min":{"1min":0.36199999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.29,"5min":6.8940000000000001,"15min":6.8940000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.67800000000000005,"15min":0.501},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.248},"max":{"1min":2.9260000000000002,"5min":2.9260000000000002,"15min":2.9260000000000002},"last":0.38700000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.72099999999999997,"15min":0.57499999999999996},"min":{"1min":0.33800000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.873,"5min":6.0640000000000001,"15min":6.0640000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.65700000000000003,"15min":0.54600000000000004},"min":{"1min":0.32500000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":1.8260000000000001}]}]},{"osd":5,"up_from":41,"seq":176093659245,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4352520,"kb_used_data":4064504,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":89392632,"statfs":{"total":95995035648,"available":91538055168,"internally_reserved":0,"allocated":4162052096,"data_stored":4161315701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:25:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":3.3679999999999999,"15min":1.4750000000000001},"min":{"1min":0.29199999999999998,"5min":0.219,"15min":0.219},"max":{"1min":2.1549999999999998,"5min":69.855999999999995,"15min":69.855999999999995},"last":1.115},{"interface":"front","average":{"1min":0.66400000000000003,"5min":3.0880000000000001,"15min":1.383},"min":{"1min":0.27800000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.4359999999999999,"5min":71.513999999999996,"15min":71.513999999999996},"last":0.57599999999999996}]},{"osd":1,"last update":"Mon Feb 5 07:25:56 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.4159999999999999,"15min":0.84299999999999997},"min":{"1min":0.34300000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":10.178000000000001,"5min":11.839,"15min":11.839},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.98699999999999999,"15min":0.68100000000000005},"min":{"1min":0.35099999999999998,"5min":0.253,"15min":0.253},"max":{"1min":9.4049999999999994,"5min":9.5950000000000006,"15min":9.5950000000000006},"last":0.68200000000000005}]},{"osd":2,"last update":"Mon Feb 5 07:25:56 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":2.1659999999999999,"15min":1.0720000000000001},"min":{"1min":0.34999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":9.7149999999999999,"5min":70.108000000000004,"15min":70.108000000000004},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.216,"5min":2.3079999999999998,"15min":1.1339999999999999},"min":{"1min":0.35499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":9.4600000000000009,"5min":72.817999999999998,"15min":72.817999999999998},"last":1.075}]},{"osd":3,"last update":"Mon Feb 5 07:25:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":2.234,"15min":1.107},"min":{"1min":0.33300000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.8330000000000002,"5min":69.793000000000006,"15min":69.793000000000006},"last":0.441},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.4399999999999999,"15min":1.181},"min":{"1min":0.35199999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.2679999999999998,"5min":69.820999999999998,"15min":69.820999999999998},"last":0.93000000000000005}]},{"osd":4,"last update":"Mon Feb 5 07:25:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.53700000000000003,"15min":0.42099999999999999},"min":{"1min":0.27400000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.5939999999999999,"5min":3.5939999999999999,"15min":3.5939999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.60199999999999998,"15min":0.46100000000000002},"min":{"1min":0.31,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":2.8780000000000001,"5min":3.6360000000000001,"15min":3.6360000000000001},"last":0.41199999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:25:26 2024","interfaces":[{"interface":"back","average":{"1min":5.6280000000000001,"5min":2.8340000000000001,"15min":1.216},"min":{"1min":0.30199999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":109.09399999999999,"5min":109.09399999999999,"15min":109.09399999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":5.6710000000000003,"5min":2.8210000000000002,"15min":1.246},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":108.965,"5min":108.965,"15min":108.965},"last":0.40400000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":3.6480000000000001,"15min":1.5580000000000001},"min":{"1min":0.32400000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":3.4180000000000001,"5min":182.87799999999999,"15min":182.87799999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":3.6019999999999999,"15min":1.548},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.3280000000000001,"5min":183.02199999999999,"15min":183.02199999999999},"last":0.71999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3265855488,"data_stored":3265855488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3286302720,"data_stored":3286302720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3779133440,"data_stored":3779133440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3276668928,"data_stored":3276668928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4186308608,"data_stored":4186308608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1085865984,"data_stored":1085865984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2331312128,"data_stored":2331312128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:26:42.923 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-05T07:26:42.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set unique_pool_1 pg_num 26 2024-02-05T07:26:42.991 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:42 smithi090 ceph-mon[33213]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:26:43.025 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:43.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:43.030 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 72 16 117913 117897 102.325 88.25 0.00447606 0.00976829 2024-02-05T07:26:43.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:42 smithi007 ceph-mon[29098]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:26:43.094 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:42 smithi007 ceph-mon[37158]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:26:43.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:43.172 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:43.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:43.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:43.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:43.313 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:43.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:43.415 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:43.454 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:43.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:43.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:43.593 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:43.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:43.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:43.674 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:26:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:26:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:26:43.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:43.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:43.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:43.871 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:43.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:43.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:43.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:44.008 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:44.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:44.030 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 73 15 119369 119354 102.171 91.0625 0.0116068 0.00978314 2024-02-05T07:26:44.110 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:44.147 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:44.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:43 smithi007 ceph-mon[29098]: pgmap v379: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.85k op/s; 35726/342861 objects degraded (10.420%) 2024-02-05T07:26:44.152 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:43 smithi007 ceph-mon[37158]: pgmap v379: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.85k op/s; 35726/342861 objects degraded (10.420%) 2024-02-05T07:26:44.196 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:43 smithi090 ceph-mon[33213]: pgmap v379: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.85k op/s; 35726/342861 objects degraded (10.420%) 2024-02-05T07:26:44.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:44.304 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:44.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:44.340 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:44.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:44.489 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:44.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:44.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:44.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:44.632 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:44.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:44.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:44.776 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:44.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:44.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:44.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:44.913 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:44.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:44.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:44.927 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:26:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:26:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:120628 8.5 8:ab689ceb:::benchmark_data_smithi007_105755_object120627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-02-05T07:26:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:44.919419+0000", 2024-02-05T07:26:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0049490350000000001, 2024-02-05T07:26:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049859450000000003, 2024-02-05T07:26:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:26:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:26:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:26:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 120628 2024-02-05T07:26:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.919419+0000", 2024-02-05T07:26:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.919419+0000", 2024-02-05T07:26:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.919420+0000", 2024-02-05T07:26:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6810000000000001e-06 2024-02-05T07:26:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.919482+0000", 2024-02-05T07:26:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1878999999999995e-05 2024-02-05T07:26:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.919494+0000", 2024-02-05T07:26:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.217e-05 2024-02-05T07:26:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.919503+0000", 2024-02-05T07:26:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.132e-06 2024-02-05T07:26:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.919573+0000", 2024-02-05T07:26:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9639000000000002e-05 2024-02-05T07:26:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.919731+0000", 2024-02-05T07:26:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015808400000000001 2024-02-05T07:26:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-02-05T07:26:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.919898+0000", 2024-02-05T07:26:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000166976 2024-02-05T07:26:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:120634 8.5 8.467051b5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-02-05T07:26:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:44.920900+0000", 2024-02-05T07:26:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0034677340000000001, 2024-02-05T07:26:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003591506, 2024-02-05T07:26:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-02-05T07:26:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:26:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:26:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 120634 2024-02-05T07:26:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.920900+0000", 2024-02-05T07:26:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.920900+0000", 2024-02-05T07:26:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.920901+0000", 2024-02-05T07:26:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2160000000000001e-06 2024-02-05T07:26:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.920929+0000", 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8299000000000001e-05 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.920948+0000", 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8331000000000001e-05 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.920956+0000", 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5080000000000005e-06 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:120635 8.5 8.d31a1145 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-02-05T07:26:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:44.920976+0000", 2024-02-05T07:26:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.003391266, 2024-02-05T07:26:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003569203, 2024-02-05T07:26:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-02-05T07:26:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:26:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:26:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 120635 2024-02-05T07:26:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.920976+0000", 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.920976+0000", 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.920977+0000", 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1099999999999996e-07 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.921038+0000", 2024-02-05T07:26:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1305999999999999e-05 2024-02-05T07:26:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.921046+0000", 2024-02-05T07:26:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6389999999999992e-06 2024-02-05T07:26:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.921052+0000", 2024-02-05T07:26:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9610000000000002e-06 2024-02-05T07:26:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:120636 8.b 8:db52c7c5:::benchmark_data_smithi007_105755_object120635:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:44.921061+0000", 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0033065619999999999, 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035431130000000001, 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 120636 2024-02-05T07:26:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.921061+0000", 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.921061+0000", 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.921062+0000", 2024-02-05T07:26:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8400000000000005e-07 2024-02-05T07:26:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.921110+0000", 2024-02-05T07:26:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8807999999999998e-05 2024-02-05T07:26:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.921117+0000", 2024-02-05T07:26:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4629999999999998e-06 2024-02-05T07:26:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.921122+0000", 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9960000000000001e-06 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.921168+0000", 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6078000000000002e-05 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.921278+0000", 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011056 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.921407+0000", 2024-02-05T07:26:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012824700000000001 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24977.0:120637 8.f e244/209 8:fcb0d110:::benchmark_data_smithi007_105755_object120636:head v 244'7652, mlcod=244'7649)", 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:44.923877+0000", 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.00049104300000000004, 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080639500000000001, 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.923877+0000", 2024-02-05T07:26:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.923877+0000", 2024-02-05T07:26:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.923877+0000", 2024-02-05T07:26:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7400000000000002e-07 2024-02-05T07:26:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.923917+0000", 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9156999999999998e-05 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.923926+0000", 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8500000000000006e-06 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.923934+0000", 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2289999999999997e-06 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.924091+0000", 2024-02-05T07:26:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157374 2024-02-05T07:26:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:44.924123+0000", 2024-02-05T07:26:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1921999999999997e-05 2024-02-05T07:26:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-02-05T07:26:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 5 2024-02-05T07:26:44.946 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:26:45.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:45.030 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 74 16 120764 120748 101.968 87.125 0.00281316 0.00979954 2024-02-05T07:26:45.039 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:45.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:45.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:45.141 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:45.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:44 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 35726/342861 objects degraded (10.420%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:45.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:44 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:26:45.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:44 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:26:45.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:44 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 35726/342861 objects degraded (10.420%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:45.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:44 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:26:45.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:44 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:26:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:45.179 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:45.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:45.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:44 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 35726/342861 objects degraded (10.420%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:45.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:44 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:26:45.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:44 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:26:45.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:45.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:45.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:45.402 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:45.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:45.466 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:45.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:45.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:45.476 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:26:45.525 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:26:45.565 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:26:45.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:45.606 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:26:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:45.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:45.651 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:26:45.698 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:26:45.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:45.739 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:26:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:26:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:26:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:26:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:26:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:26:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:26:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-02-05T07:26:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:26:45.749 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:45.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:45.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:45.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T07:26:45.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:45.912 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:45.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:46.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:45 smithi090 ceph-mon[33213]: pgmap v380: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 917 op/s; 35726/342861 objects degraded (10.420%) 2024-02-05T07:26:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:45 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3222826975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:46.030 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 75 16 122071 122055 101.697 81.6875 0.0186851 0.00982861 2024-02-05T07:26:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:46.067 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:46.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:46.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:45 smithi007 ceph-mon[37158]: pgmap v380: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 917 op/s; 35726/342861 objects degraded (10.420%) 2024-02-05T07:26:46.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:45 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3222826975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:46.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:45 smithi007 ceph-mon[29098]: pgmap v380: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 917 op/s; 35726/342861 objects degraded (10.420%) 2024-02-05T07:26:46.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:45 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3222826975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:46.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:46.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:46.215 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:46.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:46.260 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:26:46.316 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:46.350 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:46.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:46.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:46.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:46.490 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:46.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:46.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:46.639 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:46.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:46.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:46.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:46.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:46.778 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:46.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:46.879 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:46.917 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:46.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:46.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:47.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:47.032 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 76 16 123495 123479 101.528 89 0.00506766 0.00984507 2024-02-05T07:26:47.051 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:47.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:47.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:47.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:47.193 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:47.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:47.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:47.325 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:47.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:47.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:47.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:47.463 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:47.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:47.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:47.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:47.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:47.721 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:47.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:47.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:47.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:47.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:47.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:47.958 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:47.994 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:47.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:47.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:48.032 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 77 15 124621 124606 101.124 70.4375 0.0119303 0.00988422 2024-02-05T07:26:48.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:48.131 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:48.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:48.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:48.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:48.268 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:48.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:48.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:48.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:48.403 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:48.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:48.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:48 smithi007 ceph-mon[29098]: pgmap v381: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 37890/364035 objects degraded (10.408%) 2024-02-05T07:26:48.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:48 smithi007 ceph-mon[37158]: pgmap v381: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 37890/364035 objects degraded (10.408%) 2024-02-05T07:26:48.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:48.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:48 smithi090 ceph-mon[33213]: pgmap v381: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 37890/364035 objects degraded (10.408%) 2024-02-05T07:26:48.527 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:48.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:48.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:48.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:48.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:48.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:48.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:48.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:48.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:48.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:48.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:48.932 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:48.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:48.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:49.033 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 78 16 125760 125744 100.739 71.125 0.0112035 0.00991423 2024-02-05T07:26:49.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:49.064 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:49.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:49.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:49.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:49.207 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:49.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:49.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:49.346 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:49.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:49.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:49.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:49.486 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:49.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:49.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:49.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:49.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:49.623 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:49.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:49.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:49.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:49.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:49.891 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:49.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:49.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T07:26:49.941 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T07:26:49.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:50.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:50.018 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:50.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:50.032 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 79 16 126785 126769 100.275 64.0625 0.00351086 0.0099594 2024-02-05T07:26:50.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:50.151 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:50.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:50.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:50.281 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:50.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:50.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:50.418 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:50.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:50.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:50.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:50 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:26:50.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:50 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:26:50.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:50 smithi007 ceph-mon[29098]: pgmap v382: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s; 37890/364035 objects degraded (10.408%) 2024-02-05T07:26:50.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:50 smithi007 ceph-mon[29098]: osdmap e245: 8 total, 7 up, 7 in 2024-02-05T07:26:50.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:50 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 37890/364035 objects degraded (10.408%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:50.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:50 smithi007 ceph-mon[29098]: osdmap e246: 8 total, 7 up, 7 in 2024-02-05T07:26:50.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:50 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:26:50.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:50 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:26:50.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:50 smithi007 ceph-mon[37158]: pgmap v382: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s; 37890/364035 objects degraded (10.408%) 2024-02-05T07:26:50.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:50 smithi007 ceph-mon[37158]: osdmap e245: 8 total, 7 up, 7 in 2024-02-05T07:26:50.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:50 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 37890/364035 objects degraded (10.408%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:50.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:50 smithi007 ceph-mon[37158]: osdmap e246: 8 total, 7 up, 7 in 2024-02-05T07:26:50.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:50.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:26:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:26:50.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:50 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:26:50.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:50 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:26:50.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:50 smithi090 ceph-mon[33213]: pgmap v382: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s; 37890/364035 objects degraded (10.408%) 2024-02-05T07:26:50.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:50 smithi090 ceph-mon[33213]: osdmap e245: 8 total, 7 up, 7 in 2024-02-05T07:26:50.527 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:50 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 37890/364035 objects degraded (10.408%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:50.527 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:50 smithi090 ceph-mon[33213]: osdmap e246: 8 total, 7 up, 7 in 2024-02-05T07:26:50.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:50.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:50.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:50.728 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:50.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:50.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:50.863 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:50.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:50.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:50.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:50.997 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:50.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:51.032 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:26:51.032038+0000 min lat: 0.00128428 max lat: 0.975632 avg lat: 0.00999863 2024-02-05T07:26:51.033 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:26:51.033 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 80 16 127997 127981 99.9683 75.75 0.0173518 0.00999863 2024-02-05T07:26:51.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:51.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:51.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:51.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:51.274 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:51.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:51.330 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:51 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:51.330 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:51 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:51.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:51.399 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:51.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:51.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:51.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:51.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:51 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:26:51.535 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:51.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:51.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:51.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:51.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:51.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:51.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:51.804 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:51.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:51.904 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:51.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:51.929 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:51.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:51.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:26:52.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:52.033 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 81 16 128904 128888 99.434 56.6875 0.022878 0.0100524 2024-02-05T07:26:52.064 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:52.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:52.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:52.192 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 8 pg_num to 26 2024-02-05T07:26:52.195 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:26:52.219 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:52.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:52.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:52.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:52.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:52.348 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:52.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:52.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:52 smithi007 ceph-mon[29098]: pgmap v385: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 39827/381984 objects degraded (10.426%) 2024-02-05T07:26:52.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:52 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/546549816' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:26:52.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:52 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:26:52.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:52 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2903533380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:52.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:52 smithi007 ceph-mon[37158]: pgmap v385: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 39827/381984 objects degraded (10.426%) 2024-02-05T07:26:52.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:52 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/546549816' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:26:52.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:52 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:26:52.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:52 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2903533380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:52.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:52.478 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:52.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:52.527 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:52 smithi090 ceph-mon[33213]: pgmap v385: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 39827/381984 objects degraded (10.426%) 2024-02-05T07:26:52.527 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:52 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/546549816' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:26:52.528 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:52 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:26:52.528 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:52 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2903533380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:26:52.580 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:52.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:52.618 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:52.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:52.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:52.757 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:52.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:52.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:52.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:52.904 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:52.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:53.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:53.029 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:53.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:53.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:53.032 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 82 16 129854 129838 98.9455 59.375 0.00347802 0.0100925 2024-02-05T07:26:53.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:53.166 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:53.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:53.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:53.268 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:53.325 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:53.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:53.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:53 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:53.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:53 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-02-05T07:26:53.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:53 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2903533380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:26:53.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:53 smithi007 ceph-mon[29098]: osdmap e247: 8 total, 7 up, 7 in 2024-02-05T07:26:53.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:53 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:26:53.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:53 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:26:53.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:53 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:53.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:53 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-02-05T07:26:53.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:53 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2903533380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:26:53.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:53 smithi007 ceph-mon[37158]: osdmap e247: 8 total, 7 up, 7 in 2024-02-05T07:26:53.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:53 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:26:53.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:53 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:26:53.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:53.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:53.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:53.512 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:53.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:53 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:53.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:53 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-02-05T07:26:53.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:53 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2903533380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:26:53.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:53 smithi090 ceph-mon[33213]: osdmap e247: 8 total, 7 up, 7 in 2024-02-05T07:26:53.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:53 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:26:53.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:53 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:26:53.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:53.576 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:53.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:53.651 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:26:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:26:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:26:53.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:53.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:53.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:53.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:53.826 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:53.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:53.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:53.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:53.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:53.952 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:53.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:54.033 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 83 16 130042 130026 97.895 11.75 0.0132639 0.0101089 2024-02-05T07:26:54.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:54.081 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:54.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:54.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:54.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:54.214 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:54.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:54.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:54.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:54.352 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:54.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:54.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:54 smithi007 ceph-mon[29098]: pgmap v387: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 62 MiB/s wr, 997 op/s; 39827/381984 objects degraded (10.426%) 2024-02-05T07:26:54.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:54 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:26:54.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:54 smithi007 ceph-mon[29098]: osdmap e248: 8 total, 7 up, 7 in 2024-02-05T07:26:54.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:54 smithi007 ceph-mon[37158]: pgmap v387: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 62 MiB/s wr, 997 op/s; 39827/381984 objects degraded (10.426%) 2024-02-05T07:26:54.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:54 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:26:54.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:54 smithi007 ceph-mon[37158]: osdmap e248: 8 total, 7 up, 7 in 2024-02-05T07:26:54.454 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:54.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:54.495 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:54.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:54.520 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:54 smithi090 ceph-mon[33213]: pgmap v387: 158 pgs: 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 62 MiB/s wr, 997 op/s; 39827/381984 objects degraded (10.426%) 2024-02-05T07:26:54.520 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:54 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:26:54.520 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:54 smithi090 ceph-mon[33213]: osdmap e248: 8 total, 7 up, 7 in 2024-02-05T07:26:54.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:54.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:54.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:54.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:54.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:54.803 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:54.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:54.904 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:54.951 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:54.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:55.033 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 84 16 131518 131502 97.8277 92.25 0.00189091 0.0102141 2024-02-05T07:26:55.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:55.090 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:55.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:55.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:55.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:55.237 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:55.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:55.338 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:26:55.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:55.357 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:26:55.366 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:55.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:55.379 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:26:55.399 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:26:55.419 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:26:55.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:26:55.439 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:26:55.450 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:55 smithi007 ceph-mon[37158]: osdmap e249: 8 total, 7 up, 7 in 2024-02-05T07:26:55.450 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:55 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 39827/381984 objects degraded (10.426%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:55.450 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:55 smithi007 ceph-mon[37158]: osdmap e250: 8 total, 7 up, 7 in 2024-02-05T07:26:55.450 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:55 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:26:55.450 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:55 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:26:55.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:55 smithi007 ceph-mon[29098]: osdmap e249: 8 total, 7 up, 7 in 2024-02-05T07:26:55.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:55 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 39827/381984 objects degraded (10.426%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:55.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:55 smithi007 ceph-mon[29098]: osdmap e250: 8 total, 7 up, 7 in 2024-02-05T07:26:55.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:55 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:26:55.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:55 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:26:55.461 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-02-05T07:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:26:55.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:55.479 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:55 smithi090 ceph-mon[33213]: osdmap e249: 8 total, 7 up, 7 in 2024-02-05T07:26:55.479 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:55 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 39827/381984 objects degraded (10.426%), 29 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:55.479 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:55 smithi090 ceph-mon[33213]: osdmap e250: 8 total, 7 up, 7 in 2024-02-05T07:26:55.480 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:55 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:26:55.480 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:55 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:26:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:55.499 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:55.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:55.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:55.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:55.626 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:55.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:55.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:55.753 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:55.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:55.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:55.865 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:26:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:26:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:26:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:26:55.865 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:26:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:55.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:55.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:55.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:56.006 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:56.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:56.033 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 85 16 131887 131871 96.9481 23.0625 0.00224326 0.0102199 2024-02-05T07:26:56.046 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T07:26:56.046 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T07:26:56.064 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:56.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:56.133 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:56.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:56.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:56.264 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:56.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:56.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:56.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:56 smithi007 ceph-mon[29098]: pgmap v391: 168 pgs: 10 unknown, 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 39827/381984 objects degraded (10.426%) 2024-02-05T07:26:56.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:56 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T07:26:56.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:56 smithi007 ceph-mon[29098]: osdmap e251: 8 total, 7 up, 7 in 2024-02-05T07:26:56.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:56 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/861656168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:56.285 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:56 smithi007 ceph-mon[37158]: pgmap v391: 168 pgs: 10 unknown, 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 39827/381984 objects degraded (10.426%) 2024-02-05T07:26:56.285 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:56 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T07:26:56.285 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:56 smithi007 ceph-mon[37158]: osdmap e251: 8 total, 7 up, 7 in 2024-02-05T07:26:56.285 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:56 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/861656168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:56.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:56.390 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:56.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:56.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:56.429 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:56 smithi090 ceph-mon[33213]: pgmap v391: 168 pgs: 10 unknown, 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 39827/381984 objects degraded (10.426%) 2024-02-05T07:26:56.429 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:56 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T07:26:56.429 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:56 smithi090 ceph-mon[33213]: osdmap e251: 8 total, 7 up, 7 in 2024-02-05T07:26:56.429 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:56 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/861656168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:26:56.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T07:26:56.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:56.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:56.518 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:56.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:56.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:56.643 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:56.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:56.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:56.730 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:26:56.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:56.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:56.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:56.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:56.896 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:56.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:56.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:56.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:26:57.022 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:57.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:26:57.033 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 86 16 131887 131871 95.8209 0 - 0.0102199 2024-02-05T07:26:57.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:57.148 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:57.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:57.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:57.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:57.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:57.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:57.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:57.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:57.419 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:57.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:57.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:57.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:57 smithi090 ceph-mon[33213]: osdmap e252: 8 total, 7 up, 7 in 2024-02-05T07:26:57.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:57 smithi007 ceph-mon[29098]: osdmap e252: 8 total, 7 up, 7 in 2024-02-05T07:26:57.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:57 smithi007 ceph-mon[37158]: osdmap e252: 8 total, 7 up, 7 in 2024-02-05T07:26:57.560 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:57.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:57.663 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:57.717 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:57.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:57.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:57.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:57.856 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:57.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:57.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:57.871 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-02-05T07:26:57.871 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:26:57.872 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 4, 5, 6, 7] 2024-02-05T07:26:57.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd out 5 2024-02-05T07:26:57.957 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:58.003 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:58.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:58.033 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 87 16 132918 132902 95.4601 32.2188 0.0159486 0.0104711 2024-02-05T07:26:58.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:26:58.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:58.142 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:58.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:58.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:58.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:58.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:58.397 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:58.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:58.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:58.463 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:58.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:58.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[29098]: pgmap v394: 168 pgs: 7 unknown, 30 active+undersized+degraded, 40 active+undersized, 91 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 24899/322059 objects degraded (7.731%) 2024-02-05T07:26:58.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[29098]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 250'8128 2024-02-05T07:26:58.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[29098]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 250'8204 2024-02-05T07:26:58.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[29098]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 250'8279 2024-02-05T07:26:58.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[29098]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 250'8237 2024-02-05T07:26:58.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[29098]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 250'8181 2024-02-05T07:26:58.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[29098]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 250'7996 2024-02-05T07:26:58.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[29098]: osdmap e253: 8 total, 7 up, 7 in 2024-02-05T07:26:58.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[29098]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 250'8167 2024-02-05T07:26:58.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[29098]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 250'8314 2024-02-05T07:26:58.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[29098]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 250'8277 2024-02-05T07:26:58.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[29098]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 250'8157 2024-02-05T07:26:58.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[29098]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 250'8277 2024-02-05T07:26:58.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[29098]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 250'8157 2024-02-05T07:26:58.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[37158]: pgmap v394: 168 pgs: 7 unknown, 30 active+undersized+degraded, 40 active+undersized, 91 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 24899/322059 objects degraded (7.731%) 2024-02-05T07:26:58.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[37158]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 250'8128 2024-02-05T07:26:58.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[37158]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 250'8204 2024-02-05T07:26:58.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[37158]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 250'8279 2024-02-05T07:26:58.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[37158]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 250'8237 2024-02-05T07:26:58.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[37158]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 250'8181 2024-02-05T07:26:58.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[37158]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 250'7996 2024-02-05T07:26:58.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[37158]: osdmap e253: 8 total, 7 up, 7 in 2024-02-05T07:26:58.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[37158]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 250'8167 2024-02-05T07:26:58.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[37158]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 250'8314 2024-02-05T07:26:58.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[37158]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 250'8277 2024-02-05T07:26:58.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[37158]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 250'8157 2024-02-05T07:26:58.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[37158]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 250'8277 2024-02-05T07:26:58.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:58 smithi007 ceph-mon[37158]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 250'8157 2024-02-05T07:26:58.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:26:58.572 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:58 smithi090 ceph-mon[33213]: pgmap v394: 168 pgs: 7 unknown, 30 active+undersized+degraded, 40 active+undersized, 91 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 24899/322059 objects degraded (7.731%) 2024-02-05T07:26:58.572 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:58 smithi090 ceph-mon[33213]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 250'8128 2024-02-05T07:26:58.572 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:58 smithi090 ceph-mon[33213]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 250'8204 2024-02-05T07:26:58.572 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:58 smithi090 ceph-mon[33213]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 250'8279 2024-02-05T07:26:58.572 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:58 smithi090 ceph-mon[33213]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 250'8237 2024-02-05T07:26:58.572 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:58 smithi090 ceph-mon[33213]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 250'8181 2024-02-05T07:26:58.573 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:58 smithi090 ceph-mon[33213]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 250'7996 2024-02-05T07:26:58.573 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:58 smithi090 ceph-mon[33213]: osdmap e253: 8 total, 7 up, 7 in 2024-02-05T07:26:58.573 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:58 smithi090 ceph-mon[33213]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 250'8167 2024-02-05T07:26:58.573 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:58 smithi090 ceph-mon[33213]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 250'8314 2024-02-05T07:26:58.573 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:58 smithi090 ceph-mon[33213]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 250'8277 2024-02-05T07:26:58.573 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:58 smithi090 ceph-mon[33213]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 250'8157 2024-02-05T07:26:58.573 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:58 smithi090 ceph-mon[33213]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 250'8277 2024-02-05T07:26:58.573 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:58 smithi090 ceph-mon[33213]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 250'8157 2024-02-05T07:26:58.602 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:58.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:58.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:26:58.677 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:26:58.677 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-02-05T07:26:58.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-02-05T07:26:58.677 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:26:58.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.677 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:28545 7.1 7:8e820b4b:::benchmark_data_smithi007_77086_object28544:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-02-05T07:26:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:21:09.276713+0000", 2024-02-05T07:26:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 349.39339681600001, 2024-02-05T07:26:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16734980899999999, 2024-02-05T07:26:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:26:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 28545 2024-02-05T07:26:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:09.276713+0000", 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:09.276713+0000", 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:09.276714+0000", 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4249999999999999e-06 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:09.276771+0000", 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6282000000000001e-05 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:09.276783+0000", 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2413e-05 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:09.276791+0000", 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8979999999999996e-06 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:09.276851+0000", 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9515999999999997e-05 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:09.276975+0000", 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012400200000000001 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:09.277118+0000", 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014309500000000001 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:09.278189+0000", 2024-02-05T07:26:58.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001071801 2024-02-05T07:26:58.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:09.278628+0000", 2024-02-05T07:26:58.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043828699999999999 2024-02-05T07:26:58.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:09.443978+0000", 2024-02-05T07:26:58.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16535069099999999 2024-02-05T07:26:58.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:09.444023+0000", 2024-02-05T07:26:58.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5046000000000002e-05 2024-02-05T07:26:58.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:09.444063+0000", 2024-02-05T07:26:58.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9353000000000001e-05 2024-02-05T07:26:58.683 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.683 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.683 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.683 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:75596 7.16 7.8a89f676 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:21:43.610078+0000", 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 315.06003229700002, 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52161623000000001, 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 75596 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.610078+0000", 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.610078+0000", 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.610079+0000", 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9e-06 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.610108+0000", 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8960999999999999e-05 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.610124+0000", 2024-02-05T07:26:58.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5863999999999999e-05 2024-02-05T07:26:58.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.610133+0000", 2024-02-05T07:26:58.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2500000000000006e-06 2024-02-05T07:26:58.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.611468+0000", 2024-02-05T07:26:58.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001335101 2024-02-05T07:26:58.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.611476+0000", 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0860000000000002e-06 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:44.131681+0000", 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52020523200000002 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:44.131694+0000", 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2836e-05 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.687 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.688 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.688 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:75638 7.16 7.35066536 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-02-05T07:26:58.688 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:21:43.670802+0000", 2024-02-05T07:26:58.688 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 314.99930769999997, 2024-02-05T07:26:58.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.460921578, 2024-02-05T07:26:58.688 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.688 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:26:58.688 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.688 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:26:58.688 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:58.688 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 75638 2024-02-05T07:26:58.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.689 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.670802+0000", 2024-02-05T07:26:58.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.670802+0000", 2024-02-05T07:26:58.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.670804+0000", 2024-02-05T07:26:58.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.424e-06 2024-02-05T07:26:58.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.670832+0000", 2024-02-05T07:26:58.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.847e-05 2024-02-05T07:26:58.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.670844+0000", 2024-02-05T07:26:58.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2139e-05 2024-02-05T07:26:58.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.670851+0000", 2024-02-05T07:26:58.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9160000000000004e-06 2024-02-05T07:26:58.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.670900+0000", 2024-02-05T07:26:58.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8653000000000003e-05 2024-02-05T07:26:58.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.670906+0000", 2024-02-05T07:26:58.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2609999999999999e-06 2024-02-05T07:26:58.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:44.131717+0000", 2024-02-05T07:26:58.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.460811154 2024-02-05T07:26:58.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:44.131724+0000", 2024-02-05T07:26:58.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5610000000000004e-06 2024-02-05T07:26:58.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.692 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.693 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:175604 7.1 7:82a7e554:::benchmark_data_smithi007_77086_object71009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-05T07:26:58.693 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:03.357587+0000", 2024-02-05T07:26:58.693 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 175.31252277199999, 2024-02-05T07:26:58.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14676336100000001, 2024-02-05T07:26:58.693 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.693 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.693 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.693 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:26:58.693 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:58.693 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 175604 2024-02-05T07:26:58.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.693 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.357587+0000", 2024-02-05T07:26:58.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.357587+0000", 2024-02-05T07:26:58.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.357588+0000", 2024-02-05T07:26:58.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4410000000000001e-06 2024-02-05T07:26:58.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.357594+0000", 2024-02-05T07:26:58.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6180000000000003e-06 2024-02-05T07:26:58.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.357604+0000", 2024-02-05T07:26:58.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4679999999999998e-06 2024-02-05T07:26:58.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.357609+0000", 2024-02-05T07:26:58.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.463e-06 2024-02-05T07:26:58.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.357653+0000", 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4249999999999998e-05 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.357742+0000", 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9104999999999995e-05 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3,6", 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.358057+0000", 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031432900000000002 2024-02-05T07:26:58.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.359583+0000", 2024-02-05T07:26:58.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015264160000000001 2024-02-05T07:26:58.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.359644+0000", 2024-02-05T07:26:58.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0559000000000001e-05 2024-02-05T07:26:58.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.360407+0000", 2024-02-05T07:26:58.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000763778 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.504122+0000", 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14371465999999999 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.504226+0000", 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103708 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:03.504350+0000", 2024-02-05T07:26:58.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124566 2024-02-05T07:26:58.699 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.699 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.699 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.699 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:177443 7.1 7:81e76abb:::benchmark_data_smithi007_77086_object72848:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-05T07:26:58.699 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:05.702070+0000", 2024-02-05T07:26:58.699 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.96803961800001, 2024-02-05T07:26:58.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16194760899999999, 2024-02-05T07:26:58.699 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.699 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.699 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.699 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:26:58.699 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:58.700 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 177443 2024-02-05T07:26:58.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.700 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.702070+0000", 2024-02-05T07:26:58.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.702070+0000", 2024-02-05T07:26:58.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.702071+0000", 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.274e-06 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.702075+0000", 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8199999999999998e-06 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.702084+0000", 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8189999999999998e-06 2024-02-05T07:26:58.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.702089+0000", 2024-02-05T07:26:58.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3580000000000002e-06 2024-02-05T07:26:58.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.702160+0000", 2024-02-05T07:26:58.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0619999999999998e-05 2024-02-05T07:26:58.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.702260+0000", 2024-02-05T07:26:58.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010001 2024-02-05T07:26:58.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3,6", 2024-02-05T07:26:58.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.702591+0000", 2024-02-05T07:26:58.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000331105 2024-02-05T07:26:58.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.705577+0000", 2024-02-05T07:26:58.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029853610000000002 2024-02-05T07:26:58.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.706903+0000", 2024-02-05T07:26:58.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00132676 2024-02-05T07:26:58.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.707614+0000", 2024-02-05T07:26:58.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:26:58.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071026100000000001 2024-02-05T07:26:58.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.863851+0000", 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15623695700000001 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.863919+0000", 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8806999999999994e-05 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.864018+0000", 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8456999999999996e-05 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.706 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.707 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.707 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:177444 7.1 7:848d2ddc:::benchmark_data_smithi007_77086_object72849:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-05T07:26:58.707 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:05.703331+0000", 2024-02-05T07:26:58.707 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.966778527, 2024-02-05T07:26:58.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.160856633, 2024-02-05T07:26:58.707 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.707 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.707 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.707 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:26:58.707 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:58.707 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 177444 2024-02-05T07:26:58.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.708 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.703331+0000", 2024-02-05T07:26:58.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.703331+0000", 2024-02-05T07:26:58.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.703332+0000", 2024-02-05T07:26:58.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2099999999999995e-07 2024-02-05T07:26:58.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.703336+0000", 2024-02-05T07:26:58.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0389999999999998e-06 2024-02-05T07:26:58.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.703345+0000", 2024-02-05T07:26:58.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3130000000000004e-06 2024-02-05T07:26:58.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.703353+0000", 2024-02-05T07:26:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1749999999999999e-06 2024-02-05T07:26:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.704198+0000", 2024-02-05T07:26:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084553000000000004 2024-02-05T07:26:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.704280+0000", 2024-02-05T07:26:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1557999999999994e-05 2024-02-05T07:26:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3,6", 2024-02-05T07:26:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.704493+0000", 2024-02-05T07:26:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000213475 2024-02-05T07:26:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.705692+0000", 2024-02-05T07:26:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001198373 2024-02-05T07:26:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.707206+0000", 2024-02-05T07:26:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015144119999999999 2024-02-05T07:26:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.707782+0000", 2024-02-05T07:26:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057572199999999997 2024-02-05T07:26:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.864056+0000", 2024-02-05T07:26:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15627392200000001 2024-02-05T07:26:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.864089+0000", 2024-02-05T07:26:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3772999999999997e-05 2024-02-05T07:26:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:05.864188+0000", 2024-02-05T07:26:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8519999999999999e-05 2024-02-05T07:26:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:178372 7.16 7:6df6d163:::benchmark_data_smithi007_77086_object73777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-05T07:26:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:06.778185+0000", 2024-02-05T07:26:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 171.891924609, 2024-02-05T07:26:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15182461899999999, 2024-02-05T07:26:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:26:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 178372 2024-02-05T07:26:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:06.778185+0000", 2024-02-05T07:26:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:06.778185+0000", 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:06.778186+0000", 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1880000000000001e-06 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:06.778191+0000", 2024-02-05T07:26:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7160000000000002e-06 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:06.778201+0000", 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7289999999999995e-06 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:06.778208+0000", 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4660000000000002e-06 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:06.778242+0000", 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3234000000000001e-05 2024-02-05T07:26:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:06.778319+0000", 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7746999999999999e-05 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:06.778553+0000", 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023392500000000001 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:06.780504+0000", 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001950855 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:06.781490+0000", 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00098579200000000009 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:06.929856+0000", 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14836617099999999 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:06.929941+0000", 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5364999999999993e-05 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:06.930010+0000", 2024-02-05T07:26:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8430999999999996e-05 2024-02-05T07:26:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:178859 7.1 7:840ffd1f:::benchmark_data_smithi007_77086_object74264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-05T07:26:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:07.302340+0000", 2024-02-05T07:26:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 171.367769329, 2024-02-05T07:26:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18359255099999999, 2024-02-05T07:26:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 178859 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.302340+0000", 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.302340+0000", 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.302342+0000", 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3200000000000001e-06 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.302347+0000", 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1440000000000002e-06 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.302359+0000", 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2002e-05 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.302367+0000", 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5110000000000003e-06 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.302430+0000", 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2796e-05 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.302566+0000", 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013560800000000001 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3,6", 2024-02-05T07:26:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.302941+0000", 2024-02-05T07:26:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037543100000000002 2024-02-05T07:26:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.304945+0000", 2024-02-05T07:26:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002004062 2024-02-05T07:26:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.305317+0000", 2024-02-05T07:26:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037176199999999998 2024-02-05T07:26:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.306042+0000", 2024-02-05T07:26:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072519900000000005 2024-02-05T07:26:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.485288+0000", 2024-02-05T07:26:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17924576 2024-02-05T07:26:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.485359+0000", 2024-02-05T07:26:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1043000000000004e-05 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.485933+0000", 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057391300000000001 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:178866 7.16 7:6e2d0d11:::benchmark_data_smithi007_77086_object74271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:07.309571+0000", 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 171.36053849499999, 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.181544235, 2024-02-05T07:26:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 178866 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.309571+0000", 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.309571+0000", 2024-02-05T07:26:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.309572+0000", 2024-02-05T07:26:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.161e-06 2024-02-05T07:26:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.309576+0000", 2024-02-05T07:26:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9179999999999996e-06 2024-02-05T07:26:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.309583+0000", 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0600000000000002e-06 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.309588+0000", 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7949999999999998e-06 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.309637+0000", 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8785000000000001e-05 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.309717+0000", 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0311999999999999e-05 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.310004+0000", 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028623599999999999 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.311391+0000", 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001387847 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.312509+0000", 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001117706 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.490950+0000", 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17844036099999999 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.491067+0000", 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011709099999999999 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:07.491116+0000", 2024-02-05T07:26:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8962999999999999e-05 2024-02-05T07:26:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192530 7.1 7:825252f9:::benchmark_data_smithi007_77086_object87935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-02-05T07:26:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:19.535985+0000", 2024-02-05T07:26:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 159.134125294, 2024-02-05T07:26:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0065911670000001, 2024-02-05T07:26:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:26:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192530 2024-02-05T07:26:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.535985+0000", 2024-02-05T07:26:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.535985+0000", 2024-02-05T07:26:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.535987+0000", 2024-02-05T07:26:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2989999999999999e-06 2024-02-05T07:26:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.535994+0000", 2024-02-05T07:26:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0600000000000002e-06 2024-02-05T07:26:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.536013+0000", 2024-02-05T07:26:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8879e-05 2024-02-05T07:26:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.536024+0000", 2024-02-05T07:26:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0939e-05 2024-02-05T07:26:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.605023+0000", 2024-02-05T07:26:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068999788000000006 2024-02-05T07:26:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.605036+0000", 2024-02-05T07:26:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2096000000000001e-05 2024-02-05T07:26:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.532765+0000", 2024-02-05T07:26:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.92772923200000001 2024-02-05T07:26:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.532903+0000", 2024-02-05T07:26:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013795800000000001 2024-02-05T07:26:58.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3,6", 2024-02-05T07:26:58.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.533185+0000", 2024-02-05T07:26:58.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028267800000000002 2024-02-05T07:26:58.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.535365+0000", 2024-02-05T07:26:58.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021794639999999999 2024-02-05T07:26:58.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.535821+0000", 2024-02-05T07:26:58.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000456367 2024-02-05T07:26:58.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.541731+0000", 2024-02-05T07:26:58.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005909882 2024-02-05T07:26:58.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.542377+0000", 2024-02-05T07:26:58.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064597999999999999 2024-02-05T07:26:58.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.542446+0000", 2024-02-05T07:26:58.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8876999999999998e-05 2024-02-05T07:26:58.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.542576+0000", 2024-02-05T07:26:58.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012966799999999999 2024-02-05T07:26:58.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.739 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.739 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192550 7.21 7:842b01ab:::benchmark_data_smithi007_77086_object87955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:26:58.739 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:20.583755+0000", 2024-02-05T07:26:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 158.086355201, 2024-02-05T07:26:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9895773640000001, 2024-02-05T07:26:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:26:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192550 2024-02-05T07:26:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583755+0000", 2024-02-05T07:26:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583755+0000", 2024-02-05T07:26:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583756+0000", 2024-02-05T07:26:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.84e-06 2024-02-05T07:26:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583764+0000", 2024-02-05T07:26:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6990000000000003e-06 2024-02-05T07:26:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583785+0000", 2024-02-05T07:26:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0372000000000001e-05 2024-02-05T07:26:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583795+0000", 2024-02-05T07:26:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0169e-05 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583870+0000", 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5587999999999995e-05 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583882+0000", 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1932e-05 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511843+0000", 2024-02-05T07:26:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.92796104800000001 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511855+0000", 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1688999999999999e-05 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.569583+0000", 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.057728024 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.569672+0000", 2024-02-05T07:26:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9382000000000006e-05 2024-02-05T07:26:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3,5,6,7", 2024-02-05T07:26:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.569832+0000", 2024-02-05T07:26:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159643 2024-02-05T07:26:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.572963+0000", 2024-02-05T07:26:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031314300000000002 2024-02-05T07:26:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.572979+0000", 2024-02-05T07:26:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5194999999999999e-05 2024-02-05T07:26:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573014+0000", 2024-02-05T07:26:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5487000000000001e-05 2024-02-05T07:26:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573063+0000", 2024-02-05T07:26:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8714999999999997e-05 2024-02-05T07:26:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573110+0000", 2024-02-05T07:26:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7335e-05 2024-02-05T07:26:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573153+0000", 2024-02-05T07:26:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2536000000000001e-05 2024-02-05T07:26:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573215+0000", 2024-02-05T07:26:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2613000000000005e-05 2024-02-05T07:26:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573332+0000", 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116667 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192659 7.21 7:877b6383:::benchmark_data_smithi007_77086_object88064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:20.678146+0000", 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 157.991964317, 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8957312879999999, 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192659 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678146+0000", 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678146+0000", 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678147+0000", 2024-02-05T07:26:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.238e-06 2024-02-05T07:26:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678151+0000", 2024-02-05T07:26:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5249999999999999e-06 2024-02-05T07:26:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678165+0000", 2024-02-05T07:26:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3574000000000001e-05 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678172+0000", 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5730000000000002e-06 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678231+0000", 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8307000000000003e-05 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678242+0000", 2024-02-05T07:26:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1629e-05 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511879+0000", 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.83363663499999996 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511886+0000", 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0520000000000004e-06 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.570887+0000", 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.059001149 2024-02-05T07:26:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.570958+0000", 2024-02-05T07:26:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0562000000000006e-05 2024-02-05T07:26:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3,5,6,7", 2024-02-05T07:26:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.571083+0000", 2024-02-05T07:26:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124936 2024-02-05T07:26:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573027+0000", 2024-02-05T07:26:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019443329999999999 2024-02-05T07:26:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573502+0000", 2024-02-05T07:26:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047502299999999998 2024-02-05T07:26:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573538+0000", 2024-02-05T07:26:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6158000000000002e-05 2024-02-05T07:26:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573572+0000", 2024-02-05T07:26:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3832999999999999e-05 2024-02-05T07:26:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573604+0000", 2024-02-05T07:26:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1816000000000002e-05 2024-02-05T07:26:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573759+0000", 2024-02-05T07:26:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015536099999999999 2024-02-05T07:26:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573792+0000", 2024-02-05T07:26:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2295999999999997e-05 2024-02-05T07:26:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573877+0000", 2024-02-05T07:26:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5288000000000001e-05 2024-02-05T07:26:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192660 7.21 7:842a1e92:::benchmark_data_smithi007_77086_object88065:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:26:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:20.678525+0000", 2024-02-05T07:26:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 157.99158513699999, 2024-02-05T07:26:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.895505679, 2024-02-05T07:26:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:26:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192660 2024-02-05T07:26:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678525+0000", 2024-02-05T07:26:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678525+0000", 2024-02-05T07:26:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678526+0000", 2024-02-05T07:26:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.034e-06 2024-02-05T07:26:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678533+0000", 2024-02-05T07:26:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3159999999999999e-06 2024-02-05T07:26:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678547+0000", 2024-02-05T07:26:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4080000000000001e-05 2024-02-05T07:26:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678555+0000", 2024-02-05T07:26:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0960000000000001e-06 2024-02-05T07:26:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678630+0000", 2024-02-05T07:26:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.483e-05 2024-02-05T07:26:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678641+0000", 2024-02-05T07:26:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1337e-05 2024-02-05T07:26:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511906+0000", 2024-02-05T07:26:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.83326460499999999 2024-02-05T07:26:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511912+0000", 2024-02-05T07:26:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.772e-06 2024-02-05T07:26:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.571802+0000", 2024-02-05T07:26:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0598903289999999 2024-02-05T07:26:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.571888+0000", 2024-02-05T07:26:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5929999999999999e-05 2024-02-05T07:26:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3,5,6,7", 2024-02-05T07:26:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.572061+0000", 2024-02-05T07:26:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000173215 2024-02-05T07:26:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573076+0000", 2024-02-05T07:26:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001014694 2024-02-05T07:26:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573637+0000", 2024-02-05T07:26:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056068600000000002 2024-02-05T07:26:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573668+0000", 2024-02-05T07:26:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1408000000000002e-05 2024-02-05T07:26:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573698+0000", 2024-02-05T07:26:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9501000000000001e-05 2024-02-05T07:26:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573729+0000", 2024-02-05T07:26:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1223000000000002e-05 2024-02-05T07:26:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573915+0000", 2024-02-05T07:26:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018607699999999999 2024-02-05T07:26:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573946+0000", 2024-02-05T07:26:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1276999999999999e-05 2024-02-05T07:26:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.574030+0000", 2024-02-05T07:26:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4269000000000002e-05 2024-02-05T07:26:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192734 7.21 7:8768831f:::benchmark_data_smithi007_77086_object88139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:26:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:20.790637+0000", 2024-02-05T07:26:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 157.879472894, 2024-02-05T07:26:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.793210755, 2024-02-05T07:26:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:26:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:26:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192734 2024-02-05T07:26:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790637+0000", 2024-02-05T07:26:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790637+0000", 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790638+0000", 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.514e-06 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790645+0000", 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1449999999999996e-06 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790658+0000", 2024-02-05T07:26:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3069e-05 2024-02-05T07:26:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790666+0000", 2024-02-05T07:26:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6710000000000005e-06 2024-02-05T07:26:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790717+0000", 2024-02-05T07:26:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0182000000000002e-05 2024-02-05T07:26:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790723+0000", 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1070000000000001e-06 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511933+0000", 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72121072500000005 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511940+0000", 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0199999999999997e-06 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.574092+0000", 2024-02-05T07:26:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0621517819999999 2024-02-05T07:26:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.574168+0000", 2024-02-05T07:26:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5499000000000003e-05 2024-02-05T07:26:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3,5,6,7", 2024-02-05T07:26:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.574287+0000", 2024-02-05T07:26:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118989 2024-02-05T07:26:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.576095+0000", 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018082619999999999 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.576134+0000", 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8884999999999998e-05 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.576145+0000", 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1083e-05 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.576175+0000", 2024-02-05T07:26:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0042000000000001e-05 2024-02-05T07:26:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.576199+0000", 2024-02-05T07:26:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4209000000000001e-05 2024-02-05T07:26:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.583728+0000", 2024-02-05T07:26:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075287940000000001 2024-02-05T07:26:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.583766+0000", 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7783000000000001e-05 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.583848+0000", 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1994000000000001e-05 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:43570 8.b 8:dedbd279:::benchmark_data_smithi007_105755_object43569:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-02-05T07:26:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:25:49.617185+0000", 2024-02-05T07:26:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.052925278999993, 2024-02-05T07:26:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14880183799999999, 2024-02-05T07:26:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:26:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:26:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 43570 2024-02-05T07:26:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.617185+0000", 2024-02-05T07:26:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.617185+0000", 2024-02-05T07:26:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.617185+0000", 2024-02-05T07:26:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1299999999999999e-07 2024-02-05T07:26:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.617242+0000", 2024-02-05T07:26:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6925999999999997e-05 2024-02-05T07:26:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.617250+0000", 2024-02-05T07:26:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4479999999999997e-06 2024-02-05T07:26:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.617257+0000", 2024-02-05T07:26:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.985e-06 2024-02-05T07:26:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.617308+0000", 2024-02-05T07:26:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1351999999999999e-05 2024-02-05T07:26:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.617404+0000", 2024-02-05T07:26:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6206999999999996e-05 2024-02-05T07:26:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.756324+0000", 2024-02-05T07:26:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13891964400000001 2024-02-05T07:26:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.762462+0000", 2024-02-05T07:26:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061386280000000001 2024-02-05T07:26:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.762504+0000", 2024-02-05T07:26:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1511000000000003e-05 2024-02-05T07:26:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.765928+0000", 2024-02-05T07:26:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034235559999999999 2024-02-05T07:26:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.765963+0000", 2024-02-05T07:26:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5012000000000003e-05 2024-02-05T07:26:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.765986+0000", 2024-02-05T07:26:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3856e-05 2024-02-05T07:26:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:43573 8.b 8:da32ce35:::benchmark_data_smithi007_105755_object43572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-02-05T07:26:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:25:49.617721+0000", 2024-02-05T07:26:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.052388716999999, 2024-02-05T07:26:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.148395783, 2024-02-05T07:26:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:26:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:26:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 43573 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.617721+0000", 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.617721+0000", 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.617722+0000", 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7900000000000007e-07 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.617786+0000", 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4348000000000001e-05 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.754502+0000", 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13671530100000001 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.754523+0000", 2024-02-05T07:26:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1262000000000001e-05 2024-02-05T07:26:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.758039+0000", 2024-02-05T07:26:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035160439999999999 2024-02-05T07:26:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.758160+0000", 2024-02-05T07:26:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121217 2024-02-05T07:26:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:26:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.758295+0000", 2024-02-05T07:26:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013490799999999999 2024-02-05T07:26:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.762548+0000", 2024-02-05T07:26:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042529630000000002 2024-02-05T07:26:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.765994+0000", 2024-02-05T07:26:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034459170000000002 2024-02-05T07:26:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.766037+0000", 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2676000000000003e-05 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.766084+0000", 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7345000000000002e-05 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:25:49.766117+0000", 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2823000000000001e-05 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.787 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.788 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.788 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:131844 8.12 8:4b3718d0:::benchmark_data_smithi007_105755_object131843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:26:58.788 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:56.264093+0000", 2024-02-05T07:26:58.788 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.4060173150000002, 2024-02-05T07:26:58.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.079539735, 2024-02-05T07:26:58.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.788 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:26:58.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:26:58.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:26:58.789 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131844 2024-02-05T07:26:58.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.789 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264093+0000", 2024-02-05T07:26:58.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264093+0000", 2024-02-05T07:26:58.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264094+0000", 2024-02-05T07:26:58.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.443e-06 2024-02-05T07:26:58.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264295+0000", 2024-02-05T07:26:58.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020106599999999999 2024-02-05T07:26:58.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264323+0000", 2024-02-05T07:26:58.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7719999999999999e-05 2024-02-05T07:26:58.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264343+0000", 2024-02-05T07:26:58.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0304000000000001e-05 2024-02-05T07:26:58.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265133+0000", 2024-02-05T07:26:58.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079006200000000001 2024-02-05T07:26:58.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265141+0000", 2024-02-05T07:26:58.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3299999999999999e-06 2024-02-05T07:26:58.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.337772+0000", 2024-02-05T07:26:58.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0726301060000001 2024-02-05T07:26:58.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-02-05T07:26:58.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.339454+0000", 2024-02-05T07:26:58.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001682565 2024-02-05T07:26:58.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343536+0000", 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040822530000000001 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343632+0000", 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5885999999999996e-05 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:131867 8.12 8:4f7063db:::benchmark_data_smithi007_105755_object131866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:56.264377+0000", 2024-02-05T07:26:58.793 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.4057331990000002, 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.089266753, 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131867 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264377+0000", 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264377+0000", 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264377+0000", 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3799999999999996e-07 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264419+0000", 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1569000000000001e-05 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264426+0000", 2024-02-05T07:26:58.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1890000000000001e-06 2024-02-05T07:26:58.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264433+0000", 2024-02-05T07:26:58.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5039999999999999e-06 2024-02-05T07:26:58.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265185+0000", 2024-02-05T07:26:58.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075286500000000002 2024-02-05T07:26:58.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265194+0000", 2024-02-05T07:26:58.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6389999999999998e-06 2024-02-05T07:26:58.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.339734+0000", 2024-02-05T07:26:58.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0745397999999999 2024-02-05T07:26:58.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.339874+0000", 2024-02-05T07:26:58.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014015699999999999 2024-02-05T07:26:58.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2,5,7", 2024-02-05T07:26:58.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.340138+0000", 2024-02-05T07:26:58.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026435199999999998 2024-02-05T07:26:58.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343625+0000", 2024-02-05T07:26:58.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003486669 2024-02-05T07:26:58.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.348328+0000", 2024-02-05T07:26:58.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047024069999999996 2024-02-05T07:26:58.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.348374+0000", 2024-02-05T07:26:58.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6669999999999999e-05 2024-02-05T07:26:58.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.351568+0000", 2024-02-05T07:26:58.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031942749999999999 2024-02-05T07:26:58.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353402+0000", 2024-02-05T07:26:58.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018333970000000001 2024-02-05T07:26:58.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353476+0000", 2024-02-05T07:26:58.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4153000000000006e-05 2024-02-05T07:26:58.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353643+0000", 2024-02-05T07:26:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000167369 2024-02-05T07:26:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:131870 8.12 8:4c9bd4f8:::benchmark_data_smithi007_105755_object131869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:26:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:56.264452+0000", 2024-02-05T07:26:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.4056581399999999, 2024-02-05T07:26:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0894224100000001, 2024-02-05T07:26:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.803 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:26:58.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:26:58.803 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131870 2024-02-05T07:26:58.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.803 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264452+0000", 2024-02-05T07:26:58.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264452+0000", 2024-02-05T07:26:58.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264452+0000", 2024-02-05T07:26:58.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7800000000000001e-07 2024-02-05T07:26:58.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264506+0000", 2024-02-05T07:26:58.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4051000000000001e-05 2024-02-05T07:26:58.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264513+0000", 2024-02-05T07:26:58.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8170000000000001e-06 2024-02-05T07:26:58.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264519+0000", 2024-02-05T07:26:58.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3029999999999996e-06 2024-02-05T07:26:58.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265227+0000", 2024-02-05T07:26:58.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070795899999999998 2024-02-05T07:26:58.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265234+0000", 2024-02-05T07:26:58.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7989999999999996e-06 2024-02-05T07:26:58.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.341221+0000", 2024-02-05T07:26:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.075986812 2024-02-05T07:26:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.341349+0000", 2024-02-05T07:26:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012759699999999999 2024-02-05T07:26:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2,5,7", 2024-02-05T07:26:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.341592+0000", 2024-02-05T07:26:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024358299999999999 2024-02-05T07:26:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343631+0000", 2024-02-05T07:26:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020391020000000001 2024-02-05T07:26:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.348414+0000", 2024-02-05T07:26:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047826880000000002 2024-02-05T07:26:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.349943+0000", 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015293609999999999 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.351610+0000", 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001667099 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353687+0000", 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00207632 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353750+0000", 2024-02-05T07:26:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2897000000000004e-05 2024-02-05T07:26:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353874+0000", 2024-02-05T07:26:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124444 2024-02-05T07:26:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:131873 8.12 8:4dbc2e4d:::benchmark_data_smithi007_105755_object131872:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:26:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:56.264529+0000", 2024-02-05T07:26:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.4055803820000001, 2024-02-05T07:26:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0895603920000001, 2024-02-05T07:26:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:26:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:26:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:26:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:26:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:26:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131873 2024-02-05T07:26:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:26:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:26:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264529+0000", 2024-02-05T07:26:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:26:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264529+0000", 2024-02-05T07:26:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:26:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:26:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264530+0000", 2024-02-05T07:26:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1900000000000002e-07 2024-02-05T07:26:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:26:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264555+0000", 2024-02-05T07:26:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5378e-05 2024-02-05T07:26:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:26:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264565+0000", 2024-02-05T07:26:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3559999999999999e-06 2024-02-05T07:26:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:26:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264571+0000", 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9100000000000002e-06 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265270+0000", 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069908399999999999 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265277+0000", 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8519999999999998e-06 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:26:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343935+0000", 2024-02-05T07:26:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0786580370000001 2024-02-05T07:26:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:26:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.344059+0000", 2024-02-05T07:26:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124413 2024-02-05T07:26:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2,5,7", 2024-02-05T07:26:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.344449+0000", 2024-02-05T07:26:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039028899999999999 2024-02-05T07:26:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:26:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.348341+0000", 2024-02-05T07:26:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038918870000000001 2024-02-05T07:26:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.349845+0000", 2024-02-05T07:26:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015036400000000001 2024-02-05T07:26:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.349985+0000", 2024-02-05T07:26:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139598 2024-02-05T07:26:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.351684+0000", 2024-02-05T07:26:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001699267 2024-02-05T07:26:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:26:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353913+0000", 2024-02-05T07:26:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022296479999999999 2024-02-05T07:26:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:26:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353968+0000", 2024-02-05T07:26:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4391999999999997e-05 2024-02-05T07:26:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:26:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:26:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:26:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.354090+0000", 2024-02-05T07:26:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012202200000000001 2024-02-05T07:26:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:26:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:26:58.818 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:26:58.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:58.820 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:58.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:26:58.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:58.953 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:58.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:58.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:59.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 88 16 134061 134045 95.1862 71.4375 0.00528407 0.0105013 2024-02-05T07:26:59.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:59.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:59.092 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:59.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:26:59.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:26:59.213 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:26:59.264 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:26:59.269 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:59.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:26:59.371 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:26:59.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:59.431 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:59.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:26:59.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:59.581 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:59.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:59.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[29098]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 250'8181 2024-02-05T07:26:59.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[29098]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 250'8237 2024-02-05T07:26:59.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3191236699' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:26:59.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:26:59.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:26:59.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 24899/322059 objects degraded (7.731%), 30 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:59.636 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:59.636 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:26:59.636 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[29098]: osdmap e254: 8 total, 7 up, 7 in 2024-02-05T07:26:59.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[37158]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 250'8181 2024-02-05T07:26:59.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[37158]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 250'8237 2024-02-05T07:26:59.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3191236699' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:26:59.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:26:59.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:26:59.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 24899/322059 objects degraded (7.731%), 30 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:59.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:59.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:26:59.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:26:59 smithi007 ceph-mon[37158]: osdmap e254: 8 total, 7 up, 7 in 2024-02-05T07:26:59.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:26:59.769 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:26:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:59.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:26:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:59 smithi090 ceph-mon[33213]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 250'8181 2024-02-05T07:26:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:59 smithi090 ceph-mon[33213]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 250'8237 2024-02-05T07:26:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:59 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3191236699' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:26:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:59 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:26:59.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:59 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:26:59.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:59 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 24899/322059 objects degraded (7.731%), 30 pgs degraded (PG_DEGRADED) 2024-02-05T07:26:59.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:59 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:26:59.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:59 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:26:59.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:26:59 smithi090 ceph-mon[33213]: osdmap e254: 8 total, 7 up, 7 in 2024-02-05T07:26:59.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:26:59.909 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:26:59.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:26:59.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:00.011 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:00.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 89 16 135196 135180 94.9137 70.9375 0.0126404 0.0105316 2024-02-05T07:27:00.049 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:00.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:00.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:00.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:00.177 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:00.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:00.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:00.318 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:00.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:00.366 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:00 smithi007 ceph-mon[29098]: pgmap v396: 168 pgs: 7 unknown, 30 active+undersized+degraded, 40 active+undersized, 91 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 24899/322059 objects degraded (7.731%) 2024-02-05T07:27:00.366 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:00 smithi007 ceph-mon[29098]: 5.17 scrub starts 2024-02-05T07:27:00.367 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:00 smithi007 ceph-mon[29098]: 5.17 scrub ok 2024-02-05T07:27:00.367 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:00 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:27:00.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:00 smithi007 ceph-mon[37158]: pgmap v396: 168 pgs: 7 unknown, 30 active+undersized+degraded, 40 active+undersized, 91 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 24899/322059 objects degraded (7.731%) 2024-02-05T07:27:00.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:00 smithi007 ceph-mon[37158]: 5.17 scrub starts 2024-02-05T07:27:00.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:00 smithi007 ceph-mon[37158]: 5.17 scrub ok 2024-02-05T07:27:00.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:00 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:27:00.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:00.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:00.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:00.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:00.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:27:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:27:00.572 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:00 smithi090 ceph-mon[33213]: pgmap v396: 168 pgs: 7 unknown, 30 active+undersized+degraded, 40 active+undersized, 91 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 24899/322059 objects degraded (7.731%) 2024-02-05T07:27:00.573 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:00 smithi090 ceph-mon[33213]: 5.17 scrub starts 2024-02-05T07:27:00.573 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:00 smithi090 ceph-mon[33213]: 5.17 scrub ok 2024-02-05T07:27:00.573 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:00 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:27:00.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:00.610 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:00.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:00.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T07:27:00.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:00.751 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:00.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:00.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:00.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:00.904 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:00.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:01.002 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:27:01.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:01.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total time run: 90.0094 2024-02-05T07:27:01.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total writes made: 136307 2024-02-05T07:27:01.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Write size: 65536 2024-02-05T07:27:01.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object size: 65536 2024-02-05T07:27:01.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Bandwidth (MB/sec): 94.6477 2024-02-05T07:27:01.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Bandwidth: 37.2254 2024-02-05T07:27:01.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max bandwidth (MB/sec): 238.312 2024-02-05T07:27:01.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min bandwidth (MB/sec): 0 2024-02-05T07:27:01.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average IOPS: 1514 2024-02-05T07:27:01.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev IOPS: 595.615 2024-02-05T07:27:01.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max IOPS: 3813 2024-02-05T07:27:01.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min IOPS: 0 2024-02-05T07:27:01.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average Latency(s): 0.0105611 2024-02-05T07:27:01.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Latency(s): 0.0289718 2024-02-05T07:27:01.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max latency(s): 2.10185 2024-02-05T07:27:01.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min latency(s): 0.00128428 2024-02-05T07:27:01.040 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:01.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:01.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:01.042 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T07:27:01.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:01.176 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:01.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:01.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:01.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:01.321 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:01.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:01.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-mon[29098]: 5.1a scrub starts 2024-02-05T07:27:01.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-mon[29098]: 5.1a scrub ok 2024-02-05T07:27:01.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:01.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-mon[29098]: 8.1 scrub starts 2024-02-05T07:27:01.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:27:01.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:27:01.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:27:01.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-mon[37158]: 5.1a scrub starts 2024-02-05T07:27:01.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-mon[37158]: 5.1a scrub ok 2024-02-05T07:27:01.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:01.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-mon[37158]: 8.1 scrub starts 2024-02-05T07:27:01.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:27:01.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:27:01.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:27:01.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:01.455 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:01.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:01.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:01 smithi090 ceph-mon[33213]: 5.1a scrub starts 2024-02-05T07:27:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:01 smithi090 ceph-mon[33213]: 5.1a scrub ok 2024-02-05T07:27:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:01 smithi090 ceph-mon[33213]: 8.1 scrub starts 2024-02-05T07:27:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:01 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:27:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:01 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:27:01.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:01 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:27:01.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:01.596 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:01.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:01.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:01.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:01.737 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:01.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:01.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:01.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:01.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:01.882 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:01.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:01.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:02.022 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:02.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:02.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:02.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:02.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:01 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a[29094]: 2024-02-05T07:27:01.856+0000 7fe3564d3700 -1 mon.a@0(leader).osd e254 definitely_dead 0 2024-02-05T07:27:02.161 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:02.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:02.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:02.299 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:02.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:02.343 INFO:teuthology.orchestra.run.smithi007.stderr:marked out osd.5. 2024-02-05T07:27:02.401 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:02.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:27:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[29098]: 6.16 scrub starts 2024-02-05T07:27:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[29098]: 6.16 scrub ok 2024-02-05T07:27:02.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[29098]: pgmap v398: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfill_toofull, 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 8.3 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 1.7 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 41780/407166 objects degraded (10.261%); 54500/407166 objects misplaced (13.385%); 17 MiB/s, 267 objects/s recovering 2024-02-05T07:27:02.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[29098]: 8.1 scrub ok 2024-02-05T07:27:02.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1222042974' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-05T07:27:02.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-05T07:27:02.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[29098]: Client client.admin marked osd.5 out, while it was still marked up 2024-02-05T07:27:02.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[29098]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:02.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[37158]: 6.16 scrub starts 2024-02-05T07:27:02.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[37158]: 6.16 scrub ok 2024-02-05T07:27:02.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[37158]: pgmap v398: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfill_toofull, 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 8.3 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 1.7 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 41780/407166 objects degraded (10.261%); 54500/407166 objects misplaced (13.385%); 17 MiB/s, 267 objects/s recovering 2024-02-05T07:27:02.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[37158]: 8.1 scrub ok 2024-02-05T07:27:02.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1222042974' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-05T07:27:02.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-05T07:27:02.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[37158]: Client client.admin marked osd.5 out, while it was still marked up 2024-02-05T07:27:02.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:02 smithi007 ceph-mon[37158]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:02.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:02.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:02.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:02.553 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:02.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:02.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:02.676 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:02 smithi090 ceph-mon[33213]: 6.16 scrub starts 2024-02-05T07:27:02.676 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:02 smithi090 ceph-mon[33213]: 6.16 scrub ok 2024-02-05T07:27:02.677 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:02 smithi090 ceph-mon[33213]: pgmap v398: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfill_toofull, 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 8.3 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 1.7 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 41780/407166 objects degraded (10.261%); 54500/407166 objects misplaced (13.385%); 17 MiB/s, 267 objects/s recovering 2024-02-05T07:27:02.677 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:02 smithi090 ceph-mon[33213]: 8.1 scrub ok 2024-02-05T07:27:02.677 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:02 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1222042974' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-05T07:27:02.677 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:02 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-05T07:27:02.677 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:02 smithi090 ceph-mon[33213]: Client client.admin marked osd.5 out, while it was still marked up 2024-02-05T07:27:02.677 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:02 smithi090 ceph-mon[33213]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:02.678 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:02.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:02.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:02.781 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:02.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:02.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:02.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:02.932 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:02.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:02.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:03.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:03.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:03.061 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:03.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:03.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:03.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:03.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:03.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:03.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:03.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:03.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:03.324 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:27:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:27:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-02-05T07:27:03.324 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:27:03.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:03.448 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:03.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:03.498 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:27:03.526 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:27:03.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:03.551 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:27:03.564 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:27:03.587 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:03.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:03.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:03.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:03 smithi007 ceph-mon[29098]: 2.1 scrub starts 2024-02-05T07:27:03.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:03 smithi007 ceph-mon[29098]: 2.1 scrub ok 2024-02-05T07:27:03.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:03 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-02-05T07:27:03.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:03 smithi007 ceph-mon[29098]: osdmap e255: 8 total, 7 up, 6 in 2024-02-05T07:27:03.601 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:27:03.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:03 smithi007 ceph-mon[37158]: 2.1 scrub starts 2024-02-05T07:27:03.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:03 smithi007 ceph-mon[37158]: 2.1 scrub ok 2024-02-05T07:27:03.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:03 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-02-05T07:27:03.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:03 smithi007 ceph-mon[37158]: osdmap e255: 8 total, 7 up, 6 in 2024-02-05T07:27:03.602 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:27:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:27:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:27:03.627 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:27:03.660 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:27:03.676 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:03 smithi090 ceph-mon[33213]: 2.1 scrub starts 2024-02-05T07:27:03.677 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:03 smithi090 ceph-mon[33213]: 2.1 scrub ok 2024-02-05T07:27:03.677 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:03 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-02-05T07:27:03.677 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:03 smithi090 ceph-mon[33213]: osdmap e255: 8 total, 7 up, 6 in 2024-02-05T07:27:03.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:03.692 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:27:03.693 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:27:03.693 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:27:03.693 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:27:03.693 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:27:03.693 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:27:03.693 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:27:03.693 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-02-05T07:27:03.693 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:27:03.724 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:03.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:03.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:03.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:03.857 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:03.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:03.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:03.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:03.990 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:03.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:04.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T07:27:04.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:04.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:04.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:04.261 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:04.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:04.306 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:27:04.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:04.406 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:04.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:04.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:04.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:04.546 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:04.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:04.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[29098]: pgmap v400: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfill_toofull, 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 8.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 852 B/s rd, 295 MiB/s wr, 4.73k op/s; 41780/407166 objects degraded (10.261%); 54500/407166 objects misplaced (13.385%); 16 MiB/s, 261 objects/s recovering 2024-02-05T07:27:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[29098]: 8.19 continuing backfill to osd.4 from (232'3700,254'8349] 8:9a5bad4e:::benchmark_data_smithi007_105755_object135614:head to 254'8349 2024-02-05T07:27:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[29098]: 8.13 starting backfill to osd.2 from (232'3700,254'8200] MIN to 254'8201 2024-02-05T07:27:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[29098]: osdmap e256: 8 total, 7 up, 6 in 2024-02-05T07:27:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[29098]: 8.11 continuing backfill to osd.7 from (232'4000,254'8436] MIN to 254'8436 2024-02-05T07:27:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[29098]: 8.16 starting backfill to osd.1 from (0'0,0'0] MAX to 254'8392 2024-02-05T07:27:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[29098]: 8.13 starting backfill to osd.4 from (0'0,0'0] MAX to 254'8201 2024-02-05T07:27:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[29098]: 8.16 continuing backfill to osd.2 from (230'3500,254'8392] MIN to 254'8392 2024-02-05T07:27:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/339654766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:27:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 41780/407166 objects degraded (10.261%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[37158]: pgmap v400: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfill_toofull, 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 8.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 852 B/s rd, 295 MiB/s wr, 4.73k op/s; 41780/407166 objects degraded (10.261%); 54500/407166 objects misplaced (13.385%); 16 MiB/s, 261 objects/s recovering 2024-02-05T07:27:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[37158]: 8.19 continuing backfill to osd.4 from (232'3700,254'8349] 8:9a5bad4e:::benchmark_data_smithi007_105755_object135614:head to 254'8349 2024-02-05T07:27:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[37158]: 8.13 starting backfill to osd.2 from (232'3700,254'8200] MIN to 254'8201 2024-02-05T07:27:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[37158]: osdmap e256: 8 total, 7 up, 6 in 2024-02-05T07:27:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[37158]: 8.11 continuing backfill to osd.7 from (232'4000,254'8436] MIN to 254'8436 2024-02-05T07:27:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[37158]: 8.16 starting backfill to osd.1 from (0'0,0'0] MAX to 254'8392 2024-02-05T07:27:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[37158]: 8.13 starting backfill to osd.4 from (0'0,0'0] MAX to 254'8201 2024-02-05T07:27:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[37158]: 8.16 continuing backfill to osd.2 from (230'3500,254'8392] MIN to 254'8392 2024-02-05T07:27:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/339654766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:27:04.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:04 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 41780/407166 objects degraded (10.261%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:04.690 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:04.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:04.695 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:04 smithi090 ceph-mon[33213]: pgmap v400: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfill_toofull, 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 8.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 852 B/s rd, 295 MiB/s wr, 4.73k op/s; 41780/407166 objects degraded (10.261%); 54500/407166 objects misplaced (13.385%); 16 MiB/s, 261 objects/s recovering 2024-02-05T07:27:04.695 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:04 smithi090 ceph-mon[33213]: 8.19 continuing backfill to osd.4 from (232'3700,254'8349] 8:9a5bad4e:::benchmark_data_smithi007_105755_object135614:head to 254'8349 2024-02-05T07:27:04.695 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:04 smithi090 ceph-mon[33213]: 8.13 starting backfill to osd.2 from (232'3700,254'8200] MIN to 254'8201 2024-02-05T07:27:04.695 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:04 smithi090 ceph-mon[33213]: osdmap e256: 8 total, 7 up, 6 in 2024-02-05T07:27:04.695 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:04 smithi090 ceph-mon[33213]: 8.11 continuing backfill to osd.7 from (232'4000,254'8436] MIN to 254'8436 2024-02-05T07:27:04.695 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:04 smithi090 ceph-mon[33213]: 8.16 starting backfill to osd.1 from (0'0,0'0] MAX to 254'8392 2024-02-05T07:27:04.695 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:04 smithi090 ceph-mon[33213]: 8.13 starting backfill to osd.4 from (0'0,0'0] MAX to 254'8201 2024-02-05T07:27:04.696 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:04 smithi090 ceph-mon[33213]: 8.16 continuing backfill to osd.2 from (230'3500,254'8392] MIN to 254'8392 2024-02-05T07:27:04.696 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:04 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/339654766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:27:04.696 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:04 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 41780/407166 objects degraded (10.261%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:04.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:04.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:04.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:04.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:04.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:04.976 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:04.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:05.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:05.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:05.128 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:05.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:05.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:05.264 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:05.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:05.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:05.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:05.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:05.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:05.504 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:05.544 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:05.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:05.615 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:05 smithi007 ceph-mon[29098]: osdmap e257: 8 total, 7 up, 6 in 2024-02-05T07:27:05.616 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:05 smithi007 ceph-mon[29098]: 8.a starting backfill to osd.4 from (0'0,0'0] MAX to 254'8817 2024-02-05T07:27:05.616 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:05 smithi007 ceph-mon[29098]: 8.8 starting backfill to osd.2 from (0'0,0'0] MAX to 254'8315 2024-02-05T07:27:05.616 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:05 smithi007 ceph-mon[29098]: 8.c starting backfill to osd.4 from (0'0,0'0] MAX to 254'8411 2024-02-05T07:27:05.616 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:05 smithi007 ceph-mon[29098]: 8.10 continuing backfill to osd.7 from (232'3700,254'8404] MIN to 254'8404 2024-02-05T07:27:05.616 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:05 smithi007 ceph-mon[29098]: 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 254'8165 2024-02-05T07:27:05.616 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:05 smithi007 ceph-mon[29098]: 8.9 starting backfill to osd.4 from (0'0,0'0] MAX to 254'8373 2024-02-05T07:27:05.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:05 smithi007 ceph-mon[37158]: osdmap e257: 8 total, 7 up, 6 in 2024-02-05T07:27:05.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:05 smithi007 ceph-mon[37158]: 8.a starting backfill to osd.4 from (0'0,0'0] MAX to 254'8817 2024-02-05T07:27:05.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:05 smithi007 ceph-mon[37158]: 8.8 starting backfill to osd.2 from (0'0,0'0] MAX to 254'8315 2024-02-05T07:27:05.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:05 smithi007 ceph-mon[37158]: 8.c starting backfill to osd.4 from (0'0,0'0] MAX to 254'8411 2024-02-05T07:27:05.617 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:05 smithi007 ceph-mon[37158]: 8.10 continuing backfill to osd.7 from (232'3700,254'8404] MIN to 254'8404 2024-02-05T07:27:05.617 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:05 smithi007 ceph-mon[37158]: 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 254'8165 2024-02-05T07:27:05.617 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:05 smithi007 ceph-mon[37158]: 8.9 starting backfill to osd.4 from (0'0,0'0] MAX to 254'8373 2024-02-05T07:27:05.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:05.651 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:05 smithi090 ceph-mon[33213]: osdmap e257: 8 total, 7 up, 6 in 2024-02-05T07:27:05.651 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:05 smithi090 ceph-mon[33213]: 8.a starting backfill to osd.4 from (0'0,0'0] MAX to 254'8817 2024-02-05T07:27:05.651 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:05 smithi090 ceph-mon[33213]: 8.8 starting backfill to osd.2 from (0'0,0'0] MAX to 254'8315 2024-02-05T07:27:05.652 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:05 smithi090 ceph-mon[33213]: 8.c starting backfill to osd.4 from (0'0,0'0] MAX to 254'8411 2024-02-05T07:27:05.652 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:05 smithi090 ceph-mon[33213]: 8.10 continuing backfill to osd.7 from (232'3700,254'8404] MIN to 254'8404 2024-02-05T07:27:05.652 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:05 smithi090 ceph-mon[33213]: 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 254'8165 2024-02-05T07:27:05.652 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:05 smithi090 ceph-mon[33213]: 8.9 starting backfill to osd.4 from (0'0,0'0] MAX to 254'8373 2024-02-05T07:27:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T07:27:05.663 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T07:27:05.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:05.703 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:05.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:05.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:05.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:05.846 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:05.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:05.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:05.982 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:05.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:06.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:06.121 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:06.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:06.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:06.188 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:27:06.188 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:27:06.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:06.262 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:06.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:06.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:06.409 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:06.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:06.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:06.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:06 smithi090 ceph-mon[33213]: pgmap v403: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfill_toofull, 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 8.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 874 B/s rd, 303 MiB/s wr, 4.85k op/s; 41780/407166 objects degraded (10.261%); 54500/407166 objects misplaced (13.385%); 17 MiB/s, 267 objects/s recovering 2024-02-05T07:27:06.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:06 smithi090 ceph-mon[33213]: osdmap e258: 8 total, 7 up, 6 in 2024-02-05T07:27:06.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:06.544 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:06.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:06.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:06 smithi007 ceph-mon[29098]: pgmap v403: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfill_toofull, 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 8.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 874 B/s rd, 303 MiB/s wr, 4.85k op/s; 41780/407166 objects degraded (10.261%); 54500/407166 objects misplaced (13.385%); 17 MiB/s, 267 objects/s recovering 2024-02-05T07:27:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:06 smithi007 ceph-mon[29098]: osdmap e258: 8 total, 7 up, 6 in 2024-02-05T07:27:06.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:06 smithi007 ceph-mon[37158]: pgmap v403: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfill_toofull, 29 active+undersized+degraded, 40 active+undersized, 89 active+clean; 8.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 874 B/s rd, 303 MiB/s wr, 4.85k op/s; 41780/407166 objects degraded (10.261%); 54500/407166 objects misplaced (13.385%); 17 MiB/s, 267 objects/s recovering 2024-02-05T07:27:06.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:06 smithi007 ceph-mon[37158]: osdmap e258: 8 total, 7 up, 6 in 2024-02-05T07:27:06.682 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:06.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:06.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:06.826 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:06.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:06.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:06.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:06.952 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:06.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:06.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:07.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:07.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:07.093 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:07.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:07.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:07.217 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:07.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:07.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T07:27:07.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:07.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:07.416 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:27:07.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:07.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:07.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:07 smithi090 ceph-mon[33213]: osdmap e259: 8 total, 7 up, 6 in 2024-02-05T07:27:07.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:07 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/577127383' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:27:07.527 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:07 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:27:07.600 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:27:07.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:07.642 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:07.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:07.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:07.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:27:07.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:07 smithi007 ceph-mon[29098]: osdmap e259: 8 total, 7 up, 6 in 2024-02-05T07:27:07.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:07 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/577127383' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:27:07.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:07 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:27:07.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:07 smithi007 ceph-mon[37158]: osdmap e259: 8 total, 7 up, 6 in 2024-02-05T07:27:07.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:07 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/577127383' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:27:07.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:07 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:27:07.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:07.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:07.771 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:07.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:07.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:07.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:07.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:07.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:08.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:08.051 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:08.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:08.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:08.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:08.191 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:08.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:08.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:08.198 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7] out_osds: [3, 5] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-02-05T07:27:08.198 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:27:08.199 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-02-05T07:27:08.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd in 3 2024-02-05T07:27:08.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:08.315 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:08.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:08.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:08.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:08.450 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:08.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:08.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:08.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:08 smithi090 ceph-mon[33213]: pgmap v406: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 2 remapped+peering, 1 peering, 33 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+undersized+remapped, 17 active+undersized+degraded, 23 active+undersized, 75 active+clean; 8.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 38663/399420 objects degraded (9.680%); 112234/399420 objects misplaced (28.099%); 72 MiB/s, 1.16k objects/s recovering 2024-02-05T07:27:08.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:08 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:08.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:08 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:27:08.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:08 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:27:08.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:08 smithi090 ceph-mon[33213]: osdmap e260: 8 total, 7 up, 6 in 2024-02-05T07:27:08.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:08.587 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:08.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:08.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:08 smithi007 ceph-mon[29098]: pgmap v406: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 2 remapped+peering, 1 peering, 33 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+undersized+remapped, 17 active+undersized+degraded, 23 active+undersized, 75 active+clean; 8.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 38663/399420 objects degraded (9.680%); 112234/399420 objects misplaced (28.099%); 72 MiB/s, 1.16k objects/s recovering 2024-02-05T07:27:08.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:08 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:08.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:08 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:27:08.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:08 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:27:08.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:08 smithi007 ceph-mon[29098]: osdmap e260: 8 total, 7 up, 6 in 2024-02-05T07:27:08.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:08 smithi007 ceph-mon[37158]: pgmap v406: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 2 remapped+peering, 1 peering, 33 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+undersized+remapped, 17 active+undersized+degraded, 23 active+undersized, 75 active+clean; 8.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 38663/399420 objects degraded (9.680%); 112234/399420 objects misplaced (28.099%); 72 MiB/s, 1.16k objects/s recovering 2024-02-05T07:27:08.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:08 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:08.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:08 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:27:08.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:08 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:27:08.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:08 smithi007 ceph-mon[37158]: osdmap e260: 8 total, 7 up, 6 in 2024-02-05T07:27:08.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:08.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:08.731 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:08.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:08.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:08.859 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:08.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:08.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:08.997 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:08.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:08.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:09.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:09.122 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:09.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:09.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:09.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:09.267 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:09.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:09.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:09.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:09.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:09.411 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:09.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:09.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:09.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:09.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:09.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:09.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:09 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:27:09.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:09 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:27:09.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:09 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 38663/399420 objects degraded (9.680%), 24 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:09.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:09 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:27:09.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:09 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:27:09.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:09 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 38663/399420 objects degraded (9.680%), 24 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:09.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:09.740 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:27:09.740 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:27:09.740 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:09 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:27:09.740 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:09 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:27:09.740 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:09 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 38663/399420 objects degraded (9.680%), 24 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:09.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:09.820 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:09.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:09.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:09.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:09.956 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:09.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:10.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:10.099 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:10.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:10.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:10.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:10.233 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:10.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:10.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:10.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:10.373 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:10.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:10.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T07:27:10.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:10.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:27:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:27:10.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:10 smithi090 ceph-mon[33213]: pgmap v408: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 2 remapped+peering, 1 peering, 33 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+undersized+remapped, 17 active+undersized+degraded, 23 active+undersized, 75 active+clean; 8.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 38663/399420 objects degraded (9.680%); 112234/399420 objects misplaced (28.099%); 60 MiB/s, 973 objects/s recovering 2024-02-05T07:27:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:10 smithi090 ceph-mon[33213]: osdmap e261: 8 total, 7 up, 6 in 2024-02-05T07:27:10.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:10.527 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:10.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:10.572 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:27:10.581 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:27:10.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:10.665 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:10.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:10.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:10 smithi007 ceph-mon[29098]: pgmap v408: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 2 remapped+peering, 1 peering, 33 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+undersized+remapped, 17 active+undersized+degraded, 23 active+undersized, 75 active+clean; 8.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 38663/399420 objects degraded (9.680%); 112234/399420 objects misplaced (28.099%); 60 MiB/s, 973 objects/s recovering 2024-02-05T07:27:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:10 smithi007 ceph-mon[29098]: osdmap e261: 8 total, 7 up, 6 in 2024-02-05T07:27:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:10 smithi007 ceph-mon[37158]: pgmap v408: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 2 remapped+peering, 1 peering, 33 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+undersized+remapped, 17 active+undersized+degraded, 23 active+undersized, 75 active+clean; 8.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 38663/399420 objects degraded (9.680%); 112234/399420 objects misplaced (28.099%); 60 MiB/s, 973 objects/s recovering 2024-02-05T07:27:10.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:10 smithi007 ceph-mon[37158]: osdmap e261: 8 total, 7 up, 6 in 2024-02-05T07:27:10.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:10.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:10.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:10.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:10.839 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:27:10.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:10.947 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:10.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:10.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:11.049 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:11.083 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:11.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:11.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:11.223 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:11.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:11.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:11.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:11.362 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:11.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:11.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:11.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:11.504 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:11.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:11.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:11.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:11.638 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:11 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:11.642 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:11.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:11.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:11.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:11 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:11.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:11 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:11.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:11.792 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:11.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:11.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:11.932 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:11.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:12.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:12.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:12.075 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:12.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:12.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:12.210 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:12.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:12.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:12.346 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:12.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:12.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:12.493 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:27:12.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:12.495 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:12.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:12.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:12.654 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:12 smithi090 ceph-mon[33213]: pgmap v410: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 17 active+undersized+degraded, 23 active+undersized, 76 active+clean; 7.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 714 B/s rd, 0 B/s wr, 660 op/s; 35003/381855 objects degraded (9.167%); 104011/381855 objects misplaced (27.238%); 77 MiB/s, 1.24k objects/s recovering 2024-02-05T07:27:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:12.655 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:12.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:12.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:12 smithi007 ceph-mon[29098]: pgmap v410: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 17 active+undersized+degraded, 23 active+undersized, 76 active+clean; 7.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 714 B/s rd, 0 B/s wr, 660 op/s; 35003/381855 objects degraded (9.167%); 104011/381855 objects misplaced (27.238%); 77 MiB/s, 1.24k objects/s recovering 2024-02-05T07:27:12.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:12 smithi007 ceph-mon[37158]: pgmap v410: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 17 active+undersized+degraded, 23 active+undersized, 76 active+clean; 7.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 714 B/s rd, 0 B/s wr, 660 op/s; 35003/381855 objects degraded (9.167%); 104011/381855 objects misplaced (27.238%); 77 MiB/s, 1.24k objects/s recovering 2024-02-05T07:27:12.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:12.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:12.803 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:12.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:12.905 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:12.954 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:12.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:13.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:13.114 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:13.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:13.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:13.133 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:27:13.133 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:27:13.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:13.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:13.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:13.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:13.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:13.387 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:13.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:13.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:13.530 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:13.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:13.579 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:13 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:27:13.579 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:13 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:27:13.579 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:27:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:27:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:27:13.580 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:13 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:27:13.580 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:13 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:27:13.632 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:13.670 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:13.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:13.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:13.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:13 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:27:13.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:13 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:27:13.807 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:13.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:13.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:13.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:13.948 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:13.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:13.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:14.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:14.075 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:14.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:14.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:14.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:14.213 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:14.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:14.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:14.348 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:14.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:14.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:14.449 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:14.485 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:14.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:14.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:14.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:14.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:14.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:14.773 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:14.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:14.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:14 smithi090 ceph-mon[33213]: pgmap v411: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 17 active+undersized+degraded, 23 active+undersized, 76 active+clean; 7.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 760 B/s rd, 0 B/s wr, 870 op/s; 35003/381855 objects degraded (9.167%); 104011/381855 objects misplaced (27.238%); 42 MiB/s, 679 objects/s recovering 2024-02-05T07:27:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:14 smithi090 ceph-mon[33213]: osdmap e262: 8 total, 7 up, 6 in 2024-02-05T07:27:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:14 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3462824665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:27:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:14 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 35003/381855 objects degraded (9.167%), 24 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:14.875 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:14.909 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:14.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:14.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:14 smithi007 ceph-mon[29098]: pgmap v411: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 17 active+undersized+degraded, 23 active+undersized, 76 active+clean; 7.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 760 B/s rd, 0 B/s wr, 870 op/s; 35003/381855 objects degraded (9.167%); 104011/381855 objects misplaced (27.238%); 42 MiB/s, 679 objects/s recovering 2024-02-05T07:27:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:14 smithi007 ceph-mon[29098]: osdmap e262: 8 total, 7 up, 6 in 2024-02-05T07:27:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:14 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3462824665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:27:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:14 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 35003/381855 objects degraded (9.167%), 24 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:14.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:14 smithi007 ceph-mon[37158]: pgmap v411: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 17 active+undersized+degraded, 23 active+undersized, 76 active+clean; 7.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 760 B/s rd, 0 B/s wr, 870 op/s; 35003/381855 objects degraded (9.167%); 104011/381855 objects misplaced (27.238%); 42 MiB/s, 679 objects/s recovering 2024-02-05T07:27:14.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:14 smithi007 ceph-mon[37158]: osdmap e262: 8 total, 7 up, 6 in 2024-02-05T07:27:14.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:14 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3462824665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:27:14.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:14 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 35003/381855 objects degraded (9.167%), 24 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:15.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:15.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:15.053 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:15.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:15.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:15.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:15.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:15.362 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:15.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:15.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:15.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:15 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a[29094]: 2024-02-05T07:27:15.092+0000 7fe3564d3700 -1 mon.a@0(leader).osd e262 definitely_dead 0 2024-02-05T07:27:15.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:15.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:15.500 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:15.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:15.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:15.630 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:15.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:15.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:15.759 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:15.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:15.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:15.830 INFO:teuthology.orchestra.run.smithi007.stderr:marked in osd.3. 2024-02-05T07:27:15.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:15.887 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:15.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:15.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:15 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:27:15.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:27:15.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:15 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4247565064' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-05T07:27:15.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:15 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-05T07:27:15.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:15 smithi007 ceph-mon[29098]: pgmap v413: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 17 active+undersized+degraded, 23 active+undersized, 76 active+clean; 7.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 664 B/s rd, 0 B/s wr, 760 op/s; 35003/381855 objects degraded (9.167%); 104011/381855 objects misplaced (27.238%); 37 MiB/s, 593 objects/s recovering 2024-02-05T07:27:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:15 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:27:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:27:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:15 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4247565064' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-05T07:27:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:15 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-05T07:27:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:15 smithi007 ceph-mon[37158]: pgmap v413: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 17 active+undersized+degraded, 23 active+undersized, 76 active+clean; 7.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 664 B/s rd, 0 B/s wr, 760 op/s; 35003/381855 objects degraded (9.167%); 104011/381855 objects misplaced (27.238%); 37 MiB/s, 593 objects/s recovering 2024-02-05T07:27:15.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:16.012 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:16.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:16.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:15 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:27:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:27:16.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:15 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4247565064' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-05T07:27:16.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:15 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-05T07:27:16.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:15 smithi090 ceph-mon[33213]: pgmap v413: 168 pgs: 4 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 17 active+undersized+degraded, 23 active+undersized, 76 active+clean; 7.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 664 B/s rd, 0 B/s wr, 760 op/s; 35003/381855 objects degraded (9.167%); 104011/381855 objects misplaced (27.238%); 37 MiB/s, 593 objects/s recovering 2024-02-05T07:27:16.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:16.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:16.241 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:16.265 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:16.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:16.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:16.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:16.392 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:16.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:16.493 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:16.518 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:16.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:16.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:16.650 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:16.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:16.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:16.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:16.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:16.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:16 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-05T07:27:16.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:16 smithi007 ceph-mon[29098]: osdmap e263: 8 total, 7 up, 7 in 2024-02-05T07:27:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:16 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-05T07:27:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:16 smithi007 ceph-mon[37158]: osdmap e263: 8 total, 7 up, 7 in 2024-02-05T07:27:16.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:16.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:16.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:16 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-05T07:27:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:16 smithi090 ceph-mon[33213]: osdmap e263: 8 total, 7 up, 7 in 2024-02-05T07:27:17.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:17.067 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:17.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:17.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:17.191 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:17.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:17.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:17.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:17.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:17.329 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:17.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:17.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:17.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:17.459 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:17.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:17.560 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:17.593 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:17.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:17.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:17.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T07:27:17.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:17.726 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:17.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:17.827 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:17.850 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:17.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:17.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:17.891 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:27:17.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:17.985 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:17.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:18.020 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:17 smithi090 ceph-mon[33213]: 8.13 continuing backfill to osd.2 from (232'3700,262'8633] 8:ca7234c4:::benchmark_data_smithi007_105755_object129282:head to 262'8633 2024-02-05T07:27:18.020 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:17 smithi090 ceph-mon[33213]: 8.2 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8772 2024-02-05T07:27:18.020 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:17 smithi090 ceph-mon[33213]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8804 2024-02-05T07:27:18.020 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:17 smithi090 ceph-mon[33213]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8633 2024-02-05T07:27:18.020 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:17 smithi090 ceph-mon[33213]: 8.0 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8871 2024-02-05T07:27:18.020 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:17 smithi090 ceph-mon[33213]: 8.16 continuing backfill to osd.2 from (230'3500,262'8797] 8:6b2e8713:::benchmark_data_smithi007_105755_object47435:head to 262'8797 2024-02-05T07:27:18.020 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:17 smithi090 ceph-mon[33213]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8797 2024-02-05T07:27:18.020 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:17 smithi090 ceph-mon[33213]: 8.18 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8703 2024-02-05T07:27:18.020 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:17 smithi090 ceph-mon[33213]: osdmap e264: 8 total, 7 up, 7 in 2024-02-05T07:27:18.020 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:17 smithi090 ceph-mon[33213]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 262'9686 2024-02-05T07:27:18.020 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:17 smithi090 ceph-mon[33213]: 8.c starting backfill to osd.3 from (0'0,0'0] MAX to 262'9211 2024-02-05T07:27:18.020 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:17 smithi090 ceph-mon[33213]: 8.8 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8737 2024-02-05T07:27:18.021 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:17 smithi090 ceph-mon[33213]: pgmap v416: 168 pgs: 4 peering, 7 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 33 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 17 active+undersized+degraded, 22 active+undersized, 73 active+clean; 7.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 26515/363606 objects degraded (7.292%); 82761/363606 objects misplaced (22.761%); 48 MiB/s, 772 objects/s recovering 2024-02-05T07:27:18.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:18.117 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:18.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:18.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[29098]: 8.13 continuing backfill to osd.2 from (232'3700,262'8633] 8:ca7234c4:::benchmark_data_smithi007_105755_object129282:head to 262'8633 2024-02-05T07:27:18.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[29098]: 8.2 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8772 2024-02-05T07:27:18.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[29098]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8804 2024-02-05T07:27:18.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[29098]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8633 2024-02-05T07:27:18.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[29098]: 8.0 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8871 2024-02-05T07:27:18.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[29098]: 8.16 continuing backfill to osd.2 from (230'3500,262'8797] 8:6b2e8713:::benchmark_data_smithi007_105755_object47435:head to 262'8797 2024-02-05T07:27:18.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[29098]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8797 2024-02-05T07:27:18.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[29098]: 8.18 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8703 2024-02-05T07:27:18.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[29098]: osdmap e264: 8 total, 7 up, 7 in 2024-02-05T07:27:18.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[29098]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 262'9686 2024-02-05T07:27:18.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[29098]: 8.c starting backfill to osd.3 from (0'0,0'0] MAX to 262'9211 2024-02-05T07:27:18.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[29098]: 8.8 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8737 2024-02-05T07:27:18.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[29098]: pgmap v416: 168 pgs: 4 peering, 7 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 33 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 17 active+undersized+degraded, 22 active+undersized, 73 active+clean; 7.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 26515/363606 objects degraded (7.292%); 82761/363606 objects misplaced (22.761%); 48 MiB/s, 772 objects/s recovering 2024-02-05T07:27:18.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[37158]: 8.13 continuing backfill to osd.2 from (232'3700,262'8633] 8:ca7234c4:::benchmark_data_smithi007_105755_object129282:head to 262'8633 2024-02-05T07:27:18.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[37158]: 8.2 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8772 2024-02-05T07:27:18.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[37158]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8804 2024-02-05T07:27:18.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[37158]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8633 2024-02-05T07:27:18.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[37158]: 8.0 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8871 2024-02-05T07:27:18.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[37158]: 8.16 continuing backfill to osd.2 from (230'3500,262'8797] 8:6b2e8713:::benchmark_data_smithi007_105755_object47435:head to 262'8797 2024-02-05T07:27:18.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[37158]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8797 2024-02-05T07:27:18.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[37158]: 8.18 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8703 2024-02-05T07:27:18.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[37158]: osdmap e264: 8 total, 7 up, 7 in 2024-02-05T07:27:18.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[37158]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 262'9686 2024-02-05T07:27:18.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[37158]: 8.c starting backfill to osd.3 from (0'0,0'0] MAX to 262'9211 2024-02-05T07:27:18.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[37158]: 8.8 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8737 2024-02-05T07:27:18.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:17 smithi007 ceph-mon[37158]: pgmap v416: 168 pgs: 4 peering, 7 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 33 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 17 active+undersized+degraded, 22 active+undersized, 73 active+clean; 7.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 26515/363606 objects degraded (7.292%); 82761/363606 objects misplaced (22.761%); 48 MiB/s, 772 objects/s recovering 2024-02-05T07:27:18.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:18.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:18.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:18.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:18.374 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:18.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:18.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:18.475 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:18.512 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:18.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:18.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:18.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:18.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:18.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:18.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:18.784 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:18.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:18.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:18.908 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:18.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:18.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:19.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:18 smithi090 ceph-mon[33213]: osdmap e265: 8 total, 7 up, 7 in 2024-02-05T07:27:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:18 smithi090 ceph-mon[33213]: 8.f starting backfill to osd.3 from (0'0,0'0] MAX to 262'9663 2024-02-05T07:27:19.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:18 smithi090 ceph-mon[33213]: 8.b starting backfill to osd.3 from (0'0,0'0] MAX to 262'9416 2024-02-05T07:27:19.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:18 smithi090 ceph-mon[33213]: 8.7 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8960 2024-02-05T07:27:19.052 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:19.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:19.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:19.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:18 smithi007 ceph-mon[29098]: osdmap e265: 8 total, 7 up, 7 in 2024-02-05T07:27:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:18 smithi007 ceph-mon[29098]: 8.f starting backfill to osd.3 from (0'0,0'0] MAX to 262'9663 2024-02-05T07:27:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:18 smithi007 ceph-mon[29098]: 8.b starting backfill to osd.3 from (0'0,0'0] MAX to 262'9416 2024-02-05T07:27:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:18 smithi007 ceph-mon[29098]: 8.7 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8960 2024-02-05T07:27:19.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:18 smithi007 ceph-mon[37158]: osdmap e265: 8 total, 7 up, 7 in 2024-02-05T07:27:19.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:18 smithi007 ceph-mon[37158]: 8.f starting backfill to osd.3 from (0'0,0'0] MAX to 262'9663 2024-02-05T07:27:19.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:18 smithi007 ceph-mon[37158]: 8.b starting backfill to osd.3 from (0'0,0'0] MAX to 262'9416 2024-02-05T07:27:19.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:18 smithi007 ceph-mon[37158]: 8.7 starting backfill to osd.3 from (0'0,0'0] MAX to 262'8960 2024-02-05T07:27:19.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:19.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:19.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:19.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:19.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:19.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:19.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:19.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:19.463 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:19.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:19.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:19.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:19.604 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:19.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:19.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:19.734 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:19.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:19.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:19.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:19.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:19.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:20.001 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:20.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:20.021 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:19 smithi090 ceph-mon[33213]: osdmap e266: 8 total, 7 up, 7 in 2024-02-05T07:27:20.021 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:19 smithi090 ceph-mon[33213]: pgmap v419: 168 pgs: 4 peering, 7 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 33 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 17 active+undersized+degraded, 22 active+undersized, 73 active+clean; 7.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 26515/363606 objects degraded (7.292%); 82761/363606 objects misplaced (22.761%); 72 MiB/s, 1.16k objects/s recovering 2024-02-05T07:27:20.022 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:19 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 26515/363606 objects degraded (7.292%), 22 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:20.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:20.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:20.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:20.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:20.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:19 smithi007 ceph-mon[29098]: osdmap e266: 8 total, 7 up, 7 in 2024-02-05T07:27:20.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:19 smithi007 ceph-mon[29098]: pgmap v419: 168 pgs: 4 peering, 7 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 33 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 17 active+undersized+degraded, 22 active+undersized, 73 active+clean; 7.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 26515/363606 objects degraded (7.292%); 82761/363606 objects misplaced (22.761%); 72 MiB/s, 1.16k objects/s recovering 2024-02-05T07:27:20.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:19 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 26515/363606 objects degraded (7.292%), 22 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:20.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:19 smithi007 ceph-mon[37158]: osdmap e266: 8 total, 7 up, 7 in 2024-02-05T07:27:20.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:19 smithi007 ceph-mon[37158]: pgmap v419: 168 pgs: 4 peering, 7 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 33 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 17 active+undersized+degraded, 22 active+undersized, 73 active+clean; 7.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 26515/363606 objects degraded (7.292%); 82761/363606 objects misplaced (22.761%); 72 MiB/s, 1.16k objects/s recovering 2024-02-05T07:27:20.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:19 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 26515/363606 objects degraded (7.292%), 22 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:20.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:20.275 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:20.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:20.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:20.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:20.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:20.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:20.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:20.502 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:27:20.502 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:27:20.502 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:27:20.502 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:27:20.502 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:27:20.502 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:27:20.502 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:27:20.502 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:27:20.502 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:27:20.503 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:27:20.503 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:27:20.503 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:27:20.503 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:27:20.503 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-02-05T07:27:20.503 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:27:20.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:20.535 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:20.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:20.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:20.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:20.669 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:20.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:20.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:20.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:27:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:27:20.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:20.801 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:20.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:20.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:20.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:20.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:20 smithi007 ceph-mon[29098]: osdmap e267: 8 total, 7 up, 7 in 2024-02-05T07:27:20.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:20 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:20 smithi007 ceph-mon[37158]: osdmap e267: 8 total, 7 up, 7 in 2024-02-05T07:27:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:20 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:20.927 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:20.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:20.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:20 smithi090 ceph-mon[33213]: osdmap e267: 8 total, 7 up, 7 in 2024-02-05T07:27:21.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:20 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:21.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:21.058 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:21.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:21.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:21.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:21.193 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:21.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:21.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:21.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:21.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:21.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:21.470 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:21.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:21.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:21.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:21.613 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:21.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:21.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:21.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:21.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:21.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:21.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:21.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:21.898 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:21.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:21.999 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:22.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:21 smithi090 ceph-mon[33213]: pgmap v421: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 431 op/s; 29817/357894 objects degraded (8.331%); 115720/357894 objects misplaced (32.334%); 63 MiB/s, 1.03k objects/s recovering 2024-02-05T07:27:22.030 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:22.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:22.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:22.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:22.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:22.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:21 smithi007 ceph-mon[29098]: pgmap v421: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 431 op/s; 29817/357894 objects degraded (8.331%); 115720/357894 objects misplaced (32.334%); 63 MiB/s, 1.03k objects/s recovering 2024-02-05T07:27:22.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:21 smithi007 ceph-mon[37158]: pgmap v421: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 431 op/s; 29817/357894 objects degraded (8.331%); 115720/357894 objects misplaced (32.334%); 63 MiB/s, 1.03k objects/s recovering 2024-02-05T07:27:22.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:22.287 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:22.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:22.389 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:22.421 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:22.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:22.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:22.557 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:22.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:22.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:22.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:22.700 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:22.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:22.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:22.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:22.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:22.827 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:22.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:22.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:22.969 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:22.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:22 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:23.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:23.111 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:23.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:23.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:23.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:22 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:23.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:22 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:23.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:23.239 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:23.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:23.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:23.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:23.382 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:23.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:23.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:23.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:23.515 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:23.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:23.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:23.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:23.669 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:23.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:23.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:23.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:23.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:23.802 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:23.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:23.891 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:23 smithi090 ceph-mon[33213]: pgmap v422: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 318 op/s; 29817/357894 objects degraded (8.331%); 115720/357894 objects misplaced (32.334%); 47 MiB/s, 761 objects/s recovering 2024-02-05T07:27:23.903 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:23.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:23 smithi007 ceph-mon[37158]: pgmap v422: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 318 op/s; 29817/357894 objects degraded (8.331%); 115720/357894 objects misplaced (32.334%); 47 MiB/s, 761 objects/s recovering 2024-02-05T07:27:23.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:27:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:27:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:27:23.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:23 smithi007 ceph-mon[29098]: pgmap v422: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 318 op/s; 29817/357894 objects degraded (8.331%); 115720/357894 objects misplaced (32.334%); 47 MiB/s, 761 objects/s recovering 2024-02-05T07:27:23.944 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:23.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:23.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:24.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:24.073 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:24.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:24.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:24.207 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:24.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:24.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:24.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:24.346 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:24.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:24.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:24.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:24.482 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:24.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:24.534 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:27:24.534 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:27:24.534 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:27:24.534 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:27:24.534 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:27:24.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:24.623 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:24.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:24.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:24.748 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:24.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:24.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:24.883 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:24.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:24.899 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:24 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 29817/357894 objects degraded (8.331%), 18 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:24.899 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:24 smithi090 ceph-mon[33213]: osdmap e268: 8 total, 7 up, 7 in 2024-02-05T07:27:24.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:25.009 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:25.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:25.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:25.151 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:25.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:25.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:24 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 29817/357894 objects degraded (8.331%), 18 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:25.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:24 smithi007 ceph-mon[29098]: osdmap e268: 8 total, 7 up, 7 in 2024-02-05T07:27:25.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:24 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 29817/357894 objects degraded (8.331%), 18 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:25.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:24 smithi007 ceph-mon[37158]: osdmap e268: 8 total, 7 up, 7 in 2024-02-05T07:27:25.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:25.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T07:27:25.292 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:25.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:25.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:25.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:25.431 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:25.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:25.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:25.582 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:25.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:25.620 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:27:25.684 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:25.718 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:25.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:25.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:25.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:25.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:25.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:25.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:25.987 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:25.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:26.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:25 smithi090 ceph-mon[33213]: pgmap v424: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 960 B/s rd, 0 B/s wr, 298 op/s; 29817/357894 objects degraded (8.331%); 115720/357894 objects misplaced (32.334%); 44 MiB/s, 713 objects/s recovering 2024-02-05T07:27:26.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:26.123 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:26.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:26.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:25 smithi007 ceph-mon[29098]: pgmap v424: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 960 B/s rd, 0 B/s wr, 298 op/s; 29817/357894 objects degraded (8.331%); 115720/357894 objects misplaced (32.334%); 44 MiB/s, 713 objects/s recovering 2024-02-05T07:27:26.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:25 smithi007 ceph-mon[37158]: pgmap v424: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 960 B/s rd, 0 B/s wr, 298 op/s; 29817/357894 objects degraded (8.331%); 115720/357894 objects misplaced (32.334%); 44 MiB/s, 713 objects/s recovering 2024-02-05T07:27:26.225 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:26.257 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:26.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:26.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:26.390 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:26.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:26.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:26.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:26.529 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:26.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:26.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:26.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:26.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:26.656 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:26.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:26.757 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:26.789 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:26.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:26.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:26.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:26.921 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:26.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:26.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:27.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:27.069 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:27.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:27.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:27.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:27.195 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:27.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:27.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:27.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:27.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:27.341 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:27.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:27.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:27.472 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:27.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:27.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:27.474 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:27:27.474 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:74439 7.5 7:a4181962:::benchmark_data_smithi007_77086_object74438:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:21:41.796370+0000", 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 345.67493883899999, 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.80113790600000001, 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 74439 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.475 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.796370+0000", 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.796370+0000", 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.796372+0000", 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.922e-06 2024-02-05T07:27:27.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.796441+0000", 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8774000000000003e-05 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.796461+0000", 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.986e-05 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.796469+0000", 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9680000000000006e-06 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.796521+0000", 2024-02-05T07:27:27.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2539e-05 2024-02-05T07:27:27.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.796531+0000", 2024-02-05T07:27:27.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0117e-05 2024-02-05T07:27:27.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.575295+0000", 2024-02-05T07:27:27.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.77876346299999999 2024-02-05T07:27:27.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.575464+0000", 2024-02-05T07:27:27.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169197 2024-02-05T07:27:27.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2024-02-05T07:27:27.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.575630+0000", 2024-02-05T07:27:27.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000166238 2024-02-05T07:27:27.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.577269+0000", 2024-02-05T07:27:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016386250000000001 2024-02-05T07:27:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.597040+0000", 2024-02-05T07:27:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019770923999999999 2024-02-05T07:27:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.597345+0000", 2024-02-05T07:27:27.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030518899999999998 2024-02-05T07:27:27.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:27:27.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.597466+0000", 2024-02-05T07:27:27.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120759 2024-02-05T07:27:27.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.597508+0000", 2024-02-05T07:27:27.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2330999999999998e-05 2024-02-05T07:27:27.481 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.481 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.482 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.482 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:74440 7.15 7:ae9e418e:::benchmark_data_smithi007_77086_object74439:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-02-05T07:27:27.482 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:21:41.818240+0000", 2024-02-05T07:27:27.482 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 345.65306917800001, 2024-02-05T07:27:27.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79467710199999997, 2024-02-05T07:27:27.482 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.482 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:27:27.482 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.482 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:27:27.482 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 74440 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.818240+0000", 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.818240+0000", 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.818244+0000", 2024-02-05T07:27:27.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8130000000000002e-06 2024-02-05T07:27:27.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.818293+0000", 2024-02-05T07:27:27.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9759000000000003e-05 2024-02-05T07:27:27.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.818310+0000", 2024-02-05T07:27:27.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6997e-05 2024-02-05T07:27:27.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.818336+0000", 2024-02-05T07:27:27.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5423e-05 2024-02-05T07:27:27.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.818387+0000", 2024-02-05T07:27:27.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1394000000000003e-05 2024-02-05T07:27:27.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:41.818396+0000", 2024-02-05T07:27:27.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3110000000000006e-06 2024-02-05T07:27:27.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.607843+0000", 2024-02-05T07:27:27.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.78944771899999999 2024-02-05T07:27:27.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.607984+0000", 2024-02-05T07:27:27.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014056 2024-02-05T07:27:27.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2024-02-05T07:27:27.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.608153+0000", 2024-02-05T07:27:27.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016920600000000001 2024-02-05T07:27:27.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.609967+0000", 2024-02-05T07:27:27.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018137400000000001 2024-02-05T07:27:27.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.610274+0000", 2024-02-05T07:27:27.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030728199999999998 2024-02-05T07:27:27.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.612818+0000", 2024-02-05T07:27:27.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025438689999999998 2024-02-05T07:27:27.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:27:27.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.612890+0000", 2024-02-05T07:27:27.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2476000000000002e-05 2024-02-05T07:27:27.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:42.612917+0000", 2024-02-05T07:27:27.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6553000000000001e-05 2024-02-05T07:27:27.490 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.490 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.490 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.490 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:75611 7.17 7.9497757 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-02-05T07:27:27.490 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:21:43.612968+0000", 2024-02-05T07:27:27.490 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 343.85834144500001, 2024-02-05T07:27:27.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52292803399999999, 2024-02-05T07:27:27.490 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.490 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 75611 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.612968+0000", 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.612968+0000", 2024-02-05T07:27:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.612969+0000", 2024-02-05T07:27:27.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3319999999999999e-06 2024-02-05T07:27:27.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.613025+0000", 2024-02-05T07:27:27.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6344000000000001e-05 2024-02-05T07:27:27.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.613035+0000", 2024-02-05T07:27:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2769999999999994e-06 2024-02-05T07:27:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.613041+0000", 2024-02-05T07:27:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0530000000000003e-06 2024-02-05T07:27:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.614963+0000", 2024-02-05T07:27:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019224960000000001 2024-02-05T07:27:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.614971+0000", 2024-02-05T07:27:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2190000000000004e-06 2024-02-05T07:27:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:44.135886+0000", 2024-02-05T07:27:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52091499100000005 2024-02-05T07:27:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:44.135896+0000", 2024-02-05T07:27:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3219999999999999e-06 2024-02-05T07:27:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:75626 7.11 7.99100491 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-02-05T07:27:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:21:43.630440+0000", 2024-02-05T07:27:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 343.84086859600001, 2024-02-05T07:27:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.503654673, 2024-02-05T07:27:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:27:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:27:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:27:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 75626 2024-02-05T07:27:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.630440+0000", 2024-02-05T07:27:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.630440+0000", 2024-02-05T07:27:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.630442+0000", 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2309999999999999e-06 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.630494+0000", 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1906000000000001e-05 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.630502+0000", 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7809999999999994e-06 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.630508+0000", 2024-02-05T07:27:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.502e-06 2024-02-05T07:27:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.630551+0000", 2024-02-05T07:27:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2951000000000003e-05 2024-02-05T07:27:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.630556+0000", 2024-02-05T07:27:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3469999999999998e-06 2024-02-05T07:27:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:44.134085+0000", 2024-02-05T07:27:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50352892599999999 2024-02-05T07:27:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:44.134095+0000", 2024-02-05T07:27:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0029e-05 2024-02-05T07:27:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:75630 7.15 7:abba0814:::benchmark_data_smithi007_77086_object75629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-02-05T07:27:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:21:43.637172+0000", 2024-02-05T07:27:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 343.834137005, 2024-02-05T07:27:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6317031210000001, 2024-02-05T07:27:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:27:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:27:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:27:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 75630 2024-02-05T07:27:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.637172+0000", 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.637172+0000", 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.637173+0000", 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5600000000000005e-07 2024-02-05T07:27:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.637208+0000", 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5472000000000002e-05 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.637218+0000", 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5170000000000005e-06 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.637223+0000", 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5589999999999996e-06 2024-02-05T07:27:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.641716+0000", 2024-02-05T07:27:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044922449999999997 2024-02-05T07:27:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.641724+0000", 2024-02-05T07:27:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1110000000000008e-06 2024-02-05T07:27:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:44.135904+0000", 2024-02-05T07:27:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49417987099999999 2024-02-05T07:27:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:44.135914+0000", 2024-02-05T07:27:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0665e-05 2024-02-05T07:27:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.261071+0000", 2024-02-05T07:27:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.125156998 2024-02-05T07:27:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.261210+0000", 2024-02-05T07:27:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013857400000000001 2024-02-05T07:27:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:27:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.261473+0000", 2024-02-05T07:27:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026326399999999998 2024-02-05T07:27:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.263707+0000", 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022340120000000001 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.265811+0000", 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021035540000000001 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.267296+0000", 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014858689999999999 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:27:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.267368+0000", 2024-02-05T07:27:27.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1462000000000001e-05 2024-02-05T07:27:27.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.268875+0000", 2024-02-05T07:27:27.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015071920000000001 2024-02-05T07:27:27.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.507 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.508 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:172756 7.11 7:8d3d4e1b:::benchmark_data_smithi007_77086_object68161:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-05T07:27:27.508 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:23:57.456991+0000", 2024-02-05T07:27:27.508 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 210.014318293, 2024-02-05T07:27:27.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.92245448500000005, 2024-02-05T07:27:27.508 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.508 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:27:27.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:27:27.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:27:27.508 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172756 2024-02-05T07:27:27.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.508 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.456991+0000", 2024-02-05T07:27:27.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.456991+0000", 2024-02-05T07:27:27.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.456991+0000", 2024-02-05T07:27:27.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8699999999999995e-07 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.456994+0000", 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7159999999999999e-06 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.457000+0000", 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2960000000000004e-06 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.457006+0000", 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5709999999999997e-06 2024-02-05T07:27:27.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.457060+0000", 2024-02-05T07:27:27.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3748000000000001e-05 2024-02-05T07:27:27.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.457068+0000", 2024-02-05T07:27:27.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6289999999999999e-06 2024-02-05T07:27:27.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.372790+0000", 2024-02-05T07:27:27.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.91572212099999994 2024-02-05T07:27:27.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.372925+0000", 2024-02-05T07:27:27.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134871 2024-02-05T07:27:27.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,6", 2024-02-05T07:27:27.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.373296+0000", 2024-02-05T07:27:27.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037044999999999998 2024-02-05T07:27:27.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379211+0000", 2024-02-05T07:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0059157919999999996 2024-02-05T07:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379225+0000", 2024-02-05T07:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4049000000000001e-05 2024-02-05T07:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379381+0000", 2024-02-05T07:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015567599999999999 2024-02-05T07:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379419+0000", 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7629999999999997e-05 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379445+0000", 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6449e-05 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:172762 7.11 7:8a322388:::benchmark_data_smithi007_77086_object68167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-05T07:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:23:57.460318+0000", 2024-02-05T07:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 210.01099126299999, 2024-02-05T07:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.91924548800000005, 2024-02-05T07:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172762 2024-02-05T07:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460318+0000", 2024-02-05T07:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460318+0000", 2024-02-05T07:27:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460319+0000", 2024-02-05T07:27:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5409999999999999e-06 2024-02-05T07:27:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460327+0000", 2024-02-05T07:27:27.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1359999999999997e-06 2024-02-05T07:27:27.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460339+0000", 2024-02-05T07:27:27.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1644999999999999e-05 2024-02-05T07:27:27.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460348+0000", 2024-02-05T07:27:27.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.568e-06 2024-02-05T07:27:27.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460641+0000", 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029384099999999998 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460658+0000", 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6575999999999999e-05 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.375350+0000", 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.91469182500000001 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.375469+0000", 2024-02-05T07:27:27.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119292 2024-02-05T07:27:27.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,6", 2024-02-05T07:27:27.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.375697+0000", 2024-02-05T07:27:27.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022779300000000001 2024-02-05T07:27:27.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379336+0000", 2024-02-05T07:27:27.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003638694 2024-02-05T07:27:27.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379348+0000", 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2131e-05 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379492+0000", 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014466699999999999 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379526+0000", 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3408999999999998e-05 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379563+0000", 2024-02-05T07:27:27.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7370000000000003e-05 2024-02-05T07:27:27.521 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.521 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.521 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.521 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:172771 7.11 7:89039f95:::benchmark_data_smithi007_77086_object68176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-05T07:27:27.521 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:23:57.468804+0000", 2024-02-05T07:27:27.521 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 210.00250486100001, 2024-02-05T07:27:27.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.91441402699999996, 2024-02-05T07:27:27.521 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172771 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468804+0000", 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468804+0000", 2024-02-05T07:27:27.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468805+0000", 2024-02-05T07:27:27.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.052e-06 2024-02-05T07:27:27.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468808+0000", 2024-02-05T07:27:27.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0529999999999998e-06 2024-02-05T07:27:27.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468817+0000", 2024-02-05T07:27:27.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2649999999999996e-06 2024-02-05T07:27:27.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468825+0000", 2024-02-05T07:27:27.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0199999999999994e-06 2024-02-05T07:27:27.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468884+0000", 2024-02-05T07:27:27.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9929999999999997e-05 2024-02-05T07:27:27.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468894+0000", 2024-02-05T07:27:27.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1120000000000002e-06 2024-02-05T07:27:27.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.377141+0000", 2024-02-05T07:27:27.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.90824729199999998 2024-02-05T07:27:27.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.377259+0000", 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118332 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,6", 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.377519+0000", 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025967600000000003 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379608+0000", 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020893439999999999 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.381150+0000", 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015417339999999999 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.383116+0000", 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001966392 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.383173+0000", 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6802000000000003e-05 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.383218+0000", 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5022999999999999e-05 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:172812 7.11 7:88d799b2:::benchmark_data_smithi007_77086_object68217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:23:57.525416+0000", 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 209.94589253999999, 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.86240439700000004, 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:27:27.528 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172812 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.525416+0000", 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.525416+0000", 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.525417+0000", 2024-02-05T07:27:27.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9500000000000001e-07 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.525424+0000", 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2809999999999997e-06 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.525435+0000", 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1734e-05 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.525443+0000", 2024-02-05T07:27:27.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1189999999999997e-06 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.525499+0000", 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5525000000000001e-05 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.525508+0000", 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9420000000000001e-06 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379641+0000", 2024-02-05T07:27:27.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.85413319600000004 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379706+0000", 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.533e-05 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,6", 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379872+0000", 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165367 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.383282+0000", 2024-02-05T07:27:27.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034099320000000001 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.384724+0000", 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014422320000000001 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.387706+0000", 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029824320000000001 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.387767+0000", 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0368999999999997e-05 2024-02-05T07:27:27.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.387821+0000", 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4143e-05 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:172849 7.11 7:8d04114d:::benchmark_data_smithi007_77086_object68254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:23:57.581661+0000", 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 209.88964752000001, 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.80775439800000004, 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:27:27.534 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172849 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.581661+0000", 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.581661+0000", 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.581663+0000", 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6920000000000001e-06 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.581668+0000", 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5859999999999998e-06 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.581682+0000", 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3917000000000001e-05 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.581691+0000", 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1649999999999995e-06 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.581759+0000", 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7951e-05 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.581773+0000", 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4138000000000001e-05 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.381191+0000", 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79941774200000004 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.381318+0000", 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127663 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,6", 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.381551+0000", 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023293899999999999 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.384731+0000", 2024-02-05T07:27:27.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031793059999999998 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.384769+0000", 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8237e-05 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.389356+0000", 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045870049999999999 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.389393+0000", 2024-02-05T07:27:27.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6695e-05 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.389416+0000", 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3362000000000001e-05 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:172900 7.11 7:8aacc524:::benchmark_data_smithi007_77086_object68305:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:23:57.630450+0000", 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 209.84085923999999, 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.75908857799999996, 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.540 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:27:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:27:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:27:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172900 2024-02-05T07:27:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.630450+0000", 2024-02-05T07:27:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.630450+0000", 2024-02-05T07:27:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.630452+0000", 2024-02-05T07:27:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8640000000000001e-06 2024-02-05T07:27:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.630462+0000", 2024-02-05T07:27:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0176e-05 2024-02-05T07:27:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.630475+0000", 2024-02-05T07:27:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3373000000000001e-05 2024-02-05T07:27:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.630484+0000", 2024-02-05T07:27:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7420000000000003e-06 2024-02-05T07:27:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.630550+0000", 2024-02-05T07:27:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6486e-05 2024-02-05T07:27:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.630561+0000", 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0396e-05 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.383330+0000", 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.75276894800000005 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.383496+0000", 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016581000000000001 2024-02-05T07:27:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,6", 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.383702+0000", 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206863 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.384864+0000", 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011612199999999999 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.385887+0000", 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010230180000000001 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.389490+0000", 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036031359999999998 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.389520+0000", 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0490000000000001e-05 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.389538+0000", 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8056000000000001e-05 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:151887 8.c 8:324b0199:::benchmark_data_smithi007_105755_object15577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:15.622747+0000", 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.848561781000001, 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.370190494, 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151887 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.622747+0000", 2024-02-05T07:27:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.622747+0000", 2024-02-05T07:27:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.622748+0000", 2024-02-05T07:27:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2160000000000001e-06 2024-02-05T07:27:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.622755+0000", 2024-02-05T07:27:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5930000000000002e-06 2024-02-05T07:27:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.622765+0000", 2024-02-05T07:27:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9049999999999992e-06 2024-02-05T07:27:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.622773+0000", 2024-02-05T07:27:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9219999999999997e-06 2024-02-05T07:27:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.622921+0000", 2024-02-05T07:27:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148221 2024-02-05T07:27:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.623051+0000", 2024-02-05T07:27:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013038599999999999 2024-02-05T07:27:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,5", 2024-02-05T07:27:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.623186+0000", 2024-02-05T07:27:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013459400000000001 2024-02-05T07:27:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.626745+0000", 2024-02-05T07:27:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035590420000000001 2024-02-05T07:27:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.627939+0000", 2024-02-05T07:27:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011934039999999999 2024-02-05T07:27:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629311+0000", 2024-02-05T07:27:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013721580000000001 2024-02-05T07:27:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.691547+0000", 2024-02-05T07:27:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062236106999999999 2024-02-05T07:27:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.691557+0000", 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7720000000000006e-06 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.987383+0000", 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2958265010000001 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.987490+0000", 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010739 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.992884+0000", 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053939920000000002 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.992938+0000", 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3291000000000002e-05 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:151900 8.8 8:15dd58bd:::benchmark_data_smithi007_105755_object15590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:15.630983+0000", 2024-02-05T07:27:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.840326234999999, 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3621680999999999, 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151900 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.630983+0000", 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.630983+0000", 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.630984+0000", 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.33e-06 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.630989+0000", 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8099999999999997e-06 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.631004+0000", 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5407999999999999e-05 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.631012+0000", 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0730000000000005e-06 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.631076+0000", 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3590000000000006e-05 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.631210+0000", 2024-02-05T07:27:27.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133666 2024-02-05T07:27:27.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2,5", 2024-02-05T07:27:27.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.631529+0000", 2024-02-05T07:27:27.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031965100000000001 2024-02-05T07:27:27.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.634358+0000", 2024-02-05T07:27:27.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028284569999999999 2024-02-05T07:27:27.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.634826+0000", 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046775199999999999 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.684540+0000", 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049714248000000003 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.684555+0000", 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5350999999999999e-05 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.987354+0000", 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3027992239999999 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.987472+0000", 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011736299999999999 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.993094+0000", 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056223510000000003 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.993151+0000", 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6826000000000001e-05 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:151902 8.c 8:32e5c9c1:::benchmark_data_smithi007_105755_object15592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:15.632873+0000", 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.838436073, 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3627677999999999, 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:27:27.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151902 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.632873+0000", 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.632873+0000", 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.632874+0000", 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.204e-06 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.632878+0000", 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6320000000000001e-06 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.632887+0000", 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8359999999999998e-06 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.632893+0000", 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8639999999999998e-06 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.632948+0000", 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4605000000000003e-05 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.633073+0000", 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124626 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,5", 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.633200+0000", 2024-02-05T07:27:27.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127651 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.637692+0000", 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044912149999999998 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.637749+0000", 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7438000000000001e-05 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.637813+0000", 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3830000000000004e-05 2024-02-05T07:27:27.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.691600+0000", 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.053787029 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.691609+0000", 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2089999999999994e-06 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.987907+0000", 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.296297872 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.988009+0000", 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010227599999999999 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.995549+0000", 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075395009999999997 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.995641+0000", 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2011999999999999e-05 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.568 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:151908 8.8 8:11579385:::benchmark_data_smithi007_105755_object15598:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:15.644089+0000", 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.827220271, 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.348958485, 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151908 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644089+0000", 2024-02-05T07:27:27.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644089+0000", 2024-02-05T07:27:27.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644089+0000", 2024-02-05T07:27:27.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0999999999999998e-07 2024-02-05T07:27:27.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644093+0000", 2024-02-05T07:27:27.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0630000000000002e-06 2024-02-05T07:27:27.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644100+0000", 2024-02-05T07:27:27.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1849999999999998e-06 2024-02-05T07:27:27.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644104+0000", 2024-02-05T07:27:27.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6450000000000004e-06 2024-02-05T07:27:27.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644136+0000", 2024-02-05T07:27:27.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1736000000000002e-05 2024-02-05T07:27:27.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644219+0000", 2024-02-05T07:27:27.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2546999999999994e-05 2024-02-05T07:27:27.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2,5", 2024-02-05T07:27:27.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644484+0000", 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026491600000000002 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.645628+0000", 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001144436 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.646211+0000", 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058267900000000001 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.687357+0000", 2024-02-05T07:27:27.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041146481999999998 2024-02-05T07:27:27.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:27:27.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.687369+0000", 2024-02-05T07:27:27.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1972e-05 2024-02-05T07:27:27.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.987534+0000", 2024-02-05T07:27:27.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3001650279999999 2024-02-05T07:27:27.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.987600+0000", 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6271000000000003e-05 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.992998+0000", 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053975610000000004 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.993047+0000", 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9254e-05 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.575 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.576 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:152380 8.c 8:3d06f3f2:::benchmark_data_smithi007_105755_object16070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-02-05T07:27:27.576 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:18.383697+0000", 2024-02-05T07:27:27.576 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.0876117779999994, 2024-02-05T07:27:27.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23066635499999999, 2024-02-05T07:27:27.576 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.576 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:27:27.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:27:27.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:27:27.576 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 152380 2024-02-05T07:27:27.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.383697+0000", 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.383697+0000", 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.383698+0000", 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.82e-07 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.383701+0000", 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9129999999999999e-06 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.383707+0000", 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8969999999999996e-06 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.383711+0000", 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3189999999999996e-06 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384299+0000", 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058829499999999996 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384412+0000", 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113247 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3,5", 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384537+0000", 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124184 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.387875+0000", 2024-02-05T07:27:27.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033387360000000001 2024-02-05T07:27:27.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.388008+0000", 2024-02-05T07:27:27.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000132299 2024-02-05T07:27:27.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.391743+0000", 2024-02-05T07:27:27.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037350859999999999 2024-02-05T07:27:27.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.614099+0000", 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22235581400000001 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.614160+0000", 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1311999999999998e-05 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.614364+0000", 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000203671 2024-02-05T07:27:27.581 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:152381 8.d 8:b9addfe6:::benchmark_data_smithi007_105755_object16071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:18.384106+0000", 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.0872031100000008, 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23315881399999999, 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 152381 2024-02-05T07:27:27.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384106+0000", 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384106+0000", 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384107+0000", 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5099999999999998e-07 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384111+0000", 2024-02-05T07:27:27.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0929999999999996e-06 2024-02-05T07:27:27.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384120+0000", 2024-02-05T07:27:27.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4800000000000007e-06 2024-02-05T07:27:27.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384128+0000", 2024-02-05T07:27:27.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9130000000000003e-06 2024-02-05T07:27:27.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384204+0000", 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6058999999999998e-05 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384308+0000", 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103836 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384403+0000", 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4918000000000002e-05 2024-02-05T07:27:27.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.389196+0000", 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047925839999999999 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.389670+0000", 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047423400000000003 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.617104+0000", 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22743376800000001 2024-02-05T07:27:27.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:27:27.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.617161+0000", 2024-02-05T07:27:27.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7052999999999997e-05 2024-02-05T07:27:27.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.617265+0000", 2024-02-05T07:27:27.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010402499999999999 2024-02-05T07:27:27.587 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.587 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.587 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.588 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:152384 8.d 8:b49e50bc:::benchmark_data_smithi007_105755_object16074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-02-05T07:27:27.588 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:18.384468+0000", 2024-02-05T07:27:27.588 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.0868406069999992, 2024-02-05T07:27:27.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23363980200000001, 2024-02-05T07:27:27.588 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.588 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:27:27.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:27:27.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:27:27.588 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 152384 2024-02-05T07:27:27.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.588 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384468+0000", 2024-02-05T07:27:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384468+0000", 2024-02-05T07:27:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384469+0000", 2024-02-05T07:27:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0999999999999997e-07 2024-02-05T07:27:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384473+0000", 2024-02-05T07:27:27.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9960000000000004e-06 2024-02-05T07:27:27.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384483+0000", 2024-02-05T07:27:27.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4159999999999993e-06 2024-02-05T07:27:27.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.384490+0000", 2024-02-05T07:27:27.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3259999999999998e-06 2024-02-05T07:27:27.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.386777+0000", 2024-02-05T07:27:27.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022874670000000001 2024-02-05T07:27:27.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.386895+0000", 2024-02-05T07:27:27.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117108 2024-02-05T07:27:27.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2024-02-05T07:27:27.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.386995+0000", 2024-02-05T07:27:27.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100518 2024-02-05T07:27:27.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.389299+0000", 2024-02-05T07:27:27.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303949 2024-02-05T07:27:27.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.389705+0000", 2024-02-05T07:27:27.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040649700000000002 2024-02-05T07:27:27.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.617317+0000", 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.227611224 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.617428+0000", 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011111 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.618108+0000", 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068038099999999998 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.593 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:152393 8.d 8:ba8e2f3a:::benchmark_data_smithi007_105755_object16083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:18.390710+0000", 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.0805985790000001, 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.230577693, 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 152393 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.390710+0000", 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.390710+0000", 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.390711+0000", 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7399999999999998e-07 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.390713+0000", 2024-02-05T07:27:27.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1210000000000001e-06 2024-02-05T07:27:27.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.390720+0000", 2024-02-05T07:27:27.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1960000000000002e-06 2024-02-05T07:27:27.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.390724+0000", 2024-02-05T07:27:27.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6040000000000001e-06 2024-02-05T07:27:27.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.613868+0000", 2024-02-05T07:27:27.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22314441900000001 2024-02-05T07:27:27.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.614008+0000", 2024-02-05T07:27:27.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014011399999999999 2024-02-05T07:27:27.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2024-02-05T07:27:27.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.614144+0000", 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013609200000000001 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.618345+0000", 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042000880000000003 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.618383+0000", 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8189000000000003e-05 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.621146+0000", 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027630710000000002 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.621189+0000", 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3337999999999997e-05 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.621288+0000", 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8987000000000006e-05 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.600 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:152394 8.8 8:1755ca8c:::benchmark_data_smithi007_105755_object16084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-02-05T07:27:27.600 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:18.391914+0000", 2024-02-05T07:27:27.600 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.0793951919999998, 2024-02-05T07:27:27.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27303995599999997, 2024-02-05T07:27:27.600 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:27:27.600 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:27:27.600 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:27:27.600 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:27:27.600 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:27:27.600 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 152394 2024-02-05T07:27:27.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.600 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:27:27.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.391914+0000", 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.391914+0000", 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.391915+0000", 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.06e-07 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:27:27.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.391918+0000", 2024-02-05T07:27:27.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0539999999999999e-06 2024-02-05T07:27:27.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:27:27.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.391925+0000", 2024-02-05T07:27:27.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5750000000000003e-06 2024-02-05T07:27:27.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:27:27.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.391934+0000", 2024-02-05T07:27:27.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2110000000000004e-06 2024-02-05T07:27:27.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.392061+0000", 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012684 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.392182+0000", 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012105 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3,5", 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.615791+0000", 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.223608996 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.618080+0000", 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022887369999999999 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.620645+0000", 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025651699999999999 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.620702+0000", 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6529e-05 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:27:27.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.664786+0000", 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.044084506000000002 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.664827+0000", 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1269000000000001e-05 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:18.664954+0000", 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012631299999999999 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:27:27.605 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:27:27.606 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:27:27.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:27.645 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:27.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:27.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:27.747 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:27.778 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:27.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:27.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:27.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:27.911 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:27.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:28.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:28.036 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:28.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:28.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:28.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:28.184 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:28.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:28.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:28.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:28.322 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:28.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:28.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:28.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:28 smithi007 ceph-mon[29098]: pgmap v425: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 803 op/s; 27594/344343 objects degraded (8.014%); 110245/344343 objects misplaced (32.016%); 51 MiB/s, 830 objects/s recovering 2024-02-05T07:27:28.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:28 smithi007 ceph-mon[37158]: pgmap v425: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 803 op/s; 27594/344343 objects degraded (8.014%); 110245/344343 objects misplaced (32.016%); 51 MiB/s, 830 objects/s recovering 2024-02-05T07:27:28.454 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:28.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:28.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:28.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:28 smithi090 ceph-mon[33213]: pgmap v425: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 803 op/s; 27594/344343 objects degraded (8.014%); 110245/344343 objects misplaced (32.016%); 51 MiB/s, 830 objects/s recovering 2024-02-05T07:27:28.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:28.585 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:28.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:28.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:28.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:28.713 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:28.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:28.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:28.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:28.847 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:28.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:28.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:28.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:28.991 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:28.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:28.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:29.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:29.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:29.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:29.225 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T07:27:29.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:29.247 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-02-05T07:27:29.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:29.339 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:29.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:29.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:29 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:27:29.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:29 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:27:29.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:29 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:27:29.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:29 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:27:29.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:29.471 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:29.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:29.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:29.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:29 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:27:29.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:29 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:27:29.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:29.626 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:29.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:29.728 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:29.772 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:29.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:29.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:29.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:29.906 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:29.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:30.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:30.038 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:30.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:30.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:30.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:30.165 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:30.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:30.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:30.296 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:30.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:30.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:30.413 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T07:27:30.414 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T07:27:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:30.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:30.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:30.438 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:30 smithi090 ceph-mon[33213]: pgmap v426: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 685 op/s; 27594/344343 objects degraded (8.014%); 110245/344343 objects misplaced (32.016%); 44 MiB/s, 707 objects/s recovering 2024-02-05T07:27:30.438 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:30 smithi090 ceph-mon[33213]: osdmap e269: 8 total, 7 up, 7 in 2024-02-05T07:27:30.438 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:30 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 27594/344343 objects degraded (8.014%), 18 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:30.438 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:30 smithi090 ceph-mon[33213]: osdmap e270: 8 total, 7 up, 7 in 2024-02-05T07:27:30.438 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:27:30.520 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:27:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:27:30.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:30.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:30.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:30.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:30.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:30.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:30 smithi007 ceph-mon[37158]: pgmap v426: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 685 op/s; 27594/344343 objects degraded (8.014%); 110245/344343 objects misplaced (32.016%); 44 MiB/s, 707 objects/s recovering 2024-02-05T07:27:30.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:30 smithi007 ceph-mon[37158]: osdmap e269: 8 total, 7 up, 7 in 2024-02-05T07:27:30.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:30 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 27594/344343 objects degraded (8.014%), 18 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:30.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:30 smithi007 ceph-mon[37158]: osdmap e270: 8 total, 7 up, 7 in 2024-02-05T07:27:30.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:27:30.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:30 smithi007 ceph-mon[29098]: pgmap v426: 168 pgs: 2 active+undersized+degraded+remapped, 2 active+remapped+backfilling, 3 active+undersized+degraded+remapped+backfilling, 29 active+clean+remapped, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 7.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 685 op/s; 27594/344343 objects degraded (8.014%); 110245/344343 objects misplaced (32.016%); 44 MiB/s, 707 objects/s recovering 2024-02-05T07:27:30.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:30 smithi007 ceph-mon[29098]: osdmap e269: 8 total, 7 up, 7 in 2024-02-05T07:27:30.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:30 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 27594/344343 objects degraded (8.014%), 18 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:30.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:30 smithi007 ceph-mon[29098]: osdmap e270: 8 total, 7 up, 7 in 2024-02-05T07:27:30.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:27:30.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:30.711 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:30.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:30.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:27:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:27:30.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:30.842 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:30.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:30.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:30.976 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:30.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:30.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:31.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:31.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:31.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:31.255 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:31.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:31.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:31.310 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:31 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:31.310 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:31 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:31.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:31.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:31.397 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:31.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:31.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:31.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:31 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:31.534 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:31.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:31.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:31.660 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:31.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:31.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:31.786 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:31.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:31.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:31.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:31.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:31.923 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:31.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:32.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:32.048 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:32.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:32.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:32.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:32.174 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:32.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:32.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:32.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:32.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:32.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:27:32.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:32.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:32 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:27:32.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:32 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:27:32.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:32 smithi007 ceph-mon[29098]: pgmap v429: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 30 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.32k op/s; 24947/330783 objects degraded (7.542%); 104601/330783 objects misplaced (31.622%); 42 MiB/s, 678 objects/s recovering 2024-02-05T07:27:32.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:32 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:32.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:32 smithi007 ceph-mon[29098]: osdmap e271: 8 total, 7 up, 7 in 2024-02-05T07:27:32.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:32 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:27:32.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:32 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:27:32.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:32 smithi007 ceph-mon[37158]: pgmap v429: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 30 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.32k op/s; 24947/330783 objects degraded (7.542%); 104601/330783 objects misplaced (31.622%); 42 MiB/s, 678 objects/s recovering 2024-02-05T07:27:32.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:32 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:32.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:32 smithi007 ceph-mon[37158]: osdmap e271: 8 total, 7 up, 7 in 2024-02-05T07:27:32.450 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:32.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:32.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:32 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:27:32.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:32 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:27:32.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:32 smithi090 ceph-mon[33213]: pgmap v429: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 30 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.32k op/s; 24947/330783 objects degraded (7.542%); 104601/330783 objects misplaced (31.622%); 42 MiB/s, 678 objects/s recovering 2024-02-05T07:27:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:32 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:32 smithi090 ceph-mon[33213]: osdmap e271: 8 total, 7 up, 7 in 2024-02-05T07:27:32.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:32.610 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:32.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:32.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:32.772 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:32.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:32.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:32.905 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:32.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:33.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:33.035 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:33.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:33.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:33.168 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:33.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:33.270 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:33.315 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:33.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:33.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:33.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:33 smithi007 ceph-mon[37158]: osdmap e272: 8 total, 7 up, 7 in 2024-02-05T07:27:33.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:33 smithi007 ceph-mon[29098]: osdmap e272: 8 total, 7 up, 7 in 2024-02-05T07:27:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:33.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:33.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:33.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:33 smithi090 ceph-mon[33213]: osdmap e272: 8 total, 7 up, 7 in 2024-02-05T07:27:33.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:33.636 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:27:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:33.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:33.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:33.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:33.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:33.791 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:27:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:27:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:27:33.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:33.911 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:33.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:33.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:34.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:34.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:34.064 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:34.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:34.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:34.208 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:34.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:34.248 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7, 3] out_osds: [5] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-02-05T07:27:34.248 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:27:34.248 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-02-05T07:27:34.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:27:34.249 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:27:34.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:34.335 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:34.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:34.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:34.469 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:34.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:34.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:34.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:34 smithi090 ceph-mon[33213]: pgmap v432: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 30 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 24947/330783 objects degraded (7.542%); 104601/330783 objects misplaced (31.622%); 40 MiB/s, 643 objects/s recovering 2024-02-05T07:27:34.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:34 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2187351916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:27:34.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:34.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:34.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:34.636 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:34 smithi007 ceph-mon[29098]: pgmap v432: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 30 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 24947/330783 objects degraded (7.542%); 104601/330783 objects misplaced (31.622%); 40 MiB/s, 643 objects/s recovering 2024-02-05T07:27:34.636 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:34 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2187351916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:27:34.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:34 smithi007 ceph-mon[37158]: pgmap v432: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 30 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 24947/330783 objects degraded (7.542%); 104601/330783 objects misplaced (31.622%); 40 MiB/s, 643 objects/s recovering 2024-02-05T07:27:34.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:34 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2187351916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:27:34.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:34.747 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:34.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:34.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:34.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:34.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:34.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:34.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:35.009 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:35.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:35.110 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:35.137 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:35.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:35.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:35.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:35.272 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:35.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:35.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:35.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:35.410 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:35.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:35.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:35 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:27:35.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:35 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2187351916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:27:35.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:35 smithi007 ceph-mon[29098]: osdmap e273: 8 total, 7 up, 7 in 2024-02-05T07:27:35.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:35 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 24947/330783 objects degraded (7.542%), 18 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:35.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:35 smithi007 ceph-mon[29098]: osdmap e274: 8 total, 7 up, 7 in 2024-02-05T07:27:35.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:35 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:27:35.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:35 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2187351916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:27:35.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:35 smithi007 ceph-mon[37158]: osdmap e273: 8 total, 7 up, 7 in 2024-02-05T07:27:35.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:35 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 24947/330783 objects degraded (7.542%), 18 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:35.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:35 smithi007 ceph-mon[37158]: osdmap e274: 8 total, 7 up, 7 in 2024-02-05T07:27:35.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:35.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:35 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:27:35.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:35 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2187351916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:27:35.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:35 smithi090 ceph-mon[33213]: osdmap e273: 8 total, 7 up, 7 in 2024-02-05T07:27:35.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:35 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 24947/330783 objects degraded (7.542%), 18 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:35.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:35 smithi090 ceph-mon[33213]: osdmap e274: 8 total, 7 up, 7 in 2024-02-05T07:27:35.557 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:35.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:35.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:35.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:35.691 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:35.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:35.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:35.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:35.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:35.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:35.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:36.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:36.115 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:36.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:36.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:36.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:36.257 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:36.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:36.359 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:36.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:36 smithi007 ceph-mon[29098]: pgmap v435: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 30 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 24947/330783 objects degraded (7.542%); 104601/330783 objects misplaced (31.622%) 2024-02-05T07:27:36.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:36 smithi007 ceph-mon[29098]: osdmap e275: 8 total, 7 up, 7 in 2024-02-05T07:27:36.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:36 smithi007 ceph-mon[37158]: pgmap v435: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 30 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 24947/330783 objects degraded (7.542%); 104601/330783 objects misplaced (31.622%) 2024-02-05T07:27:36.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:36 smithi007 ceph-mon[37158]: osdmap e275: 8 total, 7 up, 7 in 2024-02-05T07:27:36.415 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:36.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:36.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:36.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:36 smithi090 ceph-mon[33213]: pgmap v435: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 30 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 24947/330783 objects degraded (7.542%); 104601/330783 objects misplaced (31.622%) 2024-02-05T07:27:36.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:36 smithi090 ceph-mon[33213]: osdmap e275: 8 total, 7 up, 7 in 2024-02-05T07:27:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:36.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:36.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:36.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:36.695 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:36.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:36.773 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:27:36.773 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:27:36.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.773 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:167479 8.18 8:18afb300:::benchmark_data_smithi007_105755_object31169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-02-05T07:27:36.773 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:36.714093+0000", 2024-02-05T07:27:36.773 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.001521271, 2024-02-05T07:27:36.774 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0015547239999999999, 2024-02-05T07:27:36.774 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:27:36.774 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:27:36.774 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:27:36.774 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:27:36.774 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:27:36.774 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 167479 2024-02-05T07:27:36.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.774 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:27:36.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.774 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:27:36.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714093+0000", 2024-02-05T07:27:36.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:27:36.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:27:36.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714093+0000", 2024-02-05T07:27:36.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:27:36.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:27:36.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714095+0000", 2024-02-05T07:27:36.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.497e-06 2024-02-05T07:27:36.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:27:36.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714097+0000", 2024-02-05T07:27:36.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6189999999999998e-06 2024-02-05T07:27:36.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:27:36.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714104+0000", 2024-02-05T07:27:36.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0639999999999996e-06 2024-02-05T07:27:36.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:27:36.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714110+0000", 2024-02-05T07:27:36.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7100000000000004e-06 2024-02-05T07:27:36.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:27:36.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.715266+0000", 2024-02-05T07:27:36.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0011561710000000001 2024-02-05T07:27:36.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:27:36.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.715388+0000", 2024-02-05T07:27:36.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121613 2024-02-05T07:27:36.777 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:27:36.778 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:27:36.778 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:27:36.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.778 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "MOSDPGPushReply(8.18 275/263 [PushReplyOp(8:1d59498e:::benchmark_data_smithi007_105755_object64054:head)])", 2024-02-05T07:27:36.778 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:36.714376+0000", 2024-02-05T07:27:36.778 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0012380830000000001, 2024-02-05T07:27:36.778 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0013494449999999999, 2024-02-05T07:27:36.778 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:27:36.778 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-02-05T07:27:36.778 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:27:36.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.778 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:27:36.778 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714376+0000", 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714376+0000", 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714377+0000", 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1699999999999998e-07 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:27:36.779 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714381+0000", 2024-02-05T07:27:36.780 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9570000000000004e-06 2024-02-05T07:27:36.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.780 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:27:36.780 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714389+0000", 2024-02-05T07:27:36.780 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2050000000000002e-06 2024-02-05T07:27:36.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.780 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:27:36.780 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714396+0000", 2024-02-05T07:27:36.780 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3769999999999998e-06 2024-02-05T07:27:36.781 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:27:36.781 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:27:36.781 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:27:36.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.781 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.24977.0:167482 8.14 e275/252)", 2024-02-05T07:27:36.781 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:36.714966+0000", 2024-02-05T07:27:36.781 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00064833700000000002, 2024-02-05T07:27:36.781 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00081147000000000001, 2024-02-05T07:27:36.781 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:27:36.781 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-02-05T07:27:36.781 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:27:36.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.781 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714966+0000", 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714966+0000", 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714967+0000", 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5300000000000003e-07 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.782 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714975+0000", 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3270000000000001e-06 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714985+0000", 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7580000000000004e-06 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714992+0000", 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3409999999999997e-06 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:27:36.783 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:27:36.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.784 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.24977.0:167475 8.c e275/263 8:3c166f0c:::benchmark_data_smithi007_105755_object31165:head v 275'10145, mlcod=275'10142)", 2024-02-05T07:27:36.784 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:36.708452+0000", 2024-02-05T07:27:36.784 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0071628300000000002, 2024-02-05T07:27:36.784 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.007378562, 2024-02-05T07:27:36.784 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:27:36.784 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:27:36.784 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:27:36.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.784 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:27:36.784 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.708452+0000", 2024-02-05T07:27:36.784 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.708452+0000", 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.708452+0000", 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1399999999999996e-07 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.708467+0000", 2024-02-05T07:27:36.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4972e-05 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.708479+0000", 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1231e-05 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.708486+0000", 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.486e-06 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714581+0000", 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0060953680000000003 2024-02-05T07:27:36.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714606+0000", 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5021999999999999e-05 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.24977.0:167476 8.c e275/263)", 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:36.714030+0000", 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0015846409999999999, 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001857823, 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:27:36.787 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-02-05T07:27:36.788 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:27:36.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:27:36.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714030+0000", 2024-02-05T07:27:36.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:27:36.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:27:36.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714030+0000", 2024-02-05T07:27:36.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:27:36.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:27:36.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714030+0000", 2024-02-05T07:27:36.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9499999999999998e-07 2024-02-05T07:27:36.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:27:36.789 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714036+0000", 2024-02-05T07:27:36.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2909999999999998e-06 2024-02-05T07:27:36.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:27:36.789 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714041+0000", 2024-02-05T07:27:36.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4160000000000003e-06 2024-02-05T07:27:36.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:27:36.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:27:36.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:27:36.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:36.714046+0000", 2024-02-05T07:27:36.790 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.583e-06 2024-02-05T07:27:36.790 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:27:36.790 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:27:36.790 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:27:36.790 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:27:36.790 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:27:36.790 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 5 2024-02-05T07:27:36.790 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:27:36.797 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:36.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:36.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:36.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:36.976 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:36.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:37.078 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:37.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:37.122 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:37.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:37.223 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:37.264 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:37.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:37.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:37.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:37.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:37 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:37.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:37 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:37.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:37.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:37 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:37.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:37.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:37.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:37.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T07:27:37.687 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:37.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:37.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:37.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:37.819 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:37.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:37.921 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:37.953 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:37.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:37.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:38.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:38.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:38.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:38.234 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:38.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:38.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:38.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:38.368 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:38.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:38.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:38.511 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:38.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:38.530 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:38 smithi007 ceph-mon[29098]: pgmap v437: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 734 op/s; 19977/319962 objects degraded (6.244%); 99412/319962 objects misplaced (31.070%); 63 MiB/s, 1.01k objects/s recovering 2024-02-05T07:27:38.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:38 smithi007 ceph-mon[37158]: pgmap v437: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 734 op/s; 19977/319962 objects degraded (6.244%); 99412/319962 objects misplaced (31.070%); 63 MiB/s, 1.01k objects/s recovering 2024-02-05T07:27:38.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:38.670 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:38.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:38.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:38.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:38.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:38 smithi090 ceph-mon[33213]: pgmap v437: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 734 op/s; 19977/319962 objects degraded (6.244%); 99412/319962 objects misplaced (31.070%); 63 MiB/s, 1.01k objects/s recovering 2024-02-05T07:27:38.806 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:38.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:38.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:38.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:38.946 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:38.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:39.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:39.093 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:27:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:39.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:39.212 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:39.244 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:39.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:39.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:39.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:39.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:39.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:39.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:39.528 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:39.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:39.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:39.630 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:39.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:39 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 19977/319962 objects degraded (6.244%), 17 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:39.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:39 smithi007 ceph-mon[37158]: osdmap e276: 8 total, 7 up, 7 in 2024-02-05T07:27:39.647 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:39 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 19977/319962 objects degraded (6.244%), 17 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:39.647 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:39 smithi007 ceph-mon[29098]: osdmap e276: 8 total, 7 up, 7 in 2024-02-05T07:27:39.672 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:39.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:39.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:39.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:39 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 19977/319962 objects degraded (6.244%), 17 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:39.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:39 smithi090 ceph-mon[33213]: osdmap e276: 8 total, 7 up, 7 in 2024-02-05T07:27:39.800 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:39.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:39.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:39.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:27:39.946 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:39.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:39.985 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:27:40.020 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:27:40.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:40.057 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:27:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:40.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:40.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:40.095 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:27:40.133 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:27:40.171 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:27:40.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:40.200 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:40.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:40.208 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:27:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:27:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:27:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:27:40.209 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:27:40.209 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:27:40.209 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:27:40.209 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-02-05T07:27:40.209 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:27:40.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:40.333 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:40.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:40.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:40.463 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:40.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:40.540 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:40 smithi007 ceph-mon[29098]: pgmap v438: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 601 op/s; 19977/319962 objects degraded (6.244%); 99412/319962 objects misplaced (31.070%); 52 MiB/s, 826 objects/s recovering 2024-02-05T07:27:40.540 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:40 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/91949023' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:27:40.540 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:40 smithi007 ceph-mon[29098]: osdmap e277: 8 total, 7 up, 7 in 2024-02-05T07:27:40.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:40 smithi007 ceph-mon[37158]: pgmap v438: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 601 op/s; 19977/319962 objects degraded (6.244%); 99412/319962 objects misplaced (31.070%); 52 MiB/s, 826 objects/s recovering 2024-02-05T07:27:40.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:40 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/91949023' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:27:40.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:40 smithi007 ceph-mon[37158]: osdmap e277: 8 total, 7 up, 7 in 2024-02-05T07:27:40.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:40.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:27:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:40.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:40.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:40.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:40.760 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:40.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:40.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:40.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:27:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:27:40.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:40 smithi090 ceph-mon[33213]: pgmap v438: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 601 op/s; 19977/319962 objects degraded (6.244%); 99412/319962 objects misplaced (31.070%); 52 MiB/s, 826 objects/s recovering 2024-02-05T07:27:40.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:40 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/91949023' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:27:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:40 smithi090 ceph-mon[33213]: osdmap e277: 8 total, 7 up, 7 in 2024-02-05T07:27:40.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:40.930 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:40.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:41.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:41.066 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:41.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:41.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:41.212 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:41.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:41.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:41.342 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:41.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:41.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:41.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:41.474 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:41.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:41.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:41.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:41 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:27:41.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:41 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:27:41.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:41 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:41.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:41 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:27:41.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:41 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:27:41.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:41 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:41.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:41.606 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:41.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:41.708 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:41.751 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:41.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:41.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:41 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:27:41.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:41 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:27:41.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:41 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:41.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:41.901 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:41.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:42.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:42.044 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:42.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:42.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:42.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:42.177 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:42.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:42.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:42.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:42.326 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:42.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:42.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:42.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:42.476 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:42.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:42.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:42.608 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:42.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:42.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:42.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:42 smithi007 ceph-mon[29098]: pgmap v441: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 2 active+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.41k op/s; 15359/305478 objects degraded (5.028%); 90492/305478 objects misplaced (29.623%); 98 MiB/s, 1.57k objects/s recovering 2024-02-05T07:27:42.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:42 smithi007 ceph-mon[37158]: pgmap v441: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 2 active+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.41k op/s; 15359/305478 objects degraded (5.028%); 90492/305478 objects misplaced (29.623%); 98 MiB/s, 1.57k objects/s recovering 2024-02-05T07:27:42.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:42.747 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:42.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:42 smithi090 ceph-mon[33213]: pgmap v441: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 2 active+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.41k op/s; 15359/305478 objects degraded (5.028%); 90492/305478 objects misplaced (29.623%); 98 MiB/s, 1.57k objects/s recovering 2024-02-05T07:27:42.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:42.892 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:42.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:42.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:43.019 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:43.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:43.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:43.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:43.146 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:43.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:43.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:43.272 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:43.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T07:27:43.341 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T07:27:43.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:43.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:43.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:43.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:43.537 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:43 smithi007 ceph-mon[29098]: osdmap e278: 8 total, 7 up, 7 in 2024-02-05T07:27:43.537 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:43 smithi007 ceph-mon[37158]: osdmap e278: 8 total, 7 up, 7 in 2024-02-05T07:27:43.542 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:43.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:43.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:43.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:43.679 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:43.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:43.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:43 smithi090 ceph-mon[33213]: osdmap e278: 8 total, 7 up, 7 in 2024-02-05T07:27:43.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:43.813 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:43.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:43.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:43.822 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:27:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:27:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:27:43.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:43.926 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:27:43.926 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:27:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:43.949 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:43.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:44.050 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:44.084 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:44.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:44.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:44.217 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:44.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:44.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:44.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:44.359 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:44.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:44.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:44.460 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:44.505 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:44.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:44.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:44.632 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:27:44.642 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:44.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:44.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:44.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:44 smithi007 ceph-mon[29098]: pgmap v443: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 2 active+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 805 op/s; 15359/305478 objects degraded (5.028%); 90492/305478 objects misplaced (29.623%); 47 MiB/s, 744 objects/s recovering 2024-02-05T07:27:44.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:44 smithi007 ceph-mon[29098]: osdmap e279: 8 total, 7 up, 7 in 2024-02-05T07:27:44.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:44 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 15359/305478 objects degraded (5.028%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:44.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:44 smithi007 ceph-mon[29098]: osdmap e280: 8 total, 7 up, 7 in 2024-02-05T07:27:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:44 smithi007 ceph-mon[37158]: pgmap v443: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 2 active+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 805 op/s; 15359/305478 objects degraded (5.028%); 90492/305478 objects misplaced (29.623%); 47 MiB/s, 744 objects/s recovering 2024-02-05T07:27:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:44 smithi007 ceph-mon[37158]: osdmap e279: 8 total, 7 up, 7 in 2024-02-05T07:27:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:44 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 15359/305478 objects degraded (5.028%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:44 smithi007 ceph-mon[37158]: osdmap e280: 8 total, 7 up, 7 in 2024-02-05T07:27:44.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:44.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:44 smithi090 ceph-mon[33213]: pgmap v443: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 2 active+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 805 op/s; 15359/305478 objects degraded (5.028%); 90492/305478 objects misplaced (29.623%); 47 MiB/s, 744 objects/s recovering 2024-02-05T07:27:44.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:44 smithi090 ceph-mon[33213]: osdmap e279: 8 total, 7 up, 7 in 2024-02-05T07:27:44.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:44 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 15359/305478 objects degraded (5.028%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:44.780 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:44 smithi090 ceph-mon[33213]: osdmap e280: 8 total, 7 up, 7 in 2024-02-05T07:27:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:27:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:44.786 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:44.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:44.786 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:27:44.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:44.917 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:44.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:44.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:45.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:45.052 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:45.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:45.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:45.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:45.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:45.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:45.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:45.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:27:45.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:45.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:45.459 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:45.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:45.560 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:45.605 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:45.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:45.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:45.762 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:45.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:45.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:45.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:45.891 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:45.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:45.993 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:46.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:46.166 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:46.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:46.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:46.298 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:46.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:46.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:46.399 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:46.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:46 smithi007 ceph-mon[29098]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:27:46.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:46 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:27:46.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:27:46.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:46 smithi007 ceph-mon[29098]: pgmap v446: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 2 active+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 816 op/s; 15359/305478 objects degraded (5.028%); 90492/305478 objects misplaced (29.623%); 39 MiB/s, 626 objects/s recovering 2024-02-05T07:27:46.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:46 smithi007 ceph-mon[29098]: osdmap e281: 8 total, 7 up, 7 in 2024-02-05T07:27:46.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:46 smithi007 ceph-mon[37158]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:27:46.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:46 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:27:46.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:27:46.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:46 smithi007 ceph-mon[37158]: pgmap v446: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 2 active+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 816 op/s; 15359/305478 objects degraded (5.028%); 90492/305478 objects misplaced (29.623%); 39 MiB/s, 626 objects/s recovering 2024-02-05T07:27:46.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:46 smithi007 ceph-mon[37158]: osdmap e281: 8 total, 7 up, 7 in 2024-02-05T07:27:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:46.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:46.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:46 smithi090 ceph-mon[33213]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:27:46.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:46 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:27:46.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:27:46.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:46 smithi090 ceph-mon[33213]: pgmap v446: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 2 active+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 31 active+clean+remapped, 5 active+remapped+backfill_toofull, 5 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 96 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 816 op/s; 15359/305478 objects degraded (5.028%); 90492/305478 objects misplaced (29.623%); 39 MiB/s, 626 objects/s recovering 2024-02-05T07:27:46.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:46 smithi090 ceph-mon[33213]: osdmap e281: 8 total, 7 up, 7 in 2024-02-05T07:27:46.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:46.565 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:46.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:46.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:46.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:46.702 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:46.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:46.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:46.837 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:46.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:46.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:46.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:46.979 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:46.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:47.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:47.104 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:47.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:47.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:47.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:47.229 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:47.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:47.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:47.331 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:47.365 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:47.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:47.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:47.467 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:47.496 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:47.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:47.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:47.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:47.621 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:47.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:47.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:47.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:47 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:27:47.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:47 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:27:47.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:47 smithi007 ceph-mon[29098]: osdmap e282: 8 total, 7 up, 7 in 2024-02-05T07:27:47.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:47 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:27:47.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:47 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:27:47.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:47 smithi007 ceph-mon[37158]: osdmap e282: 8 total, 7 up, 7 in 2024-02-05T07:27:47.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:47.760 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:47.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:47.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:47 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:27:47.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:47 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:27:47.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:47 smithi090 ceph-mon[33213]: osdmap e282: 8 total, 7 up, 7 in 2024-02-05T07:27:47.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:47.886 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:47.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:47.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:47.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:48.022 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:48.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:48.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:48.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:48.150 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:48.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:48.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:48.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:48.294 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:48.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:48.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:48.436 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:48.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:48.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:48.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:48.583 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:48.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:48.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:48.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:48 smithi007 ceph-mon[29098]: pgmap v449: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 35 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 902 op/s; 13239/294663 objects degraded (4.493%); 86613/294663 objects misplaced (29.394%); 77 MiB/s, 1.23k objects/s recovering 2024-02-05T07:27:48.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:48 smithi007 ceph-mon[37158]: pgmap v449: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 35 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 902 op/s; 13239/294663 objects degraded (4.493%); 86613/294663 objects misplaced (29.394%); 77 MiB/s, 1.23k objects/s recovering 2024-02-05T07:27:48.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:48.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:48.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:48.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:48.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:48 smithi090 ceph-mon[33213]: pgmap v449: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 35 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 902 op/s; 13239/294663 objects degraded (4.493%); 86613/294663 objects misplaced (29.394%); 77 MiB/s, 1.23k objects/s recovering 2024-02-05T07:27:48.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:48.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:48.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:48.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:48.976 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:48.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:49.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:49.112 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:49.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:49.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:49.251 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:49.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:49.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:49.395 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:49.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:49.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:49.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:49.542 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:49.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:49.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:49.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:49.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:49 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 13239/294663 objects degraded (4.493%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:49.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:49 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 13239/294663 objects degraded (4.493%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:49.682 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:49.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:49.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T07:27:49.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:49 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 13239/294663 objects degraded (4.493%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:49.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:49.840 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:49.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:49.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:49.973 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:49.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:49.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:50.075 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:50.099 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:50.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:50.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:50.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:50.232 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:50.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:50.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:50.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:50.368 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:50.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:50.470 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:50.511 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:50.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:50.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:50.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:50.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:50.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:50.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:50 smithi007 ceph-mon[29098]: pgmap v450: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 35 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 890 B/s rd, 0 B/s wr, 627 op/s; 13239/294663 objects degraded (4.493%); 86613/294663 objects misplaced (29.394%); 54 MiB/s, 859 objects/s recovering 2024-02-05T07:27:50.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:50 smithi007 ceph-mon[37158]: pgmap v450: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 35 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 890 B/s rd, 0 B/s wr, 627 op/s; 13239/294663 objects degraded (4.493%); 86613/294663 objects misplaced (29.394%); 54 MiB/s, 859 objects/s recovering 2024-02-05T07:27:50.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:50.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:50 smithi090 ceph-mon[33213]: pgmap v450: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 35 active+clean+remapped, 5 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 890 B/s rd, 0 B/s wr, 627 op/s; 13239/294663 objects degraded (4.493%); 86613/294663 objects misplaced (29.394%); 54 MiB/s, 859 objects/s recovering 2024-02-05T07:27:50.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:27:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:27:50.795 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:50.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:50.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:50.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:50.928 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:50.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:50.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:51.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:51.071 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:51.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:51.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:51.225 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:51.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:51.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:51.354 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:51.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:51.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:51.366 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2024-02-05T07:27:44.261163+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6660739805,"num_objects":101826,"num_object_clones":0,"num_object_copies":305478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15359,"num_objects_misplaced":90492,"num_objects_unfound":0,"num_objects_dirty":101826,"num_whiteouts":0,"num_read":1351,"num_read_kb":1165,"num_write":167220,"num_write_kb":8451376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34180,"num_bytes_recovered":2226409760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136847,"ondisk_log_size":136847,"up":433,"acting":475,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":548,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19564976,"kb_used_data":17069844,"kb_used_omap":10,"kb_used_meta":2494965,"kb_avail":636651088,"statfs":{"total":671965249536,"available":651930714112,"internally_reserved":0,"allocated":17479520256,"data_stored":17473049604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":2554844267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,3,6,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-316407808,"num_objects":-4828,"num_object_clones":0,"num_object_copies":-14484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4618,"num_objects_misplaced":-8920,"num_objects_unfound":0,"num_objects_dirty":-4828,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4828,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4467,"num_bytes_recovered":292749312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.997696"},"pg_stats":[{"pgid":"1.9","version":"86'128","reported_seq":744,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.270681+0000","last_change":"2024-02-05T07:27:19.752065+0000","last_active":"2024-02-05T07:27:40.270681+0000","last_peered":"2024-02-05T07:27:40.270681+0000","last_clean":"2024-02-05T07:27:40.270681+0000","last_became_active":"2024-02-05T07:27:19.751740+0000","last_became_peered":"2024-02-05T07:27:19.751740+0000","last_unstale":"2024-02-05T07:27:40.270681+0000","last_undegraded":"2024-02-05T07:27:40.270681+0000","last_fullsized":"2024-02-05T07:27:40.270681+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:26:18.993912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.993912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:44.273129+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"277'9483","reported_seq":9974,"reported_epoch":277,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-05T07:27:40.386888+0000","last_change":"2024-02-05T07:27:17.712853+0000","last_active":"2024-02-05T07:27:40.386888+0000","last_peered":"2024-02-05T07:27:40.386888+0000","last_clean":"2024-02-05T07:27:15.629309+0000","last_became_active":"2024-02-05T07:27:16.888635+0000","last_became_peered":"2024-02-05T07:27:16.888635+0000","last_unstale":"2024-02-05T07:27:40.386888+0000","last_undegraded":"2024-02-05T07:27:40.386888+0000","last_fullsized":"2024-02-05T07:27:40.386888+0000","mapping_epoch":263,"log_start":"232'3700","ondisk_log_start":"232'3700","created":209,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5783,"log_dups_size":0,"ondisk_log_size":5783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203816960,"num_objects":3110,"num_object_clones":0,"num_object_copies":9330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2972,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5416,"num_write_kb":272832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":3110}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"221'15","reported_seq":507,"reported_epoch":276,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:39.268898+0000","last_change":"2024-02-05T07:27:17.723973+0000","last_active":"2024-02-05T07:27:39.268898+0000","last_peered":"2024-02-05T07:27:39.268898+0000","last_clean":"2024-02-05T07:27:39.268898+0000","last_became_active":"2024-02-05T07:27:17.704761+0000","last_became_peered":"2024-02-05T07:27:17.704761+0000","last_unstale":"2024-02-05T07:27:39.268898+0000","last_undegraded":"2024-02-05T07:27:39.268898+0000","last_fullsized":"2024-02-05T07:27:39.268898+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'15","last_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:25:44.205475+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,1],"avail_no_missing":["4","1","3"],"object_location_counts":[{"shards":"1,3,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":37,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.277733+0000","last_change":"2024-02-05T07:27:16.716749+0000","last_active":"2024-02-05T07:27:40.277733+0000","last_peered":"2024-02-05T07:27:40.277733+0000","last_clean":"2024-02-05T07:27:40.277733+0000","last_became_active":"2024-02-05T07:27:16.716461+0000","last_became_peered":"2024-02-05T07:27:16.716461+0000","last_unstale":"2024-02-05T07:27:40.277733+0000","last_undegraded":"2024-02-05T07:27:40.277733+0000","last_fullsized":"2024-02-05T07:27:40.277733+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.258469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.258469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"257'9","reported_seq":575,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.281762+0000","last_change":"2024-02-05T07:27:03.390168+0000","last_active":"2024-02-05T07:27:40.281762+0000","last_peered":"2024-02-05T07:27:40.281762+0000","last_clean":"2024-02-05T07:27:40.281762+0000","last_became_active":"2024-02-05T07:27:03.374124+0000","last_became_peered":"2024-02-05T07:27:03.374124+0000","last_unstale":"2024-02-05T07:27:40.281762+0000","last_undegraded":"2024-02-05T07:27:40.281762+0000","last_fullsized":"2024-02-05T07:27:40.281762+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:25:45.823432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:45.823432+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.900655+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":441,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-02-05T07:27:39.269621+0000","last_change":"2024-02-05T07:25:43.273140+0000","last_active":"2024-02-05T07:27:39.269621+0000","last_peered":"2024-02-05T07:27:39.269621+0000","last_clean":"2024-02-05T07:27:39.269621+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:27:39.269621+0000","last_undegraded":"2024-02-05T07:27:39.269621+0000","last_fullsized":"2024-02-05T07:27:39.269621+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:25:43.273019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":439,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.280585+0000","last_change":"2024-02-05T07:27:19.750855+0000","last_active":"2024-02-05T07:27:40.280585+0000","last_peered":"2024-02-05T07:27:40.280585+0000","last_clean":"2024-02-05T07:27:40.280585+0000","last_became_active":"2024-02-05T07:27:19.749493+0000","last_became_peered":"2024-02-05T07:27:19.749493+0000","last_unstale":"2024-02-05T07:27:40.280585+0000","last_undegraded":"2024-02-05T07:27:40.280585+0000","last_fullsized":"2024-02-05T07:27:40.280585+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:41.902162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":949073,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3,4],"avail_no_missing":["2","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"277'9441","reported_seq":9647,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.402022+0000","last_change":"2024-02-05T07:27:01.628112+0000","last_active":"2024-02-05T07:27:40.402022+0000","last_peered":"2024-02-05T07:27:40.402022+0000","last_clean":"2024-02-05T07:27:40.402022+0000","last_became_active":"2024-02-05T07:26:54.214408+0000","last_became_peered":"2024-02-05T07:26:54.214408+0000","last_unstale":"2024-02-05T07:27:40.402022+0000","last_undegraded":"2024-02-05T07:27:40.402022+0000","last_fullsized":"2024-02-05T07:27:40.402022+0000","mapping_epoch":209,"log_start":"232'4000","ondisk_log_start":"232'4000","created":209,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'8402","last_scrub_stamp":"2024-02-05T07:27:01.627945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:27:01.627945+0000","objects_scrubbed":4243,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:28:16.232755+0000","scrub_duration":1206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208863232,"num_objects":3187,"num_object_clones":0,"num_object_copies":9561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1064,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":485,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-02-05T07:27:39.268621+0000","last_change":"2024-02-05T07:26:36.275060+0000","last_active":"2024-02-05T07:27:39.268621+0000","last_peered":"2024-02-05T07:27:39.268621+0000","last_clean":"2024-02-05T07:27:39.268621+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:27:39.268621+0000","last_undegraded":"2024-02-05T07:27:39.268621+0000","last_fullsized":"2024-02-05T07:27:39.268621+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:26:36.274939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.274939+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:45.343791+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":97,"reported_epoch":276,"state":"active+undersized","last_fresh":"2024-02-05T07:27:39.268239+0000","last_change":"2024-02-05T07:25:59.233701+0000","last_active":"2024-02-05T07:27:39.268239+0000","last_peered":"2024-02-05T07:27:39.268239+0000","last_clean":"2024-02-05T07:18:48.410364+0000","last_became_active":"2024-02-05T07:25:59.233701+0000","last_became_peered":"2024-02-05T07:25:59.233701+0000","last_unstale":"2024-02-05T07:27:39.268239+0000","last_undegraded":"2024-02-05T07:27:39.268239+0000","last_fullsized":"2024-02-05T07:25:59.218441+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:35.518301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:35.518301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":448,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.272878+0000","last_change":"2024-02-05T07:25:39.466584+0000","last_active":"2024-02-05T07:27:40.272878+0000","last_peered":"2024-02-05T07:27:40.272878+0000","last_clean":"2024-02-05T07:27:40.272878+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:27:40.272878+0000","last_undegraded":"2024-02-05T07:27:40.272878+0000","last_fullsized":"2024-02-05T07:27:40.272878+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:39.466451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:39.466451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":450,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.282560+0000","last_change":"2024-02-05T07:27:16.709170+0000","last_active":"2024-02-05T07:27:40.282560+0000","last_peered":"2024-02-05T07:27:40.282560+0000","last_clean":"2024-02-05T07:27:40.282560+0000","last_became_active":"2024-02-05T07:27:16.708888+0000","last_became_peered":"2024-02-05T07:27:16.708888+0000","last_unstale":"2024-02-05T07:27:40.282560+0000","last_undegraded":"2024-02-05T07:27:40.282560+0000","last_fullsized":"2024-02-05T07:27:40.282560+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:54.743499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:54.743499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.900452+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"276'9339","reported_seq":9504,"reported_epoch":276,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:27:39.575673+0000","last_change":"2024-02-05T07:27:16.975559+0000","last_active":"2024-02-05T07:27:39.575673+0000","last_peered":"2024-02-05T07:27:39.575673+0000","last_clean":"2024-02-05T07:27:15.554041+0000","last_became_active":"2024-02-05T07:27:16.962319+0000","last_became_peered":"2024-02-05T07:27:16.962319+0000","last_unstale":"2024-02-05T07:27:39.575673+0000","last_undegraded":"2024-02-05T07:27:39.575673+0000","last_fullsized":"2024-02-05T07:27:39.575673+0000","mapping_epoch":263,"log_start":"232'3900","ondisk_log_start":"232'3900","created":209,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5439,"log_dups_size":0,"ondisk_log_size":5439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209027072,"num_objects":3189,"num_object_clones":0,"num_object_copies":9567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3189,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5289,"num_write_kb":271328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,5],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4,5","objects":3189}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"90'16","reported_seq":476,"reported_epoch":276,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:27:39.269146+0000","last_change":"2024-02-05T07:25:59.232148+0000","last_active":"2024-02-05T07:27:39.269146+0000","last_peered":"2024-02-05T07:27:39.269146+0000","last_clean":"2024-02-05T07:25:57.230984+0000","last_became_active":"2024-02-05T07:25:59.232148+0000","last_became_peered":"2024-02-05T07:25:59.232148+0000","last_unstale":"2024-02-05T07:27:39.269146+0000","last_undegraded":"2024-02-05T07:25:59.222004+0000","last_fullsized":"2024-02-05T07:25:59.221213+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-02-05T07:25:27.129600+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:25:27.129600+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":454,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-02-05T07:27:39.269729+0000","last_change":"2024-02-05T07:25:46.149566+0000","last_active":"2024-02-05T07:27:39.269729+0000","last_peered":"2024-02-05T07:27:39.269729+0000","last_clean":"2024-02-05T07:27:39.269729+0000","last_became_active":"2024-02-05T07:18:49.434074+0000","last_became_peered":"2024-02-05T07:18:49.434074+0000","last_unstale":"2024-02-05T07:27:39.269729+0000","last_undegraded":"2024-02-05T07:27:39.269729+0000","last_fullsized":"2024-02-05T07:27:39.269729+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:25:46.149431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":275,"reported_epoch":276,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:39.268292+0000","last_change":"2024-02-05T07:27:04.352624+0000","last_active":"2024-02-05T07:27:39.268292+0000","last_peered":"2024-02-05T07:27:39.268292+0000","last_clean":"2024-02-05T07:27:39.268292+0000","last_became_active":"2024-02-05T07:27:04.352217+0000","last_became_peered":"2024-02-05T07:27:04.352217+0000","last_unstale":"2024-02-05T07:27:39.268292+0000","last_undegraded":"2024-02-05T07:27:39.268292+0000","last_fullsized":"2024-02-05T07:27:39.268292+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:49.760543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.760543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.277685+0000","last_change":"2024-02-05T07:27:17.711103+0000","last_active":"2024-02-05T07:27:40.277685+0000","last_peered":"2024-02-05T07:27:40.277685+0000","last_clean":"2024-02-05T07:27:40.277685+0000","last_became_active":"2024-02-05T07:27:17.710673+0000","last_became_peered":"2024-02-05T07:27:17.710673+0000","last_unstale":"2024-02-05T07:27:40.277685+0000","last_undegraded":"2024-02-05T07:27:40.277685+0000","last_fullsized":"2024-02-05T07:27:40.277685+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:47.885294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.885294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":422,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.281858+0000","last_change":"2024-02-05T07:27:06.398203+0000","last_active":"2024-02-05T07:27:40.281858+0000","last_peered":"2024-02-05T07:27:40.281858+0000","last_clean":"2024-02-05T07:27:40.281858+0000","last_became_active":"2024-02-05T07:27:06.397364+0000","last_became_peered":"2024-02-05T07:27:06.397364+0000","last_unstale":"2024-02-05T07:27:40.281858+0000","last_undegraded":"2024-02-05T07:27:40.281858+0000","last_fullsized":"2024-02-05T07:27:40.281858+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:40.878713+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4,5],"avail_no_missing":["2","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3","version":"277'9141","reported_seq":16401,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.424587+0000","last_change":"2024-02-05T07:27:35.349048+0000","last_active":"2024-02-05T07:27:40.424587+0000","last_peered":"2024-02-05T07:27:40.424587+0000","last_clean":"2024-02-05T07:27:40.424587+0000","last_became_active":"2024-02-05T07:27:35.270174+0000","last_became_peered":"2024-02-05T07:27:35.270174+0000","last_unstale":"2024-02-05T07:27:40.424587+0000","last_undegraded":"2024-02-05T07:27:40.424587+0000","last_fullsized":"2024-02-05T07:27:40.424587+0000","mapping_epoch":273,"log_start":"232'3700","ondisk_log_start":"232'3700","created":209,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:41.381820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204832768,"num_objects":3125,"num_object_clones":0,"num_object_copies":9375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3125,"num_objects_unfound":0,"num_objects_dirty":3125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5173,"num_write_kb":265568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3543,"num_bytes_recovered":232194048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":3125}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"98'22","reported_seq":528,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.274407+0000","last_change":"2024-02-05T07:27:16.982664+0000","last_active":"2024-02-05T07:27:40.274407+0000","last_peered":"2024-02-05T07:27:40.274407+0000","last_clean":"2024-02-05T07:27:40.274407+0000","last_became_active":"2024-02-05T07:27:16.768454+0000","last_became_peered":"2024-02-05T07:27:16.768454+0000","last_unstale":"2024-02-05T07:27:40.274407+0000","last_undegraded":"2024-02-05T07:27:40.274407+0000","last_fullsized":"2024-02-05T07:27:40.274407+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'22","last_scrub_stamp":"2024-02-05T07:26:25.602159+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:26:25.602159+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":37,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.277241+0000","last_change":"2024-02-05T07:27:16.830852+0000","last_active":"2024-02-05T07:27:40.277241+0000","last_peered":"2024-02-05T07:27:40.277241+0000","last_clean":"2024-02-05T07:27:40.277241+0000","last_became_active":"2024-02-05T07:27:16.800735+0000","last_became_peered":"2024-02-05T07:27:16.800735+0000","last_unstale":"2024-02-05T07:27:40.277241+0000","last_undegraded":"2024-02-05T07:27:40.277241+0000","last_fullsized":"2024-02-05T07:27:40.277241+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:48.513614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:48.513614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"257'9","reported_seq":570,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-02-05T07:27:39.269554+0000","last_change":"2024-02-05T07:27:03.386101+0000","last_active":"2024-02-05T07:27:39.269554+0000","last_peered":"2024-02-05T07:27:39.269554+0000","last_clean":"2024-02-05T07:27:39.269554+0000","last_became_active":"2024-02-05T07:27:03.375195+0000","last_became_peered":"2024-02-05T07:27:03.375195+0000","last_unstale":"2024-02-05T07:27:39.269554+0000","last_undegraded":"2024-02-05T07:27:39.269554+0000","last_fullsized":"2024-02-05T07:27:39.269554+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:26:22.295732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:22.295732+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":55,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.273915+0000","last_change":"2024-02-05T07:27:03.368067+0000","last_active":"2024-02-05T07:27:40.273915+0000","last_peered":"2024-02-05T07:27:40.273915+0000","last_clean":"2024-02-05T07:27:40.273915+0000","last_became_active":"2024-02-05T07:27:03.367753+0000","last_became_peered":"2024-02-05T07:27:03.367753+0000","last_unstale":"2024-02-05T07:27:40.273915+0000","last_undegraded":"2024-02-05T07:27:40.273915+0000","last_fullsized":"2024-02-05T07:27:40.273915+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.724944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.724944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":411,"reported_epoch":276,"state":"active+undersized","last_fresh":"2024-02-05T07:27:39.267653+0000","last_change":"2024-02-05T07:25:59.232121+0000","last_active":"2024-02-05T07:27:39.267653+0000","last_peered":"2024-02-05T07:27:39.267653+0000","last_clean":"2024-02-05T07:25:57.249010+0000","last_became_active":"2024-02-05T07:25:59.232121+0000","last_became_peered":"2024-02-05T07:25:59.232121+0000","last_unstale":"2024-02-05T07:27:39.267653+0000","last_undegraded":"2024-02-05T07:27:39.267653+0000","last_fullsized":"2024-02-05T07:25:59.217731+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:39.858635+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"277'10367","reported_seq":15995,"reported_epoch":277,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-05T07:27:40.315901+0000","last_change":"2024-02-05T07:27:17.027920+0000","last_active":"2024-02-05T07:27:40.315901+0000","last_peered":"2024-02-05T07:27:40.315901+0000","last_clean":"2024-02-05T07:27:02.321559+0000","last_became_active":"2024-02-05T07:27:16.963581+0000","last_became_peered":"2024-02-05T07:27:16.963581+0000","last_unstale":"2024-02-05T07:27:40.315901+0000","last_undegraded":"2024-02-05T07:27:40.315901+0000","last_fullsized":"2024-02-05T07:27:40.315901+0000","mapping_epoch":263,"log_start":"242'6361","ondisk_log_start":"242'6361","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":4006,"log_dups_size":0,"ondisk_log_size":4006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408223744,"num_objects":6229,"num_object_clones":0,"num_object_copies":18687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6229,"num_objects_unfound":0,"num_objects_dirty":6229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10367,"num_write_kb":531072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2733,"num_bytes_recovered":178978816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[7,0,5],"avail_no_missing":["7","0","4","5"],"object_location_counts":[{"shards":"0,5,7","objects":6229}],"blocked_by":[],"up_primary":3,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":458,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-02-05T07:27:39.269663+0000","last_change":"2024-02-05T07:27:16.706337+0000","last_active":"2024-02-05T07:27:39.269663+0000","last_peered":"2024-02-05T07:27:39.269663+0000","last_clean":"2024-02-05T07:27:39.269663+0000","last_became_active":"2024-02-05T07:27:16.705976+0000","last_became_peered":"2024-02-05T07:27:16.705976+0000","last_unstale":"2024-02-05T07:27:39.269663+0000","last_undegraded":"2024-02-05T07:27:39.269663+0000","last_fullsized":"2024-02-05T07:27:39.269663+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:51.211877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:51.211877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":55,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.277247+0000","last_change":"2024-02-05T07:27:17.056320+0000","last_active":"2024-02-05T07:27:40.277247+0000","last_peered":"2024-02-05T07:27:40.277247+0000","last_clean":"2024-02-05T07:27:40.277247+0000","last_became_active":"2024-02-05T07:27:16.830389+0000","last_became_peered":"2024-02-05T07:27:16.830389+0000","last_unstale":"2024-02-05T07:27:40.277247+0000","last_undegraded":"2024-02-05T07:27:40.277247+0000","last_fullsized":"2024-02-05T07:27:40.277247+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:24:14.359892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:14.359892+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":37,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.279153+0000","last_change":"2024-02-05T07:27:16.720166+0000","last_active":"2024-02-05T07:27:40.279153+0000","last_peered":"2024-02-05T07:27:40.279153+0000","last_clean":"2024-02-05T07:27:40.279153+0000","last_became_active":"2024-02-05T07:27:16.719678+0000","last_became_peered":"2024-02-05T07:27:16.719678+0000","last_unstale":"2024-02-05T07:27:40.279153+0000","last_undegraded":"2024-02-05T07:27:40.279153+0000","last_fullsized":"2024-02-05T07:27:40.279153+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:33.084016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:33.084016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:48.299897+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":449,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-02-05T07:27:39.268139+0000","last_change":"2024-02-05T07:27:16.707313+0000","last_active":"2024-02-05T07:27:39.268139+0000","last_peered":"2024-02-05T07:27:39.268139+0000","last_clean":"2024-02-05T07:27:39.268139+0000","last_became_active":"2024-02-05T07:27:16.706666+0000","last_became_peered":"2024-02-05T07:27:16.706666+0000","last_unstale":"2024-02-05T07:27:39.268139+0000","last_undegraded":"2024-02-05T07:27:39.268139+0000","last_fullsized":"2024-02-05T07:27:39.268139+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:16.499591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:16.499591+0000","last_clean_scrub_stamp":"2024-02-05T07:23:16.499591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":673,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.277817+0000","last_change":"2024-02-05T07:27:19.752409+0000","last_active":"2024-02-05T07:27:40.277817+0000","last_peered":"2024-02-05T07:27:40.277817+0000","last_clean":"2024-02-05T07:27:40.277817+0000","last_became_active":"2024-02-05T07:27:19.751772+0000","last_became_peered":"2024-02-05T07:27:19.751772+0000","last_unstale":"2024-02-05T07:27:40.277817+0000","last_undegraded":"2024-02-05T07:27:40.277817+0000","last_fullsized":"2024-02-05T07:27:40.277817+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.792569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,4],"avail_no_missing":["3","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"277'10728","reported_seq":10883,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.327038+0000","last_change":"2024-02-05T07:27:04.366098+0000","last_active":"2024-02-05T07:27:40.327038+0000","last_peered":"2024-02-05T07:27:40.327038+0000","last_clean":"2024-02-05T07:27:40.327038+0000","last_became_active":"2024-02-05T07:27:04.352207+0000","last_became_peered":"2024-02-05T07:27:04.352207+0000","last_unstale":"2024-02-05T07:27:40.327038+0000","last_undegraded":"2024-02-05T07:27:40.327038+0000","last_fullsized":"2024-02-05T07:27:40.327038+0000","mapping_epoch":256,"log_start":"242'6741","ondisk_log_start":"242'6741","created":209,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":3987,"log_dups_size":0,"ondisk_log_size":3987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419102744,"num_objects":6396,"num_object_clones":0,"num_object_copies":19188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6396,"num_objects_unfound":0,"num_objects_dirty":6396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10728,"num_write_kb":547905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,5],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"2,5,7","objects":6396}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"266'10","reported_seq":46,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.277329+0000","last_change":"2024-02-05T07:27:17.017592+0000","last_active":"2024-02-05T07:27:40.277329+0000","last_peered":"2024-02-05T07:27:40.277329+0000","last_clean":"2024-02-05T07:27:40.277329+0000","last_became_active":"2024-02-05T07:27:16.800207+0000","last_became_peered":"2024-02-05T07:27:16.800207+0000","last_unstale":"2024-02-05T07:27:40.277329+0000","last_undegraded":"2024-02-05T07:27:40.277329+0000","last_fullsized":"2024-02-05T07:27:40.277329+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:26:34.535447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:34.535447+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:45.356054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"202'21","reported_seq":494,"reported_epoch":276,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:39.269811+0000","last_change":"2024-02-05T07:27:04.396336+0000","last_active":"2024-02-05T07:27:39.269811+0000","last_peered":"2024-02-05T07:27:39.269811+0000","last_clean":"2024-02-05T07:27:39.269811+0000","last_became_active":"2024-02-05T07:27:04.368117+0000","last_became_peered":"2024-02-05T07:27:04.368117+0000","last_unstale":"2024-02-05T07:27:39.269811+0000","last_undegraded":"2024-02-05T07:27:39.269811+0000","last_fullsized":"2024-02-05T07:27:39.269811+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'21","last_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:24:59.870542+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,5],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4,5","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":209,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.275067+0000","last_change":"2024-02-05T07:25:33.462720+0000","last_active":"2024-02-05T07:27:40.275067+0000","last_peered":"2024-02-05T07:27:40.275067+0000","last_clean":"2024-02-05T07:27:40.275067+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:27:40.275067+0000","last_undegraded":"2024-02-05T07:27:40.275067+0000","last_fullsized":"2024-02-05T07:27:40.275067+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:25:33.462588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:33.462588+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":439,"reported_epoch":277,"state":"active+undersized","last_fresh":"2024-02-05T07:27:40.271290+0000","last_change":"2024-02-05T07:25:59.226701+0000","last_active":"2024-02-05T07:27:40.271290+0000","last_peered":"2024-02-05T07:27:40.271290+0000","last_clean":"2024-02-05T07:25:57.239687+0000","last_became_active":"2024-02-05T07:25:59.226701+0000","last_became_peered":"2024-02-05T07:25:59.226701+0000","last_unstale":"2024-02-05T07:27:40.271290+0000","last_undegraded":"2024-02-05T07:27:40.271290+0000","last_fullsized":"2024-02-05T07:25:59.222593+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:59.124133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:59.124133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":720,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.277863+0000","last_change":"2024-02-05T07:27:19.754072+0000","last_active":"2024-02-05T07:27:40.277863+0000","last_peered":"2024-02-05T07:27:40.277863+0000","last_clean":"2024-02-05T07:27:40.277863+0000","last_became_active":"2024-02-05T07:27:19.753426+0000","last_became_peered":"2024-02-05T07:27:19.753426+0000","last_unstale":"2024-02-05T07:27:40.277863+0000","last_undegraded":"2024-02-05T07:27:40.277863+0000","last_fullsized":"2024-02-05T07:27:40.277863+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.792592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,2],"avail_no_missing":["3","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"277'10592","reported_seq":10802,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.725650+0000","last_change":"2024-02-05T07:26:19.695559+0000","last_active":"2024-02-05T07:27:40.725650+0000","last_peered":"2024-02-05T07:27:40.725650+0000","last_clean":"2024-02-05T07:27:40.725650+0000","last_became_active":"2024-02-05T07:25:16.685782+0000","last_became_peered":"2024-02-05T07:25:16.685782+0000","last_unstale":"2024-02-05T07:27:40.725650+0000","last_undegraded":"2024-02-05T07:27:40.725650+0000","last_fullsized":"2024-02-05T07:27:40.725650+0000","mapping_epoch":209,"log_start":"242'6519","ondisk_log_start":"242'6519","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'5314","last_scrub_stamp":"2024-02-05T07:26:19.695409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:26:19.695409+0000","objects_scrubbed":5255,"log_size":4073,"log_dups_size":0,"ondisk_log_size":4073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:46.635667+0000","scrub_duration":1692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407502848,"num_objects":6218,"num_object_clones":0,"num_object_copies":18654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10592,"num_write_kb":537920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"266'14","reported_seq":45,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.277258+0000","last_change":"2024-02-05T07:27:17.739147+0000","last_active":"2024-02-05T07:27:40.277258+0000","last_peered":"2024-02-05T07:27:40.277258+0000","last_clean":"2024-02-05T07:27:40.277258+0000","last_became_active":"2024-02-05T07:27:17.731781+0000","last_became_peered":"2024-02-05T07:27:17.731781+0000","last_unstale":"2024-02-05T07:27:40.277258+0000","last_undegraded":"2024-02-05T07:27:40.277258+0000","last_fullsized":"2024-02-05T07:27:40.277258+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'9","last_scrub_stamp":"2024-02-05T07:24:59.438123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:59.438123+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"151'15","reported_seq":499,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.273417+0000","last_change":"2024-02-05T07:27:03.387977+0000","last_active":"2024-02-05T07:27:40.273417+0000","last_peered":"2024-02-05T07:27:40.273417+0000","last_clean":"2024-02-05T07:27:40.273417+0000","last_became_active":"2024-02-05T07:27:03.361342+0000","last_became_peered":"2024-02-05T07:27:03.361342+0000","last_unstale":"2024-02-05T07:27:40.273417+0000","last_undegraded":"2024-02-05T07:27:40.273417+0000","last_fullsized":"2024-02-05T07:27:40.273417+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-02-05T07:26:26.616180+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:26:26.616180+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":35,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.277543+0000","last_change":"2024-02-05T07:27:17.702081+0000","last_active":"2024-02-05T07:27:40.277543+0000","last_peered":"2024-02-05T07:27:40.277543+0000","last_clean":"2024-02-05T07:27:40.277543+0000","last_became_active":"2024-02-05T07:27:17.701630+0000","last_became_peered":"2024-02-05T07:27:17.701630+0000","last_unstale":"2024-02-05T07:27:40.277543+0000","last_undegraded":"2024-02-05T07:27:40.277543+0000","last_fullsized":"2024-02-05T07:27:40.277543+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:57.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:24:57.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":37,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.277448+0000","last_change":"2024-02-05T07:27:16.831397+0000","last_active":"2024-02-05T07:27:40.277448+0000","last_peered":"2024-02-05T07:27:40.277448+0000","last_clean":"2024-02-05T07:27:40.277448+0000","last_became_active":"2024-02-05T07:27:16.829863+0000","last_became_peered":"2024-02-05T07:27:16.829863+0000","last_unstale":"2024-02-05T07:27:40.277448+0000","last_undegraded":"2024-02-05T07:27:40.277448+0000","last_fullsized":"2024-02-05T07:27:40.277448+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:52.847946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:52.847946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":336,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.278578+0000","last_change":"2024-02-05T07:27:19.750507+0000","last_active":"2024-02-05T07:27:40.278578+0000","last_peered":"2024-02-05T07:27:40.278578+0000","last_clean":"2024-02-05T07:27:40.278578+0000","last_became_active":"2024-02-05T07:27:19.749915+0000","last_became_peered":"2024-02-05T07:27:19.749915+0000","last_unstale":"2024-02-05T07:27:40.278578+0000","last_undegraded":"2024-02-05T07:27:40.278578+0000","last_fullsized":"2024-02-05T07:27:40.278578+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.792544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,4],"avail_no_missing":["3","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"277'10854","reported_seq":11006,"reported_epoch":277,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-05T07:27:40.449623+0000","last_change":"2024-02-05T07:27:17.904744+0000","last_active":"2024-02-05T07:27:40.449623+0000","last_peered":"2024-02-05T07:27:40.449623+0000","last_clean":"2024-02-05T07:27:15.556434+0000","last_became_active":"2024-02-05T07:27:17.888782+0000","last_became_peered":"2024-02-05T07:27:17.888782+0000","last_unstale":"2024-02-05T07:27:40.449623+0000","last_undegraded":"2024-02-05T07:27:40.449623+0000","last_fullsized":"2024-02-05T07:27:40.449623+0000","mapping_epoch":264,"log_start":"243'7092","ondisk_log_start":"243'7092","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":3762,"log_dups_size":0,"ondisk_log_size":3762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425066496,"num_objects":6486,"num_object_clones":0,"num_object_copies":19458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6486,"num_objects_unfound":0,"num_objects_dirty":6486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10854,"num_write_kb":554880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[1,2,4],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":6486}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":52,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.281573+0000","last_change":"2024-02-05T07:27:04.350003+0000","last_active":"2024-02-05T07:27:40.281573+0000","last_peered":"2024-02-05T07:27:40.281573+0000","last_clean":"2024-02-05T07:27:40.281573+0000","last_became_active":"2024-02-05T07:27:04.348768+0000","last_became_peered":"2024-02-05T07:27:04.348768+0000","last_unstale":"2024-02-05T07:27:40.281573+0000","last_undegraded":"2024-02-05T07:27:40.281573+0000","last_fullsized":"2024-02-05T07:27:40.281573+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:30.535958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:25:30.535958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.900523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0,5],"avail_no_missing":["2","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":127,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.271186+0000","last_change":"2024-02-05T07:27:03.398624+0000","last_active":"2024-02-05T07:27:40.271186+0000","last_peered":"2024-02-05T07:27:40.271186+0000","last_clean":"2024-02-05T07:27:40.271186+0000","last_became_active":"2024-02-05T07:27:03.382876+0000","last_became_peered":"2024-02-05T07:27:03.382876+0000","last_unstale":"2024-02-05T07:27:40.271186+0000","last_undegraded":"2024-02-05T07:27:40.271186+0000","last_fullsized":"2024-02-05T07:27:40.271186+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:26:20.705363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:20.705363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":463,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.272790+0000","last_change":"2024-02-05T07:27:03.361745+0000","last_active":"2024-02-05T07:27:40.272790+0000","last_peered":"2024-02-05T07:27:40.272790+0000","last_clean":"2024-02-05T07:27:40.272790+0000","last_became_active":"2024-02-05T07:27:03.361028+0000","last_became_peered":"2024-02-05T07:27:03.361028+0000","last_unstale":"2024-02-05T07:27:40.272790+0000","last_undegraded":"2024-02-05T07:27:40.272790+0000","last_fullsized":"2024-02-05T07:27:40.272790+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:49.513404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:25:49.513404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":440,"reported_epoch":277,"state":"active+undersized","last_fresh":"2024-02-05T07:27:40.273700+0000","last_change":"2024-02-05T07:25:59.224828+0000","last_active":"2024-02-05T07:27:40.273700+0000","last_peered":"2024-02-05T07:27:40.273700+0000","last_clean":"2024-02-05T07:25:57.237073+0000","last_became_active":"2024-02-05T07:25:59.224828+0000","last_became_peered":"2024-02-05T07:25:59.224828+0000","last_unstale":"2024-02-05T07:27:40.273700+0000","last_undegraded":"2024-02-05T07:27:40.273700+0000","last_fullsized":"2024-02-05T07:25:59.217054+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:01.849346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:01.849346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":413,"reported_epoch":277,"state":"active+undersized","last_fresh":"2024-02-05T07:27:40.274008+0000","last_change":"2024-02-05T07:25:59.232229+0000","last_active":"2024-02-05T07:27:40.274008+0000","last_peered":"2024-02-05T07:27:40.274008+0000","last_clean":"2024-02-05T07:25:57.253533+0000","last_became_active":"2024-02-05T07:25:59.232229+0000","last_became_peered":"2024-02-05T07:25:59.232229+0000","last_unstale":"2024-02-05T07:27:40.274008+0000","last_undegraded":"2024-02-05T07:27:40.274008+0000","last_fullsized":"2024-02-05T07:25:59.220509+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:24:21.294546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:24:21.294546+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.374859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.a","version":"277'10906","reported_seq":11391,"reported_epoch":277,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-05T07:27:40.458041+0000","last_change":"2024-02-05T07:27:17.714525+0000","last_active":"2024-02-05T07:27:40.458041+0000","last_peered":"2024-02-05T07:27:40.458041+0000","last_clean":"2024-02-05T07:27:02.326534+0000","last_became_active":"2024-02-05T07:27:16.831253+0000","last_became_peered":"2024-02-05T07:27:16.831253+0000","last_unstale":"2024-02-05T07:27:40.458041+0000","last_undegraded":"2024-02-05T07:27:40.458041+0000","last_fullsized":"2024-02-05T07:27:40.458041+0000","mapping_epoch":263,"log_start":"243'7108","ondisk_log_start":"243'7108","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":3798,"log_dups_size":0,"ondisk_log_size":3798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423886848,"num_objects":6468,"num_object_clones":0,"num_object_copies":19404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6327,"num_objects_unfound":0,"num_objects_dirty":6468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10906,"num_write_kb":555968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":10616832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":6468}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":54,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.274890+0000","last_change":"2024-02-05T07:27:03.370712+0000","last_active":"2024-02-05T07:27:40.274890+0000","last_peered":"2024-02-05T07:27:40.274890+0000","last_clean":"2024-02-05T07:27:40.274890+0000","last_became_active":"2024-02-05T07:27:03.370297+0000","last_became_peered":"2024-02-05T07:27:03.370297+0000","last_unstale":"2024-02-05T07:27:40.274890+0000","last_undegraded":"2024-02-05T07:27:40.274890+0000","last_fullsized":"2024-02-05T07:27:40.274890+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:32.652537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:32.652537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:44.487540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":238,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.275471+0000","last_change":"2024-02-05T07:25:24.528053+0000","last_active":"2024-02-05T07:27:40.275471+0000","last_peered":"2024-02-05T07:27:40.275471+0000","last_clean":"2024-02-05T07:27:40.275471+0000","last_became_active":"2024-02-05T07:24:34.209716+0000","last_became_peered":"2024-02-05T07:24:34.209716+0000","last_unstale":"2024-02-05T07:27:40.275471+0000","last_undegraded":"2024-02-05T07:27:40.275471+0000","last_fullsized":"2024-02-05T07:27:40.275471+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:25:24.527960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:24.527960+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":983,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.270689+0000","last_change":"2024-02-05T07:24:50.043491+0000","last_active":"2024-02-05T07:27:40.270689+0000","last_peered":"2024-02-05T07:27:40.270689+0000","last_clean":"2024-02-05T07:27:40.270689+0000","last_became_active":"2024-02-05T07:18:48.424620+0000","last_became_peered":"2024-02-05T07:18:48.424620+0000","last_unstale":"2024-02-05T07:27:40.270689+0000","last_undegraded":"2024-02-05T07:27:40.270689+0000","last_fullsized":"2024-02-05T07:27:40.270689+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:24:50.043397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:24:50.043397+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":523,"num_read_kb":518,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":426,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-02-05T07:27:39.269816+0000","last_change":"2024-02-05T07:27:16.724562+0000","last_active":"2024-02-05T07:27:39.269816+0000","last_peered":"2024-02-05T07:27:39.269816+0000","last_clean":"2024-02-05T07:27:39.269816+0000","last_became_active":"2024-02-05T07:27:16.724280+0000","last_became_peered":"2024-02-05T07:27:16.724280+0000","last_unstale":"2024-02-05T07:27:39.269816+0000","last_undegraded":"2024-02-05T07:27:39.269816+0000","last_fullsized":"2024-02-05T07:27:39.269816+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:30.351655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:26:30.351655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:59.452467+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":469,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.272443+0000","last_change":"2024-02-05T07:27:17.735731+0000","last_active":"2024-02-05T07:27:40.272443+0000","last_peered":"2024-02-05T07:27:40.272443+0000","last_clean":"2024-02-05T07:27:40.272443+0000","last_became_active":"2024-02-05T07:27:17.704213+0000","last_became_peered":"2024-02-05T07:27:17.704213+0000","last_unstale":"2024-02-05T07:27:40.272443+0000","last_undegraded":"2024-02-05T07:27:40.272443+0000","last_fullsized":"2024-02-05T07:27:40.272443+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:23:22.071923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:23:22.071923+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,4],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"1,3,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"277'9399","reported_seq":18938,"reported_epoch":277,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-05T07:27:40.849989+0000","last_change":"2024-02-05T07:27:17.721643+0000","last_active":"2024-02-05T07:27:40.849989+0000","last_peered":"2024-02-05T07:27:40.849989+0000","last_clean":"2024-02-05T07:25:58.264786+0000","last_became_active":"2024-02-05T07:27:16.798515+0000","last_became_peered":"2024-02-05T07:27:16.798515+0000","last_unstale":"2024-02-05T07:27:40.849989+0000","last_undegraded":"2024-02-05T07:27:16.718511+0000","last_fullsized":"2024-02-05T07:27:16.682396+0000","mapping_epoch":263,"log_start":"232'3700","ondisk_log_start":"232'3700","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5699,"log_dups_size":0,"ondisk_log_size":5699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207585280,"num_objects":3167,"num_object_clones":0,"num_object_copies":9501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":117,"num_objects_misplaced":3167,"num_objects_unfound":0,"num_objects_dirty":3167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5345,"num_write_kb":272416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4687,"num_bytes_recovered":307167232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":3167}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":446,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.271976+0000","last_change":"2024-02-05T07:27:03.363357+0000","last_active":"2024-02-05T07:27:40.271976+0000","last_peered":"2024-02-05T07:27:40.271976+0000","last_clean":"2024-02-05T07:27:40.271976+0000","last_became_active":"2024-02-05T07:27:03.363025+0000","last_became_peered":"2024-02-05T07:27:03.363025+0000","last_unstale":"2024-02-05T07:27:40.271976+0000","last_undegraded":"2024-02-05T07:27:40.271976+0000","last_fullsized":"2024-02-05T07:27:40.271976+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:35.850761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:35.850761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"232'17","reported_seq":162,"reported_epoch":276,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:27:39.268333+0000","last_change":"2024-02-05T07:25:59.239638+0000","last_active":"2024-02-05T07:27:39.268333+0000","last_peered":"2024-02-05T07:27:39.268333+0000","last_clean":"2024-02-05T07:18:56.805155+0000","last_became_active":"2024-02-05T07:25:59.239638+0000","last_became_peered":"2024-02-05T07:25:59.239638+0000","last_unstale":"2024-02-05T07:27:39.268333+0000","last_undegraded":"2024-02-05T07:25:59.221950+0000","last_fullsized":"2024-02-05T07:25:59.221844+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-02-05T07:25:46.826417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:46.826417+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":35,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.277536+0000","last_change":"2024-02-05T07:27:17.711026+0000","last_active":"2024-02-05T07:27:40.277536+0000","last_peered":"2024-02-05T07:27:40.277536+0000","last_clean":"2024-02-05T07:27:40.277536+0000","last_became_active":"2024-02-05T07:27:17.710544+0000","last_became_peered":"2024-02-05T07:27:17.710544+0000","last_unstale":"2024-02-05T07:27:40.277536+0000","last_undegraded":"2024-02-05T07:27:40.277536+0000","last_fullsized":"2024-02-05T07:27:40.277536+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:45.761597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:45.761597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":726,"reported_epoch":277,"state":"active+undersized","last_fresh":"2024-02-05T07:27:40.270786+0000","last_change":"2024-02-05T07:25:59.226600+0000","last_active":"2024-02-05T07:27:40.270786+0000","last_peered":"2024-02-05T07:27:40.270786+0000","last_clean":"2024-02-05T07:25:57.239270+0000","last_became_active":"2024-02-05T07:25:59.226600+0000","last_became_peered":"2024-02-05T07:25:59.226600+0000","last_unstale":"2024-02-05T07:27:40.270786+0000","last_undegraded":"2024-02-05T07:27:40.270786+0000","last_fullsized":"2024-02-05T07:25:59.224267+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:25:52.772049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:25:52.772049+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":37,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.279103+0000","last_change":"2024-02-05T07:27:16.751712+0000","last_active":"2024-02-05T07:27:40.279103+0000","last_peered":"2024-02-05T07:27:40.279103+0000","last_clean":"2024-02-05T07:27:40.279103+0000","last_became_active":"2024-02-05T07:27:16.751341+0000","last_became_peered":"2024-02-05T07:27:16.751341+0000","last_unstale":"2024-02-05T07:27:40.279103+0000","last_undegraded":"2024-02-05T07:27:40.279103+0000","last_fullsized":"2024-02-05T07:27:40.279103+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:36.753098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.753098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:48.689757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":730,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.271203+0000","last_change":"2024-02-05T07:26:19.993252+0000","last_active":"2024-02-05T07:27:40.271203+0000","last_peered":"2024-02-05T07:27:40.271203+0000","last_clean":"2024-02-05T07:27:40.271203+0000","last_became_active":"2024-02-05T07:25:57.252020+0000","last_became_peered":"2024-02-05T07:25:57.252020+0000","last_unstale":"2024-02-05T07:27:40.271203+0000","last_undegraded":"2024-02-05T07:27:40.271203+0000","last_fullsized":"2024-02-05T07:27:40.271203+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:26:19.993124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:26:19.993124+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"277'10606","reported_seq":10757,"reported_epoch":277,"state":"active+remapped+backfill_wait","last_fresh":"2024-02-05T07:27:40.447785+0000","last_change":"2024-02-05T07:27:17.893741+0000","last_active":"2024-02-05T07:27:40.447785+0000","last_peered":"2024-02-05T07:27:40.447785+0000","last_clean":"2024-02-05T07:27:15.631238+0000","last_became_active":"2024-02-05T07:27:17.882039+0000","last_became_peered":"2024-02-05T07:27:17.882039+0000","last_unstale":"2024-02-05T07:27:40.447785+0000","last_undegraded":"2024-02-05T07:27:40.447785+0000","last_fullsized":"2024-02-05T07:27:40.447785+0000","mapping_epoch":264,"log_start":"242'6760","ondisk_log_start":"242'6760","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":3846,"log_dups_size":0,"ondisk_log_size":3846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407764992,"num_objects":6222,"num_object_clones":0,"num_object_copies":18666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6222,"num_objects_unfound":0,"num_objects_dirty":6222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10606,"num_write_kb":538496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":6222}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":454,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.282002+0000","last_change":"2024-02-05T07:27:03.358169+0000","last_active":"2024-02-05T07:27:40.282002+0000","last_peered":"2024-02-05T07:27:40.282002+0000","last_clean":"2024-02-05T07:27:40.282002+0000","last_became_active":"2024-02-05T07:27:03.357849+0000","last_became_peered":"2024-02-05T07:27:03.357849+0000","last_unstale":"2024-02-05T07:27:40.282002+0000","last_undegraded":"2024-02-05T07:27:40.282002+0000","last_fullsized":"2024-02-05T07:27:40.282002+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:42.863654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.863654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.900590+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"96'18","reported_seq":503,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.274699+0000","last_change":"2024-02-05T07:27:03.383286+0000","last_active":"2024-02-05T07:27:40.274699+0000","last_peered":"2024-02-05T07:27:40.274699+0000","last_clean":"2024-02-05T07:27:40.274699+0000","last_became_active":"2024-02-05T07:27:03.360661+0000","last_became_peered":"2024-02-05T07:27:03.360661+0000","last_unstale":"2024-02-05T07:27:40.274699+0000","last_undegraded":"2024-02-05T07:27:40.274699+0000","last_fullsized":"2024-02-05T07:27:40.274699+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'18","last_scrub_stamp":"2024-02-05T07:26:24.578301+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.578301+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:41.169040+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":485,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.281687+0000","last_change":"2024-02-05T07:27:16.729211+0000","last_active":"2024-02-05T07:27:40.281687+0000","last_peered":"2024-02-05T07:27:40.281687+0000","last_clean":"2024-02-05T07:27:40.281687+0000","last_became_active":"2024-02-05T07:27:16.728869+0000","last_became_peered":"2024-02-05T07:27:16.728869+0000","last_unstale":"2024-02-05T07:27:40.281687+0000","last_undegraded":"2024-02-05T07:27:40.281687+0000","last_fullsized":"2024-02-05T07:27:40.281687+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:27:02.065505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:27:02.065505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:28:08.541547+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":466,"reported_epoch":277,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:27:40.273056+0000","last_change":"2024-02-05T07:25:59.223287+0000","last_active":"2024-02-05T07:27:40.273056+0000","last_peered":"2024-02-05T07:27:40.273056+0000","last_clean":"2024-02-05T07:25:57.238772+0000","last_became_active":"2024-02-05T07:25:59.223287+0000","last_became_peered":"2024-02-05T07:25:59.223287+0000","last_unstale":"2024-02-05T07:27:40.273056+0000","last_undegraded":"2024-02-05T07:25:59.217360+0000","last_fullsized":"2024-02-05T07:25:59.217254+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:25:47.537014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.537014+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":450,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.272248+0000","last_change":"2024-02-05T07:27:17.702961+0000","last_active":"2024-02-05T07:27:40.272248+0000","last_peered":"2024-02-05T07:27:40.272248+0000","last_clean":"2024-02-05T07:27:40.272248+0000","last_became_active":"2024-02-05T07:27:17.702518+0000","last_became_peered":"2024-02-05T07:27:17.702518+0000","last_unstale":"2024-02-05T07:27:40.272248+0000","last_undegraded":"2024-02-05T07:27:40.272248+0000","last_fullsized":"2024-02-05T07:27:40.272248+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:38.531265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:21:38.531265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,2],"avail_no_missing":["7","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":426,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.281364+0000","last_change":"2024-02-05T07:27:06.396507+0000","last_active":"2024-02-05T07:27:40.281364+0000","last_peered":"2024-02-05T07:27:40.281364+0000","last_clean":"2024-02-05T07:27:40.281364+0000","last_became_active":"2024-02-05T07:27:06.396163+0000","last_became_peered":"2024-02-05T07:27:06.396163+0000","last_unstale":"2024-02-05T07:27:40.281364+0000","last_undegraded":"2024-02-05T07:27:40.281364+0000","last_fullsized":"2024-02-05T07:27:40.281364+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:26:19.017521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:26:19.017521+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.900720+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"277'9297","reported_seq":9519,"reported_epoch":277,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:27:40.270675+0000","last_change":"2024-02-05T07:27:17.907290+0000","last_active":"2024-02-05T07:27:40.270675+0000","last_peered":"2024-02-05T07:27:40.270675+0000","last_clean":"2024-02-05T07:27:02.304967+0000","last_became_active":"2024-02-05T07:27:16.964574+0000","last_became_peered":"2024-02-05T07:27:16.964574+0000","last_unstale":"2024-02-05T07:27:40.270675+0000","last_undegraded":"2024-02-05T07:27:40.270675+0000","last_fullsized":"2024-02-05T07:27:40.270675+0000","mapping_epoch":263,"log_start":"232'3700","ondisk_log_start":"232'3700","created":209,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5597,"log_dups_size":0,"ondisk_log_size":5597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207650816,"num_objects":3168,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3158,"num_objects_unfound":0,"num_objects_dirty":3168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5262,"num_write_kb":269792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[7,0,5],"avail_no_missing":["7","0","2","5"],"object_location_counts":[{"shards":"0,5,7","objects":3168}],"blocked_by":[],"up_primary":3,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":201,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.274254+0000","last_change":"2024-02-05T07:25:30.477223+0000","last_active":"2024-02-05T07:27:40.274254+0000","last_peered":"2024-02-05T07:27:40.274254+0000","last_clean":"2024-02-05T07:27:40.274254+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:27:40.274254+0000","last_undegraded":"2024-02-05T07:27:40.274254+0000","last_fullsized":"2024-02-05T07:27:40.274254+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:30.477135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:30.477135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"266'32","reported_seq":281,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.282455+0000","last_change":"2024-02-05T07:27:17.023380+0000","last_active":"2024-02-05T07:27:40.282455+0000","last_peered":"2024-02-05T07:27:40.282455+0000","last_clean":"2024-02-05T07:27:40.282455+0000","last_became_active":"2024-02-05T07:27:16.708475+0000","last_became_peered":"2024-02-05T07:27:16.708475+0000","last_unstale":"2024-02-05T07:27:40.282455+0000","last_undegraded":"2024-02-05T07:27:40.282455+0000","last_fullsized":"2024-02-05T07:27:40.282455+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-02-05T07:23:27.310935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:27.310935+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.900377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":457,"reported_epoch":276,"state":"active+undersized","last_fresh":"2024-02-05T07:27:39.268044+0000","last_change":"2024-02-05T07:25:59.229812+0000","last_active":"2024-02-05T07:27:39.268044+0000","last_peered":"2024-02-05T07:27:39.268044+0000","last_clean":"2024-02-05T07:25:57.231952+0000","last_became_active":"2024-02-05T07:25:59.229812+0000","last_became_peered":"2024-02-05T07:25:59.229812+0000","last_unstale":"2024-02-05T07:27:39.268044+0000","last_undegraded":"2024-02-05T07:27:39.268044+0000","last_fullsized":"2024-02-05T07:25:59.217783+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:53.185503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:53.185503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":366,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.272701+0000","last_change":"2024-02-05T07:27:04.466722+0000","last_active":"2024-02-05T07:27:40.272701+0000","last_peered":"2024-02-05T07:27:40.272701+0000","last_clean":"2024-02-05T07:27:40.272701+0000","last_became_active":"2024-02-05T07:27:04.377061+0000","last_became_peered":"2024-02-05T07:27:04.377061+0000","last_unstale":"2024-02-05T07:27:40.272701+0000","last_undegraded":"2024-02-05T07:27:40.272701+0000","last_fullsized":"2024-02-05T07:27:40.272701+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:23:29.340876+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:20:38.993688+0000","last_clean_scrub_stamp":"2024-02-05T07:23:29.340876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0,5],"avail_no_missing":["1","0","5"],"object_location_counts":[{"shards":"0,1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":37,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.278534+0000","last_change":"2024-02-05T07:27:16.715244+0000","last_active":"2024-02-05T07:27:40.278534+0000","last_peered":"2024-02-05T07:27:40.278534+0000","last_clean":"2024-02-05T07:27:40.278534+0000","last_became_active":"2024-02-05T07:27:16.714924+0000","last_became_peered":"2024-02-05T07:27:16.714924+0000","last_unstale":"2024-02-05T07:27:40.278534+0000","last_undegraded":"2024-02-05T07:27:40.278534+0000","last_fullsized":"2024-02-05T07:27:40.278534+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:05.412661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:05.412661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.7","version":"277'9545","reported_seq":15045,"reported_epoch":277,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-05T07:27:40.449316+0000","last_change":"2024-02-05T07:27:17.907101+0000","last_active":"2024-02-05T07:27:40.449316+0000","last_peered":"2024-02-05T07:27:40.449316+0000","last_clean":"2024-02-05T07:25:58.271170+0000","last_became_active":"2024-02-05T07:27:17.879953+0000","last_became_peered":"2024-02-05T07:27:17.879953+0000","last_unstale":"2024-02-05T07:27:40.449316+0000","last_undegraded":"2024-02-05T07:27:17.720733+0000","last_fullsized":"2024-02-05T07:27:17.709191+0000","mapping_epoch":264,"log_start":"232'4000","ondisk_log_start":"232'4000","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5545,"log_dups_size":0,"ondisk_log_size":5545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211058688,"num_objects":3220,"num_object_clones":0,"num_object_copies":9660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":751,"num_objects_misplaced":3220,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5440,"num_write_kb":277152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2670,"num_bytes_recovered":174981120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":3220}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":235,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.274695+0000","last_change":"2024-02-05T07:25:44.515426+0000","last_active":"2024-02-05T07:27:40.274695+0000","last_peered":"2024-02-05T07:27:40.274695+0000","last_clean":"2024-02-05T07:27:40.274695+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:27:40.274695+0000","last_undegraded":"2024-02-05T07:27:40.274695+0000","last_fullsized":"2024-02-05T07:27:40.274695+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:25:44.515297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:44.515297+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":35,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.278484+0000","last_change":"2024-02-05T07:27:17.706484+0000","last_active":"2024-02-05T07:27:40.278484+0000","last_peered":"2024-02-05T07:27:40.278484+0000","last_clean":"2024-02-05T07:27:40.278484+0000","last_became_active":"2024-02-05T07:27:17.705452+0000","last_became_peered":"2024-02-05T07:27:17.705452+0000","last_unstale":"2024-02-05T07:27:40.278484+0000","last_undegraded":"2024-02-05T07:27:40.278484+0000","last_fullsized":"2024-02-05T07:27:40.278484+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:38.539777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:38.539777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":460,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.282106+0000","last_change":"2024-02-05T07:27:16.708505+0000","last_active":"2024-02-05T07:27:40.282106+0000","last_peered":"2024-02-05T07:27:40.282106+0000","last_clean":"2024-02-05T07:27:40.282106+0000","last_became_active":"2024-02-05T07:27:16.708205+0000","last_became_peered":"2024-02-05T07:27:16.708205+0000","last_unstale":"2024-02-05T07:27:40.282106+0000","last_undegraded":"2024-02-05T07:27:40.282106+0000","last_fullsized":"2024-02-05T07:27:40.282106+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:43.842850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:43.842850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.900620+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":201,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.270785+0000","last_change":"2024-02-05T07:26:30.076326+0000","last_active":"2024-02-05T07:27:40.270785+0000","last_peered":"2024-02-05T07:27:40.270785+0000","last_clean":"2024-02-05T07:27:40.270785+0000","last_became_active":"2024-02-05T07:24:34.235899+0000","last_became_peered":"2024-02-05T07:24:34.235899+0000","last_unstale":"2024-02-05T07:27:40.270785+0000","last_undegraded":"2024-02-05T07:27:40.270785+0000","last_fullsized":"2024-02-05T07:27:40.270785+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:30.076226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:30.076226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"277'9465","reported_seq":9624,"reported_epoch":277,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:27:40.271679+0000","last_change":"2024-02-05T07:26:54.210583+0000","last_active":"2024-02-05T07:27:40.271679+0000","last_peered":"2024-02-05T07:27:40.271679+0000","last_clean":"2024-02-05T07:25:58.252183+0000","last_became_active":"2024-02-05T07:26:54.210583+0000","last_became_peered":"2024-02-05T07:26:54.210583+0000","last_unstale":"2024-02-05T07:27:40.271679+0000","last_undegraded":"2024-02-05T07:26:54.199271+0000","last_fullsized":"2024-02-05T07:26:54.199144+0000","mapping_epoch":231,"log_start":"230'3500","ondisk_log_start":"230'3500","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5965,"log_dups_size":0,"ondisk_log_size":5965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206798848,"num_objects":3155,"num_object_clones":0,"num_object_copies":9465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":273760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":32768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":3155}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":149,"reported_epoch":277,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:27:40.272866+0000","last_change":"2024-02-05T07:25:59.228711+0000","last_active":"2024-02-05T07:27:40.272866+0000","last_peered":"2024-02-05T07:27:40.272866+0000","last_clean":"2024-02-05T07:18:56.828300+0000","last_became_active":"2024-02-05T07:25:59.228711+0000","last_became_peered":"2024-02-05T07:25:59.228711+0000","last_unstale":"2024-02-05T07:27:40.272866+0000","last_undegraded":"2024-02-05T07:25:59.219065+0000","last_fullsized":"2024-02-05T07:25:59.218949+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:24:54.413486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:54.413486+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":460,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.272064+0000","last_change":"2024-02-05T07:27:17.703089+0000","last_active":"2024-02-05T07:27:40.272064+0000","last_peered":"2024-02-05T07:27:40.272064+0000","last_clean":"2024-02-05T07:27:40.272064+0000","last_became_active":"2024-02-05T07:27:17.702284+0000","last_became_peered":"2024-02-05T07:27:17.702284+0000","last_unstale":"2024-02-05T07:27:40.272064+0000","last_undegraded":"2024-02-05T07:27:40.272064+0000","last_fullsized":"2024-02-05T07:27:40.272064+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:41.444193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:21:41.444193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,4],"avail_no_missing":["7","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":453,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.282041+0000","last_change":"2024-02-05T07:27:03.356403+0000","last_active":"2024-02-05T07:27:40.282041+0000","last_peered":"2024-02-05T07:27:40.282041+0000","last_clean":"2024-02-05T07:27:40.282041+0000","last_became_active":"2024-02-05T07:27:03.355957+0000","last_became_peered":"2024-02-05T07:27:03.355957+0000","last_unstale":"2024-02-05T07:27:40.282041+0000","last_undegraded":"2024-02-05T07:27:40.282041+0000","last_fullsized":"2024-02-05T07:27:40.282041+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:22.018572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:22.018572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.900751+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":451,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.270600+0000","last_change":"2024-02-05T07:27:16.710549+0000","last_active":"2024-02-05T07:27:40.270600+0000","last_peered":"2024-02-05T07:27:40.270600+0000","last_clean":"2024-02-05T07:27:40.270600+0000","last_became_active":"2024-02-05T07:27:16.710267+0000","last_became_peered":"2024-02-05T07:27:16.710267+0000","last_unstale":"2024-02-05T07:27:40.270600+0000","last_undegraded":"2024-02-05T07:27:40.270600+0000","last_fullsized":"2024-02-05T07:27:40.270600+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:48.036033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:48.036033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"276'9570","reported_seq":9722,"reported_epoch":276,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:27:40.035281+0000","last_change":"2024-02-05T07:26:54.208081+0000","last_active":"2024-02-05T07:27:40.035281+0000","last_peered":"2024-02-05T07:27:40.035281+0000","last_clean":"2024-02-05T07:25:58.207147+0000","last_became_active":"2024-02-05T07:26:54.208081+0000","last_became_peered":"2024-02-05T07:26:54.208081+0000","last_unstale":"2024-02-05T07:27:40.035281+0000","last_undegraded":"2024-02-05T07:26:54.201147+0000","last_fullsized":"2024-02-05T07:26:54.197886+0000","mapping_epoch":231,"log_start":"232'4129","ondisk_log_start":"232'4129","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:43.830128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207224832,"num_objects":3162,"num_object_clones":0,"num_object_copies":9486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5442,"num_write_kb":275328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":3162}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":456,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.280707+0000","last_change":"2024-02-05T07:26:26.018740+0000","last_active":"2024-02-05T07:27:40.280707+0000","last_peered":"2024-02-05T07:27:40.280707+0000","last_clean":"2024-02-05T07:27:40.280707+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:27:40.280707+0000","last_undegraded":"2024-02-05T07:27:40.280707+0000","last_fullsized":"2024-02-05T07:27:40.280707+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:26.018629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:26.018629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:49.532442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":533,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-02-05T07:27:39.270041+0000","last_change":"2024-02-05T07:27:17.019007+0000","last_active":"2024-02-05T07:27:39.270041+0000","last_peered":"2024-02-05T07:27:39.270041+0000","last_clean":"2024-02-05T07:27:39.270041+0000","last_became_active":"2024-02-05T07:27:16.706871+0000","last_became_peered":"2024-02-05T07:27:16.706871+0000","last_unstale":"2024-02-05T07:27:39.270041+0000","last_undegraded":"2024-02-05T07:27:39.270041+0000","last_fullsized":"2024-02-05T07:27:39.270041+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:25:32.094000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:32.094000+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":455,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.271957+0000","last_change":"2024-02-05T07:26:24.102911+0000","last_active":"2024-02-05T07:27:40.271957+0000","last_peered":"2024-02-05T07:27:40.271957+0000","last_clean":"2024-02-05T07:27:40.271957+0000","last_became_active":"2024-02-05T07:18:49.435621+0000","last_became_peered":"2024-02-05T07:18:49.435621+0000","last_unstale":"2024-02-05T07:27:40.271957+0000","last_undegraded":"2024-02-05T07:27:40.271957+0000","last_fullsized":"2024-02-05T07:27:40.271957+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:24.102597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.102597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:53.808064+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":37,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.277567+0000","last_change":"2024-02-05T07:27:16.765294+0000","last_active":"2024-02-05T07:27:40.277567+0000","last_peered":"2024-02-05T07:27:40.277567+0000","last_clean":"2024-02-05T07:27:40.277567+0000","last_became_active":"2024-02-05T07:27:16.764767+0000","last_became_peered":"2024-02-05T07:27:16.764767+0000","last_unstale":"2024-02-05T07:27:40.277567+0000","last_undegraded":"2024-02-05T07:27:40.277567+0000","last_fullsized":"2024-02-05T07:27:40.277567+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:42.108043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:42.108043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"276'9532","reported_seq":9691,"reported_epoch":276,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.189201+0000","last_change":"2024-02-05T07:27:04.352191+0000","last_active":"2024-02-05T07:27:40.189201+0000","last_peered":"2024-02-05T07:27:40.189201+0000","last_clean":"2024-02-05T07:27:40.189201+0000","last_became_active":"2024-02-05T07:27:04.348830+0000","last_became_peered":"2024-02-05T07:27:04.348830+0000","last_unstale":"2024-02-05T07:27:40.189201+0000","last_undegraded":"2024-02-05T07:27:40.189201+0000","last_fullsized":"2024-02-05T07:27:40.189201+0000","mapping_epoch":256,"log_start":"232'4145","ondisk_log_start":"232'4145","created":209,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5387,"log_dups_size":0,"ondisk_log_size":5387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:41.835274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207519744,"num_objects":3166,"num_object_clones":0,"num_object_copies":9498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3166,"num_objects_unfound":0,"num_objects_dirty":3166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5418,"num_write_kb":274720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[{"shards":"2,4,5","objects":3166}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"232'10","reported_seq":560,"reported_epoch":277,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:27:40.271209+0000","last_change":"2024-02-05T07:25:59.233152+0000","last_active":"2024-02-05T07:27:40.271209+0000","last_peered":"2024-02-05T07:27:40.271209+0000","last_clean":"2024-02-05T07:25:57.241390+0000","last_became_active":"2024-02-05T07:25:59.233152+0000","last_became_peered":"2024-02-05T07:25:59.233152+0000","last_unstale":"2024-02-05T07:27:40.271209+0000","last_undegraded":"2024-02-05T07:25:59.221022+0000","last_fullsized":"2024-02-05T07:25:59.220901+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-02-05T07:21:26.505161+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:21:26.505161+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":113,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.270871+0000","last_change":"2024-02-05T07:27:04.393011+0000","last_active":"2024-02-05T07:27:40.270871+0000","last_peered":"2024-02-05T07:27:40.270871+0000","last_clean":"2024-02-05T07:27:40.270871+0000","last_became_active":"2024-02-05T07:27:04.360756+0000","last_became_peered":"2024-02-05T07:27:04.360756+0000","last_unstale":"2024-02-05T07:27:40.270871+0000","last_undegraded":"2024-02-05T07:27:40.270871+0000","last_fullsized":"2024-02-05T07:27:40.270871+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:21:32.875346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:21:32.875346+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,5],"avail_no_missing":["7","4","5"],"object_location_counts":[{"shards":"4,5,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":467,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.272065+0000","last_change":"2024-02-05T07:26:37.022248+0000","last_active":"2024-02-05T07:27:40.272065+0000","last_peered":"2024-02-05T07:27:40.272065+0000","last_clean":"2024-02-05T07:27:40.272065+0000","last_became_active":"2024-02-05T07:18:49.437663+0000","last_became_peered":"2024-02-05T07:18:49.437663+0000","last_unstale":"2024-02-05T07:27:40.272065+0000","last_undegraded":"2024-02-05T07:27:40.272065+0000","last_fullsized":"2024-02-05T07:27:40.272065+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:26:37.022129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:37.022129+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:46.252824+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":52,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.281360+0000","last_change":"2024-02-05T07:27:04.349193+0000","last_active":"2024-02-05T07:27:40.281360+0000","last_peered":"2024-02-05T07:27:40.281360+0000","last_clean":"2024-02-05T07:27:40.281360+0000","last_became_active":"2024-02-05T07:27:04.348320+0000","last_became_peered":"2024-02-05T07:27:04.348320+0000","last_unstale":"2024-02-05T07:27:40.281360+0000","last_undegraded":"2024-02-05T07:27:40.281360+0000","last_fullsized":"2024-02-05T07:27:40.281360+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:34.561259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:37.032056+0000","last_clean_scrub_stamp":"2024-02-05T07:25:34.561259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.900488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0,5],"avail_no_missing":["2","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":45,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.278928+0000","last_change":"2024-02-05T07:27:16.969586+0000","last_active":"2024-02-05T07:27:40.278928+0000","last_peered":"2024-02-05T07:27:40.278928+0000","last_clean":"2024-02-05T07:27:40.278928+0000","last_became_active":"2024-02-05T07:27:16.750507+0000","last_became_peered":"2024-02-05T07:27:16.750507+0000","last_unstale":"2024-02-05T07:27:40.278928+0000","last_undegraded":"2024-02-05T07:27:40.278928+0000","last_fullsized":"2024-02-05T07:27:40.278928+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-02-05T07:26:36.068507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.068507+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:46.177671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'1","reported_seq":39,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.277025+0000","last_change":"2024-02-05T07:27:17.716237+0000","last_active":"2024-02-05T07:27:40.277025+0000","last_peered":"2024-02-05T07:27:40.277025+0000","last_clean":"2024-02-05T07:27:40.277025+0000","last_became_active":"2024-02-05T07:27:17.708379+0000","last_became_peered":"2024-02-05T07:27:17.708379+0000","last_unstale":"2024-02-05T07:27:40.277025+0000","last_undegraded":"2024-02-05T07:27:40.277025+0000","last_fullsized":"2024-02-05T07:27:40.277025+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:24:49.359534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.359534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":35,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.277817+0000","last_change":"2024-02-05T07:27:17.711699+0000","last_active":"2024-02-05T07:27:40.277817+0000","last_peered":"2024-02-05T07:27:40.277817+0000","last_clean":"2024-02-05T07:27:40.277817+0000","last_became_active":"2024-02-05T07:27:17.711120+0000","last_became_peered":"2024-02-05T07:27:17.711120+0000","last_unstale":"2024-02-05T07:27:40.277817+0000","last_undegraded":"2024-02-05T07:27:40.277817+0000","last_fullsized":"2024-02-05T07:27:40.277817+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:28.567844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:28.567844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":438,"reported_epoch":277,"state":"active+undersized","last_fresh":"2024-02-05T07:27:40.271580+0000","last_change":"2024-02-05T07:25:59.235975+0000","last_active":"2024-02-05T07:27:40.271580+0000","last_peered":"2024-02-05T07:27:40.271580+0000","last_clean":"2024-02-05T07:25:57.239964+0000","last_became_active":"2024-02-05T07:25:59.235975+0000","last_became_peered":"2024-02-05T07:25:59.235975+0000","last_unstale":"2024-02-05T07:27:40.271580+0000","last_undegraded":"2024-02-05T07:27:40.271580+0000","last_fullsized":"2024-02-05T07:25:59.226429+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:21:44.435826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":47,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.278401+0000","last_change":"2024-02-05T07:27:17.009879+0000","last_active":"2024-02-05T07:27:40.278401+0000","last_peered":"2024-02-05T07:27:40.278401+0000","last_clean":"2024-02-05T07:27:40.278401+0000","last_became_active":"2024-02-05T07:27:16.750671+0000","last_became_peered":"2024-02-05T07:27:16.750671+0000","last_unstale":"2024-02-05T07:27:40.278401+0000","last_undegraded":"2024-02-05T07:27:40.278401+0000","last_fullsized":"2024-02-05T07:27:40.278401+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:26:28.037945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:28.037945+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:44.952559+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":458,"reported_epoch":277,"state":"active+undersized","last_fresh":"2024-02-05T07:27:40.271041+0000","last_change":"2024-02-05T07:25:59.234625+0000","last_active":"2024-02-05T07:27:40.271041+0000","last_peered":"2024-02-05T07:27:40.271041+0000","last_clean":"2024-02-05T07:25:57.241166+0000","last_became_active":"2024-02-05T07:25:59.234625+0000","last_became_peered":"2024-02-05T07:25:59.234625+0000","last_unstale":"2024-02-05T07:27:40.271041+0000","last_undegraded":"2024-02-05T07:27:40.271041+0000","last_fullsized":"2024-02-05T07:25:59.221239+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:51.811993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:25:51.811993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":37,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.279066+0000","last_change":"2024-02-05T07:27:16.750873+0000","last_active":"2024-02-05T07:27:40.279066+0000","last_peered":"2024-02-05T07:27:40.279066+0000","last_clean":"2024-02-05T07:27:40.279066+0000","last_became_active":"2024-02-05T07:27:16.749995+0000","last_became_peered":"2024-02-05T07:27:16.749995+0000","last_unstale":"2024-02-05T07:27:40.279066+0000","last_undegraded":"2024-02-05T07:27:40.279066+0000","last_fullsized":"2024-02-05T07:27:40.279066+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:59.450191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:59.450191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:28:00.341423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":443,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.282601+0000","last_change":"2024-02-05T07:25:47.862699+0000","last_active":"2024-02-05T07:27:40.282601+0000","last_peered":"2024-02-05T07:27:40.282601+0000","last_clean":"2024-02-05T07:27:40.282601+0000","last_became_active":"2024-02-05T07:18:55.469044+0000","last_became_peered":"2024-02-05T07:18:55.469044+0000","last_unstale":"2024-02-05T07:27:40.282601+0000","last_undegraded":"2024-02-05T07:27:40.282601+0000","last_fullsized":"2024-02-05T07:27:40.282601+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.862362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.900685+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"275'9390","reported_seq":16781,"reported_epoch":276,"state":"remapped+peering","last_fresh":"2024-02-05T07:27:39.241175+0000","last_change":"2024-02-05T07:27:40.244876+0000","last_active":"2024-02-05T07:27:40.244876+0000","last_peered":"2024-02-05T07:27:39.241175+0000","last_clean":"2024-02-05T07:25:58.264786+0000","last_became_active":"2024-02-05T07:27:03.356080+0000","last_became_peered":"2024-02-05T07:27:03.356080+0000","last_unstale":"2024-02-05T07:27:39.241175+0000","last_undegraded":"2024-02-05T07:27:39.241175+0000","last_fullsized":"2024-02-05T07:27:03.336907+0000","mapping_epoch":276,"log_start":"232'3700","ondisk_log_start":"232'3700","created":248,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5690,"log_dups_size":0,"ondisk_log_size":5690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205553664,"num_objects":3137,"num_object_clones":0,"num_object_copies":9411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5337,"num_write_kb":271136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3609,"num_bytes_recovered":236519424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"222'13","reported_seq":445,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.272115+0000","last_change":"2024-02-05T07:27:17.719739+0000","last_active":"2024-02-05T07:27:40.272115+0000","last_peered":"2024-02-05T07:27:40.272115+0000","last_clean":"2024-02-05T07:27:40.272115+0000","last_became_active":"2024-02-05T07:27:17.701850+0000","last_became_peered":"2024-02-05T07:27:17.701850+0000","last_unstale":"2024-02-05T07:27:40.272115+0000","last_undegraded":"2024-02-05T07:27:40.272115+0000","last_fullsized":"2024-02-05T07:27:40.272115+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'13","last_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:25:46.891183+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,0],"avail_no_missing":["7","0","3"],"object_location_counts":[{"shards":"0,3,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":414,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.270969+0000","last_change":"2024-02-05T07:27:16.724442+0000","last_active":"2024-02-05T07:27:40.270969+0000","last_peered":"2024-02-05T07:27:40.270969+0000","last_clean":"2024-02-05T07:27:40.270969+0000","last_became_active":"2024-02-05T07:27:16.724154+0000","last_became_peered":"2024-02-05T07:27:16.724154+0000","last_unstale":"2024-02-05T07:27:40.270969+0000","last_undegraded":"2024-02-05T07:27:40.270969+0000","last_fullsized":"2024-02-05T07:27:40.270969+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:49.035385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.035385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"266'10","reported_seq":46,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.278163+0000","last_change":"2024-02-05T07:27:17.013712+0000","last_active":"2024-02-05T07:27:40.278163+0000","last_peered":"2024-02-05T07:27:40.278163+0000","last_clean":"2024-02-05T07:27:40.278163+0000","last_became_active":"2024-02-05T07:27:16.816293+0000","last_became_peered":"2024-02-05T07:27:16.816293+0000","last_unstale":"2024-02-05T07:27:40.278163+0000","last_undegraded":"2024-02-05T07:27:40.278163+0000","last_fullsized":"2024-02-05T07:27:40.278163+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:26:35.091894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.091894+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:28:03.914982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":447,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-02-05T07:27:39.268493+0000","last_change":"2024-02-05T07:27:03.366653+0000","last_active":"2024-02-05T07:27:39.268493+0000","last_peered":"2024-02-05T07:27:39.268493+0000","last_clean":"2024-02-05T07:27:39.268493+0000","last_became_active":"2024-02-05T07:27:03.364334+0000","last_became_peered":"2024-02-05T07:27:03.364334+0000","last_unstale":"2024-02-05T07:27:39.268493+0000","last_undegraded":"2024-02-05T07:27:39.268493+0000","last_fullsized":"2024-02-05T07:27:39.268493+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:25.323384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:25.323384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:51.356397+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.18","version":"277'9283","reported_seq":19168,"reported_epoch":277,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T07:27:40.849570+0000","last_change":"2024-02-05T07:27:17.724991+0000","last_active":"2024-02-05T07:27:40.849570+0000","last_peered":"2024-02-05T07:27:40.849570+0000","last_clean":"2024-02-05T07:26:55.227542+0000","last_became_active":"2024-02-05T07:27:16.832467+0000","last_became_peered":"2024-02-05T07:27:16.832467+0000","last_unstale":"2024-02-05T07:27:40.849570+0000","last_undegraded":"2024-02-05T07:27:40.849570+0000","last_fullsized":"2024-02-05T07:27:40.849570+0000","mapping_epoch":263,"log_start":"232'3700","ondisk_log_start":"232'3700","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5583,"log_dups_size":0,"ondisk_log_size":5583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209223680,"num_objects":3193,"num_object_clones":0,"num_object_copies":9579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1015,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5249,"num_write_kb":270112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3753,"num_bytes_recovered":245891072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":3193}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"113'15","reported_seq":49,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.278388+0000","last_change":"2024-02-05T07:27:17.026183+0000","last_active":"2024-02-05T07:27:40.278388+0000","last_peered":"2024-02-05T07:27:40.278388+0000","last_clean":"2024-02-05T07:27:40.278388+0000","last_became_active":"2024-02-05T07:27:16.817057+0000","last_became_peered":"2024-02-05T07:27:16.817057+0000","last_unstale":"2024-02-05T07:27:40.278388+0000","last_undegraded":"2024-02-05T07:27:40.278388+0000","last_fullsized":"2024-02-05T07:27:40.278388+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'15","last_scrub_stamp":"2024-02-05T07:25:47.927943+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.927943+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":447,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.271023+0000","last_change":"2024-02-05T07:24:53.108878+0000","last_active":"2024-02-05T07:27:40.271023+0000","last_peered":"2024-02-05T07:27:40.271023+0000","last_clean":"2024-02-05T07:27:40.271023+0000","last_became_active":"2024-02-05T07:18:49.436131+0000","last_became_peered":"2024-02-05T07:18:49.436131+0000","last_unstale":"2024-02-05T07:27:40.271023+0000","last_undegraded":"2024-02-05T07:27:40.271023+0000","last_fullsized":"2024-02-05T07:27:40.271023+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:53.108801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:53.108801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"217'9","reported_seq":174,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.281608+0000","last_change":"2024-02-05T07:25:31.727799+0000","last_active":"2024-02-05T07:27:40.281608+0000","last_peered":"2024-02-05T07:27:40.281608+0000","last_clean":"2024-02-05T07:27:40.281608+0000","last_became_active":"2024-02-05T07:25:27.246616+0000","last_became_peered":"2024-02-05T07:25:27.246616+0000","last_unstale":"2024-02-05T07:27:40.281608+0000","last_undegraded":"2024-02-05T07:27:40.281608+0000","last_fullsized":"2024-02-05T07:27:40.281608+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9","last_scrub_stamp":"2024-02-05T07:25:31.727658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:31.727658+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.900559+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":402,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.271724+0000","last_change":"2024-02-05T07:27:16.727137+0000","last_active":"2024-02-05T07:27:40.271724+0000","last_peered":"2024-02-05T07:27:40.271724+0000","last_clean":"2024-02-05T07:27:40.271724+0000","last_became_active":"2024-02-05T07:27:16.726934+0000","last_became_peered":"2024-02-05T07:27:16.726934+0000","last_unstale":"2024-02-05T07:27:40.271724+0000","last_undegraded":"2024-02-05T07:27:40.271724+0000","last_fullsized":"2024-02-05T07:27:40.271724+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:27:01.069970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:27:01.069970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:28:11.318926+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":489,"reported_epoch":276,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:27:39.268381+0000","last_change":"2024-02-05T07:25:59.232617+0000","last_active":"2024-02-05T07:27:39.268381+0000","last_peered":"2024-02-05T07:27:39.268381+0000","last_clean":"2024-02-05T07:25:57.232059+0000","last_became_active":"2024-02-05T07:25:59.232617+0000","last_became_peered":"2024-02-05T07:25:59.232617+0000","last_unstale":"2024-02-05T07:27:39.268381+0000","last_undegraded":"2024-02-05T07:25:59.218091+0000","last_fullsized":"2024-02-05T07:25:59.217973+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:25:42.192387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.192387+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":448,"reported_epoch":276,"state":"active+undersized","last_fresh":"2024-02-05T07:27:39.268671+0000","last_change":"2024-02-05T07:25:59.236021+0000","last_active":"2024-02-05T07:27:39.268671+0000","last_peered":"2024-02-05T07:27:39.268671+0000","last_clean":"2024-02-05T07:25:57.230023+0000","last_became_active":"2024-02-05T07:25:59.236021+0000","last_became_peered":"2024-02-05T07:25:59.236021+0000","last_unstale":"2024-02-05T07:27:39.268671+0000","last_undegraded":"2024-02-05T07:27:39.268671+0000","last_fullsized":"2024-02-05T07:25:59.217540+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:23:23.435135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":464,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.273326+0000","last_change":"2024-02-05T07:27:17.718683+0000","last_active":"2024-02-05T07:27:40.273326+0000","last_peered":"2024-02-05T07:27:40.273326+0000","last_clean":"2024-02-05T07:27:40.273326+0000","last_became_active":"2024-02-05T07:27:17.716649+0000","last_became_peered":"2024-02-05T07:27:17.716649+0000","last_unstale":"2024-02-05T07:27:40.273326+0000","last_undegraded":"2024-02-05T07:27:40.273326+0000","last_fullsized":"2024-02-05T07:27:40.273326+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.555824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.555824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":35,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.278086+0000","last_change":"2024-02-05T07:27:17.708497+0000","last_active":"2024-02-05T07:27:40.278086+0000","last_peered":"2024-02-05T07:27:40.278086+0000","last_clean":"2024-02-05T07:27:40.278086+0000","last_became_active":"2024-02-05T07:27:17.705910+0000","last_became_peered":"2024-02-05T07:27:17.705910+0000","last_unstale":"2024-02-05T07:27:40.278086+0000","last_undegraded":"2024-02-05T07:27:40.278086+0000","last_fullsized":"2024-02-05T07:27:40.278086+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.120512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":62,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.272010+0000","last_change":"2024-02-05T07:27:04.404006+0000","last_active":"2024-02-05T07:27:40.272010+0000","last_peered":"2024-02-05T07:27:40.272010+0000","last_clean":"2024-02-05T07:27:40.272010+0000","last_became_active":"2024-02-05T07:27:04.369843+0000","last_became_peered":"2024-02-05T07:27:04.369843+0000","last_unstale":"2024-02-05T07:27:40.272010+0000","last_undegraded":"2024-02-05T07:27:40.272010+0000","last_fullsized":"2024-02-05T07:27:40.272010+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:25:50.880674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:50.880674+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":459,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.271962+0000","last_change":"2024-02-05T07:27:16.707941+0000","last_active":"2024-02-05T07:27:40.271962+0000","last_peered":"2024-02-05T07:27:40.271962+0000","last_clean":"2024-02-05T07:27:40.271962+0000","last_became_active":"2024-02-05T07:27:16.707627+0000","last_became_peered":"2024-02-05T07:27:16.707627+0000","last_unstale":"2024-02-05T07:27:40.271962+0000","last_undegraded":"2024-02-05T07:27:40.271962+0000","last_fullsized":"2024-02-05T07:27:40.271962+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:36.822835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:36.822835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":37,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.278802+0000","last_change":"2024-02-05T07:27:16.765064+0000","last_active":"2024-02-05T07:27:40.278802+0000","last_peered":"2024-02-05T07:27:40.278802+0000","last_clean":"2024-02-05T07:27:40.278802+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:27:40.278802+0000","last_undegraded":"2024-02-05T07:27:40.278802+0000","last_fullsized":"2024-02-05T07:27:40.278802+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:26:31.506641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:49.281752+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":206,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.272493+0000","last_change":"2024-02-05T07:27:04.371074+0000","last_active":"2024-02-05T07:27:40.272493+0000","last_peered":"2024-02-05T07:27:40.272493+0000","last_clean":"2024-02-05T07:27:40.272493+0000","last_became_active":"2024-02-05T07:27:04.370730+0000","last_became_peered":"2024-02-05T07:27:04.370730+0000","last_unstale":"2024-02-05T07:27:40.272493+0000","last_undegraded":"2024-02-05T07:27:40.272493+0000","last_fullsized":"2024-02-05T07:27:40.272493+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:37.616737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:26:37.616737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0,5],"avail_no_missing":["1","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":47,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.278260+0000","last_change":"2024-02-05T07:27:17.728435+0000","last_active":"2024-02-05T07:27:40.278260+0000","last_peered":"2024-02-05T07:27:40.278260+0000","last_clean":"2024-02-05T07:27:40.278260+0000","last_became_active":"2024-02-05T07:27:17.702741+0000","last_became_peered":"2024-02-05T07:27:17.702741+0000","last_unstale":"2024-02-05T07:27:40.278260+0000","last_undegraded":"2024-02-05T07:27:40.278260+0000","last_fullsized":"2024-02-05T07:27:40.278260+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:25:42.761130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.761130+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":60,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.271857+0000","last_change":"2024-02-05T07:27:16.707735+0000","last_active":"2024-02-05T07:27:40.271857+0000","last_peered":"2024-02-05T07:27:40.271857+0000","last_clean":"2024-02-05T07:27:40.271857+0000","last_became_active":"2024-02-05T07:27:16.707402+0000","last_became_peered":"2024-02-05T07:27:16.707402+0000","last_unstale":"2024-02-05T07:27:40.271857+0000","last_undegraded":"2024-02-05T07:27:40.271857+0000","last_fullsized":"2024-02-05T07:27:40.271857+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:16.644911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:16.644911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":450,"reported_epoch":277,"state":"active+undersized","last_fresh":"2024-02-05T07:27:40.271041+0000","last_change":"2024-02-05T07:25:59.235562+0000","last_active":"2024-02-05T07:27:40.271041+0000","last_peered":"2024-02-05T07:27:40.271041+0000","last_clean":"2024-02-05T07:25:57.239615+0000","last_became_active":"2024-02-05T07:25:59.235562+0000","last_became_peered":"2024-02-05T07:25:59.235562+0000","last_unstale":"2024-02-05T07:27:40.271041+0000","last_undegraded":"2024-02-05T07:27:40.271041+0000","last_fullsized":"2024-02-05T07:25:59.225564+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:34.862131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:05.673983+0000","last_clean_scrub_stamp":"2024-02-05T07:25:34.862131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":36,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.277371+0000","last_change":"2024-02-05T07:27:17.704836+0000","last_active":"2024-02-05T07:27:40.277371+0000","last_peered":"2024-02-05T07:27:40.277371+0000","last_clean":"2024-02-05T07:27:40.277371+0000","last_became_active":"2024-02-05T07:27:17.702372+0000","last_became_peered":"2024-02-05T07:27:17.702372+0000","last_unstale":"2024-02-05T07:27:40.277371+0000","last_undegraded":"2024-02-05T07:27:40.277371+0000","last_fullsized":"2024-02-05T07:27:40.277371+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:46.423760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:46.423760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,2],"avail_no_missing":["3","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":42,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.277984+0000","last_change":"2024-02-05T07:27:17.739497+0000","last_active":"2024-02-05T07:27:40.277984+0000","last_peered":"2024-02-05T07:27:40.277984+0000","last_clean":"2024-02-05T07:27:40.277984+0000","last_became_active":"2024-02-05T07:27:17.712538+0000","last_became_peered":"2024-02-05T07:27:17.712538+0000","last_unstale":"2024-02-05T07:27:40.277984+0000","last_undegraded":"2024-02-05T07:27:40.277984+0000","last_fullsized":"2024-02-05T07:27:40.277984+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-02-05T07:25:31.462787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:31.462787+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,4],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":35,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.278656+0000","last_change":"2024-02-05T07:27:17.878386+0000","last_active":"2024-02-05T07:27:40.278656+0000","last_peered":"2024-02-05T07:27:40.278656+0000","last_clean":"2024-02-05T07:27:40.278656+0000","last_became_active":"2024-02-05T07:27:17.877512+0000","last_became_peered":"2024-02-05T07:27:17.877512+0000","last_unstale":"2024-02-05T07:27:40.278656+0000","last_undegraded":"2024-02-05T07:27:40.278656+0000","last_fullsized":"2024-02-05T07:27:40.278656+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:17.202540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:17.202540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":97,"reported_epoch":276,"state":"active+undersized","last_fresh":"2024-02-05T07:27:39.269512+0000","last_change":"2024-02-05T07:25:59.229147+0000","last_active":"2024-02-05T07:27:39.269512+0000","last_peered":"2024-02-05T07:27:39.269512+0000","last_clean":"2024-02-05T07:18:52.436288+0000","last_became_active":"2024-02-05T07:25:59.229147+0000","last_became_peered":"2024-02-05T07:25:59.229147+0000","last_unstale":"2024-02-05T07:27:39.269512+0000","last_undegraded":"2024-02-05T07:27:39.269512+0000","last_fullsized":"2024-02-05T07:25:59.216730+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:32.554622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:25:32.554622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":443,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.271234+0000","last_change":"2024-02-05T07:26:27.014044+0000","last_active":"2024-02-05T07:27:40.271234+0000","last_peered":"2024-02-05T07:27:40.271234+0000","last_clean":"2024-02-05T07:27:40.271234+0000","last_became_active":"2024-02-05T07:18:55.468086+0000","last_became_peered":"2024-02-05T07:18:55.468086+0000","last_unstale":"2024-02-05T07:27:40.271234+0000","last_undegraded":"2024-02-05T07:27:40.271234+0000","last_fullsized":"2024-02-05T07:27:40.271234+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:27.013942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:27.013942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.13","version":"277'9220","reported_seq":13111,"reported_epoch":277,"state":"active+undersized+degraded+remapped","last_fresh":"2024-02-05T07:27:40.358538+0000","last_change":"2024-02-05T07:27:16.963237+0000","last_active":"2024-02-05T07:27:40.358538+0000","last_peered":"2024-02-05T07:27:40.358538+0000","last_clean":"2024-02-05T07:25:58.269991+0000","last_became_active":"2024-02-05T07:27:16.816332+0000","last_became_peered":"2024-02-05T07:27:16.816332+0000","last_unstale":"2024-02-05T07:27:40.358538+0000","last_undegraded":"2024-02-05T07:27:16.741544+0000","last_fullsized":"2024-02-05T07:27:16.680683+0000","mapping_epoch":263,"log_start":"232'3700","ondisk_log_start":"232'3700","created":248,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204242944,"num_objects":3117,"num_object_clones":0,"num_object_copies":9351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2170,"num_objects_misplaced":3117,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5253,"num_write_kb":267808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1238,"num_bytes_recovered":81133568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":3117}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":154,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.274803+0000","last_change":"2024-02-05T07:27:04.438517+0000","last_active":"2024-02-05T07:27:40.274803+0000","last_peered":"2024-02-05T07:27:40.274803+0000","last_clean":"2024-02-05T07:27:40.274803+0000","last_became_active":"2024-02-05T07:27:04.372649+0000","last_became_peered":"2024-02-05T07:27:04.372649+0000","last_unstale":"2024-02-05T07:27:40.274803+0000","last_undegraded":"2024-02-05T07:27:40.274803+0000","last_fullsized":"2024-02-05T07:27:40.274803+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:24:53.457167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:53.457167+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0,5],"avail_no_missing":["1","0","5"],"object_location_counts":[{"shards":"0,1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":37,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.277129+0000","last_change":"2024-02-05T07:27:16.831639+0000","last_active":"2024-02-05T07:27:40.277129+0000","last_peered":"2024-02-05T07:27:40.277129+0000","last_clean":"2024-02-05T07:27:40.277129+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:27:40.277129+0000","last_undegraded":"2024-02-05T07:27:40.277129+0000","last_fullsized":"2024-02-05T07:27:40.277129+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:54.528848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:54.528848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":201,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.272027+0000","last_change":"2024-02-05T07:25:30.943037+0000","last_active":"2024-02-05T07:27:40.272027+0000","last_peered":"2024-02-05T07:27:40.272027+0000","last_clean":"2024-02-05T07:27:40.272027+0000","last_became_active":"2024-02-05T07:24:34.240115+0000","last_became_peered":"2024-02-05T07:24:34.240115+0000","last_unstale":"2024-02-05T07:27:40.272027+0000","last_undegraded":"2024-02-05T07:27:40.272027+0000","last_fullsized":"2024-02-05T07:27:40.272027+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:30.942942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:30.942942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":458,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.274502+0000","last_change":"2024-02-05T07:27:03.360786+0000","last_active":"2024-02-05T07:27:40.274502+0000","last_peered":"2024-02-05T07:27:40.274502+0000","last_clean":"2024-02-05T07:27:40.274502+0000","last_became_active":"2024-02-05T07:27:03.360468+0000","last_became_peered":"2024-02-05T07:27:03.360468+0000","last_unstale":"2024-02-05T07:27:40.274502+0000","last_undegraded":"2024-02-05T07:27:40.274502+0000","last_fullsized":"2024-02-05T07:27:40.274502+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:51.546973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:51.546973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"276'9387","reported_seq":9555,"reported_epoch":276,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:27:40.109695+0000","last_change":"2024-02-05T07:27:27.373558+0000","last_active":"2024-02-05T07:27:40.109695+0000","last_peered":"2024-02-05T07:27:40.109695+0000","last_clean":"2024-02-05T07:26:55.224899+0000","last_became_active":"2024-02-05T07:26:57.335953+0000","last_became_peered":"2024-02-05T07:26:57.335953+0000","last_unstale":"2024-02-05T07:27:40.109695+0000","last_undegraded":"2024-02-05T07:27:40.109695+0000","last_fullsized":"2024-02-05T07:27:40.109695+0000","mapping_epoch":252,"log_start":"232'4042","ondisk_log_start":"232'4042","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206929920,"num_objects":3158,"num_object_clones":0,"num_object_copies":9474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9474,"num_objects_unfound":0,"num_objects_dirty":3158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5338,"num_write_kb":271840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[4,5,0],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4,5","objects":3158}],"blocked_by":[],"up_primary":7,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":253,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.272123+0000","last_change":"2024-02-05T07:27:16.968924+0000","last_active":"2024-02-05T07:27:40.272123+0000","last_peered":"2024-02-05T07:27:40.272123+0000","last_clean":"2024-02-05T07:27:40.272123+0000","last_became_active":"2024-02-05T07:27:16.726999+0000","last_became_peered":"2024-02-05T07:27:16.726999+0000","last_unstale":"2024-02-05T07:27:40.272123+0000","last_undegraded":"2024-02-05T07:27:40.272123+0000","last_fullsized":"2024-02-05T07:27:40.272123+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-02-05T07:24:46.018340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:46.018340+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":480,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.273512+0000","last_change":"2024-02-05T07:27:16.982641+0000","last_active":"2024-02-05T07:27:40.273512+0000","last_peered":"2024-02-05T07:27:40.273512+0000","last_clean":"2024-02-05T07:27:40.273512+0000","last_became_active":"2024-02-05T07:27:16.769199+0000","last_became_peered":"2024-02-05T07:27:16.769199+0000","last_unstale":"2024-02-05T07:27:40.273512+0000","last_undegraded":"2024-02-05T07:27:40.273512+0000","last_fullsized":"2024-02-05T07:27:40.273512+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:25:42.504403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.504403+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"232'9","reported_seq":147,"reported_epoch":276,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:27:39.268773+0000","last_change":"2024-02-05T07:25:59.236281+0000","last_active":"2024-02-05T07:27:39.268773+0000","last_peered":"2024-02-05T07:27:39.268773+0000","last_clean":"2024-02-05T07:18:56.648590+0000","last_became_active":"2024-02-05T07:25:59.236281+0000","last_became_peered":"2024-02-05T07:25:59.236281+0000","last_unstale":"2024-02-05T07:27:39.268773+0000","last_undegraded":"2024-02-05T07:25:59.219241+0000","last_fullsized":"2024-02-05T07:25:59.218638+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:25:31.524507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:31.524507+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":465,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.271908+0000","last_change":"2024-02-05T07:27:16.996449+0000","last_active":"2024-02-05T07:27:40.271908+0000","last_peered":"2024-02-05T07:27:40.271908+0000","last_clean":"2024-02-05T07:27:40.271908+0000","last_became_active":"2024-02-05T07:27:16.713368+0000","last_became_peered":"2024-02-05T07:27:16.713368+0000","last_unstale":"2024-02-05T07:27:40.271908+0000","last_undegraded":"2024-02-05T07:27:40.271908+0000","last_fullsized":"2024-02-05T07:27:40.271908+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:26:18.019957+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.019957+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.534435+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.11","version":"276'9438","reported_seq":17507,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.270744+0000","last_change":"2024-02-05T07:27:32.217226+0000","last_active":"2024-02-05T07:27:40.270744+0000","last_peered":"2024-02-05T07:27:40.270744+0000","last_clean":"2024-02-05T07:27:40.270744+0000","last_became_active":"2024-02-05T07:27:32.206836+0000","last_became_peered":"2024-02-05T07:27:32.206836+0000","last_unstale":"2024-02-05T07:27:40.270744+0000","last_undegraded":"2024-02-05T07:27:40.270744+0000","last_fullsized":"2024-02-05T07:27:40.270744+0000","mapping_epoch":271,"log_start":"232'4000","ondisk_log_start":"232'4000","created":248,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5438,"log_dups_size":0,"ondisk_log_size":5438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:44.097823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211681280,"num_objects":3230,"num_object_clones":0,"num_object_copies":9690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3230,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5346,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3936,"num_bytes_recovered":257949696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,0],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"0,1,7","objects":3230}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":49,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.277565+0000","last_change":"2024-02-05T07:27:16.980628+0000","last_active":"2024-02-05T07:27:40.277565+0000","last_peered":"2024-02-05T07:27:40.277565+0000","last_clean":"2024-02-05T07:27:40.277565+0000","last_became_active":"2024-02-05T07:27:16.747998+0000","last_became_peered":"2024-02-05T07:27:16.747998+0000","last_unstale":"2024-02-05T07:27:40.277565+0000","last_undegraded":"2024-02-05T07:27:40.277565+0000","last_fullsized":"2024-02-05T07:27:40.277565+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-02-05T07:24:46.667757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:24:46.667757+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":138,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.273075+0000","last_change":"2024-02-05T07:26:29.558558+0000","last_active":"2024-02-05T07:27:40.273075+0000","last_peered":"2024-02-05T07:27:40.273075+0000","last_clean":"2024-02-05T07:27:40.273075+0000","last_became_active":"2024-02-05T07:25:30.261037+0000","last_became_peered":"2024-02-05T07:25:30.261037+0000","last_unstale":"2024-02-05T07:27:40.273075+0000","last_undegraded":"2024-02-05T07:27:40.273075+0000","last_fullsized":"2024-02-05T07:27:40.273075+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:29.558353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:29.558353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":457,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-02-05T07:27:39.269314+0000","last_change":"2024-02-05T07:27:16.707095+0000","last_active":"2024-02-05T07:27:39.269314+0000","last_peered":"2024-02-05T07:27:39.269314+0000","last_clean":"2024-02-05T07:27:39.269314+0000","last_became_active":"2024-02-05T07:27:16.706777+0000","last_became_peered":"2024-02-05T07:27:16.706777+0000","last_unstale":"2024-02-05T07:27:39.269314+0000","last_undegraded":"2024-02-05T07:27:39.269314+0000","last_fullsized":"2024-02-05T07:27:39.269314+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:34.068335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:34.068335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":35,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.278826+0000","last_change":"2024-02-05T07:27:17.878009+0000","last_active":"2024-02-05T07:27:40.278826+0000","last_peered":"2024-02-05T07:27:40.278826+0000","last_clean":"2024-02-05T07:27:40.278826+0000","last_became_active":"2024-02-05T07:27:17.877027+0000","last_became_peered":"2024-02-05T07:27:17.877027+0000","last_unstale":"2024-02-05T07:27:40.278826+0000","last_undegraded":"2024-02-05T07:27:40.278826+0000","last_fullsized":"2024-02-05T07:27:40.278826+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:58.410667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:58.410667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"277'9406","reported_seq":13646,"reported_epoch":277,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:27:40.412995+0000","last_change":"2024-02-05T07:27:16.890470+0000","last_active":"2024-02-05T07:27:40.412995+0000","last_peered":"2024-02-05T07:27:40.412995+0000","last_clean":"2024-02-05T07:26:55.184307+0000","last_became_active":"2024-02-05T07:27:04.366630+0000","last_became_peered":"2024-02-05T07:27:04.366630+0000","last_unstale":"2024-02-05T07:27:40.412995+0000","last_undegraded":"2024-02-05T07:27:40.412995+0000","last_fullsized":"2024-02-05T07:27:40.412995+0000","mapping_epoch":256,"log_start":"232'3700","ondisk_log_start":"232'3700","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5706,"log_dups_size":0,"ondisk_log_size":5706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210698240,"num_objects":3215,"num_object_clones":0,"num_object_copies":9645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4766,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5339,"num_write_kb":273728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2034,"num_bytes_recovered":133234688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":3215}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":520,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.282347+0000","last_change":"2024-02-05T07:27:17.036519+0000","last_active":"2024-02-05T07:27:40.282347+0000","last_peered":"2024-02-05T07:27:40.282347+0000","last_clean":"2024-02-05T07:27:40.282347+0000","last_became_active":"2024-02-05T07:27:16.736296+0000","last_became_peered":"2024-02-05T07:27:16.736296+0000","last_unstale":"2024-02-05T07:27:40.282347+0000","last_undegraded":"2024-02-05T07:27:40.282347+0000","last_fullsized":"2024-02-05T07:27:40.282347+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:26:24.030302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.030302+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:53.131323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":199,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-02-05T07:27:39.268237+0000","last_change":"2024-02-05T07:25:33.059237+0000","last_active":"2024-02-05T07:27:39.268237+0000","last_peered":"2024-02-05T07:27:39.268237+0000","last_clean":"2024-02-05T07:27:39.268237+0000","last_became_active":"2024-02-05T07:24:34.234199+0000","last_became_peered":"2024-02-05T07:24:34.234199+0000","last_unstale":"2024-02-05T07:27:39.268237+0000","last_undegraded":"2024-02-05T07:27:39.268237+0000","last_fullsized":"2024-02-05T07:27:39.268237+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:33.059121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:33.059121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":448,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.274202+0000","last_change":"2024-02-05T07:26:21.548894+0000","last_active":"2024-02-05T07:27:40.274202+0000","last_peered":"2024-02-05T07:27:40.274202+0000","last_clean":"2024-02-05T07:27:40.274202+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:27:40.274202+0000","last_undegraded":"2024-02-05T07:27:40.274202+0000","last_fullsized":"2024-02-05T07:27:40.274202+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:21.548785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:21.548785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.889253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":455,"reported_epoch":276,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:39.269384+0000","last_change":"2024-02-05T07:27:17.695119+0000","last_active":"2024-02-05T07:27:39.269384+0000","last_peered":"2024-02-05T07:27:39.269384+0000","last_clean":"2024-02-05T07:27:39.269384+0000","last_became_active":"2024-02-05T07:27:17.694720+0000","last_became_peered":"2024-02-05T07:27:17.694720+0000","last_unstale":"2024-02-05T07:27:39.269384+0000","last_undegraded":"2024-02-05T07:27:39.269384+0000","last_fullsized":"2024-02-05T07:27:39.269384+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:37.074827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:37.074827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,0],"avail_no_missing":["4","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"277'9493","reported_seq":9656,"reported_epoch":277,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-02-05T07:27:40.413398+0000","last_change":"2024-02-05T07:27:27.347302+0000","last_active":"2024-02-05T07:27:40.413398+0000","last_peered":"2024-02-05T07:27:40.413398+0000","last_clean":"2024-02-05T07:25:58.271170+0000","last_became_active":"2024-02-05T07:26:57.322681+0000","last_became_peered":"2024-02-05T07:26:57.322681+0000","last_unstale":"2024-02-05T07:27:40.413398+0000","last_undegraded":"2024-02-05T07:26:57.284528+0000","last_fullsized":"2024-02-05T07:26:57.264609+0000","mapping_epoch":252,"log_start":"232'4000","ondisk_log_start":"232'4000","created":248,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5493,"log_dups_size":0,"ondisk_log_size":5493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212500480,"num_objects":3243,"num_object_clones":0,"num_object_copies":9729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3243,"num_objects_misplaced":6486,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5389,"num_write_kb":276192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":3243}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":483,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-02-05T07:27:39.268127+0000","last_change":"2024-02-05T07:27:16.970973+0000","last_active":"2024-02-05T07:27:39.268127+0000","last_peered":"2024-02-05T07:27:39.268127+0000","last_clean":"2024-02-05T07:27:39.268127+0000","last_became_active":"2024-02-05T07:27:16.745897+0000","last_became_peered":"2024-02-05T07:27:16.745897+0000","last_unstale":"2024-02-05T07:27:39.268127+0000","last_undegraded":"2024-02-05T07:27:39.268127+0000","last_fullsized":"2024-02-05T07:27:39.268127+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:26:35.253452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.253452+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:41.656623+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":272,"reported_epoch":276,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:39.268486+0000","last_change":"2024-02-05T07:27:04.352953+0000","last_active":"2024-02-05T07:27:39.268486+0000","last_peered":"2024-02-05T07:27:39.268486+0000","last_clean":"2024-02-05T07:27:39.268486+0000","last_became_active":"2024-02-05T07:27:04.352619+0000","last_became_peered":"2024-02-05T07:27:04.352619+0000","last_unstale":"2024-02-05T07:27:39.268486+0000","last_undegraded":"2024-02-05T07:27:39.268486+0000","last_fullsized":"2024-02-05T07:27:39.268486+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:35.058925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:35.058925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":447,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.271145+0000","last_change":"2024-02-05T07:27:00.183499+0000","last_active":"2024-02-05T07:27:40.271145+0000","last_peered":"2024-02-05T07:27:40.271145+0000","last_clean":"2024-02-05T07:27:40.271145+0000","last_became_active":"2024-02-05T07:18:53.456881+0000","last_became_peered":"2024-02-05T07:18:53.456881+0000","last_unstale":"2024-02-05T07:27:40.271145+0000","last_undegraded":"2024-02-05T07:27:40.271145+0000","last_fullsized":"2024-02-05T07:27:40.271145+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:27:00.183281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:27:00.183281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:28:09.219274+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":255,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.273877+0000","last_change":"2024-02-05T07:27:16.770005+0000","last_active":"2024-02-05T07:27:40.273877+0000","last_peered":"2024-02-05T07:27:40.273877+0000","last_clean":"2024-02-05T07:27:40.273877+0000","last_became_active":"2024-02-05T07:27:16.769581+0000","last_became_peered":"2024-02-05T07:27:16.769581+0000","last_unstale":"2024-02-05T07:27:40.273877+0000","last_undegraded":"2024-02-05T07:27:40.273877+0000","last_fullsized":"2024-02-05T07:27:40.273877+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:43.606209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:43.606209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.16","version":"277'9414","reported_seq":14365,"reported_epoch":277,"state":"active+undersized+degraded+remapped","last_fresh":"2024-02-05T07:27:40.794235+0000","last_change":"2024-02-05T07:27:16.980722+0000","last_active":"2024-02-05T07:27:40.794235+0000","last_peered":"2024-02-05T07:27:40.794235+0000","last_clean":"2024-02-05T07:25:58.252183+0000","last_became_active":"2024-02-05T07:27:16.778126+0000","last_became_peered":"2024-02-05T07:27:16.778126+0000","last_unstale":"2024-02-05T07:27:40.794235+0000","last_undegraded":"2024-02-05T07:27:16.721664+0000","last_fullsized":"2024-02-05T07:27:16.689282+0000","mapping_epoch":263,"log_start":"230'3500","ondisk_log_start":"230'3500","created":248,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5914,"log_dups_size":0,"ondisk_log_size":5914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208306176,"num_objects":3179,"num_object_clones":0,"num_object_copies":9537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":3179,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5349,"num_write_kb":272864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1592,"num_bytes_recovered":104300544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":3179}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"89'13","reported_seq":501,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-02-05T07:27:39.268938+0000","last_change":"2024-02-05T07:27:16.979411+0000","last_active":"2024-02-05T07:27:39.268938+0000","last_peered":"2024-02-05T07:27:39.268938+0000","last_clean":"2024-02-05T07:27:39.268938+0000","last_became_active":"2024-02-05T07:27:16.707036+0000","last_became_peered":"2024-02-05T07:27:16.707036+0000","last_unstale":"2024-02-05T07:27:39.268938+0000","last_undegraded":"2024-02-05T07:27:39.268938+0000","last_fullsized":"2024-02-05T07:27:39.268938+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-05T07:25:26.134110+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:25:26.134110+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":275,"reported_epoch":276,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:39.268433+0000","last_change":"2024-02-05T07:27:04.348280+0000","last_active":"2024-02-05T07:27:39.268433+0000","last_peered":"2024-02-05T07:27:39.268433+0000","last_clean":"2024-02-05T07:27:39.268433+0000","last_became_active":"2024-02-05T07:27:04.347869+0000","last_became_peered":"2024-02-05T07:27:04.347869+0000","last_unstale":"2024-02-05T07:27:39.268433+0000","last_undegraded":"2024-02-05T07:27:39.268433+0000","last_fullsized":"2024-02-05T07:27:39.268433+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:28.169335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:28.169335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":54,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.271448+0000","last_change":"2024-02-05T07:27:03.361023+0000","last_active":"2024-02-05T07:27:40.271448+0000","last_peered":"2024-02-05T07:27:40.271448+0000","last_clean":"2024-02-05T07:27:40.271448+0000","last_became_active":"2024-02-05T07:27:03.360715+0000","last_became_peered":"2024-02-05T07:27:03.360715+0000","last_unstale":"2024-02-05T07:27:40.271448+0000","last_undegraded":"2024-02-05T07:27:40.271448+0000","last_fullsized":"2024-02-05T07:27:40.271448+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:42.577533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.577533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":201,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.272748+0000","last_change":"2024-02-05T07:24:49.612194+0000","last_active":"2024-02-05T07:27:40.272748+0000","last_peered":"2024-02-05T07:27:40.272748+0000","last_clean":"2024-02-05T07:27:40.272748+0000","last_became_active":"2024-02-05T07:24:34.230537+0000","last_became_peered":"2024-02-05T07:24:34.230537+0000","last_unstale":"2024-02-05T07:27:40.272748+0000","last_undegraded":"2024-02-05T07:27:40.272748+0000","last_fullsized":"2024-02-05T07:27:40.272748+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:49.612116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:49.612116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"276'9569","reported_seq":14854,"reported_epoch":277,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-05T07:27:40.324300+0000","last_change":"2024-02-05T07:27:27.329684+0000","last_active":"2024-02-05T07:27:40.324300+0000","last_peered":"2024-02-05T07:27:40.324300+0000","last_clean":"2024-02-05T07:25:58.207147+0000","last_became_active":"2024-02-05T07:26:57.315087+0000","last_became_peered":"2024-02-05T07:26:57.315087+0000","last_unstale":"2024-02-05T07:27:40.324300+0000","last_undegraded":"2024-02-05T07:26:57.272819+0000","last_fullsized":"2024-02-05T07:26:57.264613+0000","mapping_epoch":252,"log_start":"232'4000","ondisk_log_start":"232'4000","created":248,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5569,"log_dups_size":0,"ondisk_log_size":5569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209649664,"num_objects":3199,"num_object_clones":0,"num_object_copies":9597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":777,"num_objects_misplaced":3199,"num_objects_unfound":0,"num_objects_dirty":3199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5441,"num_write_kb":276480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2562,"num_bytes_recovered":167903232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":3199}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":53,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.278437+0000","last_change":"2024-02-05T07:27:17.002461+0000","last_active":"2024-02-05T07:27:40.278437+0000","last_peered":"2024-02-05T07:27:40.278437+0000","last_clean":"2024-02-05T07:27:40.278437+0000","last_became_active":"2024-02-05T07:27:16.751179+0000","last_became_peered":"2024-02-05T07:27:16.751179+0000","last_unstale":"2024-02-05T07:27:40.278437+0000","last_undegraded":"2024-02-05T07:27:40.278437+0000","last_fullsized":"2024-02-05T07:27:40.278437+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:25:38.772812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:38.772812+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":36,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.278725+0000","last_change":"2024-02-05T07:27:17.711606+0000","last_active":"2024-02-05T07:27:40.278725+0000","last_peered":"2024-02-05T07:27:40.278725+0000","last_clean":"2024-02-05T07:27:40.278725+0000","last_became_active":"2024-02-05T07:27:17.710308+0000","last_became_peered":"2024-02-05T07:27:17.710308+0000","last_unstale":"2024-02-05T07:27:40.278725+0000","last_undegraded":"2024-02-05T07:27:40.278725+0000","last_fullsized":"2024-02-05T07:27:40.278725+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:50.859511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:50.859511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,1],"avail_no_missing":["3","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":448,"reported_epoch":276,"state":"active+undersized","last_fresh":"2024-02-05T07:27:39.267254+0000","last_change":"2024-02-05T07:25:59.223932+0000","last_active":"2024-02-05T07:27:39.267254+0000","last_peered":"2024-02-05T07:27:39.267254+0000","last_clean":"2024-02-05T07:25:57.229914+0000","last_became_active":"2024-02-05T07:25:59.223932+0000","last_became_peered":"2024-02-05T07:25:59.223932+0000","last_unstale":"2024-02-05T07:27:39.267254+0000","last_undegraded":"2024-02-05T07:27:39.267254+0000","last_fullsized":"2024-02-05T07:25:59.216590+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:51.823640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:24:51.823640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":36,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.278607+0000","last_change":"2024-02-05T07:27:17.732257+0000","last_active":"2024-02-05T07:27:40.278607+0000","last_peered":"2024-02-05T07:27:40.278607+0000","last_clean":"2024-02-05T07:27:40.278607+0000","last_became_active":"2024-02-05T07:27:17.731270+0000","last_became_peered":"2024-02-05T07:27:17.731270+0000","last_unstale":"2024-02-05T07:27:40.278607+0000","last_undegraded":"2024-02-05T07:27:40.278607+0000","last_fullsized":"2024-02-05T07:27:40.278607+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:09.289989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:24:09.289989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"276'9442","reported_seq":12102,"reported_epoch":276,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:27:40.024498+0000","last_change":"2024-02-05T07:27:33.378014+0000","last_active":"2024-02-05T07:27:40.024498+0000","last_peered":"2024-02-05T07:27:40.024498+0000","last_clean":"2024-02-05T07:26:55.224933+0000","last_became_active":"2024-02-05T07:26:57.310231+0000","last_became_peered":"2024-02-05T07:26:57.310231+0000","last_unstale":"2024-02-05T07:27:40.024498+0000","last_undegraded":"2024-02-05T07:27:40.024498+0000","last_fullsized":"2024-02-05T07:27:40.024498+0000","mapping_epoch":252,"log_start":"232'4013","ondisk_log_start":"232'4013","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:15.465577+0000","last_clean_scrub_stamp":"2024-02-05T07:25:15.465577+0000","objects_scrubbed":0,"log_size":5429,"log_dups_size":0,"ondisk_log_size":5429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215646208,"num_objects":3291,"num_object_clones":0,"num_object_copies":9873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2333,"num_objects_unfound":0,"num_objects_dirty":3291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5329,"num_write_kb":275808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1247,"num_bytes_recovered":81723392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[{"shards":"2,4,5","objects":3291}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":460,"reported_epoch":276,"state":"active+undersized+degraded","last_fresh":"2024-02-05T07:27:39.267618+0000","last_change":"2024-02-05T07:25:59.232137+0000","last_active":"2024-02-05T07:27:39.267618+0000","last_peered":"2024-02-05T07:27:39.267618+0000","last_clean":"2024-02-05T07:25:57.230142+0000","last_became_active":"2024-02-05T07:25:59.232137+0000","last_became_peered":"2024-02-05T07:25:59.232137+0000","last_unstale":"2024-02-05T07:27:39.267618+0000","last_undegraded":"2024-02-05T07:25:59.223036+0000","last_fullsized":"2024-02-05T07:25:59.222942+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:23:15.477555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:23:15.477555+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":206,"reported_epoch":277,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:27:40.282316+0000","last_change":"2024-02-05T07:27:04.351180+0000","last_active":"2024-02-05T07:27:40.282316+0000","last_peered":"2024-02-05T07:27:40.282316+0000","last_clean":"2024-02-05T07:27:40.282316+0000","last_became_active":"2024-02-05T07:27:04.350767+0000","last_became_peered":"2024-02-05T07:27:04.350767+0000","last_unstale":"2024-02-05T07:27:40.282316+0000","last_undegraded":"2024-02-05T07:27:40.282316+0000","last_fullsized":"2024-02-05T07:27:40.282316+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:24:56.703642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:24:56.703642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:27:42.900418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0,5],"avail_no_missing":["2","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":454,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-02-05T07:27:40.273611+0000","last_change":"2024-02-05T07:27:03.354044+0000","last_active":"2024-02-05T07:27:40.273611+0000","last_peered":"2024-02-05T07:27:40.273611+0000","last_clean":"2024-02-05T07:27:40.273611+0000","last_became_active":"2024-02-05T07:27:03.353249+0000","last_became_peered":"2024-02-05T07:27:03.353249+0000","last_unstale":"2024-02-05T07:27:40.273611+0000","last_undegraded":"2024-02-05T07:27:40.273611+0000","last_fullsized":"2024-02-05T07:27:40.273611+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:28.575859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:28.575859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":447,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-02-05T07:27:39.267577+0000","last_change":"2024-02-05T07:27:03.366854+0000","last_active":"2024-02-05T07:27:39.267577+0000","last_peered":"2024-02-05T07:27:39.267577+0000","last_clean":"2024-02-05T07:27:39.267577+0000","last_became_active":"2024-02-05T07:27:03.366576+0000","last_became_peered":"2024-02-05T07:27:03.366576+0000","last_unstale":"2024-02-05T07:27:39.267577+0000","last_undegraded":"2024-02-05T07:27:39.267577+0000","last_fullsized":"2024-02-05T07:27:39.267577+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:21.317417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:21.317417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":26,"stat_sum":{"num_bytes":6660358168,"num_objects":101630,"num_object_clones":0,"num_object_copies":304890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15325,"num_objects_misplaced":90456,"num_objects_unfound":0,"num_objects_dirty":101630,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166671,"num_write_kb":8449793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33950,"num_bytes_recovered":2224488448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21222076416,"data_stored":21222064200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134897,"ondisk_log_size":134897,"up":65,"acting":70,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":523,"num_read_kb":518,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":375689,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":128,"num_read_kb":190,"num_write":113,"num_write_kb":1538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1915130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":25,"acting":30,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":81,"acting":91,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31,"num_objects_misplaced":34,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":682,"num_read_kb":439,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":4896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":83,"acting":91,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":87,"acting":91,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":84,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037871,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2758816,"kb_used_data":2348628,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":90986336,"statfs":{"total":95995035648,"available":93170008064,"internally_reserved":0,"allocated":2404995072,"data_stored":2404040164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":420018634},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:26:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":3.5190000000000001,"15min":1.534},"min":{"1min":0.27400000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.1219999999999999,"5min":63.039000000000001,"15min":63.039000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":3.4790000000000001,"15min":1.54},"min":{"1min":0.26400000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.3860000000000001,"5min":65.296000000000006,"15min":65.296000000000006},"last":0.63200000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:26:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":2.0680000000000001,"15min":1.0620000000000001},"min":{"1min":0.29499999999999998,"5min":0.222,"15min":0.222},"max":{"1min":3.48,"5min":32.127000000000002,"15min":32.127000000000002},"last":0.219},{"interface":"front","average":{"1min":1.149,"5min":2.1469999999999998,"15min":1.107},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":6.0090000000000003,"5min":32.173999999999999,"15min":32.173999999999999},"last":1.651}]},{"osd":2,"last update":"Mon Feb 5 07:26:54 2024","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":2.4750000000000001,"15min":1.196},"min":{"1min":0.28299999999999997,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":10.359,"5min":51.127000000000002,"15min":51.127000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.9670000000000001,"5min":2.806,"15min":1.319},"min":{"1min":0.33500000000000002,"5min":0.27700000000000002,"15min":0.246},"max":{"1min":10.404999999999999,"5min":51.113999999999997,"15min":51.113999999999997},"last":1.036}]},{"osd":3,"last update":"Mon Feb 5 07:26:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.859,"15min":1.0289999999999999},"min":{"1min":0.35999999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":8.3699999999999992,"5min":58.341999999999999,"15min":58.341999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.7350000000000001,"15min":0.99099999999999999},"min":{"1min":0.32000000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.419,"5min":57.765000000000001,"15min":57.765000000000001},"last":0.88200000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.69699999999999995,"15min":0.52700000000000002},"min":{"1min":0.26000000000000001,"5min":0.25,"15min":0.25},"max":{"1min":2.0299999999999998,"5min":4.8659999999999997,"15min":4.8659999999999997},"last":0.44},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.68200000000000005,"15min":0.53000000000000003},"min":{"1min":0.28699999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.1699999999999999,"5min":4.577,"15min":4.577},"last":0.35499999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.78500000000000003,"15min":0.55300000000000005},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.3450000000000002,"5min":6.9020000000000001,"15min":6.9020000000000001},"last":0.309},{"interface":"front","average":{"1min":0.88,"5min":0.91600000000000004,"15min":0.63500000000000001},"min":{"1min":0.29799999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.097,"5min":6.3449999999999998,"15min":6.3449999999999998},"last":0.51100000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:25:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.77600000000000002,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.6310000000000002,"5min":4.9630000000000001,"15min":4.9630000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.872,"5min":0.71299999999999997,"15min":0.58099999999999996},"min":{"1min":0.36699999999999999,"5min":0.318,"15min":0.318},"max":{"1min":3.5150000000000001,"5min":5.5179999999999998,"15min":5.5179999999999998},"last":1.956}]}]},{"osd":6,"up_from":52,"seq":223338299488,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1295576,"kb_used_data":1057544,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92449576,"statfs":{"total":95995035648,"available":94668365824,"internally_reserved":0,"allocated":1082925056,"data_stored":1082249111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":243726790},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":6.6749999999999998,"5min":4.2910000000000004,"15min":1.7929999999999999},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":111.81699999999999,"5min":111.81699999999999,"15min":111.81699999999999},"last":2.2709999999999999},{"interface":"front","average":{"1min":6.8140000000000001,"5min":4.1550000000000002,"15min":1.738},"min":{"1min":0.34000000000000002,"5min":0.22,"15min":0.22},"max":{"1min":111.09699999999999,"5min":111.09699999999999,"15min":111.09699999999999},"last":1.772}]},{"osd":1,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":2.387,"5min":1.254,"15min":0.78100000000000003},"min":{"1min":0.26700000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":33.984999999999999,"5min":33.984999999999999,"15min":33.984999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":2.3220000000000001,"5min":1.1859999999999999,"15min":0.76400000000000001},"min":{"1min":0.374,"5min":0.249,"15min":0.249},"max":{"1min":32.134,"5min":32.134,"15min":32.134},"last":0.94499999999999995}]},{"osd":2,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":1.9019999999999999,"15min":1.0169999999999999},"min":{"1min":0.36399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":31.878,"5min":32.509999999999998,"15min":32.509999999999998},"last":0.79900000000000004},{"interface":"front","average":{"1min":2.214,"5min":2.1309999999999998,"15min":1.0640000000000001},"min":{"1min":0.36699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":31.507000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"last":2.524}]},{"osd":3,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":2.0299999999999998,"15min":1.0469999999999999},"min":{"1min":0.376,"5min":0.22,"15min":0.22},"max":{"1min":1.4470000000000001,"5min":36.976999999999997,"15min":36.976999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.835,"15min":0.98799999999999999},"min":{"1min":0.34899999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72099999999999997,"5min":34.222999999999999,"15min":34.222999999999999},"last":0.83399999999999996}]},{"osd":4,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.59999999999999998,"15min":0.47099999999999997},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.173,"5min":4.5609999999999999,"15min":4.5609999999999999},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.876,"5min":0.63400000000000001,"15min":0.51500000000000001},"min":{"1min":0.28599999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":7.7960000000000003,"5min":7.7960000000000003,"15min":7.7960000000000003},"last":0.871}]},{"osd":5,"last update":"Mon Feb 5 07:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.90000000000000002,"15min":0.59899999999999998},"min":{"1min":0.307,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.1369999999999996,"5min":27.021999999999998,"15min":27.021999999999998},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.94899999999999995,"15min":0.623},"min":{"1min":0.34300000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.0629999999999997,"5min":23.611999999999998,"15min":23.611999999999998},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Feb 5 07:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.66600000000000004,"15min":0.55800000000000005},"min":{"1min":0.29399999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.044,"5min":4.2130000000000001,"15min":4.2130000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.98999999999999999,"15min":0.69299999999999995},"min":{"1min":0.33300000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.9740000000000002,"5min":25.207000000000001,"15min":25.207000000000001},"last":0.69599999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574989,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3483784,"kb_used_data":3113848,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":90261368,"statfs":{"total":95995035648,"available":92427640832,"internally_reserved":0,"allocated":3188580352,"data_stored":3187598388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:27:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":2.5880000000000001,"15min":1.6060000000000001},"min":{"1min":0.191,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.0960000000000001,"5min":30.536999999999999,"15min":45.393000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.704,"5min":2.7200000000000002,"15min":1.603},"min":{"1min":0.318,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":7.5449999999999999,"5min":31.853999999999999,"15min":45.031999999999996},"last":4.9139999999999997}]},{"osd":2,"last update":"Mon Feb 5 07:26:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":2.4729999999999999,"15min":1.1399999999999999},"min":{"1min":0.25600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.798,"5min":39.259999999999998,"15min":39.259999999999998},"last":1.996},{"interface":"front","average":{"1min":2.1059999999999999,"5min":2.464,"15min":1.4279999999999999},"min":{"1min":0.27300000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.379,"5min":40.972000000000001,"15min":40.972000000000001},"last":5.7000000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":1.669,"15min":1.119},"min":{"1min":0.27800000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.8140000000000001,"5min":20.446000000000002,"15min":43.823},"last":4.8250000000000002},{"interface":"front","average":{"1min":1.5960000000000001,"5min":2.0979999999999999,"15min":1.2589999999999999},"min":{"1min":0.26600000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.8150000000000004,"5min":21.228000000000002,"15min":39.314999999999998},"last":5.6619999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:27:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8999999999999999,"5min":2.9630000000000001,"15min":1.4590000000000001},"min":{"1min":0.46700000000000003,"5min":0.312,"15min":0.312},"max":{"1min":9.4149999999999991,"5min":61.722000000000001,"15min":61.722000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.2150000000000001,"5min":2.6880000000000002,"15min":1.482},"min":{"1min":0.311,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":7.6070000000000002,"5min":61.786000000000001,"15min":61.786000000000001},"last":0.75800000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.8380000000000001,"5min":3.2200000000000002,"15min":1.5049999999999999},"min":{"1min":0.34499999999999997,"5min":0.33600000000000002,"15min":0.32700000000000001},"max":{"1min":9.5960000000000001,"5min":106.672,"15min":106.672},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.98299999999999998,"5min":3.1829999999999998,"15min":1.5700000000000001},"min":{"1min":0.39500000000000002,"5min":0.307,"15min":0.307},"max":{"1min":2.9620000000000002,"5min":107.348,"15min":107.348},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.4330000000000001,"15min":0.93000000000000005},"min":{"1min":0.42499999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.5739999999999998,"5min":18.456,"15min":18.456},"last":1.016},{"interface":"front","average":{"1min":0.86499999999999999,"5min":1.395,"15min":0.91400000000000003},"min":{"1min":0.46300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.4340000000000002,"5min":5.9930000000000003,"15min":5.9930000000000003},"last":1.5740000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:26:54 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":2.2850000000000001,"15min":1.153},"min":{"1min":0.35799999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.8940000000000001,"5min":66.703999999999994,"15min":66.703999999999994},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.3260000000000001,"5min":2.2370000000000001,"15min":1.1619999999999999},"min":{"1min":0.371,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":3.6680000000000001,"5min":61.594999999999999,"15min":61.594999999999999},"last":0.73299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738514,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3324296,"kb_used_data":2895996,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":90420856,"statfs":{"total":95995035648,"available":92590956544,"internally_reserved":0,"allocated":2965499904,"data_stored":2964513429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:27:27 2024","interfaces":[{"interface":"back","average":{"1min":2.0179999999999998,"5min":2.2650000000000001,"15min":1.1819999999999999},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.7189999999999994,"5min":45.372,"15min":45.372},"last":5.0419999999999998},{"interface":"front","average":{"1min":2.383,"5min":2.3380000000000001,"15min":1.3009999999999999},"min":{"1min":0.22,"5min":0.186,"15min":0.186},"max":{"1min":12.089,"5min":45.231999999999999,"15min":45.231999999999999},"last":2.4500000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:26:56 2024","interfaces":[{"interface":"back","average":{"1min":3.3599999999999999,"5min":2.9980000000000002,"15min":1.3899999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":44.920000000000002,"5min":65.903999999999996,"15min":65.903999999999996},"last":2.3460000000000001},{"interface":"front","average":{"1min":3.1850000000000001,"5min":2.9609999999999999,"15min":1.349},"min":{"1min":0.23300000000000001,"5min":0.186,"15min":0.182},"max":{"1min":44.947000000000003,"5min":65.694000000000003,"15min":65.694000000000003},"last":0.51100000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:27:08 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":2.5750000000000002,"15min":1.4410000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.6199999999999992,"5min":65.155000000000001,"15min":65.155000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.97499999999999998,"5min":2.581,"15min":1.472},"min":{"1min":0.251,"5min":0.245,"15min":0.245},"max":{"1min":3.0579999999999998,"5min":65.114000000000004,"15min":65.114000000000004},"last":0.90700000000000003}]},{"osd":4,"last update":"Mon Feb 5 07:27:31 2024","interfaces":[{"interface":"back","average":{"1min":3.0569999999999999,"5min":2.8889999999999998,"15min":1.498},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":33.683999999999997,"5min":65.188000000000002,"15min":65.188000000000002},"last":0.65000000000000002},{"interface":"front","average":{"1min":2.9500000000000002,"5min":2.8820000000000001,"15min":1.52},"min":{"1min":0.29199999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":33.640000000000001,"5min":64.873000000000005,"15min":64.873000000000005},"last":2.3050000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:27:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":3.125,"15min":1.488},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.0590000000000002,"5min":95.772999999999996,"15min":95.772999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.4450000000000001,"5min":3.2989999999999999,"15min":1.5580000000000001},"min":{"1min":0.34699999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.5720000000000001,"5min":96.028000000000006,"15min":96.028000000000006},"last":2.379}]},{"osd":6,"last update":"Mon Feb 5 07:25:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":2.2610000000000001,"15min":1.155},"min":{"1min":0.45600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":25.681999999999999,"5min":65.215000000000003,"15min":65.215000000000003},"last":0.63500000000000001},{"interface":"front","average":{"1min":2.2530000000000001,"5min":2.4489999999999998,"15min":1.242},"min":{"1min":0.41899999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":31.524000000000001,"5min":65.230000000000004,"15min":65.230000000000004},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:26:42 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":2.3769999999999998,"15min":1.248},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.9080000000000004,"5min":65.263000000000005,"15min":65.263000000000005},"last":0.62},{"interface":"front","average":{"1min":1.0609999999999999,"5min":2.3239999999999998,"15min":1.2509999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":3.1379999999999999,"5min":64.906000000000006,"15min":64.906000000000006},"last":2.419}]}]},{"osd":2,"up_from":18,"seq":77309411464,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4089812,"kb_used_data":3617160,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":89655340,"statfs":{"total":95995035648,"available":91807068160,"internally_reserved":0,"allocated":3703971840,"data_stored":3703030099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:26:53 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":2.3969999999999998,"15min":1.1319999999999999},"min":{"1min":0.28499999999999998,"5min":0.182,"15min":0.182},"max":{"1min":6.0540000000000003,"5min":67.418999999999997,"15min":67.418999999999997},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.25,"5min":2.7770000000000001,"15min":1.244},"min":{"1min":0.25700000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":4.8479999999999999,"5min":68.171000000000006,"15min":68.171000000000006},"last":1.855}]},{"osd":1,"last update":"Mon Feb 5 07:26:53 2024","interfaces":[{"interface":"back","average":{"1min":1.8560000000000001,"5min":1.544,"15min":0.82999999999999996},"min":{"1min":0.27700000000000002,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.2030000000000003,"5min":12.609,"15min":12.609},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.603,"15min":0.88},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":5.6970000000000001,"5min":9.641,"15min":9.641},"last":1.415}]},{"osd":3,"last update":"Mon Feb 5 07:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.579,"15min":1.008},"min":{"1min":0.32300000000000001,"5min":0.214,"15min":0.214},"max":{"1min":5.0460000000000003,"5min":9.6790000000000003,"15min":14.756},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.333,"5min":1.4930000000000001,"15min":0.99099999999999999},"min":{"1min":0.28199999999999997,"5min":0.193,"15min":0.193},"max":{"1min":5.4199999999999999,"5min":8.5370000000000008,"15min":9.6669999999999998},"last":0.29999999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:27:40 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.607,"15min":0.98299999999999998},"min":{"1min":0.32900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.1890000000000001,"5min":37.479999999999997,"15min":37.479999999999997},"last":1.7689999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.514,"15min":1.008},"min":{"1min":0.34300000000000003,"5min":0.317,"15min":0.317},"max":{"1min":5.0300000000000002,"5min":38.853999999999999,"15min":38.853999999999999},"last":0.98499999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:26:55 2024","interfaces":[{"interface":"back","average":{"1min":3.4620000000000002,"5min":1.778,"15min":1.052},"min":{"1min":0.45100000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":39.923000000000002,"5min":39.923000000000002,"15min":39.923000000000002},"last":1.02},{"interface":"front","average":{"1min":3.4769999999999999,"5min":1.6830000000000001,"15min":1.018},"min":{"1min":0.45100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":38.920999999999999,"5min":38.920999999999999,"15min":38.920999999999999},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Feb 5 07:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":1.3109999999999999,"15min":0.93700000000000006},"min":{"1min":0.42799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.3069999999999999,"5min":7.6829999999999998,"15min":7.6829999999999998},"last":2.7400000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.381,"15min":0.86399999999999999},"min":{"1min":0.41699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":2.0489999999999999,"5min":7.3150000000000004,"15min":7.3150000000000004},"last":1.9830000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:26:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.532,"15min":0.97399999999999998},"min":{"1min":0.41999999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.1609999999999996,"5min":7.2590000000000003,"15min":7.2590000000000003},"last":1.8240000000000001},{"interface":"front","average":{"1min":1.8779999999999999,"5min":1.4530000000000001,"15min":0.94199999999999995},"min":{"1min":0.48799999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":5.3970000000000002,"5min":7.7619999999999996,"15min":7.7619999999999996},"last":0.60399999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149827,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":742192,"kb_used_data":500004,"kb_used_omap":1,"kb_used_meta":242174,"kb_avail":93002960,"statfs":{"total":95995035648,"available":95235031040,"internally_reserved":0,"allocated":512004096,"data_stored":511033614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":247986631},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:27:26 2024","interfaces":[{"interface":"back","average":{"1min":3.1930000000000001,"5min":2.8199999999999998,"15min":1.8640000000000001},"min":{"1min":0.245,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":32.078000000000003,"5min":45.805,"15min":74.114999999999995},"last":2.7789999999999999},{"interface":"front","average":{"1min":3.8919999999999999,"5min":3.048,"15min":1.6279999999999999},"min":{"1min":0.27000000000000002,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":34.598999999999997,"5min":48.740000000000002,"15min":48.740000000000002},"last":2.6709999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:27:26 2024","interfaces":[{"interface":"back","average":{"1min":3.6400000000000001,"5min":2.6779999999999999,"15min":1.486},"min":{"1min":0.36399999999999999,"5min":0.27300000000000002,"15min":0.253},"max":{"1min":29.318999999999999,"5min":58.746000000000002,"15min":58.746000000000002},"last":2.468},{"interface":"front","average":{"1min":3.855,"5min":2.9980000000000002,"15min":1.8340000000000001},"min":{"1min":0.35599999999999998,"5min":0.185,"15min":0.185},"max":{"1min":29.393000000000001,"5min":56.244999999999997,"15min":74.084999999999994},"last":2.734}]},{"osd":2,"last update":"Mon Feb 5 07:27:26 2024","interfaces":[{"interface":"back","average":{"1min":2.54,"5min":3.2709999999999999,"15min":1.8300000000000001},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":29.123999999999999,"5min":66.268000000000001,"15min":74.051000000000002},"last":4.7960000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":2.77,"15min":1.3640000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.2440000000000002,"5min":65.861999999999995,"15min":65.861999999999995},"last":1.8999999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:27:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.2130000000000001,"15min":0.85599999999999998},"min":{"1min":0.32800000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.6749999999999998,"5min":8.4339999999999993,"15min":8.4339999999999993},"last":1.0900000000000001},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.2330000000000001,"15min":0.84399999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.8999999999999999,"5min":12.896000000000001,"15min":12.896000000000001},"last":0.28599999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":2.5169999999999999,"15min":1.2829999999999999},"min":{"1min":0.39400000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.6120000000000001,"5min":73.727000000000004,"15min":73.727000000000004},"last":1.9219999999999999},{"interface":"front","average":{"1min":1.22,"5min":1.6459999999999999,"15min":0.98199999999999998},"min":{"1min":0.33700000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":5.8490000000000002,"5min":21.126000000000001,"15min":21.126000000000001},"last":1.135}]},{"osd":6,"last update":"Mon Feb 5 07:25:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":1.1579999999999999,"15min":0.77800000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.34200000000000003},"max":{"1min":0.81899999999999995,"5min":12.864000000000001,"15min":12.864000000000001},"last":1.036},{"interface":"front","average":{"1min":0.61299999999999999,"5min":1.2649999999999999,"15min":0.82399999999999995},"min":{"1min":0.42799999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.745,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":5.7670000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:26:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.5129999999999999,"15min":0.90900000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.36499999999999999},"max":{"1min":3.1120000000000001,"5min":15.163,"15min":15.163},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.0289999999999999,"5min":2.476,"15min":1.256},"min":{"1min":0.44600000000000001,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":2.5070000000000001,"5min":71.283000000000001,"15min":71.283000000000001},"last":1.863}]}]},{"osd":4,"up_from":32,"seq":137438953598,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3870500,"kb_used_data":3536664,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":89874652,"statfs":{"total":95995035648,"available":92031643648,"internally_reserved":0,"allocated":3621543936,"data_stored":3620584799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:27:35 2024","interfaces":[{"interface":"back","average":{"1min":9.2690000000000001,"5min":3.3439999999999999,"15min":1.536},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":67.718999999999994,"5min":67.719999999999999,"15min":67.719999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":8.9499999999999993,"5min":3.5299999999999998,"15min":1.6279999999999999},"min":{"1min":0.317,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":68.829999999999998,"5min":68.829999999999998,"15min":68.829999999999998},"last":0.23899999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:27:35 2024","interfaces":[{"interface":"back","average":{"1min":8.5860000000000003,"5min":2.363,"15min":1.202},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":69.102000000000004,"5min":69.102000000000004,"15min":69.102000000000004},"last":0.26100000000000001},{"interface":"front","average":{"1min":8.8840000000000003,"5min":2.6659999999999999,"15min":1.286},"min":{"1min":0.26200000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":67.828999999999994,"5min":67.828999999999994,"15min":67.828999999999994},"last":0.753}]},{"osd":2,"last update":"Mon Feb 5 07:27:35 2024","interfaces":[{"interface":"back","average":{"1min":2.4430000000000001,"5min":2.1200000000000001,"15min":1.105},"min":{"1min":0.29999999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":26.045000000000002,"5min":67.144999999999996,"15min":67.144999999999996},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.919,"15min":1.0529999999999999},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":25.960999999999999,"5min":66.959999999999994,"15min":66.959999999999994},"last":1.5109999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.76800000000000002,"15min":0.71499999999999997},"min":{"1min":0.31,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.766,"5min":2.8740000000000001,"15min":8.8019999999999996},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.872,"5min":0.85199999999999998,"15min":0.76900000000000002},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.27800000000000002},"max":{"1min":2.7869999999999999,"5min":5.3449999999999998,"15min":10.106},"last":0.79500000000000004}]},{"osd":5,"last update":"Mon Feb 5 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.66800000000000004,"15min":0.48699999999999999},"min":{"1min":0.29099999999999998,"5min":0.25900000000000001,"15min":0.25700000000000001},"max":{"1min":2.5790000000000002,"5min":2.8420000000000001,"15min":2.8420000000000001},"last":0.309},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.66900000000000004,"15min":0.496},"min":{"1min":0.27800000000000002,"5min":0.255,"15min":0.254},"max":{"1min":2.645,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":0.34300000000000003}]},{"osd":6,"last update":"Mon Feb 5 07:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.70999999999999996,"15min":0.51600000000000001},"min":{"1min":0.36199999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.29,"5min":6.8940000000000001,"15min":6.8940000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.67800000000000005,"15min":0.501},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.248},"max":{"1min":2.9260000000000002,"5min":2.9260000000000002,"15min":2.9260000000000002},"last":0.38700000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.746,"15min":0.58999999999999997},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.835,"5min":6.0640000000000001,"15min":6.0640000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.72699999999999998,"15min":0.57099999999999995},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2.532,"5min":2.532,"15min":2.532},"last":0.32400000000000001}]}]},{"osd":5,"up_from":0,"seq":176093659257,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2949840896,"data_stored":2949840896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3182624768,"data_stored":3182624768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3678343168,"data_stored":3678339096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":551550976,"data_stored":551550976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3625582592,"data_stored":3625582592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3747352576,"data_stored":3747348504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1085865984,"data_stored":1085865984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2400915456,"data_stored":2400911384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:27:51.375 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-05T07:27:51.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-02-05T07:27:51.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:51.478 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:51.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:51.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:51.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:51.586 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:51 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:51.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:51 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:51.607 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:51.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:51.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:51.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:51.739 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:51.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:51.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:51 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:27:51.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:51.872 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:51.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:51.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:51.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:52.022 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:52.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:52.047 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:27:52.118 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:27:52.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:52.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:52.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:52.289 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:52.341 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:52.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:52.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:52.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:52.488 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:52.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:52.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:52.497 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:52 smithi007 ceph-mon[29098]: pgmap v451: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.17k op/s; 12672/281457 objects degraded (4.502%); 80827/281457 objects misplaced (28.717%); 62 MiB/s, 996 objects/s recovering 2024-02-05T07:27:52.497 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:52 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:52.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:52 smithi007 ceph-mon[37158]: pgmap v451: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.17k op/s; 12672/281457 objects degraded (4.502%); 80827/281457 objects misplaced (28.717%); 62 MiB/s, 996 objects/s recovering 2024-02-05T07:27:52.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:52 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:52.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:52.638 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:52.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:52.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:52.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:52 smithi090 ceph-mon[33213]: pgmap v451: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.17k op/s; 12672/281457 objects degraded (4.502%); 80827/281457 objects misplaced (28.717%); 62 MiB/s, 996 objects/s recovering 2024-02-05T07:27:52.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:52 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:27:52.779 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:52.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:52.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:52.907 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:52.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:53.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:53.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:53.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:53.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:53.174 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:53.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:53.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:53.299 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:53.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:53.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:53.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:53.434 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:53.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:53.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:53.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:53.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:53.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:53.707 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:53.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:53.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:53.850 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:53.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:53.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:53.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:27:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:27:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:27:53.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:53.982 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:53.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:53.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:54.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:54.109 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:54.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:54.211 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:54.242 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:54.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:54.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:54.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:54.374 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:54.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:54.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:54.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:54.507 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:54.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:54.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:54.652 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:54.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:54.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:54.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:54 smithi007 ceph-mon[29098]: pgmap v452: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 12672/281457 objects degraded (4.502%); 80827/281457 objects misplaced (28.717%); 53 MiB/s, 854 objects/s recovering 2024-02-05T07:27:54.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:54 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 12672/281457 objects degraded (4.502%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:54.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:54 smithi007 ceph-mon[37158]: pgmap v452: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 12672/281457 objects degraded (4.502%); 80827/281457 objects misplaced (28.717%); 53 MiB/s, 854 objects/s recovering 2024-02-05T07:27:54.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:54 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 12672/281457 objects degraded (4.502%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:54.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:54.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:54 smithi090 ceph-mon[33213]: pgmap v452: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 12672/281457 objects degraded (4.502%); 80827/281457 objects misplaced (28.717%); 53 MiB/s, 854 objects/s recovering 2024-02-05T07:27:54.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:54 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 12672/281457 objects degraded (4.502%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:54.784 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:54.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:54.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:54.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:54.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:55.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:55.063 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:55.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:55.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:55.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:55.204 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:55.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:55.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:55.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:55.334 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:55.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:55.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:55.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:55.482 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:55.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:55.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:55.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:55.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:55.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:55.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:55.752 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:55.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:55.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:55.903 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:55.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:56.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:56.063 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:56.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:56.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:56.207 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:56.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:56.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:56.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:56.339 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:56.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:56.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:56.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:56.481 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:56.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:56.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:56.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:56.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:56.621 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:56.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:56.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:56 smithi007 ceph-mon[29098]: pgmap v453: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 732 op/s; 12672/281457 objects degraded (4.502%); 80827/281457 objects misplaced (28.717%); 40 MiB/s, 647 objects/s recovering 2024-02-05T07:27:56.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:56 smithi007 ceph-mon[37158]: pgmap v453: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 732 op/s; 12672/281457 objects degraded (4.502%); 80827/281457 objects misplaced (28.717%); 40 MiB/s, 647 objects/s recovering 2024-02-05T07:27:56.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:56.755 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:56.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:56.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:56.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:56 smithi090 ceph-mon[33213]: pgmap v453: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 732 op/s; 12672/281457 objects degraded (4.502%); 80827/281457 objects misplaced (28.717%); 40 MiB/s, 647 objects/s recovering 2024-02-05T07:27:56.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:56.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:56.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:56.982 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:57.020 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:57.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:57.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:57.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:57.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:57.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:57.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:57.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:27:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:57.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:27:57.544 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:27:57.568 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:57.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:57.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:27:57.670 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:57.713 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:57.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:57.815 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:57.849 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:57.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:57.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:57.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:57.974 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:57.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:57.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:58.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:58.115 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:58.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:58.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:58.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:58.251 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:58.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:58.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:58.390 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:58.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:58.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:58.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:58.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:58.539 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:58.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:58.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:58.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:58 smithi007 ceph-mon[29098]: pgmap v454: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 943 B/s rd, 0 B/s wr, 886 op/s; 11969/265833 objects degraded (4.502%); 73994/265833 objects misplaced (27.835%); 25 MiB/s, 392 objects/s recovering 2024-02-05T07:27:58.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:58 smithi007 ceph-mon[37158]: pgmap v454: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 943 B/s rd, 0 B/s wr, 886 op/s; 11969/265833 objects degraded (4.502%); 73994/265833 objects misplaced (27.835%); 25 MiB/s, 392 objects/s recovering 2024-02-05T07:27:58.680 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:58.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:58.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:58.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:58 smithi090 ceph-mon[33213]: pgmap v454: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 943 B/s rd, 0 B/s wr, 886 op/s; 11969/265833 objects degraded (4.502%); 73994/265833 objects misplaced (27.835%); 25 MiB/s, 392 objects/s recovering 2024-02-05T07:27:58.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:58.820 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:58.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:58.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:58.956 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:58.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:59.057 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:27:59.081 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:59.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:59.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:27:59.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:27:59.218 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:59.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:59.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:27:59.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:59.362 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:59.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:59.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:59.472 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:27:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:59.531 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:59.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:27:59.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:27:59.662 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:27:59.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:59.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:27:59.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:59 smithi007 ceph-mon[29098]: osdmap e283: 8 total, 7 up, 7 in 2024-02-05T07:27:59.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:59 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 11969/265833 objects degraded (4.502%), 13 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:59.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:27:59 smithi007 ceph-mon[29098]: osdmap e284: 8 total, 7 up, 7 in 2024-02-05T07:27:59.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:59 smithi007 ceph-mon[37158]: osdmap e283: 8 total, 7 up, 7 in 2024-02-05T07:27:59.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:59 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 11969/265833 objects degraded (4.502%), 13 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:59.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:27:59 smithi007 ceph-mon[37158]: osdmap e284: 8 total, 7 up, 7 in 2024-02-05T07:27:59.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:27:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:59 smithi090 ceph-mon[33213]: osdmap e283: 8 total, 7 up, 7 in 2024-02-05T07:27:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:59 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 11969/265833 objects degraded (4.502%), 13 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-02-05T07:27:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:27:59 smithi090 ceph-mon[33213]: osdmap e284: 8 total, 7 up, 7 in 2024-02-05T07:27:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:59.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:59.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:27:59.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:27:59.954 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:27:59.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:27:59.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:00.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:00.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:00.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:00.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:00.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:00.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:00.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:00.366 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:00.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:00.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:00.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:00.518 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:00.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:00.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:00.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:00.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:00 smithi007 ceph-mon[29098]: pgmap v456: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 961 op/s; 11969/265833 objects degraded (4.502%); 73994/265833 objects misplaced (27.835%); 27 MiB/s, 425 objects/s recovering 2024-02-05T07:28:00.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:00 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:28:00.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:00 smithi007 ceph-mon[37158]: pgmap v456: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 961 op/s; 11969/265833 objects degraded (4.502%); 73994/265833 objects misplaced (27.835%); 27 MiB/s, 425 objects/s recovering 2024-02-05T07:28:00.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:00 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:28:00.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:00.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:00 smithi090 ceph-mon[33213]: pgmap v456: 168 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 35 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 961 op/s; 11969/265833 objects degraded (4.502%); 73994/265833 objects misplaced (27.835%); 27 MiB/s, 425 objects/s recovering 2024-02-05T07:28:00.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:00 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:28:00.775 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:28:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:28:00.779 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:00.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:00.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:00.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:00.911 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:00.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:01.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:01.040 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:01.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:01.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:01.142 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:01.169 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:01.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:01.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:01.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:01.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:01.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:01.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:01.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:01.460 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:01.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:01.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:01.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:01.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:01.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:01.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:01.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:01 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:28:01.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:01 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2509997563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:28:01.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:01 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:28:01.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:01.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:01 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:28:01.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:01 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2509997563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:28:01.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:01 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:28:01.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:01.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:01.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:01.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:01.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:01.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:01 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:28:01.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:01 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2509997563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:28:01.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:01 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:28:01.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:01.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:01.903 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:01.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:02.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:02.048 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:02.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:02.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:02.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:02.191 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:02.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:02.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:02.319 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:02.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:02.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:02.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:02.464 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:02.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:02.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:02.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:02.602 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:02.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:02.655 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:28:02.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:02 smithi007 ceph-mon[29098]: pgmap v458: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 10936/252237 objects degraded (4.336%); 67451/252237 objects misplaced (26.741%); 40 MiB/s, 640 objects/s recovering 2024-02-05T07:28:02.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:02 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:02.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:02 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:28:02.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:02 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:28:02.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:02 smithi007 ceph-mon[29098]: osdmap e285: 8 total, 7 up, 7 in 2024-02-05T07:28:02.657 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:02 smithi007 ceph-mon[37158]: pgmap v458: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 10936/252237 objects degraded (4.336%); 67451/252237 objects misplaced (26.741%); 40 MiB/s, 640 objects/s recovering 2024-02-05T07:28:02.657 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:02 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:02.657 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:02 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:28:02.657 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:02 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:28:02.657 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:02 smithi007 ceph-mon[37158]: osdmap e285: 8 total, 7 up, 7 in 2024-02-05T07:28:02.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:02.748 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:02.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:02 smithi090 ceph-mon[33213]: pgmap v458: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 10936/252237 objects degraded (4.336%); 67451/252237 objects misplaced (26.741%); 40 MiB/s, 640 objects/s recovering 2024-02-05T07:28:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:02 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:02 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:28:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:02 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:28:02.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:02 smithi090 ceph-mon[33213]: osdmap e285: 8 total, 7 up, 7 in 2024-02-05T07:28:02.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:02.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:02.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:02.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:02.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:03.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:03.023 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:03.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:03.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:03.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:03.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:03.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:03.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:03.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:03.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:03.479 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:03.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:03.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:03.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:03.621 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:03.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:03.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:03 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:28:03.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:03 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:28:03.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:03 smithi007 ceph-mon[29098]: osdmap e286: 8 total, 7 up, 7 in 2024-02-05T07:28:03.671 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:28:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:28:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:28:03.671 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:03 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:28:03.671 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:03 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:28:03.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:03 smithi007 ceph-mon[37158]: osdmap e286: 8 total, 7 up, 7 in 2024-02-05T07:28:03.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:03.762 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:03.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:03.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:03 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:28:03.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:03 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:28:03.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:03 smithi090 ceph-mon[33213]: osdmap e286: 8 total, 7 up, 7 in 2024-02-05T07:28:03.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:03.896 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:03.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:03.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:04.039 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:04.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:04.141 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:04.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:04.184 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:04.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:04.285 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:04.333 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:04.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:04.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:04.464 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:04.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:04.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:04.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:04.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:04.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:04.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:04.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:04 smithi007 ceph-mon[29098]: pgmap v461: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 949 op/s; 10936/252237 objects degraded (4.336%); 67451/252237 objects misplaced (26.741%); 36 MiB/s, 578 objects/s recovering 2024-02-05T07:28:04.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:04 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 10936/252237 objects degraded (4.336%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:04.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:04 smithi007 ceph-mon[37158]: pgmap v461: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 949 op/s; 10936/252237 objects degraded (4.336%); 67451/252237 objects misplaced (26.741%); 36 MiB/s, 578 objects/s recovering 2024-02-05T07:28:04.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:04 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 10936/252237 objects degraded (4.336%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:04.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:04.731 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:04.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:04.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:04.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:04 smithi090 ceph-mon[33213]: pgmap v461: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 949 op/s; 10936/252237 objects degraded (4.336%); 67451/252237 objects misplaced (26.741%); 36 MiB/s, 578 objects/s recovering 2024-02-05T07:28:04.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:04 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 10936/252237 objects degraded (4.336%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:04.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:04.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:04.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:04.983 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:05.052 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:05.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:05.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:05.182 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:05.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:05.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:05.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:05.326 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:05.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:05.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:05.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:05.457 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:05.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:05.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:05 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:28:05.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:05 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:28:05.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:05 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:28:05.505 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:05 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:28:05.505 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:05 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:28:05.505 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:05 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:28:05.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:05.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:05.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:05.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:05.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:05.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:05.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:05 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:28:05.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:05 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:28:05.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:05 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:28:05.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:05.872 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:05.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:05.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:05.882 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:28:05.923 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:28:05.963 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:28:05.975 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:05.999 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:28:06.008 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:06.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:06.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:06.035 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:28:06.072 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:28:06.108 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:28:06.109 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:28:06.109 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:28:06.109 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:28:06.109 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:28:06.109 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:28:06.109 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:28:06.109 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-02-05T07:28:06.109 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:28:06.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:06.135 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:06.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:06.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:28:06.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:06.270 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:06.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:06.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:06.372 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:06.404 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:06.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:06.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:06.506 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:06.547 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:06.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:06.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:06.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:06.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:06 smithi007 ceph-mon[29098]: pgmap v462: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 755 op/s; 10936/252237 objects degraded (4.336%); 67451/252237 objects misplaced (26.741%); 29 MiB/s, 460 objects/s recovering 2024-02-05T07:28:06.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:06 smithi007 ceph-mon[29098]: osdmap e287: 8 total, 7 up, 7 in 2024-02-05T07:28:06.666 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:06 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2461938104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:28:06.667 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:06 smithi007 ceph-mon[37158]: pgmap v462: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 755 op/s; 10936/252237 objects degraded (4.336%); 67451/252237 objects misplaced (26.741%); 29 MiB/s, 460 objects/s recovering 2024-02-05T07:28:06.667 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:06 smithi007 ceph-mon[37158]: osdmap e287: 8 total, 7 up, 7 in 2024-02-05T07:28:06.667 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:06 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2461938104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:28:06.716 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:06.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:06 smithi090 ceph-mon[33213]: pgmap v462: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 755 op/s; 10936/252237 objects degraded (4.336%); 67451/252237 objects misplaced (26.741%); 29 MiB/s, 460 objects/s recovering 2024-02-05T07:28:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:06 smithi090 ceph-mon[33213]: osdmap e287: 8 total, 7 up, 7 in 2024-02-05T07:28:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:06 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2461938104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:28:06.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:06.844 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:06.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:06.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:07.062 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:07.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:07.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:07.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:07.194 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:07.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:07.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:07.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:07.332 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:07.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:07.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:07.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:07.460 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:07.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:07.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:07.507 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T07:28:07.508 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T07:28:07.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:07.592 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:07.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:07.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:07.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:07 smithi007 ceph-mon[29098]: osdmap e288: 8 total, 7 up, 7 in 2024-02-05T07:28:07.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:07 smithi007 ceph-mon[37158]: osdmap e288: 8 total, 7 up, 7 in 2024-02-05T07:28:07.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:07.724 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:07.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:07.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:07.731 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:28:07.731 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-02-05T07:28:07.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-02-05T07:28:07.731 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:28:07.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.731 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75590 7.19 7:9952febd:::benchmark_data_smithi007_77086_object75589:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T07:28:07.731 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:44.134010+0000", 2024-02-05T07:28:07.731 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 382.773567453, 2024-02-05T07:28:07.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5177792879999998, 2024-02-05T07:28:07.731 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.731 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:28:07.731 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.732 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.732 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.732 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75590 2024-02-05T07:28:07.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.732 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.732 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134010+0000", 2024-02-05T07:28:07.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.732 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134010+0000", 2024-02-05T07:28:07.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.733 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134012+0000", 2024-02-05T07:28:07.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8339999999999999e-06 2024-02-05T07:28:07.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.733 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134038+0000", 2024-02-05T07:28:07.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.597e-05 2024-02-05T07:28:07.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134049+0000", 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0908e-05 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134060+0000", 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1168e-05 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135332+0000", 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012723949999999999 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135339+0000", 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1609999999999997e-06 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.576532+0000", 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.44119245 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.577277+0000", 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00074551000000000001 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.651542+0000", 2024-02-05T07:28:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.074265268999999995 2024-02-05T07:28:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.651789+0000", 2024-02-05T07:28:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00024662300000000001 2024-02-05T07:28:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75596 7.16 7:6e6f9151:::benchmark_data_smithi007_77086_object75595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T07:28:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:44.134788+0000", 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 382.77278919100002, 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4751571809999999, 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75596 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134788+0000", 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134788+0000", 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134789+0000", 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8700000000000005e-07 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134815+0000", 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6341000000000001e-05 2024-02-05T07:28:07.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134828+0000", 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2418999999999999e-05 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134836+0000", 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6640000000000004e-06 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134949+0000", 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113095 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134961+0000", 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1243000000000001e-05 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.487398+0000", 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3524372200000001 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.493549+0000", 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0061515739999999999 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.609843+0000", 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.116293277 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.609945+0000", 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000102661 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75610 7.14 7:2f8487fd:::benchmark_data_smithi007_77086_object75609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:44.134855+0000", 2024-02-05T07:28:07.741 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 382.77272214599998, 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.518808038, 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75610 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134855+0000", 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134855+0000", 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134856+0000", 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1900000000000003e-07 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134879+0000", 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3056000000000001e-05 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134889+0000", 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0370000000000001e-05 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134896+0000", 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0260000000000002e-06 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134995+0000", 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8620000000000001e-05 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135002+0000", 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7100000000000007e-06 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.500775+0000", 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.365772905 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.500958+0000", 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018317 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,3,5", 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.501145+0000", 2024-02-05T07:28:07.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000186088 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.651737+0000", 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.150592541 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.651825+0000", 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8287000000000004e-05 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.651878+0000", 2024-02-05T07:28:07.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2494000000000003e-05 2024-02-05T07:28:07.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.651930+0000", 2024-02-05T07:28:07.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1625999999999997e-05 2024-02-05T07:28:07.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.651993+0000", 2024-02-05T07:28:07.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3958e-05 2024-02-05T07:28:07.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.653663+0000", 2024-02-05T07:28:07.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001669668 2024-02-05T07:28:07.748 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.748 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.748 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.748 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75611 7.17 7:eaee9290:::benchmark_data_smithi007_77086_object75610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T07:28:07.748 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:44.134919+0000", 2024-02-05T07:28:07.748 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 382.77265855299999, 2024-02-05T07:28:07.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5402228170000001, 2024-02-05T07:28:07.748 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.748 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.748 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75611 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134919+0000", 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134919+0000", 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134919+0000", 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1799999999999995e-07 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134951+0000", 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1977999999999998e-05 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134959+0000", 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.272e-06 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134965+0000", 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3099999999999997e-06 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135017+0000", 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2383000000000003e-05 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135028+0000", 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0305e-05 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.501395+0000", 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3663675230000001 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.501531+0000", 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013617800000000001 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5,7", 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.501704+0000", 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017320700000000001 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.671123+0000", 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16941824899999999 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.671316+0000", 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019292300000000001 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.671366+0000", 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0813000000000003e-05 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.672728+0000", 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0013617830000000001 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.673117+0000", 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00038867600000000002 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.675142+0000", 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020245990000000002 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75620 7.14 7:2cf872c6:::benchmark_data_smithi007_77086_object75619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:44.134981+0000", 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 382.77259577400002, 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5372422220000002, 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75620 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134981+0000", 2024-02-05T07:28:07.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134981+0000", 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134982+0000", 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.51e-07 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135007+0000", 2024-02-05T07:28:07.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5318999999999999e-05 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135017+0000", 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2459999999999992e-06 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135024+0000", 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3000000000000004e-06 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135178+0000", 2024-02-05T07:28:07.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015381300000000001 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135187+0000", 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1700000000000003e-06 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.653853+0000", 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5186663070000002 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.654024+0000", 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000171032 2024-02-05T07:28:07.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,3,5", 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.654215+0000", 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019113299999999999 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.667211+0000", 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012995744 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.667253+0000", 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1900999999999997e-05 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.667304+0000", 2024-02-05T07:28:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1031e-05 2024-02-05T07:28:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.670356+0000", 2024-02-05T07:28:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030519499999999999 2024-02-05T07:28:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.670417+0000", 2024-02-05T07:28:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1375000000000001e-05 2024-02-05T07:28:07.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.672224+0000", 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00180635 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75627 7.14 7:2edca002:::benchmark_data_smithi007_77086_object75626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:44.135054+0000", 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 382.77252329999999, 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5390644570000001, 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.761 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.762 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.762 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.762 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.762 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75627 2024-02-05T07:28:07.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.762 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135054+0000", 2024-02-05T07:28:07.762 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135054+0000", 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135055+0000", 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0500000000000003e-07 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135079+0000", 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4865999999999999e-05 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135087+0000", 2024-02-05T07:28:07.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8210000000000002e-06 2024-02-05T07:28:07.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.764 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135094+0000", 2024-02-05T07:28:07.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8549999999999996e-06 2024-02-05T07:28:07.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.764 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135217+0000", 2024-02-05T07:28:07.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000123153 2024-02-05T07:28:07.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135224+0000", 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1250000000000004e-06 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.655529+0000", 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5203043040000002 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.655679+0000", 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000150385 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,3,5", 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.655847+0000", 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016762100000000001 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.667268+0000", 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011421205 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.667347+0000", 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9171999999999993e-05 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.672275+0000", 2024-02-05T07:28:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0049282809999999996 2024-02-05T07:28:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.672325+0000", 2024-02-05T07:28:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9632000000000002e-05 2024-02-05T07:28:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.672384+0000", 2024-02-05T07:28:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8656000000000002e-05 2024-02-05T07:28:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.768 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.674118+0000", 2024-02-05T07:28:07.768 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017347759999999999 2024-02-05T07:28:07.768 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.768 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.768 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.768 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.768 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.768 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75638 7.16 7:6ca660ac:::benchmark_data_smithi007_77086_object75637:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T07:28:07.768 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:44.135103+0000", 2024-02-05T07:28:07.768 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 382.77247400800002, 2024-02-05T07:28:07.768 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4970371789999999, 2024-02-05T07:28:07.768 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.768 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.768 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75638 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135103+0000", 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135103+0000", 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135104+0000", 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.89e-07 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135126+0000", 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.249e-05 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135133+0000", 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.353e-06 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135139+0000", 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9290000000000003e-06 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135467+0000", 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032830799999999997 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135479+0000", 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2478e-05 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.614856+0000", 2024-02-05T07:28:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4793767609999999 2024-02-05T07:28:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.615035+0000", 2024-02-05T07:28:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000179068 2024-02-05T07:28:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,7", 2024-02-05T07:28:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.615204+0000", 2024-02-05T07:28:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016881700000000001 2024-02-05T07:28:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.620046+0000", 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0048419509999999997 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.622377+0000", 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023309720000000002 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.629760+0000", 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0073832960000000001 2024-02-05T07:28:07.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.629882+0000", 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012164599999999999 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.632140+0000", 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0022586210000000002 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.774 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:192556 7.1f 7:f96b5564:::benchmark_data_smithi007_77086_object87961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:28:07.775 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:20.588439+0000", 2024-02-05T07:28:07.775 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 226.319138106, 2024-02-05T07:28:07.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0030279879999999, 2024-02-05T07:28:07.775 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.775 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.775 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.775 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.775 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.775 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 192556 2024-02-05T07:28:07.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.775 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.588439+0000", 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.588439+0000", 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.588440+0000", 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1230000000000001e-06 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.588445+0000", 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3420000000000001e-06 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.588452+0000", 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4150000000000002e-06 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.588458+0000", 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.409e-06 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.599325+0000", 2024-02-05T07:28:07.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010866506 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.599333+0000", 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6830000000000008e-06 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.524865+0000", 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.925532468 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.524879+0000", 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3448999999999999e-05 2024-02-05T07:28:07.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.779 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.779 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.586571+0000", 2024-02-05T07:28:07.779 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0616921969999999 2024-02-05T07:28:07.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.779 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.779 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.586656+0000", 2024-02-05T07:28:07.779 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5366999999999998e-05 2024-02-05T07:28:07.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.779 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4,5,6,7", 2024-02-05T07:28:07.779 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.586916+0000", 2024-02-05T07:28:07.779 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026004799999999997 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.589644+0000", 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0027280709999999999 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.590801+0000", 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0011563649999999999 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.590865+0000", 2024-02-05T07:28:07.780 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4060999999999995e-05 2024-02-05T07:28:07.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.781 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.781 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591169+0000", 2024-02-05T07:28:07.781 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030400499999999999 2024-02-05T07:28:07.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.781 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.781 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591237+0000", 2024-02-05T07:28:07.783 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8666999999999998e-05 2024-02-05T07:28:07.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591300+0000", 2024-02-05T07:28:07.783 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2621999999999998e-05 2024-02-05T07:28:07.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591348+0000", 2024-02-05T07:28:07.783 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8140999999999999e-05 2024-02-05T07:28:07.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.784 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591467+0000", 2024-02-05T07:28:07.784 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000119049 2024-02-05T07:28:07.784 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.784 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.784 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.784 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:192562 7.1f 7:fbe7dd9e:::benchmark_data_smithi007_77086_object87967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:28:07.784 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:20.593662+0000", 2024-02-05T07:28:07.784 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 226.31391493800001, 2024-02-05T07:28:07.784 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.998197354, 2024-02-05T07:28:07.784 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.784 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.784 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 192562 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.593662+0000", 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.593662+0000", 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.593664+0000", 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5409999999999999e-06 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.593669+0000", 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7029999999999997e-06 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.593680+0000", 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1046999999999999e-05 2024-02-05T07:28:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.593687+0000", 2024-02-05T07:28:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6109999999999996e-06 2024-02-05T07:28:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.599354+0000", 2024-02-05T07:28:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0056668609999999996 2024-02-05T07:28:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.599360+0000", 2024-02-05T07:28:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5289999999999999e-06 2024-02-05T07:28:07.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.526155+0000", 2024-02-05T07:28:07.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.92679503699999999 2024-02-05T07:28:07.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.526166+0000", 2024-02-05T07:28:07.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1715e-05 2024-02-05T07:28:07.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.588257+0000", 2024-02-05T07:28:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0620901789999999 2024-02-05T07:28:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.588420+0000", 2024-02-05T07:28:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016377399999999999 2024-02-05T07:28:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4,5,6,7", 2024-02-05T07:28:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.588741+0000", 2024-02-05T07:28:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032104600000000002 2024-02-05T07:28:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591105+0000", 2024-02-05T07:28:07.790 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023635219999999998 2024-02-05T07:28:07.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.790 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591505+0000", 2024-02-05T07:28:07.790 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00040054100000000003 2024-02-05T07:28:07.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.790 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591612+0000", 2024-02-05T07:28:07.790 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000106741 2024-02-05T07:28:07.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591650+0000", 2024-02-05T07:28:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7985000000000003e-05 2024-02-05T07:28:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591684+0000", 2024-02-05T07:28:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3581999999999998e-05 2024-02-05T07:28:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591717+0000", 2024-02-05T07:28:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2901000000000002e-05 2024-02-05T07:28:07.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591752+0000", 2024-02-05T07:28:07.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.561e-05 2024-02-05T07:28:07.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591860+0000", 2024-02-05T07:28:07.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000107429 2024-02-05T07:28:07.792 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.792 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.792 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:192577 7.1f 7:fcec0b30:::benchmark_data_smithi007_77086_object87982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:20.610517+0000", 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 226.297060402, 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9817076330000001, 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 192577 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.793 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610517+0000", 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610517+0000", 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610518+0000", 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.119e-06 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.794 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610528+0000", 2024-02-05T07:28:07.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9350000000000006e-06 2024-02-05T07:28:07.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.795 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610540+0000", 2024-02-05T07:28:07.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1599e-05 2024-02-05T07:28:07.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.795 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610548+0000", 2024-02-05T07:28:07.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0120000000000005e-06 2024-02-05T07:28:07.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.796 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610620+0000", 2024-02-05T07:28:07.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2121999999999998e-05 2024-02-05T07:28:07.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.796 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.796 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610628+0000", 2024-02-05T07:28:07.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1699999999999997e-06 2024-02-05T07:28:07.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.796 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.796 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.526203+0000", 2024-02-05T07:28:07.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.91557498599999998 2024-02-05T07:28:07.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.526211+0000", 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0339999999999997e-06 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.589684+0000", 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.063473584 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.589758+0000", 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3858999999999999e-05 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.798 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4,5,6,7", 2024-02-05T07:28:07.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.589945+0000", 2024-02-05T07:28:07.798 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018713200000000001 2024-02-05T07:28:07.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.798 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591254+0000", 2024-02-05T07:28:07.798 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00130828 2024-02-05T07:28:07.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.798 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591915+0000", 2024-02-05T07:28:07.798 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00066152400000000003 2024-02-05T07:28:07.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.591954+0000", 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8529000000000003e-05 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.592028+0000", 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4333999999999997e-05 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.592064+0000", 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5840000000000002e-05 2024-02-05T07:28:07.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.800 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.800 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.592098+0000", 2024-02-05T07:28:07.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4511000000000002e-05 2024-02-05T07:28:07.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.800 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.800 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.592133+0000", 2024-02-05T07:28:07.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4848000000000002e-05 2024-02-05T07:28:07.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.800 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.800 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.592224+0000", 2024-02-05T07:28:07.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1215e-05 2024-02-05T07:28:07.801 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.801 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.801 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.801 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.801 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:192578 7.1f 7:fd264c35:::benchmark_data_smithi007_77086_object87983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:28:07.801 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:20.610564+0000", 2024-02-05T07:28:07.801 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 226.29701309800001, 2024-02-05T07:28:07.801 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.989701841, 2024-02-05T07:28:07.801 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.801 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.801 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.801 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.801 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.802 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 192578 2024-02-05T07:28:07.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.802 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.802 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.802 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610564+0000", 2024-02-05T07:28:07.802 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.802 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.802 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610564+0000", 2024-02-05T07:28:07.802 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.803 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.803 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610565+0000", 2024-02-05T07:28:07.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9100000000000004e-07 2024-02-05T07:28:07.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.803 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.803 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610567+0000", 2024-02-05T07:28:07.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.576e-06 2024-02-05T07:28:07.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.803 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.803 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610573+0000", 2024-02-05T07:28:07.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8200000000000002e-06 2024-02-05T07:28:07.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.804 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.804 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610579+0000", 2024-02-05T07:28:07.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5369999999999997e-06 2024-02-05T07:28:07.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.804 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.804 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610649+0000", 2024-02-05T07:28:07.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0722000000000005e-05 2024-02-05T07:28:07.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.804 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.804 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.610655+0000", 2024-02-05T07:28:07.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9529999999999996e-06 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.526238+0000", 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.91558293400000001 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.526245+0000", 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0060000000000003e-06 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.592268+0000", 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.066023033 2024-02-05T07:28:07.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.592390+0000", 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121502 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4,5,6,7", 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.592525+0000", 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013515099999999999 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.595636+0000", 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0031114010000000002 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.595678+0000", 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1653000000000003e-05 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.595778+0000", 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000100046 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.595830+0000", 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2227e-05 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.808 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.808 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.599896+0000", 2024-02-05T07:28:07.808 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0040659090000000004 2024-02-05T07:28:07.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.808 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.808 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.600020+0000", 2024-02-05T07:28:07.808 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012383699999999999 2024-02-05T07:28:07.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.808 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.808 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.600088+0000", 2024-02-05T07:28:07.808 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7711000000000003e-05 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.600266+0000", 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017833200000000001 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:192601 7.1f 7:fa13395f:::benchmark_data_smithi007_77086_object88006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:20.635775+0000", 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 226.27180179199999, 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9650190620000001, 2024-02-05T07:28:07.809 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 192601 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.635775+0000", 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.635775+0000", 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.635777+0000", 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.291e-06 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.635781+0000", 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6199999999999998e-06 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.635791+0000", 2024-02-05T07:28:07.811 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7119999999999995e-06 2024-02-05T07:28:07.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.635797+0000", 2024-02-05T07:28:07.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8470000000000001e-06 2024-02-05T07:28:07.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.635845+0000", 2024-02-05T07:28:07.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8149999999999998e-05 2024-02-05T07:28:07.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.635854+0000", 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2e-06 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.526272+0000", 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.89041812200000003 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.526279+0000", 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.968e-06 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.813 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.593385+0000", 2024-02-05T07:28:07.814 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0671054250000001 2024-02-05T07:28:07.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.814 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.814 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.593507+0000", 2024-02-05T07:28:07.814 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000122366 2024-02-05T07:28:07.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.814 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4,5,6,7", 2024-02-05T07:28:07.814 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.593692+0000", 2024-02-05T07:28:07.814 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018456700000000001 2024-02-05T07:28:07.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.814 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.595738+0000", 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020465629999999999 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.595756+0000", 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7490999999999999e-05 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.599971+0000", 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0042150290000000003 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.816 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.600371+0000", 2024-02-05T07:28:07.816 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00039996299999999998 2024-02-05T07:28:07.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.816 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.816 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.816 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.600499+0000", 2024-02-05T07:28:07.816 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000128064 2024-02-05T07:28:07.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.816 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.816 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.816 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.600621+0000", 2024-02-05T07:28:07.816 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012238599999999999 2024-02-05T07:28:07.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.816 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.817 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.817 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.600670+0000", 2024-02-05T07:28:07.817 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8319999999999998e-05 2024-02-05T07:28:07.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.817 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.817 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.600795+0000", 2024-02-05T07:28:07.817 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012497800000000001 2024-02-05T07:28:07.817 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.817 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.817 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.817 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:192627 7.1f 7:fb110e6b:::benchmark_data_smithi007_77086_object88032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:20.653564+0000", 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 226.25401315799999, 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.947703628, 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 192627 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.653564+0000", 2024-02-05T07:28:07.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.653564+0000", 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.653565+0000", 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.06e-06 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.653570+0000", 2024-02-05T07:28:07.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9339999999999997e-06 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.653581+0000", 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0464e-05 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.653589+0000", 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0360000000000006e-06 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.657175+0000", 2024-02-05T07:28:07.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0035866460000000002 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.657186+0000", 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1083999999999999e-05 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.526306+0000", 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.86912005699999995 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.526313+0000", 2024-02-05T07:28:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7399999999999998e-06 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.594712+0000", 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.068399257 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.594793+0000", 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0718999999999997e-05 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4,5,6,7", 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.594996+0000", 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020318199999999999 2024-02-05T07:28:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.595838+0000", 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00084207599999999998 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.599823+0000", 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039843750000000001 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.600554+0000", 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00073089100000000001 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.824 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.824 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.600884+0000", 2024-02-05T07:28:07.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00033028400000000002 2024-02-05T07:28:07.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.824 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.824 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.601024+0000", 2024-02-05T07:28:07.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000140161 2024-02-05T07:28:07.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.824 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.824 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.601110+0000", 2024-02-05T07:28:07.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5999000000000001e-05 2024-02-05T07:28:07.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.601147+0000", 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7177e-05 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.601268+0000", 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000120486 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:192675 7.1f 7:fac13596:::benchmark_data_smithi007_77086_object88080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:28:07.825 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:20.692069+0000", 2024-02-05T07:28:07.826 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 226.21550800099999, 2024-02-05T07:28:07.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9097871180000001, 2024-02-05T07:28:07.826 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.826 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.826 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.826 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.826 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.826 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 192675 2024-02-05T07:28:07.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.826 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.826 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.692069+0000", 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.692069+0000", 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.692071+0000", 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3120000000000001e-06 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.692075+0000", 2024-02-05T07:28:07.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7770000000000002e-06 2024-02-05T07:28:07.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.692086+0000", 2024-02-05T07:28:07.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0462000000000001e-05 2024-02-05T07:28:07.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.692094+0000", 2024-02-05T07:28:07.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1340000000000004e-06 2024-02-05T07:28:07.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.692153+0000", 2024-02-05T07:28:07.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8903999999999997e-05 2024-02-05T07:28:07.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.692164+0000", 2024-02-05T07:28:07.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0812000000000001e-05 2024-02-05T07:28:07.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.526338+0000", 2024-02-05T07:28:07.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.83417427899999996 2024-02-05T07:28:07.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.526344+0000", 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9379999999999997e-06 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.595875+0000", 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.069531499 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.596018+0000", 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014296799999999999 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4,5,6,7", 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.596221+0000", 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020250499999999999 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.600447+0000", 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0042256760000000003 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.600580+0000", 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013361300000000001 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.831 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.601067+0000", 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00048685500000000001 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.601322+0000", 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00025485500000000001 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.601371+0000", 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9421e-05 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.601558+0000", 2024-02-05T07:28:07.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018667900000000001 2024-02-05T07:28:07.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.833 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.601711+0000", 2024-02-05T07:28:07.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000153141 2024-02-05T07:28:07.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.833 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.601856+0000", 2024-02-05T07:28:07.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014528800000000001 2024-02-05T07:28:07.833 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.833 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.834 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.834 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:192680 7.1f 7:f8553875:::benchmark_data_smithi007_77086_object88085:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:28:07.834 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:20.694987+0000", 2024-02-05T07:28:07.834 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 226.21259029399999, 2024-02-05T07:28:07.834 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9073627, 2024-02-05T07:28:07.834 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.834 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.834 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.834 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:07.834 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:07.834 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 192680 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.694987+0000", 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.694987+0000", 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.694988+0000", 2024-02-05T07:28:07.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8299999999999995e-07 2024-02-05T07:28:07.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.694992+0000", 2024-02-05T07:28:07.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3869999999999997e-06 2024-02-05T07:28:07.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.695003+0000", 2024-02-05T07:28:07.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0641e-05 2024-02-05T07:28:07.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.695011+0000", 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.588e-06 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.697276+0000", 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0022657509999999999 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.697287+0000", 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.046e-05 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.526372+0000", 2024-02-05T07:28:07.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.82908526999999999 2024-02-05T07:28:07.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:21.526378+0000", 2024-02-05T07:28:07.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1700000000000002e-06 2024-02-05T07:28:07.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.597579+0000", 2024-02-05T07:28:07.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0712010510000001 2024-02-05T07:28:07.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.597762+0000", 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000182716 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4,5,6,7", 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.598092+0000", 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00033010100000000002 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.600585+0000", 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002493328 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.600970+0000", 2024-02-05T07:28:07.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00038463400000000002 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.601492+0000", 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000522344 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.601918+0000", 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00042580199999999997 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.601980+0000", 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1378999999999996e-05 2024-02-05T07:28:07.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.602091+0000", 2024-02-05T07:28:07.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011186100000000001 2024-02-05T07:28:07.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.602191+0000", 2024-02-05T07:28:07.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9359999999999997e-05 2024-02-05T07:28:07.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:22.602350+0000", 2024-02-05T07:28:07.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015887400000000001 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151878 8.18 8:1b9df18c:::benchmark_data_smithi007_105755_object15568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:15.617665+0000", 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 51.289912325000003, 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.218317409, 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151878 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.842 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.617665+0000", 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.617665+0000", 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.617667+0000", 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.567e-06 2024-02-05T07:28:07.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.617673+0000", 2024-02-05T07:28:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2530000000000001e-06 2024-02-05T07:28:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.617684+0000", 2024-02-05T07:28:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1e-05 2024-02-05T07:28:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.617692+0000", 2024-02-05T07:28:07.844 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9240000000000007e-06 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.618154+0000", 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00046236600000000002 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.618283+0000", 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012861 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,5,7", 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.618581+0000", 2024-02-05T07:28:07.845 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029788500000000001 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.619902+0000", 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001321521 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621361+0000", 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001459019 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.704079+0000", 2024-02-05T07:28:07.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.082718268999999997 2024-02-05T07:28:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.736869+0000", 2024-02-05T07:28:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.032790080999999999 2024-02-05T07:28:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.736883+0000", 2024-02-05T07:28:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3345e-05 2024-02-05T07:28:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.847 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.835934+0000", 2024-02-05T07:28:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.099051094 2024-02-05T07:28:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.835982+0000", 2024-02-05T07:28:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8474999999999999e-05 2024-02-05T07:28:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151885 8.18 8:18501e19:::benchmark_data_smithi007_105755_object15575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:28:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:15.621172+0000", 2024-02-05T07:28:07.848 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 51.286405672000001, 2024-02-05T07:28:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.375381483, 2024-02-05T07:28:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:28:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151885 2024-02-05T07:28:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621172+0000", 2024-02-05T07:28:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621172+0000", 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621173+0000", 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3069999999999999e-06 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621211+0000", 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7966000000000002e-05 2024-02-05T07:28:07.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621228+0000", 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.685e-05 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621236+0000", 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3129999999999999e-06 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621393+0000", 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015687599999999999 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621526+0000", 2024-02-05T07:28:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013271100000000001 2024-02-05T07:28:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,5,7", 2024-02-05T07:28:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.702361+0000", 2024-02-05T07:28:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.080835844000000004 2024-02-05T07:28:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.705711+0000", 2024-02-05T07:28:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033500320000000002 2024-02-05T07:28:07.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.736929+0000", 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.03121765 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.736939+0000", 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7610000000000002e-06 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.888509+0000", 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1515699399999999 2024-02-05T07:28:07.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for missing object", 2024-02-05T07:28:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.888700+0000", 2024-02-05T07:28:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000190903 2024-02-05T07:28:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.996413+0000", 2024-02-05T07:28:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10771344099999999 2024-02-05T07:28:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.996553+0000", 2024-02-05T07:28:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000139889 2024-02-05T07:28:07.855 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.855 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.855 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.855 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.855 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151889 8.9 8:97173e06:::benchmark_data_smithi007_105755_object15579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:28:07.855 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:15.708625+0000", 2024-02-05T07:28:07.855 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 51.198952026999997, 2024-02-05T07:28:07.855 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1227700549999999, 2024-02-05T07:28:07.855 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.855 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.855 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.855 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:07.855 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:07.856 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151889 2024-02-05T07:28:07.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.856 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.856 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.856 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708625+0000", 2024-02-05T07:28:07.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.856 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.856 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708625+0000", 2024-02-05T07:28:07.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708627+0000", 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6530000000000001e-06 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708630+0000", 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7609999999999999e-06 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708638+0000", 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3839999999999997e-06 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708645+0000", 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9190000000000002e-06 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.721400+0000", 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012754557999999999 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.721411+0000", 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1426999999999999e-05 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.826103+0000", 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.104692137 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.826226+0000", 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012312 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,5", 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.826614+0000", 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00038772999999999998 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.828167+0000", 2024-02-05T07:28:07.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001553432 2024-02-05T07:28:07.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.860 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.860 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.828629+0000", 2024-02-05T07:28:07.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00046157499999999998 2024-02-05T07:28:07.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.860 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.860 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.831136+0000", 2024-02-05T07:28:07.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025074310000000001 2024-02-05T07:28:07.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.860 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.860 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.831256+0000", 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000119261 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.831395+0000", 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000139667 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151899 8.16 8:69ded5dd:::benchmark_data_smithi007_105755_object15589:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:15.708654+0000", 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 51.198923022999999, 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1801824700000001, 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.861 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151899 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708654+0000", 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708654+0000", 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708655+0000", 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5299999999999999e-07 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708657+0000", 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5979999999999999e-06 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708665+0000", 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4510000000000003e-06 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708672+0000", 2024-02-05T07:28:07.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8510000000000001e-06 2024-02-05T07:28:07.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.728610+0000", 2024-02-05T07:28:07.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.019938852 2024-02-05T07:28:07.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.728624+0000", 2024-02-05T07:28:07.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.367e-05 2024-02-05T07:28:07.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.783950+0000", 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0553263230000001 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.784094+0000", 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000143125 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,3,7", 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.784438+0000", 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000344684 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.787789+0000", 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003350853 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.794404+0000", 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0066144910000000001 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.796128+0000", 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00172437 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.888636+0000", 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.092508198999999999 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.888696+0000", 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9998999999999999e-05 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.888837+0000", 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014055099999999999 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.868 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151911 8.16 8:6ed6f866:::benchmark_data_smithi007_105755_object15601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-02-05T07:28:07.868 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:15.723119+0000", 2024-02-05T07:28:07.868 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 51.184457895000001, 2024-02-05T07:28:07.868 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1659391960000001, 2024-02-05T07:28:07.868 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:07.868 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:07.868 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:07.868 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:07.868 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:07.868 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151911 2024-02-05T07:28:07.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.868 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.723119+0000", 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.723119+0000", 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.723121+0000", 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.395e-06 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.723125+0000", 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9469999999999996e-06 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.723134+0000", 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3139999999999993e-06 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.723143+0000", 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7320000000000004e-06 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.728656+0000", 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0055137060000000002 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.728662+0000", 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9109999999999998e-06 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.785892+0000", 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.057230085 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:07.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.786022+0000", 2024-02-05T07:28:07.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012957499999999999 2024-02-05T07:28:07.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,3,7", 2024-02-05T07:28:07.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.786266+0000", 2024-02-05T07:28:07.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00024366200000000001 2024-02-05T07:28:07.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:07.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.787796+0000", 2024-02-05T07:28:07.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001529835 2024-02-05T07:28:07.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.794445+0000", 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0066497789999999998 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.796168+0000", 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00172317 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.888881+0000", 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.092712185000000003 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.888938+0000", 2024-02-05T07:28:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7420000000000003e-05 2024-02-05T07:28:07.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:07.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:07.874 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:07.874 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.889059+0000", 2024-02-05T07:28:07.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012048 2024-02-05T07:28:07.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.874 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:07.874 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:07.874 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:28:07.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:07.875 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:07 smithi090 ceph-mon[33213]: osdmap e288: 8 total, 7 up, 7 in 2024-02-05T07:28:07.906 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:07.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:07.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:08.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:08.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:08.039 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:08.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:08.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:08.182 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:08.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:08.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:08.314 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:08.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:08.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:08.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:08.458 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:08.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:08.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:08.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:08.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:08.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:08 smithi007 ceph-mon[29098]: pgmap v465: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 890 B/s rd, 0 B/s wr, 882 op/s; 9884/237033 objects degraded (4.170%); 61770/237033 objects misplaced (26.060%); 17 MiB/s, 268 objects/s recovering 2024-02-05T07:28:08.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:08 smithi007 ceph-mon[37158]: pgmap v465: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 890 B/s rd, 0 B/s wr, 882 op/s; 9884/237033 objects degraded (4.170%); 61770/237033 objects misplaced (26.060%); 17 MiB/s, 268 objects/s recovering 2024-02-05T07:28:08.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:08.746 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:08.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:08.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:08 smithi090 ceph-mon[33213]: pgmap v465: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 890 B/s rd, 0 B/s wr, 882 op/s; 9884/237033 objects degraded (4.170%); 61770/237033 objects misplaced (26.060%); 17 MiB/s, 268 objects/s recovering 2024-02-05T07:28:08.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:08.882 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:08.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:08.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:09.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:09.016 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:09.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:09.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:09.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:09.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:09.278 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:09.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:09.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:09.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:28:09.691 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:09 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 9884/237033 objects degraded (4.170%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:09.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:09 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 9884/237033 objects degraded (4.170%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:09.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:09.716 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:09.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:09 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 9884/237033 objects degraded (4.170%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:09.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:09.848 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:09.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:09.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T07:28:09.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:10.023 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:10.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:10.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:10.233 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:10.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:10.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:10.240 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:28:10.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:10.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:10.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:10.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:10.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:10.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:10.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:10.635 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:10.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:10 smithi007 ceph-mon[29098]: pgmap v466: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 758 B/s rd, 0 B/s wr, 751 op/s; 9884/237033 objects degraded (4.170%); 61770/237033 objects misplaced (26.060%); 14 MiB/s, 228 objects/s recovering 2024-02-05T07:28:10.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:10 smithi007 ceph-mon[37158]: pgmap v466: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 758 B/s rd, 0 B/s wr, 751 op/s; 9884/237033 objects degraded (4.170%); 61770/237033 objects misplaced (26.060%); 14 MiB/s, 228 objects/s recovering 2024-02-05T07:28:10.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:10.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:10.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:10.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:28:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:28:10.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:10 smithi090 ceph-mon[33213]: pgmap v466: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 97 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 758 B/s rd, 0 B/s wr, 751 op/s; 9884/237033 objects degraded (4.170%); 61770/237033 objects misplaced (26.060%); 14 MiB/s, 228 objects/s recovering 2024-02-05T07:28:10.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:10.838 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:10.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:10.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:10.940 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:10.980 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:10.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:10.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:11.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:11.113 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:28:11.139 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:11.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:11.150 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:28:11.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:11.279 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:11.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:11.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:11.381 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:11.423 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:11.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:11.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:11.473 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pg_num to 21 2024-02-05T07:28:11.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:11.531 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:11 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:11.531 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:11 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/172759073' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:28:11.531 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:11 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:28:11.531 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:11 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:11.531 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:11 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/172759073' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:28:11.531 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:11 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:28:11.553 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:11.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:11.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:11.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:11.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:11.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:11.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:11 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:11.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:11 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/172759073' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:28:11.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:11 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:28:11.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:11.838 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:11.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:11.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:11.970 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:11.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:12.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:12.112 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:12.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:12.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:12.271 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:12.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:12.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:12.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:12.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:12.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:12.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:12.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:12.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:12.546 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:12.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:12.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:12.674 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:12.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:12 smithi090 ceph-mon[33213]: pgmap v467: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 98 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 8780/223242 objects degraded (3.933%); 55467/223242 objects misplaced (24.846%); 35 MiB/s, 558 objects/s recovering 2024-02-05T07:28:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:12 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T07:28:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:12 smithi090 ceph-mon[33213]: osdmap e289: 8 total, 7 up, 7 in 2024-02-05T07:28:12.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:12.812 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:12.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:12.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:12.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:12 smithi007 ceph-mon[29098]: pgmap v467: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 98 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 8780/223242 objects degraded (3.933%); 55467/223242 objects misplaced (24.846%); 35 MiB/s, 558 objects/s recovering 2024-02-05T07:28:12.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:12 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T07:28:12.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:12 smithi007 ceph-mon[29098]: osdmap e289: 8 total, 7 up, 7 in 2024-02-05T07:28:12.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:12 smithi007 ceph-mon[37158]: pgmap v467: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 98 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 8780/223242 objects degraded (3.933%); 55467/223242 objects misplaced (24.846%); 35 MiB/s, 558 objects/s recovering 2024-02-05T07:28:12.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:12 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T07:28:12.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:12 smithi007 ceph-mon[37158]: osdmap e289: 8 total, 7 up, 7 in 2024-02-05T07:28:12.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:12.957 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:12.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:12.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:13.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:13.082 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:13.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:13.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:13.217 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:13.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:13.318 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:13.352 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:13.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:13.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:13.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:13.477 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:13.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:13.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:13.579 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:13.622 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:13.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:13.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:13.700 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:13 smithi007 ceph-mon[29098]: osdmap e290: 8 total, 7 up, 7 in 2024-02-05T07:28:13.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:13 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:28:13.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:13 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:28:13.701 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:28:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:28:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:28:13.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:13 smithi007 ceph-mon[37158]: osdmap e290: 8 total, 7 up, 7 in 2024-02-05T07:28:13.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:13 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:28:13.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:13 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:28:13.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:13.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:13.772 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:13.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:13.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:13 smithi090 ceph-mon[33213]: osdmap e290: 8 total, 7 up, 7 in 2024-02-05T07:28:13.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:13 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:28:13.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:13 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:28:13.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:13.903 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:13.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:13.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:14.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:14.031 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:14.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:14.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:14.160 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:14.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:14.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:14.288 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:28:14.306 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:14.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:14.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:14.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:14.436 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:14.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:14.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:14.582 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:14.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:14.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:14.684 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:14.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:14.726 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:14.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:14.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:14 smithi090 ceph-mon[33213]: pgmap v470: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 98 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 762 B/s rd, 0 B/s wr, 684 op/s; 8780/223242 objects degraded (3.933%); 55467/223242 objects misplaced (24.846%); 27 MiB/s, 436 objects/s recovering 2024-02-05T07:28:14.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:14 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T07:28:14.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:14 smithi090 ceph-mon[33213]: osdmap e291: 8 total, 7 up, 7 in 2024-02-05T07:28:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:14 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3004503793' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:28:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:14 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:28:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:14 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 8780/223242 objects degraded (3.933%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:14 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:28:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:14 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:28:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:14 smithi090 ceph-mon[33213]: osdmap e292: 8 total, 7 up, 7 in 2024-02-05T07:28:14.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:14.863 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:14.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:14.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[29098]: pgmap v470: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 98 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 762 B/s rd, 0 B/s wr, 684 op/s; 8780/223242 objects degraded (3.933%); 55467/223242 objects misplaced (24.846%); 27 MiB/s, 436 objects/s recovering 2024-02-05T07:28:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T07:28:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[29098]: osdmap e291: 8 total, 7 up, 7 in 2024-02-05T07:28:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3004503793' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:28:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:28:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 8780/223242 objects degraded (3.933%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:28:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:28:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[29098]: osdmap e292: 8 total, 7 up, 7 in 2024-02-05T07:28:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[37158]: pgmap v470: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 98 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 762 B/s rd, 0 B/s wr, 684 op/s; 8780/223242 objects degraded (3.933%); 55467/223242 objects misplaced (24.846%); 27 MiB/s, 436 objects/s recovering 2024-02-05T07:28:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T07:28:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[37158]: osdmap e291: 8 total, 7 up, 7 in 2024-02-05T07:28:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3004503793' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:28:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:28:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 8780/223242 objects degraded (3.933%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:28:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:28:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:14 smithi007 ceph-mon[37158]: osdmap e292: 8 total, 7 up, 7 in 2024-02-05T07:28:14.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:15.000 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:15.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:15.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:15.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:15.144 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:15.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:15.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:15.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:15.274 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:15.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:15.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:15.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:15.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:15.533 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:15.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:15.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:15.666 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:15.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:15.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:28:15.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:28:15.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:15 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:28:15.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:15 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T07:28:15.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:15 smithi007 ceph-mon[29098]: osdmap e293: 8 total, 7 up, 7 in 2024-02-05T07:28:15.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:28:15.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:28:15.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:15 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:28:15.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:15 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T07:28:15.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:15 smithi007 ceph-mon[37158]: osdmap e293: 8 total, 7 up, 7 in 2024-02-05T07:28:15.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:15.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:28:15.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:28:15.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:15 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:28:15.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:15 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T07:28:15.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:15 smithi090 ceph-mon[33213]: osdmap e293: 8 total, 7 up, 7 in 2024-02-05T07:28:15.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:15.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:15.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:15.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:15.956 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:15.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:16.057 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:16.086 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:16.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:16.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:16.188 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:16.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:16.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:16.330 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:16.377 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:16.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:16.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:16.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:16.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:16.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:16.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:16.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:16.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:16 smithi007 ceph-mon[29098]: pgmap v473: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 10 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 98 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 8780/223242 objects degraded (3.933%); 55467/223242 objects misplaced (24.846%) 2024-02-05T07:28:16.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:16 smithi007 ceph-mon[29098]: 1.1 scrub starts 2024-02-05T07:28:16.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:16 smithi007 ceph-mon[29098]: 1.1 scrub ok 2024-02-05T07:28:16.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:16 smithi007 ceph-mon[29098]: osdmap e294: 8 total, 7 up, 7 in 2024-02-05T07:28:16.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:16 smithi007 ceph-mon[37158]: pgmap v473: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 10 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 98 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 8780/223242 objects degraded (3.933%); 55467/223242 objects misplaced (24.846%) 2024-02-05T07:28:16.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:16 smithi007 ceph-mon[37158]: 1.1 scrub starts 2024-02-05T07:28:16.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:16 smithi007 ceph-mon[37158]: 1.1 scrub ok 2024-02-05T07:28:16.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:16 smithi007 ceph-mon[37158]: osdmap e294: 8 total, 7 up, 7 in 2024-02-05T07:28:16.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:16.691 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:16.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:16.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:16 smithi090 ceph-mon[33213]: pgmap v473: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 10 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 98 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 8780/223242 objects degraded (3.933%); 55467/223242 objects misplaced (24.846%) 2024-02-05T07:28:16.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:16 smithi090 ceph-mon[33213]: 1.1 scrub starts 2024-02-05T07:28:16.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:16 smithi090 ceph-mon[33213]: 1.1 scrub ok 2024-02-05T07:28:16.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:16 smithi090 ceph-mon[33213]: osdmap e294: 8 total, 7 up, 7 in 2024-02-05T07:28:16.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:16.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:16.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:16.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:16.990 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:16.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:17.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:17.130 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:17.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:17.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:17.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:17.256 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:17.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:17.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:17.257 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 7, 3] out_osds: [5] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-02-05T07:28:17.257 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-02-05T07:28:17.258 INFO:tasks.cephadm.osd.6:Restarting osd.6 (starting--it wasn't running)... 2024-02-05T07:28:17.258 DEBUG:teuthology.orchestra.run.smithi090:osd.6> sudo journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.6.service 2024-02-05T07:28:17.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl start ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.6 2024-02-05T07:28:17.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:17.417 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:17.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:17.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:17.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:28:17.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:17.549 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:28:17.549 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:28:17.549 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:28:17.549 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:28:17.549 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:28:17.566 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:28:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:17.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:17.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:17.607 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:28:17.607 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:17 smithi007 ceph-mon[29098]: 1.8 scrub starts 2024-02-05T07:28:17.608 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:17 smithi007 ceph-mon[29098]: 1.8 scrub ok 2024-02-05T07:28:17.608 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:17 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1191093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:28:17.608 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:17 smithi007 ceph-mon[29098]: osdmap e295: 8 total, 7 up, 7 in 2024-02-05T07:28:17.608 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:17 smithi007 ceph-mon[37158]: 1.8 scrub starts 2024-02-05T07:28:17.608 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:17 smithi007 ceph-mon[37158]: 1.8 scrub ok 2024-02-05T07:28:17.608 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:17 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1191093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:28:17.608 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:17 smithi007 ceph-mon[37158]: osdmap e295: 8 total, 7 up, 7 in 2024-02-05T07:28:17.609 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:28:17.609 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:28:17.609 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:28:17.609 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:28:17.609 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:28:17.609 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:28:17.609 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-02-05T07:28:17.609 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:28:17.651 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:17 smithi090 ceph-mon[33213]: 1.8 scrub starts 2024-02-05T07:28:17.651 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:17 smithi090 ceph-mon[33213]: 1.8 scrub ok 2024-02-05T07:28:17.651 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:17 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1191093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:28:17.651 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:17 smithi090 ceph-mon[33213]: osdmap e295: 8 total, 7 up, 7 in 2024-02-05T07:28:17.651 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:17 smithi090 systemd[1]: Starting Ceph osd.6 for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:28:17.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:17.724 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:17.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:17.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:17.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:17.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:17.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:17.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:17.979 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:17 smithi090 podman[99191]: 2024-02-05 07:28:17.555712794 +0000 UTC m=+0.024873687 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T07:28:17.980 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:17 smithi090 podman[99191]: 2024-02-05 07:28:17.707561007 +0000 UTC m=+0.176721885 container create 8ce85e50e5865818a9527856d0cd67354c4e1224261d0f030562822cd2fee4a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-02-05T07:28:17.980 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:17 smithi090 podman[99191]: 2024-02-05 07:28:17.977634234 +0000 UTC m=+0.446795107 container init 8ce85e50e5865818a9527856d0cd67354c4e1224261d0f030562822cd2fee4a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T07:28:17.994 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:17.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:17.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:18.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:18.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:18.136 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:18.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:18.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:18.268 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:18.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:18.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:18.274 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:17 smithi090 podman[99191]: 2024-02-05 07:28:17.983986241 +0000 UTC m=+0.453147116 container start 8ce85e50e5865818a9527856d0cd67354c4e1224261d0f030562822cd2fee4a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-activate, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-05T07:28:18.274 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 podman[99191]: 2024-02-05 07:28:18.086251236 +0000 UTC m=+0.555412114 container attach 8ce85e50e5865818a9527856d0cd67354c4e1224261d0f030562822cd2fee4a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-activate, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T07:28:18.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:18.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:18.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:18.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:18.462 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:28:18.462 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:28:18.504 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:18.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:18.550 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:18.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:18.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:18.681 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:18 smithi090 ceph-mon[33213]: pgmap v476: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 remapped+peering, 2 active+remapped+backfill_wait+backfill_toofull, 6 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 3 remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 13 active+undersized, 99 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 7577/209973 objects degraded (3.609%); 49531/209973 objects misplaced (23.589%); 43 MiB/s, 694 objects/s recovering 2024-02-05T07:28:18.681 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:18 smithi090 ceph-mon[33213]: osdmap e296: 8 total, 7 up, 7 in 2024-02-05T07:28:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:18.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:18.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:18.854 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:18.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:18 smithi007 ceph-mon[29098]: pgmap v476: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 remapped+peering, 2 active+remapped+backfill_wait+backfill_toofull, 6 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 3 remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 13 active+undersized, 99 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 7577/209973 objects degraded (3.609%); 49531/209973 objects misplaced (23.589%); 43 MiB/s, 694 objects/s recovering 2024-02-05T07:28:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:18 smithi007 ceph-mon[29098]: osdmap e296: 8 total, 7 up, 7 in 2024-02-05T07:28:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:18 smithi007 ceph-mon[37158]: pgmap v476: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 remapped+peering, 2 active+remapped+backfill_wait+backfill_toofull, 6 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 3 remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 13 active+undersized, 99 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 7577/209973 objects degraded (3.609%); 49531/209973 objects misplaced (23.589%); 43 MiB/s, 694 objects/s recovering 2024-02-05T07:28:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:18 smithi007 ceph-mon[37158]: osdmap e296: 8 total, 7 up, 7 in 2024-02-05T07:28:18.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T07:28:18.996 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T07:28:19.009 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:19.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:19.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:19.027 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-activate[99206]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-05T07:28:19.027 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-activate[99206]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-05T07:28:19.027 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-activate[99206]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-05T07:28:19.027 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-activate[99206]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-05T07:28:19.027 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-activate[99206]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-05T07:28:19.027 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-activate[99206]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-05T07:28:19.028 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-activate[99206]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-05T07:28:19.028 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 bash[99191]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-05T07:28:19.028 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 bash[99191]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-05T07:28:19.028 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 bash[99191]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-05T07:28:19.028 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 bash[99191]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-05T07:28:19.028 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 bash[99191]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-05T07:28:19.028 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 bash[99191]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-05T07:28:19.028 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 bash[99191]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-05T07:28:19.028 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:18 smithi090 podman[99395]: 2024-02-05 07:28:18.760412622 +0000 UTC m=+0.028662181 container died 8ce85e50e5865818a9527856d0cd67354c4e1224261d0f030562822cd2fee4a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-activate, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T07:28:19.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:19.151 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:19.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:19.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:19.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:19.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:19.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:19.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:19.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:19.430 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:19.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:19.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:19.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:19.632 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:19 smithi090 podman[99395]: 2024-02-05 07:28:19.36895088 +0000 UTC m=+0.637200437 container remove 8ce85e50e5865818a9527856d0cd67354c4e1224261d0f030562822cd2fee4a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-05T07:28:19.632 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:19 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 7577/209973 objects degraded (3.609%), 13 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:19.632 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:19 smithi090 ceph-mon[33213]: osdmap e297: 8 total, 7 up, 7 in 2024-02-05T07:28:19.634 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:19 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 7577/209973 objects degraded (3.609%), 13 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:19.634 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:19 smithi007 ceph-mon[37158]: osdmap e297: 8 total, 7 up, 7 in 2024-02-05T07:28:19.634 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:19 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 7577/209973 objects degraded (3.609%), 13 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:19.634 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:19 smithi007 ceph-mon[29098]: osdmap e297: 8 total, 7 up, 7 in 2024-02-05T07:28:19.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:19.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T07:28:19.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:19.705 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:19.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:19.733 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:28:19.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:19.831 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:19.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:19.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:19.926 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:19 smithi090 podman[99468]: 2024-02-05 07:28:19.535236164 +0000 UTC m=+0.025834299 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T07:28:19.926 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:19 smithi090 podman[99468]: 2024-02-05 07:28:19.662030484 +0000 UTC m=+0.152628617 container create d164ba3156345f5d9b1a905782af7988560137fefbd132c44879882d55f63b68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-05T07:28:19.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:19.965 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:19.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:20.066 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:20.096 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:20.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:20.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:20.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T07:28:20.177 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:19 smithi090 podman[99468]: 2024-02-05 07:28:19.92523158 +0000 UTC m=+0.415829718 container init d164ba3156345f5d9b1a905782af7988560137fefbd132c44879882d55f63b68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-05T07:28:20.177 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:19 smithi090 podman[99468]: 2024-02-05 07:28:19.928327905 +0000 UTC m=+0.418926039 container start d164ba3156345f5d9b1a905782af7988560137fefbd132c44879882d55f63b68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T07:28:20.177 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:19 smithi090 ceph-osd[99522]: -- 172.21.15.90:0/2064205055 <== mon.1 v2:172.21.15.90:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558e702b76c0 con 0x558e70288c00 2024-02-05T07:28:20.177 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:20 smithi090 bash[99468]: d164ba3156345f5d9b1a905782af7988560137fefbd132c44879882d55f63b68 2024-02-05T07:28:20.177 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:20 smithi090 systemd[1]: Started Ceph osd.6 for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:28:20.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:20.228 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:20.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:20.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:20.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:20.359 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:20.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:20.419 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:28:20.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:20.500 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:20.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:20.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:20.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:20.625 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:20.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:20.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:20.757 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:20.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:20.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:28:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:28:20.775 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:28:20.467+0000 7fb0fd0877c0 -1 Falling back to public interface 2024-02-05T07:28:20.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:20 smithi090 ceph-mon[33213]: pgmap v479: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 remapped+peering, 2 active+remapped+backfill_wait+backfill_toofull, 6 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 3 remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 13 active+undersized, 99 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 7577/209973 objects degraded (3.609%); 49531/209973 objects misplaced (23.589%); 43 MiB/s, 694 objects/s recovering 2024-02-05T07:28:20.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:20 smithi090 ceph-mon[33213]: osdmap e298: 8 total, 7 up, 7 in 2024-02-05T07:28:20.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:20.889 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:20.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:20 smithi007 ceph-mon[29098]: pgmap v479: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 remapped+peering, 2 active+remapped+backfill_wait+backfill_toofull, 6 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 3 remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 13 active+undersized, 99 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 7577/209973 objects degraded (3.609%); 49531/209973 objects misplaced (23.589%); 43 MiB/s, 694 objects/s recovering 2024-02-05T07:28:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:20 smithi007 ceph-mon[29098]: osdmap e298: 8 total, 7 up, 7 in 2024-02-05T07:28:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:20 smithi007 ceph-mon[37158]: pgmap v479: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 remapped+peering, 2 active+remapped+backfill_wait+backfill_toofull, 6 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 36 active+clean+remapped, 3 remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 13 active+undersized, 99 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 7577/209973 objects degraded (3.609%); 49531/209973 objects misplaced (23.589%); 43 MiB/s, 694 objects/s recovering 2024-02-05T07:28:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:20 smithi007 ceph-mon[37158]: osdmap e298: 8 total, 7 up, 7 in 2024-02-05T07:28:20.991 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:20.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:28:21.019 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:21.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:21.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:21.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:21.151 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:21.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:21.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:21.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:21.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:21.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:21.458 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:21.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:21.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:21.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:21.583 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:21.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:21.684 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:21.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:21.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:21.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:21.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:21.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:21 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:21.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:21 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:21.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:21.985 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:21.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:21.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:22.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:21 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:22.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:22.128 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:22.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:22.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:22.254 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:22.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:22.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:22.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:22.390 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:22.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:22.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:22.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:22.521 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:22.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:22.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:22.646 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:22.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:22.747 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:22.773 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:22.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:22.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:22.874 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:22.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:22 smithi007 ceph-mon[29098]: pgmap v482: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 41 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 105 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 5.0 KiB/s rd, 28 KiB/s wr, 1.21k op/s; 6677/195495 objects degraded (3.415%); 43959/195495 objects misplaced (22.486%); 33 MiB/s, 535 objects/s recovering 2024-02-05T07:28:22.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:22 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:28:22.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:22 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:28:22.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:22 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:22.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:22 smithi007 ceph-mon[29098]: osdmap e299: 8 total, 7 up, 7 in 2024-02-05T07:28:22.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:22 smithi007 ceph-mon[37158]: pgmap v482: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 41 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 105 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 5.0 KiB/s rd, 28 KiB/s wr, 1.21k op/s; 6677/195495 objects degraded (3.415%); 43959/195495 objects misplaced (22.486%); 33 MiB/s, 535 objects/s recovering 2024-02-05T07:28:22.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:22 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:28:22.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:22 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:28:22.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:22 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:22.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:22 smithi007 ceph-mon[37158]: osdmap e299: 8 total, 7 up, 7 in 2024-02-05T07:28:22.966 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:22.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:22.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:22 smithi090 ceph-mon[33213]: pgmap v482: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 41 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 105 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 5.0 KiB/s rd, 28 KiB/s wr, 1.21k op/s; 6677/195495 objects degraded (3.415%); 43959/195495 objects misplaced (22.486%); 33 MiB/s, 535 objects/s recovering 2024-02-05T07:28:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:22 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:28:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:22 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:28:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:22 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:22 smithi090 ceph-mon[33213]: osdmap e299: 8 total, 7 up, 7 in 2024-02-05T07:28:23.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:23.112 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:23.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:23.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:23.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:23.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:23.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:23.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:23.386 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:23.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:23.487 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:23.516 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:23.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:23.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:23.643 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:23.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:23.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:23.740 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:28:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:28:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:28:23.741 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:23 smithi007 ceph-mon[29098]: from='osd.6 v2:172.21.15.90:6808/313196533' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T07:28:23.741 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:23 smithi007 ceph-mon[37158]: from='osd.6 v2:172.21.15.90:6808/313196533' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T07:28:23.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:23.774 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:28:23.465+0000 7fb0fd0877c0 -1 osd.6 230 log_to_monitors true 2024-02-05T07:28:23.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:23 smithi090 ceph-mon[33213]: from='osd.6 v2:172.21.15.90:6808/313196533' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T07:28:23.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:23 smithi090 ceph-mon[33213]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T07:28:23.780 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:23.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:23.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:23.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:23.916 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:23.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:24.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:24.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:24.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:24.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:24.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:23 smithi007 ceph-mon[29098]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T07:28:24.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:23 smithi007 ceph-mon[37158]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T07:28:24.183 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:24.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:24.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:24.274 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:28:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:28:23.945+0000 7fb0ea4de700 -1 osd.6 300 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T07:28:24.285 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:24.312 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:24.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:24.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:24.436 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:24.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:24.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:24.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:24.564 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:24.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:24.665 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:24.692 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:24.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:24.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:24.825 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:24.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:24.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:24.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:24 smithi007 ceph-mon[29098]: pgmap v484: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 41 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 105 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 4.1 KiB/s rd, 23 KiB/s wr, 1.00k op/s; 6677/195495 objects degraded (3.415%); 43959/195495 objects misplaced (22.486%); 28 MiB/s, 443 objects/s recovering 2024-02-05T07:28:24.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:24 smithi007 ceph-mon[29098]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-05T07:28:24.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:24 smithi007 ceph-mon[29098]: from='osd.6 v2:172.21.15.90:6808/313196533' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:28:24.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:24 smithi007 ceph-mon[29098]: osdmap e300: 8 total, 7 up, 7 in 2024-02-05T07:28:24.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:24 smithi007 ceph-mon[29098]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:28:24.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:24 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 6677/195495 objects degraded (3.415%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:24.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:24 smithi007 ceph-mon[37158]: pgmap v484: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 41 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 105 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 4.1 KiB/s rd, 23 KiB/s wr, 1.00k op/s; 6677/195495 objects degraded (3.415%); 43959/195495 objects misplaced (22.486%); 28 MiB/s, 443 objects/s recovering 2024-02-05T07:28:24.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:24 smithi007 ceph-mon[37158]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-05T07:28:24.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:24 smithi007 ceph-mon[37158]: from='osd.6 v2:172.21.15.90:6808/313196533' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:28:24.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:24 smithi007 ceph-mon[37158]: osdmap e300: 8 total, 7 up, 7 in 2024-02-05T07:28:24.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:24 smithi007 ceph-mon[37158]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:28:24.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:24 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 6677/195495 objects degraded (3.415%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:24.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:24.951 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:24.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:24.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:24 smithi090 ceph-mon[33213]: pgmap v484: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 41 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 105 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 4.1 KiB/s rd, 23 KiB/s wr, 1.00k op/s; 6677/195495 objects degraded (3.415%); 43959/195495 objects misplaced (22.486%); 28 MiB/s, 443 objects/s recovering 2024-02-05T07:28:25.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:24 smithi090 ceph-mon[33213]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-05T07:28:25.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:24 smithi090 ceph-mon[33213]: from='osd.6 v2:172.21.15.90:6808/313196533' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:28:25.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:24 smithi090 ceph-mon[33213]: osdmap e300: 8 total, 7 up, 7 in 2024-02-05T07:28:25.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:24 smithi090 ceph-mon[33213]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-05T07:28:25.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:24 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 6677/195495 objects degraded (3.415%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:25.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:25.085 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:25.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:25.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:25.215 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:25.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:25.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:25.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:25.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:25.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:25.470 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:25.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:25.572 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:25.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:25.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:25.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:25.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:25.731 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:25.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:25.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:25.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:25 smithi090 ceph-mon[33213]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-05T07:28:25.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:25 smithi090 ceph-mon[33213]: osd.6 v2:172.21.15.90:6808/313196533 boot 2024-02-05T07:28:25.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:25 smithi090 ceph-mon[33213]: osdmap e301: 8 total, 8 up, 7 in 2024-02-05T07:28:25.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:25 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:28:25.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:25.863 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:25.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:25.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:25.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:25 smithi007 ceph-mon[29098]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-05T07:28:25.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:25 smithi007 ceph-mon[29098]: osd.6 v2:172.21.15.90:6808/313196533 boot 2024-02-05T07:28:25.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:25 smithi007 ceph-mon[29098]: osdmap e301: 8 total, 8 up, 7 in 2024-02-05T07:28:25.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:25 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:28:25.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:25 smithi007 ceph-mon[37158]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-05T07:28:25.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:25 smithi007 ceph-mon[37158]: osd.6 v2:172.21.15.90:6808/313196533 boot 2024-02-05T07:28:25.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:25 smithi007 ceph-mon[37158]: osdmap e301: 8 total, 8 up, 7 in 2024-02-05T07:28:25.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:25 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T07:28:25.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:26.136 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:26.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:26.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:26.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:26.264 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:26.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:26.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:26.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:26.389 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:26.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:26.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:26.521 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:26.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:26.623 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:26.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:26.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:26.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:26.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:26.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:26.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:26.869 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:28:26.869 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:28:26.869 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: pgmap v487: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 41 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 105 active+clean; 4.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 4.1 KiB/s rd, 23 KiB/s wr, 777 op/s; 6677/195495 objects degraded (3.415%); 43959/195495 objects misplaced (22.486%); 27 MiB/s, 440 objects/s recovering 2024-02-05T07:28:26.869 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 300'10716 2024-02-05T07:28:26.869 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: 8.10 starting backfill to osd.6 from (0'0,0'0] MAX to 300'10662 2024-02-05T07:28:26.869 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: osdmap e302: 8 total, 8 up, 7 in 2024-02-05T07:28:26.869 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: 8.9 starting backfill to osd.6 from (0'0,230'3624] MAX to 300'10634 2024-02-05T07:28:26.869 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: 8.7 starting backfill to osd.6 from (0'0,230'3581] MAX to 300'10772 2024-02-05T07:28:26.869 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: 8.3 starting backfill to osd.6 from (0'0,230'3544] MAX to 300'10347 2024-02-05T07:28:26.869 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: 8.17 starting backfill to osd.6 from (0'0,230'3581] MAX to 300'10711 2024-02-05T07:28:26.869 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: 8.15 starting backfill to osd.6 from (0'0,230'3746] MAX to 300'10903 2024-02-05T07:28:26.869 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 300'13288 2024-02-05T07:28:26.869 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: 8.12 starting backfill to osd.2 from (243'6896,300'10658] MIN to 300'10659 2024-02-05T07:28:26.870 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: 8.19 starting backfill to osd.6 from (0'0,230'3624] MAX to 300'10625 2024-02-05T07:28:26.870 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 300'10771 2024-02-05T07:28:26.870 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 300'10659 2024-02-05T07:28:26.870 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:26 smithi090 ceph-mon[33213]: 8.12 starting backfill to osd.7 from (243'6896,300'10658] MIN to 300'10659 2024-02-05T07:28:26.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:26.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:28:26.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:28:26.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: pgmap v487: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 41 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 105 active+clean; 4.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 4.1 KiB/s rd, 23 KiB/s wr, 777 op/s; 6677/195495 objects degraded (3.415%); 43959/195495 objects misplaced (22.486%); 27 MiB/s, 440 objects/s recovering 2024-02-05T07:28:26.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 300'10716 2024-02-05T07:28:26.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: 8.10 starting backfill to osd.6 from (0'0,0'0] MAX to 300'10662 2024-02-05T07:28:26.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: osdmap e302: 8 total, 8 up, 7 in 2024-02-05T07:28:26.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: 8.9 starting backfill to osd.6 from (0'0,230'3624] MAX to 300'10634 2024-02-05T07:28:26.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: 8.7 starting backfill to osd.6 from (0'0,230'3581] MAX to 300'10772 2024-02-05T07:28:26.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: 8.3 starting backfill to osd.6 from (0'0,230'3544] MAX to 300'10347 2024-02-05T07:28:26.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: 8.17 starting backfill to osd.6 from (0'0,230'3581] MAX to 300'10711 2024-02-05T07:28:26.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: 8.15 starting backfill to osd.6 from (0'0,230'3746] MAX to 300'10903 2024-02-05T07:28:26.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 300'13288 2024-02-05T07:28:26.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: 8.12 starting backfill to osd.2 from (243'6896,300'10658] MIN to 300'10659 2024-02-05T07:28:26.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: 8.19 starting backfill to osd.6 from (0'0,230'3624] MAX to 300'10625 2024-02-05T07:28:26.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 300'10771 2024-02-05T07:28:26.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 300'10659 2024-02-05T07:28:26.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[29098]: 8.12 starting backfill to osd.7 from (243'6896,300'10658] MIN to 300'10659 2024-02-05T07:28:26.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:28:26.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:28:26.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: pgmap v487: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 41 active+clean+remapped, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 active+undersized+degraded, 13 active+undersized, 105 active+clean; 4.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 4.1 KiB/s rd, 23 KiB/s wr, 777 op/s; 6677/195495 objects degraded (3.415%); 43959/195495 objects misplaced (22.486%); 27 MiB/s, 440 objects/s recovering 2024-02-05T07:28:26.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 300'10716 2024-02-05T07:28:26.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: 8.10 starting backfill to osd.6 from (0'0,0'0] MAX to 300'10662 2024-02-05T07:28:26.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: osdmap e302: 8 total, 8 up, 7 in 2024-02-05T07:28:26.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: 8.9 starting backfill to osd.6 from (0'0,230'3624] MAX to 300'10634 2024-02-05T07:28:26.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: 8.7 starting backfill to osd.6 from (0'0,230'3581] MAX to 300'10772 2024-02-05T07:28:26.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: 8.3 starting backfill to osd.6 from (0'0,230'3544] MAX to 300'10347 2024-02-05T07:28:26.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: 8.17 starting backfill to osd.6 from (0'0,230'3581] MAX to 300'10711 2024-02-05T07:28:26.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: 8.15 starting backfill to osd.6 from (0'0,230'3746] MAX to 300'10903 2024-02-05T07:28:26.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 300'13288 2024-02-05T07:28:26.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: 8.12 starting backfill to osd.2 from (243'6896,300'10658] MIN to 300'10659 2024-02-05T07:28:26.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: 8.19 starting backfill to osd.6 from (0'0,230'3624] MAX to 300'10625 2024-02-05T07:28:26.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 300'10771 2024-02-05T07:28:26.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 300'10659 2024-02-05T07:28:26.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:26 smithi007 ceph-mon[37158]: 8.12 starting backfill to osd.7 from (243'6896,300'10658] MIN to 300'10659 2024-02-05T07:28:26.929 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:26.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:27.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:27.057 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:27.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:27.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:27.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:27.191 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:27.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:27.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:27.323 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:27.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:27.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:27.457 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:27.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:27.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:27.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:27.582 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:27.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:27.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:27.719 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:27.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:27.820 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:27.849 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:27.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:27.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:27.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:27 smithi007 ceph-mon[29098]: 8.5 starting backfill to osd.6 from (0'0,230'3746] MAX to 300'10834 2024-02-05T07:28:27.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:27 smithi007 ceph-mon[29098]: osdmap e303: 8 total, 8 up, 7 in 2024-02-05T07:28:27.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:27 smithi007 ceph-mon[29098]: 8.13 continuing backfill to osd.2 from (242'6135,301'10511] 8:cd4471b6:::benchmark_data_smithi007_105755_object74945:head to 301'10511 2024-02-05T07:28:27.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:27 smithi007 ceph-mon[29098]: 8.13 continuing backfill to osd.3 from (242'6135,301'10511] 8:cd4471b6:::benchmark_data_smithi007_105755_object74945:head to 301'10511 2024-02-05T07:28:27.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:27 smithi007 ceph-mon[29098]: 8.6 starting backfill to osd.6 from (0'0,230'3570] MAX to 300'10716 2024-02-05T07:28:27.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:27 smithi007 ceph-mon[37158]: 8.5 starting backfill to osd.6 from (0'0,230'3746] MAX to 300'10834 2024-02-05T07:28:27.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:27 smithi007 ceph-mon[37158]: osdmap e303: 8 total, 8 up, 7 in 2024-02-05T07:28:27.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:27 smithi007 ceph-mon[37158]: 8.13 continuing backfill to osd.2 from (242'6135,301'10511] 8:cd4471b6:::benchmark_data_smithi007_105755_object74945:head to 301'10511 2024-02-05T07:28:27.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:27 smithi007 ceph-mon[37158]: 8.13 continuing backfill to osd.3 from (242'6135,301'10511] 8:cd4471b6:::benchmark_data_smithi007_105755_object74945:head to 301'10511 2024-02-05T07:28:27.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:27 smithi007 ceph-mon[37158]: 8.6 starting backfill to osd.6 from (0'0,230'3570] MAX to 300'10716 2024-02-05T07:28:27.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:27.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:27.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:27 smithi090 ceph-mon[33213]: 8.5 starting backfill to osd.6 from (0'0,230'3746] MAX to 300'10834 2024-02-05T07:28:28.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:27 smithi090 ceph-mon[33213]: osdmap e303: 8 total, 8 up, 7 in 2024-02-05T07:28:28.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:27 smithi090 ceph-mon[33213]: 8.13 continuing backfill to osd.2 from (242'6135,301'10511] 8:cd4471b6:::benchmark_data_smithi007_105755_object74945:head to 301'10511 2024-02-05T07:28:28.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:27 smithi090 ceph-mon[33213]: 8.13 continuing backfill to osd.3 from (242'6135,301'10511] 8:cd4471b6:::benchmark_data_smithi007_105755_object74945:head to 301'10511 2024-02-05T07:28:28.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:27 smithi090 ceph-mon[33213]: 8.6 starting backfill to osd.6 from (0'0,230'3570] MAX to 300'10716 2024-02-05T07:28:28.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:28.114 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:28.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:28.171 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:28:28.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:28.252 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:28.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:28.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:28.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:28.389 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:28.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:28.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:28.540 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:28.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:28.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:28.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:28.680 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:28.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:28.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:28.757 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:28 smithi090 ceph-mon[33213]: pgmap v490: 178 pgs: 1 active+remapped+backfill_toofull, 41 remapped+peering, 20 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 3 activating, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 activating+remapped, 2 active+remapped+backfilling, 3 active+clean+remapped, 1 active+remapped+backfill_wait, 4 active+undersized+degraded, 2 active+undersized, 95 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 765 op/s; 4638/186321 objects degraded (2.489%); 16750/186321 objects misplaced (8.990%); 31 MiB/s, 500 objects/s recovering 2024-02-05T07:28:28.757 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:28 smithi090 ceph-mon[33213]: 1.a scrub starts 2024-02-05T07:28:28.757 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:28 smithi090 ceph-mon[33213]: 1.a scrub ok 2024-02-05T07:28:28.757 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:28 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:28.757 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:28 smithi090 ceph-mon[33213]: osdmap e304: 8 total, 8 up, 7 in 2024-02-05T07:28:28.757 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:28 smithi090 ceph-mon[33213]: 8.16 continuing backfill to osd.2 from (238'5741,301'10615] 8:6b2e8713:::benchmark_data_smithi007_105755_object47435:head to 301'10615 2024-02-05T07:28:28.757 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:28 smithi090 ceph-mon[33213]: 8.16 continuing backfill to osd.3 from (238'5741,301'10615] MIN to 301'10615 2024-02-05T07:28:28.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:28.821 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:28.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:28.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:28.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:28 smithi007 ceph-mon[29098]: pgmap v490: 178 pgs: 1 active+remapped+backfill_toofull, 41 remapped+peering, 20 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 3 activating, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 activating+remapped, 2 active+remapped+backfilling, 3 active+clean+remapped, 1 active+remapped+backfill_wait, 4 active+undersized+degraded, 2 active+undersized, 95 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 765 op/s; 4638/186321 objects degraded (2.489%); 16750/186321 objects misplaced (8.990%); 31 MiB/s, 500 objects/s recovering 2024-02-05T07:28:28.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:28 smithi007 ceph-mon[29098]: 1.a scrub starts 2024-02-05T07:28:28.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:28 smithi007 ceph-mon[29098]: 1.a scrub ok 2024-02-05T07:28:28.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:28 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:28.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:28 smithi007 ceph-mon[29098]: osdmap e304: 8 total, 8 up, 7 in 2024-02-05T07:28:28.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:28 smithi007 ceph-mon[29098]: 8.16 continuing backfill to osd.2 from (238'5741,301'10615] 8:6b2e8713:::benchmark_data_smithi007_105755_object47435:head to 301'10615 2024-02-05T07:28:28.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:28 smithi007 ceph-mon[29098]: 8.16 continuing backfill to osd.3 from (238'5741,301'10615] MIN to 301'10615 2024-02-05T07:28:28.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:28 smithi007 ceph-mon[37158]: pgmap v490: 178 pgs: 1 active+remapped+backfill_toofull, 41 remapped+peering, 20 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 3 activating, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 activating+remapped, 2 active+remapped+backfilling, 3 active+clean+remapped, 1 active+remapped+backfill_wait, 4 active+undersized+degraded, 2 active+undersized, 95 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 765 op/s; 4638/186321 objects degraded (2.489%); 16750/186321 objects misplaced (8.990%); 31 MiB/s, 500 objects/s recovering 2024-02-05T07:28:28.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:28 smithi007 ceph-mon[37158]: 1.a scrub starts 2024-02-05T07:28:28.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:28 smithi007 ceph-mon[37158]: 1.a scrub ok 2024-02-05T07:28:28.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:28 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:28.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:28 smithi007 ceph-mon[37158]: osdmap e304: 8 total, 8 up, 7 in 2024-02-05T07:28:28.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:28 smithi007 ceph-mon[37158]: 8.16 continuing backfill to osd.2 from (238'5741,301'10615] 8:6b2e8713:::benchmark_data_smithi007_105755_object47435:head to 301'10615 2024-02-05T07:28:28.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:28 smithi007 ceph-mon[37158]: 8.16 continuing backfill to osd.3 from (238'5741,301'10615] MIN to 301'10615 2024-02-05T07:28:28.961 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:28.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:28.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:29.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:29.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:29.116 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:29.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:29.218 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:29.254 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:29.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:29.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:29.279 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:28:29.356 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:29.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:29.396 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:29.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:29.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:29.511 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:28:29.511 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:28:29.511 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-02-05T07:28:29.511 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:28:29.534 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:29.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:29.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:29.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:29.680 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:29.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:29.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:29.817 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:29.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:29.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:29.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:29 smithi007 ceph-mon[29098]: osdmap e305: 8 total, 8 up, 7 in 2024-02-05T07:28:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:29 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 4638/186321 objects degraded (2.489%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:29 smithi007 ceph-mon[29098]: osdmap e306: 8 total, 8 up, 7 in 2024-02-05T07:28:29.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:29 smithi007 ceph-mon[37158]: osdmap e305: 8 total, 8 up, 7 in 2024-02-05T07:28:29.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:29 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 4638/186321 objects degraded (2.489%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:29.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:29 smithi007 ceph-mon[37158]: osdmap e306: 8 total, 8 up, 7 in 2024-02-05T07:28:29.951 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:29.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:29.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:29 smithi090 ceph-mon[33213]: osdmap e305: 8 total, 8 up, 7 in 2024-02-05T07:28:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:29 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 4638/186321 objects degraded (2.489%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-05T07:28:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:29 smithi090 ceph-mon[33213]: osdmap e306: 8 total, 8 up, 7 in 2024-02-05T07:28:30.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:30.084 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:30.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:30.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:30.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:30.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:30.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:30.364 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:30.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:30.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:30.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:30.489 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:30.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:30.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:30.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:30.628 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:30.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:30.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:30.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:30.765 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:30.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:30.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:30.775 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:28:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:28:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:30 smithi090 ceph-mon[33213]: pgmap v493: 178 pgs: 1 active+remapped+backfill_toofull, 36 remapped+peering, 20 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 3 activating, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 activating+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded, 108 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 765 op/s; 4620/186321 objects degraded (2.480%); 16745/186321 objects misplaced (8.987%); 31 MiB/s, 499 objects/s recovering 2024-02-05T07:28:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:30 smithi090 ceph-mon[33213]: 1.14 scrub starts 2024-02-05T07:28:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:30 smithi090 ceph-mon[33213]: 1.14 scrub ok 2024-02-05T07:28:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:28:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:28:30.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:30.904 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:30.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:30.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:30 smithi007 ceph-mon[29098]: pgmap v493: 178 pgs: 1 active+remapped+backfill_toofull, 36 remapped+peering, 20 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 3 activating, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 activating+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded, 108 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 765 op/s; 4620/186321 objects degraded (2.480%); 16745/186321 objects misplaced (8.987%); 31 MiB/s, 499 objects/s recovering 2024-02-05T07:28:30.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:30 smithi007 ceph-mon[29098]: 1.14 scrub starts 2024-02-05T07:28:30.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:30 smithi007 ceph-mon[29098]: 1.14 scrub ok 2024-02-05T07:28:30.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:28:30.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:28:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:30 smithi007 ceph-mon[37158]: pgmap v493: 178 pgs: 1 active+remapped+backfill_toofull, 36 remapped+peering, 20 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 3 activating, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 2 activating+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded, 108 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 765 op/s; 4620/186321 objects degraded (2.480%); 16745/186321 objects misplaced (8.987%); 31 MiB/s, 499 objects/s recovering 2024-02-05T07:28:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:30 smithi007 ceph-mon[37158]: 1.14 scrub starts 2024-02-05T07:28:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:30 smithi007 ceph-mon[37158]: 1.14 scrub ok 2024-02-05T07:28:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:28:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:28:31.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:31.036 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:31.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:31.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:31.161 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:31.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:31.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:31.287 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:31.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:31.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:31.413 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:31.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:31.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:31.514 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:31.539 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:31.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:31.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:31.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:31.669 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:31.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:31.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:31.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:31.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:31.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:31.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:31.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:31 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:31 smithi007 ceph-mon[29098]: osdmap e307: 8 total, 8 up, 7 in 2024-02-05T07:28:31.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:31 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:31.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:31 smithi007 ceph-mon[37158]: osdmap e307: 8 total, 8 up, 7 in 2024-02-05T07:28:31.931 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:31.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:31.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:31 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:31 smithi090 ceph-mon[33213]: osdmap e307: 8 total, 8 up, 7 in 2024-02-05T07:28:32.033 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:32.059 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:32.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:32.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:32.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:32.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:32.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:32.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:32.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:32.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:32.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:32.443 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:32.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:32.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:32.568 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:32.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:32.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:32.704 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:32.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:32.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:32.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:32.834 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:32.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:32 smithi007 ceph-mon[29098]: pgmap v496: 178 pgs: 8 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 4 active+remapped+backfill_wait, 157 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 4745/176559 objects degraded (2.687%); 33219/176559 objects misplaced (18.815%); 78 MiB/s, 1.24k objects/s recovering 2024-02-05T07:28:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:32 smithi007 ceph-mon[29098]: 4.4 scrub starts 2024-02-05T07:28:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:32 smithi007 ceph-mon[29098]: 4.4 scrub ok 2024-02-05T07:28:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:32 smithi007 ceph-mon[29098]: osdmap e308: 8 total, 8 up, 7 in 2024-02-05T07:28:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:32 smithi007 ceph-mon[37158]: pgmap v496: 178 pgs: 8 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 4 active+remapped+backfill_wait, 157 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 4745/176559 objects degraded (2.687%); 33219/176559 objects misplaced (18.815%); 78 MiB/s, 1.24k objects/s recovering 2024-02-05T07:28:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:32 smithi007 ceph-mon[37158]: 4.4 scrub starts 2024-02-05T07:28:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:32 smithi007 ceph-mon[37158]: 4.4 scrub ok 2024-02-05T07:28:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:32 smithi007 ceph-mon[37158]: osdmap e308: 8 total, 8 up, 7 in 2024-02-05T07:28:32.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:32.969 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:32.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:32 smithi090 ceph-mon[33213]: pgmap v496: 178 pgs: 8 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 4 active+remapped+backfill_wait, 157 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 4745/176559 objects degraded (2.687%); 33219/176559 objects misplaced (18.815%); 78 MiB/s, 1.24k objects/s recovering 2024-02-05T07:28:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:32 smithi090 ceph-mon[33213]: 4.4 scrub starts 2024-02-05T07:28:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:32 smithi090 ceph-mon[33213]: 4.4 scrub ok 2024-02-05T07:28:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:32 smithi090 ceph-mon[33213]: osdmap e308: 8 total, 8 up, 7 in 2024-02-05T07:28:33.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:33.107 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:33.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:33.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:33.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:33.246 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:33.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:33.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:33.370 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:33.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:33.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:33.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:33.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:33.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:33.812 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:33.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:33.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:33.850 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:33 smithi007 ceph-mon[29098]: osdmap e309: 8 total, 8 up, 7 in 2024-02-05T07:28:33.850 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:28:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:28:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:28:33.850 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:33 smithi007 ceph-mon[37158]: osdmap e309: 8 total, 8 up, 7 in 2024-02-05T07:28:33.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:33.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:33.958 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:33.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:33 smithi090 ceph-mon[33213]: osdmap e309: 8 total, 8 up, 7 in 2024-02-05T07:28:34.059 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:34.099 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:34.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:34.200 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:34.239 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:34.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:34.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:34.380 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:34.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:34.481 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:34.515 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:34.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:34.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:34.617 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:34.639 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:34.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:34.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:34.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:34.781 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:34.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:34.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:34.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:34.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:34.912 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:34.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:34 smithi007 ceph-mon[29098]: pgmap v499: 178 pgs: 8 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 4 active+remapped+backfill_wait, 157 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 4745/176559 objects degraded (2.687%); 33219/176559 objects misplaced (18.815%); 78 MiB/s, 1.25k objects/s recovering 2024-02-05T07:28:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:34 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:34 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 4745/176559 objects degraded (2.687%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:34 smithi007 ceph-mon[29098]: osdmap e310: 8 total, 8 up, 7 in 2024-02-05T07:28:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:34 smithi007 ceph-mon[37158]: pgmap v499: 178 pgs: 8 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 4 active+remapped+backfill_wait, 157 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 4745/176559 objects degraded (2.687%); 33219/176559 objects misplaced (18.815%); 78 MiB/s, 1.25k objects/s recovering 2024-02-05T07:28:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:34 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:34 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 4745/176559 objects degraded (2.687%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:34 smithi007 ceph-mon[37158]: osdmap e310: 8 total, 8 up, 7 in 2024-02-05T07:28:35.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:34 smithi090 ceph-mon[33213]: pgmap v499: 178 pgs: 8 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 4 active+remapped+backfill_wait, 157 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 4745/176559 objects degraded (2.687%); 33219/176559 objects misplaced (18.815%); 78 MiB/s, 1.25k objects/s recovering 2024-02-05T07:28:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:34 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:34 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 4745/176559 objects degraded (2.687%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:34 smithi090 ceph-mon[33213]: osdmap e310: 8 total, 8 up, 7 in 2024-02-05T07:28:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:35.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:35.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:35.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:35.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 0 1 2024-02-05T07:28:35.197 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:35.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:35.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:35.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:35.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:35.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:35.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:35.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:35.482 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:35.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:35.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:35.617 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:35.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:35.719 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:35.761 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:35.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:35.862 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:35.899 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:35.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:35.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:35.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:35 smithi007 ceph-mon[29098]: pgmap v501: 178 pgs: 8 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 4 active+remapped+backfill_wait, 157 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 224 op/s; 4745/176559 objects degraded (2.687%); 33219/176559 objects misplaced (18.815%); 21 MiB/s, 333 objects/s recovering 2024-02-05T07:28:35.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:35 smithi007 ceph-mon[37158]: pgmap v501: 178 pgs: 8 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 4 active+remapped+backfill_wait, 157 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 224 op/s; 4745/176559 objects degraded (2.687%); 33219/176559 objects misplaced (18.815%); 21 MiB/s, 333 objects/s recovering 2024-02-05T07:28:36.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:35 smithi090 ceph-mon[33213]: pgmap v501: 178 pgs: 8 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 4 active+remapped+backfill_wait, 157 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 224 op/s; 4745/176559 objects degraded (2.687%); 33219/176559 objects misplaced (18.815%); 21 MiB/s, 333 objects/s recovering 2024-02-05T07:28:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:36.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:36.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:36.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:36.160 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:36.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:36.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:36.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:36.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:36.289 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:36.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:36.389 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:36.419 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:36.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:36.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:36.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:36.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:36.553 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:36.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:36.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:36.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:36.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:36.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:36.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:36.812 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:36.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:36.913 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:36.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:36 smithi007 ceph-mon[29098]: 3.1f scrub starts 2024-02-05T07:28:36.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:36 smithi007 ceph-mon[29098]: 3.1f scrub ok 2024-02-05T07:28:36.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:36 smithi007 ceph-mon[29098]: osdmap e311: 8 total, 8 up, 7 in 2024-02-05T07:28:36.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:36 smithi007 ceph-mon[37158]: 3.1f scrub starts 2024-02-05T07:28:36.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:36 smithi007 ceph-mon[37158]: 3.1f scrub ok 2024-02-05T07:28:36.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:36 smithi007 ceph-mon[37158]: osdmap e311: 8 total, 8 up, 7 in 2024-02-05T07:28:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:36.963 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:36.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:36 smithi090 ceph-mon[33213]: 3.1f scrub starts 2024-02-05T07:28:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:36 smithi090 ceph-mon[33213]: 3.1f scrub ok 2024-02-05T07:28:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:36 smithi090 ceph-mon[33213]: osdmap e311: 8 total, 8 up, 7 in 2024-02-05T07:28:37.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:37.104 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:37.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:37.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:37.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:37.310 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:37.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:37.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:37.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:37.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:37.442 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:37.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:37.543 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:37.575 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:37.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:37.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:37.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:37.704 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:37.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:37.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:37.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:37.849 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:37.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:37.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:37 smithi007 ceph-mon[29098]: osdmap e312: 8 total, 8 up, 7 in 2024-02-05T07:28:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:37 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:28:37.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:37 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:28:37.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:37 smithi007 ceph-mon[29098]: pgmap v504: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 159 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 799 op/s; 3772/165780 objects degraded (2.275%); 27639/165780 objects misplaced (16.672%); 63 MiB/s, 1.00k objects/s recovering 2024-02-05T07:28:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:37 smithi007 ceph-mon[37158]: osdmap e312: 8 total, 8 up, 7 in 2024-02-05T07:28:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:37 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:28:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:37 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:28:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:37 smithi007 ceph-mon[37158]: pgmap v504: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 159 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 799 op/s; 3772/165780 objects degraded (2.275%); 27639/165780 objects misplaced (16.672%); 63 MiB/s, 1.00k objects/s recovering 2024-02-05T07:28:37.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:37.982 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:37.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:37 smithi090 ceph-mon[33213]: osdmap e312: 8 total, 8 up, 7 in 2024-02-05T07:28:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:37 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:28:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:37 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:28:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:37 smithi090 ceph-mon[33213]: pgmap v504: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 159 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 799 op/s; 3772/165780 objects degraded (2.275%); 27639/165780 objects misplaced (16.672%); 63 MiB/s, 1.00k objects/s recovering 2024-02-05T07:28:38.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:38.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:38.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:38.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:38.235 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:38.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:38.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:38.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:38.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:38.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:38.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:38.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:38.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:38.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:38.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:38.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:38.757 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:38.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:38.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:38.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:38.900 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:38.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:38.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:39.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:38 smithi090 ceph-mon[33213]: osdmap e313: 8 total, 8 up, 7 in 2024-02-05T07:28:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:38 smithi090 ceph-mon[33213]: 8.0 scrub starts 2024-02-05T07:28:39.038 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:39.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:39.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:39.167 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:39.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:39.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:39.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:38 smithi007 ceph-mon[29098]: osdmap e313: 8 total, 8 up, 7 in 2024-02-05T07:28:39.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:38 smithi007 ceph-mon[29098]: 8.0 scrub starts 2024-02-05T07:28:39.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:38 smithi007 ceph-mon[37158]: osdmap e313: 8 total, 8 up, 7 in 2024-02-05T07:28:39.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:38 smithi007 ceph-mon[37158]: 8.0 scrub starts 2024-02-05T07:28:39.268 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:39.366 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:39.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:39.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:39.467 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:39.500 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:39.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:39.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:39.602 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:39.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:39.638 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:39.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:39.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:39.766 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:39.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:39.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:39.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:39.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:40.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:39 smithi090 ceph-mon[33213]: 6.f scrub starts 2024-02-05T07:28:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:39 smithi090 ceph-mon[33213]: 6.f scrub ok 2024-02-05T07:28:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:39 smithi090 ceph-mon[33213]: 8.0 scrub ok 2024-02-05T07:28:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:39 smithi090 ceph-mon[33213]: osdmap e314: 8 total, 8 up, 7 in 2024-02-05T07:28:40.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:39 smithi090 ceph-mon[33213]: 1.4 scrub starts 2024-02-05T07:28:40.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:39 smithi090 ceph-mon[33213]: 1.4 scrub ok 2024-02-05T07:28:40.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:39 smithi090 ceph-mon[33213]: pgmap v507: 178 pgs: 1 peering, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 159 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 3277/165645 objects degraded (1.978%); 27639/165645 objects misplaced (16.686%); 77 MiB/s, 1.24k objects/s recovering 2024-02-05T07:28:40.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:39 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 10 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:40.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:39 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 3772/165780 objects degraded (2.275%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:40.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:40.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:40.137 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:40.160 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:40.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[29098]: 6.f scrub starts 2024-02-05T07:28:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[29098]: 6.f scrub ok 2024-02-05T07:28:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[29098]: 8.0 scrub ok 2024-02-05T07:28:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[29098]: osdmap e314: 8 total, 8 up, 7 in 2024-02-05T07:28:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[29098]: 1.4 scrub starts 2024-02-05T07:28:40.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[29098]: 1.4 scrub ok 2024-02-05T07:28:40.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[29098]: pgmap v507: 178 pgs: 1 peering, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 159 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 3277/165645 objects degraded (1.978%); 27639/165645 objects misplaced (16.686%); 77 MiB/s, 1.24k objects/s recovering 2024-02-05T07:28:40.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 10 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:40.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 3772/165780 objects degraded (2.275%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:40.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[37158]: 6.f scrub starts 2024-02-05T07:28:40.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[37158]: 6.f scrub ok 2024-02-05T07:28:40.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[37158]: 8.0 scrub ok 2024-02-05T07:28:40.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[37158]: osdmap e314: 8 total, 8 up, 7 in 2024-02-05T07:28:40.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[37158]: 1.4 scrub starts 2024-02-05T07:28:40.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[37158]: 1.4 scrub ok 2024-02-05T07:28:40.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[37158]: pgmap v507: 178 pgs: 1 peering, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 159 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 3277/165645 objects degraded (1.978%); 27639/165645 objects misplaced (16.686%); 77 MiB/s, 1.24k objects/s recovering 2024-02-05T07:28:40.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 10 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:28:40.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:39 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 3772/165780 objects degraded (2.275%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:40.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:40.307 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:40.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:40.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:40.441 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:40.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:40.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:40.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:40.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:40.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:40.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:40.698 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:40.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:40.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:28:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:28:40.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:40 smithi090 ceph-mon[33213]: 5.15 scrub starts 2024-02-05T07:28:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:40 smithi090 ceph-mon[33213]: 5.15 scrub ok 2024-02-05T07:28:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:40 smithi090 ceph-mon[33213]: 8.2 scrub starts 2024-02-05T07:28:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:40 smithi090 ceph-mon[33213]: 1.c scrub starts 2024-02-05T07:28:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:40 smithi090 ceph-mon[33213]: 1.c scrub ok 2024-02-05T07:28:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:40 smithi090 ceph-mon[33213]: 8.2 scrub ok 2024-02-05T07:28:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:40 smithi090 ceph-mon[33213]: osdmap e315: 8 total, 8 up, 7 in 2024-02-05T07:28:40.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:40 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:40.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:40.823 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:40.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:40.925 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:40.949 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:40.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:40.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:41.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:41.078 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:41.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:41.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[29098]: 5.15 scrub starts 2024-02-05T07:28:41.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[29098]: 5.15 scrub ok 2024-02-05T07:28:41.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[29098]: 8.2 scrub starts 2024-02-05T07:28:41.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[29098]: 1.c scrub starts 2024-02-05T07:28:41.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[29098]: 1.c scrub ok 2024-02-05T07:28:41.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[29098]: 8.2 scrub ok 2024-02-05T07:28:41.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[29098]: osdmap e315: 8 total, 8 up, 7 in 2024-02-05T07:28:41.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:41.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[37158]: 5.15 scrub starts 2024-02-05T07:28:41.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[37158]: 5.15 scrub ok 2024-02-05T07:28:41.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[37158]: 8.2 scrub starts 2024-02-05T07:28:41.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[37158]: 1.c scrub starts 2024-02-05T07:28:41.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[37158]: 1.c scrub ok 2024-02-05T07:28:41.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[37158]: 8.2 scrub ok 2024-02-05T07:28:41.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[37158]: osdmap e315: 8 total, 8 up, 7 in 2024-02-05T07:28:41.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:40 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:41.180 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:41.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:41.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:41.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:41.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:41.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:41.638 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:41.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:41.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:41.774 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:41.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:41.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:41.876 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:41.919 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:41.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:42.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:41 smithi090 ceph-mon[33213]: 5.6 scrub starts 2024-02-05T07:28:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:41 smithi090 ceph-mon[33213]: 5.6 scrub ok 2024-02-05T07:28:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:41 smithi090 ceph-mon[33213]: 1.e scrub starts 2024-02-05T07:28:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:41 smithi090 ceph-mon[33213]: 1.e scrub ok 2024-02-05T07:28:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:41 smithi090 ceph-mon[33213]: 1.11 scrub starts 2024-02-05T07:28:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:41 smithi090 ceph-mon[33213]: 1.11 scrub ok 2024-02-05T07:28:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:41 smithi090 ceph-mon[33213]: pgmap v509: 178 pgs: 1 active+clean+remapped, 1 peering, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 162 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 3089/156306 objects degraded (1.976%); 20947/156306 objects misplaced (13.401%); 14 MiB/s, 225 objects/s recovering 2024-02-05T07:28:42.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:41 smithi090 ceph-mon[33213]: osdmap e316: 8 total, 8 up, 7 in 2024-02-05T07:28:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:42.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:42.157 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:42.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[29098]: 5.6 scrub starts 2024-02-05T07:28:42.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[29098]: 5.6 scrub ok 2024-02-05T07:28:42.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[29098]: 1.e scrub starts 2024-02-05T07:28:42.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[29098]: 1.e scrub ok 2024-02-05T07:28:42.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[29098]: 1.11 scrub starts 2024-02-05T07:28:42.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[29098]: 1.11 scrub ok 2024-02-05T07:28:42.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[29098]: pgmap v509: 178 pgs: 1 active+clean+remapped, 1 peering, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 162 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 3089/156306 objects degraded (1.976%); 20947/156306 objects misplaced (13.401%); 14 MiB/s, 225 objects/s recovering 2024-02-05T07:28:42.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[29098]: osdmap e316: 8 total, 8 up, 7 in 2024-02-05T07:28:42.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[37158]: 5.6 scrub starts 2024-02-05T07:28:42.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[37158]: 5.6 scrub ok 2024-02-05T07:28:42.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[37158]: 1.e scrub starts 2024-02-05T07:28:42.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[37158]: 1.e scrub ok 2024-02-05T07:28:42.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[37158]: 1.11 scrub starts 2024-02-05T07:28:42.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[37158]: 1.11 scrub ok 2024-02-05T07:28:42.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[37158]: pgmap v509: 178 pgs: 1 active+clean+remapped, 1 peering, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 162 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 3089/156306 objects degraded (1.976%); 20947/156306 objects misplaced (13.401%); 14 MiB/s, 225 objects/s recovering 2024-02-05T07:28:42.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:41 smithi007 ceph-mon[37158]: osdmap e316: 8 total, 8 up, 7 in 2024-02-05T07:28:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:42.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:42.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:42.329 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:42.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:42.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:42.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:42.464 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:42.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:42.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:42.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:42.594 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:42.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:42.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:42.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:42.723 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:42.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:42.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:42.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:42.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:42.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:42 smithi007 ceph-mon[29098]: 5.a scrub starts 2024-02-05T07:28:42.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:42 smithi007 ceph-mon[29098]: 5.a scrub ok 2024-02-05T07:28:42.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:42 smithi007 ceph-mon[29098]: 1.5 scrub starts 2024-02-05T07:28:42.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:42 smithi007 ceph-mon[29098]: 1.5 scrub ok 2024-02-05T07:28:42.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:42 smithi007 ceph-mon[29098]: 6.6 scrub starts 2024-02-05T07:28:42.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:42 smithi007 ceph-mon[29098]: 6.6 scrub ok 2024-02-05T07:28:42.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:42 smithi007 ceph-mon[29098]: osdmap e317: 8 total, 8 up, 7 in 2024-02-05T07:28:42.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:42 smithi007 ceph-mon[37158]: 5.a scrub starts 2024-02-05T07:28:42.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:42 smithi007 ceph-mon[37158]: 5.a scrub ok 2024-02-05T07:28:42.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:42 smithi007 ceph-mon[37158]: 1.5 scrub starts 2024-02-05T07:28:42.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:42 smithi007 ceph-mon[37158]: 1.5 scrub ok 2024-02-05T07:28:42.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:42 smithi007 ceph-mon[37158]: 6.6 scrub starts 2024-02-05T07:28:42.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:42 smithi007 ceph-mon[37158]: 6.6 scrub ok 2024-02-05T07:28:42.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:42 smithi007 ceph-mon[37158]: osdmap e317: 8 total, 8 up, 7 in 2024-02-05T07:28:42.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:42.977 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:42.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:42 smithi090 ceph-mon[33213]: 5.a scrub starts 2024-02-05T07:28:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:42 smithi090 ceph-mon[33213]: 5.a scrub ok 2024-02-05T07:28:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:42 smithi090 ceph-mon[33213]: 1.5 scrub starts 2024-02-05T07:28:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:42 smithi090 ceph-mon[33213]: 1.5 scrub ok 2024-02-05T07:28:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:42 smithi090 ceph-mon[33213]: 6.6 scrub starts 2024-02-05T07:28:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:42 smithi090 ceph-mon[33213]: 6.6 scrub ok 2024-02-05T07:28:43.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:42 smithi090 ceph-mon[33213]: osdmap e317: 8 total, 8 up, 7 in 2024-02-05T07:28:43.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:43.103 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:43.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:43.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:43.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:43.230 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:43.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:43.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:43.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:43.366 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:43.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:43.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:43.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:43.502 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:43.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:43.602 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:28:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:28:43.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:43.663 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:43.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:43.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:43.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:43.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:43.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:43.897 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[29098]: 8.11 scrub starts 2024-02-05T07:28:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[29098]: 8.11 scrub ok 2024-02-05T07:28:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[29098]: 8.14 scrub starts 2024-02-05T07:28:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[29098]: 8.14 scrub ok 2024-02-05T07:28:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[29098]: pgmap v512: 178 pgs: 1 active+clean+remapped, 1 peering, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 162 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 3089/156306 objects degraded (1.976%); 20947/156306 objects misplaced (13.401%); 8.1 MiB/s, 129 objects/s recovering 2024-02-05T07:28:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[29098]: osdmap e318: 8 total, 8 up, 7 in 2024-02-05T07:28:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:28:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:28:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[37158]: 8.11 scrub starts 2024-02-05T07:28:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[37158]: 8.11 scrub ok 2024-02-05T07:28:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[37158]: 8.14 scrub starts 2024-02-05T07:28:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[37158]: 8.14 scrub ok 2024-02-05T07:28:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[37158]: pgmap v512: 178 pgs: 1 active+clean+remapped, 1 peering, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 162 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 3089/156306 objects degraded (1.976%); 20947/156306 objects misplaced (13.401%); 8.1 MiB/s, 129 objects/s recovering 2024-02-05T07:28:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[37158]: osdmap e318: 8 total, 8 up, 7 in 2024-02-05T07:28:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:28:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:43 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:28:43.925 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:43.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:43.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:43 smithi090 ceph-mon[33213]: 8.11 scrub starts 2024-02-05T07:28:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:43 smithi090 ceph-mon[33213]: 8.11 scrub ok 2024-02-05T07:28:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:43 smithi090 ceph-mon[33213]: 8.14 scrub starts 2024-02-05T07:28:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:43 smithi090 ceph-mon[33213]: 8.14 scrub ok 2024-02-05T07:28:44.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:43 smithi090 ceph-mon[33213]: pgmap v512: 178 pgs: 1 active+clean+remapped, 1 peering, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 162 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 3089/156306 objects degraded (1.976%); 20947/156306 objects misplaced (13.401%); 8.1 MiB/s, 129 objects/s recovering 2024-02-05T07:28:44.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:43 smithi090 ceph-mon[33213]: osdmap e318: 8 total, 8 up, 7 in 2024-02-05T07:28:44.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:43 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:28:44.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:43 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:28:44.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:44.052 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:44.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:44.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:44.177 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:44.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:44.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:44.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:44.299 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:28:44.307 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:44.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:44.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:44.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:44.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:44.446 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:44.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:44.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:44.584 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:44.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:44.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:44.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:44.719 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:44.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:44.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:44.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:44.857 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:44.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:44.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:44.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:44.993 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:44.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:44.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:44.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[29098]: 1.10 scrub starts 2024-02-05T07:28:44.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[29098]: 1.10 scrub ok 2024-02-05T07:28:44.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[29098]: 5.d scrub starts 2024-02-05T07:28:44.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[29098]: 5.d scrub ok 2024-02-05T07:28:44.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[29098]: 8.18 scrub starts 2024-02-05T07:28:44.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[29098]: 8.18 scrub ok 2024-02-05T07:28:44.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[29098]: 6.1b scrub starts 2024-02-05T07:28:44.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[29098]: 6.1b scrub ok 2024-02-05T07:28:44.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3098270124' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:28:44.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 3089/156306 objects degraded (1.976%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:44.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:28:44.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3098270124' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:28:44.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[29098]: osdmap e319: 8 total, 8 up, 7 in 2024-02-05T07:28:44.995 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[37158]: 1.10 scrub starts 2024-02-05T07:28:44.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[37158]: 1.10 scrub ok 2024-02-05T07:28:44.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[37158]: 5.d scrub starts 2024-02-05T07:28:44.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[37158]: 5.d scrub ok 2024-02-05T07:28:44.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[37158]: 8.18 scrub starts 2024-02-05T07:28:44.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[37158]: 8.18 scrub ok 2024-02-05T07:28:44.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[37158]: 6.1b scrub starts 2024-02-05T07:28:44.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[37158]: 6.1b scrub ok 2024-02-05T07:28:44.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3098270124' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:28:44.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 3089/156306 objects degraded (1.976%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:44.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:28:44.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3098270124' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:28:44.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:44 smithi007 ceph-mon[37158]: osdmap e319: 8 total, 8 up, 7 in 2024-02-05T07:28:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:44 smithi090 ceph-mon[33213]: 1.10 scrub starts 2024-02-05T07:28:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:44 smithi090 ceph-mon[33213]: 1.10 scrub ok 2024-02-05T07:28:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:44 smithi090 ceph-mon[33213]: 5.d scrub starts 2024-02-05T07:28:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:44 smithi090 ceph-mon[33213]: 5.d scrub ok 2024-02-05T07:28:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:44 smithi090 ceph-mon[33213]: 8.18 scrub starts 2024-02-05T07:28:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:44 smithi090 ceph-mon[33213]: 8.18 scrub ok 2024-02-05T07:28:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:44 smithi090 ceph-mon[33213]: 6.1b scrub starts 2024-02-05T07:28:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:44 smithi090 ceph-mon[33213]: 6.1b scrub ok 2024-02-05T07:28:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:44 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3098270124' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:28:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:44 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 3089/156306 objects degraded (1.976%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:44 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:28:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:44 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3098270124' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:28:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:44 smithi090 ceph-mon[33213]: osdmap e319: 8 total, 8 up, 7 in 2024-02-05T07:28:45.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:45.126 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:45.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:45.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:45.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:45.260 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:45.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:45.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:45.362 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:45.388 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:45.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:45.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:45.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:45.521 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:45.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:45.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:45.655 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:45.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:45.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:45.748 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:28:45.748 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:28:45.748 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:28:45.748 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:28:45.748 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:28:45.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:45.774 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:28:45.812 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:28:45.813 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[29098]: 8.15 scrub starts 2024-02-05T07:28:45.813 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[29098]: 8.15 scrub ok 2024-02-05T07:28:45.813 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[29098]: 8.19 scrub starts 2024-02-05T07:28:45.813 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[29098]: 8.3 scrub starts 2024-02-05T07:28:45.813 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[29098]: 8.19 scrub ok 2024-02-05T07:28:45.813 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[29098]: 4.0 scrub starts 2024-02-05T07:28:45.813 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[29098]: 4.0 scrub ok 2024-02-05T07:28:45.813 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:28:45.813 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:28:45.814 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[29098]: pgmap v515: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 163 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 28 op/s; 3089/155967 objects degraded (1.981%); 20947/155967 objects misplaced (13.430%) 2024-02-05T07:28:45.814 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[29098]: 8.3 scrub ok 2024-02-05T07:28:45.814 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3905857819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:28:45.814 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[37158]: 8.15 scrub starts 2024-02-05T07:28:45.814 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[37158]: 8.15 scrub ok 2024-02-05T07:28:45.814 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[37158]: 8.19 scrub starts 2024-02-05T07:28:45.814 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[37158]: 8.3 scrub starts 2024-02-05T07:28:45.814 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[37158]: 8.19 scrub ok 2024-02-05T07:28:45.814 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[37158]: 4.0 scrub starts 2024-02-05T07:28:45.814 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[37158]: 4.0 scrub ok 2024-02-05T07:28:45.814 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:28:45.814 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:28:45.815 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[37158]: pgmap v515: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 163 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 28 op/s; 3089/155967 objects degraded (1.981%); 20947/155967 objects misplaced (13.430%) 2024-02-05T07:28:45.815 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[37158]: 8.3 scrub ok 2024-02-05T07:28:45.815 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:45 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3905857819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:28:45.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:45.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:45.850 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:28:45.857 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:28:45.857 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:28:45.857 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:28:45.857 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:28:45.857 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:28:45.857 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:28:45.857 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-02-05T07:28:45.857 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:28:45.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:45.951 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:45.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:45.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:45 smithi090 ceph-mon[33213]: 8.15 scrub starts 2024-02-05T07:28:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:45 smithi090 ceph-mon[33213]: 8.15 scrub ok 2024-02-05T07:28:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:45 smithi090 ceph-mon[33213]: 8.19 scrub starts 2024-02-05T07:28:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:45 smithi090 ceph-mon[33213]: 8.3 scrub starts 2024-02-05T07:28:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:45 smithi090 ceph-mon[33213]: 8.19 scrub ok 2024-02-05T07:28:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:45 smithi090 ceph-mon[33213]: 4.0 scrub starts 2024-02-05T07:28:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:45 smithi090 ceph-mon[33213]: 4.0 scrub ok 2024-02-05T07:28:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:45 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:28:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:45 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:28:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:45 smithi090 ceph-mon[33213]: pgmap v515: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 163 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 28 op/s; 3089/155967 objects degraded (1.981%); 20947/155967 objects misplaced (13.430%) 2024-02-05T07:28:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:45 smithi090 ceph-mon[33213]: 8.3 scrub ok 2024-02-05T07:28:46.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:45 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3905857819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:28:46.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:46.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:46.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:46.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:46.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:46.210 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:46.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:46.302 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:28:46.302 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:28:46.302 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-02-05T07:28:46.302 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-02-05T07:28:46.302 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:28:46.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:46.335 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:46.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:46.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:46.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:46.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:46.461 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:46.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:46.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:46.588 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:46.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:46.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:46.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:46.714 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:46.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:46.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:46.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:46.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[29098]: 1.d deep-scrub starts 2024-02-05T07:28:46.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[29098]: 1.d deep-scrub ok 2024-02-05T07:28:46.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[29098]: 3.10 deep-scrub starts 2024-02-05T07:28:46.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[29098]: 3.10 deep-scrub ok 2024-02-05T07:28:46.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[29098]: 8.4 scrub starts 2024-02-05T07:28:46.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[29098]: 1.13 deep-scrub starts 2024-02-05T07:28:46.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[29098]: 1.13 deep-scrub ok 2024-02-05T07:28:46.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[29098]: osdmap e320: 8 total, 8 up, 7 in 2024-02-05T07:28:46.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[29098]: 3.17 scrub starts 2024-02-05T07:28:46.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[29098]: 3.17 scrub ok 2024-02-05T07:28:46.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[29098]: osdmap e321: 8 total, 8 up, 7 in 2024-02-05T07:28:46.852 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:46.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[37158]: 1.d deep-scrub starts 2024-02-05T07:28:46.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[37158]: 1.d deep-scrub ok 2024-02-05T07:28:46.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[37158]: 3.10 deep-scrub starts 2024-02-05T07:28:46.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[37158]: 3.10 deep-scrub ok 2024-02-05T07:28:46.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[37158]: 8.4 scrub starts 2024-02-05T07:28:46.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[37158]: 1.13 deep-scrub starts 2024-02-05T07:28:46.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[37158]: 1.13 deep-scrub ok 2024-02-05T07:28:46.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[37158]: osdmap e320: 8 total, 8 up, 7 in 2024-02-05T07:28:46.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[37158]: 3.17 scrub starts 2024-02-05T07:28:46.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[37158]: 3.17 scrub ok 2024-02-05T07:28:46.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:46 smithi007 ceph-mon[37158]: osdmap e321: 8 total, 8 up, 7 in 2024-02-05T07:28:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:46.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:46.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:46.984 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:46.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:46.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:46 smithi090 ceph-mon[33213]: 1.d deep-scrub starts 2024-02-05T07:28:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:46 smithi090 ceph-mon[33213]: 1.d deep-scrub ok 2024-02-05T07:28:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:46 smithi090 ceph-mon[33213]: 3.10 deep-scrub starts 2024-02-05T07:28:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:46 smithi090 ceph-mon[33213]: 3.10 deep-scrub ok 2024-02-05T07:28:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:46 smithi090 ceph-mon[33213]: 8.4 scrub starts 2024-02-05T07:28:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:46 smithi090 ceph-mon[33213]: 1.13 deep-scrub starts 2024-02-05T07:28:47.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:46 smithi090 ceph-mon[33213]: 1.13 deep-scrub ok 2024-02-05T07:28:47.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:46 smithi090 ceph-mon[33213]: osdmap e320: 8 total, 8 up, 7 in 2024-02-05T07:28:47.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:46 smithi090 ceph-mon[33213]: 3.17 scrub starts 2024-02-05T07:28:47.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:46 smithi090 ceph-mon[33213]: 3.17 scrub ok 2024-02-05T07:28:47.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:46 smithi090 ceph-mon[33213]: osdmap e321: 8 total, 8 up, 7 in 2024-02-05T07:28:47.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:47.115 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:47.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:47.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:47.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:47.245 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:47.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:47.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:47.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:47.390 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:47.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:47.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:47.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:47.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:47.635 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:47.663 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:47.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:47.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:47.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:47.807 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:47.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:47.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:47.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:47.944 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:47.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:47.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:47 smithi090 ceph-mon[33213]: 1.2 scrub starts 2024-02-05T07:28:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:47 smithi090 ceph-mon[33213]: 1.2 scrub ok 2024-02-05T07:28:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:47 smithi090 ceph-mon[33213]: 1.b scrub starts 2024-02-05T07:28:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:47 smithi090 ceph-mon[33213]: 1.b scrub ok 2024-02-05T07:28:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:47 smithi090 ceph-mon[33213]: 4.1d deep-scrub starts 2024-02-05T07:28:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:47 smithi090 ceph-mon[33213]: 4.1d deep-scrub ok 2024-02-05T07:28:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:47 smithi090 ceph-mon[33213]: pgmap v518: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 2868/145656 objects degraded (1.969%); 16654/145656 objects misplaced (11.434%) 2024-02-05T07:28:48.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:48.073 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:48.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:48.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:48.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:47 smithi007 ceph-mon[29098]: 1.2 scrub starts 2024-02-05T07:28:48.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:47 smithi007 ceph-mon[29098]: 1.2 scrub ok 2024-02-05T07:28:48.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:47 smithi007 ceph-mon[29098]: 1.b scrub starts 2024-02-05T07:28:48.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:47 smithi007 ceph-mon[29098]: 1.b scrub ok 2024-02-05T07:28:48.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:47 smithi007 ceph-mon[29098]: 4.1d deep-scrub starts 2024-02-05T07:28:48.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:47 smithi007 ceph-mon[29098]: 4.1d deep-scrub ok 2024-02-05T07:28:48.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:47 smithi007 ceph-mon[29098]: pgmap v518: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 2868/145656 objects degraded (1.969%); 16654/145656 objects misplaced (11.434%) 2024-02-05T07:28:48.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:47 smithi007 ceph-mon[37158]: 1.2 scrub starts 2024-02-05T07:28:48.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:47 smithi007 ceph-mon[37158]: 1.2 scrub ok 2024-02-05T07:28:48.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:47 smithi007 ceph-mon[37158]: 1.b scrub starts 2024-02-05T07:28:48.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:47 smithi007 ceph-mon[37158]: 1.b scrub ok 2024-02-05T07:28:48.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:47 smithi007 ceph-mon[37158]: 4.1d deep-scrub starts 2024-02-05T07:28:48.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:47 smithi007 ceph-mon[37158]: 4.1d deep-scrub ok 2024-02-05T07:28:48.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:47 smithi007 ceph-mon[37158]: pgmap v518: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 2868/145656 objects degraded (1.969%); 16654/145656 objects misplaced (11.434%) 2024-02-05T07:28:48.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:48.210 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:48.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:48.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:48.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:48.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:48.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:48.471 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:48.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:48.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:48.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:48.613 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:48.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:48.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:48.715 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:48.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:48.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:48.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:48.886 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:48.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:48.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[29098]: 1.0 scrub starts 2024-02-05T07:28:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[29098]: 1.0 scrub ok 2024-02-05T07:28:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[29098]: 1.f scrub starts 2024-02-05T07:28:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[29098]: 1.f scrub ok 2024-02-05T07:28:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[29098]: 1.3 scrub starts 2024-02-05T07:28:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[29098]: 1.3 scrub ok 2024-02-05T07:28:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[29098]: 8.4 scrub starts 2024-02-05T07:28:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[29098]: 8.4 scrub ok 2024-02-05T07:28:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[29098]: 6.b scrub starts 2024-02-05T07:28:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[29098]: 6.b scrub ok 2024-02-05T07:28:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[29098]: 1.12 scrub starts 2024-02-05T07:28:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[37158]: 1.0 scrub starts 2024-02-05T07:28:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[37158]: 1.0 scrub ok 2024-02-05T07:28:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[37158]: 1.f scrub starts 2024-02-05T07:28:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[37158]: 1.f scrub ok 2024-02-05T07:28:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[37158]: 1.3 scrub starts 2024-02-05T07:28:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[37158]: 1.3 scrub ok 2024-02-05T07:28:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[37158]: 8.4 scrub starts 2024-02-05T07:28:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[37158]: 8.4 scrub ok 2024-02-05T07:28:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[37158]: 6.b scrub starts 2024-02-05T07:28:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[37158]: 6.b scrub ok 2024-02-05T07:28:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:48 smithi007 ceph-mon[37158]: 1.12 scrub starts 2024-02-05T07:28:48.979 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:28:48.988 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:48 smithi090 ceph-mon[33213]: 1.0 scrub starts 2024-02-05T07:28:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:48 smithi090 ceph-mon[33213]: 1.0 scrub ok 2024-02-05T07:28:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:48 smithi090 ceph-mon[33213]: 1.f scrub starts 2024-02-05T07:28:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:48 smithi090 ceph-mon[33213]: 1.f scrub ok 2024-02-05T07:28:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:48 smithi090 ceph-mon[33213]: 1.3 scrub starts 2024-02-05T07:28:49.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:48 smithi090 ceph-mon[33213]: 1.3 scrub ok 2024-02-05T07:28:49.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:48 smithi090 ceph-mon[33213]: 8.4 scrub starts 2024-02-05T07:28:49.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:48 smithi090 ceph-mon[33213]: 8.4 scrub ok 2024-02-05T07:28:49.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:48 smithi090 ceph-mon[33213]: 6.b scrub starts 2024-02-05T07:28:49.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:48 smithi090 ceph-mon[33213]: 6.b scrub ok 2024-02-05T07:28:49.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:48 smithi090 ceph-mon[33213]: 1.12 scrub starts 2024-02-05T07:28:49.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T07:28:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:49.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:49.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:49.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:49.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:49.202 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:49.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:49.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:49.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:28:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:49.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:49.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:49.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:49.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:49.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:49.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:49.613 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:49.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:49.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:49.746 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:49.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:49.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:49.923 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:49.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:49.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[29098]: 8.10 scrub starts 2024-02-05T07:28:49.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[29098]: 1.7 scrub starts 2024-02-05T07:28:49.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[29098]: 1.7 scrub ok 2024-02-05T07:28:49.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[29098]: 8.10 scrub ok 2024-02-05T07:28:49.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[29098]: 1.12 scrub ok 2024-02-05T07:28:49.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[29098]: pgmap v519: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 2868/145215 objects degraded (1.975%); 16654/145215 objects misplaced (11.469%) 2024-02-05T07:28:49.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 2868/145656 objects degraded (1.969%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:49.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[29098]: osdmap e322: 8 total, 8 up, 7 in 2024-02-05T07:28:49.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[29098]: 6.1 scrub starts 2024-02-05T07:28:49.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[29098]: 6.1 scrub ok 2024-02-05T07:28:49.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[37158]: 8.10 scrub starts 2024-02-05T07:28:49.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[37158]: 1.7 scrub starts 2024-02-05T07:28:49.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[37158]: 1.7 scrub ok 2024-02-05T07:28:49.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[37158]: 8.10 scrub ok 2024-02-05T07:28:49.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[37158]: 1.12 scrub ok 2024-02-05T07:28:49.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[37158]: pgmap v519: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 2868/145215 objects degraded (1.975%); 16654/145215 objects misplaced (11.469%) 2024-02-05T07:28:49.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 2868/145656 objects degraded (1.969%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:49.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[37158]: osdmap e322: 8 total, 8 up, 7 in 2024-02-05T07:28:49.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[37158]: 6.1 scrub starts 2024-02-05T07:28:49.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:49 smithi007 ceph-mon[37158]: 6.1 scrub ok 2024-02-05T07:28:50.024 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:50.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:49 smithi090 ceph-mon[33213]: 8.10 scrub starts 2024-02-05T07:28:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:49 smithi090 ceph-mon[33213]: 1.7 scrub starts 2024-02-05T07:28:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:49 smithi090 ceph-mon[33213]: 1.7 scrub ok 2024-02-05T07:28:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:49 smithi090 ceph-mon[33213]: 8.10 scrub ok 2024-02-05T07:28:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:49 smithi090 ceph-mon[33213]: 1.12 scrub ok 2024-02-05T07:28:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:49 smithi090 ceph-mon[33213]: pgmap v519: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 2868/145215 objects degraded (1.975%); 16654/145215 objects misplaced (11.469%) 2024-02-05T07:28:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:49 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 2868/145656 objects degraded (1.969%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:49 smithi090 ceph-mon[33213]: osdmap e322: 8 total, 8 up, 7 in 2024-02-05T07:28:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:49 smithi090 ceph-mon[33213]: 6.1 scrub starts 2024-02-05T07:28:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:49 smithi090 ceph-mon[33213]: 6.1 scrub ok 2024-02-05T07:28:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:50.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:50.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:50.218 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:50.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:50.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:50.321 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:50.360 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:50.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:50.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:50.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:50.497 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:50.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:50.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:50.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:28:50.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:50.711 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:28:50.714 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:50.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:28:50.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:28:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:28:50.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:50 smithi090 ceph-mon[33213]: 4.8 scrub starts 2024-02-05T07:28:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:50 smithi090 ceph-mon[33213]: 4.8 scrub ok 2024-02-05T07:28:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:50 smithi090 ceph-mon[33213]: 1.6 scrub starts 2024-02-05T07:28:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:50 smithi090 ceph-mon[33213]: 1.6 scrub ok 2024-02-05T07:28:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:50 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:50 smithi090 ceph-mon[33213]: 6.4 scrub starts 2024-02-05T07:28:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:50 smithi090 ceph-mon[33213]: 6.4 scrub ok 2024-02-05T07:28:50.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:50.853 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:50.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:50.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:50 smithi007 ceph-mon[29098]: 4.8 scrub starts 2024-02-05T07:28:50.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:50 smithi007 ceph-mon[29098]: 4.8 scrub ok 2024-02-05T07:28:50.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:50 smithi007 ceph-mon[29098]: 1.6 scrub starts 2024-02-05T07:28:50.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:50 smithi007 ceph-mon[29098]: 1.6 scrub ok 2024-02-05T07:28:50.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:50 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:50.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:50 smithi007 ceph-mon[29098]: 6.4 scrub starts 2024-02-05T07:28:50.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:50 smithi007 ceph-mon[29098]: 6.4 scrub ok 2024-02-05T07:28:50.862 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:50 smithi007 ceph-mon[37158]: 4.8 scrub starts 2024-02-05T07:28:50.862 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:50 smithi007 ceph-mon[37158]: 4.8 scrub ok 2024-02-05T07:28:50.862 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:50 smithi007 ceph-mon[37158]: 1.6 scrub starts 2024-02-05T07:28:50.862 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:50 smithi007 ceph-mon[37158]: 1.6 scrub ok 2024-02-05T07:28:50.862 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:50 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:28:50.862 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:50 smithi007 ceph-mon[37158]: 6.4 scrub starts 2024-02-05T07:28:50.862 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:50 smithi007 ceph-mon[37158]: 6.4 scrub ok 2024-02-05T07:28:50.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:51.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:51.007 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:51.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:51.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:51.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:51.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:51.294 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:51.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:51.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:51.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:51.426 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:51.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:51.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:51.527 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:51.560 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:51.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:51.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:51.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:51.693 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:51.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:51.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:51.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:51.830 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:51.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:51.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:51.968 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:51.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:52.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:52.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:51 smithi007 ceph-mon[29098]: 5.9 scrub starts 2024-02-05T07:28:52.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:51 smithi007 ceph-mon[29098]: 5.9 scrub ok 2024-02-05T07:28:52.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:51 smithi007 ceph-mon[29098]: 8.5 scrub starts 2024-02-05T07:28:52.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:51 smithi007 ceph-mon[29098]: 8.5 scrub ok 2024-02-05T07:28:52.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:51 smithi007 ceph-mon[29098]: pgmap v521: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2527/130254 objects degraded (1.940%); 14874/130254 objects misplaced (11.419%) 2024-02-05T07:28:52.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:51 smithi007 ceph-mon[37158]: 5.9 scrub starts 2024-02-05T07:28:52.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:51 smithi007 ceph-mon[37158]: 5.9 scrub ok 2024-02-05T07:28:52.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:51 smithi007 ceph-mon[37158]: 8.5 scrub starts 2024-02-05T07:28:52.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:51 smithi007 ceph-mon[37158]: 8.5 scrub ok 2024-02-05T07:28:52.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:51 smithi007 ceph-mon[37158]: pgmap v521: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2527/130254 objects degraded (1.940%); 14874/130254 objects misplaced (11.419%) 2024-02-05T07:28:52.100 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:52.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:52.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:52.243 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:52.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:51 smithi090 ceph-mon[33213]: 5.9 scrub starts 2024-02-05T07:28:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:51 smithi090 ceph-mon[33213]: 5.9 scrub ok 2024-02-05T07:28:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:51 smithi090 ceph-mon[33213]: 8.5 scrub starts 2024-02-05T07:28:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:51 smithi090 ceph-mon[33213]: 8.5 scrub ok 2024-02-05T07:28:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:51 smithi090 ceph-mon[33213]: pgmap v521: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2527/130254 objects degraded (1.940%); 14874/130254 objects misplaced (11.419%) 2024-02-05T07:28:52.344 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:52.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:52.387 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:52.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:52.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:52.528 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:52.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:52.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:52.630 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:52.701 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:52.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:52.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:52.857 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:52.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:52.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:52.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:52.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:52.999 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:53.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:53.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:53.127 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:53.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:53.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:53.259 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:53.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:53.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:53.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:53.389 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:53.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:53.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:53.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:53.519 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:53.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:53.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:53.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:53.652 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:53.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:53.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:53.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:28:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:28:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:28:53.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:53.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:53.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:53.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:53.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:53.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:54.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:54.070 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:54.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:54.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:54.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:54.204 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:54.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.0 to 1 (10000) 2024-02-05T07:28:54.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:54.331 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:54.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:54.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:54.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:54.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:54.471 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:54.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:54 smithi090 ceph-mon[33213]: 6.1f scrub starts 2024-02-05T07:28:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:54 smithi090 ceph-mon[33213]: 6.14 scrub starts 2024-02-05T07:28:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:54 smithi090 ceph-mon[33213]: 6.14 scrub ok 2024-02-05T07:28:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:54 smithi090 ceph-mon[33213]: 6.1f scrub ok 2024-02-05T07:28:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:54 smithi090 ceph-mon[33213]: pgmap v522: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2527/130254 objects degraded (1.940%); 14874/130254 objects misplaced (11.419%) 2024-02-05T07:28:54.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:54 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2588034635' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:28:54.526 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:54 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:28:54.572 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:54.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:54.605 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:54.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:54.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:54 smithi007 ceph-mon[29098]: 6.1f scrub starts 2024-02-05T07:28:54.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:54 smithi007 ceph-mon[29098]: 6.14 scrub starts 2024-02-05T07:28:54.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:54 smithi007 ceph-mon[29098]: 6.14 scrub ok 2024-02-05T07:28:54.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:54 smithi007 ceph-mon[29098]: 6.1f scrub ok 2024-02-05T07:28:54.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:54 smithi007 ceph-mon[29098]: pgmap v522: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2527/130254 objects degraded (1.940%); 14874/130254 objects misplaced (11.419%) 2024-02-05T07:28:54.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:54 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2588034635' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:28:54.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:54 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:28:54.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:54 smithi007 ceph-mon[37158]: 6.1f scrub starts 2024-02-05T07:28:54.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:54 smithi007 ceph-mon[37158]: 6.14 scrub starts 2024-02-05T07:28:54.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:54 smithi007 ceph-mon[37158]: 6.14 scrub ok 2024-02-05T07:28:54.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:54 smithi007 ceph-mon[37158]: 6.1f scrub ok 2024-02-05T07:28:54.659 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:54 smithi007 ceph-mon[37158]: pgmap v522: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2527/130254 objects degraded (1.940%); 14874/130254 objects misplaced (11.419%) 2024-02-05T07:28:54.659 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:54 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2588034635' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:28:54.659 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:54 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:28:54.706 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:54.736 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:54.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:54.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:54.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:54.873 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:54.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:54.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:55.010 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:55.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:55.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:55.144 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:55.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:55.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:55.298 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:55.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:55.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:55.390 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:55 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T07:28:55.390 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:55 smithi007 ceph-mon[29098]: osdmap e323: 8 total, 8 up, 7 in 2024-02-05T07:28:55.391 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:55 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 2527/130254 objects degraded (1.940%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:55.391 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:55 smithi007 ceph-mon[29098]: osdmap e324: 8 total, 8 up, 7 in 2024-02-05T07:28:55.391 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:55 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T07:28:55.391 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:55 smithi007 ceph-mon[37158]: osdmap e323: 8 total, 8 up, 7 in 2024-02-05T07:28:55.391 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:55 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 2527/130254 objects degraded (1.940%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:55.391 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:55 smithi007 ceph-mon[37158]: osdmap e324: 8 total, 8 up, 7 in 2024-02-05T07:28:55.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:55.442 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:55.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:55.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:55 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T07:28:55.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:55 smithi090 ceph-mon[33213]: osdmap e323: 8 total, 8 up, 7 in 2024-02-05T07:28:55.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:55 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 2527/130254 objects degraded (1.940%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:55.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:55 smithi090 ceph-mon[33213]: osdmap e324: 8 total, 8 up, 7 in 2024-02-05T07:28:55.544 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:55.579 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:55.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:55.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:55.706 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:55.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:55.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:55.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:55.835 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:55.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:55.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:55.966 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:55.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:55.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:56.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:56.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:56.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:56.186 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:28:56.186 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-02-05T07:28:56.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-02-05T07:28:56.187 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:28:56.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.187 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75577 7.12 7:4efb7e56:::benchmark_data_smithi007_77086_object75576:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T07:28:56.187 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:44.132711+0000", 2024-02-05T07:28:56.187 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 432.04400789200002, 2024-02-05T07:28:56.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2883520879999999, 2024-02-05T07:28:56.187 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.187 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:28:56.187 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.187 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:56.187 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:56.187 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75577 2024-02-05T07:28:56.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.187 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.132711+0000", 2024-02-05T07:28:56.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.132711+0000", 2024-02-05T07:28:56.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.132713+0000", 2024-02-05T07:28:56.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9259999999999999e-06 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.132741+0000", 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8317e-05 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.132755+0000", 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3692999999999999e-05 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.132766+0000", 2024-02-05T07:28:56.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0781999999999999e-05 2024-02-05T07:28:56.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134112+0000", 2024-02-05T07:28:56.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0013459679999999999 2024-02-05T07:28:56.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134122+0000", 2024-02-05T07:28:56.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.028e-05 2024-02-05T07:28:56.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.420935+0000", 2024-02-05T07:28:56.191 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.286812718 2024-02-05T07:28:56.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.191 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.191 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.421063+0000", 2024-02-05T07:28:56.191 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012840400000000001 2024-02-05T07:28:56.191 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.191 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.191 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.191 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75586 7.12 7:4ad7f210:::benchmark_data_smithi007_77086_object75585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T07:28:56.191 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:44.132780+0000", 2024-02-05T07:28:56.192 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 432.04393900500003, 2024-02-05T07:28:56.192 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2884912210000001, 2024-02-05T07:28:56.192 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.192 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:28:56.192 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.192 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:56.192 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:56.192 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75586 2024-02-05T07:28:56.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.192 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.132780+0000", 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.132780+0000", 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.132780+0000", 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6700000000000003e-07 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.132801+0000", 2024-02-05T07:28:56.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1090999999999998e-05 2024-02-05T07:28:56.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.132809+0000", 2024-02-05T07:28:56.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3130000000000001e-06 2024-02-05T07:28:56.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.132815+0000", 2024-02-05T07:28:56.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5150000000000003e-06 2024-02-05T07:28:56.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134146+0000", 2024-02-05T07:28:56.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0013307880000000001 2024-02-05T07:28:56.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.134152+0000", 2024-02-05T07:28:56.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.438e-06 2024-02-05T07:28:56.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.421153+0000", 2024-02-05T07:28:56.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2870009769999999 2024-02-05T07:28:56.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.421271+0000", 2024-02-05T07:28:56.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000117532 2024-02-05T07:28:56.196 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.196 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.196 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:172546 7.2 7:4660c1cb:::benchmark_data_smithi007_77086_object67951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:56.370448+0000", 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 299.80627047799999, 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0001107970000001, 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 172546 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.370448+0000", 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.370448+0000", 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.370450+0000", 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8360000000000001e-06 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.370455+0000", 2024-02-05T07:28:56.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8520000000000003e-06 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.370466+0000", 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0978999999999999e-05 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.370474+0000", 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1289999999999996e-06 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.370533+0000", 2024-02-05T07:28:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9249000000000001e-05 2024-02-05T07:28:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.370542+0000", 2024-02-05T07:28:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8030000000000003e-06 2024-02-05T07:28:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.365602+0000", 2024-02-05T07:28:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99506004800000003 2024-02-05T07:28:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.365736+0000", 2024-02-05T07:28:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000133547 2024-02-05T07:28:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,7", 2024-02-05T07:28:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.366117+0000", 2024-02-05T07:28:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00038165999999999997 2024-02-05T07:28:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.368183+0000", 2024-02-05T07:28:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002065168 2024-02-05T07:28:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.370355+0000", 2024-02-05T07:28:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021720519999999998 2024-02-05T07:28:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.370475+0000", 2024-02-05T07:28:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000120554 2024-02-05T07:28:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.370522+0000", 2024-02-05T07:28:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7154000000000003e-05 2024-02-05T07:28:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.370559+0000", 2024-02-05T07:28:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6766e-05 2024-02-05T07:28:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:172563 7.2 7:44166808:::benchmark_data_smithi007_77086_object67968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-02-05T07:28:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:56.387010+0000", 2024-02-05T07:28:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 299.78970867800001, 2024-02-05T07:28:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98658510799999999, 2024-02-05T07:28:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 172563 2024-02-05T07:28:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.387010+0000", 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.387010+0000", 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.387012+0000", 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.59e-06 2024-02-05T07:28:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.387017+0000", 2024-02-05T07:28:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9060000000000001e-06 2024-02-05T07:28:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.387030+0000", 2024-02-05T07:28:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3402e-05 2024-02-05T07:28:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.387038+0000", 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3990000000000004e-06 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.387307+0000", 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026843 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.387315+0000", 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6770000000000001e-06 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.368212+0000", 2024-02-05T07:28:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.980896518 2024-02-05T07:28:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.368319+0000", 2024-02-05T07:28:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010734900000000001 2024-02-05T07:28:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,7", 2024-02-05T07:28:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.368610+0000", 2024-02-05T07:28:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029092799999999998 2024-02-05T07:28:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.373268+0000", 2024-02-05T07:28:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0046576689999999997 2024-02-05T07:28:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.373312+0000", 2024-02-05T07:28:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3979e-05 2024-02-05T07:28:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.373400+0000", 2024-02-05T07:28:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7693000000000002e-05 2024-02-05T07:28:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.373545+0000", 2024-02-05T07:28:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014490799999999999 2024-02-05T07:28:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.373595+0000", 2024-02-05T07:28:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0659999999999999e-05 2024-02-05T07:28:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:172588 7.2 7:40d861ec:::benchmark_data_smithi007_77086_object67993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-02-05T07:28:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:56.414182+0000", 2024-02-05T07:28:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 299.76253693500001, 2024-02-05T07:28:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96319614499999995, 2024-02-05T07:28:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:28:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:28:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 172588 2024-02-05T07:28:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414182+0000", 2024-02-05T07:28:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414182+0000", 2024-02-05T07:28:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414183+0000", 2024-02-05T07:28:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3699999999999999e-07 2024-02-05T07:28:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414187+0000", 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3379999999999998e-06 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414200+0000", 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3366e-05 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414209+0000", 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0790000000000007e-06 2024-02-05T07:28:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414301+0000", 2024-02-05T07:28:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.145e-05 2024-02-05T07:28:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414311+0000", 2024-02-05T07:28:56.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7729999999999994e-06 2024-02-05T07:28:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.370873+0000", 2024-02-05T07:28:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.95656245900000003 2024-02-05T07:28:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.370970+0000", 2024-02-05T07:28:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7344999999999997e-05 2024-02-05T07:28:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,7", 2024-02-05T07:28:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.371476+0000", 2024-02-05T07:28:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00050597299999999995 2024-02-05T07:28:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.376010+0000", 2024-02-05T07:28:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.004533451 2024-02-05T07:28:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.376024+0000", 2024-02-05T07:28:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4061e-05 2024-02-05T07:28:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.377287+0000", 2024-02-05T07:28:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012634219999999999 2024-02-05T07:28:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.377339+0000", 2024-02-05T07:28:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2003999999999999e-05 2024-02-05T07:28:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.377378+0000", 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8587000000000002e-05 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:131866 8.17 8:ea0833c6:::benchmark_data_smithi007_105755_object131865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:26:56.260384+0000", 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 119.91633469200001, 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.071054945, 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 131866 2024-02-05T07:28:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.260384+0000", 2024-02-05T07:28:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.260384+0000", 2024-02-05T07:28:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.260386+0000", 2024-02-05T07:28:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5629999999999999e-06 2024-02-05T07:28:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.260434+0000", 2024-02-05T07:28:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8469e-05 2024-02-05T07:28:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.260446+0000", 2024-02-05T07:28:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.203e-05 2024-02-05T07:28:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.260457+0000", 2024-02-05T07:28:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1324e-05 2024-02-05T07:28:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.262650+0000", 2024-02-05T07:28:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021922439999999999 2024-02-05T07:28:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.262660+0000", 2024-02-05T07:28:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0495e-05 2024-02-05T07:28:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.324707+0000", 2024-02-05T07:28:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.062046332 2024-02-05T07:28:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.324914+0000", 2024-02-05T07:28:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020736300000000001 2024-02-05T07:28:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,2,7", 2024-02-05T07:28:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.325142+0000", 2024-02-05T07:28:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000227932 2024-02-05T07:28:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.326680+0000", 2024-02-05T07:28:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0015378950000000001 2024-02-05T07:28:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.326762+0000", 2024-02-05T07:28:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2738000000000006e-05 2024-02-05T07:28:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.327188+0000", 2024-02-05T07:28:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00042566799999999998 2024-02-05T07:28:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.331289+0000", 2024-02-05T07:28:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041006549999999999 2024-02-05T07:28:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.331328+0000", 2024-02-05T07:28:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9504e-05 2024-02-05T07:28:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.331439+0000", 2024-02-05T07:28:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000110733 2024-02-05T07:28:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:131868 8.10 8:088d97f8:::benchmark_data_smithi007_105755_object131867:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:28:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:26:56.260472+0000", 2024-02-05T07:28:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 119.916247152, 2024-02-05T07:28:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0607989200000001, 2024-02-05T07:28:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 131868 2024-02-05T07:28:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.260472+0000", 2024-02-05T07:28:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.260472+0000", 2024-02-05T07:28:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.260472+0000", 2024-02-05T07:28:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2099999999999997e-07 2024-02-05T07:28:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.260504+0000", 2024-02-05T07:28:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1678999999999999e-05 2024-02-05T07:28:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.260511+0000", 2024-02-05T07:28:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5700000000000004e-06 2024-02-05T07:28:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.260518+0000", 2024-02-05T07:28:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3319999999999997e-06 2024-02-05T07:28:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.264104+0000", 2024-02-05T07:28:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0035858040000000002 2024-02-05T07:28:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.264113+0000", 2024-02-05T07:28:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7650000000000005e-06 2024-02-05T07:28:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.310841+0000", 2024-02-05T07:28:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.046727287 2024-02-05T07:28:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.311018+0000", 2024-02-05T07:28:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017785300000000001 2024-02-05T07:28:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,5,7", 2024-02-05T07:28:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.311391+0000", 2024-02-05T07:28:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000372283 2024-02-05T07:28:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.313478+0000", 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002087588 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.314681+0000", 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012030739999999999 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.314896+0000", 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00021430099999999999 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.320845+0000", 2024-02-05T07:28:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059493139999999998 2024-02-05T07:28:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.320938+0000", 2024-02-05T07:28:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3475999999999997e-05 2024-02-05T07:28:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.321271+0000", 2024-02-05T07:28:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000332073 2024-02-05T07:28:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:131876 8.13 8:cb6d1103:::benchmark_data_smithi007_105755_object131875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:28:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:26:56.263463+0000", 2024-02-05T07:28:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 119.91325622799999, 2024-02-05T07:28:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.066059544, 2024-02-05T07:28:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 131876 2024-02-05T07:28:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.263463+0000", 2024-02-05T07:28:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.263463+0000", 2024-02-05T07:28:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.263464+0000", 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2899999999999999e-06 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.263492+0000", 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.813e-05 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.263503+0000", 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0827e-05 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.263512+0000", 2024-02-05T07:28:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0140000000000004e-06 2024-02-05T07:28:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.263609+0000", 2024-02-05T07:28:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6818999999999996e-05 2024-02-05T07:28:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:56.263619+0000", 2024-02-05T07:28:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0581000000000001e-05 2024-02-05T07:28:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.320056+0000", 2024-02-05T07:28:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.056436382 2024-02-05T07:28:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.320165+0000", 2024-02-05T07:28:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010946399999999999 2024-02-05T07:28:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-02-05T07:28:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.320378+0000", 2024-02-05T07:28:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00021332499999999999 2024-02-05T07:28:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.324269+0000", 2024-02-05T07:28:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003890363 2024-02-05T07:28:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.324290+0000", 2024-02-05T07:28:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1654999999999998e-05 2024-02-05T07:28:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.329388+0000", 2024-02-05T07:28:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0050971979999999998 2024-02-05T07:28:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.329427+0000", 2024-02-05T07:28:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9436e-05 2024-02-05T07:28:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:26:57.329522+0000", 2024-02-05T07:28:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5060000000000001e-05 2024-02-05T07:28:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:138169 8.13 8:cc44bdbd:::benchmark_data_smithi007_105755_object1859:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-02-05T07:28:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:02.331588+0000", 2024-02-05T07:28:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 113.84513073799999, 2024-02-05T07:28:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0803532920000001, 2024-02-05T07:28:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:28:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.252 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.252 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.252 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 138169 2024-02-05T07:28:56.252 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.252 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.252 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.331588+0000", 2024-02-05T07:28:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.331588+0000", 2024-02-05T07:28:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.331589+0000", 2024-02-05T07:28:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0389999999999999e-06 2024-02-05T07:28:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.331594+0000", 2024-02-05T07:28:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7419999999999997e-06 2024-02-05T07:28:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.331604+0000", 2024-02-05T07:28:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0397999999999999e-05 2024-02-05T07:28:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.331613+0000", 2024-02-05T07:28:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3219999999999993e-06 2024-02-05T07:28:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.331669+0000", 2024-02-05T07:28:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6817000000000002e-05 2024-02-05T07:28:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.331803+0000", 2024-02-05T07:28:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013414199999999999 2024-02-05T07:28:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-02-05T07:28:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.332668+0000", 2024-02-05T07:28:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00086457500000000002 2024-02-05T07:28:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.361391+0000", 2024-02-05T07:28:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.028723249999999999 2024-02-05T07:28:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.361399+0000", 2024-02-05T07:28:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9610000000000005e-06 2024-02-05T07:28:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:03.411846+0000", 2024-02-05T07:28:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.050446499 2024-02-05T07:28:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:03.411941+0000", 2024-02-05T07:28:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5546999999999998e-05 2024-02-05T07:28:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:138175 8.13 8:ce020f36:::benchmark_data_smithi007_105755_object1865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-02-05T07:28:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:02.345834+0000", 2024-02-05T07:28:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 113.830884459, 2024-02-05T07:28:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0773882349999999, 2024-02-05T07:28:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 138175 2024-02-05T07:28:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.345834+0000", 2024-02-05T07:28:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.345834+0000", 2024-02-05T07:28:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.345836+0000", 2024-02-05T07:28:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5570000000000001e-06 2024-02-05T07:28:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.345841+0000", 2024-02-05T07:28:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.383e-06 2024-02-05T07:28:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.345851+0000", 2024-02-05T07:28:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5810000000000003e-06 2024-02-05T07:28:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.345860+0000", 2024-02-05T07:28:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8869999999999998e-06 2024-02-05T07:28:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.361423+0000", 2024-02-05T07:28:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.01556281 2024-02-05T07:28:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:02.361429+0000", 2024-02-05T07:28:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2790000000000004e-06 2024-02-05T07:28:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:03.412017+0000", 2024-02-05T07:28:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.050588144 2024-02-05T07:28:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:03.412336+0000", 2024-02-05T07:28:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00031929700000000002 2024-02-05T07:28:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,4,5", 2024-02-05T07:28:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:03.412609+0000", 2024-02-05T07:28:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00027261100000000002 2024-02-05T07:28:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:03.415344+0000", 2024-02-05T07:28:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002734691 2024-02-05T07:28:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:03.415381+0000", 2024-02-05T07:28:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7919000000000001e-05 2024-02-05T07:28:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:03.415424+0000", 2024-02-05T07:28:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2718e-05 2024-02-05T07:28:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:03.422980+0000", 2024-02-05T07:28:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0075562069999999997 2024-02-05T07:28:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:03.423111+0000", 2024-02-05T07:28:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013083100000000001 2024-02-05T07:28:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:03.423223+0000", 2024-02-05T07:28:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011132 2024-02-05T07:28:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151898 8.a 8:55693220:::benchmark_data_smithi007_105755_object15588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:28:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:15.630004+0000", 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 100.546715029, 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2830786679999999, 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151898 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.630004+0000", 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.630004+0000", 2024-02-05T07:28:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.630005+0000", 2024-02-05T07:28:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.274e-06 2024-02-05T07:28:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.630010+0000", 2024-02-05T07:28:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8300000000000003e-06 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.630020+0000", 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0112000000000001e-05 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.630028+0000", 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9100000000000005e-06 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.630097+0000", 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9356000000000005e-05 2024-02-05T07:28:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.630232+0000", 2024-02-05T07:28:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013509000000000001 2024-02-05T07:28:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,4,5", 2024-02-05T07:28:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.630368+0000", 2024-02-05T07:28:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013537299999999999 2024-02-05T07:28:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.632445+0000", 2024-02-05T07:28:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020777510000000001 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.634801+0000", 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00235549 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.634892+0000", 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1478999999999996e-05 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.679416+0000", 2024-02-05T07:28:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.044523435 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.679434+0000", 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8173e-05 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.913000+0000", 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.233566197 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.913082+0000", 2024-02-05T07:28:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2198000000000001e-05 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151913 8.0 8:02c1afcf:::benchmark_data_smithi007_105755_object15603:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:15.730856+0000", 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 100.445862386, 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2341711399999999, 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151913 2024-02-05T07:28:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.730856+0000", 2024-02-05T07:28:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.730856+0000", 2024-02-05T07:28:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.730858+0000", 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.576e-06 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.730864+0000", 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.609e-06 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.730877+0000", 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3689000000000001e-05 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.730888+0000", 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0665999999999999e-05 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.732078+0000", 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0011899899999999999 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.732090+0000", 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2317000000000001e-05 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.911076+0000", 2024-02-05T07:28:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.178985615 2024-02-05T07:28:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.911220+0000", 2024-02-05T07:28:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014417999999999999 2024-02-05T07:28:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4,5", 2024-02-05T07:28:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.911545+0000", 2024-02-05T07:28:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032495700000000001 2024-02-05T07:28:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.915426+0000", 2024-02-05T07:28:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038810849999999998 2024-02-05T07:28:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.915472+0000", 2024-02-05T07:28:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6054000000000003e-05 2024-02-05T07:28:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.915666+0000", 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000193678 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.964920+0000", 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.049254439999999997 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.964963+0000", 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2688000000000002e-05 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.965028+0000", 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4596000000000003e-05 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151917 8.13 8:c8bc6974:::benchmark_data_smithi007_105755_object15607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:15.742346+0000", 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 100.43437229600001, 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0830120889999999, 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151917 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.742346+0000", 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.742346+0000", 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.742347+0000", 2024-02-05T07:28:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2099999999999997e-07 2024-02-05T07:28:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.742350+0000", 2024-02-05T07:28:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3890000000000001e-06 2024-02-05T07:28:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.742357+0000", 2024-02-05T07:28:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.516e-06 2024-02-05T07:28:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.742362+0000", 2024-02-05T07:28:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.31e-06 2024-02-05T07:28:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.742875+0000", 2024-02-05T07:28:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00051323700000000005 2024-02-05T07:28:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.742884+0000", 2024-02-05T07:28:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0669999999999998e-06 2024-02-05T07:28:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.821913+0000", 2024-02-05T07:28:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.079028189 2024-02-05T07:28:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.822057+0000", 2024-02-05T07:28:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000143838 2024-02-05T07:28:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,3,5", 2024-02-05T07:28:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.822387+0000", 2024-02-05T07:28:56.295 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000330272 2024-02-05T07:28:56.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.295 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.295 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.824220+0000", 2024-02-05T07:28:56.295 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001833302 2024-02-05T07:28:56.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.295 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.295 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.824541+0000", 2024-02-05T07:28:56.295 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000321262 2024-02-05T07:28:56.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.295 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.824583+0000", 2024-02-05T07:28:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1242999999999998e-05 2024-02-05T07:28:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.825138+0000", 2024-02-05T07:28:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00055543300000000003 2024-02-05T07:28:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.296 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.825259+0000", 2024-02-05T07:28:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012102 2024-02-05T07:28:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.825359+0000", 2024-02-05T07:28:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9489999999999998e-05 2024-02-05T07:28:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151918 8.a 8:5cb43259:::benchmark_data_smithi007_105755_object15608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-02-05T07:28:56.297 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:15.746176+0000", 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 100.43054243100001, 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.170439735, 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151918 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.746176+0000", 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.298 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.299 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.299 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.299 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.746176+0000", 2024-02-05T07:28:56.299 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.299 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.299 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.299 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.299 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.746177+0000", 2024-02-05T07:28:56.299 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.178e-06 2024-02-05T07:28:56.299 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.299 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.299 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.299 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.746182+0000", 2024-02-05T07:28:56.299 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.634e-06 2024-02-05T07:28:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.746193+0000", 2024-02-05T07:28:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1198e-05 2024-02-05T07:28:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.746202+0000", 2024-02-05T07:28:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2509999999999994e-06 2024-02-05T07:28:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.746264+0000", 2024-02-05T07:28:56.301 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2872000000000004e-05 2024-02-05T07:28:56.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.301 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.301 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.746276+0000", 2024-02-05T07:28:56.301 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1212999999999999e-05 2024-02-05T07:28:56.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.301 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.301 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.913475+0000", 2024-02-05T07:28:56.301 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1671993389999999 2024-02-05T07:28:56.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.302 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.302 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.913592+0000", 2024-02-05T07:28:56.302 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000117312 2024-02-05T07:28:56.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.302 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,3,5", 2024-02-05T07:28:56.302 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.913679+0000", 2024-02-05T07:28:56.302 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6635e-05 2024-02-05T07:28:56.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.302 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.302 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.915655+0000", 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0019758190000000002 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.915954+0000", 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029876300000000001 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.915999+0000", 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5599999999999997e-05 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.303 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.916463+0000", 2024-02-05T07:28:56.304 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00046401400000000001 2024-02-05T07:28:56.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.304 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.304 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.304 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.916505+0000", 2024-02-05T07:28:56.304 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1406999999999998e-05 2024-02-05T07:28:56.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.304 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.304 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.916616+0000", 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0001115 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151906 8.b 8:d61ef370:::benchmark_data_smithi007_105755_object15596:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e264)", 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:16.705028+0000", 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 99.471691214000003, 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1884909340000001, 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.305 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.306 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.306 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151906 2024-02-05T07:28:56.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.306 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.306 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.306 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.306 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.705028+0000", 2024-02-05T07:28:56.306 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.306 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.306 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.307 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.705028+0000", 2024-02-05T07:28:56.307 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.307 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.307 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.307 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.309 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.705029+0000", 2024-02-05T07:28:56.309 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7060000000000001e-06 2024-02-05T07:28:56.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.310 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.310 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.310 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.705037+0000", 2024-02-05T07:28:56.310 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3370000000000002e-06 2024-02-05T07:28:56.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.311 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.311 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.311 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.705049+0000", 2024-02-05T07:28:56.311 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2427e-05 2024-02-05T07:28:56.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.312 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.312 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.312 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.705060+0000", 2024-02-05T07:28:56.312 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.11e-05 2024-02-05T07:28:56.312 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.312 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.312 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.312 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.743469+0000", 2024-02-05T07:28:56.312 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.038408956000000001 2024-02-05T07:28:56.312 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.312 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.313 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.313 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.743480+0000", 2024-02-05T07:28:56.313 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0526000000000001e-05 2024-02-05T07:28:56.313 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.313 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.314 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.314 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:17.892561+0000", 2024-02-05T07:28:56.314 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.149081601 2024-02-05T07:28:56.314 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.314 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.314 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.314 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:17.893518+0000", 2024-02-05T07:28:56.314 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00095728100000000004 2024-02-05T07:28:56.314 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.314 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.314 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.314 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.314 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151914 8.b 8:d49b7596:::benchmark_data_smithi007_105755_object15604:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e264)", 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:16.705075+0000", 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 99.471643689999993, 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.207748225, 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151914 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.315 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.705075+0000", 2024-02-05T07:28:56.316 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.316 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.316 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.316 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.316 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.705075+0000", 2024-02-05T07:28:56.316 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.316 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.316 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.316 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.316 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.705076+0000", 2024-02-05T07:28:56.316 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9800000000000004e-07 2024-02-05T07:28:56.316 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.316 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.317 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.317 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.705078+0000", 2024-02-05T07:28:56.317 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.937e-06 2024-02-05T07:28:56.317 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.317 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.317 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.317 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.705085+0000", 2024-02-05T07:28:56.317 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8020000000000003e-06 2024-02-05T07:28:56.317 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.317 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.317 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.317 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.705092+0000", 2024-02-05T07:28:56.317 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3219999999999998e-06 2024-02-05T07:28:56.317 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.318 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.318 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.318 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.743500+0000", 2024-02-05T07:28:56.318 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.038408711999999998 2024-02-05T07:28:56.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.318 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.318 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.318 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.743506+0000", 2024-02-05T07:28:56.318 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6250000000000004e-06 2024-02-05T07:28:56.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.318 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.318 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:17.893880+0000", 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1503737940000001 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:17.894262+0000", 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00038185900000000002 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,3,4", 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:17.894398+0000", 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000136756 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.319 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:17.902224+0000", 2024-02-05T07:28:56.320 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0078252370000000005 2024-02-05T07:28:56.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.320 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.320 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:17.902268+0000", 2024-02-05T07:28:56.320 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4894000000000001e-05 2024-02-05T07:28:56.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.320 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.320 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:17.908195+0000", 2024-02-05T07:28:56.320 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059266539999999999 2024-02-05T07:28:56.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.321 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.321 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:17.912648+0000", 2024-02-05T07:28:56.321 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044530309999999997 2024-02-05T07:28:56.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.321 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.321 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.321 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:17.912708+0000", 2024-02-05T07:28:56.321 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0315000000000001e-05 2024-02-05T07:28:56.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.321 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.321 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.322 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:17.912823+0000", 2024-02-05T07:28:56.322 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000114789 2024-02-05T07:28:56.322 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.322 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.322 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.322 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.322 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.322 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:165730 8.3 8:c218261b:::benchmark_data_smithi007_105755_object29420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-02-05T07:28:56.322 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:34.316652+0000", 2024-02-05T07:28:56.322 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 81.860067154000006, 2024-02-05T07:28:56.322 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.03501668, 2024-02-05T07:28:56.322 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.322 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.322 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 165730 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.316652+0000", 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.316652+0000", 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.323 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.324 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.324 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.324 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.316652+0000", 2024-02-05T07:28:56.324 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2999999999999999e-07 2024-02-05T07:28:56.324 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.324 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.324 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.324 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.316657+0000", 2024-02-05T07:28:56.324 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7539999999999997e-06 2024-02-05T07:28:56.324 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.324 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.324 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.324 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.316666+0000", 2024-02-05T07:28:56.325 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5750000000000001e-06 2024-02-05T07:28:56.325 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.325 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.325 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.325 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.316673+0000", 2024-02-05T07:28:56.325 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6690000000000006e-06 2024-02-05T07:28:56.325 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.325 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.325 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.325 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.316728+0000", 2024-02-05T07:28:56.325 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4447000000000002e-05 2024-02-05T07:28:56.325 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.325 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.325 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.316737+0000", 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.346e-06 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.346566+0000", 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.029828728 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.346697+0000", 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013077499999999999 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.326 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-02-05T07:28:56.327 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.347085+0000", 2024-02-05T07:28:56.327 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000388176 2024-02-05T07:28:56.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.327 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.327 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.327 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.349337+0000", 2024-02-05T07:28:56.327 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002252179 2024-02-05T07:28:56.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.327 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.327 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.327 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.351460+0000", 2024-02-05T07:28:56.327 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021227049999999999 2024-02-05T07:28:56.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.327 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.328 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.328 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.351503+0000", 2024-02-05T07:28:56.328 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3015000000000001e-05 2024-02-05T07:28:56.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.328 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.328 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.351565+0000", 2024-02-05T07:28:56.328 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2655000000000002e-05 2024-02-05T07:28:56.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.328 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.328 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.351668+0000", 2024-02-05T07:28:56.328 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010282599999999999 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:165753 8.3 8:c2fd3b2a:::benchmark_data_smithi007_105755_object29443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:34.350777+0000", 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 81.825941482999994, 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.015832048, 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.329 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 165753 2024-02-05T07:28:56.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.330 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.330 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.330 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.330 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.350777+0000", 2024-02-05T07:28:56.330 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.330 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.330 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.330 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.350777+0000", 2024-02-05T07:28:56.330 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.330 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.330 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.331 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.350778+0000", 2024-02-05T07:28:56.331 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.035e-06 2024-02-05T07:28:56.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.331 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.331 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.331 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.350783+0000", 2024-02-05T07:28:56.331 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0810000000000001e-06 2024-02-05T07:28:56.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.331 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.331 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.331 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.350795+0000", 2024-02-05T07:28:56.331 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1863e-05 2024-02-05T07:28:56.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.331 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.332 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.332 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.350803+0000", 2024-02-05T07:28:56.332 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8970000000000008e-06 2024-02-05T07:28:56.332 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.332 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.332 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.332 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.350884+0000", 2024-02-05T07:28:56.332 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0569999999999996e-05 2024-02-05T07:28:56.332 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.332 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.332 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.332 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.350893+0000", 2024-02-05T07:28:56.332 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9730000000000003e-06 2024-02-05T07:28:56.332 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.333 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.333 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.333 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.349368+0000", 2024-02-05T07:28:56.333 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99847542 2024-02-05T07:28:56.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.333 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.333 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.333 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.349489+0000", 2024-02-05T07:28:56.333 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121131 2024-02-05T07:28:56.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.333 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.333 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-02-05T07:28:56.333 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.349872+0000", 2024-02-05T07:28:56.333 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000382977 2024-02-05T07:28:56.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.334 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.334 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.334 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.353580+0000", 2024-02-05T07:28:56.334 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037082270000000001 2024-02-05T07:28:56.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.334 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.334 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.334 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.353621+0000", 2024-02-05T07:28:56.334 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0899999999999998e-05 2024-02-05T07:28:56.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.334 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.334 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.334 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.366458+0000", 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012836587999999999 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.366513+0000", 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4873999999999997e-05 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.366609+0000", 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6512e-05 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:165760 8.3 8:c5c40ee8:::benchmark_data_smithi007_105755_object29450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:34.353909+0000", 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 81.822809708999998, 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0132686049999999, 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 165760 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.336 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.353909+0000", 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.353909+0000", 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.353910+0000", 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0249999999999999e-06 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.337 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.338 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.353915+0000", 2024-02-05T07:28:56.338 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6659999999999999e-06 2024-02-05T07:28:56.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.338 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.338 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.338 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.353925+0000", 2024-02-05T07:28:56.338 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0361e-05 2024-02-05T07:28:56.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.338 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.338 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.338 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.353933+0000", 2024-02-05T07:28:56.338 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4479999999999997e-06 2024-02-05T07:28:56.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.338 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.356073+0000", 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021403329999999999 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.356082+0000", 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6980000000000004e-06 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.351699+0000", 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99561760300000002 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.340 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.340 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.351818+0000", 2024-02-05T07:28:56.340 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000119041 2024-02-05T07:28:56.340 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.340 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.340 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-02-05T07:28:56.340 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.352104+0000", 2024-02-05T07:28:56.340 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028534200000000001 2024-02-05T07:28:56.340 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.340 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.340 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.340 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.358808+0000", 2024-02-05T07:28:56.340 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0067043090000000003 2024-02-05T07:28:56.341 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.341 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.341 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.341 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.358855+0000", 2024-02-05T07:28:56.341 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7153000000000001e-05 2024-02-05T07:28:56.341 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.341 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.341 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.341 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.367020+0000", 2024-02-05T07:28:56.341 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0081654020000000004 2024-02-05T07:28:56.341 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.341 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.341 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.367068+0000", 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7695000000000003e-05 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.367178+0000", 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000109529 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:165820 8.3 8:c42dad74:::benchmark_data_smithi007_105755_object29510:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:34.422328+0000", 2024-02-05T07:28:56.342 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 81.754390334999997, 2024-02-05T07:28:56.343 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.94501602600000001, 2024-02-05T07:28:56.343 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:28:56.343 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:28:56.343 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:28:56.343 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:28:56.343 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:28:56.343 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 165820 2024-02-05T07:28:56.343 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.343 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:28:56.343 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.343 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:28:56.343 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.422328+0000", 2024-02-05T07:28:56.343 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.343 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.344 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.344 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:28:56.344 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.422328+0000", 2024-02-05T07:28:56.344 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:28:56.344 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.344 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.344 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:28:56.344 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.422329+0000", 2024-02-05T07:28:56.344 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.032e-06 2024-02-05T07:28:56.344 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.344 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.344 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:28:56.344 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.422334+0000", 2024-02-05T07:28:56.344 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8489999999999997e-06 2024-02-05T07:28:56.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.345 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.345 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:28:56.345 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.422346+0000", 2024-02-05T07:28:56.345 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1963e-05 2024-02-05T07:28:56.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.345 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.345 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:28:56.345 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.422355+0000", 2024-02-05T07:28:56.345 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8459999999999997e-06 2024-02-05T07:28:56.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.345 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.345 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.345 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.422972+0000", 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00061725400000000002 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:34.422984+0000", 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1276e-05 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.353650+0000", 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.930665992 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:28:56.346 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.353766+0000", 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011686199999999999 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.354043+0000", 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000276197 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.358894+0000", 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0048517810000000003 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.358939+0000", 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4814000000000002e-05 2024-02-05T07:28:56.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.348 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.348 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:28:56.348 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.367186+0000", 2024-02-05T07:28:56.348 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0082466620000000001 2024-02-05T07:28:56.348 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.348 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.348 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:28:56.348 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.367235+0000", 2024-02-05T07:28:56.348 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8714999999999997e-05 2024-02-05T07:28:56.348 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:28:56.348 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:28:56.348 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:28:56.348 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:35.367344+0000", 2024-02-05T07:28:56.348 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000109783 2024-02-05T07:28:56.349 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.349 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.349 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.349 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:28:56.349 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:28:56.349 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:28:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:56.350 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:56.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:56.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:56.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:56 smithi007 ceph-mon[29098]: pgmap v525: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2527/129492 objects degraded (1.951%); 14874/129492 objects misplaced (11.486%) 2024-02-05T07:28:56.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:56 smithi007 ceph-mon[29098]: 5.b scrub starts 2024-02-05T07:28:56.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:56 smithi007 ceph-mon[29098]: 5.b scrub ok 2024-02-05T07:28:56.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:56 smithi007 ceph-mon[29098]: 6.15 scrub starts 2024-02-05T07:28:56.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:56 smithi007 ceph-mon[29098]: 6.15 scrub ok 2024-02-05T07:28:56.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:56 smithi007 ceph-mon[29098]: 2.2 deep-scrub starts 2024-02-05T07:28:56.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:56 smithi007 ceph-mon[29098]: 2.2 deep-scrub ok 2024-02-05T07:28:56.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:56 smithi007 ceph-mon[37158]: pgmap v525: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2527/129492 objects degraded (1.951%); 14874/129492 objects misplaced (11.486%) 2024-02-05T07:28:56.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:56 smithi007 ceph-mon[37158]: 5.b scrub starts 2024-02-05T07:28:56.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:56 smithi007 ceph-mon[37158]: 5.b scrub ok 2024-02-05T07:28:56.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:56 smithi007 ceph-mon[37158]: 6.15 scrub starts 2024-02-05T07:28:56.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:56 smithi007 ceph-mon[37158]: 6.15 scrub ok 2024-02-05T07:28:56.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:56 smithi007 ceph-mon[37158]: 2.2 deep-scrub starts 2024-02-05T07:28:56.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:56 smithi007 ceph-mon[37158]: 2.2 deep-scrub ok 2024-02-05T07:28:56.481 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:56.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:56.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:56.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:56.623 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:56.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:56.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:56.725 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:56.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:56.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:56 smithi090 ceph-mon[33213]: pgmap v525: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2527/129492 objects degraded (1.951%); 14874/129492 objects misplaced (11.486%) 2024-02-05T07:28:56.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:56 smithi090 ceph-mon[33213]: 5.b scrub starts 2024-02-05T07:28:56.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:56 smithi090 ceph-mon[33213]: 5.b scrub ok 2024-02-05T07:28:56.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:56 smithi090 ceph-mon[33213]: 6.15 scrub starts 2024-02-05T07:28:56.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:56 smithi090 ceph-mon[33213]: 6.15 scrub ok 2024-02-05T07:28:56.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:56 smithi090 ceph-mon[33213]: 2.2 deep-scrub starts 2024-02-05T07:28:56.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:56 smithi090 ceph-mon[33213]: 2.2 deep-scrub ok 2024-02-05T07:28:56.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:56.901 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:56.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:56.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:57.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:57.061 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:57.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:57.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:57.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:57.197 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:57.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:57.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:57.329 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:28:57.333 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:57.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:57.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:57.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:57.473 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:57.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:57.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:28:57.615 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:57.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:57.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:28:57.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:57 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4125719983' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:28:57.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:57 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:28:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:57 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4125719983' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:28:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:57 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:28:57.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:57.763 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:57.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:57.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:57 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4125719983' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:28:57.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:57 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:28:57.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T07:28:57.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:57.900 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:57.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:58.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:58.044 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:58.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:58.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:58.178 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:58.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:58.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 1 1 2024-02-05T07:28:58.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:58.304 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:58.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:58.368 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:28:58.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:28:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:58.461 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:58.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:28:58.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:28:58.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:28:58.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[37158]: 6.2 deep-scrub starts 2024-02-05T07:28:58.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[37158]: 6.2 deep-scrub ok 2024-02-05T07:28:58.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[37158]: pgmap v526: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2168/111549 objects degraded (1.944%); 12715/111549 objects misplaced (11.399%) 2024-02-05T07:28:58.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:28:58.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:28:58.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[37158]: osdmap e325: 8 total, 8 up, 7 in 2024-02-05T07:28:58.498 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:28:58.498 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:28:58.498 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[29098]: 6.2 deep-scrub starts 2024-02-05T07:28:58.498 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[29098]: 6.2 deep-scrub ok 2024-02-05T07:28:58.498 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[29098]: pgmap v526: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2168/111549 objects degraded (1.944%); 12715/111549 objects misplaced (11.399%) 2024-02-05T07:28:58.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:28:58.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:28:58.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:58 smithi007 ceph-mon[29098]: osdmap e325: 8 total, 8 up, 7 in 2024-02-05T07:28:58.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:58.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:58.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:28:58.764 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:58.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:58.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:28:58.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:58 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:28:58.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:58 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:28:58.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:58 smithi090 ceph-mon[33213]: 6.2 deep-scrub starts 2024-02-05T07:28:58.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:58 smithi090 ceph-mon[33213]: 6.2 deep-scrub ok 2024-02-05T07:28:58.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:58 smithi090 ceph-mon[33213]: pgmap v526: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2168/111549 objects degraded (1.944%); 12715/111549 objects misplaced (11.399%) 2024-02-05T07:28:58.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:58 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:28:58.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:58 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:28:58.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:58 smithi090 ceph-mon[33213]: osdmap e325: 8 total, 8 up, 7 in 2024-02-05T07:28:58.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:58.902 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:58.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:59.003 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:28:59.036 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:59.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:28:59.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:59.173 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:59.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:59.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:59.311 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:59.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:59.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:59.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:59.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:59.447 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:59.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:28:59.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:28:59.591 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:28:59.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:59.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:28:59.614 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:28:59.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:59 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:28:59.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:59 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:28:59.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:59 smithi007 ceph-mon[29098]: osdmap e326: 8 total, 8 up, 7 in 2024-02-05T07:28:59.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:28:59 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 2168/111549 objects degraded (1.944%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:59.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:59 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:28:59.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:59 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:28:59.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:59 smithi007 ceph-mon[37158]: osdmap e326: 8 total, 8 up, 7 in 2024-02-05T07:28:59.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:28:59 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 2168/111549 objects degraded (1.944%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:59.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:28:59.725 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:59.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:28:59.757 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:28:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:59 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:28:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:59 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:28:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:59 smithi090 ceph-mon[33213]: osdmap e326: 8 total, 8 up, 7 in 2024-02-05T07:28:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:28:59 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 2168/111549 objects degraded (1.944%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:28:59.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:28:59.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:59.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:59.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:28:59.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:28:59.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:28:59.989 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:28:59.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:00.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:00.118 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:00.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:00.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:00.254 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:00.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:00.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:00.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:00.386 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:00.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:00.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:00.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:00.524 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:00.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:00.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:00.650 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:00.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:00.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:00.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:00 smithi007 ceph-mon[29098]: pgmap v529: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 8.9 KiB/s rd, 0 B/s wr, 1.26k op/s; 2168/111006 objects degraded (1.953%); 12715/111006 objects misplaced (11.454%) 2024-02-05T07:29:00.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:00 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:29:00.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:00 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:29:00.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:00 smithi007 ceph-mon[37158]: pgmap v529: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 8.9 KiB/s rd, 0 B/s wr, 1.26k op/s; 2168/111006 objects degraded (1.953%); 12715/111006 objects misplaced (11.454%) 2024-02-05T07:29:00.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:00 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:29:00.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:00 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:29:00.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:00.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:29:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:29:00.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:00 smithi090 ceph-mon[33213]: pgmap v529: 178 pgs: 7 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 8.9 KiB/s rd, 0 B/s wr, 1.26k op/s; 2168/111006 objects degraded (1.953%); 12715/111006 objects misplaced (11.454%) 2024-02-05T07:29:00.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:00 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:29:00.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:00 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:29:00.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:00.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:00.897 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:00.931 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:00.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:00.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:00.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:29:01.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:01.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:01.083 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:01.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:01.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:01.208 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:01.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:01.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:01.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:01.343 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:01.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:01.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:01.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:01.490 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:01.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:01.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:01.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:01.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:01.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:01.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:01.757 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:01.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:01.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:01.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:01.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:01.893 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:01.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:01.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:01.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:02.030 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:02.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:02.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:02.167 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:02.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:02.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:02.269 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:02.305 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:02.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:02.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:02.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:02.446 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:02.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:02.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:02.548 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:02.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:02 smithi007 ceph-mon[29098]: pgmap v530: 178 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 1.9 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 62 KiB/s rd, 0 B/s wr, 1.77k op/s; 1440/95118 objects degraded (1.514%); 9483/95118 objects misplaced (9.970%); 18 MiB/s, 281 objects/s recovering 2024-02-05T07:29:02.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:02 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:29:02.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:02 smithi007 ceph-mon[37158]: pgmap v530: 178 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 1.9 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 62 KiB/s rd, 0 B/s wr, 1.77k op/s; 1440/95118 objects degraded (1.514%); 9483/95118 objects misplaced (9.970%); 18 MiB/s, 281 objects/s recovering 2024-02-05T07:29:02.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:02 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:29:02.578 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:02.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:02.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:02.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:02.729 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:02.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:02 smithi090 ceph-mon[33213]: pgmap v530: 178 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 1.9 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 62 KiB/s rd, 0 B/s wr, 1.77k op/s; 1440/95118 objects degraded (1.514%); 9483/95118 objects misplaced (9.970%); 18 MiB/s, 281 objects/s recovering 2024-02-05T07:29:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:02 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:29:02.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:02.860 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:02.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:02.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:02.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:03.001 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:03.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:03.103 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:03.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:03.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:03.297 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:03.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:03.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:03.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:03.434 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:03.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:03.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:03.568 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:03.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:03.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:03.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:03.701 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:03.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:03.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:03.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:03.843 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:03.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:03.873 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:29:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:29:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:29:03.882 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:03.905 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:29:03.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:03.951 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:29:03.975 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:03.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:03.995 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:29:04.045 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:29:04.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:04.090 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:29:04.132 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:04.133 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:29:04.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:04.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:04.176 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:29:04.213 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:29:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:29:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:29:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:29:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:29:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:29:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:29:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:29:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:29:04.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:04.286 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:04.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:04.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:04.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:04.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:04.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:04.553 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:04.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:04.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:04.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:04.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:04 smithi007 ceph-mon[29098]: pgmap v531: 178 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 1.9 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 53 KiB/s rd, 0 B/s wr, 1.52k op/s; 1440/95118 objects degraded (1.514%); 9483/95118 objects misplaced (9.970%); 15 MiB/s, 241 objects/s recovering 2024-02-05T07:29:04.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:04 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2863549898' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:29:04.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:04 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 1440/95118 objects degraded (1.514%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:29:04.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:04 smithi007 ceph-mon[29098]: osdmap e327: 8 total, 8 up, 7 in 2024-02-05T07:29:04.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:04 smithi007 ceph-mon[37158]: pgmap v531: 178 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 1.9 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 53 KiB/s rd, 0 B/s wr, 1.52k op/s; 1440/95118 objects degraded (1.514%); 9483/95118 objects misplaced (9.970%); 15 MiB/s, 241 objects/s recovering 2024-02-05T07:29:04.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:04 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2863549898' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:29:04.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:04 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 1440/95118 objects degraded (1.514%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:29:04.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:04 smithi007 ceph-mon[37158]: osdmap e327: 8 total, 8 up, 7 in 2024-02-05T07:29:04.679 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:04.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:04.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:04.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:04 smithi090 ceph-mon[33213]: pgmap v531: 178 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 1.9 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 53 KiB/s rd, 0 B/s wr, 1.52k op/s; 1440/95118 objects degraded (1.514%); 9483/95118 objects misplaced (9.970%); 15 MiB/s, 241 objects/s recovering 2024-02-05T07:29:04.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:04 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2863549898' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:29:04.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:04 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 1440/95118 objects degraded (1.514%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:29:04.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:04 smithi090 ceph-mon[33213]: osdmap e327: 8 total, 8 up, 7 in 2024-02-05T07:29:04.781 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:04.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:04.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:04.931 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:04.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:05.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:05.057 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:05.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:05.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:05.181 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:05.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:05.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:05.283 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:05.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:05.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:05.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:05.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:05.450 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:05.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:05.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:05.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:05.579 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:05.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:05.640 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:05 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:29:05.640 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:05 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:29:05.640 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:05 smithi007 ceph-mon[29098]: osdmap e328: 8 total, 8 up, 7 in 2024-02-05T07:29:05.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:05 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:29:05.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:05 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:29:05.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:05 smithi007 ceph-mon[37158]: osdmap e328: 8 total, 8 up, 7 in 2024-02-05T07:29:05.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:05.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:05.705 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:05.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:05.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:05 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:29:05.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:05 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:29:05.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:05 smithi090 ceph-mon[33213]: osdmap e328: 8 total, 8 up, 7 in 2024-02-05T07:29:05.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:05.829 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:05.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:05.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:05.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:05.954 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:05.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:05.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:06.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:06.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:06.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:06.209 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:06.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:06.311 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:06.337 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:06.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:06.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:06.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:06.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:06.481 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:06.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:06.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:06.613 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:06.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:06.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:06.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:06 smithi007 ceph-mon[29098]: pgmap v533: 178 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 1.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 811 op/s; 1440/94605 objects degraded (1.522%); 9483/94605 objects misplaced (10.024%); 15 MiB/s, 246 objects/s recovering 2024-02-05T07:29:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:06 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:29:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:06 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:29:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:06 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:29:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:06 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:29:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:06 smithi007 ceph-mon[29098]: osdmap e329: 8 total, 8 up, 7 in 2024-02-05T07:29:06.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:06 smithi007 ceph-mon[37158]: pgmap v533: 178 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 1.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 811 op/s; 1440/94605 objects degraded (1.522%); 9483/94605 objects misplaced (10.024%); 15 MiB/s, 246 objects/s recovering 2024-02-05T07:29:06.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:06 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:29:06.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:06 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:29:06.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:06 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:29:06.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:06 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:29:06.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:06 smithi007 ceph-mon[37158]: osdmap e329: 8 total, 8 up, 7 in 2024-02-05T07:29:06.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:06.751 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:06.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:06.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:06 smithi090 ceph-mon[33213]: pgmap v533: 178 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 168 active+clean; 1.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 811 op/s; 1440/94605 objects degraded (1.522%); 9483/94605 objects misplaced (10.024%); 15 MiB/s, 246 objects/s recovering 2024-02-05T07:29:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:06 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:29:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:06 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:29:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:06 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:29:06.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:06 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:29:06.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:06 smithi090 ceph-mon[33213]: osdmap e329: 8 total, 8 up, 7 in 2024-02-05T07:29:06.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:06.885 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:06.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:06.987 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:07.023 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:07.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:07.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:07.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:07.168 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:07.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:07.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:07.269 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:07.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:07.310 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:07.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:07.411 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:07.453 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:07.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:07.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:07.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:29:07.554 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:07.586 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:07.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:07.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:07.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:07.721 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:07.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:07.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:07.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:07.866 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:07.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:07.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:07.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:08.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:08.001 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:08.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:08.102 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:08.130 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:08.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:08.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:08.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:08.263 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:08.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:08.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:08.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:08.400 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:08.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:08.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:08.546 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:08.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:08.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:08.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:08 smithi007 ceph-mon[29098]: pgmap v536: 178 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 169 active+clean; 1.7 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 740 op/s; 1301/81879 objects degraded (1.589%); 7297/81879 objects misplaced (8.912%); 10 MiB/s, 160 objects/s recovering 2024-02-05T07:29:08.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:08 smithi007 ceph-mon[37158]: pgmap v536: 178 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 169 active+clean; 1.7 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 740 op/s; 1301/81879 objects degraded (1.589%); 7297/81879 objects misplaced (8.912%); 10 MiB/s, 160 objects/s recovering 2024-02-05T07:29:08.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:08.677 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:08.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:08 smithi090 ceph-mon[33213]: pgmap v536: 178 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 169 active+clean; 1.7 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 740 op/s; 1301/81879 objects degraded (1.589%); 7297/81879 objects misplaced (8.912%); 10 MiB/s, 160 objects/s recovering 2024-02-05T07:29:08.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:08.819 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:08.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:08.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:08.962 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:08.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:09.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:09.092 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:09.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:09.195 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:09.234 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:09.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:09.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:09.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:09.369 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:09.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:09.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:09.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:09.502 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:09.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:09.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:09.640 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:09.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:09.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:09.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:09.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:09 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 1301/81879 objects degraded (1.589%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:29:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:09 smithi090 ceph-mon[33213]: osdmap e330: 8 total, 8 up, 7 in 2024-02-05T07:29:09.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:09.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:09.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:09.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:09.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:09.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:09 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 1301/81879 objects degraded (1.589%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:29:09.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:09 smithi007 ceph-mon[29098]: osdmap e330: 8 total, 8 up, 7 in 2024-02-05T07:29:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:09 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 1301/81879 objects degraded (1.589%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T07:29:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:09 smithi007 ceph-mon[37158]: osdmap e330: 8 total, 8 up, 7 in 2024-02-05T07:29:10.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:10.044 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:10.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:10.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:10.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:10.169 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:10.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:10.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:10.306 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:10.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:10.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:10.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:10.454 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:10.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:10.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:10.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:10.589 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:10.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:10.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:10.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:10.726 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:10.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:10.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:10.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:29:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:29:10.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:10 smithi090 ceph-mon[33213]: pgmap v537: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 1.6 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 832 op/s; 760/80214 objects degraded (0.947%); 6732/80214 objects misplaced (8.393%); 21 MiB/s, 337 objects/s recovering 2024-02-05T07:29:10.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:10.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:10.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:10.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:10 smithi007 ceph-mon[29098]: pgmap v537: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 1.6 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 832 op/s; 760/80214 objects degraded (0.947%); 6732/80214 objects misplaced (8.393%); 21 MiB/s, 337 objects/s recovering 2024-02-05T07:29:10.881 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:10 smithi007 ceph-mon[37158]: pgmap v537: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 1.6 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 832 op/s; 760/80214 objects degraded (0.947%); 6732/80214 objects misplaced (8.393%); 21 MiB/s, 337 objects/s recovering 2024-02-05T07:29:10.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:11.010 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:11.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:11.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:11.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:11.142 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:11.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:11.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:11.244 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:11.272 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:11.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:11.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:11.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:11.424 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:11.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:11.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:11.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:11.557 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:11.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:11.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:11.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:11.682 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:11.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:11.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:11.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:11 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:11.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:11.825 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:11.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:11.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:11.827 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:11 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:11.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:11 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:11.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:11.982 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:11.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:12.050 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:29:12.050 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:29:12.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:12.112 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:12.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:12.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:12.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:12.245 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:12.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:12.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:12.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:12.379 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:12.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:12.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:12.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:12.514 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:12.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:12.616 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:12.661 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:12.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:12.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:12.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:12 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:29:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:12 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:29:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:12 smithi090 ceph-mon[33213]: pgmap v539: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 1.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.59k op/s; 599/65940 objects degraded (0.908%); 5363/65940 objects misplaced (8.133%); 21 MiB/s, 337 objects/s recovering 2024-02-05T07:29:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:12 smithi090 ceph-mon[33213]: osdmap e331: 8 total, 8 up, 7 in 2024-02-05T07:29:12.799 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:12.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:12.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:12.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:12 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:29:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:12 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:29:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:12 smithi007 ceph-mon[29098]: pgmap v539: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 1.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.59k op/s; 599/65940 objects degraded (0.908%); 5363/65940 objects misplaced (8.133%); 21 MiB/s, 337 objects/s recovering 2024-02-05T07:29:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:12 smithi007 ceph-mon[29098]: osdmap e331: 8 total, 8 up, 7 in 2024-02-05T07:29:12.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:12 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:29:12.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:12 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:29:12.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:12 smithi007 ceph-mon[37158]: pgmap v539: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 1.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.59k op/s; 599/65940 objects degraded (0.908%); 5363/65940 objects misplaced (8.133%); 21 MiB/s, 337 objects/s recovering 2024-02-05T07:29:12.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:12 smithi007 ceph-mon[37158]: osdmap e331: 8 total, 8 up, 7 in 2024-02-05T07:29:12.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:12.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:12.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:13.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:13.062 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:13.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:13.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:13.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:13.203 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:13.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:13.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:13.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:13.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:13.343 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:13.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:13.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:13.473 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:13.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:13.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:13.482 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.1 to 1 (10000) 2024-02-05T07:29:13.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:13.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:13.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:13.752 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:13.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:13.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:13.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:13 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3851154759' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:29:13.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:13.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:13.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:13.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:13 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3851154759' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:29:13.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:29:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:29:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:29:13.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:13 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3851154759' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:29:13.991 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:14.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T07:29:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:14.024 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:14.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:14.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:14.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:14.157 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:14.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:14.245 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:14.258 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:14.327 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:14.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:14.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:14.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:14.459 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:14.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:14.496 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:29:14.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:14.598 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:14.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:14.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:14.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:14 smithi007 ceph-mon[29098]: pgmap v541: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 1.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 745 B/s rd, 0 B/s wr, 773 op/s; 599/65940 objects degraded (0.908%); 5363/65940 objects misplaced (8.133%); 9.7 MiB/s, 154 objects/s recovering 2024-02-05T07:29:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:14 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3851154759' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T07:29:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:14 smithi007 ceph-mon[29098]: osdmap e332: 8 total, 8 up, 7 in 2024-02-05T07:29:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:14 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3467525021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:29:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:14 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:29:14.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:14 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 599/65940 objects degraded (0.908%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T07:29:14.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:14 smithi007 ceph-mon[37158]: pgmap v541: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 1.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 745 B/s rd, 0 B/s wr, 773 op/s; 599/65940 objects degraded (0.908%); 5363/65940 objects misplaced (8.133%); 9.7 MiB/s, 154 objects/s recovering 2024-02-05T07:29:14.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:14 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3851154759' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T07:29:14.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:14 smithi007 ceph-mon[37158]: osdmap e332: 8 total, 8 up, 7 in 2024-02-05T07:29:14.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:14 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3467525021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:29:14.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:14 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:29:14.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:14 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 599/65940 objects degraded (0.908%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T07:29:14.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:14.733 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:14.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:14.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:14 smithi090 ceph-mon[33213]: pgmap v541: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 1.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 745 B/s rd, 0 B/s wr, 773 op/s; 599/65940 objects degraded (0.908%); 5363/65940 objects misplaced (8.133%); 9.7 MiB/s, 154 objects/s recovering 2024-02-05T07:29:14.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:14 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3851154759' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T07:29:14.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:14 smithi090 ceph-mon[33213]: osdmap e332: 8 total, 8 up, 7 in 2024-02-05T07:29:14.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:14 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3467525021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:29:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:14 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:29:14.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:14 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 599/65940 objects degraded (0.908%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T07:29:14.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:14.861 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:14.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:14.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 2 1 2024-02-05T07:29:14.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:14.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:15.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:15.138 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:15.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:15.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:15.240 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:15.312 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:15.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:15.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:15.447 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:15.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:15.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:15.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:15.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:15.591 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:15.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:15.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:15.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:15.735 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:15.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:15.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:15 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:29:15.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:15 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:29:15.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:15 smithi090 ceph-mon[33213]: osdmap e333: 8 total, 8 up, 7 in 2024-02-05T07:29:15.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:29:15.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:15 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:29:15.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:15 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:29:15.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:15 smithi007 ceph-mon[29098]: osdmap e333: 8 total, 8 up, 7 in 2024-02-05T07:29:15.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:29:15.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:15 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:29:15.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:15 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:29:15.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:15 smithi007 ceph-mon[37158]: osdmap e333: 8 total, 8 up, 7 in 2024-02-05T07:29:15.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:29:15.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:15.891 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:15.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:15.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:15.993 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:16.030 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:16.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:16.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:16.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:16.174 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:16.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:16.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:16.310 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:16.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:16.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:16.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:16.442 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:16.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:16.544 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:16.576 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:16.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:16.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:16.716 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:16.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:16.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:16.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:16 smithi090 ceph-mon[33213]: pgmap v544: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 1.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 872 B/s rd, 0 B/s wr, 907 op/s; 599/64245 objects degraded (0.932%); 5363/64245 objects misplaced (8.348%) 2024-02-05T07:29:16.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:16.857 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:16.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:16 smithi007 ceph-mon[29098]: pgmap v544: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 1.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 872 B/s rd, 0 B/s wr, 907 op/s; 599/64245 objects degraded (0.932%); 5363/64245 objects misplaced (8.348%) 2024-02-05T07:29:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:16 smithi007 ceph-mon[37158]: pgmap v544: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 1.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 872 B/s rd, 0 B/s wr, 907 op/s; 599/64245 objects degraded (0.932%); 5363/64245 objects misplaced (8.348%) 2024-02-05T07:29:16.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:17.001 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:17.016 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:17.033 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:17.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:17.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:17.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:17.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:17.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:17.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:17.320 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:17.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:17.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:17.457 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:17.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:17.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:17.601 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:17.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:17.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:17.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:17.727 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:17.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:17.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:17.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:17.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:29:17.867 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:17.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:17.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:17.993 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:17.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:18.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:18.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:18.225 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:18.252 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:18.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:18.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:18.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:18.394 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:18.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:18.495 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:18.536 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:18.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:18.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:18.667 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:18.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:18.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:18.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:18.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:18 smithi090 ceph-mon[33213]: pgmap v545: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 985 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 436/47835 objects degraded (0.911%); 3865/47835 objects misplaced (8.080%) 2024-02-05T07:29:18.812 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:18.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:18.813 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:18 smithi007 ceph-mon[29098]: pgmap v545: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 985 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 436/47835 objects degraded (0.911%); 3865/47835 objects misplaced (8.080%) 2024-02-05T07:29:18.814 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:18 smithi007 ceph-mon[37158]: pgmap v545: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 985 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 436/47835 objects degraded (0.911%); 3865/47835 objects misplaced (8.080%) 2024-02-05T07:29:18.913 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:18.960 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:18.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:18.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:19.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:19.106 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:19.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:19.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:19.207 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:19.246 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:19.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:19.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:19.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:19.378 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:19.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:19.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:19.512 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:19.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:19.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:19.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:19.643 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:19.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:19.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:19.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:19 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 436/47835 objects degraded (0.911%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T07:29:19.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:19 smithi090 ceph-mon[33213]: osdmap e334: 8 total, 8 up, 7 in 2024-02-05T07:29:19.784 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:19.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:19.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:19.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:19 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 436/47835 objects degraded (0.911%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T07:29:19.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:19 smithi007 ceph-mon[29098]: osdmap e334: 8 total, 8 up, 7 in 2024-02-05T07:29:19.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:19 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 436/47835 objects degraded (0.911%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T07:29:19.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:19 smithi007 ceph-mon[37158]: osdmap e334: 8 total, 8 up, 7 in 2024-02-05T07:29:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:19.935 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:19.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:20.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:20.063 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:20.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:20.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:20.213 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:20.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:20.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:20.343 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:20.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:20.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:20.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:20.480 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:20.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:20.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:20.632 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:20.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:20.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:20.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:20.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:20.770 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:20.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:20.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:29:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:29:20.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:20 smithi090 ceph-mon[33213]: pgmap v546: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 951 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 852 op/s; 436/46227 objects degraded (0.943%); 3865/46227 objects misplaced (8.361%) 2024-02-05T07:29:20.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:20 smithi007 ceph-mon[29098]: pgmap v546: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 951 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 852 op/s; 436/46227 objects degraded (0.943%); 3865/46227 objects misplaced (8.361%) 2024-02-05T07:29:20.849 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:20 smithi007 ceph-mon[37158]: pgmap v546: 178 pgs: 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 171 active+clean; 951 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 852 op/s; 436/46227 objects degraded (0.943%); 3865/46227 objects misplaced (8.361%) 2024-02-05T07:29:20.871 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:20.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:20.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:20.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:21.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:21.061 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:21.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:21.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:21.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:21.193 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:21.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:21.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:21.335 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:21.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:21.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:21.485 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:21.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:21.536 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:29:21.537 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:29:21.557 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:29:21.557 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:29:21.581 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:29:21.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:21.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:21.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:21.632 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:29:21.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:21 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:21.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:21 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/826011906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:29:21.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:21 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:21.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:21 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/826011906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:29:21.680 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:29:21.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:21.719 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:29:21.758 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:21.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:21.762 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:29:21.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:21 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:21.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:21 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/826011906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:29:21.786 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:29:21.788 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:29:21.788 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:29:21.788 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:29:21.788 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:29:21.788 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:29:21.788 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:29:21.788 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:29:21.788 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:29:21.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:21.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:21.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:21.996 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:22.026 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:22.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:22.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:22.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:22.160 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:22.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:22.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:22.262 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:22.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:22.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:22.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:22.437 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:22.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:22.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:22.570 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:22.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:22.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:22.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:22.714 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:22.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:22.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:22.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:22 smithi007 ceph-mon[29098]: pgmap v548: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 599 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.59k op/s; 259/29310 objects degraded (0.884%); 2299/29310 objects misplaced (7.844%) 2024-02-05T07:29:22.856 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:22 smithi007 ceph-mon[37158]: pgmap v548: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 599 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.59k op/s; 259/29310 objects degraded (0.884%); 2299/29310 objects misplaced (7.844%) 2024-02-05T07:29:22.859 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:22.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:22.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:22.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T07:29:22.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:23.006 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:23.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:22 smithi090 ceph-mon[33213]: pgmap v548: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 599 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.59k op/s; 259/29310 objects degraded (0.884%); 2299/29310 objects misplaced (7.844%) 2024-02-05T07:29:23.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:23.136 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:23.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:23.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:23.269 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:23.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:23.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:23.401 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:23.442 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:23.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:23.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:23.544 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:23.576 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:23.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:23.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:23.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:23.710 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:23.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:23.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:23.846 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:23.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:23.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:29:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:29:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:29:23.947 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:23.988 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:23.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:23.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:24.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:24.128 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:24.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:24.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:24.262 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:24.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:24.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:24.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:24.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:24.416 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:24.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:24.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:24.557 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:24.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:24.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:24.688 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:24.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:24.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:24.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:24.827 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:24.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:24.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:24 smithi007 ceph-mon[29098]: pgmap v549: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 599 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 259/29310 objects degraded (0.884%); 2299/29310 objects misplaced (7.844%) 2024-02-05T07:29:24.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:24 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:29:24.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:24 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:29:24.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:24 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 259/29310 objects degraded (0.884%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:29:24.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:24 smithi007 ceph-mon[37158]: pgmap v549: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 599 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 259/29310 objects degraded (0.884%); 2299/29310 objects misplaced (7.844%) 2024-02-05T07:29:24.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:24 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:29:24.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:24 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:29:24.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:24 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 259/29310 objects degraded (0.884%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:29:24.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:24.959 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:24.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:24.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:24 smithi090 ceph-mon[33213]: pgmap v549: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 599 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 259/29310 objects degraded (0.884%); 2299/29310 objects misplaced (7.844%) 2024-02-05T07:29:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:24 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:29:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:24 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:29:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:24 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 259/29310 objects degraded (0.884%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:29:25.061 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:25.100 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:25.100 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:25.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:25.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:25.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:25.257 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:25.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:25.358 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:25.398 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:25.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:25.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:25.554 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:25.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:25.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:25.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:25.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:25 smithi007 ceph-mon[29098]: osdmap e335: 8 total, 8 up, 7 in 2024-02-05T07:29:25.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:25 smithi007 ceph-mon[37158]: osdmap e335: 8 total, 8 up, 7 in 2024-02-05T07:29:25.699 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:25.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:25.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:25.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:25.831 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:25.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:25.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:25.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:25.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:26.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:25 smithi090 ceph-mon[33213]: osdmap e335: 8 total, 8 up, 7 in 2024-02-05T07:29:26.065 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:26.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:26.098 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:26.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:26.107 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:26.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:26.232 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:26.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:26.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:26.373 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:26.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:26.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:26.475 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:26.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:26.521 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:26.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:26.583 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.2 to 1 (10000) 2024-02-05T07:29:26.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:26.654 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:26.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:26.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:26.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:26.804 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:26.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:26.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:26.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:29:26.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:26.922 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:26 smithi007 ceph-mon[29098]: pgmap v551: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 559 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 851 op/s; 259/27402 objects degraded (0.945%); 2299/27402 objects misplaced (8.390%) 2024-02-05T07:29:26.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:26 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2648468548' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:29:26.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:26 smithi007 ceph-mon[37158]: pgmap v551: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 559 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 851 op/s; 259/27402 objects degraded (0.945%); 2299/27402 objects misplaced (8.390%) 2024-02-05T07:29:26.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:26 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2648468548' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:29:26.926 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:26 smithi090 ceph-mon[33213]: pgmap v551: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 559 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 851 op/s; 259/27402 objects degraded (0.945%); 2299/27402 objects misplaced (8.390%) 2024-02-05T07:29:26.926 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:26 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2648468548' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:29:26.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:26.938 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:26.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:27.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:27.111 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:27.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:27.212 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:27.268 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:27.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:27.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:27.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:27.419 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:27.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:27.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:27.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:27.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:27.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:27.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:27.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:27.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:27.818 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:27.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:27.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:27.920 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:27.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:27 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2648468548' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T07:29:27.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:27 smithi007 ceph-mon[29098]: osdmap e336: 8 total, 8 up, 7 in 2024-02-05T07:29:27.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:27 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2648468548' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T07:29:27.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:27 smithi007 ceph-mon[37158]: osdmap e336: 8 total, 8 up, 7 in 2024-02-05T07:29:27.944 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:27.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:27 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2648468548' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T07:29:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:27 smithi090 ceph-mon[33213]: osdmap e336: 8 total, 8 up, 7 in 2024-02-05T07:29:28.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:28.069 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:28.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:28.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:28.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:28.194 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:28.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:28.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:28.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:28.318 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:28.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:28.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:28.445 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:28.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:28.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:28.572 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:28.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:28.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:28.719 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:28.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:28.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:28.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 4 1 2024-02-05T07:29:28.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:28 smithi090 ceph-mon[33213]: pgmap v553: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 201 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.53k op/s; 147/10215 objects degraded (1.439%); 709/10215 objects misplaced (6.941%) 2024-02-05T07:29:28.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:28 smithi090 ceph-mon[33213]: osdmap e337: 8 total, 8 up, 7 in 2024-02-05T07:29:28.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:28 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:29:28.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:28 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:29:28.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:28.849 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:28.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:28.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:28 smithi007 ceph-mon[29098]: pgmap v553: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 201 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.53k op/s; 147/10215 objects degraded (1.439%); 709/10215 objects misplaced (6.941%) 2024-02-05T07:29:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:28 smithi007 ceph-mon[29098]: osdmap e337: 8 total, 8 up, 7 in 2024-02-05T07:29:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:28 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:29:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:28 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:29:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:28 smithi007 ceph-mon[37158]: pgmap v553: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 201 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.53k op/s; 147/10215 objects degraded (1.439%); 709/10215 objects misplaced (6.941%) 2024-02-05T07:29:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:28 smithi007 ceph-mon[37158]: osdmap e337: 8 total, 8 up, 7 in 2024-02-05T07:29:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:28 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:29:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:28 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:29:28.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:28.975 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:28.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:28.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:29.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:29.100 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:29.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:29.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:29.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:29.225 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:29.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:29.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:29.351 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:29.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:29.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:29.403 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:29.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:29.476 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:29.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:29.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:29.602 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:29.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:29.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:29.742 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:29.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:29.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:29.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:29.870 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:29.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:29.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:29 smithi007 ceph-mon[29098]: osdmap e338: 8 total, 8 up, 7 in 2024-02-05T07:29:29.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:29 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 147/10215 objects degraded (1.439%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-05T07:29:29.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:29 smithi007 ceph-mon[37158]: osdmap e338: 8 total, 8 up, 7 in 2024-02-05T07:29:29.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:29 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 147/10215 objects degraded (1.439%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-05T07:29:29.972 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:29.977 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Removed 136307 objects 2024-02-05T07:29:29.977 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Clean up completed and total clean up time :148.936 2024-02-05T07:29:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:29.996 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:29.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:29 smithi090 ceph-mon[33213]: osdmap e338: 8 total, 8 up, 7 in 2024-02-05T07:29:30.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:29 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 147/10215 objects degraded (1.439%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-05T07:29:30.098 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:30.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:30.225 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:30.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:30.251 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:30.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:30.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:30.376 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:30.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:30.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:30.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:30.502 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:30.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:30.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:30.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:30.627 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:29:30.635 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:30.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:30.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:30.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:30.765 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:30.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:30.837 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:29:30.837 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-02-05T07:29:30.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-02-05T07:29:30.838 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:29:30.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.838 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75617 7.13 7:cf0c022b:::benchmark_data_smithi007_77086_object75616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T07:29:30.838 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:44.133001+0000", 2024-02-05T07:29:30.838 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 466.70158593000002, 2024-02-05T07:29:30.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3484787149999999, 2024-02-05T07:29:30.838 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.838 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.838 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.838 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.838 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.838 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75617 2024-02-05T07:29:30.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133001+0000", 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133001+0000", 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133003+0000", 2024-02-05T07:29:30.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7689999999999999e-06 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133028+0000", 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5148999999999999e-05 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133044+0000", 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6588000000000001e-05 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133054+0000", 2024-02-05T07:29:30.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0030000000000001e-05 2024-02-05T07:29:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135655+0000", 2024-02-05T07:29:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002600406 2024-02-05T07:29:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135664+0000", 2024-02-05T07:29:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0839999999999998e-06 2024-02-05T07:29:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.441044+0000", 2024-02-05T07:29:30.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3053797330000001 2024-02-05T07:29:30.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.842 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.842 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.441211+0000", 2024-02-05T07:29:30.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016769699999999999 2024-02-05T07:29:30.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.842 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:29:30.842 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.444465+0000", 2024-02-05T07:29:30.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032534159999999999 2024-02-05T07:29:30.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.449312+0000", 2024-02-05T07:29:30.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0048477559999999999 2024-02-05T07:29:30.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.456573+0000", 2024-02-05T07:29:30.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0072603210000000001 2024-02-05T07:29:30.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.844 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.844 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.479561+0000", 2024-02-05T07:29:30.844 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.022988044999999999 2024-02-05T07:29:30.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.844 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.844 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.479628+0000", 2024-02-05T07:29:30.844 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6933000000000004e-05 2024-02-05T07:29:30.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.844 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.844 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.481480+0000", 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001851788 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:75621 7.13 7:ce4ef747:::benchmark_data_smithi007_77086_object75620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:21:44.133075+0000", 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 466.70151220499997, 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.350304054, 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.845 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 75621 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133075+0000", 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133075+0000", 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133075+0000", 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4700000000000001e-07 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133096+0000", 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0849999999999999e-05 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133103+0000", 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.289e-06 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.847 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.848 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.133110+0000", 2024-02-05T07:29:30.848 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2559999999999999e-06 2024-02-05T07:29:30.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.848 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.848 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135691+0000", 2024-02-05T07:29:30.848 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025816580000000001 2024-02-05T07:29:30.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.848 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.848 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:44.135699+0000", 2024-02-05T07:29:30.848 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5569999999999998e-06 2024-02-05T07:29:30.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.849 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.849 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.447693+0000", 2024-02-05T07:29:30.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.311994023 2024-02-05T07:29:30.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.849 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.849 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.447870+0000", 2024-02-05T07:29:30.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017759699999999999 2024-02-05T07:29:30.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:29:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.448090+0000", 2024-02-05T07:29:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000219566 2024-02-05T07:29:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.453587+0000", 2024-02-05T07:29:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054966609999999999 2024-02-05T07:29:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.456653+0000", 2024-02-05T07:29:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030662329999999998 2024-02-05T07:29:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.481496+0000", 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.024843124000000001 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.481553+0000", 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6889999999999999e-05 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:21:45.483379+0000", 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001825803 2024-02-05T07:29:30.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.852 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.852 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.852 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:167698 7.14 7:29493818:::benchmark_data_smithi007_77086_object63103:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e164)", 2024-02-05T07:29:30.852 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:50.618760+0000", 2024-02-05T07:29:30.852 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 340.2158263, 2024-02-05T07:29:30.852 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.052304715, 2024-02-05T07:29:30.852 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.852 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:29:30.852 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.852 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.852 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.852 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 167698 2024-02-05T07:29:30.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.853 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.853 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.853 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.618760+0000", 2024-02-05T07:29:30.853 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.853 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.853 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.618760+0000", 2024-02-05T07:29:30.853 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.854 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.854 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.618761+0000", 2024-02-05T07:29:30.854 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9599999999999998e-07 2024-02-05T07:29:30.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.854 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.854 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.618765+0000", 2024-02-05T07:29:30.854 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.189e-06 2024-02-05T07:29:30.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.854 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.854 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.618776+0000", 2024-02-05T07:29:30.854 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0902e-05 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.618785+0000", 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1710000000000002e-06 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.619268+0000", 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00048305199999999998 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.619387+0000", 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011914899999999999 2024-02-05T07:29:30.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,3,5", 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.619751+0000", 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00036416399999999999 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.622694+0000", 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0029429730000000002 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.629424+0000", 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00673056 2024-02-05T07:29:30.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.634303+0000", 2024-02-05T07:29:30.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0048785130000000001 2024-02-05T07:29:30.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.652101+0000", 2024-02-05T07:29:30.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.017798293999999999 2024-02-05T07:29:30.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.652116+0000", 2024-02-05T07:29:30.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4260999999999999e-05 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.670979+0000", 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.01886342 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.671065+0000", 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6271000000000001e-05 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.859 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:167774 7.14 7:2c569e8c:::benchmark_data_smithi007_77086_object63179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T07:29:30.859 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:50.714711+0000", 2024-02-05T07:29:30.859 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 340.11987626000001, 2024-02-05T07:29:30.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99667040399999995, 2024-02-05T07:29:30.859 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.859 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.859 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.859 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.859 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.859 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 167774 2024-02-05T07:29:30.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.859 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714711+0000", 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714711+0000", 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714711+0000", 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9400000000000004e-07 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.860 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714715+0000", 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8029999999999998e-06 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714724+0000", 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7360000000000007e-06 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714731+0000", 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8360000000000003e-06 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714778+0000", 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6859e-05 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.714785+0000", 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.255e-06 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.674406+0000", 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.95962156499999995 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.674516+0000", 2024-02-05T07:29:30.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000109478 2024-02-05T07:29:30.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-02-05T07:29:30.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.674803+0000", 2024-02-05T07:29:30.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028674499999999998 2024-02-05T07:29:30.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.679654+0000", 2024-02-05T07:29:30.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0048510940000000002 2024-02-05T07:29:30.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.679715+0000", 2024-02-05T07:29:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1455e-05 2024-02-05T07:29:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.711096+0000", 2024-02-05T07:29:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.031380907999999999 2024-02-05T07:29:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.711335+0000", 2024-02-05T07:29:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000239344 2024-02-05T07:29:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.711381+0000", 2024-02-05T07:29:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5531999999999997e-05 2024-02-05T07:29:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:167790 7.14 7:2e0a6b7a:::benchmark_data_smithi007_77086_object63195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T07:29:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:50.728786+0000", 2024-02-05T07:29:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 340.10580060900003, 2024-02-05T07:29:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98310929400000002, 2024-02-05T07:29:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.866 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 167790 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.728786+0000", 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.728786+0000", 2024-02-05T07:29:30.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.728787+0000", 2024-02-05T07:29:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4300000000000001e-07 2024-02-05T07:29:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.728792+0000", 2024-02-05T07:29:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.544e-06 2024-02-05T07:29:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.868 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.728802+0000", 2024-02-05T07:29:30.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0753e-05 2024-02-05T07:29:30.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.728811+0000", 2024-02-05T07:29:30.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0989999999999999e-06 2024-02-05T07:29:30.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.870 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.729151+0000", 2024-02-05T07:29:30.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00034059900000000002 2024-02-05T07:29:30.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:50.729160+0000", 2024-02-05T07:29:30.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2699999999999993e-06 2024-02-05T07:29:30.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.676673+0000", 2024-02-05T07:29:30.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.94751278999999999 2024-02-05T07:29:30.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.676786+0000", 2024-02-05T07:29:30.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011234300000000001 2024-02-05T07:29:30.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-02-05T07:29:30.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.677150+0000", 2024-02-05T07:29:30.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00036407799999999999 2024-02-05T07:29:30.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.683604+0000", 2024-02-05T07:29:30.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0064540830000000002 2024-02-05T07:29:30.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.690849+0000", 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0072456550000000002 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.711389+0000", 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.020539458999999999 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.711857+0000", 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00046811200000000002 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:51.711895+0000", 2024-02-05T07:29:30.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8566000000000003e-05 2024-02-05T07:29:30.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.874 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.874 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:172539 7.3 7:c1cc9372:::benchmark_data_smithi007_77086_object67944:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-02-05T07:29:30.874 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:56.366961+0000", 2024-02-05T07:29:30.874 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 334.46762528800002, 2024-02-05T07:29:30.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0132783910000001, 2024-02-05T07:29:30.874 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 172539 2024-02-05T07:29:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.366961+0000", 2024-02-05T07:29:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.366961+0000", 2024-02-05T07:29:30.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.366964+0000", 2024-02-05T07:29:30.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0540000000000002e-06 2024-02-05T07:29:30.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.366968+0000", 2024-02-05T07:29:30.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8180000000000003e-06 2024-02-05T07:29:30.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.877 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.366989+0000", 2024-02-05T07:29:30.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0443000000000001e-05 2024-02-05T07:29:30.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.877 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.366998+0000", 2024-02-05T07:29:30.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0429999999999996e-06 2024-02-05T07:29:30.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.877 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.367057+0000", 2024-02-05T07:29:30.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9054000000000001e-05 2024-02-05T07:29:30.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.367066+0000", 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8720000000000008e-06 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.367124+0000", 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0000582739999999 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.367240+0000", 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011589199999999999 2024-02-05T07:29:30.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6", 2024-02-05T07:29:30.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.367712+0000", 2024-02-05T07:29:30.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00047228400000000001 2024-02-05T07:29:30.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.376730+0000", 2024-02-05T07:29:30.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0090178670000000006 2024-02-05T07:29:30.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.376768+0000", 2024-02-05T07:29:30.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7564999999999997e-05 2024-02-05T07:29:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.380151+0000", 2024-02-05T07:29:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033831460000000001 2024-02-05T07:29:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.380196+0000", 2024-02-05T07:29:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5605999999999997e-05 2024-02-05T07:29:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.380240+0000", 2024-02-05T07:29:30.881 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3473000000000002e-05 2024-02-05T07:29:30.881 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.881 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.881 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.881 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:172576 7.3 7:c184961a:::benchmark_data_smithi007_77086_object67981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-02-05T07:29:30.881 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:56.399380+0000", 2024-02-05T07:29:30.881 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 334.43520645799998, 2024-02-05T07:29:30.881 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98101083600000005, 2024-02-05T07:29:30.881 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.881 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.881 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.881 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.882 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.882 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 172576 2024-02-05T07:29:30.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.882 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399380+0000", 2024-02-05T07:29:30.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399380+0000", 2024-02-05T07:29:30.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399382+0000", 2024-02-05T07:29:30.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5489999999999999e-06 2024-02-05T07:29:30.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399388+0000", 2024-02-05T07:29:30.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9610000000000002e-06 2024-02-05T07:29:30.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399400+0000", 2024-02-05T07:29:30.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1773999999999999e-05 2024-02-05T07:29:30.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399408+0000", 2024-02-05T07:29:30.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.089e-06 2024-02-05T07:29:30.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399465+0000", 2024-02-05T07:29:30.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7015999999999998e-05 2024-02-05T07:29:30.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.885 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.399474+0000", 2024-02-05T07:29:30.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8449999999999992e-06 2024-02-05T07:29:30.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.885 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.370277+0000", 2024-02-05T07:29:30.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97080362399999998 2024-02-05T07:29:30.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.885 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.370383+0000", 2024-02-05T07:29:30.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010588 2024-02-05T07:29:30.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6", 2024-02-05T07:29:30.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.370860+0000", 2024-02-05T07:29:30.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00047653399999999997 2024-02-05T07:29:30.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.376807+0000", 2024-02-05T07:29:30.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059478839999999996 2024-02-05T07:29:30.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.376849+0000", 2024-02-05T07:29:30.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1213e-05 2024-02-05T07:29:30.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.887 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.380314+0000", 2024-02-05T07:29:30.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0034657580000000002 2024-02-05T07:29:30.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.887 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.380356+0000", 2024-02-05T07:29:30.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1798000000000002e-05 2024-02-05T07:29:30.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.380391+0000", 2024-02-05T07:29:30.888 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4910999999999998e-05 2024-02-05T07:29:30.888 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.888 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.888 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.888 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:172590 7.3 7:c2af45be:::benchmark_data_smithi007_77086_object67995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-02-05T07:29:30.888 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:23:56.414416+0000", 2024-02-05T07:29:30.888 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 334.42017127499997, 2024-02-05T07:29:30.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96900304500000001, 2024-02-05T07:29:30.889 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.889 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.889 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.889 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.889 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.889 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 172590 2024-02-05T07:29:30.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.889 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414416+0000", 2024-02-05T07:29:30.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414416+0000", 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414417+0000", 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5680000000000001e-06 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414422+0000", 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7090000000000001e-06 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414434+0000", 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1782e-05 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414442+0000", 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1440000000000003e-06 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414568+0000", 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012621400000000001 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:56.414578+0000", 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6120000000000005e-06 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.373764+0000", 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.95918598799999999 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.373873+0000", 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000109548 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6", 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.374423+0000", 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00054968200000000001 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.380397+0000", 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059747280000000003 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.381204+0000", 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00080686199999999999 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.383267+0000", 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020622879999999998 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.383377+0000", 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011037 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:23:57.383419+0000", 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.155e-05 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.894 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.895 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.895 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174548 7.a 7:51469907:::benchmark_data_smithi007_77086_object69953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:29:30.895 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.374698+0000", 2024-02-05T07:29:30.895 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 330.45988918699999, 2024-02-05T07:29:30.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1298535379999999, 2024-02-05T07:29:30.895 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.895 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.895 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.896 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.896 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.896 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174548 2024-02-05T07:29:30.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.896 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.896 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.374698+0000", 2024-02-05T07:29:30.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.374698+0000", 2024-02-05T07:29:30.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.374699+0000", 2024-02-05T07:29:30.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.747e-06 2024-02-05T07:29:30.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.374707+0000", 2024-02-05T07:29:30.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2269999999999996e-06 2024-02-05T07:29:30.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.374720+0000", 2024-02-05T07:29:30.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3045e-05 2024-02-05T07:29:30.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.374729+0000", 2024-02-05T07:29:30.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8390000000000002e-06 2024-02-05T07:29:30.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.438715+0000", 2024-02-05T07:29:30.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.063985095000000006 2024-02-05T07:29:30.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.438727+0000", 2024-02-05T07:29:30.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2291e-05 2024-02-05T07:29:30.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.501248+0000", 2024-02-05T07:29:30.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.062520916 2024-02-05T07:29:30.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.501389+0000", 2024-02-05T07:29:30.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014126500000000001 2024-02-05T07:29:30.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:29:30.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.501459+0000", 2024-02-05T07:29:30.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9876000000000007e-05 2024-02-05T07:29:30.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504052+0000", 2024-02-05T07:29:30.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025928729999999999 2024-02-05T07:29:30.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504402+0000", 2024-02-05T07:29:30.902 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00035052899999999998 2024-02-05T07:29:30.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.902 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.902 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504465+0000", 2024-02-05T07:29:30.902 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2397999999999994e-05 2024-02-05T07:29:30.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.903 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504516+0000", 2024-02-05T07:29:30.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1242000000000002e-05 2024-02-05T07:29:30.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.903 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.904 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504551+0000", 2024-02-05T07:29:30.904 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5194999999999998e-05 2024-02-05T07:29:30.904 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.904 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.904 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.905 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174560 7.a 7:52e66e99:::benchmark_data_smithi007_77086_object69965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:29:30.905 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.439432+0000", 2024-02-05T07:29:30.905 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 330.39515448499998, 2024-02-05T07:29:30.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.073606013, 2024-02-05T07:29:30.905 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.905 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.905 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.905 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.906 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.906 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174560 2024-02-05T07:29:30.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.906 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.906 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.906 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.906 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.439432+0000", 2024-02-05T07:29:30.906 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.907 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.907 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.439432+0000", 2024-02-05T07:29:30.907 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.907 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.907 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.439433+0000", 2024-02-05T07:29:30.907 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7800000000000002e-07 2024-02-05T07:29:30.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.908 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.908 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.439435+0000", 2024-02-05T07:29:30.908 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6589999999999999e-06 2024-02-05T07:29:30.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.908 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.908 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.439442+0000", 2024-02-05T07:29:30.908 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2339999999999999e-06 2024-02-05T07:29:30.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.908 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.439447+0000", 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.677e-06 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.441161+0000", 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017137260000000001 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.441169+0000", 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7300000000000005e-06 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504603+0000", 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0634339829999999 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504696+0000", 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3459000000000001e-05 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.504776+0000", 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0060000000000003e-05 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.910 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.512888+0000", 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0081121189999999992 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.512907+0000", 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8950999999999999e-05 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.512943+0000", 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5565999999999998e-05 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.911 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.512999+0000", 2024-02-05T07:29:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6399e-05 2024-02-05T07:29:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.513038+0000", 2024-02-05T07:29:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8971999999999999e-05 2024-02-05T07:29:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174570 7.a 7:56c3dbcb:::benchmark_data_smithi007_77086_object69975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:29:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.442757+0000", 2024-02-05T07:29:30.913 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 330.39183011300003, 2024-02-05T07:29:30.913 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0731317869999999, 2024-02-05T07:29:30.913 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.913 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.913 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.913 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.913 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.913 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174570 2024-02-05T07:29:30.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.913 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.913 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.913 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442757+0000", 2024-02-05T07:29:30.913 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.914 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.914 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442757+0000", 2024-02-05T07:29:30.914 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.914 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.914 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442758+0000", 2024-02-05T07:29:30.914 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.564e-06 2024-02-05T07:29:30.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.914 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.914 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442764+0000", 2024-02-05T07:29:30.915 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6779999999999997e-06 2024-02-05T07:29:30.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.915 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.915 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.915 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442773+0000", 2024-02-05T07:29:30.915 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3810000000000005e-06 2024-02-05T07:29:30.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.915 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.915 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.915 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442781+0000", 2024-02-05T07:29:30.915 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0390000000000005e-06 2024-02-05T07:29:30.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.915 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.915 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.916 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442954+0000", 2024-02-05T07:29:30.916 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017238600000000001 2024-02-05T07:29:30.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.916 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.916 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.916 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.442962+0000", 2024-02-05T07:29:30.916 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4090000000000003e-06 2024-02-05T07:29:30.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.916 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.916 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.916 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.507306+0000", 2024-02-05T07:29:30.916 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0643434119999999 2024-02-05T07:29:30.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.917 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.917 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.507414+0000", 2024-02-05T07:29:30.917 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108436 2024-02-05T07:29:30.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.917 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:29:30.917 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.507527+0000", 2024-02-05T07:29:30.917 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011256699999999999 2024-02-05T07:29:30.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.513048+0000", 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0055218790000000004 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.513083+0000", 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4686e-05 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.515804+0000", 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002721135 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.919 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.919 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.515862+0000", 2024-02-05T07:29:30.919 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7716000000000001e-05 2024-02-05T07:29:30.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.919 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.919 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.919 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.515888+0000", 2024-02-05T07:29:30.919 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6499000000000002e-05 2024-02-05T07:29:30.919 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.919 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.919 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.919 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.919 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174574 7.a 7:57454b6e:::benchmark_data_smithi007_77086_object69979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:29:30.920 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.444932+0000", 2024-02-05T07:29:30.920 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 330.38965472500001, 2024-02-05T07:29:30.920 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.071142161, 2024-02-05T07:29:30.920 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.920 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.920 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.920 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.920 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.920 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174574 2024-02-05T07:29:30.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.920 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.920 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.920 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.920 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.444932+0000", 2024-02-05T07:29:30.921 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.921 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.921 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.444932+0000", 2024-02-05T07:29:30.921 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.921 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.921 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.444934+0000", 2024-02-05T07:29:30.921 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2699999999999999e-06 2024-02-05T07:29:30.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.444940+0000", 2024-02-05T07:29:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3929999999999999e-06 2024-02-05T07:29:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.444952+0000", 2024-02-05T07:29:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.187e-05 2024-02-05T07:29:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.444960+0000", 2024-02-05T07:29:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8910000000000001e-06 2024-02-05T07:29:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.923 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.923 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.923 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445018+0000", 2024-02-05T07:29:30.923 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7571999999999997e-05 2024-02-05T07:29:30.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.923 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.923 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.923 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445028+0000", 2024-02-05T07:29:30.923 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0023e-05 2024-02-05T07:29:30.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.923 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.923 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.923 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.510169+0000", 2024-02-05T07:29:30.924 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0651404339999999 2024-02-05T07:29:30.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.924 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.924 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.510300+0000", 2024-02-05T07:29:30.924 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013160900000000001 2024-02-05T07:29:30.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.924 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:29:30.924 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.510424+0000", 2024-02-05T07:29:30.924 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000124313 2024-02-05T07:29:30.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.925 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.925 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.515907+0000", 2024-02-05T07:29:30.925 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.005482748 2024-02-05T07:29:30.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.925 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.925 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.515947+0000", 2024-02-05T07:29:30.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:30 smithi007 ceph-mon[29098]: pgmap v556: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 175 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s; 147/8991 objects degraded (1.635%); 709/8991 objects misplaced (7.886%) 2024-02-05T07:29:30.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:30 smithi007 ceph-mon[29098]: osdmap e339: 8 total, 8 up, 7 in 2024-02-05T07:29:30.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:30 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2136162075' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:29:30.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:29:30.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:30 smithi007 ceph-mon[37158]: pgmap v556: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 175 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s; 147/8991 objects degraded (1.635%); 709/8991 objects misplaced (7.886%) 2024-02-05T07:29:30.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:30 smithi007 ceph-mon[37158]: osdmap e339: 8 total, 8 up, 7 in 2024-02-05T07:29:30.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:30 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2136162075' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:29:30.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:29:30.928 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9425000000000003e-05 2024-02-05T07:29:30.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.928 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.928 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.928 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.515986+0000", 2024-02-05T07:29:30.928 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9833000000000003e-05 2024-02-05T07:29:30.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.928 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.928 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.928 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516038+0000", 2024-02-05T07:29:30.928 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1842000000000003e-05 2024-02-05T07:29:30.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.929 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.929 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.929 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516074+0000", 2024-02-05T07:29:30.929 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5938e-05 2024-02-05T07:29:30.929 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.929 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.929 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.929 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.930 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174575 7.a 7:546d0937:::benchmark_data_smithi007_77086_object69980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:29:30.930 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.445882+0000", 2024-02-05T07:29:30.930 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 330.38870437399999, 2024-02-05T07:29:30.930 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0703599480000001, 2024-02-05T07:29:30.930 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.930 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.930 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.930 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.931 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.931 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174575 2024-02-05T07:29:30.931 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.931 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.931 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.931 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.931 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445882+0000", 2024-02-05T07:29:30.931 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.931 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.931 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.931 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.931 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445882+0000", 2024-02-05T07:29:30.931 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.932 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.932 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445883+0000", 2024-02-05T07:29:30.932 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0800000000000003e-07 2024-02-05T07:29:30.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.932 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.932 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445887+0000", 2024-02-05T07:29:30.932 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.388e-06 2024-02-05T07:29:30.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.932 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445895+0000", 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9950000000000005e-06 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445912+0000", 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7648000000000001e-05 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445991+0000", 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8555999999999998e-05 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.933 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.445999+0000", 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1759999999999993e-06 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.513115+0000", 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0671161650000001 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.513221+0000", 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000105461 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:29:30.935 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.513337+0000", 2024-02-05T07:29:30.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115876 2024-02-05T07:29:30.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.935 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.935 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516083+0000", 2024-02-05T07:29:30.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0027464239999999999 2024-02-05T07:29:30.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.935 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.935 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516119+0000", 2024-02-05T07:29:30.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6038000000000003e-05 2024-02-05T07:29:30.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.936 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.936 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516161+0000", 2024-02-05T07:29:30.936 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1631999999999997e-05 2024-02-05T07:29:30.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.936 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.936 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516212+0000", 2024-02-05T07:29:30.936 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1264999999999998e-05 2024-02-05T07:29:30.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516242+0000", 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.0416000000000002e-05 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174577 7.a 7:5537fb92:::benchmark_data_smithi007_77086_object69982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.448795+0000", 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 330.38579221800001, 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0775387089999999, 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.937 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.938 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.938 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.938 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.938 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174577 2024-02-05T07:29:30.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.938 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.938 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.938 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.938 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448795+0000", 2024-02-05T07:29:30.938 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.938 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.938 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.938 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448795+0000", 2024-02-05T07:29:30.939 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.939 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.939 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448796+0000", 2024-02-05T07:29:30.939 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1909999999999999e-06 2024-02-05T07:29:30.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.939 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.939 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448800+0000", 2024-02-05T07:29:30.939 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8449999999999996e-06 2024-02-05T07:29:30.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448808+0000", 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5529999999999993e-06 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448815+0000", 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4710000000000004e-06 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448956+0000", 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000141304 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.448968+0000", 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1703e-05 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516273+0000", 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0673056679999999 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516386+0000", 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000112255 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.516498+0000", 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011268300000000001 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.522764+0000", 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0062659279999999996 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.522779+0000", 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4786e-05 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.942 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526239+0000", 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003459873 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526293+0000", 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3841000000000002e-05 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526333+0000", 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0608000000000002e-05 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.943 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.944 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.944 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174585 7.a 7:5d56743a:::benchmark_data_smithi007_77086_object69990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:29:30.944 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.455082+0000", 2024-02-05T07:29:30.944 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 330.37950472900002, 2024-02-05T07:29:30.944 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0802006580000001, 2024-02-05T07:29:30.944 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.944 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.944 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.944 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.944 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.944 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174585 2024-02-05T07:29:30.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.944 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.945 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.945 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455082+0000", 2024-02-05T07:29:30.945 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.945 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.945 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455082+0000", 2024-02-05T07:29:30.945 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.945 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.945 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455084+0000", 2024-02-05T07:29:30.946 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.497e-06 2024-02-05T07:29:30.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.946 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.946 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455088+0000", 2024-02-05T07:29:30.946 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1010000000000002e-06 2024-02-05T07:29:30.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.946 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.946 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455097+0000", 2024-02-05T07:29:30.946 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8950000000000004e-06 2024-02-05T07:29:30.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.947 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.947 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455104+0000", 2024-02-05T07:29:30.947 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1420000000000004e-06 2024-02-05T07:29:30.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.947 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.947 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.947 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455217+0000", 2024-02-05T07:29:30.947 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113678 2024-02-05T07:29:30.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.947 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.947 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.947 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.455226+0000", 2024-02-05T07:29:30.947 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8729999999999996e-06 2024-02-05T07:29:30.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.948 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.948 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.519622+0000", 2024-02-05T07:29:30.948 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0643955549999999 2024-02-05T07:29:30.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.948 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.948 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.519736+0000", 2024-02-05T07:29:30.948 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000114598 2024-02-05T07:29:30.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.948 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:29:30.948 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.519855+0000", 2024-02-05T07:29:30.948 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011813 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.522879+0000", 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030242789999999999 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526372+0000", 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0034927719999999999 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535191+0000", 2024-02-05T07:29:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0088197870000000008 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535240+0000", 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8751999999999997e-05 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535283+0000", 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2599000000000003e-05 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.950 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174598 7.a 7:5c33d55a:::benchmark_data_smithi007_77086_object70003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.466123+0000", 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 330.36846339700003, 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0693828480000001, 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174598 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466123+0000", 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.952 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.952 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.952 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466123+0000", 2024-02-05T07:29:30.952 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.952 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.952 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.952 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.952 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466124+0000", 2024-02-05T07:29:30.952 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7199999999999998e-07 2024-02-05T07:29:30.952 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.952 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.952 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.952 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466128+0000", 2024-02-05T07:29:30.952 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2629999999999997e-06 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466136+0000", 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8970000000000008e-06 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466143+0000", 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3430000000000001e-06 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466266+0000", 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000123196 2024-02-05T07:29:30.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.954 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.954 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.466274+0000", 2024-02-05T07:29:30.954 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8380000000000002e-06 2024-02-05T07:29:30.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.954 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.954 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.522923+0000", 2024-02-05T07:29:30.954 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.056649795 2024-02-05T07:29:30.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.954 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.954 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.523050+0000", 2024-02-05T07:29:30.954 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012692199999999999 2024-02-05T07:29:30.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.955 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.955 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:29:30.955 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.523166+0000", 2024-02-05T07:29:30.955 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011522 2024-02-05T07:29:30.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.955 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.955 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.955 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526407+0000", 2024-02-05T07:29:30.955 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032419609999999998 2024-02-05T07:29:30.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.955 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.955 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.956 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535326+0000", 2024-02-05T07:29:30.956 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0089187580000000006 2024-02-05T07:29:30.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.956 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.956 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.956 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535424+0000", 2024-02-05T07:29:30.956 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7769000000000005e-05 2024-02-05T07:29:30.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.956 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.956 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.956 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535470+0000", 2024-02-05T07:29:30.957 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6415000000000002e-05 2024-02-05T07:29:30.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.957 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.957 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.957 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535506+0000", 2024-02-05T07:29:30.957 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5799000000000001e-05 2024-02-05T07:29:30.957 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.957 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.957 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.957 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.957 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174602 7.a 7:574a2af3:::benchmark_data_smithi007_77086_object70007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:29:30.957 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.470147+0000", 2024-02-05T07:29:30.957 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 330.36443966899998, 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.065611767, 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174602 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470147+0000", 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470147+0000", 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470148+0000", 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2139999999999999e-06 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470151+0000", 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1559999999999999e-06 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.959 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.960 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.960 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470159+0000", 2024-02-05T07:29:30.960 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9540000000000004e-06 2024-02-05T07:29:30.960 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.960 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.960 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.960 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470166+0000", 2024-02-05T07:29:30.960 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4409999999999998e-06 2024-02-05T07:29:30.960 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.960 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.960 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.960 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470215+0000", 2024-02-05T07:29:30.960 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9268999999999999e-05 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.470223+0000", 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3159999999999997e-06 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526550+0000", 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0563263060000001 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526752+0000", 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020180799999999999 2024-02-05T07:29:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.526904+0000", 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000152776 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535587+0000", 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0086831110000000003 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535636+0000", 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8826999999999999e-05 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535676+0000", 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0061999999999997e-05 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535723+0000", 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7150000000000001e-05 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535759+0000", 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5376999999999997e-05 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.963 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.964 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.964 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.964 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:174607 7.a 7:59b090e0:::benchmark_data_smithi007_77086_object70012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-05T07:29:30.964 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:00.516807+0000", 2024-02-05T07:29:30.964 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 330.31777939800003, 2024-02-05T07:29:30.964 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.025446978, 2024-02-05T07:29:30.964 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.964 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.964 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.964 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.964 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 174607 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516807+0000", 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516807+0000", 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.965 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516809+0000", 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4780000000000001e-06 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516815+0000", 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.772e-06 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516826+0000", 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1617000000000001e-05 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.966 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.967 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516835+0000", 2024-02-05T07:29:30.967 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5210000000000002e-06 2024-02-05T07:29:30.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.967 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.967 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.967 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516892+0000", 2024-02-05T07:29:30.967 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7663e-05 2024-02-05T07:29:30.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.968 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.968 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:00.516913+0000", 2024-02-05T07:29:30.968 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0279e-05 2024-02-05T07:29:30.968 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.968 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.968 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.531495+0000", 2024-02-05T07:29:30.968 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0145823300000001 2024-02-05T07:29:30.968 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.968 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.969 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.531602+0000", 2024-02-05T07:29:30.969 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010724499999999999 2024-02-05T07:29:30.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.969 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:29:30.969 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.531682+0000", 2024-02-05T07:29:30.969 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9888000000000005e-05 2024-02-05T07:29:30.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.970 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.970 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535596+0000", 2024-02-05T07:29:30.970 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039139190000000001 2024-02-05T07:29:30.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.970 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.970 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.971 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.535808+0000", 2024-02-05T07:29:30.971 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00021158799999999999 2024-02-05T07:29:30.971 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.971 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.971 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.971 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.542052+0000", 2024-02-05T07:29:30.972 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0062446610000000003 2024-02-05T07:29:30.972 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.972 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.972 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.972 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.542200+0000", 2024-02-05T07:29:30.972 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014800800000000001 2024-02-05T07:29:30.972 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.972 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.972 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.972 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:01.542254+0000", 2024-02-05T07:29:30.972 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4008999999999997e-05 2024-02-05T07:29:30.973 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.973 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.973 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.973 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.973 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.973 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:192513 7.e 7:72d2a106:::benchmark_data_smithi007_77086_object87918:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e177)", 2024-02-05T07:29:30.973 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:19.509516+0000", 2024-02-05T07:29:30.974 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 311.32507088400001, 2024-02-05T07:29:30.974 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0316103679999999, 2024-02-05T07:29:30.974 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.974 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:29:30.974 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.974 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.974 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.974 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 192513 2024-02-05T07:29:30.974 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.974 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.974 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.974 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.974 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.509516+0000", 2024-02-05T07:29:30.974 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.509516+0000", 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.509517+0000", 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4810000000000001e-06 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.509521+0000", 2024-02-05T07:29:30.975 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3450000000000002e-06 2024-02-05T07:29:30.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.976 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.976 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.976 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.509530+0000", 2024-02-05T07:29:30.976 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0850000000000003e-06 2024-02-05T07:29:30.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.976 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.976 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.976 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.509536+0000", 2024-02-05T07:29:30.976 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2859999999999996e-06 2024-02-05T07:29:30.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.976 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.976 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.976 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.512281+0000", 2024-02-05T07:29:30.977 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002744486 2024-02-05T07:29:30.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.977 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-02-05T07:29:30.977 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.512287+0000", 2024-02-05T07:29:30.977 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2960000000000004e-06 2024-02-05T07:29:30.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.977 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.977 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.722402+0000", 2024-02-05T07:29:30.977 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.210115154 2024-02-05T07:29:30.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.977 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.978 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.722412+0000", 2024-02-05T07:29:30.978 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0029e-05 2024-02-05T07:29:30.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.978 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.978 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.978 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.541052+0000", 2024-02-05T07:29:30.978 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.81864034900000004 2024-02-05T07:29:30.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.978 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.978 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.978 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.541126+0000", 2024-02-05T07:29:30.978 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3856999999999994e-05 2024-02-05T07:29:30.978 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.978 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.979 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.979 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24719.0:192531 7.1a 7:5f4bc8c6:::benchmark_data_smithi007_77086_object87936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-02-05T07:29:30.979 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:24:19.636512+0000", 2024-02-05T07:29:30.979 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 311.19807438999999, 2024-02-05T07:29:30.979 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99201874700000003, 2024-02-05T07:29:30.979 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:29:30.979 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:29:30.979 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:29:30.979 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24719", 2024-02-05T07:29:30.979 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:29:30.979 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 192531 2024-02-05T07:29:30.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.980 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:29:30.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.980 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:29:30.980 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.636512+0000", 2024-02-05T07:29:30.980 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.980 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:29:30.980 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.636512+0000", 2024-02-05T07:29:30.980 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:29:30.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.980 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:29:30.981 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.636514+0000", 2024-02-05T07:29:30.981 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.392e-06 2024-02-05T07:29:30.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.981 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:29:30.981 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.636518+0000", 2024-02-05T07:29:30.981 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.121e-06 2024-02-05T07:29:30.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.981 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:29:30.981 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.636529+0000", 2024-02-05T07:29:30.981 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1238e-05 2024-02-05T07:29:30.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.636538+0000", 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9800000000000004e-06 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.736108+0000", 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.099569629000000007 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:19.736119+0000", 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1232e-05 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.982 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.620471+0000", 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.88435169400000002 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.620582+0000", 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000111731 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,5", 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.620822+0000", 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023922 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.983 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.984 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.622275+0000", 2024-02-05T07:29:30.984 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001453712 2024-02-05T07:29:30.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.984 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:29:30.984 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.626649+0000", 2024-02-05T07:29:30.984 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043739740000000001 2024-02-05T07:29:30.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.984 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:29:30.984 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.628445+0000", 2024-02-05T07:29:30.984 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001795439 2024-02-05T07:29:30.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.985 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.985 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:29:30.985 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.628494+0000", 2024-02-05T07:29:30.985 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9283999999999998e-05 2024-02-05T07:29:30.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:29:30.985 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:29:30.985 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:29:30.985 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:24:20.628531+0000", 2024-02-05T07:29:30.985 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7101000000000003e-05 2024-02-05T07:29:30.985 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.985 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.985 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.985 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:29:30.986 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:29:30.986 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:29:30.986 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:31.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:31.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:31.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:29:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:29:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:30 smithi090 ceph-mon[33213]: pgmap v556: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 171 active+clean; 175 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s; 147/8991 objects degraded (1.635%); 709/8991 objects misplaced (7.886%) 2024-02-05T07:29:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:30 smithi090 ceph-mon[33213]: osdmap e339: 8 total, 8 up, 7 in 2024-02-05T07:29:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:30 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2136162075' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:29:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:29:31.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:31.139 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:31.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:31.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:31.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:31.265 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:31.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:31.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:31.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:31.390 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:31.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:31.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:31.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:31.520 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:31.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:31.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:31.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:31.647 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:31.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:31.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:31.775 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:31.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:31.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:31.901 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:31.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:31.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:31 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:31 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:29:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:31 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2136162075' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:29:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:31 smithi007 ceph-mon[29098]: osdmap e340: 8 total, 8 up, 7 in 2024-02-05T07:29:31.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:31 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:31.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:31 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:29:31.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:31 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2136162075' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:29:31.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:31 smithi007 ceph-mon[37158]: osdmap e340: 8 total, 8 up, 7 in 2024-02-05T07:29:32.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:31 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:31 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:29:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:31 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2136162075' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:29:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:31 smithi090 ceph-mon[33213]: osdmap e340: 8 total, 8 up, 7 in 2024-02-05T07:29:32.027 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:32.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:32.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:32.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:32.153 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:32.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:32.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:32.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:32.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:32.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:32.404 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:32.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:32.506 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:32.530 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:32.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:32.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:32.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:32.661 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:32.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:32.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:32.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:32.791 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:32.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:32.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:32.893 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:32.920 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:32.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:32 smithi007 ceph-mon[29098]: pgmap v559: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 14 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 831 op/s; 2.9 MiB/s, 46 objects/s recovering 2024-02-05T07:29:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:32 smithi007 ceph-mon[29098]: 6.18 scrub starts 2024-02-05T07:29:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:32 smithi007 ceph-mon[29098]: 6.18 scrub ok 2024-02-05T07:29:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:32 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:29:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:32 smithi007 ceph-mon[29098]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 147/8991 objects degraded (1.635%), 1 pg degraded, 2 pgs undersized) 2024-02-05T07:29:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:32 smithi007 ceph-mon[37158]: pgmap v559: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 14 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 831 op/s; 2.9 MiB/s, 46 objects/s recovering 2024-02-05T07:29:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:32 smithi007 ceph-mon[37158]: 6.18 scrub starts 2024-02-05T07:29:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:32 smithi007 ceph-mon[37158]: 6.18 scrub ok 2024-02-05T07:29:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:32 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:29:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:32 smithi007 ceph-mon[37158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 147/8991 objects degraded (1.635%), 1 pg degraded, 2 pgs undersized) 2024-02-05T07:29:33.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:32 smithi090 ceph-mon[33213]: pgmap v559: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 14 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 831 op/s; 2.9 MiB/s, 46 objects/s recovering 2024-02-05T07:29:33.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:32 smithi090 ceph-mon[33213]: 6.18 scrub starts 2024-02-05T07:29:33.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:32 smithi090 ceph-mon[33213]: 6.18 scrub ok 2024-02-05T07:29:33.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:32 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:29:33.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:32 smithi090 ceph-mon[33213]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 147/8991 objects degraded (1.635%), 1 pg degraded, 2 pgs undersized) 2024-02-05T07:29:33.047 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:33.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:33.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:33.149 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:33.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:33.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:33.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:33.302 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:33.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:33.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:33.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:33.431 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:33.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:33.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:33.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:33.559 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:33.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:33.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:33.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:33.686 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:33.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:33.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:33.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:33.818 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:33.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:33.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:33.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:29:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:29:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:29:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:33.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:33.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:34.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:34.068 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:34.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:34.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:34.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:34.197 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:34.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:34.299 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:34.324 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:34.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:34.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:34.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:34.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:34.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:34.580 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:34.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:34.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:34.682 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:34.710 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:34.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:34.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:34.840 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:34.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:34.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:34 smithi007 ceph-mon[29098]: pgmap v560: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 14 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 918 B/s rd, 0 B/s wr, 597 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-02-05T07:29:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:34 smithi007 ceph-mon[29098]: 4.18 scrub starts 2024-02-05T07:29:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:34 smithi007 ceph-mon[29098]: 4.18 scrub ok 2024-02-05T07:29:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:34 smithi007 ceph-mon[29098]: 8.9 scrub starts 2024-02-05T07:29:34.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:34 smithi007 ceph-mon[29098]: 8.9 scrub ok 2024-02-05T07:29:34.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:34 smithi007 ceph-mon[29098]: 8.8 scrub starts 2024-02-05T07:29:34.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:34 smithi007 ceph-mon[29098]: 8.8 scrub ok 2024-02-05T07:29:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:34 smithi007 ceph-mon[37158]: pgmap v560: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 14 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 918 B/s rd, 0 B/s wr, 597 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-02-05T07:29:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:34 smithi007 ceph-mon[37158]: 4.18 scrub starts 2024-02-05T07:29:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:34 smithi007 ceph-mon[37158]: 4.18 scrub ok 2024-02-05T07:29:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:34 smithi007 ceph-mon[37158]: 8.9 scrub starts 2024-02-05T07:29:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:34 smithi007 ceph-mon[37158]: 8.9 scrub ok 2024-02-05T07:29:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:34 smithi007 ceph-mon[37158]: 8.8 scrub starts 2024-02-05T07:29:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:34 smithi007 ceph-mon[37158]: 8.8 scrub ok 2024-02-05T07:29:34.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:34.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:34.969 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:34.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:35.024 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-02-05T07:29:35.024 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-02-05T07:29:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:34 smithi090 ceph-mon[33213]: pgmap v560: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 14 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 918 B/s rd, 0 B/s wr, 597 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-02-05T07:29:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:34 smithi090 ceph-mon[33213]: 4.18 scrub starts 2024-02-05T07:29:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:34 smithi090 ceph-mon[33213]: 4.18 scrub ok 2024-02-05T07:29:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:34 smithi090 ceph-mon[33213]: 8.9 scrub starts 2024-02-05T07:29:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:34 smithi090 ceph-mon[33213]: 8.9 scrub ok 2024-02-05T07:29:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:34 smithi090 ceph-mon[33213]: 8.8 scrub starts 2024-02-05T07:29:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:34 smithi090 ceph-mon[33213]: 8.8 scrub ok 2024-02-05T07:29:35.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:35.096 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:35.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:35.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:35.222 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:35.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:35.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:35.350 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:35.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:35.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:35.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:35.477 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:35.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:35.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:35.580 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:29:35.602 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:35.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:35.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:35.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:35.731 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:35.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:35.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:35.860 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:35.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:35.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:35 smithi007 ceph-mon[29098]: 8.a scrub starts 2024-02-05T07:29:35.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:35 smithi007 ceph-mon[29098]: 8.a scrub ok 2024-02-05T07:29:35.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:35 smithi007 ceph-mon[29098]: 8.6 scrub starts 2024-02-05T07:29:35.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:35 smithi007 ceph-mon[29098]: 8.6 scrub ok 2024-02-05T07:29:35.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:35 smithi007 ceph-mon[29098]: 8.13 scrub starts 2024-02-05T07:29:35.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:35 smithi007 ceph-mon[29098]: 8.13 scrub ok 2024-02-05T07:29:35.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:35 smithi007 ceph-mon[37158]: 8.a scrub starts 2024-02-05T07:29:35.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:35 smithi007 ceph-mon[37158]: 8.a scrub ok 2024-02-05T07:29:35.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:35 smithi007 ceph-mon[37158]: 8.6 scrub starts 2024-02-05T07:29:35.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:35 smithi007 ceph-mon[37158]: 8.6 scrub ok 2024-02-05T07:29:35.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:35 smithi007 ceph-mon[37158]: 8.13 scrub starts 2024-02-05T07:29:35.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:35 smithi007 ceph-mon[37158]: 8.13 scrub ok 2024-02-05T07:29:35.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:35.988 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:35.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:35.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:36.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T07:29:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:35 smithi090 ceph-mon[33213]: 8.a scrub starts 2024-02-05T07:29:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:35 smithi090 ceph-mon[33213]: 8.a scrub ok 2024-02-05T07:29:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:35 smithi090 ceph-mon[33213]: 8.6 scrub starts 2024-02-05T07:29:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:35 smithi090 ceph-mon[33213]: 8.6 scrub ok 2024-02-05T07:29:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:35 smithi090 ceph-mon[33213]: 8.13 scrub starts 2024-02-05T07:29:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:35 smithi090 ceph-mon[33213]: 8.13 scrub ok 2024-02-05T07:29:36.062 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:36.065 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:36.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:36.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:36.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:36.240 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:36.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:36.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:36.369 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:36.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:36.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:36.498 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:36.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:36.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:36.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:36.628 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:36.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:36.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:36.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:36.757 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:36.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:36.859 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:36.885 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:36.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:36.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:36.886 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:29:36.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:36 smithi007 ceph-mon[29098]: pgmap v561: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 12 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 779 B/s rd, 0 B/s wr, 514 op/s; 1.7 MiB/s, 28 objects/s recovering 2024-02-05T07:29:36.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:36 smithi007 ceph-mon[29098]: 8.17 scrub starts 2024-02-05T07:29:36.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:36 smithi007 ceph-mon[29098]: 8.17 scrub ok 2024-02-05T07:29:36.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:36 smithi007 ceph-mon[29098]: 8.7 scrub starts 2024-02-05T07:29:36.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:36 smithi007 ceph-mon[29098]: 8.7 scrub ok 2024-02-05T07:29:36.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:36 smithi007 ceph-mon[29098]: 8.16 scrub starts 2024-02-05T07:29:36.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:36 smithi007 ceph-mon[29098]: 8.16 scrub ok 2024-02-05T07:29:36.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:36 smithi007 ceph-mon[37158]: pgmap v561: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 12 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 779 B/s rd, 0 B/s wr, 514 op/s; 1.7 MiB/s, 28 objects/s recovering 2024-02-05T07:29:36.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:36 smithi007 ceph-mon[37158]: 8.17 scrub starts 2024-02-05T07:29:36.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:36 smithi007 ceph-mon[37158]: 8.17 scrub ok 2024-02-05T07:29:36.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:36 smithi007 ceph-mon[37158]: 8.7 scrub starts 2024-02-05T07:29:36.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:36 smithi007 ceph-mon[37158]: 8.7 scrub ok 2024-02-05T07:29:36.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:36 smithi007 ceph-mon[37158]: 8.16 scrub starts 2024-02-05T07:29:36.888 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:36 smithi007 ceph-mon[37158]: 8.16 scrub ok 2024-02-05T07:29:36.906 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:29:36.925 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:29:36.949 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:29:36.973 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:29:36.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:36.993 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:29:37.010 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:37.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:37.015 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:29:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:36 smithi090 ceph-mon[33213]: pgmap v561: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 12 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 779 B/s rd, 0 B/s wr, 514 op/s; 1.7 MiB/s, 28 objects/s recovering 2024-02-05T07:29:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:36 smithi090 ceph-mon[33213]: 8.17 scrub starts 2024-02-05T07:29:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:36 smithi090 ceph-mon[33213]: 8.17 scrub ok 2024-02-05T07:29:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:36 smithi090 ceph-mon[33213]: 8.7 scrub starts 2024-02-05T07:29:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:36 smithi090 ceph-mon[33213]: 8.7 scrub ok 2024-02-05T07:29:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:36 smithi090 ceph-mon[33213]: 8.16 scrub starts 2024-02-05T07:29:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:36 smithi090 ceph-mon[33213]: 8.16 scrub ok 2024-02-05T07:29:37.035 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:29:37.036 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:29:37.037 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:29:37.037 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:29:37.037 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:29:37.037 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:29:37.037 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:29:37.037 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:29:37.037 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:29:37.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:37.136 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:37.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:37.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:37.266 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:37.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:37.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:37.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:37.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:37.391 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:37.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:37.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:37.520 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:37.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:37.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:37.629 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:37 smithi007 ceph-mon[37158]: 4.6 scrub starts 2024-02-05T07:29:37.629 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:37 smithi007 ceph-mon[37158]: 4.6 scrub ok 2024-02-05T07:29:37.629 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:37 smithi007 ceph-mon[37158]: 3.14 scrub starts 2024-02-05T07:29:37.629 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:37 smithi007 ceph-mon[37158]: 3.14 scrub ok 2024-02-05T07:29:37.629 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:37 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2028792558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:29:37.648 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:37.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:37.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:37.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:37.875 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:37.899 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:37.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:37 smithi007 ceph-mon[29098]: 4.6 scrub starts 2024-02-05T07:29:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:37 smithi007 ceph-mon[29098]: 4.6 scrub ok 2024-02-05T07:29:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:37 smithi007 ceph-mon[29098]: 3.14 scrub starts 2024-02-05T07:29:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:37 smithi007 ceph-mon[29098]: 3.14 scrub ok 2024-02-05T07:29:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:37 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2028792558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:29:38.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:37 smithi090 ceph-mon[33213]: 4.6 scrub starts 2024-02-05T07:29:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:37 smithi090 ceph-mon[33213]: 4.6 scrub ok 2024-02-05T07:29:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:37 smithi090 ceph-mon[33213]: 3.14 scrub starts 2024-02-05T07:29:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:37 smithi090 ceph-mon[33213]: 3.14 scrub ok 2024-02-05T07:29:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:37 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2028792558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:29:38.027 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:38.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:38.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:38.155 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:38.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:38.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:38.282 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:38.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:38.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:38.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:38.407 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:38.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:38.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:38.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:38.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:38.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:38.707 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:38.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:38.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:38.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:38 smithi090 ceph-mon[33213]: pgmap v562: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 6.0 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:29:38.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:38 smithi090 ceph-mon[33213]: 3.18 scrub starts 2024-02-05T07:29:38.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:38 smithi090 ceph-mon[33213]: 3.18 scrub ok 2024-02-05T07:29:38.810 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:38.834 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:38.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:38.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:38 smithi007 ceph-mon[29098]: pgmap v562: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 6.0 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:29:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:38 smithi007 ceph-mon[29098]: 3.18 scrub starts 2024-02-05T07:29:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:38 smithi007 ceph-mon[29098]: 3.18 scrub ok 2024-02-05T07:29:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:38 smithi007 ceph-mon[37158]: pgmap v562: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 6.0 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:29:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:38 smithi007 ceph-mon[37158]: 3.18 scrub starts 2024-02-05T07:29:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:38 smithi007 ceph-mon[37158]: 3.18 scrub ok 2024-02-05T07:29:38.935 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:38.959 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:38.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:38.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:39.061 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:39.086 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:39.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:39.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:39.212 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:39.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:39.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:39.314 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:39.339 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:39.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:39.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:39.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:39.465 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:39.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:39.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:39.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:39.591 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:39.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:39.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:39.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:39.717 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:39.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:39.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:39.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:39.843 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:39.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:39.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:39 smithi007 ceph-mon[29098]: 6.5 scrub starts 2024-02-05T07:29:39.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:39 smithi007 ceph-mon[29098]: 6.5 scrub ok 2024-02-05T07:29:39.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:39 smithi007 ceph-mon[29098]: 8.e scrub starts 2024-02-05T07:29:39.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:39 smithi007 ceph-mon[29098]: 8.e scrub ok 2024-02-05T07:29:39.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:39 smithi007 ceph-mon[37158]: 6.5 scrub starts 2024-02-05T07:29:39.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:39 smithi007 ceph-mon[37158]: 6.5 scrub ok 2024-02-05T07:29:39.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:39 smithi007 ceph-mon[37158]: 8.e scrub starts 2024-02-05T07:29:39.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:39 smithi007 ceph-mon[37158]: 8.e scrub ok 2024-02-05T07:29:39.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:39.970 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:39.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:39.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:39 smithi090 ceph-mon[33213]: 6.5 scrub starts 2024-02-05T07:29:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:39 smithi090 ceph-mon[33213]: 6.5 scrub ok 2024-02-05T07:29:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:39 smithi090 ceph-mon[33213]: 8.e scrub starts 2024-02-05T07:29:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:39 smithi090 ceph-mon[33213]: 8.e scrub ok 2024-02-05T07:29:40.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:40.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:40.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:40.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:40.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:40.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:40.224 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:40.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:40.325 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:40.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:40.352 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:40.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:40.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:40.482 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:40.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:40.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:40.488 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:40.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:40.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:40.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:40.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:40.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:40.734 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:40.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:40.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:40.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:40.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:40.865 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:40.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[29098]: 3.d scrub starts 2024-02-05T07:29:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[29098]: 3.d scrub ok 2024-02-05T07:29:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[29098]: pgmap v563: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 5.0 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:29:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[29098]: 5.8 scrub starts 2024-02-05T07:29:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[29098]: 5.8 scrub ok 2024-02-05T07:29:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:29:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:29:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[29098]: 5.3 scrub starts 2024-02-05T07:29:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[29098]: 5.3 scrub ok 2024-02-05T07:29:40.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[29098]: osdmap e341: 8 total, 8 up, 7 in 2024-02-05T07:29:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[37158]: 3.d scrub starts 2024-02-05T07:29:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[37158]: 3.d scrub ok 2024-02-05T07:29:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[37158]: pgmap v563: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 5.0 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:29:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[37158]: 5.8 scrub starts 2024-02-05T07:29:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[37158]: 5.8 scrub ok 2024-02-05T07:29:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:29:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:29:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[37158]: 5.3 scrub starts 2024-02-05T07:29:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[37158]: 5.3 scrub ok 2024-02-05T07:29:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:40 smithi007 ceph-mon[37158]: osdmap e341: 8 total, 8 up, 7 in 2024-02-05T07:29:40.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:40.994 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:40.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:40.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:41.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:29:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:29:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:40 smithi090 ceph-mon[33213]: 3.d scrub starts 2024-02-05T07:29:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:40 smithi090 ceph-mon[33213]: 3.d scrub ok 2024-02-05T07:29:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:40 smithi090 ceph-mon[33213]: pgmap v563: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 5.0 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:29:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:40 smithi090 ceph-mon[33213]: 5.8 scrub starts 2024-02-05T07:29:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:40 smithi090 ceph-mon[33213]: 5.8 scrub ok 2024-02-05T07:29:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:40 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:29:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:40 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:29:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:40 smithi090 ceph-mon[33213]: 5.3 scrub starts 2024-02-05T07:29:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:40 smithi090 ceph-mon[33213]: 5.3 scrub ok 2024-02-05T07:29:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:40 smithi090 ceph-mon[33213]: osdmap e341: 8 total, 8 up, 7 in 2024-02-05T07:29:41.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:41.119 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:41.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:41.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:41.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:41.247 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:41.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:41.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:41.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:41.373 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:41.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:41.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:41.503 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:41.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:41.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:41.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:41.629 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:41.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:41.649 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.4 to 1 (10000) 2024-02-05T07:29:41.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:41.757 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:41.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:41.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:29:41.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:41.888 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:41.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:41.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:41 smithi007 ceph-mon[29098]: 4.1e scrub starts 2024-02-05T07:29:41.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:41 smithi007 ceph-mon[29098]: 4.1e scrub ok 2024-02-05T07:29:41.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:41 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:41.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:41 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3223291771' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:29:41.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:41 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:29:41.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:41 smithi007 ceph-mon[29098]: 6.13 scrub starts 2024-02-05T07:29:41.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:41 smithi007 ceph-mon[29098]: 6.13 scrub ok 2024-02-05T07:29:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:41 smithi007 ceph-mon[37158]: 4.1e scrub starts 2024-02-05T07:29:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:41 smithi007 ceph-mon[37158]: 4.1e scrub ok 2024-02-05T07:29:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:41 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:41 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3223291771' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:29:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:41 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:29:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:41 smithi007 ceph-mon[37158]: 6.13 scrub starts 2024-02-05T07:29:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:41 smithi007 ceph-mon[37158]: 6.13 scrub ok 2024-02-05T07:29:41.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:42.015 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:42.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:41 smithi090 ceph-mon[33213]: 4.1e scrub starts 2024-02-05T07:29:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:41 smithi090 ceph-mon[33213]: 4.1e scrub ok 2024-02-05T07:29:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:41 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:41 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3223291771' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:29:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:41 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:29:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:41 smithi090 ceph-mon[33213]: 6.13 scrub starts 2024-02-05T07:29:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:41 smithi090 ceph-mon[33213]: 6.13 scrub ok 2024-02-05T07:29:42.116 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:42.143 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:42.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:42.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:42.271 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:42.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:42.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:42.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:42.397 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:42.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:42.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:42.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:42.522 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:42.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:42.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:42.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:42.652 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:29:42.654 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:42.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:42.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:42.782 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:42.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:42.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:42.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[29098]: pgmap v565: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:29:42.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[29098]: 4.1c scrub starts 2024-02-05T07:29:42.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[29098]: 4.1c scrub ok 2024-02-05T07:29:42.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[29098]: 3.6 scrub starts 2024-02-05T07:29:42.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[29098]: 3.6 scrub ok 2024-02-05T07:29:42.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T07:29:42.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[29098]: osdmap e342: 8 total, 8 up, 7 in 2024-02-05T07:29:42.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3617531423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:29:42.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:29:42.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[37158]: pgmap v565: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:29:42.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[37158]: 4.1c scrub starts 2024-02-05T07:29:42.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[37158]: 4.1c scrub ok 2024-02-05T07:29:42.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[37158]: 3.6 scrub starts 2024-02-05T07:29:42.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[37158]: 3.6 scrub ok 2024-02-05T07:29:42.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T07:29:42.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[37158]: osdmap e342: 8 total, 8 up, 7 in 2024-02-05T07:29:42.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3617531423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:29:42.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:42 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:29:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:42.914 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:42.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:43.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:42 smithi090 ceph-mon[33213]: pgmap v565: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:29:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:42 smithi090 ceph-mon[33213]: 4.1c scrub starts 2024-02-05T07:29:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:42 smithi090 ceph-mon[33213]: 4.1c scrub ok 2024-02-05T07:29:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:42 smithi090 ceph-mon[33213]: 3.6 scrub starts 2024-02-05T07:29:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:42 smithi090 ceph-mon[33213]: 3.6 scrub ok 2024-02-05T07:29:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:42 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T07:29:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:42 smithi090 ceph-mon[33213]: osdmap e342: 8 total, 8 up, 7 in 2024-02-05T07:29:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:42 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3617531423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:29:43.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:42 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:29:43.043 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:43.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:43.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:43.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:43.170 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:43.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:43.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:43.272 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:43.299 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:43.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:43.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:43.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:43.428 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:43.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:43.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:43.557 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:43.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:43.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 6 1 2024-02-05T07:29:43.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:43.682 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:43.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:43.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:43.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:43.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:43.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:43.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:43.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:29:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:29:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-mon[29098]: 2.1 scrub starts 2024-02-05T07:29:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-mon[29098]: 2.1 scrub ok 2024-02-05T07:29:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:29:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:29:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-mon[29098]: osdmap e343: 8 total, 8 up, 7 in 2024-02-05T07:29:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-mon[29098]: 3.2 scrub starts 2024-02-05T07:29:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-mon[29098]: 3.2 scrub ok 2024-02-05T07:29:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-mon[37158]: 2.1 scrub starts 2024-02-05T07:29:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-mon[37158]: 2.1 scrub ok 2024-02-05T07:29:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:29:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:29:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-mon[37158]: osdmap e343: 8 total, 8 up, 7 in 2024-02-05T07:29:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-mon[37158]: 3.2 scrub starts 2024-02-05T07:29:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:43 smithi007 ceph-mon[37158]: 3.2 scrub ok 2024-02-05T07:29:43.934 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:43.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:43.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:43 smithi090 ceph-mon[33213]: 2.1 scrub starts 2024-02-05T07:29:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:43 smithi090 ceph-mon[33213]: 2.1 scrub ok 2024-02-05T07:29:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:43 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:29:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:43 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:29:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:43 smithi090 ceph-mon[33213]: osdmap e343: 8 total, 8 up, 7 in 2024-02-05T07:29:44.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:43 smithi090 ceph-mon[33213]: 3.2 scrub starts 2024-02-05T07:29:44.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:43 smithi090 ceph-mon[33213]: 3.2 scrub ok 2024-02-05T07:29:44.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:44.064 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:44.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:44.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:44.129 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:44.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:44.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:44.194 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:44.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:44.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:44.324 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:44.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:44.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:44.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:44.453 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:44.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:44.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:44.553 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:44.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:44.580 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:44.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:44.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:44.601 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:44.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:44.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:44.712 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:44.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:44.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:44.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:44 smithi007 ceph-mon[29098]: pgmap v568: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:29:44.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:44 smithi007 ceph-mon[29098]: 3.5 scrub starts 2024-02-05T07:29:44.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:44 smithi007 ceph-mon[29098]: 3.5 scrub ok 2024-02-05T07:29:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:44 smithi007 ceph-mon[29098]: osdmap e344: 8 total, 8 up, 7 in 2024-02-05T07:29:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:44 smithi007 ceph-mon[37158]: pgmap v568: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:29:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:44 smithi007 ceph-mon[37158]: 3.5 scrub starts 2024-02-05T07:29:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:44 smithi007 ceph-mon[37158]: 3.5 scrub ok 2024-02-05T07:29:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:44 smithi007 ceph-mon[37158]: osdmap e344: 8 total, 8 up, 7 in 2024-02-05T07:29:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:44.840 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:44.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:44.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:44.965 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:44.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:44.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:44 smithi090 ceph-mon[33213]: pgmap v568: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:29:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:44 smithi090 ceph-mon[33213]: 3.5 scrub starts 2024-02-05T07:29:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:44 smithi090 ceph-mon[33213]: 3.5 scrub ok 2024-02-05T07:29:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:44 smithi090 ceph-mon[33213]: osdmap e344: 8 total, 8 up, 7 in 2024-02-05T07:29:45.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:45.094 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:45.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:45.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:45.224 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:45.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:45.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:45.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:45.353 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:45.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:45.454 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:45.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:45.479 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:45.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:45.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:45.609 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:45.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:45.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:45.735 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:45.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:45.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:45.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:45.865 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:45.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:45.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:45.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:45.996 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:45.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:46.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:29:46.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:46.124 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:46.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:46.130 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:29:46.130 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:29:46.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:46.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:46.254 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:46.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:46.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:46.381 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:46.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:46.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[29098]: 5.f deep-scrub starts 2024-02-05T07:29:46.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[29098]: 5.f deep-scrub ok 2024-02-05T07:29:46.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:29:46.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:29:46.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[29098]: pgmap v570: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:29:46.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[29098]: 6.3 scrub starts 2024-02-05T07:29:46.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:29:46.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:29:46.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[29098]: 6.3 scrub ok 2024-02-05T07:29:46.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[29098]: 3.0 scrub starts 2024-02-05T07:29:46.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[29098]: 3.0 scrub ok 2024-02-05T07:29:46.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[29098]: osdmap e345: 8 total, 8 up, 7 in 2024-02-05T07:29:46.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[37158]: 5.f deep-scrub starts 2024-02-05T07:29:46.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[37158]: 5.f deep-scrub ok 2024-02-05T07:29:46.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:29:46.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:29:46.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[37158]: pgmap v570: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:29:46.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[37158]: 6.3 scrub starts 2024-02-05T07:29:46.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:29:46.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:29:46.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[37158]: 6.3 scrub ok 2024-02-05T07:29:46.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[37158]: 3.0 scrub starts 2024-02-05T07:29:46.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[37158]: 3.0 scrub ok 2024-02-05T07:29:46.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:46 smithi007 ceph-mon[37158]: osdmap e345: 8 total, 8 up, 7 in 2024-02-05T07:29:46.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:46.505 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:46.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:46.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:46.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:46 smithi090 ceph-mon[33213]: 5.f deep-scrub starts 2024-02-05T07:29:46.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:46 smithi090 ceph-mon[33213]: 5.f deep-scrub ok 2024-02-05T07:29:46.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:46 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:29:46.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:46 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:29:46.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:46 smithi090 ceph-mon[33213]: pgmap v570: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:29:46.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:46 smithi090 ceph-mon[33213]: 6.3 scrub starts 2024-02-05T07:29:46.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:46 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:29:46.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:29:46.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:46 smithi090 ceph-mon[33213]: 6.3 scrub ok 2024-02-05T07:29:46.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:46 smithi090 ceph-mon[33213]: 3.0 scrub starts 2024-02-05T07:29:46.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:46 smithi090 ceph-mon[33213]: 3.0 scrub ok 2024-02-05T07:29:46.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:46 smithi090 ceph-mon[33213]: osdmap e345: 8 total, 8 up, 7 in 2024-02-05T07:29:46.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:46.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:46.632 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:46.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:46.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:46.761 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:46.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:46.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:46.890 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:46.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:46.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:47.019 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:47.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:47.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:47.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:47.149 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:47.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:47.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:47.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:47.274 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:47.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:47.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:47.402 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:47.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:47.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:47.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:47 smithi007 ceph-mon[29098]: 4.1 scrub starts 2024-02-05T07:29:47.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:47 smithi007 ceph-mon[29098]: 4.1 scrub ok 2024-02-05T07:29:47.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:47 smithi007 ceph-mon[29098]: 4.a scrub starts 2024-02-05T07:29:47.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:47 smithi007 ceph-mon[29098]: 4.a scrub ok 2024-02-05T07:29:47.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:47 smithi007 ceph-mon[29098]: 3.13 scrub starts 2024-02-05T07:29:47.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:47 smithi007 ceph-mon[29098]: 3.13 scrub ok 2024-02-05T07:29:47.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:47 smithi007 ceph-mon[37158]: 4.1 scrub starts 2024-02-05T07:29:47.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:47 smithi007 ceph-mon[37158]: 4.1 scrub ok 2024-02-05T07:29:47.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:47 smithi007 ceph-mon[37158]: 4.a scrub starts 2024-02-05T07:29:47.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:47 smithi007 ceph-mon[37158]: 4.a scrub ok 2024-02-05T07:29:47.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:47 smithi007 ceph-mon[37158]: 3.13 scrub starts 2024-02-05T07:29:47.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:47 smithi007 ceph-mon[37158]: 3.13 scrub ok 2024-02-05T07:29:47.505 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:47.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:47 smithi090 ceph-mon[33213]: 4.1 scrub starts 2024-02-05T07:29:47.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:47 smithi090 ceph-mon[33213]: 4.1 scrub ok 2024-02-05T07:29:47.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:47 smithi090 ceph-mon[33213]: 4.a scrub starts 2024-02-05T07:29:47.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:47 smithi090 ceph-mon[33213]: 4.a scrub ok 2024-02-05T07:29:47.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:47 smithi090 ceph-mon[33213]: 3.13 scrub starts 2024-02-05T07:29:47.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:47 smithi090 ceph-mon[33213]: 3.13 scrub ok 2024-02-05T07:29:47.528 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:47.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:47.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:47.630 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:47.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:47.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:47.788 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:47.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:47.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:47.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:47.918 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:47.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:48.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:48.048 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:48.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:48.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:48.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:48.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:48.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:48.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:48.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:48.429 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:48.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:48.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:48.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:48 smithi090 ceph-mon[33213]: 4.1f scrub starts 2024-02-05T07:29:48.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:48 smithi090 ceph-mon[33213]: 4.1f scrub ok 2024-02-05T07:29:48.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:48 smithi090 ceph-mon[33213]: pgmap v572: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 924 B/s rd, 0 op/s 2024-02-05T07:29:48.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:48 smithi007 ceph-mon[29098]: 4.1f scrub starts 2024-02-05T07:29:48.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:48 smithi007 ceph-mon[29098]: 4.1f scrub ok 2024-02-05T07:29:48.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:48 smithi007 ceph-mon[29098]: pgmap v572: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 924 B/s rd, 0 op/s 2024-02-05T07:29:48.526 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:48 smithi007 ceph-mon[37158]: 4.1f scrub starts 2024-02-05T07:29:48.526 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:48 smithi007 ceph-mon[37158]: 4.1f scrub ok 2024-02-05T07:29:48.527 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:48 smithi007 ceph-mon[37158]: pgmap v572: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 924 B/s rd, 0 op/s 2024-02-05T07:29:48.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:48.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:48.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:48.684 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:48.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:48.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:48.814 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:48.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:48.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:48.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:48.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:48.941 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:48.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:49.042 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:49.066 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:49.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:49.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:49.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T07:29:49.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:49.225 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:49.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:49.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:49.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:49.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:49.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:49.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[29098]: 5.18 scrub starts 2024-02-05T07:29:49.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[29098]: 5.18 scrub ok 2024-02-05T07:29:49.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[29098]: 1.8 scrub starts 2024-02-05T07:29:49.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[29098]: 1.8 scrub ok 2024-02-05T07:29:49.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[29098]: 6.8 deep-scrub starts 2024-02-05T07:29:49.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[29098]: 6.8 deep-scrub ok 2024-02-05T07:29:49.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[29098]: 5.5 scrub starts 2024-02-05T07:29:49.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[29098]: 5.5 scrub ok 2024-02-05T07:29:49.413 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[37158]: 5.18 scrub starts 2024-02-05T07:29:49.413 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[37158]: 5.18 scrub ok 2024-02-05T07:29:49.413 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[37158]: 1.8 scrub starts 2024-02-05T07:29:49.413 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[37158]: 1.8 scrub ok 2024-02-05T07:29:49.413 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[37158]: 6.8 deep-scrub starts 2024-02-05T07:29:49.414 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[37158]: 6.8 deep-scrub ok 2024-02-05T07:29:49.414 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[37158]: 5.5 scrub starts 2024-02-05T07:29:49.414 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:49 smithi007 ceph-mon[37158]: 5.5 scrub ok 2024-02-05T07:29:49.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:49.484 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:49.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:49.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:49.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:49 smithi090 ceph-mon[33213]: 5.18 scrub starts 2024-02-05T07:29:49.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:49 smithi090 ceph-mon[33213]: 5.18 scrub ok 2024-02-05T07:29:49.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:49 smithi090 ceph-mon[33213]: 1.8 scrub starts 2024-02-05T07:29:49.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:49 smithi090 ceph-mon[33213]: 1.8 scrub ok 2024-02-05T07:29:49.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:49 smithi090 ceph-mon[33213]: 6.8 deep-scrub starts 2024-02-05T07:29:49.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:49 smithi090 ceph-mon[33213]: 6.8 deep-scrub ok 2024-02-05T07:29:49.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:49 smithi090 ceph-mon[33213]: 5.5 scrub starts 2024-02-05T07:29:49.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:49 smithi090 ceph-mon[33213]: 5.5 scrub ok 2024-02-05T07:29:49.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:49.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:49.615 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:49.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:49.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:49.741 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:49.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:49.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:49.843 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:49.871 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:49.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:49.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:49.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:50.001 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:50.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:50.103 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:50.129 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:50.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:50.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:50.258 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:50.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:50.325 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_1' removed 2024-02-05T07:29:50.342 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.6 to 1 (10000) 2024-02-05T07:29:50.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:50.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:50.389 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:50.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:50.490 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:50.553 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:50.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:50.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:50.580 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:50 smithi090 ceph-mon[33213]: 8.c scrub starts 2024-02-05T07:29:50.580 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:50 smithi090 ceph-mon[33213]: 8.c scrub ok 2024-02-05T07:29:50.580 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:50 smithi090 ceph-mon[33213]: 4.9 scrub starts 2024-02-05T07:29:50.580 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:50 smithi090 ceph-mon[33213]: 4.9 scrub ok 2024-02-05T07:29:50.580 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:50 smithi090 ceph-mon[33213]: pgmap v573: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 783 B/s rd, 0 op/s 2024-02-05T07:29:50.580 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:50 smithi090 ceph-mon[33213]: osdmap e346: 8 total, 8 up, 7 in 2024-02-05T07:29:50.580 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:50 smithi090 ceph-mon[33213]: 3.12 scrub starts 2024-02-05T07:29:50.580 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:50 smithi090 ceph-mon[33213]: 3.12 scrub ok 2024-02-05T07:29:50.580 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:50 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3486252354' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:29:50.580 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:50 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:29:50.580 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:50 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2182721162' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:29:50.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[37158]: 8.c scrub starts 2024-02-05T07:29:50.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[37158]: 8.c scrub ok 2024-02-05T07:29:50.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[37158]: 4.9 scrub starts 2024-02-05T07:29:50.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[37158]: 4.9 scrub ok 2024-02-05T07:29:50.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[37158]: pgmap v573: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 783 B/s rd, 0 op/s 2024-02-05T07:29:50.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[37158]: osdmap e346: 8 total, 8 up, 7 in 2024-02-05T07:29:50.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[37158]: 3.12 scrub starts 2024-02-05T07:29:50.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[37158]: 3.12 scrub ok 2024-02-05T07:29:50.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3486252354' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:29:50.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:29:50.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2182721162' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:29:50.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[29098]: 8.c scrub starts 2024-02-05T07:29:50.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[29098]: 8.c scrub ok 2024-02-05T07:29:50.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[29098]: 4.9 scrub starts 2024-02-05T07:29:50.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[29098]: 4.9 scrub ok 2024-02-05T07:29:50.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[29098]: pgmap v573: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 783 B/s rd, 0 op/s 2024-02-05T07:29:50.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[29098]: osdmap e346: 8 total, 8 up, 7 in 2024-02-05T07:29:50.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[29098]: 3.12 scrub starts 2024-02-05T07:29:50.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[29098]: 3.12 scrub ok 2024-02-05T07:29:50.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3486252354' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:29:50.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:29:50.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:50 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2182721162' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:29:50.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:50.680 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:50.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:50.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:50.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:50.806 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:50.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:50.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:50.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:50.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:50.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:51.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:29:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:29:51.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:51.056 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:51.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:51.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:51.157 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T07:29:51.157 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T07:29:51.158 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-02-05T07:29:51.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-02-05T07:29:51.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:51.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:51.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:51.310 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:51.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:51.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:51.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:51.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[29098]: 5.c scrub starts 2024-02-05T07:29:51.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[29098]: 5.c scrub ok 2024-02-05T07:29:51.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[29098]: 4.e scrub starts 2024-02-05T07:29:51.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[29098]: 4.e scrub ok 2024-02-05T07:29:51.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[29098]: 3.1f deep-scrub starts 2024-02-05T07:29:51.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[29098]: 3.1f deep-scrub ok 2024-02-05T07:29:51.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T07:29:51.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2182721162' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:29:51.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[29098]: osdmap e347: 8 total, 8 up, 7 in 2024-02-05T07:29:51.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:51.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[37158]: 5.c scrub starts 2024-02-05T07:29:51.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[37158]: 5.c scrub ok 2024-02-05T07:29:51.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[37158]: 4.e scrub starts 2024-02-05T07:29:51.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[37158]: 4.e scrub ok 2024-02-05T07:29:51.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[37158]: 3.1f deep-scrub starts 2024-02-05T07:29:51.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[37158]: 3.1f deep-scrub ok 2024-02-05T07:29:51.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T07:29:51.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2182721162' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:29:51.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[37158]: osdmap e347: 8 total, 8 up, 7 in 2024-02-05T07:29:51.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:51 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:51.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:51.437 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:51.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:51.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:51.562 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:51.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:51.664 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:51.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:51.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:51 smithi090 ceph-mon[33213]: 5.c scrub starts 2024-02-05T07:29:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:51 smithi090 ceph-mon[33213]: 5.c scrub ok 2024-02-05T07:29:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:51 smithi090 ceph-mon[33213]: 4.e scrub starts 2024-02-05T07:29:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:51 smithi090 ceph-mon[33213]: 4.e scrub ok 2024-02-05T07:29:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:51 smithi090 ceph-mon[33213]: 3.1f deep-scrub starts 2024-02-05T07:29:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:51 smithi090 ceph-mon[33213]: 3.1f deep-scrub ok 2024-02-05T07:29:51.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:51 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T07:29:51.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:51 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2182721162' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:29:51.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:51 smithi090 ceph-mon[33213]: osdmap e347: 8 total, 8 up, 7 in 2024-02-05T07:29:51.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:51 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:29:51.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:51.813 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:51.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:51.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:51.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:51.940 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:51.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:52.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:52.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:52.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:52.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 7 1 2024-02-05T07:29:52.193 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:52.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:52.240 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:29:52.240 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:29:52.240 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:29:52.241 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:29:52.241 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:29:52.241 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:29:52.241 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:29:52.241 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:29:52.241 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:29:52.241 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:29:52.241 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:29:52.241 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:29:52.241 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:29:52.242 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:29:52.242 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:29:52.242 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:29:52.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:52.320 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:52.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:52.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:52.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:52.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[37158]: 6.19 scrub starts 2024-02-05T07:29:52.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[37158]: 6.19 scrub ok 2024-02-05T07:29:52.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[37158]: pgmap v576: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T07:29:52.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[37158]: 2.0 scrub starts 2024-02-05T07:29:52.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[37158]: 2.0 scrub ok 2024-02-05T07:29:52.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[37158]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-02-05T07:29:52.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:29:52.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/136527675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:29:52.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[29098]: 6.19 scrub starts 2024-02-05T07:29:52.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[29098]: 6.19 scrub ok 2024-02-05T07:29:52.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[29098]: pgmap v576: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T07:29:52.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[29098]: 2.0 scrub starts 2024-02-05T07:29:52.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[29098]: 2.0 scrub ok 2024-02-05T07:29:52.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[29098]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-02-05T07:29:52.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:29:52.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:52 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/136527675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:29:52.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:52.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:52.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:52.575 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:52.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:52.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:52.704 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:52.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:52.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:52.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:52 smithi090 ceph-mon[33213]: 6.19 scrub starts 2024-02-05T07:29:52.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:52 smithi090 ceph-mon[33213]: 6.19 scrub ok 2024-02-05T07:29:52.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:52 smithi090 ceph-mon[33213]: pgmap v576: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T07:29:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:52 smithi090 ceph-mon[33213]: 2.0 scrub starts 2024-02-05T07:29:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:52 smithi090 ceph-mon[33213]: 2.0 scrub ok 2024-02-05T07:29:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:52 smithi090 ceph-mon[33213]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-02-05T07:29:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:52 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:29:52.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:52 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/136527675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:29:52.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:52.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:52.831 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:52.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:52.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:52.955 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:52.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:52.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:53.057 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:53.079 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:53.080 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:53.085 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:53.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:53.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:53.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:53.211 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:53.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:53.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:53.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:53.342 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:53.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:53.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:53.471 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:53.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:53.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:53.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:53 smithi007 ceph-mon[29098]: 4.b scrub starts 2024-02-05T07:29:53.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:53 smithi007 ceph-mon[29098]: 4.b scrub ok 2024-02-05T07:29:53.536 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:53 smithi007 ceph-mon[37158]: 4.b scrub starts 2024-02-05T07:29:53.536 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:53 smithi007 ceph-mon[37158]: 4.b scrub ok 2024-02-05T07:29:53.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:53.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:53.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:53.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:53.727 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:53.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:53.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:53.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:53 smithi090 ceph-mon[33213]: 4.b scrub starts 2024-02-05T07:29:53.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:53 smithi090 ceph-mon[33213]: 4.b scrub ok 2024-02-05T07:29:53.806 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:29:53.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:53.856 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:53.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:53.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:53.897 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:29:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:29:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:29:53.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:53.984 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:54.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:54.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:54.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:54.155 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:54.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:54.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:54.282 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:54.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:54.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:54.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:54.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:54.409 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:54.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:54.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[29098]: 3.19 scrub starts 2024-02-05T07:29:54.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[29098]: 3.19 scrub ok 2024-02-05T07:29:54.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[29098]: 3.1d scrub starts 2024-02-05T07:29:54.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[29098]: 3.1d scrub ok 2024-02-05T07:29:54.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[29098]: pgmap v577: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 544 B/s rd, 0 op/s 2024-02-05T07:29:54.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[29098]: 3.15 scrub starts 2024-02-05T07:29:54.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[29098]: 3.15 scrub ok 2024-02-05T07:29:54.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[29098]: osdmap e348: 8 total, 8 up, 7 in 2024-02-05T07:29:54.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[37158]: 3.19 scrub starts 2024-02-05T07:29:54.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[37158]: 3.19 scrub ok 2024-02-05T07:29:54.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[37158]: 3.1d scrub starts 2024-02-05T07:29:54.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[37158]: 3.1d scrub ok 2024-02-05T07:29:54.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[37158]: pgmap v577: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 544 B/s rd, 0 op/s 2024-02-05T07:29:54.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[37158]: 3.15 scrub starts 2024-02-05T07:29:54.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[37158]: 3.15 scrub ok 2024-02-05T07:29:54.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:54 smithi007 ceph-mon[37158]: osdmap e348: 8 total, 8 up, 7 in 2024-02-05T07:29:54.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:54.536 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:54.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:54.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:54.660 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:54.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:54.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:54.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:54 smithi090 ceph-mon[33213]: 3.19 scrub starts 2024-02-05T07:29:54.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:54 smithi090 ceph-mon[33213]: 3.19 scrub ok 2024-02-05T07:29:54.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:54 smithi090 ceph-mon[33213]: 3.1d scrub starts 2024-02-05T07:29:54.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:54 smithi090 ceph-mon[33213]: 3.1d scrub ok 2024-02-05T07:29:54.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:54 smithi090 ceph-mon[33213]: pgmap v577: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 544 B/s rd, 0 op/s 2024-02-05T07:29:54.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:54 smithi090 ceph-mon[33213]: 3.15 scrub starts 2024-02-05T07:29:54.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:54 smithi090 ceph-mon[33213]: 3.15 scrub ok 2024-02-05T07:29:54.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:54 smithi090 ceph-mon[33213]: osdmap e348: 8 total, 8 up, 7 in 2024-02-05T07:29:54.790 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:54.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:54.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:54.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:54.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:54.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:55.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:29:55.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:55.079 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:55.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:55.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:55.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:55.209 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:55.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:55.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:55.311 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:55.338 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:55.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:55.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:55.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:55.468 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:55.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:55.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:55.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:55.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:55.595 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:55.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:55.657 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:55 smithi007 ceph-mon[37158]: 3.4 scrub starts 2024-02-05T07:29:55.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:55 smithi007 ceph-mon[37158]: 3.4 scrub ok 2024-02-05T07:29:55.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:55 smithi007 ceph-mon[37158]: 4.10 scrub starts 2024-02-05T07:29:55.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:55 smithi007 ceph-mon[37158]: 4.10 scrub ok 2024-02-05T07:29:55.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:55 smithi007 ceph-mon[37158]: 5.1b scrub starts 2024-02-05T07:29:55.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:55 smithi007 ceph-mon[37158]: 5.1b scrub ok 2024-02-05T07:29:55.658 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:55 smithi007 ceph-mon[29098]: 3.4 scrub starts 2024-02-05T07:29:55.658 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:55 smithi007 ceph-mon[29098]: 3.4 scrub ok 2024-02-05T07:29:55.658 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:55 smithi007 ceph-mon[29098]: 4.10 scrub starts 2024-02-05T07:29:55.658 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:55 smithi007 ceph-mon[29098]: 4.10 scrub ok 2024-02-05T07:29:55.658 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:55 smithi007 ceph-mon[29098]: 5.1b scrub starts 2024-02-05T07:29:55.659 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:55 smithi007 ceph-mon[29098]: 5.1b scrub ok 2024-02-05T07:29:55.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:55.720 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:55.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:55.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:55 smithi090 ceph-mon[33213]: 3.4 scrub starts 2024-02-05T07:29:55.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:55 smithi090 ceph-mon[33213]: 3.4 scrub ok 2024-02-05T07:29:55.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:55 smithi090 ceph-mon[33213]: 4.10 scrub starts 2024-02-05T07:29:55.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:55 smithi090 ceph-mon[33213]: 4.10 scrub ok 2024-02-05T07:29:55.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:55 smithi090 ceph-mon[33213]: 5.1b scrub starts 2024-02-05T07:29:55.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:55 smithi090 ceph-mon[33213]: 5.1b scrub ok 2024-02-05T07:29:55.822 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:55.846 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:55.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:55.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:55.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:55.976 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:55.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:56.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:56.109 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:56.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:56.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:56.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:56.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:56.236 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:56.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:56.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:56.360 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:56.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:56.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:56.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:56.490 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:56.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:56.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:56 smithi007 ceph-mon[29098]: 5.1f scrub starts 2024-02-05T07:29:56.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:56 smithi007 ceph-mon[29098]: 5.1f scrub ok 2024-02-05T07:29:56.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:56 smithi007 ceph-mon[29098]: pgmap v579: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-02-05T07:29:56.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:56 smithi007 ceph-mon[29098]: 1.1 scrub starts 2024-02-05T07:29:56.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:56 smithi007 ceph-mon[29098]: 1.1 scrub ok 2024-02-05T07:29:56.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:56 smithi007 ceph-mon[37158]: 5.1f scrub starts 2024-02-05T07:29:56.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:56 smithi007 ceph-mon[37158]: 5.1f scrub ok 2024-02-05T07:29:56.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:56 smithi007 ceph-mon[37158]: pgmap v579: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-02-05T07:29:56.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:56 smithi007 ceph-mon[37158]: 1.1 scrub starts 2024-02-05T07:29:56.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:56 smithi007 ceph-mon[37158]: 1.1 scrub ok 2024-02-05T07:29:56.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:56.618 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:56.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:56.675 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:56 smithi090 ceph-mon[33213]: 5.1f scrub starts 2024-02-05T07:29:56.675 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:56 smithi090 ceph-mon[33213]: 5.1f scrub ok 2024-02-05T07:29:56.675 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:56 smithi090 ceph-mon[33213]: pgmap v579: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-02-05T07:29:56.675 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:56 smithi090 ceph-mon[33213]: 1.1 scrub starts 2024-02-05T07:29:56.675 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:56 smithi090 ceph-mon[33213]: 1.1 scrub ok 2024-02-05T07:29:56.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:56.747 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:56.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:56.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:56.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:56.876 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:56.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:56.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:56.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:57.005 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:57.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:57.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:29:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:57.131 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:57.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:29:57.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:57.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:57.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:57.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:57.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:57.386 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:57.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:57.487 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:57.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:57.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:57.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:57.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:57.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:57.659 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:57 smithi007 ceph-mon[29098]: 6.1d scrub starts 2024-02-05T07:29:57.659 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:57 smithi007 ceph-mon[29098]: 6.1d scrub ok 2024-02-05T07:29:57.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:57 smithi007 ceph-mon[37158]: 6.1d scrub starts 2024-02-05T07:29:57.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:57 smithi007 ceph-mon[37158]: 6.1d scrub ok 2024-02-05T07:29:57.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:57.770 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:57.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:57.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:57.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:57 smithi090 ceph-mon[33213]: 6.1d scrub starts 2024-02-05T07:29:57.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:57 smithi090 ceph-mon[33213]: 6.1d scrub ok 2024-02-05T07:29:57.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:57.897 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:57.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:57.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:58.022 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:58.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:58.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:58.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:58.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:58.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:58.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:58.273 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:58.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:58.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:58.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:29:58.399 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:58.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:58.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:29:58.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:58 smithi007 ceph-mon[29098]: pgmap v580: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:29:58.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:58 smithi007 ceph-mon[37158]: pgmap v580: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:29:58.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:29:58.530 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:58.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:29:58.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:58.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:58.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:58.757 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:58.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:58 smithi090 ceph-mon[33213]: pgmap v580: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:29:58.785 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:58.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:58.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:58.887 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:29:58.914 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:58.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:58.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:29:59.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:59.041 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:59.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:59.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:59.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:59.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:59.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:59.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:59.307 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:59.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:59.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:59.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:59.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:59.434 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:59.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:29:59.507 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_2' created 2024-02-05T07:29:59.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:29:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:59.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:59.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:29:59.663 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:29:59.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:59.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:59.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:29:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:59 smithi090 ceph-mon[33213]: 4.12 scrub starts 2024-02-05T07:29:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:59 smithi090 ceph-mon[33213]: 4.12 scrub ok 2024-02-05T07:29:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:59 smithi090 ceph-mon[33213]: 5.11 scrub starts 2024-02-05T07:29:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:59 smithi090 ceph-mon[33213]: 5.11 scrub ok 2024-02-05T07:29:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:59 smithi090 ceph-mon[33213]: 5.10 deep-scrub starts 2024-02-05T07:29:59.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:29:59 smithi090 ceph-mon[33213]: 5.10 deep-scrub ok 2024-02-05T07:29:59.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:29:59.817 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:29:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:59.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:29:59.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:59 smithi007 ceph-mon[29098]: 4.12 scrub starts 2024-02-05T07:29:59.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:59 smithi007 ceph-mon[29098]: 4.12 scrub ok 2024-02-05T07:29:59.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:59 smithi007 ceph-mon[29098]: 5.11 scrub starts 2024-02-05T07:29:59.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:59 smithi007 ceph-mon[29098]: 5.11 scrub ok 2024-02-05T07:29:59.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:59 smithi007 ceph-mon[29098]: 5.10 deep-scrub starts 2024-02-05T07:29:59.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:29:59 smithi007 ceph-mon[29098]: 5.10 deep-scrub ok 2024-02-05T07:29:59.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:59 smithi007 ceph-mon[37158]: 4.12 scrub starts 2024-02-05T07:29:59.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:59 smithi007 ceph-mon[37158]: 4.12 scrub ok 2024-02-05T07:29:59.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:59 smithi007 ceph-mon[37158]: 5.11 scrub starts 2024-02-05T07:29:59.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:59 smithi007 ceph-mon[37158]: 5.11 scrub ok 2024-02-05T07:29:59.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:59 smithi007 ceph-mon[37158]: 5.10 deep-scrub starts 2024-02-05T07:29:59.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:29:59 smithi007 ceph-mon[37158]: 5.10 deep-scrub ok 2024-02-05T07:29:59.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:29:59.943 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:29:59.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:29:59.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:00.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:00.068 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:00.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:00.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:00.199 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:00.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:00.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:00.328 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:00.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:00.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:00.454 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:00.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:00.508 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:30:00.540 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:30:00.540 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:30:00.540 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-02-05T07:30:00.540 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-02-05T07:30:00.540 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:30:00.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:00.584 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:00.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:00.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:00.713 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:00.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:00.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:00.731 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:00.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:30:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:30:00.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:00 smithi090 ceph-mon[33213]: 6.1e scrub starts 2024-02-05T07:30:00.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:00 smithi090 ceph-mon[33213]: 6.1e scrub ok 2024-02-05T07:30:00.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:00 smithi090 ceph-mon[33213]: pgmap v581: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:30:00.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:00 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/858313719' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-05T07:30:00.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:00 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-05T07:30:00.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:00 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-05T07:30:00.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:00 smithi090 ceph-mon[33213]: osdmap e349: 8 total, 8 up, 7 in 2024-02-05T07:30:00.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:00 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2492114878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:30:00.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:00 smithi090 ceph-mon[33213]: overall HEALTH_OK 2024-02-05T07:30:00.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:00 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:00.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:00 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:30:00.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:00.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[29098]: 6.1e scrub starts 2024-02-05T07:30:00.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[29098]: 6.1e scrub ok 2024-02-05T07:30:00.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[29098]: pgmap v581: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:30:00.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/858313719' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-05T07:30:00.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-05T07:30:00.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-05T07:30:00.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[29098]: osdmap e349: 8 total, 8 up, 7 in 2024-02-05T07:30:00.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2492114878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:30:00.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[29098]: overall HEALTH_OK 2024-02-05T07:30:00.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:00.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[37158]: 6.1e scrub starts 2024-02-05T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[37158]: 6.1e scrub ok 2024-02-05T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[37158]: pgmap v581: 152 pgs: 152 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/858313719' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-05T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-05T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-05T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[37158]: osdmap e349: 8 total, 8 up, 7 in 2024-02-05T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2492114878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[37158]: overall HEALTH_OK 2024-02-05T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:00 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:30:00.843 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:00.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:00.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:00.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:00.969 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:00.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:00.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:01.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:01.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:01.100 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:01.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:01.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:01.228 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:01.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:01.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T07:30:01.330 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:01.354 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:01.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:01.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:01.455 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:01.483 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:01.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:01.511 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.7 to 1 (10000) 2024-02-05T07:30:01.511 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[29098]: 4.1a scrub starts 2024-02-05T07:30:01.511 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[29098]: 4.1a scrub ok 2024-02-05T07:30:01.511 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:30:01.511 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2492114878' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:30:01.511 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[29098]: osdmap e350: 8 total, 8 up, 7 in 2024-02-05T07:30:01.511 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1233865620' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:30:01.511 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:30:01.512 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:01.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[37158]: 4.1a scrub starts 2024-02-05T07:30:01.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[37158]: 4.1a scrub ok 2024-02-05T07:30:01.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:30:01.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2492114878' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:30:01.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[37158]: osdmap e350: 8 total, 8 up, 7 in 2024-02-05T07:30:01.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1233865620' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:30:01.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:30:01.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:01.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:01.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:01.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:01.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:01.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:01.739 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:01.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:01.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:01 smithi090 ceph-mon[33213]: 4.1a scrub starts 2024-02-05T07:30:01.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:01 smithi090 ceph-mon[33213]: 4.1a scrub ok 2024-02-05T07:30:01.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:01 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:30:01.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:01 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2492114878' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:30:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:01 smithi090 ceph-mon[33213]: osdmap e350: 8 total, 8 up, 7 in 2024-02-05T07:30:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:01 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1233865620' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:30:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:01 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:30:01.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:01.842 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:01.866 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:01.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:01.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:01.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:01.995 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:01.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:01.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:02.097 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:02.121 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:02.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:02.223 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:02.246 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:02.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:02.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:02.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:02.376 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:02.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:02.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:02.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:02.502 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:02.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:02.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:02.574 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:02 smithi007 ceph-mon[29098]: 5.1c scrub starts 2024-02-05T07:30:02.575 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:02 smithi007 ceph-mon[29098]: pgmap v584: 168 pgs: 7 creating+activating, 1 unknown, 160 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T07:30:02.575 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:02 smithi007 ceph-mon[29098]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:30:02.575 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:02 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T07:30:02.575 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:02 smithi007 ceph-mon[29098]: osdmap e351: 8 total, 8 up, 7 in 2024-02-05T07:30:02.575 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:02 smithi007 ceph-mon[37158]: 5.1c scrub starts 2024-02-05T07:30:02.575 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:02 smithi007 ceph-mon[37158]: pgmap v584: 168 pgs: 7 creating+activating, 1 unknown, 160 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T07:30:02.575 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:02 smithi007 ceph-mon[37158]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:30:02.575 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:02 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T07:30:02.576 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:02 smithi007 ceph-mon[37158]: osdmap e351: 8 total, 8 up, 7 in 2024-02-05T07:30:02.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:02.630 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:02.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:02.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:02.759 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:02.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:02.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:02 smithi090 ceph-mon[33213]: 5.1c scrub starts 2024-02-05T07:30:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:02 smithi090 ceph-mon[33213]: pgmap v584: 168 pgs: 7 creating+activating, 1 unknown, 160 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T07:30:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:02 smithi090 ceph-mon[33213]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:30:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:02 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T07:30:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:02 smithi090 ceph-mon[33213]: osdmap e351: 8 total, 8 up, 7 in 2024-02-05T07:30:02.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:02.886 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:02.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:02.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:03.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 3 1 2024-02-05T07:30:03.015 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:03.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:03.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:03.145 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:03.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:03.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:03.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:03.270 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:03.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:03.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:03.372 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:03.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:03.396 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:03.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:03.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:03.526 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:03.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:03.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:03.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:03.653 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:03.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:03.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:03.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-02-05T07:30:03.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:03.777 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:03.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:03.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:03.864 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:30:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:30:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:30:03.879 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:03.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:03.984 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:30:04.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:04.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:04.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:04.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:04.180 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:04.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:04.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:04.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:04.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:04.390 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:04.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:04.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:04.515 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:04.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:04.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:04.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:04.640 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:04.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:04.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:04.742 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:04.758 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:04.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:04.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:04.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:04 smithi090 ceph-mon[33213]: pgmap v586: 168 pgs: 7 creating+activating, 1 unknown, 160 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:30:04.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:04 smithi090 ceph-mon[33213]: osdmap e352: 8 total, 8 up, 7 in 2024-02-05T07:30:04.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:04.880 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:04 smithi007 ceph-mon[29098]: pgmap v586: 168 pgs: 7 creating+activating, 1 unknown, 160 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:30:04.880 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:04 smithi007 ceph-mon[29098]: osdmap e352: 8 total, 8 up, 7 in 2024-02-05T07:30:04.880 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:04 smithi007 ceph-mon[37158]: pgmap v586: 168 pgs: 7 creating+activating, 1 unknown, 160 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:30:04.880 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:04 smithi007 ceph-mon[37158]: osdmap e352: 8 total, 8 up, 7 in 2024-02-05T07:30:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:04.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:04.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:04.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:05.021 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:05.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:05.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:05.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:05.148 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:05.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:05.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:05.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:05.278 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:05.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:05.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:05.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:05.405 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:05.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:05.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:05.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:05.530 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:05.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:05.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:05.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:05.660 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:05.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:05.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:05.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:05.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:05.914 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:05.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:06.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:06.046 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:06.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:06.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:06.171 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:06.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:06.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:06.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:06.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:06.399 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:06.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:06.431 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:06.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:06.507 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:06 smithi007 ceph-mon[29098]: pgmap v588: 168 pgs: 7 creating+activating, 161 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 900 B/s rd, 0 op/s 2024-02-05T07:30:06.507 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:06 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:30:06.507 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:06 smithi007 ceph-mon[37158]: pgmap v588: 168 pgs: 7 creating+activating, 161 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 900 B/s rd, 0 op/s 2024-02-05T07:30:06.507 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:06 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:30:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:06.528 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:30:06.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:06.552 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:30:06.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:06.560 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:06.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:06.571 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:30:06.591 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:30:06.611 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:30:06.632 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:30:06.652 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:30:06.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:06.672 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:30:06.673 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:30:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:30:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:30:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:30:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:30:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:30:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:30:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:30:06.684 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:06.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:06.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:06 smithi090 ceph-mon[33213]: pgmap v588: 168 pgs: 7 creating+activating, 161 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 900 B/s rd, 0 op/s 2024-02-05T07:30:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:06 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:30:06.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:06.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:06.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:06.936 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:06.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:07.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:07.062 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:07.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:07.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:07.186 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:07.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:07.288 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:07.312 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:07.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:07.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:07.414 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:07.438 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:07.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:07.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:07.525 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2886264339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:30:07.525 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.525 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.527 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.527 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.527 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.527 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.527 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:07.527 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:07.527 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:30:07.527 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:30:07.527 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:07.528 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2886264339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:30:07.528 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.528 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.528 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.528 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.528 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.528 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.528 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.528 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.529 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.529 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.529 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.529 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.529 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.529 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.529 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:07.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:07.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:30:07.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:30:07.530 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:07 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:07.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:07.557 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:07.564 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:07.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:07.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:07.690 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:07.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:07.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:07.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2886264339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:30:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T07:30:07.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:07.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:07.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:30:07.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:30:07.776 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:07 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:07.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:07.815 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:07.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:07.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:07.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:07.942 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:07.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:08.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:08.068 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:08.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:08.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:08.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:08.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:08.195 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:08.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:08.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:08.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:08.321 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:08.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:08.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:08.446 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:08.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:08.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:08 smithi007 ceph-mon[29098]: Adjusting osd_memory_target on smithi090 to 3248M 2024-02-05T07:30:08.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:08 smithi007 ceph-mon[29098]: Adjusting osd_memory_target on smithi007 to 3249M 2024-02-05T07:30:08.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:08 smithi007 ceph-mon[29098]: pgmap v589: 168 pgs: 168 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T07:30:08.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:08 smithi007 ceph-mon[37158]: Adjusting osd_memory_target on smithi090 to 3248M 2024-02-05T07:30:08.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:08 smithi007 ceph-mon[37158]: Adjusting osd_memory_target on smithi007 to 3249M 2024-02-05T07:30:08.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:08 smithi007 ceph-mon[37158]: pgmap v589: 168 pgs: 168 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T07:30:08.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:08.572 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:08.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:08.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:08.698 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:08.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:08.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:08 smithi090 ceph-mon[33213]: Adjusting osd_memory_target on smithi090 to 3248M 2024-02-05T07:30:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:08 smithi090 ceph-mon[33213]: Adjusting osd_memory_target on smithi007 to 3249M 2024-02-05T07:30:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:08 smithi090 ceph-mon[33213]: pgmap v589: 168 pgs: 168 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T07:30:08.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:08.823 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:08.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:08.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:08.949 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:08.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:09.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:09.075 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:09.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:09.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:09.203 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:09.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:09.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:09.329 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:09.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:09.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:09.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:09.455 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:09.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:09.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:09.586 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:09.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:09.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:09.713 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:09.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:09.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:09.839 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:09.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:09.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:09.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:09.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:30:10.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:10.092 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:10.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:10.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:10.218 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:10.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:10.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:10.344 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:10.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:10.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:10.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:10.471 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:10.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:10.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:10.550 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:10 smithi007 ceph-mon[37158]: pgmap v590: 168 pgs: 168 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:30:10.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:10 smithi007 ceph-mon[29098]: pgmap v590: 168 pgs: 168 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:30:10.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:10.597 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:10.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:10.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:10.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:10.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:10.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:30:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:30:10.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:10 smithi090 ceph-mon[33213]: pgmap v590: 168 pgs: 168 active+clean; 582 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:30:10.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:10.852 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:10.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:10.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:10.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:10.980 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:10.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:10.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:11.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:11.109 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:11.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:11.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:11.235 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:11.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:11.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:11.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:11.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:11.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:11.471 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.3 to 1 (10000) 2024-02-05T07:30:11.491 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:11.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:11.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:11.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:11.617 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:11.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:11.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:11.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:11 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:11.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:11 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/10361880' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:30:11.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:11 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:11.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:11 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/10361880' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:30:11.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:11.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:11.744 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:11.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:11.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:11 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:11.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:11 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/10361880' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:30:11.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:11.870 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:11.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:11.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:11.972 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:12.001 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:12.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:12.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:12.103 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:12.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:12.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:12.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:12.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:12.258 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:12.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:12.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:12.388 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:12.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:12.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:12.413 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-05T07:30:12.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:30:12.475 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-02-05T07:30:12.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:12.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:12.522 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:12.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:12.623 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:12.647 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:12.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:12.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:12 smithi007 ceph-mon[29098]: pgmap v591: 168 pgs: 168 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:30:12.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:12 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/10361880' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T07:30:12.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:12 smithi007 ceph-mon[29098]: osdmap e353: 8 total, 8 up, 7 in 2024-02-05T07:30:12.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:12 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1542769111' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:30:12.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:12 smithi007 ceph-mon[37158]: pgmap v591: 168 pgs: 168 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:30:12.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:12 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/10361880' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T07:30:12.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:12 smithi007 ceph-mon[37158]: osdmap e353: 8 total, 8 up, 7 in 2024-02-05T07:30:12.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:12 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1542769111' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:30:12.748 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:30:12.748 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:30:12.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:12 smithi090 ceph-mon[33213]: pgmap v591: 168 pgs: 168 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:30:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:12 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/10361880' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T07:30:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:12 smithi090 ceph-mon[33213]: osdmap e353: 8 total, 8 up, 7 in 2024-02-05T07:30:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:12 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1542769111' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:30:12.777 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:12.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:12.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:12.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:12.903 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:12.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:13.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:13.029 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:13.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:13.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:13.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:13.154 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:13.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:13.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:13.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:13.284 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:13.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:13.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T07:30:13.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:13.414 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:13.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:13.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:13.541 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:13.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:13.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:13.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:13 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1542769111' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:30:13.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:13 smithi007 ceph-mon[29098]: osdmap e354: 8 total, 8 up, 7 in 2024-02-05T07:30:13.571 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:30:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:30:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:30:13.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:13 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1542769111' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:30:13.572 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:13 smithi007 ceph-mon[37158]: osdmap e354: 8 total, 8 up, 7 in 2024-02-05T07:30:13.602 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:30:13.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:13.667 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:13.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:13.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:13.725 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:13 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1542769111' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:30:13.725 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:13 smithi090 ceph-mon[33213]: osdmap e354: 8 total, 8 up, 7 in 2024-02-05T07:30:13.768 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:13.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:13.793 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:13.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:13.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:13.918 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:13.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:13.921 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:14.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:14.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:14.045 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:14.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:14.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:14.170 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:14.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:14.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:14.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:14.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:14.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:14.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:14.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:14.427 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:14.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:14.485 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:30:14.528 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:14.551 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:14.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:14.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:14.618 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:14 smithi007 ceph-mon[29098]: pgmap v594: 168 pgs: 168 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:30:14.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:14 smithi007 ceph-mon[29098]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:30:14.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:14 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1897497539' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:30:14.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:14 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:30:14.619 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:14 smithi007 ceph-mon[37158]: pgmap v594: 168 pgs: 168 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:30:14.619 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:14 smithi007 ceph-mon[37158]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:30:14.619 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:14 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1897497539' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:30:14.619 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:14 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:30:14.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:14.677 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:14.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:14.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:14.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:14 smithi090 ceph-mon[33213]: pgmap v594: 168 pgs: 168 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:30:14.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:14 smithi090 ceph-mon[33213]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:30:14.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:14 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1897497539' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:30:14.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:14 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:30:14.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:14.802 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:14.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:14.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:14.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:14.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:15.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:15.054 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:15.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:15.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:15.179 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:15.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:15.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:15.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:15.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:15.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:15.366 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:30:15.367 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:30:15.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:15.432 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:15.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:15.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:15.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:15.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:15.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:15.638 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:15 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:30:15.639 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:15 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:30:15.639 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:15 smithi090 ceph-mon[33213]: osdmap e355: 8 total, 8 up, 7 in 2024-02-05T07:30:15.639 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:15 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:15.639 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:30:15.645 DEBUG:teuthology.orchestra.run.smithi007:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-02-05T07:30:15.648 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T07:30:15.648 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T07:30:15.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:15.683 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:15.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:15.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:15.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:15.807 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:15.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:15.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:15.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:15 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:30:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:15 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:30:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:15 smithi007 ceph-mon[29098]: osdmap e355: 8 total, 8 up, 7 in 2024-02-05T07:30:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:15 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:30:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:15 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:30:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:15 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:30:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:15 smithi007 ceph-mon[37158]: osdmap e355: 8 total, 8 up, 7 in 2024-02-05T07:30:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:15 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:30:15.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:hints = 1 2024-02-05T07:30:15.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T07:30:15.927 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object prefix: benchmark_data_smithi007_136200 2024-02-05T07:30:15.928 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:30:15.928 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 0 0 0 0 0 0 - 0 2024-02-05T07:30:15.939 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:15.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:16.041 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:16.065 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T07:30:16.077 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:16.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:16.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:16.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:16.223 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:16.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:16.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:16.325 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:16.368 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:16.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:16.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:16.448 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:30:16.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:16.508 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:16.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:16.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:16.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:16.646 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:16.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:16.734 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:16 smithi090 ceph-mon[33213]: pgmap v596: 168 pgs: 168 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:30:16.745 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:16 smithi007 ceph-mon[29098]: pgmap v596: 168 pgs: 168 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:30:16.745 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:16 smithi007 ceph-mon[37158]: pgmap v596: 168 pgs: 168 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:30:16.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:16.781 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:16.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:16.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:16.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:16.928 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 1 16 3265 3249 203.043 203.062 0.00411451 0.00490247 2024-02-05T07:30:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:16.931 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:16.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:17.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:17.074 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:17.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:17.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:30:17.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:17.208 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:17.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:17.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:17.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:17.342 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:17.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:17.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:17.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:17.493 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:17.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:17.594 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:17.636 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:17.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:17.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:17.778 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:17.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:17.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:17.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:17.926 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:17.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:17.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:17.928 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 2 16 6220 6204 193.857 184.688 0.010748 0.0051462 2024-02-05T07:30:18.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:18.055 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:18.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:18.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:18.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:18.199 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:18.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:18.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:18.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:18.302 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:30:18.302 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:30:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:18.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:18.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:18.475 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:30:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:18.488 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:18.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:18.523 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:30:18.562 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:30:18.590 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:18.615 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:30:18.615 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:18 smithi007 ceph-mon[29098]: pgmap v597: 168 pgs: 168 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:30:18.615 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:18 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/562701680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:30:18.615 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:18 smithi007 ceph-mon[37158]: pgmap v597: 168 pgs: 168 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:30:18.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:18 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/562701680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:30:18.622 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:18.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:18.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:18.660 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:30:18.683 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:18 smithi090 ceph-mon[33213]: pgmap v597: 168 pgs: 168 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:30:18.683 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:18 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/562701680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:30:18.692 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:30:18.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:18.735 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:30:18.767 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:18.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:18.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:18.781 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:30:18.784 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:30:18.784 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:30:18.784 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:30:18.785 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:30:18.785 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:30:18.785 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:30:18.785 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:30:18.785 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:30:18.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:18.912 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:18.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:18.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:18.928 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 3 16 8635 8619 179.546 150.938 0.00804242 0.00555766 2024-02-05T07:30:19.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:19.055 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:19.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:19.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:19.124 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:19.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:19.206 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:19.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:19.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:19.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T07:30:19.350 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:30:19.352 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:19.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:19.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:19.356 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:30:19.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:19.494 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:19.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:19.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:19.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:19.647 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:19.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:19.667 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:30:19.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:19.790 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:19.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:19.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:19.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:19.925 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:19.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:19.928 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 4 16 11124 11108 173.547 155.562 0.00774236 0.00575447 2024-02-05T07:30:20.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:20.059 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:20.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:20.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:20.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:20.201 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:20.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:20.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:20.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:20.338 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:20.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:20.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:20.467 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:20.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:20.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:20.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:20.603 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:20.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:20.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:20.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:20.738 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:20.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:20.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:20.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:30:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:30:20.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:20 smithi090 ceph-mon[33213]: pgmap v598: 168 pgs: 168 active+clean; 30 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 663 B/s rd, 3.7 MiB/s wr, 60 op/s 2024-02-05T07:30:20.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:20 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:30:20.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:20 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:30:20.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:20 smithi090 ceph-mon[33213]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:30:20.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:20 smithi090 ceph-mon[33213]: osdmap e356: 8 total, 8 up, 7 in 2024-02-05T07:30:20.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:20.874 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:20.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:20 smithi007 ceph-mon[29098]: pgmap v598: 168 pgs: 168 active+clean; 30 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 663 B/s rd, 3.7 MiB/s wr, 60 op/s 2024-02-05T07:30:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:20 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:30:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:20 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:30:20.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:20 smithi007 ceph-mon[29098]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:30:20.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:20 smithi007 ceph-mon[29098]: osdmap e356: 8 total, 8 up, 7 in 2024-02-05T07:30:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:20 smithi007 ceph-mon[37158]: pgmap v598: 168 pgs: 168 active+clean; 30 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 663 B/s rd, 3.7 MiB/s wr, 60 op/s 2024-02-05T07:30:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:20 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:30:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:20 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:30:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:20 smithi007 ceph-mon[37158]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:30:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:20 smithi007 ceph-mon[37158]: osdmap e356: 8 total, 8 up, 7 in 2024-02-05T07:30:20.928 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 5 16 13776 13760 171.985 165.75 0.0027597 0.00580724 2024-02-05T07:30:20.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:21.011 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:21.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:21.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:21.109 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":598,"stamp":"2024-02-05T07:30:19.185640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30939484,"num_objects":661,"num_object_clones":0,"num_object_copies":1983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":1805,"num_read_kb":1557,"num_write":1185,"num_write_kb":29677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":5674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3989,"ondisk_log_size":3989,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":507,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3264920,"kb_used_data":195196,"kb_used_omap":14,"kb_used_meta":3069617,"kb_avail":652951144,"statfs":{"total":671965249536,"available":668621971456,"internally_reserved":0,"allocated":199880704,"data_stored":193307725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15190,"internal_metadata":3143287978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30343168,"num_objects":463,"num_object_clones":0,"num_object_copies":1389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":463,"num_write_kb":29632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.721532"},"pg_stats":[{"pgid":"9.1","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.715977+0000","last_change":"2024-02-05T07:30:00.509854+0000","last_active":"2024-02-05T07:30:14.715977+0000","last_peered":"2024-02-05T07:30:14.715977+0000","last_clean":"2024-02-05T07:30:14.715977+0000","last_became_active":"2024-02-05T07:30:00.509481+0000","last_became_peered":"2024-02-05T07:30:00.509481+0000","last_unstale":"2024-02-05T07:30:14.715977+0000","last_undegraded":"2024-02-05T07:30:14.715977+0000","last_fullsized":"2024-02-05T07:30:14.715977+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:16.743319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":905,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716715+0000","last_change":"2024-02-05T07:27:19.752065+0000","last_active":"2024-02-05T07:30:14.716715+0000","last_peered":"2024-02-05T07:30:14.716715+0000","last_clean":"2024-02-05T07:30:14.716715+0000","last_became_active":"2024-02-05T07:27:19.751740+0000","last_became_peered":"2024-02-05T07:27:19.751740+0000","last_unstale":"2024-02-05T07:30:14.716715+0000","last_undegraded":"2024-02-05T07:30:14.716715+0000","last_fullsized":"2024-02-05T07:30:14.716715+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:26:18.993912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.993912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"326'25","reported_seq":687,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481879+0000","last_change":"2024-02-05T07:28:26.605879+0000","last_active":"2024-02-05T07:30:14.481879+0000","last_peered":"2024-02-05T07:30:14.481879+0000","last_clean":"2024-02-05T07:30:14.481879+0000","last_became_active":"2024-02-05T07:28:26.605082+0000","last_became_peered":"2024-02-05T07:28:26.605082+0000","last_unstale":"2024-02-05T07:30:14.481879+0000","last_undegraded":"2024-02-05T07:30:14.481879+0000","last_fullsized":"2024-02-05T07:30:14.481879+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'15","last_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:25:44.205475+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906522+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":194,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717123+0000","last_change":"2024-02-05T07:27:16.716749+0000","last_active":"2024-02-05T07:30:14.717123+0000","last_peered":"2024-02-05T07:30:14.717123+0000","last_clean":"2024-02-05T07:30:14.717123+0000","last_became_active":"2024-02-05T07:27:16.716461+0000","last_became_peered":"2024-02-05T07:27:16.716461+0000","last_unstale":"2024-02-05T07:30:14.717123+0000","last_undegraded":"2024-02-05T07:30:14.717123+0000","last_fullsized":"2024-02-05T07:30:14.717123+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.258469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.258469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"257'9","reported_seq":772,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717467+0000","last_change":"2024-02-05T07:28:43.135097+0000","last_active":"2024-02-05T07:30:14.717467+0000","last_peered":"2024-02-05T07:30:14.717467+0000","last_clean":"2024-02-05T07:30:14.717467+0000","last_became_active":"2024-02-05T07:27:03.374124+0000","last_became_peered":"2024-02-05T07:27:03.374124+0000","last_unstale":"2024-02-05T07:30:14.717467+0000","last_undegraded":"2024-02-05T07:30:14.717467+0000","last_fullsized":"2024-02-05T07:30:14.717467+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'9","last_scrub_stamp":"2024-02-05T07:28:43.135015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:43.135015+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:20.016019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":601,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482324+0000","last_change":"2024-02-05T07:25:43.273140+0000","last_active":"2024-02-05T07:30:14.482324+0000","last_peered":"2024-02-05T07:30:14.482324+0000","last_clean":"2024-02-05T07:30:14.482324+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:30:14.482324+0000","last_undegraded":"2024-02-05T07:30:14.482324+0000","last_fullsized":"2024-02-05T07:30:14.482324+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:25:43.273019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906582+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.479928+0000","last_change":"2024-02-05T07:30:00.506880+0000","last_active":"2024-02-05T07:30:14.479928+0000","last_peered":"2024-02-05T07:30:14.479928+0000","last_clean":"2024-02-05T07:30:14.479928+0000","last_became_active":"2024-02-05T07:30:00.506586+0000","last_became_peered":"2024-02-05T07:30:00.506586+0000","last_unstale":"2024-02-05T07:30:14.479928+0000","last_undegraded":"2024-02-05T07:30:14.479928+0000","last_fullsized":"2024-02-05T07:30:14.479928+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:21.843677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":620,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.715646+0000","last_change":"2024-02-05T07:29:48.249173+0000","last_active":"2024-02-05T07:30:14.715646+0000","last_peered":"2024-02-05T07:30:14.715646+0000","last_clean":"2024-02-05T07:30:14.715646+0000","last_became_active":"2024-02-05T07:28:26.593333+0000","last_became_peered":"2024-02-05T07:28:26.593333+0000","last_unstale":"2024-02-05T07:30:14.715646+0000","last_undegraded":"2024-02-05T07:30:14.715646+0000","last_fullsized":"2024-02-05T07:30:14.715646+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:29:48.249082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.249082+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:14.621210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"326'31","reported_seq":666,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482062+0000","last_change":"2024-02-05T07:26:36.275060+0000","last_active":"2024-02-05T07:30:14.482062+0000","last_peered":"2024-02-05T07:30:14.482062+0000","last_clean":"2024-02-05T07:30:14.482062+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:30:14.482062+0000","last_undegraded":"2024-02-05T07:30:14.482062+0000","last_fullsized":"2024-02-05T07:30:14.482062+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:26:36.274939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.274939+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906801+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":457,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716872+0000","last_change":"2024-02-05T07:28:25.791721+0000","last_active":"2024-02-05T07:30:14.716872+0000","last_peered":"2024-02-05T07:30:14.716872+0000","last_clean":"2024-02-05T07:30:14.716872+0000","last_became_active":"2024-02-05T07:28:25.791005+0000","last_became_peered":"2024-02-05T07:28:25.791005+0000","last_unstale":"2024-02-05T07:30:14.716872+0000","last_undegraded":"2024-02-05T07:30:14.716872+0000","last_fullsized":"2024-02-05T07:30:14.716872+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:35.518301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:35.518301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":614,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481850+0000","last_change":"2024-02-05T07:29:49.674581+0000","last_active":"2024-02-05T07:30:14.481850+0000","last_peered":"2024-02-05T07:30:14.481850+0000","last_clean":"2024-02-05T07:30:14.481850+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:30:14.481850+0000","last_undegraded":"2024-02-05T07:30:14.481850+0000","last_fullsized":"2024-02-05T07:30:14.481850+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.674493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.674493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:05.012130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":616,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717778+0000","last_change":"2024-02-05T07:28:38.294618+0000","last_active":"2024-02-05T07:30:14.717778+0000","last_peered":"2024-02-05T07:30:14.717778+0000","last_clean":"2024-02-05T07:30:14.717778+0000","last_became_active":"2024-02-05T07:27:16.708888+0000","last_became_peered":"2024-02-05T07:27:16.708888+0000","last_unstale":"2024-02-05T07:30:14.717778+0000","last_undegraded":"2024-02-05T07:30:14.717778+0000","last_fullsized":"2024-02-05T07:30:14.717778+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:38.294470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:38.294470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:20.015966+0000","scrub_duration":232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717366+0000","last_change":"2024-02-05T07:30:00.508306+0000","last_active":"2024-02-05T07:30:14.717366+0000","last_peered":"2024-02-05T07:30:14.717366+0000","last_clean":"2024-02-05T07:30:14.717366+0000","last_became_active":"2024-02-05T07:30:00.507929+0000","last_became_peered":"2024-02-05T07:30:00.507929+0000","last_unstale":"2024-02-05T07:30:14.717366+0000","last_undegraded":"2024-02-05T07:30:14.717366+0000","last_fullsized":"2024-02-05T07:30:14.717366+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.268882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":591,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481525+0000","last_change":"2024-02-05T07:28:46.611825+0000","last_active":"2024-02-05T07:30:14.481525+0000","last_peered":"2024-02-05T07:30:14.481525+0000","last_clean":"2024-02-05T07:30:14.481525+0000","last_became_active":"2024-02-05T07:28:19.312714+0000","last_became_peered":"2024-02-05T07:28:19.312714+0000","last_unstale":"2024-02-05T07:30:14.481525+0000","last_undegraded":"2024-02-05T07:30:14.481525+0000","last_fullsized":"2024-02-05T07:30:14.481525+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:46.611747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:46.611747+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:17.799522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"326'24","reported_seq":662,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482017+0000","last_change":"2024-02-05T07:29:50.053697+0000","last_active":"2024-02-05T07:30:14.482017+0000","last_peered":"2024-02-05T07:30:14.482017+0000","last_clean":"2024-02-05T07:30:14.482017+0000","last_became_active":"2024-02-05T07:28:25.781007+0000","last_became_peered":"2024-02-05T07:28:25.781007+0000","last_unstale":"2024-02-05T07:30:14.482017+0000","last_undegraded":"2024-02-05T07:30:14.482017+0000","last_fullsized":"2024-02-05T07:30:14.482017+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:29:50.053583+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.053583+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:01.795266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":613,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481195+0000","last_change":"2024-02-05T07:25:46.149566+0000","last_active":"2024-02-05T07:30:14.481195+0000","last_peered":"2024-02-05T07:30:14.481195+0000","last_clean":"2024-02-05T07:30:14.481195+0000","last_became_active":"2024-02-05T07:18:49.434074+0000","last_became_peered":"2024-02-05T07:18:49.434074+0000","last_unstale":"2024-02-05T07:30:14.481195+0000","last_undegraded":"2024-02-05T07:30:14.481195+0000","last_fullsized":"2024-02-05T07:30:14.481195+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:25:46.149431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":448,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482146+0000","last_change":"2024-02-05T07:29:45.012958+0000","last_active":"2024-02-05T07:30:14.482146+0000","last_peered":"2024-02-05T07:30:14.482146+0000","last_clean":"2024-02-05T07:30:14.482146+0000","last_became_active":"2024-02-05T07:28:26.601713+0000","last_became_peered":"2024-02-05T07:28:26.601713+0000","last_unstale":"2024-02-05T07:30:14.482146+0000","last_undegraded":"2024-02-05T07:30:14.482146+0000","last_fullsized":"2024-02-05T07:30:14.482146+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.012821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:02.146624+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":197,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717073+0000","last_change":"2024-02-05T07:28:26.598062+0000","last_active":"2024-02-05T07:30:14.717073+0000","last_peered":"2024-02-05T07:30:14.717073+0000","last_clean":"2024-02-05T07:30:14.717073+0000","last_became_active":"2024-02-05T07:28:26.595495+0000","last_became_peered":"2024-02-05T07:28:26.595495+0000","last_unstale":"2024-02-05T07:30:14.717073+0000","last_undegraded":"2024-02-05T07:30:14.717073+0000","last_fullsized":"2024-02-05T07:30:14.717073+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:47.885294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.885294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.715919+0000","last_change":"2024-02-05T07:30:00.512144+0000","last_active":"2024-02-05T07:30:14.715919+0000","last_peered":"2024-02-05T07:30:14.715919+0000","last_clean":"2024-02-05T07:30:14.715919+0000","last_became_active":"2024-02-05T07:30:00.511808+0000","last_became_peered":"2024-02-05T07:30:00.511808+0000","last_unstale":"2024-02-05T07:30:14.715919+0000","last_undegraded":"2024-02-05T07:30:14.715919+0000","last_fullsized":"2024-02-05T07:30:14.715919+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:21.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":595,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716090+0000","last_change":"2024-02-05T07:28:27.218610+0000","last_active":"2024-02-05T07:30:14.716090+0000","last_peered":"2024-02-05T07:30:14.716090+0000","last_clean":"2024-02-05T07:30:14.716090+0000","last_became_active":"2024-02-05T07:28:26.588864+0000","last_became_peered":"2024-02-05T07:28:26.588864+0000","last_unstale":"2024-02-05T07:30:14.716090+0000","last_undegraded":"2024-02-05T07:30:14.716090+0000","last_fullsized":"2024-02-05T07:30:14.716090+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:27.218488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:27.218488+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"326'34","reported_seq":704,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481814+0000","last_change":"2024-02-05T07:27:16.982664+0000","last_active":"2024-02-05T07:30:14.481814+0000","last_peered":"2024-02-05T07:30:14.481814+0000","last_clean":"2024-02-05T07:30:14.481814+0000","last_became_active":"2024-02-05T07:27:16.768454+0000","last_became_peered":"2024-02-05T07:27:16.768454+0000","last_unstale":"2024-02-05T07:30:14.481814+0000","last_undegraded":"2024-02-05T07:30:14.481814+0000","last_fullsized":"2024-02-05T07:30:14.481814+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'22","last_scrub_stamp":"2024-02-05T07:26:25.602159+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:26:25.602159+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744032+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":194,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716230+0000","last_change":"2024-02-05T07:27:16.830852+0000","last_active":"2024-02-05T07:30:14.716230+0000","last_peered":"2024-02-05T07:30:14.716230+0000","last_clean":"2024-02-05T07:30:14.716230+0000","last_became_active":"2024-02-05T07:27:16.800735+0000","last_became_peered":"2024-02-05T07:27:16.800735+0000","last_unstale":"2024-02-05T07:30:14.716230+0000","last_undegraded":"2024-02-05T07:30:14.716230+0000","last_fullsized":"2024-02-05T07:30:14.716230+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:48.513614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:48.513614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"257'9","reported_seq":761,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482245+0000","last_change":"2024-02-05T07:27:03.386101+0000","last_active":"2024-02-05T07:30:14.482245+0000","last_peered":"2024-02-05T07:30:14.482245+0000","last_clean":"2024-02-05T07:30:14.482245+0000","last_became_active":"2024-02-05T07:27:03.375195+0000","last_became_peered":"2024-02-05T07:27:03.375195+0000","last_unstale":"2024-02-05T07:30:14.482245+0000","last_undegraded":"2024-02-05T07:30:14.482245+0000","last_fullsized":"2024-02-05T07:30:14.482245+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:26:22.295732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:22.295732+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906712+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":212,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482379+0000","last_change":"2024-02-05T07:27:03.368067+0000","last_active":"2024-02-05T07:30:14.482379+0000","last_peered":"2024-02-05T07:30:14.482379+0000","last_clean":"2024-02-05T07:30:14.482379+0000","last_became_active":"2024-02-05T07:27:03.367753+0000","last_became_peered":"2024-02-05T07:27:03.367753+0000","last_unstale":"2024-02-05T07:30:14.482379+0000","last_undegraded":"2024-02-05T07:30:14.482379+0000","last_fullsized":"2024-02-05T07:30:14.482379+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.724944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.724944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717933+0000","last_change":"2024-02-05T07:30:01.030341+0000","last_active":"2024-02-05T07:30:14.717933+0000","last_peered":"2024-02-05T07:30:14.717933+0000","last_clean":"2024-02-05T07:30:14.717933+0000","last_became_active":"2024-02-05T07:30:01.029929+0000","last_became_peered":"2024-02-05T07:30:01.029929+0000","last_unstale":"2024-02-05T07:30:14.717933+0000","last_undegraded":"2024-02-05T07:30:14.717933+0000","last_fullsized":"2024-02-05T07:30:14.717933+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:05.041540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":590,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481654+0000","last_change":"2024-02-05T07:28:41.657737+0000","last_active":"2024-02-05T07:30:14.481654+0000","last_peered":"2024-02-05T07:30:14.481654+0000","last_clean":"2024-02-05T07:30:14.481654+0000","last_became_active":"2024-02-05T07:28:25.780676+0000","last_became_peered":"2024-02-05T07:28:25.780676+0000","last_unstale":"2024-02-05T07:30:14.481654+0000","last_undegraded":"2024-02-05T07:30:14.481654+0000","last_fullsized":"2024-02-05T07:30:14.481654+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:41.657622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:41.657622+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906930+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":617,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481130+0000","last_change":"2024-02-05T07:27:16.706337+0000","last_active":"2024-02-05T07:30:14.481130+0000","last_peered":"2024-02-05T07:30:14.481130+0000","last_clean":"2024-02-05T07:30:14.481130+0000","last_became_active":"2024-02-05T07:27:16.705976+0000","last_became_peered":"2024-02-05T07:27:16.705976+0000","last_unstale":"2024-02-05T07:30:14.481130+0000","last_undegraded":"2024-02-05T07:30:14.481130+0000","last_fullsized":"2024-02-05T07:30:14.481130+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:51.211877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:51.211877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906448+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"326'30","reported_seq":238,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716131+0000","last_change":"2024-02-05T07:28:45.029429+0000","last_active":"2024-02-05T07:30:14.716131+0000","last_peered":"2024-02-05T07:30:14.716131+0000","last_clean":"2024-02-05T07:30:14.716131+0000","last_became_active":"2024-02-05T07:27:16.830389+0000","last_became_peered":"2024-02-05T07:27:16.830389+0000","last_unstale":"2024-02-05T07:30:14.716131+0000","last_undegraded":"2024-02-05T07:30:14.716131+0000","last_fullsized":"2024-02-05T07:30:14.716131+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:28:45.029295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.029295+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190914+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":194,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717822+0000","last_change":"2024-02-05T07:27:16.720166+0000","last_active":"2024-02-05T07:30:14.717822+0000","last_peered":"2024-02-05T07:30:14.717822+0000","last_clean":"2024-02-05T07:30:14.717822+0000","last_became_active":"2024-02-05T07:27:16.719678+0000","last_became_peered":"2024-02-05T07:27:16.719678+0000","last_unstale":"2024-02-05T07:30:14.717822+0000","last_undegraded":"2024-02-05T07:30:14.717822+0000","last_fullsized":"2024-02-05T07:30:14.717822+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:33.084016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:33.084016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190766+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":616,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481526+0000","last_change":"2024-02-05T07:28:56.783280+0000","last_active":"2024-02-05T07:30:14.481526+0000","last_peered":"2024-02-05T07:30:14.481526+0000","last_clean":"2024-02-05T07:30:14.481526+0000","last_became_active":"2024-02-05T07:27:16.706666+0000","last_became_peered":"2024-02-05T07:27:16.706666+0000","last_unstale":"2024-02-05T07:30:14.481526+0000","last_undegraded":"2024-02-05T07:30:14.481526+0000","last_fullsized":"2024-02-05T07:30:14.481526+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:28:56.783168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.528819+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717235+0000","last_change":"2024-02-05T07:30:01.031178+0000","last_active":"2024-02-05T07:30:14.717235+0000","last_peered":"2024-02-05T07:30:14.717235+0000","last_clean":"2024-02-05T07:30:14.717235+0000","last_became_active":"2024-02-05T07:30:01.030826+0000","last_became_peered":"2024-02-05T07:30:01.030826+0000","last_unstale":"2024-02-05T07:30:14.717235+0000","last_undegraded":"2024-02-05T07:30:14.717235+0000","last_fullsized":"2024-02-05T07:30:14.717235+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:17.743657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":851,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717344+0000","last_change":"2024-02-05T07:28:39.042617+0000","last_active":"2024-02-05T07:30:14.717344+0000","last_peered":"2024-02-05T07:30:14.717344+0000","last_clean":"2024-02-05T07:30:14.717344+0000","last_became_active":"2024-02-05T07:28:26.596040+0000","last_became_peered":"2024-02-05T07:28:26.596040+0000","last_unstale":"2024-02-05T07:30:14.717344+0000","last_undegraded":"2024-02-05T07:30:14.717344+0000","last_fullsized":"2024-02-05T07:30:14.717344+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:39.042281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:39.042281+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190935+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"266'10","reported_seq":234,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716316+0000","last_change":"2024-02-05T07:27:17.017592+0000","last_active":"2024-02-05T07:30:14.716316+0000","last_peered":"2024-02-05T07:30:14.716316+0000","last_clean":"2024-02-05T07:30:14.716316+0000","last_became_active":"2024-02-05T07:27:16.800207+0000","last_became_peered":"2024-02-05T07:27:16.800207+0000","last_unstale":"2024-02-05T07:30:14.716316+0000","last_undegraded":"2024-02-05T07:30:14.716316+0000","last_fullsized":"2024-02-05T07:30:14.716316+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:26:34.535447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:34.535447+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"326'35","reported_seq":687,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481298+0000","last_change":"2024-02-05T07:29:45.972745+0000","last_active":"2024-02-05T07:30:14.481298+0000","last_peered":"2024-02-05T07:30:14.481298+0000","last_clean":"2024-02-05T07:30:14.481298+0000","last_became_active":"2024-02-05T07:28:26.598831+0000","last_became_peered":"2024-02-05T07:28:26.598831+0000","last_unstale":"2024-02-05T07:30:14.481298+0000","last_undegraded":"2024-02-05T07:30:14.481298+0000","last_fullsized":"2024-02-05T07:30:14.481298+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:29:45.972589+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.972589+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:11.921496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":374,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482263+0000","last_change":"2024-02-05T07:29:41.587097+0000","last_active":"2024-02-05T07:30:14.482263+0000","last_peered":"2024-02-05T07:30:14.482263+0000","last_clean":"2024-02-05T07:30:14.482263+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:30:14.482263+0000","last_undegraded":"2024-02-05T07:30:14.482263+0000","last_fullsized":"2024-02-05T07:30:14.482263+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:41.587022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:41.587022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:03.158718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":610,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717777+0000","last_change":"2024-02-05T07:29:45.254768+0000","last_active":"2024-02-05T07:30:14.717777+0000","last_peered":"2024-02-05T07:30:14.717777+0000","last_clean":"2024-02-05T07:30:14.717777+0000","last_became_active":"2024-02-05T07:28:25.788941+0000","last_became_peered":"2024-02-05T07:28:25.788941+0000","last_unstale":"2024-02-05T07:30:14.717777+0000","last_undegraded":"2024-02-05T07:30:14.717777+0000","last_fullsized":"2024-02-05T07:30:14.717777+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:45.254637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.254637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:47.878497+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481551+0000","last_change":"2024-02-05T07:30:00.509399+0000","last_active":"2024-02-05T07:30:14.481551+0000","last_peered":"2024-02-05T07:30:14.481551+0000","last_clean":"2024-02-05T07:30:14.481551+0000","last_became_active":"2024-02-05T07:30:00.509120+0000","last_became_peered":"2024-02-05T07:30:00.509120+0000","last_unstale":"2024-02-05T07:30:14.481551+0000","last_undegraded":"2024-02-05T07:30:14.481551+0000","last_fullsized":"2024-02-05T07:30:14.481551+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:24.514798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":445,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717663+0000","last_change":"2024-02-05T07:28:48.540110+0000","last_active":"2024-02-05T07:30:14.717663+0000","last_peered":"2024-02-05T07:30:14.717663+0000","last_clean":"2024-02-05T07:30:14.717663+0000","last_became_active":"2024-02-05T07:28:26.608674+0000","last_became_peered":"2024-02-05T07:28:26.608674+0000","last_unstale":"2024-02-05T07:30:14.717663+0000","last_undegraded":"2024-02-05T07:30:14.717663+0000","last_fullsized":"2024-02-05T07:30:14.717663+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:48.539780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.539780+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.046021+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":226,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716725+0000","last_change":"2024-02-05T07:29:39.912134+0000","last_active":"2024-02-05T07:30:14.716725+0000","last_peered":"2024-02-05T07:30:14.716725+0000","last_clean":"2024-02-05T07:30:14.716725+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:30:14.716725+0000","last_undegraded":"2024-02-05T07:30:14.716725+0000","last_fullsized":"2024-02-05T07:30:14.716725+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:29:39.912035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:39.912035+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:58.559919+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"326'25","reported_seq":671,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481407+0000","last_change":"2024-02-05T07:27:03.387977+0000","last_active":"2024-02-05T07:30:14.481407+0000","last_peered":"2024-02-05T07:30:14.481407+0000","last_clean":"2024-02-05T07:30:14.481407+0000","last_became_active":"2024-02-05T07:27:03.361342+0000","last_became_peered":"2024-02-05T07:27:03.361342+0000","last_unstale":"2024-02-05T07:30:14.481407+0000","last_undegraded":"2024-02-05T07:30:14.481407+0000","last_fullsized":"2024-02-05T07:30:14.481407+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-02-05T07:26:26.616180+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:26:26.616180+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744072+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":205,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717176+0000","last_change":"2024-02-05T07:29:44.015577+0000","last_active":"2024-02-05T07:30:14.717176+0000","last_peered":"2024-02-05T07:30:14.717176+0000","last_clean":"2024-02-05T07:30:14.717176+0000","last_became_active":"2024-02-05T07:28:26.596096+0000","last_became_peered":"2024-02-05T07:28:26.596096+0000","last_unstale":"2024-02-05T07:30:14.717176+0000","last_undegraded":"2024-02-05T07:30:14.717176+0000","last_fullsized":"2024-02-05T07:30:14.717176+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:44.015468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:29:44.015468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:08.715933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":194,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716495+0000","last_change":"2024-02-05T07:27:16.831397+0000","last_active":"2024-02-05T07:30:14.716495+0000","last_peered":"2024-02-05T07:30:14.716495+0000","last_clean":"2024-02-05T07:30:14.716495+0000","last_became_active":"2024-02-05T07:27:16.829863+0000","last_became_peered":"2024-02-05T07:27:16.829863+0000","last_unstale":"2024-02-05T07:30:14.716495+0000","last_undegraded":"2024-02-05T07:30:14.716495+0000","last_fullsized":"2024-02-05T07:30:14.716495+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:52.847946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:52.847946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190615+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716963+0000","last_change":"2024-02-05T07:30:01.030294+0000","last_active":"2024-02-05T07:30:14.716963+0000","last_peered":"2024-02-05T07:30:14.716963+0000","last_clean":"2024-02-05T07:30:14.716963+0000","last_became_active":"2024-02-05T07:30:01.029966+0000","last_became_peered":"2024-02-05T07:30:01.029966+0000","last_unstale":"2024-02-05T07:30:14.716963+0000","last_undegraded":"2024-02-05T07:30:14.716963+0000","last_fullsized":"2024-02-05T07:30:14.716963+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.884581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":445,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716647+0000","last_change":"2024-02-05T07:28:49.521571+0000","last_active":"2024-02-05T07:30:14.716647+0000","last_peered":"2024-02-05T07:30:14.716647+0000","last_clean":"2024-02-05T07:30:14.716647+0000","last_became_active":"2024-02-05T07:28:26.624868+0000","last_became_peered":"2024-02-05T07:28:26.624868+0000","last_unstale":"2024-02-05T07:30:14.716647+0000","last_undegraded":"2024-02-05T07:30:14.716647+0000","last_fullsized":"2024-02-05T07:30:14.716647+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:49.521450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:49.521450+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:18.611027+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":444,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716646+0000","last_change":"2024-02-05T07:28:26.590000+0000","last_active":"2024-02-05T07:30:14.716646+0000","last_peered":"2024-02-05T07:30:14.716646+0000","last_clean":"2024-02-05T07:30:14.716646+0000","last_became_active":"2024-02-05T07:28:26.589673+0000","last_became_peered":"2024-02-05T07:28:26.589673+0000","last_unstale":"2024-02-05T07:30:14.716646+0000","last_undegraded":"2024-02-05T07:30:14.716646+0000","last_fullsized":"2024-02-05T07:30:14.716646+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:30.535958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:25:30.535958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"326'29","reported_seq":299,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717709+0000","last_change":"2024-02-05T07:27:03.398624+0000","last_active":"2024-02-05T07:30:14.717709+0000","last_peered":"2024-02-05T07:30:14.717709+0000","last_clean":"2024-02-05T07:30:14.717709+0000","last_became_active":"2024-02-05T07:27:03.382876+0000","last_became_peered":"2024-02-05T07:27:03.382876+0000","last_unstale":"2024-02-05T07:30:14.717709+0000","last_undegraded":"2024-02-05T07:30:14.717709+0000","last_fullsized":"2024-02-05T07:30:14.717709+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:26:20.705363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:20.705363+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":628,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481845+0000","last_change":"2024-02-05T07:29:53.699733+0000","last_active":"2024-02-05T07:30:14.481845+0000","last_peered":"2024-02-05T07:30:14.481845+0000","last_clean":"2024-02-05T07:30:14.481845+0000","last_became_active":"2024-02-05T07:27:03.361028+0000","last_became_peered":"2024-02-05T07:27:03.361028+0000","last_unstale":"2024-02-05T07:30:14.481845+0000","last_undegraded":"2024-02-05T07:30:14.481845+0000","last_fullsized":"2024-02-05T07:30:14.481845+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.699651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.699651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:06.234119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":611,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.480622+0000","last_change":"2024-02-05T07:28:49.555346+0000","last_active":"2024-02-05T07:30:14.480622+0000","last_peered":"2024-02-05T07:30:14.480622+0000","last_clean":"2024-02-05T07:30:14.480622+0000","last_became_active":"2024-02-05T07:28:25.840850+0000","last_became_peered":"2024-02-05T07:28:25.840850+0000","last_unstale":"2024-02-05T07:30:14.480622+0000","last_undegraded":"2024-02-05T07:30:14.480622+0000","last_fullsized":"2024-02-05T07:30:14.480622+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:49.555018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:49.555018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744399+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"355'463","reported_seq":485,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:18.616414+0000","last_change":"2024-02-05T07:30:00.869708+0000","last_active":"2024-02-05T07:30:18.616414+0000","last_peered":"2024-02-05T07:30:18.616414+0000","last_clean":"2024-02-05T07:30:18.616414+0000","last_became_active":"2024-02-05T07:30:00.869295+0000","last_became_peered":"2024-02-05T07:30:00.869295+0000","last_unstale":"2024-02-05T07:30:18.616414+0000","last_undegraded":"2024-02-05T07:30:18.616414+0000","last_fullsized":"2024-02-05T07:30:18.616414+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:11.435954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30343168,"num_objects":463,"num_object_clones":0,"num_object_copies":1389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":463,"num_write_kb":29632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":590,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481624+0000","last_change":"2024-02-05T07:28:47.605953+0000","last_active":"2024-02-05T07:30:14.481624+0000","last_peered":"2024-02-05T07:30:14.481624+0000","last_clean":"2024-02-05T07:30:14.481624+0000","last_became_active":"2024-02-05T07:28:25.756613+0000","last_became_peered":"2024-02-05T07:28:25.756613+0000","last_unstale":"2024-02-05T07:30:14.481624+0000","last_undegraded":"2024-02-05T07:30:14.481624+0000","last_fullsized":"2024-02-05T07:30:14.481624+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:47.605803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.605803+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744368+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":211,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482274+0000","last_change":"2024-02-05T07:27:03.370712+0000","last_active":"2024-02-05T07:30:14.482274+0000","last_peered":"2024-02-05T07:30:14.482274+0000","last_clean":"2024-02-05T07:30:14.482274+0000","last_became_active":"2024-02-05T07:27:03.370297+0000","last_became_peered":"2024-02-05T07:27:03.370297+0000","last_unstale":"2024-02-05T07:30:14.482274+0000","last_undegraded":"2024-02-05T07:30:14.482274+0000","last_fullsized":"2024-02-05T07:30:14.482274+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:32.652537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:32.652537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"326'24","reported_seq":421,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482747+0000","last_change":"2024-02-05T07:29:36.583093+0000","last_active":"2024-02-05T07:30:14.482747+0000","last_peered":"2024-02-05T07:30:14.482747+0000","last_clean":"2024-02-05T07:30:14.482747+0000","last_became_active":"2024-02-05T07:24:34.209716+0000","last_became_peered":"2024-02-05T07:24:34.209716+0000","last_unstale":"2024-02-05T07:30:14.482747+0000","last_undegraded":"2024-02-05T07:30:14.482747+0000","last_fullsized":"2024-02-05T07:30:14.482747+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:29:36.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:36.583005+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:41.205092+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1302,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716215+0000","last_change":"2024-02-05T07:29:51.291231+0000","last_active":"2024-02-05T07:30:14.716215+0000","last_peered":"2024-02-05T07:30:14.716215+0000","last_clean":"2024-02-05T07:30:14.716215+0000","last_became_active":"2024-02-05T07:18:48.424620+0000","last_became_peered":"2024-02-05T07:18:48.424620+0000","last_unstale":"2024-02-05T07:30:14.716215+0000","last_undegraded":"2024-02-05T07:30:14.716215+0000","last_fullsized":"2024-02-05T07:30:14.716215+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:29:51.291146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.291146+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:11.742257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":677,"num_read_kb":672,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":585,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481315+0000","last_change":"2024-02-05T07:27:16.724562+0000","last_active":"2024-02-05T07:30:14.481315+0000","last_peered":"2024-02-05T07:30:14.481315+0000","last_clean":"2024-02-05T07:30:14.481315+0000","last_became_active":"2024-02-05T07:27:16.724280+0000","last_became_peered":"2024-02-05T07:27:16.724280+0000","last_unstale":"2024-02-05T07:30:14.481315+0000","last_undegraded":"2024-02-05T07:30:14.481315+0000","last_fullsized":"2024-02-05T07:30:14.481315+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:30.351655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:26:30.351655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906859+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":639,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481644+0000","last_change":"2024-02-05T07:28:50.545548+0000","last_active":"2024-02-05T07:30:14.481644+0000","last_peered":"2024-02-05T07:30:14.481644+0000","last_clean":"2024-02-05T07:30:14.481644+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:30:14.481644+0000","last_undegraded":"2024-02-05T07:30:14.481644+0000","last_fullsized":"2024-02-05T07:30:14.481644+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:28:50.545437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:50.545437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744471+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.715685+0000","last_change":"2024-02-05T07:30:00.869079+0000","last_active":"2024-02-05T07:30:14.715685+0000","last_peered":"2024-02-05T07:30:14.715685+0000","last_clean":"2024-02-05T07:30:14.715685+0000","last_became_active":"2024-02-05T07:30:00.868747+0000","last_became_peered":"2024-02-05T07:30:00.868747+0000","last_unstale":"2024-02-05T07:30:14.715685+0000","last_undegraded":"2024-02-05T07:30:14.715685+0000","last_fullsized":"2024-02-05T07:30:14.715685+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:13.582746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":603,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.718228+0000","last_change":"2024-02-05T07:27:03.363357+0000","last_active":"2024-02-05T07:30:14.718228+0000","last_peered":"2024-02-05T07:30:14.718228+0000","last_clean":"2024-02-05T07:30:14.718228+0000","last_became_active":"2024-02-05T07:27:03.363025+0000","last_became_peered":"2024-02-05T07:27:03.363025+0000","last_unstale":"2024-02-05T07:30:14.718228+0000","last_undegraded":"2024-02-05T07:30:14.718228+0000","last_fullsized":"2024-02-05T07:30:14.718228+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:35.850761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:35.850761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102260+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"326'30","reported_seq":606,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716830+0000","last_change":"2024-02-05T07:28:25.868774+0000","last_active":"2024-02-05T07:30:14.716830+0000","last_peered":"2024-02-05T07:30:14.716830+0000","last_clean":"2024-02-05T07:30:14.716830+0000","last_became_active":"2024-02-05T07:28:25.868358+0000","last_became_peered":"2024-02-05T07:28:25.868358+0000","last_unstale":"2024-02-05T07:30:14.716830+0000","last_undegraded":"2024-02-05T07:30:14.716830+0000","last_fullsized":"2024-02-05T07:30:14.716830+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-02-05T07:25:46.826417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:46.826417+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":205,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716655+0000","last_change":"2024-02-05T07:29:43.000854+0000","last_active":"2024-02-05T07:30:14.716655+0000","last_peered":"2024-02-05T07:30:14.716655+0000","last_clean":"2024-02-05T07:30:14.716655+0000","last_became_active":"2024-02-05T07:28:26.595640+0000","last_became_peered":"2024-02-05T07:28:26.595640+0000","last_unstale":"2024-02-05T07:30:14.716655+0000","last_undegraded":"2024-02-05T07:30:14.716655+0000","last_fullsized":"2024-02-05T07:30:14.716655+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:43.000778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:43.000778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:58.850387+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":902,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716332+0000","last_change":"2024-02-05T07:28:47.119137+0000","last_active":"2024-02-05T07:30:14.716332+0000","last_peered":"2024-02-05T07:30:14.716332+0000","last_clean":"2024-02-05T07:30:14.716332+0000","last_became_active":"2024-02-05T07:28:25.788899+0000","last_became_peered":"2024-02-05T07:28:25.788899+0000","last_unstale":"2024-02-05T07:30:14.716332+0000","last_undegraded":"2024-02-05T07:30:14.716332+0000","last_fullsized":"2024-02-05T07:30:14.716332+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:47.118901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.118901+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":194,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717775+0000","last_change":"2024-02-05T07:27:16.751712+0000","last_active":"2024-02-05T07:30:14.717775+0000","last_peered":"2024-02-05T07:30:14.717775+0000","last_clean":"2024-02-05T07:30:14.717775+0000","last_became_active":"2024-02-05T07:27:16.751341+0000","last_became_peered":"2024-02-05T07:27:16.751341+0000","last_unstale":"2024-02-05T07:30:14.717775+0000","last_undegraded":"2024-02-05T07:30:14.717775+0000","last_fullsized":"2024-02-05T07:30:14.717775+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:36.753098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.753098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.480762+0000","last_change":"2024-02-05T07:30:00.510569+0000","last_active":"2024-02-05T07:30:14.480762+0000","last_peered":"2024-02-05T07:30:14.480762+0000","last_clean":"2024-02-05T07:30:14.480762+0000","last_became_active":"2024-02-05T07:30:00.509652+0000","last_became_peered":"2024-02-05T07:30:00.509652+0000","last_unstale":"2024-02-05T07:30:14.480762+0000","last_undegraded":"2024-02-05T07:30:14.480762+0000","last_fullsized":"2024-02-05T07:30:14.480762+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:01.265428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":904,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717679+0000","last_change":"2024-02-05T07:28:46.132405+0000","last_active":"2024-02-05T07:30:14.717679+0000","last_peered":"2024-02-05T07:30:14.717679+0000","last_clean":"2024-02-05T07:30:14.717679+0000","last_became_active":"2024-02-05T07:28:14.294305+0000","last_became_peered":"2024-02-05T07:28:14.294305+0000","last_unstale":"2024-02-05T07:30:14.717679+0000","last_undegraded":"2024-02-05T07:30:14.717679+0000","last_fullsized":"2024-02-05T07:30:14.717679+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:46.132295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:46.132295+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":619,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717284+0000","last_change":"2024-02-05T07:28:40.103698+0000","last_active":"2024-02-05T07:30:14.717284+0000","last_peered":"2024-02-05T07:30:14.717284+0000","last_clean":"2024-02-05T07:30:14.717284+0000","last_became_active":"2024-02-05T07:27:03.357849+0000","last_became_peered":"2024-02-05T07:27:03.357849+0000","last_unstale":"2024-02-05T07:30:14.717284+0000","last_undegraded":"2024-02-05T07:30:14.717284+0000","last_fullsized":"2024-02-05T07:30:14.717284+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:40.103572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:40.103572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"326'22","reported_seq":666,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482047+0000","last_change":"2024-02-05T07:27:03.383286+0000","last_active":"2024-02-05T07:30:14.482047+0000","last_peered":"2024-02-05T07:30:14.482047+0000","last_clean":"2024-02-05T07:30:14.482047+0000","last_became_active":"2024-02-05T07:27:03.360661+0000","last_became_peered":"2024-02-05T07:27:03.360661+0000","last_unstale":"2024-02-05T07:30:14.482047+0000","last_undegraded":"2024-02-05T07:30:14.482047+0000","last_fullsized":"2024-02-05T07:30:14.482047+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'18","last_scrub_stamp":"2024-02-05T07:26:24.578301+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.578301+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744239+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":650,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716014+0000","last_change":"2024-02-05T07:29:42.383585+0000","last_active":"2024-02-05T07:30:14.716014+0000","last_peered":"2024-02-05T07:30:14.716014+0000","last_clean":"2024-02-05T07:30:14.716014+0000","last_became_active":"2024-02-05T07:27:16.728869+0000","last_became_peered":"2024-02-05T07:27:16.728869+0000","last_unstale":"2024-02-05T07:30:14.716014+0000","last_undegraded":"2024-02-05T07:30:14.716014+0000","last_fullsized":"2024-02-05T07:30:14.716014+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:42.383469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:29:42.383469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:06.897824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":637,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.480426+0000","last_change":"2024-02-05T07:29:45.546235+0000","last_active":"2024-02-05T07:30:14.480426+0000","last_peered":"2024-02-05T07:30:14.480426+0000","last_clean":"2024-02-05T07:30:14.480426+0000","last_became_active":"2024-02-05T07:28:25.750373+0000","last_became_peered":"2024-02-05T07:28:25.750373+0000","last_unstale":"2024-02-05T07:30:14.480426+0000","last_undegraded":"2024-02-05T07:30:14.480426+0000","last_fullsized":"2024-02-05T07:30:14.480426+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:45.546152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.546152+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:52.093292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":620,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.718343+0000","last_change":"2024-02-05T07:29:38.191933+0000","last_active":"2024-02-05T07:30:14.718343+0000","last_peered":"2024-02-05T07:30:14.718343+0000","last_clean":"2024-02-05T07:30:14.718343+0000","last_became_active":"2024-02-05T07:28:26.593644+0000","last_became_peered":"2024-02-05T07:28:26.593644+0000","last_unstale":"2024-02-05T07:30:14.718343+0000","last_undegraded":"2024-02-05T07:30:14.718343+0000","last_fullsized":"2024-02-05T07:30:14.718343+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:38.191811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:29:38.191811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:05.792018+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717257+0000","last_change":"2024-02-05T07:30:00.869947+0000","last_active":"2024-02-05T07:30:14.717257+0000","last_peered":"2024-02-05T07:30:14.717257+0000","last_clean":"2024-02-05T07:30:14.717257+0000","last_became_active":"2024-02-05T07:30:00.869532+0000","last_became_peered":"2024-02-05T07:30:00.869532+0000","last_unstale":"2024-02-05T07:30:14.717257+0000","last_undegraded":"2024-02-05T07:30:14.717257+0000","last_fullsized":"2024-02-05T07:30:14.717257+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.927010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":608,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.715773+0000","last_change":"2024-02-05T07:29:55.246298+0000","last_active":"2024-02-05T07:30:14.715773+0000","last_peered":"2024-02-05T07:30:14.715773+0000","last_clean":"2024-02-05T07:30:14.715773+0000","last_became_active":"2024-02-05T07:28:14.305034+0000","last_became_peered":"2024-02-05T07:28:14.305034+0000","last_unstale":"2024-02-05T07:30:14.715773+0000","last_undegraded":"2024-02-05T07:30:14.715773+0000","last_fullsized":"2024-02-05T07:30:14.715773+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:29:55.246214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.246214+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.223711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":366,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482626+0000","last_change":"2024-02-05T07:29:48.627198+0000","last_active":"2024-02-05T07:30:14.482626+0000","last_peered":"2024-02-05T07:30:14.482626+0000","last_clean":"2024-02-05T07:30:14.482626+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:30:14.482626+0000","last_undegraded":"2024-02-05T07:30:14.482626+0000","last_fullsized":"2024-02-05T07:30:14.482626+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.627107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.627107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:56.222054+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"326'44","reported_seq":495,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716457+0000","last_change":"2024-02-05T07:28:31.177038+0000","last_active":"2024-02-05T07:30:14.716457+0000","last_peered":"2024-02-05T07:30:14.716457+0000","last_clean":"2024-02-05T07:30:14.716457+0000","last_became_active":"2024-02-05T07:27:16.708475+0000","last_became_peered":"2024-02-05T07:27:16.708475+0000","last_unstale":"2024-02-05T07:30:14.716457+0000","last_undegraded":"2024-02-05T07:30:14.716457+0000","last_fullsized":"2024-02-05T07:30:14.716457+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'32","last_scrub_stamp":"2024-02-05T07:28:31.176950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:28:31.176950+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":622,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481447+0000","last_change":"2024-02-05T07:28:25.779148+0000","last_active":"2024-02-05T07:30:14.481447+0000","last_peered":"2024-02-05T07:30:14.481447+0000","last_clean":"2024-02-05T07:30:14.481447+0000","last_became_active":"2024-02-05T07:28:25.778852+0000","last_became_peered":"2024-02-05T07:28:25.778852+0000","last_unstale":"2024-02-05T07:30:14.481447+0000","last_undegraded":"2024-02-05T07:30:14.481447+0000","last_fullsized":"2024-02-05T07:30:14.481447+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:53.185503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:53.185503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906647+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":536,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481055+0000","last_change":"2024-02-05T07:28:55.477529+0000","last_active":"2024-02-05T07:30:14.481055+0000","last_peered":"2024-02-05T07:30:14.481055+0000","last_clean":"2024-02-05T07:30:14.481055+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:30:14.481055+0000","last_undegraded":"2024-02-05T07:30:14.481055+0000","last_fullsized":"2024-02-05T07:30:14.481055+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:28:55.477280+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744505+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":202,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717786+0000","last_change":"2024-02-05T07:28:42.075567+0000","last_active":"2024-02-05T07:30:14.717786+0000","last_peered":"2024-02-05T07:30:14.717786+0000","last_clean":"2024-02-05T07:30:14.717786+0000","last_became_active":"2024-02-05T07:27:16.714924+0000","last_became_peered":"2024-02-05T07:27:16.714924+0000","last_unstale":"2024-02-05T07:30:14.717786+0000","last_undegraded":"2024-02-05T07:30:14.717786+0000","last_fullsized":"2024-02-05T07:30:14.717786+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:42.075445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:42.075445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190950+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716965+0000","last_change":"2024-02-05T07:30:00.512361+0000","last_active":"2024-02-05T07:30:14.716965+0000","last_peered":"2024-02-05T07:30:14.716965+0000","last_clean":"2024-02-05T07:30:14.716965+0000","last_became_active":"2024-02-05T07:30:00.511728+0000","last_became_peered":"2024-02-05T07:30:00.511728+0000","last_unstale":"2024-02-05T07:30:14.716965+0000","last_undegraded":"2024-02-05T07:30:14.716965+0000","last_fullsized":"2024-02-05T07:30:14.716965+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:24.957975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":517,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481762+0000","last_change":"2024-02-05T07:28:40.616425+0000","last_active":"2024-02-05T07:30:14.481762+0000","last_peered":"2024-02-05T07:30:14.481762+0000","last_clean":"2024-02-05T07:30:14.481762+0000","last_became_active":"2024-02-05T07:28:19.302415+0000","last_became_peered":"2024-02-05T07:28:19.302415+0000","last_unstale":"2024-02-05T07:30:14.481762+0000","last_undegraded":"2024-02-05T07:30:14.481762+0000","last_fullsized":"2024-02-05T07:30:14.481762+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:40.616302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:40.616302+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"326'15","reported_seq":409,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481972+0000","last_change":"2024-02-05T07:29:51.665907+0000","last_active":"2024-02-05T07:30:14.481972+0000","last_peered":"2024-02-05T07:30:14.481972+0000","last_clean":"2024-02-05T07:30:14.481972+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:30:14.481972+0000","last_undegraded":"2024-02-05T07:30:14.481972+0000","last_fullsized":"2024-02-05T07:30:14.481972+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:29:51.665808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.665808+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:13.278749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":197,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717662+0000","last_change":"2024-02-05T07:28:26.602895+0000","last_active":"2024-02-05T07:30:14.717662+0000","last_peered":"2024-02-05T07:30:14.717662+0000","last_clean":"2024-02-05T07:30:14.717662+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:30:14.717662+0000","last_undegraded":"2024-02-05T07:30:14.717662+0000","last_fullsized":"2024-02-05T07:30:14.717662+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:38.539777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:38.539777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":625,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716242+0000","last_change":"2024-02-05T07:28:41.103849+0000","last_active":"2024-02-05T07:30:14.716242+0000","last_peered":"2024-02-05T07:30:14.716242+0000","last_clean":"2024-02-05T07:30:14.716242+0000","last_became_active":"2024-02-05T07:27:16.708205+0000","last_became_peered":"2024-02-05T07:27:16.708205+0000","last_unstale":"2024-02-05T07:30:14.716242+0000","last_undegraded":"2024-02-05T07:30:14.716242+0000","last_fullsized":"2024-02-05T07:30:14.716242+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:41.103739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:41.103739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":358,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716764+0000","last_change":"2024-02-05T07:26:30.076326+0000","last_active":"2024-02-05T07:30:14.716764+0000","last_peered":"2024-02-05T07:30:14.716764+0000","last_clean":"2024-02-05T07:30:14.716764+0000","last_became_active":"2024-02-05T07:24:34.235899+0000","last_became_peered":"2024-02-05T07:24:34.235899+0000","last_unstale":"2024-02-05T07:30:14.716764+0000","last_undegraded":"2024-02-05T07:30:14.716764+0000","last_fullsized":"2024-02-05T07:30:14.716764+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:30.076226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:30.076226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102348+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717950+0000","last_change":"2024-02-05T07:30:00.509083+0000","last_active":"2024-02-05T07:30:14.717950+0000","last_peered":"2024-02-05T07:30:14.717950+0000","last_clean":"2024-02-05T07:30:14.717950+0000","last_became_active":"2024-02-05T07:30:00.508712+0000","last_became_peered":"2024-02-05T07:30:00.508712+0000","last_unstale":"2024-02-05T07:30:14.717950+0000","last_undegraded":"2024-02-05T07:30:14.717950+0000","last_fullsized":"2024-02-05T07:30:14.717950+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:01.622332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":446,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717223+0000","last_change":"2024-02-05T07:28:47.560599+0000","last_active":"2024-02-05T07:30:14.717223+0000","last_peered":"2024-02-05T07:30:14.717223+0000","last_clean":"2024-02-05T07:30:14.717223+0000","last_became_active":"2024-02-05T07:28:26.596711+0000","last_became_peered":"2024-02-05T07:28:26.596711+0000","last_unstale":"2024-02-05T07:30:14.717223+0000","last_undegraded":"2024-02-05T07:30:14.717223+0000","last_fullsized":"2024-02-05T07:30:14.717223+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:47.560475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.560475+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"326'35","reported_seq":475,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716730+0000","last_change":"2024-02-05T07:29:45.981636+0000","last_active":"2024-02-05T07:30:14.716730+0000","last_peered":"2024-02-05T07:30:14.716730+0000","last_clean":"2024-02-05T07:30:14.716730+0000","last_became_active":"2024-02-05T07:28:25.791406+0000","last_became_peered":"2024-02-05T07:28:25.791406+0000","last_unstale":"2024-02-05T07:30:14.716730+0000","last_undegraded":"2024-02-05T07:30:14.716730+0000","last_fullsized":"2024-02-05T07:30:14.716730+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:29:45.981520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.981520+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:03.718981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":631,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.718291+0000","last_change":"2024-02-05T07:29:39.170107+0000","last_active":"2024-02-05T07:30:14.718291+0000","last_peered":"2024-02-05T07:30:14.718291+0000","last_clean":"2024-02-05T07:30:14.718291+0000","last_became_active":"2024-02-05T07:28:26.609352+0000","last_became_peered":"2024-02-05T07:28:26.609352+0000","last_unstale":"2024-02-05T07:30:14.718291+0000","last_undegraded":"2024-02-05T07:30:14.718291+0000","last_fullsized":"2024-02-05T07:30:14.718291+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:39.169995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:39.169995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:47.789620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":619,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716820+0000","last_change":"2024-02-05T07:28:55.173750+0000","last_active":"2024-02-05T07:30:14.716820+0000","last_peered":"2024-02-05T07:30:14.716820+0000","last_clean":"2024-02-05T07:30:14.716820+0000","last_became_active":"2024-02-05T07:27:03.355957+0000","last_became_peered":"2024-02-05T07:27:03.355957+0000","last_unstale":"2024-02-05T07:30:14.716820+0000","last_undegraded":"2024-02-05T07:30:14.716820+0000","last_fullsized":"2024-02-05T07:30:14.716820+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:55.173628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:55.173628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":616,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716168+0000","last_change":"2024-02-05T07:29:48.262661+0000","last_active":"2024-02-05T07:30:14.716168+0000","last_peered":"2024-02-05T07:30:14.716168+0000","last_clean":"2024-02-05T07:30:14.716168+0000","last_became_active":"2024-02-05T07:27:16.710267+0000","last_became_peered":"2024-02-05T07:27:16.710267+0000","last_unstale":"2024-02-05T07:30:14.716168+0000","last_undegraded":"2024-02-05T07:30:14.716168+0000","last_fullsized":"2024-02-05T07:30:14.716168+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.262537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:55.590811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481438+0000","last_change":"2024-02-05T07:30:01.030823+0000","last_active":"2024-02-05T07:30:14.481438+0000","last_peered":"2024-02-05T07:30:14.481438+0000","last_clean":"2024-02-05T07:30:14.481438+0000","last_became_active":"2024-02-05T07:30:01.030502+0000","last_became_peered":"2024-02-05T07:30:01.030502+0000","last_unstale":"2024-02-05T07:30:14.481438+0000","last_undegraded":"2024-02-05T07:30:14.481438+0000","last_fullsized":"2024-02-05T07:30:14.481438+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:05.204271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":856,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716834+0000","last_change":"2024-02-05T07:28:40.182489+0000","last_active":"2024-02-05T07:30:14.716834+0000","last_peered":"2024-02-05T07:30:14.716834+0000","last_clean":"2024-02-05T07:30:14.716834+0000","last_became_active":"2024-02-05T07:28:26.597412+0000","last_became_peered":"2024-02-05T07:28:26.597412+0000","last_unstale":"2024-02-05T07:30:14.716834+0000","last_undegraded":"2024-02-05T07:30:14.716834+0000","last_fullsized":"2024-02-05T07:30:14.716834+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:40.181881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:40.181881+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190870+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":621,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.715702+0000","last_change":"2024-02-05T07:29:39.290437+0000","last_active":"2024-02-05T07:30:14.715702+0000","last_peered":"2024-02-05T07:30:14.715702+0000","last_clean":"2024-02-05T07:30:14.715702+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:30:14.715702+0000","last_undegraded":"2024-02-05T07:30:14.715702+0000","last_fullsized":"2024-02-05T07:30:14.715702+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:39.290320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:39.290320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:45.538623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"325'18","reported_seq":709,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481418+0000","last_change":"2024-02-05T07:29:49.022612+0000","last_active":"2024-02-05T07:30:14.481418+0000","last_peered":"2024-02-05T07:30:14.481418+0000","last_clean":"2024-02-05T07:30:14.481418+0000","last_became_active":"2024-02-05T07:27:16.706871+0000","last_became_peered":"2024-02-05T07:27:16.706871+0000","last_unstale":"2024-02-05T07:30:14.481418+0000","last_undegraded":"2024-02-05T07:30:14.481418+0000","last_fullsized":"2024-02-05T07:30:14.481418+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'18","last_scrub_stamp":"2024-02-05T07:29:49.021772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.021772+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:49.051858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":613,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717319+0000","last_change":"2024-02-05T07:26:24.102911+0000","last_active":"2024-02-05T07:30:14.717319+0000","last_peered":"2024-02-05T07:30:14.717319+0000","last_clean":"2024-02-05T07:30:14.717319+0000","last_became_active":"2024-02-05T07:18:49.435621+0000","last_became_peered":"2024-02-05T07:18:49.435621+0000","last_unstale":"2024-02-05T07:30:14.717319+0000","last_undegraded":"2024-02-05T07:30:14.717319+0000","last_fullsized":"2024-02-05T07:30:14.717319+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:24.102597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.102597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102439+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":202,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716975+0000","last_change":"2024-02-05T07:28:48.088461+0000","last_active":"2024-02-05T07:30:14.716975+0000","last_peered":"2024-02-05T07:30:14.716975+0000","last_clean":"2024-02-05T07:30:14.716975+0000","last_became_active":"2024-02-05T07:27:16.764767+0000","last_became_peered":"2024-02-05T07:27:16.764767+0000","last_unstale":"2024-02-05T07:30:14.716975+0000","last_undegraded":"2024-02-05T07:30:14.716975+0000","last_fullsized":"2024-02-05T07:30:14.716975+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:48.088229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.088229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.191031+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":22,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717921+0000","last_change":"2024-02-05T07:30:00.869259+0000","last_active":"2024-02-05T07:30:14.717921+0000","last_peered":"2024-02-05T07:30:14.717921+0000","last_clean":"2024-02-05T07:30:14.717921+0000","last_became_active":"2024-02-05T07:30:00.868882+0000","last_became_peered":"2024-02-05T07:30:00.868882+0000","last_unstale":"2024-02-05T07:30:14.717921+0000","last_undegraded":"2024-02-05T07:30:14.717921+0000","last_fullsized":"2024-02-05T07:30:14.717921+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:23.261947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":441,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.718029+0000","last_change":"2024-02-05T07:28:45.145176+0000","last_active":"2024-02-05T07:30:14.718029+0000","last_peered":"2024-02-05T07:30:14.718029+0000","last_clean":"2024-02-05T07:30:14.718029+0000","last_became_active":"2024-02-05T07:28:19.292332+0000","last_became_peered":"2024-02-05T07:28:19.292332+0000","last_unstale":"2024-02-05T07:30:14.718029+0000","last_undegraded":"2024-02-05T07:30:14.718029+0000","last_fullsized":"2024-02-05T07:30:14.718029+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.145047+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102586+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"302'11","reported_seq":763,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717176+0000","last_change":"2024-02-05T07:28:50.147621+0000","last_active":"2024-02-05T07:30:14.717176+0000","last_peered":"2024-02-05T07:30:14.717176+0000","last_clean":"2024-02-05T07:30:14.717176+0000","last_became_active":"2024-02-05T07:28:25.869896+0000","last_became_peered":"2024-02-05T07:28:25.869896+0000","last_unstale":"2024-02-05T07:30:14.717176+0000","last_undegraded":"2024-02-05T07:30:14.717176+0000","last_fullsized":"2024-02-05T07:30:14.717176+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'11","last_scrub_stamp":"2024-02-05T07:28:50.147503+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:28:50.147503+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102606+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"326'27","reported_seq":301,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716388+0000","last_change":"2024-02-05T07:28:49.179596+0000","last_active":"2024-02-05T07:30:14.716388+0000","last_peered":"2024-02-05T07:30:14.716388+0000","last_clean":"2024-02-05T07:30:14.716388+0000","last_became_active":"2024-02-05T07:28:26.591522+0000","last_became_peered":"2024-02-05T07:28:26.591522+0000","last_unstale":"2024-02-05T07:30:14.716388+0000","last_undegraded":"2024-02-05T07:30:14.716388+0000","last_fullsized":"2024-02-05T07:30:14.716388+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:28:49.179472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:28:49.179472+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102549+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":625,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.718127+0000","last_change":"2024-02-05T07:26:37.022248+0000","last_active":"2024-02-05T07:30:14.718127+0000","last_peered":"2024-02-05T07:30:14.718127+0000","last_clean":"2024-02-05T07:30:14.718127+0000","last_became_active":"2024-02-05T07:18:49.437663+0000","last_became_peered":"2024-02-05T07:18:49.437663+0000","last_unstale":"2024-02-05T07:30:14.718127+0000","last_undegraded":"2024-02-05T07:30:14.718127+0000","last_fullsized":"2024-02-05T07:30:14.718127+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:26:37.022129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:37.022129+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102420+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":274,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716564+0000","last_change":"2024-02-05T07:28:26.589968+0000","last_active":"2024-02-05T07:30:14.716564+0000","last_peered":"2024-02-05T07:30:14.716564+0000","last_clean":"2024-02-05T07:30:14.716564+0000","last_became_active":"2024-02-05T07:28:26.589671+0000","last_became_peered":"2024-02-05T07:28:26.589671+0000","last_unstale":"2024-02-05T07:30:14.716564+0000","last_undegraded":"2024-02-05T07:30:14.716564+0000","last_fullsized":"2024-02-05T07:30:14.716564+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:34.561259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:37.032056+0000","last_clean_scrub_stamp":"2024-02-05T07:25:34.561259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":448,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.480948+0000","last_change":"2024-02-05T07:28:48.594936+0000","last_active":"2024-02-05T07:30:14.480948+0000","last_peered":"2024-02-05T07:30:14.480948+0000","last_clean":"2024-02-05T07:30:14.480948+0000","last_became_active":"2024-02-05T07:28:28.631649+0000","last_became_peered":"2024-02-05T07:28:28.631649+0000","last_unstale":"2024-02-05T07:30:14.480948+0000","last_undegraded":"2024-02-05T07:30:14.480948+0000","last_fullsized":"2024-02-05T07:30:14.480948+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:48.594280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.594280+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744436+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"326'12","reported_seq":211,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717596+0000","last_change":"2024-02-05T07:27:16.969586+0000","last_active":"2024-02-05T07:30:14.717596+0000","last_peered":"2024-02-05T07:30:14.717596+0000","last_clean":"2024-02-05T07:30:14.717596+0000","last_became_active":"2024-02-05T07:27:16.750507+0000","last_became_peered":"2024-02-05T07:27:16.750507+0000","last_unstale":"2024-02-05T07:30:14.717596+0000","last_undegraded":"2024-02-05T07:30:14.717596+0000","last_fullsized":"2024-02-05T07:30:14.717596+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-02-05T07:26:36.068507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.068507+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":418,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716340+0000","last_change":"2024-02-05T07:28:45.572051+0000","last_active":"2024-02-05T07:30:14.716340+0000","last_peered":"2024-02-05T07:30:14.716340+0000","last_clean":"2024-02-05T07:30:14.716340+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:30:14.716340+0000","last_undegraded":"2024-02-05T07:30:14.716340+0000","last_fullsized":"2024-02-05T07:30:14.716340+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.571917+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.045980+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":197,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717498+0000","last_change":"2024-02-05T07:28:26.599846+0000","last_active":"2024-02-05T07:30:14.717498+0000","last_peered":"2024-02-05T07:30:14.717498+0000","last_clean":"2024-02-05T07:30:14.717498+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:30:14.717498+0000","last_undegraded":"2024-02-05T07:30:14.717498+0000","last_fullsized":"2024-02-05T07:30:14.717498+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:28.567844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:28.567844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":609,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.718147+0000","last_change":"2024-02-05T07:28:55.256192+0000","last_active":"2024-02-05T07:30:14.718147+0000","last_peered":"2024-02-05T07:30:14.718147+0000","last_clean":"2024-02-05T07:30:14.718147+0000","last_became_active":"2024-02-05T07:28:25.791864+0000","last_became_peered":"2024-02-05T07:28:25.791864+0000","last_unstale":"2024-02-05T07:30:14.718147+0000","last_undegraded":"2024-02-05T07:30:14.718147+0000","last_fullsized":"2024-02-05T07:30:14.718147+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:55.256094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:28:55.256094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:24.848234+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":591,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.480868+0000","last_change":"2024-02-05T07:28:45.702694+0000","last_active":"2024-02-05T07:30:14.480868+0000","last_peered":"2024-02-05T07:30:14.480868+0000","last_clean":"2024-02-05T07:30:14.480868+0000","last_became_active":"2024-02-05T07:28:19.311737+0000","last_became_peered":"2024-02-05T07:28:19.311737+0000","last_unstale":"2024-02-05T07:30:14.480868+0000","last_undegraded":"2024-02-05T07:30:14.480868+0000","last_fullsized":"2024-02-05T07:30:14.480868+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:45.702565+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.702565+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.702565+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.162227+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"325'15","reported_seq":213,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717693+0000","last_change":"2024-02-05T07:27:17.009879+0000","last_active":"2024-02-05T07:30:14.717693+0000","last_peered":"2024-02-05T07:30:14.717693+0000","last_clean":"2024-02-05T07:30:14.717693+0000","last_became_active":"2024-02-05T07:27:16.750671+0000","last_became_peered":"2024-02-05T07:27:16.750671+0000","last_unstale":"2024-02-05T07:30:14.717693+0000","last_undegraded":"2024-02-05T07:30:14.717693+0000","last_fullsized":"2024-02-05T07:30:14.717693+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:26:28.037945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:28.037945+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190716+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":621,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716995+0000","last_change":"2024-02-05T07:28:25.868721+0000","last_active":"2024-02-05T07:30:14.716995+0000","last_peered":"2024-02-05T07:30:14.716995+0000","last_clean":"2024-02-05T07:30:14.716995+0000","last_became_active":"2024-02-05T07:28:25.868400+0000","last_became_peered":"2024-02-05T07:28:25.868400+0000","last_unstale":"2024-02-05T07:30:14.716995+0000","last_undegraded":"2024-02-05T07:30:14.716995+0000","last_fullsized":"2024-02-05T07:30:14.716995+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:51.811993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:25:51.811993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102276+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":194,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717689+0000","last_change":"2024-02-05T07:27:16.750873+0000","last_active":"2024-02-05T07:30:14.717689+0000","last_peered":"2024-02-05T07:30:14.717689+0000","last_clean":"2024-02-05T07:30:14.717689+0000","last_became_active":"2024-02-05T07:27:16.749995+0000","last_became_peered":"2024-02-05T07:27:16.749995+0000","last_unstale":"2024-02-05T07:30:14.717689+0000","last_undegraded":"2024-02-05T07:30:14.717689+0000","last_fullsized":"2024-02-05T07:30:14.717689+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:59.450191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:59.450191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":608,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716603+0000","last_change":"2024-02-05T07:28:53.115208+0000","last_active":"2024-02-05T07:30:14.716603+0000","last_peered":"2024-02-05T07:30:14.716603+0000","last_clean":"2024-02-05T07:30:14.716603+0000","last_became_active":"2024-02-05T07:18:55.469044+0000","last_became_peered":"2024-02-05T07:18:55.469044+0000","last_unstale":"2024-02-05T07:30:14.716603+0000","last_undegraded":"2024-02-05T07:30:14.716603+0000","last_fullsized":"2024-02-05T07:30:14.716603+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:53.115096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:28:53.115096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":908,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716057+0000","last_change":"2024-02-05T07:28:43.069683+0000","last_active":"2024-02-05T07:30:14.716057+0000","last_peered":"2024-02-05T07:30:14.716057+0000","last_clean":"2024-02-05T07:30:14.716057+0000","last_became_active":"2024-02-05T07:28:26.591761+0000","last_became_peered":"2024-02-05T07:28:26.591761+0000","last_unstale":"2024-02-05T07:30:14.716057+0000","last_undegraded":"2024-02-05T07:30:14.716057+0000","last_fullsized":"2024-02-05T07:30:14.716057+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:43.069567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:43.069567+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"326'19","reported_seq":616,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.718296+0000","last_change":"2024-02-05T07:28:26.596454+0000","last_active":"2024-02-05T07:30:14.718296+0000","last_peered":"2024-02-05T07:30:14.718296+0000","last_clean":"2024-02-05T07:30:14.718296+0000","last_became_active":"2024-02-05T07:28:26.595964+0000","last_became_peered":"2024-02-05T07:28:26.595964+0000","last_unstale":"2024-02-05T07:30:14.718296+0000","last_undegraded":"2024-02-05T07:30:14.718296+0000","last_fullsized":"2024-02-05T07:30:14.718296+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'13","last_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:25:46.891183+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102295+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":579,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717359+0000","last_change":"2024-02-05T07:29:49.314863+0000","last_active":"2024-02-05T07:30:14.717359+0000","last_peered":"2024-02-05T07:30:14.717359+0000","last_clean":"2024-02-05T07:30:14.717359+0000","last_became_active":"2024-02-05T07:27:16.724154+0000","last_became_peered":"2024-02-05T07:27:16.724154+0000","last_unstale":"2024-02-05T07:30:14.717359+0000","last_undegraded":"2024-02-05T07:30:14.717359+0000","last_fullsized":"2024-02-05T07:30:14.717359+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.314735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.314735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:18.065315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"266'10","reported_seq":234,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717493+0000","last_change":"2024-02-05T07:27:17.013712+0000","last_active":"2024-02-05T07:30:14.717493+0000","last_peered":"2024-02-05T07:30:14.717493+0000","last_clean":"2024-02-05T07:30:14.717493+0000","last_became_active":"2024-02-05T07:27:16.816293+0000","last_became_peered":"2024-02-05T07:27:16.816293+0000","last_unstale":"2024-02-05T07:30:14.717493+0000","last_undegraded":"2024-02-05T07:30:14.717493+0000","last_fullsized":"2024-02-05T07:30:14.717493+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:26:35.091894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.091894+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":606,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482162+0000","last_change":"2024-02-05T07:27:03.366653+0000","last_active":"2024-02-05T07:30:14.482162+0000","last_peered":"2024-02-05T07:30:14.482162+0000","last_clean":"2024-02-05T07:30:14.482162+0000","last_became_active":"2024-02-05T07:27:03.364334+0000","last_became_peered":"2024-02-05T07:27:03.364334+0000","last_unstale":"2024-02-05T07:30:14.482162+0000","last_undegraded":"2024-02-05T07:30:14.482162+0000","last_fullsized":"2024-02-05T07:30:14.482162+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:25.323384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:25.323384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906831+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":448,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716627+0000","last_change":"2024-02-05T07:28:41.013449+0000","last_active":"2024-02-05T07:30:14.716627+0000","last_peered":"2024-02-05T07:30:14.716627+0000","last_clean":"2024-02-05T07:30:14.716627+0000","last_became_active":"2024-02-05T07:28:19.302393+0000","last_became_peered":"2024-02-05T07:28:19.302393+0000","last_unstale":"2024-02-05T07:30:14.716627+0000","last_undegraded":"2024-02-05T07:30:14.716627+0000","last_fullsized":"2024-02-05T07:30:14.716627+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:41.013360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:41.013360+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"326'25","reported_seq":221,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717622+0000","last_change":"2024-02-05T07:27:17.026183+0000","last_active":"2024-02-05T07:30:14.717622+0000","last_peered":"2024-02-05T07:30:14.717622+0000","last_clean":"2024-02-05T07:30:14.717622+0000","last_became_active":"2024-02-05T07:27:16.817057+0000","last_became_peered":"2024-02-05T07:27:16.817057+0000","last_unstale":"2024-02-05T07:30:14.717622+0000","last_undegraded":"2024-02-05T07:30:14.717622+0000","last_fullsized":"2024-02-05T07:30:14.717622+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'15","last_scrub_stamp":"2024-02-05T07:25:47.927943+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.927943+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":612,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717581+0000","last_change":"2024-02-05T07:29:46.210262+0000","last_active":"2024-02-05T07:30:14.717581+0000","last_peered":"2024-02-05T07:30:14.717581+0000","last_clean":"2024-02-05T07:30:14.717581+0000","last_became_active":"2024-02-05T07:18:49.436131+0000","last_became_peered":"2024-02-05T07:18:49.436131+0000","last_unstale":"2024-02-05T07:30:14.717581+0000","last_undegraded":"2024-02-05T07:30:14.717581+0000","last_fullsized":"2024-02-05T07:30:14.717581+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:46.210169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:46.210169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:10.460063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"217'9","reported_seq":370,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717211+0000","last_change":"2024-02-05T07:28:39.104196+0000","last_active":"2024-02-05T07:30:14.717211+0000","last_peered":"2024-02-05T07:30:14.717211+0000","last_clean":"2024-02-05T07:30:14.717211+0000","last_became_active":"2024-02-05T07:25:27.246616+0000","last_became_peered":"2024-02-05T07:25:27.246616+0000","last_unstale":"2024-02-05T07:30:14.717211+0000","last_undegraded":"2024-02-05T07:30:14.717211+0000","last_fullsized":"2024-02-05T07:30:14.717211+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9","last_scrub_stamp":"2024-02-05T07:28:39.104087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:39.104087+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":559,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717102+0000","last_change":"2024-02-05T07:27:16.727137+0000","last_active":"2024-02-05T07:30:14.717102+0000","last_peered":"2024-02-05T07:30:14.717102+0000","last_clean":"2024-02-05T07:30:14.717102+0000","last_became_active":"2024-02-05T07:27:16.726934+0000","last_became_peered":"2024-02-05T07:27:16.726934+0000","last_unstale":"2024-02-05T07:30:14.717102+0000","last_undegraded":"2024-02-05T07:30:14.717102+0000","last_fullsized":"2024-02-05T07:30:14.717102+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:27:01.069970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:27:01.069970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102494+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"326'19","reported_seq":666,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481626+0000","last_change":"2024-02-05T07:28:25.780182+0000","last_active":"2024-02-05T07:30:14.481626+0000","last_peered":"2024-02-05T07:30:14.481626+0000","last_clean":"2024-02-05T07:30:14.481626+0000","last_became_active":"2024-02-05T07:28:25.779886+0000","last_became_peered":"2024-02-05T07:28:25.779886+0000","last_unstale":"2024-02-05T07:30:14.481626+0000","last_undegraded":"2024-02-05T07:30:14.481626+0000","last_fullsized":"2024-02-05T07:30:14.481626+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:25:42.192387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.192387+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906617+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":621,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481580+0000","last_change":"2024-02-05T07:29:36.798378+0000","last_active":"2024-02-05T07:30:14.481580+0000","last_peered":"2024-02-05T07:30:14.481580+0000","last_clean":"2024-02-05T07:30:14.481580+0000","last_became_active":"2024-02-05T07:28:25.887058+0000","last_became_peered":"2024-02-05T07:28:25.887058+0000","last_unstale":"2024-02-05T07:30:14.481580+0000","last_undegraded":"2024-02-05T07:30:14.481580+0000","last_fullsized":"2024-02-05T07:30:14.481580+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:36.798282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:29:36.798282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:40.237281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":621,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481314+0000","last_change":"2024-02-05T07:27:17.718683+0000","last_active":"2024-02-05T07:30:14.481314+0000","last_peered":"2024-02-05T07:30:14.481314+0000","last_clean":"2024-02-05T07:30:14.481314+0000","last_became_active":"2024-02-05T07:27:17.716649+0000","last_became_peered":"2024-02-05T07:27:17.716649+0000","last_unstale":"2024-02-05T07:30:14.481314+0000","last_undegraded":"2024-02-05T07:30:14.481314+0000","last_fullsized":"2024-02-05T07:30:14.481314+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.555824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.555824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744175+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":192,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716175+0000","last_change":"2024-02-05T07:27:17.708497+0000","last_active":"2024-02-05T07:30:14.716175+0000","last_peered":"2024-02-05T07:30:14.716175+0000","last_clean":"2024-02-05T07:30:14.716175+0000","last_became_active":"2024-02-05T07:27:17.705910+0000","last_became_peered":"2024-02-05T07:27:17.705910+0000","last_unstale":"2024-02-05T07:30:14.716175+0000","last_undegraded":"2024-02-05T07:30:14.716175+0000","last_fullsized":"2024-02-05T07:30:14.716175+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.120512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190698+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"325'15","reported_seq":241,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.480861+0000","last_change":"2024-02-05T07:29:57.775013+0000","last_active":"2024-02-05T07:30:14.480861+0000","last_peered":"2024-02-05T07:30:14.480861+0000","last_clean":"2024-02-05T07:30:14.480861+0000","last_became_active":"2024-02-05T07:28:26.589705+0000","last_became_peered":"2024-02-05T07:28:26.589705+0000","last_unstale":"2024-02-05T07:30:14.480861+0000","last_undegraded":"2024-02-05T07:30:14.480861+0000","last_fullsized":"2024-02-05T07:30:14.480861+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:29:57.774913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:57.774913+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.777452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":624,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.718194+0000","last_change":"2024-02-05T07:29:53.220944+0000","last_active":"2024-02-05T07:30:14.718194+0000","last_peered":"2024-02-05T07:30:14.718194+0000","last_clean":"2024-02-05T07:30:14.718194+0000","last_became_active":"2024-02-05T07:27:16.707627+0000","last_became_peered":"2024-02-05T07:27:16.707627+0000","last_unstale":"2024-02-05T07:30:14.718194+0000","last_undegraded":"2024-02-05T07:30:14.718194+0000","last_fullsized":"2024-02-05T07:30:14.718194+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.220841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.220841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:06.861122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":194,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717301+0000","last_change":"2024-02-05T07:27:16.765064+0000","last_active":"2024-02-05T07:30:14.717301+0000","last_peered":"2024-02-05T07:30:14.717301+0000","last_clean":"2024-02-05T07:30:14.717301+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:30:14.717301+0000","last_undegraded":"2024-02-05T07:30:14.717301+0000","last_fullsized":"2024-02-05T07:30:14.717301+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:26:31.506641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190797+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":368,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.480022+0000","last_change":"2024-02-05T07:28:26.588019+0000","last_active":"2024-02-05T07:30:14.480022+0000","last_peered":"2024-02-05T07:30:14.480022+0000","last_clean":"2024-02-05T07:30:14.480022+0000","last_became_active":"2024-02-05T07:28:26.587701+0000","last_became_peered":"2024-02-05T07:28:26.587701+0000","last_unstale":"2024-02-05T07:30:14.480022+0000","last_undegraded":"2024-02-05T07:30:14.480022+0000","last_fullsized":"2024-02-05T07:30:14.480022+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:37.616737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:26:37.616737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744332+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":842,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716117+0000","last_change":"2024-02-05T07:28:29.271316+0000","last_active":"2024-02-05T07:30:14.716117+0000","last_peered":"2024-02-05T07:30:14.716117+0000","last_clean":"2024-02-05T07:30:14.716117+0000","last_became_active":"2024-02-05T07:28:18.334429+0000","last_became_peered":"2024-02-05T07:28:18.334429+0000","last_unstale":"2024-02-05T07:30:14.716117+0000","last_undegraded":"2024-02-05T07:30:14.716117+0000","last_fullsized":"2024-02-05T07:30:14.716117+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:29.271046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:29.271046+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"326'19","reported_seq":221,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716380+0000","last_change":"2024-02-05T07:28:26.597041+0000","last_active":"2024-02-05T07:30:14.716380+0000","last_peered":"2024-02-05T07:30:14.716380+0000","last_clean":"2024-02-05T07:30:14.716380+0000","last_became_active":"2024-02-05T07:28:26.596304+0000","last_became_peered":"2024-02-05T07:28:26.596304+0000","last_unstale":"2024-02-05T07:30:14.716380+0000","last_undegraded":"2024-02-05T07:30:14.716380+0000","last_fullsized":"2024-02-05T07:30:14.716380+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:25:42.761130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.761130+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190633+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":217,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717240+0000","last_change":"2024-02-05T07:27:16.707735+0000","last_active":"2024-02-05T07:30:14.717240+0000","last_peered":"2024-02-05T07:30:14.717240+0000","last_clean":"2024-02-05T07:30:14.717240+0000","last_became_active":"2024-02-05T07:27:16.707402+0000","last_became_peered":"2024-02-05T07:27:16.707402+0000","last_unstale":"2024-02-05T07:30:14.717240+0000","last_undegraded":"2024-02-05T07:30:14.717240+0000","last_fullsized":"2024-02-05T07:30:14.717240+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:16.644911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:16.644911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":621,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716491+0000","last_change":"2024-02-05T07:29:58.191907+0000","last_active":"2024-02-05T07:30:14.716491+0000","last_peered":"2024-02-05T07:30:14.716491+0000","last_clean":"2024-02-05T07:30:14.716491+0000","last_became_active":"2024-02-05T07:28:25.789309+0000","last_became_peered":"2024-02-05T07:28:25.789309+0000","last_unstale":"2024-02-05T07:30:14.716491+0000","last_undegraded":"2024-02-05T07:30:14.716491+0000","last_fullsized":"2024-02-05T07:30:14.716491+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:29:58.191815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:02.876214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":206,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716874+0000","last_change":"2024-02-05T07:29:40.983327+0000","last_active":"2024-02-05T07:30:14.716874+0000","last_peered":"2024-02-05T07:30:14.716874+0000","last_clean":"2024-02-05T07:30:14.716874+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:30:14.716874+0000","last_undegraded":"2024-02-05T07:30:14.716874+0000","last_fullsized":"2024-02-05T07:30:14.716874+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:40.983227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.983227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:04.158002+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"325'8","reported_seq":218,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716411+0000","last_change":"2024-02-05T07:29:53.958469+0000","last_active":"2024-02-05T07:30:14.716411+0000","last_peered":"2024-02-05T07:30:14.716411+0000","last_clean":"2024-02-05T07:30:14.716411+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:30:14.716411+0000","last_undegraded":"2024-02-05T07:30:14.716411+0000","last_fullsized":"2024-02-05T07:30:14.716411+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:29:53.958384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.958384+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:10.373415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":205,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717356+0000","last_change":"2024-02-05T07:28:46.056835+0000","last_active":"2024-02-05T07:30:14.717356+0000","last_peered":"2024-02-05T07:30:14.717356+0000","last_clean":"2024-02-05T07:30:14.717356+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:30:14.717356+0000","last_undegraded":"2024-02-05T07:30:14.717356+0000","last_fullsized":"2024-02-05T07:30:14.717356+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:46.056702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:28:46.056702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":454,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716217+0000","last_change":"2024-02-05T07:29:58.034675+0000","last_active":"2024-02-05T07:30:14.716217+0000","last_peered":"2024-02-05T07:30:14.716217+0000","last_clean":"2024-02-05T07:30:14.716217+0000","last_became_active":"2024-02-05T07:28:25.840270+0000","last_became_peered":"2024-02-05T07:28:25.840270+0000","last_unstale":"2024-02-05T07:30:14.716217+0000","last_undegraded":"2024-02-05T07:30:14.716217+0000","last_fullsized":"2024-02-05T07:30:14.716217+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:58.034570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:29:58.034570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:16.175688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":600,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717773+0000","last_change":"2024-02-05T07:26:27.014044+0000","last_active":"2024-02-05T07:30:14.717773+0000","last_peered":"2024-02-05T07:30:14.717773+0000","last_clean":"2024-02-05T07:30:14.717773+0000","last_became_active":"2024-02-05T07:18:55.468086+0000","last_became_peered":"2024-02-05T07:18:55.468086+0000","last_unstale":"2024-02-05T07:30:14.717773+0000","last_undegraded":"2024-02-05T07:30:14.717773+0000","last_fullsized":"2024-02-05T07:30:14.717773+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:27.013942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:27.013942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102365+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"326'19","reported_seq":446,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717373+0000","last_change":"2024-02-05T07:29:46.947670+0000","last_active":"2024-02-05T07:30:14.717373+0000","last_peered":"2024-02-05T07:30:14.717373+0000","last_clean":"2024-02-05T07:30:14.717373+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:30:14.717373+0000","last_undegraded":"2024-02-05T07:30:14.717373+0000","last_fullsized":"2024-02-05T07:30:14.717373+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:29:46.947559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:46.947559+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:51.889423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":202,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716041+0000","last_change":"2024-02-05T07:29:38.000417+0000","last_active":"2024-02-05T07:30:14.716041+0000","last_peered":"2024-02-05T07:30:14.716041+0000","last_clean":"2024-02-05T07:30:14.716041+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:30:14.716041+0000","last_undegraded":"2024-02-05T07:30:14.716041+0000","last_fullsized":"2024-02-05T07:30:14.716041+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:38.000345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:38.000345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:42.161255+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":358,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717399+0000","last_change":"2024-02-05T07:25:30.943037+0000","last_active":"2024-02-05T07:30:14.717399+0000","last_peered":"2024-02-05T07:30:14.717399+0000","last_clean":"2024-02-05T07:30:14.717399+0000","last_became_active":"2024-02-05T07:24:34.240115+0000","last_became_peered":"2024-02-05T07:24:34.240115+0000","last_unstale":"2024-02-05T07:30:14.717399+0000","last_undegraded":"2024-02-05T07:30:14.717399+0000","last_fullsized":"2024-02-05T07:30:14.717399+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:30.942942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:30.942942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":623,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482711+0000","last_change":"2024-02-05T07:29:55.800941+0000","last_active":"2024-02-05T07:30:14.482711+0000","last_peered":"2024-02-05T07:30:14.482711+0000","last_clean":"2024-02-05T07:30:14.482711+0000","last_became_active":"2024-02-05T07:27:03.360468+0000","last_became_peered":"2024-02-05T07:27:03.360468+0000","last_unstale":"2024-02-05T07:30:14.482711+0000","last_undegraded":"2024-02-05T07:30:14.482711+0000","last_fullsized":"2024-02-05T07:30:14.482711+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:55.800824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.800824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:15.924868+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"326'20","reported_seq":433,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717551+0000","last_change":"2024-02-05T07:29:40.154942+0000","last_active":"2024-02-05T07:30:14.717551+0000","last_peered":"2024-02-05T07:30:14.717551+0000","last_clean":"2024-02-05T07:30:14.717551+0000","last_became_active":"2024-02-05T07:27:16.726999+0000","last_became_peered":"2024-02-05T07:27:16.726999+0000","last_unstale":"2024-02-05T07:30:14.717551+0000","last_undegraded":"2024-02-05T07:30:14.717551+0000","last_fullsized":"2024-02-05T07:30:14.717551+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:29:40.154854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.154854+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:53.386629+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":645,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.480004+0000","last_change":"2024-02-05T07:29:52.715347+0000","last_active":"2024-02-05T07:30:14.480004+0000","last_peered":"2024-02-05T07:30:14.480004+0000","last_clean":"2024-02-05T07:30:14.480004+0000","last_became_active":"2024-02-05T07:27:16.769199+0000","last_became_peered":"2024-02-05T07:27:16.769199+0000","last_unstale":"2024-02-05T07:30:14.480004+0000","last_undegraded":"2024-02-05T07:30:14.480004+0000","last_fullsized":"2024-02-05T07:30:14.480004+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:52.715259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:52.715259+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:11.340931+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"302'10","reported_seq":570,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717522+0000","last_change":"2024-02-05T07:29:55.058155+0000","last_active":"2024-02-05T07:30:14.717522+0000","last_peered":"2024-02-05T07:30:14.717522+0000","last_clean":"2024-02-05T07:30:14.717522+0000","last_became_active":"2024-02-05T07:28:25.843199+0000","last_became_peered":"2024-02-05T07:28:25.843199+0000","last_unstale":"2024-02-05T07:30:14.717522+0000","last_undegraded":"2024-02-05T07:30:14.717522+0000","last_fullsized":"2024-02-05T07:30:14.717522+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'10","last_scrub_stamp":"2024-02-05T07:29:55.058040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.058040+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:21.632309+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":622,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.718117+0000","last_change":"2024-02-05T07:27:16.996449+0000","last_active":"2024-02-05T07:30:14.718117+0000","last_peered":"2024-02-05T07:30:14.718117+0000","last_clean":"2024-02-05T07:30:14.718117+0000","last_became_active":"2024-02-05T07:27:16.713368+0000","last_became_peered":"2024-02-05T07:27:16.713368+0000","last_unstale":"2024-02-05T07:30:14.718117+0000","last_undegraded":"2024-02-05T07:30:14.718117+0000","last_fullsized":"2024-02-05T07:30:14.718117+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:26:18.019957+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.019957+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102402+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"326'23","reported_seq":228,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717264+0000","last_change":"2024-02-05T07:28:47.062765+0000","last_active":"2024-02-05T07:30:14.717264+0000","last_peered":"2024-02-05T07:30:14.717264+0000","last_clean":"2024-02-05T07:30:14.717264+0000","last_became_active":"2024-02-05T07:27:16.747998+0000","last_became_peered":"2024-02-05T07:27:16.747998+0000","last_unstale":"2024-02-05T07:30:14.717264+0000","last_undegraded":"2024-02-05T07:30:14.717264+0000","last_fullsized":"2024-02-05T07:30:14.717264+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'17","last_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.062618+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.191016+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":295,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481175+0000","last_change":"2024-02-05T07:26:29.558558+0000","last_active":"2024-02-05T07:30:14.481175+0000","last_peered":"2024-02-05T07:30:14.481175+0000","last_clean":"2024-02-05T07:30:14.481175+0000","last_became_active":"2024-02-05T07:25:30.261037+0000","last_became_peered":"2024-02-05T07:25:30.261037+0000","last_unstale":"2024-02-05T07:30:14.481175+0000","last_undegraded":"2024-02-05T07:30:14.481175+0000","last_fullsized":"2024-02-05T07:30:14.481175+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:29.558353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:29.558353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744204+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":619,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481980+0000","last_change":"2024-02-05T07:30:01.504025+0000","last_active":"2024-02-05T07:30:14.481980+0000","last_peered":"2024-02-05T07:30:14.481980+0000","last_clean":"2024-02-05T07:30:14.481980+0000","last_became_active":"2024-02-05T07:27:16.706777+0000","last_became_peered":"2024-02-05T07:27:16.706777+0000","last_unstale":"2024-02-05T07:30:14.481980+0000","last_undegraded":"2024-02-05T07:30:14.481980+0000","last_fullsized":"2024-02-05T07:30:14.481980+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:34.068335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:34.068335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906355+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":205,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717508+0000","last_change":"2024-02-05T07:28:53.138071+0000","last_active":"2024-02-05T07:30:14.717508+0000","last_peered":"2024-02-05T07:30:14.717508+0000","last_clean":"2024-02-05T07:30:14.717508+0000","last_became_active":"2024-02-05T07:28:26.603067+0000","last_became_peered":"2024-02-05T07:28:26.603067+0000","last_unstale":"2024-02-05T07:30:14.717508+0000","last_undegraded":"2024-02-05T07:30:14.717508+0000","last_fullsized":"2024-02-05T07:30:14.717508+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:53.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:53.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190966+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"326'27","reported_seq":703,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716359+0000","last_change":"2024-02-05T07:29:41.354568+0000","last_active":"2024-02-05T07:30:14.716359+0000","last_peered":"2024-02-05T07:30:14.716359+0000","last_clean":"2024-02-05T07:30:14.716359+0000","last_became_active":"2024-02-05T07:27:16.736296+0000","last_became_peered":"2024-02-05T07:27:16.736296+0000","last_unstale":"2024-02-05T07:30:14.716359+0000","last_undegraded":"2024-02-05T07:30:14.716359+0000","last_fullsized":"2024-02-05T07:30:14.716359+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:29:41.354464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:41.354464+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:56.393313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":358,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481513+0000","last_change":"2024-02-05T07:25:33.059237+0000","last_active":"2024-02-05T07:30:14.481513+0000","last_peered":"2024-02-05T07:30:14.481513+0000","last_clean":"2024-02-05T07:30:14.481513+0000","last_became_active":"2024-02-05T07:24:34.234199+0000","last_became_peered":"2024-02-05T07:24:34.234199+0000","last_unstale":"2024-02-05T07:30:14.481513+0000","last_undegraded":"2024-02-05T07:30:14.481513+0000","last_fullsized":"2024-02-05T07:30:14.481513+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:33.059121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:33.059121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":605,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482583+0000","last_change":"2024-02-05T07:26:21.548894+0000","last_active":"2024-02-05T07:30:14.482583+0000","last_peered":"2024-02-05T07:30:14.482583+0000","last_clean":"2024-02-05T07:30:14.482583+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:30:14.482583+0000","last_undegraded":"2024-02-05T07:30:14.482583+0000","last_fullsized":"2024-02-05T07:30:14.482583+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:21.548785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:21.548785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":627,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482174+0000","last_change":"2024-02-05T07:29:59.002788+0000","last_active":"2024-02-05T07:30:14.482174+0000","last_peered":"2024-02-05T07:30:14.482174+0000","last_clean":"2024-02-05T07:30:14.482174+0000","last_became_active":"2024-02-05T07:28:26.590448+0000","last_became_peered":"2024-02-05T07:28:26.590448+0000","last_unstale":"2024-02-05T07:30:14.482174+0000","last_undegraded":"2024-02-05T07:30:14.482174+0000","last_fullsized":"2024-02-05T07:30:14.482174+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.002677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.002677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:25.104559+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"325'7","reported_seq":645,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481335+0000","last_change":"2024-02-05T07:27:16.970973+0000","last_active":"2024-02-05T07:30:14.481335+0000","last_peered":"2024-02-05T07:30:14.481335+0000","last_clean":"2024-02-05T07:30:14.481335+0000","last_became_active":"2024-02-05T07:27:16.745897+0000","last_became_peered":"2024-02-05T07:27:16.745897+0000","last_unstale":"2024-02-05T07:30:14.481335+0000","last_undegraded":"2024-02-05T07:30:14.481335+0000","last_fullsized":"2024-02-05T07:30:14.481335+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:26:35.253452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.253452+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906752+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":436,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481623+0000","last_change":"2024-02-05T07:28:26.591843+0000","last_active":"2024-02-05T07:30:14.481623+0000","last_peered":"2024-02-05T07:30:14.481623+0000","last_clean":"2024-02-05T07:30:14.481623+0000","last_became_active":"2024-02-05T07:28:26.590322+0000","last_became_peered":"2024-02-05T07:28:26.590322+0000","last_unstale":"2024-02-05T07:30:14.481623+0000","last_undegraded":"2024-02-05T07:30:14.481623+0000","last_fullsized":"2024-02-05T07:30:14.481623+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:35.058925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:35.058925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":604,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717530+0000","last_change":"2024-02-05T07:27:00.183499+0000","last_active":"2024-02-05T07:30:14.717530+0000","last_peered":"2024-02-05T07:30:14.717530+0000","last_clean":"2024-02-05T07:30:14.717530+0000","last_became_active":"2024-02-05T07:18:53.456881+0000","last_became_peered":"2024-02-05T07:18:53.456881+0000","last_unstale":"2024-02-05T07:30:14.717530+0000","last_undegraded":"2024-02-05T07:30:14.717530+0000","last_fullsized":"2024-02-05T07:30:14.717530+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:27:00.183281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:27:00.183281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:15.102460+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":420,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.480734+0000","last_change":"2024-02-05T07:29:50.742042+0000","last_active":"2024-02-05T07:30:14.480734+0000","last_peered":"2024-02-05T07:30:14.480734+0000","last_clean":"2024-02-05T07:30:14.480734+0000","last_became_active":"2024-02-05T07:27:16.769581+0000","last_became_peered":"2024-02-05T07:27:16.769581+0000","last_unstale":"2024-02-05T07:30:14.480734+0000","last_undegraded":"2024-02-05T07:30:14.480734+0000","last_fullsized":"2024-02-05T07:30:14.480734+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:50.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:18.589144+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"325'19","reported_seq":677,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481629+0000","last_change":"2024-02-05T07:30:00.001976+0000","last_active":"2024-02-05T07:30:14.481629+0000","last_peered":"2024-02-05T07:30:14.481629+0000","last_clean":"2024-02-05T07:30:14.481629+0000","last_became_active":"2024-02-05T07:27:16.707036+0000","last_became_peered":"2024-02-05T07:27:16.707036+0000","last_unstale":"2024-02-05T07:30:14.481629+0000","last_undegraded":"2024-02-05T07:30:14.481629+0000","last_fullsized":"2024-02-05T07:30:14.481629+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'19","last_scrub_stamp":"2024-02-05T07:30:00.001863+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:30:00.001863+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:04.904819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":447,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.482054+0000","last_change":"2024-02-05T07:29:53.022701+0000","last_active":"2024-02-05T07:30:14.482054+0000","last_peered":"2024-02-05T07:30:14.482054+0000","last_clean":"2024-02-05T07:30:14.482054+0000","last_became_active":"2024-02-05T07:28:26.616107+0000","last_became_peered":"2024-02-05T07:28:26.616107+0000","last_unstale":"2024-02-05T07:30:14.482054+0000","last_undegraded":"2024-02-05T07:30:14.482054+0000","last_fullsized":"2024-02-05T07:30:14.482054+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.022606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.022606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:18.495044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":219,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717853+0000","last_change":"2024-02-05T07:29:54.207175+0000","last_active":"2024-02-05T07:30:14.717853+0000","last_peered":"2024-02-05T07:30:14.717853+0000","last_clean":"2024-02-05T07:30:14.717853+0000","last_became_active":"2024-02-05T07:27:03.360715+0000","last_became_peered":"2024-02-05T07:27:03.360715+0000","last_unstale":"2024-02-05T07:30:14.717853+0000","last_undegraded":"2024-02-05T07:30:14.717853+0000","last_fullsized":"2024-02-05T07:30:14.717853+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:54.207089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:54.207089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:05.053234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":366,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.480165+0000","last_change":"2024-02-05T07:29:31.484681+0000","last_active":"2024-02-05T07:30:14.480165+0000","last_peered":"2024-02-05T07:30:14.480165+0000","last_clean":"2024-02-05T07:30:14.480165+0000","last_became_active":"2024-02-05T07:24:34.230537+0000","last_became_peered":"2024-02-05T07:24:34.230537+0000","last_unstale":"2024-02-05T07:30:14.480165+0000","last_undegraded":"2024-02-05T07:30:14.480165+0000","last_fullsized":"2024-02-05T07:30:14.480165+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:31.484580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:31.484580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:59.560324+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"326'27","reported_seq":228,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.716528+0000","last_change":"2024-02-05T07:27:17.002461+0000","last_active":"2024-02-05T07:30:14.716528+0000","last_peered":"2024-02-05T07:30:14.716528+0000","last_clean":"2024-02-05T07:30:14.716528+0000","last_became_active":"2024-02-05T07:27:16.751179+0000","last_became_peered":"2024-02-05T07:27:16.751179+0000","last_unstale":"2024-02-05T07:30:14.716528+0000","last_undegraded":"2024-02-05T07:30:14.716528+0000","last_fullsized":"2024-02-05T07:30:14.716528+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:25:38.772812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:38.772812+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":198,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717915+0000","last_change":"2024-02-05T07:28:26.596258+0000","last_active":"2024-02-05T07:30:14.717915+0000","last_peered":"2024-02-05T07:30:14.717915+0000","last_clean":"2024-02-05T07:30:14.717915+0000","last_became_active":"2024-02-05T07:28:26.595627+0000","last_became_peered":"2024-02-05T07:28:26.595627+0000","last_unstale":"2024-02-05T07:30:14.717915+0000","last_undegraded":"2024-02-05T07:30:14.717915+0000","last_fullsized":"2024-02-05T07:30:14.717915+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:50.859511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:50.859511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":621,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481023+0000","last_change":"2024-02-05T07:29:48.045121+0000","last_active":"2024-02-05T07:30:14.481023+0000","last_peered":"2024-02-05T07:30:14.481023+0000","last_clean":"2024-02-05T07:30:14.481023+0000","last_became_active":"2024-02-05T07:28:25.778935+0000","last_became_peered":"2024-02-05T07:28:25.778935+0000","last_unstale":"2024-02-05T07:30:14.481023+0000","last_undegraded":"2024-02-05T07:30:14.481023+0000","last_fullsized":"2024-02-05T07:30:14.481023+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.045013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.045013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:48.765921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":206,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717210+0000","last_change":"2024-02-05T07:28:44.050127+0000","last_active":"2024-02-05T07:30:14.717210+0000","last_peered":"2024-02-05T07:30:14.717210+0000","last_clean":"2024-02-05T07:30:14.717210+0000","last_became_active":"2024-02-05T07:28:26.598232+0000","last_became_peered":"2024-02-05T07:28:26.598232+0000","last_unstale":"2024-02-05T07:30:14.717210+0000","last_undegraded":"2024-02-05T07:30:14.717210+0000","last_fullsized":"2024-02-05T07:30:14.717210+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:44.050019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:44.050019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:19.190846+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"325'15","reported_seq":642,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481118+0000","last_change":"2024-02-05T07:29:33.839515+0000","last_active":"2024-02-05T07:30:14.481118+0000","last_peered":"2024-02-05T07:30:14.481118+0000","last_clean":"2024-02-05T07:30:14.481118+0000","last_became_active":"2024-02-05T07:28:25.777815+0000","last_became_peered":"2024-02-05T07:28:25.777815+0000","last_unstale":"2024-02-05T07:30:14.481118+0000","last_undegraded":"2024-02-05T07:30:14.481118+0000","last_fullsized":"2024-02-05T07:30:14.481118+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:29:33.839402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:33.839402+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:01.714433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":384,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.717733+0000","last_change":"2024-02-05T07:29:50.274850+0000","last_active":"2024-02-05T07:30:14.717733+0000","last_peered":"2024-02-05T07:30:14.717733+0000","last_clean":"2024-02-05T07:30:14.717733+0000","last_became_active":"2024-02-05T07:28:26.588496+0000","last_became_peered":"2024-02-05T07:28:26.588496+0000","last_unstale":"2024-02-05T07:30:14.717733+0000","last_undegraded":"2024-02-05T07:30:14.717733+0000","last_fullsized":"2024-02-05T07:30:14.717733+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.274768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.936836+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":611,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.480615+0000","last_change":"2024-02-05T07:27:03.354044+0000","last_active":"2024-02-05T07:30:14.480615+0000","last_peered":"2024-02-05T07:30:14.480615+0000","last_clean":"2024-02-05T07:30:14.480615+0000","last_became_active":"2024-02-05T07:27:03.353249+0000","last_became_peered":"2024-02-05T07:27:03.353249+0000","last_unstale":"2024-02-05T07:30:14.480615+0000","last_undegraded":"2024-02-05T07:30:14.480615+0000","last_fullsized":"2024-02-05T07:30:14.480615+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:28.575859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:28.575859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:14.744145+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":606,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-05T07:30:14.481373+0000","last_change":"2024-02-05T07:27:03.366854+0000","last_active":"2024-02-05T07:30:14.481373+0000","last_peered":"2024-02-05T07:30:14.481373+0000","last_clean":"2024-02-05T07:30:14.481373+0000","last_became_active":"2024-02-05T07:27:03.366576+0000","last_became_peered":"2024-02-05T07:27:03.366576+0000","last_unstale":"2024-02-05T07:30:14.481373+0000","last_undegraded":"2024-02-05T07:30:14.481373+0000","last_fullsized":"2024-02-05T07:30:14.481373+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:21.317417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:21.317417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:16.906678+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":30343168,"num_objects":463,"num_object_clones":0,"num_object_copies":1389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":463,"num_write_kb":29632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131727360,"data_stored":131727360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":463,"ondisk_log_size":463,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":677,"num_read_kb":672,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":4786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":11436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":737,"ondisk_log_size":737,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037902,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":466364,"kb_used_data":2024,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93278788,"statfs":{"total":95995035648,"available":95517478912,"internally_reserved":0,"allocated":2072576,"data_stored":1138497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":475462090},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":2.1179999999999999,"15min":2.0289999999999999},"min":{"1min":0.255,"5min":0.214,"15min":0.20000000000000001},"max":{"1min":4.9320000000000004,"5min":56.420999999999999,"15min":63.039000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.95699999999999996,"5min":2.1469999999999998,"15min":2.0310000000000001},"min":{"1min":0.41899999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.4160000000000004,"5min":56.433,"15min":65.296000000000006},"last":0.69499999999999995}]},{"osd":1,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":3.5110000000000001,"15min":1.9990000000000001},"min":{"1min":0.33900000000000002,"5min":0.219,"15min":0.219},"max":{"1min":4.766,"5min":57.502000000000002,"15min":57.502000000000002},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.2190000000000001,"5min":3.234,"15min":1.9450000000000001},"min":{"1min":0.28999999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.9489999999999998,"5min":57.479999999999997,"15min":57.479999999999997},"last":1.5569999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.1519999999999999,"15min":1.3100000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.249},"max":{"1min":2.8839999999999999,"5min":12.131,"15min":51.127000000000002},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.2190000000000001,"15min":1.4410000000000001},"min":{"1min":0.35599999999999998,"5min":0.27200000000000002,"15min":0.246},"max":{"1min":1.756,"5min":11.141,"15min":51.113999999999997},"last":1.4179999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":2.4540000000000002,"15min":1.6100000000000001},"min":{"1min":0.41099999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":7.3979999999999997,"5min":56.527999999999999,"15min":58.341999999999999},"last":1.7310000000000001},{"interface":"front","average":{"1min":0.89000000000000001,"5min":2.052,"15min":1.4590000000000001},"min":{"1min":0.46400000000000002,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":4.415,"5min":56.328000000000003,"15min":57.765000000000001},"last":1.679}]},{"osd":4,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.63,"15min":0.54600000000000004},"min":{"1min":0.31900000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.8520000000000001,"5min":4.5999999999999996,"15min":4.8659999999999997},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.75600000000000001,"15min":0.58699999999999997},"min":{"1min":0.32500000000000001,"5min":0.221,"15min":0.221},"max":{"1min":1.595,"5min":6.6219999999999999,"15min":6.6219999999999999},"last":0.40699999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":1,"15min":0.69799999999999995},"min":{"1min":0.32500000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.5669999999999999,"5min":18.890999999999998,"15min":18.890999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.91100000000000003,"15min":0.70299999999999996},"min":{"1min":0.29299999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.8129999999999999,"5min":17.081,"15min":17.081},"last":0.64800000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:29:25 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.8969999999999998,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":0.63}]}]},{"osd":6,"up_from":301,"seq":1292785156120,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":288996,"kb_used_data":112156,"kb_used_omap":5,"kb_used_meta":176826,"kb_avail":93456156,"statfs":{"total":95995035648,"available":95699103744,"internally_reserved":0,"allocated":114847744,"data_stored":113926511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":181070341},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":14.218,"5min":14.218,"15min":14.218},"last":0.51400000000000001},{"interface":"front","average":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":14.175000000000001,"5min":14.175000000000001,"15min":14.175000000000001},"last":0.59299999999999997}]},{"osd":1,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":15.473000000000001,"5min":15.473000000000001,"15min":15.473000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":2.6080000000000001,"5min":2.6080000000000001,"15min":2.6080000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":14.784000000000001,"5min":14.784000000000001,"15min":14.784000000000001},"last":0.56599999999999995}]},{"osd":2,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":2.984,"5min":2.984,"15min":2.984},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":34.689999999999998,"5min":34.689999999999998,"15min":34.689999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":34.646999999999998,"5min":34.646999999999998,"15min":34.646999999999998},"last":0.57799999999999996}]},{"osd":3,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.1890000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":17.600000000000001,"5min":17.600000000000001,"15min":17.600000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":23.425000000000001,"5min":23.425000000000001,"15min":23.425000000000001},"last":0.52200000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.869,"5min":1.869,"15min":1.869},"last":0.40500000000000003}]},{"osd":5,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":0.436}]},{"osd":7,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":5.5949999999999998,"5min":5.5949999999999998,"15min":5.5949999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":84.158000000000001,"5min":84.158000000000001,"15min":84.158000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":5.7720000000000002,"5min":5.7720000000000002,"15min":5.7720000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":85.018000000000001,"5min":85.018000000000001,"15min":85.018000000000001},"last":0.42299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575020,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":465164,"kb_used_data":10560,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93279988,"statfs":{"total":95995035648,"available":95518707712,"internally_reserved":0,"allocated":10813440,"data_stored":9855718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:29:40 2024","interfaces":[{"interface":"back","average":{"1min":2.7330000000000001,"5min":1.7150000000000001,"15min":1.9359999999999999},"min":{"1min":0.28299999999999997,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.09,"5min":15.09,"15min":45.393000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":2.3380000000000001,"5min":1.593,"15min":1.883},"min":{"1min":0.23799999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.207000000000001,"5min":14.207000000000001,"15min":45.031999999999996},"last":0.52700000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":2.0830000000000002,"15min":1.5900000000000001},"min":{"1min":0.26900000000000002,"5min":0.218,"15min":0.20699999999999999},"max":{"1min":15.137,"5min":15.137,"15min":39.259999999999998},"last":0.48099999999999998},{"interface":"front","average":{"1min":2.5870000000000002,"5min":1.98,"15min":1.651},"min":{"1min":0.29899999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.906000000000001,"5min":14.906000000000001,"15min":40.972000000000001},"last":0.377}]},{"osd":3,"last update":"Mon Feb 5 07:29:17 2024","interfaces":[{"interface":"back","average":{"1min":3.0070000000000001,"5min":1.605,"15min":1.452},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.20999999999999999},"max":{"1min":19.829999999999998,"5min":19.829999999999998,"15min":43.823},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.548,"15min":1.488},"min":{"1min":0.30299999999999999,"5min":0.26600000000000001,"15min":0.24199999999999999},"max":{"1min":5.4139999999999997,"5min":9.7880000000000003,"15min":39.314999999999998},"last":0.502}]},{"osd":4,"last update":"Mon Feb 5 07:29:40 2024","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":2.1949999999999998,"15min":1.579},"min":{"1min":0.38800000000000001,"5min":0.34799999999999998,"15min":0.312},"max":{"1min":4.7320000000000002,"5min":61.722000000000001,"15min":61.722000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.853,"5min":2.1629999999999998,"15min":1.6539999999999999},"min":{"1min":0.39200000000000002,"5min":0.311,"15min":0.29399999999999998},"max":{"1min":5.5590000000000002,"5min":61.622999999999998,"15min":61.786000000000001},"last":0.56100000000000005}]},{"osd":5,"last update":"Mon Feb 5 07:30:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":2.4649999999999999,"15min":1.6639999999999999},"min":{"1min":0.34999999999999998,"5min":0.309,"15min":0.309},"max":{"1min":6.319,"5min":106.672,"15min":106.672},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.048,"5min":2.5009999999999999,"15min":1.7709999999999999},"min":{"1min":0.38400000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.0869999999999997,"5min":107.348,"15min":107.348},"last":0.68999999999999995}]},{"osd":6,"last update":"Mon Feb 5 07:29:26 2024","interfaces":[{"interface":"back","average":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":5.5990000000000002,"5min":5.5990000000000002,"15min":5.5990000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":6.3559999999999999},"last":0.57799999999999996}]},{"osd":7,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.5469999999999999,"15min":1.4219999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":5.0519999999999996,"5min":20.309000000000001,"15min":66.703999999999994},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.456,"5min":1.6379999999999999,"15min":1.4370000000000001},"min":{"1min":0.47599999999999998,"5min":0.34000000000000002,"15min":0.33500000000000002},"max":{"1min":5.6539999999999999,"5min":11.263999999999999,"15min":61.594999999999999},"last":0.71899999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738545,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":506952,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93238200,"statfs":{"total":95995035648,"available":95475916800,"internally_reserved":0,"allocated":2420736,"data_stored":1466475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:29:30 2024","interfaces":[{"interface":"back","average":{"1min":3.972,"5min":2.7719999999999998,"15min":1.581},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":32.546999999999997,"5min":45.372,"15min":45.372},"last":0.42099999999999999},{"interface":"front","average":{"1min":3.9580000000000002,"5min":2.75,"15min":1.6619999999999999},"min":{"1min":0.23999999999999999,"5min":0.217,"15min":0.186},"max":{"1min":29.675999999999998,"5min":45.231999999999999,"15min":45.231999999999999},"last":0.40400000000000003}]},{"osd":2,"last update":"Mon Feb 5 07:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.575,"15min":1.5589999999999999},"min":{"1min":0.23799999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.1760000000000002,"5min":44.920000000000002,"15min":65.903999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":1.579,"15min":1.534},"min":{"1min":0.224,"5min":0.224,"15min":0.182},"max":{"1min":4.8220000000000001,"5min":44.947000000000003,"15min":65.694000000000003},"last":0.438}]},{"osd":3,"last update":"Mon Feb 5 07:30:14 2024","interfaces":[{"interface":"back","average":{"1min":3.6709999999999998,"5min":1.8320000000000001,"15min":1.8169999999999999},"min":{"1min":0.254,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":29.629000000000001,"5min":29.629000000000001,"15min":65.155000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":3.1930000000000001,"5min":1.871,"15min":1.903},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":29.463999999999999,"5min":29.463999999999999,"15min":65.114000000000004},"last":0.47299999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:29:35 2024","interfaces":[{"interface":"back","average":{"1min":3.3149999999999999,"5min":2.9969999999999999,"15min":1.8540000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":42.901000000000003,"5min":50.545000000000002,"15min":65.188000000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":3.7410000000000001,"5min":3.0059999999999998,"15min":1.8939999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.23699999999999999},"max":{"1min":42.960999999999999,"5min":50.310000000000002,"15min":64.873000000000005},"last":0.65400000000000003}]},{"osd":5,"last update":"Mon Feb 5 07:30:14 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":2.113,"15min":1.613},"min":{"1min":0.32100000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.1800000000000002,"5min":95.772999999999996,"15min":95.772999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":2.081,"15min":1.663},"min":{"1min":0.29299999999999998,"5min":0.28199999999999997,"15min":0.27600000000000002},"max":{"1min":3.9860000000000002,"5min":96.028000000000006,"15min":96.028000000000006},"last":0.76300000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:29:25 2024","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.411,"15min":1.411},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.6929999999999996,"5min":4.6929999999999996,"15min":4.6929999999999996},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":5.806,"5min":5.806,"15min":5.806},"last":0.69599999999999995}]},{"osd":7,"last update":"Mon Feb 5 07:29:45 2024","interfaces":[{"interface":"back","average":{"1min":3.3399999999999999,"5min":1.923,"15min":1.645},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":43.012,"5min":43.012,"15min":65.263000000000005},"last":0.72599999999999998},{"interface":"front","average":{"1min":3.7040000000000002,"5min":1.9199999999999999,"15min":1.6419999999999999},"min":{"1min":0.34000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":43.512,"5min":43.512,"15min":64.906000000000006},"last":0.80600000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411495,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":608292,"kb_used_data":63448,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93136860,"statfs":{"total":95995035648,"available":95372144640,"internally_reserved":0,"allocated":64970752,"data_stored":64053538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":2.8260000000000001,"15min":1.875},"min":{"1min":0.23100000000000001,"5min":0.20699999999999999,"15min":0.182},"max":{"1min":12.759,"5min":92.638000000000005,"15min":92.638000000000005},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.879,"5min":2.7559999999999998,"15min":1.962},"min":{"1min":0.29099999999999998,"5min":0.22600000000000001,"15min":0.20699999999999999},"max":{"1min":12.647,"5min":93.481999999999999,"15min":93.481999999999999},"last":0.26300000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.9830000000000001,"5min":3.1560000000000001,"15min":1.635},"min":{"1min":0.26200000000000001,"5min":0.20999999999999999,"15min":0.19400000000000001},"max":{"1min":17.911999999999999,"5min":93.504000000000005,"15min":93.504000000000005},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.9399999999999999,"5min":3.2650000000000001,"15min":1.7330000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":11.464,"5min":93.004000000000005,"15min":93.004000000000005},"last":0.372}]},{"osd":3,"last update":"Mon Feb 5 07:29:20 2024","interfaces":[{"interface":"back","average":{"1min":1.887,"5min":1.9490000000000001,"15min":1.3919999999999999},"min":{"1min":0.27500000000000002,"5min":0.25600000000000001,"15min":0.214},"max":{"1min":6.2000000000000002,"5min":26.114999999999998,"15min":26.114999999999998},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.5049999999999999,"15min":1.266},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.193},"max":{"1min":5.6399999999999997,"5min":26.216000000000001,"15min":26.216000000000001},"last":0.34799999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":4.085,"15min":1.944},"min":{"1min":0.377,"5min":0.318,"15min":0.24099999999999999},"max":{"1min":5.4829999999999997,"5min":168.11699999999999,"15min":168.11699999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":3.9780000000000002,"15min":1.927},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":5.6680000000000001,"5min":168.589,"15min":168.589},"last":0.58499999999999996}]},{"osd":5,"last update":"Mon Feb 5 07:30:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":1.597,"15min":1.1659999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":3.2549999999999999,"5min":39.923000000000002,"15min":39.923000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.554,"15min":1.113},"min":{"1min":0.35199999999999998,"5min":0.33600000000000002,"15min":0.26400000000000001},"max":{"1min":3.5950000000000002,"5min":38.920999999999999,"15min":38.920999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Mon Feb 5 07:29:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":3.6749999999999998,"5min":3.6749999999999998,"15min":3.6749999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.512,"5min":1.512,"15min":1.512},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":5.7240000000000002,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.495}]},{"osd":7,"last update":"Mon Feb 5 07:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.744,"15min":1.256},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":5.5099999999999998,"5min":37.573999999999998,"15min":37.573999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.89900000000000002,"5min":1.6799999999999999,"15min":1.1930000000000001},"min":{"1min":0.35999999999999999,"5min":0.34699999999999998,"15min":0.33800000000000002},"max":{"1min":3.6480000000000001,"5min":37.432000000000002,"15min":37.432000000000002},"last":0.53600000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":503048,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93242104,"statfs":{"total":95995035648,"available":95479914496,"internally_reserved":0,"allocated":2682880,"data_stored":1729155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":512424391},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:29:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.575,"15min":2.0310000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":6.1349999999999998,"5min":32.078000000000003,"15min":74.114999999999995},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.7869999999999999,"15min":1.8200000000000001},"min":{"1min":0.24199999999999999,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":4.8449999999999998,"5min":34.598999999999997,"15min":48.740000000000002},"last":0.33100000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:29:29 2024","interfaces":[{"interface":"back","average":{"1min":5.6319999999999997,"5min":2.5329999999999999,"15min":1.931},"min":{"1min":0.27900000000000003,"5min":0.27300000000000002,"15min":0.253},"max":{"1min":59.298999999999999,"5min":59.298999999999999,"15min":59.298999999999999},"last":0.219},{"interface":"front","average":{"1min":4.798,"5min":2.5449999999999999,"15min":2.198},"min":{"1min":0.29499999999999998,"5min":0.27900000000000003,"15min":0.185},"max":{"1min":59.634,"5min":59.634,"15min":74.084999999999994},"last":0.29499999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:29:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.3740000000000001,"15min":1.97},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.6529999999999996,"5min":29.123999999999999,"15min":74.051000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.23,"5min":1.262,"15min":1.5209999999999999},"min":{"1min":0.27000000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2709999999999999,"5min":7.8769999999999998,"15min":65.861999999999995},"last":0.378}]},{"osd":4,"last update":"Mon Feb 5 07:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.4670000000000001,"15min":1.0880000000000001},"min":{"1min":0.41399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.7290000000000001,"5min":34.981000000000002,"15min":34.981000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.4830000000000001,"15min":1.083},"min":{"1min":0.38800000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":7.9500000000000002,"5min":36.987000000000002,"15min":36.987000000000002},"last":0.75900000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:30:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.876,"15min":1.3520000000000001},"min":{"1min":0.38800000000000001,"5min":0.32900000000000001,"15min":0.28499999999999998},"max":{"1min":2.6579999999999999,"5min":4.1100000000000003,"15min":73.727000000000004},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":1.1279999999999999,"15min":1.1000000000000001},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.27400000000000002},"max":{"1min":2.6840000000000002,"5min":6.3319999999999999,"15min":21.126000000000001},"last":0.67600000000000005}]},{"osd":6,"last update":"Mon Feb 5 07:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.77,"5min":2.77,"15min":2.77},"last":0.71799999999999997}]},{"osd":7,"last update":"Mon Feb 5 07:29:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":2.4870000000000001,"15min":1.498},"min":{"1min":0.42699999999999999,"5min":0.318,"15min":0.318},"max":{"1min":7.984,"5min":91.051000000000002,"15min":91.051000000000002},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.999,"5min":2.431,"15min":1.833},"min":{"1min":0.46700000000000003,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":4.0599999999999996,"5min":91.924000000000007,"15min":91.924000000000007},"last":0.54500000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953629,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":426104,"kb_used_data":2024,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93319048,"statfs":{"total":95995035648,"available":95558705152,"internally_reserved":0,"allocated":2072576,"data_stored":1137831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:29:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":4.0430000000000001,"15min":2.0819999999999999},"min":{"1min":0.22,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.103999999999999,"5min":101.212,"15min":101.212},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.9039999999999999,"5min":4.0060000000000002,"15min":2.1640000000000001},"min":{"1min":0.27300000000000002,"5min":0.23899999999999999,"15min":0.23300000000000001},"max":{"1min":13.169,"5min":102.07599999999999,"15min":102.07599999999999},"last":0.36599999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:29:40 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":4.0439999999999996,"15min":1.8089999999999999},"min":{"1min":0.28699999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.343999999999999,"5min":69.102000000000004,"15min":69.102000000000004},"last":0.59899999999999998},{"interface":"front","average":{"1min":2.9830000000000001,"5min":4.7249999999999996,"15min":2.0179999999999998},"min":{"1min":0.24399999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":18.638999999999999,"5min":68.108999999999995,"15min":68.108999999999995},"last":0.63300000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:29:40 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.841,"15min":1.3089999999999999},"min":{"1min":0.27600000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.266,"5min":26.045000000000002,"15min":67.144999999999996},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.724,"15min":1.2809999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.1689999999999996,"5min":25.960999999999999,"15min":66.959999999999994},"last":0.61599999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:29:40 2024","interfaces":[{"interface":"back","average":{"1min":2.3410000000000002,"5min":2.2000000000000002,"15min":1.206},"min":{"1min":0.40500000000000003,"5min":0.31,"15min":0.27500000000000002},"max":{"1min":13.124000000000001,"5min":66.019999999999996,"15min":66.019999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.5390000000000001,"15min":1.3879999999999999},"min":{"1min":0.36299999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.202999999999999,"5min":101.001,"15min":101.001},"last":0.70599999999999996}]},{"osd":5,"last update":"Mon Feb 5 07:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.82599999999999996,"15min":0.57899999999999996},"min":{"1min":0.28199999999999997,"5min":0.25800000000000001,"15min":0.25700000000000001},"max":{"1min":1.907,"5min":10.917999999999999,"15min":10.917999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.83399999999999996,"15min":0.60499999999999998},"min":{"1min":0.317,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.2410000000000001,"5min":15.105,"15min":15.105},"last":0.501}]},{"osd":6,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":14.606,"5min":14.606,"15min":14.606},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":13.114000000000001,"5min":13.114000000000001,"15min":13.114000000000001},"last":0.74099999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.97699999999999998,"15min":0.72199999999999998},"min":{"1min":0.308,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.8260000000000001,"5min":15.073,"15min":15.073},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.94799999999999995,"15min":0.67900000000000005},"min":{"1min":0.39500000000000002,"5min":0.248,"15min":0.248},"max":{"1min":3.0670000000000002,"5min":14.581,"15min":14.581},"last":0.77100000000000002}]}]},{"osd":5,"up_from":0,"seq":176093659288,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":851968,"data_stored":851968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":130875392,"data_stored":130875392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:30:21.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:30:21.118 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:21.148 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:21.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:21.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:21.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:21.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:21.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:21.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:21.415 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:21.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:21.487 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:21 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:21.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:21.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:21.550 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:21.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:21.568 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:30:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-02-05T07:30:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-02-05T07:30:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:30:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192530 7.1 7:825252f9:::benchmark_data_smithi007_77086_object87935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-02-05T07:30:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:19.535985+0000", 2024-02-05T07:30:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 362.028337034, 2024-02-05T07:30:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0065911670000001, 2024-02-05T07:30:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:30:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:30:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192530 2024-02-05T07:30:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.535985+0000", 2024-02-05T07:30:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.535985+0000", 2024-02-05T07:30:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.535987+0000", 2024-02-05T07:30:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2989999999999999e-06 2024-02-05T07:30:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.535994+0000", 2024-02-05T07:30:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0600000000000002e-06 2024-02-05T07:30:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.536013+0000", 2024-02-05T07:30:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8879e-05 2024-02-05T07:30:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.536024+0000", 2024-02-05T07:30:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0939e-05 2024-02-05T07:30:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.605023+0000", 2024-02-05T07:30:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068999788000000006 2024-02-05T07:30:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:19.605036+0000", 2024-02-05T07:30:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2096000000000001e-05 2024-02-05T07:30:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.532765+0000", 2024-02-05T07:30:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.92772923200000001 2024-02-05T07:30:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.532903+0000", 2024-02-05T07:30:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013795800000000001 2024-02-05T07:30:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3,6", 2024-02-05T07:30:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.533185+0000", 2024-02-05T07:30:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028267800000000002 2024-02-05T07:30:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.535365+0000", 2024-02-05T07:30:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021794639999999999 2024-02-05T07:30:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.535821+0000", 2024-02-05T07:30:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000456367 2024-02-05T07:30:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.541731+0000", 2024-02-05T07:30:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005909882 2024-02-05T07:30:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.542377+0000", 2024-02-05T07:30:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064597999999999999 2024-02-05T07:30:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.542446+0000", 2024-02-05T07:30:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8876999999999998e-05 2024-02-05T07:30:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.542576+0000", 2024-02-05T07:30:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012966799999999999 2024-02-05T07:30:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192550 7.21 7:842b01ab:::benchmark_data_smithi007_77086_object87955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:30:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:20.583755+0000", 2024-02-05T07:30:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 360.98056694100001, 2024-02-05T07:30:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9895773640000001, 2024-02-05T07:30:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:30:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:30:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192550 2024-02-05T07:30:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583755+0000", 2024-02-05T07:30:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583755+0000", 2024-02-05T07:30:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583756+0000", 2024-02-05T07:30:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.84e-06 2024-02-05T07:30:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583764+0000", 2024-02-05T07:30:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6990000000000003e-06 2024-02-05T07:30:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583785+0000", 2024-02-05T07:30:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0372000000000001e-05 2024-02-05T07:30:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583795+0000", 2024-02-05T07:30:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0169e-05 2024-02-05T07:30:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583870+0000", 2024-02-05T07:30:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5587999999999995e-05 2024-02-05T07:30:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.583882+0000", 2024-02-05T07:30:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1932e-05 2024-02-05T07:30:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511843+0000", 2024-02-05T07:30:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.92796104800000001 2024-02-05T07:30:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511855+0000", 2024-02-05T07:30:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1688999999999999e-05 2024-02-05T07:30:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.569583+0000", 2024-02-05T07:30:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.057728024 2024-02-05T07:30:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.569672+0000", 2024-02-05T07:30:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9382000000000006e-05 2024-02-05T07:30:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3,5,6,7", 2024-02-05T07:30:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.569832+0000", 2024-02-05T07:30:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159643 2024-02-05T07:30:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.572963+0000", 2024-02-05T07:30:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031314300000000002 2024-02-05T07:30:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.572979+0000", 2024-02-05T07:30:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5194999999999999e-05 2024-02-05T07:30:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573014+0000", 2024-02-05T07:30:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5487000000000001e-05 2024-02-05T07:30:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573063+0000", 2024-02-05T07:30:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8714999999999997e-05 2024-02-05T07:30:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573110+0000", 2024-02-05T07:30:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7335e-05 2024-02-05T07:30:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573153+0000", 2024-02-05T07:30:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2536000000000001e-05 2024-02-05T07:30:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573215+0000", 2024-02-05T07:30:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2613000000000005e-05 2024-02-05T07:30:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573332+0000", 2024-02-05T07:30:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116667 2024-02-05T07:30:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192659 7.21 7:877b6383:::benchmark_data_smithi007_77086_object88064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:30:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:20.678146+0000", 2024-02-05T07:30:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 360.886176057, 2024-02-05T07:30:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8957312879999999, 2024-02-05T07:30:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:30:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:30:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192659 2024-02-05T07:30:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678146+0000", 2024-02-05T07:30:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678146+0000", 2024-02-05T07:30:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678147+0000", 2024-02-05T07:30:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.238e-06 2024-02-05T07:30:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678151+0000", 2024-02-05T07:30:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5249999999999999e-06 2024-02-05T07:30:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678165+0000", 2024-02-05T07:30:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3574000000000001e-05 2024-02-05T07:30:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678172+0000", 2024-02-05T07:30:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5730000000000002e-06 2024-02-05T07:30:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678231+0000", 2024-02-05T07:30:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8307000000000003e-05 2024-02-05T07:30:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678242+0000", 2024-02-05T07:30:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1629e-05 2024-02-05T07:30:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511879+0000", 2024-02-05T07:30:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.83363663499999996 2024-02-05T07:30:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511886+0000", 2024-02-05T07:30:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0520000000000004e-06 2024-02-05T07:30:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.570887+0000", 2024-02-05T07:30:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.059001149 2024-02-05T07:30:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.570958+0000", 2024-02-05T07:30:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0562000000000006e-05 2024-02-05T07:30:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3,5,6,7", 2024-02-05T07:30:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.571083+0000", 2024-02-05T07:30:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124936 2024-02-05T07:30:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573027+0000", 2024-02-05T07:30:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019443329999999999 2024-02-05T07:30:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573502+0000", 2024-02-05T07:30:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047502299999999998 2024-02-05T07:30:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573538+0000", 2024-02-05T07:30:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6158000000000002e-05 2024-02-05T07:30:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573572+0000", 2024-02-05T07:30:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3832999999999999e-05 2024-02-05T07:30:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573604+0000", 2024-02-05T07:30:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1816000000000002e-05 2024-02-05T07:30:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573759+0000", 2024-02-05T07:30:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015536099999999999 2024-02-05T07:30:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573792+0000", 2024-02-05T07:30:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2295999999999997e-05 2024-02-05T07:30:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573877+0000", 2024-02-05T07:30:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5288000000000001e-05 2024-02-05T07:30:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192660 7.21 7:842a1e92:::benchmark_data_smithi007_77086_object88065:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:30:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:20.678525+0000", 2024-02-05T07:30:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 360.88579687700002, 2024-02-05T07:30:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.895505679, 2024-02-05T07:30:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:30:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:30:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192660 2024-02-05T07:30:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678525+0000", 2024-02-05T07:30:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678525+0000", 2024-02-05T07:30:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678526+0000", 2024-02-05T07:30:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.034e-06 2024-02-05T07:30:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678533+0000", 2024-02-05T07:30:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3159999999999999e-06 2024-02-05T07:30:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678547+0000", 2024-02-05T07:30:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4080000000000001e-05 2024-02-05T07:30:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678555+0000", 2024-02-05T07:30:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0960000000000001e-06 2024-02-05T07:30:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678630+0000", 2024-02-05T07:30:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.483e-05 2024-02-05T07:30:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.678641+0000", 2024-02-05T07:30:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1337e-05 2024-02-05T07:30:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511906+0000", 2024-02-05T07:30:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.83326460499999999 2024-02-05T07:30:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511912+0000", 2024-02-05T07:30:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.772e-06 2024-02-05T07:30:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.571802+0000", 2024-02-05T07:30:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0598903289999999 2024-02-05T07:30:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.571888+0000", 2024-02-05T07:30:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5929999999999999e-05 2024-02-05T07:30:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3,5,6,7", 2024-02-05T07:30:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.572061+0000", 2024-02-05T07:30:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000173215 2024-02-05T07:30:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573076+0000", 2024-02-05T07:30:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001014694 2024-02-05T07:30:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573637+0000", 2024-02-05T07:30:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056068600000000002 2024-02-05T07:30:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573668+0000", 2024-02-05T07:30:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1408000000000002e-05 2024-02-05T07:30:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573698+0000", 2024-02-05T07:30:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9501000000000001e-05 2024-02-05T07:30:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573729+0000", 2024-02-05T07:30:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1223000000000002e-05 2024-02-05T07:30:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573915+0000", 2024-02-05T07:30:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018607699999999999 2024-02-05T07:30:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.573946+0000", 2024-02-05T07:30:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1276999999999999e-05 2024-02-05T07:30:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.574030+0000", 2024-02-05T07:30:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4269000000000002e-05 2024-02-05T07:30:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:192734 7.21 7:8768831f:::benchmark_data_smithi007_77086_object88139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-02-05T07:30:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:24:20.790637+0000", 2024-02-05T07:30:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 360.77368463400001, 2024-02-05T07:30:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.793210755, 2024-02-05T07:30:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:30:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:30:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192734 2024-02-05T07:30:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790637+0000", 2024-02-05T07:30:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790637+0000", 2024-02-05T07:30:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790638+0000", 2024-02-05T07:30:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.514e-06 2024-02-05T07:30:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790645+0000", 2024-02-05T07:30:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1449999999999996e-06 2024-02-05T07:30:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790658+0000", 2024-02-05T07:30:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3069e-05 2024-02-05T07:30:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790666+0000", 2024-02-05T07:30:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6710000000000005e-06 2024-02-05T07:30:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790717+0000", 2024-02-05T07:30:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0182000000000002e-05 2024-02-05T07:30:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:20.790723+0000", 2024-02-05T07:30:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1070000000000001e-06 2024-02-05T07:30:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511933+0000", 2024-02-05T07:30:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72121072500000005 2024-02-05T07:30:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:21.511940+0000", 2024-02-05T07:30:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0199999999999997e-06 2024-02-05T07:30:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.574092+0000", 2024-02-05T07:30:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0621517819999999 2024-02-05T07:30:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.574168+0000", 2024-02-05T07:30:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5499000000000003e-05 2024-02-05T07:30:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3,5,6,7", 2024-02-05T07:30:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.574287+0000", 2024-02-05T07:30:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118989 2024-02-05T07:30:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.576095+0000", 2024-02-05T07:30:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018082619999999999 2024-02-05T07:30:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.576134+0000", 2024-02-05T07:30:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8884999999999998e-05 2024-02-05T07:30:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.576145+0000", 2024-02-05T07:30:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1083e-05 2024-02-05T07:30:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.576175+0000", 2024-02-05T07:30:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0042000000000001e-05 2024-02-05T07:30:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.576199+0000", 2024-02-05T07:30:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4209000000000001e-05 2024-02-05T07:30:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.583728+0000", 2024-02-05T07:30:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075287940000000001 2024-02-05T07:30:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.583766+0000", 2024-02-05T07:30:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7783000000000001e-05 2024-02-05T07:30:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:24:22.583848+0000", 2024-02-05T07:30:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1994000000000001e-05 2024-02-05T07:30:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:131844 8.12 8:4b3718d0:::benchmark_data_smithi007_105755_object131843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:30:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:56.264093+0000", 2024-02-05T07:30:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 205.30022905499999, 2024-02-05T07:30:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.079539735, 2024-02-05T07:30:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:30:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131844 2024-02-05T07:30:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264093+0000", 2024-02-05T07:30:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264093+0000", 2024-02-05T07:30:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264094+0000", 2024-02-05T07:30:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.443e-06 2024-02-05T07:30:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264295+0000", 2024-02-05T07:30:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020106599999999999 2024-02-05T07:30:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264323+0000", 2024-02-05T07:30:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7719999999999999e-05 2024-02-05T07:30:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264343+0000", 2024-02-05T07:30:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0304000000000001e-05 2024-02-05T07:30:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265133+0000", 2024-02-05T07:30:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079006200000000001 2024-02-05T07:30:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265141+0000", 2024-02-05T07:30:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3299999999999999e-06 2024-02-05T07:30:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.337772+0000", 2024-02-05T07:30:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0726301060000001 2024-02-05T07:30:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-02-05T07:30:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.339454+0000", 2024-02-05T07:30:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001682565 2024-02-05T07:30:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343536+0000", 2024-02-05T07:30:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040822530000000001 2024-02-05T07:30:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343632+0000", 2024-02-05T07:30:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5885999999999996e-05 2024-02-05T07:30:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:131867 8.12 8:4f7063db:::benchmark_data_smithi007_105755_object131866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:30:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:56.264377+0000", 2024-02-05T07:30:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 205.299944939, 2024-02-05T07:30:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.089266753, 2024-02-05T07:30:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131867 2024-02-05T07:30:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264377+0000", 2024-02-05T07:30:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264377+0000", 2024-02-05T07:30:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264377+0000", 2024-02-05T07:30:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3799999999999996e-07 2024-02-05T07:30:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264419+0000", 2024-02-05T07:30:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1569000000000001e-05 2024-02-05T07:30:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264426+0000", 2024-02-05T07:30:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1890000000000001e-06 2024-02-05T07:30:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264433+0000", 2024-02-05T07:30:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5039999999999999e-06 2024-02-05T07:30:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265185+0000", 2024-02-05T07:30:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075286500000000002 2024-02-05T07:30:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265194+0000", 2024-02-05T07:30:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6389999999999998e-06 2024-02-05T07:30:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.339734+0000", 2024-02-05T07:30:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0745397999999999 2024-02-05T07:30:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.339874+0000", 2024-02-05T07:30:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014015699999999999 2024-02-05T07:30:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2,5,7", 2024-02-05T07:30:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.340138+0000", 2024-02-05T07:30:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026435199999999998 2024-02-05T07:30:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343625+0000", 2024-02-05T07:30:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003486669 2024-02-05T07:30:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.348328+0000", 2024-02-05T07:30:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047024069999999996 2024-02-05T07:30:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.348374+0000", 2024-02-05T07:30:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6669999999999999e-05 2024-02-05T07:30:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.351568+0000", 2024-02-05T07:30:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031942749999999999 2024-02-05T07:30:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353402+0000", 2024-02-05T07:30:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018333970000000001 2024-02-05T07:30:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353476+0000", 2024-02-05T07:30:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4153000000000006e-05 2024-02-05T07:30:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353643+0000", 2024-02-05T07:30:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000167369 2024-02-05T07:30:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:131870 8.12 8:4c9bd4f8:::benchmark_data_smithi007_105755_object131869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:30:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:56.264452+0000", 2024-02-05T07:30:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 205.29986987999999, 2024-02-05T07:30:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0894224100000001, 2024-02-05T07:30:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131870 2024-02-05T07:30:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264452+0000", 2024-02-05T07:30:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264452+0000", 2024-02-05T07:30:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264452+0000", 2024-02-05T07:30:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7800000000000001e-07 2024-02-05T07:30:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264506+0000", 2024-02-05T07:30:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4051000000000001e-05 2024-02-05T07:30:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264513+0000", 2024-02-05T07:30:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8170000000000001e-06 2024-02-05T07:30:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264519+0000", 2024-02-05T07:30:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3029999999999996e-06 2024-02-05T07:30:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265227+0000", 2024-02-05T07:30:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070795899999999998 2024-02-05T07:30:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265234+0000", 2024-02-05T07:30:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7989999999999996e-06 2024-02-05T07:30:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.341221+0000", 2024-02-05T07:30:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.075986812 2024-02-05T07:30:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.341349+0000", 2024-02-05T07:30:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012759699999999999 2024-02-05T07:30:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2,5,7", 2024-02-05T07:30:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.341592+0000", 2024-02-05T07:30:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024358299999999999 2024-02-05T07:30:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343631+0000", 2024-02-05T07:30:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020391020000000001 2024-02-05T07:30:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.348414+0000", 2024-02-05T07:30:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047826880000000002 2024-02-05T07:30:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.349943+0000", 2024-02-05T07:30:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015293609999999999 2024-02-05T07:30:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.351610+0000", 2024-02-05T07:30:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001667099 2024-02-05T07:30:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353687+0000", 2024-02-05T07:30:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00207632 2024-02-05T07:30:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353750+0000", 2024-02-05T07:30:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2897000000000004e-05 2024-02-05T07:30:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353874+0000", 2024-02-05T07:30:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124444 2024-02-05T07:30:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:131873 8.12 8:4dbc2e4d:::benchmark_data_smithi007_105755_object131872:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:30:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:56.264529+0000", 2024-02-05T07:30:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 205.29979212200001, 2024-02-05T07:30:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0895603920000001, 2024-02-05T07:30:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131873 2024-02-05T07:30:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264529+0000", 2024-02-05T07:30:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264529+0000", 2024-02-05T07:30:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264530+0000", 2024-02-05T07:30:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1900000000000002e-07 2024-02-05T07:30:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264555+0000", 2024-02-05T07:30:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5378e-05 2024-02-05T07:30:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264565+0000", 2024-02-05T07:30:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3559999999999999e-06 2024-02-05T07:30:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264571+0000", 2024-02-05T07:30:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9100000000000002e-06 2024-02-05T07:30:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265270+0000", 2024-02-05T07:30:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069908399999999999 2024-02-05T07:30:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265277+0000", 2024-02-05T07:30:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8519999999999998e-06 2024-02-05T07:30:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343935+0000", 2024-02-05T07:30:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0786580370000001 2024-02-05T07:30:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.344059+0000", 2024-02-05T07:30:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124413 2024-02-05T07:30:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2,5,7", 2024-02-05T07:30:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.344449+0000", 2024-02-05T07:30:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039028899999999999 2024-02-05T07:30:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.348341+0000", 2024-02-05T07:30:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038918870000000001 2024-02-05T07:30:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.349845+0000", 2024-02-05T07:30:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015036400000000001 2024-02-05T07:30:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.349985+0000", 2024-02-05T07:30:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139598 2024-02-05T07:30:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.351684+0000", 2024-02-05T07:30:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001699267 2024-02-05T07:30:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353913+0000", 2024-02-05T07:30:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022296479999999999 2024-02-05T07:30:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353968+0000", 2024-02-05T07:30:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4391999999999997e-05 2024-02-05T07:30:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.354090+0000", 2024-02-05T07:30:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012202200000000001 2024-02-05T07:30:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:151895 8.2 8:46433b01:::benchmark_data_smithi007_105755_object15585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:30:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:15.629580+0000", 2024-02-05T07:30:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 185.93474184300001, 2024-02-05T07:30:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.341571794, 2024-02-05T07:30:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:30:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151895 2024-02-05T07:30:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629580+0000", 2024-02-05T07:30:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629580+0000", 2024-02-05T07:30:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629581+0000", 2024-02-05T07:30:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1650000000000001e-06 2024-02-05T07:30:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629585+0000", 2024-02-05T07:30:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4329999999999997e-06 2024-02-05T07:30:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629593+0000", 2024-02-05T07:30:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.14e-06 2024-02-05T07:30:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629600+0000", 2024-02-05T07:30:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1820000000000003e-06 2024-02-05T07:30:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629662+0000", 2024-02-05T07:30:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2609999999999999e-05 2024-02-05T07:30:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629791+0000", 2024-02-05T07:30:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128362 2024-02-05T07:30:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,5", 2024-02-05T07:30:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.630331+0000", 2024-02-05T07:30:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054002399999999995 2024-02-05T07:30:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.634382+0000", 2024-02-05T07:30:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040511710000000001 2024-02-05T07:30:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.634627+0000", 2024-02-05T07:30:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024509 2024-02-05T07:30:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.691063+0000", 2024-02-05T07:30:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.056435762 2024-02-05T07:30:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.691077+0000", 2024-02-05T07:30:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4270999999999999e-05 2024-02-05T07:30:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.971083+0000", 2024-02-05T07:30:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2800058949999999 2024-02-05T07:30:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.971152+0000", 2024-02-05T07:30:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8689000000000006e-05 2024-02-05T07:30:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:224444 8.4 8:23e19df8:::benchmark_data_smithi007_105755_object88134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-02-05T07:30:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:45.765107+0000", 2024-02-05T07:30:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.799214294999999, 2024-02-05T07:30:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.987133759, 2024-02-05T07:30:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224444 2024-02-05T07:30:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765107+0000", 2024-02-05T07:30:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765107+0000", 2024-02-05T07:30:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765108+0000", 2024-02-05T07:30:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4600000000000001e-07 2024-02-05T07:30:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765110+0000", 2024-02-05T07:30:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9970000000000001e-06 2024-02-05T07:30:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765113+0000", 2024-02-05T07:30:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2024-02-05T07:30:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765116+0000", 2024-02-05T07:30:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0589999999999998e-06 2024-02-05T07:30:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765147+0000", 2024-02-05T07:30:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0747999999999998e-05 2024-02-05T07:30:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765152+0000", 2024-02-05T07:30:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-02-05T07:30:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.742202+0000", 2024-02-05T07:30:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.97704997400000004 2024-02-05T07:30:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.742312+0000", 2024-02-05T07:30:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110503 2024-02-05T07:30:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:30:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.742585+0000", 2024-02-05T07:30:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027314400000000001 2024-02-05T07:30:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.745284+0000", 2024-02-05T07:30:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026988189999999999 2024-02-05T07:30:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.746606+0000", 2024-02-05T07:30:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013217039999999999 2024-02-05T07:30:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.752149+0000", 2024-02-05T07:30:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055428370000000001 2024-02-05T07:30:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.752208+0000", 2024-02-05T07:30:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8742000000000001e-05 2024-02-05T07:30:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.752241+0000", 2024-02-05T07:30:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3494000000000002e-05 2024-02-05T07:30:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:224465 8.4 8:21f25b55:::benchmark_data_smithi007_105755_object88155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-02-05T07:30:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:45.790686+0000", 2024-02-05T07:30:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.773635468999998, 2024-02-05T07:30:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.96721204100000002, 2024-02-05T07:30:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224465 2024-02-05T07:30:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790686+0000", 2024-02-05T07:30:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790686+0000", 2024-02-05T07:30:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790687+0000", 2024-02-05T07:30:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4799999999999997e-07 2024-02-05T07:30:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790691+0000", 2024-02-05T07:30:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7340000000000002e-06 2024-02-05T07:30:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790701+0000", 2024-02-05T07:30:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7399999999999999e-06 2024-02-05T07:30:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790708+0000", 2024-02-05T07:30:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7649999999999993e-06 2024-02-05T07:30:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790770+0000", 2024-02-05T07:30:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1412000000000001e-05 2024-02-05T07:30:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790780+0000", 2024-02-05T07:30:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9580000000000002e-06 2024-02-05T07:30:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.743922+0000", 2024-02-05T07:30:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95314281099999998 2024-02-05T07:30:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.744022+0000", 2024-02-05T07:30:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9419000000000005e-05 2024-02-05T07:30:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:30:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.744187+0000", 2024-02-05T07:30:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016518699999999999 2024-02-05T07:30:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.746615+0000", 2024-02-05T07:30:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024280299999999999 2024-02-05T07:30:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.746674+0000", 2024-02-05T07:30:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9366e-05 2024-02-05T07:30:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.757803+0000", 2024-02-05T07:30:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011128209999999999 2024-02-05T07:30:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.757860+0000", 2024-02-05T07:30:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7358000000000002e-05 2024-02-05T07:30:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.757898+0000", 2024-02-05T07:30:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8102999999999998e-05 2024-02-05T07:30:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:224477 8.4 8:277e5eae:::benchmark_data_smithi007_105755_object88167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-02-05T07:30:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:45.800502+0000", 2024-02-05T07:30:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.763819427000001, 2024-02-05T07:30:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95752454200000003, 2024-02-05T07:30:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224477 2024-02-05T07:30:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800502+0000", 2024-02-05T07:30:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800502+0000", 2024-02-05T07:30:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800503+0000", 2024-02-05T07:30:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.048e-06 2024-02-05T07:30:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800508+0000", 2024-02-05T07:30:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3370000000000001e-06 2024-02-05T07:30:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800517+0000", 2024-02-05T07:30:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0189999999999995e-06 2024-02-05T07:30:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800524+0000", 2024-02-05T07:30:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6029999999999999e-06 2024-02-05T07:30:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800573+0000", 2024-02-05T07:30:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8501999999999997e-05 2024-02-05T07:30:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800581+0000", 2024-02-05T07:30:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4249999999999998e-06 2024-02-05T07:30:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.745336+0000", 2024-02-05T07:30:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94475443999999997 2024-02-05T07:30:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.745409+0000", 2024-02-05T07:30:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3042999999999998e-05 2024-02-05T07:30:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:30:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.745628+0000", 2024-02-05T07:30:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021931300000000001 2024-02-05T07:30:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.749129+0000", 2024-02-05T07:30:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035011999999999999 2024-02-05T07:30:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.749137+0000", 2024-02-05T07:30:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3610000000000001e-06 2024-02-05T07:30:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.757945+0000", 2024-02-05T07:30:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008807601 2024-02-05T07:30:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.757991+0000", 2024-02-05T07:30:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5973999999999997e-05 2024-02-05T07:30:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758027+0000", 2024-02-05T07:30:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5676000000000002e-05 2024-02-05T07:30:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:224478 8.4 8:2023b659:::benchmark_data_smithi007_105755_object88168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-02-05T07:30:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:45.800541+0000", 2024-02-05T07:30:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.763780472999997, 2024-02-05T07:30:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95762280600000005, 2024-02-05T07:30:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224478 2024-02-05T07:30:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800541+0000", 2024-02-05T07:30:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800541+0000", 2024-02-05T07:30:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800542+0000", 2024-02-05T07:30:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0299999999999999e-07 2024-02-05T07:30:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800544+0000", 2024-02-05T07:30:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5210000000000001e-06 2024-02-05T07:30:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800550+0000", 2024-02-05T07:30:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9950000000000002e-06 2024-02-05T07:30:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800556+0000", 2024-02-05T07:30:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8459999999999996e-06 2024-02-05T07:30:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800599+0000", 2024-02-05T07:30:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2948000000000003e-05 2024-02-05T07:30:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800605+0000", 2024-02-05T07:30:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8320000000000002e-06 2024-02-05T07:30:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.746699+0000", 2024-02-05T07:30:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94609387 2024-02-05T07:30:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.746798+0000", 2024-02-05T07:30:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8925999999999994e-05 2024-02-05T07:30:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:30:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.746973+0000", 2024-02-05T07:30:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017586699999999999 2024-02-05T07:30:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.749175+0000", 2024-02-05T07:30:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022016969999999999 2024-02-05T07:30:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.749325+0000", 2024-02-05T07:30:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015013200000000001 2024-02-05T07:30:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758080+0000", 2024-02-05T07:30:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087546289999999999 2024-02-05T07:30:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758129+0000", 2024-02-05T07:30:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8982e-05 2024-02-05T07:30:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758164+0000", 2024-02-05T07:30:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5058000000000002e-05 2024-02-05T07:30:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:224513 8.4 8:225f6048:::benchmark_data_smithi007_105755_object88203:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-02-05T07:30:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:45.837274+0000", 2024-02-05T07:30:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.727047255000002, 2024-02-05T07:30:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.92101294, 2024-02-05T07:30:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224513 2024-02-05T07:30:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.837274+0000", 2024-02-05T07:30:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.837274+0000", 2024-02-05T07:30:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.837276+0000", 2024-02-05T07:30:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3289999999999999e-06 2024-02-05T07:30:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.837280+0000", 2024-02-05T07:30:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3170000000000003e-06 2024-02-05T07:30:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.837291+0000", 2024-02-05T07:30:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0658e-05 2024-02-05T07:30:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.837297+0000", 2024-02-05T07:30:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6630000000000004e-06 2024-02-05T07:30:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.837344+0000", 2024-02-05T07:30:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6697999999999999e-05 2024-02-05T07:30:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.837351+0000", 2024-02-05T07:30:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.55e-06 2024-02-05T07:30:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.747894+0000", 2024-02-05T07:30:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.910543726 2024-02-05T07:30:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.748010+0000", 2024-02-05T07:30:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011617599999999999 2024-02-05T07:30:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:30:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.748202+0000", 2024-02-05T07:30:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126700000000001 2024-02-05T07:30:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.752019+0000", 2024-02-05T07:30:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038174160000000001 2024-02-05T07:30:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.752063+0000", 2024-02-05T07:30:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4085999999999998e-05 2024-02-05T07:30:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758210+0000", 2024-02-05T07:30:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061464220000000003 2024-02-05T07:30:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758256+0000", 2024-02-05T07:30:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5904e-05 2024-02-05T07:30:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758287+0000", 2024-02-05T07:30:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1727999999999998e-05 2024-02-05T07:30:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:224582 8.4 8:279ad87e:::benchmark_data_smithi007_105755_object88272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-02-05T07:30:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:45.899899+0000", 2024-02-05T07:30:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.664422095999996, 2024-02-05T07:30:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.85850582900000005, 2024-02-05T07:30:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224582 2024-02-05T07:30:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.899899+0000", 2024-02-05T07:30:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.899899+0000", 2024-02-05T07:30:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.899900+0000", 2024-02-05T07:30:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4000000000000002e-07 2024-02-05T07:30:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.899902+0000", 2024-02-05T07:30:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5450000000000002e-06 2024-02-05T07:30:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.899912+0000", 2024-02-05T07:30:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4730000000000006e-06 2024-02-05T07:30:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.899918+0000", 2024-02-05T07:30:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2500000000000003e-06 2024-02-05T07:30:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.899966+0000", 2024-02-05T07:30:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7771000000000001e-05 2024-02-05T07:30:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.899976+0000", 2024-02-05T07:30:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0079e-05 2024-02-05T07:30:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.749380+0000", 2024-02-05T07:30:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.84940352200000002 2024-02-05T07:30:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.749502+0000", 2024-02-05T07:30:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122657 2024-02-05T07:30:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:30:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.749728+0000", 2024-02-05T07:30:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022627500000000001 2024-02-05T07:30:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.752083+0000", 2024-02-05T07:30:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023540639999999999 2024-02-05T07:30:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.752281+0000", 2024-02-05T07:30:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198822 2024-02-05T07:30:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758331+0000", 2024-02-05T07:30:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0060494429999999998 2024-02-05T07:30:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758372+0000", 2024-02-05T07:30:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1230000000000003e-05 2024-02-05T07:30:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758405+0000", 2024-02-05T07:30:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3257999999999999e-05 2024-02-05T07:30:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:224637 8.4 8:224c93fa:::benchmark_data_smithi007_105755_object88327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-02-05T07:30:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:45.959217+0000", 2024-02-05T07:30:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.605104726999997, 2024-02-05T07:30:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79984418999999995, 2024-02-05T07:30:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224637 2024-02-05T07:30:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.959217+0000", 2024-02-05T07:30:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.959217+0000", 2024-02-05T07:30:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.959217+0000", 2024-02-05T07:30:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6699999999999999e-07 2024-02-05T07:30:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.959219+0000", 2024-02-05T07:30:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.857e-06 2024-02-05T07:30:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.959224+0000", 2024-02-05T07:30:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4750000000000004e-06 2024-02-05T07:30:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.959227+0000", 2024-02-05T07:30:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.388e-06 2024-02-05T07:30:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.959272+0000", 2024-02-05T07:30:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5071000000000003e-05 2024-02-05T07:30:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.959281+0000", 2024-02-05T07:30:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6689999999999995e-06 2024-02-05T07:30:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.750839+0000", 2024-02-05T07:30:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79155810999999998 2024-02-05T07:30:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.750911+0000", 2024-02-05T07:30:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2512e-05 2024-02-05T07:30:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:30:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.751077+0000", 2024-02-05T07:30:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016571399999999999 2024-02-05T07:30:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.757760+0000", 2024-02-05T07:30:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0066827140000000002 2024-02-05T07:30:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758175+0000", 2024-02-05T07:30:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041552700000000001 2024-02-05T07:30:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758849+0000", 2024-02-05T07:30:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067333099999999999 2024-02-05T07:30:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758898+0000", 2024-02-05T07:30:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9190000000000002e-05 2024-02-05T07:30:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.759061+0000", 2024-02-05T07:30:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163265 2024-02-05T07:30:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:224655 8.4 8:26f1af1a:::benchmark_data_smithi007_105755_object88345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-02-05T07:30:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:45.974696+0000", 2024-02-05T07:30:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.589625303999995, 2024-02-05T07:30:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.78449247700000002, 2024-02-05T07:30:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224655 2024-02-05T07:30:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.974696+0000", 2024-02-05T07:30:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.974696+0000", 2024-02-05T07:30:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.974698+0000", 2024-02-05T07:30:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.595e-06 2024-02-05T07:30:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.974702+0000", 2024-02-05T07:30:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4889999999999997e-06 2024-02-05T07:30:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.974713+0000", 2024-02-05T07:30:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0889e-05 2024-02-05T07:30:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.974726+0000", 2024-02-05T07:30:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3019e-05 2024-02-05T07:30:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.974793+0000", 2024-02-05T07:30:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6581999999999994e-05 2024-02-05T07:30:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.974804+0000", 2024-02-05T07:30:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.077e-05 2024-02-05T07:30:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.752382+0000", 2024-02-05T07:30:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.77757874999999999 2024-02-05T07:30:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.752497+0000", 2024-02-05T07:30:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114839 2024-02-05T07:30:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:30:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.753918+0000", 2024-02-05T07:30:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001420796 2024-02-05T07:30:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758299+0000", 2024-02-05T07:30:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043810890000000003 2024-02-05T07:30:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758450+0000", 2024-02-05T07:30:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150481 2024-02-05T07:30:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.759110+0000", 2024-02-05T07:30:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066089000000000002 2024-02-05T07:30:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.759154+0000", 2024-02-05T07:30:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3764999999999999e-05 2024-02-05T07:30:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.759189+0000", 2024-02-05T07:30:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4523000000000001e-05 2024-02-05T07:30:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:224667 8.4 8:24424027:::benchmark_data_smithi007_105755_object88357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-02-05T07:30:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:45.986682+0000", 2024-02-05T07:30:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.577639771999998, 2024-02-05T07:30:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.77263725000000005, 2024-02-05T07:30:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224667 2024-02-05T07:30:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.986682+0000", 2024-02-05T07:30:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.986682+0000", 2024-02-05T07:30:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.986683+0000", 2024-02-05T07:30:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1960000000000001e-06 2024-02-05T07:30:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.986687+0000", 2024-02-05T07:30:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4139999999999996e-06 2024-02-05T07:30:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.986697+0000", 2024-02-05T07:30:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7289999999999995e-06 2024-02-05T07:30:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.986707+0000", 2024-02-05T07:30:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0064999999999999e-05 2024-02-05T07:30:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.986760+0000", 2024-02-05T07:30:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3001000000000003e-05 2024-02-05T07:30:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:21.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:21 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.986769+0000", 2024-02-05T07:30:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0450000000000006e-06 2024-02-05T07:30:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.755734+0000", 2024-02-05T07:30:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76896463199999998 2024-02-05T07:30:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.755871+0000", 2024-02-05T07:30:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013748800000000001 2024-02-05T07:30:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:30:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.756169+0000", 2024-02-05T07:30:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000297495 2024-02-05T07:30:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.759071+0000", 2024-02-05T07:30:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029021089999999999 2024-02-05T07:30:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.759226+0000", 2024-02-05T07:30:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155415 2024-02-05T07:30:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.759262+0000", 2024-02-05T07:30:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5961999999999999e-05 2024-02-05T07:30:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.759288+0000", 2024-02-05T07:30:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5993999999999999e-05 2024-02-05T07:30:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.759319+0000", 2024-02-05T07:30:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0704999999999998e-05 2024-02-05T07:30:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:224680 8.4 8:25c2d4b1:::benchmark_data_smithi007_105755_object88370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-02-05T07:30:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:46.000322+0000", 2024-02-05T07:30:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.563999092000003, 2024-02-05T07:30:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.77388131500000001, 2024-02-05T07:30:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224680 2024-02-05T07:30:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.000322+0000", 2024-02-05T07:30:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.000322+0000", 2024-02-05T07:30:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.000324+0000", 2024-02-05T07:30:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1659999999999999e-06 2024-02-05T07:30:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.000328+0000", 2024-02-05T07:30:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4569999999999998e-06 2024-02-05T07:30:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.000339+0000", 2024-02-05T07:30:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0662000000000001e-05 2024-02-05T07:30:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.000345+0000", 2024-02-05T07:30:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1879999999999999e-06 2024-02-05T07:30:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.000405+0000", 2024-02-05T07:30:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0476000000000002e-05 2024-02-05T07:30:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.000415+0000", 2024-02-05T07:30:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8030000000000008e-06 2024-02-05T07:30:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.759349+0000", 2024-02-05T07:30:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.75893333900000004 2024-02-05T07:30:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.759486+0000", 2024-02-05T07:30:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013740000000000001 2024-02-05T07:30:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:30:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.759849+0000", 2024-02-05T07:30:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036299900000000003 2024-02-05T07:30:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.766350+0000", 2024-02-05T07:30:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0065010229999999999 2024-02-05T07:30:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.770988+0000", 2024-02-05T07:30:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046381199999999999 2024-02-05T07:30:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.774112+0000", 2024-02-05T07:30:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031239459999999998 2024-02-05T07:30:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.774164+0000", 2024-02-05T07:30:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1524999999999999e-05 2024-02-05T07:30:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.774204+0000", 2024-02-05T07:30:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0210999999999998e-05 2024-02-05T07:30:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:21.743 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:30:21.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:21 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:21.775 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:21.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:21.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:21.877 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:21.909 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:21.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:21.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:21.928 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 6 16 16131 16115 167.85 147.188 0.00633419 0.00594992 2024-02-05T07:30:21.999 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:22.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:22.055 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:22.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:22.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:22.097 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:30:22.157 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:22.187 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:22.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:22.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:22.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:22.313 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:22.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:22.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:22.415 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:22.445 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:22.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:22.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:22.610 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:22.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:22.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:22.735 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:22.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:22.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:22 smithi090 ceph-mon[33213]: pgmap v600: 168 pgs: 168 active+clean; 798 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-02-05T07:30:22.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:22.875 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:22.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:22.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:22 smithi007 ceph-mon[29098]: pgmap v600: 168 pgs: 168 active+clean; 798 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-02-05T07:30:22.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:22 smithi007 ceph-mon[37158]: pgmap v600: 168 pgs: 168 active+clean; 798 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-02-05T07:30:22.928 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 7 16 17949 17933 160.102 113.625 0.00729374 0.00623906 2024-02-05T07:30:22.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:23.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:23.017 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:23.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:23.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:23.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:23.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:23.263 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:23.296 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:23.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:23.399 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:23.433 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:23.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:23.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T07:30:23.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:23.573 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:23.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:23.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:30:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:30:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:30:23.676 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:23.711 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:23.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:23.774 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:30:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=cleanup t=2024-02-05T07:30:23.416784507Z level=info msg="Completed cleanup jobs" duration=3.682497ms 2024-02-05T07:30:23.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:23.835 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:30:23.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:23.854 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:23.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:23.928 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 8 16 20229 20213 157.9 142.5 0.00553792 0.00632447 2024-02-05T07:30:23.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:23.987 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:23.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:24.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:24.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:24.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:24.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:24.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:24.270 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:24.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:24.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:24.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:24.403 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:24.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:24.505 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:24.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:24 smithi090 ceph-mon[33213]: pgmap v601: 168 pgs: 168 active+clean; 798 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-05T07:30:24.544 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:24.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:24.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:24.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:24.698 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:24.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:24.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:24.790 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:24 smithi007 ceph-mon[29098]: pgmap v601: 168 pgs: 168 active+clean; 798 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-05T07:30:24.790 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:24 smithi007 ceph-mon[37158]: pgmap v601: 168 pgs: 168 active+clean; 798 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-05T07:30:24.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:24.840 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:24.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:24.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:24.929 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 9 16 22234 22218 154.278 125.312 0.00529487 0.00647503 2024-02-05T07:30:24.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:24.982 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:24.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:24.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:25.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:25.126 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:25.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:25.228 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:25.266 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:25.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:25.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:25.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:25.400 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:25.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:25.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:25.501 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:30:25.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:25.584 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:25 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3284502187' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:30:25.647 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:25.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:25.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:25.753 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:30:25.753 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:30:25.753 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-02-05T07:30:25.754 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:30:25.759 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:25 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3284502187' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:30:25.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:25 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3284502187' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:30:25.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:25.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:25.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:25.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:25.929 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 10 16 23962 23946 149.649 108 0.00699396 0.00667564 2024-02-05T07:30:25.938 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:25.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:25.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:26.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:26.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:26.065 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:26.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:26.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:26.198 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:26.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:26.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:26.299 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:26.342 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:26.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:26.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T07:30:26.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:26.485 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:26.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:26.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:26.621 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:26.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:26.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:26.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:26.738 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:26 smithi007 ceph-mon[29098]: pgmap v602: 168 pgs: 168 active+clean; 842 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-05T07:30:26.738 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:26 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:30:26.738 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:26 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:30:26.738 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:26 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:30:26.738 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:26 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3284502187' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:30:26.738 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:26 smithi007 ceph-mon[29098]: osdmap e357: 8 total, 8 up, 7 in 2024-02-05T07:30:26.738 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:26 smithi007 ceph-mon[37158]: pgmap v602: 168 pgs: 168 active+clean; 842 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-05T07:30:26.738 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:26 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:30:26.739 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:26 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:30:26.739 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:26 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:30:26.739 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:26 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3284502187' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:30:26.739 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:26 smithi007 ceph-mon[37158]: osdmap e357: 8 total, 8 up, 7 in 2024-02-05T07:30:26.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:26.767 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:26.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:26.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:26 smithi090 ceph-mon[33213]: pgmap v602: 168 pgs: 168 active+clean; 842 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-05T07:30:26.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:26 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:30:26.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:26 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:30:26.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:26 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:30:26.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:26 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3284502187' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:30:26.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:26 smithi090 ceph-mon[33213]: osdmap e357: 8 total, 8 up, 7 in 2024-02-05T07:30:26.784 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:30:26.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:26.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:26.899 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:26.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:26.929 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 11 16 25676 25660 145.783 107.125 0.0113404 0.00685299 2024-02-05T07:30:27.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:27.037 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:27.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:27.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:27.139 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:27.182 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:27.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:27.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:27.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:27.321 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:27.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:27.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:27.464 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:27.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:27.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:27.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:27.593 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:27.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:27.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:27.719 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:27.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:27.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:27 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:30:27.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:27 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:30:27.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:27 smithi090 ceph-mon[33213]: osdmap e358: 8 total, 8 up, 7 in 2024-02-05T07:30:27.820 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:27.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:27 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:30:27.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:27 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:30:27.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:27 smithi007 ceph-mon[29098]: osdmap e358: 8 total, 8 up, 7 in 2024-02-05T07:30:27.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:27 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:30:27.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:27 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:30:27.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:27 smithi007 ceph-mon[37158]: osdmap e358: 8 total, 8 up, 7 in 2024-02-05T07:30:27.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:27.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:27.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:27.929 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 12 16 26807 26791 139.524 70.6875 0.00704139 0.0071592 2024-02-05T07:30:27.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:27.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:27.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:28.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:28.135 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:28.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:28.163 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:28.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:28.261 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:28.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:28.328 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:30:28.331 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:30:28.362 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:28.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:28.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:28.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:28.530 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:28.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:28.632 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:28.659 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:28.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:28.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:28 smithi007 ceph-mon[29098]: 5.1c scrub starts 2024-02-05T07:30:28.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:28 smithi007 ceph-mon[29098]: 5.1c scrub ok 2024-02-05T07:30:28.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:28 smithi007 ceph-mon[29098]: pgmap v605: 168 pgs: 168 active+clean; 1.4 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 184 MiB/s wr, 2.95k op/s 2024-02-05T07:30:28.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:28 smithi007 ceph-mon[29098]: 5.16 scrub starts 2024-02-05T07:30:28.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:28 smithi007 ceph-mon[29098]: 5.16 scrub ok 2024-02-05T07:30:28.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:28 smithi007 ceph-mon[37158]: 5.1c scrub starts 2024-02-05T07:30:28.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:28 smithi007 ceph-mon[37158]: 5.1c scrub ok 2024-02-05T07:30:28.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:28 smithi007 ceph-mon[37158]: pgmap v605: 168 pgs: 168 active+clean; 1.4 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 184 MiB/s wr, 2.95k op/s 2024-02-05T07:30:28.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:28 smithi007 ceph-mon[37158]: 5.16 scrub starts 2024-02-05T07:30:28.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:28 smithi007 ceph-mon[37158]: 5.16 scrub ok 2024-02-05T07:30:28.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:28.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:28 smithi090 ceph-mon[33213]: 5.1c scrub starts 2024-02-05T07:30:28.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:28 smithi090 ceph-mon[33213]: 5.1c scrub ok 2024-02-05T07:30:28.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:28 smithi090 ceph-mon[33213]: pgmap v605: 168 pgs: 168 active+clean; 1.4 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 184 MiB/s wr, 2.95k op/s 2024-02-05T07:30:28.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:28 smithi090 ceph-mon[33213]: 5.16 scrub starts 2024-02-05T07:30:28.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:28 smithi090 ceph-mon[33213]: 5.16 scrub ok 2024-02-05T07:30:28.792 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:28.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:28.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:28.930 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 13 16 28202 28186 135.498 87.1875 0.0106433 0.00737262 2024-02-05T07:30:28.932 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:28.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:28.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:29.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:29.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:29.077 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:29.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:29.178 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:29.208 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:29.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:29.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:29.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:29.333 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:29.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:29.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:29.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:29.466 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:29.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:29.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:29.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:29.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:29.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:29.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:29.727 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:29.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:29.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:29 smithi090 ceph-mon[33213]: 5.15 scrub starts 2024-02-05T07:30:29.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:29 smithi090 ceph-mon[33213]: 5.15 scrub ok 2024-02-05T07:30:29.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:29 smithi090 ceph-mon[33213]: 5.1e scrub starts 2024-02-05T07:30:29.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:29 smithi090 ceph-mon[33213]: 5.1e scrub ok 2024-02-05T07:30:29.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:29 smithi090 ceph-mon[33213]: from='client.25147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:30:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2024-02-05T07:30:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]: dispatch 2024-02-05T07:30:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:29 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2024-02-05T07:30:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-05T07:30:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:29 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]: dispatch 2024-02-05T07:30:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:29 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-05T07:30:29.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:29.858 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:29.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[29098]: 5.15 scrub starts 2024-02-05T07:30:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[29098]: 5.15 scrub ok 2024-02-05T07:30:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[29098]: 5.1e scrub starts 2024-02-05T07:30:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[29098]: 5.1e scrub ok 2024-02-05T07:30:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[29098]: from='client.25147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:30:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2024-02-05T07:30:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]: dispatch 2024-02-05T07:30:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2024-02-05T07:30:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-05T07:30:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]: dispatch 2024-02-05T07:30:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-05T07:30:29.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[37158]: 5.15 scrub starts 2024-02-05T07:30:29.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[37158]: 5.15 scrub ok 2024-02-05T07:30:29.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[37158]: 5.1e scrub starts 2024-02-05T07:30:29.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[37158]: 5.1e scrub ok 2024-02-05T07:30:29.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[37158]: from='client.25147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:30:29.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2024-02-05T07:30:29.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]: dispatch 2024-02-05T07:30:29.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2024-02-05T07:30:29.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-05T07:30:29.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]: dispatch 2024-02-05T07:30:29.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:29 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-05T07:30:29.929 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 14 16 29367 29351 131.02 72.8125 0.0098479 0.00762626 2024-02-05T07:30:29.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:29.994 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:29.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:30.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:30.118 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:30.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:30.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:30.251 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:30.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:30.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:30.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:30.383 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:30.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:30.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:30.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:30.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:30.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:30.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:30.639 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:30.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:30.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:30.741 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:30.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:30:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:30:30.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:30 smithi090 ceph-mon[33213]: pgmap v606: 168 pgs: 168 active+clean; 1.5 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-02-05T07:30:30.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]': finished 2024-02-05T07:30:30.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]': finished 2024-02-05T07:30:30.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-05T07:30:30.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:30 smithi090 ceph-mon[33213]: osdmap e359: 8 total, 8 up, 7 in 2024-02-05T07:30:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:30 smithi090 ceph-mon[33213]: 3.a scrub starts 2024-02-05T07:30:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:30 smithi090 ceph-mon[33213]: 3.a scrub ok 2024-02-05T07:30:30.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:30:30.775 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[37158]: pgmap v606: 168 pgs: 168 active+clean; 1.5 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-02-05T07:30:30.775 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]': finished 2024-02-05T07:30:30.775 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]': finished 2024-02-05T07:30:30.775 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-05T07:30:30.775 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[37158]: osdmap e359: 8 total, 8 up, 7 in 2024-02-05T07:30:30.775 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[37158]: 3.a scrub starts 2024-02-05T07:30:30.775 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[37158]: 3.a scrub ok 2024-02-05T07:30:30.775 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:30:30.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[29098]: pgmap v606: 168 pgs: 168 active+clean; 1.5 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-02-05T07:30:30.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]': finished 2024-02-05T07:30:30.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [4, 7]}]': finished 2024-02-05T07:30:30.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-05T07:30:30.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[29098]: osdmap e359: 8 total, 8 up, 7 in 2024-02-05T07:30:30.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[29098]: 3.a scrub starts 2024-02-05T07:30:30.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[29098]: 3.a scrub ok 2024-02-05T07:30:30.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:30:30.783 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:30.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:30.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:30.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:30.909 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:30.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:30.929 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 15 16 30428 30412 126.705 66.3125 0.00187394 0.00784096 2024-02-05T07:30:31.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:31.051 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:31.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:31.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:31.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:31.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:31.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:31.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:31.327 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:31.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:31.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:31.459 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:31.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:31.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:31.593 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:31.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:31.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[37158]: 4.f scrub starts 2024-02-05T07:30:31.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[37158]: 4.f scrub ok 2024-02-05T07:30:31.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[37158]: 5.4 scrub starts 2024-02-05T07:30:31.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[37158]: 5.4 scrub ok 2024-02-05T07:30:31.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[37158]: osdmap e360: 8 total, 8 up, 7 in 2024-02-05T07:30:31.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:31.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[37158]: 3.8 scrub starts 2024-02-05T07:30:31.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[37158]: 3.8 scrub ok 2024-02-05T07:30:31.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[29098]: 4.f scrub starts 2024-02-05T07:30:31.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[29098]: 4.f scrub ok 2024-02-05T07:30:31.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[29098]: 5.4 scrub starts 2024-02-05T07:30:31.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[29098]: 5.4 scrub ok 2024-02-05T07:30:31.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[29098]: osdmap e360: 8 total, 8 up, 7 in 2024-02-05T07:30:31.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:31.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[29098]: 3.8 scrub starts 2024-02-05T07:30:31.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:31 smithi007 ceph-mon[29098]: 3.8 scrub ok 2024-02-05T07:30:31.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:31.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:31.725 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:31.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:31.773 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":605,"stamp":"2024-02-05T07:30:27.188071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1524832604,"num_objects":23456,"num_object_clones":0,"num_object_copies":70368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23456,"num_whiteouts":0,"num_read":1815,"num_read_kb":1567,"num_write":23980,"num_write_kb":1488557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":5674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26784,"ondisk_log_size":26784,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":506,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7425368,"kb_used_data":4284892,"kb_used_omap":14,"kb_used_meta":3140401,"kb_avail":648790696,"statfs":{"total":671965249536,"available":664361672704,"internally_reserved":0,"allocated":4387729408,"data_stored":4381113869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15190,"internal_metadata":3215770794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,9,6,5,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1493893120,"num_objects":22795,"num_object_clones":0,"num_object_copies":68385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22795,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":22795,"num_write_kb":1458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.738548"},"pg_stats":[{"pgid":"9.1","version":"357'1438","reported_seq":1464,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.604532+0000","last_change":"2024-02-05T07:30:00.509854+0000","last_active":"2024-02-05T07:30:25.604532+0000","last_peered":"2024-02-05T07:30:25.604532+0000","last_clean":"2024-02-05T07:30:25.604532+0000","last_became_active":"2024-02-05T07:30:00.509481+0000","last_became_peered":"2024-02-05T07:30:00.509481+0000","last_unstale":"2024-02-05T07:30:25.604532+0000","last_undegraded":"2024-02-05T07:30:25.604532+0000","last_fullsized":"2024-02-05T07:30:25.604532+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1438,"log_dups_size":0,"ondisk_log_size":1438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:16.743319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94240768,"num_objects":1438,"num_object_clones":0,"num_object_copies":4314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1438,"num_write_kb":92032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":909,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.501738+0000","last_change":"2024-02-05T07:27:19.752065+0000","last_active":"2024-02-05T07:30:25.501738+0000","last_peered":"2024-02-05T07:30:25.501738+0000","last_clean":"2024-02-05T07:30:25.501738+0000","last_became_active":"2024-02-05T07:27:19.751740+0000","last_became_peered":"2024-02-05T07:27:19.751740+0000","last_unstale":"2024-02-05T07:30:25.501738+0000","last_undegraded":"2024-02-05T07:30:25.501738+0000","last_fullsized":"2024-02-05T07:30:25.501738+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:26:18.993912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.993912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122218+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"326'25","reported_seq":689,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.454912+0000","last_change":"2024-02-05T07:28:26.605879+0000","last_active":"2024-02-05T07:30:19.454912+0000","last_peered":"2024-02-05T07:30:19.454912+0000","last_clean":"2024-02-05T07:30:19.454912+0000","last_became_active":"2024-02-05T07:28:26.605082+0000","last_became_peered":"2024-02-05T07:28:26.605082+0000","last_unstale":"2024-02-05T07:30:19.454912+0000","last_undegraded":"2024-02-05T07:30:19.454912+0000","last_fullsized":"2024-02-05T07:30:19.454912+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'15","last_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:25:44.205475+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863183+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":198,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.511770+0000","last_change":"2024-02-05T07:27:16.716749+0000","last_active":"2024-02-05T07:30:25.511770+0000","last_peered":"2024-02-05T07:30:25.511770+0000","last_clean":"2024-02-05T07:30:25.511770+0000","last_became_active":"2024-02-05T07:27:16.716461+0000","last_became_peered":"2024-02-05T07:27:16.716461+0000","last_unstale":"2024-02-05T07:30:25.511770+0000","last_undegraded":"2024-02-05T07:30:25.511770+0000","last_fullsized":"2024-02-05T07:30:25.511770+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.258469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.258469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"257'9","reported_seq":778,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.506860+0000","last_change":"2024-02-05T07:28:43.135097+0000","last_active":"2024-02-05T07:30:25.506860+0000","last_peered":"2024-02-05T07:30:25.506860+0000","last_clean":"2024-02-05T07:30:25.506860+0000","last_became_active":"2024-02-05T07:27:03.374124+0000","last_became_peered":"2024-02-05T07:27:03.374124+0000","last_unstale":"2024-02-05T07:30:25.506860+0000","last_undegraded":"2024-02-05T07:30:25.506860+0000","last_fullsized":"2024-02-05T07:30:25.506860+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'9","last_scrub_stamp":"2024-02-05T07:28:43.135015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:43.135015+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:26.926889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":603,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.455266+0000","last_change":"2024-02-05T07:25:43.273140+0000","last_active":"2024-02-05T07:30:19.455266+0000","last_peered":"2024-02-05T07:30:19.455266+0000","last_clean":"2024-02-05T07:30:19.455266+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:30:19.455266+0000","last_undegraded":"2024-02-05T07:30:19.455266+0000","last_fullsized":"2024-02-05T07:30:19.455266+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:25:43.273019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863239+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"357'1459","reported_seq":1484,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.874824+0000","last_change":"2024-02-05T07:30:00.506880+0000","last_active":"2024-02-05T07:30:25.874824+0000","last_peered":"2024-02-05T07:30:25.874824+0000","last_clean":"2024-02-05T07:30:25.874824+0000","last_became_active":"2024-02-05T07:30:00.506586+0000","last_became_peered":"2024-02-05T07:30:00.506586+0000","last_unstale":"2024-02-05T07:30:25.874824+0000","last_undegraded":"2024-02-05T07:30:25.874824+0000","last_fullsized":"2024-02-05T07:30:25.874824+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:21.843677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95617024,"num_objects":1459,"num_object_clones":0,"num_object_copies":4377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1459,"num_write_kb":93376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":625,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.505120+0000","last_change":"2024-02-05T07:29:48.249173+0000","last_active":"2024-02-05T07:30:25.505120+0000","last_peered":"2024-02-05T07:30:25.505120+0000","last_clean":"2024-02-05T07:30:25.505120+0000","last_became_active":"2024-02-05T07:28:26.593333+0000","last_became_peered":"2024-02-05T07:28:26.593333+0000","last_unstale":"2024-02-05T07:30:25.505120+0000","last_undegraded":"2024-02-05T07:30:25.505120+0000","last_fullsized":"2024-02-05T07:30:25.505120+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:29:48.249082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.249082+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:14.621210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"326'31","reported_seq":668,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.455949+0000","last_change":"2024-02-05T07:26:36.275060+0000","last_active":"2024-02-05T07:30:19.455949+0000","last_peered":"2024-02-05T07:30:19.455949+0000","last_clean":"2024-02-05T07:30:19.455949+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:30:19.455949+0000","last_undegraded":"2024-02-05T07:30:19.455949+0000","last_fullsized":"2024-02-05T07:30:19.455949+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:26:36.274939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.274939+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863440+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":459,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.463179+0000","last_change":"2024-02-05T07:28:25.791721+0000","last_active":"2024-02-05T07:30:19.463179+0000","last_peered":"2024-02-05T07:30:19.463179+0000","last_clean":"2024-02-05T07:30:19.463179+0000","last_became_active":"2024-02-05T07:28:25.791005+0000","last_became_peered":"2024-02-05T07:28:25.791005+0000","last_unstale":"2024-02-05T07:30:19.463179+0000","last_undegraded":"2024-02-05T07:30:19.463179+0000","last_fullsized":"2024-02-05T07:30:19.463179+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:35.518301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:35.518301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:22.161632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":618,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.506513+0000","last_change":"2024-02-05T07:29:49.674581+0000","last_active":"2024-02-05T07:30:25.506513+0000","last_peered":"2024-02-05T07:30:25.506513+0000","last_clean":"2024-02-05T07:30:25.506513+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:30:25.506513+0000","last_undegraded":"2024-02-05T07:30:25.506513+0000","last_fullsized":"2024-02-05T07:30:25.506513+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.674493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.674493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:05.012130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":620,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.507613+0000","last_change":"2024-02-05T07:28:38.294618+0000","last_active":"2024-02-05T07:30:25.507613+0000","last_peered":"2024-02-05T07:30:25.507613+0000","last_clean":"2024-02-05T07:30:25.507613+0000","last_became_active":"2024-02-05T07:27:16.708888+0000","last_became_peered":"2024-02-05T07:27:16.708888+0000","last_unstale":"2024-02-05T07:30:25.507613+0000","last_undegraded":"2024-02-05T07:30:25.507613+0000","last_fullsized":"2024-02-05T07:30:25.507613+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:38.294470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:38.294470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:26.926790+0000","scrub_duration":232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"357'1476","reported_seq":1501,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.510103+0000","last_change":"2024-02-05T07:30:00.508306+0000","last_active":"2024-02-05T07:30:25.510103+0000","last_peered":"2024-02-05T07:30:25.510103+0000","last_clean":"2024-02-05T07:30:25.510103+0000","last_became_active":"2024-02-05T07:30:00.507929+0000","last_became_peered":"2024-02-05T07:30:00.507929+0000","last_unstale":"2024-02-05T07:30:25.510103+0000","last_undegraded":"2024-02-05T07:30:25.510103+0000","last_fullsized":"2024-02-05T07:30:25.510103+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.268882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96731136,"num_objects":1476,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1476,"num_write_kb":94464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":595,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502765+0000","last_change":"2024-02-05T07:28:46.611825+0000","last_active":"2024-02-05T07:30:25.502765+0000","last_peered":"2024-02-05T07:30:25.502765+0000","last_clean":"2024-02-05T07:30:25.502765+0000","last_became_active":"2024-02-05T07:28:19.312714+0000","last_became_peered":"2024-02-05T07:28:19.312714+0000","last_unstale":"2024-02-05T07:30:25.502765+0000","last_undegraded":"2024-02-05T07:30:25.502765+0000","last_fullsized":"2024-02-05T07:30:25.502765+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:46.611747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:46.611747+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"326'24","reported_seq":664,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.455008+0000","last_change":"2024-02-05T07:29:50.053697+0000","last_active":"2024-02-05T07:30:19.455008+0000","last_peered":"2024-02-05T07:30:19.455008+0000","last_clean":"2024-02-05T07:30:19.455008+0000","last_became_active":"2024-02-05T07:28:25.781007+0000","last_became_peered":"2024-02-05T07:28:25.781007+0000","last_unstale":"2024-02-05T07:30:19.455008+0000","last_undegraded":"2024-02-05T07:30:19.455008+0000","last_fullsized":"2024-02-05T07:30:19.455008+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:29:50.053583+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.053583+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:01.795266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":615,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.453346+0000","last_change":"2024-02-05T07:25:46.149566+0000","last_active":"2024-02-05T07:30:19.453346+0000","last_peered":"2024-02-05T07:30:19.453346+0000","last_clean":"2024-02-05T07:30:19.453346+0000","last_became_active":"2024-02-05T07:18:49.434074+0000","last_became_peered":"2024-02-05T07:18:49.434074+0000","last_unstale":"2024-02-05T07:30:19.453346+0000","last_undegraded":"2024-02-05T07:30:19.453346+0000","last_fullsized":"2024-02-05T07:30:19.453346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:25:46.149431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":450,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.455277+0000","last_change":"2024-02-05T07:29:45.012958+0000","last_active":"2024-02-05T07:30:19.455277+0000","last_peered":"2024-02-05T07:30:19.455277+0000","last_clean":"2024-02-05T07:30:19.455277+0000","last_became_active":"2024-02-05T07:28:26.601713+0000","last_became_peered":"2024-02-05T07:28:26.601713+0000","last_unstale":"2024-02-05T07:30:19.455277+0000","last_undegraded":"2024-02-05T07:30:19.455277+0000","last_fullsized":"2024-02-05T07:30:19.455277+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.012821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:02.146624+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":201,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.515016+0000","last_change":"2024-02-05T07:28:26.598062+0000","last_active":"2024-02-05T07:30:25.515016+0000","last_peered":"2024-02-05T07:30:25.515016+0000","last_clean":"2024-02-05T07:30:25.515016+0000","last_became_active":"2024-02-05T07:28:26.595495+0000","last_became_peered":"2024-02-05T07:28:26.595495+0000","last_unstale":"2024-02-05T07:30:25.515016+0000","last_undegraded":"2024-02-05T07:30:25.515016+0000","last_fullsized":"2024-02-05T07:30:25.515016+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:47.885294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.885294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"357'1470","reported_seq":1495,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.589384+0000","last_change":"2024-02-05T07:30:00.512144+0000","last_active":"2024-02-05T07:30:25.589384+0000","last_peered":"2024-02-05T07:30:25.589384+0000","last_clean":"2024-02-05T07:30:25.589384+0000","last_became_active":"2024-02-05T07:30:00.511808+0000","last_became_peered":"2024-02-05T07:30:00.511808+0000","last_unstale":"2024-02-05T07:30:25.589384+0000","last_undegraded":"2024-02-05T07:30:25.589384+0000","last_fullsized":"2024-02-05T07:30:25.589384+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1470,"log_dups_size":0,"ondisk_log_size":1470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:21.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96337920,"num_objects":1470,"num_object_clones":0,"num_object_copies":4410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1470,"num_write_kb":94080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":600,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.505720+0000","last_change":"2024-02-05T07:28:27.218610+0000","last_active":"2024-02-05T07:30:25.505720+0000","last_peered":"2024-02-05T07:30:25.505720+0000","last_clean":"2024-02-05T07:30:25.505720+0000","last_became_active":"2024-02-05T07:28:26.588864+0000","last_became_peered":"2024-02-05T07:28:26.588864+0000","last_unstale":"2024-02-05T07:30:25.505720+0000","last_undegraded":"2024-02-05T07:30:25.505720+0000","last_fullsized":"2024-02-05T07:30:25.505720+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:27.218488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:27.218488+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:26.926962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"326'34","reported_seq":708,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.508812+0000","last_change":"2024-02-05T07:27:16.982664+0000","last_active":"2024-02-05T07:30:25.508812+0000","last_peered":"2024-02-05T07:30:25.508812+0000","last_clean":"2024-02-05T07:30:25.508812+0000","last_became_active":"2024-02-05T07:27:16.768454+0000","last_became_peered":"2024-02-05T07:27:16.768454+0000","last_unstale":"2024-02-05T07:30:25.508812+0000","last_undegraded":"2024-02-05T07:30:25.508812+0000","last_fullsized":"2024-02-05T07:30:25.508812+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'22","last_scrub_stamp":"2024-02-05T07:26:25.602159+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:26:25.602159+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":198,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.512891+0000","last_change":"2024-02-05T07:27:16.830852+0000","last_active":"2024-02-05T07:30:25.512891+0000","last_peered":"2024-02-05T07:30:25.512891+0000","last_clean":"2024-02-05T07:30:25.512891+0000","last_became_active":"2024-02-05T07:27:16.800735+0000","last_became_peered":"2024-02-05T07:27:16.800735+0000","last_unstale":"2024-02-05T07:30:25.512891+0000","last_undegraded":"2024-02-05T07:30:25.512891+0000","last_fullsized":"2024-02-05T07:30:25.512891+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:48.513614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:48.513614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"257'9","reported_seq":765,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.608462+0000","last_change":"2024-02-05T07:27:03.386101+0000","last_active":"2024-02-05T07:30:19.608462+0000","last_peered":"2024-02-05T07:30:19.608462+0000","last_clean":"2024-02-05T07:30:19.608462+0000","last_became_active":"2024-02-05T07:27:03.375195+0000","last_became_peered":"2024-02-05T07:27:03.375195+0000","last_unstale":"2024-02-05T07:30:19.608462+0000","last_undegraded":"2024-02-05T07:30:19.608462+0000","last_fullsized":"2024-02-05T07:30:19.608462+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:26:22.295732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:22.295732+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":216,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.507670+0000","last_change":"2024-02-05T07:27:03.368067+0000","last_active":"2024-02-05T07:30:25.507670+0000","last_peered":"2024-02-05T07:30:25.507670+0000","last_clean":"2024-02-05T07:30:25.507670+0000","last_became_active":"2024-02-05T07:27:03.367753+0000","last_became_peered":"2024-02-05T07:27:03.367753+0000","last_unstale":"2024-02-05T07:30:25.507670+0000","last_undegraded":"2024-02-05T07:30:25.507670+0000","last_fullsized":"2024-02-05T07:30:25.507670+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.724944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.724944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"357'1558","reported_seq":1584,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.543448+0000","last_change":"2024-02-05T07:30:01.030341+0000","last_active":"2024-02-05T07:30:25.543448+0000","last_peered":"2024-02-05T07:30:25.543448+0000","last_clean":"2024-02-05T07:30:25.543448+0000","last_became_active":"2024-02-05T07:30:01.029929+0000","last_became_peered":"2024-02-05T07:30:01.029929+0000","last_unstale":"2024-02-05T07:30:25.543448+0000","last_undegraded":"2024-02-05T07:30:25.543448+0000","last_fullsized":"2024-02-05T07:30:25.543448+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1558,"log_dups_size":0,"ondisk_log_size":1558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:05.041540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102105088,"num_objects":1558,"num_object_clones":0,"num_object_copies":4674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1558,"num_write_kb":99712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":592,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.454658+0000","last_change":"2024-02-05T07:28:41.657737+0000","last_active":"2024-02-05T07:30:19.454658+0000","last_peered":"2024-02-05T07:30:19.454658+0000","last_clean":"2024-02-05T07:30:19.454658+0000","last_became_active":"2024-02-05T07:28:25.780676+0000","last_became_peered":"2024-02-05T07:28:25.780676+0000","last_unstale":"2024-02-05T07:30:19.454658+0000","last_undegraded":"2024-02-05T07:30:19.454658+0000","last_fullsized":"2024-02-05T07:30:19.454658+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:41.657622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:41.657622+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863578+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":619,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.453264+0000","last_change":"2024-02-05T07:27:16.706337+0000","last_active":"2024-02-05T07:30:19.453264+0000","last_peered":"2024-02-05T07:30:19.453264+0000","last_clean":"2024-02-05T07:30:19.453264+0000","last_became_active":"2024-02-05T07:27:16.705976+0000","last_became_peered":"2024-02-05T07:27:16.705976+0000","last_unstale":"2024-02-05T07:30:19.453264+0000","last_undegraded":"2024-02-05T07:30:19.453264+0000","last_fullsized":"2024-02-05T07:30:19.453264+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:51.211877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:51.211877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863149+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"326'30","reported_seq":242,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.512801+0000","last_change":"2024-02-05T07:28:45.029429+0000","last_active":"2024-02-05T07:30:25.512801+0000","last_peered":"2024-02-05T07:30:25.512801+0000","last_clean":"2024-02-05T07:30:25.512801+0000","last_became_active":"2024-02-05T07:27:16.830389+0000","last_became_peered":"2024-02-05T07:27:16.830389+0000","last_unstale":"2024-02-05T07:30:25.512801+0000","last_undegraded":"2024-02-05T07:30:25.512801+0000","last_fullsized":"2024-02-05T07:30:25.512801+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:28:45.029295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.029295+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":198,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.514567+0000","last_change":"2024-02-05T07:27:16.720166+0000","last_active":"2024-02-05T07:30:25.514567+0000","last_peered":"2024-02-05T07:30:25.514567+0000","last_clean":"2024-02-05T07:30:25.514567+0000","last_became_active":"2024-02-05T07:27:16.719678+0000","last_became_peered":"2024-02-05T07:27:16.719678+0000","last_unstale":"2024-02-05T07:30:25.514567+0000","last_undegraded":"2024-02-05T07:30:25.514567+0000","last_fullsized":"2024-02-05T07:30:25.514567+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:33.084016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:33.084016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":618,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.454175+0000","last_change":"2024-02-05T07:28:56.783280+0000","last_active":"2024-02-05T07:30:19.454175+0000","last_peered":"2024-02-05T07:30:19.454175+0000","last_clean":"2024-02-05T07:30:19.454175+0000","last_became_active":"2024-02-05T07:27:16.706666+0000","last_became_peered":"2024-02-05T07:27:16.706666+0000","last_unstale":"2024-02-05T07:30:19.454175+0000","last_undegraded":"2024-02-05T07:30:19.454175+0000","last_fullsized":"2024-02-05T07:30:19.454175+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:28:56.783168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863605+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"357'1446","reported_seq":1472,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.600371+0000","last_change":"2024-02-05T07:30:01.031178+0000","last_active":"2024-02-05T07:30:25.600371+0000","last_peered":"2024-02-05T07:30:25.600371+0000","last_clean":"2024-02-05T07:30:25.600371+0000","last_became_active":"2024-02-05T07:30:01.030826+0000","last_became_peered":"2024-02-05T07:30:01.030826+0000","last_unstale":"2024-02-05T07:30:25.600371+0000","last_undegraded":"2024-02-05T07:30:25.600371+0000","last_fullsized":"2024-02-05T07:30:25.600371+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1446,"log_dups_size":0,"ondisk_log_size":1446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:17.743657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94765056,"num_objects":1446,"num_object_clones":0,"num_object_copies":4338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1446,"num_write_kb":92544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":855,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.517431+0000","last_change":"2024-02-05T07:28:39.042617+0000","last_active":"2024-02-05T07:30:25.517431+0000","last_peered":"2024-02-05T07:30:25.517431+0000","last_clean":"2024-02-05T07:30:25.517431+0000","last_became_active":"2024-02-05T07:28:26.596040+0000","last_became_peered":"2024-02-05T07:28:26.596040+0000","last_unstale":"2024-02-05T07:30:25.517431+0000","last_undegraded":"2024-02-05T07:30:25.517431+0000","last_fullsized":"2024-02-05T07:30:25.517431+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:39.042281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:39.042281+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"266'10","reported_seq":240,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.512985+0000","last_change":"2024-02-05T07:27:17.017592+0000","last_active":"2024-02-05T07:30:25.512985+0000","last_peered":"2024-02-05T07:30:25.512985+0000","last_clean":"2024-02-05T07:30:25.512985+0000","last_became_active":"2024-02-05T07:27:16.800207+0000","last_became_peered":"2024-02-05T07:27:16.800207+0000","last_unstale":"2024-02-05T07:30:25.512985+0000","last_undegraded":"2024-02-05T07:30:25.512985+0000","last_fullsized":"2024-02-05T07:30:25.512985+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:26:34.535447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:34.535447+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"326'35","reported_seq":689,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.453486+0000","last_change":"2024-02-05T07:29:45.972745+0000","last_active":"2024-02-05T07:30:19.453486+0000","last_peered":"2024-02-05T07:30:19.453486+0000","last_clean":"2024-02-05T07:30:19.453486+0000","last_became_active":"2024-02-05T07:28:26.598831+0000","last_became_peered":"2024-02-05T07:28:26.598831+0000","last_unstale":"2024-02-05T07:30:19.453486+0000","last_undegraded":"2024-02-05T07:30:19.453486+0000","last_fullsized":"2024-02-05T07:30:19.453486+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:29:45.972589+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.972589+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:11.921496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":378,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502740+0000","last_change":"2024-02-05T07:29:41.587097+0000","last_active":"2024-02-05T07:30:25.502740+0000","last_peered":"2024-02-05T07:30:25.502740+0000","last_clean":"2024-02-05T07:30:25.502740+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:30:25.502740+0000","last_undegraded":"2024-02-05T07:30:25.502740+0000","last_fullsized":"2024-02-05T07:30:25.502740+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:41.587022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:41.587022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:03.158718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":614,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503073+0000","last_change":"2024-02-05T07:29:45.254768+0000","last_active":"2024-02-05T07:30:25.503073+0000","last_peered":"2024-02-05T07:30:25.503073+0000","last_clean":"2024-02-05T07:30:25.503073+0000","last_became_active":"2024-02-05T07:28:25.788941+0000","last_became_peered":"2024-02-05T07:28:25.788941+0000","last_unstale":"2024-02-05T07:30:25.503073+0000","last_undegraded":"2024-02-05T07:30:25.503073+0000","last_fullsized":"2024-02-05T07:30:25.503073+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:45.254637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.254637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:47.878497+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"357'1469","reported_seq":1495,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.872577+0000","last_change":"2024-02-05T07:30:00.509399+0000","last_active":"2024-02-05T07:30:25.872577+0000","last_peered":"2024-02-05T07:30:25.872577+0000","last_clean":"2024-02-05T07:30:25.872577+0000","last_became_active":"2024-02-05T07:30:00.509120+0000","last_became_peered":"2024-02-05T07:30:00.509120+0000","last_unstale":"2024-02-05T07:30:25.872577+0000","last_undegraded":"2024-02-05T07:30:25.872577+0000","last_fullsized":"2024-02-05T07:30:25.872577+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1469,"log_dups_size":0,"ondisk_log_size":1469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:24.514798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96272384,"num_objects":1469,"num_object_clones":0,"num_object_copies":4407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1469,"num_write_kb":94016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":447,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.464182+0000","last_change":"2024-02-05T07:28:48.540110+0000","last_active":"2024-02-05T07:30:19.464182+0000","last_peered":"2024-02-05T07:30:19.464182+0000","last_clean":"2024-02-05T07:30:19.464182+0000","last_became_active":"2024-02-05T07:28:26.608674+0000","last_became_peered":"2024-02-05T07:28:26.608674+0000","last_unstale":"2024-02-05T07:30:19.464182+0000","last_undegraded":"2024-02-05T07:30:19.464182+0000","last_fullsized":"2024-02-05T07:30:19.464182+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:48.539780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.539780+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:22.161810+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":229,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.462853+0000","last_change":"2024-02-05T07:29:39.912134+0000","last_active":"2024-02-05T07:30:19.462853+0000","last_peered":"2024-02-05T07:30:19.462853+0000","last_clean":"2024-02-05T07:30:19.462853+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:30:19.462853+0000","last_undegraded":"2024-02-05T07:30:19.462853+0000","last_fullsized":"2024-02-05T07:30:19.462853+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:29:39.912035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:39.912035+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:58.559919+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"326'25","reported_seq":675,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.504372+0000","last_change":"2024-02-05T07:27:03.387977+0000","last_active":"2024-02-05T07:30:25.504372+0000","last_peered":"2024-02-05T07:30:25.504372+0000","last_clean":"2024-02-05T07:30:25.504372+0000","last_became_active":"2024-02-05T07:27:03.361342+0000","last_became_peered":"2024-02-05T07:27:03.361342+0000","last_unstale":"2024-02-05T07:30:25.504372+0000","last_undegraded":"2024-02-05T07:30:25.504372+0000","last_fullsized":"2024-02-05T07:30:25.504372+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-02-05T07:26:26.616180+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:26:26.616180+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":209,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.512193+0000","last_change":"2024-02-05T07:29:44.015577+0000","last_active":"2024-02-05T07:30:25.512193+0000","last_peered":"2024-02-05T07:30:25.512193+0000","last_clean":"2024-02-05T07:30:25.512193+0000","last_became_active":"2024-02-05T07:28:26.596096+0000","last_became_peered":"2024-02-05T07:28:26.596096+0000","last_unstale":"2024-02-05T07:30:25.512193+0000","last_undegraded":"2024-02-05T07:30:25.512193+0000","last_fullsized":"2024-02-05T07:30:25.512193+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:44.015468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:29:44.015468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:08.715933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":198,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.513170+0000","last_change":"2024-02-05T07:27:16.831397+0000","last_active":"2024-02-05T07:30:25.513170+0000","last_peered":"2024-02-05T07:30:25.513170+0000","last_clean":"2024-02-05T07:30:25.513170+0000","last_became_active":"2024-02-05T07:27:16.829863+0000","last_became_peered":"2024-02-05T07:27:16.829863+0000","last_unstale":"2024-02-05T07:30:25.513170+0000","last_undegraded":"2024-02-05T07:30:25.513170+0000","last_fullsized":"2024-02-05T07:30:25.513170+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:52.847946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:52.847946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"357'1507","reported_seq":1533,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.537608+0000","last_change":"2024-02-05T07:30:01.030294+0000","last_active":"2024-02-05T07:30:25.537608+0000","last_peered":"2024-02-05T07:30:25.537608+0000","last_clean":"2024-02-05T07:30:25.537608+0000","last_became_active":"2024-02-05T07:30:01.029966+0000","last_became_peered":"2024-02-05T07:30:01.029966+0000","last_unstale":"2024-02-05T07:30:25.537608+0000","last_undegraded":"2024-02-05T07:30:25.537608+0000","last_fullsized":"2024-02-05T07:30:25.537608+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.884581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98762752,"num_objects":1507,"num_object_clones":0,"num_object_copies":4521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1507,"num_write_kb":96448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":447,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.463625+0000","last_change":"2024-02-05T07:28:49.521571+0000","last_active":"2024-02-05T07:30:19.463625+0000","last_peered":"2024-02-05T07:30:19.463625+0000","last_clean":"2024-02-05T07:30:19.463625+0000","last_became_active":"2024-02-05T07:28:26.624868+0000","last_became_peered":"2024-02-05T07:28:26.624868+0000","last_unstale":"2024-02-05T07:30:19.463625+0000","last_undegraded":"2024-02-05T07:30:19.463625+0000","last_fullsized":"2024-02-05T07:30:19.463625+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:49.521450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:49.521450+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":446,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.462876+0000","last_change":"2024-02-05T07:28:26.590000+0000","last_active":"2024-02-05T07:30:19.462876+0000","last_peered":"2024-02-05T07:30:19.462876+0000","last_clean":"2024-02-05T07:30:19.462876+0000","last_became_active":"2024-02-05T07:28:26.589673+0000","last_became_peered":"2024-02-05T07:28:26.589673+0000","last_unstale":"2024-02-05T07:30:19.462876+0000","last_undegraded":"2024-02-05T07:30:19.462876+0000","last_fullsized":"2024-02-05T07:30:19.462876+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:30.535958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:25:30.535958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:22.161659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"326'29","reported_seq":303,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502919+0000","last_change":"2024-02-05T07:27:03.398624+0000","last_active":"2024-02-05T07:30:25.502919+0000","last_peered":"2024-02-05T07:30:25.502919+0000","last_clean":"2024-02-05T07:30:25.502919+0000","last_became_active":"2024-02-05T07:27:03.382876+0000","last_became_peered":"2024-02-05T07:27:03.382876+0000","last_unstale":"2024-02-05T07:30:25.502919+0000","last_undegraded":"2024-02-05T07:30:25.502919+0000","last_fullsized":"2024-02-05T07:30:25.502919+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-02-05T07:26:20.705363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:20.705363+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":632,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.506428+0000","last_change":"2024-02-05T07:29:53.699733+0000","last_active":"2024-02-05T07:30:25.506428+0000","last_peered":"2024-02-05T07:30:25.506428+0000","last_clean":"2024-02-05T07:30:25.506428+0000","last_became_active":"2024-02-05T07:27:03.361028+0000","last_became_peered":"2024-02-05T07:27:03.361028+0000","last_unstale":"2024-02-05T07:30:25.506428+0000","last_undegraded":"2024-02-05T07:30:25.506428+0000","last_fullsized":"2024-02-05T07:30:25.506428+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.699651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.699651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:06.234119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":615,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502148+0000","last_change":"2024-02-05T07:28:49.555346+0000","last_active":"2024-02-05T07:30:25.502148+0000","last_peered":"2024-02-05T07:30:25.502148+0000","last_clean":"2024-02-05T07:30:25.502148+0000","last_became_active":"2024-02-05T07:28:25.840850+0000","last_became_peered":"2024-02-05T07:28:25.840850+0000","last_unstale":"2024-02-05T07:30:25.502148+0000","last_undegraded":"2024-02-05T07:30:25.502148+0000","last_fullsized":"2024-02-05T07:30:25.502148+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:49.555018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:49.555018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"356'1169","reported_seq":1193,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:23.639435+0000","last_change":"2024-02-05T07:30:00.869708+0000","last_active":"2024-02-05T07:30:23.639435+0000","last_peered":"2024-02-05T07:30:23.639435+0000","last_clean":"2024-02-05T07:30:23.639435+0000","last_became_active":"2024-02-05T07:30:00.869295+0000","last_became_peered":"2024-02-05T07:30:00.869295+0000","last_unstale":"2024-02-05T07:30:23.639435+0000","last_undegraded":"2024-02-05T07:30:23.639435+0000","last_fullsized":"2024-02-05T07:30:23.639435+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1169,"log_dups_size":0,"ondisk_log_size":1169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:11.435954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76611584,"num_objects":1169,"num_object_clones":0,"num_object_copies":3507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1169,"num_write_kb":74816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":594,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502874+0000","last_change":"2024-02-05T07:28:47.605953+0000","last_active":"2024-02-05T07:30:25.502874+0000","last_peered":"2024-02-05T07:30:25.502874+0000","last_clean":"2024-02-05T07:30:25.502874+0000","last_became_active":"2024-02-05T07:28:25.756613+0000","last_became_peered":"2024-02-05T07:28:25.756613+0000","last_unstale":"2024-02-05T07:30:25.502874+0000","last_undegraded":"2024-02-05T07:30:25.502874+0000","last_fullsized":"2024-02-05T07:30:25.502874+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:47.605803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.605803+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":215,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.509178+0000","last_change":"2024-02-05T07:27:03.370712+0000","last_active":"2024-02-05T07:30:25.509178+0000","last_peered":"2024-02-05T07:30:25.509178+0000","last_clean":"2024-02-05T07:30:25.509178+0000","last_became_active":"2024-02-05T07:27:03.370297+0000","last_became_peered":"2024-02-05T07:27:03.370297+0000","last_unstale":"2024-02-05T07:30:25.509178+0000","last_undegraded":"2024-02-05T07:30:25.509178+0000","last_fullsized":"2024-02-05T07:30:25.509178+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:32.652537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:32.652537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"326'24","reported_seq":425,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503094+0000","last_change":"2024-02-05T07:29:36.583093+0000","last_active":"2024-02-05T07:30:25.503094+0000","last_peered":"2024-02-05T07:30:25.503094+0000","last_clean":"2024-02-05T07:30:25.503094+0000","last_became_active":"2024-02-05T07:24:34.209716+0000","last_became_peered":"2024-02-05T07:24:34.209716+0000","last_unstale":"2024-02-05T07:30:25.503094+0000","last_undegraded":"2024-02-05T07:30:25.503094+0000","last_fullsized":"2024-02-05T07:30:25.503094+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:29:36.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:36.583005+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:41.205092+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1316,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.501483+0000","last_change":"2024-02-05T07:29:51.291231+0000","last_active":"2024-02-05T07:30:25.501483+0000","last_peered":"2024-02-05T07:30:25.501483+0000","last_clean":"2024-02-05T07:30:25.501483+0000","last_became_active":"2024-02-05T07:18:48.424620+0000","last_became_peered":"2024-02-05T07:18:48.424620+0000","last_unstale":"2024-02-05T07:30:25.501483+0000","last_undegraded":"2024-02-05T07:30:25.501483+0000","last_fullsized":"2024-02-05T07:30:25.501483+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:29:51.291146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.291146+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:11.742257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":687,"num_read_kb":682,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":587,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.453574+0000","last_change":"2024-02-05T07:27:16.724562+0000","last_active":"2024-02-05T07:30:19.453574+0000","last_peered":"2024-02-05T07:30:19.453574+0000","last_clean":"2024-02-05T07:30:19.453574+0000","last_became_active":"2024-02-05T07:27:16.724280+0000","last_became_peered":"2024-02-05T07:27:16.724280+0000","last_unstale":"2024-02-05T07:30:19.453574+0000","last_undegraded":"2024-02-05T07:30:19.453574+0000","last_fullsized":"2024-02-05T07:30:19.453574+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:30.351655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:26:30.351655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863511+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":643,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.506133+0000","last_change":"2024-02-05T07:28:50.545548+0000","last_active":"2024-02-05T07:30:25.506133+0000","last_peered":"2024-02-05T07:30:25.506133+0000","last_clean":"2024-02-05T07:30:25.506133+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:30:25.506133+0000","last_undegraded":"2024-02-05T07:30:25.506133+0000","last_fullsized":"2024-02-05T07:30:25.506133+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:28:50.545437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:50.545437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"357'1475","reported_seq":1501,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.594935+0000","last_change":"2024-02-05T07:30:00.869079+0000","last_active":"2024-02-05T07:30:25.594935+0000","last_peered":"2024-02-05T07:30:25.594935+0000","last_clean":"2024-02-05T07:30:25.594935+0000","last_became_active":"2024-02-05T07:30:00.868747+0000","last_became_peered":"2024-02-05T07:30:00.868747+0000","last_unstale":"2024-02-05T07:30:25.594935+0000","last_undegraded":"2024-02-05T07:30:25.594935+0000","last_fullsized":"2024-02-05T07:30:25.594935+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1475,"log_dups_size":0,"ondisk_log_size":1475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:13.582746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96665600,"num_objects":1475,"num_object_clones":0,"num_object_copies":4425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1475,"num_write_kb":94400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":607,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503395+0000","last_change":"2024-02-05T07:27:03.363357+0000","last_active":"2024-02-05T07:30:25.503395+0000","last_peered":"2024-02-05T07:30:25.503395+0000","last_clean":"2024-02-05T07:30:25.503395+0000","last_became_active":"2024-02-05T07:27:03.363025+0000","last_became_peered":"2024-02-05T07:27:03.363025+0000","last_unstale":"2024-02-05T07:30:25.503395+0000","last_undegraded":"2024-02-05T07:30:25.503395+0000","last_fullsized":"2024-02-05T07:30:25.503395+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:35.850761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:35.850761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.121995+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"326'30","reported_seq":609,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.463289+0000","last_change":"2024-02-05T07:28:25.868774+0000","last_active":"2024-02-05T07:30:19.463289+0000","last_peered":"2024-02-05T07:30:19.463289+0000","last_clean":"2024-02-05T07:30:19.463289+0000","last_became_active":"2024-02-05T07:28:25.868358+0000","last_became_peered":"2024-02-05T07:28:25.868358+0000","last_unstale":"2024-02-05T07:30:19.463289+0000","last_undegraded":"2024-02-05T07:30:19.463289+0000","last_fullsized":"2024-02-05T07:30:19.463289+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-02-05T07:25:46.826417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:46.826417+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:22.161683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":209,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.515380+0000","last_change":"2024-02-05T07:29:43.000854+0000","last_active":"2024-02-05T07:30:25.515380+0000","last_peered":"2024-02-05T07:30:25.515380+0000","last_clean":"2024-02-05T07:30:25.515380+0000","last_became_active":"2024-02-05T07:28:26.595640+0000","last_became_peered":"2024-02-05T07:28:26.595640+0000","last_unstale":"2024-02-05T07:30:25.515380+0000","last_undegraded":"2024-02-05T07:30:25.515380+0000","last_fullsized":"2024-02-05T07:30:25.515380+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:43.000778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:43.000778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:58.850387+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":906,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.501655+0000","last_change":"2024-02-05T07:28:47.119137+0000","last_active":"2024-02-05T07:30:25.501655+0000","last_peered":"2024-02-05T07:30:25.501655+0000","last_clean":"2024-02-05T07:30:25.501655+0000","last_became_active":"2024-02-05T07:28:25.788899+0000","last_became_peered":"2024-02-05T07:28:25.788899+0000","last_unstale":"2024-02-05T07:30:25.501655+0000","last_undegraded":"2024-02-05T07:30:25.501655+0000","last_fullsized":"2024-02-05T07:30:25.501655+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:47.118901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.118901+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":198,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.514536+0000","last_change":"2024-02-05T07:27:16.751712+0000","last_active":"2024-02-05T07:30:25.514536+0000","last_peered":"2024-02-05T07:30:25.514536+0000","last_clean":"2024-02-05T07:30:25.514536+0000","last_became_active":"2024-02-05T07:27:16.751341+0000","last_became_peered":"2024-02-05T07:27:16.751341+0000","last_unstale":"2024-02-05T07:30:25.514536+0000","last_undegraded":"2024-02-05T07:30:25.514536+0000","last_fullsized":"2024-02-05T07:30:25.514536+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:36.753098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.753098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"357'1467","reported_seq":1493,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.814004+0000","last_change":"2024-02-05T07:30:00.510569+0000","last_active":"2024-02-05T07:30:25.814004+0000","last_peered":"2024-02-05T07:30:25.814004+0000","last_clean":"2024-02-05T07:30:25.814004+0000","last_became_active":"2024-02-05T07:30:00.509652+0000","last_became_peered":"2024-02-05T07:30:00.509652+0000","last_unstale":"2024-02-05T07:30:25.814004+0000","last_undegraded":"2024-02-05T07:30:25.814004+0000","last_fullsized":"2024-02-05T07:30:25.814004+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1467,"log_dups_size":0,"ondisk_log_size":1467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:01.265428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96141312,"num_objects":1467,"num_object_clones":0,"num_object_copies":4401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1467,"num_write_kb":93888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":908,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502006+0000","last_change":"2024-02-05T07:28:46.132405+0000","last_active":"2024-02-05T07:30:25.502006+0000","last_peered":"2024-02-05T07:30:25.502006+0000","last_clean":"2024-02-05T07:30:25.502006+0000","last_became_active":"2024-02-05T07:28:14.294305+0000","last_became_peered":"2024-02-05T07:28:14.294305+0000","last_unstale":"2024-02-05T07:30:25.502006+0000","last_undegraded":"2024-02-05T07:30:25.502006+0000","last_fullsized":"2024-02-05T07:30:25.502006+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:46.132295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:46.132295+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":623,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.509762+0000","last_change":"2024-02-05T07:28:40.103698+0000","last_active":"2024-02-05T07:30:25.509762+0000","last_peered":"2024-02-05T07:30:25.509762+0000","last_clean":"2024-02-05T07:30:25.509762+0000","last_became_active":"2024-02-05T07:27:03.357849+0000","last_became_peered":"2024-02-05T07:27:03.357849+0000","last_unstale":"2024-02-05T07:30:25.509762+0000","last_undegraded":"2024-02-05T07:30:25.509762+0000","last_fullsized":"2024-02-05T07:30:25.509762+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:40.103572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:40.103572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:26.927037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"326'22","reported_seq":670,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.509079+0000","last_change":"2024-02-05T07:27:03.383286+0000","last_active":"2024-02-05T07:30:25.509079+0000","last_peered":"2024-02-05T07:30:25.509079+0000","last_clean":"2024-02-05T07:30:25.509079+0000","last_became_active":"2024-02-05T07:27:03.360661+0000","last_became_peered":"2024-02-05T07:27:03.360661+0000","last_unstale":"2024-02-05T07:30:25.509079+0000","last_undegraded":"2024-02-05T07:30:25.509079+0000","last_fullsized":"2024-02-05T07:30:25.509079+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'18","last_scrub_stamp":"2024-02-05T07:26:24.578301+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.578301+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":654,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.505468+0000","last_change":"2024-02-05T07:29:42.383585+0000","last_active":"2024-02-05T07:30:25.505468+0000","last_peered":"2024-02-05T07:30:25.505468+0000","last_clean":"2024-02-05T07:30:25.505468+0000","last_became_active":"2024-02-05T07:27:16.728869+0000","last_became_peered":"2024-02-05T07:27:16.728869+0000","last_unstale":"2024-02-05T07:30:25.505468+0000","last_undegraded":"2024-02-05T07:30:25.505468+0000","last_fullsized":"2024-02-05T07:30:25.505468+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:42.383469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:29:42.383469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:06.897824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":641,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502622+0000","last_change":"2024-02-05T07:29:45.546235+0000","last_active":"2024-02-05T07:30:25.502622+0000","last_peered":"2024-02-05T07:30:25.502622+0000","last_clean":"2024-02-05T07:30:25.502622+0000","last_became_active":"2024-02-05T07:28:25.750373+0000","last_became_peered":"2024-02-05T07:28:25.750373+0000","last_unstale":"2024-02-05T07:30:25.502622+0000","last_undegraded":"2024-02-05T07:30:25.502622+0000","last_fullsized":"2024-02-05T07:30:25.502622+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:45.546152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.546152+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:52.093292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":624,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503758+0000","last_change":"2024-02-05T07:29:38.191933+0000","last_active":"2024-02-05T07:30:25.503758+0000","last_peered":"2024-02-05T07:30:25.503758+0000","last_clean":"2024-02-05T07:30:25.503758+0000","last_became_active":"2024-02-05T07:28:26.593644+0000","last_became_peered":"2024-02-05T07:28:26.593644+0000","last_unstale":"2024-02-05T07:30:25.503758+0000","last_undegraded":"2024-02-05T07:30:25.503758+0000","last_fullsized":"2024-02-05T07:30:25.503758+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:38.191811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:29:38.191811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:05.792018+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"357'1384","reported_seq":1410,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.536270+0000","last_change":"2024-02-05T07:30:00.869947+0000","last_active":"2024-02-05T07:30:25.536270+0000","last_peered":"2024-02-05T07:30:25.536270+0000","last_clean":"2024-02-05T07:30:25.536270+0000","last_became_active":"2024-02-05T07:30:00.869532+0000","last_became_peered":"2024-02-05T07:30:00.869532+0000","last_unstale":"2024-02-05T07:30:25.536270+0000","last_undegraded":"2024-02-05T07:30:25.536270+0000","last_fullsized":"2024-02-05T07:30:25.536270+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1384,"log_dups_size":0,"ondisk_log_size":1384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.927010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90701824,"num_objects":1384,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1384,"num_write_kb":88576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":612,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.505197+0000","last_change":"2024-02-05T07:29:55.246298+0000","last_active":"2024-02-05T07:30:25.505197+0000","last_peered":"2024-02-05T07:30:25.505197+0000","last_clean":"2024-02-05T07:30:25.505197+0000","last_became_active":"2024-02-05T07:28:14.305034+0000","last_became_peered":"2024-02-05T07:28:14.305034+0000","last_unstale":"2024-02-05T07:30:25.505197+0000","last_undegraded":"2024-02-05T07:30:25.505197+0000","last_fullsized":"2024-02-05T07:30:25.505197+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:29:55.246214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.246214+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.223711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":370,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.508251+0000","last_change":"2024-02-05T07:29:48.627198+0000","last_active":"2024-02-05T07:30:25.508251+0000","last_peered":"2024-02-05T07:30:25.508251+0000","last_clean":"2024-02-05T07:30:25.508251+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:30:25.508251+0000","last_undegraded":"2024-02-05T07:30:25.508251+0000","last_fullsized":"2024-02-05T07:30:25.508251+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.627107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.627107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:56.222054+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"326'44","reported_seq":501,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.505849+0000","last_change":"2024-02-05T07:28:31.177038+0000","last_active":"2024-02-05T07:30:25.505849+0000","last_peered":"2024-02-05T07:30:25.505849+0000","last_clean":"2024-02-05T07:30:25.505849+0000","last_became_active":"2024-02-05T07:27:16.708475+0000","last_became_peered":"2024-02-05T07:27:16.708475+0000","last_unstale":"2024-02-05T07:30:25.505849+0000","last_undegraded":"2024-02-05T07:30:25.505849+0000","last_fullsized":"2024-02-05T07:30:25.505849+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'32","last_scrub_stamp":"2024-02-05T07:28:31.176950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:28:31.176950+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:26.926829+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":624,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.454361+0000","last_change":"2024-02-05T07:28:25.779148+0000","last_active":"2024-02-05T07:30:19.454361+0000","last_peered":"2024-02-05T07:30:19.454361+0000","last_clean":"2024-02-05T07:30:19.454361+0000","last_became_active":"2024-02-05T07:28:25.778852+0000","last_became_peered":"2024-02-05T07:28:25.778852+0000","last_unstale":"2024-02-05T07:30:19.454361+0000","last_undegraded":"2024-02-05T07:30:19.454361+0000","last_fullsized":"2024-02-05T07:30:19.454361+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:53.185503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:53.185503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863301+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":540,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503945+0000","last_change":"2024-02-05T07:28:55.477529+0000","last_active":"2024-02-05T07:30:25.503945+0000","last_peered":"2024-02-05T07:30:25.503945+0000","last_clean":"2024-02-05T07:30:25.503945+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:30:25.503945+0000","last_undegraded":"2024-02-05T07:30:25.503945+0000","last_fullsized":"2024-02-05T07:30:25.503945+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:28:55.477280+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":206,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.512776+0000","last_change":"2024-02-05T07:28:42.075567+0000","last_active":"2024-02-05T07:30:25.512776+0000","last_peered":"2024-02-05T07:30:25.512776+0000","last_clean":"2024-02-05T07:30:25.512776+0000","last_became_active":"2024-02-05T07:27:16.714924+0000","last_became_peered":"2024-02-05T07:27:16.714924+0000","last_unstale":"2024-02-05T07:30:25.512776+0000","last_undegraded":"2024-02-05T07:30:25.512776+0000","last_fullsized":"2024-02-05T07:30:25.512776+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:42.075445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:42.075445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.6","version":"357'1472","reported_seq":1496,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.806796+0000","last_change":"2024-02-05T07:30:00.512361+0000","last_active":"2024-02-05T07:30:25.806796+0000","last_peered":"2024-02-05T07:30:25.806796+0000","last_clean":"2024-02-05T07:30:25.806796+0000","last_became_active":"2024-02-05T07:30:00.511728+0000","last_became_peered":"2024-02-05T07:30:00.511728+0000","last_unstale":"2024-02-05T07:30:25.806796+0000","last_undegraded":"2024-02-05T07:30:25.806796+0000","last_fullsized":"2024-02-05T07:30:25.806796+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1472,"log_dups_size":0,"ondisk_log_size":1472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:24.957975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":1472,"num_object_clones":0,"num_object_copies":4416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1472,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":519,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.454690+0000","last_change":"2024-02-05T07:28:40.616425+0000","last_active":"2024-02-05T07:30:19.454690+0000","last_peered":"2024-02-05T07:30:19.454690+0000","last_clean":"2024-02-05T07:30:19.454690+0000","last_became_active":"2024-02-05T07:28:19.302415+0000","last_became_peered":"2024-02-05T07:28:19.302415+0000","last_unstale":"2024-02-05T07:30:19.454690+0000","last_undegraded":"2024-02-05T07:30:19.454690+0000","last_fullsized":"2024-02-05T07:30:19.454690+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:40.616302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:40.616302+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"326'15","reported_seq":413,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502985+0000","last_change":"2024-02-05T07:29:51.665907+0000","last_active":"2024-02-05T07:30:25.502985+0000","last_peered":"2024-02-05T07:30:25.502985+0000","last_clean":"2024-02-05T07:30:25.502985+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:30:25.502985+0000","last_undegraded":"2024-02-05T07:30:25.502985+0000","last_fullsized":"2024-02-05T07:30:25.502985+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:29:51.665808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.665808+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:13.278749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":201,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.519828+0000","last_change":"2024-02-05T07:28:26.602895+0000","last_active":"2024-02-05T07:30:25.519828+0000","last_peered":"2024-02-05T07:30:25.519828+0000","last_clean":"2024-02-05T07:30:25.519828+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:30:25.519828+0000","last_undegraded":"2024-02-05T07:30:25.519828+0000","last_fullsized":"2024-02-05T07:30:25.519828+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:38.539777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:38.539777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":629,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.505879+0000","last_change":"2024-02-05T07:28:41.103849+0000","last_active":"2024-02-05T07:30:25.505879+0000","last_peered":"2024-02-05T07:30:25.505879+0000","last_clean":"2024-02-05T07:30:25.505879+0000","last_became_active":"2024-02-05T07:27:16.708205+0000","last_became_peered":"2024-02-05T07:27:16.708205+0000","last_unstale":"2024-02-05T07:30:25.505879+0000","last_undegraded":"2024-02-05T07:30:25.505879+0000","last_fullsized":"2024-02-05T07:30:25.505879+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:41.103739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:41.103739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:26.926932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":362,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.501732+0000","last_change":"2024-02-05T07:26:30.076326+0000","last_active":"2024-02-05T07:30:25.501732+0000","last_peered":"2024-02-05T07:30:25.501732+0000","last_clean":"2024-02-05T07:30:25.501732+0000","last_became_active":"2024-02-05T07:24:34.235899+0000","last_became_peered":"2024-02-05T07:24:34.235899+0000","last_unstale":"2024-02-05T07:30:25.501732+0000","last_undegraded":"2024-02-05T07:30:25.501732+0000","last_fullsized":"2024-02-05T07:30:25.501732+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:30.076226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:30.076226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122153+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"357'1450","reported_seq":1476,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.510212+0000","last_change":"2024-02-05T07:30:00.509083+0000","last_active":"2024-02-05T07:30:25.510212+0000","last_peered":"2024-02-05T07:30:25.510212+0000","last_clean":"2024-02-05T07:30:25.510212+0000","last_became_active":"2024-02-05T07:30:00.508712+0000","last_became_peered":"2024-02-05T07:30:00.508712+0000","last_unstale":"2024-02-05T07:30:25.510212+0000","last_undegraded":"2024-02-05T07:30:25.510212+0000","last_fullsized":"2024-02-05T07:30:25.510212+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1450,"log_dups_size":0,"ondisk_log_size":1450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:01.622332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95027200,"num_objects":1450,"num_object_clones":0,"num_object_copies":4350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1450,"num_write_kb":92800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":448,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.463714+0000","last_change":"2024-02-05T07:28:47.560599+0000","last_active":"2024-02-05T07:30:19.463714+0000","last_peered":"2024-02-05T07:30:19.463714+0000","last_clean":"2024-02-05T07:30:19.463714+0000","last_became_active":"2024-02-05T07:28:26.596711+0000","last_became_peered":"2024-02-05T07:28:26.596711+0000","last_unstale":"2024-02-05T07:30:19.463714+0000","last_undegraded":"2024-02-05T07:30:19.463714+0000","last_fullsized":"2024-02-05T07:30:19.463714+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:47.560475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.560475+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:22.161775+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"326'35","reported_seq":477,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.462969+0000","last_change":"2024-02-05T07:29:45.981636+0000","last_active":"2024-02-05T07:30:19.462969+0000","last_peered":"2024-02-05T07:30:19.462969+0000","last_clean":"2024-02-05T07:30:19.462969+0000","last_became_active":"2024-02-05T07:28:25.791406+0000","last_became_peered":"2024-02-05T07:28:25.791406+0000","last_unstale":"2024-02-05T07:30:19.462969+0000","last_undegraded":"2024-02-05T07:30:19.462969+0000","last_fullsized":"2024-02-05T07:30:19.462969+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:29:45.981520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.981520+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:03.718981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":635,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503534+0000","last_change":"2024-02-05T07:29:39.170107+0000","last_active":"2024-02-05T07:30:25.503534+0000","last_peered":"2024-02-05T07:30:25.503534+0000","last_clean":"2024-02-05T07:30:25.503534+0000","last_became_active":"2024-02-05T07:28:26.609352+0000","last_became_peered":"2024-02-05T07:28:26.609352+0000","last_unstale":"2024-02-05T07:30:25.503534+0000","last_undegraded":"2024-02-05T07:30:25.503534+0000","last_fullsized":"2024-02-05T07:30:25.503534+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:39.169995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:39.169995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:47.789620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":623,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.505981+0000","last_change":"2024-02-05T07:28:55.173750+0000","last_active":"2024-02-05T07:30:25.505981+0000","last_peered":"2024-02-05T07:30:25.505981+0000","last_clean":"2024-02-05T07:30:25.505981+0000","last_became_active":"2024-02-05T07:27:03.355957+0000","last_became_peered":"2024-02-05T07:27:03.355957+0000","last_unstale":"2024-02-05T07:30:25.505981+0000","last_undegraded":"2024-02-05T07:30:25.505981+0000","last_fullsized":"2024-02-05T07:30:25.505981+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:55.173628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:55.173628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:26.927012+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":620,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.501381+0000","last_change":"2024-02-05T07:29:48.262661+0000","last_active":"2024-02-05T07:30:25.501381+0000","last_peered":"2024-02-05T07:30:25.501381+0000","last_clean":"2024-02-05T07:30:25.501381+0000","last_became_active":"2024-02-05T07:27:16.710267+0000","last_became_peered":"2024-02-05T07:27:16.710267+0000","last_unstale":"2024-02-05T07:30:25.501381+0000","last_undegraded":"2024-02-05T07:30:25.501381+0000","last_fullsized":"2024-02-05T07:30:25.501381+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.262537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:55.590811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"357'1547","reported_seq":1573,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.874793+0000","last_change":"2024-02-05T07:30:01.030823+0000","last_active":"2024-02-05T07:30:25.874793+0000","last_peered":"2024-02-05T07:30:25.874793+0000","last_clean":"2024-02-05T07:30:25.874793+0000","last_became_active":"2024-02-05T07:30:01.030502+0000","last_became_peered":"2024-02-05T07:30:01.030502+0000","last_unstale":"2024-02-05T07:30:25.874793+0000","last_undegraded":"2024-02-05T07:30:25.874793+0000","last_fullsized":"2024-02-05T07:30:25.874793+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1547,"log_dups_size":0,"ondisk_log_size":1547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:05.204271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101384192,"num_objects":1547,"num_object_clones":0,"num_object_copies":4641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1547,"num_write_kb":99008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":860,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.517228+0000","last_change":"2024-02-05T07:28:40.182489+0000","last_active":"2024-02-05T07:30:25.517228+0000","last_peered":"2024-02-05T07:30:25.517228+0000","last_clean":"2024-02-05T07:30:25.517228+0000","last_became_active":"2024-02-05T07:28:26.597412+0000","last_became_peered":"2024-02-05T07:28:26.597412+0000","last_unstale":"2024-02-05T07:30:25.517228+0000","last_undegraded":"2024-02-05T07:30:25.517228+0000","last_fullsized":"2024-02-05T07:30:25.517228+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:40.181881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:40.181881+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":625,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.505122+0000","last_change":"2024-02-05T07:29:39.290437+0000","last_active":"2024-02-05T07:30:25.505122+0000","last_peered":"2024-02-05T07:30:25.505122+0000","last_clean":"2024-02-05T07:30:25.505122+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:30:25.505122+0000","last_undegraded":"2024-02-05T07:30:25.505122+0000","last_fullsized":"2024-02-05T07:30:25.505122+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:39.290320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:39.290320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:45.538623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"325'18","reported_seq":711,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.453722+0000","last_change":"2024-02-05T07:29:49.022612+0000","last_active":"2024-02-05T07:30:19.453722+0000","last_peered":"2024-02-05T07:30:19.453722+0000","last_clean":"2024-02-05T07:30:19.453722+0000","last_became_active":"2024-02-05T07:27:16.706871+0000","last_became_peered":"2024-02-05T07:27:16.706871+0000","last_unstale":"2024-02-05T07:30:19.453722+0000","last_undegraded":"2024-02-05T07:30:19.453722+0000","last_fullsized":"2024-02-05T07:30:19.453722+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'18","last_scrub_stamp":"2024-02-05T07:29:49.021772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.021772+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:49.051858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":617,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503763+0000","last_change":"2024-02-05T07:26:24.102911+0000","last_active":"2024-02-05T07:30:25.503763+0000","last_peered":"2024-02-05T07:30:25.503763+0000","last_clean":"2024-02-05T07:30:25.503763+0000","last_became_active":"2024-02-05T07:18:49.435621+0000","last_became_peered":"2024-02-05T07:18:49.435621+0000","last_unstale":"2024-02-05T07:30:25.503763+0000","last_undegraded":"2024-02-05T07:30:25.503763+0000","last_fullsized":"2024-02-05T07:30:25.503763+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:24.102597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.102597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122315+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":206,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.511628+0000","last_change":"2024-02-05T07:28:48.088461+0000","last_active":"2024-02-05T07:30:25.511628+0000","last_peered":"2024-02-05T07:30:25.511628+0000","last_clean":"2024-02-05T07:30:25.511628+0000","last_became_active":"2024-02-05T07:27:16.764767+0000","last_became_peered":"2024-02-05T07:27:16.764767+0000","last_unstale":"2024-02-05T07:30:25.511628+0000","last_undegraded":"2024-02-05T07:30:25.511628+0000","last_fullsized":"2024-02-05T07:30:25.511628+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:48.088229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.088229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"357'1471","reported_seq":1497,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.530490+0000","last_change":"2024-02-05T07:30:00.869259+0000","last_active":"2024-02-05T07:30:25.530490+0000","last_peered":"2024-02-05T07:30:25.530490+0000","last_clean":"2024-02-05T07:30:25.530490+0000","last_became_active":"2024-02-05T07:30:00.868882+0000","last_became_peered":"2024-02-05T07:30:00.868882+0000","last_unstale":"2024-02-05T07:30:25.530490+0000","last_undegraded":"2024-02-05T07:30:25.530490+0000","last_fullsized":"2024-02-05T07:30:25.530490+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":1471,"log_dups_size":0,"ondisk_log_size":1471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:23.261947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96403456,"num_objects":1471,"num_object_clones":0,"num_object_copies":4413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1471,"num_write_kb":94144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":445,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503327+0000","last_change":"2024-02-05T07:28:45.145176+0000","last_active":"2024-02-05T07:30:25.503327+0000","last_peered":"2024-02-05T07:30:25.503327+0000","last_clean":"2024-02-05T07:30:25.503327+0000","last_became_active":"2024-02-05T07:28:19.292332+0000","last_became_peered":"2024-02-05T07:28:19.292332+0000","last_unstale":"2024-02-05T07:30:25.503327+0000","last_undegraded":"2024-02-05T07:30:25.503327+0000","last_fullsized":"2024-02-05T07:30:25.503327+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.145047+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122549+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"302'11","reported_seq":769,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502280+0000","last_change":"2024-02-05T07:28:50.147621+0000","last_active":"2024-02-05T07:30:25.502280+0000","last_peered":"2024-02-05T07:30:25.502280+0000","last_clean":"2024-02-05T07:30:25.502280+0000","last_became_active":"2024-02-05T07:28:25.869896+0000","last_became_peered":"2024-02-05T07:28:25.869896+0000","last_unstale":"2024-02-05T07:30:25.502280+0000","last_undegraded":"2024-02-05T07:30:25.502280+0000","last_fullsized":"2024-02-05T07:30:25.502280+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'11","last_scrub_stamp":"2024-02-05T07:28:50.147503+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:28:50.147503+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122584+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"326'27","reported_seq":305,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.501837+0000","last_change":"2024-02-05T07:28:49.179596+0000","last_active":"2024-02-05T07:30:25.501837+0000","last_peered":"2024-02-05T07:30:25.501837+0000","last_clean":"2024-02-05T07:30:25.501837+0000","last_became_active":"2024-02-05T07:28:26.591522+0000","last_became_peered":"2024-02-05T07:28:26.591522+0000","last_unstale":"2024-02-05T07:30:25.501837+0000","last_undegraded":"2024-02-05T07:30:25.501837+0000","last_fullsized":"2024-02-05T07:30:25.501837+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:28:49.179472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:28:49.179472+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122481+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":629,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503997+0000","last_change":"2024-02-05T07:26:37.022248+0000","last_active":"2024-02-05T07:30:25.503997+0000","last_peered":"2024-02-05T07:30:25.503997+0000","last_clean":"2024-02-05T07:30:25.503997+0000","last_became_active":"2024-02-05T07:18:49.437663+0000","last_became_peered":"2024-02-05T07:18:49.437663+0000","last_unstale":"2024-02-05T07:30:25.503997+0000","last_undegraded":"2024-02-05T07:30:25.503997+0000","last_fullsized":"2024-02-05T07:30:25.503997+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:26:37.022129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:37.022129+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122287+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":276,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.462772+0000","last_change":"2024-02-05T07:28:26.589968+0000","last_active":"2024-02-05T07:30:19.462772+0000","last_peered":"2024-02-05T07:30:19.462772+0000","last_clean":"2024-02-05T07:30:19.462772+0000","last_became_active":"2024-02-05T07:28:26.589671+0000","last_became_peered":"2024-02-05T07:28:26.589671+0000","last_unstale":"2024-02-05T07:30:19.462772+0000","last_undegraded":"2024-02-05T07:30:19.462772+0000","last_fullsized":"2024-02-05T07:30:19.462772+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:34.561259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:37.032056+0000","last_clean_scrub_stamp":"2024-02-05T07:25:34.561259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:22.161705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":452,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503722+0000","last_change":"2024-02-05T07:28:48.594936+0000","last_active":"2024-02-05T07:30:25.503722+0000","last_peered":"2024-02-05T07:30:25.503722+0000","last_clean":"2024-02-05T07:30:25.503722+0000","last_became_active":"2024-02-05T07:28:28.631649+0000","last_became_peered":"2024-02-05T07:28:28.631649+0000","last_unstale":"2024-02-05T07:30:25.503722+0000","last_undegraded":"2024-02-05T07:30:25.503722+0000","last_fullsized":"2024-02-05T07:30:25.503722+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:48.594280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.594280+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"326'12","reported_seq":215,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.514440+0000","last_change":"2024-02-05T07:27:16.969586+0000","last_active":"2024-02-05T07:30:25.514440+0000","last_peered":"2024-02-05T07:30:25.514440+0000","last_clean":"2024-02-05T07:30:25.514440+0000","last_became_active":"2024-02-05T07:27:16.750507+0000","last_became_peered":"2024-02-05T07:27:16.750507+0000","last_unstale":"2024-02-05T07:30:25.514440+0000","last_undegraded":"2024-02-05T07:30:25.514440+0000","last_fullsized":"2024-02-05T07:30:25.514440+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-02-05T07:26:36.068507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.068507+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":420,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.462487+0000","last_change":"2024-02-05T07:28:45.572051+0000","last_active":"2024-02-05T07:30:19.462487+0000","last_peered":"2024-02-05T07:30:19.462487+0000","last_clean":"2024-02-05T07:30:19.462487+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:30:19.462487+0000","last_undegraded":"2024-02-05T07:30:19.462487+0000","last_fullsized":"2024-02-05T07:30:19.462487+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.571917+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:22.161738+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":201,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.512512+0000","last_change":"2024-02-05T07:28:26.599846+0000","last_active":"2024-02-05T07:30:25.512512+0000","last_peered":"2024-02-05T07:30:25.512512+0000","last_clean":"2024-02-05T07:30:25.512512+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:30:25.512512+0000","last_undegraded":"2024-02-05T07:30:25.512512+0000","last_fullsized":"2024-02-05T07:30:25.512512+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:28.567844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:28.567844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":613,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503341+0000","last_change":"2024-02-05T07:28:55.256192+0000","last_active":"2024-02-05T07:30:25.503341+0000","last_peered":"2024-02-05T07:30:25.503341+0000","last_clean":"2024-02-05T07:30:25.503341+0000","last_became_active":"2024-02-05T07:28:25.791864+0000","last_became_peered":"2024-02-05T07:28:25.791864+0000","last_unstale":"2024-02-05T07:30:25.503341+0000","last_undegraded":"2024-02-05T07:30:25.503341+0000","last_fullsized":"2024-02-05T07:30:25.503341+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:55.256094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:28:55.256094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:30.185520+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":595,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502429+0000","last_change":"2024-02-05T07:28:45.702694+0000","last_active":"2024-02-05T07:30:25.502429+0000","last_peered":"2024-02-05T07:30:25.502429+0000","last_clean":"2024-02-05T07:30:25.502429+0000","last_became_active":"2024-02-05T07:28:19.311737+0000","last_became_peered":"2024-02-05T07:28:19.311737+0000","last_unstale":"2024-02-05T07:30:25.502429+0000","last_undegraded":"2024-02-05T07:30:25.502429+0000","last_fullsized":"2024-02-05T07:30:25.502429+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:45.702565+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.702565+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.702565+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"325'15","reported_seq":217,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.512691+0000","last_change":"2024-02-05T07:27:17.009879+0000","last_active":"2024-02-05T07:30:25.512691+0000","last_peered":"2024-02-05T07:30:25.512691+0000","last_clean":"2024-02-05T07:30:25.512691+0000","last_became_active":"2024-02-05T07:27:16.750671+0000","last_became_peered":"2024-02-05T07:27:16.750671+0000","last_unstale":"2024-02-05T07:30:25.512691+0000","last_undegraded":"2024-02-05T07:30:25.512691+0000","last_fullsized":"2024-02-05T07:30:25.512691+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:26:28.037945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:28.037945+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":625,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.501995+0000","last_change":"2024-02-05T07:28:25.868721+0000","last_active":"2024-02-05T07:30:25.501995+0000","last_peered":"2024-02-05T07:30:25.501995+0000","last_clean":"2024-02-05T07:30:25.501995+0000","last_became_active":"2024-02-05T07:28:25.868400+0000","last_became_peered":"2024-02-05T07:28:25.868400+0000","last_unstale":"2024-02-05T07:30:25.501995+0000","last_undegraded":"2024-02-05T07:30:25.501995+0000","last_fullsized":"2024-02-05T07:30:25.501995+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:51.811993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:25:51.811993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122024+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":198,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.514486+0000","last_change":"2024-02-05T07:27:16.750873+0000","last_active":"2024-02-05T07:30:25.514486+0000","last_peered":"2024-02-05T07:30:25.514486+0000","last_clean":"2024-02-05T07:30:25.514486+0000","last_became_active":"2024-02-05T07:27:16.749995+0000","last_became_peered":"2024-02-05T07:27:16.749995+0000","last_unstale":"2024-02-05T07:30:25.514486+0000","last_undegraded":"2024-02-05T07:30:25.514486+0000","last_fullsized":"2024-02-05T07:30:25.514486+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:59.450191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:59.450191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":612,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.506007+0000","last_change":"2024-02-05T07:28:53.115208+0000","last_active":"2024-02-05T07:30:25.506007+0000","last_peered":"2024-02-05T07:30:25.506007+0000","last_clean":"2024-02-05T07:30:25.506007+0000","last_became_active":"2024-02-05T07:18:55.469044+0000","last_became_peered":"2024-02-05T07:18:55.469044+0000","last_unstale":"2024-02-05T07:30:25.506007+0000","last_undegraded":"2024-02-05T07:30:25.506007+0000","last_fullsized":"2024-02-05T07:30:25.506007+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:53.115096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:28:53.115096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:26.926987+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":912,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.501213+0000","last_change":"2024-02-05T07:28:43.069683+0000","last_active":"2024-02-05T07:30:25.501213+0000","last_peered":"2024-02-05T07:30:25.501213+0000","last_clean":"2024-02-05T07:30:25.501213+0000","last_became_active":"2024-02-05T07:28:26.591761+0000","last_became_peered":"2024-02-05T07:28:26.591761+0000","last_unstale":"2024-02-05T07:30:25.501213+0000","last_undegraded":"2024-02-05T07:30:25.501213+0000","last_fullsized":"2024-02-05T07:30:25.501213+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:43.069567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:43.069567+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"326'19","reported_seq":620,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503762+0000","last_change":"2024-02-05T07:28:26.596454+0000","last_active":"2024-02-05T07:30:25.503762+0000","last_peered":"2024-02-05T07:30:25.503762+0000","last_clean":"2024-02-05T07:30:25.503762+0000","last_became_active":"2024-02-05T07:28:26.595964+0000","last_became_peered":"2024-02-05T07:28:26.595964+0000","last_unstale":"2024-02-05T07:30:25.503762+0000","last_undegraded":"2024-02-05T07:30:25.503762+0000","last_fullsized":"2024-02-05T07:30:25.503762+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'13","last_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:25:46.891183+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122059+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":583,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.501783+0000","last_change":"2024-02-05T07:29:49.314863+0000","last_active":"2024-02-05T07:30:25.501783+0000","last_peered":"2024-02-05T07:30:25.501783+0000","last_clean":"2024-02-05T07:30:25.501783+0000","last_became_active":"2024-02-05T07:27:16.724154+0000","last_became_peered":"2024-02-05T07:27:16.724154+0000","last_unstale":"2024-02-05T07:30:25.501783+0000","last_undegraded":"2024-02-05T07:30:25.501783+0000","last_fullsized":"2024-02-05T07:30:25.501783+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.314735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.314735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:18.065315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"266'10","reported_seq":240,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.520055+0000","last_change":"2024-02-05T07:27:17.013712+0000","last_active":"2024-02-05T07:30:25.520055+0000","last_peered":"2024-02-05T07:30:25.520055+0000","last_clean":"2024-02-05T07:30:25.520055+0000","last_became_active":"2024-02-05T07:27:16.816293+0000","last_became_peered":"2024-02-05T07:27:16.816293+0000","last_unstale":"2024-02-05T07:30:25.520055+0000","last_undegraded":"2024-02-05T07:30:25.520055+0000","last_fullsized":"2024-02-05T07:30:25.520055+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:26:35.091894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.091894+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":608,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.455360+0000","last_change":"2024-02-05T07:27:03.366653+0000","last_active":"2024-02-05T07:30:19.455360+0000","last_peered":"2024-02-05T07:30:19.455360+0000","last_clean":"2024-02-05T07:30:19.455360+0000","last_became_active":"2024-02-05T07:27:03.364334+0000","last_became_peered":"2024-02-05T07:27:03.364334+0000","last_unstale":"2024-02-05T07:30:19.455360+0000","last_undegraded":"2024-02-05T07:30:19.455360+0000","last_fullsized":"2024-02-05T07:30:19.455360+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:25.323384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:25.323384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863482+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":452,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.515287+0000","last_change":"2024-02-05T07:28:41.013449+0000","last_active":"2024-02-05T07:30:25.515287+0000","last_peered":"2024-02-05T07:30:25.515287+0000","last_clean":"2024-02-05T07:30:25.515287+0000","last_became_active":"2024-02-05T07:28:19.302393+0000","last_became_peered":"2024-02-05T07:28:19.302393+0000","last_unstale":"2024-02-05T07:30:25.515287+0000","last_undegraded":"2024-02-05T07:30:25.515287+0000","last_fullsized":"2024-02-05T07:30:25.515287+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:41.013360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:41.013360+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"326'25","reported_seq":225,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.519734+0000","last_change":"2024-02-05T07:27:17.026183+0000","last_active":"2024-02-05T07:30:25.519734+0000","last_peered":"2024-02-05T07:30:25.519734+0000","last_clean":"2024-02-05T07:30:25.519734+0000","last_became_active":"2024-02-05T07:27:16.817057+0000","last_became_peered":"2024-02-05T07:27:16.817057+0000","last_unstale":"2024-02-05T07:30:25.519734+0000","last_undegraded":"2024-02-05T07:30:25.519734+0000","last_fullsized":"2024-02-05T07:30:25.519734+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'15","last_scrub_stamp":"2024-02-05T07:25:47.927943+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.927943+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":616,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502706+0000","last_change":"2024-02-05T07:29:46.210262+0000","last_active":"2024-02-05T07:30:25.502706+0000","last_peered":"2024-02-05T07:30:25.502706+0000","last_clean":"2024-02-05T07:30:25.502706+0000","last_became_active":"2024-02-05T07:18:49.436131+0000","last_became_peered":"2024-02-05T07:18:49.436131+0000","last_unstale":"2024-02-05T07:30:25.502706+0000","last_undegraded":"2024-02-05T07:30:25.502706+0000","last_fullsized":"2024-02-05T07:30:25.502706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:46.210169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:46.210169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:10.460063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"217'9","reported_seq":377,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.506722+0000","last_change":"2024-02-05T07:28:39.104196+0000","last_active":"2024-02-05T07:30:25.506722+0000","last_peered":"2024-02-05T07:30:25.506722+0000","last_clean":"2024-02-05T07:30:25.506722+0000","last_became_active":"2024-02-05T07:25:27.246616+0000","last_became_peered":"2024-02-05T07:25:27.246616+0000","last_unstale":"2024-02-05T07:30:25.506722+0000","last_undegraded":"2024-02-05T07:30:25.506722+0000","last_fullsized":"2024-02-05T07:30:25.506722+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9","last_scrub_stamp":"2024-02-05T07:28:39.104087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:39.104087+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:26.926757+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":563,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503623+0000","last_change":"2024-02-05T07:27:16.727137+0000","last_active":"2024-02-05T07:30:25.503623+0000","last_peered":"2024-02-05T07:30:25.503623+0000","last_clean":"2024-02-05T07:30:25.503623+0000","last_became_active":"2024-02-05T07:27:16.726934+0000","last_became_peered":"2024-02-05T07:27:16.726934+0000","last_unstale":"2024-02-05T07:30:25.503623+0000","last_undegraded":"2024-02-05T07:30:25.503623+0000","last_fullsized":"2024-02-05T07:30:25.503623+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:27:01.069970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:27:01.069970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122373+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"326'19","reported_seq":668,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.454478+0000","last_change":"2024-02-05T07:28:25.780182+0000","last_active":"2024-02-05T07:30:19.454478+0000","last_peered":"2024-02-05T07:30:19.454478+0000","last_clean":"2024-02-05T07:30:19.454478+0000","last_became_active":"2024-02-05T07:28:25.779886+0000","last_became_peered":"2024-02-05T07:28:25.779886+0000","last_unstale":"2024-02-05T07:30:19.454478+0000","last_undegraded":"2024-02-05T07:30:19.454478+0000","last_fullsized":"2024-02-05T07:30:19.454478+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:25:42.192387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.192387+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863270+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":623,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.454765+0000","last_change":"2024-02-05T07:29:36.798378+0000","last_active":"2024-02-05T07:30:19.454765+0000","last_peered":"2024-02-05T07:30:19.454765+0000","last_clean":"2024-02-05T07:30:19.454765+0000","last_became_active":"2024-02-05T07:28:25.887058+0000","last_became_peered":"2024-02-05T07:28:25.887058+0000","last_unstale":"2024-02-05T07:30:19.454765+0000","last_undegraded":"2024-02-05T07:30:19.454765+0000","last_fullsized":"2024-02-05T07:30:19.454765+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:36.798282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:29:36.798282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:40.237281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":625,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.504287+0000","last_change":"2024-02-05T07:27:17.718683+0000","last_active":"2024-02-05T07:30:25.504287+0000","last_peered":"2024-02-05T07:30:25.504287+0000","last_clean":"2024-02-05T07:30:25.504287+0000","last_became_active":"2024-02-05T07:27:17.716649+0000","last_became_peered":"2024-02-05T07:27:17.716649+0000","last_unstale":"2024-02-05T07:30:25.504287+0000","last_undegraded":"2024-02-05T07:30:25.504287+0000","last_fullsized":"2024-02-05T07:30:25.504287+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.555824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.555824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":196,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.515674+0000","last_change":"2024-02-05T07:27:17.708497+0000","last_active":"2024-02-05T07:30:25.515674+0000","last_peered":"2024-02-05T07:30:25.515674+0000","last_clean":"2024-02-05T07:30:25.515674+0000","last_became_active":"2024-02-05T07:27:17.705910+0000","last_became_peered":"2024-02-05T07:27:17.705910+0000","last_unstale":"2024-02-05T07:30:25.515674+0000","last_undegraded":"2024-02-05T07:30:25.515674+0000","last_fullsized":"2024-02-05T07:30:25.515674+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.120512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"325'15","reported_seq":245,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503628+0000","last_change":"2024-02-05T07:29:57.775013+0000","last_active":"2024-02-05T07:30:25.503628+0000","last_peered":"2024-02-05T07:30:25.503628+0000","last_clean":"2024-02-05T07:30:25.503628+0000","last_became_active":"2024-02-05T07:28:26.589705+0000","last_became_peered":"2024-02-05T07:28:26.589705+0000","last_unstale":"2024-02-05T07:30:25.503628+0000","last_undegraded":"2024-02-05T07:30:25.503628+0000","last_fullsized":"2024-02-05T07:30:25.503628+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:29:57.774913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:57.774913+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.777452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":628,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503430+0000","last_change":"2024-02-05T07:29:53.220944+0000","last_active":"2024-02-05T07:30:25.503430+0000","last_peered":"2024-02-05T07:30:25.503430+0000","last_clean":"2024-02-05T07:30:25.503430+0000","last_became_active":"2024-02-05T07:27:16.707627+0000","last_became_peered":"2024-02-05T07:27:16.707627+0000","last_unstale":"2024-02-05T07:30:25.503430+0000","last_undegraded":"2024-02-05T07:30:25.503430+0000","last_fullsized":"2024-02-05T07:30:25.503430+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.220841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.220841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:06.861122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":198,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.511872+0000","last_change":"2024-02-05T07:27:16.765064+0000","last_active":"2024-02-05T07:30:25.511872+0000","last_peered":"2024-02-05T07:30:25.511872+0000","last_clean":"2024-02-05T07:30:25.511872+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:30:25.511872+0000","last_undegraded":"2024-02-05T07:30:25.511872+0000","last_fullsized":"2024-02-05T07:30:25.511872+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:26:31.506641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":372,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502110+0000","last_change":"2024-02-05T07:28:26.588019+0000","last_active":"2024-02-05T07:30:25.502110+0000","last_peered":"2024-02-05T07:30:25.502110+0000","last_clean":"2024-02-05T07:30:25.502110+0000","last_became_active":"2024-02-05T07:28:26.587701+0000","last_became_peered":"2024-02-05T07:28:26.587701+0000","last_unstale":"2024-02-05T07:30:25.502110+0000","last_undegraded":"2024-02-05T07:30:25.502110+0000","last_fullsized":"2024-02-05T07:30:25.502110+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:37.616737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:26:37.616737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":846,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.505533+0000","last_change":"2024-02-05T07:28:29.271316+0000","last_active":"2024-02-05T07:30:25.505533+0000","last_peered":"2024-02-05T07:30:25.505533+0000","last_clean":"2024-02-05T07:30:25.505533+0000","last_became_active":"2024-02-05T07:28:18.334429+0000","last_became_peered":"2024-02-05T07:28:18.334429+0000","last_unstale":"2024-02-05T07:30:25.505533+0000","last_undegraded":"2024-02-05T07:30:25.505533+0000","last_fullsized":"2024-02-05T07:30:25.505533+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:29.271046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:29.271046+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:26.926860+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"326'19","reported_seq":225,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.515853+0000","last_change":"2024-02-05T07:28:26.597041+0000","last_active":"2024-02-05T07:30:25.515853+0000","last_peered":"2024-02-05T07:30:25.515853+0000","last_clean":"2024-02-05T07:30:25.515853+0000","last_became_active":"2024-02-05T07:28:26.596304+0000","last_became_peered":"2024-02-05T07:28:26.596304+0000","last_unstale":"2024-02-05T07:30:25.515853+0000","last_undegraded":"2024-02-05T07:30:25.515853+0000","last_fullsized":"2024-02-05T07:30:25.515853+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:25:42.761130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.761130+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":221,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503711+0000","last_change":"2024-02-05T07:27:16.707735+0000","last_active":"2024-02-05T07:30:25.503711+0000","last_peered":"2024-02-05T07:30:25.503711+0000","last_clean":"2024-02-05T07:30:25.503711+0000","last_became_active":"2024-02-05T07:27:16.707402+0000","last_became_peered":"2024-02-05T07:27:16.707402+0000","last_unstale":"2024-02-05T07:30:25.503711+0000","last_undegraded":"2024-02-05T07:30:25.503711+0000","last_fullsized":"2024-02-05T07:30:25.503711+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:16.644911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:16.644911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":625,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502000+0000","last_change":"2024-02-05T07:29:58.191907+0000","last_active":"2024-02-05T07:30:25.502000+0000","last_peered":"2024-02-05T07:30:25.502000+0000","last_clean":"2024-02-05T07:30:25.502000+0000","last_became_active":"2024-02-05T07:28:25.789309+0000","last_became_peered":"2024-02-05T07:28:25.789309+0000","last_unstale":"2024-02-05T07:30:25.502000+0000","last_undegraded":"2024-02-05T07:30:25.502000+0000","last_fullsized":"2024-02-05T07:30:25.502000+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:29:58.191815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:02.876214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":210,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.511528+0000","last_change":"2024-02-05T07:29:40.983327+0000","last_active":"2024-02-05T07:30:25.511528+0000","last_peered":"2024-02-05T07:30:25.511528+0000","last_clean":"2024-02-05T07:30:25.511528+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:30:25.511528+0000","last_undegraded":"2024-02-05T07:30:25.511528+0000","last_fullsized":"2024-02-05T07:30:25.511528+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:40.983227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.983227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:04.158002+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"325'8","reported_seq":222,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.513081+0000","last_change":"2024-02-05T07:29:53.958469+0000","last_active":"2024-02-05T07:30:25.513081+0000","last_peered":"2024-02-05T07:30:25.513081+0000","last_clean":"2024-02-05T07:30:25.513081+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:30:25.513081+0000","last_undegraded":"2024-02-05T07:30:25.513081+0000","last_fullsized":"2024-02-05T07:30:25.513081+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:29:53.958384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.958384+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:10.373415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":209,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.513853+0000","last_change":"2024-02-05T07:28:46.056835+0000","last_active":"2024-02-05T07:30:25.513853+0000","last_peered":"2024-02-05T07:30:25.513853+0000","last_clean":"2024-02-05T07:30:25.513853+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:30:25.513853+0000","last_undegraded":"2024-02-05T07:30:25.513853+0000","last_fullsized":"2024-02-05T07:30:25.513853+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:46.056702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:28:46.056702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":456,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.462376+0000","last_change":"2024-02-05T07:29:58.034675+0000","last_active":"2024-02-05T07:30:19.462376+0000","last_peered":"2024-02-05T07:30:19.462376+0000","last_clean":"2024-02-05T07:30:19.462376+0000","last_became_active":"2024-02-05T07:28:25.840270+0000","last_became_peered":"2024-02-05T07:28:25.840270+0000","last_unstale":"2024-02-05T07:30:19.462376+0000","last_undegraded":"2024-02-05T07:30:19.462376+0000","last_fullsized":"2024-02-05T07:30:19.462376+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:58.034570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:29:58.034570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:16.175688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":604,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502108+0000","last_change":"2024-02-05T07:26:27.014044+0000","last_active":"2024-02-05T07:30:25.502108+0000","last_peered":"2024-02-05T07:30:25.502108+0000","last_clean":"2024-02-05T07:30:25.502108+0000","last_became_active":"2024-02-05T07:18:55.468086+0000","last_became_peered":"2024-02-05T07:18:55.468086+0000","last_unstale":"2024-02-05T07:30:25.502108+0000","last_undegraded":"2024-02-05T07:30:25.502108+0000","last_fullsized":"2024-02-05T07:30:25.502108+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:27.013942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:27.013942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122184+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"326'19","reported_seq":448,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.463862+0000","last_change":"2024-02-05T07:29:46.947670+0000","last_active":"2024-02-05T07:30:19.463862+0000","last_peered":"2024-02-05T07:30:19.463862+0000","last_clean":"2024-02-05T07:30:19.463862+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:30:19.463862+0000","last_undegraded":"2024-02-05T07:30:19.463862+0000","last_fullsized":"2024-02-05T07:30:19.463862+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:29:46.947559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:46.947559+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:51.889423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":206,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.512711+0000","last_change":"2024-02-05T07:29:38.000417+0000","last_active":"2024-02-05T07:30:25.512711+0000","last_peered":"2024-02-05T07:30:25.512711+0000","last_clean":"2024-02-05T07:30:25.512711+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:30:25.512711+0000","last_undegraded":"2024-02-05T07:30:25.512711+0000","last_fullsized":"2024-02-05T07:30:25.512711+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:38.000345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:38.000345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:42.161255+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":362,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503859+0000","last_change":"2024-02-05T07:25:30.943037+0000","last_active":"2024-02-05T07:30:25.503859+0000","last_peered":"2024-02-05T07:30:25.503859+0000","last_clean":"2024-02-05T07:30:25.503859+0000","last_became_active":"2024-02-05T07:24:34.240115+0000","last_became_peered":"2024-02-05T07:24:34.240115+0000","last_unstale":"2024-02-05T07:30:25.503859+0000","last_undegraded":"2024-02-05T07:30:25.503859+0000","last_fullsized":"2024-02-05T07:30:25.503859+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:30.942942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:30.942942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.121959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":627,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.508397+0000","last_change":"2024-02-05T07:29:55.800941+0000","last_active":"2024-02-05T07:30:25.508397+0000","last_peered":"2024-02-05T07:30:25.508397+0000","last_clean":"2024-02-05T07:30:25.508397+0000","last_became_active":"2024-02-05T07:27:03.360468+0000","last_became_peered":"2024-02-05T07:27:03.360468+0000","last_unstale":"2024-02-05T07:30:25.508397+0000","last_undegraded":"2024-02-05T07:30:25.508397+0000","last_fullsized":"2024-02-05T07:30:25.508397+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:55.800824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.800824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:15.924868+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"326'20","reported_seq":437,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503989+0000","last_change":"2024-02-05T07:29:40.154942+0000","last_active":"2024-02-05T07:30:25.503989+0000","last_peered":"2024-02-05T07:30:25.503989+0000","last_clean":"2024-02-05T07:30:25.503989+0000","last_became_active":"2024-02-05T07:27:16.726999+0000","last_became_peered":"2024-02-05T07:27:16.726999+0000","last_unstale":"2024-02-05T07:30:25.503989+0000","last_undegraded":"2024-02-05T07:30:25.503989+0000","last_fullsized":"2024-02-05T07:30:25.503989+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:29:40.154854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.154854+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:53.386629+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":649,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502004+0000","last_change":"2024-02-05T07:29:52.715347+0000","last_active":"2024-02-05T07:30:25.502004+0000","last_peered":"2024-02-05T07:30:25.502004+0000","last_clean":"2024-02-05T07:30:25.502004+0000","last_became_active":"2024-02-05T07:27:16.769199+0000","last_became_peered":"2024-02-05T07:27:16.769199+0000","last_unstale":"2024-02-05T07:30:25.502004+0000","last_undegraded":"2024-02-05T07:30:25.502004+0000","last_fullsized":"2024-02-05T07:30:25.502004+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:52.715259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:52.715259+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:11.340931+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"302'10","reported_seq":573,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.464080+0000","last_change":"2024-02-05T07:29:55.058155+0000","last_active":"2024-02-05T07:30:19.464080+0000","last_peered":"2024-02-05T07:30:19.464080+0000","last_clean":"2024-02-05T07:30:19.464080+0000","last_became_active":"2024-02-05T07:28:25.843199+0000","last_became_peered":"2024-02-05T07:28:25.843199+0000","last_unstale":"2024-02-05T07:30:19.464080+0000","last_undegraded":"2024-02-05T07:30:19.464080+0000","last_fullsized":"2024-02-05T07:30:19.464080+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'10","last_scrub_stamp":"2024-02-05T07:29:55.058040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.058040+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:21.632309+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":626,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503283+0000","last_change":"2024-02-05T07:27:16.996449+0000","last_active":"2024-02-05T07:30:25.503283+0000","last_peered":"2024-02-05T07:30:25.503283+0000","last_clean":"2024-02-05T07:30:25.503283+0000","last_became_active":"2024-02-05T07:27:16.713368+0000","last_became_peered":"2024-02-05T07:27:16.713368+0000","last_unstale":"2024-02-05T07:30:25.503283+0000","last_undegraded":"2024-02-05T07:30:25.503283+0000","last_fullsized":"2024-02-05T07:30:25.503283+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:26:18.019957+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.019957+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122251+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"326'23","reported_seq":232,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.512279+0000","last_change":"2024-02-05T07:28:47.062765+0000","last_active":"2024-02-05T07:30:25.512279+0000","last_peered":"2024-02-05T07:30:25.512279+0000","last_clean":"2024-02-05T07:30:25.512279+0000","last_became_active":"2024-02-05T07:27:16.747998+0000","last_became_peered":"2024-02-05T07:27:16.747998+0000","last_unstale":"2024-02-05T07:30:25.512279+0000","last_undegraded":"2024-02-05T07:30:25.512279+0000","last_fullsized":"2024-02-05T07:30:25.512279+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'17","last_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.062618+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":299,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.504149+0000","last_change":"2024-02-05T07:26:29.558558+0000","last_active":"2024-02-05T07:30:25.504149+0000","last_peered":"2024-02-05T07:30:25.504149+0000","last_clean":"2024-02-05T07:30:25.504149+0000","last_became_active":"2024-02-05T07:25:30.261037+0000","last_became_peered":"2024-02-05T07:25:30.261037+0000","last_unstale":"2024-02-05T07:30:25.504149+0000","last_undegraded":"2024-02-05T07:30:25.504149+0000","last_fullsized":"2024-02-05T07:30:25.504149+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:29.558353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:29.558353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":621,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.455009+0000","last_change":"2024-02-05T07:30:01.504025+0000","last_active":"2024-02-05T07:30:19.455009+0000","last_peered":"2024-02-05T07:30:19.455009+0000","last_clean":"2024-02-05T07:30:19.455009+0000","last_became_active":"2024-02-05T07:27:16.706777+0000","last_became_peered":"2024-02-05T07:27:16.706777+0000","last_unstale":"2024-02-05T07:30:19.455009+0000","last_undegraded":"2024-02-05T07:30:19.455009+0000","last_fullsized":"2024-02-05T07:30:19.455009+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:34.068335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:34.068335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863051+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":209,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.514393+0000","last_change":"2024-02-05T07:28:53.138071+0000","last_active":"2024-02-05T07:30:25.514393+0000","last_peered":"2024-02-05T07:30:25.514393+0000","last_clean":"2024-02-05T07:30:25.514393+0000","last_became_active":"2024-02-05T07:28:26.603067+0000","last_became_peered":"2024-02-05T07:28:26.603067+0000","last_unstale":"2024-02-05T07:30:25.514393+0000","last_undegraded":"2024-02-05T07:30:25.514393+0000","last_fullsized":"2024-02-05T07:30:25.514393+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:53.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:53.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"326'27","reported_seq":707,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.505751+0000","last_change":"2024-02-05T07:29:41.354568+0000","last_active":"2024-02-05T07:30:25.505751+0000","last_peered":"2024-02-05T07:30:25.505751+0000","last_clean":"2024-02-05T07:30:25.505751+0000","last_became_active":"2024-02-05T07:27:16.736296+0000","last_became_peered":"2024-02-05T07:27:16.736296+0000","last_unstale":"2024-02-05T07:30:25.505751+0000","last_undegraded":"2024-02-05T07:30:25.505751+0000","last_fullsized":"2024-02-05T07:30:25.505751+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:29:41.354464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:41.354464+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:56.393313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":360,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.454382+0000","last_change":"2024-02-05T07:25:33.059237+0000","last_active":"2024-02-05T07:30:19.454382+0000","last_peered":"2024-02-05T07:30:19.454382+0000","last_clean":"2024-02-05T07:30:19.454382+0000","last_became_active":"2024-02-05T07:24:34.234199+0000","last_became_peered":"2024-02-05T07:24:34.234199+0000","last_unstale":"2024-02-05T07:30:19.454382+0000","last_undegraded":"2024-02-05T07:30:19.454382+0000","last_fullsized":"2024-02-05T07:30:19.454382+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:33.059121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:33.059121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":609,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.507955+0000","last_change":"2024-02-05T07:26:21.548894+0000","last_active":"2024-02-05T07:30:25.507955+0000","last_peered":"2024-02-05T07:30:25.507955+0000","last_clean":"2024-02-05T07:30:25.507955+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:30:25.507955+0000","last_undegraded":"2024-02-05T07:30:25.507955+0000","last_fullsized":"2024-02-05T07:30:25.507955+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:21.548785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:21.548785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":629,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.455183+0000","last_change":"2024-02-05T07:29:59.002788+0000","last_active":"2024-02-05T07:30:19.455183+0000","last_peered":"2024-02-05T07:30:19.455183+0000","last_clean":"2024-02-05T07:30:19.455183+0000","last_became_active":"2024-02-05T07:28:26.590448+0000","last_became_peered":"2024-02-05T07:28:26.590448+0000","last_unstale":"2024-02-05T07:30:19.455183+0000","last_undegraded":"2024-02-05T07:30:19.455183+0000","last_fullsized":"2024-02-05T07:30:19.455183+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.002677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.002677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:25.104559+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"325'7","reported_seq":647,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.454262+0000","last_change":"2024-02-05T07:27:16.970973+0000","last_active":"2024-02-05T07:30:19.454262+0000","last_peered":"2024-02-05T07:30:19.454262+0000","last_clean":"2024-02-05T07:30:19.454262+0000","last_became_active":"2024-02-05T07:27:16.745897+0000","last_became_peered":"2024-02-05T07:27:16.745897+0000","last_unstale":"2024-02-05T07:30:19.454262+0000","last_undegraded":"2024-02-05T07:30:19.454262+0000","last_fullsized":"2024-02-05T07:30:19.454262+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:26:35.253452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.253452+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863406+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":438,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.454678+0000","last_change":"2024-02-05T07:28:26.591843+0000","last_active":"2024-02-05T07:30:19.454678+0000","last_peered":"2024-02-05T07:30:19.454678+0000","last_clean":"2024-02-05T07:30:19.454678+0000","last_became_active":"2024-02-05T07:28:26.590322+0000","last_became_peered":"2024-02-05T07:28:26.590322+0000","last_unstale":"2024-02-05T07:30:19.454678+0000","last_undegraded":"2024-02-05T07:30:19.454678+0000","last_fullsized":"2024-02-05T07:30:19.454678+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:35.058925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:35.058925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":608,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.501927+0000","last_change":"2024-02-05T07:27:00.183499+0000","last_active":"2024-02-05T07:30:25.501927+0000","last_peered":"2024-02-05T07:30:25.501927+0000","last_clean":"2024-02-05T07:30:25.501927+0000","last_became_active":"2024-02-05T07:18:53.456881+0000","last_became_peered":"2024-02-05T07:18:53.456881+0000","last_unstale":"2024-02-05T07:30:25.501927+0000","last_undegraded":"2024-02-05T07:30:25.501927+0000","last_fullsized":"2024-02-05T07:30:25.501927+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:27:00.183281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:27:00.183281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:28.122344+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":424,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502327+0000","last_change":"2024-02-05T07:29:50.742042+0000","last_active":"2024-02-05T07:30:25.502327+0000","last_peered":"2024-02-05T07:30:25.502327+0000","last_clean":"2024-02-05T07:30:25.502327+0000","last_became_active":"2024-02-05T07:27:16.769581+0000","last_became_peered":"2024-02-05T07:27:16.769581+0000","last_unstale":"2024-02-05T07:30:25.502327+0000","last_undegraded":"2024-02-05T07:30:25.502327+0000","last_fullsized":"2024-02-05T07:30:25.502327+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:50.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:18.589144+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"325'19","reported_seq":679,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.454319+0000","last_change":"2024-02-05T07:30:00.001976+0000","last_active":"2024-02-05T07:30:19.454319+0000","last_peered":"2024-02-05T07:30:19.454319+0000","last_clean":"2024-02-05T07:30:19.454319+0000","last_became_active":"2024-02-05T07:27:16.707036+0000","last_became_peered":"2024-02-05T07:27:16.707036+0000","last_unstale":"2024-02-05T07:30:19.454319+0000","last_undegraded":"2024-02-05T07:30:19.454319+0000","last_fullsized":"2024-02-05T07:30:19.454319+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'19","last_scrub_stamp":"2024-02-05T07:30:00.001863+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:30:00.001863+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:04.904819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":449,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.455864+0000","last_change":"2024-02-05T07:29:53.022701+0000","last_active":"2024-02-05T07:30:19.455864+0000","last_peered":"2024-02-05T07:30:19.455864+0000","last_clean":"2024-02-05T07:30:19.455864+0000","last_became_active":"2024-02-05T07:28:26.616107+0000","last_became_peered":"2024-02-05T07:28:26.616107+0000","last_unstale":"2024-02-05T07:30:19.455864+0000","last_undegraded":"2024-02-05T07:30:19.455864+0000","last_fullsized":"2024-02-05T07:30:19.455864+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.022606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.022606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:18.495044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":223,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.503230+0000","last_change":"2024-02-05T07:29:54.207175+0000","last_active":"2024-02-05T07:30:25.503230+0000","last_peered":"2024-02-05T07:30:25.503230+0000","last_clean":"2024-02-05T07:30:25.503230+0000","last_became_active":"2024-02-05T07:27:03.360715+0000","last_became_peered":"2024-02-05T07:27:03.360715+0000","last_unstale":"2024-02-05T07:30:25.503230+0000","last_undegraded":"2024-02-05T07:30:25.503230+0000","last_fullsized":"2024-02-05T07:30:25.503230+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:54.207089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:54.207089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:05.053234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":370,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502281+0000","last_change":"2024-02-05T07:29:31.484681+0000","last_active":"2024-02-05T07:30:25.502281+0000","last_peered":"2024-02-05T07:30:25.502281+0000","last_clean":"2024-02-05T07:30:25.502281+0000","last_became_active":"2024-02-05T07:24:34.230537+0000","last_became_peered":"2024-02-05T07:24:34.230537+0000","last_unstale":"2024-02-05T07:30:25.502281+0000","last_undegraded":"2024-02-05T07:30:25.502281+0000","last_fullsized":"2024-02-05T07:30:25.502281+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:31.484580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:31.484580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:59.560324+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"326'27","reported_seq":232,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.516143+0000","last_change":"2024-02-05T07:27:17.002461+0000","last_active":"2024-02-05T07:30:25.516143+0000","last_peered":"2024-02-05T07:30:25.516143+0000","last_clean":"2024-02-05T07:30:25.516143+0000","last_became_active":"2024-02-05T07:27:16.751179+0000","last_became_peered":"2024-02-05T07:27:16.751179+0000","last_unstale":"2024-02-05T07:30:25.516143+0000","last_undegraded":"2024-02-05T07:30:25.516143+0000","last_fullsized":"2024-02-05T07:30:25.516143+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:25:38.772812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:38.772812+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":202,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.513043+0000","last_change":"2024-02-05T07:28:26.596258+0000","last_active":"2024-02-05T07:30:25.513043+0000","last_peered":"2024-02-05T07:30:25.513043+0000","last_clean":"2024-02-05T07:30:25.513043+0000","last_became_active":"2024-02-05T07:28:26.595627+0000","last_became_peered":"2024-02-05T07:28:26.595627+0000","last_unstale":"2024-02-05T07:30:25.513043+0000","last_undegraded":"2024-02-05T07:30:25.513043+0000","last_fullsized":"2024-02-05T07:30:25.513043+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:50.859511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:50.859511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":623,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.454068+0000","last_change":"2024-02-05T07:29:48.045121+0000","last_active":"2024-02-05T07:30:19.454068+0000","last_peered":"2024-02-05T07:30:19.454068+0000","last_clean":"2024-02-05T07:30:19.454068+0000","last_became_active":"2024-02-05T07:28:25.778935+0000","last_became_peered":"2024-02-05T07:28:25.778935+0000","last_unstale":"2024-02-05T07:30:19.454068+0000","last_undegraded":"2024-02-05T07:30:19.454068+0000","last_fullsized":"2024-02-05T07:30:19.454068+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.045013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.045013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:48.765921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":210,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.511872+0000","last_change":"2024-02-05T07:28:44.050127+0000","last_active":"2024-02-05T07:30:25.511872+0000","last_peered":"2024-02-05T07:30:25.511872+0000","last_clean":"2024-02-05T07:30:25.511872+0000","last_became_active":"2024-02-05T07:28:26.598232+0000","last_became_peered":"2024-02-05T07:28:26.598232+0000","last_unstale":"2024-02-05T07:30:25.511872+0000","last_undegraded":"2024-02-05T07:30:25.511872+0000","last_fullsized":"2024-02-05T07:30:25.511872+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:44.050019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:44.050019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"325'15","reported_seq":644,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.453060+0000","last_change":"2024-02-05T07:29:33.839515+0000","last_active":"2024-02-05T07:30:19.453060+0000","last_peered":"2024-02-05T07:30:19.453060+0000","last_clean":"2024-02-05T07:30:19.453060+0000","last_became_active":"2024-02-05T07:28:25.777815+0000","last_became_peered":"2024-02-05T07:28:25.777815+0000","last_unstale":"2024-02-05T07:30:19.453060+0000","last_undegraded":"2024-02-05T07:30:19.453060+0000","last_fullsized":"2024-02-05T07:30:19.453060+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:29:33.839402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:33.839402+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:01.714433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":388,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.507532+0000","last_change":"2024-02-05T07:29:50.274850+0000","last_active":"2024-02-05T07:30:25.507532+0000","last_peered":"2024-02-05T07:30:25.507532+0000","last_clean":"2024-02-05T07:30:25.507532+0000","last_became_active":"2024-02-05T07:28:26.588496+0000","last_became_peered":"2024-02-05T07:28:26.588496+0000","last_unstale":"2024-02-05T07:30:25.507532+0000","last_undegraded":"2024-02-05T07:30:25.507532+0000","last_fullsized":"2024-02-05T07:30:25.507532+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.274768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.936836+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":615,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T07:30:25.502056+0000","last_change":"2024-02-05T07:27:03.354044+0000","last_active":"2024-02-05T07:30:25.502056+0000","last_peered":"2024-02-05T07:30:25.502056+0000","last_clean":"2024-02-05T07:30:25.502056+0000","last_became_active":"2024-02-05T07:27:03.353249+0000","last_became_peered":"2024-02-05T07:27:03.353249+0000","last_unstale":"2024-02-05T07:30:25.502056+0000","last_undegraded":"2024-02-05T07:30:25.502056+0000","last_fullsized":"2024-02-05T07:30:25.502056+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:28.575859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:28.575859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":608,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-05T07:30:19.454087+0000","last_change":"2024-02-05T07:27:03.366854+0000","last_active":"2024-02-05T07:30:19.454087+0000","last_peered":"2024-02-05T07:30:19.454087+0000","last_clean":"2024-02-05T07:30:19.454087+0000","last_became_active":"2024-02-05T07:27:03.366576+0000","last_became_peered":"2024-02-05T07:27:03.366576+0000","last_unstale":"2024-02-05T07:30:19.454087+0000","last_undegraded":"2024-02-05T07:30:19.454087+0000","last_fullsized":"2024-02-05T07:30:19.454087+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:21.317417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:21.317417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:23.863337+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1524236288,"num_objects":23258,"num_object_clones":0,"num_object_copies":69774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23258,"num_write_kb":1488512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4562354176,"data_stored":4562354176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23258,"ondisk_log_size":23258,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":687,"num_read_kb":682,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":4786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":11436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":737,"ondisk_log_size":737,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037904,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1399868,"kb_used_data":913332,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92345284,"statfs":{"total":95995035648,"available":94561570816,"internally_reserved":0,"allocated":935251968,"data_stored":934312569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":498203082},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":2.1179999999999999,"15min":2.0289999999999999},"min":{"1min":0.255,"5min":0.214,"15min":0.20000000000000001},"max":{"1min":4.9320000000000004,"5min":56.420999999999999,"15min":63.039000000000001},"last":3.206},{"interface":"front","average":{"1min":0.95699999999999996,"5min":2.1469999999999998,"15min":2.0310000000000001},"min":{"1min":0.41899999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.4160000000000004,"5min":56.433,"15min":65.296000000000006},"last":0.88900000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":3.5110000000000001,"15min":1.9990000000000001},"min":{"1min":0.33900000000000002,"5min":0.219,"15min":0.219},"max":{"1min":4.766,"5min":57.502000000000002,"15min":57.502000000000002},"last":1.234},{"interface":"front","average":{"1min":1.2190000000000001,"5min":3.234,"15min":1.9450000000000001},"min":{"1min":0.28999999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.9489999999999998,"5min":57.479999999999997,"15min":57.479999999999997},"last":0.85299999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.1519999999999999,"15min":1.3100000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.249},"max":{"1min":2.8839999999999999,"5min":12.131,"15min":51.127000000000002},"last":4.899},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.2190000000000001,"15min":1.4410000000000001},"min":{"1min":0.35599999999999998,"5min":0.27200000000000002,"15min":0.246},"max":{"1min":1.756,"5min":11.141,"15min":51.113999999999997},"last":3.3250000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":2.4540000000000002,"15min":1.6100000000000001},"min":{"1min":0.41099999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":7.3979999999999997,"5min":56.527999999999999,"15min":58.341999999999999},"last":0.437},{"interface":"front","average":{"1min":0.89000000000000001,"5min":2.052,"15min":1.4590000000000001},"min":{"1min":0.46400000000000002,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":4.415,"5min":56.328000000000003,"15min":57.765000000000001},"last":1.7490000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.63,"15min":0.54600000000000004},"min":{"1min":0.31900000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.8520000000000001,"5min":4.5999999999999996,"15min":4.8659999999999997},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.75600000000000001,"15min":0.58699999999999997},"min":{"1min":0.32500000000000001,"5min":0.221,"15min":0.221},"max":{"1min":1.595,"5min":6.6219999999999999,"15min":6.6219999999999999},"last":0.376}]},{"osd":5,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":1,"15min":0.69799999999999995},"min":{"1min":0.32500000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.5669999999999999,"5min":18.890999999999998,"15min":18.890999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.91100000000000003,"15min":0.70299999999999996},"min":{"1min":0.29299999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.8129999999999999,"5min":17.081,"15min":17.081},"last":0.36499999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:29:25 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.8969999999999998,"5min":2.8969999999999998,"15min":2.8969999999999998},"last":0.41499999999999998}]}]},{"osd":6,"up_from":301,"seq":1292785156121,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":466416,"kb_used_data":293736,"kb_used_omap":5,"kb_used_meta":172666,"kb_avail":93278736,"statfs":{"total":95995035648,"available":95517425664,"internally_reserved":0,"allocated":300785664,"data_stored":299859111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":176810501},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":14.218,"5min":14.218,"15min":14.218},"last":0.50900000000000001},{"interface":"front","average":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":14.175000000000001,"5min":14.175000000000001,"15min":14.175000000000001},"last":1.012}]},{"osd":1,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":15.473000000000001,"5min":15.473000000000001,"15min":15.473000000000001},"last":0.629},{"interface":"front","average":{"1min":2.6080000000000001,"5min":2.6080000000000001,"15min":2.6080000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":14.784000000000001,"5min":14.784000000000001,"15min":14.784000000000001},"last":0.80100000000000005}]},{"osd":2,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":2.984,"5min":2.984,"15min":2.984},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":34.689999999999998,"5min":34.689999999999998,"15min":34.689999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":34.646999999999998,"5min":34.646999999999998,"15min":34.646999999999998},"last":0.88600000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.1890000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":17.600000000000001,"5min":17.600000000000001,"15min":17.600000000000001},"last":2.5499999999999998},{"interface":"front","average":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":23.425000000000001,"5min":23.425000000000001,"15min":23.425000000000001},"last":2.1429999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.869,"5min":1.869,"15min":1.869},"last":1.0720000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":0.39300000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":5.5949999999999998,"5min":5.5949999999999998,"15min":5.5949999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":84.158000000000001,"5min":84.158000000000001,"15min":84.158000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":5.7720000000000002,"5min":5.7720000000000002,"15min":5.7720000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":85.018000000000001,"5min":85.018000000000001,"15min":85.018000000000001},"last":0.87}]}]},{"osd":1,"up_from":13,"seq":55834575022,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1376596,"kb_used_data":899788,"kb_used_omap":1,"kb_used_meta":476798,"kb_avail":92368556,"statfs":{"total":95995035648,"available":94585401344,"internally_reserved":0,"allocated":921382912,"data_stored":920419870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":488241605},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:29:40 2024","interfaces":[{"interface":"back","average":{"1min":2.7330000000000001,"5min":1.7150000000000001,"15min":1.9359999999999999},"min":{"1min":0.28299999999999997,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.09,"5min":15.09,"15min":45.393000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":2.3380000000000001,"5min":1.593,"15min":1.883},"min":{"1min":0.23799999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":14.207000000000001,"5min":14.207000000000001,"15min":45.031999999999996},"last":5.242}]},{"osd":2,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":2.0830000000000002,"15min":1.5900000000000001},"min":{"1min":0.26900000000000002,"5min":0.218,"15min":0.20699999999999999},"max":{"1min":15.137,"5min":15.137,"15min":39.259999999999998},"last":0.442},{"interface":"front","average":{"1min":2.5870000000000002,"5min":1.98,"15min":1.651},"min":{"1min":0.29899999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.906000000000001,"5min":14.906000000000001,"15min":40.972000000000001},"last":0.26700000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:30:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.7050000000000001,"15min":1.4930000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20999999999999999},"max":{"1min":5.2389999999999999,"5min":19.829999999999998,"15min":43.823},"last":5.343},{"interface":"front","average":{"1min":1.034,"5min":1.6120000000000001,"15min":1.524},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24199999999999999},"max":{"1min":6.3949999999999996,"5min":9.7880000000000003,"15min":39.314999999999998},"last":0.67100000000000004}]},{"osd":4,"last update":"Mon Feb 5 07:29:40 2024","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":2.1949999999999998,"15min":1.579},"min":{"1min":0.38800000000000001,"5min":0.34799999999999998,"15min":0.312},"max":{"1min":4.7320000000000002,"5min":61.722000000000001,"15min":61.722000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.853,"5min":2.1629999999999998,"15min":1.6539999999999999},"min":{"1min":0.39200000000000002,"5min":0.311,"15min":0.29399999999999998},"max":{"1min":5.5590000000000002,"5min":61.622999999999998,"15min":61.786000000000001},"last":0.38400000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:30:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":2.4649999999999999,"15min":1.6639999999999999},"min":{"1min":0.34999999999999998,"5min":0.309,"15min":0.309},"max":{"1min":6.319,"5min":106.672,"15min":106.672},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.048,"5min":2.5009999999999999,"15min":1.7709999999999999},"min":{"1min":0.38400000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.0869999999999997,"5min":107.348,"15min":107.348},"last":5.2830000000000004}]},{"osd":6,"last update":"Mon Feb 5 07:29:26 2024","interfaces":[{"interface":"back","average":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":5.5990000000000002,"5min":5.5990000000000002,"15min":5.5990000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":6.3559999999999999},"last":0.47599999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.5469999999999999,"15min":1.4219999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":5.0519999999999996,"5min":20.309000000000001,"15min":66.703999999999994},"last":5.3159999999999998},{"interface":"front","average":{"1min":1.456,"5min":1.6379999999999999,"15min":1.4370000000000001},"min":{"1min":0.47599999999999998,"5min":0.34000000000000002,"15min":0.33500000000000002},"max":{"1min":5.6539999999999999,"5min":11.263999999999999,"15min":61.594999999999999},"last":0.56899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738547,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887136,"kb_used_data":374228,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":92858016,"statfs":{"total":95995035648,"available":95086608384,"internally_reserved":0,"allocated":383209472,"data_stored":382244571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":525203909},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:29:30 2024","interfaces":[{"interface":"back","average":{"1min":3.972,"5min":2.7719999999999998,"15min":1.581},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":32.546999999999997,"5min":45.372,"15min":45.372},"last":0.63300000000000001},{"interface":"front","average":{"1min":3.9580000000000002,"5min":2.75,"15min":1.6619999999999999},"min":{"1min":0.23999999999999999,"5min":0.217,"15min":0.186},"max":{"1min":29.675999999999998,"5min":45.231999999999999,"15min":45.231999999999999},"last":0.26200000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.575,"15min":1.5589999999999999},"min":{"1min":0.23799999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.1760000000000002,"5min":44.920000000000002,"15min":65.903999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":1.579,"15min":1.534},"min":{"1min":0.224,"5min":0.224,"15min":0.182},"max":{"1min":4.8220000000000001,"5min":44.947000000000003,"15min":65.694000000000003},"last":0.42199999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:30:14 2024","interfaces":[{"interface":"back","average":{"1min":3.6709999999999998,"5min":1.8320000000000001,"15min":1.8169999999999999},"min":{"1min":0.254,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":29.629000000000001,"5min":29.629000000000001,"15min":65.155000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":3.1930000000000001,"5min":1.871,"15min":1.903},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":29.463999999999999,"5min":29.463999999999999,"15min":65.114000000000004},"last":0.50900000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:29:35 2024","interfaces":[{"interface":"back","average":{"1min":3.3149999999999999,"5min":2.9969999999999999,"15min":1.8540000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":42.901000000000003,"5min":50.545000000000002,"15min":65.188000000000002},"last":0.374},{"interface":"front","average":{"1min":3.7410000000000001,"5min":3.0059999999999998,"15min":1.8939999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.23699999999999999},"max":{"1min":42.960999999999999,"5min":50.310000000000002,"15min":64.873000000000005},"last":0.39400000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:30:14 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":2.113,"15min":1.613},"min":{"1min":0.32100000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.1800000000000002,"5min":95.772999999999996,"15min":95.772999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.76900000000000002,"5min":2.081,"15min":1.663},"min":{"1min":0.29299999999999998,"5min":0.28199999999999997,"15min":0.27600000000000002},"max":{"1min":3.9860000000000002,"5min":96.028000000000006,"15min":96.028000000000006},"last":0.52400000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:29:25 2024","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.411,"15min":1.411},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.6929999999999996,"5min":4.6929999999999996,"15min":4.6929999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":5.806,"5min":5.806,"15min":5.806},"last":0.48599999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:29:45 2024","interfaces":[{"interface":"back","average":{"1min":3.3399999999999999,"5min":1.923,"15min":1.645},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":43.012,"5min":43.012,"15min":65.263000000000005},"last":0.78600000000000003},{"interface":"front","average":{"1min":3.7040000000000002,"5min":1.9199999999999999,"15min":1.6419999999999999},"min":{"1min":0.34000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":43.512,"5min":43.512,"15min":64.906000000000006},"last":0.58999999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411497,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1090796,"kb_used_data":554276,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92654356,"statfs":{"total":95995035648,"available":94878060544,"internally_reserved":0,"allocated":567578624,"data_stored":566656090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":2.8260000000000001,"15min":1.875},"min":{"1min":0.23100000000000001,"5min":0.20699999999999999,"15min":0.182},"max":{"1min":12.759,"5min":92.638000000000005,"15min":92.638000000000005},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.879,"5min":2.7559999999999998,"15min":1.962},"min":{"1min":0.29099999999999998,"5min":0.22600000000000001,"15min":0.20699999999999999},"max":{"1min":12.647,"5min":93.481999999999999,"15min":93.481999999999999},"last":0.59199999999999997}]},{"osd":1,"last update":"Mon Feb 5 07:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.9830000000000001,"5min":3.1560000000000001,"15min":1.635},"min":{"1min":0.26200000000000001,"5min":0.20999999999999999,"15min":0.19400000000000001},"max":{"1min":17.911999999999999,"5min":93.504000000000005,"15min":93.504000000000005},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.9399999999999999,"5min":3.2650000000000001,"15min":1.7330000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":11.464,"5min":93.004000000000005,"15min":93.004000000000005},"last":0.376}]},{"osd":3,"last update":"Mon Feb 5 07:30:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":1.9399999999999999,"15min":1.3939999999999999},"min":{"1min":0.27900000000000003,"5min":0.25600000000000001,"15min":0.214},"max":{"1min":2.8580000000000001,"5min":26.114999999999998,"15min":26.114999999999998},"last":1.012},{"interface":"front","average":{"1min":0.505,"5min":1.5109999999999999,"15min":1.2689999999999999},"min":{"1min":0.28100000000000003,"5min":0.22600000000000001,"15min":0.193},"max":{"1min":1.5509999999999999,"5min":26.216000000000001,"15min":26.216000000000001},"last":0.71899999999999997}]},{"osd":4,"last update":"Mon Feb 5 07:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":4.085,"15min":1.944},"min":{"1min":0.377,"5min":0.318,"15min":0.24099999999999999},"max":{"1min":5.4829999999999997,"5min":168.11699999999999,"15min":168.11699999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.2290000000000001,"5min":3.9780000000000002,"15min":1.927},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":5.6680000000000001,"5min":168.589,"15min":168.589},"last":0.40899999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:30:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":1.597,"15min":1.1659999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":3.2549999999999999,"5min":39.923000000000002,"15min":39.923000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.554,"15min":1.113},"min":{"1min":0.35199999999999998,"5min":0.33600000000000002,"15min":0.26400000000000001},"max":{"1min":3.5950000000000002,"5min":38.920999999999999,"15min":38.920999999999999},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:29:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":3.6749999999999998,"5min":3.6749999999999998,"15min":3.6749999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.512,"5min":1.512,"15min":1.512},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":5.7240000000000002,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.49399999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.744,"15min":1.256},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":5.5099999999999998,"5min":37.573999999999998,"15min":37.573999999999998},"last":0.628},{"interface":"front","average":{"1min":0.89900000000000002,"5min":1.6799999999999999,"15min":1.1930000000000001},"min":{"1min":0.35999999999999999,"5min":0.34699999999999998,"15min":0.33800000000000002},"max":{"1min":3.6480000000000001,"5min":37.432000000000002,"15min":37.432000000000002},"last":0.436}]}]},{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1411216,"kb_used_data":888584,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92333936,"statfs":{"total":95995035648,"available":94549950464,"internally_reserved":0,"allocated":909910016,"data_stored":908950971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":535165383},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:29:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.575,"15min":2.0310000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":6.1349999999999998,"5min":32.078000000000003,"15min":74.114999999999995},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.7869999999999999,"15min":1.8200000000000001},"min":{"1min":0.24199999999999999,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":4.8449999999999998,"5min":34.598999999999997,"15min":48.740000000000002},"last":0.32100000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:29:29 2024","interfaces":[{"interface":"back","average":{"1min":5.6319999999999997,"5min":2.5329999999999999,"15min":1.931},"min":{"1min":0.27900000000000003,"5min":0.27300000000000002,"15min":0.253},"max":{"1min":59.298999999999999,"5min":59.298999999999999,"15min":59.298999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":4.798,"5min":2.5449999999999999,"15min":2.198},"min":{"1min":0.29499999999999998,"5min":0.27900000000000003,"15min":0.185},"max":{"1min":59.634,"5min":59.634,"15min":74.084999999999994},"last":0.255}]},{"osd":2,"last update":"Mon Feb 5 07:29:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.3740000000000001,"15min":1.97},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.6529999999999996,"5min":29.123999999999999,"15min":74.051000000000002},"last":0.30499999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.262,"15min":1.5209999999999999},"min":{"1min":0.27000000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2709999999999999,"5min":7.8769999999999998,"15min":65.861999999999995},"last":0.28599999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.4670000000000001,"15min":1.0880000000000001},"min":{"1min":0.41399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.7290000000000001,"5min":34.981000000000002,"15min":34.981000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.4830000000000001,"15min":1.083},"min":{"1min":0.38800000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":7.9500000000000002,"5min":36.987000000000002,"15min":36.987000000000002},"last":0.41399999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:30:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.876,"15min":1.3520000000000001},"min":{"1min":0.38800000000000001,"5min":0.32900000000000001,"15min":0.28499999999999998},"max":{"1min":2.6579999999999999,"5min":4.1100000000000003,"15min":73.727000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":1.1279999999999999,"15min":1.1000000000000001},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.27400000000000002},"max":{"1min":2.6840000000000002,"5min":6.3319999999999999,"15min":21.126000000000001},"last":0.437}]},{"osd":6,"last update":"Mon Feb 5 07:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.77,"5min":2.77,"15min":2.77},"last":0.57799999999999996}]},{"osd":7,"last update":"Mon Feb 5 07:29:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":2.4870000000000001,"15min":1.498},"min":{"1min":0.42699999999999999,"5min":0.318,"15min":0.318},"max":{"1min":7.984,"5min":91.051000000000002,"15min":91.051000000000002},"last":29.141999999999999},{"interface":"front","average":{"1min":0.999,"5min":2.431,"15min":1.833},"min":{"1min":0.46700000000000003,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":4.0599999999999996,"5min":91.924000000000007,"15min":91.924000000000007},"last":29.603000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953631,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":793340,"kb_used_data":360948,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92951812,"statfs":{"total":95995035648,"available":95182655488,"internally_reserved":0,"allocated":369610752,"data_stored":368670687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:29:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":4.0430000000000001,"15min":2.0819999999999999},"min":{"1min":0.22,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.103999999999999,"5min":101.212,"15min":101.212},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.9039999999999999,"5min":4.0060000000000002,"15min":2.1640000000000001},"min":{"1min":0.27300000000000002,"5min":0.23899999999999999,"15min":0.23300000000000001},"max":{"1min":13.169,"5min":102.07599999999999,"15min":102.07599999999999},"last":0.58399999999999996}]},{"osd":1,"last update":"Mon Feb 5 07:29:40 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":4.0439999999999996,"15min":1.8089999999999999},"min":{"1min":0.28699999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.343999999999999,"5min":69.102000000000004,"15min":69.102000000000004},"last":0.53900000000000003},{"interface":"front","average":{"1min":2.9830000000000001,"5min":4.7249999999999996,"15min":2.0179999999999998},"min":{"1min":0.24399999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":18.638999999999999,"5min":68.108999999999995,"15min":68.108999999999995},"last":0.69399999999999995}]},{"osd":2,"last update":"Mon Feb 5 07:29:40 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.841,"15min":1.3089999999999999},"min":{"1min":0.27600000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.266,"5min":26.045000000000002,"15min":67.144999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.724,"15min":1.2809999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.1689999999999996,"5min":25.960999999999999,"15min":66.959999999999994},"last":0.91500000000000004}]},{"osd":3,"last update":"Mon Feb 5 07:29:40 2024","interfaces":[{"interface":"back","average":{"1min":2.3410000000000002,"5min":2.2000000000000002,"15min":1.206},"min":{"1min":0.40500000000000003,"5min":0.31,"15min":0.27500000000000002},"max":{"1min":13.124000000000001,"5min":66.019999999999996,"15min":66.019999999999996},"last":1.0409999999999999},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.5390000000000001,"15min":1.3879999999999999},"min":{"1min":0.36299999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.202999999999999,"5min":101.001,"15min":101.001},"last":0.48899999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.82599999999999996,"15min":0.57899999999999996},"min":{"1min":0.28199999999999997,"5min":0.25800000000000001,"15min":0.25700000000000001},"max":{"1min":1.907,"5min":10.917999999999999,"15min":10.917999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.83399999999999996,"15min":0.60499999999999998},"min":{"1min":0.317,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.2410000000000001,"5min":15.105,"15min":15.105},"last":0.39000000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":14.606,"5min":14.606,"15min":14.606},"last":0.36499999999999999},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":13.114000000000001,"5min":13.114000000000001,"15min":13.114000000000001},"last":0.55600000000000005}]},{"osd":7,"last update":"Mon Feb 5 07:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.97699999999999998,"15min":0.72199999999999998},"min":{"1min":0.308,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.8260000000000001,"5min":15.073,"15min":15.073},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.94799999999999995,"15min":0.67900000000000005},"min":{"1min":0.39500000000000002,"5min":0.248,"15min":0.248},"max":{"1min":3.0670000000000002,"5min":14.581,"15min":14.581},"last":0.45600000000000002}]}]},{"osd":5,"up_from":0,"seq":176093659290,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":386138112,"data_stored":386138112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":988086272,"data_stored":988086272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":577699840,"data_stored":577699840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":954400768,"data_stored":954400768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":385548288,"data_stored":385548288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":312410112,"data_stored":312410112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":958070784,"data_stored":958070784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:30:31.780 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-05T07:30:31.781 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:30:31.781 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 4, 6, 7, 3] 2024-02-05T07:30:31.781 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd out 7 2024-02-05T07:30:31.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:31.874 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:31.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:31.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:31.930 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 16 16 32044 32028 125.095 101 0.0125094 0.00798675 2024-02-05T07:30:31.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:32.007 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:32.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:31 smithi090 ceph-mon[33213]: 4.f scrub starts 2024-02-05T07:30:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:31 smithi090 ceph-mon[33213]: 4.f scrub ok 2024-02-05T07:30:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:31 smithi090 ceph-mon[33213]: 5.4 scrub starts 2024-02-05T07:30:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:31 smithi090 ceph-mon[33213]: 5.4 scrub ok 2024-02-05T07:30:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:31 smithi090 ceph-mon[33213]: osdmap e360: 8 total, 8 up, 7 in 2024-02-05T07:30:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:31 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:31 smithi090 ceph-mon[33213]: 3.8 scrub starts 2024-02-05T07:30:32.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:31 smithi090 ceph-mon[33213]: 3.8 scrub ok 2024-02-05T07:30:32.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:32.140 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:32.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:32.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:32.272 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:32.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:32.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:32.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:32.415 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:32.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:32.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:32.567 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:32.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:32.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:32.700 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:32.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:32.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:32.802 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:32.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:30:32.834 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:32.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:32.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:32 smithi007 ceph-mon[29098]: pgmap v609: 168 pgs: 168 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.8 KiB/s rd, 182 MiB/s wr, 2.91k op/s; 8 B/s, 0 objects/s recovering 2024-02-05T07:30:32.846 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:32 smithi007 ceph-mon[37158]: pgmap v609: 168 pgs: 168 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.8 KiB/s rd, 182 MiB/s wr, 2.91k op/s; 8 B/s, 0 objects/s recovering 2024-02-05T07:30:32.930 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 17 16 33720 33704 123.897 104.75 0.00219111 0.00805551 2024-02-05T07:30:32.936 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:32.966 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:32.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:32.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:32 smithi090 ceph-mon[33213]: pgmap v609: 168 pgs: 168 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.8 KiB/s rd, 182 MiB/s wr, 2.91k op/s; 8 B/s, 0 objects/s recovering 2024-02-05T07:30:33.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:33.102 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:33.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:33.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:33.237 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:33.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:33.339 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:33.387 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:33.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:33.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:33.517 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:33.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:33.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:33.569 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:33.574 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:33.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:33.630 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:30:33.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:33 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:30:33.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:33 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:30:33.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:33 smithi007 ceph-mon[29098]: osdmap e361: 8 total, 8 up, 7 in 2024-02-05T07:30:33.667 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:33 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:30:33.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:33 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:30:33.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:33 smithi007 ceph-mon[37158]: osdmap e361: 8 total, 8 up, 7 in 2024-02-05T07:30:33.668 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:30:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:30:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:30:33.670 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:30:33.670 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:33.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:33.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:33.715 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:30:33.756 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:30:33.772 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:33.798 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:30:33.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:33.820 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:33.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:33.822 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:33 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:30:33.822 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:33 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:30:33.822 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:33 smithi090 ceph-mon[33213]: osdmap e361: 8 total, 8 up, 7 in 2024-02-05T07:30:33.839 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:30:33.882 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:30:33.887 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:30:33.887 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:30:33.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:33.929 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:30:33.930 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 18 16 35046 35030 121.616 82.875 0.0130742 0.00821657 2024-02-05T07:30:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:30:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:30:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:30:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:30:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:30:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:30:33.939 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:30:33.939 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:30:33.952 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:33.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:33.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:34.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:34.085 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:34.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:34.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:34.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:34.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:34.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:34.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:34.349 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:34.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:34.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:34.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:34.481 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:34.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:34.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:34.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:34.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:34.625 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:34.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:34.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T07:30:34.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[29098]: 3.3 scrub starts 2024-02-05T07:30:34.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[29098]: 3.3 scrub ok 2024-02-05T07:30:34.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[29098]: 3.11 scrub starts 2024-02-05T07:30:34.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[29098]: 3.11 scrub ok 2024-02-05T07:30:34.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[29098]: pgmap v611: 168 pgs: 168 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.13k op/s; 7 B/s, 0 objects/s recovering 2024-02-05T07:30:34.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4206751001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:30:34.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[29098]: 4.14 scrub starts 2024-02-05T07:30:34.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[29098]: 4.14 scrub ok 2024-02-05T07:30:34.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[29098]: osdmap e362: 8 total, 8 up, 7 in 2024-02-05T07:30:34.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[37158]: 3.3 scrub starts 2024-02-05T07:30:34.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[37158]: 3.3 scrub ok 2024-02-05T07:30:34.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[37158]: 3.11 scrub starts 2024-02-05T07:30:34.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[37158]: 3.11 scrub ok 2024-02-05T07:30:34.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[37158]: pgmap v611: 168 pgs: 168 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.13k op/s; 7 B/s, 0 objects/s recovering 2024-02-05T07:30:34.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4206751001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:30:34.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[37158]: 4.14 scrub starts 2024-02-05T07:30:34.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[37158]: 4.14 scrub ok 2024-02-05T07:30:34.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:34 smithi007 ceph-mon[37158]: osdmap e362: 8 total, 8 up, 7 in 2024-02-05T07:30:34.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:34.766 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:34.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:34.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:34.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:34 smithi090 ceph-mon[33213]: 3.3 scrub starts 2024-02-05T07:30:34.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:34 smithi090 ceph-mon[33213]: 3.3 scrub ok 2024-02-05T07:30:34.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:34 smithi090 ceph-mon[33213]: 3.11 scrub starts 2024-02-05T07:30:34.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:34 smithi090 ceph-mon[33213]: 3.11 scrub ok 2024-02-05T07:30:34.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:34 smithi090 ceph-mon[33213]: pgmap v611: 168 pgs: 168 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.13k op/s; 7 B/s, 0 objects/s recovering 2024-02-05T07:30:34.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:34 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4206751001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:30:34.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:34 smithi090 ceph-mon[33213]: 4.14 scrub starts 2024-02-05T07:30:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:34 smithi090 ceph-mon[33213]: 4.14 scrub ok 2024-02-05T07:30:34.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:34 smithi090 ceph-mon[33213]: osdmap e362: 8 total, 8 up, 7 in 2024-02-05T07:30:34.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:34.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:34.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:34.930 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 19 15 36800 36785 120.988 109.688 0.00768564 0.00825952 2024-02-05T07:30:34.968 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:30:35.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:35.077 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:35.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:35.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:35.221 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:35.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:35.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:35.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:35.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:35.506 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:35.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:35.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:35.629 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:35 smithi090 ceph-mon[33213]: 3.1b scrub starts 2024-02-05T07:30:35.629 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:35 smithi090 ceph-mon[33213]: 3.1b scrub ok 2024-02-05T07:30:35.629 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:35 smithi090 ceph-mon[33213]: 5.2 scrub starts 2024-02-05T07:30:35.629 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:35 smithi090 ceph-mon[33213]: 5.2 scrub ok 2024-02-05T07:30:35.641 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:35.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:35.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:35.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:35.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:35.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:35.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:35 smithi007 ceph-mon[29098]: 3.1b scrub starts 2024-02-05T07:30:35.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:35 smithi007 ceph-mon[29098]: 3.1b scrub ok 2024-02-05T07:30:35.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:35 smithi007 ceph-mon[29098]: 5.2 scrub starts 2024-02-05T07:30:35.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:35 smithi007 ceph-mon[29098]: 5.2 scrub ok 2024-02-05T07:30:35.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:35 smithi007 ceph-mon[37158]: 3.1b scrub starts 2024-02-05T07:30:35.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:35 smithi007 ceph-mon[37158]: 3.1b scrub ok 2024-02-05T07:30:35.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:35 smithi007 ceph-mon[37158]: 5.2 scrub starts 2024-02-05T07:30:35.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:35 smithi007 ceph-mon[37158]: 5.2 scrub ok 2024-02-05T07:30:35.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:35.904 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:35.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:35.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:35.930 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:30:35.930252+0000 min lat: 0.00140221 max lat: 0.247585 avg lat: 0.00831679 2024-02-05T07:30:35.930 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:30:35.930 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 20 15 38471 38456 120.16 104.438 0.0126704 0.00831679 2024-02-05T07:30:36.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:36.036 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:36.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:36.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:36.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:36.170 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:36.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:36.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:36.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:36.315 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:36.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:36.416 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:36.446 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:36.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:36.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:36.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:36.585 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:36.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:36.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:36.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:36.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:36 smithi007 ceph-mon[29098]: 6.f scrub starts 2024-02-05T07:30:36.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:36 smithi007 ceph-mon[29098]: 4.15 scrub starts 2024-02-05T07:30:36.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:36 smithi007 ceph-mon[29098]: 4.15 scrub ok 2024-02-05T07:30:36.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:36 smithi007 ceph-mon[29098]: 6.f scrub ok 2024-02-05T07:30:36.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:36 smithi007 ceph-mon[29098]: pgmap v613: 168 pgs: 168 active+clean; 1.9 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 901 B/s rd, 73 MiB/s wr, 1.18k op/s; 27 B/s, 1 objects/s recovering 2024-02-05T07:30:36.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:36 smithi007 ceph-mon[37158]: 6.f scrub starts 2024-02-05T07:30:36.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:36 smithi007 ceph-mon[37158]: 4.15 scrub starts 2024-02-05T07:30:36.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:36 smithi007 ceph-mon[37158]: 4.15 scrub ok 2024-02-05T07:30:36.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:36 smithi007 ceph-mon[37158]: 6.f scrub ok 2024-02-05T07:30:36.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:36 smithi007 ceph-mon[37158]: pgmap v613: 168 pgs: 168 active+clean; 1.9 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 901 B/s rd, 73 MiB/s wr, 1.18k op/s; 27 B/s, 1 objects/s recovering 2024-02-05T07:30:36.729 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:36.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:36.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:36.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:36.863 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:36.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:36.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:36.930 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 21 16 40125 40109 119.357 103.312 0.0112807 0.00837239 2024-02-05T07:30:36.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:36.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:36.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:37.009 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:36 smithi090 ceph-mon[33213]: 6.f scrub starts 2024-02-05T07:30:37.009 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:36 smithi090 ceph-mon[33213]: 4.15 scrub starts 2024-02-05T07:30:37.010 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:36 smithi090 ceph-mon[33213]: 4.15 scrub ok 2024-02-05T07:30:37.010 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:36 smithi090 ceph-mon[33213]: 6.f scrub ok 2024-02-05T07:30:37.010 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:36 smithi090 ceph-mon[33213]: pgmap v613: 168 pgs: 168 active+clean; 1.9 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 901 B/s rd, 73 MiB/s wr, 1.18k op/s; 27 B/s, 1 objects/s recovering 2024-02-05T07:30:37.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:37.137 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:37.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:37.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:37.281 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:37.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:37.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:37.415 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:37.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:37.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:37.460 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:30:37.460 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:30:37.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:37.548 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:37.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:37.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:37.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:37.696 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:37.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:37.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:37.860 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:37.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:37.885 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:37 smithi090 ceph-mon[33213]: 4.4 scrub starts 2024-02-05T07:30:37.885 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:37 smithi090 ceph-mon[33213]: 4.4 scrub ok 2024-02-05T07:30:37.885 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:37 smithi090 ceph-mon[33213]: 4.5 scrub starts 2024-02-05T07:30:37.885 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:37 smithi090 ceph-mon[33213]: 4.5 scrub ok 2024-02-05T07:30:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:37 smithi007 ceph-mon[29098]: 4.4 scrub starts 2024-02-05T07:30:37.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:37 smithi007 ceph-mon[29098]: 4.4 scrub ok 2024-02-05T07:30:37.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:37 smithi007 ceph-mon[29098]: 4.5 scrub starts 2024-02-05T07:30:37.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:37 smithi007 ceph-mon[29098]: 4.5 scrub ok 2024-02-05T07:30:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:37 smithi007 ceph-mon[37158]: 4.4 scrub starts 2024-02-05T07:30:37.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:37 smithi007 ceph-mon[37158]: 4.4 scrub ok 2024-02-05T07:30:37.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:37 smithi007 ceph-mon[37158]: 4.5 scrub starts 2024-02-05T07:30:37.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:37 smithi007 ceph-mon[37158]: 4.5 scrub ok 2024-02-05T07:30:37.931 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 22 16 41526 41510 117.912 87.5625 0.0119981 0.00847471 2024-02-05T07:30:37.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:37.985 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:37.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:37.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:38.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:38.119 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:38.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:38.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:38.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T07:30:38.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:38.268 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:38.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:38.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:38.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:38.405 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:38.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:38.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:30:38.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:30:38.547 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:38.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:38.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:38.601 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:30:38.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:38.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:38.715 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:38.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:38.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:38.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[29098]: 4.2 scrub starts 2024-02-05T07:30:38.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[29098]: 4.2 scrub ok 2024-02-05T07:30:38.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[29098]: 1.14 scrub starts 2024-02-05T07:30:38.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[29098]: 1.14 scrub ok 2024-02-05T07:30:38.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[29098]: 4.3 scrub starts 2024-02-05T07:30:38.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[29098]: 4.3 scrub ok 2024-02-05T07:30:38.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[29098]: pgmap v614: 168 pgs: 168 active+clean; 2.3 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 132 MiB/s wr, 2.11k op/s; 23 B/s, 1 objects/s recovering 2024-02-05T07:30:38.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3382143520' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:30:38.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[37158]: 4.2 scrub starts 2024-02-05T07:30:38.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[37158]: 4.2 scrub ok 2024-02-05T07:30:38.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[37158]: 1.14 scrub starts 2024-02-05T07:30:38.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[37158]: 1.14 scrub ok 2024-02-05T07:30:38.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[37158]: 4.3 scrub starts 2024-02-05T07:30:38.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[37158]: 4.3 scrub ok 2024-02-05T07:30:38.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[37158]: pgmap v614: 168 pgs: 168 active+clean; 2.3 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 132 MiB/s wr, 2.11k op/s; 23 B/s, 1 objects/s recovering 2024-02-05T07:30:38.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:38 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3382143520' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:30:38.869 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:38.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:38.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:38.911 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:38 smithi090 ceph-mon[33213]: 4.2 scrub starts 2024-02-05T07:30:38.911 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:38 smithi090 ceph-mon[33213]: 4.2 scrub ok 2024-02-05T07:30:38.911 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:38 smithi090 ceph-mon[33213]: 1.14 scrub starts 2024-02-05T07:30:38.911 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:38 smithi090 ceph-mon[33213]: 1.14 scrub ok 2024-02-05T07:30:38.911 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:38 smithi090 ceph-mon[33213]: 4.3 scrub starts 2024-02-05T07:30:38.911 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:38 smithi090 ceph-mon[33213]: 4.3 scrub ok 2024-02-05T07:30:38.911 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:38 smithi090 ceph-mon[33213]: pgmap v614: 168 pgs: 168 active+clean; 2.3 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 132 MiB/s wr, 2.11k op/s; 23 B/s, 1 objects/s recovering 2024-02-05T07:30:38.911 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:38 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3382143520' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:30:38.931 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 23 16 42806 42790 116.263 80 0.0220933 0.00859408 2024-02-05T07:30:38.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:39.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:39.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:39.118 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:39.147 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:39.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:39.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:39.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a[29094]: 2024-02-05T07:30:39.063+0000 7fe3564d3700 -1 mon.a@0(leader).osd e363 definitely_dead 0 2024-02-05T07:30:39.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:39.294 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:39.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:39.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:39.423 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:39.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:39.469 INFO:teuthology.orchestra.run.smithi007.stderr:marked out osd.7. 2024-02-05T07:30:39.476 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:39.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:39.548 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:39.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:39.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:39.676 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:39.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:39.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:39.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:39.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:39.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[29098]: 6.d scrub starts 2024-02-05T07:30:39.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[29098]: 6.d scrub ok 2024-02-05T07:30:39.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:30:39.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:30:39.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3382143520' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:30:39.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[29098]: osdmap e363: 8 total, 8 up, 7 in 2024-02-05T07:30:39.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[29098]: 5.d deep-scrub starts 2024-02-05T07:30:39.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[29098]: 5.d deep-scrub ok 2024-02-05T07:30:39.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[29098]: 3.c scrub starts 2024-02-05T07:30:39.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[29098]: 3.c scrub ok 2024-02-05T07:30:39.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2826439438' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-05T07:30:39.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[29098]: Client client.admin marked osd.7 out, while it was still marked up 2024-02-05T07:30:39.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2826439438' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-05T07:30:39.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[29098]: osdmap e364: 8 total, 8 up, 6 in 2024-02-05T07:30:39.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[37158]: 6.d scrub starts 2024-02-05T07:30:39.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[37158]: 6.d scrub ok 2024-02-05T07:30:39.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:30:39.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:30:39.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3382143520' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:30:39.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[37158]: osdmap e363: 8 total, 8 up, 7 in 2024-02-05T07:30:39.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[37158]: 5.d deep-scrub starts 2024-02-05T07:30:39.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[37158]: 5.d deep-scrub ok 2024-02-05T07:30:39.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[37158]: 3.c scrub starts 2024-02-05T07:30:39.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[37158]: 3.c scrub ok 2024-02-05T07:30:39.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2826439438' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-05T07:30:39.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[37158]: Client client.admin marked osd.7 out, while it was still marked up 2024-02-05T07:30:39.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2826439438' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-05T07:30:39.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:39 smithi007 ceph-mon[37158]: osdmap e364: 8 total, 8 up, 6 in 2024-02-05T07:30:39.903 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:39.926 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:39.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:39.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:39.930 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 24 16 43530 43514 113.304 45.25 0.00668683 0.00865247 2024-02-05T07:30:39.974 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:39 smithi090 ceph-mon[33213]: 6.d scrub starts 2024-02-05T07:30:39.974 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:39 smithi090 ceph-mon[33213]: 6.d scrub ok 2024-02-05T07:30:39.974 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:39 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:30:39.974 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:39 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:30:39.975 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:39 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3382143520' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:30:39.975 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:39 smithi090 ceph-mon[33213]: osdmap e363: 8 total, 8 up, 7 in 2024-02-05T07:30:39.975 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:39 smithi090 ceph-mon[33213]: 5.d deep-scrub starts 2024-02-05T07:30:39.975 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:39 smithi090 ceph-mon[33213]: 5.d deep-scrub ok 2024-02-05T07:30:39.975 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:39 smithi090 ceph-mon[33213]: 3.c scrub starts 2024-02-05T07:30:39.975 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:39 smithi090 ceph-mon[33213]: 3.c scrub ok 2024-02-05T07:30:39.975 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:39 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2826439438' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-05T07:30:39.975 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:39 smithi090 ceph-mon[33213]: Client client.admin marked osd.7 out, while it was still marked up 2024-02-05T07:30:39.975 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:39 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2826439438' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-05T07:30:39.975 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:39 smithi090 ceph-mon[33213]: osdmap e364: 8 total, 8 up, 6 in 2024-02-05T07:30:40.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:40.051 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:40.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:40.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:40.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:40.237 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:30:40.238 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:30:40.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:40.302 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:40.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:40.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:40.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:40.427 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:40.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:40.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:40.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:40.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:40.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:40.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:40.680 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:40.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:40.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:40.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:40.805 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:40.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:40.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:40.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[29098]: 3.16 scrub starts 2024-02-05T07:30:40.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[29098]: 3.16 scrub ok 2024-02-05T07:30:40.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[29098]: pgmap v616: 168 pgs: 168 active+clean; 2.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 769 B/s rd, 78 MiB/s wr, 1.25k op/s; 16 B/s, 1 objects/s recovering 2024-02-05T07:30:40.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[29098]: 3.1c scrub starts 2024-02-05T07:30:40.847 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[29098]: 3.1c scrub ok 2024-02-05T07:30:40.847 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[29098]: 6.a deep-scrub starts 2024-02-05T07:30:40.847 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[29098]: 6.a deep-scrub ok 2024-02-05T07:30:40.847 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[29098]: osdmap e365: 8 total, 8 up, 6 in 2024-02-05T07:30:40.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[37158]: 3.16 scrub starts 2024-02-05T07:30:40.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[37158]: 3.16 scrub ok 2024-02-05T07:30:40.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[37158]: pgmap v616: 168 pgs: 168 active+clean; 2.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 769 B/s rd, 78 MiB/s wr, 1.25k op/s; 16 B/s, 1 objects/s recovering 2024-02-05T07:30:40.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[37158]: 3.1c scrub starts 2024-02-05T07:30:40.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[37158]: 3.1c scrub ok 2024-02-05T07:30:40.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[37158]: 6.a deep-scrub starts 2024-02-05T07:30:40.848 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[37158]: 6.a deep-scrub ok 2024-02-05T07:30:40.848 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:40 smithi007 ceph-mon[37158]: osdmap e365: 8 total, 8 up, 6 in 2024-02-05T07:30:40.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:40.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:40.931 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 25 16 43530 43514 108.772 0 - 0.00865247 2024-02-05T07:30:40.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:40.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:41.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:30:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:30:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:40 smithi090 ceph-mon[33213]: 3.16 scrub starts 2024-02-05T07:30:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:40 smithi090 ceph-mon[33213]: 3.16 scrub ok 2024-02-05T07:30:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:40 smithi090 ceph-mon[33213]: pgmap v616: 168 pgs: 168 active+clean; 2.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 769 B/s rd, 78 MiB/s wr, 1.25k op/s; 16 B/s, 1 objects/s recovering 2024-02-05T07:30:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:40 smithi090 ceph-mon[33213]: 3.1c scrub starts 2024-02-05T07:30:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:40 smithi090 ceph-mon[33213]: 3.1c scrub ok 2024-02-05T07:30:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:40 smithi090 ceph-mon[33213]: 6.a deep-scrub starts 2024-02-05T07:30:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:40 smithi090 ceph-mon[33213]: 6.a deep-scrub ok 2024-02-05T07:30:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:40 smithi090 ceph-mon[33213]: osdmap e365: 8 total, 8 up, 6 in 2024-02-05T07:30:41.033 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:41.056 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:41.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:41.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:41.149 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T07:30:41.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:41.217 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:41.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:41.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:41.343 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:41.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:41.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:41.388 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:30:41.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:30:41.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:41.472 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:41.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:41.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:41.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:41.637 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:41.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:41.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:41 smithi007 ceph-mon[29098]: 3.1a deep-scrub starts 2024-02-05T07:30:41.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:41 smithi007 ceph-mon[29098]: 3.1a deep-scrub ok 2024-02-05T07:30:41.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:41 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:41.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:41 smithi007 ceph-mon[29098]: 4.11 scrub starts 2024-02-05T07:30:41.711 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:41 smithi007 ceph-mon[29098]: 4.11 scrub ok 2024-02-05T07:30:41.711 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:41 smithi007 ceph-mon[29098]: osdmap e366: 8 total, 8 up, 6 in 2024-02-05T07:30:41.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:41 smithi007 ceph-mon[37158]: 3.1a deep-scrub starts 2024-02-05T07:30:41.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:41 smithi007 ceph-mon[37158]: 3.1a deep-scrub ok 2024-02-05T07:30:41.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:41 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:41.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:41 smithi007 ceph-mon[37158]: 4.11 scrub starts 2024-02-05T07:30:41.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:41 smithi007 ceph-mon[37158]: 4.11 scrub ok 2024-02-05T07:30:41.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:41 smithi007 ceph-mon[37158]: osdmap e366: 8 total, 8 up, 6 in 2024-02-05T07:30:41.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:41.765 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:41 smithi090 ceph-mon[33213]: 3.1a deep-scrub starts 2024-02-05T07:30:41.766 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:41 smithi090 ceph-mon[33213]: 3.1a deep-scrub ok 2024-02-05T07:30:41.766 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:41 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:41.766 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:41 smithi090 ceph-mon[33213]: 4.11 scrub starts 2024-02-05T07:30:41.766 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:41 smithi090 ceph-mon[33213]: 4.11 scrub ok 2024-02-05T07:30:41.766 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:41 smithi090 ceph-mon[33213]: osdmap e366: 8 total, 8 up, 6 in 2024-02-05T07:30:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:41.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:41.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:41.870 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:41.923 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:41.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:41.932 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 26 16 43618 43602 104.795 2.75 0.00824109 0.00935063 2024-02-05T07:30:42.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:42.051 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:42.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:42.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:42.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:42.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:42.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:42.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:42.317 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:42.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:42.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:42.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:42.443 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:42.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:42.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:42.583 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:42.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:42.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:42.670 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:30:42.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:42.709 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:30:42.722 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:42.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:42.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:42.745 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:30:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:42 smithi090 ceph-mon[33213]: 5.1d scrub starts 2024-02-05T07:30:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:42 smithi090 ceph-mon[33213]: 5.1d scrub ok 2024-02-05T07:30:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:42 smithi090 ceph-mon[33213]: pgmap v619: 168 pgs: 13 remapped+peering, 13 peering, 3 unknown, 139 active+clean; 2.7 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 136 MiB/s wr, 2.17k op/s; 26 KiB/s, 8 objects/s recovering 2024-02-05T07:30:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:42 smithi090 ceph-mon[33213]: Health check failed: Reduced data availability: 8 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-02-05T07:30:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:42 smithi090 ceph-mon[33213]: 6.1c scrub starts 2024-02-05T07:30:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:42 smithi090 ceph-mon[33213]: 6.1c scrub ok 2024-02-05T07:30:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:42 smithi090 ceph-mon[33213]: osdmap e367: 8 total, 8 up, 6 in 2024-02-05T07:30:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:42 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3041700040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:30:42.778 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:30:42.810 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:30:42.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:42.845 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:30:42.845 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[29098]: 5.1d scrub starts 2024-02-05T07:30:42.845 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[29098]: 5.1d scrub ok 2024-02-05T07:30:42.845 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[29098]: pgmap v619: 168 pgs: 13 remapped+peering, 13 peering, 3 unknown, 139 active+clean; 2.7 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 136 MiB/s wr, 2.17k op/s; 26 KiB/s, 8 objects/s recovering 2024-02-05T07:30:42.845 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[29098]: Health check failed: Reduced data availability: 8 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-02-05T07:30:42.845 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[29098]: 6.1c scrub starts 2024-02-05T07:30:42.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[29098]: 6.1c scrub ok 2024-02-05T07:30:42.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[29098]: osdmap e367: 8 total, 8 up, 6 in 2024-02-05T07:30:42.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3041700040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:30:42.846 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[37158]: 5.1d scrub starts 2024-02-05T07:30:42.846 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[37158]: 5.1d scrub ok 2024-02-05T07:30:42.846 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[37158]: pgmap v619: 168 pgs: 13 remapped+peering, 13 peering, 3 unknown, 139 active+clean; 2.7 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 136 MiB/s wr, 2.17k op/s; 26 KiB/s, 8 objects/s recovering 2024-02-05T07:30:42.846 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[37158]: Health check failed: Reduced data availability: 8 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-02-05T07:30:42.846 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[37158]: 6.1c scrub starts 2024-02-05T07:30:42.846 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[37158]: 6.1c scrub ok 2024-02-05T07:30:42.846 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[37158]: osdmap e367: 8 total, 8 up, 6 in 2024-02-05T07:30:42.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:42 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3041700040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:30:42.852 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:42.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:42.881 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:30:42.913 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:30:42.915 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:30:42.915 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:30:42.915 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:30:42.915 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:30:42.915 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:30:42.915 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:30:42.915 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:30:42.915 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:30:42.932 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 27 16 43764 43748 101.252 9.125 0.0100116 0.00934847 2024-02-05T07:30:42.953 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:42.982 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:42.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:42.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:43.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:43.115 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:43.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:43.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:43.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:43.242 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:30:43.242 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:30:43.243 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:30:43.243 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:30:43.243 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:30:43.243 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:43.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:43.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:43.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:43.370 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:43.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:43.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:43.495 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:43.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:43.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:43.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:43.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:43.621 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:43.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:43.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:43.746 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:43.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:43.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:43.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:43.882 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:43.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:43.891 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:30:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:30:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:30:43.932 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 28 16 43764 43748 97.6364 0 - 0.00934847 2024-02-05T07:30:43.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T07:30:43.984 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:44.015 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:44.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:44.117 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:44.148 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:44.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:44.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:44.230 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:30:44.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:44.280 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:44.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:44.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:44.381 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:44.412 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:44.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:44.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:44.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:44.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:44.545 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:44.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:44.615 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:44 smithi090 ceph-mon[33213]: pgmap v622: 168 pgs: 13 remapped+peering, 13 peering, 3 unknown, 139 active+clean; 2.7 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 80 MiB/s wr, 1.28k op/s; 40 KiB/s, 12 objects/s recovering 2024-02-05T07:30:44.615 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:44 smithi090 ceph-mon[33213]: osdmap e368: 8 total, 8 up, 6 in 2024-02-05T07:30:44.615 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:44 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:30:44.615 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:44 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:30:44.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:44.672 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:44.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:44.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:44.799 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:44.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:44.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:44.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:44.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:44 smithi007 ceph-mon[29098]: pgmap v622: 168 pgs: 13 remapped+peering, 13 peering, 3 unknown, 139 active+clean; 2.7 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 80 MiB/s wr, 1.28k op/s; 40 KiB/s, 12 objects/s recovering 2024-02-05T07:30:44.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:44 smithi007 ceph-mon[29098]: osdmap e368: 8 total, 8 up, 6 in 2024-02-05T07:30:44.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:44 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:30:44.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:44 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:30:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:44 smithi007 ceph-mon[37158]: pgmap v622: 168 pgs: 13 remapped+peering, 13 peering, 3 unknown, 139 active+clean; 2.7 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 80 MiB/s wr, 1.28k op/s; 40 KiB/s, 12 objects/s recovering 2024-02-05T07:30:44.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:44 smithi007 ceph-mon[37158]: osdmap e368: 8 total, 8 up, 6 in 2024-02-05T07:30:44.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:44 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:30:44.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:44 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:30:44.932 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:44.932 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 29 16 43764 43748 94.2699 0 - 0.00934847 2024-02-05T07:30:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:44.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:45.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:45.059 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:45.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:45.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:45.186 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:45.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:45.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:45.353 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:45.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:45.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:45.455 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:45.483 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:45.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:45.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:45.545 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 3] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:30:45.545 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:30:45.546 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-02-05T07:30:45.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:30:45.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:45.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:45.659 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:45.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:45.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:45.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:45.823 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:45.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:45.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:45.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:45 smithi007 ceph-mon[29098]: osdmap e369: 8 total, 8 up, 6 in 2024-02-05T07:30:45.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:45 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:45.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:45 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:30:45.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:45 smithi007 ceph-mon[37158]: osdmap e369: 8 total, 8 up, 6 in 2024-02-05T07:30:45.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:45 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:45.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:45 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:30:45.933 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 30 16 44273 44257 92.188 10.6042 0.0399267 0.010836 2024-02-05T07:30:45.935 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:45 smithi090 ceph-mon[33213]: osdmap e369: 8 total, 8 up, 6 in 2024-02-05T07:30:45.935 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:45 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:30:45.935 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:45 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:30:45.970 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:30:45.970 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-02-05T07:30:45.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-02-05T07:30:45.971 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:30:45.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.971 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:75630 7.15 7:abba0814:::benchmark_data_smithi007_77086_object75629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-02-05T07:30:45.971 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:21:43.637172+0000", 2024-02-05T07:30:45.971 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 542.33069126400005, 2024-02-05T07:30:45.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6317031210000001, 2024-02-05T07:30:45.971 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:45.971 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:45.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:45.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:30:45.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:30:45.972 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 75630 2024-02-05T07:30:45.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.972 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:45.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:45.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.637172+0000", 2024-02-05T07:30:45.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:45.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:45.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.637172+0000", 2024-02-05T07:30:45.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:45.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:45.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.637173+0000", 2024-02-05T07:30:45.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5600000000000005e-07 2024-02-05T07:30:45.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:45.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.637208+0000", 2024-02-05T07:30:45.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5472000000000002e-05 2024-02-05T07:30:45.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:45.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.637218+0000", 2024-02-05T07:30:45.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5170000000000005e-06 2024-02-05T07:30:45.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:45.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.637223+0000", 2024-02-05T07:30:45.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5589999999999996e-06 2024-02-05T07:30:45.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:45.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.641716+0000", 2024-02-05T07:30:45.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044922449999999997 2024-02-05T07:30:45.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:45.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:43.641724+0000", 2024-02-05T07:30:45.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1110000000000008e-06 2024-02-05T07:30:45.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:45.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:44.135904+0000", 2024-02-05T07:30:45.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49417987099999999 2024-02-05T07:30:45.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:45.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:44.135914+0000", 2024-02-05T07:30:45.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0665e-05 2024-02-05T07:30:45.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:45.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.261071+0000", 2024-02-05T07:30:45.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.125156998 2024-02-05T07:30:45.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:45.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.261210+0000", 2024-02-05T07:30:45.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013857400000000001 2024-02-05T07:30:45.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:30:45.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.261473+0000", 2024-02-05T07:30:45.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026326399999999998 2024-02-05T07:30:45.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:45.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.263707+0000", 2024-02-05T07:30:45.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022340120000000001 2024-02-05T07:30:45.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:45.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.265811+0000", 2024-02-05T07:30:45.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021035540000000001 2024-02-05T07:30:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.267296+0000", 2024-02-05T07:30:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014858689999999999 2024-02-05T07:30:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.267368+0000", 2024-02-05T07:30:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1462000000000001e-05 2024-02-05T07:30:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:21:45.268875+0000", 2024-02-05T07:30:45.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015071920000000001 2024-02-05T07:30:45.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:45.979 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:45.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:45.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.979 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:172756 7.11 7:8d3d4e1b:::benchmark_data_smithi007_77086_object68161:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-05T07:30:45.979 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:23:57.456991+0000", 2024-02-05T07:30:45.979 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 408.51087255200002, 2024-02-05T07:30:45.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.92245448500000005, 2024-02-05T07:30:45.979 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:45.979 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:45.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:45.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:30:45.980 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:30:45.980 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172756 2024-02-05T07:30:45.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.980 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:45.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:45.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.456991+0000", 2024-02-05T07:30:45.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:45.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:45.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.456991+0000", 2024-02-05T07:30:45.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:45.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:45.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.456991+0000", 2024-02-05T07:30:45.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8699999999999995e-07 2024-02-05T07:30:45.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:45.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.456994+0000", 2024-02-05T07:30:45.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7159999999999999e-06 2024-02-05T07:30:45.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:45.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.457000+0000", 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2960000000000004e-06 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.457006+0000", 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5709999999999997e-06 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.457060+0000", 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3748000000000001e-05 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:45.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.457068+0000", 2024-02-05T07:30:45.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6289999999999999e-06 2024-02-05T07:30:45.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:45.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.372790+0000", 2024-02-05T07:30:45.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.91572212099999994 2024-02-05T07:30:45.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:45.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.372925+0000", 2024-02-05T07:30:45.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134871 2024-02-05T07:30:45.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,6", 2024-02-05T07:30:45.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.373296+0000", 2024-02-05T07:30:45.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037044999999999998 2024-02-05T07:30:45.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:45.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379211+0000", 2024-02-05T07:30:45.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0059157919999999996 2024-02-05T07:30:45.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:45.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379225+0000", 2024-02-05T07:30:45.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4049000000000001e-05 2024-02-05T07:30:45.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:45.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379381+0000", 2024-02-05T07:30:45.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015567599999999999 2024-02-05T07:30:45.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:45.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379419+0000", 2024-02-05T07:30:45.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7629999999999997e-05 2024-02-05T07:30:45.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:45.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379445+0000", 2024-02-05T07:30:45.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6449e-05 2024-02-05T07:30:45.986 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:45.986 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:45.986 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:45.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.987 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:172762 7.11 7:8a322388:::benchmark_data_smithi007_77086_object68167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-05T07:30:45.987 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:23:57.460318+0000", 2024-02-05T07:30:45.987 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 408.50754552199999, 2024-02-05T07:30:45.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.91924548800000005, 2024-02-05T07:30:45.987 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:45.987 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:45.987 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:45.987 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:30:45.987 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:30:45.987 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172762 2024-02-05T07:30:45.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.987 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:45.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:45.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460318+0000", 2024-02-05T07:30:45.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:45.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:45.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460318+0000", 2024-02-05T07:30:45.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:45.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:45.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460319+0000", 2024-02-05T07:30:45.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5409999999999999e-06 2024-02-05T07:30:45.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:45.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460327+0000", 2024-02-05T07:30:45.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1359999999999997e-06 2024-02-05T07:30:45.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:45.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460339+0000", 2024-02-05T07:30:45.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1644999999999999e-05 2024-02-05T07:30:45.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:45.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460348+0000", 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.568e-06 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460641+0000", 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029384099999999998 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.460658+0000", 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6575999999999999e-05 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.375350+0000", 2024-02-05T07:30:45.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.91469182500000001 2024-02-05T07:30:45.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:45.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.375469+0000", 2024-02-05T07:30:45.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119292 2024-02-05T07:30:45.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,6", 2024-02-05T07:30:45.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.375697+0000", 2024-02-05T07:30:45.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022779300000000001 2024-02-05T07:30:45.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379336+0000", 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003638694 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379348+0000", 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2131e-05 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379492+0000", 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014466699999999999 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:45.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379526+0000", 2024-02-05T07:30:45.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3408999999999998e-05 2024-02-05T07:30:45.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:45.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379563+0000", 2024-02-05T07:30:45.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7370000000000003e-05 2024-02-05T07:30:45.993 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:45.993 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:45.993 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:45.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.993 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24719.0:172771 7.11 7:89039f95:::benchmark_data_smithi007_77086_object68176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-05T07:30:45.993 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:23:57.468804+0000", 2024-02-05T07:30:45.994 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 408.49905912000003, 2024-02-05T07:30:45.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.91441402699999996, 2024-02-05T07:30:45.994 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:45.994 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:45.994 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:45.994 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24719", 2024-02-05T07:30:45.994 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2218879804", 2024-02-05T07:30:45.994 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172771 2024-02-05T07:30:45.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.994 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:45.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:45.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468804+0000", 2024-02-05T07:30:45.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:45.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:45.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468804+0000", 2024-02-05T07:30:45.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:45.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:45.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468805+0000", 2024-02-05T07:30:45.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.052e-06 2024-02-05T07:30:45.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:45.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468808+0000", 2024-02-05T07:30:45.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0529999999999998e-06 2024-02-05T07:30:45.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:45.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468817+0000", 2024-02-05T07:30:45.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2649999999999996e-06 2024-02-05T07:30:45.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:45.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468825+0000", 2024-02-05T07:30:45.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0199999999999994e-06 2024-02-05T07:30:45.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:45.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468884+0000", 2024-02-05T07:30:45.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9929999999999997e-05 2024-02-05T07:30:45.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:45.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:45.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:57.468894+0000", 2024-02-05T07:30:45.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1120000000000002e-06 2024-02-05T07:30:45.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:45.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.377141+0000", 2024-02-05T07:30:46.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.90824729199999998 2024-02-05T07:30:46.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.377259+0000", 2024-02-05T07:30:46.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118332 2024-02-05T07:30:46.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,6", 2024-02-05T07:30:46.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.377519+0000", 2024-02-05T07:30:46.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025967600000000003 2024-02-05T07:30:46.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.379608+0000", 2024-02-05T07:30:46.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020893439999999999 2024-02-05T07:30:46.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.381150+0000", 2024-02-05T07:30:46.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015417339999999999 2024-02-05T07:30:46.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.383116+0000", 2024-02-05T07:30:46.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001966392 2024-02-05T07:30:46.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:46.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.383173+0000", 2024-02-05T07:30:46.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6802000000000003e-05 2024-02-05T07:30:46.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:23:58.383218+0000", 2024-02-05T07:30:46.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5022999999999999e-05 2024-02-05T07:30:46.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.005 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.006 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:151887 8.c 8:324b0199:::benchmark_data_smithi007_105755_object15577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:30:46.006 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:15.622747+0000", 2024-02-05T07:30:46.006 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 210.34511603999999, 2024-02-05T07:30:46.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.370190494, 2024-02-05T07:30:46.006 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.006 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:30:46.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.006 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151887 2024-02-05T07:30:46.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.006 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.622747+0000", 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.622747+0000", 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.622748+0000", 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2160000000000001e-06 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.622755+0000", 2024-02-05T07:30:46.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5930000000000002e-06 2024-02-05T07:30:46.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.622765+0000", 2024-02-05T07:30:46.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9049999999999992e-06 2024-02-05T07:30:46.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.622773+0000", 2024-02-05T07:30:46.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9219999999999997e-06 2024-02-05T07:30:46.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.622921+0000", 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148221 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.623051+0000", 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013038599999999999 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,5", 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.623186+0000", 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013459400000000001 2024-02-05T07:30:46.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.626745+0000", 2024-02-05T07:30:46.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035590420000000001 2024-02-05T07:30:46.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.627939+0000", 2024-02-05T07:30:46.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011934039999999999 2024-02-05T07:30:46.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629311+0000", 2024-02-05T07:30:46.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013721580000000001 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.691547+0000", 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062236106999999999 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.691557+0000", 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7720000000000006e-06 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.987383+0000", 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2958265010000001 2024-02-05T07:30:46.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-02-05T07:30:46.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.987490+0000", 2024-02-05T07:30:46.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010739 2024-02-05T07:30:46.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.992884+0000", 2024-02-05T07:30:46.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053939920000000002 2024-02-05T07:30:46.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.992938+0000", 2024-02-05T07:30:46.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3291000000000002e-05 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:151900 8.8 8:15dd58bd:::benchmark_data_smithi007_105755_object15590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:15.630983+0000", 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 210.33688049400001, 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3621680999999999, 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.013 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151900 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.630983+0000", 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.630983+0000", 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.630984+0000", 2024-02-05T07:30:46.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.33e-06 2024-02-05T07:30:46.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.630989+0000", 2024-02-05T07:30:46.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8099999999999997e-06 2024-02-05T07:30:46.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.631004+0000", 2024-02-05T07:30:46.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5407999999999999e-05 2024-02-05T07:30:46.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.631012+0000", 2024-02-05T07:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0730000000000005e-06 2024-02-05T07:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.631076+0000", 2024-02-05T07:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3590000000000006e-05 2024-02-05T07:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.631210+0000", 2024-02-05T07:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133666 2024-02-05T07:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2,5", 2024-02-05T07:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.631529+0000", 2024-02-05T07:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031965100000000001 2024-02-05T07:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.634358+0000", 2024-02-05T07:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028284569999999999 2024-02-05T07:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.634826+0000", 2024-02-05T07:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046775199999999999 2024-02-05T07:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.684540+0000", 2024-02-05T07:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049714248000000003 2024-02-05T07:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.684555+0000", 2024-02-05T07:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5350999999999999e-05 2024-02-05T07:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.987354+0000", 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3027992239999999 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.987472+0000", 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011736299999999999 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.993094+0000", 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056223510000000003 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.993151+0000", 2024-02-05T07:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6826000000000001e-05 2024-02-05T07:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:151902 8.c 8:32e5c9c1:::benchmark_data_smithi007_105755_object15592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:15.632873+0000", 2024-02-05T07:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 210.33499033199999, 2024-02-05T07:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3627677999999999, 2024-02-05T07:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151902 2024-02-05T07:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.632873+0000", 2024-02-05T07:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.632873+0000", 2024-02-05T07:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.632874+0000", 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.204e-06 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.632878+0000", 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6320000000000001e-06 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.632887+0000", 2024-02-05T07:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8359999999999998e-06 2024-02-05T07:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.632893+0000", 2024-02-05T07:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8639999999999998e-06 2024-02-05T07:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.632948+0000", 2024-02-05T07:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4605000000000003e-05 2024-02-05T07:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.633073+0000", 2024-02-05T07:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124626 2024-02-05T07:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,5", 2024-02-05T07:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.633200+0000", 2024-02-05T07:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127651 2024-02-05T07:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.637692+0000", 2024-02-05T07:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044912149999999998 2024-02-05T07:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.637749+0000", 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7438000000000001e-05 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.637813+0000", 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3830000000000004e-05 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.691600+0000", 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.053787029 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.691609+0000", 2024-02-05T07:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2089999999999994e-06 2024-02-05T07:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.987907+0000", 2024-02-05T07:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.296297872 2024-02-05T07:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-02-05T07:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.988009+0000", 2024-02-05T07:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010227599999999999 2024-02-05T07:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.995549+0000", 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075395009999999997 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.995641+0000", 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2011999999999999e-05 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:151908 8.8 8:11579385:::benchmark_data_smithi007_105755_object15598:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:15.644089+0000", 2024-02-05T07:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 210.32377453000001, 2024-02-05T07:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.348958485, 2024-02-05T07:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151908 2024-02-05T07:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644089+0000", 2024-02-05T07:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644089+0000", 2024-02-05T07:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644089+0000", 2024-02-05T07:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0999999999999998e-07 2024-02-05T07:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644093+0000", 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0630000000000002e-06 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644100+0000", 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1849999999999998e-06 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644104+0000", 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6450000000000004e-06 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644136+0000", 2024-02-05T07:30:46.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1736000000000002e-05 2024-02-05T07:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644219+0000", 2024-02-05T07:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2546999999999994e-05 2024-02-05T07:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2,5", 2024-02-05T07:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.644484+0000", 2024-02-05T07:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026491600000000002 2024-02-05T07:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.645628+0000", 2024-02-05T07:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001144436 2024-02-05T07:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.646211+0000", 2024-02-05T07:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058267900000000001 2024-02-05T07:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.687357+0000", 2024-02-05T07:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041146481999999998 2024-02-05T07:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.687369+0000", 2024-02-05T07:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1972e-05 2024-02-05T07:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.987534+0000", 2024-02-05T07:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3001650279999999 2024-02-05T07:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-02-05T07:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.987600+0000", 2024-02-05T07:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6271000000000003e-05 2024-02-05T07:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.992998+0000", 2024-02-05T07:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053975610000000004 2024-02-05T07:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.993047+0000", 2024-02-05T07:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9254e-05 2024-02-05T07:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:163385 8.11 8:8b198359:::benchmark_data_smithi007_105755_object27075:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e271)", 2024-02-05T07:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:31.201355+0000", 2024-02-05T07:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 194.76650861300001, 2024-02-05T07:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.020621497, 2024-02-05T07:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 163385 2024-02-05T07:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.201355+0000", 2024-02-05T07:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.201355+0000", 2024-02-05T07:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.201356+0000", 2024-02-05T07:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5829999999999999e-06 2024-02-05T07:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.201360+0000", 2024-02-05T07:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8160000000000004e-06 2024-02-05T07:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.201368+0000", 2024-02-05T07:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2379999999999997e-06 2024-02-05T07:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.201374+0000", 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3159999999999996e-06 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.201413+0000", 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9063999999999997e-05 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.201418+0000", 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2050000000000001e-06 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.215332+0000", 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0139138219999999 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.215466+0000", 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013426900000000001 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.215741+0000", 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027515800000000001 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.219611+0000", 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038703679999999999 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.219651+0000", 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9841e-05 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.221880+0000", 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002228692 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.221916+0000", 2024-02-05T07:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6350000000000003e-05 2024-02-05T07:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.221976+0000", 2024-02-05T07:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9775000000000003e-05 2024-02-05T07:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:163391 8.11 8:8caf1aef:::benchmark_data_smithi007_105755_object27081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-02-05T07:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:31.208914+0000", 2024-02-05T07:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 194.75894894800001, 2024-02-05T07:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0176787860000001, 2024-02-05T07:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 163391 2024-02-05T07:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.208914+0000", 2024-02-05T07:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.208914+0000", 2024-02-05T07:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.208916+0000", 2024-02-05T07:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.364e-06 2024-02-05T07:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.208920+0000", 2024-02-05T07:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1629999999999998e-06 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.208930+0000", 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0618999999999999e-05 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.208936+0000", 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8200000000000002e-06 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.208992+0000", 2024-02-05T07:30:46.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5968999999999999e-05 2024-02-05T07:30:46.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.209005+0000", 2024-02-05T07:30:46.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.287e-05 2024-02-05T07:30:46.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.217439+0000", 2024-02-05T07:30:46.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.008434026 2024-02-05T07:30:46.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.217518+0000", 2024-02-05T07:30:46.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8767e-05 2024-02-05T07:30:46.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:30:46.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.217676+0000", 2024-02-05T07:30:46.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015824400000000001 2024-02-05T07:30:46.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.221837+0000", 2024-02-05T07:30:46.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041604060000000002 2024-02-05T07:30:46.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.222004+0000", 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000167446 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.226443+0000", 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044392370000000004 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.226503+0000", 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0164999999999997e-05 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.226593+0000", 2024-02-05T07:30:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9690000000000004e-05 2024-02-05T07:30:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:163403 8.11 8:8c5a11d3:::benchmark_data_smithi007_105755_object27093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-02-05T07:30:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:31.229335+0000", 2024-02-05T07:30:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 194.738527777, 2024-02-05T07:30:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0047851800000001, 2024-02-05T07:30:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 163403 2024-02-05T07:30:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.229335+0000", 2024-02-05T07:30:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.229335+0000", 2024-02-05T07:30:46.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.229336+0000", 2024-02-05T07:30:46.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6100000000000002e-07 2024-02-05T07:30:46.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.229338+0000", 2024-02-05T07:30:46.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.249e-06 2024-02-05T07:30:46.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.229343+0000", 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.763e-06 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.229347+0000", 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1080000000000003e-06 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.229404+0000", 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6910999999999998e-05 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.229414+0000", 2024-02-05T07:30:46.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0540000000000001e-05 2024-02-05T07:30:46.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.218615+0000", 2024-02-05T07:30:46.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98920036499999997 2024-02-05T07:30:46.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.218736+0000", 2024-02-05T07:30:46.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121408 2024-02-05T07:30:46.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:30:46.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.218921+0000", 2024-02-05T07:30:46.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000184511 2024-02-05T07:30:46.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.223172+0000", 2024-02-05T07:30:46.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042508390000000002 2024-02-05T07:30:46.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.226370+0000", 2024-02-05T07:30:46.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031980120000000001 2024-02-05T07:30:46.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.234002+0000", 2024-02-05T07:30:46.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0076324160000000004 2024-02-05T07:30:46.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:46.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.234061+0000", 2024-02-05T07:30:46.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9345000000000002e-05 2024-02-05T07:30:46.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.234121+0000", 2024-02-05T07:30:46.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9351999999999997e-05 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:163421 8.11 8:8c1af58a:::benchmark_data_smithi007_105755_object27111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:31.250121+0000", 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 194.71774195399999, 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98414484400000002, 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 163421 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.250121+0000", 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.250121+0000", 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.250122+0000", 2024-02-05T07:30:46.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0640000000000001e-06 2024-02-05T07:30:46.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.250126+0000", 2024-02-05T07:30:46.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0679999999999998e-06 2024-02-05T07:30:46.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.250136+0000", 2024-02-05T07:30:46.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6879999999999994e-06 2024-02-05T07:30:46.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.250142+0000", 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6239999999999999e-06 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.250192+0000", 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9948999999999999e-05 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.250199+0000", 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7109999999999995e-06 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.219693+0000", 2024-02-05T07:30:46.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.96949389699999999 2024-02-05T07:30:46.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.219807+0000", 2024-02-05T07:30:46.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113717 2024-02-05T07:30:46.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:30:46.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.220016+0000", 2024-02-05T07:30:46.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020870599999999999 2024-02-05T07:30:46.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.223178+0000", 2024-02-05T07:30:46.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031622400000000002 2024-02-05T07:30:46.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.226641+0000", 2024-02-05T07:30:46.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034629690000000002 2024-02-05T07:30:46.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.234163+0000", 2024-02-05T07:30:46.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.007522015 2024-02-05T07:30:46.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:46.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.234203+0000", 2024-02-05T07:30:46.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0497000000000002e-05 2024-02-05T07:30:46.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.234266+0000", 2024-02-05T07:30:46.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2699000000000004e-05 2024-02-05T07:30:46.064 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.064 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.064 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.065 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:163425 8.11 8:8a7e1718:::benchmark_data_smithi007_105755_object27115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-02-05T07:30:46.065 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:31.253063+0000", 2024-02-05T07:30:46.065 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 194.71480023699999, 2024-02-05T07:30:46.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.982797064, 2024-02-05T07:30:46.065 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.065 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:46.065 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.065 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.065 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.065 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 163425 2024-02-05T07:30:46.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.065 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.253063+0000", 2024-02-05T07:30:46.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.253063+0000", 2024-02-05T07:30:46.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.253064+0000", 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.023e-06 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.253069+0000", 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.096e-06 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.253081+0000", 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2286000000000001e-05 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.253089+0000", 2024-02-05T07:30:46.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9410000000000007e-06 2024-02-05T07:30:46.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.253149+0000", 2024-02-05T07:30:46.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0035999999999999e-05 2024-02-05T07:30:46.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.253159+0000", 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4490000000000005e-06 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.220866+0000", 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.96770688800000004 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.220939+0000", 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3151000000000005e-05 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:30:46.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.221068+0000", 2024-02-05T07:30:46.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128668 2024-02-05T07:30:46.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.223184+0000", 2024-02-05T07:30:46.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021168570000000002 2024-02-05T07:30:46.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.230577+0000", 2024-02-05T07:30:46.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073927669999999997 2024-02-05T07:30:46.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.235725+0000", 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051474659999999998 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.235782+0000", 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7593000000000002e-05 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.235860+0000", 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7843000000000007e-05 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.072 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:163461 8.11 8:89ef237c:::benchmark_data_smithi007_105755_object27151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-02-05T07:30:46.072 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:31.288873+0000", 2024-02-05T07:30:46.072 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 194.67899020600001, 2024-02-05T07:30:46.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94719473099999996, 2024-02-05T07:30:46.072 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.072 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:46.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.072 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 163461 2024-02-05T07:30:46.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.073 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.288873+0000", 2024-02-05T07:30:46.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.288873+0000", 2024-02-05T07:30:46.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.288874+0000", 2024-02-05T07:30:46.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1680000000000001e-06 2024-02-05T07:30:46.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.288878+0000", 2024-02-05T07:30:46.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3969999999999998e-06 2024-02-05T07:30:46.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.288889+0000", 2024-02-05T07:30:46.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1025e-05 2024-02-05T07:30:46.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.288894+0000", 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.835e-06 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.288945+0000", 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0973000000000001e-05 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.288953+0000", 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4580000000000004e-06 2024-02-05T07:30:46.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.222041+0000", 2024-02-05T07:30:46.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93308844199999996 2024-02-05T07:30:46.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.222157+0000", 2024-02-05T07:30:46.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115264 2024-02-05T07:30:46.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:30:46.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.222352+0000", 2024-02-05T07:30:46.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019531099999999999 2024-02-05T07:30:46.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.226386+0000", 2024-02-05T07:30:46.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004034223 2024-02-05T07:30:46.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.230629+0000", 2024-02-05T07:30:46.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042429479999999999 2024-02-05T07:30:46.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.235913+0000", 2024-02-05T07:30:46.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052841269999999996 2024-02-05T07:30:46.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:46.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.235971+0000", 2024-02-05T07:30:46.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7682000000000001e-05 2024-02-05T07:30:46.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.236068+0000", 2024-02-05T07:30:46.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6878000000000003e-05 2024-02-05T07:30:46.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.079 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.079 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:163500 8.11 8:8fda3e75:::benchmark_data_smithi007_105755_object27190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-02-05T07:30:46.079 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:31.319868+0000", 2024-02-05T07:30:46.079 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 194.64799488899999, 2024-02-05T07:30:46.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.91900468099999999, 2024-02-05T07:30:46.079 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 163500 2024-02-05T07:30:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.319868+0000", 2024-02-05T07:30:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.319868+0000", 2024-02-05T07:30:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.319869+0000", 2024-02-05T07:30:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8000000000000005e-07 2024-02-05T07:30:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.319874+0000", 2024-02-05T07:30:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6079999999999998e-06 2024-02-05T07:30:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.319881+0000", 2024-02-05T07:30:46.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4270000000000002e-06 2024-02-05T07:30:46.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.319886+0000", 2024-02-05T07:30:46.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2050000000000001e-06 2024-02-05T07:30:46.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.319930+0000", 2024-02-05T07:30:46.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3455000000000003e-05 2024-02-05T07:30:46.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.319937+0000", 2024-02-05T07:30:46.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7139999999999996e-06 2024-02-05T07:30:46.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.223218+0000", 2024-02-05T07:30:46.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.90328113600000004 2024-02-05T07:30:46.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.223330+0000", 2024-02-05T07:30:46.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112446 2024-02-05T07:30:46.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:30:46.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.223525+0000", 2024-02-05T07:30:46.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019513899999999999 2024-02-05T07:30:46.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.226392+0000", 2024-02-05T07:30:46.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028670969999999999 2024-02-05T07:30:46.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.230720+0000", 2024-02-05T07:30:46.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043280510000000003 2024-02-05T07:30:46.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.238727+0000", 2024-02-05T07:30:46.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0080069640000000001 2024-02-05T07:30:46.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:46.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.238782+0000", 2024-02-05T07:30:46.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2024-02-05T07:30:46.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.238873+0000", 2024-02-05T07:30:46.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1334000000000003e-05 2024-02-05T07:30:46.088 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.088 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.088 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.088 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:163503 8.11 8:8be25dfd:::benchmark_data_smithi007_105755_object27193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-02-05T07:30:46.089 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:31.323594+0000", 2024-02-05T07:30:46.089 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 194.64426901799999, 2024-02-05T07:30:46.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.91546838699999999, 2024-02-05T07:30:46.089 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.089 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:46.089 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.090 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.090 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.090 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 163503 2024-02-05T07:30:46.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.090 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.323594+0000", 2024-02-05T07:30:46.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.323594+0000", 2024-02-05T07:30:46.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.323596+0000", 2024-02-05T07:30:46.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4929999999999999e-06 2024-02-05T07:30:46.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.323601+0000", 2024-02-05T07:30:46.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9490000000000004e-06 2024-02-05T07:30:46.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.323614+0000", 2024-02-05T07:30:46.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3499000000000001e-05 2024-02-05T07:30:46.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.323623+0000", 2024-02-05T07:30:46.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5490000000000006e-06 2024-02-05T07:30:46.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.323689+0000", 2024-02-05T07:30:46.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6112000000000007e-05 2024-02-05T07:30:46.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.323700+0000", 2024-02-05T07:30:46.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1307e-05 2024-02-05T07:30:46.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.224407+0000", 2024-02-05T07:30:46.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.90070708300000002 2024-02-05T07:30:46.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.224499+0000", 2024-02-05T07:30:46.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2237999999999993e-05 2024-02-05T07:30:46.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:30:46.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.224633+0000", 2024-02-05T07:30:46.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133876 2024-02-05T07:30:46.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.230534+0000", 2024-02-05T07:30:46.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0059010649999999996 2024-02-05T07:30:46.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.230790+0000", 2024-02-05T07:30:46.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025535899999999998 2024-02-05T07:30:46.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.238915+0000", 2024-02-05T07:30:46.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0081252500000000005 2024-02-05T07:30:46.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:46.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.238948+0000", 2024-02-05T07:30:46.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3219999999999997e-05 2024-02-05T07:30:46.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.239063+0000", 2024-02-05T07:30:46.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114387 2024-02-05T07:30:46.104 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.104 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.104 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.105 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:163525 8.11 8:88763a89:::benchmark_data_smithi007_105755_object27215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-02-05T07:30:46.105 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:31.343462+0000", 2024-02-05T07:30:46.105 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 194.624400979, 2024-02-05T07:30:46.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.89572998599999998, 2024-02-05T07:30:46.105 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.105 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:46.106 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.106 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.106 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.106 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 163525 2024-02-05T07:30:46.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.106 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.343462+0000", 2024-02-05T07:30:46.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.343462+0000", 2024-02-05T07:30:46.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.343464+0000", 2024-02-05T07:30:46.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.39e-06 2024-02-05T07:30:46.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.343469+0000", 2024-02-05T07:30:46.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.485e-06 2024-02-05T07:30:46.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.343482+0000", 2024-02-05T07:30:46.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.271e-05 2024-02-05T07:30:46.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.343491+0000", 2024-02-05T07:30:46.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0720000000000006e-06 2024-02-05T07:30:46.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.343553+0000", 2024-02-05T07:30:46.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1631999999999995e-05 2024-02-05T07:30:46.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:31.343585+0000", 2024-02-05T07:30:46.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2051999999999997e-05 2024-02-05T07:30:46.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.225352+0000", 2024-02-05T07:30:46.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.88176712199999996 2024-02-05T07:30:46.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.225443+0000", 2024-02-05T07:30:46.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1719999999999996e-05 2024-02-05T07:30:46.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:30:46.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.225581+0000", 2024-02-05T07:30:46.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013709700000000001 2024-02-05T07:30:46.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.230649+0000", 2024-02-05T07:30:46.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050683129999999996 2024-02-05T07:30:46.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.234315+0000", 2024-02-05T07:30:46.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036658670000000002 2024-02-05T07:30:46.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.239100+0000", 2024-02-05T07:30:46.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047852900000000002 2024-02-05T07:30:46.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:46.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.239130+0000", 2024-02-05T07:30:46.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0205e-05 2024-02-05T07:30:46.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:32.239192+0000", 2024-02-05T07:30:46.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2031000000000003e-05 2024-02-05T07:30:46.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.120 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.120 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:271957 8.17 8:e915d766:::benchmark_data_smithi007_105755_object135647:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:30:46.120 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:29:27.647635+0000", 2024-02-05T07:30:46.120 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 78.320228588000006, 2024-02-05T07:30:46.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98865452799999998, 2024-02-05T07:30:46.120 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.120 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:46.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.121 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 271957 2024-02-05T07:30:46.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.121 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647635+0000", 2024-02-05T07:30:46.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647635+0000", 2024-02-05T07:30:46.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647636+0000", 2024-02-05T07:30:46.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3880000000000001e-06 2024-02-05T07:30:46.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647642+0000", 2024-02-05T07:30:46.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8270000000000003e-06 2024-02-05T07:30:46.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647655+0000", 2024-02-05T07:30:46.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.332e-05 2024-02-05T07:30:46.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647664+0000", 2024-02-05T07:30:46.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2099999999999999e-06 2024-02-05T07:30:46.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647726+0000", 2024-02-05T07:30:46.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1384999999999996e-05 2024-02-05T07:30:46.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647735+0000", 2024-02-05T07:30:46.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2024-02-05T07:30:46.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.631569+0000", 2024-02-05T07:30:46.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98383460599999994 2024-02-05T07:30:46.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.631666+0000", 2024-02-05T07:30:46.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6846000000000001e-05 2024-02-05T07:30:46.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,6", 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.631751+0000", 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5303000000000006e-05 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.635981+0000", 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042294289999999998 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636138+0000", 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156988 2024-02-05T07:30:46.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636232+0000", 2024-02-05T07:30:46.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4400000000000004e-05 2024-02-05T07:30:46.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:46.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636266+0000", 2024-02-05T07:30:46.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3318999999999997e-05 2024-02-05T07:30:46.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636289+0000", 2024-02-05T07:30:46.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3609999999999999e-05 2024-02-05T07:30:46.127 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.127 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.127 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.127 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:271993 8.17 8:eb340f93:::benchmark_data_smithi007_105755_object135683:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:30:46.127 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:29:27.688048+0000", 2024-02-05T07:30:46.127 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 78.279814905999999, 2024-02-05T07:30:46.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94901903399999998, 2024-02-05T07:30:46.127 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.128 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:46.128 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.128 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.128 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.128 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 271993 2024-02-05T07:30:46.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.128 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688048+0000", 2024-02-05T07:30:46.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688048+0000", 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688049+0000", 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.096e-06 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688054+0000", 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3980000000000001e-06 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688064+0000", 2024-02-05T07:30:46.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0614999999999999e-05 2024-02-05T07:30:46.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688072+0000", 2024-02-05T07:30:46.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5299999999999999e-06 2024-02-05T07:30:46.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688162+0000", 2024-02-05T07:30:46.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0325e-05 2024-02-05T07:30:46.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688171+0000", 2024-02-05T07:30:46.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1689999999999998e-06 2024-02-05T07:30:46.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.633668+0000", 2024-02-05T07:30:46.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94549656900000001 2024-02-05T07:30:46.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.633783+0000", 2024-02-05T07:30:46.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115307 2024-02-05T07:30:46.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,6", 2024-02-05T07:30:46.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.633905+0000", 2024-02-05T07:30:46.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121409 2024-02-05T07:30:46.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636196+0000", 2024-02-05T07:30:46.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022911559999999999 2024-02-05T07:30:46.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636210+0000", 2024-02-05T07:30:46.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3899e-05 2024-02-05T07:30:46.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.637021+0000", 2024-02-05T07:30:46.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081141799999999995 2024-02-05T07:30:46.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:46.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.637047+0000", 2024-02-05T07:30:46.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5973e-05 2024-02-05T07:30:46.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.637067+0000", 2024-02-05T07:30:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0169999999999998e-05 2024-02-05T07:30:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:272015 8.17 8:ed63ad4e:::benchmark_data_smithi007_105755_object135705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:30:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:29:27.722017+0000", 2024-02-05T07:30:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 78.245846470999993, 2024-02-05T07:30:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.91513220399999995, 2024-02-05T07:30:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 272015 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722017+0000", 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722017+0000", 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722018+0000", 2024-02-05T07:30:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7600000000000006e-07 2024-02-05T07:30:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722021+0000", 2024-02-05T07:30:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2024-02-05T07:30:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722034+0000", 2024-02-05T07:30:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2901999999999999e-05 2024-02-05T07:30:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722039+0000", 2024-02-05T07:30:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5430000000000001e-06 2024-02-05T07:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722093+0000", 2024-02-05T07:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4280999999999997e-05 2024-02-05T07:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722102+0000", 2024-02-05T07:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8189999999999998e-06 2024-02-05T07:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.634969+0000", 2024-02-05T07:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.91286651699999999 2024-02-05T07:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.635072+0000", 2024-02-05T07:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103607 2024-02-05T07:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,6", 2024-02-05T07:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.635152+0000", 2024-02-05T07:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9505000000000005e-05 2024-02-05T07:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636325+0000", 2024-02-05T07:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011735739999999999 2024-02-05T07:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636339+0000", 2024-02-05T07:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3173000000000001e-05 2024-02-05T07:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.637105+0000", 2024-02-05T07:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076605499999999997 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.637128+0000", 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3546000000000001e-05 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.637149+0000", 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0764e-05 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:46.142 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:30:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:46.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:46.143 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:46.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:46.272 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:46.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:46.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:46.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:46.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:46.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:46.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:46.532 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:46.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:46.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:46.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:30:46.660 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:46.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:46.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:46.704 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:30:46.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:46.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:46 smithi007 ceph-mon[29098]: pgmap v625: 168 pgs: 13 remapped+peering, 13 peering, 3 unknown, 139 active+clean; 2.7 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 6.2 KiB/s rd, 1.5 MiB/s wr, 30 op/s; 39 KiB/s, 6 objects/s recovering 2024-02-05T07:30:46.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:46 smithi007 ceph-mon[29098]: osdmap e370: 8 total, 8 up, 6 in 2024-02-05T07:30:46.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:46 smithi007 ceph-mon[29098]: 6.12 scrub starts 2024-02-05T07:30:46.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:46 smithi007 ceph-mon[29098]: 6.12 scrub ok 2024-02-05T07:30:46.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:46 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4071799959' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:30:46.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:46 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:30:46.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:46 smithi007 ceph-mon[37158]: pgmap v625: 168 pgs: 13 remapped+peering, 13 peering, 3 unknown, 139 active+clean; 2.7 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 6.2 KiB/s rd, 1.5 MiB/s wr, 30 op/s; 39 KiB/s, 6 objects/s recovering 2024-02-05T07:30:46.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:46 smithi007 ceph-mon[37158]: osdmap e370: 8 total, 8 up, 6 in 2024-02-05T07:30:46.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:46 smithi007 ceph-mon[37158]: 6.12 scrub starts 2024-02-05T07:30:46.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:46 smithi007 ceph-mon[37158]: 6.12 scrub ok 2024-02-05T07:30:46.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:46 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4071799959' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:30:46.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:46 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:30:46.833 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:46.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:46.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:46.932 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 31 16 44943 44927 90.565 41.875 0.0182548 0.0110338 2024-02-05T07:30:46.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:46.965 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:46.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:46.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:46 smithi090 ceph-mon[33213]: pgmap v625: 168 pgs: 13 remapped+peering, 13 peering, 3 unknown, 139 active+clean; 2.7 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 6.2 KiB/s rd, 1.5 MiB/s wr, 30 op/s; 39 KiB/s, 6 objects/s recovering 2024-02-05T07:30:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:46 smithi090 ceph-mon[33213]: osdmap e370: 8 total, 8 up, 6 in 2024-02-05T07:30:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:46 smithi090 ceph-mon[33213]: 6.12 scrub starts 2024-02-05T07:30:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:46 smithi090 ceph-mon[33213]: 6.12 scrub ok 2024-02-05T07:30:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:46 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4071799959' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:30:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:46 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:30:47.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:47.094 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:47.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:47.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:47.195 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:47.227 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:47.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:47.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:47.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T07:30:47.329 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:47.383 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:47.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:47.485 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:47.517 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:47.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:47.591 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:30:47.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:47.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:47.661 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:47.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:47.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:47.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:47.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:47.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:47 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:30:47.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:47 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:30:47.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:47 smithi007 ceph-mon[29098]: osdmap e371: 8 total, 8 up, 6 in 2024-02-05T07:30:47.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:47 smithi007 ceph-mon[29098]: pgmap v628: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 157 active+clean; 2.7 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 7.5 KiB/s rd, 11 MiB/s wr, 185 op/s; 13734/133035 objects degraded (10.324%); 11736/133035 objects misplaced (8.822%); 28 MiB/s, 452 objects/s recovering 2024-02-05T07:30:47.844 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:47 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:30:47.844 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:47 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:30:47.845 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:47 smithi007 ceph-mon[37158]: osdmap e371: 8 total, 8 up, 6 in 2024-02-05T07:30:47.845 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:47 smithi007 ceph-mon[37158]: pgmap v628: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 157 active+clean; 2.7 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 7.5 KiB/s rd, 11 MiB/s wr, 185 op/s; 13734/133035 objects degraded (10.324%); 11736/133035 objects misplaced (8.822%); 28 MiB/s, 452 objects/s recovering 2024-02-05T07:30:47.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:47.932 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 32 16 45680 45664 89.1742 46.0625 0.0368996 0.0112061 2024-02-05T07:30:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:47.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:47.968 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:47 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:30:47.968 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:47 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:30:47.968 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:47 smithi090 ceph-mon[33213]: osdmap e371: 8 total, 8 up, 6 in 2024-02-05T07:30:47.968 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:47 smithi090 ceph-mon[33213]: pgmap v628: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 157 active+clean; 2.7 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 7.5 KiB/s rd, 11 MiB/s wr, 185 op/s; 13734/133035 objects degraded (10.324%); 11736/133035 objects misplaced (8.822%); 28 MiB/s, 452 objects/s recovering 2024-02-05T07:30:48.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:48.064 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:48.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:48.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:48.205 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:48.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:48.307 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:48.332 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:48.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:48.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:48.461 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:48.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:48.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:48.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:48.611 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:48.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:48.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:48.712 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:48.767 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:48.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:48.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:48.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:48 smithi007 ceph-mon[29098]: Health check failed: Degraded data redundancy: 13734/133035 objects degraded (10.324%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:30:48.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:48 smithi007 ceph-mon[29098]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 8 pgs peering) 2024-02-05T07:30:48.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:48 smithi007 ceph-mon[37158]: Health check failed: Degraded data redundancy: 13734/133035 objects degraded (10.324%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:30:48.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:48 smithi007 ceph-mon[37158]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 8 pgs peering) 2024-02-05T07:30:48.922 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:48.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:48.935 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 33 16 46399 46383 87.8275 44.9375 0.0474174 0.0113781 2024-02-05T07:30:48.959 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:48 smithi090 ceph-mon[33213]: Health check failed: Degraded data redundancy: 13734/133035 objects degraded (10.324%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:30:48.959 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:48 smithi090 ceph-mon[33213]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 8 pgs peering) 2024-02-05T07:30:49.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:49.052 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:49.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:49.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:49.203 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:49.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:49.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:49.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:49.342 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:49.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:49.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:49.407 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:30:49.407 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:30:49.407 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-02-05T07:30:49.407 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:30:49.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:49.470 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:49.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:49.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:49.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:49.615 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:49.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:49.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:49.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:49.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:49.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:49.869 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:49.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:49.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:30:49.935 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 34 16 47010 46994 86.3672 38.1875 0.010949 0.0115712 2024-02-05T07:30:49.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:49.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:49.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:50.098 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:50.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T07:30:50.155 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:50.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:50.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:50.217 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:30:50.229 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:30:50.234 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:30:50.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:50.372 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:50.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:50.387 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:30:50.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:50.523 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:50.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:50.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:50.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:50.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:50.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:50.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:30:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:30:50.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:50 smithi090 ceph-mon[33213]: pgmap v629: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 157 active+clean; 2.7 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 5.2 KiB/s rd, 9.5 MiB/s wr, 158 op/s; 13734/133536 objects degraded (10.285%); 11736/133536 objects misplaced (8.789%); 19 MiB/s, 315 objects/s recovering 2024-02-05T07:30:50.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:50 smithi090 ceph-mon[33213]: osdmap e372: 8 total, 8 up, 6 in 2024-02-05T07:30:50.798 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:50.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:50.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:50.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:50 smithi007 ceph-mon[29098]: pgmap v629: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 157 active+clean; 2.7 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 5.2 KiB/s rd, 9.5 MiB/s wr, 158 op/s; 13734/133536 objects degraded (10.285%); 11736/133536 objects misplaced (8.789%); 19 MiB/s, 315 objects/s recovering 2024-02-05T07:30:50.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:50 smithi007 ceph-mon[29098]: osdmap e372: 8 total, 8 up, 6 in 2024-02-05T07:30:50.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:50 smithi007 ceph-mon[37158]: pgmap v629: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 157 active+clean; 2.7 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 5.2 KiB/s rd, 9.5 MiB/s wr, 158 op/s; 13734/133536 objects degraded (10.285%); 11736/133536 objects misplaced (8.789%); 19 MiB/s, 315 objects/s recovering 2024-02-05T07:30:50.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:50 smithi007 ceph-mon[37158]: osdmap e372: 8 total, 8 up, 6 in 2024-02-05T07:30:50.927 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:50.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:50.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:50.935 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 35 16 47650 47634 85.0425 40 0.0273615 0.0117513 2024-02-05T07:30:51.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:51.056 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:51.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:51.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:51.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:51.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:51.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:51.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:51.346 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:51.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:51.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:51.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:51.474 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:51.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:51.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:51 smithi090 ceph-mon[33213]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:30:51.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:51 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:51.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:51.603 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:51.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:51.705 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:51.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:51.756 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:51.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:51.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:51.900 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:51.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:51.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:51.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:51 smithi007 ceph-mon[29098]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:30:51.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:51 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:51.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:51 smithi007 ceph-mon[37158]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:30:51.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:51 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:30:51.935 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 36 15 48308 48293 83.8244 41.1875 0.0371225 0.0119207 2024-02-05T07:30:52.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:52.026 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:52.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:52.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:52.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:52.158 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:52.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:52.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:52.175 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:30:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:30:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:30:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:30:52.175 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:30:52.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:52.283 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:52.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:52.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:52.429 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:52.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:52.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:52.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:52.580 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:52.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:52.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:52.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:52.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:52.735 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:52.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:52.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:52 smithi090 ceph-mon[33213]: pgmap v631: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 2.9 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 1.7 KiB/s rd, 41 MiB/s wr, 658 op/s; 13734/143019 objects degraded (9.603%); 10814/143019 objects misplaced (7.561%); 28 MiB/s, 450 objects/s recovering 2024-02-05T07:30:52.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:52.886 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:52.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:52.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:52.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:52 smithi007 ceph-mon[29098]: pgmap v631: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 2.9 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 1.7 KiB/s rd, 41 MiB/s wr, 658 op/s; 13734/143019 objects degraded (9.603%); 10814/143019 objects misplaced (7.561%); 28 MiB/s, 450 objects/s recovering 2024-02-05T07:30:52.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:52 smithi007 ceph-mon[37158]: pgmap v631: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 2.9 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 1.7 KiB/s rd, 41 MiB/s wr, 658 op/s; 13734/143019 objects degraded (9.603%); 10814/143019 objects misplaced (7.561%); 28 MiB/s, 450 objects/s recovering 2024-02-05T07:30:52.935 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 37 16 48854 48838 82.4795 34.0625 0.0130613 0.0121153 2024-02-05T07:30:52.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T07:30:52.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:53.013 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:53.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:53.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:53.141 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:53.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:53.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:53.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:30:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:53.284 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:53.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:53.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:53.409 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:53.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:53.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:53.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:53.542 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:53.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:53.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:53.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:53.671 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:53.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:53.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:53.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:53.798 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:53.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:53.872 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:30:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:30:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:30:53.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:53.927 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:53.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:53.935 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 38 16 49401 49385 81.2088 34.1875 0.0337908 0.0123064 2024-02-05T07:30:54.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:54.052 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:54.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:54.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:54.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:54.179 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:54.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:54.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:54.281 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:54.304 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:54.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:54.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:54.431 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:54.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:54.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:54.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:54.560 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:54.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:54.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:54.701 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:54.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:54.733 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:54 smithi090 ceph-mon[33213]: pgmap v632: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 2.9 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 1.3 KiB/s rd, 32 MiB/s wr, 521 op/s; 13734/143019 objects degraded (9.603%); 10814/143019 objects misplaced (7.561%); 22 MiB/s, 358 objects/s recovering 2024-02-05T07:30:54.733 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:54 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 13734/143019 objects degraded (9.603%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:30:54.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:54.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:54.828 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:54.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:54.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:54 smithi007 ceph-mon[29098]: pgmap v632: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 2.9 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 1.3 KiB/s rd, 32 MiB/s wr, 521 op/s; 13734/143019 objects degraded (9.603%); 10814/143019 objects misplaced (7.561%); 22 MiB/s, 358 objects/s recovering 2024-02-05T07:30:54.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:54 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 13734/143019 objects degraded (9.603%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:30:54.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:54 smithi007 ceph-mon[37158]: pgmap v632: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 2.9 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 1.3 KiB/s rd, 32 MiB/s wr, 521 op/s; 13734/143019 objects degraded (9.603%); 10814/143019 objects misplaced (7.561%); 22 MiB/s, 358 objects/s recovering 2024-02-05T07:30:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:54 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 13734/143019 objects degraded (9.603%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:30:54.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:54.935 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 39 16 49831 49815 79.8157 26.875 0.00380854 0.0125208 2024-02-05T07:30:54.958 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:54.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:55.025 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:30:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-02-05T07:30:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-02-05T07:30:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:30:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:210982 8.5 8.3d282b45 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e301)", 2024-02-05T07:30:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:24.576942+0000", 2024-02-05T07:30:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 150.44453277, 2024-02-05T07:30:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1733628220000001, 2024-02-05T07:30:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:30:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 210982 2024-02-05T07:30:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.576942+0000", 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.576942+0000", 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.576943+0000", 2024-02-05T07:30:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7659999999999999e-06 2024-02-05T07:30:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.576949+0000", 2024-02-05T07:30:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3510000000000001e-06 2024-02-05T07:30:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.576958+0000", 2024-02-05T07:30:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7780000000000002e-06 2024-02-05T07:30:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.576970+0000", 2024-02-05T07:30:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253999999999999e-05 2024-02-05T07:30:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.579239+0000", 2024-02-05T07:30:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022689939999999999 2024-02-05T07:30:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.579251+0000", 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2125e-05 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.750291+0000", 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1710403359999999 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.750304+0000", 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3217999999999999e-05 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24356.0:2078 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 93862608739072 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-02-05T07:30:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:24.587417+0000", 2024-02-05T07:30:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 150.434057465, 2024-02-05T07:30:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2858643619999999, 2024-02-05T07:30:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:30:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24356", 2024-02-05T07:30:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2716729371", 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2078 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587417+0000", 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587417+0000", 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587418+0000", 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.083e-06 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587421+0000", 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9119999999999998e-06 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587428+0000", 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8379999999999996e-06 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587435+0000", 2024-02-05T07:30:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9210000000000003e-06 2024-02-05T07:30:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587488+0000", 2024-02-05T07:30:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3013999999999998e-05 2024-02-05T07:30:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587498+0000", 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9599999999999995e-06 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.844368+0000", 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2568704500000001 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.844526+0000", 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015806899999999999 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.873281+0000", 2024-02-05T07:30:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028755115000000001 2024-02-05T07:30:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24356.0:2079 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 93862613159296 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-02-05T07:30:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:24.587448+0000", 2024-02-05T07:30:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 150.43402635300001, 2024-02-05T07:30:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4188860320000001, 2024-02-05T07:30:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:30:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24356", 2024-02-05T07:30:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2716729371", 2024-02-05T07:30:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2079 2024-02-05T07:30:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587448+0000", 2024-02-05T07:30:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587448+0000", 2024-02-05T07:30:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587448+0000", 2024-02-05T07:30:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.91e-07 2024-02-05T07:30:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587450+0000", 2024-02-05T07:30:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.691e-06 2024-02-05T07:30:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587466+0000", 2024-02-05T07:30:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5739e-05 2024-02-05T07:30:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587471+0000", 2024-02-05T07:30:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0479999999999998e-06 2024-02-05T07:30:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587505+0000", 2024-02-05T07:30:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4044000000000001e-05 2024-02-05T07:30:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587512+0000", 2024-02-05T07:30:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2370000000000003e-06 2024-02-05T07:30:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.868905+0000", 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2813931219999999 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.869089+0000", 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018377199999999999 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:26.006334+0000", 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13724508799999999 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24356.0:2081 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 93862608734464 gen 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-02-05T07:30:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:25.742864+0000", 2024-02-05T07:30:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 149.278610458, 2024-02-05T07:30:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87074127499999998, 2024-02-05T07:30:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:30:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24356", 2024-02-05T07:30:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2716729371", 2024-02-05T07:30:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2081 2024-02-05T07:30:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.742864+0000", 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.742864+0000", 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.742865+0000", 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4190000000000001e-06 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.742876+0000", 2024-02-05T07:30:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1001e-05 2024-02-05T07:30:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.742886+0000", 2024-02-05T07:30:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8519999999999999e-06 2024-02-05T07:30:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.742900+0000", 2024-02-05T07:30:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3908e-05 2024-02-05T07:30:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.748077+0000", 2024-02-05T07:30:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051767239999999997 2024-02-05T07:30:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.748087+0000", 2024-02-05T07:30:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0609e-05 2024-02-05T07:30:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:26.597167+0000", 2024-02-05T07:30:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.84907927599999999 2024-02-05T07:30:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:26.597325+0000", 2024-02-05T07:30:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015865599999999999 2024-02-05T07:30:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:26.613605+0000", 2024-02-05T07:30:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016279829999999999 2024-02-05T07:30:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(8.5 308/302 [PushOp(8:a39c4052:::benchmark_data_smithi007_105755_object93948:head, version: 239'6033, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:a39c4052:::benchmark_data_smithi007_105755_object93948:head@239'6033, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:30:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:31.932956+0000", 2024-02-05T07:30:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 143.08851819500001, 2024-02-05T07:30:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120018678, 2024-02-05T07:30:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:30:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:31.932956+0000", 2024-02-05T07:30:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:31.932956+0000", 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:31.932957+0000", 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6199999999999998e-07 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:31.932992+0000", 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5160000000000002e-05 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:32.051536+0000", 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.118543747 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:32.051570+0000", 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4275e-05 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:32.052781+0000", 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001211476 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:32.052795+0000", 2024-02-05T07:30:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3311e-05 2024-02-05T07:30:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:32.052975+0000", 2024-02-05T07:30:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018014699999999999 2024-02-05T07:30:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:218870 8.5 8:a7634133:::benchmark_data_smithi007_105755_object82560:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-02-05T07:30:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:37.819605+0000", 2024-02-05T07:30:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 137.201869187, 2024-02-05T07:30:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.88666498199999999, 2024-02-05T07:30:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218870 2024-02-05T07:30:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819605+0000", 2024-02-05T07:30:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819605+0000", 2024-02-05T07:30:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819607+0000", 2024-02-05T07:30:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5069999999999999e-06 2024-02-05T07:30:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819612+0000", 2024-02-05T07:30:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0130000000000001e-06 2024-02-05T07:30:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819625+0000", 2024-02-05T07:30:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3754000000000001e-05 2024-02-05T07:30:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819633+0000", 2024-02-05T07:30:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9869999999999999e-06 2024-02-05T07:30:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819696+0000", 2024-02-05T07:30:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2669999999999995e-05 2024-02-05T07:30:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819706+0000", 2024-02-05T07:30:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7049999999999994e-06 2024-02-05T07:30:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.700281+0000", 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.88057515200000003 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.700417+0000", 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013600400000000001 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.700693+0000", 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027553400000000002 2024-02-05T07:30:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.702831+0000", 2024-02-05T07:30:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021382390000000001 2024-02-05T07:30:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.706057+0000", 2024-02-05T07:30:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032261880000000001 2024-02-05T07:30:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.706167+0000", 2024-02-05T07:30:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110256 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.706232+0000", 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5079000000000005e-05 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.706270+0000", 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7894e-05 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:218941 8.5 8:a24ccea1:::benchmark_data_smithi007_105755_object82631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-02-05T07:30:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:37.877696+0000", 2024-02-05T07:30:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 137.14377869500001, 2024-02-05T07:30:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.83040171799999996, 2024-02-05T07:30:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218941 2024-02-05T07:30:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877696+0000", 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877696+0000", 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877697+0000", 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6750000000000001e-06 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877703+0000", 2024-02-05T07:30:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7450000000000001e-06 2024-02-05T07:30:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877714+0000", 2024-02-05T07:30:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1246e-05 2024-02-05T07:30:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877722+0000", 2024-02-05T07:30:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5780000000000001e-06 2024-02-05T07:30:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877950+0000", 2024-02-05T07:30:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022855 2024-02-05T07:30:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877962+0000", 2024-02-05T07:30:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1847e-05 2024-02-05T07:30:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.702860+0000", 2024-02-05T07:30:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82489743699999996 2024-02-05T07:30:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.702981+0000", 2024-02-05T07:30:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120763 2024-02-05T07:30:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-02-05T07:30:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.703200+0000", 2024-02-05T07:30:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021935999999999999 2024-02-05T07:30:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.707893+0000", 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046932520000000002 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.707931+0000", 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.756e-05 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.707973+0000", 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1836999999999999e-05 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708059+0000", 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6238999999999999e-05 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708097+0000", 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8628999999999999e-05 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:218996 8.5 8:a17532da:::benchmark_data_smithi007_105755_object82686:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:37.922624+0000", 2024-02-05T07:30:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 137.098849882, 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.78597759899999997, 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218996 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.922624+0000", 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.922624+0000", 2024-02-05T07:30:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.922626+0000", 2024-02-05T07:30:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9130000000000001e-06 2024-02-05T07:30:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.922631+0000", 2024-02-05T07:30:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6450000000000004e-06 2024-02-05T07:30:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.922645+0000", 2024-02-05T07:30:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3974e-05 2024-02-05T07:30:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.922653+0000", 2024-02-05T07:30:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9330000000000001e-06 2024-02-05T07:30:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.923228+0000", 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057506799999999998 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.923236+0000", 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3049999999999993e-06 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.704144+0000", 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.78090789400000005 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.704253+0000", 2024-02-05T07:30:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108874 2024-02-05T07:30:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-02-05T07:30:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.704445+0000", 2024-02-05T07:30:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192177 2024-02-05T07:30:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.707941+0000", 2024-02-05T07:30:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034960429999999999 2024-02-05T07:30:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708161+0000", 2024-02-05T07:30:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000220168 2024-02-05T07:30:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708210+0000", 2024-02-05T07:30:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8848999999999999e-05 2024-02-05T07:30:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708555+0000", 2024-02-05T07:30:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034451700000000002 2024-02-05T07:30:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708602+0000", 2024-02-05T07:30:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7239e-05 2024-02-05T07:30:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:219029 8.5 8:a492476d:::benchmark_data_smithi007_105755_object82719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-02-05T07:30:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:37.952103+0000", 2024-02-05T07:30:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 137.06937098399999, 2024-02-05T07:30:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.75848268500000005, 2024-02-05T07:30:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 219029 2024-02-05T07:30:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952103+0000", 2024-02-05T07:30:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952103+0000", 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952105+0000", 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3969999999999999e-06 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952109+0000", 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.464e-06 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952122+0000", 2024-02-05T07:30:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3064999999999999e-05 2024-02-05T07:30:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952131+0000", 2024-02-05T07:30:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5879999999999998e-06 2024-02-05T07:30:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952181+0000", 2024-02-05T07:30:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9895000000000003e-05 2024-02-05T07:30:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952190+0000", 2024-02-05T07:30:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8799999999999997e-06 2024-02-05T07:30:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.706326+0000", 2024-02-05T07:30:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.75413592699999998 2024-02-05T07:30:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.706428+0000", 2024-02-05T07:30:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102396 2024-02-05T07:30:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-02-05T07:30:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.706617+0000", 2024-02-05T07:30:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018896900000000001 2024-02-05T07:30:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.710420+0000", 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038025670000000002 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.710430+0000", 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.024e-05 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.710471+0000", 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1331000000000001e-05 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.710528+0000", 2024-02-05T07:30:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6611999999999999e-05 2024-02-05T07:30:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.710586+0000", 2024-02-05T07:30:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8353999999999997e-05 2024-02-05T07:30:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:219073 8.5 8:a5d9854b:::benchmark_data_smithi007_105755_object82763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:38.040105+0000", 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 136.98136899100001, 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.672581702, 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 219073 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040105+0000", 2024-02-05T07:30:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040105+0000", 2024-02-05T07:30:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040107+0000", 2024-02-05T07:30:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.153e-06 2024-02-05T07:30:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040111+0000", 2024-02-05T07:30:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8899999999999998e-06 2024-02-05T07:30:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040123+0000", 2024-02-05T07:30:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1719000000000001e-05 2024-02-05T07:30:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040130+0000", 2024-02-05T07:30:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2024-02-05T07:30:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040200+0000", 2024-02-05T07:30:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0444999999999994e-05 2024-02-05T07:30:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:30:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040209+0000", 2024-02-05T07:30:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6780000000000006e-06 2024-02-05T07:30:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708697+0000", 2024-02-05T07:30:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66848853500000005 2024-02-05T07:30:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708831+0000", 2024-02-05T07:30:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133588 2024-02-05T07:30:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-02-05T07:30:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.709080+0000", 2024-02-05T07:30:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024862199999999999 2024-02-05T07:30:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.710602+0000", 2024-02-05T07:30:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015226879999999999 2024-02-05T07:30:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.711434+0000", 2024-02-05T07:30:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083224299999999998 2024-02-05T07:30:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.712628+0000", 2024-02-05T07:30:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011935959999999999 2024-02-05T07:30:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.712667+0000", 2024-02-05T07:30:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8506e-05 2024-02-05T07:30:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.712687+0000", 2024-02-05T07:30:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0537999999999999e-05 2024-02-05T07:30:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:226786 8.5 8:a32fc456:::benchmark_data_smithi007_105755_object90476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-02-05T07:30:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:48.299952+0000", 2024-02-05T07:30:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 126.72152239099999, 2024-02-05T07:30:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11645621, 2024-02-05T07:30:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 226786 2024-02-05T07:30:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:48.299952+0000", 2024-02-05T07:30:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:48.299952+0000", 2024-02-05T07:30:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:48.299953+0000", 2024-02-05T07:30:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.099e-06 2024-02-05T07:30:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:48.299957+0000", 2024-02-05T07:30:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9829999999999998e-06 2024-02-05T07:30:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:48.299966+0000", 2024-02-05T07:30:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2620000000000004e-06 2024-02-05T07:30:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:48.299972+0000", 2024-02-05T07:30:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5770000000000001e-06 2024-02-05T07:30:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:48.300020+0000", 2024-02-05T07:30:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8300000000000002e-05 2024-02-05T07:30:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:48.300109+0000", 2024-02-05T07:30:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8471e-05 2024-02-05T07:30:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-02-05T07:30:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:48.300347+0000", 2024-02-05T07:30:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023788499999999999 2024-02-05T07:30:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:48.301913+0000", 2024-02-05T07:30:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001566907 2024-02-05T07:30:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:48.302367+0000", 2024-02-05T07:30:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045393400000000002 2024-02-05T07:30:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:48.416253+0000", 2024-02-05T07:30:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.113885691 2024-02-05T07:30:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:48.416355+0000", 2024-02-05T07:30:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101857 2024-02-05T07:30:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:48.416408+0000", 2024-02-05T07:30:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3244e-05 2024-02-05T07:30:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:238577 8.5 8:a48344d7:::benchmark_data_smithi007_105755_object102267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-02-05T07:30:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:58.236143+0000", 2024-02-05T07:30:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 116.785330914, 2024-02-05T07:30:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15159524899999999, 2024-02-05T07:30:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 238577 2024-02-05T07:30:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.236143+0000", 2024-02-05T07:30:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.236143+0000", 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.236144+0000", 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0470000000000001e-06 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.236149+0000", 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6789999999999996e-06 2024-02-05T07:30:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.236158+0000", 2024-02-05T07:30:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1600000000000004e-06 2024-02-05T07:30:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.236164+0000", 2024-02-05T07:30:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5459999999999999e-06 2024-02-05T07:30:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.236204+0000", 2024-02-05T07:30:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9628000000000001e-05 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.236277+0000", 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3120999999999993e-05 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.236539+0000", 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000262749 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.237823+0000", 2024-02-05T07:30:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012832519999999999 2024-02-05T07:30:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.238812+0000", 2024-02-05T07:30:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00098924699999999996 2024-02-05T07:30:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.387620+0000", 2024-02-05T07:30:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.148808311 2024-02-05T07:30:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.387683+0000", 2024-02-05T07:30:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3188000000000006e-05 2024-02-05T07:30:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.387739+0000", 2024-02-05T07:30:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5321000000000001e-05 2024-02-05T07:30:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:238578 8.5 8:a43ad9ad:::benchmark_data_smithi007_105755_object102268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:58.243192+0000", 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 116.778282672, 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14466709, 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 238578 2024-02-05T07:30:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.243192+0000", 2024-02-05T07:30:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.243192+0000", 2024-02-05T07:30:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.243193+0000", 2024-02-05T07:30:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2610000000000001e-06 2024-02-05T07:30:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.243197+0000", 2024-02-05T07:30:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.527e-06 2024-02-05T07:30:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.243206+0000", 2024-02-05T07:30:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2539999999999992e-06 2024-02-05T07:30:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.243212+0000", 2024-02-05T07:30:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0109999999999997e-06 2024-02-05T07:30:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.243254+0000", 2024-02-05T07:30:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2194000000000003e-05 2024-02-05T07:30:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.243372+0000", 2024-02-05T07:30:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118232 2024-02-05T07:30:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-02-05T07:30:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.243636+0000", 2024-02-05T07:30:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026404500000000002 2024-02-05T07:30:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.245258+0000", 2024-02-05T07:30:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016221650000000001 2024-02-05T07:30:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.245748+0000", 2024-02-05T07:30:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048930800000000004 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.387781+0000", 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.142033465 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.387827+0000", 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5864e-05 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.387859+0000", 2024-02-05T07:30:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1764000000000003e-05 2024-02-05T07:30:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(8.9 325/301 [PushOp(8:91901038:::benchmark_data_smithi007_105755_object104863:head, version: 242'6546, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:91901038:::benchmark_data_smithi007_105755_object104863:head@242'6546, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:30:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:58.249754+0000", 2024-02-05T07:30:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 116.77172064200001, 2024-02-05T07:30:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12785632199999999, 2024-02-05T07:30:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:30:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.249754+0000", 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.249754+0000", 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.249754+0000", 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7599999999999997e-07 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.249820+0000", 2024-02-05T07:30:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6131999999999996e-05 2024-02-05T07:30:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.377041+0000", 2024-02-05T07:30:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12722109100000001 2024-02-05T07:30:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.377078+0000", 2024-02-05T07:30:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6977000000000002e-05 2024-02-05T07:30:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.377350+0000", 2024-02-05T07:30:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027174600000000001 2024-02-05T07:30:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.377364+0000", 2024-02-05T07:30:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3583e-05 2024-02-05T07:30:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.377610+0000", 2024-02-05T07:30:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024631700000000002 2024-02-05T07:30:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24977.0:238584 8.6 e325/302 8:6425a2a4:::benchmark_data_smithi007_105755_object102274:head v 325'11580, mlcod=325'11578)", 2024-02-05T07:30:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:58.250930+0000", 2024-02-05T07:30:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 116.77054474400001, 2024-02-05T07:30:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.130911953, 2024-02-05T07:30:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.250930+0000", 2024-02-05T07:30:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.250930+0000", 2024-02-05T07:30:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.250930+0000", 2024-02-05T07:30:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0500000000000004e-07 2024-02-05T07:30:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.250935+0000", 2024-02-05T07:30:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5940000000000004e-06 2024-02-05T07:30:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.377649+0000", 2024-02-05T07:30:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12671429200000001 2024-02-05T07:30:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.377657+0000", 2024-02-05T07:30:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5939999999999996e-06 2024-02-05T07:30:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.377707+0000", 2024-02-05T07:30:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0834000000000002e-05 2024-02-05T07:30:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.377728+0000", 2024-02-05T07:30:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.067e-05 2024-02-05T07:30:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.381784+0000", 2024-02-05T07:30:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040557910000000004 2024-02-05T07:30:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:58.381842+0000", 2024-02-05T07:30:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7673000000000002e-05 2024-02-05T07:30:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:26455 9.b 9:dba02e71:::benchmark_data_smithi007_136200_object26454:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-02-05T07:30:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:30:27.478187+0000", 2024-02-05T07:30:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.543287466999999, 2024-02-05T07:30:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15831493999999999, 2024-02-05T07:30:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:30:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:30:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 26455 2024-02-05T07:30:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.478187+0000", 2024-02-05T07:30:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.478187+0000", 2024-02-05T07:30:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.478188+0000", 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4190000000000001e-06 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.478223+0000", 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4511000000000002e-05 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.478234+0000", 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1139e-05 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.478242+0000", 2024-02-05T07:30:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8809999999999996e-06 2024-02-05T07:30:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.478298+0000", 2024-02-05T07:30:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6063000000000002e-05 2024-02-05T07:30:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.478422+0000", 2024-02-05T07:30:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000123724 2024-02-05T07:30:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2024-02-05T07:30:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.478563+0000", 2024-02-05T07:30:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014093100000000001 2024-02-05T07:30:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.480626+0000", 2024-02-05T07:30:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020630610000000001 2024-02-05T07:30:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481126+0000", 2024-02-05T07:30:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050072099999999998 2024-02-05T07:30:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.636395+0000", 2024-02-05T07:30:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.155269133 2024-02-05T07:30:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.636453+0000", 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7692000000000003e-05 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.636502+0000", 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8665000000000002e-05 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:26458 9.b 9:d70b51c3:::benchmark_data_smithi007_136200_object26457:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:30:27.479034+0000", 2024-02-05T07:30:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.542440066000001, 2024-02-05T07:30:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15757099199999999, 2024-02-05T07:30:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:30:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:30:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 26458 2024-02-05T07:30:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.479034+0000", 2024-02-05T07:30:55.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.479034+0000", 2024-02-05T07:30:55.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.479035+0000", 2024-02-05T07:30:55.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.088e-06 2024-02-05T07:30:55.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.479086+0000", 2024-02-05T07:30:55.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0192000000000003e-05 2024-02-05T07:30:55.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:55.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.479100+0000", 2024-02-05T07:30:55.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.43e-05 2024-02-05T07:30:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.479108+0000", 2024-02-05T07:30:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9580000000000007e-06 2024-02-05T07:30:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.479551+0000", 2024-02-05T07:30:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044335499999999999 2024-02-05T07:30:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.479667+0000", 2024-02-05T07:30:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011607000000000001 2024-02-05T07:30:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2024-02-05T07:30:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.479763+0000", 2024-02-05T07:30:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6088000000000006e-05 2024-02-05T07:30:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.480813+0000", 2024-02-05T07:30:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010493410000000001 2024-02-05T07:30:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481281+0000", 2024-02-05T07:30:55.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046874200000000002 2024-02-05T07:30:55.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.636537+0000", 2024-02-05T07:30:55.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15525549899999999 2024-02-05T07:30:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.636572+0000", 2024-02-05T07:30:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4944999999999998e-05 2024-02-05T07:30:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.636605+0000", 2024-02-05T07:30:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3414000000000002e-05 2024-02-05T07:30:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:26460 9.b 9:d51ce91c:::benchmark_data_smithi007_136200_object26459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-02-05T07:30:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:30:27.481413+0000", 2024-02-05T07:30:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.540061467000001, 2024-02-05T07:30:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15584008999999999, 2024-02-05T07:30:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:30:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:30:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 26460 2024-02-05T07:30:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481413+0000", 2024-02-05T07:30:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481413+0000", 2024-02-05T07:30:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481414+0000", 2024-02-05T07:30:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2500000000000001e-06 2024-02-05T07:30:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481478+0000", 2024-02-05T07:30:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3583000000000004e-05 2024-02-05T07:30:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481491+0000", 2024-02-05T07:30:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3502e-05 2024-02-05T07:30:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481499+0000", 2024-02-05T07:30:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8509999999999999e-06 2024-02-05T07:30:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481567+0000", 2024-02-05T07:30:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8180000000000001e-05 2024-02-05T07:30:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481706+0000", 2024-02-05T07:30:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013912500000000001 2024-02-05T07:30:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2024-02-05T07:30:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481863+0000", 2024-02-05T07:30:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156921 2024-02-05T07:30:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.483827+0000", 2024-02-05T07:30:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019638400000000001 2024-02-05T07:30:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.484723+0000", 2024-02-05T07:30:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089602400000000004 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.637200+0000", 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15247664599999999 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.637232+0000", 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1930000000000001e-05 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.637253+0000", 2024-02-05T07:30:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1237999999999999e-05 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:26463 9.b 9:dbdfe128:::benchmark_data_smithi007_136200_object26462:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:30:27.481800+0000", 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.539674497, 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15557917399999999, 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 26463 2024-02-05T07:30:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481800+0000", 2024-02-05T07:30:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481800+0000", 2024-02-05T07:30:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481801+0000", 2024-02-05T07:30:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3399999999999997e-07 2024-02-05T07:30:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481834+0000", 2024-02-05T07:30:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2777999999999997e-05 2024-02-05T07:30:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481846+0000", 2024-02-05T07:30:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2690999999999999e-05 2024-02-05T07:30:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.481854+0000", 2024-02-05T07:30:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0059999999999992e-06 2024-02-05T07:30:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.482796+0000", 2024-02-05T07:30:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00094196000000000004 2024-02-05T07:30:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.482925+0000", 2024-02-05T07:30:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012914600000000001 2024-02-05T07:30:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2024-02-05T07:30:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.483063+0000", 2024-02-05T07:30:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013784499999999999 2024-02-05T07:30:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:55.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.483918+0000", 2024-02-05T07:30:55.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085473000000000005 2024-02-05T07:30:55.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.484771+0000", 2024-02-05T07:30:55.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085347600000000004 2024-02-05T07:30:55.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.637334+0000", 2024-02-05T07:30:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15256214900000001 2024-02-05T07:30:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.637361+0000", 2024-02-05T07:30:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7212999999999998e-05 2024-02-05T07:30:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:27.637379+0000", 2024-02-05T07:30:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8246000000000001e-05 2024-02-05T07:30:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:29133 9.b 9:da365f33:::benchmark_data_smithi007_136200_object29132:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-02-05T07:30:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:30:29.751595+0000", 2024-02-05T07:30:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.269879523, 2024-02-05T07:30:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094752032, 2024-02-05T07:30:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:30:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:30:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:30:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:30:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:30:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 29133 2024-02-05T07:30:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:30:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:30:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:29.751595+0000", 2024-02-05T07:30:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:30:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:29.751595+0000", 2024-02-05T07:30:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:30:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:29.751597+0000", 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7010000000000001e-06 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:29.751665+0000", 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8935000000000003e-05 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:29.751678+0000", 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2921999999999999e-05 2024-02-05T07:30:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:29.751687+0000", 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6580000000000008e-06 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:29.751754+0000", 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7230000000000005e-05 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:29.751901+0000", 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014679999999999999 2024-02-05T07:30:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:29.842012+0000", 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090111202000000001 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:29.846179+0000", 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041664720000000001 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:29.846193+0000", 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4606e-05 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:29.846251+0000", 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7982000000000002e-05 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:29.846303+0000", 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2011000000000001e-05 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:29.846347+0000", 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3513000000000002e-05 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.156 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.156 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:30:55.156 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:30:55.156 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:30:55.185 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:55.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:55.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:55.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:55.440 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:55.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:55.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:55.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:55.604 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:55.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:55.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:55.706 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:55.732 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:55.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:55.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:55.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:55.858 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:55.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:55.936 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:30:55.935670+0000 min lat: 0.00140221 max lat: 5.65097 avg lat: 0.0126786 2024-02-05T07:30:55.936 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:30:55.936 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 40 16 50471 50455 78.8203 40 0.0238584 0.0126786 2024-02-05T07:30:55.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:56.004 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:56.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:56.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:56.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T07:30:56.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:56.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:56.134 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:56.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:56.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:56.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:56.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:56.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:56.410 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:56.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:56.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:56.535 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:56.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:56.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:56.662 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:56.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:56.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:56.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:56.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:56 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:30:56.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:56 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:30:56.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:56 smithi090 ceph-mon[33213]: pgmap v633: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 2.9 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 600 B/s rd, 26 MiB/s wr, 411 op/s; 13734/143535 objects degraded (9.568%); 10814/143535 objects misplaced (7.534%); 6.8 MiB/s, 108 objects/s recovering 2024-02-05T07:30:56.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:56 smithi090 ceph-mon[33213]: osdmap e373: 8 total, 8 up, 6 in 2024-02-05T07:30:56.817 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:56.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:56.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:56.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:56.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:56 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:30:56.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:56 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:30:56.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:56 smithi007 ceph-mon[29098]: pgmap v633: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 2.9 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 600 B/s rd, 26 MiB/s wr, 411 op/s; 13734/143535 objects degraded (9.568%); 10814/143535 objects misplaced (7.534%); 6.8 MiB/s, 108 objects/s recovering 2024-02-05T07:30:56.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:56 smithi007 ceph-mon[29098]: osdmap e373: 8 total, 8 up, 6 in 2024-02-05T07:30:56.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:56 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:30:56.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:56 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:30:56.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:56 smithi007 ceph-mon[37158]: pgmap v633: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 2.9 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 600 B/s rd, 26 MiB/s wr, 411 op/s; 13734/143535 objects degraded (9.568%); 10814/143535 objects misplaced (7.534%); 6.8 MiB/s, 108 objects/s recovering 2024-02-05T07:30:56.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:56 smithi007 ceph-mon[37158]: osdmap e373: 8 total, 8 up, 6 in 2024-02-05T07:30:56.937 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 41 16 50924 50908 77.5884 28.3125 0.0546197 0.0128813 2024-02-05T07:30:56.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:56.995 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:56.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:57.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:57.120 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:57.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:57.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:57.249 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:57.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:57.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:57.351 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:57.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:57.376 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:57.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:57.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:57.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:57.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:57.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:57.653 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:57.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:57.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:57.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:57.805 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:57.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:57.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:57.933 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:57.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:57.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:57.936 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 42 15 51495 51480 76.5923 35.75 0.0175383 0.013047 2024-02-05T07:30:58.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:58.057 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:58.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:58.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:58.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:58.188 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:58.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:58.289 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:58.338 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:58.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:58.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:30:58.463 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:58.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:58.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:30:58.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:58.591 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:58.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:58.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:58.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:30:58.716 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:58.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:58.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:30:58.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:58 smithi090 ceph-mon[33213]: pgmap v635: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 752 op/s; 13734/151566 objects degraded (9.061%); 10019/151566 objects misplaced (6.610%); 13 MiB/s, 214 objects/s recovering 2024-02-05T07:30:58.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:58.843 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:58.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:58.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:58.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:58 smithi007 ceph-mon[29098]: pgmap v635: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 752 op/s; 13734/151566 objects degraded (9.061%); 10019/151566 objects misplaced (6.610%); 13 MiB/s, 214 objects/s recovering 2024-02-05T07:30:58.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:58 smithi007 ceph-mon[37158]: pgmap v635: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 752 op/s; 13734/151566 objects degraded (9.061%); 10019/151566 objects misplaced (6.610%); 13 MiB/s, 214 objects/s recovering 2024-02-05T07:30:58.936 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 43 16 52046 52030 75.6105 34.375 0.0240079 0.013219 2024-02-05T07:30:58.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:58.989 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:58.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:59.091 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:59.120 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:59.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:59.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:30:59.247 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:59.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:59.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:30:59.349 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:30:59.372 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:59.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:59.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:30:59.474 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:30:59.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:59.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:30:59.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:30:59.658 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:30:59.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:59.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:30:59.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:30:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:59 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 13734/151566 objects degraded (9.061%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:30:59.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:30:59 smithi090 ceph-mon[33213]: osdmap e374: 8 total, 8 up, 6 in 2024-02-05T07:30:59.785 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:59.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:59.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:30:59.888 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:30:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:30:59.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:30:59.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:30:59.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:59 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 13734/151566 objects degraded (9.061%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:30:59.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:30:59 smithi007 ceph-mon[29098]: osdmap e374: 8 total, 8 up, 6 in 2024-02-05T07:30:59.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:59 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 13734/151566 objects degraded (9.061%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:30:59.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:30:59 smithi007 ceph-mon[37158]: osdmap e374: 8 total, 8 up, 6 in 2024-02-05T07:30:59.936 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 44 16 52565 52549 74.6293 32.4375 0.00322712 0.0133868 2024-02-05T07:31:00.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:00.041 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:00.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:00.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:00.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:00.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:00.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:00.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:00.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:00.335 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:00.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:00.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:00.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:00.489 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:00.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:00.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:00.613 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:00.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:00.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:00.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:00.740 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:00.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:00.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:00.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:31:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:31:00.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:00 smithi090 ceph-mon[33213]: pgmap v636: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.0 KiB/s rd, 40 MiB/s wr, 636 op/s; 13734/152106 objects degraded (9.029%); 10019/152106 objects misplaced (6.587%); 11 MiB/s, 176 objects/s recovering 2024-02-05T07:31:00.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:00 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:31:00.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:00 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:31:00.843 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:00.869 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:00.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:00.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:00.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:00 smithi007 ceph-mon[29098]: pgmap v636: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.0 KiB/s rd, 40 MiB/s wr, 636 op/s; 13734/152106 objects degraded (9.029%); 10019/152106 objects misplaced (6.587%); 11 MiB/s, 176 objects/s recovering 2024-02-05T07:31:00.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:00 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:31:00.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:00 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:31:00.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:00 smithi007 ceph-mon[37158]: pgmap v636: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.0 KiB/s rd, 40 MiB/s wr, 636 op/s; 13734/152106 objects degraded (9.029%); 10019/152106 objects misplaced (6.587%); 11 MiB/s, 176 objects/s recovering 2024-02-05T07:31:00.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:00 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:31:00.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:00 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:31:00.936 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 45 16 53078 53062 73.6834 32.0625 0.0457188 0.0135624 2024-02-05T07:31:00.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:01.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:01.023 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:01.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:01.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:01.175 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:01.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:01.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:01.311 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:01.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:01.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:01.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:01.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:01.539 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:01.580 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:01.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:01.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:01.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:01.722 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:01.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:01.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:01.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:01.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:01.872 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:01.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:01.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:01.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:01.936 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 46 16 53726 53710 72.962 40.5 0.0134171 0.0136976 2024-02-05T07:31:01.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:02.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:02.002 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:02.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:02.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:02.127 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:02.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:02.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:02.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:02.280 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:02.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:02.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:02.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:02.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:02.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:02.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:02.542 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:02.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:02.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:02.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:02.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:02.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:02.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:02 smithi090 ceph-mon[33213]: pgmap v638: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 687 op/s; 13734/159486 objects degraded (8.611%); 9128/159486 objects misplaced (5.723%); 13 MiB/s, 210 objects/s recovering 2024-02-05T07:31:02.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:02.815 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:02.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:02.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:02.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:02 smithi007 ceph-mon[29098]: pgmap v638: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 687 op/s; 13734/159486 objects degraded (8.611%); 9128/159486 objects misplaced (5.723%); 13 MiB/s, 210 objects/s recovering 2024-02-05T07:31:02.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:02 smithi007 ceph-mon[37158]: pgmap v638: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 687 op/s; 13734/159486 objects degraded (8.611%); 9128/159486 objects misplaced (5.723%); 13 MiB/s, 210 objects/s recovering 2024-02-05T07:31:02.937 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 47 16 54405 54389 72.3125 42.4375 0.0063891 0.0138209 2024-02-05T07:31:02.957 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:02.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:03.059 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:03.105 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:03.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:03.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:03.207 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:03.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:03.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:03.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:03.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:03.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:03.477 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:03.504 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:03.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:03.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:03.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:03.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:03.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:03.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:03.775 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:03.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:03.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:03.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:03.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:31:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:31:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:31:03.936 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 48 16 54964 54948 71.5339 34.9375 0.0320708 0.0139718 2024-02-05T07:31:04.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:04.041 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:04.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:04.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:04.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:04.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:04.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:04.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:04.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:04.465 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:04.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:04.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:04.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:04.591 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:04.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:04.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:04.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:04.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:04.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:04 smithi007 ceph-mon[29098]: pgmap v639: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 42 MiB/s wr, 665 op/s; 13734/159486 objects degraded (8.611%); 9128/159486 objects misplaced (5.723%); 13 MiB/s, 210 objects/s recovering 2024-02-05T07:31:04.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:04 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 13734/159486 objects degraded (8.611%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:04.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:04 smithi007 ceph-mon[37158]: pgmap v639: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 42 MiB/s wr, 665 op/s; 13734/159486 objects degraded (8.611%); 9128/159486 objects misplaced (5.723%); 13 MiB/s, 210 objects/s recovering 2024-02-05T07:31:04.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:04 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 13734/159486 objects degraded (8.611%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:04.843 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:04.867 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:04.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:04.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:04.936 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 49 16 55474 55458 70.7245 31.875 0.00624861 0.014126 2024-02-05T07:31:04.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:04.993 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:04.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:04.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:05.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:04 smithi090 ceph-mon[33213]: pgmap v639: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 42 MiB/s wr, 665 op/s; 13734/159486 objects degraded (8.611%); 9128/159486 objects misplaced (5.723%); 13 MiB/s, 210 objects/s recovering 2024-02-05T07:31:05.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:04 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 13734/159486 objects degraded (8.611%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:05.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:05.135 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:05.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:05.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:05.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:05.262 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:05.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:05.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:05.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:05.390 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:05.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:05.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:05.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:05.535 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:05.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:05.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:05.661 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:05.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:05.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:05.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:05.837 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:05.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:05.937 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 50 16 56042 56026 70.02 35.5 0.00769723 0.014273 2024-02-05T07:31:05.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:05.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:05.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:06.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:06.136 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:06.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:06.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:06.262 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:06.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:06.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:06.389 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:06.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:06.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:06.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:06.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:06.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:06.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:06.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:06.662 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:06.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:06.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:06.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:06.788 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:06.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:06.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:06.914 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:06.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:06.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:06 smithi007 ceph-mon[29098]: pgmap v640: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.0 KiB/s rd, 34 MiB/s wr, 548 op/s; 13734/160044 objects degraded (8.581%); 9128/160044 objects misplaced (5.703%); 11 MiB/s, 173 objects/s recovering 2024-02-05T07:31:06.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:06 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:31:06.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:06 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:31:06.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:06 smithi007 ceph-mon[37158]: pgmap v640: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.0 KiB/s rd, 34 MiB/s wr, 548 op/s; 13734/160044 objects degraded (8.581%); 9128/160044 objects misplaced (5.703%); 11 MiB/s, 173 objects/s recovering 2024-02-05T07:31:06.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:06 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:31:06.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:06 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:31:06.936 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 51 16 56607 56591 69.3395 35.3125 0.0208262 0.0144149 2024-02-05T07:31:07.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:07.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:06 smithi090 ceph-mon[33213]: pgmap v640: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.0 KiB/s rd, 34 MiB/s wr, 548 op/s; 13734/160044 objects degraded (8.581%); 9128/160044 objects misplaced (5.703%); 11 MiB/s, 173 objects/s recovering 2024-02-05T07:31:07.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:06 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:31:07.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:06 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:31:07.063 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:07.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:07.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:07.165 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:07.195 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:07.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:07.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:07.324 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:07.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:07.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:07.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:07.498 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:07.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:07.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:07.626 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:07.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:07.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:07.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:07.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:07.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:07 smithi090 ceph-mon[33213]: osdmap e375: 8 total, 8 up, 6 in 2024-02-05T07:31:07.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:31:07.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:07.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:07.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:07 smithi007 ceph-mon[29098]: osdmap e375: 8 total, 8 up, 6 in 2024-02-05T07:31:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:31:07.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:07 smithi007 ceph-mon[37158]: osdmap e375: 8 total, 8 up, 6 in 2024-02-05T07:31:07.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:31:07.937 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 52 16 57210 57194 68.7305 37.6875 0.00625343 0.014544 2024-02-05T07:31:07.982 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:08.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:08.028 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:08.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:08.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:08.154 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:08.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:08.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:08.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:08.280 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:08.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:08.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:08.410 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:08.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:08.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:08.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:08.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:08.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:08.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:08.703 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:08.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:08.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:08.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:08.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:08.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:08.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:08 smithi007 ceph-mon[29098]: pgmap v642: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 687 op/s; 13734/168591 objects degraded (8.146%); 8164/168591 objects misplaced (4.842%); 14 MiB/s, 231 objects/s recovering 2024-02-05T07:31:08.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:08 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:31:08.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:08 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:31:08.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:08 smithi007 ceph-mon[29098]: osdmap e376: 8 total, 8 up, 6 in 2024-02-05T07:31:08.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:08 smithi007 ceph-mon[37158]: pgmap v642: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 687 op/s; 13734/168591 objects degraded (8.146%); 8164/168591 objects misplaced (4.842%); 14 MiB/s, 231 objects/s recovering 2024-02-05T07:31:08.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:08 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:31:08.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:08 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:31:08.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:08 smithi007 ceph-mon[37158]: osdmap e376: 8 total, 8 up, 6 in 2024-02-05T07:31:08.937 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 53 16 57750 57734 68.0705 33.75 0.00431588 0.014666 2024-02-05T07:31:08.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:08.972 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:08.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:08.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:08 smithi090 ceph-mon[33213]: pgmap v642: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 687 op/s; 13734/168591 objects degraded (8.146%); 8164/168591 objects misplaced (4.842%); 14 MiB/s, 231 objects/s recovering 2024-02-05T07:31:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:08 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:31:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:08 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:31:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:08 smithi090 ceph-mon[33213]: osdmap e376: 8 total, 8 up, 6 in 2024-02-05T07:31:09.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:09.098 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:09.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:09.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:09.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:09.226 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:09.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:09.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:09.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:09.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:09.353 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:09.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:09.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:09.509 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:09.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:09.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:09.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:09.655 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:09.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:09.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:09.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:09.788 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:09.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:09.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:09.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:09 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 13734/168591 objects degraded (8.146%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:09.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:09 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 13734/168591 objects degraded (8.146%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:09.930 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:09.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:09.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:09.937 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 54 16 58367 58351 67.5241 38.5625 0.0523037 0.0148014 2024-02-05T07:31:10.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:09 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 13734/168591 objects degraded (8.146%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:10.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:10.079 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:10.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:10.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:10.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:10.225 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:10.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:10.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:10.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:10.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:10.352 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:10.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:10.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:10.478 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:10.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:10.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:10.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:10.606 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:10.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:10.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:10.734 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:10.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:10.758 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":630,"stamp":"2024-02-05T07:30:49.462460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2904865959,"num_objects":44512,"num_object_clones":0,"num_object_copies":133536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13734,"num_objects_misplaced":11736,"num_objects_unfound":0,"num_objects_dirty":44512,"num_whiteouts":0,"num_read":1163,"num_read_kb":932,"num_write":45068,"num_write_kb":2836580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":117049207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47846,"ondisk_log_size":47846,"up":504,"acting":493,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":9669688,"kb_used_data":6874084,"kb_used_omap":13,"kb_used_meta":2795506,"kb_avail":552801224,"statfs":{"total":575970213888,"available":566068453376,"internally_reserved":0,"allocated":7039062016,"data_stored":7032926153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":2862598880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,5,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":51118080,"num_objects":780,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13734,"num_objects_misplaced":11736,"num_objects_unfound":0,"num_objects_dirty":780,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":780,"num_write_kb":49920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1783,"num_bytes_recovered":116719708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.810192"},"pg_stats":[{"pgid":"9.1","version":"369'2716","reported_seq":2769,"reported_epoch":369,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-02-05T07:30:45.642615+0000","last_change":"2024-02-05T07:30:41.670846+0000","last_active":"2024-02-05T07:30:45.642615+0000","last_peered":"2024-02-05T07:30:45.642615+0000","last_clean":"2024-02-05T07:30:39.453308+0000","last_became_active":"2024-02-05T07:30:41.655732+0000","last_became_peered":"2024-02-05T07:30:41.655732+0000","last_unstale":"2024-02-05T07:30:45.642615+0000","last_undegraded":"2024-02-05T07:30:41.510704+0000","last_fullsized":"2024-02-05T07:30:41.466560+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2716,"log_dups_size":0,"ondisk_log_size":2716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177995776,"num_objects":2716,"num_object_clones":0,"num_object_copies":8148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2716,"num_write_kb":173824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6","objects":38},{"shards":"3,6,7","objects":2678}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":730,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.673894+0000","last_change":"2024-02-05T07:30:43.521908+0000","last_active":"2024-02-05T07:30:44.673894+0000","last_peered":"2024-02-05T07:30:44.673894+0000","last_clean":"2024-02-05T07:30:44.673894+0000","last_became_active":"2024-02-05T07:30:43.520874+0000","last_became_peered":"2024-02-05T07:30:43.520874+0000","last_unstale":"2024-02-05T07:30:44.673894+0000","last_undegraded":"2024-02-05T07:30:44.673894+0000","last_fullsized":"2024-02-05T07:30:44.673894+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:26:18.993912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.993912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"326'25","reported_seq":715,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.655739+0000","last_change":"2024-02-05T07:28:26.605879+0000","last_active":"2024-02-05T07:30:44.655739+0000","last_peered":"2024-02-05T07:30:44.655739+0000","last_clean":"2024-02-05T07:30:44.655739+0000","last_became_active":"2024-02-05T07:28:26.605082+0000","last_became_peered":"2024-02-05T07:28:26.605082+0000","last_unstale":"2024-02-05T07:30:44.655739+0000","last_undegraded":"2024-02-05T07:30:44.655739+0000","last_fullsized":"2024-02-05T07:30:44.655739+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'15","last_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:25:44.205475+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":222,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.673092+0000","last_change":"2024-02-05T07:27:16.716749+0000","last_active":"2024-02-05T07:30:44.673092+0000","last_peered":"2024-02-05T07:30:44.673092+0000","last_clean":"2024-02-05T07:30:44.673092+0000","last_became_active":"2024-02-05T07:27:16.716461+0000","last_became_peered":"2024-02-05T07:27:16.716461+0000","last_unstale":"2024-02-05T07:30:44.673092+0000","last_undegraded":"2024-02-05T07:30:44.673092+0000","last_fullsized":"2024-02-05T07:30:44.673092+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.258469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.258469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"365'11","reported_seq":827,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.677239+0000","last_change":"2024-02-05T07:30:40.506050+0000","last_active":"2024-02-05T07:30:45.677239+0000","last_peered":"2024-02-05T07:30:45.677239+0000","last_clean":"2024-02-05T07:30:45.677239+0000","last_became_active":"2024-02-05T07:30:40.484389+0000","last_became_peered":"2024-02-05T07:30:40.484389+0000","last_unstale":"2024-02-05T07:30:45.677239+0000","last_undegraded":"2024-02-05T07:30:45.677239+0000","last_fullsized":"2024-02-05T07:30:45.677239+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'10","last_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:30:38.946065+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:41.606908+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":628,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.054441+0000","last_change":"2024-02-05T07:25:43.273140+0000","last_active":"2024-02-05T07:30:45.054441+0000","last_peered":"2024-02-05T07:30:45.054441+0000","last_clean":"2024-02-05T07:30:45.054441+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:30:45.054441+0000","last_undegraded":"2024-02-05T07:30:45.054441+0000","last_fullsized":"2024-02-05T07:30:45.054441+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:25:43.273019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"370'2721","reported_seq":2773,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.882690+0000","last_change":"2024-02-05T07:30:00.506880+0000","last_active":"2024-02-05T07:30:45.882690+0000","last_peered":"2024-02-05T07:30:45.882690+0000","last_clean":"2024-02-05T07:30:45.882690+0000","last_became_active":"2024-02-05T07:30:00.506586+0000","last_became_peered":"2024-02-05T07:30:00.506586+0000","last_unstale":"2024-02-05T07:30:45.882690+0000","last_undegraded":"2024-02-05T07:30:45.882690+0000","last_fullsized":"2024-02-05T07:30:45.882690+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2721,"log_dups_size":0,"ondisk_log_size":2721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:21.843677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178323456,"num_objects":2721,"num_object_clones":0,"num_object_copies":8163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2721,"num_write_kb":174144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":651,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.676883+0000","last_change":"2024-02-05T07:29:48.249173+0000","last_active":"2024-02-05T07:30:45.676883+0000","last_peered":"2024-02-05T07:30:45.676883+0000","last_clean":"2024-02-05T07:30:45.676883+0000","last_became_active":"2024-02-05T07:28:26.593333+0000","last_became_peered":"2024-02-05T07:28:26.593333+0000","last_unstale":"2024-02-05T07:30:45.676883+0000","last_undegraded":"2024-02-05T07:30:45.676883+0000","last_fullsized":"2024-02-05T07:30:45.676883+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:29:48.249082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.249082+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:14.621210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"326'31","reported_seq":694,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.656215+0000","last_change":"2024-02-05T07:26:36.275060+0000","last_active":"2024-02-05T07:30:44.656215+0000","last_peered":"2024-02-05T07:30:44.656215+0000","last_clean":"2024-02-05T07:30:44.656215+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:30:44.656215+0000","last_undegraded":"2024-02-05T07:30:44.656215+0000","last_fullsized":"2024-02-05T07:30:44.656215+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:26:36.274939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.274939+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":498,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669822+0000","last_change":"2024-02-05T07:30:30.212483+0000","last_active":"2024-02-05T07:30:46.669822+0000","last_peered":"2024-02-05T07:30:46.669822+0000","last_clean":"2024-02-05T07:30:46.669822+0000","last_became_active":"2024-02-05T07:28:25.791005+0000","last_became_peered":"2024-02-05T07:28:25.791005+0000","last_unstale":"2024-02-05T07:30:46.669822+0000","last_undegraded":"2024-02-05T07:30:46.669822+0000","last_fullsized":"2024-02-05T07:30:46.669822+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:30.212401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.212401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:49.902414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":644,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.680024+0000","last_change":"2024-02-05T07:29:49.674581+0000","last_active":"2024-02-05T07:30:45.680024+0000","last_peered":"2024-02-05T07:30:45.680024+0000","last_clean":"2024-02-05T07:30:45.680024+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:30:45.680024+0000","last_undegraded":"2024-02-05T07:30:45.680024+0000","last_fullsized":"2024-02-05T07:30:45.680024+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.674493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.674493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:05.012130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":654,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.677827+0000","last_change":"2024-02-05T07:30:35.156491+0000","last_active":"2024-02-05T07:30:45.677827+0000","last_peered":"2024-02-05T07:30:45.677827+0000","last_clean":"2024-02-05T07:30:45.677827+0000","last_became_active":"2024-02-05T07:27:16.708888+0000","last_became_peered":"2024-02-05T07:27:16.708888+0000","last_unstale":"2024-02-05T07:30:45.677827+0000","last_undegraded":"2024-02-05T07:30:45.677827+0000","last_fullsized":"2024-02-05T07:30:45.677827+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:35.156384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:30:35.156384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:37.780463+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"369'2802","reported_seq":3583,"reported_epoch":369,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-05T07:30:45.640242+0000","last_change":"2024-02-05T07:30:41.691796+0000","last_active":"2024-02-05T07:30:45.640242+0000","last_peered":"2024-02-05T07:30:45.640242+0000","last_clean":"2024-02-05T07:30:39.443474+0000","last_became_active":"2024-02-05T07:30:41.642093+0000","last_became_peered":"2024-02-05T07:30:41.642093+0000","last_unstale":"2024-02-05T07:30:45.640242+0000","last_undegraded":"2024-02-05T07:30:45.640242+0000","last_fullsized":"2024-02-05T07:30:41.465298+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2802,"log_dups_size":0,"ondisk_log_size":2802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183631872,"num_objects":2802,"num_object_clones":0,"num_object_copies":8406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2403,"num_objects_unfound":0,"num_objects_dirty":2802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2802,"num_write_kb":179328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":365,"num_bytes_recovered":23920640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3","objects":399},{"shards":"1,3,7","objects":2403}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":622,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.678259+0000","last_change":"2024-02-05T07:28:46.611825+0000","last_active":"2024-02-05T07:30:45.678259+0000","last_peered":"2024-02-05T07:30:45.678259+0000","last_clean":"2024-02-05T07:30:45.678259+0000","last_became_active":"2024-02-05T07:28:19.312714+0000","last_became_peered":"2024-02-05T07:28:19.312714+0000","last_unstale":"2024-02-05T07:30:45.678259+0000","last_undegraded":"2024-02-05T07:30:45.678259+0000","last_fullsized":"2024-02-05T07:30:45.678259+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:46.611747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:46.611747+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"326'24","reported_seq":689,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.015796+0000","last_change":"2024-02-05T07:29:50.053697+0000","last_active":"2024-02-05T07:30:45.015796+0000","last_peered":"2024-02-05T07:30:45.015796+0000","last_clean":"2024-02-05T07:30:45.015796+0000","last_became_active":"2024-02-05T07:28:25.781007+0000","last_became_peered":"2024-02-05T07:28:25.781007+0000","last_unstale":"2024-02-05T07:30:45.015796+0000","last_undegraded":"2024-02-05T07:30:45.015796+0000","last_fullsized":"2024-02-05T07:30:45.015796+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:29:50.053583+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.053583+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:01.795266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":647,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.654665+0000","last_change":"2024-02-05T07:30:40.486435+0000","last_active":"2024-02-05T07:30:44.654665+0000","last_peered":"2024-02-05T07:30:44.654665+0000","last_clean":"2024-02-05T07:30:44.654665+0000","last_became_active":"2024-02-05T07:30:40.486134+0000","last_became_peered":"2024-02-05T07:30:40.486134+0000","last_unstale":"2024-02-05T07:30:44.654665+0000","last_undegraded":"2024-02-05T07:30:44.654665+0000","last_fullsized":"2024-02-05T07:30:44.654665+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:25:46.149431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":476,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.656086+0000","last_change":"2024-02-05T07:29:45.012958+0000","last_active":"2024-02-05T07:30:44.656086+0000","last_peered":"2024-02-05T07:30:44.656086+0000","last_clean":"2024-02-05T07:30:44.656086+0000","last_became_active":"2024-02-05T07:28:26.601713+0000","last_became_peered":"2024-02-05T07:28:26.601713+0000","last_unstale":"2024-02-05T07:30:44.656086+0000","last_undegraded":"2024-02-05T07:30:44.656086+0000","last_fullsized":"2024-02-05T07:30:44.656086+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.012821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:02.146624+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":231,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.670885+0000","last_change":"2024-02-05T07:30:40.499008+0000","last_active":"2024-02-05T07:30:44.670885+0000","last_peered":"2024-02-05T07:30:44.670885+0000","last_clean":"2024-02-05T07:30:44.670885+0000","last_became_active":"2024-02-05T07:30:40.498529+0000","last_became_peered":"2024-02-05T07:30:40.498529+0000","last_unstale":"2024-02-05T07:30:44.670885+0000","last_undegraded":"2024-02-05T07:30:44.670885+0000","last_fullsized":"2024-02-05T07:30:44.670885+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:47.885294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.885294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"369'2737","reported_seq":2787,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.595443+0000","last_change":"2024-02-05T07:30:00.512144+0000","last_active":"2024-02-05T07:30:45.595443+0000","last_peered":"2024-02-05T07:30:45.595443+0000","last_clean":"2024-02-05T07:30:45.595443+0000","last_became_active":"2024-02-05T07:30:00.511808+0000","last_became_peered":"2024-02-05T07:30:00.511808+0000","last_unstale":"2024-02-05T07:30:45.595443+0000","last_undegraded":"2024-02-05T07:30:45.595443+0000","last_fullsized":"2024-02-05T07:30:45.595443+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2737,"log_dups_size":0,"ondisk_log_size":2737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:21.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179372032,"num_objects":2737,"num_object_clones":0,"num_object_copies":8211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2737,"num_write_kb":175168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":626,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.676355+0000","last_change":"2024-02-05T07:28:27.218610+0000","last_active":"2024-02-05T07:30:45.676355+0000","last_peered":"2024-02-05T07:30:45.676355+0000","last_clean":"2024-02-05T07:30:45.676355+0000","last_became_active":"2024-02-05T07:28:26.588864+0000","last_became_peered":"2024-02-05T07:28:26.588864+0000","last_unstale":"2024-02-05T07:30:45.676355+0000","last_undegraded":"2024-02-05T07:30:45.676355+0000","last_fullsized":"2024-02-05T07:30:45.676355+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:27.218488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:27.218488+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"326'34","reported_seq":742,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.681377+0000","last_change":"2024-02-05T07:30:30.006087+0000","last_active":"2024-02-05T07:30:45.681377+0000","last_peered":"2024-02-05T07:30:45.681377+0000","last_clean":"2024-02-05T07:30:45.681377+0000","last_became_active":"2024-02-05T07:27:16.768454+0000","last_became_peered":"2024-02-05T07:27:16.768454+0000","last_unstale":"2024-02-05T07:30:45.681377+0000","last_undegraded":"2024-02-05T07:30:45.681377+0000","last_fullsized":"2024-02-05T07:30:45.681377+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'34","last_scrub_stamp":"2024-02-05T07:30:30.005988+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.005988+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:46.026624+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":236,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.669739+0000","last_change":"2024-02-05T07:30:40.508975+0000","last_active":"2024-02-05T07:30:44.669739+0000","last_peered":"2024-02-05T07:30:44.669739+0000","last_clean":"2024-02-05T07:30:44.669739+0000","last_became_active":"2024-02-05T07:30:40.508561+0000","last_became_peered":"2024-02-05T07:30:40.508561+0000","last_unstale":"2024-02-05T07:30:44.669739+0000","last_undegraded":"2024-02-05T07:30:44.669739+0000","last_fullsized":"2024-02-05T07:30:44.669739+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:30.982457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.982457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:38.021604+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"257'9","reported_seq":794,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.054381+0000","last_change":"2024-02-05T07:27:03.386101+0000","last_active":"2024-02-05T07:30:45.054381+0000","last_peered":"2024-02-05T07:30:45.054381+0000","last_clean":"2024-02-05T07:30:45.054381+0000","last_became_active":"2024-02-05T07:27:03.375195+0000","last_became_peered":"2024-02-05T07:27:03.375195+0000","last_unstale":"2024-02-05T07:30:45.054381+0000","last_undegraded":"2024-02-05T07:30:45.054381+0000","last_fullsized":"2024-02-05T07:30:45.054381+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:26:22.295732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:22.295732+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":250,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.729409+0000","last_change":"2024-02-05T07:30:37.998622+0000","last_active":"2024-02-05T07:30:45.729409+0000","last_peered":"2024-02-05T07:30:45.729409+0000","last_clean":"2024-02-05T07:30:45.729409+0000","last_became_active":"2024-02-05T07:27:03.367753+0000","last_became_peered":"2024-02-05T07:27:03.367753+0000","last_unstale":"2024-02-05T07:30:45.729409+0000","last_undegraded":"2024-02-05T07:30:45.729409+0000","last_fullsized":"2024-02-05T07:30:45.729409+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:37.998383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:30:37.998383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:32:01.160522+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"370'2866","reported_seq":2925,"reported_epoch":370,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-02-05T07:30:45.895125+0000","last_change":"2024-02-05T07:30:41.745452+0000","last_active":"2024-02-05T07:30:45.895125+0000","last_peered":"2024-02-05T07:30:45.895125+0000","last_clean":"2024-02-05T07:30:39.447605+0000","last_became_active":"2024-02-05T07:30:41.702210+0000","last_became_peered":"2024-02-05T07:30:41.702210+0000","last_unstale":"2024-02-05T07:30:45.895125+0000","last_undegraded":"2024-02-05T07:30:41.663668+0000","last_fullsized":"2024-02-05T07:30:41.461576+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2866,"log_dups_size":0,"ondisk_log_size":2866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187826176,"num_objects":2866,"num_object_clones":0,"num_object_copies":8598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2866,"num_write_kb":183424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4","objects":43},{"shards":"1,4,7","objects":2823}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":618,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.655838+0000","last_change":"2024-02-05T07:28:41.657737+0000","last_active":"2024-02-05T07:30:44.655838+0000","last_peered":"2024-02-05T07:30:44.655838+0000","last_clean":"2024-02-05T07:30:44.655838+0000","last_became_active":"2024-02-05T07:28:25.780676+0000","last_became_peered":"2024-02-05T07:28:25.780676+0000","last_unstale":"2024-02-05T07:30:44.655838+0000","last_undegraded":"2024-02-05T07:30:44.655838+0000","last_fullsized":"2024-02-05T07:30:44.655838+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:41.657622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:41.657622+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":651,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.654572+0000","last_change":"2024-02-05T07:30:40.489530+0000","last_active":"2024-02-05T07:30:44.654572+0000","last_peered":"2024-02-05T07:30:44.654572+0000","last_clean":"2024-02-05T07:30:44.654572+0000","last_became_active":"2024-02-05T07:30:40.489212+0000","last_became_peered":"2024-02-05T07:30:40.489212+0000","last_unstale":"2024-02-05T07:30:44.654572+0000","last_undegraded":"2024-02-05T07:30:44.654572+0000","last_fullsized":"2024-02-05T07:30:44.654572+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:51.211877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:51.211877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"326'30","reported_seq":290,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.669649+0000","last_change":"2024-02-05T07:30:40.543094+0000","last_active":"2024-02-05T07:30:44.669649+0000","last_peered":"2024-02-05T07:30:44.669649+0000","last_clean":"2024-02-05T07:30:44.669649+0000","last_became_active":"2024-02-05T07:30:40.496940+0000","last_became_peered":"2024-02-05T07:30:40.496940+0000","last_unstale":"2024-02-05T07:30:44.669649+0000","last_undegraded":"2024-02-05T07:30:44.669649+0000","last_fullsized":"2024-02-05T07:30:44.669649+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:28:45.029295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.029295+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":228,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.672584+0000","last_change":"2024-02-05T07:30:40.509232+0000","last_active":"2024-02-05T07:30:44.672584+0000","last_peered":"2024-02-05T07:30:44.672584+0000","last_clean":"2024-02-05T07:30:44.672584+0000","last_became_active":"2024-02-05T07:30:40.508607+0000","last_became_peered":"2024-02-05T07:30:40.508607+0000","last_unstale":"2024-02-05T07:30:44.672584+0000","last_undegraded":"2024-02-05T07:30:44.672584+0000","last_fullsized":"2024-02-05T07:30:44.672584+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:33.084016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:33.084016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":644,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.654684+0000","last_change":"2024-02-05T07:28:56.783280+0000","last_active":"2024-02-05T07:30:44.654684+0000","last_peered":"2024-02-05T07:30:44.654684+0000","last_clean":"2024-02-05T07:30:44.654684+0000","last_became_active":"2024-02-05T07:27:16.706666+0000","last_became_peered":"2024-02-05T07:27:16.706666+0000","last_unstale":"2024-02-05T07:30:44.654684+0000","last_undegraded":"2024-02-05T07:30:44.654684+0000","last_fullsized":"2024-02-05T07:30:44.654684+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:28:56.783168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"369'2703","reported_seq":3448,"reported_epoch":369,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-05T07:30:45.682817+0000","last_change":"2024-02-05T07:30:41.662688+0000","last_active":"2024-02-05T07:30:45.682817+0000","last_peered":"2024-02-05T07:30:45.682817+0000","last_clean":"2024-02-05T07:30:39.467841+0000","last_became_active":"2024-02-05T07:30:41.630359+0000","last_became_peered":"2024-02-05T07:30:41.630359+0000","last_unstale":"2024-02-05T07:30:45.682817+0000","last_undegraded":"2024-02-05T07:30:45.682817+0000","last_fullsized":"2024-02-05T07:30:41.471878+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2703,"log_dups_size":0,"ondisk_log_size":2703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177143808,"num_objects":2703,"num_object_clones":0,"num_object_copies":8109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2313,"num_objects_unfound":0,"num_objects_dirty":2703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2703,"num_write_kb":172992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":22675456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4","objects":390},{"shards":"3,4,7","objects":2313}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":888,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.674677+0000","last_change":"2024-02-05T07:30:43.524967+0000","last_active":"2024-02-05T07:30:44.674677+0000","last_peered":"2024-02-05T07:30:44.674677+0000","last_clean":"2024-02-05T07:30:44.674677+0000","last_became_active":"2024-02-05T07:30:43.524588+0000","last_became_peered":"2024-02-05T07:30:43.524588+0000","last_unstale":"2024-02-05T07:30:44.674677+0000","last_undegraded":"2024-02-05T07:30:44.674677+0000","last_fullsized":"2024-02-05T07:30:44.674677+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:39.042281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:39.042281+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"266'10","reported_seq":268,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.669820+0000","last_change":"2024-02-05T07:27:17.017592+0000","last_active":"2024-02-05T07:30:44.669820+0000","last_peered":"2024-02-05T07:30:44.669820+0000","last_clean":"2024-02-05T07:30:44.669820+0000","last_became_active":"2024-02-05T07:27:16.800207+0000","last_became_peered":"2024-02-05T07:27:16.800207+0000","last_unstale":"2024-02-05T07:30:44.669820+0000","last_undegraded":"2024-02-05T07:30:44.669820+0000","last_fullsized":"2024-02-05T07:30:44.669820+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:26:34.535447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:34.535447+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"326'35","reported_seq":715,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.654757+0000","last_change":"2024-02-05T07:29:45.972745+0000","last_active":"2024-02-05T07:30:44.654757+0000","last_peered":"2024-02-05T07:30:44.654757+0000","last_clean":"2024-02-05T07:30:44.654757+0000","last_became_active":"2024-02-05T07:28:26.598831+0000","last_became_peered":"2024-02-05T07:28:26.598831+0000","last_unstale":"2024-02-05T07:30:44.654757+0000","last_undegraded":"2024-02-05T07:30:44.654757+0000","last_fullsized":"2024-02-05T07:30:44.654757+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:29:45.972589+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.972589+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:11.921496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":404,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.679566+0000","last_change":"2024-02-05T07:29:41.587097+0000","last_active":"2024-02-05T07:30:45.679566+0000","last_peered":"2024-02-05T07:30:45.679566+0000","last_clean":"2024-02-05T07:30:45.679566+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:30:45.679566+0000","last_undegraded":"2024-02-05T07:30:45.679566+0000","last_fullsized":"2024-02-05T07:30:45.679566+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:41.587022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:41.587022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:03.158718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":510,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.670205+0000","last_change":"2024-02-05T07:30:40.489610+0000","last_active":"2024-02-05T07:30:46.670205+0000","last_peered":"2024-02-05T07:30:46.670205+0000","last_clean":"2024-02-05T07:30:46.670205+0000","last_became_active":"2024-02-05T07:30:40.489120+0000","last_became_peered":"2024-02-05T07:30:40.489120+0000","last_unstale":"2024-02-05T07:30:46.670205+0000","last_undegraded":"2024-02-05T07:30:46.670205+0000","last_fullsized":"2024-02-05T07:30:46.670205+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:45.254637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.254637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"370'2768","reported_seq":2820,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.882448+0000","last_change":"2024-02-05T07:30:00.509399+0000","last_active":"2024-02-05T07:30:45.882448+0000","last_peered":"2024-02-05T07:30:45.882448+0000","last_clean":"2024-02-05T07:30:45.882448+0000","last_became_active":"2024-02-05T07:30:00.509120+0000","last_became_peered":"2024-02-05T07:30:00.509120+0000","last_unstale":"2024-02-05T07:30:45.882448+0000","last_undegraded":"2024-02-05T07:30:45.882448+0000","last_fullsized":"2024-02-05T07:30:45.882448+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2768,"log_dups_size":0,"ondisk_log_size":2768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:24.514798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181403648,"num_objects":2768,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":177152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":487,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669931+0000","last_change":"2024-02-05T07:30:43.513209+0000","last_active":"2024-02-05T07:30:46.669931+0000","last_peered":"2024-02-05T07:30:46.669931+0000","last_clean":"2024-02-05T07:30:46.669931+0000","last_became_active":"2024-02-05T07:30:43.512898+0000","last_became_peered":"2024-02-05T07:30:43.512898+0000","last_unstale":"2024-02-05T07:30:46.669931+0000","last_undegraded":"2024-02-05T07:30:46.669931+0000","last_fullsized":"2024-02-05T07:30:46.669931+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:48.539780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.539780+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":265,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.670265+0000","last_change":"2024-02-05T07:29:39.912134+0000","last_active":"2024-02-05T07:30:46.670265+0000","last_peered":"2024-02-05T07:30:46.670265+0000","last_clean":"2024-02-05T07:30:46.670265+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:30:46.670265+0000","last_undegraded":"2024-02-05T07:30:46.670265+0000","last_fullsized":"2024-02-05T07:30:46.670265+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:29:39.912035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:39.912035+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:58.559919+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"326'25","reported_seq":709,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.688762+0000","last_change":"2024-02-05T07:30:36.908113+0000","last_active":"2024-02-05T07:30:45.688762+0000","last_peered":"2024-02-05T07:30:45.688762+0000","last_clean":"2024-02-05T07:30:45.688762+0000","last_became_active":"2024-02-05T07:27:03.361342+0000","last_became_peered":"2024-02-05T07:27:03.361342+0000","last_unstale":"2024-02-05T07:30:45.688762+0000","last_undegraded":"2024-02-05T07:30:45.688762+0000","last_fullsized":"2024-02-05T07:30:45.688762+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:30:36.907989+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.907989+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:32:04.256517+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":233,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.670256+0000","last_change":"2024-02-05T07:29:44.015577+0000","last_active":"2024-02-05T07:30:44.670256+0000","last_peered":"2024-02-05T07:30:44.670256+0000","last_clean":"2024-02-05T07:30:44.670256+0000","last_became_active":"2024-02-05T07:28:26.596096+0000","last_became_peered":"2024-02-05T07:28:26.596096+0000","last_unstale":"2024-02-05T07:30:44.670256+0000","last_undegraded":"2024-02-05T07:30:44.670256+0000","last_fullsized":"2024-02-05T07:30:44.670256+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:44.015468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:29:44.015468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:08.715933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":222,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.670015+0000","last_change":"2024-02-05T07:27:16.831397+0000","last_active":"2024-02-05T07:30:44.670015+0000","last_peered":"2024-02-05T07:30:44.670015+0000","last_clean":"2024-02-05T07:30:44.670015+0000","last_became_active":"2024-02-05T07:27:16.829863+0000","last_became_peered":"2024-02-05T07:27:16.829863+0000","last_unstale":"2024-02-05T07:30:44.670015+0000","last_undegraded":"2024-02-05T07:30:44.670015+0000","last_fullsized":"2024-02-05T07:30:44.670015+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:52.847946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:52.847946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"369'2746","reported_seq":2801,"reported_epoch":369,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-02-05T07:30:45.285730+0000","last_change":"2024-02-05T07:30:45.058294+0000","last_active":"2024-02-05T07:30:45.285730+0000","last_peered":"2024-02-05T07:30:45.285730+0000","last_clean":"2024-02-05T07:30:39.436762+0000","last_became_active":"2024-02-05T07:30:45.035966+0000","last_became_peered":"2024-02-05T07:30:45.035966+0000","last_unstale":"2024-02-05T07:30:45.285730+0000","last_undegraded":"2024-02-05T07:30:44.022643+0000","last_fullsized":"2024-02-05T07:30:41.464489+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2746,"log_dups_size":0,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179961856,"num_objects":2746,"num_object_clones":0,"num_object_copies":8238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":175744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1],"avail_no_missing":["4","1","7"],"object_location_counts":[{"shards":"1,4","objects":26},{"shards":"1,4,7","objects":2720}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":478,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669418+0000","last_change":"2024-02-05T07:28:49.521571+0000","last_active":"2024-02-05T07:30:46.669418+0000","last_peered":"2024-02-05T07:30:46.669418+0000","last_clean":"2024-02-05T07:30:46.669418+0000","last_became_active":"2024-02-05T07:28:26.624868+0000","last_became_peered":"2024-02-05T07:28:26.624868+0000","last_unstale":"2024-02-05T07:30:46.669418+0000","last_undegraded":"2024-02-05T07:30:46.669418+0000","last_fullsized":"2024-02-05T07:30:46.669418+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:49.521450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:49.521450+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":485,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669560+0000","last_change":"2024-02-05T07:30:35.140369+0000","last_active":"2024-02-05T07:30:46.669560+0000","last_peered":"2024-02-05T07:30:46.669560+0000","last_clean":"2024-02-05T07:30:46.669560+0000","last_became_active":"2024-02-05T07:28:26.589673+0000","last_became_peered":"2024-02-05T07:28:26.589673+0000","last_unstale":"2024-02-05T07:30:46.669560+0000","last_undegraded":"2024-02-05T07:30:46.669560+0000","last_fullsized":"2024-02-05T07:30:46.669560+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:35.140291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:30:35.140291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:42.480404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"326'29","reported_seq":278,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.669480+0000","last_change":"2024-02-05T07:30:40.516301+0000","last_active":"2024-02-05T07:30:45.669480+0000","last_peered":"2024-02-05T07:30:45.669480+0000","last_clean":"2024-02-05T07:30:45.669480+0000","last_became_active":"2024-02-05T07:30:40.481234+0000","last_became_peered":"2024-02-05T07:30:40.481234+0000","last_unstale":"2024-02-05T07:30:45.669480+0000","last_undegraded":"2024-02-05T07:30:45.669480+0000","last_fullsized":"2024-02-05T07:30:45.669480+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'29","last_scrub_stamp":"2024-02-05T07:30:37.106672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:37.106672+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:44.853993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":658,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.677859+0000","last_change":"2024-02-05T07:29:53.699733+0000","last_active":"2024-02-05T07:30:45.677859+0000","last_peered":"2024-02-05T07:30:45.677859+0000","last_clean":"2024-02-05T07:30:45.677859+0000","last_became_active":"2024-02-05T07:27:03.361028+0000","last_became_peered":"2024-02-05T07:27:03.361028+0000","last_unstale":"2024-02-05T07:30:45.677859+0000","last_undegraded":"2024-02-05T07:30:45.677859+0000","last_fullsized":"2024-02-05T07:30:45.677859+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.699651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.699651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:06.234119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":641,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.669122+0000","last_change":"2024-02-05T07:28:49.555346+0000","last_active":"2024-02-05T07:30:45.669122+0000","last_peered":"2024-02-05T07:30:45.669122+0000","last_clean":"2024-02-05T07:30:45.669122+0000","last_became_active":"2024-02-05T07:28:25.840850+0000","last_became_peered":"2024-02-05T07:28:25.840850+0000","last_unstale":"2024-02-05T07:30:45.669122+0000","last_undegraded":"2024-02-05T07:30:45.669122+0000","last_fullsized":"2024-02-05T07:30:45.669122+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:49.555018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:49.555018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"371'2821","reported_seq":2875,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:48.615253+0000","last_change":"2024-02-05T07:30:00.869708+0000","last_active":"2024-02-05T07:30:48.615253+0000","last_peered":"2024-02-05T07:30:48.615253+0000","last_clean":"2024-02-05T07:30:48.615253+0000","last_became_active":"2024-02-05T07:30:00.869295+0000","last_became_peered":"2024-02-05T07:30:00.869295+0000","last_unstale":"2024-02-05T07:30:48.615253+0000","last_undegraded":"2024-02-05T07:30:48.615253+0000","last_fullsized":"2024-02-05T07:30:48.615253+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2821,"log_dups_size":0,"ondisk_log_size":2821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:11.435954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184877056,"num_objects":2821,"num_object_clones":0,"num_object_copies":8463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2821,"num_write_kb":180544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":620,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.678351+0000","last_change":"2024-02-05T07:28:47.605953+0000","last_active":"2024-02-05T07:30:45.678351+0000","last_peered":"2024-02-05T07:30:45.678351+0000","last_clean":"2024-02-05T07:30:45.678351+0000","last_became_active":"2024-02-05T07:28:25.756613+0000","last_became_peered":"2024-02-05T07:28:25.756613+0000","last_unstale":"2024-02-05T07:30:45.678351+0000","last_undegraded":"2024-02-05T07:30:45.678351+0000","last_fullsized":"2024-02-05T07:30:45.678351+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:47.605803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.605803+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":241,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.681777+0000","last_change":"2024-02-05T07:27:03.370712+0000","last_active":"2024-02-05T07:30:45.681777+0000","last_peered":"2024-02-05T07:30:45.681777+0000","last_clean":"2024-02-05T07:30:45.681777+0000","last_became_active":"2024-02-05T07:27:03.370297+0000","last_became_peered":"2024-02-05T07:27:03.370297+0000","last_unstale":"2024-02-05T07:30:45.681777+0000","last_undegraded":"2024-02-05T07:30:45.681777+0000","last_fullsized":"2024-02-05T07:30:45.681777+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:32.652537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:32.652537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"326'24","reported_seq":451,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.680464+0000","last_change":"2024-02-05T07:29:36.583093+0000","last_active":"2024-02-05T07:30:45.680464+0000","last_peered":"2024-02-05T07:30:45.680464+0000","last_clean":"2024-02-05T07:30:45.680464+0000","last_became_active":"2024-02-05T07:24:34.209716+0000","last_became_peered":"2024-02-05T07:24:34.209716+0000","last_unstale":"2024-02-05T07:30:45.680464+0000","last_undegraded":"2024-02-05T07:30:45.680464+0000","last_fullsized":"2024-02-05T07:30:45.680464+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:29:36.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:36.583005+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":73,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.670027+0000","last_change":"2024-02-05T07:30:40.520395+0000","last_active":"2024-02-05T07:30:45.670027+0000","last_peered":"2024-02-05T07:30:45.670027+0000","last_clean":"2024-02-05T07:30:45.670027+0000","last_became_active":"2024-02-05T07:30:40.493770+0000","last_became_peered":"2024-02-05T07:30:40.493770+0000","last_unstale":"2024-02-05T07:30:45.670027+0000","last_undegraded":"2024-02-05T07:30:45.670027+0000","last_fullsized":"2024-02-05T07:30:45.670027+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:29:51.291146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.291146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:05.269025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":613,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.654855+0000","last_change":"2024-02-05T07:27:16.724562+0000","last_active":"2024-02-05T07:30:44.654855+0000","last_peered":"2024-02-05T07:30:44.654855+0000","last_clean":"2024-02-05T07:30:44.654855+0000","last_became_active":"2024-02-05T07:27:16.724280+0000","last_became_peered":"2024-02-05T07:27:16.724280+0000","last_unstale":"2024-02-05T07:30:44.654855+0000","last_undegraded":"2024-02-05T07:30:44.654855+0000","last_fullsized":"2024-02-05T07:30:44.654855+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:30.351655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:26:30.351655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":669,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.677650+0000","last_change":"2024-02-05T07:28:50.545548+0000","last_active":"2024-02-05T07:30:45.677650+0000","last_peered":"2024-02-05T07:30:45.677650+0000","last_clean":"2024-02-05T07:30:45.677650+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:30:45.677650+0000","last_undegraded":"2024-02-05T07:30:45.677650+0000","last_fullsized":"2024-02-05T07:30:45.677650+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:28:50.545437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:50.545437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"369'2811","reported_seq":2863,"reported_epoch":369,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-02-05T07:30:45.634205+0000","last_change":"2024-02-05T07:30:41.745424+0000","last_active":"2024-02-05T07:30:45.634205+0000","last_peered":"2024-02-05T07:30:45.634205+0000","last_clean":"2024-02-05T07:30:39.467673+0000","last_became_active":"2024-02-05T07:30:41.737377+0000","last_became_peered":"2024-02-05T07:30:41.737377+0000","last_unstale":"2024-02-05T07:30:45.634205+0000","last_undegraded":"2024-02-05T07:30:41.549610+0000","last_fullsized":"2024-02-05T07:30:41.467588+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2811,"log_dups_size":0,"ondisk_log_size":2811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184221696,"num_objects":2811,"num_object_clones":0,"num_object_copies":8433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2811,"num_write_kb":179904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3","objects":39},{"shards":"0,3,7","objects":2771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":416,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.676485+0000","last_change":"2024-02-05T07:30:40.474637+0000","last_active":"2024-02-05T07:30:45.676485+0000","last_peered":"2024-02-05T07:30:45.676485+0000","last_clean":"2024-02-05T07:30:45.676485+0000","last_became_active":"2024-02-05T07:30:40.474169+0000","last_became_peered":"2024-02-05T07:30:40.474169+0000","last_unstale":"2024-02-05T07:30:45.676485+0000","last_undegraded":"2024-02-05T07:30:45.676485+0000","last_fullsized":"2024-02-05T07:30:45.676485+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:30.221253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.221253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:31.131808+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"365'31","reported_seq":677,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669622+0000","last_change":"2024-02-05T07:30:40.539387+0000","last_active":"2024-02-05T07:30:46.669622+0000","last_peered":"2024-02-05T07:30:46.669622+0000","last_clean":"2024-02-05T07:30:46.669622+0000","last_became_active":"2024-02-05T07:30:40.508686+0000","last_became_peered":"2024-02-05T07:30:40.508686+0000","last_unstale":"2024-02-05T07:30:46.669622+0000","last_undegraded":"2024-02-05T07:30:46.669622+0000","last_fullsized":"2024-02-05T07:30:46.669622+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'30","last_scrub_stamp":"2024-02-05T07:30:37.201930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:37.201930+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:58.017999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":233,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.669990+0000","last_change":"2024-02-05T07:29:43.000854+0000","last_active":"2024-02-05T07:30:44.669990+0000","last_peered":"2024-02-05T07:30:44.669990+0000","last_clean":"2024-02-05T07:30:44.669990+0000","last_became_active":"2024-02-05T07:28:26.595640+0000","last_became_peered":"2024-02-05T07:28:26.595640+0000","last_unstale":"2024-02-05T07:30:44.669990+0000","last_undegraded":"2024-02-05T07:30:44.669990+0000","last_fullsized":"2024-02-05T07:30:44.669990+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:43.000778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:43.000778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:58.850387+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":328,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669640+0000","last_change":"2024-02-05T07:30:43.516579+0000","last_active":"2024-02-05T07:30:46.669640+0000","last_peered":"2024-02-05T07:30:46.669640+0000","last_clean":"2024-02-05T07:30:46.669640+0000","last_became_active":"2024-02-05T07:30:43.516031+0000","last_became_peered":"2024-02-05T07:30:43.516031+0000","last_unstale":"2024-02-05T07:30:46.669640+0000","last_undegraded":"2024-02-05T07:30:46.669640+0000","last_fullsized":"2024-02-05T07:30:46.669640+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:47.118901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.118901+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:49.103427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":222,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.672495+0000","last_change":"2024-02-05T07:27:16.751712+0000","last_active":"2024-02-05T07:30:44.672495+0000","last_peered":"2024-02-05T07:30:44.672495+0000","last_clean":"2024-02-05T07:30:44.672495+0000","last_became_active":"2024-02-05T07:27:16.751341+0000","last_became_peered":"2024-02-05T07:27:16.751341+0000","last_unstale":"2024-02-05T07:30:44.672495+0000","last_undegraded":"2024-02-05T07:30:44.672495+0000","last_fullsized":"2024-02-05T07:30:44.672495+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:36.753098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.753098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"370'2774","reported_seq":3538,"reported_epoch":370,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-05T07:30:45.902047+0000","last_change":"2024-02-05T07:30:41.639937+0000","last_active":"2024-02-05T07:30:45.902047+0000","last_peered":"2024-02-05T07:30:45.902047+0000","last_clean":"2024-02-05T07:30:39.445383+0000","last_became_active":"2024-02-05T07:30:41.615125+0000","last_became_peered":"2024-02-05T07:30:41.615125+0000","last_unstale":"2024-02-05T07:30:45.902047+0000","last_undegraded":"2024-02-05T07:30:45.902047+0000","last_fullsized":"2024-02-05T07:30:41.460160+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2774,"log_dups_size":0,"ondisk_log_size":2774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181796864,"num_objects":2774,"num_object_clones":0,"num_object_copies":8322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2361,"num_objects_unfound":0,"num_objects_dirty":2774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2774,"num_write_kb":177536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":355,"num_bytes_recovered":23265280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3","objects":413},{"shards":"1,3,7","objects":2361}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":634,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.654552+0000","last_change":"2024-02-05T07:30:43.510032+0000","last_active":"2024-02-05T07:30:44.654552+0000","last_peered":"2024-02-05T07:30:44.654552+0000","last_clean":"2024-02-05T07:30:44.654552+0000","last_became_active":"2024-02-05T07:30:43.509733+0000","last_became_peered":"2024-02-05T07:30:43.509733+0000","last_unstale":"2024-02-05T07:30:44.654552+0000","last_undegraded":"2024-02-05T07:30:44.654552+0000","last_fullsized":"2024-02-05T07:30:44.654552+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:46.132295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:46.132295+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:48.710919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":14,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":655,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.678983+0000","last_change":"2024-02-05T07:30:40.504250+0000","last_active":"2024-02-05T07:30:45.678983+0000","last_peered":"2024-02-05T07:30:45.678983+0000","last_clean":"2024-02-05T07:30:45.678983+0000","last_became_active":"2024-02-05T07:30:40.503816+0000","last_became_peered":"2024-02-05T07:30:40.503816+0000","last_unstale":"2024-02-05T07:30:45.678983+0000","last_undegraded":"2024-02-05T07:30:45.678983+0000","last_fullsized":"2024-02-05T07:30:45.678983+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:40.103572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:40.103572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"326'22","reported_seq":714,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.681573+0000","last_change":"2024-02-05T07:30:40.533683+0000","last_active":"2024-02-05T07:30:45.681573+0000","last_peered":"2024-02-05T07:30:45.681573+0000","last_clean":"2024-02-05T07:30:45.681573+0000","last_became_active":"2024-02-05T07:30:40.494460+0000","last_became_peered":"2024-02-05T07:30:40.494460+0000","last_unstale":"2024-02-05T07:30:45.681573+0000","last_undegraded":"2024-02-05T07:30:45.681573+0000","last_fullsized":"2024-02-05T07:30:45.681573+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'18","last_scrub_stamp":"2024-02-05T07:26:24.578301+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.578301+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":680,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.675566+0000","last_change":"2024-02-05T07:29:42.383585+0000","last_active":"2024-02-05T07:30:45.675566+0000","last_peered":"2024-02-05T07:30:45.675566+0000","last_clean":"2024-02-05T07:30:45.675566+0000","last_became_active":"2024-02-05T07:27:16.728869+0000","last_became_peered":"2024-02-05T07:27:16.728869+0000","last_unstale":"2024-02-05T07:30:45.675566+0000","last_undegraded":"2024-02-05T07:30:45.675566+0000","last_fullsized":"2024-02-05T07:30:45.675566+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:42.383469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:29:42.383469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:06.897824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":667,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.670165+0000","last_change":"2024-02-05T07:29:45.546235+0000","last_active":"2024-02-05T07:30:45.670165+0000","last_peered":"2024-02-05T07:30:45.670165+0000","last_clean":"2024-02-05T07:30:45.670165+0000","last_became_active":"2024-02-05T07:28:25.750373+0000","last_became_peered":"2024-02-05T07:28:25.750373+0000","last_unstale":"2024-02-05T07:30:45.670165+0000","last_undegraded":"2024-02-05T07:30:45.670165+0000","last_fullsized":"2024-02-05T07:30:45.670165+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:45.546152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.546152+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:52.093292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":522,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.670403+0000","last_change":"2024-02-05T07:30:40.489722+0000","last_active":"2024-02-05T07:30:46.670403+0000","last_peered":"2024-02-05T07:30:46.670403+0000","last_clean":"2024-02-05T07:30:46.670403+0000","last_became_active":"2024-02-05T07:30:40.489314+0000","last_became_peered":"2024-02-05T07:30:40.489314+0000","last_unstale":"2024-02-05T07:30:46.670403+0000","last_undegraded":"2024-02-05T07:30:46.670403+0000","last_fullsized":"2024-02-05T07:30:46.670403+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:38.191811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:29:38.191811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:48.579383+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"369'2654","reported_seq":3412,"reported_epoch":369,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-05T07:30:45.658757+0000","last_change":"2024-02-05T07:30:41.698959+0000","last_active":"2024-02-05T07:30:45.658757+0000","last_peered":"2024-02-05T07:30:45.658757+0000","last_clean":"2024-02-05T07:30:39.441404+0000","last_became_active":"2024-02-05T07:30:41.638290+0000","last_became_peered":"2024-02-05T07:30:41.638290+0000","last_unstale":"2024-02-05T07:30:45.658757+0000","last_undegraded":"2024-02-05T07:30:45.658757+0000","last_fullsized":"2024-02-05T07:30:41.466167+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2654,"log_dups_size":0,"ondisk_log_size":2654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173932544,"num_objects":2654,"num_object_clones":0,"num_object_copies":7962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2275,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2654,"num_write_kb":169856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":352,"num_bytes_recovered":23068672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3","objects":379},{"shards":"0,3,7","objects":2275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":638,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.675446+0000","last_change":"2024-02-05T07:29:55.246298+0000","last_active":"2024-02-05T07:30:45.675446+0000","last_peered":"2024-02-05T07:30:45.675446+0000","last_clean":"2024-02-05T07:30:45.675446+0000","last_became_active":"2024-02-05T07:28:14.305034+0000","last_became_peered":"2024-02-05T07:28:14.305034+0000","last_unstale":"2024-02-05T07:30:45.675446+0000","last_undegraded":"2024-02-05T07:30:45.675446+0000","last_fullsized":"2024-02-05T07:30:45.675446+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:29:55.246214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.246214+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.223711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":396,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.729819+0000","last_change":"2024-02-05T07:29:48.627198+0000","last_active":"2024-02-05T07:30:45.729819+0000","last_peered":"2024-02-05T07:30:45.729819+0000","last_clean":"2024-02-05T07:30:45.729819+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:30:45.729819+0000","last_undegraded":"2024-02-05T07:30:45.729819+0000","last_fullsized":"2024-02-05T07:30:45.729819+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.627107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.627107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:56.222054+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"326'44","reported_seq":539,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.676227+0000","last_change":"2024-02-05T07:30:36.009087+0000","last_active":"2024-02-05T07:30:45.676227+0000","last_peered":"2024-02-05T07:30:45.676227+0000","last_clean":"2024-02-05T07:30:45.676227+0000","last_became_active":"2024-02-05T07:27:16.708475+0000","last_became_peered":"2024-02-05T07:27:16.708475+0000","last_unstale":"2024-02-05T07:30:45.676227+0000","last_undegraded":"2024-02-05T07:30:45.676227+0000","last_fullsized":"2024-02-05T07:30:45.676227+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'44","last_scrub_stamp":"2024-02-05T07:30:36.008885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.008885+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:32:03.664756+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":39,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.677419+0000","last_change":"2024-02-05T07:30:32.964745+0000","last_active":"2024-02-05T07:30:45.677419+0000","last_peered":"2024-02-05T07:30:45.677419+0000","last_clean":"2024-02-05T07:30:45.677419+0000","last_became_active":"2024-02-05T07:30:30.524566+0000","last_became_peered":"2024-02-05T07:30:30.524566+0000","last_unstale":"2024-02-05T07:30:45.677419+0000","last_undegraded":"2024-02-05T07:30:45.677419+0000","last_fullsized":"2024-02-05T07:30:45.677419+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:32.964622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:32.964622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:54.761305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":566,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.688390+0000","last_change":"2024-02-05T07:28:55.477529+0000","last_active":"2024-02-05T07:30:45.688390+0000","last_peered":"2024-02-05T07:30:45.688390+0000","last_clean":"2024-02-05T07:30:45.688390+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:30:45.688390+0000","last_undegraded":"2024-02-05T07:30:45.688390+0000","last_fullsized":"2024-02-05T07:30:45.688390+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:28:55.477280+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":236,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.672112+0000","last_change":"2024-02-05T07:30:40.508412+0000","last_active":"2024-02-05T07:30:44.672112+0000","last_peered":"2024-02-05T07:30:44.672112+0000","last_clean":"2024-02-05T07:30:44.672112+0000","last_became_active":"2024-02-05T07:30:40.508071+0000","last_became_peered":"2024-02-05T07:30:40.508071+0000","last_unstale":"2024-02-05T07:30:44.672112+0000","last_undegraded":"2024-02-05T07:30:44.672112+0000","last_fullsized":"2024-02-05T07:30:44.672112+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:42.075445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:42.075445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.6","version":"370'2798","reported_seq":2850,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.770468+0000","last_change":"2024-02-05T07:30:00.512361+0000","last_active":"2024-02-05T07:30:45.770468+0000","last_peered":"2024-02-05T07:30:45.770468+0000","last_clean":"2024-02-05T07:30:45.770468+0000","last_became_active":"2024-02-05T07:30:00.511728+0000","last_became_peered":"2024-02-05T07:30:00.511728+0000","last_unstale":"2024-02-05T07:30:45.770468+0000","last_undegraded":"2024-02-05T07:30:45.770468+0000","last_fullsized":"2024-02-05T07:30:45.770468+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2798,"log_dups_size":0,"ondisk_log_size":2798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:24.957975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183369728,"num_objects":2798,"num_object_clones":0,"num_object_copies":8394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2798,"num_write_kb":179072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":548,"reported_epoch":369,"state":"active+undersized+remapped","last_fresh":"2024-02-05T07:30:45.016125+0000","last_change":"2024-02-05T07:30:44.063673+0000","last_active":"2024-02-05T07:30:45.016125+0000","last_peered":"2024-02-05T07:30:45.016125+0000","last_clean":"2024-02-05T07:30:38.596366+0000","last_became_active":"2024-02-05T07:30:44.063673+0000","last_became_peered":"2024-02-05T07:30:44.063673+0000","last_unstale":"2024-02-05T07:30:45.016125+0000","last_undegraded":"2024-02-05T07:30:45.016125+0000","last_fullsized":"2024-02-05T07:30:41.464135+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:40.616302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:40.616302+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3],"avail_no_missing":["4","2","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"326'15","reported_seq":439,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.678436+0000","last_change":"2024-02-05T07:29:51.665907+0000","last_active":"2024-02-05T07:30:45.678436+0000","last_peered":"2024-02-05T07:30:45.678436+0000","last_clean":"2024-02-05T07:30:45.678436+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:30:45.678436+0000","last_undegraded":"2024-02-05T07:30:45.678436+0000","last_fullsized":"2024-02-05T07:30:45.678436+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:29:51.665808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.665808+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:13.278749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":233,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.675168+0000","last_change":"2024-02-05T07:30:38.998491+0000","last_active":"2024-02-05T07:30:44.675168+0000","last_peered":"2024-02-05T07:30:44.675168+0000","last_clean":"2024-02-05T07:30:44.675168+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:30:44.675168+0000","last_undegraded":"2024-02-05T07:30:44.675168+0000","last_fullsized":"2024-02-05T07:30:44.675168+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:38.995586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:38.995586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:42.178681+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":655,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.676501+0000","last_change":"2024-02-05T07:28:41.103849+0000","last_active":"2024-02-05T07:30:45.676501+0000","last_peered":"2024-02-05T07:30:45.676501+0000","last_clean":"2024-02-05T07:30:45.676501+0000","last_became_active":"2024-02-05T07:27:16.708205+0000","last_became_peered":"2024-02-05T07:27:16.708205+0000","last_unstale":"2024-02-05T07:30:45.676501+0000","last_undegraded":"2024-02-05T07:30:45.676501+0000","last_fullsized":"2024-02-05T07:30:45.676501+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:41.103739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:41.103739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":24,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.675681+0000","last_change":"2024-02-05T07:30:40.488052+0000","last_active":"2024-02-05T07:30:45.675681+0000","last_peered":"2024-02-05T07:30:45.675681+0000","last_clean":"2024-02-05T07:30:45.675681+0000","last_became_active":"2024-02-05T07:30:40.486401+0000","last_became_peered":"2024-02-05T07:30:40.486401+0000","last_unstale":"2024-02-05T07:30:45.675681+0000","last_undegraded":"2024-02-05T07:30:45.675681+0000","last_fullsized":"2024-02-05T07:30:45.675681+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:30.076226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:30.076226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"369'2781","reported_seq":2832,"reported_epoch":369,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-02-05T07:30:45.575894+0000","last_change":"2024-02-05T07:30:41.745634+0000","last_active":"2024-02-05T07:30:45.575894+0000","last_peered":"2024-02-05T07:30:45.575894+0000","last_clean":"2024-02-05T07:30:39.445489+0000","last_became_active":"2024-02-05T07:30:41.737849+0000","last_became_peered":"2024-02-05T07:30:41.737849+0000","last_unstale":"2024-02-05T07:30:45.575894+0000","last_undegraded":"2024-02-05T07:30:41.540462+0000","last_fullsized":"2024-02-05T07:30:41.471627+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2781,"log_dups_size":0,"ondisk_log_size":2781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182255616,"num_objects":2781,"num_object_clones":0,"num_object_copies":8343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2781,"num_write_kb":177984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":40},{"shards":"2,3,7","objects":2741}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":479,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669509+0000","last_change":"2024-02-05T07:28:47.560599+0000","last_active":"2024-02-05T07:30:46.669509+0000","last_peered":"2024-02-05T07:30:46.669509+0000","last_clean":"2024-02-05T07:30:46.669509+0000","last_became_active":"2024-02-05T07:28:26.596711+0000","last_became_peered":"2024-02-05T07:28:26.596711+0000","last_unstale":"2024-02-05T07:30:46.669509+0000","last_undegraded":"2024-02-05T07:30:46.669509+0000","last_fullsized":"2024-02-05T07:30:46.669509+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:47.560475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.560475+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"326'35","reported_seq":560,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669651+0000","last_change":"2024-02-05T07:30:40.520826+0000","last_active":"2024-02-05T07:30:46.669651+0000","last_peered":"2024-02-05T07:30:46.669651+0000","last_clean":"2024-02-05T07:30:46.669651+0000","last_became_active":"2024-02-05T07:30:40.482753+0000","last_became_peered":"2024-02-05T07:30:40.482753+0000","last_unstale":"2024-02-05T07:30:46.669651+0000","last_undegraded":"2024-02-05T07:30:46.669651+0000","last_fullsized":"2024-02-05T07:30:46.669651+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:29:45.981520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.981520+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:08.120152+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":533,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.670314+0000","last_change":"2024-02-05T07:30:40.508017+0000","last_active":"2024-02-05T07:30:46.670314+0000","last_peered":"2024-02-05T07:30:46.670314+0000","last_clean":"2024-02-05T07:30:46.670314+0000","last_became_active":"2024-02-05T07:30:40.507705+0000","last_became_peered":"2024-02-05T07:30:40.507705+0000","last_unstale":"2024-02-05T07:30:46.670314+0000","last_undegraded":"2024-02-05T07:30:46.670314+0000","last_fullsized":"2024-02-05T07:30:46.670314+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:39.169995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:39.169995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:51.481267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":655,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.677652+0000","last_change":"2024-02-05T07:30:40.480847+0000","last_active":"2024-02-05T07:30:45.677652+0000","last_peered":"2024-02-05T07:30:45.677652+0000","last_clean":"2024-02-05T07:30:45.677652+0000","last_became_active":"2024-02-05T07:30:40.479288+0000","last_became_peered":"2024-02-05T07:30:40.479288+0000","last_unstale":"2024-02-05T07:30:45.677652+0000","last_undegraded":"2024-02-05T07:30:45.677652+0000","last_fullsized":"2024-02-05T07:30:45.677652+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:55.173628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:55.173628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":437,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.677225+0000","last_change":"2024-02-05T07:30:40.497142+0000","last_active":"2024-02-05T07:30:45.677225+0000","last_peered":"2024-02-05T07:30:45.677225+0000","last_clean":"2024-02-05T07:30:45.677225+0000","last_became_active":"2024-02-05T07:30:40.496822+0000","last_became_peered":"2024-02-05T07:30:40.496822+0000","last_unstale":"2024-02-05T07:30:45.677225+0000","last_undegraded":"2024-02-05T07:30:45.677225+0000","last_fullsized":"2024-02-05T07:30:45.677225+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.262537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:54.218130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.4","version":"370'2842","reported_seq":2894,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.777064+0000","last_change":"2024-02-05T07:30:01.030823+0000","last_active":"2024-02-05T07:30:45.777064+0000","last_peered":"2024-02-05T07:30:45.777064+0000","last_clean":"2024-02-05T07:30:45.777064+0000","last_became_active":"2024-02-05T07:30:01.030502+0000","last_became_peered":"2024-02-05T07:30:01.030502+0000","last_unstale":"2024-02-05T07:30:45.777064+0000","last_undegraded":"2024-02-05T07:30:45.777064+0000","last_fullsized":"2024-02-05T07:30:45.777064+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2842,"log_dups_size":0,"ondisk_log_size":2842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:05.204271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186253312,"num_objects":2842,"num_object_clones":0,"num_object_copies":8526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2842,"num_write_kb":181888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":885,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.670742+0000","last_change":"2024-02-05T07:28:40.182489+0000","last_active":"2024-02-05T07:30:44.670742+0000","last_peered":"2024-02-05T07:30:44.670742+0000","last_clean":"2024-02-05T07:30:44.670742+0000","last_became_active":"2024-02-05T07:28:26.597412+0000","last_became_peered":"2024-02-05T07:28:26.597412+0000","last_unstale":"2024-02-05T07:30:44.670742+0000","last_undegraded":"2024-02-05T07:30:44.670742+0000","last_fullsized":"2024-02-05T07:30:44.670742+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:40.181881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:40.181881+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":651,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.677034+0000","last_change":"2024-02-05T07:29:39.290437+0000","last_active":"2024-02-05T07:30:45.677034+0000","last_peered":"2024-02-05T07:30:45.677034+0000","last_clean":"2024-02-05T07:30:45.677034+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:30:45.677034+0000","last_undegraded":"2024-02-05T07:30:45.677034+0000","last_fullsized":"2024-02-05T07:30:45.677034+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:39.290320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:39.290320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"325'18","reported_seq":737,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.654939+0000","last_change":"2024-02-05T07:29:49.022612+0000","last_active":"2024-02-05T07:30:44.654939+0000","last_peered":"2024-02-05T07:30:44.654939+0000","last_clean":"2024-02-05T07:30:44.654939+0000","last_became_active":"2024-02-05T07:27:16.706871+0000","last_became_peered":"2024-02-05T07:27:16.706871+0000","last_unstale":"2024-02-05T07:30:44.654939+0000","last_undegraded":"2024-02-05T07:30:44.654939+0000","last_fullsized":"2024-02-05T07:30:44.654939+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'18","last_scrub_stamp":"2024-02-05T07:29:49.021772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.021772+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:49.051858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.053787+0000","last_change":"2024-02-05T07:30:40.519397+0000","last_active":"2024-02-05T07:30:45.053787+0000","last_peered":"2024-02-05T07:30:45.053787+0000","last_clean":"2024-02-05T07:30:45.053787+0000","last_became_active":"2024-02-05T07:30:40.518949+0000","last_became_peered":"2024-02-05T07:30:40.518949+0000","last_unstale":"2024-02-05T07:30:45.053787+0000","last_undegraded":"2024-02-05T07:30:45.053787+0000","last_fullsized":"2024-02-05T07:30:45.053787+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:24.102597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.102597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":236,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.672959+0000","last_change":"2024-02-05T07:30:40.486334+0000","last_active":"2024-02-05T07:30:44.672959+0000","last_peered":"2024-02-05T07:30:44.672959+0000","last_clean":"2024-02-05T07:30:44.672959+0000","last_became_active":"2024-02-05T07:30:40.485991+0000","last_became_peered":"2024-02-05T07:30:40.485991+0000","last_unstale":"2024-02-05T07:30:44.672959+0000","last_undegraded":"2024-02-05T07:30:44.672959+0000","last_fullsized":"2024-02-05T07:30:44.672959+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:48.088229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.088229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"370'2774","reported_seq":3550,"reported_epoch":370,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-05T07:30:45.919473+0000","last_change":"2024-02-05T07:30:41.811091+0000","last_active":"2024-02-05T07:30:45.919473+0000","last_peered":"2024-02-05T07:30:45.919473+0000","last_clean":"2024-02-05T07:30:39.433649+0000","last_became_active":"2024-02-05T07:30:41.718293+0000","last_became_peered":"2024-02-05T07:30:41.718293+0000","last_unstale":"2024-02-05T07:30:45.919473+0000","last_undegraded":"2024-02-05T07:30:45.919473+0000","last_fullsized":"2024-02-05T07:30:41.470038+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":2774,"log_dups_size":0,"ondisk_log_size":2774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181796864,"num_objects":2774,"num_object_clones":0,"num_object_copies":8322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2384,"num_objects_unfound":0,"num_objects_dirty":2774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2774,"num_write_kb":177536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":23658496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0],"avail_no_missing":["1","0","7"],"object_location_counts":[{"shards":"0,1","objects":390},{"shards":"0,1,7","objects":2384}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":470,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.655674+0000","last_change":"2024-02-05T07:30:43.529334+0000","last_active":"2024-02-05T07:30:44.655674+0000","last_peered":"2024-02-05T07:30:44.655674+0000","last_clean":"2024-02-05T07:30:44.655674+0000","last_became_active":"2024-02-05T07:30:43.529109+0000","last_became_peered":"2024-02-05T07:30:43.529109+0000","last_unstale":"2024-02-05T07:30:44.655674+0000","last_undegraded":"2024-02-05T07:30:44.655674+0000","last_fullsized":"2024-02-05T07:30:44.655674+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.145047+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:48.710935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"365'12","reported_seq":646,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669885+0000","last_change":"2024-02-05T07:30:40.519998+0000","last_active":"2024-02-05T07:30:46.669885+0000","last_peered":"2024-02-05T07:30:46.669885+0000","last_clean":"2024-02-05T07:30:46.669885+0000","last_became_active":"2024-02-05T07:30:40.500113+0000","last_became_peered":"2024-02-05T07:30:40.500113+0000","last_unstale":"2024-02-05T07:30:46.669885+0000","last_undegraded":"2024-02-05T07:30:46.669885+0000","last_fullsized":"2024-02-05T07:30:46.669885+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'11","last_scrub_stamp":"2024-02-05T07:28:50.147503+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:28:50.147503+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"326'27","reported_seq":280,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669163+0000","last_change":"2024-02-05T07:30:40.526838+0000","last_active":"2024-02-05T07:30:46.669163+0000","last_peered":"2024-02-05T07:30:46.669163+0000","last_clean":"2024-02-05T07:30:46.669163+0000","last_became_active":"2024-02-05T07:30:40.497982+0000","last_became_peered":"2024-02-05T07:30:40.497982+0000","last_unstale":"2024-02-05T07:30:46.669163+0000","last_undegraded":"2024-02-05T07:30:46.669163+0000","last_fullsized":"2024-02-05T07:30:46.669163+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:28:49.179472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:28:49.179472+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":35,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.656046+0000","last_change":"2024-02-05T07:30:40.530544+0000","last_active":"2024-02-05T07:30:44.656046+0000","last_peered":"2024-02-05T07:30:44.656046+0000","last_clean":"2024-02-05T07:30:44.656046+0000","last_became_active":"2024-02-05T07:30:40.503903+0000","last_became_peered":"2024-02-05T07:30:40.503903+0000","last_unstale":"2024-02-05T07:30:44.656046+0000","last_undegraded":"2024-02-05T07:30:44.656046+0000","last_fullsized":"2024-02-05T07:30:44.656046+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:26:37.022129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:37.022129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":315,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669437+0000","last_change":"2024-02-05T07:30:40.130268+0000","last_active":"2024-02-05T07:30:46.669437+0000","last_peered":"2024-02-05T07:30:46.669437+0000","last_clean":"2024-02-05T07:30:46.669437+0000","last_became_active":"2024-02-05T07:28:26.589671+0000","last_became_peered":"2024-02-05T07:28:26.589671+0000","last_unstale":"2024-02-05T07:30:46.669437+0000","last_undegraded":"2024-02-05T07:30:46.669437+0000","last_fullsized":"2024-02-05T07:30:46.669437+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.130170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:47.049334+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":488,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.688107+0000","last_change":"2024-02-05T07:30:43.523848+0000","last_active":"2024-02-05T07:30:45.688107+0000","last_peered":"2024-02-05T07:30:45.688107+0000","last_clean":"2024-02-05T07:30:45.688107+0000","last_became_active":"2024-02-05T07:30:43.523108+0000","last_became_peered":"2024-02-05T07:30:43.523108+0000","last_unstale":"2024-02-05T07:30:45.688107+0000","last_undegraded":"2024-02-05T07:30:45.688107+0000","last_fullsized":"2024-02-05T07:30:45.688107+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:48.594280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.594280+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"326'12","reported_seq":253,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.672294+0000","last_change":"2024-02-05T07:30:40.526400+0000","last_active":"2024-02-05T07:30:44.672294+0000","last_peered":"2024-02-05T07:30:44.672294+0000","last_clean":"2024-02-05T07:30:44.672294+0000","last_became_active":"2024-02-05T07:30:40.496980+0000","last_became_peered":"2024-02-05T07:30:40.496980+0000","last_unstale":"2024-02-05T07:30:44.672294+0000","last_undegraded":"2024-02-05T07:30:44.672294+0000","last_fullsized":"2024-02-05T07:30:44.672294+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-02-05T07:26:36.068507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.068507+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":451,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669379+0000","last_change":"2024-02-05T07:28:45.572051+0000","last_active":"2024-02-05T07:30:46.669379+0000","last_peered":"2024-02-05T07:30:46.669379+0000","last_clean":"2024-02-05T07:30:46.669379+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:30:46.669379+0000","last_undegraded":"2024-02-05T07:30:46.669379+0000","last_fullsized":"2024-02-05T07:30:46.669379+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.571917+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":233,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.671841+0000","last_change":"2024-02-05T07:30:27.965535+0000","last_active":"2024-02-05T07:30:44.671841+0000","last_peered":"2024-02-05T07:30:44.671841+0000","last_clean":"2024-02-05T07:30:44.671841+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:30:44.671841+0000","last_undegraded":"2024-02-05T07:30:44.671841+0000","last_fullsized":"2024-02-05T07:30:44.671841+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:27.965449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:27.965449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:47.638625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":509,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.670599+0000","last_change":"2024-02-05T07:30:40.509687+0000","last_active":"2024-02-05T07:30:46.670599+0000","last_peered":"2024-02-05T07:30:46.670599+0000","last_clean":"2024-02-05T07:30:46.670599+0000","last_became_active":"2024-02-05T07:30:40.509216+0000","last_became_peered":"2024-02-05T07:30:40.509216+0000","last_unstale":"2024-02-05T07:30:46.670599+0000","last_undegraded":"2024-02-05T07:30:46.670599+0000","last_fullsized":"2024-02-05T07:30:46.670599+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:55.256094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:28:55.256094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":622,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.668679+0000","last_change":"2024-02-05T07:28:45.702694+0000","last_active":"2024-02-05T07:30:45.668679+0000","last_peered":"2024-02-05T07:30:45.668679+0000","last_clean":"2024-02-05T07:30:45.668679+0000","last_became_active":"2024-02-05T07:28:19.311737+0000","last_became_peered":"2024-02-05T07:28:19.311737+0000","last_unstale":"2024-02-05T07:30:45.668679+0000","last_undegraded":"2024-02-05T07:30:45.668679+0000","last_fullsized":"2024-02-05T07:30:45.668679+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:45.702565+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.702565+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.702565+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"325'15","reported_seq":257,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.672025+0000","last_change":"2024-02-05T07:30:40.524523+0000","last_active":"2024-02-05T07:30:44.672025+0000","last_peered":"2024-02-05T07:30:44.672025+0000","last_clean":"2024-02-05T07:30:44.672025+0000","last_became_active":"2024-02-05T07:30:40.485837+0000","last_became_peered":"2024-02-05T07:30:40.485837+0000","last_unstale":"2024-02-05T07:30:44.672025+0000","last_undegraded":"2024-02-05T07:30:44.672025+0000","last_fullsized":"2024-02-05T07:30:44.672025+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:26:28.037945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:28.037945+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":526,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.654930+0000","last_change":"2024-02-05T07:30:40.498263+0000","last_active":"2024-02-05T07:30:44.654930+0000","last_peered":"2024-02-05T07:30:44.654930+0000","last_clean":"2024-02-05T07:30:44.654930+0000","last_became_active":"2024-02-05T07:30:40.497956+0000","last_became_peered":"2024-02-05T07:30:40.497956+0000","last_unstale":"2024-02-05T07:30:44.654930+0000","last_undegraded":"2024-02-05T07:30:44.654930+0000","last_fullsized":"2024-02-05T07:30:44.654930+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:33.183427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:30:33.183427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:36.436239+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":228,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.672405+0000","last_change":"2024-02-05T07:30:40.496550+0000","last_active":"2024-02-05T07:30:44.672405+0000","last_peered":"2024-02-05T07:30:44.672405+0000","last_clean":"2024-02-05T07:30:44.672405+0000","last_became_active":"2024-02-05T07:30:40.496076+0000","last_became_peered":"2024-02-05T07:30:40.496076+0000","last_unstale":"2024-02-05T07:30:44.672405+0000","last_undegraded":"2024-02-05T07:30:44.672405+0000","last_fullsized":"2024-02-05T07:30:44.672405+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:59.450191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:59.450191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":645,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.676384+0000","last_change":"2024-02-05T07:30:40.488677+0000","last_active":"2024-02-05T07:30:45.676384+0000","last_peered":"2024-02-05T07:30:45.676384+0000","last_clean":"2024-02-05T07:30:45.676384+0000","last_became_active":"2024-02-05T07:30:40.487455+0000","last_became_peered":"2024-02-05T07:30:40.487455+0000","last_unstale":"2024-02-05T07:30:45.676384+0000","last_undegraded":"2024-02-05T07:30:45.676384+0000","last_fullsized":"2024-02-05T07:30:45.676384+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:53.115096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:28:53.115096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":328,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.668979+0000","last_change":"2024-02-05T07:30:43.467206+0000","last_active":"2024-02-05T07:30:46.668979+0000","last_peered":"2024-02-05T07:30:46.668979+0000","last_clean":"2024-02-05T07:30:46.668979+0000","last_became_active":"2024-02-05T07:30:43.466854+0000","last_became_peered":"2024-02-05T07:30:43.466854+0000","last_unstale":"2024-02-05T07:30:46.668979+0000","last_undegraded":"2024-02-05T07:30:46.668979+0000","last_fullsized":"2024-02-05T07:30:46.668979+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:43.069567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:43.069567+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:49.103397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"326'19","reported_seq":585,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.670383+0000","last_change":"2024-02-05T07:30:40.503823+0000","last_active":"2024-02-05T07:30:44.670383+0000","last_peered":"2024-02-05T07:30:44.670383+0000","last_clean":"2024-02-05T07:30:44.670383+0000","last_became_active":"2024-02-05T07:30:40.483700+0000","last_became_peered":"2024-02-05T07:30:40.483700+0000","last_unstale":"2024-02-05T07:30:44.670383+0000","last_undegraded":"2024-02-05T07:30:44.670383+0000","last_fullsized":"2024-02-05T07:30:44.670383+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:30:35.138361+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:30:35.138361+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:52.186504+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":399,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.669825+0000","last_change":"2024-02-05T07:30:40.476394+0000","last_active":"2024-02-05T07:30:44.669825+0000","last_peered":"2024-02-05T07:30:44.669825+0000","last_clean":"2024-02-05T07:30:44.669825+0000","last_became_active":"2024-02-05T07:30:40.476058+0000","last_became_peered":"2024-02-05T07:30:40.476058+0000","last_unstale":"2024-02-05T07:30:44.669825+0000","last_undegraded":"2024-02-05T07:30:44.669825+0000","last_fullsized":"2024-02-05T07:30:44.669825+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.314735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.314735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:18.682614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"365'11","reported_seq":278,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.674873+0000","last_change":"2024-02-05T07:30:40.517194+0000","last_active":"2024-02-05T07:30:44.674873+0000","last_peered":"2024-02-05T07:30:44.674873+0000","last_clean":"2024-02-05T07:30:44.674873+0000","last_became_active":"2024-02-05T07:30:40.495708+0000","last_became_peered":"2024-02-05T07:30:40.495708+0000","last_unstale":"2024-02-05T07:30:44.674873+0000","last_undegraded":"2024-02-05T07:30:44.674873+0000","last_fullsized":"2024-02-05T07:30:44.674873+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:26:35.091894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.091894+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":634,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.656133+0000","last_change":"2024-02-05T07:27:03.366653+0000","last_active":"2024-02-05T07:30:44.656133+0000","last_peered":"2024-02-05T07:30:44.656133+0000","last_clean":"2024-02-05T07:30:44.656133+0000","last_became_active":"2024-02-05T07:27:03.364334+0000","last_became_peered":"2024-02-05T07:27:03.364334+0000","last_unstale":"2024-02-05T07:30:44.656133+0000","last_undegraded":"2024-02-05T07:30:44.656133+0000","last_fullsized":"2024-02-05T07:30:44.656133+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:25.323384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:25.323384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":477,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.670078+0000","last_change":"2024-02-05T07:28:41.013449+0000","last_active":"2024-02-05T07:30:44.670078+0000","last_peered":"2024-02-05T07:30:44.670078+0000","last_clean":"2024-02-05T07:30:44.670078+0000","last_became_active":"2024-02-05T07:28:19.302393+0000","last_became_peered":"2024-02-05T07:28:19.302393+0000","last_unstale":"2024-02-05T07:30:44.670078+0000","last_undegraded":"2024-02-05T07:30:44.670078+0000","last_fullsized":"2024-02-05T07:30:44.670078+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:41.013360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:41.013360+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"326'25","reported_seq":275,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.675077+0000","last_change":"2024-02-05T07:30:40.512630+0000","last_active":"2024-02-05T07:30:44.675077+0000","last_peered":"2024-02-05T07:30:44.675077+0000","last_clean":"2024-02-05T07:30:44.675077+0000","last_became_active":"2024-02-05T07:30:40.486394+0000","last_became_peered":"2024-02-05T07:30:40.486394+0000","last_unstale":"2024-02-05T07:30:44.675077+0000","last_undegraded":"2024-02-05T07:30:44.675077+0000","last_fullsized":"2024-02-05T07:30:44.675077+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:30:33.959124+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:30:33.959124+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:36.817784+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.655362+0000","last_change":"2024-02-05T07:30:40.505910+0000","last_active":"2024-02-05T07:30:44.655362+0000","last_peered":"2024-02-05T07:30:44.655362+0000","last_clean":"2024-02-05T07:30:44.655362+0000","last_became_active":"2024-02-05T07:30:40.505600+0000","last_became_peered":"2024-02-05T07:30:40.505600+0000","last_unstale":"2024-02-05T07:30:44.655362+0000","last_undegraded":"2024-02-05T07:30:44.655362+0000","last_fullsized":"2024-02-05T07:30:44.655362+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:46.210169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:46.210169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:54.529348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"217'9","reported_seq":415,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.677077+0000","last_change":"2024-02-05T07:30:27.993196+0000","last_active":"2024-02-05T07:30:45.677077+0000","last_peered":"2024-02-05T07:30:45.677077+0000","last_clean":"2024-02-05T07:30:45.677077+0000","last_became_active":"2024-02-05T07:25:27.246616+0000","last_became_peered":"2024-02-05T07:25:27.246616+0000","last_unstale":"2024-02-05T07:30:45.677077+0000","last_undegraded":"2024-02-05T07:30:45.677077+0000","last_fullsized":"2024-02-05T07:30:45.677077+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9","last_scrub_stamp":"2024-02-05T07:30:27.993000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:27.993000+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:40.940475+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":387,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.672367+0000","last_change":"2024-02-05T07:30:40.508206+0000","last_active":"2024-02-05T07:30:44.672367+0000","last_peered":"2024-02-05T07:30:44.672367+0000","last_clean":"2024-02-05T07:30:44.672367+0000","last_became_active":"2024-02-05T07:30:40.507831+0000","last_became_peered":"2024-02-05T07:30:40.507831+0000","last_unstale":"2024-02-05T07:30:44.672367+0000","last_undegraded":"2024-02-05T07:30:44.672367+0000","last_fullsized":"2024-02-05T07:30:44.672367+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:27:01.069970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:27:01.069970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"326'19","reported_seq":695,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.655368+0000","last_change":"2024-02-05T07:28:25.780182+0000","last_active":"2024-02-05T07:30:44.655368+0000","last_peered":"2024-02-05T07:30:44.655368+0000","last_clean":"2024-02-05T07:30:44.655368+0000","last_became_active":"2024-02-05T07:28:25.779886+0000","last_became_peered":"2024-02-05T07:28:25.779886+0000","last_unstale":"2024-02-05T07:30:44.655368+0000","last_undegraded":"2024-02-05T07:30:44.655368+0000","last_fullsized":"2024-02-05T07:30:44.655368+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:25:42.192387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.192387+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":656,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.655637+0000","last_change":"2024-02-05T07:30:40.510773+0000","last_active":"2024-02-05T07:30:44.655637+0000","last_peered":"2024-02-05T07:30:44.655637+0000","last_clean":"2024-02-05T07:30:44.655637+0000","last_became_active":"2024-02-05T07:30:40.510301+0000","last_became_peered":"2024-02-05T07:30:40.510301+0000","last_unstale":"2024-02-05T07:30:44.655637+0000","last_undegraded":"2024-02-05T07:30:44.655637+0000","last_fullsized":"2024-02-05T07:30:44.655637+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:36.798282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:29:36.798282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:52.592293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":658,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.688676+0000","last_change":"2024-02-05T07:30:40.487555+0000","last_active":"2024-02-05T07:30:45.688676+0000","last_peered":"2024-02-05T07:30:45.688676+0000","last_clean":"2024-02-05T07:30:45.688676+0000","last_became_active":"2024-02-05T07:30:40.487141+0000","last_became_peered":"2024-02-05T07:30:40.487141+0000","last_unstale":"2024-02-05T07:30:45.688676+0000","last_undegraded":"2024-02-05T07:30:45.688676+0000","last_fullsized":"2024-02-05T07:30:45.688676+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.555824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.555824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":220,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.674044+0000","last_change":"2024-02-05T07:27:17.708497+0000","last_active":"2024-02-05T07:30:44.674044+0000","last_peered":"2024-02-05T07:30:44.674044+0000","last_clean":"2024-02-05T07:30:44.674044+0000","last_became_active":"2024-02-05T07:27:17.705910+0000","last_became_peered":"2024-02-05T07:27:17.705910+0000","last_unstale":"2024-02-05T07:30:44.674044+0000","last_undegraded":"2024-02-05T07:30:44.674044+0000","last_fullsized":"2024-02-05T07:30:44.674044+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.120512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"325'15","reported_seq":271,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.688002+0000","last_change":"2024-02-05T07:29:57.775013+0000","last_active":"2024-02-05T07:30:45.688002+0000","last_peered":"2024-02-05T07:30:45.688002+0000","last_clean":"2024-02-05T07:30:45.688002+0000","last_became_active":"2024-02-05T07:28:26.589705+0000","last_became_peered":"2024-02-05T07:28:26.589705+0000","last_unstale":"2024-02-05T07:30:45.688002+0000","last_undegraded":"2024-02-05T07:30:45.688002+0000","last_fullsized":"2024-02-05T07:30:45.688002+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:29:57.774913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:57.774913+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.777452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":444,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.670375+0000","last_change":"2024-02-05T07:30:40.489183+0000","last_active":"2024-02-05T07:30:44.670375+0000","last_peered":"2024-02-05T07:30:44.670375+0000","last_clean":"2024-02-05T07:30:44.670375+0000","last_became_active":"2024-02-05T07:30:40.488789+0000","last_became_peered":"2024-02-05T07:30:40.488789+0000","last_unstale":"2024-02-05T07:30:44.670375+0000","last_undegraded":"2024-02-05T07:30:44.670375+0000","last_fullsized":"2024-02-05T07:30:44.670375+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.220841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.220841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:12.818967+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":222,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.673243+0000","last_change":"2024-02-05T07:27:16.765064+0000","last_active":"2024-02-05T07:30:44.673243+0000","last_peered":"2024-02-05T07:30:44.673243+0000","last_clean":"2024-02-05T07:30:44.673243+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:30:44.673243+0000","last_undegraded":"2024-02-05T07:30:44.673243+0000","last_fullsized":"2024-02-05T07:30:44.673243+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:26:31.506641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":398,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.669620+0000","last_change":"2024-02-05T07:28:26.588019+0000","last_active":"2024-02-05T07:30:45.669620+0000","last_peered":"2024-02-05T07:30:45.669620+0000","last_clean":"2024-02-05T07:30:45.669620+0000","last_became_active":"2024-02-05T07:28:26.587701+0000","last_became_peered":"2024-02-05T07:28:26.587701+0000","last_unstale":"2024-02-05T07:30:45.669620+0000","last_undegraded":"2024-02-05T07:30:45.669620+0000","last_fullsized":"2024-02-05T07:30:45.669620+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:37.616737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:26:37.616737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":890,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.675940+0000","last_change":"2024-02-05T07:30:43.515841+0000","last_active":"2024-02-05T07:30:45.675940+0000","last_peered":"2024-02-05T07:30:45.675940+0000","last_clean":"2024-02-05T07:30:45.675940+0000","last_became_active":"2024-02-05T07:30:43.515488+0000","last_became_peered":"2024-02-05T07:30:43.515488+0000","last_unstale":"2024-02-05T07:30:45.675940+0000","last_undegraded":"2024-02-05T07:30:45.675940+0000","last_fullsized":"2024-02-05T07:30:45.675940+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:30:36.995330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.995330+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:32:06.145775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"326'19","reported_seq":275,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.674227+0000","last_change":"2024-02-05T07:30:40.906805+0000","last_active":"2024-02-05T07:30:44.674227+0000","last_peered":"2024-02-05T07:30:44.674227+0000","last_clean":"2024-02-05T07:30:44.674227+0000","last_became_active":"2024-02-05T07:30:40.501893+0000","last_became_peered":"2024-02-05T07:30:40.501893+0000","last_unstale":"2024-02-05T07:30:44.674227+0000","last_undegraded":"2024-02-05T07:30:44.674227+0000","last_fullsized":"2024-02-05T07:30:44.674227+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:30:40.906693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.906693+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:41.737495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":47,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.679655+0000","last_change":"2024-02-05T07:30:40.492005+0000","last_active":"2024-02-05T07:30:45.679655+0000","last_peered":"2024-02-05T07:30:45.679655+0000","last_clean":"2024-02-05T07:30:45.679655+0000","last_became_active":"2024-02-05T07:30:40.491686+0000","last_became_peered":"2024-02-05T07:30:40.491686+0000","last_unstale":"2024-02-05T07:30:45.679655+0000","last_undegraded":"2024-02-05T07:30:45.679655+0000","last_fullsized":"2024-02-05T07:30:45.679655+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.097532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.097532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:50.701071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":518,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.654270+0000","last_change":"2024-02-05T07:30:40.497509+0000","last_active":"2024-02-05T07:30:44.654270+0000","last_peered":"2024-02-05T07:30:44.654270+0000","last_clean":"2024-02-05T07:30:44.654270+0000","last_became_active":"2024-02-05T07:30:40.497190+0000","last_became_peered":"2024-02-05T07:30:40.497190+0000","last_unstale":"2024-02-05T07:30:44.654270+0000","last_undegraded":"2024-02-05T07:30:44.654270+0000","last_fullsized":"2024-02-05T07:30:44.654270+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:29:58.191815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:20.355601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":234,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.672862+0000","last_change":"2024-02-05T07:29:40.983327+0000","last_active":"2024-02-05T07:30:44.672862+0000","last_peered":"2024-02-05T07:30:44.672862+0000","last_clean":"2024-02-05T07:30:44.672862+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:30:44.672862+0000","last_undegraded":"2024-02-05T07:30:44.672862+0000","last_fullsized":"2024-02-05T07:30:44.672862+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:40.983227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.983227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:04.158002+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"325'8","reported_seq":246,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.669922+0000","last_change":"2024-02-05T07:29:53.958469+0000","last_active":"2024-02-05T07:30:44.669922+0000","last_peered":"2024-02-05T07:30:44.669922+0000","last_clean":"2024-02-05T07:30:44.669922+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:30:44.669922+0000","last_undegraded":"2024-02-05T07:30:44.669922+0000","last_fullsized":"2024-02-05T07:30:44.669922+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:29:53.958384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.958384+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:10.373415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":233,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.672052+0000","last_change":"2024-02-05T07:28:46.056835+0000","last_active":"2024-02-05T07:30:44.672052+0000","last_peered":"2024-02-05T07:30:44.672052+0000","last_clean":"2024-02-05T07:30:44.672052+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:30:44.672052+0000","last_undegraded":"2024-02-05T07:30:44.672052+0000","last_fullsized":"2024-02-05T07:30:44.672052+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:46.056702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:28:46.056702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":493,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669847+0000","last_change":"2024-02-05T07:30:40.500892+0000","last_active":"2024-02-05T07:30:46.669847+0000","last_peered":"2024-02-05T07:30:46.669847+0000","last_clean":"2024-02-05T07:30:46.669847+0000","last_became_active":"2024-02-05T07:30:40.500403+0000","last_became_peered":"2024-02-05T07:30:40.500403+0000","last_unstale":"2024-02-05T07:30:46.669847+0000","last_undegraded":"2024-02-05T07:30:46.669847+0000","last_fullsized":"2024-02-05T07:30:46.669847+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:58.034570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:29:58.034570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:09.895599+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":23,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.676734+0000","last_change":"2024-02-05T07:30:40.489534+0000","last_active":"2024-02-05T07:30:45.676734+0000","last_peered":"2024-02-05T07:30:45.676734+0000","last_clean":"2024-02-05T07:30:45.676734+0000","last_became_active":"2024-02-05T07:30:40.489230+0000","last_became_peered":"2024-02-05T07:30:40.489230+0000","last_unstale":"2024-02-05T07:30:45.676734+0000","last_undegraded":"2024-02-05T07:30:45.676734+0000","last_fullsized":"2024-02-05T07:30:45.676734+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:27.013942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:27.013942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"326'19","reported_seq":479,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669630+0000","last_change":"2024-02-05T07:29:46.947670+0000","last_active":"2024-02-05T07:30:46.669630+0000","last_peered":"2024-02-05T07:30:46.669630+0000","last_clean":"2024-02-05T07:30:46.669630+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:30:46.669630+0000","last_undegraded":"2024-02-05T07:30:46.669630+0000","last_fullsized":"2024-02-05T07:30:46.669630+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:29:46.947559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:46.947559+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:51.889423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":230,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.669554+0000","last_change":"2024-02-05T07:29:38.000417+0000","last_active":"2024-02-05T07:30:44.669554+0000","last_peered":"2024-02-05T07:30:44.669554+0000","last_clean":"2024-02-05T07:30:44.669554+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:30:44.669554+0000","last_undegraded":"2024-02-05T07:30:44.669554+0000","last_fullsized":"2024-02-05T07:30:44.669554+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:38.000345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:38.000345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":24,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.679160+0000","last_change":"2024-02-05T07:30:40.480962+0000","last_active":"2024-02-05T07:30:45.679160+0000","last_peered":"2024-02-05T07:30:45.679160+0000","last_clean":"2024-02-05T07:30:45.679160+0000","last_became_active":"2024-02-05T07:30:40.479162+0000","last_became_peered":"2024-02-05T07:30:40.479162+0000","last_unstale":"2024-02-05T07:30:45.679160+0000","last_undegraded":"2024-02-05T07:30:45.679160+0000","last_fullsized":"2024-02-05T07:30:45.679160+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:28.153881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:28.153881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:54.994975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":653,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.729936+0000","last_change":"2024-02-05T07:29:55.800941+0000","last_active":"2024-02-05T07:30:45.729936+0000","last_peered":"2024-02-05T07:30:45.729936+0000","last_clean":"2024-02-05T07:30:45.729936+0000","last_became_active":"2024-02-05T07:27:03.360468+0000","last_became_peered":"2024-02-05T07:27:03.360468+0000","last_unstale":"2024-02-05T07:30:45.729936+0000","last_undegraded":"2024-02-05T07:30:45.729936+0000","last_fullsized":"2024-02-05T07:30:45.729936+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:55.800824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.800824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:15.924868+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"326'20","reported_seq":397,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.671743+0000","last_change":"2024-02-05T07:30:40.534624+0000","last_active":"2024-02-05T07:30:44.671743+0000","last_peered":"2024-02-05T07:30:44.671743+0000","last_clean":"2024-02-05T07:30:44.671743+0000","last_became_active":"2024-02-05T07:30:40.495954+0000","last_became_peered":"2024-02-05T07:30:40.495954+0000","last_unstale":"2024-02-05T07:30:44.671743+0000","last_undegraded":"2024-02-05T07:30:44.671743+0000","last_fullsized":"2024-02-05T07:30:44.671743+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:29:40.154854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.154854+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:01.254529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":695,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.668944+0000","last_change":"2024-02-05T07:30:40.526955+0000","last_active":"2024-02-05T07:30:45.668944+0000","last_peered":"2024-02-05T07:30:45.668944+0000","last_clean":"2024-02-05T07:30:45.668944+0000","last_became_active":"2024-02-05T07:30:40.495135+0000","last_became_peered":"2024-02-05T07:30:40.495135+0000","last_unstale":"2024-02-05T07:30:45.668944+0000","last_undegraded":"2024-02-05T07:30:45.668944+0000","last_fullsized":"2024-02-05T07:30:45.668944+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:52.715259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:52.715259+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:02.319506+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"365'11","reported_seq":619,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-05T07:30:46.669782+0000","last_change":"2024-02-05T07:30:40.516172+0000","last_active":"2024-02-05T07:30:46.669782+0000","last_peered":"2024-02-05T07:30:46.669782+0000","last_clean":"2024-02-05T07:30:46.669782+0000","last_became_active":"2024-02-05T07:30:40.497371+0000","last_became_peered":"2024-02-05T07:30:40.497371+0000","last_unstale":"2024-02-05T07:30:46.669782+0000","last_undegraded":"2024-02-05T07:30:46.669782+0000","last_fullsized":"2024-02-05T07:30:46.669782+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'10","last_scrub_stamp":"2024-02-05T07:29:55.058040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.058040+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:16.734609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":141,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.676034+0000","last_change":"2024-02-05T07:30:41.950393+0000","last_active":"2024-02-05T07:30:45.676034+0000","last_peered":"2024-02-05T07:30:45.676034+0000","last_clean":"2024-02-05T07:30:45.676034+0000","last_became_active":"2024-02-05T07:30:40.486920+0000","last_became_peered":"2024-02-05T07:30:40.486920+0000","last_unstale":"2024-02-05T07:30:45.676034+0000","last_undegraded":"2024-02-05T07:30:45.676034+0000","last_fullsized":"2024-02-05T07:30:45.676034+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:30:41.950253+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:30:41.950253+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:53.731570+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"326'23","reported_seq":256,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.670219+0000","last_change":"2024-02-05T07:28:47.062765+0000","last_active":"2024-02-05T07:30:44.670219+0000","last_peered":"2024-02-05T07:30:44.670219+0000","last_clean":"2024-02-05T07:30:44.670219+0000","last_became_active":"2024-02-05T07:27:16.747998+0000","last_became_peered":"2024-02-05T07:27:16.747998+0000","last_unstale":"2024-02-05T07:30:44.670219+0000","last_undegraded":"2024-02-05T07:30:44.670219+0000","last_fullsized":"2024-02-05T07:30:44.670219+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'17","last_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.062618+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":334,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.688542+0000","last_change":"2024-02-05T07:30:39.874170+0000","last_active":"2024-02-05T07:30:45.688542+0000","last_peered":"2024-02-05T07:30:45.688542+0000","last_clean":"2024-02-05T07:30:45.688542+0000","last_became_active":"2024-02-05T07:25:30.261037+0000","last_became_peered":"2024-02-05T07:25:30.261037+0000","last_unstale":"2024-02-05T07:30:45.688542+0000","last_undegraded":"2024-02-05T07:30:45.688542+0000","last_fullsized":"2024-02-05T07:30:45.688542+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.874081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:54.591947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":655,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.655768+0000","last_change":"2024-02-05T07:30:26.669849+0000","last_active":"2024-02-05T07:30:44.655768+0000","last_peered":"2024-02-05T07:30:44.655768+0000","last_clean":"2024-02-05T07:30:44.655768+0000","last_became_active":"2024-02-05T07:27:16.706777+0000","last_became_peered":"2024-02-05T07:27:16.706777+0000","last_unstale":"2024-02-05T07:30:44.655768+0000","last_undegraded":"2024-02-05T07:30:44.655768+0000","last_fullsized":"2024-02-05T07:30:44.655768+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:26.669576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:26.669576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:32.569089+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":233,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.672200+0000","last_change":"2024-02-05T07:28:53.138071+0000","last_active":"2024-02-05T07:30:44.672200+0000","last_peered":"2024-02-05T07:30:44.672200+0000","last_clean":"2024-02-05T07:30:44.672200+0000","last_became_active":"2024-02-05T07:28:26.603067+0000","last_became_peered":"2024-02-05T07:28:26.603067+0000","last_unstale":"2024-02-05T07:30:44.672200+0000","last_undegraded":"2024-02-05T07:30:44.672200+0000","last_fullsized":"2024-02-05T07:30:44.672200+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:53.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:53.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"326'27","reported_seq":733,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.676129+0000","last_change":"2024-02-05T07:29:41.354568+0000","last_active":"2024-02-05T07:30:45.676129+0000","last_peered":"2024-02-05T07:30:45.676129+0000","last_clean":"2024-02-05T07:30:45.676129+0000","last_became_active":"2024-02-05T07:27:16.736296+0000","last_became_peered":"2024-02-05T07:27:16.736296+0000","last_unstale":"2024-02-05T07:30:45.676129+0000","last_undegraded":"2024-02-05T07:30:45.676129+0000","last_fullsized":"2024-02-05T07:30:45.676129+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:29:41.354464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:41.354464+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:56.393313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":400,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.655259+0000","last_change":"2024-02-05T07:30:40.503947+0000","last_active":"2024-02-05T07:30:44.655259+0000","last_peered":"2024-02-05T07:30:44.655259+0000","last_clean":"2024-02-05T07:30:44.655259+0000","last_became_active":"2024-02-05T07:30:40.503627+0000","last_became_peered":"2024-02-05T07:30:40.503627+0000","last_unstale":"2024-02-05T07:30:44.655259+0000","last_undegraded":"2024-02-05T07:30:44.655259+0000","last_fullsized":"2024-02-05T07:30:44.655259+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:34.663987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:34.663987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:57.807204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":643,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.729726+0000","last_change":"2024-02-05T07:30:40.884273+0000","last_active":"2024-02-05T07:30:45.729726+0000","last_peered":"2024-02-05T07:30:45.729726+0000","last_clean":"2024-02-05T07:30:45.729726+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:30:45.729726+0000","last_undegraded":"2024-02-05T07:30:45.729726+0000","last_fullsized":"2024-02-05T07:30:45.729726+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:40.884197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.884197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:48.024898+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":654,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.015893+0000","last_change":"2024-02-05T07:29:59.002788+0000","last_active":"2024-02-05T07:30:45.015893+0000","last_peered":"2024-02-05T07:30:45.015893+0000","last_clean":"2024-02-05T07:30:45.015893+0000","last_became_active":"2024-02-05T07:28:26.590448+0000","last_became_peered":"2024-02-05T07:28:26.590448+0000","last_unstale":"2024-02-05T07:30:45.015893+0000","last_undegraded":"2024-02-05T07:30:45.015893+0000","last_fullsized":"2024-02-05T07:30:45.015893+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.002677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.002677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:25.104559+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"325'7","reported_seq":673,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.655265+0000","last_change":"2024-02-05T07:27:16.970973+0000","last_active":"2024-02-05T07:30:44.655265+0000","last_peered":"2024-02-05T07:30:44.655265+0000","last_clean":"2024-02-05T07:30:44.655265+0000","last_became_active":"2024-02-05T07:27:16.745897+0000","last_became_peered":"2024-02-05T07:27:16.745897+0000","last_unstale":"2024-02-05T07:30:44.655265+0000","last_undegraded":"2024-02-05T07:30:44.655265+0000","last_fullsized":"2024-02-05T07:30:44.655265+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:26:35.253452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.253452+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":472,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.655557+0000","last_change":"2024-02-05T07:30:39.726603+0000","last_active":"2024-02-05T07:30:44.655557+0000","last_peered":"2024-02-05T07:30:44.655557+0000","last_clean":"2024-02-05T07:30:44.655557+0000","last_became_active":"2024-02-05T07:28:26.590322+0000","last_became_peered":"2024-02-05T07:28:26.590322+0000","last_unstale":"2024-02-05T07:30:44.655557+0000","last_undegraded":"2024-02-05T07:30:44.655557+0000","last_fullsized":"2024-02-05T07:30:44.655557+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.726525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.726525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:32:09.212327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":20,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.655083+0000","last_change":"2024-02-05T07:30:40.504663+0000","last_active":"2024-02-05T07:30:44.655083+0000","last_peered":"2024-02-05T07:30:44.655083+0000","last_clean":"2024-02-05T07:30:44.655083+0000","last_became_active":"2024-02-05T07:30:40.504365+0000","last_became_peered":"2024-02-05T07:30:40.504365+0000","last_unstale":"2024-02-05T07:30:44.655083+0000","last_undegraded":"2024-02-05T07:30:44.655083+0000","last_fullsized":"2024-02-05T07:30:44.655083+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:27:00.183281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:27:00.183281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":450,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.669213+0000","last_change":"2024-02-05T07:29:50.742042+0000","last_active":"2024-02-05T07:30:45.669213+0000","last_peered":"2024-02-05T07:30:45.669213+0000","last_clean":"2024-02-05T07:30:45.669213+0000","last_became_active":"2024-02-05T07:27:16.769581+0000","last_became_peered":"2024-02-05T07:27:16.769581+0000","last_unstale":"2024-02-05T07:30:45.669213+0000","last_undegraded":"2024-02-05T07:30:45.669213+0000","last_fullsized":"2024-02-05T07:30:45.669213+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:50.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:18.589144+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"325'19","reported_seq":705,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.654792+0000","last_change":"2024-02-05T07:30:00.001976+0000","last_active":"2024-02-05T07:30:44.654792+0000","last_peered":"2024-02-05T07:30:44.654792+0000","last_clean":"2024-02-05T07:30:44.654792+0000","last_became_active":"2024-02-05T07:27:16.707036+0000","last_became_peered":"2024-02-05T07:27:16.707036+0000","last_unstale":"2024-02-05T07:30:44.654792+0000","last_undegraded":"2024-02-05T07:30:44.654792+0000","last_fullsized":"2024-02-05T07:30:44.654792+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'19","last_scrub_stamp":"2024-02-05T07:30:00.001863+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:30:00.001863+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:04.904819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":475,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.656134+0000","last_change":"2024-02-05T07:29:53.022701+0000","last_active":"2024-02-05T07:30:44.656134+0000","last_peered":"2024-02-05T07:30:44.656134+0000","last_clean":"2024-02-05T07:30:44.656134+0000","last_became_active":"2024-02-05T07:28:26.616107+0000","last_became_peered":"2024-02-05T07:28:26.616107+0000","last_unstale":"2024-02-05T07:30:44.656134+0000","last_undegraded":"2024-02-05T07:30:44.656134+0000","last_fullsized":"2024-02-05T07:30:44.656134+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.022606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.022606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:18.495044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":22,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.655141+0000","last_change":"2024-02-05T07:30:40.491944+0000","last_active":"2024-02-05T07:30:44.655141+0000","last_peered":"2024-02-05T07:30:44.655141+0000","last_clean":"2024-02-05T07:30:44.655141+0000","last_became_active":"2024-02-05T07:30:40.491630+0000","last_became_peered":"2024-02-05T07:30:40.491630+0000","last_unstale":"2024-02-05T07:30:44.655141+0000","last_undegraded":"2024-02-05T07:30:44.655141+0000","last_fullsized":"2024-02-05T07:30:44.655141+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:54.207089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:54.207089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:04.317669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":402,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.669829+0000","last_change":"2024-02-05T07:30:40.476139+0000","last_active":"2024-02-05T07:30:45.669829+0000","last_peered":"2024-02-05T07:30:45.669829+0000","last_clean":"2024-02-05T07:30:45.669829+0000","last_became_active":"2024-02-05T07:30:40.475535+0000","last_became_peered":"2024-02-05T07:30:40.475535+0000","last_unstale":"2024-02-05T07:30:45.669829+0000","last_undegraded":"2024-02-05T07:30:45.669829+0000","last_fullsized":"2024-02-05T07:30:45.669829+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:31.484580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:31.484580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"326'27","reported_seq":256,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.674359+0000","last_change":"2024-02-05T07:27:17.002461+0000","last_active":"2024-02-05T07:30:44.674359+0000","last_peered":"2024-02-05T07:30:44.674359+0000","last_clean":"2024-02-05T07:30:44.674359+0000","last_became_active":"2024-02-05T07:27:16.751179+0000","last_became_peered":"2024-02-05T07:27:16.751179+0000","last_unstale":"2024-02-05T07:30:44.674359+0000","last_undegraded":"2024-02-05T07:30:44.674359+0000","last_fullsized":"2024-02-05T07:30:44.674359+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:25:38.772812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:38.772812+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":226,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.672293+0000","last_change":"2024-02-05T07:28:26.596258+0000","last_active":"2024-02-05T07:30:44.672293+0000","last_peered":"2024-02-05T07:30:44.672293+0000","last_clean":"2024-02-05T07:30:44.672293+0000","last_became_active":"2024-02-05T07:28:26.595627+0000","last_became_peered":"2024-02-05T07:28:26.595627+0000","last_unstale":"2024-02-05T07:30:44.672293+0000","last_undegraded":"2024-02-05T07:30:44.672293+0000","last_fullsized":"2024-02-05T07:30:44.672293+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:50.859511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:50.859511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":649,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.654263+0000","last_change":"2024-02-05T07:29:48.045121+0000","last_active":"2024-02-05T07:30:44.654263+0000","last_peered":"2024-02-05T07:30:44.654263+0000","last_clean":"2024-02-05T07:30:44.654263+0000","last_became_active":"2024-02-05T07:28:25.778935+0000","last_became_peered":"2024-02-05T07:28:25.778935+0000","last_unstale":"2024-02-05T07:30:44.654263+0000","last_undegraded":"2024-02-05T07:30:44.654263+0000","last_fullsized":"2024-02-05T07:30:44.654263+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.045013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.045013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:30:48.765921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":240,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.673166+0000","last_change":"2024-02-05T07:30:40.496738+0000","last_active":"2024-02-05T07:30:44.673166+0000","last_peered":"2024-02-05T07:30:44.673166+0000","last_clean":"2024-02-05T07:30:44.673166+0000","last_became_active":"2024-02-05T07:30:40.493086+0000","last_became_peered":"2024-02-05T07:30:40.493086+0000","last_unstale":"2024-02-05T07:30:44.673166+0000","last_undegraded":"2024-02-05T07:30:44.673166+0000","last_fullsized":"2024-02-05T07:30:44.673166+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:44.050019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:44.050019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"325'15","reported_seq":670,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.654421+0000","last_change":"2024-02-05T07:29:33.839515+0000","last_active":"2024-02-05T07:30:44.654421+0000","last_peered":"2024-02-05T07:30:44.654421+0000","last_clean":"2024-02-05T07:30:44.654421+0000","last_became_active":"2024-02-05T07:28:25.777815+0000","last_became_peered":"2024-02-05T07:28:25.777815+0000","last_unstale":"2024-02-05T07:30:44.654421+0000","last_undegraded":"2024-02-05T07:30:44.654421+0000","last_fullsized":"2024-02-05T07:30:44.654421+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:29:33.839402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:33.839402+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:01.714433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":414,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.677684+0000","last_change":"2024-02-05T07:29:50.274850+0000","last_active":"2024-02-05T07:30:45.677684+0000","last_peered":"2024-02-05T07:30:45.677684+0000","last_clean":"2024-02-05T07:30:45.677684+0000","last_became_active":"2024-02-05T07:28:26.588496+0000","last_became_peered":"2024-02-05T07:28:26.588496+0000","last_unstale":"2024-02-05T07:30:45.677684+0000","last_undegraded":"2024-02-05T07:30:45.677684+0000","last_fullsized":"2024-02-05T07:30:45.677684+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.274768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:31:07.936836+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":647,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-05T07:30:45.669042+0000","last_change":"2024-02-05T07:30:40.496063+0000","last_active":"2024-02-05T07:30:45.669042+0000","last_peered":"2024-02-05T07:30:45.669042+0000","last_clean":"2024-02-05T07:30:45.669042+0000","last_became_active":"2024-02-05T07:30:40.495711+0000","last_became_peered":"2024-02-05T07:30:40.495711+0000","last_unstale":"2024-02-05T07:30:45.669042+0000","last_undegraded":"2024-02-05T07:30:45.669042+0000","last_fullsized":"2024-02-05T07:30:45.669042+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:28.575859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:28.575859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":640,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T07:30:44.654653+0000","last_change":"2024-02-05T07:30:40.490593+0000","last_active":"2024-02-05T07:30:44.654653+0000","last_peered":"2024-02-05T07:30:44.654653+0000","last_clean":"2024-02-05T07:30:44.654653+0000","last_became_active":"2024-02-05T07:30:40.490288+0000","last_became_peered":"2024-02-05T07:30:40.490288+0000","last_unstale":"2024-02-05T07:30:44.654653+0000","last_undegraded":"2024-02-05T07:30:44.654653+0000","last_fullsized":"2024-02-05T07:30:44.654653+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:21.317417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:21.317417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2904162304,"num_objects":44314,"num_object_clones":0,"num_object_copies":132942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13734,"num_objects_misplaced":11736,"num_objects_unfound":0,"num_objects_dirty":44314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44314,"num_write_kb":2836096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1781,"num_bytes_recovered":116719616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8895397888,"data_stored":8895397888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44314,"ondisk_log_size":44314,"up":48,"acting":38,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":697707,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":54,"num_write":32,"num_write_kb":439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":322019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":62,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1107,"num_read_kb":864,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":6480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":11876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":738,"ondisk_log_size":738,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":257698037908,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":301,"seq":1292785156126,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1058404,"kb_used_data":849628,"kb_used_omap":5,"kb_used_meta":208762,"kb_avail":92686748,"statfs":{"total":95995035648,"available":94911229952,"internally_reserved":0,"allocated":870019072,"data_stored":869018555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":213772805},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:30:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.8839999999999999,"15min":2.1120000000000001},"min":{"1min":0.32300000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.72299999999999998,"5min":14.218,"15min":14.218},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.8700000000000001,"15min":2.0790000000000002},"min":{"1min":0.36499999999999999,"5min":0.245,"15min":0.245},"max":{"1min":1.5800000000000001,"5min":14.175000000000001,"15min":14.175000000000001},"last":0.84899999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:30:30 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":1.48,"15min":1.6279999999999999},"min":{"1min":0.34599999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.736,"5min":15.473000000000001,"15min":15.473000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":2.2000000000000002,"15min":2.472},"min":{"1min":0.33500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80100000000000005,"5min":14.784000000000001,"15min":14.784000000000001},"last":0.999}]},{"osd":2,"last update":"Mon Feb 5 07:30:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":2.4969999999999999,"15min":2.8210000000000002},"min":{"1min":0.34499999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.1120000000000001,"5min":34.689999999999998,"15min":34.689999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":2.9870000000000001,"15min":3.395},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88600000000000001,"5min":34.646999999999998,"15min":34.646999999999998},"last":0.33700000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:30:30 2024","interfaces":[{"interface":"back","average":{"1min":3.6989999999999998,"5min":2.4910000000000001,"15min":2.2890000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":53.878,"5min":53.878,"15min":53.878},"last":0.495},{"interface":"front","average":{"1min":3.6190000000000002,"5min":2.7400000000000002,"15min":2.5939999999999999},"min":{"1min":0.38800000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":53.752000000000002,"5min":53.752000000000002,"15min":53.752000000000002},"last":1.49}]},{"osd":4,"last update":"Mon Feb 5 07:30:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.72099999999999997,"15min":0.751},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.90900000000000003,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.314},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.63500000000000001,"15min":0.65000000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.0720000000000001,"5min":1.869,"15min":1.869},"last":0.42599999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:30:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59899999999999998,"15min":0.60899999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73499999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.39300000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.83899999999999997,"5min":1.526,"15min":1.526},"last":0.45700000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:30:30 2024","interfaces":[{"interface":"back","average":{"1min":2.3220000000000001,"5min":4.9400000000000004,"15min":5.3760000000000003},"min":{"1min":0.35599999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":30.998999999999999,"5min":84.158000000000001,"15min":84.158000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":2.3519999999999999,"5min":5.0880000000000001,"15min":5.5439999999999996},"min":{"1min":0.40100000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":31.067,"5min":85.018000000000001,"15min":85.018000000000001},"last":0.439}]}]},{"osd":1,"up_from":13,"seq":55834575026,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2253700,"kb_used_data":1767156,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91491452,"statfs":{"total":95995035648,"available":93687246848,"internally_reserved":0,"allocated":1809567744,"data_stored":1808528659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":1.7170000000000001,"15min":1.9530000000000001},"min":{"1min":0.20899999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.8480000000000001,"5min":15.09,"15min":45.393000000000001},"last":2.6269999999999998},{"interface":"front","average":{"1min":1.254,"5min":1.7330000000000001,"15min":1.9450000000000001},"min":{"1min":0.26500000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":5.242,"5min":14.207000000000001,"15min":45.031999999999996},"last":2.718}]},{"osd":2,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":2.0830000000000002,"15min":1.5900000000000001},"min":{"1min":0.26900000000000002,"5min":0.218,"15min":0.20699999999999999},"max":{"1min":15.137,"5min":15.137,"15min":39.259999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":2.5870000000000002,"5min":1.98,"15min":1.651},"min":{"1min":0.29899999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.906000000000001,"5min":14.906000000000001,"15min":40.972000000000001},"last":1.6479999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:30:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.7050000000000001,"15min":1.4930000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20999999999999999},"max":{"1min":5.2389999999999999,"5min":19.829999999999998,"15min":43.823},"last":3.2879999999999998},{"interface":"front","average":{"1min":1.034,"5min":1.6120000000000001,"15min":1.524},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24199999999999999},"max":{"1min":6.3949999999999996,"5min":9.7880000000000003,"15min":39.314999999999998},"last":1.792}]},{"osd":4,"last update":"Mon Feb 5 07:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":2.21,"15min":1.595},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.312},"max":{"1min":3.133,"5min":61.722000000000001,"15min":61.722000000000001},"last":1.6970000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":2.149,"15min":1.6599999999999999},"min":{"1min":0.38400000000000001,"5min":0.311,"15min":0.29399999999999998},"max":{"1min":2.673,"5min":61.622999999999998,"15min":61.786000000000001},"last":0.44700000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:30:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":2.4649999999999999,"15min":1.6639999999999999},"min":{"1min":0.34999999999999998,"5min":0.309,"15min":0.309},"max":{"1min":6.319,"5min":106.672,"15min":106.672},"last":1.734},{"interface":"front","average":{"1min":1.048,"5min":2.5009999999999999,"15min":1.7709999999999999},"min":{"1min":0.38400000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.0869999999999997,"5min":107.348,"15min":107.348},"last":0.46200000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":1.651,"15min":1.786},"min":{"1min":0.47199999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.903,"5min":5.5990000000000002,"15min":5.5990000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.8620000000000001,"15min":2.0569999999999999},"min":{"1min":0.47599999999999998,"5min":0.437,"15min":0.437},"max":{"1min":1.778,"5min":6.3559999999999999,"15min":6.3559999999999999},"last":1.766}]},{"osd":7,"last update":"Mon Feb 5 07:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.5469999999999999,"15min":1.4219999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":5.0519999999999996,"5min":20.309000000000001,"15min":66.703999999999994},"last":2.548},{"interface":"front","average":{"1min":1.456,"5min":1.6379999999999999,"15min":1.4370000000000001},"min":{"1min":0.47599999999999998,"5min":0.34000000000000002,"15min":0.33500000000000002},"max":{"1min":5.6539999999999999,"5min":11.263999999999999,"15min":61.594999999999999},"last":0.50900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738551,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1264960,"kb_used_data":728436,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92480192,"statfs":{"total":95995035648,"available":94699716608,"internally_reserved":0,"allocated":745918464,"data_stored":744879026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:30:32 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":2.3620000000000001,"15min":1.5840000000000001},"min":{"1min":0.222,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.63300000000000001,"5min":45.372,"15min":45.372},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":2.3889999999999998,"15min":1.6679999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.186},"max":{"1min":1.0629999999999999,"5min":45.231999999999999,"15min":45.231999999999999},"last":0.499}]},{"osd":2,"last update":"Mon Feb 5 07:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.575,"15min":1.5589999999999999},"min":{"1min":0.23799999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.1760000000000002,"5min":44.920000000000002,"15min":65.903999999999996},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":1.579,"15min":1.534},"min":{"1min":0.224,"5min":0.224,"15min":0.182},"max":{"1min":4.8220000000000001,"5min":44.947000000000003,"15min":65.694000000000003},"last":0.34899999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:30:14 2024","interfaces":[{"interface":"back","average":{"1min":3.6709999999999998,"5min":1.8320000000000001,"15min":1.8169999999999999},"min":{"1min":0.254,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":29.629000000000001,"5min":29.629000000000001,"15min":65.155000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":3.1930000000000001,"5min":1.871,"15min":1.903},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":29.463999999999999,"5min":29.463999999999999,"15min":65.114000000000004},"last":0.94999999999999996}]},{"osd":4,"last update":"Mon Feb 5 07:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":2.6850000000000001,"15min":1.855},"min":{"1min":0.32500000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.94399999999999995,"5min":50.545000000000002,"15min":65.188000000000002},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":2.7690000000000001,"15min":1.903},"min":{"1min":0.32700000000000001,"5min":0.27100000000000002,"15min":0.23699999999999999},"max":{"1min":1.766,"5min":50.310000000000002,"15min":64.873000000000005},"last":0.59799999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:30:14 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":2.113,"15min":1.613},"min":{"1min":0.32100000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.1800000000000002,"5min":95.772999999999996,"15min":95.772999999999996},"last":0.754},{"interface":"front","average":{"1min":0.76900000000000002,"5min":2.081,"15min":1.663},"min":{"1min":0.29299999999999998,"5min":0.28199999999999997,"15min":0.27600000000000002},"max":{"1min":3.9860000000000002,"5min":96.028000000000006,"15min":96.028000000000006},"last":0.63600000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":1.2430000000000001,"15min":1.355},"min":{"1min":0.308,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.99299999999999999,"5min":4.6929999999999996,"15min":4.6929999999999996},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":1.4890000000000001,"15min":1.629},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.389,"5min":5.806,"15min":5.806},"last":0.70699999999999996}]},{"osd":7,"last update":"Mon Feb 5 07:29:45 2024","interfaces":[{"interface":"back","average":{"1min":3.3399999999999999,"5min":1.923,"15min":1.645},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":43.012,"5min":43.012,"15min":65.263000000000005},"last":0.84899999999999998},{"interface":"front","average":{"1min":3.7040000000000002,"5min":1.9199999999999999,"15min":1.6419999999999999},"min":{"1min":0.34000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":43.512,"5min":43.512,"15min":64.906000000000006},"last":0.55600000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411501,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1600996,"kb_used_data":1046420,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92144156,"statfs":{"total":95995035648,"available":94355615744,"internally_reserved":0,"allocated":1071534080,"data_stored":1070527659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":2.8260000000000001,"15min":1.875},"min":{"1min":0.23100000000000001,"5min":0.20699999999999999,"15min":0.182},"max":{"1min":12.759,"5min":92.638000000000005,"15min":92.638000000000005},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.879,"5min":2.7559999999999998,"15min":1.962},"min":{"1min":0.29099999999999998,"5min":0.22600000000000001,"15min":0.20699999999999999},"max":{"1min":12.647,"5min":93.481999999999999,"15min":93.481999999999999},"last":0.41599999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.9830000000000001,"5min":3.1560000000000001,"15min":1.635},"min":{"1min":0.26200000000000001,"5min":0.20999999999999999,"15min":0.19400000000000001},"max":{"1min":17.911999999999999,"5min":93.504000000000005,"15min":93.504000000000005},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.9399999999999999,"5min":3.2650000000000001,"15min":1.7330000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":11.464,"5min":93.004000000000005,"15min":93.004000000000005},"last":0.69999999999999996}]},{"osd":3,"last update":"Mon Feb 5 07:30:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":1.9399999999999999,"15min":1.3939999999999999},"min":{"1min":0.27900000000000003,"5min":0.25600000000000001,"15min":0.214},"max":{"1min":2.8580000000000001,"5min":26.114999999999998,"15min":26.114999999999998},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.505,"5min":1.5109999999999999,"15min":1.2689999999999999},"min":{"1min":0.28100000000000003,"5min":0.22600000000000001,"15min":0.193},"max":{"1min":1.5509999999999999,"5min":26.216000000000001,"15min":26.216000000000001},"last":0.74299999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":4.085,"15min":1.944},"min":{"1min":0.377,"5min":0.318,"15min":0.24099999999999999},"max":{"1min":5.4829999999999997,"5min":168.11699999999999,"15min":168.11699999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":3.9780000000000002,"15min":1.927},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":5.6680000000000001,"5min":168.589,"15min":168.589},"last":0.33900000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:30:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":1.597,"15min":1.1659999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":3.2549999999999999,"5min":39.923000000000002,"15min":39.923000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.554,"15min":1.113},"min":{"1min":0.35199999999999998,"5min":0.33600000000000002,"15min":0.26400000000000001},"max":{"1min":3.5950000000000002,"5min":38.920999999999999,"15min":38.920999999999999},"last":0.498}]},{"osd":6,"last update":"Mon Feb 5 07:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":1.133,"15min":1.2270000000000001},"min":{"1min":0.33700000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.98599999999999999,"5min":3.6749999999999998,"15min":3.6749999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.323,"15min":1.4490000000000001},"min":{"1min":0.379,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78700000000000003,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.54000000000000004}]},{"osd":7,"last update":"Mon Feb 5 07:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.744,"15min":1.256},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":5.5099999999999998,"5min":37.573999999999998,"15min":37.573999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.89900000000000002,"5min":1.6799999999999999,"15min":1.1930000000000001},"min":{"1min":0.35999999999999999,"5min":0.34699999999999998,"15min":0.33800000000000002},"max":{"1min":3.6480000000000001,"5min":37.432000000000002,"15min":37.432000000000002},"last":0.63800000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2309060,"kb_used_data":1772536,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91436092,"statfs":{"total":95995035648,"available":93630558208,"internally_reserved":0,"allocated":1815076864,"data_stored":1814034211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":549386695},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":1.5589999999999999,"15min":2.032},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":1.278,"5min":32.078000000000003,"15min":74.114999999999995},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":1.7330000000000001,"15min":1.8120000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.85399999999999998,"5min":34.598999999999997,"15min":48.740000000000002},"last":0.46600000000000003}]},{"osd":1,"last update":"Mon Feb 5 07:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":2.5529999999999999,"15min":1.9339999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":3.8170000000000002,"5min":59.298999999999999,"15min":59.298999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.63,"5min":2.5739999999999998,"15min":2.206},"min":{"1min":0.255,"5min":0.255,"15min":0.185},"max":{"1min":4.2889999999999997,"5min":59.634,"15min":74.084999999999994},"last":1.101}]},{"osd":2,"last update":"Mon Feb 5 07:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.407,"15min":1.978},"min":{"1min":0.245,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.2229999999999999,"5min":29.123999999999999,"15min":74.051000000000002},"last":1.1319999999999999},{"interface":"front","average":{"1min":0.436,"5min":1.2509999999999999,"15min":1.5169999999999999},"min":{"1min":0.26800000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.474,"5min":7.8769999999999998,"15min":65.861999999999995},"last":0.59199999999999997}]},{"osd":4,"last update":"Mon Feb 5 07:30:42 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":1.5,"15min":1.097},"min":{"1min":0.38700000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.859,"5min":34.981000000000002,"15min":34.981000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":1.4870000000000001,"15min":1.0860000000000001},"min":{"1min":0.30399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.375,"5min":36.987000000000002,"15min":36.987000000000002},"last":0.39100000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:30:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.876,"15min":1.3520000000000001},"min":{"1min":0.38800000000000001,"5min":0.32900000000000001,"15min":0.28499999999999998},"max":{"1min":2.6579999999999999,"5min":4.1100000000000003,"15min":73.727000000000004},"last":1.2290000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":1.1279999999999999,"15min":1.1000000000000001},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.27400000000000002},"max":{"1min":2.6840000000000002,"5min":6.3319999999999999,"15min":21.126000000000001},"last":0.56699999999999995}]},{"osd":6,"last update":"Mon Feb 5 07:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.88900000000000001,"15min":0.91600000000000004},"min":{"1min":0.42399999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":1.198},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.91700000000000004,"15min":0.97299999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80200000000000005,"5min":2.77,"15min":2.77},"last":0.49299999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:29:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":2.4870000000000001,"15min":1.498},"min":{"1min":0.42699999999999999,"5min":0.318,"15min":0.318},"max":{"1min":7.984,"5min":91.051000000000002,"15min":91.051000000000002},"last":0.621},{"interface":"front","average":{"1min":0.999,"5min":2.431,"15min":1.833},"min":{"1min":0.46700000000000003,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":4.0599999999999996,"5min":91.924000000000007,"15min":91.924000000000007},"last":1.2529999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953635,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1182568,"kb_used_data":709908,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92562584,"statfs":{"total":95995035648,"available":94784086016,"internally_reserved":0,"allocated":726945792,"data_stored":725938043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:30:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":4.0460000000000003,"15min":2.0870000000000002},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.3540000000000001,"5min":101.212,"15min":101.212},"last":0.315},{"interface":"front","average":{"1min":0.73899999999999999,"5min":4.0019999999999998,"15min":2.1760000000000002},"min":{"1min":0.24099999999999999,"5min":0.23899999999999999,"15min":0.23300000000000001},"max":{"1min":4.0490000000000004,"5min":102.07599999999999,"15min":102.07599999999999},"last":0.34000000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:30:43 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":4.1580000000000004,"15min":1.8480000000000001},"min":{"1min":0.307,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":6.3319999999999999,"5min":69.102000000000004,"15min":69.102000000000004},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":4.6340000000000003,"15min":2.0459999999999998},"min":{"1min":0.27600000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.3680000000000003,"5min":68.108999999999995,"15min":68.108999999999995},"last":0.40999999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:30:43 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":1.7170000000000001,"15min":1.327},"min":{"1min":0.29699999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.7679999999999998,"5min":26.045000000000002,"15min":67.144999999999996},"last":0.376},{"interface":"front","average":{"1min":1.022,"5min":1.768,"15min":1.3129999999999999},"min":{"1min":0.28000000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":9.8209999999999997,"5min":25.960999999999999,"15min":66.959999999999994},"last":0.432}]},{"osd":3,"last update":"Mon Feb 5 07:30:43 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":2.2509999999999999,"15min":1.236},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.27500000000000002},"max":{"1min":5.7469999999999999,"5min":66.019999999999996,"15min":66.019999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.996,"5min":2.6320000000000001,"15min":1.4159999999999999},"min":{"1min":0.40600000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.5140000000000002,"5min":101.001,"15min":101.001},"last":0.57199999999999995}]},{"osd":5,"last update":"Mon Feb 5 07:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.82599999999999996,"15min":0.57899999999999996},"min":{"1min":0.28199999999999997,"5min":0.25800000000000001,"15min":0.25700000000000001},"max":{"1min":1.907,"5min":10.917999999999999,"15min":10.917999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.83399999999999996,"15min":0.60499999999999998},"min":{"1min":0.317,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.2410000000000001,"5min":15.105,"15min":15.105},"last":0.44900000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":1.1950000000000001,"15min":1.3100000000000001},"min":{"1min":0.316,"5min":0.245,"15min":0.245},"max":{"1min":1.1679999999999999,"5min":14.606,"15min":14.606},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.4630000000000001,"15min":1.6160000000000001},"min":{"1min":0.35399999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.83899999999999997,"5min":13.114000000000001,"15min":13.114000000000001},"last":0.63200000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.97699999999999998,"15min":0.72199999999999998},"min":{"1min":0.308,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.8260000000000001,"5min":15.073,"15min":15.073},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.94799999999999995,"15min":0.67900000000000005},"min":{"1min":0.39500000000000002,"5min":0.248,"15min":0.248},"max":{"1min":3.0670000000000002,"5min":14.581,"15min":14.581},"last":0.55900000000000005}]}]},{"osd":5,"up_from":0,"seq":176093659294,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":747372544,"data_stored":747372544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1833500672,"data_stored":1833500672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1081606144,"data_stored":1081606144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1831600128,"data_stored":1831600128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":728760320,"data_stored":728760320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":886702080,"data_stored":886702080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1785856000,"data_stored":1785856000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:31:10.807 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-05T07:31:10.808 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pg_num 31 2024-02-05T07:31:10.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:10 smithi007 ceph-mon[29098]: pgmap v644: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 25 MiB/s wr, 401 op/s; 13734/169116 objects degraded (8.121%); 8164/169116 objects misplaced (4.827%); 7.5 MiB/s, 120 objects/s recovering 2024-02-05T07:31:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:10 smithi007 ceph-mon[37158]: pgmap v644: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 25 MiB/s wr, 401 op/s; 13734/169116 objects degraded (8.121%); 8164/169116 objects misplaced (4.827%); 7.5 MiB/s, 120 objects/s recovering 2024-02-05T07:31:10.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:10.862 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:10.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:10.937 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 55 16 59018 59002 67.0361 40.6875 0.0271169 0.0149082 2024-02-05T07:31:10.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:10.990 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:10.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:10.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:10 smithi090 ceph-mon[33213]: pgmap v644: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 25 MiB/s wr, 401 op/s; 13734/169116 objects degraded (8.121%); 8164/169116 objects misplaced (4.827%); 7.5 MiB/s, 120 objects/s recovering 2024-02-05T07:31:11.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:31:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:31:11.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:11.115 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:11.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:11.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:11.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:11.268 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:11.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:11.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:11.393 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:11.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:11.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:11.495 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:11.541 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:11.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:11.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:11.685 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:11.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:11.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:11.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:11.810 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:11.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:11.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:11.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:11.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:11 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:11.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:11 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:11.937 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 56 16 59514 59498 66.3925 31 0.00716932 0.0150547 2024-02-05T07:31:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:11.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:11.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:11 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:12.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:12.083 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:12.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:12.092 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:31:12.093 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:31:12.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:12.212 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:12.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:12.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:12.340 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:31:12.373 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:12.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:12.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:12.500 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:12.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:12.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:12.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:12.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:12.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:12.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:12.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:12.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:12.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:12.901 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:12.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:12.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:12 smithi007 ceph-mon[29098]: pgmap v645: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 38 MiB/s wr, 608 op/s; 13734/174072 objects degraded (7.890%); 7579/174072 objects misplaced (4.354%); 12 MiB/s, 193 objects/s recovering 2024-02-05T07:31:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:12 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:31:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:12 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:31:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:12 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:31:12.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:12 smithi007 ceph-mon[37158]: pgmap v645: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 38 MiB/s wr, 608 op/s; 13734/174072 objects degraded (7.890%); 7579/174072 objects misplaced (4.354%); 12 MiB/s, 193 objects/s recovering 2024-02-05T07:31:12.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:12 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:31:12.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:12 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:31:12.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:12 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:31:12.937 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 57 15 60058 60043 65.8253 34.0625 0.0282916 0.0151844 2024-02-05T07:31:13.003 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:12 smithi090 ceph-mon[33213]: pgmap v645: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 38 MiB/s wr, 608 op/s; 13734/174072 objects degraded (7.890%); 7579/174072 objects misplaced (4.354%); 12 MiB/s, 193 objects/s recovering 2024-02-05T07:31:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:12 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:31:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:12 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:31:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:12 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:31:13.028 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:13.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:13.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:13.153 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:13.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:13.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:13.279 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:13.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:13.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:13.381 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:13.429 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:13.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:13.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:13.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:13.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:13.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:13.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:13.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:13.682 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:13.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:13.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:13.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:13.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:13.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:13.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:13.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:13.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:31:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:31:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:31:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:13.938 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 58 16 60599 60583 65.2722 33.75 0.0276032 0.0153134 2024-02-05T07:31:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:13.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:14.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:14.086 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:14.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:14.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:14.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:14.233 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:14.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:14.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:14.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:14.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:14.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:14.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:14.499 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:14.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:14.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:14.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:14.624 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:14.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:14.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:14.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:14.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:14.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:14.896 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:14.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:14.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:14 smithi007 ceph-mon[29098]: pgmap v646: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 731 op/s; 13734/177570 objects degraded (7.734%); 7236/177570 objects misplaced (4.075%); 15 MiB/s, 236 objects/s recovering 2024-02-05T07:31:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:14 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 13734/177570 objects degraded (7.734%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:14.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:14 smithi007 ceph-mon[37158]: pgmap v646: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 731 op/s; 13734/177570 objects degraded (7.734%); 7236/177570 objects misplaced (4.075%); 15 MiB/s, 236 objects/s recovering 2024-02-05T07:31:14.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:14 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 13734/177570 objects degraded (7.734%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:14.937 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 59 16 61221 61205 64.8248 38.875 0.015327 0.015419 2024-02-05T07:31:14.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:14 smithi090 ceph-mon[33213]: pgmap v646: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 731 op/s; 13734/177570 objects degraded (7.734%); 7236/177570 objects misplaced (4.075%); 15 MiB/s, 236 objects/s recovering 2024-02-05T07:31:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:14 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 13734/177570 objects degraded (7.734%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:15.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:15.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:15.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:15.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:15.176 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:15.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:15.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:15.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:15.304 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:15.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:15.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:15.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:15.454 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:15.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:15.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:15.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:15.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:15.582 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:15.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:15.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:15.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:15.730 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:15.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:15.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:15.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:15.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:31:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:31:15.938 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:31:15.937834+0000 min lat: 0.00140221 max lat: 5.65097 avg lat: 0.015519 2024-02-05T07:31:15.938 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:31:15.938 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 60 16 61819 61803 64.3673 37.375 0.0819565 0.015519 2024-02-05T07:31:15.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:15.998 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:15.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:15.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:31:16.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:16.142 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:16.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:16.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:16.268 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:16.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:16.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:16.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:16.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:16.396 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:16.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:16.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:16.552 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:16.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:16.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:16.678 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:16.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:16.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:16.829 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:16.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:16 smithi007 ceph-mon[29098]: pgmap v647: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 590 B/s rd, 23 MiB/s wr, 366 op/s; 13734/178101 objects degraded (7.711%); 7236/178101 objects misplaced (4.063%); 6.7 MiB/s, 107 objects/s recovering 2024-02-05T07:31:16.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:16 smithi007 ceph-mon[37158]: pgmap v647: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 590 B/s rd, 23 MiB/s wr, 366 op/s; 13734/178101 objects degraded (7.711%); 7236/178101 objects misplaced (4.063%); 6.7 MiB/s, 107 objects/s recovering 2024-02-05T07:31:16.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:16.938 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 61 16 62330 62314 63.8357 31.9375 0.0725225 0.0156563 2024-02-05T07:31:16.957 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:16.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:16 smithi090 ceph-mon[33213]: pgmap v647: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 590 B/s rd, 23 MiB/s wr, 366 op/s; 13734/178101 objects degraded (7.711%); 7236/178101 objects misplaced (4.063%); 6.7 MiB/s, 107 objects/s recovering 2024-02-05T07:31:17.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:17.086 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:17.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:17.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:17.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:17.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:17.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:17.357 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:17.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:17.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:17.464 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:31:17.483 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:17.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:17.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:17.540 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:31:17.580 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:31:17.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:17.610 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:17.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:17.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:17.633 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:31:17.671 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:31:17.711 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:31:17.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:17.752 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:31:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:17.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:17.789 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:31:17.798 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:31:17.798 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:31:17.798 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:31:17.798 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:31:17.798 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:31:17.798 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:31:17.798 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:31:17.798 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:31:17.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:17.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:17.888 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:17.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:17.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:17 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2238344883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:31:17.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:17 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2238344883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:31:17.938 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 62 16 62892 62876 63.3726 35.125 0.0142272 0.0157736 2024-02-05T07:31:17.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:17 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2238344883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:31:18.027 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:18.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:18.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:18.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:18.168 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:18.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:18.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:18.318 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:18.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:18.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:18.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:18.444 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:18.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:18.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:18.570 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:18.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:18.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:18.718 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:18.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:18.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:18.820 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:18.843 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:18.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:18 smithi007 ceph-mon[29098]: pgmap v648: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 36 MiB/s wr, 581 op/s; 13734/186012 objects degraded (7.383%); 6287/186012 objects misplaced (3.380%); 12 MiB/s, 187 objects/s recovering 2024-02-05T07:31:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:18 smithi007 ceph-mon[37158]: pgmap v648: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 36 MiB/s wr, 581 op/s; 13734/186012 objects degraded (7.383%); 6287/186012 objects misplaced (3.380%); 12 MiB/s, 187 objects/s recovering 2024-02-05T07:31:18.938 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 63 16 63515 63499 62.9847 38.9375 0.0147959 0.0158702 2024-02-05T07:31:18.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:18.970 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:18.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:18.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:18 smithi090 ceph-mon[33213]: pgmap v648: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 36 MiB/s wr, 581 op/s; 13734/186012 objects degraded (7.383%); 6287/186012 objects misplaced (3.380%); 12 MiB/s, 187 objects/s recovering 2024-02-05T07:31:19.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:19.117 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:19.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:19.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:19.269 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:19.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:19.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:19.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:19.423 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:19.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:19.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:19.550 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:19.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:19.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:19.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:19.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:19 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 13734/186012 objects degraded (7.383%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:19.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:19 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 13734/186012 objects degraded (7.383%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:19.676 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:19.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:19.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:19.804 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:19.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:19.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:19.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:19.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:19.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:19.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:19.938 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 64 16 64052 64036 62.5249 33.5625 0.00392714 0.0159595 2024-02-05T07:31:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:19 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 13734/186012 objects degraded (7.383%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:20.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:20.056 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:20.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:20.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:20.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:20.184 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:20.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:20.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:20.286 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:20.312 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:20.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:20.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:20.414 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:20.457 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:20.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:20.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:20.584 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:20.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:20.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:20.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:20.712 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:20.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:20.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:20.857 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:20.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:20 smithi007 ceph-mon[29098]: pgmap v649: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 879 B/s rd, 32 MiB/s wr, 514 op/s; 13734/186537 objects degraded (7.363%); 6287/186537 objects misplaced (3.370%); 10 MiB/s, 161 objects/s recovering 2024-02-05T07:31:20.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:20 smithi007 ceph-mon[37158]: pgmap v649: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 879 B/s rd, 32 MiB/s wr, 514 op/s; 13734/186537 objects degraded (7.363%); 6287/186537 objects misplaced (3.370%); 10 MiB/s, 161 objects/s recovering 2024-02-05T07:31:20.938 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 65 16 64635 64619 62.1236 36.4375 0.0321986 0.0160883 2024-02-05T07:31:20.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:20.986 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:20.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:20.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:20 smithi090 ceph-mon[33213]: pgmap v649: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 879 B/s rd, 32 MiB/s wr, 514 op/s; 13734/186537 objects degraded (7.363%); 6287/186537 objects misplaced (3.370%); 10 MiB/s, 161 objects/s recovering 2024-02-05T07:31:21.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:31:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:31:21.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:21.115 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:21.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:21.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:21.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:21.269 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:21.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:21.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:21.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:21.394 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:21.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:21.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:21.521 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:21.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:21.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:21.623 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:21.665 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:21.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:21.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:21.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:21.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:21.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:21.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:21.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:21 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:21.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:21 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:21.936 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:21.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:21.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:21.938 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 66 16 65244 65228 61.759 38.0625 0.0213501 0.0161852 2024-02-05T07:31:22.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:21 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:22.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:22.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:22.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:22.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:22.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:22.207 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:22.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:22.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:22.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:22.335 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:22.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:22.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:22.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:22.474 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:22.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:22.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:22.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:22.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:22.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:22.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:22.742 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:22.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:22.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:22.868 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:22.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:22.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:22.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:22 smithi007 ceph-mon[29098]: pgmap v650: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 38 MiB/s wr, 616 op/s; 13734/191295 objects degraded (7.179%); 5567/191295 objects misplaced (2.910%); 14 MiB/s, 216 objects/s recovering 2024-02-05T07:31:22.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:22 smithi007 ceph-mon[37158]: pgmap v650: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 38 MiB/s wr, 616 op/s; 13734/191295 objects degraded (7.179%); 5567/191295 objects misplaced (2.910%); 14 MiB/s, 216 objects/s recovering 2024-02-05T07:31:22.938 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 67 16 65809 65793 61.3642 35.3125 0.00517283 0.0162658 2024-02-05T07:31:22.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:22.997 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:22.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:22 smithi090 ceph-mon[33213]: pgmap v650: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 38 MiB/s wr, 616 op/s; 13734/191295 objects degraded (7.179%); 5567/191295 objects misplaced (2.910%); 14 MiB/s, 216 objects/s recovering 2024-02-05T07:31:23.098 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:23.137 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:23.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:23.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:23.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:23.262 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:23.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:23.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:23.390 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:23.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:23.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:23.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:23.518 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:23.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:23.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:23.644 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:23.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:23.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:23.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:23.798 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:23.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:23.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:23.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:23.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:31:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:31:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:31:23.927 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:23.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:23.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:23.938 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 68 16 66289 66273 60.903 30 0.00910328 0.016413 2024-02-05T07:31:24.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:24.053 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:24.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:24.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:24.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:24.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:24.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:24.309 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:24.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:24.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:24.436 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:24.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:24.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:24.590 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:24.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:24.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:24.733 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:24.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:24.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:24.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:24.860 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:24.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:24.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:24 smithi007 ceph-mon[29098]: pgmap v651: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 571 op/s; 13734/194604 objects degraded (7.057%); 5102/194604 objects misplaced (2.622%); 13 MiB/s, 206 objects/s recovering 2024-02-05T07:31:24.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:24 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 13734/194604 objects degraded (7.057%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:24.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:24 smithi007 ceph-mon[37158]: pgmap v651: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 571 op/s; 13734/194604 objects degraded (7.057%); 5102/194604 objects misplaced (2.622%); 13 MiB/s, 206 objects/s recovering 2024-02-05T07:31:24.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:24 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 13734/194604 objects degraded (7.057%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:24.938 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 69 16 66955 66939 60.6236 41.625 0.0303904 0.0164836 2024-02-05T07:31:24.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:24.986 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:24.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:24 smithi090 ceph-mon[33213]: pgmap v651: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 571 op/s; 13734/194604 objects degraded (7.057%); 5102/194604 objects misplaced (2.622%); 13 MiB/s, 206 objects/s recovering 2024-02-05T07:31:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:24 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 13734/194604 objects degraded (7.057%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:25.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:25.113 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:25.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:25.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:25.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:25.259 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:25.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:25.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:25.402 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:25.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:25.504 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:25.557 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:25.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:25.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:25.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:25.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:25.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:25.834 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:25.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:25.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:25.936 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:25.939 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 70 16 67697 67681 60.42 46.375 0.00810009 0.0165442 2024-02-05T07:31:25.982 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:25.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:26.084 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:26.135 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:26.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:26.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:26.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:26.288 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:26.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:26.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:26.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:26.434 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:26.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:26.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:26.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:26.563 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:26.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:26.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:26.708 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:26.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:26.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:26.854 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:26.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:26.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:26 smithi007 ceph-mon[37158]: pgmap v652: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 30 MiB/s wr, 488 op/s; 13734/195135 objects degraded (7.038%); 5102/195135 objects misplaced (2.615%); 11 MiB/s, 177 objects/s recovering 2024-02-05T07:31:26.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:26 smithi007 ceph-mon[29098]: pgmap v652: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 30 MiB/s wr, 488 op/s; 13734/195135 objects degraded (7.038%); 5102/195135 objects misplaced (2.615%); 11 MiB/s, 177 objects/s recovering 2024-02-05T07:31:26.939 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 71 16 68428 68412 60.2124 45.6875 0.0451976 0.0166003 2024-02-05T07:31:26.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:26.981 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:26.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:27.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:26 smithi090 ceph-mon[33213]: pgmap v652: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 30 MiB/s wr, 488 op/s; 13734/195135 objects degraded (7.038%); 5102/195135 objects misplaced (2.615%); 11 MiB/s, 177 objects/s recovering 2024-02-05T07:31:27.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:27.129 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:27.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:27.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:27.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:27.256 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:27.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:27.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:27.358 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:27.402 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:27.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:27.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:27.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:27.548 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:27.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:27.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:27.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:27.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:27.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:27.816 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:27.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:27.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:27.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:27 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:31:27.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:27 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:31:27.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:27 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:31:27.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:27 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:31:27.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:27.939 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 72 16 69184 69168 60.0323 47.25 0.023792 0.0166517 2024-02-05T07:31:27.945 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:27.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:27 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:31:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:27 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:31:28.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:28.089 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:28.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:28.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:28.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:28.216 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:28.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:28.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:28.318 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:28.374 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:28.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:28.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:28.496 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:31:28.500 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:28.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:28.602 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:28.626 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:28.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:28.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:28.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:28.753 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:28.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:28.855 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:28.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:28.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:28 smithi007 ceph-mon[29098]: pgmap v653: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 714 op/s; 13734/203805 objects degraded (6.739%); 3744/203805 objects misplaced (1.837%); 18 MiB/s, 290 objects/s recovering 2024-02-05T07:31:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:28 smithi007 ceph-mon[29098]: osdmap e377: 8 total, 8 up, 6 in 2024-02-05T07:31:28.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:28 smithi007 ceph-mon[37158]: pgmap v653: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 714 op/s; 13734/203805 objects degraded (6.739%); 3744/203805 objects misplaced (1.837%); 18 MiB/s, 290 objects/s recovering 2024-02-05T07:31:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:28 smithi007 ceph-mon[37158]: osdmap e377: 8 total, 8 up, 6 in 2024-02-05T07:31:28.939 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 73 16 69905 69889 59.8272 45.0625 0.0294528 0.016708 2024-02-05T07:31:28.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:29.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:29.007 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:29.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:28 smithi090 ceph-mon[33213]: pgmap v653: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 714 op/s; 13734/203805 objects degraded (6.739%); 3744/203805 objects misplaced (1.837%); 18 MiB/s, 290 objects/s recovering 2024-02-05T07:31:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:28 smithi090 ceph-mon[33213]: osdmap e377: 8 total, 8 up, 6 in 2024-02-05T07:31:29.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:29.136 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:29.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:29.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:29.262 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:29.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:29.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:29.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:29.390 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:29.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:29.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:29.518 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:29.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:29.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:29.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:29.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:29.659 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:29.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:29.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:29.788 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:29.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:29.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:29.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:29.917 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:29.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:29.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:29.939 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 74 16 70571 70555 59.5812 41.625 0.0128874 0.0167772 2024-02-05T07:31:30.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:30.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:29 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 13734/203805 objects degraded (6.739%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:29 smithi090 ceph-mon[33213]: osdmap e378: 8 total, 8 up, 6 in 2024-02-05T07:31:30.065 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:30.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:30.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:30.121 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:29 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 13734/203805 objects degraded (6.739%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:30.121 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:29 smithi007 ceph-mon[29098]: osdmap e378: 8 total, 8 up, 6 in 2024-02-05T07:31:30.121 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:29 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 13734/203805 objects degraded (6.739%), 5 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:30.121 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:29 smithi007 ceph-mon[37158]: osdmap e378: 8 total, 8 up, 6 in 2024-02-05T07:31:30.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:30.234 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:30.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:30.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:30.362 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:30.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:30.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:30.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:30.493 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:30.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:30.595 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:30.620 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:30.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:30.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:30.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:30.759 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:30.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:30.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:30.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:30.884 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:30.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:30.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:30.939 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 75 16 71130 71114 59.2526 34.9375 0.00683733 0.0168535 2024-02-05T07:31:30.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:31.012 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:31.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:31.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:31:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:31:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:30 smithi090 ceph-mon[33213]: pgmap v655: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 37 MiB/s wr, 597 op/s; 13734/204450 objects degraded (6.718%); 3744/204450 objects misplaced (1.831%); 16 MiB/s, 254 objects/s recovering 2024-02-05T07:31:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:31:31.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:30 smithi007 ceph-mon[29098]: pgmap v655: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 37 MiB/s wr, 597 op/s; 13734/204450 objects degraded (6.718%); 3744/204450 objects misplaced (1.831%); 16 MiB/s, 254 objects/s recovering 2024-02-05T07:31:31.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:31:31.091 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:30 smithi007 ceph-mon[37158]: pgmap v655: 168 pgs: 5 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 37 MiB/s wr, 597 op/s; 13734/204450 objects degraded (6.718%); 3744/204450 objects misplaced (1.831%); 16 MiB/s, 254 objects/s recovering 2024-02-05T07:31:31.091 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:31:31.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:31.140 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:31.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:31.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:31.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:31.270 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:31.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:31.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:31.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:31.395 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:31.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:31.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:31.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:31.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:31.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:31.649 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:31.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:31.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:31.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:31.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:31.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:31.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:31.918 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:31.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:31.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:31.939 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 76 16 71484 71468 58.7641 22.125 0.00668743 0.0169948 2024-02-05T07:31:32.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:31 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:31 smithi090 ceph-mon[33213]: osdmap e379: 8 total, 8 up, 6 in 2024-02-05T07:31:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:31 smithi090 ceph-mon[33213]: pgmap v658: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 53 MiB/s wr, 850 op/s; 13734/210444 objects degraded (6.526%); 2651/210444 objects misplaced (1.260%); 26 MiB/s, 408 objects/s recovering 2024-02-05T07:31:32.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:32.044 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:32.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:32.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:32.169 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:32.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:32.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:31 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:32.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:31 smithi007 ceph-mon[29098]: osdmap e379: 8 total, 8 up, 6 in 2024-02-05T07:31:32.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:31 smithi007 ceph-mon[29098]: pgmap v658: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 53 MiB/s wr, 850 op/s; 13734/210444 objects degraded (6.526%); 2651/210444 objects misplaced (1.260%); 26 MiB/s, 408 objects/s recovering 2024-02-05T07:31:32.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:31 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:32.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:31 smithi007 ceph-mon[37158]: osdmap e379: 8 total, 8 up, 6 in 2024-02-05T07:31:32.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:31 smithi007 ceph-mon[37158]: pgmap v658: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 53 MiB/s wr, 850 op/s; 13734/210444 objects degraded (6.526%); 2651/210444 objects misplaced (1.260%); 26 MiB/s, 408 objects/s recovering 2024-02-05T07:31:32.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:32.298 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:32.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:32.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:32.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:32.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:32.559 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:32.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:32.654 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:31:32.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:32.707 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:32.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:32.808 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:32.863 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:32.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:32.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:32.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:32 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:31:32.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:32 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:31:32.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:32 smithi007 ceph-mon[29098]: osdmap e380: 8 total, 8 up, 6 in 2024-02-05T07:31:32.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:32 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:31:32.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:32 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:31:32.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:32 smithi007 ceph-mon[37158]: osdmap e380: 8 total, 8 up, 6 in 2024-02-05T07:31:32.939 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 77 16 71653 71637 58.1382 10.5625 0.0182922 0.0171936 2024-02-05T07:31:32.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:33.007 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:33.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:33.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:32 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:31:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:32 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:31:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:32 smithi090 ceph-mon[33213]: osdmap e380: 8 total, 8 up, 6 in 2024-02-05T07:31:33.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:33.133 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:33.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:33.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:33.273 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:33.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:33.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:33.419 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:33.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:33.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:33.564 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:33.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:33.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:33.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:33.693 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:33.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:33.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:33.727 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:31:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:31:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:31:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pg_num to 31 2024-02-05T07:31:33.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:33.819 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:33.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:33.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:33.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:33.941 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 78 16 72313 72297 57.9205 41.25 0.0405283 0.0172586 2024-02-05T07:31:33.967 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:33.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:33.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:33 smithi090 ceph-mon[33213]: osdmap e381: 8 total, 8 up, 6 in 2024-02-05T07:31:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:33 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3034729521' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-02-05T07:31:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:33 smithi090 ceph-mon[33213]: pgmap v661: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 784 op/s; 10911/213849 objects degraded (5.102%); 4765/213849 objects misplaced (2.228%); 28 MiB/s, 450 objects/s recovering 2024-02-05T07:31:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:33 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:31:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:33 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:31:34.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:34.111 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:34.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:34.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:34.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:33 smithi007 ceph-mon[29098]: osdmap e381: 8 total, 8 up, 6 in 2024-02-05T07:31:34.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:33 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3034729521' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-02-05T07:31:34.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:33 smithi007 ceph-mon[29098]: pgmap v661: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 784 op/s; 10911/213849 objects degraded (5.102%); 4765/213849 objects misplaced (2.228%); 28 MiB/s, 450 objects/s recovering 2024-02-05T07:31:34.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:33 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:31:34.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:33 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:31:34.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:33 smithi007 ceph-mon[37158]: osdmap e381: 8 total, 8 up, 6 in 2024-02-05T07:31:34.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:33 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3034729521' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-02-05T07:31:34.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:33 smithi007 ceph-mon[37158]: pgmap v661: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 784 op/s; 10911/213849 objects degraded (5.102%); 4765/213849 objects misplaced (2.228%); 28 MiB/s, 450 objects/s recovering 2024-02-05T07:31:34.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:33 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:31:34.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:33 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:31:34.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:34.256 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:34.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:34.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:34.358 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:34.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:34.409 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:34.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:34.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:34.534 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:34.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:34.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:34.662 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:34.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:34.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:34.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:34.793 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:34.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:34.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:34.922 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:34.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:34.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:34.941 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 79 16 73070 73054 57.7862 47.3125 0.00551523 0.0172981 2024-02-05T07:31:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:34 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3034729521' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-02-05T07:31:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:34 smithi090 ceph-mon[33213]: osdmap e382: 8 total, 8 up, 6 in 2024-02-05T07:31:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:34 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 10911/213849 objects degraded (5.102%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:35.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:35.050 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:35.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:35.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:35.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:34 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3034729521' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-02-05T07:31:35.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:34 smithi007 ceph-mon[29098]: osdmap e382: 8 total, 8 up, 6 in 2024-02-05T07:31:35.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:34 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 10911/213849 objects degraded (5.102%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:35.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:34 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3034729521' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-02-05T07:31:35.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:34 smithi007 ceph-mon[37158]: osdmap e382: 8 total, 8 up, 6 in 2024-02-05T07:31:35.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:34 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 10911/213849 objects degraded (5.102%), 4 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:35.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:35.183 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:35.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:35.285 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:35.311 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:35.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:35.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:35.440 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:35.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:35.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:35.585 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:35.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:35.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:35.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:35.725 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:35.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:35.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:35.872 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:35.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:35.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:35.941 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:31:35.941085+0000 min lat: 0.00140221 max lat: 5.65097 avg lat: 0.0173599 2024-02-05T07:31:35.941 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:31:35.941 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 80 16 73721 73705 57.5724 40.6875 0.0453555 0.0173599 2024-02-05T07:31:35.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:36.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:35 smithi090 ceph-mon[33213]: pgmap v663: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 677 op/s; 10911/214335 objects degraded (5.091%); 4765/214335 objects misplaced (2.223%); 25 MiB/s, 398 objects/s recovering 2024-02-05T07:31:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:35 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-05T07:31:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:35 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-05T07:31:36.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:35 smithi007 ceph-mon[29098]: pgmap v663: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 677 op/s; 10911/214335 objects degraded (5.091%); 4765/214335 objects misplaced (2.223%); 25 MiB/s, 398 objects/s recovering 2024-02-05T07:31:36.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:35 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-05T07:31:36.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:35 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-05T07:31:36.036 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:35 smithi007 ceph-mon[37158]: pgmap v663: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 158 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 677 op/s; 10911/214335 objects degraded (5.091%); 4765/214335 objects misplaced (2.223%); 25 MiB/s, 398 objects/s recovering 2024-02-05T07:31:36.036 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:35 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-05T07:31:36.036 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:35 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-05T07:31:36.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:36.129 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:36.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:36.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:36.255 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:36.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:36.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:36.382 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:36.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:36.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:36.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:36.509 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:36.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:36.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:36.634 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:36.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:36.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:36.769 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:36.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:36.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:36.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:36.897 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:36.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:36.941 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 81 16 73947 73931 57.0361 14.125 0.0170366 0.0175145 2024-02-05T07:31:36.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:37.023 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:36 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-05T07:31:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:36 smithi090 ceph-mon[33213]: osdmap e383: 8 total, 8 up, 6 in 2024-02-05T07:31:37.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:37.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:37.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:37.150 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:37.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:37.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:36 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-05T07:31:37.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:36 smithi007 ceph-mon[29098]: osdmap e383: 8 total, 8 up, 6 in 2024-02-05T07:31:37.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:36 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-05T07:31:37.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:36 smithi007 ceph-mon[37158]: osdmap e383: 8 total, 8 up, 6 in 2024-02-05T07:31:37.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:37.281 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:37.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:37.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:37.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:37.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:37.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:37.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:37.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:37.534 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:37.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:37.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:37.676 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:37.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:37.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:37.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:37.825 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:37.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:37.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:37.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:37.941 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 82 16 74174 74158 56.5135 14.1875 0.0230411 0.0176875 2024-02-05T07:31:37.952 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:37.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:37.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:37.967 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:31:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:37 smithi090 ceph-mon[33213]: osdmap e384: 8 total, 8 up, 6 in 2024-02-05T07:31:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:37 smithi090 ceph-mon[33213]: pgmap v666: 178 pgs: 10 unknown, 6 peering, 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 155 active+clean; 4.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.1 KiB/s rd, 36 MiB/s wr, 581 op/s; 5513/221685 objects degraded (2.487%); 8551/221685 objects misplaced (3.857%); 22 MiB/s, 358 objects/s recovering 2024-02-05T07:31:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:37 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-05T07:31:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:37 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-05T07:31:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:37 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2099949277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:31:38.054 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:38.077 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:38.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:38.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:37 smithi007 ceph-mon[29098]: osdmap e384: 8 total, 8 up, 6 in 2024-02-05T07:31:38.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:37 smithi007 ceph-mon[29098]: pgmap v666: 178 pgs: 10 unknown, 6 peering, 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 155 active+clean; 4.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.1 KiB/s rd, 36 MiB/s wr, 581 op/s; 5513/221685 objects degraded (2.487%); 8551/221685 objects misplaced (3.857%); 22 MiB/s, 358 objects/s recovering 2024-02-05T07:31:38.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:37 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-05T07:31:38.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:37 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-05T07:31:38.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:37 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2099949277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:31:38.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:37 smithi007 ceph-mon[37158]: osdmap e384: 8 total, 8 up, 6 in 2024-02-05T07:31:38.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:37 smithi007 ceph-mon[37158]: pgmap v666: 178 pgs: 10 unknown, 6 peering, 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 155 active+clean; 4.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.1 KiB/s rd, 36 MiB/s wr, 581 op/s; 5513/221685 objects degraded (2.487%); 8551/221685 objects misplaced (3.857%); 22 MiB/s, 358 objects/s recovering 2024-02-05T07:31:38.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:37 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-05T07:31:38.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:37 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-05T07:31:38.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:37 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2099949277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:31:38.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:38.204 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:38.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:38.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:38.352 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:38.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:38.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:38.454 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:38.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:38.502 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:38.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:38.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:38.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:38.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:38.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:38.798 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:38.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:38.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:38.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:38.924 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:38.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:38.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:38.941 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 83 16 74821 74805 56.3198 40.4375 0.00285733 0.0177289 2024-02-05T07:31:39.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:39.053 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:39.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:39.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:39.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:39.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:38 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:31:39.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:38 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-05T07:31:39.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:38 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2099949277' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:31:39.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:38 smithi007 ceph-mon[29098]: osdmap e385: 8 total, 8 up, 6 in 2024-02-05T07:31:39.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:38 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:31:39.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:38 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-05T07:31:39.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:38 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2099949277' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:31:39.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:38 smithi007 ceph-mon[37158]: osdmap e385: 8 total, 8 up, 6 in 2024-02-05T07:31:39.197 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:39.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:38 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:31:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:38 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-05T07:31:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:38 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2099949277' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:31:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:38 smithi090 ceph-mon[33213]: osdmap e385: 8 total, 8 up, 6 in 2024-02-05T07:31:39.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:39.340 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:39.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:39.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:39.465 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:39.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:39.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:39.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:39.612 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:39.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:39.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:39.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:39.740 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:39.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:39.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:39.868 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:39.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:39.941 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 84 16 75444 75428 56.1129 38.9375 0.0107753 0.0178137 2024-02-05T07:31:39.969 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:40.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:40.018 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:40.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:40.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:40.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:40.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:39 smithi007 ceph-mon[29098]: osdmap e386: 8 total, 8 up, 6 in 2024-02-05T07:31:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:39 smithi007 ceph-mon[29098]: pgmap v669: 178 pgs: 10 unknown, 6 peering, 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 155 active+clean; 4.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 2.5 KiB/s rd, 41 MiB/s wr, 651 op/s; 5513/222120 objects degraded (2.482%); 8551/222120 objects misplaced (3.850%); 25 MiB/s, 403 objects/s recovering 2024-02-05T07:31:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:39 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 5513/221685 objects degraded (2.487%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:39 smithi007 ceph-mon[29098]: osdmap e387: 8 total, 8 up, 6 in 2024-02-05T07:31:40.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:39 smithi007 ceph-mon[37158]: osdmap e386: 8 total, 8 up, 6 in 2024-02-05T07:31:40.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:39 smithi007 ceph-mon[37158]: pgmap v669: 178 pgs: 10 unknown, 6 peering, 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 155 active+clean; 4.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 2.5 KiB/s rd, 41 MiB/s wr, 651 op/s; 5513/222120 objects degraded (2.482%); 8551/222120 objects misplaced (3.850%); 25 MiB/s, 403 objects/s recovering 2024-02-05T07:31:40.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:39 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 5513/221685 objects degraded (2.487%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:40.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:39 smithi007 ceph-mon[37158]: osdmap e387: 8 total, 8 up, 6 in 2024-02-05T07:31:40.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:40.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:39 smithi090 ceph-mon[33213]: osdmap e386: 8 total, 8 up, 6 in 2024-02-05T07:31:40.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:39 smithi090 ceph-mon[33213]: pgmap v669: 178 pgs: 10 unknown, 6 peering, 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 155 active+clean; 4.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 2.5 KiB/s rd, 41 MiB/s wr, 651 op/s; 5513/222120 objects degraded (2.482%); 8551/222120 objects misplaced (3.850%); 25 MiB/s, 403 objects/s recovering 2024-02-05T07:31:40.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:39 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 5513/221685 objects degraded (2.487%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T07:31:40.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:39 smithi090 ceph-mon[33213]: osdmap e387: 8 total, 8 up, 6 in 2024-02-05T07:31:40.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:40.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:40.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:40.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:40.422 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:40.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:40.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:40.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:40.549 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:40.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:40.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:40.651 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:40.676 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:40.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:40.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:40.804 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:40.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:40.905 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:40.943 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 85 16 76198 76182 56.0067 47.125 0.0437675 0.0178483 2024-02-05T07:31:40.946 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:40.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:40.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:41.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:31:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:31:41.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:41.081 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:41.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:41.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:41.227 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:41.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:41.329 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:41.352 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:41.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:41.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:41.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:41.480 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:41.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:41.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:41.607 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:41.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:41.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:41.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:41.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:41.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:41.885 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:41.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:41.942 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 86 16 76824 76808 55.8103 39.125 0.00283504 0.0179035 2024-02-05T07:31:41.987 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:42.022 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:42.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:42.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:42.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:42.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:41 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:42.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:41 smithi007 ceph-mon[29098]: osdmap e388: 8 total, 8 up, 6 in 2024-02-05T07:31:42.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:41 smithi007 ceph-mon[29098]: pgmap v672: 178 pgs: 6 unknown, 3 peering, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 161 active+clean; 4.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 4.5 KiB/s rd, 24 MiB/s wr, 393 op/s; 3/226338 objects degraded (0.001%); 13016/226338 objects misplaced (5.751%); 16 MiB/s, 261 objects/s recovering 2024-02-05T07:31:42.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:41 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:42.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:41 smithi007 ceph-mon[37158]: osdmap e388: 8 total, 8 up, 6 in 2024-02-05T07:31:42.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:41 smithi007 ceph-mon[37158]: pgmap v672: 178 pgs: 6 unknown, 3 peering, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 161 active+clean; 4.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 4.5 KiB/s rd, 24 MiB/s wr, 393 op/s; 3/226338 objects degraded (0.001%); 13016/226338 objects misplaced (5.751%); 16 MiB/s, 261 objects/s recovering 2024-02-05T07:31:42.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:42.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:41 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:42.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:41 smithi090 ceph-mon[33213]: osdmap e388: 8 total, 8 up, 6 in 2024-02-05T07:31:42.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:41 smithi090 ceph-mon[33213]: pgmap v672: 178 pgs: 6 unknown, 3 peering, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 161 active+clean; 4.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 4.5 KiB/s rd, 24 MiB/s wr, 393 op/s; 3/226338 objects degraded (0.001%); 13016/226338 objects misplaced (5.751%); 16 MiB/s, 261 objects/s recovering 2024-02-05T07:31:42.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:42.277 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:42.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:42.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:42.422 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:42.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:42.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:42.548 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:42.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:42.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:42.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:42.690 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:42.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:42.791 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:42.817 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:42.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:42.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:42.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:42.942 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 87 15 77496 77481 55.6523 42.0625 0.0166646 0.0179625 2024-02-05T07:31:42.946 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:42.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:43.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:43.072 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:43.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:43.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:43.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:42 smithi007 ceph-mon[29098]: osdmap e389: 8 total, 8 up, 6 in 2024-02-05T07:31:43.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:42 smithi007 ceph-mon[37158]: osdmap e389: 8 total, 8 up, 6 in 2024-02-05T07:31:43.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:43.200 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:43.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:43.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:43.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:42 smithi090 ceph-mon[33213]: osdmap e389: 8 total, 8 up, 6 in 2024-02-05T07:31:43.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:43.332 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:43.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:43.433 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:43.480 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:43.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:43.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:43.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:43.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:43.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:43.756 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:43.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:43.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:43.824 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:31:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:31:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:31:43.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:43.898 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:43.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:43.942 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 88 16 78274 78258 55.5717 48.5625 0.0106947 0.0179877 2024-02-05T07:31:44.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:44.044 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:44.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:44.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:43 smithi007 ceph-mon[29098]: pgmap v674: 178 pgs: 3 unknown, 5 peering, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 162 active+clean; 4.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 8.8 KiB/s rd, 34 MiB/s wr, 549 op/s; 3/229302 objects degraded (0.001%); 12485/229302 objects misplaced (5.445%); 22 MiB/s, 355 objects/s recovering 2024-02-05T07:31:44.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:43 smithi007 ceph-mon[37158]: pgmap v674: 178 pgs: 3 unknown, 5 peering, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 162 active+clean; 4.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 8.8 KiB/s rd, 34 MiB/s wr, 549 op/s; 3/229302 objects degraded (0.001%); 12485/229302 objects misplaced (5.445%); 22 MiB/s, 355 objects/s recovering 2024-02-05T07:31:44.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:44.174 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:44.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:44.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:43 smithi090 ceph-mon[33213]: pgmap v674: 178 pgs: 3 unknown, 5 peering, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 162 active+clean; 4.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 8.8 KiB/s rd, 34 MiB/s wr, 549 op/s; 3/229302 objects degraded (0.001%); 12485/229302 objects misplaced (5.445%); 22 MiB/s, 355 objects/s recovering 2024-02-05T07:31:44.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:44.300 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:44.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:44.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:44.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:44.443 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:44.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:44.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:44.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:44.591 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:44.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:44.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:44.739 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:44.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:44.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:44.841 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:44.888 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:44.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:44.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:44.942 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 89 16 79063 79047 55.5013 49.3125 0.0177534 0.0180117 2024-02-05T07:31:44.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:45.016 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:45.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:45.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:45.160 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:45.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:45.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:44 smithi007 ceph-mon[29098]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:31:45.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:44 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 3/229302 objects degraded (0.001%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-05T07:31:45.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:44 smithi007 ceph-mon[29098]: osdmap e390: 8 total, 8 up, 6 in 2024-02-05T07:31:45.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:44 smithi007 ceph-mon[37158]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:31:45.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:44 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 3/229302 objects degraded (0.001%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-05T07:31:45.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:44 smithi007 ceph-mon[37158]: osdmap e390: 8 total, 8 up, 6 in 2024-02-05T07:31:45.262 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:45.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:44 smithi090 ceph-mon[33213]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:31:45.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:44 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 3/229302 objects degraded (0.001%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-05T07:31:45.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:44 smithi090 ceph-mon[33213]: osdmap e390: 8 total, 8 up, 6 in 2024-02-05T07:31:45.289 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:45.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:45.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:45.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:45.454 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:45.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:45.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:45.600 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:45.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:45.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:45.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:45.743 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:45.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:45.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:45.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:45.871 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:45.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:45.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:45.942 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 90 1 79842 79841 55.436 49.625 0.0209544 0.0180325 2024-02-05T07:31:45.973 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:46.004 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:46.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:46.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:46.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:46.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:46.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:46.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:46.275 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:46.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:46.377 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:46.416 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:46.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:46.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:46.521 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:46.558 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:46.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:46.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:46.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:46 smithi007 ceph-mon[29098]: pgmap v676: 178 pgs: 1 unknown, 5 peering, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 164 active+clean; 4.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 7.2 KiB/s rd, 30 MiB/s wr, 489 op/s; 3/229974 objects degraded (0.001%); 12485/229974 objects misplaced (5.429%); 18 MiB/s, 289 objects/s recovering 2024-02-05T07:31:46.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:46 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:31:46.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:31:46.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:46 smithi007 ceph-mon[37158]: pgmap v676: 178 pgs: 1 unknown, 5 peering, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 164 active+clean; 4.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 7.2 KiB/s rd, 30 MiB/s wr, 489 op/s; 3/229974 objects degraded (0.001%); 12485/229974 objects misplaced (5.429%); 18 MiB/s, 289 objects/s recovering 2024-02-05T07:31:46.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:46 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:31:46.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:31:46.691 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:46.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:46.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:46 smithi090 ceph-mon[33213]: pgmap v676: 178 pgs: 1 unknown, 5 peering, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 164 active+clean; 4.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 7.2 KiB/s rd, 30 MiB/s wr, 489 op/s; 3/229974 objects degraded (0.001%); 12485/229974 objects misplaced (5.429%); 18 MiB/s, 289 objects/s recovering 2024-02-05T07:31:46.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:46 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:31:46.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:31:46.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:46.816 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:46.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:46.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:46.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:46.942 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:46.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:46.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:46.943 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total time run: 90.0155 2024-02-05T07:31:46.943 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total writes made: 79842 2024-02-05T07:31:46.944 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Write size: 65536 2024-02-05T07:31:46.944 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object size: 65536 2024-02-05T07:31:46.944 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Bandwidth (MB/sec): 55.4363 2024-02-05T07:31:46.944 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Bandwidth: 41.8658 2024-02-05T07:31:46.944 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max bandwidth (MB/sec): 203.062 2024-02-05T07:31:46.944 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min bandwidth (MB/sec): 0 2024-02-05T07:31:46.944 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average IOPS: 886 2024-02-05T07:31:46.944 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev IOPS: 669.861 2024-02-05T07:31:46.944 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max IOPS: 3249 2024-02-05T07:31:46.944 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min IOPS: 0 2024-02-05T07:31:46.944 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average Latency(s): 0.0180325 2024-02-05T07:31:46.944 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Latency(s): 0.0670315 2024-02-05T07:31:46.944 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max latency(s): 5.65097 2024-02-05T07:31:46.944 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min latency(s): 0.00140221 2024-02-05T07:31:46.957 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T07:31:47.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:47.069 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:47.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:47.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:47.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:47.196 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:47.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:47.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:47.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:31:47.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:47.357 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:47.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:47.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:47.445 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:31:47.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:47.483 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:47.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:47.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:47.498 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:31:47.504 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:31:47.505 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:31:47.547 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:31:47.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:47.587 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:31:47.629 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:31:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:47.653 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:47.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:47.667 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:31:47.706 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:31:47.706 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:47 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2228864987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:31:47.706 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:47 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2228864987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:31:47.744 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:31:47.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:31:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:31:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:31:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:31:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:31:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:31:47.757 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:31:47.757 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:31:47.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:47 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2228864987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:31:47.778 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:47.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:47.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:47.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:47.904 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:47.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:47.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:48.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:48.032 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:48.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:48.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:48.163 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:48.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:48.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:48.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:48.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T07:31:48.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:48.299 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:48.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:48.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:48.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:48.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:48.550 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:48.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:48.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:48.659 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:48 smithi007 ceph-mon[29098]: pgmap v677: 178 pgs: 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 171 active+clean; 4.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 6.7 KiB/s rd, 55 MiB/s wr, 893 op/s; 3/239643 objects degraded (0.001%); 10992/239643 objects misplaced (4.587%); 25 MiB/s, 407 objects/s recovering 2024-02-05T07:31:48.659 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:48 smithi007 ceph-mon[29098]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-05T07:31:48.659 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:48 smithi007 ceph-mon[29098]: osdmap e391: 8 total, 8 up, 6 in 2024-02-05T07:31:48.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:48 smithi007 ceph-mon[37158]: pgmap v677: 178 pgs: 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 171 active+clean; 4.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 6.7 KiB/s rd, 55 MiB/s wr, 893 op/s; 3/239643 objects degraded (0.001%); 10992/239643 objects misplaced (4.587%); 25 MiB/s, 407 objects/s recovering 2024-02-05T07:31:48.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:48 smithi007 ceph-mon[37158]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-05T07:31:48.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:48 smithi007 ceph-mon[37158]: osdmap e391: 8 total, 8 up, 6 in 2024-02-05T07:31:48.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:48.695 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:48.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:48.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:48 smithi090 ceph-mon[33213]: pgmap v677: 178 pgs: 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 171 active+clean; 4.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 6.7 KiB/s rd, 55 MiB/s wr, 893 op/s; 3/239643 objects degraded (0.001%); 10992/239643 objects misplaced (4.587%); 25 MiB/s, 407 objects/s recovering 2024-02-05T07:31:48.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:48 smithi090 ceph-mon[33213]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-05T07:31:48.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:48 smithi090 ceph-mon[33213]: osdmap e391: 8 total, 8 up, 6 in 2024-02-05T07:31:48.795 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:48.839 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:48.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:48.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:48.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:48.967 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:48.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:48.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:49.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:49.112 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:49.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:49.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:49.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:49.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:49.264 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:31:49.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:49.405 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:49.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:49.438 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:31:49.505 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:49.530 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:49.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:49.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:49.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:49.656 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:49.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:49.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:49.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:49 smithi007 ceph-mon[29098]: osdmap e392: 8 total, 8 up, 6 in 2024-02-05T07:31:49.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:49 smithi007 ceph-mon[37158]: osdmap e392: 8 total, 8 up, 6 in 2024-02-05T07:31:49.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:49.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:49 smithi090 ceph-mon[33213]: osdmap e392: 8 total, 8 up, 6 in 2024-02-05T07:31:49.784 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:49.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:49.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:49.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:50.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:50.064 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:50.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:50.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:50.210 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:50.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:50.311 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:50.353 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:50.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:50.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:50.409 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 3] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:31:50.409 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:31:50.409 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-02-05T07:31:50.409 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-05T07:31:50.410 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-05T07:31:50.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-02-05T07:31:50.455 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:50.482 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:50.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:50.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:50.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:50.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:50.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:50.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:50.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:50 smithi007 ceph-mon[29098]: pgmap v680: 178 pgs: 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 171 active+clean; 4.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.7 KiB/s rd, 37 MiB/s wr, 603 op/s; 3/239913 objects degraded (0.001%); 10992/239913 objects misplaced (4.582%); 16 MiB/s, 248 objects/s recovering 2024-02-05T07:31:50.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:50 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 3/239913 objects degraded (0.001%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-05T07:31:50.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:50 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:31:50.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:50 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:31:50.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:50 smithi007 ceph-mon[37158]: pgmap v680: 178 pgs: 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 171 active+clean; 4.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.7 KiB/s rd, 37 MiB/s wr, 603 op/s; 3/239913 objects degraded (0.001%); 10992/239913 objects misplaced (4.582%); 16 MiB/s, 248 objects/s recovering 2024-02-05T07:31:50.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:50 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 3/239913 objects degraded (0.001%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-05T07:31:50.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:50 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:31:50.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:50 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:31:50.691 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:31:50.705 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:50 smithi090 ceph-mon[33213]: pgmap v680: 178 pgs: 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 171 active+clean; 4.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.7 KiB/s rd, 37 MiB/s wr, 603 op/s; 3/239913 objects degraded (0.001%); 10992/239913 objects misplaced (4.582%); 16 MiB/s, 248 objects/s recovering 2024-02-05T07:31:50.705 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:50 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 3/239913 objects degraded (0.001%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-05T07:31:50.705 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:50 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:31:50.705 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:50 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:31:50.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:50.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:50.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:50.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:50.876 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:50.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:50.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:50.963 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:31:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:31:50.979 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:51.026 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:51.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:51.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:51.159 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:51.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:51.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:51.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:31:51.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:51.286 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:51.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:51.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:51.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:51.413 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:51.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:51.514 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:51.540 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:51.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:51.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:51.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:51.666 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:51.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:51.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:51.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:51 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:31:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:51 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:31:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:51 smithi090 ceph-mon[33213]: osdmap e393: 8 total, 8 up, 6 in 2024-02-05T07:31:51.812 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:51.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:51.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:51.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:51 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:31:51.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:51 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:31:51.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:51 smithi007 ceph-mon[29098]: osdmap e393: 8 total, 8 up, 6 in 2024-02-05T07:31:51.856 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:51 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:31:51.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:51 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:31:51.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:51 smithi007 ceph-mon[37158]: osdmap e393: 8 total, 8 up, 6 in 2024-02-05T07:31:51.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:51.938 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:51.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:51.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:52.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:52.079 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:52.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:52.181 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:52.207 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:52.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:52.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:52.353 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:52.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:52.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:52.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:52.496 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:52.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:52.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:52.591 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:31:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-05T07:31:52.591 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:31:52.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:52.629 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:52.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:52.730 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:52.771 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:52.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:52.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:52 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:52.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:52 smithi090 ceph-mon[33213]: pgmap v682: 178 pgs: 5 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 171 active+clean; 4.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.7 KiB/s rd, 35 MiB/s wr, 615 op/s; 3/239217 objects degraded (0.001%); 10425/239217 objects misplaced (4.358%); 21 MiB/s, 343 objects/s recovering 2024-02-05T07:31:52.872 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:52.899 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:52.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:52.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:52 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:52.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:52 smithi007 ceph-mon[29098]: pgmap v682: 178 pgs: 5 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 171 active+clean; 4.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.7 KiB/s rd, 35 MiB/s wr, 615 op/s; 3/239217 objects degraded (0.001%); 10425/239217 objects misplaced (4.358%); 21 MiB/s, 343 objects/s recovering 2024-02-05T07:31:52.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:52 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:31:52.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:52 smithi007 ceph-mon[37158]: pgmap v682: 178 pgs: 5 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 171 active+clean; 4.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.7 KiB/s rd, 35 MiB/s wr, 615 op/s; 3/239217 objects degraded (0.001%); 10425/239217 objects misplaced (4.358%); 21 MiB/s, 343 objects/s recovering 2024-02-05T07:31:53.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:53.040 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:53.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:53.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:53.179 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:53.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:53.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:53.281 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:53.307 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:53.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:53.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:53.451 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:53.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:53.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:53.579 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:53.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:53.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:53.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:53.708 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:53.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:53.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:53.854 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:53.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:53.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:31:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:31:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:31:53.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:53.997 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:53.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:54.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:54.124 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:54.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:54.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:54.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:54.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:54.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:54.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:54.405 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:54.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:54.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:54.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:54.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:54.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:54.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:54.682 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:54.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:54.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:54.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:54 smithi090 ceph-mon[33213]: pgmap v683: 178 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 172 active+clean; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 1.6 MiB/s wr, 480 op/s; 3/231957 objects degraded (0.001%); 9209/231957 objects misplaced (3.970%); 19 MiB/s, 297 objects/s recovering 2024-02-05T07:31:54.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:54 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 3/231957 objects degraded (0.001%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-05T07:31:54.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:54 smithi090 ceph-mon[33213]: osdmap e394: 8 total, 8 up, 6 in 2024-02-05T07:31:54.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:54.831 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:54.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:54.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:54.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:54 smithi007 ceph-mon[29098]: pgmap v683: 178 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 172 active+clean; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 1.6 MiB/s wr, 480 op/s; 3/231957 objects degraded (0.001%); 9209/231957 objects misplaced (3.970%); 19 MiB/s, 297 objects/s recovering 2024-02-05T07:31:54.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:54 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 3/231957 objects degraded (0.001%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-05T07:31:54.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:54 smithi007 ceph-mon[29098]: osdmap e394: 8 total, 8 up, 6 in 2024-02-05T07:31:54.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:54 smithi007 ceph-mon[37158]: pgmap v683: 178 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 172 active+clean; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 1.6 MiB/s wr, 480 op/s; 3/231957 objects degraded (0.001%); 9209/231957 objects misplaced (3.970%); 19 MiB/s, 297 objects/s recovering 2024-02-05T07:31:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:54 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 3/231957 objects degraded (0.001%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-05T07:31:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:54 smithi007 ceph-mon[37158]: osdmap e394: 8 total, 8 up, 6 in 2024-02-05T07:31:54.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:54.980 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:54.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:55.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:55.105 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:55.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:55.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:55.207 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:55.258 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:55.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:55.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:55.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:55.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:55.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:55.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:55.515 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:55.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:55.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:55.617 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:55.640 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:55.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:55.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:55.742 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:55.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:55.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:31:55.893 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:55.918 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:55.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:55.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:56.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:56.045 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:56.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:56.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:56.147 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:56.193 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:56.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:56.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:56.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:56.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:56.330 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:56.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:56.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:56.457 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:56.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:56.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:56.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:56.585 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:56.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:56.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:56.712 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:56.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:56.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:56 smithi090 ceph-mon[33213]: pgmap v685: 178 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 172 active+clean; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 906 B/s rd, 1.5 MiB/s wr, 456 op/s; 3/231327 objects degraded (0.001%); 9209/231327 objects misplaced (3.981%); 16 MiB/s, 263 objects/s recovering 2024-02-05T07:31:56.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:56.858 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:56.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:56.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:56.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:56 smithi007 ceph-mon[29098]: pgmap v685: 178 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 172 active+clean; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 906 B/s rd, 1.5 MiB/s wr, 456 op/s; 3/231327 objects degraded (0.001%); 9209/231327 objects misplaced (3.981%); 16 MiB/s, 263 objects/s recovering 2024-02-05T07:31:56.935 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:56 smithi007 ceph-mon[37158]: pgmap v685: 178 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 172 active+clean; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 906 B/s rd, 1.5 MiB/s wr, 456 op/s; 3/231327 objects degraded (0.001%); 9209/231327 objects misplaced (3.981%); 16 MiB/s, 263 objects/s recovering 2024-02-05T07:31:56.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:57.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:57.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:57.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:57.143 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:57.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:57.244 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:57.270 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:57.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:57.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:57.372 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:57.417 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:57.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:57.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:57.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:57.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:57.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:57.672 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:57.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:57.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:57.817 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:57.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:57.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:57.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:57.966 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:57.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:57.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:58.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:31:58.108 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:58.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:58.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:31:58.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:58.253 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:58.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:58.256 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 3] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:31:58.256 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:31:58.256 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-05T07:31:58.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T07:31:58.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:31:58.465 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:58.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:58.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:31:58.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:58.592 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:58 smithi007 ceph-mon[29098]: pgmap v686: 178 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 172 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.4 KiB/s rd, 256 KiB/s wr, 835 op/s; 3/220089 objects degraded (0.001%); 7885/220089 objects misplaced (3.583%); 24 MiB/s, 388 objects/s recovering 2024-02-05T07:31:58.593 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:58 smithi007 ceph-mon[37158]: pgmap v686: 178 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 172 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.4 KiB/s rd, 256 KiB/s wr, 835 op/s; 3/220089 objects degraded (0.001%); 7885/220089 objects misplaced (3.583%); 24 MiB/s, 388 objects/s recovering 2024-02-05T07:31:58.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:58.594 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:58.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:58.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:58.721 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:58.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:58.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:58.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:31:58.848 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:58.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:58.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:31:58.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:58.975 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:58.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:58.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:59.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:58 smithi090 ceph-mon[33213]: pgmap v686: 178 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 172 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.4 KiB/s rd, 256 KiB/s wr, 835 op/s; 3/220089 objects degraded (0.001%); 7885/220089 objects misplaced (3.583%); 24 MiB/s, 388 objects/s recovering 2024-02-05T07:31:59.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:59.122 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:59.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:59.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:59.224 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:31:59.250 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:59.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:59.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:31:59.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:59.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:59.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:59.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:31:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:59.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:59.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:31:59.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:31:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:59.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:59.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:31:59.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:31:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:31:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:59.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:31:59.882 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:31:59.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:31:59 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 3/220089 objects degraded (0.001%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-05T07:31:59.896 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:31:59 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 3/220089 objects degraded (0.001%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-05T07:31:59.932 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:31:59.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:31:59.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:00.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:31:59 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 3/220089 objects degraded (0.001%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:00.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:00.059 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:00.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:00.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:00.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:00.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:00.199 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:00.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:00.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:00.349 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:00.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:00.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:00.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:00.497 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:00.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:00.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:00.638 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:00.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:00.646 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:32:00.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:00.765 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:00.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:00.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:00.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:00 smithi007 ceph-mon[29098]: pgmap v687: 178 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 172 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 779 op/s; 3/219321 objects degraded (0.001%); 7885/219321 objects misplaced (3.595%); 18 MiB/s, 290 objects/s recovering 2024-02-05T07:32:00.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:00 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:32:00.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:00 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:32:00.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:00 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:32:00.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:00 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:32:00.869 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:00 smithi007 ceph-mon[37158]: pgmap v687: 178 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 172 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 779 op/s; 3/219321 objects degraded (0.001%); 7885/219321 objects misplaced (3.595%); 18 MiB/s, 290 objects/s recovering 2024-02-05T07:32:00.869 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:00 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:32:00.869 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:00 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:32:00.869 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:00 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:32:00.869 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:00 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:32:00.892 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:00.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:00.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:01.017 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:01.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:01.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:32:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:32:01.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:00 smithi090 ceph-mon[33213]: pgmap v687: 178 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 172 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 779 op/s; 3/219321 objects degraded (0.001%); 7885/219321 objects misplaced (3.595%); 18 MiB/s, 290 objects/s recovering 2024-02-05T07:32:01.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:00 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:32:01.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:00 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:32:01.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:00 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:32:01.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:00 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:32:01.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:01.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:01.144 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:01.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:01.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:01.271 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:01.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:01.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:01.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:01.398 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:01.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:01.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:01.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:01.523 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:01.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:01.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:01.567 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:32:01.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:01.650 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:01.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:01.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:01.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:01.778 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:01.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:01.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:01.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:01 smithi007 ceph-mon[29098]: osdmap e395: 8 total, 8 up, 6 in 2024-02-05T07:32:01.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:01 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1262130331' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:32:01.839 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:01 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:32:01.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:01 smithi007 ceph-mon[37158]: osdmap e395: 8 total, 8 up, 6 in 2024-02-05T07:32:01.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:01 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1262130331' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:32:01.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:01 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:32:01.879 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:01.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:01.921 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:01.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:02.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:02.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:01 smithi090 ceph-mon[33213]: osdmap e395: 8 total, 8 up, 6 in 2024-02-05T07:32:02.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:01 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1262130331' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:32:02.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:01 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:32:02.060 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:02.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:02.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:02.162 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:02.187 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:02.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:02.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:02.289 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:02.332 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:02.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:02.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:02.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:02.460 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:02.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:02.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:02.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:02.588 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:02.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:02.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:02.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:02.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:02.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:02.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:02.844 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:02.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:02.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:02 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:02.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:02 smithi007 ceph-mon[29098]: pgmap v689: 178 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 172 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 593 op/s; 1/217731 objects degraded (0.000%); 7227/217731 objects misplaced (3.319%); 16 MiB/s, 247 objects/s recovering 2024-02-05T07:32:02.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:02 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:32:02.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:02 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:32:02.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:02 smithi007 ceph-mon[29098]: osdmap e396: 8 total, 8 up, 6 in 2024-02-05T07:32:02.919 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:02 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:02.919 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:02 smithi007 ceph-mon[37158]: pgmap v689: 178 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 172 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 593 op/s; 1/217731 objects degraded (0.000%); 7227/217731 objects misplaced (3.319%); 16 MiB/s, 247 objects/s recovering 2024-02-05T07:32:02.919 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:02 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:32:02.919 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:02 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:32:02.919 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:02 smithi007 ceph-mon[37158]: osdmap e396: 8 total, 8 up, 6 in 2024-02-05T07:32:02.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:03.003 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:03.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:03.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:03.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:02 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:03.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:02 smithi090 ceph-mon[33213]: pgmap v689: 178 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 172 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 593 op/s; 1/217731 objects degraded (0.000%); 7227/217731 objects misplaced (3.319%); 16 MiB/s, 247 objects/s recovering 2024-02-05T07:32:03.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:02 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:32:03.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:02 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:32:03.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:02 smithi090 ceph-mon[33213]: osdmap e396: 8 total, 8 up, 6 in 2024-02-05T07:32:03.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:03.166 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:03.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:03.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:03.268 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:03.310 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:03.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:03.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:03.412 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:03.438 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:03.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:03.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:03.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:03.566 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:03.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:03.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:03.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:03.709 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:03.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:03.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:03.835 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:03.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:03.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:03.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:32:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:32:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:32:03.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:03.961 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:03.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:03.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:04.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:04.088 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:04.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:04.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:04.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:04.216 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:04.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:04.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:04.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:04.342 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:04.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:04.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:04.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:04.469 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:04.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:04.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:04.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:04.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:04.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:04.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:04.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:04.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:04.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:04 smithi007 ceph-mon[29098]: pgmap v691: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovering+undersized+remapped, 172 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 930 op/s; 1/209019 objects degraded (0.000%); 6130/209019 objects misplaced (2.933%); 24 MiB/s, 384 objects/s recovering 2024-02-05T07:32:04.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:04 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 1/209019 objects degraded (0.000%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:04.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:04 smithi007 ceph-mon[37158]: pgmap v691: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovering+undersized+remapped, 172 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 930 op/s; 1/209019 objects degraded (0.000%); 6130/209019 objects misplaced (2.933%); 24 MiB/s, 384 objects/s recovering 2024-02-05T07:32:04.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:04 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 1/209019 objects degraded (0.000%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:04.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:04.890 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:04.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:04.991 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:05.023 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:05.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:05.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:04 smithi090 ceph-mon[33213]: pgmap v691: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovering+undersized+remapped, 172 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 930 op/s; 1/209019 objects degraded (0.000%); 6130/209019 objects misplaced (2.933%); 24 MiB/s, 384 objects/s recovering 2024-02-05T07:32:05.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:04 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 1/209019 objects degraded (0.000%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:05.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:05.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:05.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:05.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:05.300 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:05.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:05.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:05.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:05.427 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:05.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:05.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:05.554 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:05.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:05.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:05.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:05.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:05.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:05.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:05.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:05.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:05.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:05.948 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:05.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:05.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:06.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:06.092 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:06.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:06.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:06.237 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:06.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:06.339 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:06.382 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:06.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:06.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:06.526 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:06.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:06.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:06.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:06.770 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:06.810 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:06.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:06.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:06.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:06 smithi007 ceph-mon[29098]: pgmap v692: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovering+undersized+remapped, 172 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 486 op/s; 1/208413 objects degraded (0.000%); 6130/208413 objects misplaced (2.941%); 14 MiB/s, 219 objects/s recovering 2024-02-05T07:32:06.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:06 smithi007 ceph-mon[37158]: pgmap v692: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovering+undersized+remapped, 172 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 486 op/s; 1/208413 objects degraded (0.000%); 6130/208413 objects misplaced (2.941%); 14 MiB/s, 219 objects/s recovering 2024-02-05T07:32:06.953 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:06.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:07.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:06 smithi090 ceph-mon[33213]: pgmap v692: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovering+undersized+remapped, 172 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 486 op/s; 1/208413 objects degraded (0.000%); 6130/208413 objects misplaced (2.941%); 14 MiB/s, 219 objects/s recovering 2024-02-05T07:32:07.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:07.081 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:07.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:07.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:07.226 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:07.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:07.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:07.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:07.354 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:07.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:07.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:07.480 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:07.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:07.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:07.607 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:07.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:07.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:07.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:07.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:07.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:07.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:07.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:07.896 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:07.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:07.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:07.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:08.021 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:08.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:08.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:08.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:08.146 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:08.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:08.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:08.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:08.288 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:08.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:08.390 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:08.414 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:08.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:08.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:08.562 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:08.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:08.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:08.663 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:08.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:08 smithi007 ceph-mon[29098]: pgmap v693: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 913 op/s; 1/197421 objects degraded (0.001%); 5240/197421 objects misplaced (2.654%); 21 MiB/s, 330 objects/s recovering 2024-02-05T07:32:08.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:08 smithi007 ceph-mon[37158]: pgmap v693: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 913 op/s; 1/197421 objects degraded (0.001%); 5240/197421 objects misplaced (2.654%); 21 MiB/s, 330 objects/s recovering 2024-02-05T07:32:08.704 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:08.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:08.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:08.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:08.842 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:08.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:08.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:08.970 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:08.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:08 smithi090 ceph-mon[33213]: pgmap v693: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 913 op/s; 1/197421 objects degraded (0.001%); 5240/197421 objects misplaced (2.654%); 21 MiB/s, 330 objects/s recovering 2024-02-05T07:32:09.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:09.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:09.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:09.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:09.120 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:32:09.120 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:32:09.120 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-02-05T07:32:09.120 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-02-05T07:32:09.120 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:32:09.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:09.222 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:09.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:09.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:09.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:09.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:09.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:09.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:09.492 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:09.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:09.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:09.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:09.618 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:09.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:09.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:09.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:09.759 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:09.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:09.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:09.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:09.896 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:09.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:09.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:09.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:09 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 1/197421 objects degraded (0.001%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:09.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:09 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 1/197421 objects degraded (0.001%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:09.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:10.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:09 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 1/197421 objects degraded (0.001%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:10.041 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:10.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:10.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:10.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:10.169 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:10.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:10.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:10.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:10.296 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:10.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:10.398 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:10.440 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:10.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:10.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:10.567 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:10.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:10.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:10.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:10.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:10.699 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:10.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:10.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:10.824 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:10.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:10.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:10.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:10 smithi007 ceph-mon[29098]: pgmap v694: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 839 op/s; 1/196725 objects degraded (0.001%); 5240/196725 objects misplaced (2.664%); 14 MiB/s, 229 objects/s recovering 2024-02-05T07:32:10.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:10 smithi007 ceph-mon[37158]: pgmap v694: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 839 op/s; 1/196725 objects degraded (0.001%); 5240/196725 objects misplaced (2.664%); 14 MiB/s, 229 objects/s recovering 2024-02-05T07:32:10.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:10.965 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:10.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:10.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:10 smithi090 ceph-mon[33213]: pgmap v694: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 839 op/s; 1/196725 objects degraded (0.001%); 5240/196725 objects misplaced (2.664%); 14 MiB/s, 229 objects/s recovering 2024-02-05T07:32:11.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:32:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:32:11.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:11.113 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:11.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:11.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:11.257 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:11.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:11.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:11.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:11.383 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:11.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:11.485 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:11.511 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:11.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:11.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:11.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:11.653 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:11.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:11.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:11.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:11.797 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:11.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:11.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:11.899 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:11.939 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:11.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:12.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:12.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:12.066 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:12.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:12.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:12.191 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:12.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:12.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:12.319 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:12.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:12.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:12.421 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:12.465 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:12.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:12.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:12.591 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:12.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:12.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:12.719 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:12.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:12.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:12.821 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:12.869 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:12.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:12 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:12 smithi007 ceph-mon[29098]: pgmap v695: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 750 op/s; 1/195237 objects degraded (0.001%); 5240/195237 objects misplaced (2.684%); 12 MiB/s, 198 objects/s recovering 2024-02-05T07:32:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:12 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:32:12.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:12 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:12.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:12 smithi007 ceph-mon[37158]: pgmap v695: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 750 op/s; 1/195237 objects degraded (0.001%); 5240/195237 objects misplaced (2.684%); 12 MiB/s, 198 objects/s recovering 2024-02-05T07:32:12.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:12 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:32:12.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:13.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:13.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:12 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:12 smithi090 ceph-mon[33213]: pgmap v695: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 750 op/s; 1/195237 objects degraded (0.001%); 5240/195237 objects misplaced (2.684%); 12 MiB/s, 198 objects/s recovering 2024-02-05T07:32:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:12 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:32:13.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:13.172 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:13.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:13.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:13.325 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:13.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:13.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:13.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:13.454 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:13.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:13.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:13.581 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:13.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:13.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:13.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:32:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:32:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:32:13.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:13.707 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:13.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:13.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:13.833 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:13.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:13.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:13.935 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:13.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:14.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:14.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:14.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:14.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:14.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:14.225 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:14.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:14.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:14.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:14.352 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:14.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:14.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:14.481 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:14.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:14.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:14.622 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:14.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:14.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:14.749 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:14.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:14.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:14.896 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:14.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:14.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:14 smithi007 ceph-mon[29098]: pgmap v696: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 787 op/s; 1/185313 objects degraded (0.001%); 4190/185313 objects misplaced (2.261%); 12 MiB/s, 199 objects/s recovering 2024-02-05T07:32:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:14 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:32:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:14 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:32:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:14 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:32:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:14 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 1/185313 objects degraded (0.001%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:14.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:14 smithi007 ceph-mon[37158]: pgmap v696: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 787 op/s; 1/185313 objects degraded (0.001%); 4190/185313 objects misplaced (2.261%); 12 MiB/s, 199 objects/s recovering 2024-02-05T07:32:14.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:14 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:32:14.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:14 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:32:14.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:14 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:32:14.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:14 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 1/185313 objects degraded (0.001%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:14.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:14 smithi090 ceph-mon[33213]: pgmap v696: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 787 op/s; 1/185313 objects degraded (0.001%); 4190/185313 objects misplaced (2.261%); 12 MiB/s, 199 objects/s recovering 2024-02-05T07:32:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:14 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:32:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:14 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:32:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:14 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:32:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:14 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 1/185313 objects degraded (0.001%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:15.036 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:15.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:15.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:15.164 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:15.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:15.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:15.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:15.296 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:15.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:15.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:15.398 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:15.421 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:15.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:15.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:15.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:15.550 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:15.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:15.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:15.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:15.678 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:15.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:15.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:15.821 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:15.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:15.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:15.844 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:32:15.915 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:15 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:32:15.915 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:15 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:32:15.915 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:32:15.915 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:15 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:32:15.915 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:15 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:32:15.916 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:32:15.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:15.963 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:15.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:15.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:15 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:32:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:15 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:32:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:32:16.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:16.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:16.092 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:16.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:16.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:16.243 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:16.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:16.347 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:16.387 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:16.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:16.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:16.514 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:16.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:16.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:16.642 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:16.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:16.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:16 smithi007 ceph-mon[29098]: pgmap v697: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 3.7 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 682 op/s; 1/184485 objects degraded (0.001%); 4190/184485 objects misplaced (2.271%); 10 MiB/s, 161 objects/s recovering 2024-02-05T07:32:16.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:16 smithi007 ceph-mon[29098]: osdmap e397: 8 total, 8 up, 6 in 2024-02-05T07:32:16.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:16 smithi007 ceph-mon[37158]: pgmap v697: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 3.7 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 682 op/s; 1/184485 objects degraded (0.001%); 4190/184485 objects misplaced (2.271%); 10 MiB/s, 161 objects/s recovering 2024-02-05T07:32:16.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:16 smithi007 ceph-mon[37158]: osdmap e397: 8 total, 8 up, 6 in 2024-02-05T07:32:16.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:16.770 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:16.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:16.872 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:16.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:16.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:16.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:16 smithi090 ceph-mon[33213]: pgmap v697: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 174 active+clean; 3.7 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 682 op/s; 1/184485 objects degraded (0.001%); 4190/184485 objects misplaced (2.271%); 10 MiB/s, 161 objects/s recovering 2024-02-05T07:32:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:16 smithi090 ceph-mon[33213]: osdmap e397: 8 total, 8 up, 6 in 2024-02-05T07:32:17.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T07:32:17.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:17.078 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:17.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:17.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:17.204 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:17.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:17.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:17.354 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:17.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:17.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:17.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:17.531 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:17.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:17.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:17.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:17.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:17.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:17.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:17.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:17.815 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:17.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:17.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:17.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:18.050 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:18.076 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:18.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:18.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:18.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:18.205 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:18.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:18.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:18.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:18.332 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:18.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:18.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:18.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:18.458 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:18.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:18.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:32:18.560 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:18.613 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:18.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:18.715 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:18.768 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:18.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:18.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:18.906 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:18.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:18 smithi007 ceph-mon[29098]: pgmap v699: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 174 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 865 op/s; 108/171486 objects degraded (0.063%); 2862/171486 objects misplaced (1.669%); 14 MiB/s, 227 objects/s recovering 2024-02-05T07:32:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:18 smithi007 ceph-mon[29098]: osdmap e398: 8 total, 8 up, 6 in 2024-02-05T07:32:18.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:18 smithi007 ceph-mon[37158]: pgmap v699: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 174 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 865 op/s; 108/171486 objects degraded (0.063%); 2862/171486 objects misplaced (1.669%); 14 MiB/s, 227 objects/s recovering 2024-02-05T07:32:18.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:18 smithi007 ceph-mon[37158]: osdmap e398: 8 total, 8 up, 6 in 2024-02-05T07:32:18.991 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:32:19.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:19.021 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:32:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:18 smithi090 ceph-mon[33213]: pgmap v699: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 174 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 865 op/s; 108/171486 objects degraded (0.063%); 2862/171486 objects misplaced (1.669%); 14 MiB/s, 227 objects/s recovering 2024-02-05T07:32:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:18 smithi090 ceph-mon[33213]: osdmap e398: 8 total, 8 up, 6 in 2024-02-05T07:32:19.032 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:19.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:19.044 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:32:19.068 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:32:19.091 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:32:19.113 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:32:19.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:19.136 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:32:19.161 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:19.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:19.163 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:32:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:32:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:32:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:32:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:32:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:32:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:32:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:32:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:32:19.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:19.290 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:19.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:19.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:19.416 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:19.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:19.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:19.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:19.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:19.554 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:19.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:19.644 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:32:19.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:19.680 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:19.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:19.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:19.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:19 smithi007 ceph-mon[29098]: osdmap e399: 8 total, 8 up, 6 in 2024-02-05T07:32:19.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:19 smithi007 ceph-mon[29098]: 1.10 scrub starts 2024-02-05T07:32:19.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:19 smithi007 ceph-mon[29098]: 1.10 scrub ok 2024-02-05T07:32:19.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:19 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3934711867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:32:19.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:19 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 108/171486 objects degraded (0.063%), 2 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:19.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:19 smithi007 ceph-mon[29098]: osdmap e400: 8 total, 8 up, 6 in 2024-02-05T07:32:19.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:19 smithi007 ceph-mon[37158]: osdmap e399: 8 total, 8 up, 6 in 2024-02-05T07:32:19.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:19 smithi007 ceph-mon[37158]: 1.10 scrub starts 2024-02-05T07:32:19.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:19 smithi007 ceph-mon[37158]: 1.10 scrub ok 2024-02-05T07:32:19.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:19 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3934711867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:32:19.736 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:19 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 108/171486 objects degraded (0.063%), 2 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:19.736 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:19 smithi007 ceph-mon[37158]: osdmap e400: 8 total, 8 up, 6 in 2024-02-05T07:32:19.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:19.807 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:19.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:19.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:19.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:19.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:19.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:19.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:19 smithi090 ceph-mon[33213]: osdmap e399: 8 total, 8 up, 6 in 2024-02-05T07:32:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:19 smithi090 ceph-mon[33213]: 1.10 scrub starts 2024-02-05T07:32:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:19 smithi090 ceph-mon[33213]: 1.10 scrub ok 2024-02-05T07:32:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:19 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3934711867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:32:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:19 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 108/171486 objects degraded (0.063%), 2 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:19 smithi090 ceph-mon[33213]: osdmap e400: 8 total, 8 up, 6 in 2024-02-05T07:32:20.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:20.084 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:20.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:20.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:20.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:20.211 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:20.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:20.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:32:20.328 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:32:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 400, 2024-02-05T07:32:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "e3e7059e-c3f5-11ee-95b6-87774f69a715", 2024-02-05T07:32:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: "created": "2024-02-05T07:13:14.603885+0000", 2024-02-05T07:32:20.329 INFO:teuthology.orchestra.run.smithi007.stdout: "modified": "2024-02-05T07:32:19.492334+0000", 2024-02-05T07:32:20.329 INFO:teuthology.orchestra.run.smithi007.stdout: "last_up_change": "2024-02-05T07:28:24.555976+0000", 2024-02-05T07:32:20.329 INFO:teuthology.orchestra.run.smithi007.stdout: "last_in_change": "2024-02-05T07:30:39.451715+0000", 2024-02-05T07:32:20.329 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T07:32:20.329 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_num": 5804032, 2024-02-05T07:32:20.329 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_set": [ 2024-02-05T07:32:20.329 INFO:teuthology.orchestra.run.smithi007.stdout: "nodeep-scrub", 2024-02-05T07:32:20.329 INFO:teuthology.orchestra.run.smithi007.stdout: "pglog_hardlimit", 2024-02-05T07:32:20.329 INFO:teuthology.orchestra.run.smithi007.stdout: "purged_snapdirs", 2024-02-05T07:32:20.329 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_deletes", 2024-02-05T07:32:20.329 INFO:teuthology.orchestra.run.smithi007.stdout: "sortbitwise" 2024-02-05T07:32:20.329 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_version": 19, 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_snapshot": "", 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_max": 9, 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd": 8, 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: "require_min_compat_client": "luminous", 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: "min_compat_client": "luminous", 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: "require_osd_release": "squid", 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_crimson": false, 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: "pools": [ 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 1, 2024-02-05T07:32:20.330 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".mgr", 2024-02-05T07:32:20.331 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:16:32.505567+0000", 2024-02-05T07:32:20.331 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:32:20.331 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:32:20.331 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:32:20.331 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:32:20.331 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:32:20.331 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:32:20.331 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:32:20.331 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:32:20.331 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:32:20.331 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:32:20.331 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:32:20.331 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:32:20.331 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 31, 2024-02-05T07:32:20.332 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 31, 2024-02-05T07:32:20.332 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 31, 2024-02-05T07:32:20.332 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 31, 2024-02-05T07:32:20.332 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 31, 2024-02-05T07:32:20.332 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:32:20.332 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:32:20.332 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:32:20.332 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:32:20.332 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:32:20.333 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:32:20.333 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:32:20.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.333 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "385", 2024-02-05T07:32:20.333 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:32:20.333 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:32:20.333 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "383", 2024-02-05T07:32:20.333 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:32:20.333 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:32:20.333 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:32:20.333 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:32:20.333 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:32:20.333 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:32:20.333 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:32:20.334 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:32:20.334 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:32:20.334 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:32:20.334 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:32:20.334 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:32:20.334 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:32:20.334 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:32:20.334 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:32:20.334 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:32:20.334 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:32:20.334 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:32:20.334 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:32:20.334 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:32:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:32:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:32:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:32:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:32:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:32:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:32:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:32:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:32:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:32:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:32:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:32:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:32:20.335 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:32:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:32:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-02-05T07:32:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_max": 32, 2024-02-05T07:32:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_min": 1 2024-02-05T07:32:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:32:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": {} 2024-02-05T07:32:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:32:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:32:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.5700000524520874, 2024-02-05T07:32:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.5700000524520874, 2024-02-05T07:32:20.336 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:32:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.3500000238418579, 2024-02-05T07:32:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.3500000238418579, 2024-02-05T07:32:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:32:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:32:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:32:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 2, 2024-02-05T07:32:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "datapool", 2024-02-05T07:32:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:47.259462+0000", 2024-02-05T07:32:20.337 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 8193, 2024-02-05T07:32:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T07:32:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:32:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:32:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:32:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:32:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:32:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:32:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:32:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:32:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:32:20.338 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:32:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 3, 2024-02-05T07:32:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 3, 2024-02-05T07:32:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 3, 2024-02-05T07:32:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 3, 2024-02-05T07:32:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 3, 2024-02-05T07:32:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:32:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:32:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:32:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:32:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:32:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:32:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:32:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.339 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "71", 2024-02-05T07:32:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:32:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:32:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:32:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:32:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:32:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 2, 2024-02-05T07:32:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 71, 2024-02-05T07:32:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:32:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:32:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:32:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:32:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:32:20.340 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:32:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:32:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:32:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:32:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:32:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:32:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:32:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:32:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:32:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:32:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:32:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:32:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:32:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:32:20.341 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:32:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:32:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:32:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:32:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:32:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:32:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:32:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:32:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:32:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:32:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:32:20.342 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:32:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:32:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd": {} 2024-02-05T07:32:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:32:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:32:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 4.690000057220459, 2024-02-05T07:32:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 4.690000057220459, 2024-02-05T07:32:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.70999997854232788, 2024-02-05T07:32:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 3.3299999237060547, 2024-02-05T07:32:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 3.3299999237060547, 2024-02-05T07:32:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:32:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:32:20.343 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:32:20.344 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.344 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.344 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.344 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 3, 2024-02-05T07:32:20.344 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".rgw.root", 2024-02-05T07:32:20.344 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:47.762336+0000", 2024-02-05T07:32:20.344 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:32:20.344 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:32:20.344 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:32:20.344 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:32:20.344 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:32:20.344 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:32:20.344 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:32:20.344 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:32:20.345 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:32:20.345 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:32:20.345 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:32:20.345 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:32:20.345 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:32:20.345 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:32:20.345 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:32:20.345 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:32:20.345 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:32:20.345 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:32:20.345 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:32:20.345 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:32:20.346 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:32:20.346 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:32:20.346 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:32:20.346 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:32:20.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "69", 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:32:20.347 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:32:20.348 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:32:20.348 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:32:20.348 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:32:20.348 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:32:20.348 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:32:20.348 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:32:20.348 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:32:20.348 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:32:20.348 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:32:20.348 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:32:20.349 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:32:20.349 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.349 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:32:20.349 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:32:20.349 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:32:20.349 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:32:20.349 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:32:20.349 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:32:20.349 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:32:20.349 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:32:20.349 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:32:20.349 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:32:20.349 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.4000000953674316, 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.4000000953674316, 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.059999942779541, 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.059999942779541, 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.350 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 4, 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.log", 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:49.736599+0000", 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:32:20.351 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:32:20.352 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:32:20.352 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:32:20.352 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:32:20.352 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:32:20.352 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:32:20.352 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:32:20.352 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:32:20.352 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:32:20.352 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:32:20.352 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:32:20.352 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:32:20.352 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:32:20.352 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.353 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "71", 2024-02-05T07:32:20.353 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:32:20.353 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:32:20.353 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:32:20.353 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:32:20.353 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:32:20.353 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:32:20.353 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:32:20.353 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:32:20.353 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:32:20.354 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:32:20.354 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:32:20.354 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:32:20.354 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:32:20.354 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:32:20.354 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:32:20.354 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:32:20.354 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:32:20.354 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:32:20.354 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:32:20.354 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:32:20.354 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:32:20.354 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:32:20.354 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:32:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:32:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:32:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:32:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:32:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:32:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:32:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:32:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:32:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:32:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:32:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:32:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:32:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:32:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:32:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:32:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:32:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:32:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:32:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:32:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.190000057220459, 2024-02-05T07:32:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.190000057220459, 2024-02-05T07:32:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:32:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.8799999952316284, 2024-02-05T07:32:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.8799999952316284, 2024-02-05T07:32:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:32:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 5, 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.control", 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:51.696948+0000", 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:32:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:32:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:32:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:32:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:32:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:32:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:32:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:32:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:32:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:32:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:32:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:32:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:32:20.359 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:32:20.359 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:32:20.359 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:32:20.359 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:32:20.359 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:32:20.359 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.359 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "73", 2024-02-05T07:32:20.359 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:32:20.359 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:32:20.359 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:32:20.359 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:32:20.359 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:32:20.360 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:32:20.360 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:32:20.360 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:32:20.360 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:32:20.360 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:32:20.360 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:32:20.360 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:32:20.360 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:32:20.360 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:32:20.360 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:32:20.360 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:32:20.360 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:32:20.360 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:32:20.360 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:32:20.361 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.7400000095367432, 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.7400000095367432, 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.5, 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.5, 2024-02-05T07:32:20.362 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:32:20.363 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:32:20.363 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:32:20.363 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.363 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.363 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.363 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 6, 2024-02-05T07:32:20.363 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.meta", 2024-02-05T07:32:20.363 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:53.768555+0000", 2024-02-05T07:32:20.363 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:32:20.363 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:32:20.363 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:32:20.364 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:32:20.364 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:32:20.364 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:32:20.364 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:32:20.364 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:32:20.364 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:32:20.364 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:32:20.364 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:32:20.364 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:32:20.364 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:32:20.364 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:32:20.364 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "75", 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:32:20.365 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:32:20.366 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:32:20.367 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:32:20.368 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:32:20.368 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-02-05T07:32:20.368 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_bias": 4 2024-02-05T07:32:20.368 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.368 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:32:20.368 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:32:20.368 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.368 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:32:20.368 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:32:20.368 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.190000057220459, 2024-02-05T07:32:20.368 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.190000057220459, 2024-02-05T07:32:20.368 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.8799999952316284, 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.8799999952316284, 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 9, 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "unique_pool_2", 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:29:59.438957+0000", 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:32:20.369 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:32:20.370 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:32:20.370 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:32:20.370 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:32:20.370 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:32:20.370 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:32:20.370 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:32:20.370 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:32:20.370 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 16, 2024-02-05T07:32:20.370 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 16, 2024-02-05T07:32:20.370 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 16, 2024-02-05T07:32:20.370 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 16, 2024-02-05T07:32:20.370 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 16, 2024-02-05T07:32:20.370 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:32:20.370 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "354", 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:32:20.371 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:32:20.372 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:32:20.373 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:32:20.374 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:32:20.374 INFO:teuthology.orchestra.run.smithi007.stdout: "rados": {} 2024-02-05T07:32:20.374 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.374 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:32:20.374 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:32:20.374 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 3.059999942779541, 2024-02-05T07:32:20.374 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 3.059999942779541, 2024-02-05T07:32:20.374 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:32:20.374 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.630000114440918, 2024-02-05T07:32:20.374 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.630000114440918, 2024-02-05T07:32:20.375 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:32:20.375 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:32:20.375 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:32:20.375 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.375 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.375 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:32:20.375 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:32:20.375 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.375 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-02-05T07:32:20.375 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "703c3948-8003-4015-9dd8-21019cf0eeb8", 2024-02-05T07:32:20.375 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:32:20.375 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:32:20.375 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:32:20.376 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0, 2024-02-05T07:32:20.376 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:32:20.376 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:32:20.376 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 8, 2024-02-05T07:32:20.376 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 387, 2024-02-05T07:32:20.376 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:32:20.376 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:32:20.376 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:32:20.376 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.376 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.376 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-02-05T07:32:20.376 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6802", 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.377 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.378 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6804", 2024-02-05T07:32:20.378 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:32:20.378 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.378 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.378 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:32:20.378 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.378 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.378 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6803", 2024-02-05T07:32:20.378 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:32:20.378 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.379 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.379 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6801/2069016689", 2024-02-05T07:32:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:20.379 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:20.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:20.380 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6802/2069016689", 2024-02-05T07:32:20.380 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6804/2069016689", 2024-02-05T07:32:20.380 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6803/2069016689", 2024-02-05T07:32:20.380 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:32:20.380 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:32:20.380 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "5c3e7d1e-a61d-44b9-b21e-f91c99a096bc", 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 13, 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 398, 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:32:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6805", 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6806", 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.382 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6808", 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6807", 2024-02-05T07:32:20.383 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:32:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6805/865486532", 2024-02-05T07:32:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6806/865486532", 2024-02-05T07:32:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6808/865486532", 2024-02-05T07:32:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6807/865486532", 2024-02-05T07:32:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:32:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:32:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:32:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-02-05T07:32:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "f4fd930f-7bb2-46ca-9964-98235bd47b3a", 2024-02-05T07:32:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:32:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:32:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:32:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:32:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:32:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:32:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 18, 2024-02-05T07:32:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 395, 2024-02-05T07:32:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:32:20.385 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6809", 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6810", 2024-02-05T07:32:20.386 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6812", 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.387 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.388 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.388 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6811", 2024-02-05T07:32:20.388 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:32:20.388 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.388 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.388 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6809/904871370", 2024-02-05T07:32:20.388 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6810/904871370", 2024-02-05T07:32:20.388 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6812/904871370", 2024-02-05T07:32:20.388 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6811/904871370", 2024-02-05T07:32:20.389 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:32:20.389 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:32:20.389 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:32:20.389 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.389 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.389 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-02-05T07:32:20.389 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "149ed0d6-b049-4478-9819-cafc446e94d3", 2024-02-05T07:32:20.389 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:32:20.389 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:32:20.389 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:32:20.389 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:32:20.389 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 26, 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 399, 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6813", 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.390 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6814", 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6816", 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:32:20.391 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6815", 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6813/2758586242", 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6814/2758586242", 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6816/2758586242", 2024-02-05T07:32:20.392 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6815/2758586242", 2024-02-05T07:32:20.393 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:32:20.393 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:32:20.393 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:32:20.393 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.393 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-02-05T07:32:20.393 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d", 2024-02-05T07:32:20.393 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:32:20.393 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:32:20.393 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:32:20.393 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:32:20.393 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:32:20.393 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:32:20.394 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 32, 2024-02-05T07:32:20.394 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 395, 2024-02-05T07:32:20.394 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:32:20.394 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:32:20.394 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:32:20.394 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.394 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.394 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6800", 2024-02-05T07:32:20.394 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:32:20.394 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.394 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.395 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:32:20.395 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.395 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.395 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6801", 2024-02-05T07:32:20.395 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:32:20.395 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.395 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:32:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6803", 2024-02-05T07:32:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:32:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:32:20.397 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.397 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.397 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.397 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6802", 2024-02-05T07:32:20.397 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:32:20.397 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.397 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.397 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6800/1154929470", 2024-02-05T07:32:20.397 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6801/1154929470", 2024-02-05T07:32:20.397 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6803/1154929470", 2024-02-05T07:32:20.397 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6802/1154929470", 2024-02-05T07:32:20.397 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:32:20.398 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:32:20.398 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:32:20.398 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.398 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-02-05T07:32:20.398 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "244a7930-c22c-413e-86e1-11e0749f7367", 2024-02-05T07:32:20.398 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:32:20.398 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-02-05T07:32:20.398 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-02-05T07:32:20.399 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0.7362823486328125, 2024-02-05T07:32:20.399 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:32:20.399 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:32:20.399 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 41, 2024-02-05T07:32:20.399 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 336, 2024-02-05T07:32:20.399 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:32:20.399 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:32:20.399 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:32:20.399 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.399 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.399 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6804", 2024-02-05T07:32:20.400 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:32:20.400 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.400 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.400 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:32:20.400 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.400 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.400 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6805", 2024-02-05T07:32:20.400 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:32:20.400 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.401 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.401 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:32:20.401 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.401 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.401 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6807", 2024-02-05T07:32:20.401 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:32:20.401 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.401 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.401 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:32:20.401 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.402 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.402 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.402 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6806", 2024-02-05T07:32:20.402 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:32:20.402 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.402 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.402 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6804/398113577", 2024-02-05T07:32:20.402 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6805/398113577", 2024-02-05T07:32:20.402 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6807/398113577", 2024-02-05T07:32:20.402 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6806/398113577", 2024-02-05T07:32:20.402 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:32:20.402 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:32:20.402 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:32:20.403 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.403 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.403 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-02-05T07:32:20.403 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7", 2024-02-05T07:32:20.403 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:32:20.403 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:32:20.403 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:32:20.403 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:32:20.403 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 52, 2024-02-05T07:32:20.403 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 230, 2024-02-05T07:32:20.404 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 301, 2024-02-05T07:32:20.404 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 388, 2024-02-05T07:32:20.404 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 231, 2024-02-05T07:32:20.404 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:32:20.404 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:32:20.404 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.404 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.404 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.404 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6808", 2024-02-05T07:32:20.404 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:32:20.404 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.404 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.405 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.405 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:32:20.405 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.405 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.405 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.405 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6809", 2024-02-05T07:32:20.405 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:32:20.405 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.405 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.405 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.405 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:32:20.406 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.406 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.406 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.406 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6811", 2024-02-05T07:32:20.406 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:32:20.406 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.406 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.406 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:32:20.406 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.406 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.407 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.407 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6810", 2024-02-05T07:32:20.407 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:32:20.407 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.407 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.407 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6808/313196533", 2024-02-05T07:32:20.407 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6809/313196533", 2024-02-05T07:32:20.407 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6811/313196533", 2024-02-05T07:32:20.407 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6810/313196533", 2024-02-05T07:32:20.407 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:32:20.407 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:32:20.408 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:32:20.408 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.408 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-02-05T07:32:20.408 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9", 2024-02-05T07:32:20.408 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:32:20.408 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-02-05T07:32:20.408 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-02-05T07:32:20.408 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 60, 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 349, 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6812", 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6813", 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6815", 2024-02-05T07:32:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:32:20.411 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.411 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.411 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:32:20.411 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:32:20.411 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.411 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:32:20.411 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6814", 2024-02-05T07:32:20.411 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:32:20.411 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.411 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.411 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6812/2983605980", 2024-02-05T07:32:20.411 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6813/2983605980", 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2983605980", 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6814/2983605980", 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_xinfo": [ 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:32:20.412 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:32:20.413 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:32:20.413 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:31:49.882095+0000", 2024-02-05T07:32:20.413 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:32:20.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.413 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.413 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-02-05T07:32:20.413 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:32:20.413 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:32:20.413 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:32:20.413 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:32:20.413 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:32:20.413 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:32:14.901479+0000", 2024-02-05T07:32:20.413 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:32:20.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:31:26.668702+0000", 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:32:20.414 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:32:20.415 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:32:20.415 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:31:50.052170+0000", 2024-02-05T07:32:20.415 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:32:20.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.415 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.415 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-02-05T07:32:20.415 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:32:20.415 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:32:20.415 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:32:20.415 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:32:20.415 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:32:20.415 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:31:33.467981+0000", 2024-02-05T07:32:20.415 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:32:20.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-02-05T07:32:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:32:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:32:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:32:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:32:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-02-05T07:32:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:31:05.781144+0000", 2024-02-05T07:32:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:32:20.416 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.417 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.417 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-02-05T07:32:20.417 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:32:20.417 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:32:20.417 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:32:20.417 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:32:20.417 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:32:20.417 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:32:00.252211+0000", 2024-02-05T07:32:20.417 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 230 2024-02-05T07:32:20.417 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.417 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.417 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-02-05T07:32:20.417 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:32:20.417 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:32:20.418 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:32:20.418 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:32:20.418 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-02-05T07:32:20.418 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:31:31.337017+0000", 2024-02-05T07:32:20.418 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:32:20.418 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.418 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:32:20.418 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap": [ 2024-02-05T07:32:20.418 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.418 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "5.15", 2024-02-05T07:32:20.418 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:32:20.419 INFO:teuthology.orchestra.run.smithi007.stdout: 2, 2024-02-05T07:32:20.419 INFO:teuthology.orchestra.run.smithi007.stdout: 0, 2024-02-05T07:32:20.419 INFO:teuthology.orchestra.run.smithi007.stdout: 1 2024-02-05T07:32:20.419 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.419 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.419 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:32:20.419 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_items": [ 2024-02-05T07:32:20.419 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.419 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.3", 2024-02-05T07:32:20.419 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:32:20.419 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.419 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 4, 2024-02-05T07:32:20.419 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:32:20.419 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.420 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.420 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.420 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.420 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.1a", 2024-02-05T07:32:20.420 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:32:20.420 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.420 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 4, 2024-02-05T07:32:20.420 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 0 2024-02-05T07:32:20.420 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.420 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.420 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.420 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.420 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.4", 2024-02-05T07:32:20.420 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 0 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.9", 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.422 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.a", 2024-02-05T07:32:20.422 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:32:20.422 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.422 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:32:20.422 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:32:20.422 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.422 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.422 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.422 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.422 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.13", 2024-02-05T07:32:20.422 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:32:20.422 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.422 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:32:20.422 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_primaries": [], 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_temp": [ 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "9.7", 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: 3, 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: 2 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:32:20.423 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "9.8", 2024-02-05T07:32:20.424 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:32:20.424 INFO:teuthology.orchestra.run.smithi007.stdout: 3, 2024-02-05T07:32:20.424 INFO:teuthology.orchestra.run.smithi007.stdout: 0 2024-02-05T07:32:20.424 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:32:20.424 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.424 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:32:20.424 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_temp": [], 2024-02-05T07:32:20.424 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklist": { 2024-02-05T07:32:20.424 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2180087772": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:32:20.424 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2066685086": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:32:20.424 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/4100020626": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:32:20.424 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1953351182": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:32:20.424 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3203189162": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:32:20.424 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3418781614": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:32:20.425 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1518592084": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:32:20.425 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/3380493069": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:32:20.425 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2208590106": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:32:20.425 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/223579766": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:32:20.425 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/2634895574": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:32:20.425 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1462812719": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:32:20.425 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2823976078": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:32:20.425 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1905520582": "2024-02-06T07:13:47.004765+0000" 2024-02-05T07:32:20.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.425 INFO:teuthology.orchestra.run.smithi007.stdout: "range_blocklist": {}, 2024-02-05T07:32:20.425 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profiles": { 2024-02-05T07:32:20.425 INFO:teuthology.orchestra.run.smithi007.stdout: "default": { 2024-02-05T07:32:20.425 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-failure-domain": "osd", 2024-02-05T07:32:20.425 INFO:teuthology.orchestra.run.smithi007.stdout: "k": "2", 2024-02-05T07:32:20.426 INFO:teuthology.orchestra.run.smithi007.stdout: "m": "1", 2024-02-05T07:32:20.426 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin": "jerasure", 2024-02-05T07:32:20.426 INFO:teuthology.orchestra.run.smithi007.stdout: "technique": "reed_sol_van" 2024-02-05T07:32:20.426 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:32:20.426 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps_queue": [], 2024-02-05T07:32:20.426 INFO:teuthology.orchestra.run.smithi007.stdout: "new_removed_snaps": [], 2024-02-05T07:32:20.426 INFO:teuthology.orchestra.run.smithi007.stdout: "new_purged_snaps": [], 2024-02-05T07:32:20.426 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_node_flags": {}, 2024-02-05T07:32:20.427 INFO:teuthology.orchestra.run.smithi007.stdout: "device_class_flags": {}, 2024-02-05T07:32:20.427 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode": { 2024-02-05T07:32:20.427 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_enabled": false, 2024-02-05T07:32:20.427 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_bucket_count": 0, 2024-02-05T07:32:20.427 INFO:teuthology.orchestra.run.smithi007.stdout: "degraded_stretch_mode": 0, 2024-02-05T07:32:20.427 INFO:teuthology.orchestra.run.smithi007.stdout: "recovering_stretch_mode": 0, 2024-02-05T07:32:20.427 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_bucket": 0 2024-02-05T07:32:20.427 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:32:20.427 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:32:20.480 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:20.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:20.506 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:20.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:20.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:20.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:20.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:20.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:20.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:20.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:20.908 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:20.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:20.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:20 smithi007 ceph-mon[29098]: pgmap v702: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 174 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 811 op/s; 108/170715 objects degraded (0.063%); 2862/170715 objects misplaced (1.676%); 13 MiB/s, 203 objects/s recovering 2024-02-05T07:32:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:20 smithi007 ceph-mon[29098]: 1.0 scrub starts 2024-02-05T07:32:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:20 smithi007 ceph-mon[29098]: 1.0 scrub ok 2024-02-05T07:32:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:20 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/430425946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:32:20.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:20 smithi007 ceph-mon[37158]: pgmap v702: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 174 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 811 op/s; 108/170715 objects degraded (0.063%); 2862/170715 objects misplaced (1.676%); 13 MiB/s, 203 objects/s recovering 2024-02-05T07:32:20.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:20 smithi007 ceph-mon[37158]: 1.0 scrub starts 2024-02-05T07:32:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:20 smithi007 ceph-mon[37158]: 1.0 scrub ok 2024-02-05T07:32:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:20 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/430425946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:32:21.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:21.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:32:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:32:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:20 smithi090 ceph-mon[33213]: pgmap v702: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 174 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 811 op/s; 108/170715 objects degraded (0.063%); 2862/170715 objects misplaced (1.676%); 13 MiB/s, 203 objects/s recovering 2024-02-05T07:32:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:20 smithi090 ceph-mon[33213]: 1.0 scrub starts 2024-02-05T07:32:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:20 smithi090 ceph-mon[33213]: 1.0 scrub ok 2024-02-05T07:32:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:20 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/430425946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:32:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:21.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:21.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:21.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:21.183 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:21.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:21.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:21.310 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:21.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:21.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:21.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:21.438 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:21.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:21.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:21.564 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:21.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:21.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:21.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:21.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:21.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:21.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:21.820 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:21.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:21.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:21.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:21 smithi007 ceph-mon[29098]: 1.6 scrub starts 2024-02-05T07:32:21.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:21 smithi007 ceph-mon[29098]: 1.6 scrub ok 2024-02-05T07:32:21.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:21 smithi007 ceph-mon[37158]: 1.6 scrub starts 2024-02-05T07:32:21.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:21 smithi007 ceph-mon[37158]: 1.6 scrub ok 2024-02-05T07:32:21.951 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:21.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:22.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:21 smithi090 ceph-mon[33213]: 1.6 scrub starts 2024-02-05T07:32:22.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:21 smithi090 ceph-mon[33213]: 1.6 scrub ok 2024-02-05T07:32:22.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:22.094 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:22.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:22.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:22.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:22.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:22.349 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:22.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:22.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:22.502 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:22.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:22.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:22.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:22.650 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:22.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:22.752 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:22.794 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:22.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:22.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:22 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:22 smithi007 ceph-mon[29098]: pgmap v704: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 174 active+clean; 3.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 909 B/s rd, 0 B/s wr, 842 op/s; 108/169365 objects degraded (0.064%); 2862/169365 objects misplaced (1.690%); 14 MiB/s, 216 objects/s recovering 2024-02-05T07:32:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:22 smithi007 ceph-mon[29098]: 1.2 scrub starts 2024-02-05T07:32:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:22 smithi007 ceph-mon[29098]: 1.2 scrub ok 2024-02-05T07:32:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:22 smithi007 ceph-mon[29098]: 1.7 scrub starts 2024-02-05T07:32:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:22 smithi007 ceph-mon[29098]: 1.7 scrub ok 2024-02-05T07:32:22.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:22 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:22.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:22 smithi007 ceph-mon[37158]: pgmap v704: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 174 active+clean; 3.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 909 B/s rd, 0 B/s wr, 842 op/s; 108/169365 objects degraded (0.064%); 2862/169365 objects misplaced (1.690%); 14 MiB/s, 216 objects/s recovering 2024-02-05T07:32:22.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:22 smithi007 ceph-mon[37158]: 1.2 scrub starts 2024-02-05T07:32:22.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:22 smithi007 ceph-mon[37158]: 1.2 scrub ok 2024-02-05T07:32:22.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:22 smithi007 ceph-mon[37158]: 1.7 scrub starts 2024-02-05T07:32:22.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:22 smithi007 ceph-mon[37158]: 1.7 scrub ok 2024-02-05T07:32:22.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:22.921 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:22.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:22.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:23.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:22 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:22 smithi090 ceph-mon[33213]: pgmap v704: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 174 active+clean; 3.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 909 B/s rd, 0 B/s wr, 842 op/s; 108/169365 objects degraded (0.064%); 2862/169365 objects misplaced (1.690%); 14 MiB/s, 216 objects/s recovering 2024-02-05T07:32:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:22 smithi090 ceph-mon[33213]: 1.2 scrub starts 2024-02-05T07:32:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:22 smithi090 ceph-mon[33213]: 1.2 scrub ok 2024-02-05T07:32:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:22 smithi090 ceph-mon[33213]: 1.7 scrub starts 2024-02-05T07:32:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:22 smithi090 ceph-mon[33213]: 1.7 scrub ok 2024-02-05T07:32:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:23.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:23.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:23.200 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:23.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:23.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:23.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:23.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:23.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:23.445 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:32:23.479 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:23.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:23.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:23.649 INFO:tasks.thrashosds.thrasher:j is {'epoch': 400, 'fsid': 'e3e7059e-c3f5-11ee-95b6-87774f69a715', 'created': '2024-02-05T07:13:14.603885+0000', 'modified': '2024-02-05T07:32:19.492334+0000', 'last_up_change': '2024-02-05T07:28:24.555976+0000', 'last_in_change': '2024-02-05T07:30:39.451715+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T07:16:32.505567+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '385', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '383', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5700000524520874, 'score_stable': 1.5700000524520874, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.350000023841858, 'raw_score_stable': 1.350000023841858, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T07:18:47.259462+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.690000057220459, 'score_stable': 4.690000057220459, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 3.3299999237060547, 'raw_score_stable': 3.3299999237060547, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T07:18:47.762336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4000000953674316, 'score_stable': 2.4000000953674316, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T07:18:49.736599+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T07:18:51.696948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T07:18:53.768555+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-02-05T07:29:59.438957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '354', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.059999942779541, 'score_stable': 3.059999942779541, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}], 'osds': [{'osd': 0, 'uuid': '703c3948-8003-4015-9dd8-21019cf0eeb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 387, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6801', 'nonce': 2069016689}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6802', 'nonce': 2069016689}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6804', 'nonce': 2069016689}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6803', 'nonce': 2069016689}]}, 'public_addr': '172.21.15.7:6801/2069016689', 'cluster_addr': '172.21.15.7:6802/2069016689', 'heartbeat_back_addr': '172.21.15.7:6804/2069016689', 'heartbeat_front_addr': '172.21.15.7:6803/2069016689', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '5c3e7d1e-a61d-44b9-b21e-f91c99a096bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 398, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6805', 'nonce': 865486532}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6806', 'nonce': 865486532}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6808', 'nonce': 865486532}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6807', 'nonce': 865486532}]}, 'public_addr': '172.21.15.7:6805/865486532', 'cluster_addr': '172.21.15.7:6806/865486532', 'heartbeat_back_addr': '172.21.15.7:6808/865486532', 'heartbeat_front_addr': '172.21.15.7:6807/865486532', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4fd930f-7bb2-46ca-9964-98235bd47b3a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 395, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6809', 'nonce': 904871370}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6810', 'nonce': 904871370}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6812', 'nonce': 904871370}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6811', 'nonce': 904871370}]}, 'public_addr': '172.21.15.7:6809/904871370', 'cluster_addr': '172.21.15.7:6810/904871370', 'heartbeat_back_addr': '172.21.15.7:6812/904871370', 'heartbeat_front_addr': '172.21.15.7:6811/904871370', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '149ed0d6-b049-4478-9819-cafc446e94d3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 399, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6813', 'nonce': 2758586242}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6814', 'nonce': 2758586242}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6816', 'nonce': 2758586242}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6815', 'nonce': 2758586242}]}, 'public_addr': '172.21.15.7:6813/2758586242', 'cluster_addr': '172.21.15.7:6814/2758586242', 'heartbeat_back_addr': '172.21.15.7:6816/2758586242', 'heartbeat_front_addr': '172.21.15.7:6815/2758586242', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 395, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 1154929470}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6801', 'nonce': 1154929470}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6803', 'nonce': 1154929470}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 1154929470}]}, 'public_addr': '172.21.15.90:6800/1154929470', 'cluster_addr': '172.21.15.90:6801/1154929470', 'heartbeat_back_addr': '172.21.15.90:6803/1154929470', 'heartbeat_front_addr': '172.21.15.90:6802/1154929470', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '244a7930-c22c-413e-86e1-11e0749f7367', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.7362823486328125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 336, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 398113577}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6805', 'nonce': 398113577}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6807', 'nonce': 398113577}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 398113577}]}, 'public_addr': '172.21.15.90:6804/398113577', 'cluster_addr': '172.21.15.90:6805/398113577', 'heartbeat_back_addr': '172.21.15.90:6807/398113577', 'heartbeat_front_addr': '172.21.15.90:6806/398113577', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0adbb562-3012-4f60-b5ec-02550a10b2c7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 52, 'last_clean_end': 230, 'up_from': 301, 'up_thru': 388, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 313196533}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 313196533}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6811', 'nonce': 313196533}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6810', 'nonce': 313196533}]}, 'public_addr': '172.21.15.90:6808/313196533', 'cluster_addr': '172.21.15.90:6809/313196533', 'heartbeat_back_addr': '172.21.15.90:6811/313196533', 'heartbeat_front_addr': '172.21.15.90:6810/313196533', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dbc26c36-a0e9-4f21-9e98-739270cea6d9', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 349, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 2983605980}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 2983605980}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6815', 'nonce': 2983605980}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6814', 'nonce': 2983605980}]}, 'public_addr': '172.21.15.90:6812/2983605980', 'cluster_addr': '172.21.15.90:6813/2983605980', 'heartbeat_back_addr': '172.21.15.90:6815/2983605980', 'heartbeat_front_addr': '172.21.15.90:6814/2983605980', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:31:49.882095+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:32:14.901479+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:31:26.668702+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:31:50.052170+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:31:33.467981+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T07:31:05.781144+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:32:00.252211+0000', 'dead_epoch': 230}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T07:31:31.337017+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.15', 'osds': [2, 0, 1]}], 'pg_upmap_items': [{'pgid': '3.3', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '3.1a', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.7', 'osds': [3, 2]}, {'pgid': '9.8', 'osds': [3, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.7:0/2180087772': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/2066685086': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/4100020626': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/1953351182': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/3203189162': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/3418781614': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:6800/1518592084': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:6800/3380493069': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/2208590106': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:0/223579766': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:6800/2634895574': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/1462812719': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/2823976078': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/1905520582': '2024-02-06T07:13:47.004765+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T07:32:23.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:32:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:23.652 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:23.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:23.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:23 smithi007 ceph-mon[29098]: 1.d scrub starts 2024-02-05T07:32:23.687 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:23 smithi007 ceph-mon[29098]: 1.d scrub ok 2024-02-05T07:32:23.687 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:23 smithi007 ceph-mon[29098]: 1.1c scrub starts 2024-02-05T07:32:23.687 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:23 smithi007 ceph-mon[29098]: 1.1c scrub ok 2024-02-05T07:32:23.687 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:32:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:32:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:32:23.753 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:23.815 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:23.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:23.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:23.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:23.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:24.021 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:23 smithi007 ceph-mon[37158]: 1.d scrub starts 2024-02-05T07:32:24.021 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:23 smithi007 ceph-mon[37158]: 1.d scrub ok 2024-02-05T07:32:24.021 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:23 smithi007 ceph-mon[37158]: 1.1c scrub starts 2024-02-05T07:32:24.021 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:23 smithi007 ceph-mon[37158]: 1.1c scrub ok 2024-02-05T07:32:24.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:23 smithi090 ceph-mon[33213]: 1.d scrub starts 2024-02-05T07:32:24.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:23 smithi090 ceph-mon[33213]: 1.d scrub ok 2024-02-05T07:32:24.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:23 smithi090 ceph-mon[33213]: 1.1c scrub starts 2024-02-05T07:32:24.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:23 smithi090 ceph-mon[33213]: 1.1c scrub ok 2024-02-05T07:32:24.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:24.141 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:24.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:24.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:24.269 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:24.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:24.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:24.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:24.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:24.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:32:24.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:24.581 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:24.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:24.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:24.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:24.708 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:24.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:24.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:24.841 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:32:24.841 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:32:24.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:24 smithi007 ceph-mon[29098]: pgmap v705: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 608 op/s; 158/160551 objects degraded (0.098%); 1711/160551 objects misplaced (1.066%); 11 MiB/s, 183 objects/s recovering 2024-02-05T07:32:24.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:24 smithi007 ceph-mon[29098]: 1.e scrub starts 2024-02-05T07:32:24.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:24 smithi007 ceph-mon[29098]: 1.e scrub ok 2024-02-05T07:32:24.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:24 smithi007 ceph-mon[29098]: 1.17 scrub starts 2024-02-05T07:32:24.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:24 smithi007 ceph-mon[29098]: 1.17 scrub ok 2024-02-05T07:32:24.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:24 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 158/160551 objects degraded (0.098%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:24.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:24 smithi007 ceph-mon[37158]: pgmap v705: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 608 op/s; 158/160551 objects degraded (0.098%); 1711/160551 objects misplaced (1.066%); 11 MiB/s, 183 objects/s recovering 2024-02-05T07:32:24.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:24 smithi007 ceph-mon[37158]: 1.e scrub starts 2024-02-05T07:32:24.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:24 smithi007 ceph-mon[37158]: 1.e scrub ok 2024-02-05T07:32:24.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:24 smithi007 ceph-mon[37158]: 1.17 scrub starts 2024-02-05T07:32:24.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:24 smithi007 ceph-mon[37158]: 1.17 scrub ok 2024-02-05T07:32:24.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:24 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 158/160551 objects degraded (0.098%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:24.863 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:24.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:24.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:24.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:25.006 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:25.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:25.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:24 smithi090 ceph-mon[33213]: pgmap v705: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 608 op/s; 158/160551 objects degraded (0.098%); 1711/160551 objects misplaced (1.066%); 11 MiB/s, 183 objects/s recovering 2024-02-05T07:32:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:24 smithi090 ceph-mon[33213]: 1.e scrub starts 2024-02-05T07:32:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:24 smithi090 ceph-mon[33213]: 1.e scrub ok 2024-02-05T07:32:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:24 smithi090 ceph-mon[33213]: 1.17 scrub starts 2024-02-05T07:32:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:24 smithi090 ceph-mon[33213]: 1.17 scrub ok 2024-02-05T07:32:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:24 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 158/160551 objects degraded (0.098%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:25.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:25.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:25.136 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:25.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:25.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:25.280 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:25.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:25.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:25.407 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:25.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:25.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:25.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:25.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T07:32:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:25.550 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:25.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:25.651 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:25.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:25.677 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:25.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:25.778 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:25 smithi007 ceph-mon[29098]: 1.5 scrub starts 2024-02-05T07:32:25.778 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:25 smithi007 ceph-mon[29098]: 1.5 scrub ok 2024-02-05T07:32:25.778 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:25 smithi007 ceph-mon[29098]: 1.f scrub starts 2024-02-05T07:32:25.778 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:25 smithi007 ceph-mon[29098]: 1.f scrub ok 2024-02-05T07:32:25.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:25.779 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:25 smithi007 ceph-mon[37158]: 1.5 scrub starts 2024-02-05T07:32:25.779 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:25 smithi007 ceph-mon[37158]: 1.5 scrub ok 2024-02-05T07:32:25.780 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:25 smithi007 ceph-mon[37158]: 1.f scrub starts 2024-02-05T07:32:25.780 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:25 smithi007 ceph-mon[37158]: 1.f scrub ok 2024-02-05T07:32:25.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:25.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:25.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:26.007 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:26.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:26.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:26.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:25 smithi090 ceph-mon[33213]: 1.5 scrub starts 2024-02-05T07:32:26.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:25 smithi090 ceph-mon[33213]: 1.5 scrub ok 2024-02-05T07:32:26.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:25 smithi090 ceph-mon[33213]: 1.f scrub starts 2024-02-05T07:32:26.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:25 smithi090 ceph-mon[33213]: 1.f scrub ok 2024-02-05T07:32:26.057 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:32:26.074 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:32:26.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:26.118 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:32:26.133 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:26.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:26.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:26.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:26.260 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:26.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:26.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:26.416 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:26.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:26.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:26.543 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:26.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:26.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:26.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:26.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:26.689 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:26.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:26.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:26.834 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:26.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:26.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:26.936 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:26.979 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:26.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:26.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:27.006 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:26 smithi007 ceph-mon[29098]: pgmap v706: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 3.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 514 op/s; 158/159699 objects degraded (0.099%); 1711/159699 objects misplaced (1.071%); 9.0 MiB/s, 144 objects/s recovering 2024-02-05T07:32:27.006 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:26 smithi007 ceph-mon[29098]: 1.1a scrub starts 2024-02-05T07:32:27.006 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:26 smithi007 ceph-mon[29098]: 1.1a scrub ok 2024-02-05T07:32:27.006 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:26 smithi007 ceph-mon[37158]: pgmap v706: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 3.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 514 op/s; 158/159699 objects degraded (0.099%); 1711/159699 objects misplaced (1.071%); 9.0 MiB/s, 144 objects/s recovering 2024-02-05T07:32:27.007 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:26 smithi007 ceph-mon[37158]: 1.1a scrub starts 2024-02-05T07:32:27.007 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:26 smithi007 ceph-mon[37158]: 1.1a scrub ok 2024-02-05T07:32:27.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:26 smithi090 ceph-mon[33213]: pgmap v706: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 3.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 514 op/s; 158/159699 objects degraded (0.099%); 1711/159699 objects misplaced (1.071%); 9.0 MiB/s, 144 objects/s recovering 2024-02-05T07:32:27.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:26 smithi090 ceph-mon[33213]: 1.1a scrub starts 2024-02-05T07:32:27.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:26 smithi090 ceph-mon[33213]: 1.1a scrub ok 2024-02-05T07:32:27.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:27.105 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:27.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:27.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:27.243 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:27.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:27.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:27.376 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:27.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:27.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:27.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:27.506 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:27.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:27.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:27.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:27.649 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:27.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:27.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:27.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:27.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:27.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:27.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:27.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:27.902 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:27.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:28.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:27 smithi090 ceph-mon[33213]: 1.1b scrub starts 2024-02-05T07:32:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:27 smithi090 ceph-mon[33213]: 1.1b scrub ok 2024-02-05T07:32:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:27 smithi090 ceph-mon[33213]: 3.10 scrub starts 2024-02-05T07:32:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:27 smithi090 ceph-mon[33213]: 3.10 scrub ok 2024-02-05T07:32:28.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:28.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:28.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:28.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:27 smithi007 ceph-mon[29098]: 1.1b scrub starts 2024-02-05T07:32:28.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:27 smithi007 ceph-mon[29098]: 1.1b scrub ok 2024-02-05T07:32:28.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:27 smithi007 ceph-mon[29098]: 3.10 scrub starts 2024-02-05T07:32:28.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:27 smithi007 ceph-mon[29098]: 3.10 scrub ok 2024-02-05T07:32:28.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:27 smithi007 ceph-mon[37158]: 1.1b scrub starts 2024-02-05T07:32:28.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:27 smithi007 ceph-mon[37158]: 1.1b scrub ok 2024-02-05T07:32:28.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:27 smithi007 ceph-mon[37158]: 3.10 scrub starts 2024-02-05T07:32:28.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:27 smithi007 ceph-mon[37158]: 3.10 scrub ok 2024-02-05T07:32:28.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:28.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:28.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:28.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:28.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:28.337 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:28.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:28.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:28.480 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:28.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:28.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:28.608 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:28.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:28.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:28.748 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:28.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:28.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:32:28.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:28.891 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:28.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:28.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:29.008 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:28 smithi007 ceph-mon[29098]: pgmap v707: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.03k op/s; 158/144783 objects degraded (0.109%); 1262/144783 objects misplaced (0.872%); 11 MiB/s, 180 objects/s recovering 2024-02-05T07:32:29.008 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:28 smithi007 ceph-mon[29098]: 4.c scrub starts 2024-02-05T07:32:29.008 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:28 smithi007 ceph-mon[29098]: 4.c scrub ok 2024-02-05T07:32:29.008 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:28 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3937359367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:32:29.008 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:28 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:32:29.008 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:28 smithi007 ceph-mon[37158]: pgmap v707: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.03k op/s; 158/144783 objects degraded (0.109%); 1262/144783 objects misplaced (0.872%); 11 MiB/s, 180 objects/s recovering 2024-02-05T07:32:29.008 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:28 smithi007 ceph-mon[37158]: 4.c scrub starts 2024-02-05T07:32:29.008 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:28 smithi007 ceph-mon[37158]: 4.c scrub ok 2024-02-05T07:32:29.009 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:28 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3937359367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:32:29.009 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:28 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:32:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:28 smithi090 ceph-mon[33213]: pgmap v707: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.03k op/s; 158/144783 objects degraded (0.109%); 1262/144783 objects misplaced (0.872%); 11 MiB/s, 180 objects/s recovering 2024-02-05T07:32:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:28 smithi090 ceph-mon[33213]: 4.c scrub starts 2024-02-05T07:32:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:28 smithi090 ceph-mon[33213]: 4.c scrub ok 2024-02-05T07:32:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:28 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3937359367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:32:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:28 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:32:29.040 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:29.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:29.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:29.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:29.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:29.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:29.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:29.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:29.325 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:29.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:29.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:29.450 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:29.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:29.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:29.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:29.576 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:29.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:29.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:29.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:29.703 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:29.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:29.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:29.831 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:29.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:29.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:29.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:29.958 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[29098]: 3.9 scrub starts 2024-02-05T07:32:29.958 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[29098]: 3.9 scrub ok 2024-02-05T07:32:29.958 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:32:29.959 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:32:29.959 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[29098]: osdmap e401: 8 total, 8 up, 6 in 2024-02-05T07:32:29.959 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[29098]: pgmap v709: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 914 op/s; 158/143919 objects degraded (0.110%); 1262/143919 objects misplaced (0.877%); 9.9 MiB/s, 159 objects/s recovering 2024-02-05T07:32:29.959 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 158/144783 objects degraded (0.109%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:29.959 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[29098]: osdmap e402: 8 total, 8 up, 6 in 2024-02-05T07:32:29.959 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[37158]: 3.9 scrub starts 2024-02-05T07:32:29.959 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[37158]: 3.9 scrub ok 2024-02-05T07:32:29.959 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:32:29.959 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:32:29.960 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[37158]: osdmap e401: 8 total, 8 up, 6 in 2024-02-05T07:32:29.960 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[37158]: pgmap v709: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 914 op/s; 158/143919 objects degraded (0.110%); 1262/143919 objects misplaced (0.877%); 9.9 MiB/s, 159 objects/s recovering 2024-02-05T07:32:29.960 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 158/144783 objects degraded (0.109%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:29.960 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:29 smithi007 ceph-mon[37158]: osdmap e402: 8 total, 8 up, 6 in 2024-02-05T07:32:29.976 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:29.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:29.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:29 smithi090 ceph-mon[33213]: 3.9 scrub starts 2024-02-05T07:32:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:29 smithi090 ceph-mon[33213]: 3.9 scrub ok 2024-02-05T07:32:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:29 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:32:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:29 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:32:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:29 smithi090 ceph-mon[33213]: osdmap e401: 8 total, 8 up, 6 in 2024-02-05T07:32:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:29 smithi090 ceph-mon[33213]: pgmap v709: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 914 op/s; 158/143919 objects degraded (0.110%); 1262/143919 objects misplaced (0.877%); 9.9 MiB/s, 159 objects/s recovering 2024-02-05T07:32:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:29 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 158/144783 objects degraded (0.109%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:29 smithi090 ceph-mon[33213]: osdmap e402: 8 total, 8 up, 6 in 2024-02-05T07:32:30.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:30.104 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:30.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:30.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:30.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:30.248 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:30.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:30.350 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:30.375 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:30.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:30.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:30.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:30.517 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:30.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:30.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:30.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:30.644 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:30.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:30.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:30.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:30.796 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:30.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:30.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:30.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:30 smithi007 ceph-mon[29098]: 4.8 scrub starts 2024-02-05T07:32:30.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:30 smithi007 ceph-mon[29098]: 4.8 scrub ok 2024-02-05T07:32:30.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:32:30.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:32:30.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:30 smithi007 ceph-mon[37158]: 4.8 scrub starts 2024-02-05T07:32:30.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:30 smithi007 ceph-mon[37158]: 4.8 scrub ok 2024-02-05T07:32:30.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:32:30.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:32:30.953 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:30.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:30.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:31.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:32:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:32:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:30 smithi090 ceph-mon[33213]: 4.8 scrub starts 2024-02-05T07:32:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:30 smithi090 ceph-mon[33213]: 4.8 scrub ok 2024-02-05T07:32:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:32:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:32:31.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:31.107 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:31.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:31.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:31.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:31.261 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:31.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:31.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:31.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:31.401 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:31.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:31.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:31.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:31.547 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:31.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:31.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:31.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:31.686 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:31.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:31.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:31.828 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:31.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:31.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:31.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:31 smithi007 ceph-mon[29098]: 6.e scrub starts 2024-02-05T07:32:31.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:31 smithi007 ceph-mon[29098]: 6.e scrub ok 2024-02-05T07:32:31.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:31 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:31.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:31 smithi007 ceph-mon[29098]: 5.9 scrub starts 2024-02-05T07:32:31.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:31 smithi007 ceph-mon[29098]: 5.9 scrub ok 2024-02-05T07:32:31.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:31 smithi007 ceph-mon[29098]: pgmap v711: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 772 op/s; 158/141888 objects degraded (0.111%); 1262/141888 objects misplaced (0.889%); 3.5 MiB/s, 55 objects/s recovering 2024-02-05T07:32:31.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:31 smithi007 ceph-mon[37158]: 6.e scrub starts 2024-02-05T07:32:31.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:31 smithi007 ceph-mon[37158]: 6.e scrub ok 2024-02-05T07:32:31.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:31 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:31.844 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:31 smithi007 ceph-mon[37158]: 5.9 scrub starts 2024-02-05T07:32:31.844 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:31 smithi007 ceph-mon[37158]: 5.9 scrub ok 2024-02-05T07:32:31.844 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:31 smithi007 ceph-mon[37158]: pgmap v711: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 772 op/s; 158/141888 objects degraded (0.111%); 1262/141888 objects misplaced (0.889%); 3.5 MiB/s, 55 objects/s recovering 2024-02-05T07:32:31.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:31.968 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:31.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:32.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:32:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:31 smithi090 ceph-mon[33213]: 6.e scrub starts 2024-02-05T07:32:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:31 smithi090 ceph-mon[33213]: 6.e scrub ok 2024-02-05T07:32:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:31 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:31 smithi090 ceph-mon[33213]: 5.9 scrub starts 2024-02-05T07:32:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:31 smithi090 ceph-mon[33213]: 5.9 scrub ok 2024-02-05T07:32:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:31 smithi090 ceph-mon[33213]: pgmap v711: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 772 op/s; 158/141888 objects degraded (0.111%); 1262/141888 objects misplaced (0.889%); 3.5 MiB/s, 55 objects/s recovering 2024-02-05T07:32:32.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:32.094 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:32.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:32.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:32.221 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:32.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:32.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:32.348 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:32.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:32.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:32.412 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:32:32.412 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:32:32.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:32.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:32.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:32.635 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:32.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:32.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:32.771 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:32.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:32.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:32.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:32.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:32.920 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:32.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:33.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:33.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:32 smithi090 ceph-mon[33213]: 4.13 scrub starts 2024-02-05T07:32:33.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:32 smithi090 ceph-mon[33213]: 4.13 scrub ok 2024-02-05T07:32:33.045 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:33.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:33.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:33.048 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:32 smithi007 ceph-mon[29098]: 4.13 scrub starts 2024-02-05T07:32:33.048 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:32 smithi007 ceph-mon[29098]: 4.13 scrub ok 2024-02-05T07:32:33.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:32 smithi007 ceph-mon[37158]: 4.13 scrub starts 2024-02-05T07:32:33.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:32 smithi007 ceph-mon[37158]: 4.13 scrub ok 2024-02-05T07:32:33.147 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:33.191 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:33.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:33.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:33.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:33.350 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:33.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:33.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:33.495 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:33.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:33.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:33.566 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:32:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:32:33.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:33.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:33.646 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:33.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:33.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:33.750 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:32:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:32:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:32:33.789 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:33.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:33.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:33.932 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:33.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:33 smithi090 ceph-mon[33213]: 6.15 scrub starts 2024-02-05T07:32:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:33 smithi090 ceph-mon[33213]: 6.15 scrub ok 2024-02-05T07:32:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:33 smithi090 ceph-mon[33213]: pgmap v712: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.29k op/s; 210/128691 objects degraded (0.163%); 751/128691 objects misplaced (0.584%); 7.1 MiB/s, 113 objects/s recovering 2024-02-05T07:32:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:33 smithi090 ceph-mon[33213]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:32:34.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:34.058 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:34.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:34.064 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:33 smithi007 ceph-mon[29098]: 6.15 scrub starts 2024-02-05T07:32:34.064 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:33 smithi007 ceph-mon[29098]: 6.15 scrub ok 2024-02-05T07:32:34.064 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:33 smithi007 ceph-mon[29098]: pgmap v712: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.29k op/s; 210/128691 objects degraded (0.163%); 751/128691 objects misplaced (0.584%); 7.1 MiB/s, 113 objects/s recovering 2024-02-05T07:32:34.064 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:33 smithi007 ceph-mon[29098]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:32:34.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:33 smithi007 ceph-mon[37158]: 6.15 scrub starts 2024-02-05T07:32:34.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:33 smithi007 ceph-mon[37158]: 6.15 scrub ok 2024-02-05T07:32:34.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:33 smithi007 ceph-mon[37158]: pgmap v712: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.29k op/s; 210/128691 objects degraded (0.163%); 751/128691 objects misplaced (0.584%); 7.1 MiB/s, 113 objects/s recovering 2024-02-05T07:32:34.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:33 smithi007 ceph-mon[37158]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:32:34.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:34.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:34.197 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:32:34.247 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:32:34.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:34.289 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:32:34.329 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:34.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:34.335 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:32:34.376 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:32:34.421 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:32:34.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:34.456 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:34.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:34.464 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:32:34.509 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:32:34.525 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:32:34.525 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:32:34.525 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:32:34.525 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:32:34.525 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:32:34.525 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:32:34.525 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:32:34.525 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:32:34.556 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":712,"stamp":"2024-02-05T07:32:33.234436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2798807836,"num_objects":42897,"num_object_clones":0,"num_object_copies":128691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":210,"num_objects_misplaced":751,"num_objects_unfound":0,"num_objects_dirty":42897,"num_whiteouts":0,"num_read":1252,"num_read_kb":1012,"num_write":117724,"num_write_kb":5110154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26597,"num_bytes_recovered":1723372711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56457,"ondisk_log_size":56457,"up":534,"acting":532,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":11663924,"kb_used_data":8403000,"kb_used_omap":13,"kb_used_meta":3260594,"kb_avail":550806988,"statfs":{"total":575970213888,"available":564026355712,"internally_reserved":0,"allocated":8604672000,"data_stored":8599449164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":3338848992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-677456128,"num_objects":-10336,"num_object_clones":0,"num_object_copies":-31008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":-960,"num_objects_unfound":0,"num_objects_dirty":-10336,"num_whiteouts":0,"num_read":-16,"num_read_kb":-29,"num_write":10313,"num_write_kb":-312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":907,"num_bytes_recovered":59278591,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002723"},"pg_stats":[{"pgid":"9.1","version":"402'7274","reported_seq":12758,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:31.577629+0000","last_change":"2024-02-05T07:32:19.517221+0000","last_active":"2024-02-05T07:32:31.577629+0000","last_peered":"2024-02-05T07:32:31.577629+0000","last_clean":"2024-02-05T07:32:31.577629+0000","last_became_active":"2024-02-05T07:32:19.505118+0000","last_became_peered":"2024-02-05T07:32:19.505118+0000","last_unstale":"2024-02-05T07:32:31.577629+0000","last_undegraded":"2024-02-05T07:32:31.577629+0000","last_fullsized":"2024-02-05T07:32:31.577629+0000","mapping_epoch":399,"log_start":"376'4129","ondisk_log_start":"376'4129","created":349,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3145,"log_dups_size":0,"ondisk_log_size":3145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175636480,"num_objects":2680,"num_object_clones":0,"num_object_copies":8040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7274,"num_write_kb":318528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2678,"num_bytes_recovered":175505408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":804,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.523979+0000","last_change":"2024-02-05T07:31:36.857954+0000","last_active":"2024-02-05T07:32:29.523979+0000","last_peered":"2024-02-05T07:32:29.523979+0000","last_clean":"2024-02-05T07:32:29.523979+0000","last_became_active":"2024-02-05T07:31:36.857092+0000","last_became_peered":"2024-02-05T07:31:36.857092+0000","last_unstale":"2024-02-05T07:32:29.523979+0000","last_undegraded":"2024-02-05T07:32:29.523979+0000","last_fullsized":"2024-02-05T07:32:29.523979+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:26:18.993912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.993912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:32:31.684745+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"326'25","reported_seq":790,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499831+0000","last_change":"2024-02-05T07:32:27.643863+0000","last_active":"2024-02-05T07:32:29.499831+0000","last_peered":"2024-02-05T07:32:29.499831+0000","last_clean":"2024-02-05T07:32:29.499831+0000","last_became_active":"2024-02-05T07:28:26.605082+0000","last_became_peered":"2024-02-05T07:28:26.605082+0000","last_unstale":"2024-02-05T07:32:29.499831+0000","last_undegraded":"2024-02-05T07:32:29.499831+0000","last_fullsized":"2024-02-05T07:32:29.499831+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:32:27.643788+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:32:27.643788+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:47.371484+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":289,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.527018+0000","last_change":"2024-02-05T07:27:16.716749+0000","last_active":"2024-02-05T07:32:29.527018+0000","last_peered":"2024-02-05T07:32:29.527018+0000","last_clean":"2024-02-05T07:32:29.527018+0000","last_became_active":"2024-02-05T07:27:16.716461+0000","last_became_peered":"2024-02-05T07:27:16.716461+0000","last_unstale":"2024-02-05T07:32:29.527018+0000","last_undegraded":"2024-02-05T07:32:29.527018+0000","last_fullsized":"2024-02-05T07:32:29.527018+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.258469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.258469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"365'11","reported_seq":913,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.635244+0000","last_change":"2024-02-05T07:30:40.506050+0000","last_active":"2024-02-05T07:32:29.635244+0000","last_peered":"2024-02-05T07:32:29.635244+0000","last_clean":"2024-02-05T07:32:29.635244+0000","last_became_active":"2024-02-05T07:30:40.484389+0000","last_became_peered":"2024-02-05T07:30:40.484389+0000","last_unstale":"2024-02-05T07:32:29.635244+0000","last_undegraded":"2024-02-05T07:32:29.635244+0000","last_fullsized":"2024-02-05T07:32:29.635244+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'10","last_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:30:38.946065+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":695,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.500551+0000","last_change":"2024-02-05T07:25:43.273140+0000","last_active":"2024-02-05T07:32:29.500551+0000","last_peered":"2024-02-05T07:32:29.500551+0000","last_clean":"2024-02-05T07:32:29.500551+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:32:29.500551+0000","last_undegraded":"2024-02-05T07:32:29.500551+0000","last_fullsized":"2024-02-05T07:32:29.500551+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:25:43.273019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"402'7258","reported_seq":7374,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:31.854482+0000","last_change":"2024-02-05T07:30:00.506880+0000","last_active":"2024-02-05T07:32:31.854482+0000","last_peered":"2024-02-05T07:32:31.854482+0000","last_clean":"2024-02-05T07:32:31.854482+0000","last_became_active":"2024-02-05T07:30:00.506586+0000","last_became_peered":"2024-02-05T07:30:00.506586+0000","last_unstale":"2024-02-05T07:32:31.854482+0000","last_undegraded":"2024-02-05T07:32:31.854482+0000","last_fullsized":"2024-02-05T07:32:31.854482+0000","mapping_epoch":349,"log_start":"376'4169","ondisk_log_start":"376'4169","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3089,"log_dups_size":0,"ondisk_log_size":3089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174194688,"num_objects":2658,"num_object_clones":0,"num_object_copies":7974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7258,"num_write_kb":317312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":724,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.498622+0000","last_change":"2024-02-05T07:31:36.857186+0000","last_active":"2024-02-05T07:32:29.498622+0000","last_peered":"2024-02-05T07:32:29.498622+0000","last_clean":"2024-02-05T07:32:29.498622+0000","last_became_active":"2024-02-05T07:31:36.856842+0000","last_became_peered":"2024-02-05T07:31:36.856842+0000","last_unstale":"2024-02-05T07:32:29.498622+0000","last_undegraded":"2024-02-05T07:32:29.498622+0000","last_fullsized":"2024-02-05T07:32:29.498622+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:29:48.249082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.249082+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:32:34.274084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"326'31","reported_seq":761,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.500281+0000","last_change":"2024-02-05T07:26:36.275060+0000","last_active":"2024-02-05T07:32:29.500281+0000","last_peered":"2024-02-05T07:32:29.500281+0000","last_clean":"2024-02-05T07:32:29.500281+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:32:29.500281+0000","last_undegraded":"2024-02-05T07:32:29.500281+0000","last_fullsized":"2024-02-05T07:32:29.500281+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:26:36.274939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.274939+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":556,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.504569+0000","last_change":"2024-02-05T07:30:30.212483+0000","last_active":"2024-02-05T07:32:19.504569+0000","last_peered":"2024-02-05T07:32:19.504569+0000","last_clean":"2024-02-05T07:32:19.504569+0000","last_became_active":"2024-02-05T07:28:25.791005+0000","last_became_peered":"2024-02-05T07:28:25.791005+0000","last_unstale":"2024-02-05T07:32:19.504569+0000","last_undegraded":"2024-02-05T07:32:19.504569+0000","last_fullsized":"2024-02-05T07:32:19.504569+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:30.212401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.212401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":709,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.515823+0000","last_change":"2024-02-05T07:29:49.674581+0000","last_active":"2024-02-05T07:32:29.515823+0000","last_peered":"2024-02-05T07:32:29.515823+0000","last_clean":"2024-02-05T07:32:29.515823+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:32:29.515823+0000","last_undegraded":"2024-02-05T07:32:29.515823+0000","last_fullsized":"2024-02-05T07:32:29.515823+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.674493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.674493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":719,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.513706+0000","last_change":"2024-02-05T07:30:35.156491+0000","last_active":"2024-02-05T07:32:29.513706+0000","last_peered":"2024-02-05T07:32:29.513706+0000","last_clean":"2024-02-05T07:32:29.513706+0000","last_became_active":"2024-02-05T07:27:16.708888+0000","last_became_peered":"2024-02-05T07:27:16.708888+0000","last_unstale":"2024-02-05T07:32:29.513706+0000","last_undegraded":"2024-02-05T07:32:29.513706+0000","last_fullsized":"2024-02-05T07:32:29.513706+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:35.156384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:30:35.156384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"402'7353","reported_seq":13016,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:31.548938+0000","last_change":"2024-02-05T07:31:31.709575+0000","last_active":"2024-02-05T07:32:31.548938+0000","last_peered":"2024-02-05T07:32:31.548938+0000","last_clean":"2024-02-05T07:32:31.548938+0000","last_became_active":"2024-02-05T07:31:31.703194+0000","last_became_peered":"2024-02-05T07:31:31.703194+0000","last_unstale":"2024-02-05T07:32:31.548938+0000","last_undegraded":"2024-02-05T07:32:31.548938+0000","last_fullsized":"2024-02-05T07:32:31.548938+0000","mapping_epoch":379,"log_start":"376'4180","ondisk_log_start":"376'4180","created":349,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3173,"log_dups_size":0,"ondisk_log_size":3173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169148416,"num_objects":2581,"num_object_clones":0,"num_object_copies":7743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":317888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2768,"num_bytes_recovered":181403648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":694,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511751+0000","last_change":"2024-02-05T07:31:36.842480+0000","last_active":"2024-02-05T07:32:29.511751+0000","last_peered":"2024-02-05T07:32:29.511751+0000","last_clean":"2024-02-05T07:32:29.511751+0000","last_became_active":"2024-02-05T07:31:36.841690+0000","last_became_peered":"2024-02-05T07:31:36.841690+0000","last_unstale":"2024-02-05T07:32:29.511751+0000","last_undegraded":"2024-02-05T07:32:29.511751+0000","last_fullsized":"2024-02-05T07:32:29.511751+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:46.611747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:46.611747+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:32:30.389628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"326'24","reported_seq":756,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.500306+0000","last_change":"2024-02-05T07:29:50.053697+0000","last_active":"2024-02-05T07:32:29.500306+0000","last_peered":"2024-02-05T07:32:29.500306+0000","last_clean":"2024-02-05T07:32:29.500306+0000","last_became_active":"2024-02-05T07:28:25.781007+0000","last_became_peered":"2024-02-05T07:28:25.781007+0000","last_unstale":"2024-02-05T07:32:29.500306+0000","last_undegraded":"2024-02-05T07:32:29.500306+0000","last_fullsized":"2024-02-05T07:32:29.500306+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:29:50.053583+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.053583+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":722,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499158+0000","last_change":"2024-02-05T07:32:28.670360+0000","last_active":"2024-02-05T07:32:29.499158+0000","last_peered":"2024-02-05T07:32:29.499158+0000","last_clean":"2024-02-05T07:32:29.499158+0000","last_became_active":"2024-02-05T07:30:40.486134+0000","last_became_peered":"2024-02-05T07:30:40.486134+0000","last_unstale":"2024-02-05T07:32:29.499158+0000","last_undegraded":"2024-02-05T07:32:29.499158+0000","last_fullsized":"2024-02-05T07:32:29.499158+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:28.670286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:32:28.670286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:32.433468+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":543,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.500422+0000","last_change":"2024-02-05T07:29:45.012958+0000","last_active":"2024-02-05T07:32:29.500422+0000","last_peered":"2024-02-05T07:32:29.500422+0000","last_clean":"2024-02-05T07:32:29.500422+0000","last_became_active":"2024-02-05T07:28:26.601713+0000","last_became_peered":"2024-02-05T07:28:26.601713+0000","last_unstale":"2024-02-05T07:32:29.500422+0000","last_undegraded":"2024-02-05T07:32:29.500422+0000","last_fullsized":"2024-02-05T07:32:29.500422+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.012821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":298,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.522487+0000","last_change":"2024-02-05T07:30:40.499008+0000","last_active":"2024-02-05T07:32:29.522487+0000","last_peered":"2024-02-05T07:32:29.522487+0000","last_clean":"2024-02-05T07:32:29.522487+0000","last_became_active":"2024-02-05T07:30:40.498529+0000","last_became_peered":"2024-02-05T07:30:40.498529+0000","last_unstale":"2024-02-05T07:32:29.522487+0000","last_undegraded":"2024-02-05T07:32:29.522487+0000","last_fullsized":"2024-02-05T07:32:29.522487+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:47.885294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.885294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"402'7300","reported_seq":7415,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:31.585527+0000","last_change":"2024-02-05T07:30:00.512144+0000","last_active":"2024-02-05T07:32:31.585527+0000","last_peered":"2024-02-05T07:32:31.585527+0000","last_clean":"2024-02-05T07:32:31.585527+0000","last_became_active":"2024-02-05T07:30:00.511808+0000","last_became_peered":"2024-02-05T07:30:00.511808+0000","last_unstale":"2024-02-05T07:32:31.585527+0000","last_undegraded":"2024-02-05T07:32:31.585527+0000","last_fullsized":"2024-02-05T07:32:31.585527+0000","mapping_epoch":349,"log_start":"376'4083","ondisk_log_start":"376'4083","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3217,"log_dups_size":0,"ondisk_log_size":3217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171048960,"num_objects":2610,"num_object_clones":0,"num_object_copies":7830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7300,"num_write_kb":317120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":699,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.510432+0000","last_change":"2024-02-05T07:31:36.840346+0000","last_active":"2024-02-05T07:32:29.510432+0000","last_peered":"2024-02-05T07:32:29.510432+0000","last_clean":"2024-02-05T07:32:29.510432+0000","last_became_active":"2024-02-05T07:31:36.839888+0000","last_became_peered":"2024-02-05T07:31:36.839888+0000","last_unstale":"2024-02-05T07:32:29.510432+0000","last_undegraded":"2024-02-05T07:32:29.510432+0000","last_fullsized":"2024-02-05T07:32:29.510432+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:27.218488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:27.218488+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:32:34.274061+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"326'34","reported_seq":807,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.517511+0000","last_change":"2024-02-05T07:30:30.006087+0000","last_active":"2024-02-05T07:32:29.517511+0000","last_peered":"2024-02-05T07:32:29.517511+0000","last_clean":"2024-02-05T07:32:29.517511+0000","last_became_active":"2024-02-05T07:27:16.768454+0000","last_became_peered":"2024-02-05T07:27:16.768454+0000","last_unstale":"2024-02-05T07:32:29.517511+0000","last_undegraded":"2024-02-05T07:32:29.517511+0000","last_fullsized":"2024-02-05T07:32:29.517511+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'34","last_scrub_stamp":"2024-02-05T07:30:30.005988+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.005988+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":303,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.530244+0000","last_change":"2024-02-05T07:30:40.508975+0000","last_active":"2024-02-05T07:32:29.530244+0000","last_peered":"2024-02-05T07:32:29.530244+0000","last_clean":"2024-02-05T07:32:29.530244+0000","last_became_active":"2024-02-05T07:30:40.508561+0000","last_became_peered":"2024-02-05T07:30:40.508561+0000","last_unstale":"2024-02-05T07:32:29.530244+0000","last_undegraded":"2024-02-05T07:32:29.530244+0000","last_fullsized":"2024-02-05T07:32:29.530244+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:30.982457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.982457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"257'9","reported_seq":882,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.500546+0000","last_change":"2024-02-05T07:27:03.386101+0000","last_active":"2024-02-05T07:32:29.500546+0000","last_peered":"2024-02-05T07:32:29.500546+0000","last_clean":"2024-02-05T07:32:29.500546+0000","last_became_active":"2024-02-05T07:27:03.375195+0000","last_became_peered":"2024-02-05T07:27:03.375195+0000","last_unstale":"2024-02-05T07:32:29.500546+0000","last_undegraded":"2024-02-05T07:32:29.500546+0000","last_fullsized":"2024-02-05T07:32:29.500546+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-02-05T07:26:22.295732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:22.295732+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":315,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.516638+0000","last_change":"2024-02-05T07:30:37.998622+0000","last_active":"2024-02-05T07:32:29.516638+0000","last_peered":"2024-02-05T07:32:29.516638+0000","last_clean":"2024-02-05T07:32:29.516638+0000","last_became_active":"2024-02-05T07:27:03.367753+0000","last_became_peered":"2024-02-05T07:27:03.367753+0000","last_unstale":"2024-02-05T07:32:29.516638+0000","last_undegraded":"2024-02-05T07:32:29.516638+0000","last_fullsized":"2024-02-05T07:32:29.516638+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:37.998383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:30:37.998383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"402'7468","reported_seq":13245,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:31.806643+0000","last_change":"2024-02-05T07:32:18.658612+0000","last_active":"2024-02-05T07:32:31.806643+0000","last_peered":"2024-02-05T07:32:31.806643+0000","last_clean":"2024-02-05T07:32:31.806643+0000","last_became_active":"2024-02-05T07:32:18.645799+0000","last_became_peered":"2024-02-05T07:32:18.645799+0000","last_unstale":"2024-02-05T07:32:31.806643+0000","last_undegraded":"2024-02-05T07:32:31.806643+0000","last_fullsized":"2024-02-05T07:32:31.806643+0000","mapping_epoch":398,"log_start":"378'4433","ondisk_log_start":"378'4433","created":349,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3035,"log_dups_size":0,"ondisk_log_size":3035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167575576,"num_objects":2558,"num_object_clones":0,"num_object_copies":7674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7468,"num_write_kb":320769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2824,"num_bytes_recovered":185073664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":701,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499937+0000","last_change":"2024-02-05T07:32:24.656130+0000","last_active":"2024-02-05T07:32:29.499937+0000","last_peered":"2024-02-05T07:32:29.499937+0000","last_clean":"2024-02-05T07:32:29.499937+0000","last_became_active":"2024-02-05T07:31:36.758820+0000","last_became_peered":"2024-02-05T07:31:36.758820+0000","last_unstale":"2024-02-05T07:32:29.499937+0000","last_undegraded":"2024-02-05T07:32:29.499937+0000","last_fullsized":"2024-02-05T07:32:29.499937+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:24.656057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:24.656057+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:45.588976+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":718,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499103+0000","last_change":"2024-02-05T07:30:40.489530+0000","last_active":"2024-02-05T07:32:29.499103+0000","last_peered":"2024-02-05T07:32:29.499103+0000","last_clean":"2024-02-05T07:32:29.499103+0000","last_became_active":"2024-02-05T07:30:40.489212+0000","last_became_peered":"2024-02-05T07:30:40.489212+0000","last_unstale":"2024-02-05T07:32:29.499103+0000","last_undegraded":"2024-02-05T07:32:29.499103+0000","last_fullsized":"2024-02-05T07:32:29.499103+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:51.211877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:25:51.211877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"326'30","reported_seq":357,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.530188+0000","last_change":"2024-02-05T07:30:40.543094+0000","last_active":"2024-02-05T07:32:29.530188+0000","last_peered":"2024-02-05T07:32:29.530188+0000","last_clean":"2024-02-05T07:32:29.530188+0000","last_became_active":"2024-02-05T07:30:40.496940+0000","last_became_peered":"2024-02-05T07:30:40.496940+0000","last_unstale":"2024-02-05T07:32:29.530188+0000","last_undegraded":"2024-02-05T07:32:29.530188+0000","last_fullsized":"2024-02-05T07:32:29.530188+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:28:45.029295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.029295+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":295,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.531666+0000","last_change":"2024-02-05T07:30:40.509232+0000","last_active":"2024-02-05T07:32:29.531666+0000","last_peered":"2024-02-05T07:32:29.531666+0000","last_clean":"2024-02-05T07:32:29.531666+0000","last_became_active":"2024-02-05T07:30:40.508607+0000","last_became_peered":"2024-02-05T07:30:40.508607+0000","last_unstale":"2024-02-05T07:32:29.531666+0000","last_undegraded":"2024-02-05T07:32:29.531666+0000","last_fullsized":"2024-02-05T07:32:29.531666+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:33.084016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:33.084016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":711,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499313+0000","last_change":"2024-02-05T07:28:56.783280+0000","last_active":"2024-02-05T07:32:29.499313+0000","last_peered":"2024-02-05T07:32:29.499313+0000","last_clean":"2024-02-05T07:32:29.499313+0000","last_became_active":"2024-02-05T07:27:16.706666+0000","last_became_peered":"2024-02-05T07:27:16.706666+0000","last_unstale":"2024-02-05T07:32:29.499313+0000","last_undegraded":"2024-02-05T07:32:29.499313+0000","last_fullsized":"2024-02-05T07:32:29.499313+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:28:56.783168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"402'7161","reported_seq":12606,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:31.589617+0000","last_change":"2024-02-05T07:31:36.855992+0000","last_active":"2024-02-05T07:32:31.589617+0000","last_peered":"2024-02-05T07:32:31.589617+0000","last_clean":"2024-02-05T07:32:31.589617+0000","last_became_active":"2024-02-05T07:31:36.841233+0000","last_became_peered":"2024-02-05T07:31:36.841233+0000","last_unstale":"2024-02-05T07:32:31.589617+0000","last_undegraded":"2024-02-05T07:32:31.589617+0000","last_fullsized":"2024-02-05T07:32:31.589617+0000","mapping_epoch":383,"log_start":"376'4035","ondisk_log_start":"376'4035","created":349,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3126,"log_dups_size":0,"ondisk_log_size":3126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167837696,"num_objects":2561,"num_object_clones":0,"num_object_copies":7683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7161,"num_write_kb":311104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2659,"num_bytes_recovered":174260224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":958,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.522741+0000","last_change":"2024-02-05T07:30:43.524967+0000","last_active":"2024-02-05T07:32:29.522741+0000","last_peered":"2024-02-05T07:32:29.522741+0000","last_clean":"2024-02-05T07:32:29.522741+0000","last_became_active":"2024-02-05T07:30:43.524588+0000","last_became_peered":"2024-02-05T07:30:43.524588+0000","last_unstale":"2024-02-05T07:32:29.522741+0000","last_undegraded":"2024-02-05T07:32:29.522741+0000","last_fullsized":"2024-02-05T07:32:29.522741+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:39.042281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:39.042281+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"266'10","reported_seq":356,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.530337+0000","last_change":"2024-02-05T07:27:17.017592+0000","last_active":"2024-02-05T07:32:29.530337+0000","last_peered":"2024-02-05T07:32:29.530337+0000","last_clean":"2024-02-05T07:32:29.530337+0000","last_became_active":"2024-02-05T07:27:16.800207+0000","last_became_peered":"2024-02-05T07:27:16.800207+0000","last_unstale":"2024-02-05T07:32:29.530337+0000","last_undegraded":"2024-02-05T07:32:29.530337+0000","last_fullsized":"2024-02-05T07:32:29.530337+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:26:34.535447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:34.535447+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"326'35","reported_seq":782,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499198+0000","last_change":"2024-02-05T07:29:45.972745+0000","last_active":"2024-02-05T07:32:29.499198+0000","last_peered":"2024-02-05T07:32:29.499198+0000","last_clean":"2024-02-05T07:32:29.499198+0000","last_became_active":"2024-02-05T07:28:26.598831+0000","last_became_peered":"2024-02-05T07:28:26.598831+0000","last_unstale":"2024-02-05T07:32:29.499198+0000","last_undegraded":"2024-02-05T07:32:29.499198+0000","last_fullsized":"2024-02-05T07:32:29.499198+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:29:45.972589+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.972589+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":469,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.520366+0000","last_change":"2024-02-05T07:29:41.587097+0000","last_active":"2024-02-05T07:32:29.520366+0000","last_peered":"2024-02-05T07:32:29.520366+0000","last_clean":"2024-02-05T07:32:29.520366+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:32:29.520366+0000","last_undegraded":"2024-02-05T07:32:29.520366+0000","last_fullsized":"2024-02-05T07:32:29.520366+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:41.587022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:41.587022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":568,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.504705+0000","last_change":"2024-02-05T07:30:40.489610+0000","last_active":"2024-02-05T07:32:19.504705+0000","last_peered":"2024-02-05T07:32:19.504705+0000","last_clean":"2024-02-05T07:32:19.504705+0000","last_became_active":"2024-02-05T07:30:40.489120+0000","last_became_peered":"2024-02-05T07:30:40.489120+0000","last_unstale":"2024-02-05T07:32:19.504705+0000","last_undegraded":"2024-02-05T07:32:19.504705+0000","last_fullsized":"2024-02-05T07:32:19.504705+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:45.254637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.254637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"402'7262","reported_seq":7379,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:31.818215+0000","last_change":"2024-02-05T07:30:00.509399+0000","last_active":"2024-02-05T07:32:31.818215+0000","last_peered":"2024-02-05T07:32:31.818215+0000","last_clean":"2024-02-05T07:32:31.818215+0000","last_became_active":"2024-02-05T07:30:00.509120+0000","last_became_peered":"2024-02-05T07:30:00.509120+0000","last_unstale":"2024-02-05T07:32:31.818215+0000","last_undegraded":"2024-02-05T07:32:31.818215+0000","last_fullsized":"2024-02-05T07:32:31.818215+0000","mapping_epoch":349,"log_start":"376'4171","ondisk_log_start":"376'4171","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3091,"log_dups_size":0,"ondisk_log_size":3091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172228608,"num_objects":2628,"num_object_clones":0,"num_object_copies":7884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7262,"num_write_kb":316480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":561,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:21.853092+0000","last_change":"2024-02-05T07:32:21.853114+0000","last_active":"2024-02-05T07:32:21.853092+0000","last_peered":"2024-02-05T07:32:21.853092+0000","last_clean":"2024-02-05T07:32:21.853092+0000","last_became_active":"2024-02-05T07:31:36.850075+0000","last_became_peered":"2024-02-05T07:31:36.850075+0000","last_unstale":"2024-02-05T07:32:21.853092+0000","last_undegraded":"2024-02-05T07:32:21.853092+0000","last_fullsized":"2024-02-05T07:32:21.853092+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:21.853039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:21.853039+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:42.459463+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":343,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.633027+0000","last_change":"2024-02-05T07:29:39.912134+0000","last_active":"2024-02-05T07:32:19.633027+0000","last_peered":"2024-02-05T07:32:19.633027+0000","last_clean":"2024-02-05T07:32:19.633027+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:32:19.633027+0000","last_undegraded":"2024-02-05T07:32:19.633027+0000","last_fullsized":"2024-02-05T07:32:19.633027+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:29:39.912035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:39.912035+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"326'25","reported_seq":774,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.521096+0000","last_change":"2024-02-05T07:30:36.908113+0000","last_active":"2024-02-05T07:32:29.521096+0000","last_peered":"2024-02-05T07:32:29.521096+0000","last_clean":"2024-02-05T07:32:29.521096+0000","last_became_active":"2024-02-05T07:27:03.361342+0000","last_became_peered":"2024-02-05T07:27:03.361342+0000","last_unstale":"2024-02-05T07:32:29.521096+0000","last_undegraded":"2024-02-05T07:32:29.521096+0000","last_fullsized":"2024-02-05T07:32:29.521096+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:30:36.907989+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.907989+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":300,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.523648+0000","last_change":"2024-02-05T07:29:44.015577+0000","last_active":"2024-02-05T07:32:29.523648+0000","last_peered":"2024-02-05T07:32:29.523648+0000","last_clean":"2024-02-05T07:32:29.523648+0000","last_became_active":"2024-02-05T07:28:26.596096+0000","last_became_peered":"2024-02-05T07:28:26.596096+0000","last_unstale":"2024-02-05T07:32:29.523648+0000","last_undegraded":"2024-02-05T07:32:29.523648+0000","last_fullsized":"2024-02-05T07:32:29.523648+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:44.015468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:29:44.015468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":289,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.530390+0000","last_change":"2024-02-05T07:27:16.831397+0000","last_active":"2024-02-05T07:32:29.530390+0000","last_peered":"2024-02-05T07:32:29.530390+0000","last_clean":"2024-02-05T07:32:29.530390+0000","last_became_active":"2024-02-05T07:27:16.829863+0000","last_became_peered":"2024-02-05T07:27:16.829863+0000","last_unstale":"2024-02-05T07:32:29.530390+0000","last_undegraded":"2024-02-05T07:32:29.530390+0000","last_fullsized":"2024-02-05T07:32:29.530390+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:52.847946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:52.847946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"402'7286","reported_seq":12854,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:30.309886+0000","last_change":"2024-02-05T07:32:01.564512+0000","last_active":"2024-02-05T07:32:30.309886+0000","last_peered":"2024-02-05T07:32:30.309886+0000","last_clean":"2024-02-05T07:32:30.309886+0000","last_became_active":"2024-02-05T07:32:01.559320+0000","last_became_peered":"2024-02-05T07:32:01.559320+0000","last_unstale":"2024-02-05T07:32:30.309886+0000","last_undegraded":"2024-02-05T07:32:30.309886+0000","last_fullsized":"2024-02-05T07:32:30.309886+0000","mapping_epoch":395,"log_start":"376'3780","ondisk_log_start":"376'3780","created":349,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3506,"log_dups_size":0,"ondisk_log_size":3506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181534720,"num_objects":2770,"num_object_clones":0,"num_object_copies":8310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7286,"num_write_kb":321792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2721,"num_bytes_recovered":178323456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":552,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:20.818853+0000","last_change":"2024-02-05T07:32:20.818876+0000","last_active":"2024-02-05T07:32:20.818853+0000","last_peered":"2024-02-05T07:32:20.818853+0000","last_clean":"2024-02-05T07:32:20.818853+0000","last_became_active":"2024-02-05T07:31:36.840534+0000","last_became_peered":"2024-02-05T07:31:36.840534+0000","last_unstale":"2024-02-05T07:32:20.818853+0000","last_undegraded":"2024-02-05T07:32:20.818853+0000","last_fullsized":"2024-02-05T07:32:20.818853+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:20.818799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:20.818799+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:29.517723+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":543,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.504356+0000","last_change":"2024-02-05T07:30:35.140369+0000","last_active":"2024-02-05T07:32:19.504356+0000","last_peered":"2024-02-05T07:32:19.504356+0000","last_clean":"2024-02-05T07:32:19.504356+0000","last_became_active":"2024-02-05T07:28:26.589673+0000","last_became_peered":"2024-02-05T07:28:26.589673+0000","last_unstale":"2024-02-05T07:32:19.504356+0000","last_undegraded":"2024-02-05T07:32:19.504356+0000","last_fullsized":"2024-02-05T07:32:19.504356+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:35.140291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:30:35.140291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"326'29","reported_seq":342,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511586+0000","last_change":"2024-02-05T07:30:40.516301+0000","last_active":"2024-02-05T07:32:29.511586+0000","last_peered":"2024-02-05T07:32:29.511586+0000","last_clean":"2024-02-05T07:32:29.511586+0000","last_became_active":"2024-02-05T07:30:40.481234+0000","last_became_peered":"2024-02-05T07:30:40.481234+0000","last_unstale":"2024-02-05T07:32:29.511586+0000","last_undegraded":"2024-02-05T07:32:29.511586+0000","last_fullsized":"2024-02-05T07:32:29.511586+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'29","last_scrub_stamp":"2024-02-05T07:30:37.106672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:37.106672+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":723,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.515742+0000","last_change":"2024-02-05T07:29:53.699733+0000","last_active":"2024-02-05T07:32:29.515742+0000","last_peered":"2024-02-05T07:32:29.515742+0000","last_clean":"2024-02-05T07:32:29.515742+0000","last_became_active":"2024-02-05T07:27:03.361028+0000","last_became_peered":"2024-02-05T07:27:03.361028+0000","last_unstale":"2024-02-05T07:32:29.515742+0000","last_undegraded":"2024-02-05T07:32:29.515742+0000","last_fullsized":"2024-02-05T07:32:29.515742+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.699651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.699651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":706,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.515363+0000","last_change":"2024-02-05T07:28:49.555346+0000","last_active":"2024-02-05T07:32:29.515363+0000","last_peered":"2024-02-05T07:32:29.515363+0000","last_clean":"2024-02-05T07:32:29.515363+0000","last_became_active":"2024-02-05T07:28:25.840850+0000","last_became_peered":"2024-02-05T07:28:25.840850+0000","last_unstale":"2024-02-05T07:32:29.515363+0000","last_undegraded":"2024-02-05T07:32:29.515363+0000","last_fullsized":"2024-02-05T07:32:29.515363+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:49.555018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:49.555018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"400'6977","reported_seq":7090,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:28.631918+0000","last_change":"2024-02-05T07:30:00.869708+0000","last_active":"2024-02-05T07:32:28.631918+0000","last_peered":"2024-02-05T07:32:28.631918+0000","last_clean":"2024-02-05T07:32:28.631918+0000","last_became_active":"2024-02-05T07:30:00.869295+0000","last_became_peered":"2024-02-05T07:30:00.869295+0000","last_unstale":"2024-02-05T07:32:28.631918+0000","last_undegraded":"2024-02-05T07:32:28.631918+0000","last_fullsized":"2024-02-05T07:32:28.631918+0000","mapping_epoch":349,"log_start":"374'3328","ondisk_log_start":"374'3328","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3649,"log_dups_size":0,"ondisk_log_size":3649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189726720,"num_objects":2895,"num_object_clones":0,"num_object_copies":8685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6977,"num_write_kb":315904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":688,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511850+0000","last_change":"2024-02-05T07:28:47.605953+0000","last_active":"2024-02-05T07:32:29.511850+0000","last_peered":"2024-02-05T07:32:29.511850+0000","last_clean":"2024-02-05T07:32:29.511850+0000","last_became_active":"2024-02-05T07:28:25.756613+0000","last_became_peered":"2024-02-05T07:28:25.756613+0000","last_unstale":"2024-02-05T07:32:29.511850+0000","last_undegraded":"2024-02-05T07:32:29.511850+0000","last_fullsized":"2024-02-05T07:32:29.511850+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:47.605803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.605803+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":306,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.521704+0000","last_change":"2024-02-05T07:27:03.370712+0000","last_active":"2024-02-05T07:32:29.521704+0000","last_peered":"2024-02-05T07:32:29.521704+0000","last_clean":"2024-02-05T07:32:29.521704+0000","last_became_active":"2024-02-05T07:27:03.370297+0000","last_became_peered":"2024-02-05T07:27:03.370297+0000","last_unstale":"2024-02-05T07:32:29.521704+0000","last_undegraded":"2024-02-05T07:32:29.521704+0000","last_fullsized":"2024-02-05T07:32:29.521704+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:32.652537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:32.652537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"326'24","reported_seq":516,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.520638+0000","last_change":"2024-02-05T07:29:36.583093+0000","last_active":"2024-02-05T07:32:29.520638+0000","last_peered":"2024-02-05T07:32:29.520638+0000","last_clean":"2024-02-05T07:32:29.520638+0000","last_became_active":"2024-02-05T07:24:34.209716+0000","last_became_peered":"2024-02-05T07:24:34.209716+0000","last_unstale":"2024-02-05T07:32:29.520638+0000","last_undegraded":"2024-02-05T07:32:29.520638+0000","last_fullsized":"2024-02-05T07:32:29.520638+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:29:36.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:36.583005+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":242,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:30.423906+0000","last_change":"2024-02-05T07:30:40.520395+0000","last_active":"2024-02-05T07:32:30.423906+0000","last_peered":"2024-02-05T07:32:30.423906+0000","last_clean":"2024-02-05T07:32:30.423906+0000","last_became_active":"2024-02-05T07:30:40.493770+0000","last_became_peered":"2024-02-05T07:30:40.493770+0000","last_unstale":"2024-02-05T07:32:30.423906+0000","last_undegraded":"2024-02-05T07:32:30.423906+0000","last_fullsized":"2024-02-05T07:32:30.423906+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:29:51.291146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.291146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":117,"num_read_kb":112,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":680,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499246+0000","last_change":"2024-02-05T07:27:16.724562+0000","last_active":"2024-02-05T07:32:29.499246+0000","last_peered":"2024-02-05T07:32:29.499246+0000","last_clean":"2024-02-05T07:32:29.499246+0000","last_became_active":"2024-02-05T07:27:16.724280+0000","last_became_peered":"2024-02-05T07:27:16.724280+0000","last_unstale":"2024-02-05T07:32:29.499246+0000","last_undegraded":"2024-02-05T07:32:29.499246+0000","last_fullsized":"2024-02-05T07:32:29.499246+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:30.351655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:26:30.351655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":734,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.515523+0000","last_change":"2024-02-05T07:28:50.545548+0000","last_active":"2024-02-05T07:32:29.515523+0000","last_peered":"2024-02-05T07:32:29.515523+0000","last_clean":"2024-02-05T07:32:29.515523+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:32:29.515523+0000","last_undegraded":"2024-02-05T07:32:29.515523+0000","last_fullsized":"2024-02-05T07:32:29.515523+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:28:50.545437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:50.545437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"402'7480","reported_seq":12053,"reported_epoch":402,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-05T07:32:31.573534+0000","last_change":"2024-02-05T07:31:36.422030+0000","last_active":"2024-02-05T07:32:31.573534+0000","last_peered":"2024-02-05T07:32:31.573534+0000","last_clean":"2024-02-05T07:30:39.467673+0000","last_became_active":"2024-02-05T07:30:41.737377+0000","last_became_peered":"2024-02-05T07:30:41.737377+0000","last_unstale":"2024-02-05T07:32:31.573534+0000","last_undegraded":"2024-02-05T07:30:41.549610+0000","last_fullsized":"2024-02-05T07:30:41.467588+0000","mapping_epoch":365,"log_start":"376'4338","ondisk_log_start":"376'4338","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3142,"log_dups_size":0,"ondisk_log_size":3142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176816128,"num_objects":2698,"num_object_clones":0,"num_object_copies":8094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":166,"num_objects_misplaced":379,"num_objects_unfound":0,"num_objects_dirty":2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7480,"num_write_kb":325696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2227,"num_bytes_recovered":145948672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3","objects":2153},{"shards":"0,3,7","objects":379}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":480,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511678+0000","last_change":"2024-02-05T07:30:40.474637+0000","last_active":"2024-02-05T07:32:29.511678+0000","last_peered":"2024-02-05T07:32:29.511678+0000","last_clean":"2024-02-05T07:32:29.511678+0000","last_became_active":"2024-02-05T07:30:40.474169+0000","last_became_peered":"2024-02-05T07:30:40.474169+0000","last_unstale":"2024-02-05T07:32:29.511678+0000","last_undegraded":"2024-02-05T07:32:29.511678+0000","last_fullsized":"2024-02-05T07:32:29.511678+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:30.221253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.221253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"365'31","reported_seq":755,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.633070+0000","last_change":"2024-02-05T07:30:40.539387+0000","last_active":"2024-02-05T07:32:19.633070+0000","last_peered":"2024-02-05T07:32:19.633070+0000","last_clean":"2024-02-05T07:32:19.633070+0000","last_became_active":"2024-02-05T07:30:40.508686+0000","last_became_peered":"2024-02-05T07:30:40.508686+0000","last_unstale":"2024-02-05T07:32:19.633070+0000","last_undegraded":"2024-02-05T07:32:19.633070+0000","last_fullsized":"2024-02-05T07:32:19.633070+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'30","last_scrub_stamp":"2024-02-05T07:30:37.201930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:37.201930+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":300,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.527632+0000","last_change":"2024-02-05T07:29:43.000854+0000","last_active":"2024-02-05T07:32:29.527632+0000","last_peered":"2024-02-05T07:32:29.527632+0000","last_clean":"2024-02-05T07:32:29.527632+0000","last_became_active":"2024-02-05T07:28:26.595640+0000","last_became_peered":"2024-02-05T07:28:26.595640+0000","last_unstale":"2024-02-05T07:32:29.527632+0000","last_undegraded":"2024-02-05T07:32:29.527632+0000","last_fullsized":"2024-02-05T07:32:29.527632+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:43.000778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:43.000778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":397,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.775050+0000","last_change":"2024-02-05T07:32:19.775077+0000","last_active":"2024-02-05T07:32:19.775050+0000","last_peered":"2024-02-05T07:32:19.775050+0000","last_clean":"2024-02-05T07:32:19.775050+0000","last_became_active":"2024-02-05T07:30:43.516031+0000","last_became_peered":"2024-02-05T07:30:43.516031+0000","last_unstale":"2024-02-05T07:32:19.775050+0000","last_undegraded":"2024-02-05T07:32:19.775050+0000","last_fullsized":"2024-02-05T07:32:19.775050+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:19.774991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:19.774991+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:36.180536+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":289,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.531572+0000","last_change":"2024-02-05T07:27:16.751712+0000","last_active":"2024-02-05T07:32:29.531572+0000","last_peered":"2024-02-05T07:32:29.531572+0000","last_clean":"2024-02-05T07:32:29.531572+0000","last_became_active":"2024-02-05T07:27:16.751341+0000","last_became_peered":"2024-02-05T07:27:16.751341+0000","last_unstale":"2024-02-05T07:32:29.531572+0000","last_undegraded":"2024-02-05T07:32:29.531572+0000","last_fullsized":"2024-02-05T07:32:29.531572+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:36.753098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.753098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"402'7396","reported_seq":12953,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:31.846143+0000","last_change":"2024-02-05T07:31:32.742675+0000","last_active":"2024-02-05T07:32:31.846143+0000","last_peered":"2024-02-05T07:32:31.846143+0000","last_clean":"2024-02-05T07:32:31.846143+0000","last_became_active":"2024-02-05T07:31:32.736052+0000","last_became_peered":"2024-02-05T07:31:32.736052+0000","last_unstale":"2024-02-05T07:32:31.846143+0000","last_undegraded":"2024-02-05T07:32:31.846143+0000","last_fullsized":"2024-02-05T07:32:31.846143+0000","mapping_epoch":380,"log_start":"377'4352","ondisk_log_start":"377'4352","created":349,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3044,"log_dups_size":0,"ondisk_log_size":3044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174063616,"num_objects":2656,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7396,"num_write_kb":321664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2716,"num_bytes_recovered":177995776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":711,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499121+0000","last_change":"2024-02-05T07:32:21.667678+0000","last_active":"2024-02-05T07:32:29.499121+0000","last_peered":"2024-02-05T07:32:29.499121+0000","last_clean":"2024-02-05T07:32:29.499121+0000","last_became_active":"2024-02-05T07:30:43.509733+0000","last_became_peered":"2024-02-05T07:30:43.509733+0000","last_unstale":"2024-02-05T07:32:29.499121+0000","last_undegraded":"2024-02-05T07:32:29.499121+0000","last_fullsized":"2024-02-05T07:32:29.499121+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:21.667581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:21.667581+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:48.654513+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":720,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.514253+0000","last_change":"2024-02-05T07:30:40.504250+0000","last_active":"2024-02-05T07:32:29.514253+0000","last_peered":"2024-02-05T07:32:29.514253+0000","last_clean":"2024-02-05T07:32:29.514253+0000","last_became_active":"2024-02-05T07:30:40.503816+0000","last_became_peered":"2024-02-05T07:30:40.503816+0000","last_unstale":"2024-02-05T07:32:29.514253+0000","last_undegraded":"2024-02-05T07:32:29.514253+0000","last_fullsized":"2024-02-05T07:32:29.514253+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:40.103572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:40.103572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"326'22","reported_seq":779,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.521475+0000","last_change":"2024-02-05T07:30:40.533683+0000","last_active":"2024-02-05T07:32:29.521475+0000","last_peered":"2024-02-05T07:32:29.521475+0000","last_clean":"2024-02-05T07:32:29.521475+0000","last_became_active":"2024-02-05T07:30:40.494460+0000","last_became_peered":"2024-02-05T07:30:40.494460+0000","last_unstale":"2024-02-05T07:32:29.521475+0000","last_undegraded":"2024-02-05T07:32:29.521475+0000","last_fullsized":"2024-02-05T07:32:29.521475+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'18","last_scrub_stamp":"2024-02-05T07:26:24.578301+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.578301+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":745,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.510287+0000","last_change":"2024-02-05T07:29:42.383585+0000","last_active":"2024-02-05T07:32:29.510287+0000","last_peered":"2024-02-05T07:32:29.510287+0000","last_clean":"2024-02-05T07:32:29.510287+0000","last_became_active":"2024-02-05T07:27:16.728869+0000","last_became_peered":"2024-02-05T07:27:16.728869+0000","last_unstale":"2024-02-05T07:32:29.510287+0000","last_undegraded":"2024-02-05T07:32:29.510287+0000","last_fullsized":"2024-02-05T07:32:29.510287+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:42.383469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:29:42.383469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":732,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.515677+0000","last_change":"2024-02-05T07:29:45.546235+0000","last_active":"2024-02-05T07:32:29.515677+0000","last_peered":"2024-02-05T07:32:29.515677+0000","last_clean":"2024-02-05T07:32:29.515677+0000","last_became_active":"2024-02-05T07:28:25.750373+0000","last_became_peered":"2024-02-05T07:28:25.750373+0000","last_unstale":"2024-02-05T07:32:29.515677+0000","last_undegraded":"2024-02-05T07:32:29.515677+0000","last_fullsized":"2024-02-05T07:32:29.515677+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:45.546152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.546152+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":580,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.504908+0000","last_change":"2024-02-05T07:30:40.489722+0000","last_active":"2024-02-05T07:32:19.504908+0000","last_peered":"2024-02-05T07:32:19.504908+0000","last_clean":"2024-02-05T07:32:19.504908+0000","last_became_active":"2024-02-05T07:30:40.489314+0000","last_became_peered":"2024-02-05T07:30:40.489314+0000","last_unstale":"2024-02-05T07:32:19.504908+0000","last_undegraded":"2024-02-05T07:32:19.504908+0000","last_fullsized":"2024-02-05T07:32:19.504908+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:38.191811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:29:38.191811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"402'7117","reported_seq":12498,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:31.558778+0000","last_change":"2024-02-05T07:31:37.794109+0000","last_active":"2024-02-05T07:32:31.558778+0000","last_peered":"2024-02-05T07:32:31.558778+0000","last_clean":"2024-02-05T07:32:31.558778+0000","last_became_active":"2024-02-05T07:31:37.788267+0000","last_became_peered":"2024-02-05T07:31:37.788267+0000","last_unstale":"2024-02-05T07:32:31.558778+0000","last_undegraded":"2024-02-05T07:32:31.558778+0000","last_fullsized":"2024-02-05T07:32:31.558778+0000","mapping_epoch":384,"log_start":"376'3905","ondisk_log_start":"376'3905","created":349,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3212,"log_dups_size":0,"ondisk_log_size":3212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170459136,"num_objects":2601,"num_object_clones":0,"num_object_copies":7803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":310976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2627,"num_bytes_recovered":172163072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":706,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.498636+0000","last_change":"2024-02-05T07:29:55.246298+0000","last_active":"2024-02-05T07:32:29.498636+0000","last_peered":"2024-02-05T07:32:29.498636+0000","last_clean":"2024-02-05T07:32:29.498636+0000","last_became_active":"2024-02-05T07:28:14.305034+0000","last_became_peered":"2024-02-05T07:28:14.305034+0000","last_unstale":"2024-02-05T07:32:29.498636+0000","last_undegraded":"2024-02-05T07:32:29.498636+0000","last_fullsized":"2024-02-05T07:32:29.498636+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:29:55.246214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.246214+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":461,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.517007+0000","last_change":"2024-02-05T07:29:48.627198+0000","last_active":"2024-02-05T07:32:29.517007+0000","last_peered":"2024-02-05T07:32:29.517007+0000","last_clean":"2024-02-05T07:32:29.517007+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:32:29.517007+0000","last_undegraded":"2024-02-05T07:32:29.517007+0000","last_fullsized":"2024-02-05T07:32:29.517007+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.627107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.627107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"326'44","reported_seq":625,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511463+0000","last_change":"2024-02-05T07:30:36.009087+0000","last_active":"2024-02-05T07:32:29.511463+0000","last_peered":"2024-02-05T07:32:29.511463+0000","last_clean":"2024-02-05T07:32:29.511463+0000","last_became_active":"2024-02-05T07:27:16.708475+0000","last_became_peered":"2024-02-05T07:27:16.708475+0000","last_unstale":"2024-02-05T07:32:29.511463+0000","last_undegraded":"2024-02-05T07:32:29.511463+0000","last_fullsized":"2024-02-05T07:32:29.511463+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'44","last_scrub_stamp":"2024-02-05T07:30:36.008885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.008885+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":103,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.510737+0000","last_change":"2024-02-05T07:30:32.964745+0000","last_active":"2024-02-05T07:32:29.510737+0000","last_peered":"2024-02-05T07:32:29.510737+0000","last_clean":"2024-02-05T07:32:29.510737+0000","last_became_active":"2024-02-05T07:30:30.524566+0000","last_became_peered":"2024-02-05T07:30:30.524566+0000","last_unstale":"2024-02-05T07:32:29.510737+0000","last_undegraded":"2024-02-05T07:32:29.510737+0000","last_fullsized":"2024-02-05T07:32:29.510737+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:32.964622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:32.964622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":631,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.520750+0000","last_change":"2024-02-05T07:28:55.477529+0000","last_active":"2024-02-05T07:32:29.520750+0000","last_peered":"2024-02-05T07:32:29.520750+0000","last_clean":"2024-02-05T07:32:29.520750+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:32:29.520750+0000","last_undegraded":"2024-02-05T07:32:29.520750+0000","last_fullsized":"2024-02-05T07:32:29.520750+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:28:55.477280+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":303,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.523255+0000","last_change":"2024-02-05T07:30:40.508412+0000","last_active":"2024-02-05T07:32:29.523255+0000","last_peered":"2024-02-05T07:32:29.523255+0000","last_clean":"2024-02-05T07:32:29.523255+0000","last_became_active":"2024-02-05T07:30:40.508071+0000","last_became_peered":"2024-02-05T07:30:40.508071+0000","last_unstale":"2024-02-05T07:32:29.523255+0000","last_undegraded":"2024-02-05T07:32:29.523255+0000","last_fullsized":"2024-02-05T07:32:29.523255+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:42.075445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:42.075445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.6","version":"402'7379","reported_seq":7496,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:30.894819+0000","last_change":"2024-02-05T07:30:00.512361+0000","last_active":"2024-02-05T07:32:30.894819+0000","last_peered":"2024-02-05T07:32:30.894819+0000","last_clean":"2024-02-05T07:32:30.894819+0000","last_became_active":"2024-02-05T07:30:00.511728+0000","last_became_peered":"2024-02-05T07:30:00.511728+0000","last_unstale":"2024-02-05T07:32:30.894819+0000","last_undegraded":"2024-02-05T07:32:30.894819+0000","last_fullsized":"2024-02-05T07:32:30.894819+0000","mapping_epoch":349,"log_start":"374'3489","ondisk_log_start":"374'3489","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3890,"log_dups_size":0,"ondisk_log_size":3890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179503104,"num_objects":2739,"num_object_clones":0,"num_object_copies":8217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7379,"num_write_kb":323776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":636,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499997+0000","last_change":"2024-02-05T07:32:23.628807+0000","last_active":"2024-02-05T07:32:29.499997+0000","last_peered":"2024-02-05T07:32:29.499997+0000","last_clean":"2024-02-05T07:32:29.499997+0000","last_became_active":"2024-02-05T07:31:36.842497+0000","last_became_peered":"2024-02-05T07:31:36.842497+0000","last_unstale":"2024-02-05T07:32:29.499997+0000","last_undegraded":"2024-02-05T07:32:29.499997+0000","last_fullsized":"2024-02-05T07:32:29.499997+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:23.628698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:23.628698+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:45.560553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"326'15","reported_seq":504,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511957+0000","last_change":"2024-02-05T07:29:51.665907+0000","last_active":"2024-02-05T07:32:29.511957+0000","last_peered":"2024-02-05T07:32:29.511957+0000","last_clean":"2024-02-05T07:32:29.511957+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:32:29.511957+0000","last_undegraded":"2024-02-05T07:32:29.511957+0000","last_fullsized":"2024-02-05T07:32:29.511957+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:29:51.665808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.665808+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":300,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.524411+0000","last_change":"2024-02-05T07:30:38.998491+0000","last_active":"2024-02-05T07:32:29.524411+0000","last_peered":"2024-02-05T07:32:29.524411+0000","last_clean":"2024-02-05T07:32:29.524411+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:32:29.524411+0000","last_undegraded":"2024-02-05T07:32:29.524411+0000","last_fullsized":"2024-02-05T07:32:29.524411+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:38.995586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:38.995586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":720,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.510557+0000","last_change":"2024-02-05T07:28:41.103849+0000","last_active":"2024-02-05T07:32:29.510557+0000","last_peered":"2024-02-05T07:32:29.510557+0000","last_clean":"2024-02-05T07:32:29.510557+0000","last_became_active":"2024-02-05T07:27:16.708205+0000","last_became_peered":"2024-02-05T07:27:16.708205+0000","last_unstale":"2024-02-05T07:32:29.510557+0000","last_undegraded":"2024-02-05T07:32:29.510557+0000","last_fullsized":"2024-02-05T07:32:29.510557+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:41.103739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:41.103739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":88,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.510472+0000","last_change":"2024-02-05T07:30:40.488052+0000","last_active":"2024-02-05T07:32:29.510472+0000","last_peered":"2024-02-05T07:32:29.510472+0000","last_clean":"2024-02-05T07:32:29.510472+0000","last_became_active":"2024-02-05T07:30:40.486401+0000","last_became_peered":"2024-02-05T07:30:40.486401+0000","last_unstale":"2024-02-05T07:32:29.510472+0000","last_undegraded":"2024-02-05T07:32:29.510472+0000","last_fullsized":"2024-02-05T07:32:29.510472+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:30.076226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:30.076226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"402'7381","reported_seq":12149,"reported_epoch":402,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-05T07:32:31.578692+0000","last_change":"2024-02-05T07:32:16.031000+0000","last_active":"2024-02-05T07:32:31.578692+0000","last_peered":"2024-02-05T07:32:31.578692+0000","last_clean":"2024-02-05T07:30:39.445489+0000","last_became_active":"2024-02-05T07:30:41.737849+0000","last_became_peered":"2024-02-05T07:30:41.737849+0000","last_unstale":"2024-02-05T07:32:31.578692+0000","last_undegraded":"2024-02-05T07:32:16.028507+0000","last_fullsized":"2024-02-05T07:30:41.471627+0000","mapping_epoch":365,"log_start":"376'4179","ondisk_log_start":"376'4179","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3202,"log_dups_size":0,"ondisk_log_size":3202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178585600,"num_objects":2725,"num_object_clones":0,"num_object_copies":8175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":44,"num_objects_misplaced":372,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7381,"num_write_kb":323392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2325,"num_bytes_recovered":152371200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":2309},{"shards":"2,3,7","objects":372}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":548,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:24.854940+0000","last_change":"2024-02-05T07:32:24.854963+0000","last_active":"2024-02-05T07:32:24.854940+0000","last_peered":"2024-02-05T07:32:24.854940+0000","last_clean":"2024-02-05T07:32:24.854940+0000","last_became_active":"2024-02-05T07:28:26.596711+0000","last_became_peered":"2024-02-05T07:28:26.596711+0000","last_unstale":"2024-02-05T07:32:24.854940+0000","last_undegraded":"2024-02-05T07:32:24.854940+0000","last_fullsized":"2024-02-05T07:32:24.854940+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:24.854881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:24.854881+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:38.855880+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"326'35","reported_seq":618,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.504464+0000","last_change":"2024-02-05T07:30:40.520826+0000","last_active":"2024-02-05T07:32:19.504464+0000","last_peered":"2024-02-05T07:32:19.504464+0000","last_clean":"2024-02-05T07:32:19.504464+0000","last_became_active":"2024-02-05T07:30:40.482753+0000","last_became_peered":"2024-02-05T07:30:40.482753+0000","last_unstale":"2024-02-05T07:32:19.504464+0000","last_undegraded":"2024-02-05T07:32:19.504464+0000","last_fullsized":"2024-02-05T07:32:19.504464+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:29:45.981520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.981520+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":591,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.504876+0000","last_change":"2024-02-05T07:30:40.508017+0000","last_active":"2024-02-05T07:32:19.504876+0000","last_peered":"2024-02-05T07:32:19.504876+0000","last_clean":"2024-02-05T07:32:19.504876+0000","last_became_active":"2024-02-05T07:30:40.507705+0000","last_became_peered":"2024-02-05T07:30:40.507705+0000","last_unstale":"2024-02-05T07:32:19.504876+0000","last_undegraded":"2024-02-05T07:32:19.504876+0000","last_fullsized":"2024-02-05T07:32:19.504876+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:39.169995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:39.169995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":720,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511049+0000","last_change":"2024-02-05T07:30:40.480847+0000","last_active":"2024-02-05T07:32:29.511049+0000","last_peered":"2024-02-05T07:32:29.511049+0000","last_clean":"2024-02-05T07:32:29.511049+0000","last_became_active":"2024-02-05T07:30:40.479288+0000","last_became_peered":"2024-02-05T07:30:40.479288+0000","last_unstale":"2024-02-05T07:32:29.511049+0000","last_undegraded":"2024-02-05T07:32:29.511049+0000","last_fullsized":"2024-02-05T07:32:29.511049+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:55.173628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:55.173628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":501,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.498800+0000","last_change":"2024-02-05T07:30:40.497142+0000","last_active":"2024-02-05T07:32:29.498800+0000","last_peered":"2024-02-05T07:32:29.498800+0000","last_clean":"2024-02-05T07:32:29.498800+0000","last_became_active":"2024-02-05T07:30:40.496822+0000","last_became_peered":"2024-02-05T07:30:40.496822+0000","last_unstale":"2024-02-05T07:32:29.498800+0000","last_undegraded":"2024-02-05T07:32:29.498800+0000","last_fullsized":"2024-02-05T07:32:29.498800+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.262537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.4","version":"402'7505","reported_seq":7622,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:31.834497+0000","last_change":"2024-02-05T07:30:01.030823+0000","last_active":"2024-02-05T07:32:31.834497+0000","last_peered":"2024-02-05T07:32:31.834497+0000","last_clean":"2024-02-05T07:32:31.834497+0000","last_became_active":"2024-02-05T07:30:01.030502+0000","last_became_peered":"2024-02-05T07:30:01.030502+0000","last_unstale":"2024-02-05T07:32:31.834497+0000","last_undegraded":"2024-02-05T07:32:31.834497+0000","last_fullsized":"2024-02-05T07:32:31.834497+0000","mapping_epoch":349,"log_start":"376'4423","ondisk_log_start":"376'4423","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3082,"log_dups_size":0,"ondisk_log_size":3082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175570944,"num_objects":2679,"num_object_clones":0,"num_object_copies":8037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7505,"num_write_kb":325888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":960,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.522315+0000","last_change":"2024-02-05T07:31:36.840582+0000","last_active":"2024-02-05T07:32:29.522315+0000","last_peered":"2024-02-05T07:32:29.522315+0000","last_clean":"2024-02-05T07:32:29.522315+0000","last_became_active":"2024-02-05T07:31:36.840096+0000","last_became_peered":"2024-02-05T07:31:36.840096+0000","last_unstale":"2024-02-05T07:32:29.522315+0000","last_undegraded":"2024-02-05T07:32:29.522315+0000","last_fullsized":"2024-02-05T07:32:29.522315+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:40.181881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:40.181881+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:32:31.684721+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":716,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.498696+0000","last_change":"2024-02-05T07:29:39.290437+0000","last_active":"2024-02-05T07:32:29.498696+0000","last_peered":"2024-02-05T07:32:29.498696+0000","last_clean":"2024-02-05T07:32:29.498696+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:32:29.498696+0000","last_undegraded":"2024-02-05T07:32:29.498696+0000","last_fullsized":"2024-02-05T07:32:29.498696+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:39.290320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:39.290320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"325'18","reported_seq":804,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499331+0000","last_change":"2024-02-05T07:29:49.022612+0000","last_active":"2024-02-05T07:32:29.499331+0000","last_peered":"2024-02-05T07:32:29.499331+0000","last_clean":"2024-02-05T07:32:29.499331+0000","last_became_active":"2024-02-05T07:27:16.706871+0000","last_became_peered":"2024-02-05T07:27:16.706871+0000","last_unstale":"2024-02-05T07:32:29.499331+0000","last_undegraded":"2024-02-05T07:32:29.499331+0000","last_fullsized":"2024-02-05T07:32:29.499331+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'18","last_scrub_stamp":"2024-02-05T07:29:49.021772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.021772+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":85,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.500152+0000","last_change":"2024-02-05T07:30:40.519397+0000","last_active":"2024-02-05T07:32:29.500152+0000","last_peered":"2024-02-05T07:32:29.500152+0000","last_clean":"2024-02-05T07:32:29.500152+0000","last_became_active":"2024-02-05T07:30:40.518949+0000","last_became_peered":"2024-02-05T07:30:40.518949+0000","last_unstale":"2024-02-05T07:32:29.500152+0000","last_undegraded":"2024-02-05T07:32:29.500152+0000","last_fullsized":"2024-02-05T07:32:29.500152+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:24.102597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.102597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":303,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.526840+0000","last_change":"2024-02-05T07:30:40.486334+0000","last_active":"2024-02-05T07:32:29.526840+0000","last_peered":"2024-02-05T07:32:29.526840+0000","last_clean":"2024-02-05T07:32:29.526840+0000","last_became_active":"2024-02-05T07:30:40.485991+0000","last_became_peered":"2024-02-05T07:30:40.485991+0000","last_unstale":"2024-02-05T07:32:29.526840+0000","last_undegraded":"2024-02-05T07:32:29.526840+0000","last_fullsized":"2024-02-05T07:32:29.526840+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:48.088229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.088229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"402'7389","reported_seq":13004,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:31.854745+0000","last_change":"2024-02-05T07:31:48.466349+0000","last_active":"2024-02-05T07:32:31.854745+0000","last_peered":"2024-02-05T07:32:31.854745+0000","last_clean":"2024-02-05T07:32:31.854745+0000","last_became_active":"2024-02-05T07:31:48.459417+0000","last_became_peered":"2024-02-05T07:31:48.459417+0000","last_unstale":"2024-02-05T07:32:31.854745+0000","last_undegraded":"2024-02-05T07:32:31.854745+0000","last_fullsized":"2024-02-05T07:32:31.854745+0000","mapping_epoch":391,"log_start":"376'4347","ondisk_log_start":"376'4347","created":349,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3042,"log_dups_size":0,"ondisk_log_size":3042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174391296,"num_objects":2661,"num_object_clones":0,"num_object_copies":7983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7389,"num_write_kb":321600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2745,"num_bytes_recovered":179896320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":552,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499869+0000","last_change":"2024-02-05T07:32:22.621687+0000","last_active":"2024-02-05T07:32:29.499869+0000","last_peered":"2024-02-05T07:32:29.499869+0000","last_clean":"2024-02-05T07:32:29.499869+0000","last_became_active":"2024-02-05T07:31:36.837268+0000","last_became_peered":"2024-02-05T07:31:36.837268+0000","last_unstale":"2024-02-05T07:32:29.499869+0000","last_undegraded":"2024-02-05T07:32:29.499869+0000","last_fullsized":"2024-02-05T07:32:29.499869+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:22.621608+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_clean_scrub_stamp":"2024-02-05T07:32:22.621608+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:48.097590+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"365'12","reported_seq":724,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.633013+0000","last_change":"2024-02-05T07:30:40.519998+0000","last_active":"2024-02-05T07:32:19.633013+0000","last_peered":"2024-02-05T07:32:19.633013+0000","last_clean":"2024-02-05T07:32:19.633013+0000","last_became_active":"2024-02-05T07:30:40.500113+0000","last_became_peered":"2024-02-05T07:30:40.500113+0000","last_unstale":"2024-02-05T07:32:19.633013+0000","last_undegraded":"2024-02-05T07:32:19.633013+0000","last_fullsized":"2024-02-05T07:32:19.633013+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'11","last_scrub_stamp":"2024-02-05T07:28:50.147503+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:28:50.147503+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"326'27","reported_seq":338,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.504122+0000","last_change":"2024-02-05T07:30:40.526838+0000","last_active":"2024-02-05T07:32:19.504122+0000","last_peered":"2024-02-05T07:32:19.504122+0000","last_clean":"2024-02-05T07:32:19.504122+0000","last_became_active":"2024-02-05T07:30:40.497982+0000","last_became_peered":"2024-02-05T07:30:40.497982+0000","last_unstale":"2024-02-05T07:32:19.504122+0000","last_undegraded":"2024-02-05T07:32:19.504122+0000","last_fullsized":"2024-02-05T07:32:19.504122+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:28:49.179472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:28:49.179472+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":101,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.500338+0000","last_change":"2024-02-05T07:30:40.530544+0000","last_active":"2024-02-05T07:32:29.500338+0000","last_peered":"2024-02-05T07:32:29.500338+0000","last_clean":"2024-02-05T07:32:29.500338+0000","last_became_active":"2024-02-05T07:30:40.503903+0000","last_became_peered":"2024-02-05T07:30:40.503903+0000","last_unstale":"2024-02-05T07:32:29.500338+0000","last_undegraded":"2024-02-05T07:32:29.500338+0000","last_fullsized":"2024-02-05T07:32:29.500338+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:26:37.022129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:37.022129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":373,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.504330+0000","last_change":"2024-02-05T07:30:40.130268+0000","last_active":"2024-02-05T07:32:19.504330+0000","last_peered":"2024-02-05T07:32:19.504330+0000","last_clean":"2024-02-05T07:32:19.504330+0000","last_became_active":"2024-02-05T07:28:26.589671+0000","last_became_peered":"2024-02-05T07:28:26.589671+0000","last_unstale":"2024-02-05T07:32:19.504330+0000","last_undegraded":"2024-02-05T07:32:19.504330+0000","last_fullsized":"2024-02-05T07:32:19.504330+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.130170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":555,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.520488+0000","last_change":"2024-02-05T07:30:43.523848+0000","last_active":"2024-02-05T07:32:29.520488+0000","last_peered":"2024-02-05T07:32:29.520488+0000","last_clean":"2024-02-05T07:32:29.520488+0000","last_became_active":"2024-02-05T07:30:43.523108+0000","last_became_peered":"2024-02-05T07:30:43.523108+0000","last_unstale":"2024-02-05T07:32:29.520488+0000","last_undegraded":"2024-02-05T07:32:29.520488+0000","last_fullsized":"2024-02-05T07:32:29.520488+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:48.594280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.594280+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"326'12","reported_seq":320,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.531378+0000","last_change":"2024-02-05T07:30:40.526400+0000","last_active":"2024-02-05T07:32:29.531378+0000","last_peered":"2024-02-05T07:32:29.531378+0000","last_clean":"2024-02-05T07:32:29.531378+0000","last_became_active":"2024-02-05T07:30:40.496980+0000","last_became_peered":"2024-02-05T07:30:40.496980+0000","last_unstale":"2024-02-05T07:32:29.531378+0000","last_undegraded":"2024-02-05T07:32:29.531378+0000","last_fullsized":"2024-02-05T07:32:29.531378+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-02-05T07:26:36.068507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.068507+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":517,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:26.858445+0000","last_change":"2024-02-05T07:32:26.858476+0000","last_active":"2024-02-05T07:32:26.858445+0000","last_peered":"2024-02-05T07:32:26.858445+0000","last_clean":"2024-02-05T07:32:26.858445+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:32:26.858445+0000","last_undegraded":"2024-02-05T07:32:26.858445+0000","last_fullsized":"2024-02-05T07:32:26.858445+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:32:26.858391+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:32:26.858391+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:36.213638+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":300,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.522908+0000","last_change":"2024-02-05T07:30:27.965535+0000","last_active":"2024-02-05T07:32:29.522908+0000","last_peered":"2024-02-05T07:32:29.522908+0000","last_clean":"2024-02-05T07:32:29.522908+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:32:29.522908+0000","last_undegraded":"2024-02-05T07:32:29.522908+0000","last_fullsized":"2024-02-05T07:32:29.522908+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:27.965449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:27.965449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":567,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.505028+0000","last_change":"2024-02-05T07:30:40.509687+0000","last_active":"2024-02-05T07:32:19.505028+0000","last_peered":"2024-02-05T07:32:19.505028+0000","last_clean":"2024-02-05T07:32:19.505028+0000","last_became_active":"2024-02-05T07:30:40.509216+0000","last_became_peered":"2024-02-05T07:30:40.509216+0000","last_unstale":"2024-02-05T07:32:19.505028+0000","last_undegraded":"2024-02-05T07:32:19.505028+0000","last_fullsized":"2024-02-05T07:32:19.505028+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:55.256094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:28:55.256094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":689,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511434+0000","last_change":"2024-02-05T07:28:45.702694+0000","last_active":"2024-02-05T07:32:29.511434+0000","last_peered":"2024-02-05T07:32:29.511434+0000","last_clean":"2024-02-05T07:32:29.511434+0000","last_became_active":"2024-02-05T07:28:19.311737+0000","last_became_peered":"2024-02-05T07:28:19.311737+0000","last_unstale":"2024-02-05T07:32:29.511434+0000","last_undegraded":"2024-02-05T07:32:29.511434+0000","last_fullsized":"2024-02-05T07:32:29.511434+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:45.702565+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.702565+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.702565+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"325'15","reported_seq":324,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.523190+0000","last_change":"2024-02-05T07:30:40.524523+0000","last_active":"2024-02-05T07:32:29.523190+0000","last_peered":"2024-02-05T07:32:29.523190+0000","last_clean":"2024-02-05T07:32:29.523190+0000","last_became_active":"2024-02-05T07:30:40.485837+0000","last_became_peered":"2024-02-05T07:30:40.485837+0000","last_unstale":"2024-02-05T07:32:29.523190+0000","last_undegraded":"2024-02-05T07:32:29.523190+0000","last_fullsized":"2024-02-05T07:32:29.523190+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:26:28.037945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:28.037945+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":592,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499290+0000","last_change":"2024-02-05T07:30:40.498263+0000","last_active":"2024-02-05T07:32:29.499290+0000","last_peered":"2024-02-05T07:32:29.499290+0000","last_clean":"2024-02-05T07:32:29.499290+0000","last_became_active":"2024-02-05T07:30:40.497956+0000","last_became_peered":"2024-02-05T07:30:40.497956+0000","last_unstale":"2024-02-05T07:32:29.499290+0000","last_undegraded":"2024-02-05T07:32:29.499290+0000","last_fullsized":"2024-02-05T07:32:29.499290+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:33.183427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:30:33.183427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":295,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.531481+0000","last_change":"2024-02-05T07:30:40.496550+0000","last_active":"2024-02-05T07:32:29.531481+0000","last_peered":"2024-02-05T07:32:29.531481+0000","last_clean":"2024-02-05T07:32:29.531481+0000","last_became_active":"2024-02-05T07:30:40.496076+0000","last_became_peered":"2024-02-05T07:30:40.496076+0000","last_unstale":"2024-02-05T07:32:29.531481+0000","last_undegraded":"2024-02-05T07:32:29.531481+0000","last_fullsized":"2024-02-05T07:32:29.531481+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:59.450191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:59.450191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":709,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511581+0000","last_change":"2024-02-05T07:30:40.488677+0000","last_active":"2024-02-05T07:32:29.511581+0000","last_peered":"2024-02-05T07:32:29.511581+0000","last_clean":"2024-02-05T07:32:29.511581+0000","last_became_active":"2024-02-05T07:30:40.487455+0000","last_became_peered":"2024-02-05T07:30:40.487455+0000","last_unstale":"2024-02-05T07:32:29.511581+0000","last_undegraded":"2024-02-05T07:32:29.511581+0000","last_fullsized":"2024-02-05T07:32:29.511581+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:53.115096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:28:53.115096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":397,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.503954+0000","last_change":"2024-02-05T07:32:18.784193+0000","last_active":"2024-02-05T07:32:19.503954+0000","last_peered":"2024-02-05T07:32:19.503954+0000","last_clean":"2024-02-05T07:32:19.503954+0000","last_became_active":"2024-02-05T07:30:43.466854+0000","last_became_peered":"2024-02-05T07:30:43.466854+0000","last_unstale":"2024-02-05T07:32:19.503954+0000","last_undegraded":"2024-02-05T07:32:19.503954+0000","last_fullsized":"2024-02-05T07:32:19.503954+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:18.784071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:18.784071+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:42.865761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"326'19","reported_seq":651,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.523863+0000","last_change":"2024-02-05T07:30:40.503823+0000","last_active":"2024-02-05T07:32:29.523863+0000","last_peered":"2024-02-05T07:32:29.523863+0000","last_clean":"2024-02-05T07:32:29.523863+0000","last_became_active":"2024-02-05T07:30:40.483700+0000","last_became_peered":"2024-02-05T07:30:40.483700+0000","last_unstale":"2024-02-05T07:32:29.523863+0000","last_undegraded":"2024-02-05T07:32:29.523863+0000","last_fullsized":"2024-02-05T07:32:29.523863+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:30:35.138361+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:30:35.138361+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":465,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.527456+0000","last_change":"2024-02-05T07:30:40.476394+0000","last_active":"2024-02-05T07:32:29.527456+0000","last_peered":"2024-02-05T07:32:29.527456+0000","last_clean":"2024-02-05T07:32:29.527456+0000","last_became_active":"2024-02-05T07:30:40.476058+0000","last_became_peered":"2024-02-05T07:30:40.476058+0000","last_unstale":"2024-02-05T07:32:29.527456+0000","last_undegraded":"2024-02-05T07:32:29.527456+0000","last_fullsized":"2024-02-05T07:32:29.527456+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.314735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.314735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"365'11","reported_seq":366,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.524163+0000","last_change":"2024-02-05T07:30:40.517194+0000","last_active":"2024-02-05T07:32:29.524163+0000","last_peered":"2024-02-05T07:32:29.524163+0000","last_clean":"2024-02-05T07:32:29.524163+0000","last_became_active":"2024-02-05T07:30:40.495708+0000","last_became_peered":"2024-02-05T07:30:40.495708+0000","last_unstale":"2024-02-05T07:32:29.524163+0000","last_undegraded":"2024-02-05T07:32:29.524163+0000","last_fullsized":"2024-02-05T07:32:29.524163+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:26:35.091894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.091894+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":701,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.500432+0000","last_change":"2024-02-05T07:27:03.366653+0000","last_active":"2024-02-05T07:32:29.500432+0000","last_peered":"2024-02-05T07:32:29.500432+0000","last_clean":"2024-02-05T07:32:29.500432+0000","last_became_active":"2024-02-05T07:27:03.364334+0000","last_became_peered":"2024-02-05T07:27:03.364334+0000","last_unstale":"2024-02-05T07:32:29.500432+0000","last_undegraded":"2024-02-05T07:32:29.500432+0000","last_fullsized":"2024-02-05T07:32:29.500432+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:25.323384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:25.323384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":546,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.523676+0000","last_change":"2024-02-05T07:28:41.013449+0000","last_active":"2024-02-05T07:32:29.523676+0000","last_peered":"2024-02-05T07:32:29.523676+0000","last_clean":"2024-02-05T07:32:29.523676+0000","last_became_active":"2024-02-05T07:28:19.302393+0000","last_became_peered":"2024-02-05T07:28:19.302393+0000","last_unstale":"2024-02-05T07:32:29.523676+0000","last_undegraded":"2024-02-05T07:32:29.523676+0000","last_fullsized":"2024-02-05T07:32:29.523676+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:41.013360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:41.013360+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"326'25","reported_seq":342,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.524328+0000","last_change":"2024-02-05T07:30:40.512630+0000","last_active":"2024-02-05T07:32:29.524328+0000","last_peered":"2024-02-05T07:32:29.524328+0000","last_clean":"2024-02-05T07:32:29.524328+0000","last_became_active":"2024-02-05T07:30:40.486394+0000","last_became_peered":"2024-02-05T07:30:40.486394+0000","last_unstale":"2024-02-05T07:32:29.524328+0000","last_undegraded":"2024-02-05T07:32:29.524328+0000","last_fullsized":"2024-02-05T07:32:29.524328+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:30:33.959124+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:30:33.959124+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":86,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499586+0000","last_change":"2024-02-05T07:30:40.505910+0000","last_active":"2024-02-05T07:32:29.499586+0000","last_peered":"2024-02-05T07:32:29.499586+0000","last_clean":"2024-02-05T07:32:29.499586+0000","last_became_active":"2024-02-05T07:30:40.505600+0000","last_became_peered":"2024-02-05T07:30:40.505600+0000","last_unstale":"2024-02-05T07:32:29.499586+0000","last_undegraded":"2024-02-05T07:32:29.499586+0000","last_fullsized":"2024-02-05T07:32:29.499586+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:46.210169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:46.210169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"217'9","reported_seq":500,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511132+0000","last_change":"2024-02-05T07:30:27.993196+0000","last_active":"2024-02-05T07:32:29.511132+0000","last_peered":"2024-02-05T07:32:29.511132+0000","last_clean":"2024-02-05T07:32:29.511132+0000","last_became_active":"2024-02-05T07:25:27.246616+0000","last_became_peered":"2024-02-05T07:25:27.246616+0000","last_unstale":"2024-02-05T07:32:29.511132+0000","last_undegraded":"2024-02-05T07:32:29.511132+0000","last_fullsized":"2024-02-05T07:32:29.511132+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9","last_scrub_stamp":"2024-02-05T07:30:27.993000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:27.993000+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":453,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.523157+0000","last_change":"2024-02-05T07:30:40.508206+0000","last_active":"2024-02-05T07:32:29.523157+0000","last_peered":"2024-02-05T07:32:29.523157+0000","last_clean":"2024-02-05T07:32:29.523157+0000","last_became_active":"2024-02-05T07:30:40.507831+0000","last_became_peered":"2024-02-05T07:30:40.507831+0000","last_unstale":"2024-02-05T07:32:29.523157+0000","last_undegraded":"2024-02-05T07:32:29.523157+0000","last_fullsized":"2024-02-05T07:32:29.523157+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:27:01.069970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:27:01.069970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.16","version":"86'128","reported_seq":350,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.512051+0000","last_change":"2024-02-05T07:31:41.838465+0000","last_active":"2024-02-05T07:32:29.512051+0000","last_peered":"2024-02-05T07:32:29.512051+0000","last_clean":"2024-02-05T07:32:29.512051+0000","last_became_active":"2024-02-05T07:31:41.837877+0000","last_became_peered":"2024-02-05T07:31:41.837877+0000","last_unstale":"2024-02-05T07:32:29.512051+0000","last_undegraded":"2024-02-05T07:32:29.512051+0000","last_fullsized":"2024-02-05T07:32:29.512051+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:49.521450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:49.521450+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":4,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"326'19","reported_seq":761,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499666+0000","last_change":"2024-02-05T07:28:25.780182+0000","last_active":"2024-02-05T07:32:29.499666+0000","last_peered":"2024-02-05T07:32:29.499666+0000","last_clean":"2024-02-05T07:32:29.499666+0000","last_became_active":"2024-02-05T07:28:25.779886+0000","last_became_peered":"2024-02-05T07:28:25.779886+0000","last_unstale":"2024-02-05T07:32:29.499666+0000","last_undegraded":"2024-02-05T07:32:29.499666+0000","last_fullsized":"2024-02-05T07:32:29.499666+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-02-05T07:25:42.192387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:42.192387+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":722,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499704+0000","last_change":"2024-02-05T07:30:40.510773+0000","last_active":"2024-02-05T07:32:29.499704+0000","last_peered":"2024-02-05T07:32:29.499704+0000","last_clean":"2024-02-05T07:32:29.499704+0000","last_became_active":"2024-02-05T07:30:40.510301+0000","last_became_peered":"2024-02-05T07:30:40.510301+0000","last_unstale":"2024-02-05T07:32:29.499704+0000","last_undegraded":"2024-02-05T07:32:29.499704+0000","last_fullsized":"2024-02-05T07:32:29.499704+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:36.798282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:29:36.798282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":722,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.521020+0000","last_change":"2024-02-05T07:30:40.487555+0000","last_active":"2024-02-05T07:32:29.521020+0000","last_peered":"2024-02-05T07:32:29.521020+0000","last_clean":"2024-02-05T07:32:29.521020+0000","last_became_active":"2024-02-05T07:30:40.487141+0000","last_became_peered":"2024-02-05T07:30:40.487141+0000","last_unstale":"2024-02-05T07:32:29.521020+0000","last_undegraded":"2024-02-05T07:32:29.521020+0000","last_fullsized":"2024-02-05T07:32:29.521020+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.555824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.555824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":287,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.524131+0000","last_change":"2024-02-05T07:27:17.708497+0000","last_active":"2024-02-05T07:32:29.524131+0000","last_peered":"2024-02-05T07:32:29.524131+0000","last_clean":"2024-02-05T07:32:29.524131+0000","last_became_active":"2024-02-05T07:27:17.705910+0000","last_became_peered":"2024-02-05T07:27:17.705910+0000","last_unstale":"2024-02-05T07:32:29.524131+0000","last_undegraded":"2024-02-05T07:32:29.524131+0000","last_fullsized":"2024-02-05T07:32:29.524131+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.120512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"86'128","reported_seq":569,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:23.818953+0000","last_change":"2024-02-05T07:32:23.818975+0000","last_active":"2024-02-05T07:32:23.818953+0000","last_peered":"2024-02-05T07:32:23.818953+0000","last_clean":"2024-02-05T07:32:23.818953+0000","last_became_active":"2024-02-05T07:31:41.837612+0000","last_became_peered":"2024-02-05T07:31:41.837612+0000","last_unstale":"2024-02-05T07:32:23.818953+0000","last_undegraded":"2024-02-05T07:32:23.818953+0000","last_fullsized":"2024-02-05T07:32:23.818953+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:23.818901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:23.818901+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:47.869222+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"325'15","reported_seq":336,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.520385+0000","last_change":"2024-02-05T07:29:57.775013+0000","last_active":"2024-02-05T07:32:29.520385+0000","last_peered":"2024-02-05T07:32:29.520385+0000","last_clean":"2024-02-05T07:32:29.520385+0000","last_became_active":"2024-02-05T07:28:26.589705+0000","last_became_peered":"2024-02-05T07:28:26.589705+0000","last_unstale":"2024-02-05T07:32:29.520385+0000","last_undegraded":"2024-02-05T07:32:29.520385+0000","last_fullsized":"2024-02-05T07:32:29.520385+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:29:57.774913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:57.774913+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":510,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.523778+0000","last_change":"2024-02-05T07:30:40.489183+0000","last_active":"2024-02-05T07:32:29.523778+0000","last_peered":"2024-02-05T07:32:29.523778+0000","last_clean":"2024-02-05T07:32:29.523778+0000","last_became_active":"2024-02-05T07:30:40.488789+0000","last_became_peered":"2024-02-05T07:30:40.488789+0000","last_unstale":"2024-02-05T07:32:29.523778+0000","last_undegraded":"2024-02-05T07:32:29.523778+0000","last_fullsized":"2024-02-05T07:32:29.523778+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.220841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.220841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":289,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.527190+0000","last_change":"2024-02-05T07:27:16.765064+0000","last_active":"2024-02-05T07:32:29.527190+0000","last_peered":"2024-02-05T07:32:29.527190+0000","last_clean":"2024-02-05T07:32:29.527190+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:32:29.527190+0000","last_undegraded":"2024-02-05T07:32:29.527190+0000","last_fullsized":"2024-02-05T07:32:29.527190+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:26:31.506641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":463,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511152+0000","last_change":"2024-02-05T07:28:26.588019+0000","last_active":"2024-02-05T07:32:29.511152+0000","last_peered":"2024-02-05T07:32:29.511152+0000","last_clean":"2024-02-05T07:32:29.511152+0000","last_became_active":"2024-02-05T07:28:26.587701+0000","last_became_peered":"2024-02-05T07:28:26.587701+0000","last_unstale":"2024-02-05T07:32:29.511152+0000","last_undegraded":"2024-02-05T07:32:29.511152+0000","last_fullsized":"2024-02-05T07:32:29.511152+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:37.616737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:26:37.616737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":957,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511056+0000","last_change":"2024-02-05T07:30:43.515841+0000","last_active":"2024-02-05T07:32:29.511056+0000","last_peered":"2024-02-05T07:32:29.511056+0000","last_clean":"2024-02-05T07:32:29.511056+0000","last_became_active":"2024-02-05T07:30:43.515488+0000","last_became_peered":"2024-02-05T07:30:43.515488+0000","last_unstale":"2024-02-05T07:32:29.511056+0000","last_undegraded":"2024-02-05T07:32:29.511056+0000","last_fullsized":"2024-02-05T07:32:29.511056+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:30:36.995330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.995330+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"326'19","reported_seq":342,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.524328+0000","last_change":"2024-02-05T07:30:40.906805+0000","last_active":"2024-02-05T07:32:29.524328+0000","last_peered":"2024-02-05T07:32:29.524328+0000","last_clean":"2024-02-05T07:32:29.524328+0000","last_became_active":"2024-02-05T07:30:40.501893+0000","last_became_peered":"2024-02-05T07:30:40.501893+0000","last_unstale":"2024-02-05T07:32:29.524328+0000","last_undegraded":"2024-02-05T07:32:29.524328+0000","last_fullsized":"2024-02-05T07:32:29.524328+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:30:40.906693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.906693+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":111,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.520459+0000","last_change":"2024-02-05T07:30:40.492005+0000","last_active":"2024-02-05T07:32:29.520459+0000","last_peered":"2024-02-05T07:32:29.520459+0000","last_clean":"2024-02-05T07:32:29.520459+0000","last_became_active":"2024-02-05T07:30:40.491686+0000","last_became_peered":"2024-02-05T07:30:40.491686+0000","last_unstale":"2024-02-05T07:32:29.520459+0000","last_undegraded":"2024-02-05T07:32:29.520459+0000","last_fullsized":"2024-02-05T07:32:29.520459+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.097532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.097532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":584,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.498969+0000","last_change":"2024-02-05T07:30:40.497509+0000","last_active":"2024-02-05T07:32:29.498969+0000","last_peered":"2024-02-05T07:32:29.498969+0000","last_clean":"2024-02-05T07:32:29.498969+0000","last_became_active":"2024-02-05T07:30:40.497190+0000","last_became_peered":"2024-02-05T07:30:40.497190+0000","last_unstale":"2024-02-05T07:32:29.498969+0000","last_undegraded":"2024-02-05T07:32:29.498969+0000","last_fullsized":"2024-02-05T07:32:29.498969+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:29:58.191815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":301,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.527273+0000","last_change":"2024-02-05T07:29:40.983327+0000","last_active":"2024-02-05T07:32:29.527273+0000","last_peered":"2024-02-05T07:32:29.527273+0000","last_clean":"2024-02-05T07:32:29.527273+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:32:29.527273+0000","last_undegraded":"2024-02-05T07:32:29.527273+0000","last_fullsized":"2024-02-05T07:32:29.527273+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:40.983227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.983227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"86'128","reported_seq":344,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.512078+0000","last_change":"2024-02-05T07:31:41.844811+0000","last_active":"2024-02-05T07:32:29.512078+0000","last_peered":"2024-02-05T07:32:29.512078+0000","last_clean":"2024-02-05T07:32:29.512078+0000","last_became_active":"2024-02-05T07:31:41.843850+0000","last_became_peered":"2024-02-05T07:31:41.843850+0000","last_unstale":"2024-02-05T07:32:29.512078+0000","last_undegraded":"2024-02-05T07:32:29.512078+0000","last_fullsized":"2024-02-05T07:32:29.512078+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:41.657622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:41.657622+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":4,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"325'8","reported_seq":313,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.530330+0000","last_change":"2024-02-05T07:29:53.958469+0000","last_active":"2024-02-05T07:32:29.530330+0000","last_peered":"2024-02-05T07:32:29.530330+0000","last_clean":"2024-02-05T07:32:29.530330+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:32:29.530330+0000","last_undegraded":"2024-02-05T07:32:29.530330+0000","last_fullsized":"2024-02-05T07:32:29.530330+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:29:53.958384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.958384+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":300,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.531077+0000","last_change":"2024-02-05T07:28:46.056835+0000","last_active":"2024-02-05T07:32:29.531077+0000","last_peered":"2024-02-05T07:32:29.531077+0000","last_clean":"2024-02-05T07:32:29.531077+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:32:29.531077+0000","last_undegraded":"2024-02-05T07:32:29.531077+0000","last_fullsized":"2024-02-05T07:32:29.531077+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:46.056702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:28:46.056702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":551,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.504285+0000","last_change":"2024-02-05T07:30:40.500892+0000","last_active":"2024-02-05T07:32:19.504285+0000","last_peered":"2024-02-05T07:32:19.504285+0000","last_clean":"2024-02-05T07:32:19.504285+0000","last_became_active":"2024-02-05T07:30:40.500403+0000","last_became_peered":"2024-02-05T07:30:40.500403+0000","last_unstale":"2024-02-05T07:32:19.504285+0000","last_undegraded":"2024-02-05T07:32:19.504285+0000","last_fullsized":"2024-02-05T07:32:19.504285+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:58.034570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:29:58.034570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":95,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.510828+0000","last_change":"2024-02-05T07:30:45.926725+0000","last_active":"2024-02-05T07:32:29.510828+0000","last_peered":"2024-02-05T07:32:29.510828+0000","last_clean":"2024-02-05T07:32:29.510828+0000","last_became_active":"2024-02-05T07:30:40.489230+0000","last_became_peered":"2024-02-05T07:30:40.489230+0000","last_unstale":"2024-02-05T07:32:29.510828+0000","last_undegraded":"2024-02-05T07:32:29.510828+0000","last_fullsized":"2024-02-05T07:32:29.510828+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:45.926395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:30:45.926395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1a","version":"86'128","reported_seq":424,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499029+0000","last_change":"2024-02-05T07:32:25.652816+0000","last_active":"2024-02-05T07:32:29.499029+0000","last_peered":"2024-02-05T07:32:29.499029+0000","last_clean":"2024-02-05T07:32:29.499029+0000","last_became_active":"2024-02-05T07:31:41.820206+0000","last_became_peered":"2024-02-05T07:31:41.820206+0000","last_unstale":"2024-02-05T07:32:29.499029+0000","last_undegraded":"2024-02-05T07:32:29.499029+0000","last_fullsized":"2024-02-05T07:32:29.499029+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:25.652696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:25.652696+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:38.772356+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"326'19","reported_seq":537,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.505167+0000","last_change":"2024-02-05T07:29:46.947670+0000","last_active":"2024-02-05T07:32:19.505167+0000","last_peered":"2024-02-05T07:32:19.505167+0000","last_clean":"2024-02-05T07:32:19.505167+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:32:19.505167+0000","last_undegraded":"2024-02-05T07:32:19.505167+0000","last_fullsized":"2024-02-05T07:32:19.505167+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:29:46.947559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:46.947559+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":297,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.530151+0000","last_change":"2024-02-05T07:29:38.000417+0000","last_active":"2024-02-05T07:32:29.530151+0000","last_peered":"2024-02-05T07:32:29.530151+0000","last_clean":"2024-02-05T07:32:29.530151+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:32:29.530151+0000","last_undegraded":"2024-02-05T07:32:29.530151+0000","last_fullsized":"2024-02-05T07:32:29.530151+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:38.000345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:38.000345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":88,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.514335+0000","last_change":"2024-02-05T07:30:40.480962+0000","last_active":"2024-02-05T07:32:29.514335+0000","last_peered":"2024-02-05T07:32:29.514335+0000","last_clean":"2024-02-05T07:32:29.514335+0000","last_became_active":"2024-02-05T07:30:40.479162+0000","last_became_peered":"2024-02-05T07:30:40.479162+0000","last_unstale":"2024-02-05T07:32:29.514335+0000","last_undegraded":"2024-02-05T07:32:29.514335+0000","last_fullsized":"2024-02-05T07:32:29.514335+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:28.153881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:28.153881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":718,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.517097+0000","last_change":"2024-02-05T07:29:55.800941+0000","last_active":"2024-02-05T07:32:29.517097+0000","last_peered":"2024-02-05T07:32:29.517097+0000","last_clean":"2024-02-05T07:32:29.517097+0000","last_became_active":"2024-02-05T07:27:03.360468+0000","last_became_peered":"2024-02-05T07:27:03.360468+0000","last_unstale":"2024-02-05T07:32:29.517097+0000","last_undegraded":"2024-02-05T07:32:29.517097+0000","last_fullsized":"2024-02-05T07:32:29.517097+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:55.800824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.800824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1b","version":"86'128","reported_seq":499,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499472+0000","last_change":"2024-02-05T07:32:26.685749+0000","last_active":"2024-02-05T07:32:29.499472+0000","last_peered":"2024-02-05T07:32:29.499472+0000","last_clean":"2024-02-05T07:32:29.499472+0000","last_became_active":"2024-02-05T07:31:41.823891+0000","last_became_peered":"2024-02-05T07:31:41.823891+0000","last_unstale":"2024-02-05T07:32:29.499472+0000","last_undegraded":"2024-02-05T07:32:29.499472+0000","last_fullsized":"2024-02-05T07:32:29.499472+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:26.685655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:26.685655+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:39.162396+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"326'20","reported_seq":463,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.522808+0000","last_change":"2024-02-05T07:30:40.534624+0000","last_active":"2024-02-05T07:32:29.522808+0000","last_peered":"2024-02-05T07:32:29.522808+0000","last_clean":"2024-02-05T07:32:29.522808+0000","last_became_active":"2024-02-05T07:30:40.495954+0000","last_became_peered":"2024-02-05T07:30:40.495954+0000","last_unstale":"2024-02-05T07:32:29.522808+0000","last_undegraded":"2024-02-05T07:32:29.522808+0000","last_fullsized":"2024-02-05T07:32:29.522808+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:29:40.154854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.154854+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":760,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.512090+0000","last_change":"2024-02-05T07:30:40.526955+0000","last_active":"2024-02-05T07:32:29.512090+0000","last_peered":"2024-02-05T07:32:29.512090+0000","last_clean":"2024-02-05T07:32:29.512090+0000","last_became_active":"2024-02-05T07:30:40.495135+0000","last_became_peered":"2024-02-05T07:30:40.495135+0000","last_unstale":"2024-02-05T07:32:29.512090+0000","last_undegraded":"2024-02-05T07:32:29.512090+0000","last_fullsized":"2024-02-05T07:32:29.512090+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:52.715259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:52.715259+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"365'11","reported_seq":697,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:19.633005+0000","last_change":"2024-02-05T07:30:40.516172+0000","last_active":"2024-02-05T07:32:19.633005+0000","last_peered":"2024-02-05T07:32:19.633005+0000","last_clean":"2024-02-05T07:32:19.633005+0000","last_became_active":"2024-02-05T07:30:40.497371+0000","last_became_peered":"2024-02-05T07:30:40.497371+0000","last_unstale":"2024-02-05T07:32:19.633005+0000","last_undegraded":"2024-02-05T07:32:19.633005+0000","last_fullsized":"2024-02-05T07:32:19.633005+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'10","last_scrub_stamp":"2024-02-05T07:29:55.058040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.058040+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":205,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511156+0000","last_change":"2024-02-05T07:30:41.950393+0000","last_active":"2024-02-05T07:32:29.511156+0000","last_peered":"2024-02-05T07:32:29.511156+0000","last_clean":"2024-02-05T07:32:29.511156+0000","last_became_active":"2024-02-05T07:30:40.486920+0000","last_became_peered":"2024-02-05T07:30:40.486920+0000","last_unstale":"2024-02-05T07:32:29.511156+0000","last_undegraded":"2024-02-05T07:32:29.511156+0000","last_fullsized":"2024-02-05T07:32:29.511156+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:30:41.950253+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:30:41.950253+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"86'128","reported_seq":737,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.498501+0000","last_change":"2024-02-05T07:32:01.562888+0000","last_active":"2024-02-05T07:32:29.498501+0000","last_peered":"2024-02-05T07:32:29.498501+0000","last_clean":"2024-02-05T07:32:29.498501+0000","last_became_active":"2024-02-05T07:32:01.562409+0000","last_became_peered":"2024-02-05T07:32:01.562409+0000","last_unstale":"2024-02-05T07:32:29.498501+0000","last_undegraded":"2024-02-05T07:32:29.498501+0000","last_fullsized":"2024-02-05T07:32:29.498501+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:29:48.249082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.249082+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:32:34.274106+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"326'23","reported_seq":323,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.523558+0000","last_change":"2024-02-05T07:28:47.062765+0000","last_active":"2024-02-05T07:32:29.523558+0000","last_peered":"2024-02-05T07:32:29.523558+0000","last_clean":"2024-02-05T07:32:29.523558+0000","last_became_active":"2024-02-05T07:27:16.747998+0000","last_became_peered":"2024-02-05T07:27:16.747998+0000","last_unstale":"2024-02-05T07:32:29.523558+0000","last_undegraded":"2024-02-05T07:32:29.523558+0000","last_fullsized":"2024-02-05T07:32:29.523558+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'17","last_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.062618+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":398,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.520891+0000","last_change":"2024-02-05T07:30:39.874170+0000","last_active":"2024-02-05T07:32:29.520891+0000","last_peered":"2024-02-05T07:32:29.520891+0000","last_clean":"2024-02-05T07:32:29.520891+0000","last_became_active":"2024-02-05T07:25:30.261037+0000","last_became_peered":"2024-02-05T07:25:30.261037+0000","last_unstale":"2024-02-05T07:32:29.520891+0000","last_undegraded":"2024-02-05T07:32:29.520891+0000","last_fullsized":"2024-02-05T07:32:29.520891+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.874081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":722,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499850+0000","last_change":"2024-02-05T07:30:26.669849+0000","last_active":"2024-02-05T07:32:29.499850+0000","last_peered":"2024-02-05T07:32:29.499850+0000","last_clean":"2024-02-05T07:32:29.499850+0000","last_became_active":"2024-02-05T07:27:16.706777+0000","last_became_peered":"2024-02-05T07:27:16.706777+0000","last_unstale":"2024-02-05T07:32:29.499850+0000","last_undegraded":"2024-02-05T07:32:29.499850+0000","last_fullsized":"2024-02-05T07:32:29.499850+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:26.669576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:26.669576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":300,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.531284+0000","last_change":"2024-02-05T07:28:53.138071+0000","last_active":"2024-02-05T07:32:29.531284+0000","last_peered":"2024-02-05T07:32:29.531284+0000","last_clean":"2024-02-05T07:32:29.531284+0000","last_became_active":"2024-02-05T07:28:26.603067+0000","last_became_peered":"2024-02-05T07:28:26.603067+0000","last_unstale":"2024-02-05T07:32:29.531284+0000","last_undegraded":"2024-02-05T07:32:29.531284+0000","last_fullsized":"2024-02-05T07:32:29.531284+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:53.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:53.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.19","version":"86'128","reported_seq":812,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.523428+0000","last_change":"2024-02-05T07:31:41.868151+0000","last_active":"2024-02-05T07:32:29.523428+0000","last_peered":"2024-02-05T07:32:29.523428+0000","last_clean":"2024-02-05T07:32:29.523428+0000","last_became_active":"2024-02-05T07:31:41.858087+0000","last_became_peered":"2024-02-05T07:31:41.858087+0000","last_unstale":"2024-02-05T07:32:29.523428+0000","last_undegraded":"2024-02-05T07:32:29.523428+0000","last_fullsized":"2024-02-05T07:32:29.523428+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:26:18.993912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.993912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:32:31.684796+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"326'27","reported_seq":798,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511319+0000","last_change":"2024-02-05T07:29:41.354568+0000","last_active":"2024-02-05T07:32:29.511319+0000","last_peered":"2024-02-05T07:32:29.511319+0000","last_clean":"2024-02-05T07:32:29.511319+0000","last_became_active":"2024-02-05T07:27:16.736296+0000","last_became_peered":"2024-02-05T07:27:16.736296+0000","last_unstale":"2024-02-05T07:32:29.511319+0000","last_undegraded":"2024-02-05T07:32:29.511319+0000","last_fullsized":"2024-02-05T07:32:29.511319+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:29:41.354464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:41.354464+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":467,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499566+0000","last_change":"2024-02-05T07:30:40.503947+0000","last_active":"2024-02-05T07:32:29.499566+0000","last_peered":"2024-02-05T07:32:29.499566+0000","last_clean":"2024-02-05T07:32:29.499566+0000","last_became_active":"2024-02-05T07:30:40.503627+0000","last_became_peered":"2024-02-05T07:30:40.503627+0000","last_unstale":"2024-02-05T07:32:29.499566+0000","last_undegraded":"2024-02-05T07:32:29.499566+0000","last_fullsized":"2024-02-05T07:32:29.499566+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:34.663987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:34.663987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":708,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.516922+0000","last_change":"2024-02-05T07:30:40.884273+0000","last_active":"2024-02-05T07:32:29.516922+0000","last_peered":"2024-02-05T07:32:29.516922+0000","last_clean":"2024-02-05T07:32:29.516922+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:32:29.516922+0000","last_undegraded":"2024-02-05T07:32:29.516922+0000","last_fullsized":"2024-02-05T07:32:29.516922+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:40.884197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.884197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":721,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.500422+0000","last_change":"2024-02-05T07:29:59.002788+0000","last_active":"2024-02-05T07:32:29.500422+0000","last_peered":"2024-02-05T07:32:29.500422+0000","last_clean":"2024-02-05T07:32:29.500422+0000","last_became_active":"2024-02-05T07:28:26.590448+0000","last_became_peered":"2024-02-05T07:28:26.590448+0000","last_unstale":"2024-02-05T07:32:29.500422+0000","last_undegraded":"2024-02-05T07:32:29.500422+0000","last_fullsized":"2024-02-05T07:32:29.500422+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.002677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.002677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"86'128","reported_seq":412,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.522589+0000","last_change":"2024-02-05T07:31:41.835122+0000","last_active":"2024-02-05T07:32:29.522589+0000","last_peered":"2024-02-05T07:32:29.522589+0000","last_clean":"2024-02-05T07:32:29.522589+0000","last_became_active":"2024-02-05T07:31:41.834645+0000","last_became_peered":"2024-02-05T07:31:41.834645+0000","last_unstale":"2024-02-05T07:32:29.522589+0000","last_undegraded":"2024-02-05T07:32:29.522589+0000","last_fullsized":"2024-02-05T07:32:29.522589+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:40.616302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:40.616302+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:32:31.684765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":4,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":40253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"325'7","reported_seq":740,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499492+0000","last_change":"2024-02-05T07:27:16.970973+0000","last_active":"2024-02-05T07:32:29.499492+0000","last_peered":"2024-02-05T07:32:29.499492+0000","last_clean":"2024-02-05T07:32:29.499492+0000","last_became_active":"2024-02-05T07:27:16.745897+0000","last_became_peered":"2024-02-05T07:27:16.745897+0000","last_unstale":"2024-02-05T07:32:29.499492+0000","last_undegraded":"2024-02-05T07:32:29.499492+0000","last_fullsized":"2024-02-05T07:32:29.499492+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:26:35.253452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.253452+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":539,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499696+0000","last_change":"2024-02-05T07:30:39.726603+0000","last_active":"2024-02-05T07:32:29.499696+0000","last_peered":"2024-02-05T07:32:29.499696+0000","last_clean":"2024-02-05T07:32:29.499696+0000","last_became_active":"2024-02-05T07:28:26.590322+0000","last_became_peered":"2024-02-05T07:28:26.590322+0000","last_unstale":"2024-02-05T07:32:29.499696+0000","last_undegraded":"2024-02-05T07:32:29.499696+0000","last_fullsized":"2024-02-05T07:32:29.499696+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.726525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.726525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":86,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499540+0000","last_change":"2024-02-05T07:30:40.504663+0000","last_active":"2024-02-05T07:32:29.499540+0000","last_peered":"2024-02-05T07:32:29.499540+0000","last_clean":"2024-02-05T07:32:29.499540+0000","last_became_active":"2024-02-05T07:30:40.504365+0000","last_became_peered":"2024-02-05T07:30:40.504365+0000","last_unstale":"2024-02-05T07:32:29.499540+0000","last_undegraded":"2024-02-05T07:32:29.499540+0000","last_fullsized":"2024-02-05T07:32:29.499540+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:27:00.183281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:27:00.183281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":515,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.515429+0000","last_change":"2024-02-05T07:29:50.742042+0000","last_active":"2024-02-05T07:32:29.515429+0000","last_peered":"2024-02-05T07:32:29.515429+0000","last_clean":"2024-02-05T07:32:29.515429+0000","last_became_active":"2024-02-05T07:27:16.769581+0000","last_became_peered":"2024-02-05T07:27:16.769581+0000","last_unstale":"2024-02-05T07:32:29.515429+0000","last_undegraded":"2024-02-05T07:32:29.515429+0000","last_fullsized":"2024-02-05T07:32:29.515429+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:50.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"325'19","reported_seq":772,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499442+0000","last_change":"2024-02-05T07:30:00.001976+0000","last_active":"2024-02-05T07:32:29.499442+0000","last_peered":"2024-02-05T07:32:29.499442+0000","last_clean":"2024-02-05T07:32:29.499442+0000","last_became_active":"2024-02-05T07:27:16.707036+0000","last_became_peered":"2024-02-05T07:27:16.707036+0000","last_unstale":"2024-02-05T07:32:29.499442+0000","last_undegraded":"2024-02-05T07:32:29.499442+0000","last_fullsized":"2024-02-05T07:32:29.499442+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'19","last_scrub_stamp":"2024-02-05T07:30:00.001863+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:30:00.001863+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":542,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.500248+0000","last_change":"2024-02-05T07:29:53.022701+0000","last_active":"2024-02-05T07:32:29.500248+0000","last_peered":"2024-02-05T07:32:29.500248+0000","last_clean":"2024-02-05T07:32:29.500248+0000","last_became_active":"2024-02-05T07:28:26.616107+0000","last_became_peered":"2024-02-05T07:28:26.616107+0000","last_unstale":"2024-02-05T07:32:29.500248+0000","last_undegraded":"2024-02-05T07:32:29.500248+0000","last_fullsized":"2024-02-05T07:32:29.500248+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.022606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.022606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":88,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499336+0000","last_change":"2024-02-05T07:30:40.491944+0000","last_active":"2024-02-05T07:32:29.499336+0000","last_peered":"2024-02-05T07:32:29.499336+0000","last_clean":"2024-02-05T07:32:29.499336+0000","last_became_active":"2024-02-05T07:30:40.491630+0000","last_became_peered":"2024-02-05T07:30:40.491630+0000","last_unstale":"2024-02-05T07:32:29.499336+0000","last_undegraded":"2024-02-05T07:32:29.499336+0000","last_fullsized":"2024-02-05T07:32:29.499336+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:54.207089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:54.207089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":467,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511309+0000","last_change":"2024-02-05T07:30:40.476139+0000","last_active":"2024-02-05T07:32:29.511309+0000","last_peered":"2024-02-05T07:32:29.511309+0000","last_clean":"2024-02-05T07:32:29.511309+0000","last_became_active":"2024-02-05T07:30:40.475535+0000","last_became_peered":"2024-02-05T07:30:40.475535+0000","last_unstale":"2024-02-05T07:32:29.511309+0000","last_undegraded":"2024-02-05T07:32:29.511309+0000","last_fullsized":"2024-02-05T07:32:29.511309+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:31.484580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:31.484580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"86'128","reported_seq":352,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-05T07:32:22.866468+0000","last_change":"2024-02-05T07:32:22.866491+0000","last_active":"2024-02-05T07:32:22.866468+0000","last_peered":"2024-02-05T07:32:22.866468+0000","last_clean":"2024-02-05T07:32:22.866468+0000","last_became_active":"2024-02-05T07:31:41.837502+0000","last_became_peered":"2024-02-05T07:31:41.837502+0000","last_unstale":"2024-02-05T07:32:22.866468+0000","last_undegraded":"2024-02-05T07:32:22.866468+0000","last_fullsized":"2024-02-05T07:32:22.866468+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:22.866405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:22.866405+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:51.348616+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"326'27","reported_seq":323,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.524480+0000","last_change":"2024-02-05T07:27:17.002461+0000","last_active":"2024-02-05T07:32:29.524480+0000","last_peered":"2024-02-05T07:32:29.524480+0000","last_clean":"2024-02-05T07:32:29.524480+0000","last_became_active":"2024-02-05T07:27:16.751179+0000","last_became_peered":"2024-02-05T07:27:16.751179+0000","last_unstale":"2024-02-05T07:32:29.524480+0000","last_undegraded":"2024-02-05T07:32:29.524480+0000","last_fullsized":"2024-02-05T07:32:29.524480+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:25:38.772812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:38.772812+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":293,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.523422+0000","last_change":"2024-02-05T07:28:26.596258+0000","last_active":"2024-02-05T07:32:29.523422+0000","last_peered":"2024-02-05T07:32:29.523422+0000","last_clean":"2024-02-05T07:32:29.523422+0000","last_became_active":"2024-02-05T07:28:26.595627+0000","last_became_peered":"2024-02-05T07:28:26.595627+0000","last_unstale":"2024-02-05T07:32:29.523422+0000","last_undegraded":"2024-02-05T07:32:29.523422+0000","last_fullsized":"2024-02-05T07:32:29.523422+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:50.859511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:25:50.859511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":716,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.498948+0000","last_change":"2024-02-05T07:29:48.045121+0000","last_active":"2024-02-05T07:32:29.498948+0000","last_peered":"2024-02-05T07:32:29.498948+0000","last_clean":"2024-02-05T07:32:29.498948+0000","last_became_active":"2024-02-05T07:28:25.778935+0000","last_became_peered":"2024-02-05T07:28:25.778935+0000","last_unstale":"2024-02-05T07:32:29.498948+0000","last_undegraded":"2024-02-05T07:32:29.498948+0000","last_fullsized":"2024-02-05T07:32:29.498948+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.045013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.045013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":307,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.527104+0000","last_change":"2024-02-05T07:30:40.496738+0000","last_active":"2024-02-05T07:32:29.527104+0000","last_peered":"2024-02-05T07:32:29.527104+0000","last_clean":"2024-02-05T07:32:29.527104+0000","last_became_active":"2024-02-05T07:30:40.493086+0000","last_became_peered":"2024-02-05T07:30:40.493086+0000","last_unstale":"2024-02-05T07:32:29.527104+0000","last_undegraded":"2024-02-05T07:32:29.527104+0000","last_fullsized":"2024-02-05T07:32:29.527104+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:44.050019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:44.050019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"86'128","reported_seq":352,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.516534+0000","last_change":"2024-02-05T07:31:41.845278+0000","last_active":"2024-02-05T07:32:29.516534+0000","last_peered":"2024-02-05T07:32:29.516534+0000","last_clean":"2024-02-05T07:32:29.516534+0000","last_became_active":"2024-02-05T07:31:41.843342+0000","last_became_peered":"2024-02-05T07:31:41.843342+0000","last_unstale":"2024-02-05T07:32:29.516534+0000","last_undegraded":"2024-02-05T07:32:29.516534+0000","last_fullsized":"2024-02-05T07:32:29.516534+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.145047+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":4,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":40253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"325'15","reported_seq":737,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499133+0000","last_change":"2024-02-05T07:29:33.839515+0000","last_active":"2024-02-05T07:32:29.499133+0000","last_peered":"2024-02-05T07:32:29.499133+0000","last_clean":"2024-02-05T07:32:29.499133+0000","last_became_active":"2024-02-05T07:28:25.777815+0000","last_became_peered":"2024-02-05T07:28:25.777815+0000","last_unstale":"2024-02-05T07:32:29.499133+0000","last_undegraded":"2024-02-05T07:32:29.499133+0000","last_fullsized":"2024-02-05T07:32:29.499133+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:29:33.839402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:33.839402+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":479,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.511729+0000","last_change":"2024-02-05T07:29:50.274850+0000","last_active":"2024-02-05T07:32:29.511729+0000","last_peered":"2024-02-05T07:32:29.511729+0000","last_clean":"2024-02-05T07:32:29.511729+0000","last_became_active":"2024-02-05T07:28:26.588496+0000","last_became_peered":"2024-02-05T07:28:26.588496+0000","last_unstale":"2024-02-05T07:32:29.511729+0000","last_undegraded":"2024-02-05T07:32:29.511729+0000","last_fullsized":"2024-02-05T07:32:29.511729+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.274768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":712,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.515329+0000","last_change":"2024-02-05T07:30:40.496063+0000","last_active":"2024-02-05T07:32:29.515329+0000","last_peered":"2024-02-05T07:32:29.515329+0000","last_clean":"2024-02-05T07:32:29.515329+0000","last_became_active":"2024-02-05T07:30:40.495711+0000","last_became_peered":"2024-02-05T07:30:40.495711+0000","last_unstale":"2024-02-05T07:32:29.515329+0000","last_undegraded":"2024-02-05T07:32:29.515329+0000","last_fullsized":"2024-02-05T07:32:29.515329+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:28.575859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:28.575859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":707,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T07:32:29.499375+0000","last_change":"2024-02-05T07:30:40.490593+0000","last_active":"2024-02-05T07:32:29.499375+0000","last_peered":"2024-02-05T07:32:29.499375+0000","last_clean":"2024-02-05T07:32:29.499375+0000","last_became_active":"2024-02-05T07:30:40.490288+0000","last_became_peered":"2024-02-05T07:30:40.490288+0000","last_unstale":"2024-02-05T07:32:29.499375+0000","last_undegraded":"2024-02-05T07:32:29.499375+0000","last_fullsized":"2024-02-05T07:32:29.499375+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:21.317417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:21.317417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2798321688,"num_objects":42700,"num_object_clones":0,"num_object_copies":128100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":210,"num_objects_misplaced":751,"num_objects_unfound":0,"num_objects_dirty":42700,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116986,"num_write_kb":5109889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26290,"num_bytes_recovered":1722941440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9617420288,"data_stored":9617408072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51645,"ondisk_log_size":51645,"up":48,"acting":46,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":112,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":480200,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":423699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3968,"ondisk_log_size":3968,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1107,"num_read_kb":864,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":6480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":11876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":738,"ondisk_log_size":738,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":257698037929,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":301,"seq":1292785156146,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1779688,"kb_used_data":1486240,"kb_used_omap":5,"kb_used_meta":293434,"kb_avail":91965464,"statfs":{"total":95995035648,"available":94172635136,"internally_reserved":0,"allocated":1521909760,"data_stored":1521080026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":300476933},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":1.554,"15min":2.0019999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.948,"5min":14.218,"15min":14.218},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.631,"15min":1.9990000000000001},"min":{"1min":0.29599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":4.101,"5min":14.175000000000001,"15min":14.175000000000001},"last":0.40000000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:31:31 2024","interfaces":[{"interface":"back","average":{"1min":5.1719999999999997,"5min":2.1739999999999999,"15min":1.8600000000000001},"min":{"1min":0.28100000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":55.923000000000002,"5min":55.923000000000002,"15min":55.923000000000002},"last":0.32700000000000001},{"interface":"front","average":{"1min":5.8630000000000004,"5min":2.851,"15min":2.6890000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":61.103000000000002,"5min":61.103000000000002,"15min":61.103000000000002},"last":2.9100000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":1.986,"15min":2.6509999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.86599999999999999,"5min":34.689999999999998,"15min":34.689999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.81000000000000005,"5min":2.4289999999999998,"15min":3.2090000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.2039999999999997,"5min":34.646999999999998,"15min":34.646999999999998},"last":5.9119999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:31:31 2024","interfaces":[{"interface":"back","average":{"1min":4.9909999999999997,"5min":3.0510000000000002,"15min":2.476},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":55.978999999999999,"5min":55.978999999999999,"15min":55.978999999999999},"last":2.1179999999999999},{"interface":"front","average":{"1min":5.0890000000000004,"5min":3.254,"15min":2.7650000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":55.631,"5min":55.631,"15min":55.631},"last":0.499}]},{"osd":4,"last update":"Mon Feb 5 07:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.64000000000000001,"15min":0.72399999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58499999999999996,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.57599999999999996,"15min":0.63},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.48999999999999999,"5min":1.869,"15min":1.869},"last":0.375}]},{"osd":5,"last update":"Mon Feb 5 07:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.55900000000000005,"15min":0.59499999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.56399999999999995,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.56100000000000005,"15min":0.58399999999999996},"min":{"1min":0.32700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.58799999999999997,"5min":1.526,"15min":1.526},"last":0.35399999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":3.9079999999999999,"15min":5.032},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.71299999999999997,"5min":84.158000000000001,"15min":84.158000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":4.0279999999999996,"15min":5.1900000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.73199999999999998,"5min":85.018000000000001,"15min":85.018000000000001},"last":0.51900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575047,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2629240,"kb_used_data":2006576,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91115912,"statfs":{"total":95995035648,"available":93302693888,"internally_reserved":0,"allocated":2054733824,"data_stored":2053837903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:31:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.8129999999999999,"15min":2.0270000000000001},"min":{"1min":0.216,"5min":0.191,"15min":0.16800000000000001},"max":{"1min":5.4400000000000004,"5min":15.09,"15min":45.393000000000001},"last":1.206},{"interface":"front","average":{"1min":3.3740000000000001,"5min":2.2330000000000001,"15min":2.149},"min":{"1min":0.40000000000000002,"5min":0.23799999999999999,"15min":0.20899999999999999},"max":{"1min":14.313000000000001,"5min":14.313000000000001,"15min":45.031999999999996},"last":0.88800000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:32:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":2.157,"15min":1.7090000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":3.8069999999999999,"5min":15.137,"15min":39.259999999999998},"last":2.4260000000000002},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.9319999999999999,"15min":1.6819999999999999},"min":{"1min":0.308,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":8.8789999999999996,"5min":14.906000000000001,"15min":40.972000000000001},"last":1.982}]},{"osd":3,"last update":"Mon Feb 5 07:32:22 2024","interfaces":[{"interface":"back","average":{"1min":4.6929999999999996,"5min":3.069,"15min":1.9810000000000001},"min":{"1min":0.61199999999999999,"5min":0.22900000000000001,"15min":0.20999999999999999},"max":{"1min":46.777000000000001,"5min":46.777000000000001,"15min":46.777000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":4.9809999999999999,"5min":2.6890000000000001,"15min":1.9910000000000001},"min":{"1min":0.46300000000000002,"5min":0.24299999999999999,"15min":0.24199999999999999},"max":{"1min":50.164999999999999,"5min":50.164999999999999,"15min":50.164999999999999},"last":2.3769999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:31:42 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.48,"15min":1.6679999999999999},"min":{"1min":0.48199999999999998,"5min":0.32600000000000001,"15min":0.312},"max":{"1min":4.9039999999999999,"5min":9.4149999999999991,"15min":61.722000000000001},"last":2.012},{"interface":"front","average":{"1min":0.92100000000000004,"5min":1.4199999999999999,"15min":1.6699999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.3149999999999999,"5min":11.443,"15min":61.786000000000001},"last":1.2470000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:32:10 2024","interfaces":[{"interface":"back","average":{"1min":3.9009999999999998,"5min":1.9139999999999999,"15min":1.9350000000000001},"min":{"1min":0.35699999999999998,"5min":0.309,"15min":0.309},"max":{"1min":46.398000000000003,"5min":46.398000000000003,"15min":106.672},"last":2.0659999999999998},{"interface":"front","average":{"1min":2.4140000000000001,"5min":1.929,"15min":1.948},"min":{"1min":0.48099999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":8.5410000000000004,"5min":20.085999999999999,"15min":107.348},"last":0.85599999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:32:30 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.3400000000000001,"15min":1.6819999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.3809999999999998,"5min":5.5990000000000002,"15min":5.5990000000000002},"last":1.282},{"interface":"front","average":{"1min":1.607,"5min":1.657,"15min":1.9890000000000001},"min":{"1min":0.41899999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":8.923,"5min":8.923,"15min":8.923},"last":2.04}]},{"osd":7,"last update":"Mon Feb 5 07:32:06 2024","interfaces":[{"interface":"back","average":{"1min":5.0529999999999999,"5min":2.5310000000000001,"15min":1.8049999999999999},"min":{"1min":0.52500000000000002,"5min":0.318,"15min":0.318},"max":{"1min":46.351999999999997,"5min":46.351999999999997,"15min":66.703999999999994},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.266,"5min":1.635,"15min":1.5089999999999999},"min":{"1min":0.40100000000000002,"5min":0.34000000000000002,"15min":0.33500000000000002},"max":{"1min":3.5529999999999999,"5min":11.263999999999999,"15min":61.594999999999999},"last":1.3089999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738572,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1622964,"kb_used_data":1046124,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92122188,"statfs":{"total":95995035648,"available":94333120512,"internally_reserved":0,"allocated":1071230976,"data_stored":1070338774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:31:35 2024","interfaces":[{"interface":"back","average":{"1min":2.0270000000000001,"5min":2.1909999999999998,"15min":1.7},"min":{"1min":0.192,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6129999999999995,"5min":32.546999999999997,"15min":45.372},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.7829999999999999,"5min":2.121,"15min":1.766},"min":{"1min":0.26400000000000001,"5min":0.217,"15min":0.186},"max":{"1min":7.25,"5min":29.675999999999998,"15min":45.231999999999999},"last":5.8689999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:32:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8129999999999999,"5min":1.27,"15min":1.6719999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":6.0389999999999997,"5min":6.5140000000000002,"15min":65.903999999999996},"last":5.9180000000000001},{"interface":"front","average":{"1min":1.774,"5min":1.296,"15min":1.6419999999999999},"min":{"1min":0.26300000000000001,"5min":0.224,"15min":0.182},"max":{"1min":4.7450000000000001,"5min":7.1959999999999997,"15min":65.694000000000003},"last":4.7389999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.4169999999999998,"5min":2.3050000000000002,"15min":2.0659999999999998},"min":{"1min":0.28899999999999998,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":22.018000000000001,"5min":29.629000000000001,"15min":65.155000000000001},"last":0.502},{"interface":"front","average":{"1min":4.1539999999999999,"5min":2.6970000000000001,"15min":2.2309999999999999},"min":{"1min":0.30399999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":22.175000000000001,"5min":29.463999999999999,"15min":65.114000000000004},"last":0.48499999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":2.1659999999999999,"15min":1.869},"min":{"1min":0.33700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.2200000000000002,"5min":42.901000000000003,"15min":65.188000000000002},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.048,"5min":2.2789999999999999,"15min":1.9359999999999999},"min":{"1min":0.29699999999999999,"5min":0.27100000000000002,"15min":0.23699999999999999},"max":{"1min":7.0599999999999996,"5min":42.960999999999999,"15min":64.873000000000005},"last":0.31900000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.1140000000000001,"15min":1.6759999999999999},"min":{"1min":0.33300000000000002,"5min":0.30599999999999999,"15min":0.27000000000000002},"max":{"1min":4.6959999999999997,"5min":7.0730000000000004,"15min":95.772999999999996},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.248,"5min":1.069,"15min":1.7190000000000001},"min":{"1min":0.32900000000000001,"5min":0.28199999999999997,"15min":0.27600000000000002},"max":{"1min":7.1929999999999996,"5min":7.1929999999999996,"15min":96.028000000000006},"last":0.39800000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.1259999999999999,"15min":1.3160000000000001},"min":{"1min":0.378,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.258,"5min":4.6929999999999996,"15min":4.6929999999999996},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.4239999999999999,"15min":1.607},"min":{"1min":0.38900000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":7.1319999999999997,"5min":7.1319999999999997,"15min":7.1319999999999997},"last":0.46999999999999997}]},{"osd":7,"last update":"Mon Feb 5 07:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":2.1579999999999999,"15min":1.766},"min":{"1min":0.37,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.2869999999999999,"5min":43.012,"15min":65.263000000000005},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.417,"5min":2.3220000000000001,"15min":1.8109999999999999},"min":{"1min":0.36699999999999999,"5min":0.34000000000000002,"15min":0.309},"max":{"1min":7.2859999999999996,"5min":43.512,"15min":64.906000000000006},"last":0.60299999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411522,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1690384,"kb_used_data":1063560,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92054768,"statfs":{"total":95995035648,"available":94264082432,"internally_reserved":0,"allocated":1089085440,"data_stored":1088229807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:32:04 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":2.802,"15min":1.9450000000000001},"min":{"1min":0.20100000000000001,"5min":0.19600000000000001,"15min":0.182},"max":{"1min":3.2450000000000001,"5min":92.638000000000005,"15min":92.638000000000005},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.575,"5min":2.8519999999999999,"15min":2.0680000000000001},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.20699999999999999},"max":{"1min":4.8620000000000001,"5min":93.481999999999999,"15min":93.481999999999999},"last":0.69399999999999995}]},{"osd":1,"last update":"Mon Feb 5 07:32:04 2024","interfaces":[{"interface":"back","average":{"1min":8.4710000000000001,"5min":4.4880000000000004,"15min":2.1989999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.19400000000000001},"max":{"1min":63.823,"5min":93.504000000000005,"15min":93.504000000000005},"last":0.73699999999999999},{"interface":"front","average":{"1min":9.0909999999999993,"5min":4.8799999999999999,"15min":2.3679999999999999},"min":{"1min":0.35399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":64.771000000000001,"5min":93.004000000000005,"15min":93.004000000000005},"last":1.3320000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:32:27 2024","interfaces":[{"interface":"back","average":{"1min":4.859,"5min":3.431,"15min":1.996},"min":{"1min":0.26100000000000001,"5min":0.25600000000000001,"15min":0.214},"max":{"1min":63.784999999999997,"5min":63.784999999999997,"15min":63.784999999999997},"last":1.1910000000000001},{"interface":"front","average":{"1min":5.2830000000000004,"5min":3.391,"15min":1.968},"min":{"1min":0.38900000000000001,"5min":0.22600000000000001,"15min":0.193},"max":{"1min":63.521000000000001,"5min":63.521000000000001,"15min":63.521000000000001},"last":0.78700000000000003}]},{"osd":4,"last update":"Mon Feb 5 07:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":3.637,"15min":1.96},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.24099999999999999},"max":{"1min":3.7589999999999999,"5min":168.11699999999999,"15min":168.11699999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":3.5129999999999999,"15min":1.9450000000000001},"min":{"1min":0.36199999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.1309999999999998,"5min":168.589,"15min":168.589},"last":0.85899999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:32:04 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.999,"15min":1.1879999999999999},"min":{"1min":0.379,"5min":0.315,"15min":0.315},"max":{"1min":3.2090000000000001,"5min":5.6929999999999996,"15min":39.923000000000002},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.93400000000000005,"15min":1.1279999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.26400000000000001},"max":{"1min":4.0709999999999997,"5min":4.0709999999999997,"15min":38.920999999999999},"last":1.1419999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.0649999999999999,"15min":1.2050000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":4.109,"5min":4.109,"15min":4.109},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.91100000000000003,"5min":1.2030000000000001,"15min":1.409},"min":{"1min":0.44500000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.0960000000000001,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.90200000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:32:04 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.7709999999999999,"15min":1.331},"min":{"1min":0.378,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":4.21,"5min":37.573999999999998,"15min":37.573999999999998},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.874,"5min":1.5409999999999999,"15min":1.2410000000000001},"min":{"1min":0.45000000000000001,"5min":0.34699999999999998,"15min":0.33800000000000002},"max":{"1min":4.1399999999999997,"5min":37.432000000000002,"15min":37.432000000000002},"last":1.228}]}]},{"osd":3,"up_from":26,"seq":111669149885,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2558132,"kb_used_data":1913256,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":91187020,"statfs":{"total":95995035648,"available":93375508480,"internally_reserved":0,"allocated":1959174144,"data_stored":1958285831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":660273607},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:31:35 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.7090000000000001,"15min":2.1110000000000002},"min":{"1min":0.37,"5min":0.192,"15min":0.192},"max":{"1min":6.1239999999999997,"5min":32.078000000000003,"15min":74.114999999999995},"last":1.97},{"interface":"front","average":{"1min":6.867,"5min":2.9820000000000002,"15min":2.2389999999999999},"min":{"1min":0.23200000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":99.989999999999995,"5min":99.989999999999995,"15min":99.989999999999995},"last":2.278}]},{"osd":1,"last update":"Mon Feb 5 07:31:35 2024","interfaces":[{"interface":"back","average":{"1min":8.1449999999999996,"5min":3.988,"15min":2.4460000000000002},"min":{"1min":0.32600000000000001,"5min":0.219,"15min":0.219},"max":{"1min":102.20399999999999,"5min":102.20399999999999,"15min":102.20399999999999},"last":4.5940000000000003},{"interface":"front","average":{"1min":7.8959999999999999,"5min":3.7730000000000001,"15min":2.698},"min":{"1min":0.63200000000000001,"5min":0.255,"15min":0.185},"max":{"1min":102.286,"5min":102.286,"15min":102.286},"last":4.423}]},{"osd":2,"last update":"Mon Feb 5 07:31:35 2024","interfaces":[{"interface":"back","average":{"1min":6.8730000000000002,"5min":2.6120000000000001,"15min":2.4039999999999999},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":99.847999999999999,"5min":99.847999999999999,"15min":99.847999999999999},"last":1.9299999999999999},{"interface":"front","average":{"1min":6.6989999999999998,"5min":2.298,"15min":1.9299999999999999},"min":{"1min":0.309,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":99.950999999999993,"5min":99.950999999999993,"15min":99.950999999999993},"last":0.28000000000000003}]},{"osd":4,"last update":"Mon Feb 5 07:31:46 2024","interfaces":[{"interface":"back","average":{"1min":6.6900000000000004,"5min":2.6259999999999999,"15min":1.5029999999999999},"min":{"1min":0.38900000000000001,"5min":0.32800000000000001,"15min":0.29399999999999998},"max":{"1min":100.024,"5min":100.024,"15min":100.024},"last":0.432},{"interface":"front","average":{"1min":7.6849999999999996,"5min":2.798,"15min":1.5589999999999999},"min":{"1min":0.44,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":99.859999999999999,"5min":99.859999999999999,"15min":99.859999999999999},"last":4.3860000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.7669999999999999,"5min":2.222,"15min":1.8320000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.28499999999999998},"max":{"1min":5.0620000000000003,"5min":99.911000000000001,"15min":99.911000000000001},"last":2.044},{"interface":"front","average":{"1min":1.335,"5min":2.3410000000000002,"15min":1.5649999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.27400000000000002},"max":{"1min":4.8410000000000002,"5min":100.053,"15min":100.053},"last":0.495}]},{"osd":6,"last update":"Mon Feb 5 07:31:35 2024","interfaces":[{"interface":"back","average":{"1min":6.6680000000000001,"5min":2.036,"15min":1.298},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":99.807000000000002,"5min":99.807000000000002,"15min":99.807000000000002},"last":2.0150000000000001},{"interface":"front","average":{"1min":7.1040000000000001,"5min":2.137,"15min":1.3799999999999999},"min":{"1min":0.438,"5min":0.37,"15min":0.37},"max":{"1min":101.489,"5min":101.489,"15min":101.489},"last":0.46400000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.9279999999999999,"5min":4.4859999999999998,"15min":2.2120000000000002},"min":{"1min":0.34499999999999997,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":6.7649999999999997,"5min":99.893000000000001,"15min":99.893000000000001},"last":4.3410000000000002},{"interface":"front","average":{"1min":1.9339999999999999,"5min":4.2859999999999996,"15min":2.4860000000000002},"min":{"1min":0.40699999999999997,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":6.0609999999999999,"5min":99.879000000000005,"15min":99.879000000000005},"last":2.0720000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953656,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1383516,"kb_used_data":887244,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92361636,"statfs":{"total":95995035648,"available":94578315264,"internally_reserved":0,"allocated":908537856,"data_stored":907676823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":3.9369999999999998,"15min":2.089},"min":{"1min":0.22800000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.7610000000000001,"5min":101.212,"15min":101.212},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":3.8700000000000001,"15min":2.173},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23300000000000001},"max":{"1min":2.0699999999999998,"5min":102.07599999999999,"15min":102.07599999999999},"last":4.0659999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:31:47 2024","interfaces":[{"interface":"back","average":{"1min":4.758,"5min":4.9199999999999999,"15min":2.1309999999999998},"min":{"1min":0.29799999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":45.655999999999999,"5min":69.102000000000004,"15min":69.102000000000004},"last":2.4510000000000001},{"interface":"front","average":{"1min":5.3570000000000002,"5min":5.4610000000000003,"15min":2.3660000000000001},"min":{"1min":0.36199999999999999,"5min":0.24399999999999999,"15min":0.23799999999999999},"max":{"1min":46.531999999999996,"5min":68.108999999999995,"15min":68.108999999999995},"last":0.41199999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.6220000000000001,"15min":1.3380000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":2.9910000000000001,"5min":26.045000000000002,"15min":67.144999999999996},"last":2.2639999999999998},{"interface":"front","average":{"1min":1.22,"5min":1.712,"15min":1.359},"min":{"1min":0.32400000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.8630000000000004,"5min":25.960999999999999,"15min":66.959999999999994},"last":0.92100000000000004}]},{"osd":3,"last update":"Mon Feb 5 07:31:47 2024","interfaces":[{"interface":"back","average":{"1min":7.4690000000000003,"5min":3.5219999999999998,"15min":1.6970000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":46.549999999999997,"5min":66.019999999999996,"15min":66.019999999999996},"last":0.36299999999999999},{"interface":"front","average":{"1min":7.3529999999999998,"5min":3.9260000000000002,"15min":1.869},"min":{"1min":0.33300000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":46.003999999999998,"5min":101.001,"15min":101.001},"last":0.34499999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:32:17 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.64800000000000002,"15min":0.58299999999999996},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.52500000000000002,"5min":10.917999999999999,"15min":10.917999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.70799999999999996,"15min":0.60699999999999998},"min":{"1min":0.27700000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.51600000000000001,"5min":15.105,"15min":15.105},"last":0.436}]},{"osd":6,"last update":"Mon Feb 5 07:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":1.02,"15min":1.252},"min":{"1min":0.28299999999999997,"5min":0.245,"15min":0.245},"max":{"1min":1.2470000000000001,"5min":14.606,"15min":14.606},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.45000000000000001,"5min":1.214,"15min":1.5329999999999999},"min":{"1min":0.29999999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63200000000000001,"5min":13.114000000000001,"15min":13.114000000000001},"last":0.45000000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.88300000000000001,"15min":0.71999999999999997},"min":{"1min":0.30599999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.64600000000000002,"5min":15.073,"15min":15.073},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.83299999999999996,"15min":0.68799999999999994},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.248},"max":{"1min":0.69599999999999995,"5min":14.581,"15min":14.581},"last":0.379}]}]},{"osd":5,"up_from":0,"seq":176093659315,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1060372480,"data_stored":1060372480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2033586176,"data_stored":2033582104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1068498944,"data_stored":1068498944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1910374400,"data_stored":1910374400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":893063168,"data_stored":893059096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1492324352,"data_stored":1492320280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1159200768,"data_stored":1159200768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:32:34.565 INFO:tasks.thrashosds.thrasher:Setting 6.0 to [1, 0, 6] 2024-02-05T07:32:34.565 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.0', '1', '0', '6'] 2024-02-05T07:32:34.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pg-upmap 6.0 1 0 6 2024-02-05T07:32:34.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:34.641 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:34.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:34.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:34.742 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:34.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:34.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:34.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:34.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:34.919 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:34.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:34 smithi007 ceph-mon[29098]: 6.3 scrub starts 2024-02-05T07:32:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:34 smithi007 ceph-mon[29098]: 6.3 scrub ok 2024-02-05T07:32:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:34 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/54765640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:32:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:34 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 210/128691 objects degraded (0.163%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:34.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:34 smithi007 ceph-mon[37158]: 6.3 scrub starts 2024-02-05T07:32:34.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:34 smithi007 ceph-mon[37158]: 6.3 scrub ok 2024-02-05T07:32:34.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:34 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/54765640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:32:34.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:34 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 210/128691 objects degraded (0.163%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:35.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:34 smithi090 ceph-mon[33213]: 6.3 scrub starts 2024-02-05T07:32:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:34 smithi090 ceph-mon[33213]: 6.3 scrub ok 2024-02-05T07:32:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:34 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/54765640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:32:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:34 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 210/128691 objects degraded (0.163%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:35.046 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:35.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:35.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:35.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:35.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:35.192 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:35.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:35.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:35.320 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:35.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:35.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:35.421 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:35.461 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:35.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:35.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:35.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:35.588 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:35.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:35.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:35.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:35.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:35.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:35.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:35.842 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:35.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:35.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:35.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:35 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:32:35.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:35 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:32:35.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:35 smithi007 ceph-mon[29098]: pgmap v713: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 710 op/s; 210/127692 objects degraded (0.164%); 751/127692 objects misplaced (0.588%); 3.6 MiB/s, 57 objects/s recovering 2024-02-05T07:32:35.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:35 smithi007 ceph-mon[29098]: osdmap e403: 8 total, 8 up, 6 in 2024-02-05T07:32:35.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:35 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:32:35.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:35 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:32:35.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:35 smithi007 ceph-mon[37158]: pgmap v713: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 710 op/s; 210/127692 objects degraded (0.164%); 751/127692 objects misplaced (0.588%); 3.6 MiB/s, 57 objects/s recovering 2024-02-05T07:32:35.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:35 smithi007 ceph-mon[37158]: osdmap e403: 8 total, 8 up, 6 in 2024-02-05T07:32:35.943 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:35.984 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:35.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:35 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:32:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:35 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:32:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:35 smithi090 ceph-mon[33213]: pgmap v713: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 710 op/s; 210/127692 objects degraded (0.164%); 751/127692 objects misplaced (0.588%); 3.6 MiB/s, 57 objects/s recovering 2024-02-05T07:32:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:35 smithi090 ceph-mon[33213]: osdmap e403: 8 total, 8 up, 6 in 2024-02-05T07:32:36.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:36.111 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:36.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:36.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:36.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:36.256 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:36.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:36.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:36.384 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:36.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:36.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:36.524 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:36.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:36.626 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:36.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:36.666 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:36.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:36.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:36.793 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:36.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:36.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:36.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:36.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:36.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:37.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:37.062 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:37.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:37.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:37.187 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:37.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:37.289 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:37.315 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:37.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:37.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:37.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:37.442 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:37.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:37.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:37.544 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:37.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:37.669 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:37.708 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:37.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:37.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:37.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:37.849 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:37.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:37.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:37.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:37.992 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:37.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:37.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:38.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:38.119 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:38.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:38.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:38.244 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:38.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:38.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:38.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:38.362 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:38 smithi007 ceph-mon[29098]: 5.1 scrub starts 2024-02-05T07:32:38.362 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:38 smithi007 ceph-mon[37158]: 5.1 scrub starts 2024-02-05T07:32:38.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:38 smithi007 ceph-mon[37158]: 5.1 scrub ok 2024-02-05T07:32:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:38.375 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:38.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:38.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:38.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:38.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:38.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:38.629 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:38.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:38.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:38 smithi007 ceph-mon[29098]: 5.1 scrub ok 2024-02-05T07:32:38.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:38 smithi007 ceph-mon[29098]: pgmap v715: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.28k op/s; 341/113127 objects degraded (0.301%); 118/113127 objects misplaced (0.104%); 7.5 MiB/s, 119 objects/s recovering 2024-02-05T07:32:38.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:38 smithi007 ceph-mon[37158]: pgmap v715: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.28k op/s; 341/113127 objects degraded (0.301%); 118/113127 objects misplaced (0.104%); 7.5 MiB/s, 119 objects/s recovering 2024-02-05T07:32:38.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:38.769 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:38.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:38.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:38.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:38 smithi090 ceph-mon[33213]: 5.1 scrub starts 2024-02-05T07:32:38.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:38 smithi090 ceph-mon[33213]: 5.1 scrub ok 2024-02-05T07:32:38.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:38 smithi090 ceph-mon[33213]: pgmap v715: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.28k op/s; 341/113127 objects degraded (0.301%); 118/113127 objects misplaced (0.104%); 7.5 MiB/s, 119 objects/s recovering 2024-02-05T07:32:38.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:38.911 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:38.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:39.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:39.052 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:39.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:39.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:39.191 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:39.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:39.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:39.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:39.318 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:39.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:39.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:39.444 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:39.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:39.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:39.569 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:39.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:39.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:39.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:39 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:32:39.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:39 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:32:39.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:39 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:32:39.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:39 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:32:39.695 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:39.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:39.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:39.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:39 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:32:39.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:39 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:32:39.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:39.820 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:39.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:39.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:39.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:39.947 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:39.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:40.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:40.075 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:40.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:40.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:40.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:40.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:40.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:40.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:40.336 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:40.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:40.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:40.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:40.464 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:40.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:40.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:40.590 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:40.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:40.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:40.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:40 smithi007 ceph-mon[29098]: 6.5 scrub starts 2024-02-05T07:32:40.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:40 smithi007 ceph-mon[29098]: 6.5 scrub ok 2024-02-05T07:32:40.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:40 smithi007 ceph-mon[29098]: pgmap v716: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 341/112107 objects degraded (0.304%); 118/112107 objects misplaced (0.105%); 6.2 MiB/s, 98 objects/s recovering 2024-02-05T07:32:40.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:40 smithi007 ceph-mon[29098]: osdmap e404: 8 total, 8 up, 6 in 2024-02-05T07:32:40.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:40 smithi007 ceph-mon[29098]: Health check update: Degraded data redundancy: 341/112107 objects degraded (0.304%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:40.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:40 smithi007 ceph-mon[29098]: 5.e scrub starts 2024-02-05T07:32:40.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:40 smithi007 ceph-mon[37158]: 6.5 scrub starts 2024-02-05T07:32:40.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:40 smithi007 ceph-mon[37158]: 6.5 scrub ok 2024-02-05T07:32:40.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:40 smithi007 ceph-mon[37158]: pgmap v716: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 341/112107 objects degraded (0.304%); 118/112107 objects misplaced (0.105%); 6.2 MiB/s, 98 objects/s recovering 2024-02-05T07:32:40.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:40 smithi007 ceph-mon[37158]: osdmap e404: 8 total, 8 up, 6 in 2024-02-05T07:32:40.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:40 smithi007 ceph-mon[37158]: Health check update: Degraded data redundancy: 341/112107 objects degraded (0.304%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:40.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:40 smithi007 ceph-mon[37158]: 5.e scrub starts 2024-02-05T07:32:40.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:40.716 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:40.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:40.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:40.755 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:40 smithi090 ceph-mon[33213]: 6.5 scrub starts 2024-02-05T07:32:40.755 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:40 smithi090 ceph-mon[33213]: 6.5 scrub ok 2024-02-05T07:32:40.755 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:40 smithi090 ceph-mon[33213]: pgmap v716: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 341/112107 objects degraded (0.304%); 118/112107 objects misplaced (0.105%); 6.2 MiB/s, 98 objects/s recovering 2024-02-05T07:32:40.755 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:40 smithi090 ceph-mon[33213]: osdmap e404: 8 total, 8 up, 6 in 2024-02-05T07:32:40.755 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:40 smithi090 ceph-mon[33213]: Health check update: Degraded data redundancy: 341/112107 objects degraded (0.304%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-05T07:32:40.755 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:40 smithi090 ceph-mon[33213]: 5.e scrub starts 2024-02-05T07:32:40.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:40.842 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:40.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:40.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:40.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:40.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:40.969 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:40.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:41.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:32:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:32:41.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:41.094 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:41.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:41.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:41.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T07:32:41.195 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:41.218 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:41.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:41.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:41.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:41.345 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:41.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:41.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:41.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:41 smithi007 ceph-mon[29098]: 5.e scrub ok 2024-02-05T07:32:41.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:41 smithi007 ceph-mon[29098]: 3.d deep-scrub starts 2024-02-05T07:32:41.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:41 smithi007 ceph-mon[29098]: 3.d deep-scrub ok 2024-02-05T07:32:41.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:41 smithi007 ceph-mon[29098]: 1.a scrub starts 2024-02-05T07:32:41.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:41 smithi007 ceph-mon[29098]: 1.a scrub ok 2024-02-05T07:32:41.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:41 smithi007 ceph-mon[29098]: osdmap e405: 8 total, 8 up, 6 in 2024-02-05T07:32:41.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:41 smithi007 ceph-mon[37158]: 5.e scrub ok 2024-02-05T07:32:41.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:41 smithi007 ceph-mon[37158]: 3.d deep-scrub starts 2024-02-05T07:32:41.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:41 smithi007 ceph-mon[37158]: 3.d deep-scrub ok 2024-02-05T07:32:41.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:41 smithi007 ceph-mon[37158]: 1.a scrub starts 2024-02-05T07:32:41.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:41 smithi007 ceph-mon[37158]: 1.a scrub ok 2024-02-05T07:32:41.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:41 smithi007 ceph-mon[37158]: osdmap e405: 8 total, 8 up, 6 in 2024-02-05T07:32:41.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:41.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:41.490 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:41.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:41.590 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:41.616 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:41.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:41.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:41.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:41.761 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:41.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:41.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:41 smithi090 ceph-mon[33213]: 5.e scrub ok 2024-02-05T07:32:41.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:41 smithi090 ceph-mon[33213]: 3.d deep-scrub starts 2024-02-05T07:32:41.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:41 smithi090 ceph-mon[33213]: 3.d deep-scrub ok 2024-02-05T07:32:41.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:41 smithi090 ceph-mon[33213]: 1.a scrub starts 2024-02-05T07:32:41.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:41 smithi090 ceph-mon[33213]: 1.a scrub ok 2024-02-05T07:32:41.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:41 smithi090 ceph-mon[33213]: osdmap e405: 8 total, 8 up, 6 in 2024-02-05T07:32:41.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:41.874 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:32:41.920 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:41.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:41.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:42.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:42.070 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:42.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:42.162 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:32:42.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:42.205 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:42.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:42.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:42.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:42.332 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:42.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:42.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:42.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:42.458 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:42.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:42.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:42.585 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:42.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:42.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:42.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[29098]: 4.1b scrub starts 2024-02-05T07:32:42.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[29098]: 4.1b scrub ok 2024-02-05T07:32:42.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:42.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[29098]: 1.8 scrub starts 2024-02-05T07:32:42.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[29098]: 1.8 scrub ok 2024-02-05T07:32:42.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[29098]: pgmap v719: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.2 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 952 op/s; 341/110553 objects degraded (0.308%); 118/110553 objects misplaced (0.107%); 5.2 MiB/s, 83 objects/s recovering 2024-02-05T07:32:42.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[29098]: 1.b scrub starts 2024-02-05T07:32:42.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[29098]: 1.b scrub ok 2024-02-05T07:32:42.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[29098]: osdmap e406: 8 total, 8 up, 6 in 2024-02-05T07:32:42.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[29098]: 1.c scrub starts 2024-02-05T07:32:42.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[29098]: 1.c scrub ok 2024-02-05T07:32:42.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[37158]: 4.1b scrub starts 2024-02-05T07:32:42.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[37158]: 4.1b scrub ok 2024-02-05T07:32:42.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:42.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[37158]: 1.8 scrub starts 2024-02-05T07:32:42.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[37158]: 1.8 scrub ok 2024-02-05T07:32:42.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[37158]: pgmap v719: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.2 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 952 op/s; 341/110553 objects degraded (0.308%); 118/110553 objects misplaced (0.107%); 5.2 MiB/s, 83 objects/s recovering 2024-02-05T07:32:42.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[37158]: 1.b scrub starts 2024-02-05T07:32:42.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[37158]: 1.b scrub ok 2024-02-05T07:32:42.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[37158]: osdmap e406: 8 total, 8 up, 6 in 2024-02-05T07:32:42.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[37158]: 1.c scrub starts 2024-02-05T07:32:42.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:42 smithi007 ceph-mon[37158]: 1.c scrub ok 2024-02-05T07:32:42.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:42.724 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:42.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:42.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:42 smithi090 ceph-mon[33213]: 4.1b scrub starts 2024-02-05T07:32:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:42 smithi090 ceph-mon[33213]: 4.1b scrub ok 2024-02-05T07:32:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:42 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:42 smithi090 ceph-mon[33213]: 1.8 scrub starts 2024-02-05T07:32:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:42 smithi090 ceph-mon[33213]: 1.8 scrub ok 2024-02-05T07:32:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:42 smithi090 ceph-mon[33213]: pgmap v719: 178 pgs: 2 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.2 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 952 op/s; 341/110553 objects degraded (0.308%); 118/110553 objects misplaced (0.107%); 5.2 MiB/s, 83 objects/s recovering 2024-02-05T07:32:42.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:42 smithi090 ceph-mon[33213]: 1.b scrub starts 2024-02-05T07:32:42.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:42 smithi090 ceph-mon[33213]: 1.b scrub ok 2024-02-05T07:32:42.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:42 smithi090 ceph-mon[33213]: osdmap e406: 8 total, 8 up, 6 in 2024-02-05T07:32:42.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:42 smithi090 ceph-mon[33213]: 1.c scrub starts 2024-02-05T07:32:42.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:42 smithi090 ceph-mon[33213]: 1.c scrub ok 2024-02-05T07:32:42.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:42.852 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:42.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:42.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:42.981 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:42.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:42.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:43.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:43.109 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:43.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:43.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:43.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:43.249 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:43.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:43.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:43.388 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:43.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:43.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:43.490 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:43.532 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:43.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:43.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:43.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:32:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:32:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:32:43.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:43 smithi007 ceph-mon[29098]: 1.18 scrub starts 2024-02-05T07:32:43.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:43 smithi007 ceph-mon[29098]: 1.18 scrub ok 2024-02-05T07:32:43.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:43 smithi007 ceph-mon[29098]: 1.16 scrub starts 2024-02-05T07:32:43.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:43 smithi007 ceph-mon[29098]: 1.16 scrub ok 2024-02-05T07:32:43.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:43 smithi007 ceph-mon[29098]: 1.9 scrub starts 2024-02-05T07:32:43.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:43 smithi007 ceph-mon[29098]: 1.9 scrub ok 2024-02-05T07:32:43.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:43 smithi007 ceph-mon[37158]: 1.18 scrub starts 2024-02-05T07:32:43.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:43 smithi007 ceph-mon[37158]: 1.18 scrub ok 2024-02-05T07:32:43.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:43 smithi007 ceph-mon[37158]: 1.16 scrub starts 2024-02-05T07:32:43.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:43 smithi007 ceph-mon[37158]: 1.16 scrub ok 2024-02-05T07:32:43.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:43 smithi007 ceph-mon[37158]: 1.9 scrub starts 2024-02-05T07:32:43.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:43 smithi007 ceph-mon[37158]: 1.9 scrub ok 2024-02-05T07:32:43.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:43.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:43.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:43.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:43 smithi090 ceph-mon[33213]: 1.18 scrub starts 2024-02-05T07:32:43.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:43 smithi090 ceph-mon[33213]: 1.18 scrub ok 2024-02-05T07:32:43.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:43 smithi090 ceph-mon[33213]: 1.16 scrub starts 2024-02-05T07:32:43.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:43 smithi090 ceph-mon[33213]: 1.16 scrub ok 2024-02-05T07:32:43.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:43 smithi090 ceph-mon[33213]: 1.9 scrub starts 2024-02-05T07:32:43.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:43 smithi090 ceph-mon[33213]: 1.9 scrub ok 2024-02-05T07:32:43.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:43.808 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:43.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:43.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:43.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:43.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:32:43.946 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:43.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:44.049 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:44.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:44.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:44.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:44.201 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:44.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:44.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:44.347 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:44.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:44.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:44.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:44.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:44.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:44.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:44.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:44.602 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:44.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:44.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:44.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:44 smithi007 ceph-mon[29098]: pgmap v721: 178 pgs: 178 active+clean; 2.1 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 629 op/s; 970 KiB/s, 76 objects/s recovering 2024-02-05T07:32:44.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:44 smithi007 ceph-mon[29098]: 1.15 scrub starts 2024-02-05T07:32:44.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:44 smithi007 ceph-mon[29098]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 341/110553 objects degraded (0.308%), 2 pgs degraded, 2 pgs undersized) 2024-02-05T07:32:44.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:44 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:32:44.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:44 smithi007 ceph-mon[29098]: 1.15 scrub ok 2024-02-05T07:32:44.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:44 smithi007 ceph-mon[29098]: 1.1e scrub starts 2024-02-05T07:32:44.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:44 smithi007 ceph-mon[29098]: 1.1e scrub ok 2024-02-05T07:32:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:44 smithi007 ceph-mon[37158]: pgmap v721: 178 pgs: 178 active+clean; 2.1 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 629 op/s; 970 KiB/s, 76 objects/s recovering 2024-02-05T07:32:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:44 smithi007 ceph-mon[37158]: 1.15 scrub starts 2024-02-05T07:32:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:44 smithi007 ceph-mon[37158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 341/110553 objects degraded (0.308%), 2 pgs degraded, 2 pgs undersized) 2024-02-05T07:32:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:44 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:32:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:44 smithi007 ceph-mon[37158]: 1.15 scrub ok 2024-02-05T07:32:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:44 smithi007 ceph-mon[37158]: 1.1e scrub starts 2024-02-05T07:32:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:44 smithi007 ceph-mon[37158]: 1.1e scrub ok 2024-02-05T07:32:44.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:44.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:44.729 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:44.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:44.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:44 smithi090 ceph-mon[33213]: pgmap v721: 178 pgs: 178 active+clean; 2.1 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 629 op/s; 970 KiB/s, 76 objects/s recovering 2024-02-05T07:32:44.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:44 smithi090 ceph-mon[33213]: 1.15 scrub starts 2024-02-05T07:32:44.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:44 smithi090 ceph-mon[33213]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 341/110553 objects degraded (0.308%), 2 pgs degraded, 2 pgs undersized) 2024-02-05T07:32:44.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:44 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:32:44.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:44 smithi090 ceph-mon[33213]: 1.15 scrub ok 2024-02-05T07:32:44.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:44 smithi090 ceph-mon[33213]: 1.1e scrub starts 2024-02-05T07:32:44.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:44 smithi090 ceph-mon[33213]: 1.1e scrub ok 2024-02-05T07:32:44.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:44.855 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:44.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:44.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:44.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:44.996 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:44.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:44.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:45.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:45.123 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:45.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:45.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:45.225 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:45.250 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:45.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:45.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:45.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:45.375 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:45.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:45.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:45 smithi007 ceph-mon[37158]: 6.1a scrub starts 2024-02-05T07:32:45.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:45 smithi007 ceph-mon[37158]: 6.1a scrub ok 2024-02-05T07:32:45.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:45 smithi007 ceph-mon[37158]: 4.1f scrub starts 2024-02-05T07:32:45.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:45 smithi007 ceph-mon[37158]: 4.1f scrub ok 2024-02-05T07:32:45.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:45 smithi007 ceph-mon[37158]: 1.1d scrub starts 2024-02-05T07:32:45.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:45 smithi007 ceph-mon[29098]: 6.1a scrub starts 2024-02-05T07:32:45.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:45 smithi007 ceph-mon[29098]: 6.1a scrub ok 2024-02-05T07:32:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:45 smithi007 ceph-mon[29098]: 4.1f scrub starts 2024-02-05T07:32:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:45 smithi007 ceph-mon[29098]: 4.1f scrub ok 2024-02-05T07:32:45.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:45 smithi007 ceph-mon[29098]: 1.1d scrub starts 2024-02-05T07:32:45.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:45.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:45.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:45.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:45.658 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:45.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:45.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:45.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:45.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:45 smithi090 ceph-mon[33213]: 6.1a scrub starts 2024-02-05T07:32:45.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:45 smithi090 ceph-mon[33213]: 6.1a scrub ok 2024-02-05T07:32:45.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:45 smithi090 ceph-mon[33213]: 4.1f scrub starts 2024-02-05T07:32:45.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:45 smithi090 ceph-mon[33213]: 4.1f scrub ok 2024-02-05T07:32:45.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:45 smithi090 ceph-mon[33213]: 1.1d scrub starts 2024-02-05T07:32:45.799 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:45.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:45.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:45.926 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:45.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:45.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:46.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:46.053 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:46.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:46.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:46.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:46.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:46.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:46.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:46.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:46.324 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:46.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:46.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:46.452 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:46.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:46.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:46.544 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:32:46.554 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:46.579 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:46.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:46.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:46 smithi007 ceph-mon[37158]: 1.1d scrub ok 2024-02-05T07:32:46.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:46 smithi007 ceph-mon[37158]: 1.19 scrub starts 2024-02-05T07:32:46.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:46 smithi007 ceph-mon[37158]: 1.19 scrub ok 2024-02-05T07:32:46.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:46 smithi007 ceph-mon[37158]: pgmap v722: 178 pgs: 178 active+clean; 2.0 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 612 op/s; 970 KiB/s, 76 objects/s recovering 2024-02-05T07:32:46.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:46 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:32:46.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:32:46.638 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:46 smithi007 ceph-mon[29098]: 1.1d scrub ok 2024-02-05T07:32:46.638 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:46 smithi007 ceph-mon[29098]: 1.19 scrub starts 2024-02-05T07:32:46.638 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:46 smithi007 ceph-mon[29098]: 1.19 scrub ok 2024-02-05T07:32:46.638 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:46 smithi007 ceph-mon[29098]: pgmap v722: 178 pgs: 178 active+clean; 2.0 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 612 op/s; 970 KiB/s, 76 objects/s recovering 2024-02-05T07:32:46.639 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:46 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:32:46.639 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:32:46.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:46.705 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:46.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:46.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:46.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:46 smithi090 ceph-mon[33213]: 1.1d scrub ok 2024-02-05T07:32:46.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:46 smithi090 ceph-mon[33213]: 1.19 scrub starts 2024-02-05T07:32:46.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:46 smithi090 ceph-mon[33213]: 1.19 scrub ok 2024-02-05T07:32:46.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:46 smithi090 ceph-mon[33213]: pgmap v722: 178 pgs: 178 active+clean; 2.0 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 612 op/s; 970 KiB/s, 76 objects/s recovering 2024-02-05T07:32:46.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:46 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:32:46.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:32:46.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:46.832 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:46.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:46.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:46.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:46.959 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:46.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:47.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:47.084 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:47.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:47.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:47.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:47.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:47.223 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:47.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:47.324 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:47.350 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:47.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:47.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:47.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:47.481 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:47.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:47.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:47.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:47.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:47.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:47 smithi007 ceph-mon[29098]: 6.9 scrub starts 2024-02-05T07:32:47.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:47 smithi007 ceph-mon[29098]: 6.9 scrub ok 2024-02-05T07:32:47.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:47 smithi007 ceph-mon[29098]: 5.7 scrub starts 2024-02-05T07:32:47.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:47 smithi007 ceph-mon[37158]: 6.9 scrub starts 2024-02-05T07:32:47.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:47 smithi007 ceph-mon[37158]: 6.9 scrub ok 2024-02-05T07:32:47.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:47 smithi007 ceph-mon[37158]: 5.7 scrub starts 2024-02-05T07:32:47.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:47.737 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:47.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:47.773 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:47 smithi090 ceph-mon[33213]: 6.9 scrub starts 2024-02-05T07:32:47.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:47 smithi090 ceph-mon[33213]: 6.9 scrub ok 2024-02-05T07:32:47.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:47 smithi090 ceph-mon[33213]: 5.7 scrub starts 2024-02-05T07:32:47.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:47.882 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:47.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:47.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:47.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:48.009 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:48.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:48.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:48.110 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:48.136 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:48.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:48.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:48.264 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:48.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:48.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:48.392 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:48.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:48.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:48.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:48.519 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:48.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:48.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:48.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:48.666 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:48.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:48.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:48.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:48 smithi007 ceph-mon[37158]: 5.7 scrub ok 2024-02-05T07:32:48.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:48 smithi007 ceph-mon[37158]: 5.a scrub starts 2024-02-05T07:32:48.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:48 smithi007 ceph-mon[37158]: 5.a scrub ok 2024-02-05T07:32:48.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:48 smithi007 ceph-mon[37158]: pgmap v723: 178 pgs: 178 active+clean; 1.7 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 681 KiB/s, 57 objects/s recovering 2024-02-05T07:32:48.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:48 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:32:48.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:48 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:32:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:48 smithi007 ceph-mon[29098]: 5.7 scrub ok 2024-02-05T07:32:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:48 smithi007 ceph-mon[29098]: 5.a scrub starts 2024-02-05T07:32:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:48 smithi007 ceph-mon[29098]: 5.a scrub ok 2024-02-05T07:32:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:48 smithi007 ceph-mon[29098]: pgmap v723: 178 pgs: 178 active+clean; 1.7 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 681 KiB/s, 57 objects/s recovering 2024-02-05T07:32:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:48 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:32:48.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:48 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:32:48.768 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:48.773 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:48 smithi090 ceph-mon[33213]: 5.7 scrub ok 2024-02-05T07:32:48.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:48 smithi090 ceph-mon[33213]: 5.a scrub starts 2024-02-05T07:32:48.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:48 smithi090 ceph-mon[33213]: 5.a scrub ok 2024-02-05T07:32:48.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:48 smithi090 ceph-mon[33213]: pgmap v723: 178 pgs: 178 active+clean; 1.7 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 681 KiB/s, 57 objects/s recovering 2024-02-05T07:32:48.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:48 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:32:48.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:48 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:32:48.794 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:48.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:48.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:48.941 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:48.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:48.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:49.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:49.068 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:49.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:49.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:49.193 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:49.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:49.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:49.347 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:49.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:49.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:49.449 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:49.474 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:49.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:49.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:49.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:49.601 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:49.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:49.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:49 smithi007 ceph-mon[29098]: 6.14 scrub starts 2024-02-05T07:32:49.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:49 smithi007 ceph-mon[29098]: 6.14 scrub ok 2024-02-05T07:32:49.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:49 smithi007 ceph-mon[29098]: 5.12 deep-scrub starts 2024-02-05T07:32:49.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:49 smithi007 ceph-mon[29098]: 5.12 deep-scrub ok 2024-02-05T07:32:49.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:49 smithi007 ceph-mon[29098]: osdmap e407: 8 total, 8 up, 6 in 2024-02-05T07:32:49.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:49 smithi007 ceph-mon[37158]: 6.14 scrub starts 2024-02-05T07:32:49.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:49 smithi007 ceph-mon[37158]: 6.14 scrub ok 2024-02-05T07:32:49.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:49 smithi007 ceph-mon[37158]: 5.12 deep-scrub starts 2024-02-05T07:32:49.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:49 smithi007 ceph-mon[37158]: 5.12 deep-scrub ok 2024-02-05T07:32:49.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:49 smithi007 ceph-mon[37158]: osdmap e407: 8 total, 8 up, 6 in 2024-02-05T07:32:49.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:49.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:49.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:49.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:49 smithi090 ceph-mon[33213]: 6.14 scrub starts 2024-02-05T07:32:49.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:49 smithi090 ceph-mon[33213]: 6.14 scrub ok 2024-02-05T07:32:49.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:49 smithi090 ceph-mon[33213]: 5.12 deep-scrub starts 2024-02-05T07:32:49.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:49 smithi090 ceph-mon[33213]: 5.12 deep-scrub ok 2024-02-05T07:32:49.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:49 smithi090 ceph-mon[33213]: osdmap e407: 8 total, 8 up, 6 in 2024-02-05T07:32:49.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:49.883 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:49.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:49.985 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:50.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:50.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:50.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:50.170 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:50.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:50.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:50.300 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:50.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:50.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:50.429 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:50.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:50.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:50.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:50.534 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:32:50.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:50.573 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:50.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:50.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:50.698 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:50.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:50.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:50.768 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:50 smithi090 ceph-mon[33213]: pgmap v725: 178 pgs: 178 active+clean; 1.7 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 676 KiB/s, 56 objects/s recovering 2024-02-05T07:32:50.768 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:50 smithi090 ceph-mon[33213]: osdmap e408: 8 total, 8 up, 6 in 2024-02-05T07:32:50.768 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:50 smithi090 ceph-mon[33213]: 3.1e scrub starts 2024-02-05T07:32:50.768 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:50 smithi090 ceph-mon[33213]: 3.1e scrub ok 2024-02-05T07:32:50.768 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:50 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2691953613' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:32:50.768 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:50 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:32:50.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:50.839 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:50.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:50.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:50.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:50 smithi007 ceph-mon[29098]: pgmap v725: 178 pgs: 178 active+clean; 1.7 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 676 KiB/s, 56 objects/s recovering 2024-02-05T07:32:50.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:50 smithi007 ceph-mon[29098]: osdmap e408: 8 total, 8 up, 6 in 2024-02-05T07:32:50.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:50 smithi007 ceph-mon[29098]: 3.1e scrub starts 2024-02-05T07:32:50.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:50 smithi007 ceph-mon[29098]: 3.1e scrub ok 2024-02-05T07:32:50.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:50 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2691953613' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:32:50.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:50 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:32:50.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:50 smithi007 ceph-mon[37158]: pgmap v725: 178 pgs: 178 active+clean; 1.7 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 676 KiB/s, 56 objects/s recovering 2024-02-05T07:32:50.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:50 smithi007 ceph-mon[37158]: osdmap e408: 8 total, 8 up, 6 in 2024-02-05T07:32:50.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:50 smithi007 ceph-mon[37158]: 3.1e scrub starts 2024-02-05T07:32:50.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:50 smithi007 ceph-mon[37158]: 3.1e scrub ok 2024-02-05T07:32:50.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:50 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2691953613' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:32:50.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:50 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:32:50.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:50.966 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:50.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:51.023 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:32:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:32:51.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:51.091 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:51.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:51.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:51.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:51.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:51.321 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:51.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:51.365 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:51.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:51.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:51.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:51.490 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:51.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:51.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:51.631 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:51.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:51.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:51.733 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:51 smithi090 ceph-mon[33213]: 5.3 scrub starts 2024-02-05T07:32:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:51 smithi090 ceph-mon[33213]: 5.3 scrub ok 2024-02-05T07:32:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:51 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:32:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:51 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:32:51.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:51 smithi090 ceph-mon[33213]: osdmap e409: 8 total, 8 up, 6 in 2024-02-05T07:32:51.775 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:51.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:51.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:51.878 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:51.904 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:51.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:51.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:51.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:51 smithi007 ceph-mon[29098]: 5.3 scrub starts 2024-02-05T07:32:51.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:51 smithi007 ceph-mon[29098]: 5.3 scrub ok 2024-02-05T07:32:51.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:51 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:32:51.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:51 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:32:51.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:51 smithi007 ceph-mon[29098]: osdmap e409: 8 total, 8 up, 6 in 2024-02-05T07:32:51.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:51 smithi007 ceph-mon[37158]: 5.3 scrub starts 2024-02-05T07:32:51.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:51 smithi007 ceph-mon[37158]: 5.3 scrub ok 2024-02-05T07:32:51.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:51 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:32:51.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:51 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:32:51.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:51 smithi007 ceph-mon[37158]: osdmap e409: 8 total, 8 up, 6 in 2024-02-05T07:32:52.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:52.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:52.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:52.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:52.194 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:52.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:52.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:52.328 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:52.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:52.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:52.469 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:52.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:52.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:52.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:52.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:52.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:52.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:52.724 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:52.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:52.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:52 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:52.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:52 smithi090 ceph-mon[33213]: pgmap v728: 178 pgs: 178 active+clean; 1.7 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s 2024-02-05T07:32:52.807 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:52 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:52.807 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:52 smithi007 ceph-mon[29098]: pgmap v728: 178 pgs: 178 active+clean; 1.7 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s 2024-02-05T07:32:52.807 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:52 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:32:52.807 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:52 smithi007 ceph-mon[37158]: pgmap v728: 178 pgs: 178 active+clean; 1.7 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s 2024-02-05T07:32:52.825 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:52.866 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:52.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:52.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:52.968 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:53.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:53.015 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:53.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:53.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:53.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:53.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:53.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:53.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:53.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:53.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:53.421 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:53.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:53.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:53.547 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:53.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:53.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:53.696 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:53.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:53.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:53.823 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:53.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:53.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:32:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:32:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:32:53.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:53.949 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:53.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:54.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:54.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:54.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:54.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:54.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:54.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:54.349 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:54.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:54.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:54.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:54.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:54.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:54.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:54.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:54.602 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:54 smithi007 ceph-mon[29098]: pgmap v729: 178 pgs: 178 active+clean; 1.4 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 955 op/s 2024-02-05T07:32:54.603 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:54 smithi007 ceph-mon[29098]: osdmap e410: 8 total, 8 up, 6 in 2024-02-05T07:32:54.603 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:54 smithi007 ceph-mon[37158]: pgmap v729: 178 pgs: 178 active+clean; 1.4 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 955 op/s 2024-02-05T07:32:54.603 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:54 smithi007 ceph-mon[37158]: osdmap e410: 8 total, 8 up, 6 in 2024-02-05T07:32:54.625 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:54.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:54.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:54.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:54.765 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:54.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:54.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:32:54.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:54.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:55.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:54 smithi090 ceph-mon[33213]: pgmap v729: 178 pgs: 178 active+clean; 1.4 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 955 op/s 2024-02-05T07:32:55.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:54 smithi090 ceph-mon[33213]: osdmap e410: 8 total, 8 up, 6 in 2024-02-05T07:32:55.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:55.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:55.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:55.178 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:55.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:55.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:55.280 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:55.293 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:32:55.293 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:32:55.293 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-02-05T07:32:55.293 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-02-05T07:32:55.293 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:32:55.322 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:55.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:55.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:55.449 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:55.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:55.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:55.576 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:55.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:55.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:55.720 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:55.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:55.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:55.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:55.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:55.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:55.972 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T07:32:56.009 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:56.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:56.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:56.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:56.160 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:56.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:56.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:56.187 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:32:56.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:56.315 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:56.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:56.416 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:56.441 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:56.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:56.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:56.526 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:32:56.543 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:56.608 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:56.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:56.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:56.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:56.735 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:56.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:56.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:56.825 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:56 smithi090 ceph-mon[33213]: pgmap v731: 178 pgs: 178 active+clean; 1.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 955 op/s 2024-02-05T07:32:56.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:56.863 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:56.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:56.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:56.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:56 smithi007 ceph-mon[29098]: pgmap v731: 178 pgs: 178 active+clean; 1.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 955 op/s 2024-02-05T07:32:56.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:56 smithi007 ceph-mon[37158]: pgmap v731: 178 pgs: 178 active+clean; 1.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 955 op/s 2024-02-05T07:32:56.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:57.003 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:57.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:57.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:57.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:57.144 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:57.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:57.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:57.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:32:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:57.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:32:57.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:57.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:57.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:57.537 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:57.564 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:57.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:57.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:57.572 INFO:teuthology.orchestra.run.smithi007.stderr:set 6.0 pg_upmap mapping to [1,0,6] 2024-02-05T07:32:57.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:57.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:57 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:32:57.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:57 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:32:57.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:57 smithi007 ceph-mon[29098]: osdmap e411: 8 total, 8 up, 6 in 2024-02-05T07:32:57.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:57 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/821932686' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [1, 0, 6]}]: dispatch 2024-02-05T07:32:57.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:57 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [1, 0, 6]}]: dispatch 2024-02-05T07:32:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:57 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:32:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:57 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:32:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:57 smithi007 ceph-mon[37158]: osdmap e411: 8 total, 8 up, 6 in 2024-02-05T07:32:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:57 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/821932686' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [1, 0, 6]}]: dispatch 2024-02-05T07:32:57.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:57 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [1, 0, 6]}]: dispatch 2024-02-05T07:32:57.692 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:57.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:57.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:57.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:57.822 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:57.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:57.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:57.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:57.968 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:57.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:57.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:58.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:57 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:32:58.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:57 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:32:58.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:57 smithi090 ceph-mon[33213]: osdmap e411: 8 total, 8 up, 6 in 2024-02-05T07:32:58.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:57 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/821932686' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [1, 0, 6]}]: dispatch 2024-02-05T07:32:58.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:57 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [1, 0, 6]}]: dispatch 2024-02-05T07:32:58.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:58.095 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:58.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:58.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:32:58.222 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:58.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:32:58.324 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:58.350 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:58.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:58.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:58.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:58.477 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:58.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:58.579 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:32:58.637 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:58.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:58.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:32:58.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:32:58.765 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:58.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:58.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:32:58.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:58.905 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:58.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:58.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:58.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:58 smithi007 ceph-mon[29098]: pgmap v733: 178 pgs: 178 active+clean; 1.0 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-02-05T07:32:58.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:58 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [1, 0, 6]}]': finished 2024-02-05T07:32:58.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:58 smithi007 ceph-mon[29098]: osdmap e412: 8 total, 8 up, 6 in 2024-02-05T07:32:58.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:58 smithi007 ceph-mon[37158]: pgmap v733: 178 pgs: 178 active+clean; 1.0 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-02-05T07:32:58.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:58 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [1, 0, 6]}]': finished 2024-02-05T07:32:58.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:58 smithi007 ceph-mon[37158]: osdmap e412: 8 total, 8 up, 6 in 2024-02-05T07:32:59.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:59.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:58 smithi090 ceph-mon[33213]: pgmap v733: 178 pgs: 178 active+clean; 1.0 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-02-05T07:32:59.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:58 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [1, 0, 6]}]': finished 2024-02-05T07:32:59.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:58 smithi090 ceph-mon[33213]: osdmap e412: 8 total, 8 up, 6 in 2024-02-05T07:32:59.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:59.051 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:59.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:59.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:32:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:59.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:59.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:32:59.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:32:59.323 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:59.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:59.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:32:59.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:59.461 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:59.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:59.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:59.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:59.604 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:59.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:59.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:32:59.748 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:32:59.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:59.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:32:59.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:32:59.875 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:32:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:32:59.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:32:59.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:59 smithi007 ceph-mon[29098]: osdmap e413: 8 total, 8 up, 6 in 2024-02-05T07:32:59.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:59 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:32:59.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:59 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:32:59.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:32:59 smithi007 ceph-mon[29098]: osdmap e414: 8 total, 8 up, 6 in 2024-02-05T07:32:59.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:59 smithi007 ceph-mon[37158]: osdmap e413: 8 total, 8 up, 6 in 2024-02-05T07:32:59.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:59 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:32:59.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:59 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:32:59.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:32:59 smithi007 ceph-mon[37158]: osdmap e414: 8 total, 8 up, 6 in 2024-02-05T07:32:59.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:00.001 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:00.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:00.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:59 smithi090 ceph-mon[33213]: osdmap e413: 8 total, 8 up, 6 in 2024-02-05T07:33:00.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:59 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:33:00.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:59 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:33:00.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:32:59 smithi090 ceph-mon[33213]: osdmap e414: 8 total, 8 up, 6 in 2024-02-05T07:33:00.103 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:00.143 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:00.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:00.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:00.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:00.271 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:00.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:00.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:00.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:00.412 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:00.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:00.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:00.514 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:00.540 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:00.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:00.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:00.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:00.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:00.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:00.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:00.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:00.810 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:00.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:00.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:00.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:00 smithi007 ceph-mon[29098]: pgmap v736: 178 pgs: 178 active+clean; 1.0 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T07:33:00.862 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:00 smithi007 ceph-mon[37158]: pgmap v736: 178 pgs: 178 active+clean; 1.0 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T07:33:00.872 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:00.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:00.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:00.936 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:00.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:01.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:00 smithi090 ceph-mon[33213]: pgmap v736: 178 pgs: 178 active+clean; 1.0 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T07:33:01.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:33:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:33:01.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:01.062 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:01.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:01.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:01.205 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:01.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:01.307 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:01.333 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:01.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:01.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:01.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:01.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:01.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:01.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:01.608 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:01.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:01.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:01.621 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:01 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:33:01.621 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:01 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:33:01.622 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:01 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:33:01.622 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:01 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:33:01.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:01.738 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:01.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:01.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:01.865 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:01.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:01.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:02.010 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:02.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:02.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:02.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:01 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:33:02.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:01 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:33:02.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:02.152 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:02.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:02.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:02.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:02.292 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:02.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:02.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:02.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:02.420 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:02.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:02.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:02.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:02.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:02.549 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:02.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:02.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:02.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:02 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:02.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:02 smithi007 ceph-mon[29098]: pgmap v738: 178 pgs: 178 active+clean; 978 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T07:33:02.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:02 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:02.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:02 smithi007 ceph-mon[37158]: pgmap v738: 178 pgs: 178 active+clean; 978 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T07:33:02.677 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:02.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:02.779 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:02.803 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:02.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:02.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:02.905 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:02.954 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:02.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:03.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:02 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:03.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:02 smithi090 ceph-mon[33213]: pgmap v738: 178 pgs: 178 active+clean; 978 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T07:33:03.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:03.102 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:03.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:03.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:03.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:03.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:03.333 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:03.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:03.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:03.504 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:03.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:03.550 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 3] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:33:03.550 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:33:03.551 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-05T07:33:03.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T07:33:03.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:03.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:03.655 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:03.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:03.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:03.799 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:33:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:33:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:33:03.800 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:03.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:03.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:03.928 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:03.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:04.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:04.050 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:33:04.055 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:04.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:04.100 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:33:04.143 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:33:04.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:04.181 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:04.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:04.191 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:33:04.235 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:33:04.281 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:33:04.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:04.307 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:04.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:04.322 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:33:04.372 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:33:04.392 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:33:04.392 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:33:04.392 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:33:04.392 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:33:04.392 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:33:04.392 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:33:04.392 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:33:04.392 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:33:04.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:04.433 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:04.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:04.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:04.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:04.660 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:04.685 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:04.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:04.786 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:04.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:04.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:04 smithi007 ceph-mon[29098]: pgmap v739: 178 pgs: 178 active+clean; 679 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 988 op/s 2024-02-05T07:33:04.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:04 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4076777608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:04.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:04 smithi007 ceph-mon[37158]: pgmap v739: 178 pgs: 178 active+clean; 679 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 988 op/s 2024-02-05T07:33:04.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:04 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4076777608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:04.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:04.961 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:04.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:04.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:05.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:04 smithi090 ceph-mon[33213]: pgmap v739: 178 pgs: 178 active+clean; 679 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 988 op/s 2024-02-05T07:33:05.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:04 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4076777608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:05.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:05.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:05.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:05.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:05.216 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:05.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:05.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:05.343 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:05.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:05.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:05.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:05.458 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:05.470 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:05.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:05.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:05.572 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:05.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:05.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:05.617 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:33:05.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:05.726 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:05.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:05.786 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:05 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3784911102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:33:05.787 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:05 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:33:05.787 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:05 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3784911102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:33:05.787 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:05 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:33:05.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:05.852 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:05.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:05.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:05.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:05.979 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:05.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:06.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:05 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3784911102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:33:06.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:05 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:33:06.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:06.107 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:06.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:06.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:06.234 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:06.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:06.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:06.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:06.361 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:06.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:06.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:06.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:06.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:06.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:06.644 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:06.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:06.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:06.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:06.789 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:06.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:06.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:06.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:06 smithi007 ceph-mon[29098]: pgmap v740: 178 pgs: 178 active+clean; 656 MiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 665 B/s rd, 0 B/s wr, 819 op/s 2024-02-05T07:33:06.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:06 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:33:06.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:06 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:33:06.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:06 smithi007 ceph-mon[29098]: osdmap e415: 8 total, 8 up, 6 in 2024-02-05T07:33:06.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:06 smithi007 ceph-mon[37158]: pgmap v740: 178 pgs: 178 active+clean; 656 MiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 665 B/s rd, 0 B/s wr, 819 op/s 2024-02-05T07:33:06.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:06 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:33:06.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:06 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:33:06.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:06 smithi007 ceph-mon[37158]: osdmap e415: 8 total, 8 up, 6 in 2024-02-05T07:33:06.932 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:06.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:07.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:06 smithi090 ceph-mon[33213]: pgmap v740: 178 pgs: 178 active+clean; 656 MiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 665 B/s rd, 0 B/s wr, 819 op/s 2024-02-05T07:33:07.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:06 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:33:07.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:06 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:33:07.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:06 smithi090 ceph-mon[33213]: osdmap e415: 8 total, 8 up, 6 in 2024-02-05T07:33:07.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:07.060 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:07.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:07.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:07.200 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:07.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:07.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:07.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:07.342 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:07.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:07.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:07.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:07.469 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:07.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:07.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:07.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:33:07.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:07.618 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:07.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:07.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:07.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:07.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:07.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:07.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:07.896 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:07.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:07.999 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:08.024 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:08.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:08.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:08.152 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:08.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:08.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:08.232 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:33:08.232 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:33:08.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:08.280 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:08.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:08.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:08.407 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:08.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:08.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:08.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:08.551 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:08.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:08.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:08.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:08.720 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:08.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:08.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:08.822 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:08.863 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:08.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:08.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:08.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:08 smithi007 ceph-mon[29098]: pgmap v742: 178 pgs: 178 active+clean; 303 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-02-05T07:33:08.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:08 smithi007 ceph-mon[37158]: pgmap v742: 178 pgs: 178 active+clean; 303 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-02-05T07:33:08.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:08.992 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:08.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:08.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:09.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T07:33:09.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:08 smithi090 ceph-mon[33213]: pgmap v742: 178 pgs: 178 active+clean; 303 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-02-05T07:33:09.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:09.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:09.144 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:09.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:09.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:09.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:09.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:09.371 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:09.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:09.421 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:09.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:09.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:09.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:33:09.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:09.549 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:09.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:09.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:09.651 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:09.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:09.677 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:09.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:09.776 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:33:09.776 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:33:09.777 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 416, 2024-02-05T07:33:09.777 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "e3e7059e-c3f5-11ee-95b6-87774f69a715", 2024-02-05T07:33:09.777 INFO:teuthology.orchestra.run.smithi007.stdout: "created": "2024-02-05T07:13:14.603885+0000", 2024-02-05T07:33:09.777 INFO:teuthology.orchestra.run.smithi007.stdout: "modified": "2024-02-05T07:33:09.513614+0000", 2024-02-05T07:33:09.777 INFO:teuthology.orchestra.run.smithi007.stdout: "last_up_change": "2024-02-05T07:28:24.555976+0000", 2024-02-05T07:33:09.777 INFO:teuthology.orchestra.run.smithi007.stdout: "last_in_change": "2024-02-05T07:30:39.451715+0000", 2024-02-05T07:33:09.777 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T07:33:09.777 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_num": 5806080, 2024-02-05T07:33:09.777 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_set": [ 2024-02-05T07:33:09.777 INFO:teuthology.orchestra.run.smithi007.stdout: "nodeep-scrub", 2024-02-05T07:33:09.777 INFO:teuthology.orchestra.run.smithi007.stdout: "noscrub", 2024-02-05T07:33:09.777 INFO:teuthology.orchestra.run.smithi007.stdout: "pglog_hardlimit", 2024-02-05T07:33:09.777 INFO:teuthology.orchestra.run.smithi007.stdout: "purged_snapdirs", 2024-02-05T07:33:09.777 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_deletes", 2024-02-05T07:33:09.778 INFO:teuthology.orchestra.run.smithi007.stdout: "sortbitwise" 2024-02-05T07:33:09.778 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:33:09.778 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_version": 19, 2024-02-05T07:33:09.778 INFO:teuthology.orchestra.run.smithi007.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T07:33:09.778 INFO:teuthology.orchestra.run.smithi007.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T07:33:09.778 INFO:teuthology.orchestra.run.smithi007.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T07:33:09.778 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_snapshot": "", 2024-02-05T07:33:09.778 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_max": 9, 2024-02-05T07:33:09.778 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd": 8, 2024-02-05T07:33:09.778 INFO:teuthology.orchestra.run.smithi007.stdout: "require_min_compat_client": "luminous", 2024-02-05T07:33:09.778 INFO:teuthology.orchestra.run.smithi007.stdout: "min_compat_client": "luminous", 2024-02-05T07:33:09.778 INFO:teuthology.orchestra.run.smithi007.stdout: "require_osd_release": "squid", 2024-02-05T07:33:09.778 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_crimson": false, 2024-02-05T07:33:09.779 INFO:teuthology.orchestra.run.smithi007.stdout: "pools": [ 2024-02-05T07:33:09.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.779 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 1, 2024-02-05T07:33:09.779 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".mgr", 2024-02-05T07:33:09.779 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:16:32.505567+0000", 2024-02-05T07:33:09.779 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:33:09.779 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:33:09.779 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:33:09.780 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:33:09.780 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:33:09.780 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:33:09.780 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:33:09.780 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:33:09.780 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:33:09.780 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:33:09.780 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:33:09.780 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:33:09.780 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 31, 2024-02-05T07:33:09.780 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 31, 2024-02-05T07:33:09.780 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 31, 2024-02-05T07:33:09.781 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 31, 2024-02-05T07:33:09.781 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 31, 2024-02-05T07:33:09.781 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:33:09.781 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:33:09.781 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:33:09.781 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:33:09.781 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:33:09.781 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:33:09.781 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:33:09.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.782 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "385", 2024-02-05T07:33:09.782 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:33:09.782 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:33:09.782 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "383", 2024-02-05T07:33:09.782 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:33:09.782 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:33:09.782 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:33:09.782 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:33:09.782 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:33:09.782 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:33:09.782 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:33:09.782 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:33:09.782 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:33:09.782 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:33:09.783 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:33:09.783 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:33:09.783 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:33:09.783 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:33:09.783 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:33:09.783 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:33:09.783 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:33:09.783 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:33:09.783 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:33:09.783 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:33:09.783 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:33:09.783 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:33:09.784 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:33:09.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.784 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:33:09.784 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:33:09.784 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:33:09.784 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:33:09.784 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:33:09.784 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:33:09.784 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:33:09.784 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:33:09.784 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:33:09.784 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:33:09.784 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_max": 32, 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_min": 1 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": {} 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.5700000524520874, 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.5700000524520874, 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.3500000238418579, 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.3500000238418579, 2024-02-05T07:33:09.785 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 2, 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "datapool", 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:47.259462+0000", 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 8193, 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:33:09.786 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:33:09.787 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:33:09.787 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:33:09.787 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:33:09.787 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:33:09.787 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:33:09.787 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 3, 2024-02-05T07:33:09.787 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 3, 2024-02-05T07:33:09.787 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 3, 2024-02-05T07:33:09.787 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 3, 2024-02-05T07:33:09.787 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 3, 2024-02-05T07:33:09.787 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:33:09.787 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:33:09.787 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:33:09.787 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:33:09.788 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:33:09.788 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:33:09.788 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:33:09.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.788 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "71", 2024-02-05T07:33:09.788 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:33:09.788 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:33:09.788 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:33:09.788 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:33:09.788 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:33:09.788 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 2, 2024-02-05T07:33:09.788 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 71, 2024-02-05T07:33:09.788 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:33:09.789 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:33:09.790 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:33:09.790 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:33:09.790 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:33:09.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.790 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:33:09.790 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:33:09.790 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:33:09.790 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:33:09.790 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:33:09.790 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:33:09.790 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:33:09.790 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:33:09.790 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:33:09.791 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:33:09.791 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:33:09.791 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:33:09.791 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:33:09.791 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd": {} 2024-02-05T07:33:09.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.791 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:33:09.791 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:33:09.791 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 4.690000057220459, 2024-02-05T07:33:09.791 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 4.690000057220459, 2024-02-05T07:33:09.791 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.70999997854232788, 2024-02-05T07:33:09.791 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 3.3299999237060547, 2024-02-05T07:33:09.791 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 3.3299999237060547, 2024-02-05T07:33:09.791 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:33:09.792 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:33:09.792 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:33:09.792 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.792 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 3, 2024-02-05T07:33:09.792 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".rgw.root", 2024-02-05T07:33:09.792 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:47.762336+0000", 2024-02-05T07:33:09.792 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:33:09.792 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:33:09.792 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:33:09.792 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:33:09.792 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:33:09.792 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:33:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:33:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:33:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:33:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:33:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:33:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:33:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:33:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:33:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:33:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:33:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:33:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:33:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:33:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:33:09.794 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:33:09.794 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:33:09.794 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:33:09.794 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:33:09.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.794 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "69", 2024-02-05T07:33:09.794 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:33:09.794 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:33:09.794 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:33:09.794 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:33:09.794 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:33:09.794 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:33:09.794 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:33:09.795 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:33:09.795 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:33:09.795 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:33:09.795 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:33:09.795 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:33:09.795 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:33:09.795 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:33:09.795 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:33:09.795 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:33:09.795 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:33:09.795 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:33:09.795 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:33:09.795 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:33:09.795 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:33:09.796 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:33:09.796 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:33:09.796 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:33:09.796 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:33:09.796 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:33:09.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.796 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:33:09.796 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:33:09.796 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:33:09.796 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:33:09.796 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:33:09.796 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:33:09.796 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:33:09.796 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.4000000953674316, 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.4000000953674316, 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.059999942779541, 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.059999942779541, 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:33:09.797 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 4, 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.log", 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:49.736599+0000", 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:33:09.798 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:33:09.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.800 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "71", 2024-02-05T07:33:09.800 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:33:09.800 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:33:09.800 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:33:09.800 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:33:09.800 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:33:09.800 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:33:09.800 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:33:09.800 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:33:09.800 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:33:09.800 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:33:09.800 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:33:09.800 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:33:09.800 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:33:09.801 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:33:09.801 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:33:09.801 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:33:09.801 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:33:09.801 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:33:09.801 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:33:09.801 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:33:09.801 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:33:09.801 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:33:09.801 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:33:09.801 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:33:09.801 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:33:09.802 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:33:09.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.802 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:33:09.802 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:33:09.802 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:33:09.802 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:33:09.802 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:33:09.802 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:33:09.802 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:33:09.802 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:33:09.802 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:33:09.802 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:33:09.802 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:33:09.802 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.190000057220459, 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.190000057220459, 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.8799999952316284, 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.8799999952316284, 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.804 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 5, 2024-02-05T07:33:09.804 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.control", 2024-02-05T07:33:09.804 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:51.696948+0000", 2024-02-05T07:33:09.804 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:33:09.804 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:33:09.804 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:33:09.804 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:33:09.804 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:33:09.804 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:33:09.804 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:33:09.804 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:33:09.804 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:33:09.804 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:33:09.805 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:33:09.805 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:33:09.805 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:33:09.805 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:33:09.805 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:33:09.805 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:33:09.805 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:33:09.805 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:33:09.805 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:33:09.805 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:33:09.805 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:33:09.805 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:33:09.805 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:33:09.805 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "73", 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:33:09.806 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:33:09.807 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:33:09.807 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:33:09.807 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:33:09.807 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:33:09.807 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:33:09.807 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:33:09.807 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:33:09.807 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:33:09.807 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:33:09.807 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:33:09.807 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:33:09.807 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:33:09.807 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:33:09.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.808 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:33:09.808 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:33:09.808 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:33:09.808 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:33:09.808 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:33:09.808 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:33:09.808 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:33:09.808 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:33:09.808 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:33:09.808 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:33:09.808 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:33:09.808 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:33:09.808 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:33:09.809 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:33:09.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.809 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:33:09.809 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:33:09.809 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.7400000095367432, 2024-02-05T07:33:09.809 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.7400000095367432, 2024-02-05T07:33:09.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:09.810 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:33:09.810 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.5, 2024-02-05T07:33:09.810 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.5, 2024-02-05T07:33:09.810 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:33:09.810 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:33:09.810 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:33:09.810 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 6, 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.meta", 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:53.768555+0000", 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:33:09.811 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:33:09.812 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:33:09.812 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:33:09.812 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:33:09.812 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:33:09.812 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:33:09.812 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:33:09.812 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:33:09.812 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:33:09.812 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:33:09.812 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:33:09.812 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:33:09.812 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:33:09.812 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:33:09.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.813 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "75", 2024-02-05T07:33:09.813 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:33:09.813 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:33:09.813 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:33:09.813 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:33:09.813 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:33:09.813 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:33:09.813 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:33:09.813 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:33:09.813 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:33:09.813 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:33:09.813 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:33:09.813 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:33:09.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.815 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:33:09.815 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:33:09.815 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:33:09.815 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:33:09.815 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:33:09.815 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:33:09.815 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:33:09.815 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:33:09.815 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:33:09.815 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:33:09.815 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:33:09.815 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-02-05T07:33:09.815 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_bias": 4 2024-02-05T07:33:09.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.816 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:33:09.816 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:33:09.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.816 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:33:09.816 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:33:09.816 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9700000286102295, 2024-02-05T07:33:09.816 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9700000286102295, 2024-02-05T07:33:09.816 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:33:09.816 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.690000057220459, 2024-02-05T07:33:09.816 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.690000057220459, 2024-02-05T07:33:09.816 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:33:09.816 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:33:09.817 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:33:09.817 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.817 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 9, 2024-02-05T07:33:09.817 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "unique_pool_2", 2024-02-05T07:33:09.817 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:29:59.438957+0000", 2024-02-05T07:33:09.817 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:33:09.817 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:33:09.817 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:33:09.817 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:33:09.817 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 16, 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 16, 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 16, 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 16, 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 16, 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:33:09.818 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:33:09.819 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:33:09.819 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:33:09.819 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:33:09.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.819 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "354", 2024-02-05T07:33:09.819 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:33:09.819 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:33:09.819 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:33:09.819 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:33:09.819 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:33:09.819 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:33:09.819 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:33:09.819 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:33:09.820 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:33:09.821 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: "rados": {} 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 3.059999942779541, 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 3.059999942779541, 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.630000114440918, 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.630000114440918, 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84203529357910156, 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.822 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "703c3948-8003-4015-9dd8-21019cf0eeb8", 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0, 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 8, 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 387, 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:33:09.823 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:33:09.824 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.824 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.824 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-02-05T07:33:09.824 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:33:09.824 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.824 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.824 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:33:09.824 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.824 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.825 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6802", 2024-02-05T07:33:09.825 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:33:09.825 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.825 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.825 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:33:09.825 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.825 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.825 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6804", 2024-02-05T07:33:09.825 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:33:09.825 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.825 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.826 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:33:09.826 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.826 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.826 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6803", 2024-02-05T07:33:09.826 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:33:09.826 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.826 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.827 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6801/2069016689", 2024-02-05T07:33:09.827 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6802/2069016689", 2024-02-05T07:33:09.827 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6804/2069016689", 2024-02-05T07:33:09.827 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6803/2069016689", 2024-02-05T07:33:09.827 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:33:09.827 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:33:09.827 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:33:09.827 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.827 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-02-05T07:33:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "5c3e7d1e-a61d-44b9-b21e-f91c99a096bc", 2024-02-05T07:33:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:33:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:33:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:33:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:33:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:33:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:33:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 13, 2024-02-05T07:33:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 412, 2024-02-05T07:33:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:33:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:33:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:33:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.829 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.829 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6805", 2024-02-05T07:33:09.829 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:33:09.829 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.829 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.829 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:33:09.829 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.829 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.829 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6806", 2024-02-05T07:33:09.829 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:33:09.829 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.829 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.830 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:33:09.830 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.830 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.830 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6808", 2024-02-05T07:33:09.830 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:33:09.830 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.830 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.830 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:33:09.830 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.831 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.831 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6807", 2024-02-05T07:33:09.831 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:33:09.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.831 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.831 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6805/865486532", 2024-02-05T07:33:09.831 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6806/865486532", 2024-02-05T07:33:09.831 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6808/865486532", 2024-02-05T07:33:09.831 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6807/865486532", 2024-02-05T07:33:09.831 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:33:09.831 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:33:09.831 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:33:09.832 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.832 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-02-05T07:33:09.832 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "f4fd930f-7bb2-46ca-9964-98235bd47b3a", 2024-02-05T07:33:09.832 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:33:09.832 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:33:09.832 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:33:09.832 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:33:09.832 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:33:09.832 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:33:09.832 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 18, 2024-02-05T07:33:09.832 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 395, 2024-02-05T07:33:09.833 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:33:09.833 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:33:09.833 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:33:09.833 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.833 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.833 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6809", 2024-02-05T07:33:09.833 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:33:09.833 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.833 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.833 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:33:09.834 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.834 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.834 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6810", 2024-02-05T07:33:09.834 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:33:09.834 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.834 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.834 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:33:09.834 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.834 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.835 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6812", 2024-02-05T07:33:09.835 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:33:09.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.835 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.835 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:33:09.835 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.835 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.835 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6811", 2024-02-05T07:33:09.835 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:33:09.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.835 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.836 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6809/904871370", 2024-02-05T07:33:09.836 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6810/904871370", 2024-02-05T07:33:09.836 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6812/904871370", 2024-02-05T07:33:09.836 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6811/904871370", 2024-02-05T07:33:09.836 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:33:09.836 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:33:09.836 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:33:09.836 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.836 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-02-05T07:33:09.836 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "149ed0d6-b049-4478-9819-cafc446e94d3", 2024-02-05T07:33:09.837 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:33:09.837 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:33:09.837 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:33:09.837 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:33:09.837 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:33:09.837 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:33:09.837 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 26, 2024-02-05T07:33:09.837 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 405, 2024-02-05T07:33:09.837 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:33:09.837 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:33:09.837 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:33:09.837 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.837 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.838 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6813", 2024-02-05T07:33:09.838 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:33:09.838 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.838 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.838 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:33:09.838 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.838 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.838 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6814", 2024-02-05T07:33:09.838 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:33:09.838 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.839 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.839 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:33:09.839 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.839 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.839 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6816", 2024-02-05T07:33:09.839 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:33:09.839 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.839 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.839 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:33:09.839 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.840 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.840 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6815", 2024-02-05T07:33:09.840 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:33:09.840 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.840 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.840 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6813/2758586242", 2024-02-05T07:33:09.840 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6814/2758586242", 2024-02-05T07:33:09.840 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6816/2758586242", 2024-02-05T07:33:09.840 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6815/2758586242", 2024-02-05T07:33:09.840 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:33:09.840 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:33:09.841 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:33:09.841 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.841 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-02-05T07:33:09.841 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d", 2024-02-05T07:33:09.841 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:33:09.841 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:33:09.841 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:33:09.841 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:33:09.841 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:33:09.841 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:33:09.841 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 32, 2024-02-05T07:33:09.842 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 395, 2024-02-05T07:33:09.842 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:33:09.842 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:33:09.842 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:33:09.842 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.842 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.842 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6800", 2024-02-05T07:33:09.842 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:33:09.842 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.842 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.842 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:33:09.842 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6801", 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6803", 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.843 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6802", 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6800/1154929470", 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6801/1154929470", 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6803/1154929470", 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6802/1154929470", 2024-02-05T07:33:09.844 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "244a7930-c22c-413e-86e1-11e0749f7367", 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0.7362823486328125, 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 41, 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 336, 2024-02-05T07:33:09.845 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:33:09.846 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:33:09.846 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:33:09.846 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.846 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.846 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6804", 2024-02-05T07:33:09.846 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:33:09.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.847 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.847 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:33:09.847 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.847 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.848 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6805", 2024-02-05T07:33:09.848 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:33:09.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.848 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.848 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:33:09.848 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.848 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.848 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6807", 2024-02-05T07:33:09.848 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:33:09.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.848 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.849 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:33:09.849 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.849 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.849 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6806", 2024-02-05T07:33:09.849 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:33:09.849 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.849 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.849 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6804/398113577", 2024-02-05T07:33:09.849 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6805/398113577", 2024-02-05T07:33:09.849 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6807/398113577", 2024-02-05T07:33:09.849 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6806/398113577", 2024-02-05T07:33:09.850 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:33:09.850 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:33:09.850 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:33:09.850 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.850 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-02-05T07:33:09.850 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7", 2024-02-05T07:33:09.850 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:33:09.850 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:33:09.850 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:33:09.850 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:33:09.850 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 52, 2024-02-05T07:33:09.851 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 230, 2024-02-05T07:33:09.851 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 301, 2024-02-05T07:33:09.851 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 388, 2024-02-05T07:33:09.851 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 231, 2024-02-05T07:33:09.851 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:33:09.851 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:33:09.851 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.851 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.851 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6808", 2024-02-05T07:33:09.851 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:33:09.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.851 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.852 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:33:09.852 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.852 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.852 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6809", 2024-02-05T07:33:09.852 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:33:09.852 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.852 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.852 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:33:09.852 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.852 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6811", 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6810", 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.853 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6808/313196533", 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6809/313196533", 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6811/313196533", 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6810/313196533", 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9", 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-02-05T07:33:09.854 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:33:09.855 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:33:09.855 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:33:09.855 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 60, 2024-02-05T07:33:09.855 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 349, 2024-02-05T07:33:09.855 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:33:09.855 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:33:09.855 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:33:09.855 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.855 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.855 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.855 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6812", 2024-02-05T07:33:09.855 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:33:09.855 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.855 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.856 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:33:09.856 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.856 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.856 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6813", 2024-02-05T07:33:09.856 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:33:09.856 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.856 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.856 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:33:09.856 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.856 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.857 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6815", 2024-02-05T07:33:09.857 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:33:09.857 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.857 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.857 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:33:09.857 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:33:09.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.857 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:33:09.857 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6814", 2024-02-05T07:33:09.857 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:33:09.857 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.857 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6812/2983605980", 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6813/2983605980", 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2983605980", 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6814/2983605980", 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_xinfo": [ 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-02-05T07:33:09.858 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:33:09.859 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:33:09.859 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:33:09.859 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:33:09.859 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:33:09.859 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:33:09.343196+0000", 2024-02-05T07:33:09.859 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:33:09.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.859 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-02-05T07:33:09.859 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:33:09.859 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:33:09.859 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:33:09.859 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:33:09.859 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:33:09.860 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:32:14.901479+0000", 2024-02-05T07:33:09.860 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:33:09.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.860 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-02-05T07:33:09.860 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:33:09.860 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:33:09.860 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:33:09.860 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:33:09.860 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:33:09.860 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:32:47.527890+0000", 2024-02-05T07:33:09.860 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:33:09.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-02-05T07:33:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:33:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:33:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:33:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:33:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:33:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:32:59.059934+0000", 2024-02-05T07:33:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:33:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-02-05T07:33:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:33:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:32:56.508117+0000", 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:32:34.744999+0000", 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:33:09.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.863 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-02-05T07:33:09.863 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:33:09.863 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:33:09.863 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:33:09.863 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:33:09.863 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:33:09.863 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:32:00.252211+0000", 2024-02-05T07:33:09.863 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 230 2024-02-05T07:33:09.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.863 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-02-05T07:33:09.863 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:33:09.864 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:33:09.864 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:33:09.864 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:33:09.864 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-02-05T07:33:09.864 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:32:38.602805+0000", 2024-02-05T07:33:09.864 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:33:09.864 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.864 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:33:09.864 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap": [ 2024-02-05T07:33:09.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.864 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "5.15", 2024-02-05T07:33:09.864 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:33:09.864 INFO:teuthology.orchestra.run.smithi007.stdout: 2, 2024-02-05T07:33:09.865 INFO:teuthology.orchestra.run.smithi007.stdout: 0, 2024-02-05T07:33:09.865 INFO:teuthology.orchestra.run.smithi007.stdout: 1 2024-02-05T07:33:09.865 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.865 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "6.0", 2024-02-05T07:33:09.865 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:33:09.865 INFO:teuthology.orchestra.run.smithi007.stdout: 1, 2024-02-05T07:33:09.865 INFO:teuthology.orchestra.run.smithi007.stdout: 0, 2024-02-05T07:33:09.865 INFO:teuthology.orchestra.run.smithi007.stdout: 6 2024-02-05T07:33:09.865 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.865 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.865 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:33:09.865 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_items": [ 2024-02-05T07:33:09.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.866 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.3", 2024-02-05T07:33:09.866 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:33:09.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.866 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 4, 2024-02-05T07:33:09.866 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:33:09.866 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.866 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.866 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.1a", 2024-02-05T07:33:09.866 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:33:09.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.867 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 4, 2024-02-05T07:33:09.867 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 0 2024-02-05T07:33:09.867 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.867 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.867 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.4", 2024-02-05T07:33:09.867 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:33:09.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.867 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:33:09.867 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 0 2024-02-05T07:33:09.867 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.867 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.868 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.9", 2024-02-05T07:33:09.868 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:33:09.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.868 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:33:09.868 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:33:09.868 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.868 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.868 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.a", 2024-02-05T07:33:09.868 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:33:09.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.869 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:33:09.869 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:33:09.869 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.869 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.869 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.13", 2024-02-05T07:33:09.869 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:33:09.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:33:09.869 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:33:09.869 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:33:09.869 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.869 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:33:09.870 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.870 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:33:09.870 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_primaries": [], 2024-02-05T07:33:09.870 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_temp": [], 2024-02-05T07:33:09.870 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_temp": [], 2024-02-05T07:33:09.870 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklist": { 2024-02-05T07:33:09.870 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2180087772": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:33:09.870 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2066685086": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:33:09.870 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/4100020626": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:33:09.870 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1953351182": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:33:09.870 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3203189162": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:33:09.870 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3418781614": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:33:09.870 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1518592084": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:33:09.871 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/3380493069": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:33:09.871 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2208590106": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:33:09.871 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/223579766": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:33:09.871 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/2634895574": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:33:09.871 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1462812719": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:33:09.871 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2823976078": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:33:09.871 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1905520582": "2024-02-06T07:13:47.004765+0000" 2024-02-05T07:33:09.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.871 INFO:teuthology.orchestra.run.smithi007.stdout: "range_blocklist": {}, 2024-02-05T07:33:09.871 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profiles": { 2024-02-05T07:33:09.871 INFO:teuthology.orchestra.run.smithi007.stdout: "default": { 2024-02-05T07:33:09.871 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-failure-domain": "osd", 2024-02-05T07:33:09.871 INFO:teuthology.orchestra.run.smithi007.stdout: "k": "2", 2024-02-05T07:33:09.872 INFO:teuthology.orchestra.run.smithi007.stdout: "m": "1", 2024-02-05T07:33:09.872 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin": "jerasure", 2024-02-05T07:33:09.872 INFO:teuthology.orchestra.run.smithi007.stdout: "technique": "reed_sol_van" 2024-02-05T07:33:09.872 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:33:09.872 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps_queue": [], 2024-02-05T07:33:09.872 INFO:teuthology.orchestra.run.smithi007.stdout: "new_removed_snaps": [], 2024-02-05T07:33:09.872 INFO:teuthology.orchestra.run.smithi007.stdout: "new_purged_snaps": [], 2024-02-05T07:33:09.872 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_node_flags": {}, 2024-02-05T07:33:09.872 INFO:teuthology.orchestra.run.smithi007.stdout: "device_class_flags": {}, 2024-02-05T07:33:09.872 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode": { 2024-02-05T07:33:09.872 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_enabled": false, 2024-02-05T07:33:09.872 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_bucket_count": 0, 2024-02-05T07:33:09.873 INFO:teuthology.orchestra.run.smithi007.stdout: "degraded_stretch_mode": 0, 2024-02-05T07:33:09.873 INFO:teuthology.orchestra.run.smithi007.stdout: "recovering_stretch_mode": 0, 2024-02-05T07:33:09.873 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_bucket": 0 2024-02-05T07:33:09.873 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:33:09.873 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:33:09.879 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:09.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:09.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:10.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:10.006 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:10.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:10.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:10.132 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:10.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:10.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:10.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:10.272 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:10.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:10.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:10.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:10.416 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:10.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:10.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:10.542 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:10.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:10.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:10.560 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Removed 79842 objects 2024-02-05T07:33:10.560 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Clean up completed and total clean up time :83.6026 2024-02-05T07:33:10.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:10.667 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:10.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:10.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:10.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:10 smithi090 ceph-mon[33213]: pgmap v743: 178 pgs: 178 active+clean; 279 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T07:33:10.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:10 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:33:10.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:10 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:33:10.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:10 smithi090 ceph-mon[33213]: osdmap e416: 8 total, 8 up, 6 in 2024-02-05T07:33:10.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:10 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3320362130' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:33:10.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:10.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:10.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:10.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:10 smithi007 ceph-mon[29098]: pgmap v743: 178 pgs: 178 active+clean; 279 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T07:33:10.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:10 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:33:10.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:10 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:33:10.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:10 smithi007 ceph-mon[29098]: osdmap e416: 8 total, 8 up, 6 in 2024-02-05T07:33:10.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:10 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3320362130' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:33:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:10 smithi007 ceph-mon[37158]: pgmap v743: 178 pgs: 178 active+clean; 279 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T07:33:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:10 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:33:10.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:10 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:33:10.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:10 smithi007 ceph-mon[37158]: osdmap e416: 8 total, 8 up, 6 in 2024-02-05T07:33:10.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:10 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3320362130' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:33:10.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:10.918 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:10.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:10.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:11.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:11.043 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:11.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:11.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:11.115 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:11.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:11.173 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:11.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:11.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:33:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:33:11.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:11.299 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:11.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:11.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:11.401 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:11.425 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:11.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:11.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:11.527 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:11.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:11.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:11.597 INFO:tasks.thrashosds.thrasher:j is {'epoch': 416, 'fsid': 'e3e7059e-c3f5-11ee-95b6-87774f69a715', 'created': '2024-02-05T07:13:14.603885+0000', 'modified': '2024-02-05T07:33:09.513614+0000', 'last_up_change': '2024-02-05T07:28:24.555976+0000', 'last_in_change': '2024-02-05T07:30:39.451715+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T07:16:32.505567+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '385', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '383', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5700000524520874, 'score_stable': 1.5700000524520874, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.350000023841858, 'raw_score_stable': 1.350000023841858, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T07:18:47.259462+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.690000057220459, 'score_stable': 4.690000057220459, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 3.3299999237060547, 'raw_score_stable': 3.3299999237060547, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T07:18:47.762336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4000000953674316, 'score_stable': 2.4000000953674316, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T07:18:49.736599+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T07:18:51.696948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T07:18:53.768555+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-02-05T07:29:59.438957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '354', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.059999942779541, 'score_stable': 3.059999942779541, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.8420352935791016, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}], 'osds': [{'osd': 0, 'uuid': '703c3948-8003-4015-9dd8-21019cf0eeb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 387, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6801', 'nonce': 2069016689}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6802', 'nonce': 2069016689}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6804', 'nonce': 2069016689}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6803', 'nonce': 2069016689}]}, 'public_addr': '172.21.15.7:6801/2069016689', 'cluster_addr': '172.21.15.7:6802/2069016689', 'heartbeat_back_addr': '172.21.15.7:6804/2069016689', 'heartbeat_front_addr': '172.21.15.7:6803/2069016689', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '5c3e7d1e-a61d-44b9-b21e-f91c99a096bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 412, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6805', 'nonce': 865486532}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6806', 'nonce': 865486532}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6808', 'nonce': 865486532}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6807', 'nonce': 865486532}]}, 'public_addr': '172.21.15.7:6805/865486532', 'cluster_addr': '172.21.15.7:6806/865486532', 'heartbeat_back_addr': '172.21.15.7:6808/865486532', 'heartbeat_front_addr': '172.21.15.7:6807/865486532', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4fd930f-7bb2-46ca-9964-98235bd47b3a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 395, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6809', 'nonce': 904871370}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6810', 'nonce': 904871370}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6812', 'nonce': 904871370}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6811', 'nonce': 904871370}]}, 'public_addr': '172.21.15.7:6809/904871370', 'cluster_addr': '172.21.15.7:6810/904871370', 'heartbeat_back_addr': '172.21.15.7:6812/904871370', 'heartbeat_front_addr': '172.21.15.7:6811/904871370', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '149ed0d6-b049-4478-9819-cafc446e94d3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 405, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6813', 'nonce': 2758586242}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6814', 'nonce': 2758586242}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6816', 'nonce': 2758586242}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6815', 'nonce': 2758586242}]}, 'public_addr': '172.21.15.7:6813/2758586242', 'cluster_addr': '172.21.15.7:6814/2758586242', 'heartbeat_back_addr': '172.21.15.7:6816/2758586242', 'heartbeat_front_addr': '172.21.15.7:6815/2758586242', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 395, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 1154929470}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6801', 'nonce': 1154929470}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6803', 'nonce': 1154929470}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 1154929470}]}, 'public_addr': '172.21.15.90:6800/1154929470', 'cluster_addr': '172.21.15.90:6801/1154929470', 'heartbeat_back_addr': '172.21.15.90:6803/1154929470', 'heartbeat_front_addr': '172.21.15.90:6802/1154929470', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '244a7930-c22c-413e-86e1-11e0749f7367', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.7362823486328125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 336, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 398113577}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6805', 'nonce': 398113577}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6807', 'nonce': 398113577}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 398113577}]}, 'public_addr': '172.21.15.90:6804/398113577', 'cluster_addr': '172.21.15.90:6805/398113577', 'heartbeat_back_addr': '172.21.15.90:6807/398113577', 'heartbeat_front_addr': '172.21.15.90:6806/398113577', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0adbb562-3012-4f60-b5ec-02550a10b2c7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 52, 'last_clean_end': 230, 'up_from': 301, 'up_thru': 388, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 313196533}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 313196533}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6811', 'nonce': 313196533}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6810', 'nonce': 313196533}]}, 'public_addr': '172.21.15.90:6808/313196533', 'cluster_addr': '172.21.15.90:6809/313196533', 'heartbeat_back_addr': '172.21.15.90:6811/313196533', 'heartbeat_front_addr': '172.21.15.90:6810/313196533', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dbc26c36-a0e9-4f21-9e98-739270cea6d9', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 349, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 2983605980}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 2983605980}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6815', 'nonce': 2983605980}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6814', 'nonce': 2983605980}]}, 'public_addr': '172.21.15.90:6812/2983605980', 'cluster_addr': '172.21.15.90:6813/2983605980', 'heartbeat_back_addr': '172.21.15.90:6815/2983605980', 'heartbeat_front_addr': '172.21.15.90:6814/2983605980', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:33:09.343196+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:32:14.901479+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:32:47.527890+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:32:59.059934+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:32:56.508117+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T07:32:34.744999+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:32:00.252211+0000', 'dead_epoch': 230}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T07:32:38.602805+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.15', 'osds': [2, 0, 1]}, {'pgid': '6.0', 'osds': [1, 0, 6]}], 'pg_upmap_items': [{'pgid': '3.3', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '3.1a', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.7:0/2180087772': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/2066685086': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/4100020626': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/1953351182': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/3203189162': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/3418781614': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:6800/1518592084': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:6800/3380493069': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/2208590106': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:0/223579766': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:6800/2634895574': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/1462812719': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/2823976078': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/1905520582': '2024-02-06T07:13:47.004765+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T07:33:11.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:33:11.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:11.680 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:11.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:11.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:11.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:11.805 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:11.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:11.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:11.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:11.934 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:11.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:11.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:12.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:12.064 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:12.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:12.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:12.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:12.189 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:12.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:12.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:12.315 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:12.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:12.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:12.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:12.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:12.442 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:12.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:12.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:12.572 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:12.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:12.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:12.587 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-02-05T07:33:12.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-02-05T07:33:12.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:12.700 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:12.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:12.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:12.736 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:12 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:12.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:12 smithi090 ceph-mon[33213]: pgmap v745: 178 pgs: 178 active+clean; 233 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 891 op/s 2024-02-05T07:33:12.781 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:33:12.781 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:12 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:12.781 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:12 smithi007 ceph-mon[29098]: pgmap v745: 178 pgs: 178 active+clean; 233 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 891 op/s 2024-02-05T07:33:12.781 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:12 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:12.781 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:12 smithi007 ceph-mon[37158]: pgmap v745: 178 pgs: 178 active+clean; 233 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 891 op/s 2024-02-05T07:33:12.801 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:33:12.802 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:12.820 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:33:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:12.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:12.840 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:33:12.861 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:33:12.882 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:33:12.902 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:33:12.922 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:33:12.923 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:33:12.923 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:33:12.923 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:33:12.924 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:33:12.924 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:33:12.924 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:33:12.924 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:33:12.924 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:33:12.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:12.955 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:12.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:13.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:13.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:13.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:13.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:13.210 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:13.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:13.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:13.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:13.338 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:13.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:13.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:13.468 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:13.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:13.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:13.597 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:13.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:13.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:13.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:13.725 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:13.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:13.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:13.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:13.852 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:13.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:13.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:13.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:13 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1655472914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:13.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:33:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:33:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:33:13.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:13 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1655472914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:13.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:13.978 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:13.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:13.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:13 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1655472914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:14.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:14.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:14.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:14.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:14.234 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:14.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:14.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:14.361 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:14.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:14.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:14.486 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:14.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:14.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:14.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:14.612 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:14.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:14.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:14.649 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:14 smithi007 ceph-mon[29098]: pgmap v746: 178 pgs: 178 active+clean; 11 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-02-05T07:33:14.649 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:14 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:33:14.649 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:14 smithi007 ceph-mon[37158]: pgmap v746: 178 pgs: 178 active+clean; 11 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-02-05T07:33:14.649 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:14 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:33:14.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:14.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:14.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:14.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:14.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:14.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:14.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:14.991 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:14.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:14.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:15.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:14 smithi090 ceph-mon[33213]: pgmap v746: 178 pgs: 178 active+clean; 11 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-02-05T07:33:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:14 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:33:15.067 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:33:15.067 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:33:15.067 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 0 2024-02-05T07:33:15.067 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:33:15.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:15.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:15.118 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:15.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:15.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:15.242 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:15.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:15.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:15.267 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:15.344 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:15.369 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:15.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:15.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:15.495 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:15.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:15.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:15.621 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:15.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:15.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:15.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:33:15.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:33:15.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:15 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:33:15.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:33:15.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:33:15.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:15 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:33:15.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:15.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T07:33:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:15.748 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:15.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:15.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:15.877 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:15.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:15.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:15.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:16.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:16.007 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:16.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:33:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:33:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:15 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:33:16.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:16.133 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:16.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:16.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:16.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:16.262 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:16.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:16.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:16.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:16.388 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:16.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:16.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:16.518 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:16.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:16.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:16.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:16.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:16.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:16.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:16.776 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:16.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:16.809 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:16.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:16.901 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:16.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:16.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:16 smithi007 ceph-mon[29098]: pgmap v747: 178 pgs: 178 active+clean; 1.6 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T07:33:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:16 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:33:16.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:16 smithi007 ceph-mon[37158]: pgmap v747: 178 pgs: 178 active+clean; 1.6 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T07:33:16.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:16 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:33:17.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:17.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:16 smithi090 ceph-mon[33213]: pgmap v747: 178 pgs: 178 active+clean; 1.6 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T07:33:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:16 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:33:17.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:17.029 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:17.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:17.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:17.158 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:17.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:17.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:17.284 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:17.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:17.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:17.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:17.413 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:17.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:17.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:17.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:17.542 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:17.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:17.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:17.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:17.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:17.670 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:17.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:17.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:17.799 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:17.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:17.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:17.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:17.911 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:17.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:17 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:33:17.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:17 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:33:17.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:17 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:33:17.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:17 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:33:17.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:17.929 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:17.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:17 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:33:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:17 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:33:18.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:18.053 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:18.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:18.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:18.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:33:18.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:18.211 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:18.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:18.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:18.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:18.338 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:18.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:18.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:18.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:18.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:18.465 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:18.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:18.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:18.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:18.595 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:18.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:18.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:18 smithi007 ceph-mon[29098]: pgmap v748: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 484 op/s 2024-02-05T07:33:18.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:18 smithi007 ceph-mon[29098]: osdmap e417: 8 total, 8 up, 6 in 2024-02-05T07:33:18.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:18 smithi007 ceph-mon[37158]: pgmap v748: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 484 op/s 2024-02-05T07:33:18.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:18 smithi007 ceph-mon[37158]: osdmap e417: 8 total, 8 up, 6 in 2024-02-05T07:33:18.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:18.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:18.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:18.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:18.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:18.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:18.981 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:18.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:18.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:18 smithi090 ceph-mon[33213]: pgmap v748: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 484 op/s 2024-02-05T07:33:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:18 smithi090 ceph-mon[33213]: osdmap e417: 8 total, 8 up, 6 in 2024-02-05T07:33:19.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:19.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:19.112 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:19.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:19.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:19.239 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:19.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:19.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:19.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:19.369 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:19.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:19.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:19.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:19.498 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:19.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:19.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:19.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:19.626 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:19.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:19.728 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:19.754 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:19.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:19.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:19.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:20.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:20.031 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:20.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:20.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:20.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:20.157 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:20.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:20.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:20.287 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:20.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:20.389 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:20.415 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:20.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:20.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:20.547 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:20.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:20.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:20.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:20.674 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:20.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:20.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:20.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:20 smithi090 ceph-mon[33213]: pgmap v750: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 459 op/s 2024-02-05T07:33:20.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:20 smithi090 ceph-mon[33213]: osdmap e418: 8 total, 8 up, 6 in 2024-02-05T07:33:20.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:20 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:33:20.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:20 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:33:20.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:20.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:20.801 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:20.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:20.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:20 smithi007 ceph-mon[29098]: pgmap v750: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 459 op/s 2024-02-05T07:33:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:20 smithi007 ceph-mon[29098]: osdmap e418: 8 total, 8 up, 6 in 2024-02-05T07:33:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:20 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:33:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:20 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:33:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:20 smithi007 ceph-mon[37158]: pgmap v750: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 459 op/s 2024-02-05T07:33:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:20 smithi007 ceph-mon[37158]: osdmap e418: 8 total, 8 up, 6 in 2024-02-05T07:33:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:20 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:33:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:20 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:33:20.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:20.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:20.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:21.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:21.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:21.060 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:21.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:21.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:21.190 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:21.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:21.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:21.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:33:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:33:21.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:21.315 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:21.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:21.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:21.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:21.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:21.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:21.550 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:33:21.571 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:21.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:33:21.601 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:33:21.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:21.702 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:21.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:21.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:21.832 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:21.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:21.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:21 smithi007 ceph-mon[29098]: osdmap e419: 8 total, 8 up, 6 in 2024-02-05T07:33:21.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:21 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2420455963' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:33:21.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:21 smithi007 ceph-mon[37158]: osdmap e419: 8 total, 8 up, 6 in 2024-02-05T07:33:21.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:21 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2420455963' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:33:21.934 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:21.959 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:21.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:22.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:21 smithi090 ceph-mon[33213]: osdmap e419: 8 total, 8 up, 6 in 2024-02-05T07:33:22.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:21 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2420455963' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:33:22.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:22.087 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:22.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:22.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:22.216 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:22.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:22.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:22.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:22.346 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:22.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:22.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:22.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:22.473 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:22.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:22.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:22.598 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:22.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:22.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:22.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:22.723 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:22.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:22.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:22.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:22 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:22.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:22 smithi007 ceph-mon[29098]: pgmap v753: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 3 op/s 2024-02-05T07:33:22.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:22 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:33:22.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:22 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2420455963' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:33:22.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:22 smithi007 ceph-mon[29098]: osdmap e420: 8 total, 8 up, 6 in 2024-02-05T07:33:22.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:22 smithi007 ceph-mon[29098]: from='client.25397 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:33:22.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:22 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:22.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:22 smithi007 ceph-mon[37158]: pgmap v753: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 3 op/s 2024-02-05T07:33:22.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:22 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:33:22.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:22 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2420455963' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:33:22.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:22 smithi007 ceph-mon[37158]: osdmap e420: 8 total, 8 up, 6 in 2024-02-05T07:33:22.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:22 smithi007 ceph-mon[37158]: from='client.25397 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:33:22.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:22.852 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:22.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:22.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:22.977 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:22.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:22.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:22 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:22 smithi090 ceph-mon[33213]: pgmap v753: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 3 op/s 2024-02-05T07:33:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:22 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:33:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:22 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2420455963' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:33:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:22 smithi090 ceph-mon[33213]: osdmap e420: 8 total, 8 up, 6 in 2024-02-05T07:33:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:22 smithi090 ceph-mon[33213]: from='client.25397 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:33:23.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:23.103 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:23.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:23.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:23.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:23.232 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:23.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:23.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:23.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:23.362 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:23.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:23.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:23.383 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":754,"stamp":"2024-02-05T07:33:21.542185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1277,"num_read_kb":1029,"num_write":160408,"num_write_kb":5109934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27555,"num_bytes_recovered":1761811860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56820,"ondisk_log_size":56820,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":3411936,"kb_used_data":10036,"kb_used_omap":13,"kb_used_meta":3400818,"kb_avail":559058976,"statfs":{"total":575970213888,"available":572476391424,"internally_reserved":0,"allocated":10276864,"data_stored":6399756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":3482438368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.291206"},"pg_stats":[{"pgid":"9.1","version":"416'9954","reported_seq":15466,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:10.532970+0000","last_change":"2024-02-05T07:32:19.517221+0000","last_active":"2024-02-05T07:33:10.532970+0000","last_peered":"2024-02-05T07:33:10.532970+0000","last_clean":"2024-02-05T07:33:10.532970+0000","last_became_active":"2024-02-05T07:32:19.505118+0000","last_became_peered":"2024-02-05T07:32:19.505118+0000","last_unstale":"2024-02-05T07:33:10.532970+0000","last_undegraded":"2024-02-05T07:33:10.532970+0000","last_fullsized":"2024-02-05T07:33:10.532970+0000","mapping_epoch":399,"log_start":"400'6729","ondisk_log_start":"400'6729","created":349,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3225,"log_dups_size":0,"ondisk_log_size":3225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9954,"num_write_kb":318528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2678,"num_bytes_recovered":175505408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":841,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.540070+0000","last_change":"2024-02-05T07:32:42.818086+0000","last_active":"2024-02-05T07:33:09.540070+0000","last_peered":"2024-02-05T07:33:09.540070+0000","last_clean":"2024-02-05T07:33:09.540070+0000","last_became_active":"2024-02-05T07:31:36.857092+0000","last_became_peered":"2024-02-05T07:31:36.857092+0000","last_unstale":"2024-02-05T07:33:09.540070+0000","last_undegraded":"2024-02-05T07:33:09.540070+0000","last_fullsized":"2024-02-05T07:33:09.540070+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:42.817975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:42.817975+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:59.541198+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"326'25","reported_seq":822,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522072+0000","last_change":"2024-02-05T07:32:27.643863+0000","last_active":"2024-02-05T07:33:19.522072+0000","last_peered":"2024-02-05T07:33:19.522072+0000","last_clean":"2024-02-05T07:33:19.522072+0000","last_became_active":"2024-02-05T07:28:26.605082+0000","last_became_peered":"2024-02-05T07:28:26.605082+0000","last_unstale":"2024-02-05T07:33:19.522072+0000","last_undegraded":"2024-02-05T07:33:19.522072+0000","last_fullsized":"2024-02-05T07:33:19.522072+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:32:27.643788+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:32:27.643788+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:47.371484+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":317,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.534952+0000","last_change":"2024-02-05T07:27:16.716749+0000","last_active":"2024-02-05T07:33:09.534952+0000","last_peered":"2024-02-05T07:33:09.534952+0000","last_clean":"2024-02-05T07:33:09.534952+0000","last_became_active":"2024-02-05T07:27:16.716461+0000","last_became_peered":"2024-02-05T07:27:16.716461+0000","last_unstale":"2024-02-05T07:33:09.534952+0000","last_undegraded":"2024-02-05T07:33:09.534952+0000","last_fullsized":"2024-02-05T07:33:09.534952+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:18.258469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:18.258469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"365'11","reported_seq":957,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.539629+0000","last_change":"2024-02-05T07:30:40.506050+0000","last_active":"2024-02-05T07:33:20.539629+0000","last_peered":"2024-02-05T07:33:20.539629+0000","last_clean":"2024-02-05T07:33:20.539629+0000","last_became_active":"2024-02-05T07:30:40.484389+0000","last_became_peered":"2024-02-05T07:30:40.484389+0000","last_unstale":"2024-02-05T07:33:20.539629+0000","last_undegraded":"2024-02-05T07:33:20.539629+0000","last_fullsized":"2024-02-05T07:33:20.539629+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'10","last_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:30:38.946065+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":735,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522754+0000","last_change":"2024-02-05T07:32:30.708289+0000","last_active":"2024-02-05T07:33:19.522754+0000","last_peered":"2024-02-05T07:33:19.522754+0000","last_clean":"2024-02-05T07:33:19.522754+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:33:19.522754+0000","last_undegraded":"2024-02-05T07:33:19.522754+0000","last_fullsized":"2024-02-05T07:33:19.522754+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:30.708205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:32:30.708205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:49.185866+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"416'9916","reported_seq":10061,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:10.558545+0000","last_change":"2024-02-05T07:30:00.506880+0000","last_active":"2024-02-05T07:33:10.558545+0000","last_peered":"2024-02-05T07:33:10.558545+0000","last_clean":"2024-02-05T07:33:10.558545+0000","last_became_active":"2024-02-05T07:30:00.506586+0000","last_became_peered":"2024-02-05T07:30:00.506586+0000","last_unstale":"2024-02-05T07:33:10.558545+0000","last_undegraded":"2024-02-05T07:33:10.558545+0000","last_fullsized":"2024-02-05T07:33:10.558545+0000","mapping_epoch":349,"log_start":"400'6879","ondisk_log_start":"400'6879","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3037,"log_dups_size":0,"ondisk_log_size":3037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.384121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9916,"num_write_kb":317312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":766,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.539010+0000","last_change":"2024-02-05T07:32:41.228497+0000","last_active":"2024-02-05T07:33:20.539010+0000","last_peered":"2024-02-05T07:33:20.539010+0000","last_clean":"2024-02-05T07:33:20.539010+0000","last_became_active":"2024-02-05T07:31:36.856842+0000","last_became_peered":"2024-02-05T07:31:36.856842+0000","last_unstale":"2024-02-05T07:33:20.539010+0000","last_undegraded":"2024-02-05T07:33:20.539010+0000","last_fullsized":"2024-02-05T07:33:20.539010+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:41.228420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:41.228420+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:54.957938+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"326'31","reported_seq":793,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522654+0000","last_change":"2024-02-05T07:26:36.275060+0000","last_active":"2024-02-05T07:33:19.522654+0000","last_peered":"2024-02-05T07:33:19.522654+0000","last_clean":"2024-02-05T07:33:19.522654+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:33:19.522654+0000","last_undegraded":"2024-02-05T07:33:19.522654+0000","last_fullsized":"2024-02-05T07:33:19.522654+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-02-05T07:26:36.274939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.274939+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":591,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065293+0000","last_change":"2024-02-05T07:30:30.212483+0000","last_active":"2024-02-05T07:33:18.065293+0000","last_peered":"2024-02-05T07:33:18.065293+0000","last_clean":"2024-02-05T07:33:18.065293+0000","last_became_active":"2024-02-05T07:28:25.791005+0000","last_became_peered":"2024-02-05T07:28:25.791005+0000","last_unstale":"2024-02-05T07:33:18.065293+0000","last_undegraded":"2024-02-05T07:33:18.065293+0000","last_fullsized":"2024-02-05T07:33:18.065293+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:30.212401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.212401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:20.071024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":737,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.526829+0000","last_change":"2024-02-05T07:29:49.674581+0000","last_active":"2024-02-05T07:33:09.526829+0000","last_peered":"2024-02-05T07:33:09.526829+0000","last_clean":"2024-02-05T07:33:09.526829+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:33:09.526829+0000","last_undegraded":"2024-02-05T07:33:09.526829+0000","last_fullsized":"2024-02-05T07:33:09.526829+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.674493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.674493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":753,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.542380+0000","last_change":"2024-02-05T07:30:35.156491+0000","last_active":"2024-02-05T07:33:20.542380+0000","last_peered":"2024-02-05T07:33:20.542380+0000","last_clean":"2024-02-05T07:33:20.542380+0000","last_became_active":"2024-02-05T07:27:16.708888+0000","last_became_peered":"2024-02-05T07:27:16.708888+0000","last_unstale":"2024-02-05T07:33:20.542380+0000","last_undegraded":"2024-02-05T07:33:20.542380+0000","last_fullsized":"2024-02-05T07:33:20.542380+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:35.156384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:30:35.156384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"416'9934","reported_seq":15626,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:10.500011+0000","last_change":"2024-02-05T07:31:31.709575+0000","last_active":"2024-02-05T07:33:10.500011+0000","last_peered":"2024-02-05T07:33:10.500011+0000","last_clean":"2024-02-05T07:33:10.500011+0000","last_became_active":"2024-02-05T07:31:31.703194+0000","last_became_peered":"2024-02-05T07:31:31.703194+0000","last_unstale":"2024-02-05T07:33:10.500011+0000","last_undegraded":"2024-02-05T07:33:10.500011+0000","last_fullsized":"2024-02-05T07:33:10.500011+0000","mapping_epoch":379,"log_start":"400'6680","ondisk_log_start":"400'6680","created":349,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3254,"log_dups_size":0,"ondisk_log_size":3254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9934,"num_write_kb":317888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2768,"num_bytes_recovered":181403648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":731,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.525600+0000","last_change":"2024-02-05T07:32:41.303464+0000","last_active":"2024-02-05T07:33:09.525600+0000","last_peered":"2024-02-05T07:33:09.525600+0000","last_clean":"2024-02-05T07:33:09.525600+0000","last_became_active":"2024-02-05T07:31:36.841690+0000","last_became_peered":"2024-02-05T07:31:36.841690+0000","last_unstale":"2024-02-05T07:33:09.525600+0000","last_undegraded":"2024-02-05T07:33:09.525600+0000","last_fullsized":"2024-02-05T07:33:09.525600+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:41.303385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:41.303385+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:48.002757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"326'24","reported_seq":788,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522475+0000","last_change":"2024-02-05T07:29:50.053697+0000","last_active":"2024-02-05T07:33:19.522475+0000","last_peered":"2024-02-05T07:33:19.522475+0000","last_clean":"2024-02-05T07:33:19.522475+0000","last_became_active":"2024-02-05T07:28:25.781007+0000","last_became_peered":"2024-02-05T07:28:25.781007+0000","last_unstale":"2024-02-05T07:33:19.522475+0000","last_undegraded":"2024-02-05T07:33:19.522475+0000","last_fullsized":"2024-02-05T07:33:19.522475+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:29:50.053583+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.053583+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":754,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521114+0000","last_change":"2024-02-05T07:32:28.670360+0000","last_active":"2024-02-05T07:33:19.521114+0000","last_peered":"2024-02-05T07:33:19.521114+0000","last_clean":"2024-02-05T07:33:19.521114+0000","last_became_active":"2024-02-05T07:30:40.486134+0000","last_became_peered":"2024-02-05T07:30:40.486134+0000","last_unstale":"2024-02-05T07:33:19.521114+0000","last_undegraded":"2024-02-05T07:33:19.521114+0000","last_fullsized":"2024-02-05T07:33:19.521114+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:28.670286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:32:28.670286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:32.433468+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":575,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522537+0000","last_change":"2024-02-05T07:29:45.012958+0000","last_active":"2024-02-05T07:33:19.522537+0000","last_peered":"2024-02-05T07:33:19.522537+0000","last_clean":"2024-02-05T07:33:19.522537+0000","last_became_active":"2024-02-05T07:28:26.601713+0000","last_became_peered":"2024-02-05T07:28:26.601713+0000","last_unstale":"2024-02-05T07:33:19.522537+0000","last_undegraded":"2024-02-05T07:33:19.522537+0000","last_fullsized":"2024-02-05T07:33:19.522537+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.012821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":326,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.539621+0000","last_change":"2024-02-05T07:30:40.499008+0000","last_active":"2024-02-05T07:33:09.539621+0000","last_peered":"2024-02-05T07:33:09.539621+0000","last_clean":"2024-02-05T07:33:09.539621+0000","last_became_active":"2024-02-05T07:30:40.498529+0000","last_became_peered":"2024-02-05T07:30:40.498529+0000","last_unstale":"2024-02-05T07:33:09.539621+0000","last_undegraded":"2024-02-05T07:33:09.539621+0000","last_fullsized":"2024-02-05T07:33:09.539621+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:47.885294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:47.885294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"416'9910","reported_seq":10055,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:10.533201+0000","last_change":"2024-02-05T07:30:00.512144+0000","last_active":"2024-02-05T07:33:10.533201+0000","last_peered":"2024-02-05T07:33:10.533201+0000","last_clean":"2024-02-05T07:33:10.533201+0000","last_became_active":"2024-02-05T07:30:00.511808+0000","last_became_peered":"2024-02-05T07:30:00.511808+0000","last_unstale":"2024-02-05T07:33:10.533201+0000","last_undegraded":"2024-02-05T07:33:10.533201+0000","last_fullsized":"2024-02-05T07:33:10.533201+0000","mapping_epoch":349,"log_start":"400'6683","ondisk_log_start":"400'6683","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9910,"num_write_kb":317120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":741,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.541016+0000","last_change":"2024-02-05T07:32:40.217065+0000","last_active":"2024-02-05T07:33:20.541016+0000","last_peered":"2024-02-05T07:33:20.541016+0000","last_clean":"2024-02-05T07:33:20.541016+0000","last_became_active":"2024-02-05T07:31:36.839888+0000","last_became_peered":"2024-02-05T07:31:36.839888+0000","last_unstale":"2024-02-05T07:33:20.541016+0000","last_undegraded":"2024-02-05T07:33:20.541016+0000","last_fullsized":"2024-02-05T07:33:20.541016+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:40.216973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:40.216973+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:56.914996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"326'34","reported_seq":835,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.536666+0000","last_change":"2024-02-05T07:30:30.006087+0000","last_active":"2024-02-05T07:33:09.536666+0000","last_peered":"2024-02-05T07:33:09.536666+0000","last_clean":"2024-02-05T07:33:09.536666+0000","last_became_active":"2024-02-05T07:27:16.768454+0000","last_became_peered":"2024-02-05T07:27:16.768454+0000","last_unstale":"2024-02-05T07:33:09.536666+0000","last_undegraded":"2024-02-05T07:33:09.536666+0000","last_fullsized":"2024-02-05T07:33:09.536666+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'34","last_scrub_stamp":"2024-02-05T07:30:30.005988+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.005988+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.384211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":331,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.529972+0000","last_change":"2024-02-05T07:30:40.508975+0000","last_active":"2024-02-05T07:33:09.529972+0000","last_peered":"2024-02-05T07:33:09.529972+0000","last_clean":"2024-02-05T07:33:09.529972+0000","last_became_active":"2024-02-05T07:30:40.508561+0000","last_became_peered":"2024-02-05T07:30:40.508561+0000","last_unstale":"2024-02-05T07:33:09.529972+0000","last_undegraded":"2024-02-05T07:33:09.529972+0000","last_fullsized":"2024-02-05T07:33:09.529972+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:30.982457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.982457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"257'9","reported_seq":932,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522758+0000","last_change":"2024-02-05T07:32:39.565923+0000","last_active":"2024-02-05T07:33:19.522758+0000","last_peered":"2024-02-05T07:33:19.522758+0000","last_clean":"2024-02-05T07:33:19.522758+0000","last_became_active":"2024-02-05T07:27:03.375195+0000","last_became_peered":"2024-02-05T07:27:03.375195+0000","last_unstale":"2024-02-05T07:33:19.522758+0000","last_undegraded":"2024-02-05T07:33:19.522758+0000","last_fullsized":"2024-02-05T07:33:19.522758+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'9","last_scrub_stamp":"2024-02-05T07:32:39.565833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:39.565833+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:09.327994+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":343,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.528237+0000","last_change":"2024-02-05T07:30:37.998622+0000","last_active":"2024-02-05T07:33:09.528237+0000","last_peered":"2024-02-05T07:33:09.528237+0000","last_clean":"2024-02-05T07:33:09.528237+0000","last_became_active":"2024-02-05T07:27:03.367753+0000","last_became_peered":"2024-02-05T07:27:03.367753+0000","last_unstale":"2024-02-05T07:33:09.528237+0000","last_undegraded":"2024-02-05T07:33:09.528237+0000","last_fullsized":"2024-02-05T07:33:09.528237+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:37.998383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:30:37.998383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.384529+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"416'10026","reported_seq":15832,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:10.557151+0000","last_change":"2024-02-05T07:32:18.658612+0000","last_active":"2024-02-05T07:33:10.557151+0000","last_peered":"2024-02-05T07:33:10.557151+0000","last_clean":"2024-02-05T07:33:10.557151+0000","last_became_active":"2024-02-05T07:32:18.645799+0000","last_became_peered":"2024-02-05T07:32:18.645799+0000","last_unstale":"2024-02-05T07:33:10.557151+0000","last_undegraded":"2024-02-05T07:33:10.557151+0000","last_fullsized":"2024-02-05T07:33:10.557151+0000","mapping_epoch":398,"log_start":"400'6933","ondisk_log_start":"400'6933","created":349,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3093,"log_dups_size":0,"ondisk_log_size":3093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10026,"num_write_kb":320769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2824,"num_bytes_recovered":185073664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":733,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522041+0000","last_change":"2024-02-05T07:32:24.656130+0000","last_active":"2024-02-05T07:33:19.522041+0000","last_peered":"2024-02-05T07:33:19.522041+0000","last_clean":"2024-02-05T07:33:19.522041+0000","last_became_active":"2024-02-05T07:31:36.758820+0000","last_became_peered":"2024-02-05T07:31:36.758820+0000","last_unstale":"2024-02-05T07:33:19.522041+0000","last_undegraded":"2024-02-05T07:33:19.522041+0000","last_fullsized":"2024-02-05T07:33:19.522041+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:24.656057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:24.656057+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:45.588976+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":758,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521088+0000","last_change":"2024-02-05T07:32:36.623960+0000","last_active":"2024-02-05T07:33:19.521088+0000","last_peered":"2024-02-05T07:33:19.521088+0000","last_clean":"2024-02-05T07:33:19.521088+0000","last_became_active":"2024-02-05T07:30:40.489212+0000","last_became_peered":"2024-02-05T07:30:40.489212+0000","last_unstale":"2024-02-05T07:33:19.521088+0000","last_undegraded":"2024-02-05T07:33:19.521088+0000","last_fullsized":"2024-02-05T07:33:19.521088+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:36.623865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:36.623865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.446069+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"326'30","reported_seq":385,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.529861+0000","last_change":"2024-02-05T07:30:40.543094+0000","last_active":"2024-02-05T07:33:09.529861+0000","last_peered":"2024-02-05T07:33:09.529861+0000","last_clean":"2024-02-05T07:33:09.529861+0000","last_became_active":"2024-02-05T07:30:40.496940+0000","last_became_peered":"2024-02-05T07:30:40.496940+0000","last_unstale":"2024-02-05T07:33:09.529861+0000","last_undegraded":"2024-02-05T07:33:09.529861+0000","last_fullsized":"2024-02-05T07:33:09.529861+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:28:45.029295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.029295+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":323,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.533941+0000","last_change":"2024-02-05T07:30:40.509232+0000","last_active":"2024-02-05T07:33:09.533941+0000","last_peered":"2024-02-05T07:33:09.533941+0000","last_clean":"2024-02-05T07:33:09.533941+0000","last_became_active":"2024-02-05T07:30:40.508607+0000","last_became_peered":"2024-02-05T07:30:40.508607+0000","last_unstale":"2024-02-05T07:33:09.533941+0000","last_undegraded":"2024-02-05T07:33:09.533941+0000","last_fullsized":"2024-02-05T07:33:09.533941+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:33.084016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:33.084016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":743,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521343+0000","last_change":"2024-02-05T07:28:56.783280+0000","last_active":"2024-02-05T07:33:19.521343+0000","last_peered":"2024-02-05T07:33:19.521343+0000","last_clean":"2024-02-05T07:33:19.521343+0000","last_became_active":"2024-02-05T07:27:16.706666+0000","last_became_peered":"2024-02-05T07:27:16.706666+0000","last_unstale":"2024-02-05T07:33:19.521343+0000","last_undegraded":"2024-02-05T07:33:19.521343+0000","last_fullsized":"2024-02-05T07:33:19.521343+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:28:56.783168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"416'9722","reported_seq":15196,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:10.557983+0000","last_change":"2024-02-05T07:31:36.855992+0000","last_active":"2024-02-05T07:33:10.557983+0000","last_peered":"2024-02-05T07:33:10.557983+0000","last_clean":"2024-02-05T07:33:10.557983+0000","last_became_active":"2024-02-05T07:31:36.841233+0000","last_became_peered":"2024-02-05T07:31:36.841233+0000","last_unstale":"2024-02-05T07:33:10.557983+0000","last_undegraded":"2024-02-05T07:33:10.557983+0000","last_fullsized":"2024-02-05T07:33:10.557983+0000","mapping_epoch":383,"log_start":"400'6535","ondisk_log_start":"400'6535","created":349,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3187,"log_dups_size":0,"ondisk_log_size":3187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":311104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2659,"num_bytes_recovered":174260224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":986,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.539868+0000","last_change":"2024-02-05T07:30:43.524967+0000","last_active":"2024-02-05T07:33:09.539868+0000","last_peered":"2024-02-05T07:33:09.539868+0000","last_clean":"2024-02-05T07:33:09.539868+0000","last_became_active":"2024-02-05T07:30:43.524588+0000","last_became_peered":"2024-02-05T07:30:43.524588+0000","last_unstale":"2024-02-05T07:33:09.539868+0000","last_undegraded":"2024-02-05T07:33:09.539868+0000","last_fullsized":"2024-02-05T07:33:09.539868+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:39.042281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:39.042281+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"266'10","reported_seq":393,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.642430+0000","last_change":"2024-02-05T07:27:17.017592+0000","last_active":"2024-02-05T07:33:09.642430+0000","last_peered":"2024-02-05T07:33:09.642430+0000","last_clean":"2024-02-05T07:33:09.642430+0000","last_became_active":"2024-02-05T07:27:16.800207+0000","last_became_peered":"2024-02-05T07:27:16.800207+0000","last_unstale":"2024-02-05T07:33:09.642430+0000","last_undegraded":"2024-02-05T07:33:09.642430+0000","last_fullsized":"2024-02-05T07:33:09.642430+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:26:34.535447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:34.535447+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:15.746246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"326'35","reported_seq":814,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521188+0000","last_change":"2024-02-05T07:29:45.972745+0000","last_active":"2024-02-05T07:33:19.521188+0000","last_peered":"2024-02-05T07:33:19.521188+0000","last_clean":"2024-02-05T07:33:19.521188+0000","last_became_active":"2024-02-05T07:28:26.598831+0000","last_became_peered":"2024-02-05T07:28:26.598831+0000","last_unstale":"2024-02-05T07:33:19.521188+0000","last_undegraded":"2024-02-05T07:33:19.521188+0000","last_fullsized":"2024-02-05T07:33:19.521188+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:29:45.972589+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.972589+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":498,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.531794+0000","last_change":"2024-02-05T07:29:41.587097+0000","last_active":"2024-02-05T07:33:09.531794+0000","last_peered":"2024-02-05T07:33:09.531794+0000","last_clean":"2024-02-05T07:33:09.531794+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:33:09.531794+0000","last_undegraded":"2024-02-05T07:33:09.531794+0000","last_fullsized":"2024-02-05T07:33:09.531794+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:41.587022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:41.587022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":611,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065249+0000","last_change":"2024-02-05T07:32:33.839588+0000","last_active":"2024-02-05T07:33:18.065249+0000","last_peered":"2024-02-05T07:33:18.065249+0000","last_clean":"2024-02-05T07:33:18.065249+0000","last_became_active":"2024-02-05T07:30:40.489120+0000","last_became_peered":"2024-02-05T07:30:40.489120+0000","last_unstale":"2024-02-05T07:33:18.065249+0000","last_undegraded":"2024-02-05T07:33:18.065249+0000","last_fullsized":"2024-02-05T07:33:18.065249+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:33.839513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:32:33.839513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:41.186592+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"416'9890","reported_seq":10035,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:10.557580+0000","last_change":"2024-02-05T07:30:00.509399+0000","last_active":"2024-02-05T07:33:10.557580+0000","last_peered":"2024-02-05T07:33:10.557580+0000","last_clean":"2024-02-05T07:33:10.557580+0000","last_became_active":"2024-02-05T07:30:00.509120+0000","last_became_peered":"2024-02-05T07:30:00.509120+0000","last_unstale":"2024-02-05T07:33:10.557580+0000","last_undegraded":"2024-02-05T07:33:10.557580+0000","last_fullsized":"2024-02-05T07:33:10.557580+0000","mapping_epoch":349,"log_start":"400'6871","ondisk_log_start":"400'6871","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3019,"log_dups_size":0,"ondisk_log_size":3019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.384151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9890,"num_write_kb":316480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":596,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065935+0000","last_change":"2024-02-05T07:32:21.853114+0000","last_active":"2024-02-05T07:33:18.065935+0000","last_peered":"2024-02-05T07:33:18.065935+0000","last_clean":"2024-02-05T07:33:18.065935+0000","last_became_active":"2024-02-05T07:31:36.850075+0000","last_became_peered":"2024-02-05T07:31:36.850075+0000","last_unstale":"2024-02-05T07:33:18.065935+0000","last_undegraded":"2024-02-05T07:33:18.065935+0000","last_fullsized":"2024-02-05T07:33:18.065935+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:21.853039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:21.853039+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:42.459463+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":396,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065525+0000","last_change":"2024-02-05T07:32:49.752237+0000","last_active":"2024-02-05T07:33:18.065525+0000","last_peered":"2024-02-05T07:33:18.065525+0000","last_clean":"2024-02-05T07:33:18.065525+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:33:18.065525+0000","last_undegraded":"2024-02-05T07:33:18.065525+0000","last_fullsized":"2024-02-05T07:33:18.065525+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:32:49.752163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:49.752163+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:56.030580+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"326'25","reported_seq":802,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.539714+0000","last_change":"2024-02-05T07:30:36.908113+0000","last_active":"2024-02-05T07:33:09.539714+0000","last_peered":"2024-02-05T07:33:09.539714+0000","last_clean":"2024-02-05T07:33:09.539714+0000","last_became_active":"2024-02-05T07:27:03.361342+0000","last_became_peered":"2024-02-05T07:27:03.361342+0000","last_unstale":"2024-02-05T07:33:09.539714+0000","last_undegraded":"2024-02-05T07:33:09.539714+0000","last_fullsized":"2024-02-05T07:33:09.539714+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:30:36.907989+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.907989+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.384564+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":328,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.533923+0000","last_change":"2024-02-05T07:29:44.015577+0000","last_active":"2024-02-05T07:33:09.533923+0000","last_peered":"2024-02-05T07:33:09.533923+0000","last_clean":"2024-02-05T07:33:09.533923+0000","last_became_active":"2024-02-05T07:28:26.596096+0000","last_became_peered":"2024-02-05T07:28:26.596096+0000","last_unstale":"2024-02-05T07:33:09.533923+0000","last_undegraded":"2024-02-05T07:33:09.533923+0000","last_fullsized":"2024-02-05T07:33:09.533923+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:44.015468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:29:44.015468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":17,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.525069+0000","last_change":"2024-02-05T07:32:58.601524+0000","last_active":"2024-02-05T07:33:09.525069+0000","last_peered":"2024-02-05T07:33:09.525069+0000","last_clean":"2024-02-05T07:33:09.525069+0000","last_became_active":"2024-02-05T07:32:58.598709+0000","last_became_peered":"2024-02-05T07:32:58.598709+0000","last_unstale":"2024-02-05T07:33:09.525069+0000","last_undegraded":"2024-02-05T07:33:09.525069+0000","last_fullsized":"2024-02-05T07:33:09.525069+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:25:52.847946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:25:52.847946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383284+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"416'10056","reported_seq":15657,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522541+0000","last_change":"2024-02-05T07:32:01.564512+0000","last_active":"2024-02-05T07:33:19.522541+0000","last_peered":"2024-02-05T07:33:19.522541+0000","last_clean":"2024-02-05T07:33:19.522541+0000","last_became_active":"2024-02-05T07:32:01.559320+0000","last_became_peered":"2024-02-05T07:32:01.559320+0000","last_unstale":"2024-02-05T07:33:19.522541+0000","last_undegraded":"2024-02-05T07:33:19.522541+0000","last_fullsized":"2024-02-05T07:33:19.522541+0000","mapping_epoch":395,"log_start":"397'6580","ondisk_log_start":"397'6580","created":349,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3476,"log_dups_size":0,"ondisk_log_size":3476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10056,"num_write_kb":321792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2721,"num_bytes_recovered":178323456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":587,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065640+0000","last_change":"2024-02-05T07:32:20.818876+0000","last_active":"2024-02-05T07:33:18.065640+0000","last_peered":"2024-02-05T07:33:18.065640+0000","last_clean":"2024-02-05T07:33:18.065640+0000","last_became_active":"2024-02-05T07:31:36.840534+0000","last_became_peered":"2024-02-05T07:31:36.840534+0000","last_unstale":"2024-02-05T07:33:18.065640+0000","last_undegraded":"2024-02-05T07:33:18.065640+0000","last_fullsized":"2024-02-05T07:33:18.065640+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:20.818799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:20.818799+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:29.517723+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":578,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065082+0000","last_change":"2024-02-05T07:30:35.140369+0000","last_active":"2024-02-05T07:33:18.065082+0000","last_peered":"2024-02-05T07:33:18.065082+0000","last_clean":"2024-02-05T07:33:18.065082+0000","last_became_active":"2024-02-05T07:28:26.589673+0000","last_became_peered":"2024-02-05T07:28:26.589673+0000","last_unstale":"2024-02-05T07:33:18.065082+0000","last_undegraded":"2024-02-05T07:33:18.065082+0000","last_fullsized":"2024-02-05T07:33:18.065082+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:35.140291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:30:35.140291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:20.070995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"326'29","reported_seq":371,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.525441+0000","last_change":"2024-02-05T07:30:40.516301+0000","last_active":"2024-02-05T07:33:09.525441+0000","last_peered":"2024-02-05T07:33:09.525441+0000","last_clean":"2024-02-05T07:33:09.525441+0000","last_became_active":"2024-02-05T07:30:40.481234+0000","last_became_peered":"2024-02-05T07:30:40.481234+0000","last_unstale":"2024-02-05T07:33:09.525441+0000","last_undegraded":"2024-02-05T07:33:09.525441+0000","last_fullsized":"2024-02-05T07:33:09.525441+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'29","last_scrub_stamp":"2024-02-05T07:30:37.106672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:37.106672+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.384180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":751,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.526924+0000","last_change":"2024-02-05T07:29:53.699733+0000","last_active":"2024-02-05T07:33:09.526924+0000","last_peered":"2024-02-05T07:33:09.526924+0000","last_clean":"2024-02-05T07:33:09.526924+0000","last_became_active":"2024-02-05T07:27:03.361028+0000","last_became_peered":"2024-02-05T07:27:03.361028+0000","last_unstale":"2024-02-05T07:33:09.526924+0000","last_undegraded":"2024-02-05T07:33:09.526924+0000","last_fullsized":"2024-02-05T07:33:09.526924+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.699651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.699651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":734,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.526061+0000","last_change":"2024-02-05T07:28:49.555346+0000","last_active":"2024-02-05T07:33:09.526061+0000","last_peered":"2024-02-05T07:33:09.526061+0000","last_clean":"2024-02-05T07:33:09.526061+0000","last_became_active":"2024-02-05T07:28:25.840850+0000","last_became_peered":"2024-02-05T07:28:25.840850+0000","last_unstale":"2024-02-05T07:33:09.526061+0000","last_undegraded":"2024-02-05T07:33:09.526061+0000","last_fullsized":"2024-02-05T07:33:09.526061+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:49.555018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:49.555018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383423+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"416'9872","reported_seq":10019,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065588+0000","last_change":"2024-02-05T07:30:00.869708+0000","last_active":"2024-02-05T07:33:18.065588+0000","last_peered":"2024-02-05T07:33:18.065588+0000","last_clean":"2024-02-05T07:33:18.065588+0000","last_became_active":"2024-02-05T07:30:00.869295+0000","last_became_peered":"2024-02-05T07:30:00.869295+0000","last_unstale":"2024-02-05T07:33:18.065588+0000","last_undegraded":"2024-02-05T07:33:18.065588+0000","last_fullsized":"2024-02-05T07:33:18.065588+0000","mapping_epoch":349,"log_start":"396'6228","ondisk_log_start":"396'6228","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3644,"log_dups_size":0,"ondisk_log_size":3644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:20.070965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9872,"num_write_kb":315904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":716,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.525699+0000","last_change":"2024-02-05T07:28:47.605953+0000","last_active":"2024-02-05T07:33:09.525699+0000","last_peered":"2024-02-05T07:33:09.525699+0000","last_clean":"2024-02-05T07:33:09.525699+0000","last_became_active":"2024-02-05T07:28:25.756613+0000","last_became_peered":"2024-02-05T07:28:25.756613+0000","last_unstale":"2024-02-05T07:33:09.525699+0000","last_undegraded":"2024-02-05T07:33:09.525699+0000","last_fullsized":"2024-02-05T07:33:09.525699+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:47.605803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.605803+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383483+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":342,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.537651+0000","last_change":"2024-02-05T07:32:46.399673+0000","last_active":"2024-02-05T07:33:09.537651+0000","last_peered":"2024-02-05T07:33:09.537651+0000","last_clean":"2024-02-05T07:33:09.537651+0000","last_became_active":"2024-02-05T07:27:03.370297+0000","last_became_peered":"2024-02-05T07:27:03.370297+0000","last_unstale":"2024-02-05T07:33:09.537651+0000","last_undegraded":"2024-02-05T07:33:09.537651+0000","last_fullsized":"2024-02-05T07:33:09.537651+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:46.399572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:46.399572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:02.315642+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"326'24","reported_seq":545,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.532096+0000","last_change":"2024-02-05T07:29:36.583093+0000","last_active":"2024-02-05T07:33:09.532096+0000","last_peered":"2024-02-05T07:33:09.532096+0000","last_clean":"2024-02-05T07:33:09.532096+0000","last_became_active":"2024-02-05T07:24:34.209716+0000","last_became_peered":"2024-02-05T07:24:34.209716+0000","last_unstale":"2024-02-05T07:33:09.532096+0000","last_undegraded":"2024-02-05T07:33:09.532096+0000","last_fullsized":"2024-02-05T07:33:09.532096+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:29:36.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:36.583005+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":316,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:15.647457+0000","last_change":"2024-02-05T07:30:40.520395+0000","last_active":"2024-02-05T07:33:15.647457+0000","last_peered":"2024-02-05T07:33:15.647457+0000","last_clean":"2024-02-05T07:33:15.647457+0000","last_became_active":"2024-02-05T07:30:40.493770+0000","last_became_peered":"2024-02-05T07:30:40.493770+0000","last_unstale":"2024-02-05T07:33:15.647457+0000","last_undegraded":"2024-02-05T07:33:15.647457+0000","last_fullsized":"2024-02-05T07:33:15.647457+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:29:51.291146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.291146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":162,"num_read_kb":157,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":713,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521220+0000","last_change":"2024-02-05T07:27:16.724562+0000","last_active":"2024-02-05T07:33:19.521220+0000","last_peered":"2024-02-05T07:33:19.521220+0000","last_clean":"2024-02-05T07:33:19.521220+0000","last_became_active":"2024-02-05T07:27:16.724280+0000","last_became_peered":"2024-02-05T07:27:16.724280+0000","last_unstale":"2024-02-05T07:33:19.521220+0000","last_undegraded":"2024-02-05T07:33:19.521220+0000","last_fullsized":"2024-02-05T07:33:19.521220+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:30.351655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:26:30.351655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":762,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.525921+0000","last_change":"2024-02-05T07:28:50.545548+0000","last_active":"2024-02-05T07:33:09.525921+0000","last_peered":"2024-02-05T07:33:09.525921+0000","last_clean":"2024-02-05T07:33:09.525921+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:33:09.525921+0000","last_undegraded":"2024-02-05T07:33:09.525921+0000","last_fullsized":"2024-02-05T07:33:09.525921+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:28:50.545437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:50.545437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383453+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"416'10178","reported_seq":15877,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:10.550857+0000","last_change":"2024-02-05T07:32:41.344276+0000","last_active":"2024-02-05T07:33:10.550857+0000","last_peered":"2024-02-05T07:33:10.550857+0000","last_clean":"2024-02-05T07:33:10.550857+0000","last_became_active":"2024-02-05T07:32:41.320952+0000","last_became_peered":"2024-02-05T07:32:41.320952+0000","last_unstale":"2024-02-05T07:33:10.550857+0000","last_undegraded":"2024-02-05T07:33:10.550857+0000","last_fullsized":"2024-02-05T07:33:10.550857+0000","mapping_epoch":405,"log_start":"400'6938","ondisk_log_start":"400'6938","created":349,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3240,"log_dups_size":0,"ondisk_log_size":3240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10178,"num_write_kb":325696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2772,"num_bytes_recovered":165937152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":515,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.541358+0000","last_change":"2024-02-05T07:30:40.474637+0000","last_active":"2024-02-05T07:33:20.541358+0000","last_peered":"2024-02-05T07:33:20.541358+0000","last_clean":"2024-02-05T07:33:20.541358+0000","last_became_active":"2024-02-05T07:30:40.474169+0000","last_became_peered":"2024-02-05T07:30:40.474169+0000","last_unstale":"2024-02-05T07:33:20.541358+0000","last_undegraded":"2024-02-05T07:33:20.541358+0000","last_fullsized":"2024-02-05T07:33:20.541358+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:30.221253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.221253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"365'31","reported_seq":800,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065246+0000","last_change":"2024-02-05T07:30:40.539387+0000","last_active":"2024-02-05T07:33:18.065246+0000","last_peered":"2024-02-05T07:33:18.065246+0000","last_clean":"2024-02-05T07:33:18.065246+0000","last_became_active":"2024-02-05T07:30:40.508686+0000","last_became_peered":"2024-02-05T07:30:40.508686+0000","last_unstale":"2024-02-05T07:33:18.065246+0000","last_undegraded":"2024-02-05T07:33:18.065246+0000","last_fullsized":"2024-02-05T07:33:18.065246+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'30","last_scrub_stamp":"2024-02-05T07:30:37.201930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:37.201930+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:20.071042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":328,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.534952+0000","last_change":"2024-02-05T07:29:43.000854+0000","last_active":"2024-02-05T07:33:09.534952+0000","last_peered":"2024-02-05T07:33:09.534952+0000","last_clean":"2024-02-05T07:33:09.534952+0000","last_became_active":"2024-02-05T07:28:26.595640+0000","last_became_peered":"2024-02-05T07:28:26.595640+0000","last_unstale":"2024-02-05T07:33:09.534952+0000","last_undegraded":"2024-02-05T07:33:09.534952+0000","last_fullsized":"2024-02-05T07:33:09.534952+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:43.000778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:43.000778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":432,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.064753+0000","last_change":"2024-02-05T07:32:19.775077+0000","last_active":"2024-02-05T07:33:18.064753+0000","last_peered":"2024-02-05T07:33:18.064753+0000","last_clean":"2024-02-05T07:33:18.064753+0000","last_became_active":"2024-02-05T07:30:43.516031+0000","last_became_peered":"2024-02-05T07:30:43.516031+0000","last_unstale":"2024-02-05T07:33:18.064753+0000","last_undegraded":"2024-02-05T07:33:18.064753+0000","last_fullsized":"2024-02-05T07:33:18.064753+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:19.774991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:19.774991+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:36.180536+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":317,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.533850+0000","last_change":"2024-02-05T07:27:16.751712+0000","last_active":"2024-02-05T07:33:09.533850+0000","last_peered":"2024-02-05T07:33:09.533850+0000","last_clean":"2024-02-05T07:33:09.533850+0000","last_became_active":"2024-02-05T07:27:16.751341+0000","last_became_peered":"2024-02-05T07:27:16.751341+0000","last_unstale":"2024-02-05T07:33:09.533850+0000","last_undegraded":"2024-02-05T07:33:09.533850+0000","last_fullsized":"2024-02-05T07:33:09.533850+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:36.753098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.753098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"416'10052","reported_seq":15638,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:10.515147+0000","last_change":"2024-02-05T07:31:32.742675+0000","last_active":"2024-02-05T07:33:10.515147+0000","last_peered":"2024-02-05T07:33:10.515147+0000","last_clean":"2024-02-05T07:33:10.515147+0000","last_became_active":"2024-02-05T07:31:32.736052+0000","last_became_peered":"2024-02-05T07:31:32.736052+0000","last_unstale":"2024-02-05T07:33:10.515147+0000","last_undegraded":"2024-02-05T07:33:10.515147+0000","last_fullsized":"2024-02-05T07:33:10.515147+0000","mapping_epoch":380,"log_start":"400'6952","ondisk_log_start":"400'6952","created":349,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3100,"log_dups_size":0,"ondisk_log_size":3100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.384011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10052,"num_write_kb":321664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2716,"num_bytes_recovered":177995776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":744,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521186+0000","last_change":"2024-02-05T07:32:21.667678+0000","last_active":"2024-02-05T07:33:19.521186+0000","last_peered":"2024-02-05T07:33:19.521186+0000","last_clean":"2024-02-05T07:33:19.521186+0000","last_became_active":"2024-02-05T07:30:43.509733+0000","last_became_peered":"2024-02-05T07:30:43.509733+0000","last_unstale":"2024-02-05T07:33:19.521186+0000","last_undegraded":"2024-02-05T07:33:19.521186+0000","last_fullsized":"2024-02-05T07:33:19.521186+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:21.667581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:21.667581+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:48.654513+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":754,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.541243+0000","last_change":"2024-02-05T07:30:40.504250+0000","last_active":"2024-02-05T07:33:20.541243+0000","last_peered":"2024-02-05T07:33:20.541243+0000","last_clean":"2024-02-05T07:33:20.541243+0000","last_became_active":"2024-02-05T07:30:40.503816+0000","last_became_peered":"2024-02-05T07:30:40.503816+0000","last_unstale":"2024-02-05T07:33:20.541243+0000","last_undegraded":"2024-02-05T07:33:20.541243+0000","last_fullsized":"2024-02-05T07:33:20.541243+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:40.103572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:40.103572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"326'22","reported_seq":807,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.536861+0000","last_change":"2024-02-05T07:30:40.533683+0000","last_active":"2024-02-05T07:33:09.536861+0000","last_peered":"2024-02-05T07:33:09.536861+0000","last_clean":"2024-02-05T07:33:09.536861+0000","last_became_active":"2024-02-05T07:30:40.494460+0000","last_became_peered":"2024-02-05T07:30:40.494460+0000","last_unstale":"2024-02-05T07:33:09.536861+0000","last_undegraded":"2024-02-05T07:33:09.536861+0000","last_fullsized":"2024-02-05T07:33:09.536861+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'18","last_scrub_stamp":"2024-02-05T07:26:24.578301+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.578301+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383368+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":779,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.540144+0000","last_change":"2024-02-05T07:29:42.383585+0000","last_active":"2024-02-05T07:33:20.540144+0000","last_peered":"2024-02-05T07:33:20.540144+0000","last_clean":"2024-02-05T07:33:20.540144+0000","last_became_active":"2024-02-05T07:27:16.728869+0000","last_became_peered":"2024-02-05T07:27:16.728869+0000","last_unstale":"2024-02-05T07:33:20.540144+0000","last_undegraded":"2024-02-05T07:33:20.540144+0000","last_fullsized":"2024-02-05T07:33:20.540144+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:42.383469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:29:42.383469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":761,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.526172+0000","last_change":"2024-02-05T07:29:45.546235+0000","last_active":"2024-02-05T07:33:09.526172+0000","last_peered":"2024-02-05T07:33:09.526172+0000","last_clean":"2024-02-05T07:33:09.526172+0000","last_became_active":"2024-02-05T07:28:25.750373+0000","last_became_peered":"2024-02-05T07:28:25.750373+0000","last_unstale":"2024-02-05T07:33:09.526172+0000","last_undegraded":"2024-02-05T07:33:09.526172+0000","last_fullsized":"2024-02-05T07:33:09.526172+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:45.546152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.546152+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383690+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":623,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065411+0000","last_change":"2024-02-05T07:32:38.877762+0000","last_active":"2024-02-05T07:33:18.065411+0000","last_peered":"2024-02-05T07:33:18.065411+0000","last_clean":"2024-02-05T07:33:18.065411+0000","last_became_active":"2024-02-05T07:30:40.489314+0000","last_became_peered":"2024-02-05T07:30:40.489314+0000","last_unstale":"2024-02-05T07:33:18.065411+0000","last_undegraded":"2024-02-05T07:33:18.065411+0000","last_fullsized":"2024-02-05T07:33:18.065411+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:38.877685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:32:38.877685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:55.211010+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"416'9718","reported_seq":15128,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:10.525391+0000","last_change":"2024-02-05T07:31:37.794109+0000","last_active":"2024-02-05T07:33:10.525391+0000","last_peered":"2024-02-05T07:33:10.525391+0000","last_clean":"2024-02-05T07:33:10.525391+0000","last_became_active":"2024-02-05T07:31:37.788267+0000","last_became_peered":"2024-02-05T07:31:37.788267+0000","last_unstale":"2024-02-05T07:33:10.525391+0000","last_undegraded":"2024-02-05T07:33:10.525391+0000","last_fullsized":"2024-02-05T07:33:10.525391+0000","mapping_epoch":384,"log_start":"400'6505","ondisk_log_start":"400'6505","created":349,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3213,"log_dups_size":0,"ondisk_log_size":3213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9718,"num_write_kb":310976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2627,"num_bytes_recovered":172163072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":740,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.539864+0000","last_change":"2024-02-05T07:29:55.246298+0000","last_active":"2024-02-05T07:33:20.539864+0000","last_peered":"2024-02-05T07:33:20.539864+0000","last_clean":"2024-02-05T07:33:20.539864+0000","last_became_active":"2024-02-05T07:28:14.305034+0000","last_became_peered":"2024-02-05T07:28:14.305034+0000","last_unstale":"2024-02-05T07:33:20.539864+0000","last_undegraded":"2024-02-05T07:33:20.539864+0000","last_fullsized":"2024-02-05T07:33:20.539864+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:29:55.246214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.246214+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":490,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.528585+0000","last_change":"2024-02-05T07:29:48.627198+0000","last_active":"2024-02-05T07:33:09.528585+0000","last_peered":"2024-02-05T07:33:09.528585+0000","last_clean":"2024-02-05T07:33:09.528585+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:33:09.528585+0000","last_undegraded":"2024-02-05T07:33:09.528585+0000","last_fullsized":"2024-02-05T07:33:09.528585+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.627107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.627107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383715+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"326'44","reported_seq":669,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.541098+0000","last_change":"2024-02-05T07:30:36.009087+0000","last_active":"2024-02-05T07:33:20.541098+0000","last_peered":"2024-02-05T07:33:20.541098+0000","last_clean":"2024-02-05T07:33:20.541098+0000","last_became_active":"2024-02-05T07:27:16.708475+0000","last_became_peered":"2024-02-05T07:27:16.708475+0000","last_unstale":"2024-02-05T07:33:20.541098+0000","last_undegraded":"2024-02-05T07:33:20.541098+0000","last_fullsized":"2024-02-05T07:33:20.541098+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'44","last_scrub_stamp":"2024-02-05T07:30:36.008885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.008885+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":138,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.541696+0000","last_change":"2024-02-05T07:30:32.964745+0000","last_active":"2024-02-05T07:33:20.541696+0000","last_peered":"2024-02-05T07:33:20.541696+0000","last_clean":"2024-02-05T07:33:20.541696+0000","last_became_active":"2024-02-05T07:30:30.524566+0000","last_became_peered":"2024-02-05T07:30:30.524566+0000","last_unstale":"2024-02-05T07:33:20.541696+0000","last_undegraded":"2024-02-05T07:33:20.541696+0000","last_fullsized":"2024-02-05T07:33:20.541696+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:32.964622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:32.964622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":659,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.539350+0000","last_change":"2024-02-05T07:28:55.477529+0000","last_active":"2024-02-05T07:33:09.539350+0000","last_peered":"2024-02-05T07:33:09.539350+0000","last_clean":"2024-02-05T07:33:09.539350+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:33:09.539350+0000","last_undegraded":"2024-02-05T07:33:09.539350+0000","last_fullsized":"2024-02-05T07:33:09.539350+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:28:55.477280+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383573+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":331,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.530985+0000","last_change":"2024-02-05T07:30:40.508412+0000","last_active":"2024-02-05T07:33:09.530985+0000","last_peered":"2024-02-05T07:33:09.530985+0000","last_clean":"2024-02-05T07:33:09.530985+0000","last_became_active":"2024-02-05T07:30:40.508071+0000","last_became_peered":"2024-02-05T07:30:40.508071+0000","last_unstale":"2024-02-05T07:33:09.530985+0000","last_undegraded":"2024-02-05T07:33:09.530985+0000","last_fullsized":"2024-02-05T07:33:09.530985+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:42.075445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:42.075445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.6","version":"416'10118","reported_seq":10269,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.540737+0000","last_change":"2024-02-05T07:30:00.512361+0000","last_active":"2024-02-05T07:33:20.540737+0000","last_peered":"2024-02-05T07:33:20.540737+0000","last_clean":"2024-02-05T07:33:20.540737+0000","last_became_active":"2024-02-05T07:30:00.511728+0000","last_became_peered":"2024-02-05T07:30:00.511728+0000","last_unstale":"2024-02-05T07:33:20.540737+0000","last_undegraded":"2024-02-05T07:33:20.540737+0000","last_fullsized":"2024-02-05T07:33:20.540737+0000","mapping_epoch":349,"log_start":"396'6189","ondisk_log_start":"396'6189","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3929,"log_dups_size":0,"ondisk_log_size":3929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":323776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":669,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522116+0000","last_change":"2024-02-05T07:32:23.628807+0000","last_active":"2024-02-05T07:33:19.522116+0000","last_peered":"2024-02-05T07:33:19.522116+0000","last_clean":"2024-02-05T07:33:19.522116+0000","last_became_active":"2024-02-05T07:31:36.842497+0000","last_became_peered":"2024-02-05T07:31:36.842497+0000","last_unstale":"2024-02-05T07:33:19.522116+0000","last_undegraded":"2024-02-05T07:33:19.522116+0000","last_fullsized":"2024-02-05T07:33:19.522116+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:23.628698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:23.628698+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:45.560553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"326'15","reported_seq":532,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.525797+0000","last_change":"2024-02-05T07:29:51.665907+0000","last_active":"2024-02-05T07:33:09.525797+0000","last_peered":"2024-02-05T07:33:09.525797+0000","last_clean":"2024-02-05T07:33:09.525797+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:33:09.525797+0000","last_undegraded":"2024-02-05T07:33:09.525797+0000","last_fullsized":"2024-02-05T07:33:09.525797+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:29:51.665808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.665808+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.384040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":328,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.540426+0000","last_change":"2024-02-05T07:30:38.998491+0000","last_active":"2024-02-05T07:33:09.540426+0000","last_peered":"2024-02-05T07:33:09.540426+0000","last_clean":"2024-02-05T07:33:09.540426+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:33:09.540426+0000","last_undegraded":"2024-02-05T07:33:09.540426+0000","last_fullsized":"2024-02-05T07:33:09.540426+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:38.995586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:38.995586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":762,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.541179+0000","last_change":"2024-02-05T07:32:47.161468+0000","last_active":"2024-02-05T07:33:20.541179+0000","last_peered":"2024-02-05T07:33:20.541179+0000","last_clean":"2024-02-05T07:33:20.541179+0000","last_became_active":"2024-02-05T07:27:16.708205+0000","last_became_peered":"2024-02-05T07:27:16.708205+0000","last_unstale":"2024-02-05T07:33:20.541179+0000","last_undegraded":"2024-02-05T07:33:20.541179+0000","last_fullsized":"2024-02-05T07:33:20.541179+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:47.161343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:47.161343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:13.165702+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":131,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.540320+0000","last_change":"2024-02-05T07:32:46.198277+0000","last_active":"2024-02-05T07:33:20.540320+0000","last_peered":"2024-02-05T07:33:20.540320+0000","last_clean":"2024-02-05T07:33:20.540320+0000","last_became_active":"2024-02-05T07:30:40.486401+0000","last_became_peered":"2024-02-05T07:30:40.486401+0000","last_unstale":"2024-02-05T07:33:20.540320+0000","last_undegraded":"2024-02-05T07:33:20.540320+0000","last_fullsized":"2024-02-05T07:33:20.540320+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:46.198155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:32:46.198155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:48.596593+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"416'10106","reported_seq":15741,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:10.550854+0000","last_change":"2024-02-05T07:32:40.332596+0000","last_active":"2024-02-05T07:33:10.550854+0000","last_peered":"2024-02-05T07:33:10.550854+0000","last_clean":"2024-02-05T07:33:10.550854+0000","last_became_active":"2024-02-05T07:32:40.316255+0000","last_became_peered":"2024-02-05T07:32:40.316255+0000","last_unstale":"2024-02-05T07:33:10.550854+0000","last_undegraded":"2024-02-05T07:33:10.550854+0000","last_fullsized":"2024-02-05T07:33:10.550854+0000","mapping_epoch":404,"log_start":"400'6879","ondisk_log_start":"400'6879","created":349,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10106,"num_write_kb":323392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2741,"num_bytes_recovered":171245568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":583,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065684+0000","last_change":"2024-02-05T07:32:24.854963+0000","last_active":"2024-02-05T07:33:18.065684+0000","last_peered":"2024-02-05T07:33:18.065684+0000","last_clean":"2024-02-05T07:33:18.065684+0000","last_became_active":"2024-02-05T07:28:26.596711+0000","last_became_peered":"2024-02-05T07:28:26.596711+0000","last_unstale":"2024-02-05T07:33:18.065684+0000","last_undegraded":"2024-02-05T07:33:18.065684+0000","last_fullsized":"2024-02-05T07:33:18.065684+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:24.854881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:24.854881+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:38.855880+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"326'35","reported_seq":653,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065187+0000","last_change":"2024-02-05T07:30:40.520826+0000","last_active":"2024-02-05T07:33:18.065187+0000","last_peered":"2024-02-05T07:33:18.065187+0000","last_clean":"2024-02-05T07:33:18.065187+0000","last_became_active":"2024-02-05T07:30:40.482753+0000","last_became_peered":"2024-02-05T07:30:40.482753+0000","last_unstale":"2024-02-05T07:33:18.065187+0000","last_undegraded":"2024-02-05T07:33:18.065187+0000","last_fullsized":"2024-02-05T07:33:18.065187+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:29:45.981520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.981520+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:20.070929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":634,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065374+0000","last_change":"2024-02-05T07:32:39.793152+0000","last_active":"2024-02-05T07:33:18.065374+0000","last_peered":"2024-02-05T07:33:18.065374+0000","last_clean":"2024-02-05T07:33:18.065374+0000","last_became_active":"2024-02-05T07:30:40.507705+0000","last_became_peered":"2024-02-05T07:30:40.507705+0000","last_unstale":"2024-02-05T07:33:18.065374+0000","last_undegraded":"2024-02-05T07:33:18.065374+0000","last_fullsized":"2024-02-05T07:33:18.065374+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_clean_scrub_stamp":"2024-02-05T07:32:39.793075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:55.003730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":754,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.541828+0000","last_change":"2024-02-05T07:30:40.480847+0000","last_active":"2024-02-05T07:33:20.541828+0000","last_peered":"2024-02-05T07:33:20.541828+0000","last_clean":"2024-02-05T07:33:20.541828+0000","last_became_active":"2024-02-05T07:30:40.479288+0000","last_became_peered":"2024-02-05T07:30:40.479288+0000","last_unstale":"2024-02-05T07:33:20.541828+0000","last_undegraded":"2024-02-05T07:33:20.541828+0000","last_fullsized":"2024-02-05T07:33:20.541828+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:55.173628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:28:55.173628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":536,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.539340+0000","last_change":"2024-02-05T07:30:40.497142+0000","last_active":"2024-02-05T07:33:20.539340+0000","last_peered":"2024-02-05T07:33:20.539340+0000","last_clean":"2024-02-05T07:33:20.539340+0000","last_became_active":"2024-02-05T07:30:40.496822+0000","last_became_peered":"2024-02-05T07:30:40.496822+0000","last_unstale":"2024-02-05T07:33:20.539340+0000","last_undegraded":"2024-02-05T07:33:20.539340+0000","last_fullsized":"2024-02-05T07:33:20.539340+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.262537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.4","version":"416'10184","reported_seq":10329,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:10.535936+0000","last_change":"2024-02-05T07:30:01.030823+0000","last_active":"2024-02-05T07:33:10.535936+0000","last_peered":"2024-02-05T07:33:10.535936+0000","last_clean":"2024-02-05T07:33:10.535936+0000","last_became_active":"2024-02-05T07:30:01.030502+0000","last_became_peered":"2024-02-05T07:30:01.030502+0000","last_unstale":"2024-02-05T07:33:10.535936+0000","last_undegraded":"2024-02-05T07:33:10.535936+0000","last_fullsized":"2024-02-05T07:33:10.535936+0000","mapping_epoch":349,"log_start":"400'7128","ondisk_log_start":"400'7128","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3056,"log_dups_size":0,"ondisk_log_size":3056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10184,"num_write_kb":325888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":997,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.539456+0000","last_change":"2024-02-05T07:32:41.754728+0000","last_active":"2024-02-05T07:33:09.539456+0000","last_peered":"2024-02-05T07:33:09.539456+0000","last_clean":"2024-02-05T07:33:09.539456+0000","last_became_active":"2024-02-05T07:31:36.840096+0000","last_became_peered":"2024-02-05T07:31:36.840096+0000","last_unstale":"2024-02-05T07:33:09.539456+0000","last_undegraded":"2024-02-05T07:33:09.539456+0000","last_fullsized":"2024-02-05T07:33:09.539456+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:41.754605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:41.754605+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:09.050452+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":750,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.539183+0000","last_change":"2024-02-05T07:29:39.290437+0000","last_active":"2024-02-05T07:33:20.539183+0000","last_peered":"2024-02-05T07:33:20.539183+0000","last_clean":"2024-02-05T07:33:20.539183+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:33:20.539183+0000","last_undegraded":"2024-02-05T07:33:20.539183+0000","last_fullsized":"2024-02-05T07:33:20.539183+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:39.290320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:39.290320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"325'18","reported_seq":836,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521443+0000","last_change":"2024-02-05T07:29:49.022612+0000","last_active":"2024-02-05T07:33:19.521443+0000","last_peered":"2024-02-05T07:33:19.521443+0000","last_clean":"2024-02-05T07:33:19.521443+0000","last_became_active":"2024-02-05T07:27:16.706871+0000","last_became_peered":"2024-02-05T07:27:16.706871+0000","last_unstale":"2024-02-05T07:33:19.521443+0000","last_undegraded":"2024-02-05T07:33:19.521443+0000","last_fullsized":"2024-02-05T07:33:19.521443+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'18","last_scrub_stamp":"2024-02-05T07:29:49.021772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.021772+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":118,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522352+0000","last_change":"2024-02-05T07:30:40.519397+0000","last_active":"2024-02-05T07:33:19.522352+0000","last_peered":"2024-02-05T07:33:19.522352+0000","last_clean":"2024-02-05T07:33:19.522352+0000","last_became_active":"2024-02-05T07:30:40.518949+0000","last_became_peered":"2024-02-05T07:30:40.518949+0000","last_unstale":"2024-02-05T07:33:19.522352+0000","last_undegraded":"2024-02-05T07:33:19.522352+0000","last_fullsized":"2024-02-05T07:33:19.522352+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:24.102597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:24.102597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":331,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.530079+0000","last_change":"2024-02-05T07:30:40.486334+0000","last_active":"2024-02-05T07:33:09.530079+0000","last_peered":"2024-02-05T07:33:09.530079+0000","last_clean":"2024-02-05T07:33:09.530079+0000","last_became_active":"2024-02-05T07:30:40.485991+0000","last_became_peered":"2024-02-05T07:30:40.485991+0000","last_unstale":"2024-02-05T07:33:09.530079+0000","last_undegraded":"2024-02-05T07:33:09.530079+0000","last_fullsized":"2024-02-05T07:33:09.530079+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:48.088229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.088229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"416'10050","reported_seq":15694,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:10.550815+0000","last_change":"2024-02-05T07:31:48.466349+0000","last_active":"2024-02-05T07:33:10.550815+0000","last_peered":"2024-02-05T07:33:10.550815+0000","last_clean":"2024-02-05T07:33:10.550815+0000","last_became_active":"2024-02-05T07:31:48.459417+0000","last_became_peered":"2024-02-05T07:31:48.459417+0000","last_unstale":"2024-02-05T07:33:10.550815+0000","last_undegraded":"2024-02-05T07:33:10.550815+0000","last_fullsized":"2024-02-05T07:33:10.550815+0000","mapping_epoch":391,"log_start":"400'6969","ondisk_log_start":"400'6969","created":349,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:59.492295+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.492295+0000","objects_scrubbed":0,"log_size":3081,"log_dups_size":0,"ondisk_log_size":3081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10050,"num_write_kb":321600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2745,"num_bytes_recovered":179896320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":585,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521935+0000","last_change":"2024-02-05T07:32:22.621687+0000","last_active":"2024-02-05T07:33:19.521935+0000","last_peered":"2024-02-05T07:33:19.521935+0000","last_clean":"2024-02-05T07:33:19.521935+0000","last_became_active":"2024-02-05T07:31:36.837268+0000","last_became_peered":"2024-02-05T07:31:36.837268+0000","last_unstale":"2024-02-05T07:33:19.521935+0000","last_undegraded":"2024-02-05T07:33:19.521935+0000","last_fullsized":"2024-02-05T07:33:19.521935+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:22.621608+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_clean_scrub_stamp":"2024-02-05T07:32:22.621608+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:48.097590+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"365'12","reported_seq":777,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.064685+0000","last_change":"2024-02-05T07:32:30.801254+0000","last_active":"2024-02-05T07:33:18.064685+0000","last_peered":"2024-02-05T07:33:18.064685+0000","last_clean":"2024-02-05T07:33:18.064685+0000","last_became_active":"2024-02-05T07:30:40.500113+0000","last_became_peered":"2024-02-05T07:30:40.500113+0000","last_unstale":"2024-02-05T07:33:18.064685+0000","last_undegraded":"2024-02-05T07:33:18.064685+0000","last_fullsized":"2024-02-05T07:33:18.064685+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'12","last_scrub_stamp":"2024-02-05T07:32:30.801134+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:32:30.801134+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:42.756735+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"326'27","reported_seq":381,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.064655+0000","last_change":"2024-02-05T07:32:29.833732+0000","last_active":"2024-02-05T07:33:18.064655+0000","last_peered":"2024-02-05T07:33:18.064655+0000","last_clean":"2024-02-05T07:33:18.064655+0000","last_became_active":"2024-02-05T07:30:40.497982+0000","last_became_peered":"2024-02-05T07:30:40.497982+0000","last_unstale":"2024-02-05T07:33:18.064655+0000","last_undegraded":"2024-02-05T07:33:18.064655+0000","last_fullsized":"2024-02-05T07:33:18.064655+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:32:29.833636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:29.833636+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:46.742620+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":134,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522539+0000","last_change":"2024-02-05T07:30:40.530544+0000","last_active":"2024-02-05T07:33:19.522539+0000","last_peered":"2024-02-05T07:33:19.522539+0000","last_clean":"2024-02-05T07:33:19.522539+0000","last_became_active":"2024-02-05T07:30:40.503903+0000","last_became_peered":"2024-02-05T07:30:40.503903+0000","last_unstale":"2024-02-05T07:33:19.522539+0000","last_undegraded":"2024-02-05T07:33:19.522539+0000","last_fullsized":"2024-02-05T07:33:19.522539+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:26:37.022129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:26:37.022129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":408,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.064981+0000","last_change":"2024-02-05T07:30:40.130268+0000","last_active":"2024-02-05T07:33:18.064981+0000","last_peered":"2024-02-05T07:33:18.064981+0000","last_clean":"2024-02-05T07:33:18.064981+0000","last_became_active":"2024-02-05T07:28:26.589671+0000","last_became_peered":"2024-02-05T07:28:26.589671+0000","last_unstale":"2024-02-05T07:33:18.064981+0000","last_undegraded":"2024-02-05T07:33:18.064981+0000","last_fullsized":"2024-02-05T07:33:18.064981+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.130170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:20.071010+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":584,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.525812+0000","last_change":"2024-02-05T07:30:43.523848+0000","last_active":"2024-02-05T07:33:09.525812+0000","last_peered":"2024-02-05T07:33:09.525812+0000","last_clean":"2024-02-05T07:33:09.525812+0000","last_became_active":"2024-02-05T07:30:43.523108+0000","last_became_peered":"2024-02-05T07:30:43.523108+0000","last_unstale":"2024-02-05T07:33:09.525812+0000","last_undegraded":"2024-02-05T07:33:09.525812+0000","last_fullsized":"2024-02-05T07:33:09.525812+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:48.594280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.594280+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383603+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"326'12","reported_seq":348,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.533673+0000","last_change":"2024-02-05T07:30:40.526400+0000","last_active":"2024-02-05T07:33:09.533673+0000","last_peered":"2024-02-05T07:33:09.533673+0000","last_clean":"2024-02-05T07:33:09.533673+0000","last_became_active":"2024-02-05T07:30:40.496980+0000","last_became_peered":"2024-02-05T07:30:40.496980+0000","last_unstale":"2024-02-05T07:33:09.533673+0000","last_undegraded":"2024-02-05T07:33:09.533673+0000","last_fullsized":"2024-02-05T07:33:09.533673+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-02-05T07:26:36.068507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:36.068507+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":552,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065006+0000","last_change":"2024-02-05T07:32:26.858476+0000","last_active":"2024-02-05T07:33:18.065006+0000","last_peered":"2024-02-05T07:33:18.065006+0000","last_clean":"2024-02-05T07:33:18.065006+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:33:18.065006+0000","last_undegraded":"2024-02-05T07:33:18.065006+0000","last_fullsized":"2024-02-05T07:33:18.065006+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:32:26.858391+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:32:26.858391+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:36.213638+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":328,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.530637+0000","last_change":"2024-02-05T07:30:27.965535+0000","last_active":"2024-02-05T07:33:09.530637+0000","last_peered":"2024-02-05T07:33:09.530637+0000","last_clean":"2024-02-05T07:33:09.530637+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:33:09.530637+0000","last_undegraded":"2024-02-05T07:33:09.530637+0000","last_fullsized":"2024-02-05T07:33:09.530637+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:27.965449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:27.965449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":610,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065637+0000","last_change":"2024-02-05T07:32:32.844370+0000","last_active":"2024-02-05T07:33:18.065637+0000","last_peered":"2024-02-05T07:33:18.065637+0000","last_clean":"2024-02-05T07:33:18.065637+0000","last_became_active":"2024-02-05T07:30:40.509216+0000","last_became_peered":"2024-02-05T07:30:40.509216+0000","last_unstale":"2024-02-05T07:33:18.065637+0000","last_undegraded":"2024-02-05T07:33:18.065637+0000","last_fullsized":"2024-02-05T07:33:18.065637+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:32.844264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:32:32.844264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:49.940142+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":718,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.525287+0000","last_change":"2024-02-05T07:28:45.702694+0000","last_active":"2024-02-05T07:33:09.525287+0000","last_peered":"2024-02-05T07:33:09.525287+0000","last_clean":"2024-02-05T07:33:09.525287+0000","last_became_active":"2024-02-05T07:28:19.311737+0000","last_became_peered":"2024-02-05T07:28:19.311737+0000","last_unstale":"2024-02-05T07:33:09.525287+0000","last_undegraded":"2024-02-05T07:33:09.525287+0000","last_fullsized":"2024-02-05T07:33:09.525287+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:45.702565+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.702565+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.702565+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383398+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"325'15","reported_seq":352,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.530878+0000","last_change":"2024-02-05T07:30:40.524523+0000","last_active":"2024-02-05T07:33:09.530878+0000","last_peered":"2024-02-05T07:33:09.530878+0000","last_clean":"2024-02-05T07:33:09.530878+0000","last_became_active":"2024-02-05T07:30:40.485837+0000","last_became_peered":"2024-02-05T07:30:40.485837+0000","last_unstale":"2024-02-05T07:33:09.530878+0000","last_undegraded":"2024-02-05T07:33:09.530878+0000","last_fullsized":"2024-02-05T07:33:09.530878+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-02-05T07:26:28.037945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:26:28.037945+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":625,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521329+0000","last_change":"2024-02-05T07:30:40.498263+0000","last_active":"2024-02-05T07:33:19.521329+0000","last_peered":"2024-02-05T07:33:19.521329+0000","last_clean":"2024-02-05T07:33:19.521329+0000","last_became_active":"2024-02-05T07:30:40.497956+0000","last_became_peered":"2024-02-05T07:30:40.497956+0000","last_unstale":"2024-02-05T07:33:19.521329+0000","last_undegraded":"2024-02-05T07:33:19.521329+0000","last_fullsized":"2024-02-05T07:33:19.521329+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:33.183427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:30:33.183427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":323,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.533769+0000","last_change":"2024-02-05T07:30:40.496550+0000","last_active":"2024-02-05T07:33:09.533769+0000","last_peered":"2024-02-05T07:33:09.533769+0000","last_clean":"2024-02-05T07:33:09.533769+0000","last_became_active":"2024-02-05T07:30:40.496076+0000","last_became_peered":"2024-02-05T07:30:40.496076+0000","last_unstale":"2024-02-05T07:33:09.533769+0000","last_undegraded":"2024-02-05T07:33:09.533769+0000","last_fullsized":"2024-02-05T07:33:09.533769+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:59.450191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:59.450191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":752,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.541257+0000","last_change":"2024-02-05T07:32:48.164374+0000","last_active":"2024-02-05T07:33:20.541257+0000","last_peered":"2024-02-05T07:33:20.541257+0000","last_clean":"2024-02-05T07:33:20.541257+0000","last_became_active":"2024-02-05T07:30:40.487455+0000","last_became_peered":"2024-02-05T07:30:40.487455+0000","last_unstale":"2024-02-05T07:33:20.541257+0000","last_undegraded":"2024-02-05T07:33:20.541257+0000","last_fullsized":"2024-02-05T07:33:20.541257+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:48.164259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:32:48.164259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:50.898675+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":432,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.064414+0000","last_change":"2024-02-05T07:32:18.784193+0000","last_active":"2024-02-05T07:33:18.064414+0000","last_peered":"2024-02-05T07:33:18.064414+0000","last_clean":"2024-02-05T07:33:18.064414+0000","last_became_active":"2024-02-05T07:30:43.466854+0000","last_became_peered":"2024-02-05T07:30:43.466854+0000","last_unstale":"2024-02-05T07:33:18.064414+0000","last_undegraded":"2024-02-05T07:33:18.064414+0000","last_fullsized":"2024-02-05T07:33:18.064414+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:18.784071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:18.784071+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:42.865761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"326'19","reported_seq":680,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.534143+0000","last_change":"2024-02-05T07:30:40.503823+0000","last_active":"2024-02-05T07:33:09.534143+0000","last_peered":"2024-02-05T07:33:09.534143+0000","last_clean":"2024-02-05T07:33:09.534143+0000","last_became_active":"2024-02-05T07:30:40.483700+0000","last_became_peered":"2024-02-05T07:30:40.483700+0000","last_unstale":"2024-02-05T07:33:09.534143+0000","last_undegraded":"2024-02-05T07:33:09.534143+0000","last_fullsized":"2024-02-05T07:33:09.534143+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:30:35.138361+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:30:35.138361+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":494,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.534682+0000","last_change":"2024-02-05T07:30:40.476394+0000","last_active":"2024-02-05T07:33:09.534682+0000","last_peered":"2024-02-05T07:33:09.534682+0000","last_clean":"2024-02-05T07:33:09.534682+0000","last_became_active":"2024-02-05T07:30:40.476058+0000","last_became_peered":"2024-02-05T07:30:40.476058+0000","last_unstale":"2024-02-05T07:33:09.534682+0000","last_undegraded":"2024-02-05T07:33:09.534682+0000","last_fullsized":"2024-02-05T07:33:09.534682+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.314735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.314735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"365'11","reported_seq":403,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.642643+0000","last_change":"2024-02-05T07:30:40.517194+0000","last_active":"2024-02-05T07:33:09.642643+0000","last_peered":"2024-02-05T07:33:09.642643+0000","last_clean":"2024-02-05T07:33:09.642643+0000","last_became_active":"2024-02-05T07:30:40.495708+0000","last_became_peered":"2024-02-05T07:30:40.495708+0000","last_unstale":"2024-02-05T07:33:09.642643+0000","last_undegraded":"2024-02-05T07:33:09.642643+0000","last_fullsized":"2024-02-05T07:33:09.642643+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'9","last_scrub_stamp":"2024-02-05T07:26:35.091894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.091894+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:15.746210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":733,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522628+0000","last_change":"2024-02-05T07:27:03.366653+0000","last_active":"2024-02-05T07:33:19.522628+0000","last_peered":"2024-02-05T07:33:19.522628+0000","last_clean":"2024-02-05T07:33:19.522628+0000","last_became_active":"2024-02-05T07:27:03.364334+0000","last_became_peered":"2024-02-05T07:27:03.364334+0000","last_unstale":"2024-02-05T07:33:19.522628+0000","last_undegraded":"2024-02-05T07:33:19.522628+0000","last_fullsized":"2024-02-05T07:33:19.522628+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:25.323384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:26:25.323384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":575,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.539719+0000","last_change":"2024-02-05T07:28:41.013449+0000","last_active":"2024-02-05T07:33:09.539719+0000","last_peered":"2024-02-05T07:33:09.539719+0000","last_clean":"2024-02-05T07:33:09.539719+0000","last_became_active":"2024-02-05T07:28:19.302393+0000","last_became_peered":"2024-02-05T07:28:19.302393+0000","last_unstale":"2024-02-05T07:33:09.539719+0000","last_undegraded":"2024-02-05T07:33:09.539719+0000","last_fullsized":"2024-02-05T07:33:09.539719+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:41.013360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:41.013360+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"326'25","reported_seq":370,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.540323+0000","last_change":"2024-02-05T07:30:40.512630+0000","last_active":"2024-02-05T07:33:09.540323+0000","last_peered":"2024-02-05T07:33:09.540323+0000","last_clean":"2024-02-05T07:33:09.540323+0000","last_became_active":"2024-02-05T07:30:40.486394+0000","last_became_peered":"2024-02-05T07:30:40.486394+0000","last_unstale":"2024-02-05T07:33:09.540323+0000","last_undegraded":"2024-02-05T07:33:09.540323+0000","last_fullsized":"2024-02-05T07:33:09.540323+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:30:33.959124+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:30:33.959124+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":119,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521712+0000","last_change":"2024-02-05T07:30:40.505910+0000","last_active":"2024-02-05T07:33:19.521712+0000","last_peered":"2024-02-05T07:33:19.521712+0000","last_clean":"2024-02-05T07:33:19.521712+0000","last_became_active":"2024-02-05T07:30:40.505600+0000","last_became_peered":"2024-02-05T07:30:40.505600+0000","last_unstale":"2024-02-05T07:33:19.521712+0000","last_undegraded":"2024-02-05T07:33:19.521712+0000","last_fullsized":"2024-02-05T07:33:19.521712+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:46.210169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:46.210169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"217'9","reported_seq":545,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.539556+0000","last_change":"2024-02-05T07:30:27.993196+0000","last_active":"2024-02-05T07:33:20.539556+0000","last_peered":"2024-02-05T07:33:20.539556+0000","last_clean":"2024-02-05T07:33:20.539556+0000","last_became_active":"2024-02-05T07:25:27.246616+0000","last_became_peered":"2024-02-05T07:25:27.246616+0000","last_unstale":"2024-02-05T07:33:20.539556+0000","last_undegraded":"2024-02-05T07:33:20.539556+0000","last_fullsized":"2024-02-05T07:33:20.539556+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9","last_scrub_stamp":"2024-02-05T07:30:27.993000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:27.993000+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":482,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.530782+0000","last_change":"2024-02-05T07:30:40.508206+0000","last_active":"2024-02-05T07:33:09.530782+0000","last_peered":"2024-02-05T07:33:09.530782+0000","last_clean":"2024-02-05T07:33:09.530782+0000","last_became_active":"2024-02-05T07:30:40.507831+0000","last_became_peered":"2024-02-05T07:30:40.507831+0000","last_unstale":"2024-02-05T07:33:09.530782+0000","last_undegraded":"2024-02-05T07:33:09.530782+0000","last_fullsized":"2024-02-05T07:33:09.530782+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:27:01.069970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:27:01.069970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.16","version":"86'128","reported_seq":386,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.526198+0000","last_change":"2024-02-05T07:32:42.321154+0000","last_active":"2024-02-05T07:33:09.526198+0000","last_peered":"2024-02-05T07:33:09.526198+0000","last_clean":"2024-02-05T07:33:09.526198+0000","last_became_active":"2024-02-05T07:31:41.837877+0000","last_became_peered":"2024-02-05T07:31:41.837877+0000","last_unstale":"2024-02-05T07:33:09.526198+0000","last_undegraded":"2024-02-05T07:33:09.526198+0000","last_fullsized":"2024-02-05T07:33:09.526198+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:42.321045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:42.321045+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:54.490443+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"326'19","reported_seq":802,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521709+0000","last_change":"2024-02-05T07:32:31.667966+0000","last_active":"2024-02-05T07:33:19.521709+0000","last_peered":"2024-02-05T07:33:19.521709+0000","last_clean":"2024-02-05T07:33:19.521709+0000","last_became_active":"2024-02-05T07:28:25.779886+0000","last_became_peered":"2024-02-05T07:28:25.779886+0000","last_unstale":"2024-02-05T07:33:19.521709+0000","last_undegraded":"2024-02-05T07:33:19.521709+0000","last_fullsized":"2024-02-05T07:33:19.521709+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:32:31.667882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:31.667882+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:45.586856+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":755,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521907+0000","last_change":"2024-02-05T07:30:40.510773+0000","last_active":"2024-02-05T07:33:19.521907+0000","last_peered":"2024-02-05T07:33:19.521907+0000","last_clean":"2024-02-05T07:33:19.521907+0000","last_became_active":"2024-02-05T07:30:40.510301+0000","last_became_peered":"2024-02-05T07:30:40.510301+0000","last_unstale":"2024-02-05T07:33:19.521907+0000","last_undegraded":"2024-02-05T07:33:19.521907+0000","last_fullsized":"2024-02-05T07:33:19.521907+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:36.798282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:29:36.798282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":759,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.539630+0000","last_change":"2024-02-05T07:32:48.346195+0000","last_active":"2024-02-05T07:33:09.539630+0000","last_peered":"2024-02-05T07:33:09.539630+0000","last_clean":"2024-02-05T07:33:09.539630+0000","last_became_active":"2024-02-05T07:30:40.487141+0000","last_became_peered":"2024-02-05T07:30:40.487141+0000","last_unstale":"2024-02-05T07:33:09.539630+0000","last_undegraded":"2024-02-05T07:33:09.539630+0000","last_fullsized":"2024-02-05T07:33:09.539630+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_clean_scrub_stamp":"2024-02-05T07:32:48.345965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:06.833776+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":315,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.540224+0000","last_change":"2024-02-05T07:27:17.708497+0000","last_active":"2024-02-05T07:33:09.540224+0000","last_peered":"2024-02-05T07:33:09.540224+0000","last_clean":"2024-02-05T07:33:09.540224+0000","last_became_active":"2024-02-05T07:27:17.705910+0000","last_became_peered":"2024-02-05T07:27:17.705910+0000","last_unstale":"2024-02-05T07:33:09.540224+0000","last_undegraded":"2024-02-05T07:33:09.540224+0000","last_fullsized":"2024-02-05T07:33:09.540224+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:26:35.120512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"86'128","reported_seq":603,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.064293+0000","last_change":"2024-02-05T07:32:23.818975+0000","last_active":"2024-02-05T07:33:18.064293+0000","last_peered":"2024-02-05T07:33:18.064293+0000","last_clean":"2024-02-05T07:33:18.064293+0000","last_became_active":"2024-02-05T07:31:41.837612+0000","last_became_peered":"2024-02-05T07:31:41.837612+0000","last_unstale":"2024-02-05T07:33:18.064293+0000","last_undegraded":"2024-02-05T07:33:18.064293+0000","last_fullsized":"2024-02-05T07:33:18.064293+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:23.818901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:23.818901+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:47.869222+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"325'15","reported_seq":364,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.525707+0000","last_change":"2024-02-05T07:29:57.775013+0000","last_active":"2024-02-05T07:33:09.525707+0000","last_peered":"2024-02-05T07:33:09.525707+0000","last_clean":"2024-02-05T07:33:09.525707+0000","last_became_active":"2024-02-05T07:28:26.589705+0000","last_became_peered":"2024-02-05T07:28:26.589705+0000","last_unstale":"2024-02-05T07:33:09.525707+0000","last_undegraded":"2024-02-05T07:33:09.525707+0000","last_fullsized":"2024-02-05T07:33:09.525707+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:29:57.774913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:57.774913+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":539,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.534063+0000","last_change":"2024-02-05T07:30:40.489183+0000","last_active":"2024-02-05T07:33:09.534063+0000","last_peered":"2024-02-05T07:33:09.534063+0000","last_clean":"2024-02-05T07:33:09.534063+0000","last_became_active":"2024-02-05T07:30:40.488789+0000","last_became_peered":"2024-02-05T07:30:40.488789+0000","last_unstale":"2024-02-05T07:33:09.534063+0000","last_undegraded":"2024-02-05T07:33:09.534063+0000","last_fullsized":"2024-02-05T07:33:09.534063+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.220841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.220841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":317,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.538917+0000","last_change":"2024-02-05T07:27:16.765064+0000","last_active":"2024-02-05T07:33:09.538917+0000","last_peered":"2024-02-05T07:33:09.538917+0000","last_clean":"2024-02-05T07:33:09.538917+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:33:09.538917+0000","last_undegraded":"2024-02-05T07:33:09.538917+0000","last_fullsized":"2024-02-05T07:33:09.538917+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:26:31.506641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":491,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.527398+0000","last_change":"2024-02-05T07:28:26.588019+0000","last_active":"2024-02-05T07:33:09.527398+0000","last_peered":"2024-02-05T07:33:09.527398+0000","last_clean":"2024-02-05T07:33:09.527398+0000","last_became_active":"2024-02-05T07:28:26.587701+0000","last_became_peered":"2024-02-05T07:28:26.587701+0000","last_unstale":"2024-02-05T07:33:09.527398+0000","last_undegraded":"2024-02-05T07:33:09.527398+0000","last_fullsized":"2024-02-05T07:33:09.527398+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:37.616737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:26:37.616737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383312+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":992,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.540706+0000","last_change":"2024-02-05T07:30:43.515841+0000","last_active":"2024-02-05T07:33:20.540706+0000","last_peered":"2024-02-05T07:33:20.540706+0000","last_clean":"2024-02-05T07:33:20.540706+0000","last_became_active":"2024-02-05T07:30:43.515488+0000","last_became_peered":"2024-02-05T07:30:43.515488+0000","last_unstale":"2024-02-05T07:33:20.540706+0000","last_undegraded":"2024-02-05T07:33:20.540706+0000","last_fullsized":"2024-02-05T07:33:20.540706+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:30:36.995330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.995330+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"326'19","reported_seq":370,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.540446+0000","last_change":"2024-02-05T07:30:40.906805+0000","last_active":"2024-02-05T07:33:09.540446+0000","last_peered":"2024-02-05T07:33:09.540446+0000","last_clean":"2024-02-05T07:33:09.540446+0000","last_became_active":"2024-02-05T07:30:40.501893+0000","last_became_peered":"2024-02-05T07:30:40.501893+0000","last_unstale":"2024-02-05T07:33:09.540446+0000","last_undegraded":"2024-02-05T07:33:09.540446+0000","last_fullsized":"2024-02-05T07:33:09.540446+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:30:40.906693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.906693+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":140,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.531849+0000","last_change":"2024-02-05T07:30:40.492005+0000","last_active":"2024-02-05T07:33:09.531849+0000","last_peered":"2024-02-05T07:33:09.531849+0000","last_clean":"2024-02-05T07:33:09.531849+0000","last_became_active":"2024-02-05T07:30:40.491686+0000","last_became_peered":"2024-02-05T07:30:40.491686+0000","last_unstale":"2024-02-05T07:33:09.531849+0000","last_undegraded":"2024-02-05T07:33:09.531849+0000","last_fullsized":"2024-02-05T07:33:09.531849+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.097532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.097532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.384272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":617,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.520916+0000","last_change":"2024-02-05T07:30:40.497509+0000","last_active":"2024-02-05T07:33:19.520916+0000","last_peered":"2024-02-05T07:33:19.520916+0000","last_clean":"2024-02-05T07:33:19.520916+0000","last_became_active":"2024-02-05T07:30:40.497190+0000","last_became_peered":"2024-02-05T07:30:40.497190+0000","last_unstale":"2024-02-05T07:33:19.520916+0000","last_undegraded":"2024-02-05T07:33:19.520916+0000","last_fullsized":"2024-02-05T07:33:19.520916+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:29:58.191815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":329,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.529990+0000","last_change":"2024-02-05T07:29:40.983327+0000","last_active":"2024-02-05T07:33:09.529990+0000","last_peered":"2024-02-05T07:33:09.529990+0000","last_clean":"2024-02-05T07:33:09.529990+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:33:09.529990+0000","last_undegraded":"2024-02-05T07:33:09.529990+0000","last_fullsized":"2024-02-05T07:33:09.529990+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:40.983227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.983227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"86'128","reported_seq":380,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.526208+0000","last_change":"2024-02-05T07:32:43.318421+0000","last_active":"2024-02-05T07:33:09.526208+0000","last_peered":"2024-02-05T07:33:09.526208+0000","last_clean":"2024-02-05T07:33:09.526208+0000","last_became_active":"2024-02-05T07:31:41.843850+0000","last_became_peered":"2024-02-05T07:31:41.843850+0000","last_unstale":"2024-02-05T07:33:09.526208+0000","last_undegraded":"2024-02-05T07:33:09.526208+0000","last_fullsized":"2024-02-05T07:33:09.526208+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:43.318297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:43.318297+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.720359+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"325'8","reported_seq":341,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.531301+0000","last_change":"2024-02-05T07:29:53.958469+0000","last_active":"2024-02-05T07:33:09.531301+0000","last_peered":"2024-02-05T07:33:09.531301+0000","last_clean":"2024-02-05T07:33:09.531301+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:33:09.531301+0000","last_undegraded":"2024-02-05T07:33:09.531301+0000","last_fullsized":"2024-02-05T07:33:09.531301+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:29:53.958384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.958384+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":328,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.531871+0000","last_change":"2024-02-05T07:28:46.056835+0000","last_active":"2024-02-05T07:33:09.531871+0000","last_peered":"2024-02-05T07:33:09.531871+0000","last_clean":"2024-02-05T07:33:09.531871+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:33:09.531871+0000","last_undegraded":"2024-02-05T07:33:09.531871+0000","last_fullsized":"2024-02-05T07:33:09.531871+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:46.056702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:28:46.056702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":586,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.064670+0000","last_change":"2024-02-05T07:30:40.500892+0000","last_active":"2024-02-05T07:33:18.064670+0000","last_peered":"2024-02-05T07:33:18.064670+0000","last_clean":"2024-02-05T07:33:18.064670+0000","last_became_active":"2024-02-05T07:30:40.500403+0000","last_became_peered":"2024-02-05T07:30:40.500403+0000","last_unstale":"2024-02-05T07:33:18.064670+0000","last_undegraded":"2024-02-05T07:33:18.064670+0000","last_fullsized":"2024-02-05T07:33:18.064670+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:58.034570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:29:58.034570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:20.070947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":130,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.541490+0000","last_change":"2024-02-05T07:30:45.926725+0000","last_active":"2024-02-05T07:33:20.541490+0000","last_peered":"2024-02-05T07:33:20.541490+0000","last_clean":"2024-02-05T07:33:20.541490+0000","last_became_active":"2024-02-05T07:30:40.489230+0000","last_became_peered":"2024-02-05T07:30:40.489230+0000","last_unstale":"2024-02-05T07:33:20.541490+0000","last_undegraded":"2024-02-05T07:33:20.541490+0000","last_fullsized":"2024-02-05T07:33:20.541490+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:45.926395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:30:45.926395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1a","version":"86'128","reported_seq":456,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521028+0000","last_change":"2024-02-05T07:32:25.652816+0000","last_active":"2024-02-05T07:33:19.521028+0000","last_peered":"2024-02-05T07:33:19.521028+0000","last_clean":"2024-02-05T07:33:19.521028+0000","last_became_active":"2024-02-05T07:31:41.820206+0000","last_became_peered":"2024-02-05T07:31:41.820206+0000","last_unstale":"2024-02-05T07:33:19.521028+0000","last_undegraded":"2024-02-05T07:33:19.521028+0000","last_fullsized":"2024-02-05T07:33:19.521028+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:25.652696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:25.652696+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:38.772356+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"326'19","reported_seq":580,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065777+0000","last_change":"2024-02-05T07:32:43.787447+0000","last_active":"2024-02-05T07:33:18.065777+0000","last_peered":"2024-02-05T07:33:18.065777+0000","last_clean":"2024-02-05T07:33:18.065777+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:33:18.065777+0000","last_undegraded":"2024-02-05T07:33:18.065777+0000","last_fullsized":"2024-02-05T07:33:18.065777+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:32:43.787327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:43.787327+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:13.482809+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":325,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.529764+0000","last_change":"2024-02-05T07:29:38.000417+0000","last_active":"2024-02-05T07:33:09.529764+0000","last_peered":"2024-02-05T07:33:09.529764+0000","last_clean":"2024-02-05T07:33:09.529764+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:33:09.529764+0000","last_undegraded":"2024-02-05T07:33:09.529764+0000","last_fullsized":"2024-02-05T07:33:09.529764+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:38.000345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:38.000345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":123,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.541350+0000","last_change":"2024-02-05T07:30:40.480962+0000","last_active":"2024-02-05T07:33:20.541350+0000","last_peered":"2024-02-05T07:33:20.541350+0000","last_clean":"2024-02-05T07:33:20.541350+0000","last_became_active":"2024-02-05T07:30:40.479162+0000","last_became_peered":"2024-02-05T07:30:40.479162+0000","last_unstale":"2024-02-05T07:33:20.541350+0000","last_undegraded":"2024-02-05T07:33:20.541350+0000","last_fullsized":"2024-02-05T07:33:20.541350+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:28.153881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:28.153881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":746,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.528659+0000","last_change":"2024-02-05T07:29:55.800941+0000","last_active":"2024-02-05T07:33:09.528659+0000","last_peered":"2024-02-05T07:33:09.528659+0000","last_clean":"2024-02-05T07:33:09.528659+0000","last_became_active":"2024-02-05T07:27:03.360468+0000","last_became_peered":"2024-02-05T07:27:03.360468+0000","last_unstale":"2024-02-05T07:33:09.528659+0000","last_undegraded":"2024-02-05T07:33:09.528659+0000","last_fullsized":"2024-02-05T07:33:09.528659+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:55.800824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.800824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.384065+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1b","version":"86'128","reported_seq":531,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521330+0000","last_change":"2024-02-05T07:32:26.685749+0000","last_active":"2024-02-05T07:33:19.521330+0000","last_peered":"2024-02-05T07:33:19.521330+0000","last_clean":"2024-02-05T07:33:19.521330+0000","last_became_active":"2024-02-05T07:31:41.823891+0000","last_became_peered":"2024-02-05T07:31:41.823891+0000","last_unstale":"2024-02-05T07:33:19.521330+0000","last_undegraded":"2024-02-05T07:33:19.521330+0000","last_fullsized":"2024-02-05T07:33:19.521330+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:26.685655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:26.685655+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:39.162396+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"326'20","reported_seq":492,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.530522+0000","last_change":"2024-02-05T07:30:40.534624+0000","last_active":"2024-02-05T07:33:09.530522+0000","last_peered":"2024-02-05T07:33:09.530522+0000","last_clean":"2024-02-05T07:33:09.530522+0000","last_became_active":"2024-02-05T07:30:40.495954+0000","last_became_peered":"2024-02-05T07:30:40.495954+0000","last_unstale":"2024-02-05T07:33:09.530522+0000","last_undegraded":"2024-02-05T07:33:09.530522+0000","last_fullsized":"2024-02-05T07:33:09.530522+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:29:40.154854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.154854+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":788,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.525959+0000","last_change":"2024-02-05T07:30:40.526955+0000","last_active":"2024-02-05T07:33:09.525959+0000","last_peered":"2024-02-05T07:33:09.525959+0000","last_clean":"2024-02-05T07:33:09.525959+0000","last_became_active":"2024-02-05T07:30:40.495135+0000","last_became_peered":"2024-02-05T07:30:40.495135+0000","last_unstale":"2024-02-05T07:33:09.525959+0000","last_undegraded":"2024-02-05T07:33:09.525959+0000","last_fullsized":"2024-02-05T07:33:09.525959+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:52.715259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:52.715259+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383745+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"365'11","reported_seq":742,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.065844+0000","last_change":"2024-02-05T07:30:40.516172+0000","last_active":"2024-02-05T07:33:18.065844+0000","last_peered":"2024-02-05T07:33:18.065844+0000","last_clean":"2024-02-05T07:33:18.065844+0000","last_became_active":"2024-02-05T07:30:40.497371+0000","last_became_peered":"2024-02-05T07:30:40.497371+0000","last_unstale":"2024-02-05T07:33:18.065844+0000","last_undegraded":"2024-02-05T07:33:18.065844+0000","last_fullsized":"2024-02-05T07:33:18.065844+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'10","last_scrub_stamp":"2024-02-05T07:29:55.058040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.058040+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:20.070981+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":240,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.540823+0000","last_change":"2024-02-05T07:30:41.950393+0000","last_active":"2024-02-05T07:33:20.540823+0000","last_peered":"2024-02-05T07:33:20.540823+0000","last_clean":"2024-02-05T07:33:20.540823+0000","last_became_active":"2024-02-05T07:30:40.486920+0000","last_became_peered":"2024-02-05T07:30:40.486920+0000","last_unstale":"2024-02-05T07:33:20.540823+0000","last_undegraded":"2024-02-05T07:33:20.540823+0000","last_fullsized":"2024-02-05T07:33:20.540823+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:30:41.950253+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:30:41.950253+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"86'128","reported_seq":779,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.538790+0000","last_change":"2024-02-05T07:32:42.278242+0000","last_active":"2024-02-05T07:33:20.538790+0000","last_peered":"2024-02-05T07:33:20.538790+0000","last_clean":"2024-02-05T07:33:20.538790+0000","last_became_active":"2024-02-05T07:32:01.562409+0000","last_became_peered":"2024-02-05T07:32:01.562409+0000","last_unstale":"2024-02-05T07:33:20.538790+0000","last_undegraded":"2024-02-05T07:33:20.538790+0000","last_fullsized":"2024-02-05T07:33:20.538790+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:42.278112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:42.278112+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:03.855341+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"326'23","reported_seq":351,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.533824+0000","last_change":"2024-02-05T07:28:47.062765+0000","last_active":"2024-02-05T07:33:09.533824+0000","last_peered":"2024-02-05T07:33:09.533824+0000","last_clean":"2024-02-05T07:33:09.533824+0000","last_became_active":"2024-02-05T07:27:16.747998+0000","last_became_peered":"2024-02-05T07:27:16.747998+0000","last_unstale":"2024-02-05T07:33:09.533824+0000","last_undegraded":"2024-02-05T07:33:09.533824+0000","last_fullsized":"2024-02-05T07:33:09.533824+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'17","last_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.062618+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":427,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.539498+0000","last_change":"2024-02-05T07:30:39.874170+0000","last_active":"2024-02-05T07:33:09.539498+0000","last_peered":"2024-02-05T07:33:09.539498+0000","last_clean":"2024-02-05T07:33:09.539498+0000","last_became_active":"2024-02-05T07:25:30.261037+0000","last_became_peered":"2024-02-05T07:25:30.261037+0000","last_unstale":"2024-02-05T07:33:09.539498+0000","last_undegraded":"2024-02-05T07:33:09.539498+0000","last_fullsized":"2024-02-05T07:33:09.539498+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.874081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.384498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":754,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522105+0000","last_change":"2024-02-05T07:30:26.669849+0000","last_active":"2024-02-05T07:33:19.522105+0000","last_peered":"2024-02-05T07:33:19.522105+0000","last_clean":"2024-02-05T07:33:19.522105+0000","last_became_active":"2024-02-05T07:27:16.706777+0000","last_became_peered":"2024-02-05T07:27:16.706777+0000","last_unstale":"2024-02-05T07:33:19.522105+0000","last_undegraded":"2024-02-05T07:33:19.522105+0000","last_fullsized":"2024-02-05T07:33:19.522105+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:26.669576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:26.669576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":328,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.533575+0000","last_change":"2024-02-05T07:28:53.138071+0000","last_active":"2024-02-05T07:33:09.533575+0000","last_peered":"2024-02-05T07:33:09.533575+0000","last_clean":"2024-02-05T07:33:09.533575+0000","last_became_active":"2024-02-05T07:28:26.603067+0000","last_became_peered":"2024-02-05T07:28:26.603067+0000","last_unstale":"2024-02-05T07:33:09.533575+0000","last_undegraded":"2024-02-05T07:33:09.533575+0000","last_fullsized":"2024-02-05T07:33:09.533575+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:53.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:53.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.19","version":"86'128","reported_seq":848,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.535079+0000","last_change":"2024-02-05T07:32:44.746470+0000","last_active":"2024-02-05T07:33:09.535079+0000","last_peered":"2024-02-05T07:33:09.535079+0000","last_clean":"2024-02-05T07:33:09.535079+0000","last_became_active":"2024-02-05T07:31:41.858087+0000","last_became_peered":"2024-02-05T07:31:41.858087+0000","last_unstale":"2024-02-05T07:33:09.535079+0000","last_undegraded":"2024-02-05T07:33:09.535079+0000","last_fullsized":"2024-02-05T07:33:09.535079+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:44.746349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:44.746349+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:59.248511+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"326'27","reported_seq":832,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.540995+0000","last_change":"2024-02-05T07:29:41.354568+0000","last_active":"2024-02-05T07:33:20.540995+0000","last_peered":"2024-02-05T07:33:20.540995+0000","last_clean":"2024-02-05T07:33:20.540995+0000","last_became_active":"2024-02-05T07:27:16.736296+0000","last_became_peered":"2024-02-05T07:27:16.736296+0000","last_unstale":"2024-02-05T07:33:20.540995+0000","last_undegraded":"2024-02-05T07:33:20.540995+0000","last_fullsized":"2024-02-05T07:33:20.540995+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:29:41.354464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:41.354464+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":499,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521585+0000","last_change":"2024-02-05T07:30:40.503947+0000","last_active":"2024-02-05T07:33:19.521585+0000","last_peered":"2024-02-05T07:33:19.521585+0000","last_clean":"2024-02-05T07:33:19.521585+0000","last_became_active":"2024-02-05T07:30:40.503627+0000","last_became_peered":"2024-02-05T07:30:40.503627+0000","last_unstale":"2024-02-05T07:33:19.521585+0000","last_undegraded":"2024-02-05T07:33:19.521585+0000","last_fullsized":"2024-02-05T07:33:19.521585+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:34.663987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:34.663987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":736,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.528511+0000","last_change":"2024-02-05T07:30:40.884273+0000","last_active":"2024-02-05T07:33:09.528511+0000","last_peered":"2024-02-05T07:33:09.528511+0000","last_clean":"2024-02-05T07:33:09.528511+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:33:09.528511+0000","last_undegraded":"2024-02-05T07:33:09.528511+0000","last_fullsized":"2024-02-05T07:33:09.528511+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:40.884197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.884197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.384242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":754,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522601+0000","last_change":"2024-02-05T07:29:59.002788+0000","last_active":"2024-02-05T07:33:19.522601+0000","last_peered":"2024-02-05T07:33:19.522601+0000","last_clean":"2024-02-05T07:33:19.522601+0000","last_became_active":"2024-02-05T07:28:26.590448+0000","last_became_peered":"2024-02-05T07:28:26.590448+0000","last_unstale":"2024-02-05T07:33:19.522601+0000","last_undegraded":"2024-02-05T07:33:19.522601+0000","last_fullsized":"2024-02-05T07:33:19.522601+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.002677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.002677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"86'128","reported_seq":448,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.530287+0000","last_change":"2024-02-05T07:32:43.758495+0000","last_active":"2024-02-05T07:33:09.530287+0000","last_peered":"2024-02-05T07:33:09.530287+0000","last_clean":"2024-02-05T07:33:09.530287+0000","last_became_active":"2024-02-05T07:31:41.834645+0000","last_became_peered":"2024-02-05T07:31:41.834645+0000","last_unstale":"2024-02-05T07:33:09.530287+0000","last_undegraded":"2024-02-05T07:33:09.530287+0000","last_fullsized":"2024-02-05T07:33:09.530287+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:43.758385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:43.758385+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:45.089271+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"325'7","reported_seq":780,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521580+0000","last_change":"2024-02-05T07:32:40.597579+0000","last_active":"2024-02-05T07:33:19.521580+0000","last_peered":"2024-02-05T07:33:19.521580+0000","last_clean":"2024-02-05T07:33:19.521580+0000","last_became_active":"2024-02-05T07:27:16.745897+0000","last_became_peered":"2024-02-05T07:27:16.745897+0000","last_unstale":"2024-02-05T07:33:19.521580+0000","last_undegraded":"2024-02-05T07:33:19.521580+0000","last_fullsized":"2024-02-05T07:33:19.521580+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'7","last_scrub_stamp":"2024-02-05T07:32:40.597479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:40.597479+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:52.852342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":571,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521852+0000","last_change":"2024-02-05T07:30:39.726603+0000","last_active":"2024-02-05T07:33:19.521852+0000","last_peered":"2024-02-05T07:33:19.521852+0000","last_clean":"2024-02-05T07:33:19.521852+0000","last_became_active":"2024-02-05T07:28:26.590322+0000","last_became_peered":"2024-02-05T07:28:26.590322+0000","last_unstale":"2024-02-05T07:33:19.521852+0000","last_undegraded":"2024-02-05T07:33:19.521852+0000","last_fullsized":"2024-02-05T07:33:19.521852+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.726525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.726525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":119,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521686+0000","last_change":"2024-02-05T07:30:40.504663+0000","last_active":"2024-02-05T07:33:19.521686+0000","last_peered":"2024-02-05T07:33:19.521686+0000","last_clean":"2024-02-05T07:33:19.521686+0000","last_became_active":"2024-02-05T07:30:40.504365+0000","last_became_peered":"2024-02-05T07:30:40.504365+0000","last_unstale":"2024-02-05T07:33:19.521686+0000","last_undegraded":"2024-02-05T07:33:19.521686+0000","last_fullsized":"2024-02-05T07:33:19.521686+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:27:00.183281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:27:00.183281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":543,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.526089+0000","last_change":"2024-02-05T07:29:50.742042+0000","last_active":"2024-02-05T07:33:09.526089+0000","last_peered":"2024-02-05T07:33:09.526089+0000","last_clean":"2024-02-05T07:33:09.526089+0000","last_became_active":"2024-02-05T07:27:16.769581+0000","last_became_peered":"2024-02-05T07:27:16.769581+0000","last_unstale":"2024-02-05T07:33:09.526089+0000","last_undegraded":"2024-02-05T07:33:09.526089+0000","last_fullsized":"2024-02-05T07:33:09.526089+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:50.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.384091+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"325'19","reported_seq":804,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521537+0000","last_change":"2024-02-05T07:30:00.001976+0000","last_active":"2024-02-05T07:33:19.521537+0000","last_peered":"2024-02-05T07:33:19.521537+0000","last_clean":"2024-02-05T07:33:19.521537+0000","last_became_active":"2024-02-05T07:27:16.707036+0000","last_became_peered":"2024-02-05T07:27:16.707036+0000","last_unstale":"2024-02-05T07:33:19.521537+0000","last_undegraded":"2024-02-05T07:33:19.521537+0000","last_fullsized":"2024-02-05T07:33:19.521537+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'19","last_scrub_stamp":"2024-02-05T07:30:00.001863+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:30:00.001863+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":574,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.522490+0000","last_change":"2024-02-05T07:29:53.022701+0000","last_active":"2024-02-05T07:33:19.522490+0000","last_peered":"2024-02-05T07:33:19.522490+0000","last_clean":"2024-02-05T07:33:19.522490+0000","last_became_active":"2024-02-05T07:28:26.616107+0000","last_became_peered":"2024-02-05T07:28:26.616107+0000","last_unstale":"2024-02-05T07:33:19.522490+0000","last_undegraded":"2024-02-05T07:33:19.522490+0000","last_fullsized":"2024-02-05T07:33:19.522490+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.022606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.022606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":121,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521454+0000","last_change":"2024-02-05T07:30:40.491944+0000","last_active":"2024-02-05T07:33:19.521454+0000","last_peered":"2024-02-05T07:33:19.521454+0000","last_clean":"2024-02-05T07:33:19.521454+0000","last_became_active":"2024-02-05T07:30:40.491630+0000","last_became_peered":"2024-02-05T07:30:40.491630+0000","last_unstale":"2024-02-05T07:33:19.521454+0000","last_undegraded":"2024-02-05T07:33:19.521454+0000","last_fullsized":"2024-02-05T07:33:19.521454+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:54.207089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:54.207089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":495,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.527269+0000","last_change":"2024-02-05T07:30:40.476139+0000","last_active":"2024-02-05T07:33:09.527269+0000","last_peered":"2024-02-05T07:33:09.527269+0000","last_clean":"2024-02-05T07:33:09.527269+0000","last_became_active":"2024-02-05T07:30:40.475535+0000","last_became_peered":"2024-02-05T07:30:40.475535+0000","last_unstale":"2024-02-05T07:33:09.527269+0000","last_undegraded":"2024-02-05T07:33:09.527269+0000","last_fullsized":"2024-02-05T07:33:09.527269+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:31.484580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:31.484580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383630+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"86'128","reported_seq":386,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-02-05T07:33:18.064277+0000","last_change":"2024-02-05T07:32:22.866491+0000","last_active":"2024-02-05T07:33:18.064277+0000","last_peered":"2024-02-05T07:33:18.064277+0000","last_clean":"2024-02-05T07:33:18.064277+0000","last_became_active":"2024-02-05T07:31:41.837502+0000","last_became_peered":"2024-02-05T07:31:41.837502+0000","last_unstale":"2024-02-05T07:33:18.064277+0000","last_undegraded":"2024-02-05T07:33:18.064277+0000","last_fullsized":"2024-02-05T07:33:18.064277+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:22.866405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:22.866405+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:51.348616+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"326'27","reported_seq":351,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.540655+0000","last_change":"2024-02-05T07:27:17.002461+0000","last_active":"2024-02-05T07:33:09.540655+0000","last_peered":"2024-02-05T07:33:09.540655+0000","last_clean":"2024-02-05T07:33:09.540655+0000","last_became_active":"2024-02-05T07:27:16.751179+0000","last_became_peered":"2024-02-05T07:27:16.751179+0000","last_unstale":"2024-02-05T07:33:09.540655+0000","last_undegraded":"2024-02-05T07:33:09.540655+0000","last_fullsized":"2024-02-05T07:33:09.540655+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-02-05T07:25:38.772812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:25:38.772812+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":329,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.531198+0000","last_change":"2024-02-05T07:32:49.743375+0000","last_active":"2024-02-05T07:33:09.531198+0000","last_peered":"2024-02-05T07:33:09.531198+0000","last_clean":"2024-02-05T07:33:09.531198+0000","last_became_active":"2024-02-05T07:28:26.595627+0000","last_became_peered":"2024-02-05T07:28:26.595627+0000","last_unstale":"2024-02-05T07:33:09.531198+0000","last_undegraded":"2024-02-05T07:33:09.531198+0000","last_fullsized":"2024-02-05T07:33:09.531198+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:49.743255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:32:49.743255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:59.867887+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":748,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.520896+0000","last_change":"2024-02-05T07:29:48.045121+0000","last_active":"2024-02-05T07:33:19.520896+0000","last_peered":"2024-02-05T07:33:19.520896+0000","last_clean":"2024-02-05T07:33:19.520896+0000","last_became_active":"2024-02-05T07:28:25.778935+0000","last_became_peered":"2024-02-05T07:28:25.778935+0000","last_unstale":"2024-02-05T07:33:19.520896+0000","last_undegraded":"2024-02-05T07:33:19.520896+0000","last_fullsized":"2024-02-05T07:33:19.520896+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.045013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.045013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":335,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.535057+0000","last_change":"2024-02-05T07:30:40.496738+0000","last_active":"2024-02-05T07:33:09.535057+0000","last_peered":"2024-02-05T07:33:09.535057+0000","last_clean":"2024-02-05T07:33:09.535057+0000","last_became_active":"2024-02-05T07:30:40.493086+0000","last_became_peered":"2024-02-05T07:30:40.493086+0000","last_unstale":"2024-02-05T07:33:09.535057+0000","last_undegraded":"2024-02-05T07:33:09.535057+0000","last_fullsized":"2024-02-05T07:33:09.535057+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:44.050019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:44.050019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"86'128","reported_seq":388,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.528142+0000","last_change":"2024-02-05T07:32:44.373808+0000","last_active":"2024-02-05T07:33:09.528142+0000","last_peered":"2024-02-05T07:33:09.528142+0000","last_clean":"2024-02-05T07:33:09.528142+0000","last_became_active":"2024-02-05T07:31:41.843342+0000","last_became_peered":"2024-02-05T07:31:41.843342+0000","last_unstale":"2024-02-05T07:33:09.528142+0000","last_undegraded":"2024-02-05T07:33:09.528142+0000","last_fullsized":"2024-02-05T07:33:09.528142+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:44.373683+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_clean_scrub_stamp":"2024-02-05T07:32:44.373683+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:12.613629+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"325'15","reported_seq":769,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521124+0000","last_change":"2024-02-05T07:29:33.839515+0000","last_active":"2024-02-05T07:33:19.521124+0000","last_peered":"2024-02-05T07:33:19.521124+0000","last_clean":"2024-02-05T07:33:19.521124+0000","last_became_active":"2024-02-05T07:28:25.777815+0000","last_became_peered":"2024-02-05T07:28:25.777815+0000","last_unstale":"2024-02-05T07:33:19.521124+0000","last_undegraded":"2024-02-05T07:33:19.521124+0000","last_fullsized":"2024-02-05T07:33:19.521124+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:29:33.839402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:33.839402+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":514,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-05T07:33:20.541836+0000","last_change":"2024-02-05T07:29:50.274850+0000","last_active":"2024-02-05T07:33:20.541836+0000","last_peered":"2024-02-05T07:33:20.541836+0000","last_clean":"2024-02-05T07:33:20.541836+0000","last_became_active":"2024-02-05T07:28:26.588496+0000","last_became_peered":"2024-02-05T07:28:26.588496+0000","last_unstale":"2024-02-05T07:33:20.541836+0000","last_undegraded":"2024-02-05T07:33:20.541836+0000","last_fullsized":"2024-02-05T07:33:20.541836+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.274768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":740,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-02-05T07:33:09.526001+0000","last_change":"2024-02-05T07:30:40.496063+0000","last_active":"2024-02-05T07:33:09.526001+0000","last_peered":"2024-02-05T07:33:09.526001+0000","last_clean":"2024-02-05T07:33:09.526001+0000","last_became_active":"2024-02-05T07:30:40.495711+0000","last_became_peered":"2024-02-05T07:30:40.495711+0000","last_unstale":"2024-02-05T07:33:09.526001+0000","last_undegraded":"2024-02-05T07:33:09.526001+0000","last_fullsized":"2024-02-05T07:33:09.526001+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:26:28.575859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:26:28.575859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:11.383338+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":747,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-02-05T07:33:19.521321+0000","last_change":"2024-02-05T07:32:43.608109+0000","last_active":"2024-02-05T07:33:19.521321+0000","last_peered":"2024-02-05T07:33:19.521321+0000","last_clean":"2024-02-05T07:33:19.521321+0000","last_became_active":"2024-02-05T07:30:40.490288+0000","last_became_peered":"2024-02-05T07:30:40.490288+0000","last_unstale":"2024-02-05T07:33:19.521321+0000","last_undegraded":"2024-02-05T07:33:19.521321+0000","last_fullsized":"2024-02-05T07:33:19.521321+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:43.608002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:32:43.608002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:56.240139+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159686,"num_write_kb":5109889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27251,"num_bytes_recovered":1761804288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":344064000,"data_stored":344064000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52008,"ondisk_log_size":52008,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":162,"num_read_kb":157,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3968,"ondisk_log_size":3968,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1107,"num_read_kb":864,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":6480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":11876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":738,"ondisk_log_size":738,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":257698037939,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":301,"seq":1292785156156,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305044,"kb_used_data":1664,"kb_used_omap":5,"kb_used_meta":303162,"kb_avail":93440108,"statfs":{"total":95995035648,"available":95682670592,"internally_reserved":0,"allocated":1703936,"data_stored":1094187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":310438405},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":1.278,"15min":1.9099999999999999},"min":{"1min":0.25,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.4119999999999999,"5min":14.218,"15min":14.218},"last":0.249},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.389,"15min":1.919},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.4990000000000001,"5min":14.175000000000001,"15min":14.175000000000001},"last":0.48499999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:32:35 2024","interfaces":[{"interface":"back","average":{"1min":2.6789999999999998,"5min":2.3690000000000002,"15min":1.925},"min":{"1min":0.312,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":19.861000000000001,"5min":55.923000000000002,"15min":55.923000000000002},"last":0.375},{"interface":"front","average":{"1min":3.4049999999999998,"5min":3.0099999999999998,"15min":2.742},"min":{"1min":0.378,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":27.518000000000001,"5min":61.103000000000002,"15min":61.103000000000002},"last":0.32400000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":1.54,"15min":2.5019999999999998},"min":{"1min":0.27400000000000002,"5min":0.249,"15min":0.249},"max":{"1min":2.6339999999999999,"5min":34.689999999999998,"15min":34.689999999999998},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.921,"15min":3.04},"min":{"1min":0.26700000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":5.9119999999999999,"5min":34.646999999999998,"15min":34.646999999999998},"last":0.42599999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:32:35 2024","interfaces":[{"interface":"back","average":{"1min":4.7000000000000002,"5min":3.5529999999999999,"15min":2.6429999999999998},"min":{"1min":0.32800000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":36.090000000000003,"5min":55.978999999999999,"15min":55.978999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":4.3869999999999996,"5min":3.6269999999999998,"15min":2.8889999999999998},"min":{"1min":0.29799999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":35.161000000000001,"5min":55.631,"15min":55.631},"last":0.52100000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.55900000000000005,"15min":0.69699999999999995},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.65600000000000003,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.51200000000000001,"15min":0.60899999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60899999999999999,"5min":1.869,"15min":1.869},"last":0.34599999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.51300000000000001,"15min":0.57999999999999996},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56000000000000005,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.52400000000000002,"15min":0.57199999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.52100000000000002,"5min":1.526,"15min":1.526},"last":0.432}]},{"osd":7,"last update":"Mon Feb 5 07:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":2.8719999999999999,"15min":4.6870000000000003},"min":{"1min":0.32900000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.65100000000000002,"5min":84.158000000000001,"15min":84.158000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":2.9630000000000001,"15min":4.835},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71899999999999997,"5min":85.018000000000001,"15min":85.018000000000001},"last":0.57699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575056,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":650812,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93094340,"statfs":{"total":95995035648,"available":95328604160,"internally_reserved":0,"allocated":1888256,"data_stored":1146507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:32:46 2024","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.843,"15min":2.1139999999999999},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.16800000000000001},"max":{"1min":8.8819999999999997,"5min":15.09,"15min":45.393000000000001},"last":0.248},{"interface":"front","average":{"1min":1.5880000000000001,"5min":2.21,"15min":2.234},"min":{"1min":0.28000000000000003,"5min":0.23799999999999999,"15min":0.20899999999999999},"max":{"1min":4.9770000000000003,"5min":14.313000000000001,"15min":45.031999999999996},"last":0.45400000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:33:09 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":2.0139999999999998,"15min":1.7849999999999999},"min":{"1min":0.23799999999999999,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":3.5840000000000001,"5min":15.137,"15min":39.259999999999998},"last":0.311},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.8480000000000001,"15min":1.6990000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.7610000000000001,"5min":14.906000000000001,"15min":40.972000000000001},"last":0.47699999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:32:22 2024","interfaces":[{"interface":"back","average":{"1min":4.6929999999999996,"5min":3.069,"15min":1.9810000000000001},"min":{"1min":0.61199999999999999,"5min":0.22900000000000001,"15min":0.20999999999999999},"max":{"1min":46.777000000000001,"5min":46.777000000000001,"15min":46.777000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":4.9809999999999999,"5min":2.6890000000000001,"15min":1.9910000000000001},"min":{"1min":0.46300000000000002,"5min":0.24299999999999999,"15min":0.24199999999999999},"max":{"1min":50.164999999999999,"5min":50.164999999999999,"15min":50.164999999999999},"last":0.42199999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:32:46 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.395,"15min":1.7270000000000001},"min":{"1min":0.34300000000000003,"5min":0.32600000000000001,"15min":0.312},"max":{"1min":8.9939999999999998,"5min":8.9939999999999998,"15min":61.722000000000001},"last":0.622},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.3799999999999999,"15min":1.6870000000000001},"min":{"1min":0.33000000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.339,"5min":11.443,"15min":61.786000000000001},"last":0.60499999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:33:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.9299999999999999,"15min":1.974},"min":{"1min":0.44700000000000001,"5min":0.33900000000000002,"15min":0.309},"max":{"1min":3.4350000000000001,"5min":46.398000000000003,"15min":106.672},"last":0.57499999999999996},{"interface":"front","average":{"1min":2.028,"5min":2.0379999999999998,"15min":2.0299999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":7.1210000000000004,"5min":20.085999999999999,"15min":107.348},"last":0.63700000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:32:30 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.3400000000000001,"15min":1.6819999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.3809999999999998,"5min":5.5990000000000002,"15min":5.5990000000000002},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.607,"5min":1.657,"15min":1.9890000000000001},"min":{"1min":0.41899999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":8.923,"5min":8.923,"15min":8.923},"last":0.54000000000000004}]},{"osd":7,"last update":"Mon Feb 5 07:33:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1920000000000002,"5min":2.633,"15min":1.9119999999999999},"min":{"1min":0.38900000000000001,"5min":0.318,"15min":0.318},"max":{"1min":7.0899999999999999,"5min":46.351999999999997,"15min":66.703999999999994},"last":0.58599999999999997},{"interface":"front","average":{"1min":1.246,"5min":1.6040000000000001,"15min":1.5529999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.33500000000000002},"max":{"1min":3.5339999999999998,"5min":11.263999999999999,"15min":61.594999999999999},"last":0.49299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738582,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":610704,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93134448,"statfs":{"total":95995035648,"available":95369674752,"internally_reserved":0,"allocated":1851392,"data_stored":1254679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:32:39 2024","interfaces":[{"interface":"back","average":{"1min":4.9790000000000001,"5min":2.7829999999999999,"15min":2.012},"min":{"1min":0.40300000000000002,"5min":0.192,"15min":0.16500000000000001},"max":{"1min":25.277999999999999,"5min":32.546999999999997,"15min":45.372},"last":0.39800000000000002},{"interface":"front","average":{"1min":5.2149999999999999,"5min":2.6869999999999998,"15min":2.093},"min":{"1min":0.45600000000000002,"5min":0.217,"15min":0.186},"max":{"1min":24.652999999999999,"5min":29.675999999999998,"15min":45.231999999999999},"last":0.49099999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.2609999999999999,"15min":1.7430000000000001},"min":{"1min":0.23899999999999999,"5min":0.191,"15min":0.191},"max":{"1min":5.9180000000000001,"5min":6.5140000000000002,"15min":65.903999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.175,"15min":1.7090000000000001},"min":{"1min":0.32400000000000001,"5min":0.224,"15min":0.182},"max":{"1min":4.7389999999999999,"5min":4.8220000000000001,"15min":65.694000000000003},"last":0.50900000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:33:20 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":2.1309999999999998,"15min":2.1120000000000001},"min":{"1min":0.30099999999999999,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":4.7679999999999998,"5min":29.629000000000001,"15min":65.155000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.444,"5min":2.4460000000000002,"15min":2.2989999999999999},"min":{"1min":0.27300000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.9000000000000004,"5min":29.463999999999999,"15min":65.114000000000004},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":1.722,"15min":1.8879999999999999},"min":{"1min":0.27200000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.8769999999999998,"5min":42.901000000000003,"15min":65.188000000000002},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.851,"15min":1.954},"min":{"1min":0.311,"5min":0.27100000000000002,"15min":0.23699999999999999},"max":{"1min":2.2250000000000001,"5min":42.960999999999999,"15min":64.873000000000005},"last":0.83699999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.96399999999999997,"15min":1.6919999999999999},"min":{"1min":0.41699999999999998,"5min":0.30599999999999999,"15min":0.27000000000000002},"max":{"1min":2.1259999999999999,"5min":5.4669999999999996,"15min":95.772999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.0089999999999999,"15min":1.7430000000000001},"min":{"1min":0.34699999999999998,"5min":0.29299999999999998,"15min":0.27600000000000002},"max":{"1min":3.5950000000000002,"5min":7.1929999999999996,"15min":96.028000000000006},"last":0.873}]},{"osd":6,"last update":"Mon Feb 5 07:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":1.024,"15min":1.282},"min":{"1min":0.33700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.9100000000000001,"5min":4.6929999999999996,"15min":4.6929999999999996},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.254,"15min":1.55},"min":{"1min":0.40600000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.2679999999999998,"5min":7.1319999999999997,"15min":7.1319999999999997},"last":0.77100000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:32:52 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.6859999999999999,"15min":1.7909999999999999},"min":{"1min":0.34899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.9630000000000001,"5min":43.012,"15min":65.263000000000005},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.746,"5min":1.873,"15min":1.8180000000000001},"min":{"1min":0.373,"5min":0.34000000000000002,"15min":0.309},"max":{"1min":2.032,"5min":43.512,"15min":64.906000000000006},"last":0.92000000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411532,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":664680,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93080472,"statfs":{"total":95995035648,"available":95314403328,"internally_reserved":0,"allocated":1658880,"data_stored":1029668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:33:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.2749999999999999,"15min":2.0179999999999998},"min":{"1min":0.32500000000000001,"5min":0.19600000000000001,"15min":0.182},"max":{"1min":7.2519999999999998,"5min":12.759,"15min":92.638000000000005},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.3160000000000001,"15min":2.1459999999999999},"min":{"1min":0.23999999999999999,"5min":0.22,"15min":0.20699999999999999},"max":{"1min":5.1870000000000003,"5min":12.647,"15min":93.481999999999999},"last":0.31900000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:33:06 2024","interfaces":[{"interface":"back","average":{"1min":1.851,"5min":2.879,"15min":2.2970000000000002},"min":{"1min":0.307,"5min":0.215,"15min":0.19400000000000001},"max":{"1min":6.7140000000000004,"5min":63.823,"15min":93.504000000000005},"last":0.36499999999999999},{"interface":"front","average":{"1min":2.452,"5min":3.3500000000000001,"15min":2.5030000000000001},"min":{"1min":0.35099999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":7.1859999999999999,"5min":64.771000000000001,"15min":93.004000000000005},"last":0.42599999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:32:27 2024","interfaces":[{"interface":"back","average":{"1min":4.859,"5min":3.431,"15min":1.996},"min":{"1min":0.26100000000000001,"5min":0.25600000000000001,"15min":0.214},"max":{"1min":63.784999999999997,"5min":63.784999999999997,"15min":63.784999999999997},"last":0.252},{"interface":"front","average":{"1min":5.2830000000000004,"5min":3.391,"15min":1.968},"min":{"1min":0.38900000000000001,"5min":0.22600000000000001,"15min":0.193},"max":{"1min":63.521000000000001,"5min":63.521000000000001,"15min":63.521000000000001},"last":0.38100000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:32:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":3.5760000000000001,"15min":1.9990000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.24099999999999999},"max":{"1min":5.2359999999999998,"5min":168.11699999999999,"15min":168.11699999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.86799999999999999,"5min":3.4239999999999999,"15min":1.966},"min":{"1min":0.34399999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":3.1240000000000001,"5min":168.589,"15min":168.589},"last":0.45000000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:33:06 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":0.97799999999999998,"15min":1.2250000000000001},"min":{"1min":0.47199999999999998,"5min":0.315,"15min":0.315},"max":{"1min":4.742,"5min":5.6929999999999996,"15min":39.923000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.276,"5min":0.94899999999999995,"15min":1.173},"min":{"1min":0.36699999999999999,"5min":0.33000000000000002,"15min":0.26400000000000001},"max":{"1min":2.7490000000000001,"5min":4.0709999999999997,"15min":38.920999999999999},"last":0.59099999999999997}]},{"osd":6,"last update":"Mon Feb 5 07:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.96799999999999997,"15min":1.1719999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4180000000000001,"5min":4.109,"15min":4.109},"last":0.505},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.0649999999999999,"15min":1.363},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.1749999999999998,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.52000000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:33:06 2024","interfaces":[{"interface":"back","average":{"1min":1.726,"5min":1.851,"15min":1.3999999999999999},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":11.311999999999999,"5min":37.573999999999998,"15min":37.573999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.581,"15min":1.2889999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.819,"5min":37.432000000000002,"15min":37.432000000000002},"last":0.55500000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149894,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":660844,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93084308,"statfs":{"total":95995035648,"available":95318331392,"internally_reserved":0,"allocated":1949696,"data_stored":1211483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":674494919},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.3360000000000001,"15min":2.1720000000000002},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":4.2590000000000003,"5min":6.1849999999999996,"15min":74.114999999999995},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.3620000000000001,"5min":2.476,"15min":2.2989999999999999},"min":{"1min":0.27400000000000002,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.4239999999999999,"5min":99.989999999999995,"15min":99.989999999999995},"last":0.309}]},{"osd":1,"last update":"Mon Feb 5 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":3.5800000000000001,"15min":2.52},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.516,"5min":102.20399999999999,"15min":102.20399999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.976,"5min":3.3969999999999998,"15min":2.7959999999999998},"min":{"1min":0.29699999999999999,"5min":0.255,"15min":0.185},"max":{"1min":5.0700000000000003,"5min":102.286,"15min":102.286},"last":0.33300000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":2.4079999999999999,"15min":2.4740000000000002},"min":{"1min":0.33100000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.3040000000000003,"5min":99.847999999999999,"15min":99.847999999999999},"last":0.377},{"interface":"front","average":{"1min":1.631,"5min":2.4279999999999999,"15min":2.0049999999999999},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.22500000000000001},"max":{"1min":5.6710000000000003,"5min":99.950999999999993,"15min":99.950999999999993},"last":0.36699999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:32:50 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":2.677,"15min":1.5489999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.5629999999999997,"5min":100.024,"15min":100.024},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.5549999999999999,"5min":2.9300000000000002,"15min":1.623},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.28599999999999998},"max":{"1min":6.4480000000000004,"5min":99.859999999999999,"15min":99.859999999999999},"last":0.58799999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":2.2749999999999999,"15min":1.873},"min":{"1min":0.46000000000000002,"5min":0.32700000000000001,"15min":0.28499999999999998},"max":{"1min":3.3159999999999998,"5min":99.911000000000001,"15min":99.911000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.1919999999999999,"5min":2.2639999999999998,"15min":1.6040000000000001},"min":{"1min":0.45200000000000001,"5min":0.33000000000000002,"15min":0.27400000000000002},"max":{"1min":4.6369999999999996,"5min":100.053,"15min":100.053},"last":0.61199999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.169,"15min":1.343},"min":{"1min":0.437,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.4080000000000004,"5min":99.807000000000002,"15min":99.807000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.1659999999999999,"5min":2.1699999999999999,"15min":1.391},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":3.5,"5min":101.489,"15min":101.489},"last":0.63400000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:32:58 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":4.6459999999999999,"15min":2.2850000000000001},"min":{"1min":0.49199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":6.4880000000000004,"5min":99.893000000000001,"15min":99.893000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.5309999999999999,"5min":4.3540000000000001,"15min":2.5489999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":4.468,"5min":99.879000000000005,"15min":99.879000000000005},"last":0.59999999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953666,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":519852,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":93225300,"statfs":{"total":95995035648,"available":95462707200,"internally_reserved":0,"allocated":1224704,"data_stored":663232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":2.194,"15min":2.0899999999999999},"min":{"1min":0.248,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.4199999999999999,"5min":101.212,"15min":101.212},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.77100000000000002,"5min":2.234,"15min":2.1880000000000002},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23300000000000001},"max":{"1min":4.0659999999999998,"5min":102.07599999999999,"15min":102.07599999999999},"last":0.64000000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":2.742,"5min":3.7519999999999998,"15min":2.2789999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":23.335000000000001,"5min":67.838999999999999,"15min":69.102000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":2.5659999999999998,"5min":4.1980000000000004,"15min":2.5},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":23.346,"5min":68.108999999999995,"15min":68.108999999999995},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":1.2529999999999999,"15min":1.3420000000000001},"min":{"1min":0.254,"5min":0.249,"15min":0.249},"max":{"1min":2.2639999999999998,"5min":24.773,"15min":67.144999999999996},"last":0.623},{"interface":"front","average":{"1min":0.70399999999999996,"5min":1.49,"15min":1.369},"min":{"1min":0.29599999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.5790000000000002,"5min":24.693000000000001,"15min":66.959999999999994},"last":0.46300000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":2.8780000000000001,"5min":3.9729999999999999,"15min":1.851},"min":{"1min":0.27600000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":26.899999999999999,"5min":66.019999999999996,"15min":66.019999999999996},"last":0.51400000000000001},{"interface":"front","average":{"1min":3.1819999999999999,"5min":4.3879999999999999,"15min":2.044},"min":{"1min":0.34499999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":24.518999999999998,"5min":101.001,"15min":101.001},"last":0.54200000000000004}]},{"osd":5,"last update":"Mon Feb 5 07:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.63300000000000001,"15min":0.58699999999999997},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.73999999999999999,"5min":10.917999999999999,"15min":10.917999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.67700000000000005,"15min":0.60899999999999999},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.24199999999999999},"max":{"1min":0.68600000000000005,"5min":15.105,"15min":15.105},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Feb 5 07:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.82899999999999996,"15min":1.1879999999999999},"min":{"1min":0.25600000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.70899999999999996,"5min":14.606,"15min":14.606},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.96099999999999997,"15min":1.4490000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.82699999999999996,"5min":13.114000000000001,"15min":13.114000000000001},"last":0.70499999999999996}]},{"osd":7,"last update":"Mon Feb 5 07:32:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.81000000000000005,"15min":0.71499999999999997},"min":{"1min":0.34599999999999997,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.57099999999999995,"5min":15.073,"15min":15.073},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.77600000000000002,"15min":0.68600000000000005},"min":{"1min":0.35399999999999998,"5min":0.28999999999999998,"15min":0.248},"max":{"1min":0.60899999999999999,"5min":14.581,"15min":14.581},"last":0.72399999999999998}]}]},{"osd":5,"up_from":0,"seq":176093659325,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":344064000,"data_stored":344064000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:33:23.391 INFO:tasks.thrashosds.thrasher:Setting 4.d to [4, 7, 1] 2024-02-05T07:33:23.391 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.d', '4', '7', '1'] 2024-02-05T07:33:23.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pg-upmap 4.d 4 7 1 2024-02-05T07:33:23.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:23.491 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:23.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:23.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:23.591 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:23.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:23.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:23.619 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:23.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:23.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:33:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:33:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:33:23.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:23 smithi007 ceph-mon[29098]: 5.6 scrub starts 2024-02-05T07:33:23.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:23 smithi007 ceph-mon[29098]: 5.6 scrub ok 2024-02-05T07:33:23.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:23 smithi007 ceph-mon[37158]: 5.6 scrub starts 2024-02-05T07:33:23.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:23 smithi007 ceph-mon[37158]: 5.6 scrub ok 2024-02-05T07:33:23.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:23.744 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:23.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:23.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:23.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:23.874 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:23.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:23.975 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:23.999 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:24.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:24.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:24.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:23 smithi090 ceph-mon[33213]: 5.6 scrub starts 2024-02-05T07:33:24.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:23 smithi090 ceph-mon[33213]: 5.6 scrub ok 2024-02-05T07:33:24.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:24.124 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:24.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:24.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:24.225 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:24.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:24.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:24.383 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:24.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:24.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:24.485 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:24.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:24.512 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:24.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:24.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:24.638 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:24.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:24.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:24.715 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:33:24.715 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:33:24.715 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-02-05T07:33:24.715 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-02-05T07:33:24.715 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:33:24.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:24.765 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:24.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:24.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:24.811 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:24 smithi007 ceph-mon[29098]: pgmap v755: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 895 B/s rd, 0 op/s 2024-02-05T07:33:24.811 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:24 smithi007 ceph-mon[29098]: 6.0 scrub starts 2024-02-05T07:33:24.812 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:24 smithi007 ceph-mon[29098]: 6.0 scrub ok 2024-02-05T07:33:24.812 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:24 smithi007 ceph-mon[37158]: pgmap v755: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 895 B/s rd, 0 op/s 2024-02-05T07:33:24.812 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:24 smithi007 ceph-mon[37158]: 6.0 scrub starts 2024-02-05T07:33:24.812 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:24 smithi007 ceph-mon[37158]: 6.0 scrub ok 2024-02-05T07:33:24.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:24.893 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:24.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:24.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:24.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:25.019 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:25.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:25.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:24 smithi090 ceph-mon[33213]: pgmap v755: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 895 B/s rd, 0 op/s 2024-02-05T07:33:25.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:24 smithi090 ceph-mon[33213]: 6.0 scrub starts 2024-02-05T07:33:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:24 smithi090 ceph-mon[33213]: 6.0 scrub ok 2024-02-05T07:33:25.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:25.143 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:25.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:25.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:25.268 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:25.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:25.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:25.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:25.393 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:25.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:25.494 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:25.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:25.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:25.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:25.649 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:25.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:25.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:25.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:25.777 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:25.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:25.878 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:25.905 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:25.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:25.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:26.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:26.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:33:26.032 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:26.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:26.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:26.134 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:26.160 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:26.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:26.184 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:26.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:26.285 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:26.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:26.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:26.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:26.411 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:26.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:26.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:26.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:26.540 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:26.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:26.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:26.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T07:33:26.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:26.670 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:26.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:26.772 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:26.797 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:26.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:26.811 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:26 smithi007 ceph-mon[29098]: pgmap v756: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:33:26.811 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:26 smithi007 ceph-mon[29098]: 6.10 scrub starts 2024-02-05T07:33:26.811 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:26 smithi007 ceph-mon[29098]: 6.10 scrub ok 2024-02-05T07:33:26.811 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:26 smithi007 ceph-mon[37158]: pgmap v756: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:33:26.811 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:26 smithi007 ceph-mon[37158]: 6.10 scrub starts 2024-02-05T07:33:26.811 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:26 smithi007 ceph-mon[37158]: 6.10 scrub ok 2024-02-05T07:33:26.825 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:26 smithi090 ceph-mon[33213]: pgmap v756: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:33:26.826 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:26 smithi090 ceph-mon[33213]: 6.10 scrub starts 2024-02-05T07:33:26.826 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:26 smithi090 ceph-mon[33213]: 6.10 scrub ok 2024-02-05T07:33:26.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:26.927 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:26.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:26.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:27.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:27.053 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:27.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:27.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:27.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:27.182 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:27.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:27.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:27.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:27.310 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:27.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:27.412 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:27.435 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:27.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:27.537 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:27.561 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:27.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:27.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:27.571 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_2' removed 2024-02-05T07:33:27.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:27.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:27.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:27.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:27.813 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:27.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:27.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:27.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:27.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:27 smithi007 ceph-mon[29098]: 4.19 scrub starts 2024-02-05T07:33:27.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:27 smithi007 ceph-mon[29098]: 4.19 scrub ok 2024-02-05T07:33:27.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:27 smithi007 ceph-mon[29098]: 4.a scrub starts 2024-02-05T07:33:27.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:27 smithi007 ceph-mon[29098]: 4.a scrub ok 2024-02-05T07:33:27.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:27 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1219778666' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:33:27.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:27 smithi007 ceph-mon[37158]: 4.19 scrub starts 2024-02-05T07:33:27.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:27 smithi007 ceph-mon[37158]: 4.19 scrub ok 2024-02-05T07:33:27.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:27 smithi007 ceph-mon[37158]: 4.a scrub starts 2024-02-05T07:33:27.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:27 smithi007 ceph-mon[37158]: 4.a scrub ok 2024-02-05T07:33:27.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:27 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1219778666' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:33:27.939 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:27.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:27.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:27 smithi090 ceph-mon[33213]: 4.19 scrub starts 2024-02-05T07:33:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:27 smithi090 ceph-mon[33213]: 4.19 scrub ok 2024-02-05T07:33:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:27 smithi090 ceph-mon[33213]: 4.a scrub starts 2024-02-05T07:33:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:27 smithi090 ceph-mon[33213]: 4.a scrub ok 2024-02-05T07:33:28.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:27 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1219778666' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:33:28.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:28.064 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:28.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:28.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:28.193 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:28.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:28.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:28.322 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:28.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:28.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:28.451 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:28.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:28.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:28.553 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:28.578 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:28.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:28.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:28.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:28.703 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:28.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:28.808 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:28.833 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:28.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:28.853 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:28 smithi007 ceph-mon[29098]: pgmap v757: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:33:28.853 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:28 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1219778666' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:33:28.853 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:28 smithi007 ceph-mon[29098]: osdmap e421: 8 total, 8 up, 6 in 2024-02-05T07:33:28.854 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:28 smithi007 ceph-mon[29098]: 5.b scrub starts 2024-02-05T07:33:28.854 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:28 smithi007 ceph-mon[29098]: 5.b scrub ok 2024-02-05T07:33:28.854 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:28 smithi007 ceph-mon[29098]: 5.19 scrub starts 2024-02-05T07:33:28.854 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:28 smithi007 ceph-mon[29098]: 5.19 scrub ok 2024-02-05T07:33:28.854 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:28 smithi007 ceph-mon[37158]: pgmap v757: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:33:28.854 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:28 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1219778666' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:33:28.854 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:28 smithi007 ceph-mon[37158]: osdmap e421: 8 total, 8 up, 6 in 2024-02-05T07:33:28.854 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:28 smithi007 ceph-mon[37158]: 5.b scrub starts 2024-02-05T07:33:28.854 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:28 smithi007 ceph-mon[37158]: 5.b scrub ok 2024-02-05T07:33:28.854 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:28 smithi007 ceph-mon[37158]: 5.19 scrub starts 2024-02-05T07:33:28.854 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:28 smithi007 ceph-mon[37158]: 5.19 scrub ok 2024-02-05T07:33:28.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:28.960 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:28.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:28.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:28 smithi090 ceph-mon[33213]: pgmap v757: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:33:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:28 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1219778666' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:33:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:28 smithi090 ceph-mon[33213]: osdmap e421: 8 total, 8 up, 6 in 2024-02-05T07:33:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:28 smithi090 ceph-mon[33213]: 5.b scrub starts 2024-02-05T07:33:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:28 smithi090 ceph-mon[33213]: 5.b scrub ok 2024-02-05T07:33:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:28 smithi090 ceph-mon[33213]: 5.19 scrub starts 2024-02-05T07:33:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:28 smithi090 ceph-mon[33213]: 5.19 scrub ok 2024-02-05T07:33:29.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:29.086 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:29.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:29.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:29.188 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:29.216 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:29.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:29.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:29.345 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:29.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:29.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:29.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:29.471 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:29.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:29.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:29.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:29.596 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:29.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:29.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:29.600 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:33:29.619 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:33:29.638 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:33:29.658 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:33:29.678 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:33:29.699 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:29.699 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:33:29.720 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:33:29.722 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:29.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:29.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:29.740 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:33:29.740 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:29 smithi007 ceph-mon[29098]: 3.f scrub starts 2024-02-05T07:33:29.740 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:29 smithi007 ceph-mon[29098]: 3.f scrub ok 2024-02-05T07:33:29.740 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:29 smithi007 ceph-mon[29098]: 5.8 scrub starts 2024-02-05T07:33:29.740 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:29 smithi007 ceph-mon[29098]: 5.8 scrub ok 2024-02-05T07:33:29.741 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:29 smithi007 ceph-mon[37158]: 3.f scrub starts 2024-02-05T07:33:29.741 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:29 smithi007 ceph-mon[37158]: 3.f scrub ok 2024-02-05T07:33:29.741 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:29 smithi007 ceph-mon[37158]: 5.8 scrub starts 2024-02-05T07:33:29.741 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:29 smithi007 ceph-mon[37158]: 5.8 scrub ok 2024-02-05T07:33:29.741 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:33:29.741 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:33:29.741 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:33:29.741 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:33:29.742 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:33:29.742 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:33:29.742 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:33:29.742 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:33:29.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:29.847 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:29.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:29.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:29.978 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:29.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:29.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:29 smithi090 ceph-mon[33213]: 3.f scrub starts 2024-02-05T07:33:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:29 smithi090 ceph-mon[33213]: 3.f scrub ok 2024-02-05T07:33:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:29 smithi090 ceph-mon[33213]: 5.8 scrub starts 2024-02-05T07:33:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:29 smithi090 ceph-mon[33213]: 5.8 scrub ok 2024-02-05T07:33:30.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:30.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:30.108 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:30.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:30.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:30.237 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:30.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:30.263 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T07:33:30.264 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T07:33:30.264 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-02-05T07:33:30.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-02-05T07:33:30.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:30.367 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:30.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:30.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:30.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:30.494 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:30.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:30.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:30.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:30.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:30.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:30.751 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:30.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:30.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:30.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:30.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:30.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[29098]: 6.c scrub starts 2024-02-05T07:33:30.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[29098]: 6.c scrub ok 2024-02-05T07:33:30.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[29098]: pgmap v759: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:33:30.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/43577010' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:30.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[29098]: 5.11 scrub starts 2024-02-05T07:33:30.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[29098]: 5.11 scrub ok 2024-02-05T07:33:30.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[29098]: 4.7 scrub starts 2024-02-05T07:33:30.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[29098]: 4.7 scrub ok 2024-02-05T07:33:30.884 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[37158]: 6.c scrub starts 2024-02-05T07:33:30.884 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[37158]: 6.c scrub ok 2024-02-05T07:33:30.884 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[37158]: pgmap v759: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:33:30.884 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/43577010' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:30.884 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[37158]: 5.11 scrub starts 2024-02-05T07:33:30.884 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[37158]: 5.11 scrub ok 2024-02-05T07:33:30.884 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[37158]: 4.7 scrub starts 2024-02-05T07:33:30.884 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:30 smithi007 ceph-mon[37158]: 4.7 scrub ok 2024-02-05T07:33:30.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:31.008 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:31.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:31.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:31.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:33:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:33:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:30 smithi090 ceph-mon[33213]: 6.c scrub starts 2024-02-05T07:33:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:30 smithi090 ceph-mon[33213]: 6.c scrub ok 2024-02-05T07:33:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:30 smithi090 ceph-mon[33213]: pgmap v759: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:33:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:30 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/43577010' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:30 smithi090 ceph-mon[33213]: 5.11 scrub starts 2024-02-05T07:33:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:30 smithi090 ceph-mon[33213]: 5.11 scrub ok 2024-02-05T07:33:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:30 smithi090 ceph-mon[33213]: 4.7 scrub starts 2024-02-05T07:33:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:30 smithi090 ceph-mon[33213]: 4.7 scrub ok 2024-02-05T07:33:31.110 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:31.135 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:31.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:31.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:31.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:31.262 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:31.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:31.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:31.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:31.394 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:31.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:31.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:31.524 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:31.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:31.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:31.653 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:31.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:31.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:31.781 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:31.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:31.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:31.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:31 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:33:31.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:31 smithi007 ceph-mon[29098]: 6.8 scrub starts 2024-02-05T07:33:31.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:31 smithi007 ceph-mon[29098]: 6.8 scrub ok 2024-02-05T07:33:31.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:31 smithi007 ceph-mon[29098]: 5.1f scrub starts 2024-02-05T07:33:31.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:31 smithi007 ceph-mon[29098]: 5.1f scrub ok 2024-02-05T07:33:31.884 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:31 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:33:31.884 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:31 smithi007 ceph-mon[37158]: 6.8 scrub starts 2024-02-05T07:33:31.884 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:31 smithi007 ceph-mon[37158]: 6.8 scrub ok 2024-02-05T07:33:31.884 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:31 smithi007 ceph-mon[37158]: 5.1f scrub starts 2024-02-05T07:33:31.884 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:31 smithi007 ceph-mon[37158]: 5.1f scrub ok 2024-02-05T07:33:31.911 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:31.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:32.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:32.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:31 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:33:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:31 smithi090 ceph-mon[33213]: 6.8 scrub starts 2024-02-05T07:33:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:31 smithi090 ceph-mon[33213]: 6.8 scrub ok 2024-02-05T07:33:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:31 smithi090 ceph-mon[33213]: 5.1f scrub starts 2024-02-05T07:33:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:31 smithi090 ceph-mon[33213]: 5.1f scrub ok 2024-02-05T07:33:32.040 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:32.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:32.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:32.168 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:32.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:32.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:32.203 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:32.203 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:32.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:32.298 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:32.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:32.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:32.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:32.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:32.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:32.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:32.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:32.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:32.583 INFO:teuthology.orchestra.run.smithi007.stderr:set 4.d pg_upmap mapping to [4,7,1] 2024-02-05T07:33:32.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[29098]: 4.d scrub starts 2024-02-05T07:33:32.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[29098]: 4.d scrub ok 2024-02-05T07:33:32.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:32.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[29098]: pgmap v760: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 948 B/s rd, 0 op/s 2024-02-05T07:33:32.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2011315344' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.d", "id": [4, 7, 1]}]: dispatch 2024-02-05T07:33:32.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[29098]: 1.13 scrub starts 2024-02-05T07:33:32.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[29098]: 1.13 scrub ok 2024-02-05T07:33:32.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2011315344' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.d", "id": [4, 7, 1]}]': finished 2024-02-05T07:33:32.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:32.685 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:32.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:32.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:32.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:32.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:32.866 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[37158]: 4.d scrub starts 2024-02-05T07:33:32.866 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[37158]: 4.d scrub ok 2024-02-05T07:33:32.866 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:32.866 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[37158]: pgmap v760: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 948 B/s rd, 0 op/s 2024-02-05T07:33:32.866 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2011315344' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.d", "id": [4, 7, 1]}]: dispatch 2024-02-05T07:33:32.866 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[37158]: 1.13 scrub starts 2024-02-05T07:33:32.866 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[37158]: 1.13 scrub ok 2024-02-05T07:33:32.866 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:32 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2011315344' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.d", "id": [4, 7, 1]}]': finished 2024-02-05T07:33:32.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:32.941 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:32.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:32 smithi090 ceph-mon[33213]: 4.d scrub starts 2024-02-05T07:33:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:32 smithi090 ceph-mon[33213]: 4.d scrub ok 2024-02-05T07:33:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:32 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:32 smithi090 ceph-mon[33213]: pgmap v760: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 948 B/s rd, 0 op/s 2024-02-05T07:33:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:32 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2011315344' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.d", "id": [4, 7, 1]}]: dispatch 2024-02-05T07:33:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:32 smithi090 ceph-mon[33213]: 1.13 scrub starts 2024-02-05T07:33:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:32 smithi090 ceph-mon[33213]: 1.13 scrub ok 2024-02-05T07:33:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:32 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2011315344' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.d", "id": [4, 7, 1]}]': finished 2024-02-05T07:33:33.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:33.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:33.067 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:33.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:33.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:33.196 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:33.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:33.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:33.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:33.321 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:33.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:33.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:33.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:33.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:33.448 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:33.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:33.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:33.576 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:33.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:33.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:33.704 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:33.705 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:33:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:33:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:33:33.706 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:33 smithi007 ceph-mon[29098]: osdmap e422: 8 total, 8 up, 6 in 2024-02-05T07:33:33.706 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:33 smithi007 ceph-mon[29098]: 5.2 scrub starts 2024-02-05T07:33:33.706 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:33 smithi007 ceph-mon[29098]: 5.2 scrub ok 2024-02-05T07:33:33.706 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:33 smithi007 ceph-mon[29098]: 6.1 scrub starts 2024-02-05T07:33:33.706 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:33 smithi007 ceph-mon[29098]: 6.1 scrub ok 2024-02-05T07:33:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:33.707 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:33 smithi007 ceph-mon[37158]: osdmap e422: 8 total, 8 up, 6 in 2024-02-05T07:33:33.707 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:33 smithi007 ceph-mon[37158]: 5.2 scrub starts 2024-02-05T07:33:33.707 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:33 smithi007 ceph-mon[37158]: 5.2 scrub ok 2024-02-05T07:33:33.707 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:33 smithi007 ceph-mon[37158]: 6.1 scrub starts 2024-02-05T07:33:33.707 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:33 smithi007 ceph-mon[37158]: 6.1 scrub ok 2024-02-05T07:33:33.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:33.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:33:33.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:33.856 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:33.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:33.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:33.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:33.986 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:33.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:33 smithi090 ceph-mon[33213]: osdmap e422: 8 total, 8 up, 6 in 2024-02-05T07:33:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:33 smithi090 ceph-mon[33213]: 5.2 scrub starts 2024-02-05T07:33:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:33 smithi090 ceph-mon[33213]: 5.2 scrub ok 2024-02-05T07:33:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:33 smithi090 ceph-mon[33213]: 6.1 scrub starts 2024-02-05T07:33:34.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:33 smithi090 ceph-mon[33213]: 6.1 scrub ok 2024-02-05T07:33:34.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:34.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:34.113 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:34.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:34.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:34.237 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:34.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:34.339 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:34.365 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:34.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:34.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:34.467 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:34.491 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:34.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:34.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:34.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:34.616 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:34.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:34.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:34.654 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:33:34.654 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:33:34.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:34.742 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:34.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:34.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:34.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:34.869 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:34.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:34.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:34 smithi007 ceph-mon[29098]: pgmap v762: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:33:34.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:34 smithi007 ceph-mon[29098]: 3.e scrub starts 2024-02-05T07:33:34.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:34 smithi007 ceph-mon[29098]: 3.e scrub ok 2024-02-05T07:33:34.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:34 smithi007 ceph-mon[29098]: 4.1c scrub starts 2024-02-05T07:33:34.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:34 smithi007 ceph-mon[29098]: 4.1c scrub ok 2024-02-05T07:33:34.899 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:34 smithi007 ceph-mon[37158]: pgmap v762: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:33:34.899 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:34 smithi007 ceph-mon[37158]: 3.e scrub starts 2024-02-05T07:33:34.899 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:34 smithi007 ceph-mon[37158]: 3.e scrub ok 2024-02-05T07:33:34.899 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:34 smithi007 ceph-mon[37158]: 4.1c scrub starts 2024-02-05T07:33:34.899 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:34 smithi007 ceph-mon[37158]: 4.1c scrub ok 2024-02-05T07:33:34.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:34.998 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:35.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:34 smithi090 ceph-mon[33213]: pgmap v762: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:33:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:34 smithi090 ceph-mon[33213]: 3.e scrub starts 2024-02-05T07:33:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:34 smithi090 ceph-mon[33213]: 3.e scrub ok 2024-02-05T07:33:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:34 smithi090 ceph-mon[33213]: 4.1c scrub starts 2024-02-05T07:33:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:34 smithi090 ceph-mon[33213]: 4.1c scrub ok 2024-02-05T07:33:35.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:35.126 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:35.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:35.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:35.256 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:35.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:35.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:35.358 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:35.382 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:35.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:35.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:35.512 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:35.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:35.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:35.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:35.637 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:35.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:35.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:35.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:35 smithi007 ceph-mon[29098]: 3.b scrub starts 2024-02-05T07:33:35.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:35 smithi007 ceph-mon[29098]: 3.b scrub ok 2024-02-05T07:33:35.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:35 smithi007 ceph-mon[29098]: 6.a scrub starts 2024-02-05T07:33:35.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:35 smithi007 ceph-mon[29098]: 6.a scrub ok 2024-02-05T07:33:35.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:35 smithi007 ceph-mon[37158]: 3.b scrub starts 2024-02-05T07:33:35.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:35 smithi007 ceph-mon[37158]: 3.b scrub ok 2024-02-05T07:33:35.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:35 smithi007 ceph-mon[37158]: 6.a scrub starts 2024-02-05T07:33:35.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:35 smithi007 ceph-mon[37158]: 6.a scrub ok 2024-02-05T07:33:35.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:35.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:35.767 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:35.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:35.868 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:35.893 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:35.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:35.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:35.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:36.022 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:36.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:36.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:35 smithi090 ceph-mon[33213]: 3.b scrub starts 2024-02-05T07:33:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:35 smithi090 ceph-mon[33213]: 3.b scrub ok 2024-02-05T07:33:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:35 smithi090 ceph-mon[33213]: 6.a scrub starts 2024-02-05T07:33:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:35 smithi090 ceph-mon[33213]: 6.a scrub ok 2024-02-05T07:33:36.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:36.152 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:36.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:36.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:36.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:36.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:36.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:36.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T07:33:36.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:36.410 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:36.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:36.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:36.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:36.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:36.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:36.660 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:36.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:36.692 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:36 smithi007 ceph-mon[29098]: pgmap v763: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T07:33:36.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:36 smithi007 ceph-mon[37158]: pgmap v763: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T07:33:36.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:36.786 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:36.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:36.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:36.888 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:36.916 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:36.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:36.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:36.962 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:37.005 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:37.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:37.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:36 smithi090 ceph-mon[33213]: pgmap v763: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T07:33:37.042 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:37.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:37.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:37.167 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:37.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:37.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:37.269 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:37.292 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:37.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:37.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:37.394 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:37.422 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:37.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:37.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:37.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:37.551 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:37.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:37.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:37.609 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_3' created 2024-02-05T07:33:37.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:37.679 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:37.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:37.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:37.781 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:37.805 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:37.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:37.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:37.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:37 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1025917351' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-05T07:33:37.849 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:37 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1025917351' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-05T07:33:37.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:37.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:37.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:37 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1025917351' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-05T07:33:38.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:38.065 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:38.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:38.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:38.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:38.192 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:38.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:38.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:38.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:38.321 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:38.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:38.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:38.450 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:38.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:38.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:38.578 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:38.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:38.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:38 smithi007 ceph-mon[29098]: pgmap v764: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:33:38.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:38 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1025917351' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-05T07:33:38.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:38 smithi007 ceph-mon[29098]: osdmap e423: 8 total, 8 up, 6 in 2024-02-05T07:33:38.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:38 smithi007 ceph-mon[29098]: 6.11 scrub starts 2024-02-05T07:33:38.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:38 smithi007 ceph-mon[29098]: 6.11 scrub ok 2024-02-05T07:33:38.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:38 smithi007 ceph-mon[37158]: pgmap v764: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:33:38.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:38 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1025917351' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-05T07:33:38.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:38 smithi007 ceph-mon[37158]: osdmap e423: 8 total, 8 up, 6 in 2024-02-05T07:33:38.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:38 smithi007 ceph-mon[37158]: 6.11 scrub starts 2024-02-05T07:33:38.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:38 smithi007 ceph-mon[37158]: 6.11 scrub ok 2024-02-05T07:33:38.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:38.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:38.706 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:38.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:38.807 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:38.831 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:38.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:38.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:38.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:38.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:38 smithi090 ceph-mon[33213]: pgmap v764: 162 pgs: 162 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:33:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:38 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1025917351' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-05T07:33:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:38 smithi090 ceph-mon[33213]: osdmap e423: 8 total, 8 up, 6 in 2024-02-05T07:33:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:38 smithi090 ceph-mon[33213]: 6.11 scrub starts 2024-02-05T07:33:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:38 smithi090 ceph-mon[33213]: 6.11 scrub ok 2024-02-05T07:33:39.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:39.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:39.082 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:39.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:39.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:39.209 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:39.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:39.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:39.333 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:39.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:39.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:39.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:39.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:39.460 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:39.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:39.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:39.586 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:39.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:39.619 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:33:39.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:39.732 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:39.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:39.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:39.860 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:39.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:39.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:39 smithi007 ceph-mon[29098]: osdmap e424: 8 total, 8 up, 6 in 2024-02-05T07:33:39.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:39 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/524611243' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:33:39.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:39 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:33:39.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:39 smithi007 ceph-mon[29098]: 6.17 scrub starts 2024-02-05T07:33:39.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:39 smithi007 ceph-mon[29098]: 6.17 scrub ok 2024-02-05T07:33:39.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:39 smithi007 ceph-mon[37158]: osdmap e424: 8 total, 8 up, 6 in 2024-02-05T07:33:39.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:39 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/524611243' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:33:39.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:39 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:33:39.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:39 smithi007 ceph-mon[37158]: 6.17 scrub starts 2024-02-05T07:33:39.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:39 smithi007 ceph-mon[37158]: 6.17 scrub ok 2024-02-05T07:33:39.935 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 3] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:33:39.935 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:33:39.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:39.985 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:39.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:39.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:39 smithi090 ceph-mon[33213]: osdmap e424: 8 total, 8 up, 6 in 2024-02-05T07:33:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:39 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/524611243' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:33:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:39 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:33:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:39 smithi090 ceph-mon[33213]: 6.17 scrub starts 2024-02-05T07:33:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:39 smithi090 ceph-mon[33213]: 6.17 scrub ok 2024-02-05T07:33:40.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:40.115 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:40.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:40.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:40.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:40.241 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:40.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:40.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:40.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:40.366 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:40.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:40.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:40.492 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:40.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:40.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:40.565 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:33:40.584 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:33:40.595 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:40.603 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:33:40.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:40.624 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:40.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:40.625 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:33:40.645 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:33:40.666 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:33:40.687 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:33:40.707 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:33:40.709 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:33:40.709 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:33:40.709 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:33:40.709 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:33:40.709 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:33:40.709 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:33:40.709 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:33:40.709 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:33:40.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:40.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:40.752 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:40.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:40.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:40.876 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:40.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:40.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[29098]: pgmap v767: 178 pgs: 3 creating+activating, 10 unknown, 165 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 766 B/s rd, 0 op/s 2024-02-05T07:33:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[29098]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:33:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:33:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:33:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:33:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[29098]: osdmap e425: 8 total, 8 up, 6 in 2024-02-05T07:33:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[29098]: 5.14 scrub starts 2024-02-05T07:33:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[29098]: 5.14 scrub ok 2024-02-05T07:33:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[29098]: 3.1 scrub starts 2024-02-05T07:33:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[29098]: 3.1 scrub ok 2024-02-05T07:33:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2085724214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:40.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[37158]: pgmap v767: 178 pgs: 3 creating+activating, 10 unknown, 165 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 766 B/s rd, 0 op/s 2024-02-05T07:33:40.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[37158]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:33:40.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:33:40.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:33:40.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:33:40.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[37158]: osdmap e425: 8 total, 8 up, 6 in 2024-02-05T07:33:40.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[37158]: 5.14 scrub starts 2024-02-05T07:33:40.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[37158]: 5.14 scrub ok 2024-02-05T07:33:40.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[37158]: 3.1 scrub starts 2024-02-05T07:33:40.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[37158]: 3.1 scrub ok 2024-02-05T07:33:40.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:40 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2085724214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:40.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:40.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-02-05T07:33:41.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:41.003 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:41.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:41.023 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:33:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:33:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:40 smithi090 ceph-mon[33213]: pgmap v767: 178 pgs: 3 creating+activating, 10 unknown, 165 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 766 B/s rd, 0 op/s 2024-02-05T07:33:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:40 smithi090 ceph-mon[33213]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:33:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:40 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:33:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:40 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:33:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:40 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:33:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:40 smithi090 ceph-mon[33213]: osdmap e425: 8 total, 8 up, 6 in 2024-02-05T07:33:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:40 smithi090 ceph-mon[33213]: 5.14 scrub starts 2024-02-05T07:33:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:40 smithi090 ceph-mon[33213]: 5.14 scrub ok 2024-02-05T07:33:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:40 smithi090 ceph-mon[33213]: 3.1 scrub starts 2024-02-05T07:33:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:40 smithi090 ceph-mon[33213]: 3.1 scrub ok 2024-02-05T07:33:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:40 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2085724214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:41.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:41.128 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:41.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:41.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:41.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:41.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:41.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:41.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:41.388 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:41.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:41.490 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:41.514 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:41.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:41.616 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:41.641 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:41.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:41.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:41.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:41 smithi007 ceph-mon[29098]: 1.1 scrub starts 2024-02-05T07:33:41.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:41 smithi007 ceph-mon[29098]: 1.1 scrub ok 2024-02-05T07:33:41.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:41 smithi007 ceph-mon[29098]: 6.4 scrub starts 2024-02-05T07:33:41.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:41 smithi007 ceph-mon[29098]: 6.4 scrub ok 2024-02-05T07:33:41.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:41 smithi007 ceph-mon[37158]: 1.1 scrub starts 2024-02-05T07:33:41.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:41 smithi007 ceph-mon[37158]: 1.1 scrub ok 2024-02-05T07:33:41.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:41 smithi007 ceph-mon[37158]: 6.4 scrub starts 2024-02-05T07:33:41.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:41 smithi007 ceph-mon[37158]: 6.4 scrub ok 2024-02-05T07:33:41.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:41.767 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:41.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:41.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:41.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:41.896 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:41.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:41.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:41.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:42.022 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:42.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:42.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:41 smithi090 ceph-mon[33213]: 1.1 scrub starts 2024-02-05T07:33:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:41 smithi090 ceph-mon[33213]: 1.1 scrub ok 2024-02-05T07:33:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:41 smithi090 ceph-mon[33213]: 6.4 scrub starts 2024-02-05T07:33:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:41 smithi090 ceph-mon[33213]: 6.4 scrub ok 2024-02-05T07:33:42.094 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:42.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:42.151 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:42.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:42.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:42.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:42.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:42.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:33:42.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:42.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:42.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:42.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:42.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:42.536 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:42.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:42.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:42.666 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:42.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:42.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:42.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:42.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:42.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:42.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[29098]: pgmap v769: 178 pgs: 3 creating+activating, 1 unknown, 174 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:33:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[29098]: 4.17 scrub starts 2024-02-05T07:33:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[29098]: 4.17 scrub ok 2024-02-05T07:33:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[29098]: 2.1 scrub starts 2024-02-05T07:33:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[29098]: 2.1 scrub ok 2024-02-05T07:33:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[29098]: 1.3 deep-scrub starts 2024-02-05T07:33:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[29098]: 1.3 deep-scrub ok 2024-02-05T07:33:42.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[37158]: pgmap v769: 178 pgs: 3 creating+activating, 1 unknown, 174 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:33:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[37158]: 4.17 scrub starts 2024-02-05T07:33:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[37158]: 4.17 scrub ok 2024-02-05T07:33:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[37158]: 2.1 scrub starts 2024-02-05T07:33:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[37158]: 2.1 scrub ok 2024-02-05T07:33:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[37158]: 1.3 deep-scrub starts 2024-02-05T07:33:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:42 smithi007 ceph-mon[37158]: 1.3 deep-scrub ok 2024-02-05T07:33:42.924 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:42.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:42 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:42 smithi090 ceph-mon[33213]: pgmap v769: 178 pgs: 3 creating+activating, 1 unknown, 174 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:33:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:42 smithi090 ceph-mon[33213]: 4.17 scrub starts 2024-02-05T07:33:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:42 smithi090 ceph-mon[33213]: 4.17 scrub ok 2024-02-05T07:33:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:42 smithi090 ceph-mon[33213]: 2.1 scrub starts 2024-02-05T07:33:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:42 smithi090 ceph-mon[33213]: 2.1 scrub ok 2024-02-05T07:33:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:42 smithi090 ceph-mon[33213]: 1.3 deep-scrub starts 2024-02-05T07:33:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:42 smithi090 ceph-mon[33213]: 1.3 deep-scrub ok 2024-02-05T07:33:43.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:43.054 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:43.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:43.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:43.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:43.173 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:33:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:33:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 0 2024-02-05T07:33:43.173 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:33:43.184 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:43.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:43.285 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:43.309 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:43.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:43.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:43.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:43.439 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:43.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:43.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:43.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:43.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:43.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:43.696 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:43.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:43.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:43.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:43.824 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:43.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:43.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:43 smithi007 ceph-mon[29098]: 5.0 scrub starts 2024-02-05T07:33:43.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:43 smithi007 ceph-mon[29098]: 5.0 scrub ok 2024-02-05T07:33:43.826 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:33:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:33:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:33:43.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:43 smithi007 ceph-mon[37158]: 5.0 scrub starts 2024-02-05T07:33:43.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:43 smithi007 ceph-mon[37158]: 5.0 scrub ok 2024-02-05T07:33:43.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:43.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:43.951 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:43.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:43 smithi090 ceph-mon[33213]: 5.0 scrub starts 2024-02-05T07:33:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:43 smithi090 ceph-mon[33213]: 5.0 scrub ok 2024-02-05T07:33:44.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:33:44.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:44.098 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:44.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:44.200 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:44.228 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:44.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:44.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T07:33:44.329 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:44.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:44.354 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:44.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:44.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:44.479 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:44.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:44.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:44.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:44.606 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:44.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:44.664 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:44.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:44.733 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:44.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:44.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:44.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:44 smithi007 ceph-mon[29098]: pgmap v770: 178 pgs: 3 creating+activating, 175 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:33:44.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:44 smithi007 ceph-mon[29098]: 2.2 scrub starts 2024-02-05T07:33:44.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:44 smithi007 ceph-mon[29098]: 2.2 scrub ok 2024-02-05T07:33:44.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:44 smithi007 ceph-mon[29098]: 3.1f scrub starts 2024-02-05T07:33:44.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:44 smithi007 ceph-mon[29098]: 3.1f scrub ok 2024-02-05T07:33:44.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:44 smithi007 ceph-mon[29098]: 3.a scrub starts 2024-02-05T07:33:44.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:44 smithi007 ceph-mon[29098]: 3.a scrub ok 2024-02-05T07:33:44.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:44 smithi007 ceph-mon[37158]: pgmap v770: 178 pgs: 3 creating+activating, 175 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:33:44.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:44 smithi007 ceph-mon[37158]: 2.2 scrub starts 2024-02-05T07:33:44.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:44 smithi007 ceph-mon[37158]: 2.2 scrub ok 2024-02-05T07:33:44.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:44 smithi007 ceph-mon[37158]: 3.1f scrub starts 2024-02-05T07:33:44.810 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:44 smithi007 ceph-mon[37158]: 3.1f scrub ok 2024-02-05T07:33:44.810 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:44 smithi007 ceph-mon[37158]: 3.a scrub starts 2024-02-05T07:33:44.810 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:44 smithi007 ceph-mon[37158]: 3.a scrub ok 2024-02-05T07:33:44.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:44.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:44.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:44.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:44.994 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:44.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:44.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:44 smithi090 ceph-mon[33213]: pgmap v770: 178 pgs: 3 creating+activating, 175 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:33:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:44 smithi090 ceph-mon[33213]: 2.2 scrub starts 2024-02-05T07:33:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:44 smithi090 ceph-mon[33213]: 2.2 scrub ok 2024-02-05T07:33:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:44 smithi090 ceph-mon[33213]: 3.1f scrub starts 2024-02-05T07:33:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:44 smithi090 ceph-mon[33213]: 3.1f scrub ok 2024-02-05T07:33:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:44 smithi090 ceph-mon[33213]: 3.a scrub starts 2024-02-05T07:33:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:44 smithi090 ceph-mon[33213]: 3.a scrub ok 2024-02-05T07:33:45.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:45.119 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:45.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:45.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:45.244 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:45.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:45.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:45.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:45.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:45.378 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:45.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:45.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:45.507 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:45.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:45.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:45.637 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:45.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:45.638 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-02-05T07:33:45.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:45.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:45.764 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:45.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:45.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:45.890 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:45.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:45.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:45 smithi007 ceph-mon[29098]: 1.12 scrub starts 2024-02-05T07:33:45.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:45 smithi007 ceph-mon[29098]: 1.12 scrub ok 2024-02-05T07:33:45.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:45 smithi007 ceph-mon[29098]: 5.1a scrub starts 2024-02-05T07:33:45.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:45 smithi007 ceph-mon[29098]: 5.1a scrub ok 2024-02-05T07:33:45.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:45 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2032666464' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:33:45.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:45 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:33:45.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:45 smithi007 ceph-mon[37158]: 1.12 scrub starts 2024-02-05T07:33:45.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:45 smithi007 ceph-mon[37158]: 1.12 scrub ok 2024-02-05T07:33:45.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:45 smithi007 ceph-mon[37158]: 5.1a scrub starts 2024-02-05T07:33:45.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:45 smithi007 ceph-mon[37158]: 5.1a scrub ok 2024-02-05T07:33:45.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:45 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2032666464' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:33:45.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:45 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:33:45.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:46.015 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:46.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:46.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:45 smithi090 ceph-mon[33213]: 1.12 scrub starts 2024-02-05T07:33:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:45 smithi090 ceph-mon[33213]: 1.12 scrub ok 2024-02-05T07:33:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:45 smithi090 ceph-mon[33213]: 5.1a scrub starts 2024-02-05T07:33:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:45 smithi090 ceph-mon[33213]: 5.1a scrub ok 2024-02-05T07:33:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:45 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2032666464' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:33:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:45 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:33:46.115 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:46.117 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:46.143 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:46.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:46.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:46.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:46.275 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:46.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:46.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:46.405 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:46.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:46.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:46.534 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:46.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:46.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:46.581 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:46.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:46.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:46.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:46.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:46.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:46.792 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:46.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:46.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[29098]: pgmap v771: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 668 B/s rd, 0 op/s 2024-02-05T07:33:46.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:33:46.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[29098]: osdmap e426: 8 total, 8 up, 6 in 2024-02-05T07:33:46.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:33:46.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:33:46.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[29098]: 4.16 scrub starts 2024-02-05T07:33:46.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[29098]: 4.16 scrub ok 2024-02-05T07:33:46.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[29098]: 6.2 scrub starts 2024-02-05T07:33:46.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[29098]: 6.2 scrub ok 2024-02-05T07:33:46.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[29098]: 4.5 scrub starts 2024-02-05T07:33:46.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[29098]: 4.5 scrub ok 2024-02-05T07:33:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[37158]: pgmap v771: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 668 B/s rd, 0 op/s 2024-02-05T07:33:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:33:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[37158]: osdmap e426: 8 total, 8 up, 6 in 2024-02-05T07:33:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:33:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:33:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[37158]: 4.16 scrub starts 2024-02-05T07:33:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[37158]: 4.16 scrub ok 2024-02-05T07:33:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[37158]: 6.2 scrub starts 2024-02-05T07:33:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[37158]: 6.2 scrub ok 2024-02-05T07:33:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[37158]: 4.5 scrub starts 2024-02-05T07:33:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:46 smithi007 ceph-mon[37158]: 4.5 scrub ok 2024-02-05T07:33:46.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:46.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:46.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:47.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:46 smithi090 ceph-mon[33213]: pgmap v771: 178 pgs: 178 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 668 B/s rd, 0 op/s 2024-02-05T07:33:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:46 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:33:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:46 smithi090 ceph-mon[33213]: osdmap e426: 8 total, 8 up, 6 in 2024-02-05T07:33:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:46 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:33:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:33:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:46 smithi090 ceph-mon[33213]: 4.16 scrub starts 2024-02-05T07:33:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:46 smithi090 ceph-mon[33213]: 4.16 scrub ok 2024-02-05T07:33:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:46 smithi090 ceph-mon[33213]: 6.2 scrub starts 2024-02-05T07:33:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:46 smithi090 ceph-mon[33213]: 6.2 scrub ok 2024-02-05T07:33:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:46 smithi090 ceph-mon[33213]: 4.5 scrub starts 2024-02-05T07:33:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:46 smithi090 ceph-mon[33213]: 4.5 scrub ok 2024-02-05T07:33:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:47.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:47.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:47.147 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:47.176 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:47.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:47.253 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:33:47.272 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:33:47.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:47.292 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:33:47.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:47.302 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:47.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:47.312 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:33:47.332 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:33:47.352 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:33:47.372 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:33:47.392 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:33:47.394 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:33:47.394 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:33:47.394 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:33:47.394 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:33:47.394 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:33:47.394 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:33:47.394 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:33:47.394 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:33:47.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:47.431 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:47.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:47.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:47.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:47.557 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:47.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:47.590 DEBUG:teuthology.orchestra.run.smithi007:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-02-05T07:33:47.590 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-02-05T07:33:47.590 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-02-05T07:33:47.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:33:47.601 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T07:33:47.602 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T07:33:47.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:47.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:47 smithi007 ceph-mon[29098]: 6.7 scrub starts 2024-02-05T07:33:47.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:47 smithi007 ceph-mon[29098]: 6.7 scrub ok 2024-02-05T07:33:47.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:47 smithi007 ceph-mon[29098]: 5.18 scrub starts 2024-02-05T07:33:47.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:47 smithi007 ceph-mon[29098]: 5.18 scrub ok 2024-02-05T07:33:47.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:47 smithi007 ceph-mon[29098]: 1.6 scrub starts 2024-02-05T07:33:47.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:47 smithi007 ceph-mon[29098]: 1.6 scrub ok 2024-02-05T07:33:47.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:47 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4141842653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:47.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:47 smithi007 ceph-mon[37158]: 6.7 scrub starts 2024-02-05T07:33:47.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:47 smithi007 ceph-mon[37158]: 6.7 scrub ok 2024-02-05T07:33:47.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:47 smithi007 ceph-mon[37158]: 5.18 scrub starts 2024-02-05T07:33:47.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:47 smithi007 ceph-mon[37158]: 5.18 scrub ok 2024-02-05T07:33:47.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:47 smithi007 ceph-mon[37158]: 1.6 scrub starts 2024-02-05T07:33:47.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:47 smithi007 ceph-mon[37158]: 1.6 scrub ok 2024-02-05T07:33:47.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:47 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4141842653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:47.687 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:47.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:47.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:47.761 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:hints = 1 2024-02-05T07:33:47.761 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T07:33:47.762 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object prefix: benchmark_data_smithi007_157098 2024-02-05T07:33:47.762 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:33:47.762 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 0 0 0 0 0 0 - 0 2024-02-05T07:33:47.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:47.825 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:47.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:47.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:47.960 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:47.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:47.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:47 smithi090 ceph-mon[33213]: 6.7 scrub starts 2024-02-05T07:33:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:47 smithi090 ceph-mon[33213]: 6.7 scrub ok 2024-02-05T07:33:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:47 smithi090 ceph-mon[33213]: 5.18 scrub starts 2024-02-05T07:33:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:47 smithi090 ceph-mon[33213]: 5.18 scrub ok 2024-02-05T07:33:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:47 smithi090 ceph-mon[33213]: 1.6 scrub starts 2024-02-05T07:33:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:47 smithi090 ceph-mon[33213]: 1.6 scrub ok 2024-02-05T07:33:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:47 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4141842653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:48.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:48.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:48.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:48.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:48.235 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:48.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:48.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:48.376 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:48.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:48.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:48.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:48.519 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:48.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:48.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:48.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:48.658 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:48.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:48.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:48.761 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 1 16 3571 3555 222.174 222.188 0.00614423 0.00448769 2024-02-05T07:33:48.794 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:48.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:48.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[29098]: pgmap v773: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:33:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[29098]: 6.18 scrub starts 2024-02-05T07:33:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[29098]: 6.18 scrub ok 2024-02-05T07:33:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[29098]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:33:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:33:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[29098]: 5.13 scrub starts 2024-02-05T07:33:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[29098]: 5.13 scrub ok 2024-02-05T07:33:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[29098]: 1.0 deep-scrub starts 2024-02-05T07:33:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[29098]: 1.0 deep-scrub ok 2024-02-05T07:33:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[37158]: pgmap v773: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:33:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[37158]: 6.18 scrub starts 2024-02-05T07:33:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[37158]: 6.18 scrub ok 2024-02-05T07:33:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[37158]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:33:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:33:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[37158]: 5.13 scrub starts 2024-02-05T07:33:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[37158]: 5.13 scrub ok 2024-02-05T07:33:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[37158]: 1.0 deep-scrub starts 2024-02-05T07:33:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:48 smithi007 ceph-mon[37158]: 1.0 deep-scrub ok 2024-02-05T07:33:48.934 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:48.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:48.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:48 smithi090 ceph-mon[33213]: pgmap v773: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:33:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:48 smithi090 ceph-mon[33213]: 6.18 scrub starts 2024-02-05T07:33:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:48 smithi090 ceph-mon[33213]: 6.18 scrub ok 2024-02-05T07:33:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:48 smithi090 ceph-mon[33213]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:33:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:48 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:33:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:48 smithi090 ceph-mon[33213]: 5.13 scrub starts 2024-02-05T07:33:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:48 smithi090 ceph-mon[33213]: 5.13 scrub ok 2024-02-05T07:33:49.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:48 smithi090 ceph-mon[33213]: 1.0 deep-scrub starts 2024-02-05T07:33:49.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:48 smithi090 ceph-mon[33213]: 1.0 deep-scrub ok 2024-02-05T07:33:49.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:49.069 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:49.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:49.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:49.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:49.218 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:49.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:49.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:49.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:49.361 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:49.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:49.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:49.506 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:49.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:49.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:49.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:49.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:49.761 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 2 16 6851 6835 213.567 205 0.00276014 0.00467284 2024-02-05T07:33:49.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:49.855 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:49.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:49.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:49.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:50.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:50.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:50.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:50.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:50.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:33:50.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:50.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:50.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:50.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:50.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:50.410 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:50.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:50.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:50.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:50.531 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:50 smithi007 ceph-mon[29098]: 3.7 scrub starts 2024-02-05T07:33:50.531 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:50 smithi007 ceph-mon[29098]: 3.7 scrub ok 2024-02-05T07:33:50.531 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:50 smithi007 ceph-mon[29098]: pgmap v774: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 79 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 8.1 MiB/s wr, 131 op/s 2024-02-05T07:33:50.531 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:50 smithi007 ceph-mon[29098]: 4.6 scrub starts 2024-02-05T07:33:50.531 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:50 smithi007 ceph-mon[29098]: 4.6 scrub ok 2024-02-05T07:33:50.545 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:50.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:50.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:50.689 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:50.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:50.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:50.762 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 3 15 9628 9613 200.249 173.625 0.00459408 0.00498494 2024-02-05T07:33:50.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:50.818 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:50 smithi090 ceph-mon[33213]: 3.7 scrub starts 2024-02-05T07:33:50.818 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:50 smithi090 ceph-mon[33213]: 3.7 scrub ok 2024-02-05T07:33:50.818 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:50 smithi090 ceph-mon[33213]: pgmap v774: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 79 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 8.1 MiB/s wr, 131 op/s 2024-02-05T07:33:50.818 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:50 smithi090 ceph-mon[33213]: 4.6 scrub starts 2024-02-05T07:33:50.819 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:50 smithi090 ceph-mon[33213]: 4.6 scrub ok 2024-02-05T07:33:50.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:50 smithi007 ceph-mon[37158]: 3.7 scrub starts 2024-02-05T07:33:50.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:50 smithi007 ceph-mon[37158]: 3.7 scrub ok 2024-02-05T07:33:50.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:50 smithi007 ceph-mon[37158]: pgmap v774: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 79 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 8.1 MiB/s wr, 131 op/s 2024-02-05T07:33:50.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:50 smithi007 ceph-mon[37158]: 4.6 scrub starts 2024-02-05T07:33:50.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:50 smithi007 ceph-mon[37158]: 4.6 scrub ok 2024-02-05T07:33:50.834 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:50.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:50.936 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:50.978 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:50.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:51.080 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:51.137 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:51.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:51.169 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:33:51.170 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:33:51.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:51.271 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:51.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:51.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:51.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:33:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:33:51.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:51.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:51.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:51.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:51.561 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:51.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:51.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:51.564 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:33:51.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:51.704 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:51.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:51.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:51.762 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 4 16 12019 12003 187.528 149.375 0.00270157 0.00527457 2024-02-05T07:33:51.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:51.830 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:51.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:51.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:51.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:51 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:33:51.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:51 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:33:51.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:51 smithi007 ceph-mon[29098]: 5.4 scrub starts 2024-02-05T07:33:51.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:51 smithi007 ceph-mon[29098]: 5.4 scrub ok 2024-02-05T07:33:51.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:51 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2178971510' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:33:51.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:51 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:33:51.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:51 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:33:51.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:51 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:33:51.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:51 smithi007 ceph-mon[37158]: 5.4 scrub starts 2024-02-05T07:33:51.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:51 smithi007 ceph-mon[37158]: 5.4 scrub ok 2024-02-05T07:33:51.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:51 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2178971510' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:33:51.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:51 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:33:51.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:51.965 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:51.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:51.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:51 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:33:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:51 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:33:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:51 smithi090 ceph-mon[33213]: 5.4 scrub starts 2024-02-05T07:33:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:51 smithi090 ceph-mon[33213]: 5.4 scrub ok 2024-02-05T07:33:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:51 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2178971510' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:33:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:51 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:33:52.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:52.109 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:52.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:52.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:52.251 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:52.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:52.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:52.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:52.385 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:52.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:52.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:52.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T07:33:52.487 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:52.519 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:52.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:52.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:52.653 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:52.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:52.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:52.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[29098]: 5.17 deep-scrub starts 2024-02-05T07:33:52.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[29098]: 5.17 deep-scrub ok 2024-02-05T07:33:52.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:52.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[29098]: 3.10 scrub starts 2024-02-05T07:33:52.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[29098]: 3.10 scrub ok 2024-02-05T07:33:52.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[29098]: pgmap v775: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 399 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 40 MiB/s wr, 637 op/s 2024-02-05T07:33:52.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:33:52.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:33:52.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[29098]: osdmap e427: 8 total, 8 up, 6 in 2024-02-05T07:33:52.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[37158]: 5.17 deep-scrub starts 2024-02-05T07:33:52.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[37158]: 5.17 deep-scrub ok 2024-02-05T07:33:52.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:52.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[37158]: 3.10 scrub starts 2024-02-05T07:33:52.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[37158]: 3.10 scrub ok 2024-02-05T07:33:52.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[37158]: pgmap v775: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 399 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 40 MiB/s wr, 637 op/s 2024-02-05T07:33:52.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:33:52.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:33:52.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:52 smithi007 ceph-mon[37158]: osdmap e427: 8 total, 8 up, 6 in 2024-02-05T07:33:52.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:52.762 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 5 16 14458 14442 180.507 152.438 0.00605355 0.00553231 2024-02-05T07:33:52.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:52.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:52.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:52.934 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:52.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:52 smithi090 ceph-mon[33213]: 5.17 deep-scrub starts 2024-02-05T07:33:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:52 smithi090 ceph-mon[33213]: 5.17 deep-scrub ok 2024-02-05T07:33:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:52 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:33:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:52 smithi090 ceph-mon[33213]: 3.10 scrub starts 2024-02-05T07:33:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:52 smithi090 ceph-mon[33213]: 3.10 scrub ok 2024-02-05T07:33:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:52 smithi090 ceph-mon[33213]: pgmap v775: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 399 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 40 MiB/s wr, 637 op/s 2024-02-05T07:33:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:52 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:33:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:52 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:33:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:52 smithi090 ceph-mon[33213]: osdmap e427: 8 total, 8 up, 6 in 2024-02-05T07:33:53.036 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:53.075 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:53.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:53.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:53.221 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:53.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:53.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:53.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:53.356 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:53.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:53.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:53.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:53.498 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:53.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:53.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:53.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:53.633 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:53.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:53.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:53.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:53.762 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 6 16 17122 17106 178.171 166.5 0.00664268 0.0056057 2024-02-05T07:33:53.769 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:53.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:53.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:53.903 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:53.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:53.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:33:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:33:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:33:54.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:54.020 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:54.068 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:54.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:54.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:54.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:54.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:54.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:54.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:54.336 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:54.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:54.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:54.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:54.471 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:54.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:54.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:54.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:54.607 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:54.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:54.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:54.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:54 smithi007 ceph-mon[29098]: pgmap v777: 178 pgs: 178 active+clean; 449 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 897 op/s 2024-02-05T07:33:54.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:54 smithi007 ceph-mon[37158]: pgmap v777: 178 pgs: 178 active+clean; 449 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 897 op/s 2024-02-05T07:33:54.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:54.740 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:54.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:54.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:54.762 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 7 16 19561 19545 174.483 152.438 0.005691 0.00572453 2024-02-05T07:33:54.795 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:33:54.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:54.874 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:54.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:54.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:55.007 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:55.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:55.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:55.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:54 smithi090 ceph-mon[33213]: pgmap v777: 178 pgs: 178 active+clean; 449 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 897 op/s 2024-02-05T07:33:55.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:55.150 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:55.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:55.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:55.293 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:55.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:55.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:55.427 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:55.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:55.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:55.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:55.567 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:55.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:55.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:55.669 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:55.707 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:55.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:55.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:55.762 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 8 16 21886 21870 170.835 145.312 0.00541022 0.00584716 2024-02-05T07:33:55.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:55.849 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:55.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:55.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:55.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:55.989 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:55.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:55.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:56.021 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:33:56.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:56.118 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:56.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:56.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:56.220 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:56.267 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:56.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:56.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:56.342 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:33:56.342 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:33:56.342 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:33:56.342 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:33:56.342 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:33:56.343 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:33:56.343 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:33:56.343 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:33:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:33:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:33:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:33:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:33:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:33:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:33:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:33:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:33:56.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:56.426 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:56.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:56.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:56.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:56.561 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:56.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:56.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:56.663 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:56.711 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:56.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:56.762 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 9 16 23911 23895 165.915 126.562 0.00469878 0.00602146 2024-02-05T07:33:56.812 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:56.825 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:56 smithi090 ceph-mon[33213]: pgmap v778: 178 pgs: 178 active+clean; 769 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.0 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T07:33:56.825 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:56 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3391139669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:56.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:56.853 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:56.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:56.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:56 smithi007 ceph-mon[29098]: pgmap v778: 178 pgs: 178 active+clean; 769 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.0 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T07:33:56.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:56 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3391139669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:56.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:56 smithi007 ceph-mon[37158]: pgmap v778: 178 pgs: 178 active+clean; 769 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.0 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T07:33:56.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:56 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3391139669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:33:56.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:56.986 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:56.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:56.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:57.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:57.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:57.129 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:57.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:57.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:57.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:57.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:57.362 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:57.395 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:57.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:57.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:57.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:33:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:57.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:57.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:33:57.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:57.663 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:57.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:57.763 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 10 16 26121 26105 163.135 138.125 0.00638554 0.0061233 2024-02-05T07:33:57.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:33:57.797 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:57.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:33:57.899 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:57.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:58.045 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:33:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:58.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:58.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:33:58.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:58.210 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:58.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:58.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:58.358 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:58.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:58.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:58.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:58.505 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:58.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:58.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:58.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:33:58.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:58.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:33:58.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:58.763 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 11 16 28224 28208 160.251 131.438 0.0135413 0.0062335 2024-02-05T07:33:58.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:58.801 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:58.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:58.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:58.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:58 smithi007 ceph-mon[29098]: pgmap v779: 178 pgs: 178 active+clean; 1.2 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 125 MiB/s wr, 1.99k op/s 2024-02-05T07:33:58.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:58 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:33:58.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:58 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:33:58.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:33:58 smithi007 ceph-mon[29098]: osdmap e428: 8 total, 8 up, 6 in 2024-02-05T07:33:58.919 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:58 smithi007 ceph-mon[37158]: pgmap v779: 178 pgs: 178 active+clean; 1.2 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 125 MiB/s wr, 1.99k op/s 2024-02-05T07:33:58.920 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:58 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:33:58.920 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:58 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:33:58.920 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:33:58 smithi007 ceph-mon[37158]: osdmap e428: 8 total, 8 up, 6 in 2024-02-05T07:33:58.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:58.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:58.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:59.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:58 smithi090 ceph-mon[33213]: pgmap v779: 178 pgs: 178 active+clean; 1.2 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 125 MiB/s wr, 1.99k op/s 2024-02-05T07:33:59.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:58 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:33:59.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:58 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:33:59.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:33:58 smithi090 ceph-mon[33213]: osdmap e428: 8 total, 8 up, 6 in 2024-02-05T07:33:59.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:59.067 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:59.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:59.169 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:59.202 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:59.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:59.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:33:59.327 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:59.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:59.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:33:59.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:59.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:33:59.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:33:59.586 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:59.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:59.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:33:59.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:33:59.688 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:33:59.763 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 12 15 29851 29836 155.375 101.75 0.012416 0.00642905 2024-02-05T07:33:59.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:59.775 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:33:59.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:33:59.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:33:59.902 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:33:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:33:59.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:00.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:00.036 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:00.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:00.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:00.163 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:00.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:00.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:00.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:00.298 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:00.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:00.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:00.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:00.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:00.619 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:00.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:00.721 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:00.744 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:00.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:00.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:00.763 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 13 16 31127 31111 149.553 79.6875 0.00412342 0.00659479 2024-02-05T07:34:00.828 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:00 smithi090 ceph-mon[33213]: pgmap v781: 178 pgs: 178 active+clean; 1.5 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.22k op/s 2024-02-05T07:34:00.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:00.877 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:00.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:00.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:00.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:00 smithi007 ceph-mon[29098]: pgmap v781: 178 pgs: 178 active+clean; 1.5 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.22k op/s 2024-02-05T07:34:00.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:00 smithi007 ceph-mon[37158]: pgmap v781: 178 pgs: 178 active+clean; 1.5 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.22k op/s 2024-02-05T07:34:00.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:01.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:01.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:01.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:01.169 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:01.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:01.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:01.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:34:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:34:01.296 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:01.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:01.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:01.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:01.426 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:01.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:01.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:01.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:01.561 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:01.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:01.663 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:01.696 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:01.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:01.763 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 14 16 32764 32748 146.178 102.312 0.00425607 0.00683446 2024-02-05T07:34:01.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:01.832 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:01.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:01.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:01.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:01 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:01.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:01 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:34:01.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:01 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:01.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:01 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:34:01.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:01.965 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:01.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:02.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:01 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:02.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:01 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:34:02.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:02.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:02.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:02.241 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:02.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:02.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:02.385 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:02.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:02.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:02.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:02.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:02.511 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:02.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:02.612 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:02.657 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:02.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:02.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:02.764 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 15 16 34244 34228 142.59 92.5 0.00907258 0.00700768 2024-02-05T07:34:02.797 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:02.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:02.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:02.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:02.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:02 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:02.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:02 smithi007 ceph-mon[29098]: pgmap v782: 178 pgs: 178 active+clean; 1.8 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 1.0 KiB/s rd, 149 MiB/s wr, 2.38k op/s 2024-02-05T07:34:02.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:02 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:34:02.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:02 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:34:02.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:02 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:02.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:02 smithi007 ceph-mon[37158]: pgmap v782: 178 pgs: 178 active+clean; 1.8 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 1.0 KiB/s rd, 149 MiB/s wr, 2.38k op/s 2024-02-05T07:34:02.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:02 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:34:02.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:02 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:34:02.932 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:02.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:02.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:03.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:02 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:03.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:02 smithi090 ceph-mon[33213]: pgmap v782: 178 pgs: 178 active+clean; 1.8 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 1.0 KiB/s rd, 149 MiB/s wr, 2.38k op/s 2024-02-05T07:34:03.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:02 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:34:03.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:02 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:34:03.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:03.073 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:03.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:03.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:03.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:03.198 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:03.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:03.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:03.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:03.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:03.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:03.455 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:03.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:03.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:03.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:03.589 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:03.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:03.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:03.685 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:34:03.685 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-02-05T07:34:03.685 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-02-05T07:34:03.685 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:34:03.686 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.686 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:200399 8.c 8:369e3f1b:::benchmark_data_smithi007_105755_object64089:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e290)", 2024-02-05T07:34:03.686 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:28:12.480723+0000", 2024-02-05T07:34:03.686 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 350.09009961499999, 2024-02-05T07:34:03.686 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.072542493, 2024-02-05T07:34:03.686 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.686 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:34:03.686 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.686 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:34:03.686 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:03.686 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200399 2024-02-05T07:34:03.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.686 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.686 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480723+0000", 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480723+0000", 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480724+0000", 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3799999999999996e-07 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.688 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.688 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480727+0000", 2024-02-05T07:34:03.688 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6239999999999999e-06 2024-02-05T07:34:03.688 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.688 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.688 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.688 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480735+0000", 2024-02-05T07:34:03.688 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.357e-06 2024-02-05T07:34:03.688 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.688 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.688 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.688 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480742+0000", 2024-02-05T07:34:03.688 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.097e-06 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480875+0000", 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013302 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480883+0000", 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.661e-06 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.490966+0000", 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010083844999999999 2024-02-05T07:34:03.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.490976+0000", 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4399999999999994e-06 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.553187+0000", 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0622112269999999 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.553266+0000", 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8484000000000003e-05 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.690 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:200403 8.c 8:36ae280d:::benchmark_data_smithi007_105755_object64093:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e290)", 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:28:12.480754+0000", 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 350.09006911799997, 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.073251822, 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200403 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.691 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480754+0000", 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480754+0000", 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480754+0000", 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8000000000000001e-07 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480756+0000", 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9530000000000002e-06 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480761+0000", 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2469999999999999e-06 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480766+0000", 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1699999999999996e-06 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.693 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480990+0000", 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00022330000000000001 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting_for_map not empty", 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480995+0000", 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5199999999999997e-06 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.491000+0000", 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010004677 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.694 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.491006+0000", 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1020000000000002e-06 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.553938+0000", 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0629322329999999 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.554005+0000", 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.724e-05 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.696 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.696 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:200429 8.c 8:39182b98:::benchmark_data_smithi007_105755_object64119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-02-05T07:34:03.696 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:28:12.503389+0000", 2024-02-05T07:34:03.696 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 350.06743393800002, 2024-02-05T07:34:03.696 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0624662739999999, 2024-02-05T07:34:03.696 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.696 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.696 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.696 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:34:03.696 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:03.696 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200429 2024-02-05T07:34:03.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.696 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.696 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503389+0000", 2024-02-05T07:34:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503389+0000", 2024-02-05T07:34:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503390+0000", 2024-02-05T07:34:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3e-06 2024-02-05T07:34:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503395+0000", 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2469999999999999e-06 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503406+0000", 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0305e-05 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503413+0000", 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5390000000000002e-06 2024-02-05T07:34:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503474+0000", 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0556000000000001e-05 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503483+0000", 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8699999999999998e-06 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.554034+0000", 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0505512459999999 2024-02-05T07:34:03.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.554148+0000", 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113948 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.554485+0000", 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00033712500000000002 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.561433+0000", 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0069477769999999996 2024-02-05T07:34:03.700 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.561478+0000", 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5457999999999997e-05 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.565745+0000", 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.004267374 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.565805+0000", 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9747999999999998e-05 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.701 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.565855+0000", 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9781000000000003e-05 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:200448 8.c 8:3b8b152a:::benchmark_data_smithi007_105755_object64138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:28:12.525915+0000", 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 350.04490734900003, 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0402964589999999, 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.702 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200448 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.525915+0000", 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.525915+0000", 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.703 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.525917+0000", 2024-02-05T07:34:03.704 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8530000000000001e-06 2024-02-05T07:34:03.704 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.704 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.704 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.704 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.525925+0000", 2024-02-05T07:34:03.704 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7379999999999994e-06 2024-02-05T07:34:03.704 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.704 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.704 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.704 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.525946+0000", 2024-02-05T07:34:03.704 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1277e-05 2024-02-05T07:34:03.704 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.704 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.704 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.525957+0000", 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0308e-05 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.526027+0000", 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0742999999999997e-05 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.526038+0000", 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.094e-05 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.705 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.559347+0000", 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0333087940000001 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.559469+0000", 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012193600000000001 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.559585+0000", 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011568799999999999 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.706 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.565685+0000", 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0061002809999999999 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.565915+0000", 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023005000000000001 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.566126+0000", 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00021152399999999999 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.566177+0000", 2024-02-05T07:34:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0293000000000001e-05 2024-02-05T07:34:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.566212+0000", 2024-02-05T07:34:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5033999999999997e-05 2024-02-05T07:34:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:271911 8.13 8:ce51e4df:::benchmark_data_smithi007_105755_object135601:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.609391+0000", 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 274.96143166000002, 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.103982032, 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 271911 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.709 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609391+0000", 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609391+0000", 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609392+0000", 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.252e-06 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.710 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609396+0000", 2024-02-05T07:34:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7419999999999999e-06 2024-02-05T07:34:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609405+0000", 2024-02-05T07:34:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0149999999999992e-06 2024-02-05T07:34:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609413+0000", 2024-02-05T07:34:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6399999999999997e-06 2024-02-05T07:34:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.711 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609478+0000", 2024-02-05T07:34:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4993999999999995e-05 2024-02-05T07:34:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-02-05T07:34:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609485+0000", 2024-02-05T07:34:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4630000000000004e-06 2024-02-05T07:34:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.618256+0000", 2024-02-05T07:34:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0087711200000000003 2024-02-05T07:34:03.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.713 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.713 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.618265+0000", 2024-02-05T07:34:03.713 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7169999999999997e-06 2024-02-05T07:34:03.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.713 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.713 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.683143+0000", 2024-02-05T07:34:03.713 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0648781140000001 2024-02-05T07:34:03.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.713 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for missing object", 2024-02-05T07:34:03.713 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.683264+0000", 2024-02-05T07:34:03.713 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121237 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.713247+0000", 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.029982775999999999 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.713373+0000", 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000125962 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:271929 8.8 8:15173cd4:::benchmark_data_smithi007_105755_object135619:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.624345+0000", 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 274.94647798900002, 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0390269809999999, 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 271929 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.715 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624345+0000", 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624345+0000", 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624345+0000", 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4000000000000001e-07 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624349+0000", 2024-02-05T07:34:03.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9779999999999999e-06 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624359+0000", 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6309999999999998e-06 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624366+0000", 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0729999999999999e-06 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624778+0000", 2024-02-05T07:34:03.717 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00041162499999999999 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624786+0000", 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5339999999999999e-06 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.653311+0000", 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0285250699999999 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.653431+0000", 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000119527 2024-02-05T07:34:03.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.653812+0000", 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00038111800000000001 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.659043+0000", 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0052313819999999997 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.663156+0000", 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041125220000000004 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.720 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.720 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.663222+0000", 2024-02-05T07:34:03.720 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5782000000000001e-05 2024-02-05T07:34:03.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.720 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.720 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.663321+0000", 2024-02-05T07:34:03.720 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9026999999999999e-05 2024-02-05T07:34:03.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.720 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.720 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.663372+0000", 2024-02-05T07:34:03.720 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0971999999999999e-05 2024-02-05T07:34:03.720 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:271931 8.8 8:10a7a263:::benchmark_data_smithi007_105755_object135621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.630044+0000", 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 274.940778368, 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.038097405, 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 271931 2024-02-05T07:34:03.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630044+0000", 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630044+0000", 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630046+0000", 2024-02-05T07:34:03.722 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3990000000000001e-06 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630049+0000", 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3840000000000001e-06 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630057+0000", 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8429999999999993e-06 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630063+0000", 2024-02-05T07:34:03.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4949999999999996e-06 2024-02-05T07:34:03.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630175+0000", 2024-02-05T07:34:03.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000111443 2024-02-05T07:34:03.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630184+0000", 2024-02-05T07:34:03.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1260000000000004e-06 2024-02-05T07:34:03.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.659103+0000", 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.028919361 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.659235+0000", 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013129 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.659391+0000", 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015600100000000001 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.663439+0000", 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0040486960000000001 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.663567+0000", 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012772799999999999 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.668046+0000", 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044785299999999997 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.668100+0000", 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4119999999999997e-05 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.668142+0000", 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1989000000000001e-05 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:271936 8.13 8:c86abfbb:::benchmark_data_smithi007_105755_object135626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.631907+0000", 2024-02-05T07:34:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 274.93891615899997, 2024-02-05T07:34:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0919809620000001, 2024-02-05T07:34:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:34:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 271936 2024-02-05T07:34:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.631907+0000", 2024-02-05T07:34:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.631907+0000", 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.631907+0000", 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1200000000000002e-07 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.631911+0000", 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5559999999999999e-06 2024-02-05T07:34:03.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.631920+0000", 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8240000000000006e-06 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.631927+0000", 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2690000000000002e-06 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.635499+0000", 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003572062 2024-02-05T07:34:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.635509+0000", 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7799999999999995e-06 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.683531+0000", 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.048021976 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.683661+0000", 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013036600000000001 2024-02-05T07:34:03.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.684156+0000", 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00049498499999999998 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722733+0000", 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.038577127000000003 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722799+0000", 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6266999999999994e-05 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.733 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722945+0000", 2024-02-05T07:34:03.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014543100000000001 2024-02-05T07:34:03.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.733 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.723147+0000", 2024-02-05T07:34:03.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020186100000000001 2024-02-05T07:34:03.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.733 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.723887+0000", 2024-02-05T07:34:03.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00074074600000000003 2024-02-05T07:34:03.733 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:271951 8.13 8:cc87e9b1:::benchmark_data_smithi007_105755_object135641:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.641980+0000", 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 274.92884234799999, 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0821172720000001, 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 271951 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.734 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.641980+0000", 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.641980+0000", 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.641981+0000", 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.105e-06 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.641986+0000", 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4909999999999999e-06 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.642000+0000", 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4235e-05 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.642008+0000", 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6320000000000008e-06 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.736 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.643114+0000", 2024-02-05T07:34:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001106459 2024-02-05T07:34:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.643125+0000", 2024-02-05T07:34:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0823e-05 2024-02-05T07:34:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.706220+0000", 2024-02-05T07:34:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.063094722 2024-02-05T07:34:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.706363+0000", 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000142714 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.706511+0000", 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014864699999999999 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722741+0000", 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016229389 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722884+0000", 2024-02-05T07:34:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014346699999999999 2024-02-05T07:34:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.723945+0000", 2024-02-05T07:34:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001060851 2024-02-05T07:34:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.723994+0000", 2024-02-05T07:34:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9403000000000001e-05 2024-02-05T07:34:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724098+0000", 2024-02-05T07:34:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103334 2024-02-05T07:34:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:271972 8.13 8:ca91f266:::benchmark_data_smithi007_105755_object135662:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:34:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.666032+0000", 2024-02-05T07:34:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 274.90479106399999, 2024-02-05T07:34:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0584022369999999, 2024-02-05T07:34:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.740 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 271972 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666032+0000", 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666032+0000", 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666032+0000", 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5799999999999999e-07 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666036+0000", 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7079999999999999e-06 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666045+0000", 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7960000000000001e-06 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666052+0000", 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9480000000000002e-06 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666191+0000", 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013942699999999999 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666201+0000", 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8309999999999996e-06 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.708519+0000", 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.042318219 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.708648+0000", 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012891899999999999 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.708787+0000", 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013839200000000001 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722746+0000", 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013959651 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724143+0000", 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0013966180000000001 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724188+0000", 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5382e-05 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724284+0000", 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5748999999999994e-05 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724434+0000", 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000149939 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:272009 8.13 8:ce2194ad:::benchmark_data_smithi007_105755_object135699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.712912+0000", 2024-02-05T07:34:03.746 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 274.85791024000002, 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0118669250000001, 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 272009 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.712912+0000", 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.712912+0000", 2024-02-05T07:34:03.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.748 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.712914+0000", 2024-02-05T07:34:03.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3260000000000001e-06 2024-02-05T07:34:03.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.748 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.712919+0000", 2024-02-05T07:34:03.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9760000000000003e-06 2024-02-05T07:34:03.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.712935+0000", 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5977e-05 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.712943+0000", 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.551e-06 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.713018+0000", 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4529000000000004e-05 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.713029+0000", 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1487e-05 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.710502+0000", 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99747268200000005 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.710622+0000", 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000120048 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:34:03.750 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.710753+0000", 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013055200000000001 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722752+0000", 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011999157999999999 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724485+0000", 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017332140000000001 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724578+0000", 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3110999999999997e-05 2024-02-05T07:34:03.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724671+0000", 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2610999999999998e-05 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724779+0000", 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010870300000000001 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:272018 8.13 8:ce136c69:::benchmark_data_smithi007_105755_object135708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:34:03.752 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.726075+0000", 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 274.84474763600002, 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99892678000000001, 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 272018 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726075+0000", 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726075+0000", 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726076+0000", 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.235e-06 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726081+0000", 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8099999999999997e-06 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726092+0000", 2024-02-05T07:34:03.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0865e-05 2024-02-05T07:34:03.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726100+0000", 2024-02-05T07:34:03.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9969999999999998e-06 2024-02-05T07:34:03.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726173+0000", 2024-02-05T07:34:03.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3363999999999998e-05 2024-02-05T07:34:03.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726185+0000", 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1368e-05 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.713401+0000", 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98721674500000001 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.713510+0000", 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108312 2024-02-05T07:34:03.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.713648+0000", 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000137837 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722757+0000", 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0091097469999999996 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724532+0000", 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017743100000000001 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724827+0000", 2024-02-05T07:34:03.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029515699999999999 2024-02-05T07:34:03.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724880+0000", 2024-02-05T07:34:03.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3392e-05 2024-02-05T07:34:03.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.725002+0000", 2024-02-05T07:34:03.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121641 2024-02-05T07:34:03.758 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:125089 9.7 9:e4c9f568:::benchmark_data_smithi007_136200_object45244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:39.293800+0000", 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 83.277022345000006, 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0385608829999999, 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 125089 2024-02-05T07:34:03.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.760 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.760 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.293800+0000", 2024-02-05T07:34:03.760 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.760 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.293800+0000", 2024-02-05T07:34:03.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.761 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.293801+0000", 2024-02-05T07:34:03.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.13e-06 2024-02-05T07:34:03.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.761 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.293805+0000", 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.63e-06 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.293813+0000", 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0560000000000005e-06 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.293820+0000", 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5729999999999996e-06 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.293869+0000", 2024-02-05T07:34:03.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8967000000000001e-05 2024-02-05T07:34:03.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.294134+0000", 2024-02-05T07:34:03.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026571800000000002 2024-02-05T07:34:03.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:34:03.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.294253+0000", 2024-02-05T07:34:03.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011818 2024-02-05T07:34:03.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.310233+0000", 2024-02-05T07:34:03.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.015980755999999999 2024-02-05T07:34:03.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.764 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.310248+0000", 2024-02-05T07:34:03.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4812e-05 2024-02-05T07:34:03.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.764 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.326702+0000", 2024-02-05T07:34:03.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016454116000000001 2024-02-05T07:34:03.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.326712+0000", 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0074e-05 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.332273+0000", 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.005561138 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.332361+0000", 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7732999999999995e-05 2024-02-05T07:34:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.766 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.766 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.766 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:125118 9.7 9:ec3ff33e:::benchmark_data_smithi007_136200_object45273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e404)", 2024-02-05T07:34:03.766 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:39.335732+0000", 2024-02-05T07:34:03.766 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 83.235090455000005, 2024-02-05T07:34:03.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.004064861, 2024-02-05T07:34:03.766 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.766 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.766 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.766 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:34:03.766 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:34:03.766 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 125118 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335732+0000", 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335732+0000", 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335733+0000", 2024-02-05T07:34:03.767 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1650000000000001e-06 2024-02-05T07:34:03.768 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.768 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.768 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.768 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335737+0000", 2024-02-05T07:34:03.768 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3890000000000001e-06 2024-02-05T07:34:03.768 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.768 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.768 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.768 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335747+0000", 2024-02-05T07:34:03.768 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0152e-05 2024-02-05T07:34:03.768 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.768 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.768 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.768 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335754+0000", 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5379999999999997e-06 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335831+0000", 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6379999999999997e-05 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335857+0000", 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5741e-05 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.332771+0000", 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99691459800000004 2024-02-05T07:34:03.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.332894+0000", 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012235300000000001 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.333042+0000", 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014887799999999999 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.339562+0000", 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0065198979999999997 2024-02-05T07:34:03.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.339629+0000", 2024-02-05T07:34:03.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6812999999999999e-05 2024-02-05T07:34:03.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.339708+0000", 2024-02-05T07:34:03.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9349000000000002e-05 2024-02-05T07:34:03.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.339761+0000", 2024-02-05T07:34:03.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2426999999999998e-05 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.339797+0000", 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6180000000000003e-05 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:125390 9.8 9:1235e142:::benchmark_data_smithi007_136200_object45545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:40.336843+0000", 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 82.233979973000004, 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0096065670000001, 2024-02-05T07:34:03.772 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 125390 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.336843+0000", 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.336843+0000", 2024-02-05T07:34:03.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.336844+0000", 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.198e-06 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.336848+0000", 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0609999999999997e-06 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.336859+0000", 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0975000000000001e-05 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.336867+0000", 2024-02-05T07:34:03.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3140000000000004e-06 2024-02-05T07:34:03.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.336997+0000", 2024-02-05T07:34:03.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012979699999999999 2024-02-05T07:34:03.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.337013+0000", 2024-02-05T07:34:03.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6231000000000001e-05 2024-02-05T07:34:03.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.341645+0000", 2024-02-05T07:34:03.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0046315459999999 2024-02-05T07:34:03.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.341802+0000", 2024-02-05T07:34:03.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015739000000000001 2024-02-05T07:34:03.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:03.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.341952+0000", 2024-02-05T07:34:03.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015017599999999999 2024-02-05T07:34:03.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.344427+0000", 2024-02-05T07:34:03.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024746460000000001 2024-02-05T07:34:03.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.346280+0000", 2024-02-05T07:34:03.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0018532310000000001 2024-02-05T07:34:03.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.778 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.778 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.346348+0000", 2024-02-05T07:34:03.778 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7379000000000007e-05 2024-02-05T07:34:03.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.778 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.778 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.346405+0000", 2024-02-05T07:34:03.778 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7061000000000001e-05 2024-02-05T07:34:03.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.778 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.778 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.346449+0000", 2024-02-05T07:34:03.779 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4561999999999998e-05 2024-02-05T07:34:03.779 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.779 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.779 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.779 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:125406 9.8 9:19e233ec:::benchmark_data_smithi007_136200_object45561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-02-05T07:34:03.779 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:40.358368+0000", 2024-02-05T07:34:03.779 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 82.212454457000007, 2024-02-05T07:34:03.779 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99250793900000001, 2024-02-05T07:34:03.779 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.779 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.779 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:34:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:34:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 125406 2024-02-05T07:34:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358368+0000", 2024-02-05T07:34:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.780 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358368+0000", 2024-02-05T07:34:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358369+0000", 2024-02-05T07:34:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.111e-06 2024-02-05T07:34:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358373+0000", 2024-02-05T07:34:03.781 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8489999999999999e-06 2024-02-05T07:34:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358385+0000", 2024-02-05T07:34:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1976e-05 2024-02-05T07:34:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358395+0000", 2024-02-05T07:34:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6900000000000004e-06 2024-02-05T07:34:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.782 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358702+0000", 2024-02-05T07:34:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030728500000000002 2024-02-05T07:34:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358712+0000", 2024-02-05T07:34:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7149999999999993e-06 2024-02-05T07:34:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.344459+0000", 2024-02-05T07:34:03.783 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98574705799999995 2024-02-05T07:34:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.344597+0000", 2024-02-05T07:34:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013843500000000001 2024-02-05T07:34:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.344688+0000", 2024-02-05T07:34:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0884999999999995e-05 2024-02-05T07:34:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.350688+0000", 2024-02-05T07:34:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0060001389999999998 2024-02-05T07:34:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.350752+0000", 2024-02-05T07:34:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4022000000000005e-05 2024-02-05T07:34:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.350781+0000", 2024-02-05T07:34:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.813e-05 2024-02-05T07:34:03.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.350831+0000", 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0778000000000001e-05 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.350876+0000", 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4866e-05 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.786 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:125412 9.8 9:1a6cf359:::benchmark_data_smithi007_136200_object45567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-02-05T07:34:03.787 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:40.362668+0000", 2024-02-05T07:34:03.787 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 82.208155172999994, 2024-02-05T07:34:03.787 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99224961300000003, 2024-02-05T07:34:03.787 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.787 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.787 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.787 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:34:03.787 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:34:03.787 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 125412 2024-02-05T07:34:03.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.787 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.787 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.787 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362668+0000", 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362668+0000", 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362669+0000", 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.263e-06 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362672+0000", 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3000000000000002e-06 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362680+0000", 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6510000000000001e-06 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362686+0000", 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2979999999999997e-06 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362836+0000", 2024-02-05T07:34:03.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014964899999999999 2024-02-05T07:34:03.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.790 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362846+0000", 2024-02-05T07:34:03.790 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0237e-05 2024-02-05T07:34:03.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.790 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.346606+0000", 2024-02-05T07:34:03.790 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98375975400000004 2024-02-05T07:34:03.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.790 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.346719+0000", 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011328999999999999 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.346808+0000", 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8731000000000001e-05 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.350945+0000", 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041377760000000001 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.791 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.350963+0000", 2024-02-05T07:34:03.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.753e-05 2024-02-05T07:34:03.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.354797+0000", 2024-02-05T07:34:03.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003834456 2024-02-05T07:34:03.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.354860+0000", 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2292000000000006e-05 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.354917+0000", 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7386000000000002e-05 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:125415 9.8 9:1d0f3dec:::benchmark_data_smithi007_136200_object45570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:40.368752+0000", 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 82.202070919999997, 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98631972800000001, 2024-02-05T07:34:03.793 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 125415 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368752+0000", 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.794 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368752+0000", 2024-02-05T07:34:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368754+0000", 2024-02-05T07:34:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9489999999999999e-06 2024-02-05T07:34:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368759+0000", 2024-02-05T07:34:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4770000000000002e-06 2024-02-05T07:34:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.796 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368777+0000", 2024-02-05T07:34:03.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7572999999999999e-05 2024-02-05T07:34:03.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.796 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.796 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368786+0000", 2024-02-05T07:34:03.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7839999999999992e-06 2024-02-05T07:34:03.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.796 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.796 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368885+0000", 2024-02-05T07:34:03.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9445999999999996e-05 2024-02-05T07:34:03.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.797 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:34:03.797 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368895+0000", 2024-02-05T07:34:03.797 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0104e-05 2024-02-05T07:34:03.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.797 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.797 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.348592+0000", 2024-02-05T07:34:03.797 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97969647900000001 2024-02-05T07:34:03.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.797 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.348695+0000", 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103731 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.348801+0000", 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000105932 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.351003+0000", 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0022017790000000001 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.351014+0000", 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0637e-05 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.354982+0000", 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039685950000000001 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.355034+0000", 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1740000000000003e-05 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.355071+0000", 2024-02-05T07:34:03.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7502000000000001e-05 2024-02-05T07:34:03.800 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.800 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.800 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.800 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:137081 9.8 9:175c584a:::benchmark_data_smithi007_136200_object57236:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-02-05T07:34:03.800 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:51.457329+0000", 2024-02-05T07:34:03.800 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 71.113493602999995, 2024-02-05T07:34:03.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.12851328100000001, 2024-02-05T07:34:03.800 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.800 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.800 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.800 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:34:03.801 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:34:03.801 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 137081 2024-02-05T07:34:03.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.801 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.801 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.801 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.801 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:51.457329+0000", 2024-02-05T07:34:03.801 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.801 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.802 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.802 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:51.457329+0000", 2024-02-05T07:34:03.802 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.802 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.802 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:51.457330+0000", 2024-02-05T07:34:03.802 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7699999999999992e-07 2024-02-05T07:34:03.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.802 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.802 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:51.457334+0000", 2024-02-05T07:34:03.802 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9770000000000002e-06 2024-02-05T07:34:03.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.803 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.803 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:51.457400+0000", 2024-02-05T07:34:03.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6409999999999996e-05 2024-02-05T07:34:03.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.803 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.803 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:51.457451+0000", 2024-02-05T07:34:03.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0837000000000001e-05 2024-02-05T07:34:03.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.803 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.803 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:51.457533+0000", 2024-02-05T07:34:03.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1359000000000005e-05 2024-02-05T07:34:03.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.804 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.804 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:51.457637+0000", 2024-02-05T07:34:03.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010477899999999999 2024-02-05T07:34:03.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.804 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:03.804 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:51.457723+0000", 2024-02-05T07:34:03.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5677e-05 2024-02-05T07:34:03.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.805 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:51.548770+0000", 2024-02-05T07:34:03.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.091047100000000006 2024-02-05T07:34:03.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.805 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:51.550123+0000", 2024-02-05T07:34:03.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001352996 2024-02-05T07:34:03.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.805 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:51.554083+0000", 2024-02-05T07:34:03.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039599409999999998 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:51.554216+0000", 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013313499999999999 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:51.585842+0000", 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.031626093000000001 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:144110 9.7 9:e0ddc649:::benchmark_data_smithi007_136200_object64265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:57.416798+0000", 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 65.154025090000005, 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.106829065, 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 144110 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:57.416798+0000", 2024-02-05T07:34:03.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.808 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:34:03.808 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:57.416798+0000", 2024-02-05T07:34:03.808 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:34:03.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.808 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:34:03.808 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:57.416798+0000", 2024-02-05T07:34:03.808 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.75e-07 2024-02-05T07:34:03.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.808 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:34:03.808 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:57.416801+0000", 2024-02-05T07:34:03.809 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4940000000000002e-06 2024-02-05T07:34:03.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.809 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:34:03.809 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:57.416807+0000", 2024-02-05T07:34:03.809 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1020000000000002e-06 2024-02-05T07:34:03.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.809 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:34:03.809 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:57.416812+0000", 2024-02-05T07:34:03.809 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7270000000000004e-06 2024-02-05T07:34:03.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.810 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:34:03.810 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:57.431531+0000", 2024-02-05T07:34:03.810 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014718385 2024-02-05T07:34:03.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.810 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:34:03.810 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:57.431641+0000", 2024-02-05T07:34:03.810 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000110436 2024-02-05T07:34:03.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.810 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:34:03.810 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:57.431750+0000", 2024-02-05T07:34:03.811 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108656 2024-02-05T07:34:03.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.811 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.811 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:57.519637+0000", 2024-02-05T07:34:03.811 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.087887007000000003 2024-02-05T07:34:03.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.811 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:03.811 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:57.521800+0000", 2024-02-05T07:34:03.811 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002163211 2024-02-05T07:34:03.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.811 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:34:03.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:57.523534+0000", 2024-02-05T07:34:03.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017336160000000001 2024-02-05T07:34:03.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:34:03.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:57.523585+0000", 2024-02-05T07:34:03.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1768999999999999e-05 2024-02-05T07:34:03.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:34:03.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:34:03.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:34:03.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:57.523627+0000", 2024-02-05T07:34:03.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1186999999999997e-05 2024-02-05T07:34:03.812 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.812 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.813 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.813 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:34:03.813 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:34:03.813 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:34:03.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:03.814 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 16 16 35405 35389 138.213 72.5625 0.00982624 0.00722938 2024-02-05T07:34:03.856 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:03.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:03.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:03.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:03 smithi007 ceph-mon[29098]: osdmap e429: 8 total, 8 up, 6 in 2024-02-05T07:34:03.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:03 smithi007 ceph-mon[37158]: osdmap e429: 8 total, 8 up, 6 in 2024-02-05T07:34:03.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:34:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:34:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:34:03.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:04.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:03 smithi090 ceph-mon[33213]: osdmap e429: 8 total, 8 up, 6 in 2024-02-05T07:34:04.092 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:04.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:04.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:04.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:04.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:04.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:04.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:04.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:04.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:04.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:04.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:04.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:04.655 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:04.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:04.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:04.757 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:04.764 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 17 16 37204 37188 136.697 112.438 0.00788115 0.00730746 2024-02-05T07:34:04.787 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:04.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:04.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:04.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:04 smithi007 ceph-mon[29098]: pgmap v784: 178 pgs: 178 active+clean; 1.8 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-02-05T07:34:04.805 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:04 smithi007 ceph-mon[37158]: pgmap v784: 178 pgs: 178 active+clean; 1.8 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-02-05T07:34:04.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:04.954 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:04.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:05.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:04 smithi090 ceph-mon[33213]: pgmap v784: 178 pgs: 178 active+clean; 1.8 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-02-05T07:34:05.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:05.097 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:05.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:05.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:05.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:05.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:05.331 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:05.360 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:05.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:05.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:05.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:05.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T07:34:05.493 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:05.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:05.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:05.626 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:05.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:05.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:05.711 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:05.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:05.761 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:05.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:05.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:05.764 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 18 16 38927 38911 135.085 107.688 0.0132109 0.00739634 2024-02-05T07:34:05.787 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:05.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:05.912 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:05.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:05.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:05.933 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:34:06.014 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:06.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:06.056 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:06.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:06.157 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:06.167 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:34:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:34:06.189 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:06.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:06.291 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:06.322 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:06.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:06.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:06.458 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:06.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:06.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:06.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:06.591 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:06.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:06.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:06.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:06.732 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:06.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:06.764 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 19 16 40547 40531 133.303 101.25 0.0111676 0.00749653 2024-02-05T07:34:06.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:06.878 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:06.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:06.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:06 smithi007 ceph-mon[29098]: pgmap v785: 178 pgs: 178 active+clean; 2.0 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 100 MiB/s wr, 1.61k op/s 2024-02-05T07:34:06.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:06 smithi007 ceph-mon[37158]: pgmap v785: 178 pgs: 178 active+clean; 2.0 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 100 MiB/s wr, 1.61k op/s 2024-02-05T07:34:06.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:07.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:06 smithi090 ceph-mon[33213]: pgmap v785: 178 pgs: 178 active+clean; 2.0 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 100 MiB/s wr, 1.61k op/s 2024-02-05T07:34:07.030 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":785,"stamp":"2024-02-05T07:34:05.267954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2151356764,"num_objects":33016,"num_object_clones":0,"num_object_copies":99048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33016,"num_whiteouts":0,"num_read":1321,"num_read_kb":1073,"num_write":33540,"num_write_kb":2100397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":304,"num_bytes_recovered":7572,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37630,"ondisk_log_size":37630,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":9737260,"kb_used_data":6076792,"kb_used_omap":13,"kb_used_meta":3660402,"kb_avail":552733652,"statfs":{"total":575970213888,"available":565999259648,"internally_reserved":0,"allocated":6222635008,"data_stored":6220173046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":3748252384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,1,0,0,7,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":843317248,"num_objects":12868,"num_object_clones":0,"num_object_copies":38604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12868,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12868,"num_write_kb":823552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002769"},"pg_stats":[{"pgid":"10.2","version":"428'2076","reported_seq":2095,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:34:01.978329+0000","last_change":"2024-02-05T07:33:38.775613+0000","last_active":"2024-02-05T07:34:01.978329+0000","last_peered":"2024-02-05T07:34:01.978329+0000","last_clean":"2024-02-05T07:34:01.978329+0000","last_became_active":"2024-02-05T07:33:38.775201+0000","last_became_peered":"2024-02-05T07:33:38.775201+0000","last_unstale":"2024-02-05T07:34:01.978329+0000","last_undegraded":"2024-02-05T07:34:01.978329+0000","last_fullsized":"2024-02-05T07:34:01.978329+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":2076,"log_dups_size":0,"ondisk_log_size":2076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:45.287394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136052736,"num_objects":2076,"num_object_clones":0,"num_object_copies":6228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2076,"num_write_kb":132864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":865,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.563688+0000","last_change":"2024-02-05T07:32:42.818086+0000","last_active":"2024-02-05T07:33:57.563688+0000","last_peered":"2024-02-05T07:33:57.563688+0000","last_clean":"2024-02-05T07:33:57.563688+0000","last_became_active":"2024-02-05T07:31:36.857092+0000","last_became_peered":"2024-02-05T07:31:36.857092+0000","last_unstale":"2024-02-05T07:33:57.563688+0000","last_undegraded":"2024-02-05T07:33:57.563688+0000","last_fullsized":"2024-02-05T07:33:57.563688+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:42.817975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:42.817975+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:59.541198+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"326'25","reported_seq":842,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.562169+0000","last_change":"2024-02-05T07:32:27.643863+0000","last_active":"2024-02-05T07:33:57.562169+0000","last_peered":"2024-02-05T07:33:57.562169+0000","last_clean":"2024-02-05T07:33:57.562169+0000","last_became_active":"2024-02-05T07:28:26.605082+0000","last_became_peered":"2024-02-05T07:28:26.605082+0000","last_unstale":"2024-02-05T07:33:57.562169+0000","last_undegraded":"2024-02-05T07:33:57.562169+0000","last_fullsized":"2024-02-05T07:33:57.562169+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:32:27.643788+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:32:27.643788+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.931031+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":349,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.564724+0000","last_change":"2024-02-05T07:33:34.736291+0000","last_active":"2024-02-05T07:33:57.564724+0000","last_peered":"2024-02-05T07:33:57.564724+0000","last_clean":"2024-02-05T07:33:57.564724+0000","last_became_active":"2024-02-05T07:27:16.716461+0000","last_became_peered":"2024-02-05T07:27:16.716461+0000","last_unstale":"2024-02-05T07:33:57.564724+0000","last_undegraded":"2024-02-05T07:33:57.564724+0000","last_fullsized":"2024-02-05T07:33:57.564724+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:34.736160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:34.736160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:00.625101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"365'11","reported_seq":983,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.570246+0000","last_change":"2024-02-05T07:30:40.506050+0000","last_active":"2024-02-05T07:33:57.570246+0000","last_peered":"2024-02-05T07:33:57.570246+0000","last_clean":"2024-02-05T07:33:57.570246+0000","last_became_active":"2024-02-05T07:30:40.484389+0000","last_became_peered":"2024-02-05T07:30:40.484389+0000","last_unstale":"2024-02-05T07:33:57.570246+0000","last_undegraded":"2024-02-05T07:33:57.570246+0000","last_fullsized":"2024-02-05T07:33:57.570246+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'10","last_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:30:38.946065+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:01.924578+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":755,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559302+0000","last_change":"2024-02-05T07:32:30.708289+0000","last_active":"2024-02-05T07:33:57.559302+0000","last_peered":"2024-02-05T07:33:57.559302+0000","last_clean":"2024-02-05T07:33:57.559302+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:33:57.559302+0000","last_undegraded":"2024-02-05T07:33:57.559302+0000","last_fullsized":"2024-02-05T07:33:57.559302+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:30.708205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:32:30.708205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.931125+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"428'1900","reported_seq":1919,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:34:00.931957+0000","last_change":"2024-02-05T07:33:38.616173+0000","last_active":"2024-02-05T07:34:00.931957+0000","last_peered":"2024-02-05T07:34:00.931957+0000","last_clean":"2024-02-05T07:34:00.931957+0000","last_became_active":"2024-02-05T07:33:38.615825+0000","last_became_peered":"2024-02-05T07:33:38.615825+0000","last_unstale":"2024-02-05T07:34:00.931957+0000","last_undegraded":"2024-02-05T07:34:00.931957+0000","last_fullsized":"2024-02-05T07:34:00.931957+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":1900,"log_dups_size":0,"ondisk_log_size":1900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:53.838130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124518400,"num_objects":1900,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1900,"num_write_kb":121600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":784,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.570664+0000","last_change":"2024-02-05T07:32:41.228497+0000","last_active":"2024-02-05T07:33:57.570664+0000","last_peered":"2024-02-05T07:33:57.570664+0000","last_clean":"2024-02-05T07:33:57.570664+0000","last_became_active":"2024-02-05T07:31:36.856842+0000","last_became_peered":"2024-02-05T07:31:36.856842+0000","last_unstale":"2024-02-05T07:33:57.570664+0000","last_undegraded":"2024-02-05T07:33:57.570664+0000","last_fullsized":"2024-02-05T07:33:57.570664+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:41.228420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:41.228420+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:01.924880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"326'31","reported_seq":821,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558544+0000","last_change":"2024-02-05T07:33:30.750667+0000","last_active":"2024-02-05T07:33:57.558544+0000","last_peered":"2024-02-05T07:33:57.558544+0000","last_clean":"2024-02-05T07:33:57.558544+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:33:57.558544+0000","last_undegraded":"2024-02-05T07:33:57.558544+0000","last_fullsized":"2024-02-05T07:33:57.558544+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'31","last_scrub_stamp":"2024-02-05T07:33:30.750548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.750548+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:37.292765+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":623,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.680765+0000","last_change":"2024-02-05T07:33:44.229582+0000","last_active":"2024-02-05T07:34:02.680765+0000","last_peered":"2024-02-05T07:34:02.680765+0000","last_clean":"2024-02-05T07:34:02.680765+0000","last_became_active":"2024-02-05T07:28:25.791005+0000","last_became_peered":"2024-02-05T07:28:25.791005+0000","last_unstale":"2024-02-05T07:34:02.680765+0000","last_undegraded":"2024-02-05T07:34:02.680765+0000","last_fullsized":"2024-02-05T07:34:02.680765+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.229489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.229489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.986179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":761,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.563112+0000","last_change":"2024-02-05T07:29:49.674581+0000","last_active":"2024-02-05T07:33:57.563112+0000","last_peered":"2024-02-05T07:33:57.563112+0000","last_clean":"2024-02-05T07:33:57.563112+0000","last_became_active":"2024-02-05T07:18:53.457402+0000","last_became_peered":"2024-02-05T07:18:53.457402+0000","last_unstale":"2024-02-05T07:33:57.563112+0000","last_undegraded":"2024-02-05T07:33:57.563112+0000","last_fullsized":"2024-02-05T07:33:57.563112+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.674493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.674493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":771,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.569479+0000","last_change":"2024-02-05T07:30:35.156491+0000","last_active":"2024-02-05T07:33:57.569479+0000","last_peered":"2024-02-05T07:33:57.569479+0000","last_clean":"2024-02-05T07:33:57.569479+0000","last_became_active":"2024-02-05T07:27:16.708888+0000","last_became_peered":"2024-02-05T07:27:16.708888+0000","last_unstale":"2024-02-05T07:33:57.569479+0000","last_undegraded":"2024-02-05T07:33:57.569479+0000","last_fullsized":"2024-02-05T07:33:57.569479+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:35.156384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:30:35.156384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:01.924505+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"429'2224","reported_seq":2246,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:03.671410+0000","last_change":"2024-02-05T07:33:38.613283+0000","last_active":"2024-02-05T07:34:03.671410+0000","last_peered":"2024-02-05T07:34:03.671410+0000","last_clean":"2024-02-05T07:34:03.671410+0000","last_became_active":"2024-02-05T07:33:38.613051+0000","last_became_peered":"2024-02-05T07:33:38.613051+0000","last_unstale":"2024-02-05T07:34:03.671410+0000","last_undegraded":"2024-02-05T07:34:03.671410+0000","last_fullsized":"2024-02-05T07:34:03.671410+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":2224,"log_dups_size":0,"ondisk_log_size":2224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:05.930275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145752064,"num_objects":2224,"num_object_clones":0,"num_object_copies":6672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2224,"num_write_kb":142336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":755,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.565458+0000","last_change":"2024-02-05T07:32:41.303464+0000","last_active":"2024-02-05T07:33:57.565458+0000","last_peered":"2024-02-05T07:33:57.565458+0000","last_clean":"2024-02-05T07:33:57.565458+0000","last_became_active":"2024-02-05T07:31:36.841690+0000","last_became_peered":"2024-02-05T07:31:36.841690+0000","last_unstale":"2024-02-05T07:33:57.565458+0000","last_undegraded":"2024-02-05T07:33:57.565458+0000","last_fullsized":"2024-02-05T07:33:57.565458+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:41.303385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:41.303385+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"326'24","reported_seq":808,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559385+0000","last_change":"2024-02-05T07:29:50.053697+0000","last_active":"2024-02-05T07:33:57.559385+0000","last_peered":"2024-02-05T07:33:57.559385+0000","last_clean":"2024-02-05T07:33:57.559385+0000","last_became_active":"2024-02-05T07:28:25.781007+0000","last_became_peered":"2024-02-05T07:28:25.781007+0000","last_unstale":"2024-02-05T07:33:57.559385+0000","last_undegraded":"2024-02-05T07:33:57.559385+0000","last_fullsized":"2024-02-05T07:33:57.559385+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:29:50.053583+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.053583+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930530+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":774,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558638+0000","last_change":"2024-02-05T07:32:28.670360+0000","last_active":"2024-02-05T07:33:57.558638+0000","last_peered":"2024-02-05T07:33:57.558638+0000","last_clean":"2024-02-05T07:33:57.558638+0000","last_became_active":"2024-02-05T07:30:40.486134+0000","last_became_peered":"2024-02-05T07:30:40.486134+0000","last_unstale":"2024-02-05T07:33:57.558638+0000","last_undegraded":"2024-02-05T07:33:57.558638+0000","last_fullsized":"2024-02-05T07:33:57.558638+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:28.670286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:32:28.670286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930828+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":595,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558476+0000","last_change":"2024-02-05T07:29:45.012958+0000","last_active":"2024-02-05T07:33:57.558476+0000","last_peered":"2024-02-05T07:33:57.558476+0000","last_clean":"2024-02-05T07:33:57.558476+0000","last_became_active":"2024-02-05T07:28:26.601713+0000","last_became_peered":"2024-02-05T07:28:26.601713+0000","last_unstale":"2024-02-05T07:33:57.558476+0000","last_undegraded":"2024-02-05T07:33:57.558476+0000","last_fullsized":"2024-02-05T07:33:57.558476+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.012821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930554+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":358,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.566651+0000","last_change":"2024-02-05T07:33:28.696549+0000","last_active":"2024-02-05T07:33:57.566651+0000","last_peered":"2024-02-05T07:33:57.566651+0000","last_clean":"2024-02-05T07:33:57.566651+0000","last_became_active":"2024-02-05T07:30:40.498529+0000","last_became_peered":"2024-02-05T07:30:40.498529+0000","last_unstale":"2024-02-05T07:33:57.566651+0000","last_undegraded":"2024-02-05T07:33:57.566651+0000","last_fullsized":"2024-02-05T07:33:57.566651+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.696462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.696462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:28.717583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"428'1926","reported_seq":1946,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:34:00.350981+0000","last_change":"2024-02-05T07:33:38.778383+0000","last_active":"2024-02-05T07:34:00.350981+0000","last_peered":"2024-02-05T07:34:00.350981+0000","last_clean":"2024-02-05T07:34:00.350981+0000","last_became_active":"2024-02-05T07:33:38.778162+0000","last_became_peered":"2024-02-05T07:33:38.778162+0000","last_unstale":"2024-02-05T07:34:00.350981+0000","last_undegraded":"2024-02-05T07:34:00.350981+0000","last_fullsized":"2024-02-05T07:34:00.350981+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":1926,"log_dups_size":0,"ondisk_log_size":1926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.483201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126222336,"num_objects":1926,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1926,"num_write_kb":123264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":759,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567078+0000","last_change":"2024-02-05T07:32:40.217065+0000","last_active":"2024-02-05T07:33:57.567078+0000","last_peered":"2024-02-05T07:33:57.567078+0000","last_clean":"2024-02-05T07:33:57.567078+0000","last_became_active":"2024-02-05T07:31:36.839888+0000","last_became_peered":"2024-02-05T07:31:36.839888+0000","last_unstale":"2024-02-05T07:33:57.567078+0000","last_undegraded":"2024-02-05T07:33:57.567078+0000","last_fullsized":"2024-02-05T07:33:57.567078+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:40.216973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:40.216973+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:01.925099+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"326'34","reported_seq":859,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.562916+0000","last_change":"2024-02-05T07:30:30.006087+0000","last_active":"2024-02-05T07:33:57.562916+0000","last_peered":"2024-02-05T07:33:57.562916+0000","last_clean":"2024-02-05T07:33:57.562916+0000","last_became_active":"2024-02-05T07:27:16.768454+0000","last_became_peered":"2024-02-05T07:27:16.768454+0000","last_unstale":"2024-02-05T07:33:57.562916+0000","last_undegraded":"2024-02-05T07:33:57.562916+0000","last_fullsized":"2024-02-05T07:33:57.562916+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'34","last_scrub_stamp":"2024-02-05T07:30:30.005988+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.005988+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557459+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":356,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.570196+0000","last_change":"2024-02-05T07:30:40.508975+0000","last_active":"2024-02-05T07:33:57.570196+0000","last_peered":"2024-02-05T07:33:57.570196+0000","last_clean":"2024-02-05T07:33:57.570196+0000","last_became_active":"2024-02-05T07:30:40.508561+0000","last_became_peered":"2024-02-05T07:30:40.508561+0000","last_unstale":"2024-02-05T07:33:57.570196+0000","last_undegraded":"2024-02-05T07:33:57.570196+0000","last_fullsized":"2024-02-05T07:33:57.570196+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:30.982457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.982457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642322+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"257'9","reported_seq":960,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559249+0000","last_change":"2024-02-05T07:32:39.565923+0000","last_active":"2024-02-05T07:33:57.559249+0000","last_peered":"2024-02-05T07:33:57.559249+0000","last_clean":"2024-02-05T07:33:57.559249+0000","last_became_active":"2024-02-05T07:27:03.375195+0000","last_became_peered":"2024-02-05T07:27:03.375195+0000","last_unstale":"2024-02-05T07:33:57.559249+0000","last_undegraded":"2024-02-05T07:33:57.559249+0000","last_fullsized":"2024-02-05T07:33:57.559249+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'9","last_scrub_stamp":"2024-02-05T07:32:39.565833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:39.565833+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:09.327994+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":368,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559772+0000","last_change":"2024-02-05T07:30:37.998622+0000","last_active":"2024-02-05T07:33:57.559772+0000","last_peered":"2024-02-05T07:33:57.559772+0000","last_clean":"2024-02-05T07:33:57.559772+0000","last_became_active":"2024-02-05T07:27:03.367753+0000","last_became_peered":"2024-02-05T07:27:03.367753+0000","last_unstale":"2024-02-05T07:33:57.559772+0000","last_undegraded":"2024-02-05T07:33:57.559772+0000","last_fullsized":"2024-02-05T07:33:57.559772+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:37.998383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:30:37.998383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557573+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"428'1992","reported_seq":2012,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:34:00.340385+0000","last_change":"2024-02-05T07:33:38.613937+0000","last_active":"2024-02-05T07:34:00.340385+0000","last_peered":"2024-02-05T07:34:00.340385+0000","last_clean":"2024-02-05T07:34:00.340385+0000","last_became_active":"2024-02-05T07:33:38.613669+0000","last_became_peered":"2024-02-05T07:33:38.613669+0000","last_unstale":"2024-02-05T07:34:00.340385+0000","last_undegraded":"2024-02-05T07:34:00.340385+0000","last_fullsized":"2024-02-05T07:34:00.340385+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":1992,"log_dups_size":0,"ondisk_log_size":1992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:46.055957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130547712,"num_objects":1992,"num_object_clones":0,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1992,"num_write_kb":127488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":753,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.557926+0000","last_change":"2024-02-05T07:32:24.656130+0000","last_active":"2024-02-05T07:33:57.557926+0000","last_peered":"2024-02-05T07:33:57.557926+0000","last_clean":"2024-02-05T07:33:57.557926+0000","last_became_active":"2024-02-05T07:31:36.758820+0000","last_became_peered":"2024-02-05T07:31:36.758820+0000","last_unstale":"2024-02-05T07:33:57.557926+0000","last_undegraded":"2024-02-05T07:33:57.557926+0000","last_fullsized":"2024-02-05T07:33:57.557926+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:24.656057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:24.656057+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":779,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558077+0000","last_change":"2024-02-05T07:32:36.623960+0000","last_active":"2024-02-05T07:33:57.558077+0000","last_peered":"2024-02-05T07:33:57.558077+0000","last_clean":"2024-02-05T07:33:57.558077+0000","last_became_active":"2024-02-05T07:30:40.489212+0000","last_became_peered":"2024-02-05T07:30:40.489212+0000","last_unstale":"2024-02-05T07:33:57.558077+0000","last_undegraded":"2024-02-05T07:33:57.558077+0000","last_fullsized":"2024-02-05T07:33:57.558077+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:36.623865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:36.623865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.446069+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"326'30","reported_seq":410,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.570097+0000","last_change":"2024-02-05T07:30:40.543094+0000","last_active":"2024-02-05T07:33:57.570097+0000","last_peered":"2024-02-05T07:33:57.570097+0000","last_clean":"2024-02-05T07:33:57.570097+0000","last_became_active":"2024-02-05T07:30:40.496940+0000","last_became_peered":"2024-02-05T07:30:40.496940+0000","last_unstale":"2024-02-05T07:33:57.570097+0000","last_undegraded":"2024-02-05T07:33:57.570097+0000","last_fullsized":"2024-02-05T07:33:57.570097+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:28:45.029295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.029295+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.641961+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":356,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.570875+0000","last_change":"2024-02-05T07:33:48.701618+0000","last_active":"2024-02-05T07:33:57.570875+0000","last_peered":"2024-02-05T07:33:57.570875+0000","last_clean":"2024-02-05T07:33:57.570875+0000","last_became_active":"2024-02-05T07:30:40.508607+0000","last_became_peered":"2024-02-05T07:30:40.508607+0000","last_unstale":"2024-02-05T07:33:57.570875+0000","last_undegraded":"2024-02-05T07:33:57.570875+0000","last_fullsized":"2024-02-05T07:33:57.570875+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:48.701399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:48.701399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:03.427068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":772,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.557668+0000","last_change":"2024-02-05T07:33:45.908153+0000","last_active":"2024-02-05T07:33:57.557668+0000","last_peered":"2024-02-05T07:33:57.557668+0000","last_clean":"2024-02-05T07:33:57.557668+0000","last_became_active":"2024-02-05T07:27:16.706666+0000","last_became_peered":"2024-02-05T07:27:16.706666+0000","last_unstale":"2024-02-05T07:33:57.557668+0000","last_undegraded":"2024-02-05T07:33:57.557668+0000","last_fullsized":"2024-02-05T07:33:57.557668+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:45.908064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:33:45.908064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:09.071480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"429'2241","reported_seq":2263,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:03.600128+0000","last_change":"2024-02-05T07:33:38.614195+0000","last_active":"2024-02-05T07:34:03.600128+0000","last_peered":"2024-02-05T07:34:03.600128+0000","last_clean":"2024-02-05T07:34:03.600128+0000","last_became_active":"2024-02-05T07:33:38.613892+0000","last_became_peered":"2024-02-05T07:33:38.613892+0000","last_unstale":"2024-02-05T07:34:03.600128+0000","last_undegraded":"2024-02-05T07:34:03.600128+0000","last_fullsized":"2024-02-05T07:34:03.600128+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":2241,"log_dups_size":0,"ondisk_log_size":2241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.064473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146866176,"num_objects":2241,"num_object_clones":0,"num_object_copies":6723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2241,"num_write_kb":143424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":1011,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.566809+0000","last_change":"2024-02-05T07:30:43.524967+0000","last_active":"2024-02-05T07:33:57.566809+0000","last_peered":"2024-02-05T07:33:57.566809+0000","last_clean":"2024-02-05T07:33:57.566809+0000","last_became_active":"2024-02-05T07:30:43.524588+0000","last_became_peered":"2024-02-05T07:30:43.524588+0000","last_unstale":"2024-02-05T07:33:57.566809+0000","last_undegraded":"2024-02-05T07:33:57.566809+0000","last_fullsized":"2024-02-05T07:33:57.566809+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:39.042281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:39.042281+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.641796+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"266'10","reported_seq":435,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.570323+0000","last_change":"2024-02-05T07:33:42.715242+0000","last_active":"2024-02-05T07:33:57.570323+0000","last_peered":"2024-02-05T07:33:57.570323+0000","last_clean":"2024-02-05T07:33:57.570323+0000","last_became_active":"2024-02-05T07:27:16.800207+0000","last_became_peered":"2024-02-05T07:27:16.800207+0000","last_unstale":"2024-02-05T07:33:57.570323+0000","last_undegraded":"2024-02-05T07:33:57.570323+0000","last_fullsized":"2024-02-05T07:33:57.570323+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'10","last_scrub_stamp":"2024-02-05T07:33:42.715162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:42.715162+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:56.032172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"326'35","reported_seq":835,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558723+0000","last_change":"2024-02-05T07:29:45.972745+0000","last_active":"2024-02-05T07:33:57.558723+0000","last_peered":"2024-02-05T07:33:57.558723+0000","last_clean":"2024-02-05T07:33:57.558723+0000","last_became_active":"2024-02-05T07:28:26.598831+0000","last_became_peered":"2024-02-05T07:28:26.598831+0000","last_unstale":"2024-02-05T07:33:57.558723+0000","last_undegraded":"2024-02-05T07:33:57.558723+0000","last_fullsized":"2024-02-05T07:33:57.558723+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:29:45.972589+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.972589+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":522,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.563026+0000","last_change":"2024-02-05T07:29:41.587097+0000","last_active":"2024-02-05T07:33:57.563026+0000","last_peered":"2024-02-05T07:33:57.563026+0000","last_clean":"2024-02-05T07:33:57.563026+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:33:57.563026+0000","last_undegraded":"2024-02-05T07:33:57.563026+0000","last_fullsized":"2024-02-05T07:33:57.563026+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:41.587022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:41.587022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":635,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.681710+0000","last_change":"2024-02-05T07:32:33.839588+0000","last_active":"2024-02-05T07:34:02.681710+0000","last_peered":"2024-02-05T07:34:02.681710+0000","last_clean":"2024-02-05T07:34:02.681710+0000","last_became_active":"2024-02-05T07:30:40.489120+0000","last_became_peered":"2024-02-05T07:30:40.489120+0000","last_unstale":"2024-02-05T07:34:02.681710+0000","last_undegraded":"2024-02-05T07:34:02.681710+0000","last_fullsized":"2024-02-05T07:34:02.681710+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:33.839513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:32:33.839513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:07.049596+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"428'1937","reported_seq":1957,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:34:00.347917+0000","last_change":"2024-02-05T07:33:38.775156+0000","last_active":"2024-02-05T07:34:00.347917+0000","last_peered":"2024-02-05T07:34:00.347917+0000","last_clean":"2024-02-05T07:34:00.347917+0000","last_became_active":"2024-02-05T07:33:38.774713+0000","last_became_peered":"2024-02-05T07:33:38.774713+0000","last_unstale":"2024-02-05T07:34:00.347917+0000","last_undegraded":"2024-02-05T07:34:00.347917+0000","last_fullsized":"2024-02-05T07:34:00.347917+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":1937,"log_dups_size":0,"ondisk_log_size":1937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:05.742929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126943232,"num_objects":1937,"num_object_clones":0,"num_object_copies":5811,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1937,"num_write_kb":123968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":620,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.681518+0000","last_change":"2024-02-05T07:32:21.853114+0000","last_active":"2024-02-05T07:34:02.681518+0000","last_peered":"2024-02-05T07:34:02.681518+0000","last_clean":"2024-02-05T07:34:02.681518+0000","last_became_active":"2024-02-05T07:31:36.850075+0000","last_became_peered":"2024-02-05T07:31:36.850075+0000","last_unstale":"2024-02-05T07:34:02.681518+0000","last_undegraded":"2024-02-05T07:34:02.681518+0000","last_fullsized":"2024-02-05T07:34:02.681518+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:21.853039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:21.853039+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:07.049615+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":429,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.681768+0000","last_change":"2024-02-05T07:32:49.752237+0000","last_active":"2024-02-05T07:34:02.681768+0000","last_peered":"2024-02-05T07:34:02.681768+0000","last_clean":"2024-02-05T07:34:02.681768+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:34:02.681768+0000","last_undegraded":"2024-02-05T07:34:02.681768+0000","last_fullsized":"2024-02-05T07:34:02.681768+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:32:49.752163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:49.752163+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"326'25","reported_seq":827,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.561647+0000","last_change":"2024-02-05T07:30:36.908113+0000","last_active":"2024-02-05T07:33:57.561647+0000","last_peered":"2024-02-05T07:33:57.561647+0000","last_clean":"2024-02-05T07:33:57.561647+0000","last_became_active":"2024-02-05T07:27:03.361342+0000","last_became_peered":"2024-02-05T07:27:03.361342+0000","last_unstale":"2024-02-05T07:33:57.561647+0000","last_undegraded":"2024-02-05T07:33:57.561647+0000","last_fullsized":"2024-02-05T07:33:57.561647+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:30:36.907989+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.907989+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557607+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":353,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.563762+0000","last_change":"2024-02-05T07:29:44.015577+0000","last_active":"2024-02-05T07:33:57.563762+0000","last_peered":"2024-02-05T07:33:57.563762+0000","last_clean":"2024-02-05T07:33:57.563762+0000","last_became_active":"2024-02-05T07:28:26.596096+0000","last_became_peered":"2024-02-05T07:28:26.596096+0000","last_unstale":"2024-02-05T07:33:57.563762+0000","last_undegraded":"2024-02-05T07:33:57.563762+0000","last_fullsized":"2024-02-05T07:33:57.563762+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:44.015468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:29:44.015468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":49,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559006+0000","last_change":"2024-02-05T07:33:24.419375+0000","last_active":"2024-02-05T07:33:57.559006+0000","last_peered":"2024-02-05T07:33:57.559006+0000","last_clean":"2024-02-05T07:33:57.559006+0000","last_became_active":"2024-02-05T07:32:58.598709+0000","last_became_peered":"2024-02-05T07:32:58.598709+0000","last_unstale":"2024-02-05T07:33:57.559006+0000","last_undegraded":"2024-02-05T07:33:57.559006+0000","last_fullsized":"2024-02-05T07:33:57.559006+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:24.419250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:24.419250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:34.834326+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"429'2174","reported_seq":2196,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:03.663597+0000","last_change":"2024-02-05T07:33:38.775383+0000","last_active":"2024-02-05T07:34:03.663597+0000","last_peered":"2024-02-05T07:34:03.663597+0000","last_clean":"2024-02-05T07:34:03.663597+0000","last_became_active":"2024-02-05T07:33:38.775048+0000","last_became_peered":"2024-02-05T07:33:38.775048+0000","last_unstale":"2024-02-05T07:34:03.663597+0000","last_undegraded":"2024-02-05T07:34:03.663597+0000","last_fullsized":"2024-02-05T07:34:03.663597+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:00.065375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142475264,"num_objects":2174,"num_object_clones":0,"num_object_copies":6522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2174,"num_write_kb":139136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":619,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.680918+0000","last_change":"2024-02-05T07:33:47.197319+0000","last_active":"2024-02-05T07:34:02.680918+0000","last_peered":"2024-02-05T07:34:02.680918+0000","last_clean":"2024-02-05T07:34:02.680918+0000","last_became_active":"2024-02-05T07:31:36.840534+0000","last_became_peered":"2024-02-05T07:31:36.840534+0000","last_unstale":"2024-02-05T07:34:02.680918+0000","last_undegraded":"2024-02-05T07:34:02.680918+0000","last_fullsized":"2024-02-05T07:34:02.680918+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:47.197225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.197225+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:05.910268+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":610,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.680370+0000","last_change":"2024-02-05T07:33:33.212392+0000","last_active":"2024-02-05T07:34:02.680370+0000","last_peered":"2024-02-05T07:34:02.680370+0000","last_clean":"2024-02-05T07:34:02.680370+0000","last_became_active":"2024-02-05T07:28:26.589673+0000","last_became_peered":"2024-02-05T07:28:26.589673+0000","last_unstale":"2024-02-05T07:34:02.680370+0000","last_undegraded":"2024-02-05T07:34:02.680370+0000","last_fullsized":"2024-02-05T07:34:02.680370+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:33.212264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.212264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:58.757962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"326'29","reported_seq":395,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.565318+0000","last_change":"2024-02-05T07:30:40.516301+0000","last_active":"2024-02-05T07:33:57.565318+0000","last_peered":"2024-02-05T07:33:57.565318+0000","last_clean":"2024-02-05T07:33:57.565318+0000","last_became_active":"2024-02-05T07:30:40.481234+0000","last_became_peered":"2024-02-05T07:30:40.481234+0000","last_unstale":"2024-02-05T07:33:57.565318+0000","last_undegraded":"2024-02-05T07:33:57.565318+0000","last_fullsized":"2024-02-05T07:33:57.565318+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'29","last_scrub_stamp":"2024-02-05T07:30:37.106672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:37.106672+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":776,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.563029+0000","last_change":"2024-02-05T07:29:53.699733+0000","last_active":"2024-02-05T07:33:57.563029+0000","last_peered":"2024-02-05T07:33:57.563029+0000","last_clean":"2024-02-05T07:33:57.563029+0000","last_became_active":"2024-02-05T07:27:03.361028+0000","last_became_peered":"2024-02-05T07:27:03.361028+0000","last_unstale":"2024-02-05T07:33:57.563029+0000","last_undegraded":"2024-02-05T07:33:57.563029+0000","last_fullsized":"2024-02-05T07:33:57.563029+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.699651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.699651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":767,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.561577+0000","last_change":"2024-02-05T07:33:33.257106+0000","last_active":"2024-02-05T07:33:57.561577+0000","last_peered":"2024-02-05T07:33:57.561577+0000","last_clean":"2024-02-05T07:33:57.561577+0000","last_became_active":"2024-02-05T07:28:25.840850+0000","last_became_peered":"2024-02-05T07:28:25.840850+0000","last_unstale":"2024-02-05T07:33:57.561577+0000","last_undegraded":"2024-02-05T07:33:57.561577+0000","last_fullsized":"2024-02-05T07:33:57.561577+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:33.257028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.257028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.513556+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"428'1960","reported_seq":1980,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:34:00.912652+0000","last_change":"2024-02-05T07:33:38.777030+0000","last_active":"2024-02-05T07:34:00.912652+0000","last_peered":"2024-02-05T07:34:00.912652+0000","last_clean":"2024-02-05T07:34:00.912652+0000","last_became_active":"2024-02-05T07:33:38.776442+0000","last_became_peered":"2024-02-05T07:33:38.776442+0000","last_unstale":"2024-02-05T07:34:00.912652+0000","last_undegraded":"2024-02-05T07:34:00.912652+0000","last_fullsized":"2024-02-05T07:34:00.912652+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":1960,"log_dups_size":0,"ondisk_log_size":1960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:46.941588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128450560,"num_objects":1960,"num_object_clones":0,"num_object_copies":5880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1960,"num_write_kb":125440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":748,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567620+0000","last_change":"2024-02-05T07:33:42.423790+0000","last_active":"2024-02-05T07:33:57.567620+0000","last_peered":"2024-02-05T07:33:57.567620+0000","last_clean":"2024-02-05T07:33:57.567620+0000","last_became_active":"2024-02-05T07:28:25.756613+0000","last_became_peered":"2024-02-05T07:28:25.756613+0000","last_unstale":"2024-02-05T07:33:57.567620+0000","last_undegraded":"2024-02-05T07:33:57.567620+0000","last_fullsized":"2024-02-05T07:33:57.567620+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:33:42.423707+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:06.512536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":367,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.563350+0000","last_change":"2024-02-05T07:32:46.399673+0000","last_active":"2024-02-05T07:33:57.563350+0000","last_peered":"2024-02-05T07:33:57.563350+0000","last_clean":"2024-02-05T07:33:57.563350+0000","last_became_active":"2024-02-05T07:27:03.370297+0000","last_became_peered":"2024-02-05T07:27:03.370297+0000","last_unstale":"2024-02-05T07:33:57.563350+0000","last_undegraded":"2024-02-05T07:33:57.563350+0000","last_fullsized":"2024-02-05T07:33:57.563350+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:46.399572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:46.399572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:02.315642+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"326'24","reported_seq":577,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.563550+0000","last_change":"2024-02-05T07:33:49.499998+0000","last_active":"2024-02-05T07:33:57.563550+0000","last_peered":"2024-02-05T07:33:57.563550+0000","last_clean":"2024-02-05T07:33:57.563550+0000","last_became_active":"2024-02-05T07:24:34.209716+0000","last_became_peered":"2024-02-05T07:24:34.209716+0000","last_unstale":"2024-02-05T07:33:57.563550+0000","last_undegraded":"2024-02-05T07:33:57.563550+0000","last_fullsized":"2024-02-05T07:33:57.563550+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:33:49.499846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:49.499846+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:53.074156+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":385,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:34:00.746788+0000","last_change":"2024-02-05T07:30:40.520395+0000","last_active":"2024-02-05T07:34:00.746788+0000","last_peered":"2024-02-05T07:34:00.746788+0000","last_clean":"2024-02-05T07:34:00.746788+0000","last_became_active":"2024-02-05T07:30:40.493770+0000","last_became_peered":"2024-02-05T07:30:40.493770+0000","last_unstale":"2024-02-05T07:34:00.746788+0000","last_undegraded":"2024-02-05T07:34:00.746788+0000","last_fullsized":"2024-02-05T07:34:00.746788+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:29:51.291146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.291146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:05.635199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":207,"num_read_kb":202,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":741,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558836+0000","last_change":"2024-02-05T07:33:39.809630+0000","last_active":"2024-02-05T07:33:57.558836+0000","last_peered":"2024-02-05T07:33:57.558836+0000","last_clean":"2024-02-05T07:33:57.558836+0000","last_became_active":"2024-02-05T07:27:16.724280+0000","last_became_peered":"2024-02-05T07:27:16.724280+0000","last_unstale":"2024-02-05T07:33:57.558836+0000","last_undegraded":"2024-02-05T07:33:57.558836+0000","last_fullsized":"2024-02-05T07:33:57.558836+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:39.809544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:33:39.809544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:03.386365+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":795,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.562710+0000","last_change":"2024-02-05T07:33:41.379002+0000","last_active":"2024-02-05T07:33:57.562710+0000","last_peered":"2024-02-05T07:33:57.562710+0000","last_clean":"2024-02-05T07:33:57.562710+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:33:57.562710+0000","last_undegraded":"2024-02-05T07:33:57.562710+0000","last_fullsized":"2024-02-05T07:33:57.562710+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:33:41.378909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.378909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:42.516508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"429'2209","reported_seq":2231,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:03.674180+0000","last_change":"2024-02-05T07:33:38.775761+0000","last_active":"2024-02-05T07:34:03.674180+0000","last_peered":"2024-02-05T07:34:03.674180+0000","last_clean":"2024-02-05T07:34:03.674180+0000","last_became_active":"2024-02-05T07:33:38.775338+0000","last_became_peered":"2024-02-05T07:33:38.775338+0000","last_unstale":"2024-02-05T07:34:03.674180+0000","last_undegraded":"2024-02-05T07:34:03.674180+0000","last_fullsized":"2024-02-05T07:34:03.674180+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":2209,"log_dups_size":0,"ondisk_log_size":2209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:50.485312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144769024,"num_objects":2209,"num_object_clones":0,"num_object_copies":6627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2209,"num_write_kb":141376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":541,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567722+0000","last_change":"2024-02-05T07:33:50.943755+0000","last_active":"2024-02-05T07:33:57.567722+0000","last_peered":"2024-02-05T07:33:57.567722+0000","last_clean":"2024-02-05T07:33:57.567722+0000","last_became_active":"2024-02-05T07:30:40.474169+0000","last_became_peered":"2024-02-05T07:30:40.474169+0000","last_unstale":"2024-02-05T07:33:57.567722+0000","last_undegraded":"2024-02-05T07:33:57.567722+0000","last_fullsized":"2024-02-05T07:33:57.567722+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:50.943640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:50.943640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:06.509391+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"365'31","reported_seq":841,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.681104+0000","last_change":"2024-02-05T07:33:46.200250+0000","last_active":"2024-02-05T07:34:02.681104+0000","last_peered":"2024-02-05T07:34:02.681104+0000","last_clean":"2024-02-05T07:34:02.681104+0000","last_became_active":"2024-02-05T07:30:40.508686+0000","last_became_peered":"2024-02-05T07:30:40.508686+0000","last_unstale":"2024-02-05T07:34:02.681104+0000","last_undegraded":"2024-02-05T07:34:02.681104+0000","last_fullsized":"2024-02-05T07:34:02.681104+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'31","last_scrub_stamp":"2024-02-05T07:33:46.200150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.200150+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.420503+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":353,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.568351+0000","last_change":"2024-02-05T07:29:43.000854+0000","last_active":"2024-02-05T07:33:57.568351+0000","last_peered":"2024-02-05T07:33:57.568351+0000","last_clean":"2024-02-05T07:33:57.568351+0000","last_became_active":"2024-02-05T07:28:26.595640+0000","last_became_peered":"2024-02-05T07:28:26.595640+0000","last_unstale":"2024-02-05T07:33:57.568351+0000","last_undegraded":"2024-02-05T07:33:57.568351+0000","last_fullsized":"2024-02-05T07:33:57.568351+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:43.000778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:43.000778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642098+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":464,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.679850+0000","last_change":"2024-02-05T07:33:48.221978+0000","last_active":"2024-02-05T07:34:02.679850+0000","last_peered":"2024-02-05T07:34:02.679850+0000","last_clean":"2024-02-05T07:34:02.679850+0000","last_became_active":"2024-02-05T07:30:43.516031+0000","last_became_peered":"2024-02-05T07:30:43.516031+0000","last_unstale":"2024-02-05T07:34:02.679850+0000","last_undegraded":"2024-02-05T07:34:02.679850+0000","last_fullsized":"2024-02-05T07:34:02.679850+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:33:48.221845+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:03.117832+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":350,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.570778+0000","last_change":"2024-02-05T07:33:46.760555+0000","last_active":"2024-02-05T07:33:57.570778+0000","last_peered":"2024-02-05T07:33:57.570778+0000","last_clean":"2024-02-05T07:33:57.570778+0000","last_became_active":"2024-02-05T07:27:16.751341+0000","last_became_peered":"2024-02-05T07:27:16.751341+0000","last_unstale":"2024-02-05T07:33:57.570778+0000","last_undegraded":"2024-02-05T07:33:57.570778+0000","last_fullsized":"2024-02-05T07:33:57.570778+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:46.760448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.760448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:01.541210+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"428'1916","reported_seq":1936,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:34:00.910169+0000","last_change":"2024-02-05T07:33:38.615853+0000","last_active":"2024-02-05T07:34:00.910169+0000","last_peered":"2024-02-05T07:34:00.910169+0000","last_clean":"2024-02-05T07:34:00.910169+0000","last_became_active":"2024-02-05T07:33:38.615574+0000","last_became_peered":"2024-02-05T07:33:38.615574+0000","last_unstale":"2024-02-05T07:34:00.910169+0000","last_undegraded":"2024-02-05T07:34:00.910169+0000","last_fullsized":"2024-02-05T07:34:00.910169+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":1916,"log_dups_size":0,"ondisk_log_size":1916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:58.336758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125566976,"num_objects":1916,"num_object_clones":0,"num_object_copies":5748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1916,"num_write_kb":122624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":764,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.557390+0000","last_change":"2024-02-05T07:32:21.667678+0000","last_active":"2024-02-05T07:33:57.557390+0000","last_peered":"2024-02-05T07:33:57.557390+0000","last_clean":"2024-02-05T07:33:57.557390+0000","last_became_active":"2024-02-05T07:30:43.509733+0000","last_became_peered":"2024-02-05T07:30:43.509733+0000","last_unstale":"2024-02-05T07:33:57.557390+0000","last_undegraded":"2024-02-05T07:33:57.557390+0000","last_fullsized":"2024-02-05T07:33:57.557390+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:21.667581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:21.667581+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.931097+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":780,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.570460+0000","last_change":"2024-02-05T07:33:23.118646+0000","last_active":"2024-02-05T07:33:57.570460+0000","last_peered":"2024-02-05T07:33:57.570460+0000","last_clean":"2024-02-05T07:33:57.570460+0000","last_became_active":"2024-02-05T07:30:40.503816+0000","last_became_peered":"2024-02-05T07:30:40.503816+0000","last_unstale":"2024-02-05T07:33:57.570460+0000","last_undegraded":"2024-02-05T07:33:57.570460+0000","last_fullsized":"2024-02-05T07:33:57.570460+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:23.118569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:23.118569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:34.830436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"326'22","reported_seq":840,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.563170+0000","last_change":"2024-02-05T07:33:30.327114+0000","last_active":"2024-02-05T07:33:57.563170+0000","last_peered":"2024-02-05T07:33:57.563170+0000","last_clean":"2024-02-05T07:33:57.563170+0000","last_became_active":"2024-02-05T07:30:40.494460+0000","last_became_peered":"2024-02-05T07:30:40.494460+0000","last_unstale":"2024-02-05T07:33:57.563170+0000","last_undegraded":"2024-02-05T07:33:57.563170+0000","last_fullsized":"2024-02-05T07:33:57.563170+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'22","last_scrub_stamp":"2024-02-05T07:33:30.327027+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.327027+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:47.402987+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":806,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567437+0000","last_change":"2024-02-05T07:33:42.096031+0000","last_active":"2024-02-05T07:33:57.567437+0000","last_peered":"2024-02-05T07:33:57.567437+0000","last_clean":"2024-02-05T07:33:57.567437+0000","last_became_active":"2024-02-05T07:27:16.728869+0000","last_became_peered":"2024-02-05T07:27:16.728869+0000","last_unstale":"2024-02-05T07:33:57.567437+0000","last_undegraded":"2024-02-05T07:33:57.567437+0000","last_fullsized":"2024-02-05T07:33:57.567437+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:42.095953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:33:42.095953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:55.145283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":785,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559264+0000","last_change":"2024-02-05T07:29:45.546235+0000","last_active":"2024-02-05T07:33:57.559264+0000","last_peered":"2024-02-05T07:33:57.559264+0000","last_clean":"2024-02-05T07:33:57.559264+0000","last_became_active":"2024-02-05T07:28:25.750373+0000","last_became_peered":"2024-02-05T07:28:25.750373+0000","last_unstale":"2024-02-05T07:33:57.559264+0000","last_undegraded":"2024-02-05T07:33:57.559264+0000","last_fullsized":"2024-02-05T07:33:57.559264+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:45.546152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.546152+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":647,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.681347+0000","last_change":"2024-02-05T07:32:38.877762+0000","last_active":"2024-02-05T07:34:02.681347+0000","last_peered":"2024-02-05T07:34:02.681347+0000","last_clean":"2024-02-05T07:34:02.681347+0000","last_became_active":"2024-02-05T07:30:40.489314+0000","last_became_peered":"2024-02-05T07:30:40.489314+0000","last_unstale":"2024-02-05T07:34:02.681347+0000","last_undegraded":"2024-02-05T07:34:02.681347+0000","last_fullsized":"2024-02-05T07:34:02.681347+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:38.877685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:32:38.877685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"429'2245","reported_seq":2267,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:03.663720+0000","last_change":"2024-02-05T07:33:38.775531+0000","last_active":"2024-02-05T07:34:03.663720+0000","last_peered":"2024-02-05T07:34:03.663720+0000","last_clean":"2024-02-05T07:34:03.663720+0000","last_became_active":"2024-02-05T07:33:38.775221+0000","last_became_peered":"2024-02-05T07:33:38.775221+0000","last_unstale":"2024-02-05T07:34:03.663720+0000","last_undegraded":"2024-02-05T07:34:03.663720+0000","last_fullsized":"2024-02-05T07:34:03.663720+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":2245,"log_dups_size":0,"ondisk_log_size":2245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:44.600443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147128320,"num_objects":2245,"num_object_clones":0,"num_object_copies":6735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2245,"num_write_kb":143680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":766,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567067+0000","last_change":"2024-02-05T07:33:41.082128+0000","last_active":"2024-02-05T07:33:57.567067+0000","last_peered":"2024-02-05T07:33:57.567067+0000","last_clean":"2024-02-05T07:33:57.567067+0000","last_became_active":"2024-02-05T07:28:14.305034+0000","last_became_peered":"2024-02-05T07:28:14.305034+0000","last_unstale":"2024-02-05T07:33:57.567067+0000","last_undegraded":"2024-02-05T07:33:57.567067+0000","last_fullsized":"2024-02-05T07:33:57.567067+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:41.082046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.082046+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:52.796910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":514,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.560128+0000","last_change":"2024-02-05T07:29:48.627198+0000","last_active":"2024-02-05T07:33:57.560128+0000","last_peered":"2024-02-05T07:33:57.560128+0000","last_clean":"2024-02-05T07:33:57.560128+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:33:57.560128+0000","last_undegraded":"2024-02-05T07:33:57.560128+0000","last_fullsized":"2024-02-05T07:33:57.560128+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:48.627107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:48.627107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557100+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"326'44","reported_seq":696,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567448+0000","last_change":"2024-02-05T07:30:36.009087+0000","last_active":"2024-02-05T07:33:57.567448+0000","last_peered":"2024-02-05T07:33:57.567448+0000","last_clean":"2024-02-05T07:33:57.567448+0000","last_became_active":"2024-02-05T07:27:16.708475+0000","last_became_peered":"2024-02-05T07:27:16.708475+0000","last_unstale":"2024-02-05T07:33:57.567448+0000","last_undegraded":"2024-02-05T07:33:57.567448+0000","last_fullsized":"2024-02-05T07:33:57.567448+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'44","last_scrub_stamp":"2024-02-05T07:30:36.008885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.008885+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:01.924745+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":156,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567478+0000","last_change":"2024-02-05T07:30:32.964745+0000","last_active":"2024-02-05T07:33:57.567478+0000","last_peered":"2024-02-05T07:33:57.567478+0000","last_clean":"2024-02-05T07:33:57.567478+0000","last_became_active":"2024-02-05T07:30:30.524566+0000","last_became_peered":"2024-02-05T07:30:30.524566+0000","last_unstale":"2024-02-05T07:33:57.567478+0000","last_undegraded":"2024-02-05T07:33:57.567478+0000","last_fullsized":"2024-02-05T07:33:57.567478+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:32.964622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:32.964622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:01.924677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":692,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559644+0000","last_change":"2024-02-05T07:33:43.473208+0000","last_active":"2024-02-05T07:33:57.559644+0000","last_peered":"2024-02-05T07:33:57.559644+0000","last_clean":"2024-02-05T07:33:57.559644+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:33:57.559644+0000","last_undegraded":"2024-02-05T07:33:57.559644+0000","last_fullsized":"2024-02-05T07:33:57.559644+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:33:43.473107+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:33:43.473107+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:12.505533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":356,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.566738+0000","last_change":"2024-02-05T07:30:40.508412+0000","last_active":"2024-02-05T07:33:57.566738+0000","last_peered":"2024-02-05T07:33:57.566738+0000","last_clean":"2024-02-05T07:33:57.566738+0000","last_became_active":"2024-02-05T07:30:40.508071+0000","last_became_peered":"2024-02-05T07:30:40.508071+0000","last_unstale":"2024-02-05T07:33:57.566738+0000","last_undegraded":"2024-02-05T07:33:57.566738+0000","last_fullsized":"2024-02-05T07:33:57.566738+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:42.075445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:42.075445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.641763+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.5","version":"428'1915","reported_seq":1935,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:34:00.915309+0000","last_change":"2024-02-05T07:33:38.778391+0000","last_active":"2024-02-05T07:34:00.915309+0000","last_peered":"2024-02-05T07:34:00.915309+0000","last_clean":"2024-02-05T07:34:00.915309+0000","last_became_active":"2024-02-05T07:33:38.778086+0000","last_became_peered":"2024-02-05T07:33:38.778086+0000","last_unstale":"2024-02-05T07:34:00.915309+0000","last_undegraded":"2024-02-05T07:34:00.915309+0000","last_fullsized":"2024-02-05T07:34:00.915309+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":1915,"log_dups_size":0,"ondisk_log_size":1915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:41.700207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125501440,"num_objects":1915,"num_object_clones":0,"num_object_copies":5745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1915,"num_write_kb":122560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":689,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558881+0000","last_change":"2024-02-05T07:32:23.628807+0000","last_active":"2024-02-05T07:33:57.558881+0000","last_peered":"2024-02-05T07:33:57.558881+0000","last_clean":"2024-02-05T07:33:57.558881+0000","last_became_active":"2024-02-05T07:31:36.842497+0000","last_became_peered":"2024-02-05T07:31:36.842497+0000","last_unstale":"2024-02-05T07:33:57.558881+0000","last_undegraded":"2024-02-05T07:33:57.558881+0000","last_fullsized":"2024-02-05T07:33:57.558881+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:23.628698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:23.628698+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"326'15","reported_seq":557,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.565641+0000","last_change":"2024-02-05T07:29:51.665907+0000","last_active":"2024-02-05T07:33:57.565641+0000","last_peered":"2024-02-05T07:33:57.565641+0000","last_clean":"2024-02-05T07:33:57.565641+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:33:57.565641+0000","last_undegraded":"2024-02-05T07:33:57.565641+0000","last_fullsized":"2024-02-05T07:33:57.565641+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:29:51.665808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.665808+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":352,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567352+0000","last_change":"2024-02-05T07:30:38.998491+0000","last_active":"2024-02-05T07:33:57.567352+0000","last_peered":"2024-02-05T07:33:57.567352+0000","last_clean":"2024-02-05T07:33:57.567352+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:33:57.567352+0000","last_undegraded":"2024-02-05T07:33:57.567352+0000","last_fullsized":"2024-02-05T07:33:57.567352+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:38.995586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:38.995586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642398+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":780,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567181+0000","last_change":"2024-02-05T07:32:47.161468+0000","last_active":"2024-02-05T07:33:57.567181+0000","last_peered":"2024-02-05T07:33:57.567181+0000","last_clean":"2024-02-05T07:33:57.567181+0000","last_became_active":"2024-02-05T07:27:16.708205+0000","last_became_peered":"2024-02-05T07:27:16.708205+0000","last_unstale":"2024-02-05T07:33:57.567181+0000","last_undegraded":"2024-02-05T07:33:57.567181+0000","last_fullsized":"2024-02-05T07:33:57.567181+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:47.161343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:47.161343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:13.165702+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":149,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567657+0000","last_change":"2024-02-05T07:32:46.198277+0000","last_active":"2024-02-05T07:33:57.567657+0000","last_peered":"2024-02-05T07:33:57.567657+0000","last_clean":"2024-02-05T07:33:57.567657+0000","last_became_active":"2024-02-05T07:30:40.486401+0000","last_became_peered":"2024-02-05T07:30:40.486401+0000","last_unstale":"2024-02-05T07:33:57.567657+0000","last_undegraded":"2024-02-05T07:33:57.567657+0000","last_fullsized":"2024-02-05T07:33:57.567657+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:46.198155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:32:46.198155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:01.924807+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"428'1977","reported_seq":1997,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:34:00.347790+0000","last_change":"2024-02-05T07:33:38.776231+0000","last_active":"2024-02-05T07:34:00.347790+0000","last_peered":"2024-02-05T07:34:00.347790+0000","last_clean":"2024-02-05T07:34:00.347790+0000","last_became_active":"2024-02-05T07:33:38.775291+0000","last_became_peered":"2024-02-05T07:33:38.775291+0000","last_unstale":"2024-02-05T07:34:00.347790+0000","last_undegraded":"2024-02-05T07:34:00.347790+0000","last_fullsized":"2024-02-05T07:34:00.347790+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":1977,"log_dups_size":0,"ondisk_log_size":1977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:44.655160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129564672,"num_objects":1977,"num_object_clones":0,"num_object_copies":5931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1977,"num_write_kb":126528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":607,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.681113+0000","last_change":"2024-02-05T07:32:24.854963+0000","last_active":"2024-02-05T07:34:02.681113+0000","last_peered":"2024-02-05T07:34:02.681113+0000","last_clean":"2024-02-05T07:34:02.681113+0000","last_became_active":"2024-02-05T07:28:26.596711+0000","last_became_peered":"2024-02-05T07:28:26.596711+0000","last_unstale":"2024-02-05T07:34:02.681113+0000","last_undegraded":"2024-02-05T07:34:02.681113+0000","last_fullsized":"2024-02-05T07:34:02.681113+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:24.854881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:24.854881+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:07.049561+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"326'35","reported_seq":685,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.680473+0000","last_change":"2024-02-05T07:33:27.163756+0000","last_active":"2024-02-05T07:34:02.680473+0000","last_peered":"2024-02-05T07:34:02.680473+0000","last_clean":"2024-02-05T07:34:02.680473+0000","last_became_active":"2024-02-05T07:30:40.482753+0000","last_became_peered":"2024-02-05T07:30:40.482753+0000","last_unstale":"2024-02-05T07:34:02.680473+0000","last_undegraded":"2024-02-05T07:34:02.680473+0000","last_fullsized":"2024-02-05T07:34:02.680473+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:33:27.163551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:27.163551+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:31.162649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":658,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.681264+0000","last_change":"2024-02-05T07:32:39.793152+0000","last_active":"2024-02-05T07:34:02.681264+0000","last_peered":"2024-02-05T07:34:02.681264+0000","last_clean":"2024-02-05T07:34:02.681264+0000","last_became_active":"2024-02-05T07:30:40.507705+0000","last_became_peered":"2024-02-05T07:30:40.507705+0000","last_unstale":"2024-02-05T07:34:02.681264+0000","last_undegraded":"2024-02-05T07:34:02.681264+0000","last_fullsized":"2024-02-05T07:34:02.681264+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_clean_scrub_stamp":"2024-02-05T07:32:39.793075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":780,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.568030+0000","last_change":"2024-02-05T07:33:28.121535+0000","last_active":"2024-02-05T07:33:57.568030+0000","last_peered":"2024-02-05T07:33:57.568030+0000","last_clean":"2024-02-05T07:33:57.568030+0000","last_became_active":"2024-02-05T07:30:40.479288+0000","last_became_peered":"2024-02-05T07:30:40.479288+0000","last_unstale":"2024-02-05T07:33:57.568030+0000","last_undegraded":"2024-02-05T07:33:57.568030+0000","last_fullsized":"2024-02-05T07:33:57.568030+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.121427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.121427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:39.578069+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":562,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.570942+0000","last_change":"2024-02-05T07:33:31.033044+0000","last_active":"2024-02-05T07:33:57.570942+0000","last_peered":"2024-02-05T07:33:57.570942+0000","last_clean":"2024-02-05T07:33:57.570942+0000","last_became_active":"2024-02-05T07:30:40.496822+0000","last_became_peered":"2024-02-05T07:30:40.496822+0000","last_unstale":"2024-02-05T07:33:57.570942+0000","last_undegraded":"2024-02-05T07:33:57.570942+0000","last_fullsized":"2024-02-05T07:33:57.570942+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:31.032967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_clean_scrub_stamp":"2024-02-05T07:33:31.032967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:58.673311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"429'2181","reported_seq":2203,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:03.670730+0000","last_change":"2024-02-05T07:33:38.614213+0000","last_active":"2024-02-05T07:34:03.670730+0000","last_peered":"2024-02-05T07:34:03.670730+0000","last_clean":"2024-02-05T07:34:03.670730+0000","last_became_active":"2024-02-05T07:33:38.613855+0000","last_became_peered":"2024-02-05T07:33:38.613855+0000","last_unstale":"2024-02-05T07:34:03.670730+0000","last_undegraded":"2024-02-05T07:34:03.670730+0000","last_fullsized":"2024-02-05T07:34:03.670730+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":2181,"log_dups_size":0,"ondisk_log_size":2181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:42.559822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142934016,"num_objects":2181,"num_object_clones":0,"num_object_copies":6543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2181,"num_write_kb":139584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":1021,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.566481+0000","last_change":"2024-02-05T07:32:41.754728+0000","last_active":"2024-02-05T07:33:57.566481+0000","last_peered":"2024-02-05T07:33:57.566481+0000","last_clean":"2024-02-05T07:33:57.566481+0000","last_became_active":"2024-02-05T07:31:36.840096+0000","last_became_peered":"2024-02-05T07:31:36.840096+0000","last_unstale":"2024-02-05T07:33:57.566481+0000","last_undegraded":"2024-02-05T07:33:57.566481+0000","last_fullsized":"2024-02-05T07:33:57.566481+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:41.754605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:41.754605+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:09.050452+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":776,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.570812+0000","last_change":"2024-02-05T07:33:29.082880+0000","last_active":"2024-02-05T07:33:57.570812+0000","last_peered":"2024-02-05T07:33:57.570812+0000","last_clean":"2024-02-05T07:33:57.570812+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:33:57.570812+0000","last_undegraded":"2024-02-05T07:33:57.570812+0000","last_fullsized":"2024-02-05T07:33:57.570812+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:29.082803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:29.082803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:43.891007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"325'18","reported_seq":856,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559047+0000","last_change":"2024-02-05T07:29:49.022612+0000","last_active":"2024-02-05T07:33:57.559047+0000","last_peered":"2024-02-05T07:33:57.559047+0000","last_clean":"2024-02-05T07:33:57.559047+0000","last_became_active":"2024-02-05T07:27:16.706871+0000","last_became_peered":"2024-02-05T07:27:16.706871+0000","last_unstale":"2024-02-05T07:33:57.559047+0000","last_undegraded":"2024-02-05T07:33:57.559047+0000","last_fullsized":"2024-02-05T07:33:57.559047+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'18","last_scrub_stamp":"2024-02-05T07:29:49.021772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.021772+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":146,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559014+0000","last_change":"2024-02-05T07:33:33.822290+0000","last_active":"2024-02-05T07:33:57.559014+0000","last_peered":"2024-02-05T07:33:57.559014+0000","last_clean":"2024-02-05T07:33:57.559014+0000","last_became_active":"2024-02-05T07:30:40.518949+0000","last_became_peered":"2024-02-05T07:30:40.518949+0000","last_unstale":"2024-02-05T07:33:57.559014+0000","last_undegraded":"2024-02-05T07:33:57.559014+0000","last_fullsized":"2024-02-05T07:33:57.559014+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:33.822199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.822199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:39.649249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":355,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.564598+0000","last_change":"2024-02-05T07:30:40.486334+0000","last_active":"2024-02-05T07:33:57.564598+0000","last_peered":"2024-02-05T07:33:57.564598+0000","last_clean":"2024-02-05T07:33:57.564598+0000","last_became_active":"2024-02-05T07:30:40.485991+0000","last_became_peered":"2024-02-05T07:30:40.485991+0000","last_unstale":"2024-02-05T07:33:57.564598+0000","last_undegraded":"2024-02-05T07:33:57.564598+0000","last_fullsized":"2024-02-05T07:33:57.564598+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:48.088229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.088229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.641927+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"428'1945","reported_seq":1963,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:34:00.352759+0000","last_change":"2024-02-05T07:33:38.775313+0000","last_active":"2024-02-05T07:34:00.352759+0000","last_peered":"2024-02-05T07:34:00.352759+0000","last_clean":"2024-02-05T07:34:00.352759+0000","last_became_active":"2024-02-05T07:33:38.774996+0000","last_became_peered":"2024-02-05T07:33:38.774996+0000","last_unstale":"2024-02-05T07:34:00.352759+0000","last_undegraded":"2024-02-05T07:34:00.352759+0000","last_fullsized":"2024-02-05T07:34:00.352759+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":1945,"log_dups_size":0,"ondisk_log_size":1945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:37.651266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127467520,"num_objects":1945,"num_object_clones":0,"num_object_copies":5835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1945,"num_write_kb":124480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":605,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.557803+0000","last_change":"2024-02-05T07:32:22.621687+0000","last_active":"2024-02-05T07:33:57.557803+0000","last_peered":"2024-02-05T07:33:57.557803+0000","last_clean":"2024-02-05T07:33:57.557803+0000","last_became_active":"2024-02-05T07:31:36.837268+0000","last_became_peered":"2024-02-05T07:31:36.837268+0000","last_unstale":"2024-02-05T07:33:57.557803+0000","last_undegraded":"2024-02-05T07:33:57.557803+0000","last_fullsized":"2024-02-05T07:33:57.557803+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:22.621608+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_clean_scrub_stamp":"2024-02-05T07:32:22.621608+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.931064+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"365'12","reported_seq":810,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.680171+0000","last_change":"2024-02-05T07:32:30.801254+0000","last_active":"2024-02-05T07:34:02.680171+0000","last_peered":"2024-02-05T07:34:02.680171+0000","last_clean":"2024-02-05T07:34:02.680171+0000","last_became_active":"2024-02-05T07:30:40.500113+0000","last_became_peered":"2024-02-05T07:30:40.500113+0000","last_unstale":"2024-02-05T07:34:02.680171+0000","last_undegraded":"2024-02-05T07:34:02.680171+0000","last_fullsized":"2024-02-05T07:34:02.680171+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'12","last_scrub_stamp":"2024-02-05T07:32:30.801134+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:32:30.801134+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:07.049643+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"326'27","reported_seq":405,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.679786+0000","last_change":"2024-02-05T07:32:29.833732+0000","last_active":"2024-02-05T07:34:02.679786+0000","last_peered":"2024-02-05T07:34:02.679786+0000","last_clean":"2024-02-05T07:34:02.679786+0000","last_became_active":"2024-02-05T07:30:40.497982+0000","last_became_peered":"2024-02-05T07:30:40.497982+0000","last_unstale":"2024-02-05T07:34:02.679786+0000","last_undegraded":"2024-02-05T07:34:02.679786+0000","last_fullsized":"2024-02-05T07:34:02.679786+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:32:29.833636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:29.833636+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:07.049713+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":162,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558076+0000","last_change":"2024-02-05T07:33:27.649686+0000","last_active":"2024-02-05T07:33:57.558076+0000","last_peered":"2024-02-05T07:33:57.558076+0000","last_clean":"2024-02-05T07:33:57.558076+0000","last_became_active":"2024-02-05T07:30:40.503903+0000","last_became_peered":"2024-02-05T07:30:40.503903+0000","last_unstale":"2024-02-05T07:33:57.558076+0000","last_undegraded":"2024-02-05T07:33:57.558076+0000","last_fullsized":"2024-02-05T07:33:57.558076+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:33:27.649596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:27.649596+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:43.748950+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":440,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.680356+0000","last_change":"2024-02-05T07:33:35.177937+0000","last_active":"2024-02-05T07:34:02.680356+0000","last_peered":"2024-02-05T07:34:02.680356+0000","last_clean":"2024-02-05T07:34:02.680356+0000","last_became_active":"2024-02-05T07:28:26.589671+0000","last_became_peered":"2024-02-05T07:28:26.589671+0000","last_unstale":"2024-02-05T07:34:02.680356+0000","last_undegraded":"2024-02-05T07:34:02.680356+0000","last_fullsized":"2024-02-05T07:34:02.680356+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:35.177804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:33:35.177804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:44.172989+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":616,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559424+0000","last_change":"2024-02-05T07:33:44.492558+0000","last_active":"2024-02-05T07:33:57.559424+0000","last_peered":"2024-02-05T07:33:57.559424+0000","last_clean":"2024-02-05T07:33:57.559424+0000","last_became_active":"2024-02-05T07:30:43.523108+0000","last_became_peered":"2024-02-05T07:30:43.523108+0000","last_unstale":"2024-02-05T07:33:57.559424+0000","last_undegraded":"2024-02-05T07:33:57.559424+0000","last_fullsized":"2024-02-05T07:33:57.559424+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:44.492478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.492478+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:50.920619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"326'12","reported_seq":380,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567533+0000","last_change":"2024-02-05T07:33:41.762486+0000","last_active":"2024-02-05T07:33:57.567533+0000","last_peered":"2024-02-05T07:33:57.567533+0000","last_clean":"2024-02-05T07:33:57.567533+0000","last_became_active":"2024-02-05T07:30:40.496980+0000","last_became_peered":"2024-02-05T07:30:40.496980+0000","last_unstale":"2024-02-05T07:33:57.567533+0000","last_undegraded":"2024-02-05T07:33:57.567533+0000","last_fullsized":"2024-02-05T07:33:57.567533+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'12","last_scrub_stamp":"2024-02-05T07:33:41.762404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.762404+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:00.099696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":584,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.679954+0000","last_change":"2024-02-05T07:33:51.105371+0000","last_active":"2024-02-05T07:34:02.679954+0000","last_peered":"2024-02-05T07:34:02.679954+0000","last_clean":"2024-02-05T07:34:02.679954+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:34:02.679954+0000","last_undegraded":"2024-02-05T07:34:02.679954+0000","last_fullsized":"2024-02-05T07:34:02.679954+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:33:51.105251+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:33:51.105251+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:18.353438+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":352,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.566429+0000","last_change":"2024-02-05T07:30:27.965535+0000","last_active":"2024-02-05T07:33:57.566429+0000","last_peered":"2024-02-05T07:33:57.566429+0000","last_clean":"2024-02-05T07:33:57.566429+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:33:57.566429+0000","last_undegraded":"2024-02-05T07:33:57.566429+0000","last_fullsized":"2024-02-05T07:33:57.566429+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:27.965449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:27.965449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":634,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.681636+0000","last_change":"2024-02-05T07:32:32.844370+0000","last_active":"2024-02-05T07:34:02.681636+0000","last_peered":"2024-02-05T07:34:02.681636+0000","last_clean":"2024-02-05T07:34:02.681636+0000","last_became_active":"2024-02-05T07:30:40.509216+0000","last_became_peered":"2024-02-05T07:30:40.509216+0000","last_unstale":"2024-02-05T07:34:02.681636+0000","last_undegraded":"2024-02-05T07:34:02.681636+0000","last_fullsized":"2024-02-05T07:34:02.681636+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:32.844264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:32:32.844264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:07.049775+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":750,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.565298+0000","last_change":"2024-02-05T07:33:32.282627+0000","last_active":"2024-02-05T07:33:57.565298+0000","last_peered":"2024-02-05T07:33:57.565298+0000","last_clean":"2024-02-05T07:33:57.565298+0000","last_became_active":"2024-02-05T07:28:19.311737+0000","last_became_peered":"2024-02-05T07:28:19.311737+0000","last_unstale":"2024-02-05T07:33:57.565298+0000","last_undegraded":"2024-02-05T07:33:57.565298+0000","last_fullsized":"2024-02-05T07:33:57.565298+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:32.282546+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.702565+0000","last_clean_scrub_stamp":"2024-02-05T07:33:32.282546+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.423885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"325'15","reported_seq":384,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.566653+0000","last_change":"2024-02-05T07:33:45.711524+0000","last_active":"2024-02-05T07:33:57.566653+0000","last_peered":"2024-02-05T07:33:57.566653+0000","last_clean":"2024-02-05T07:33:57.566653+0000","last_became_active":"2024-02-05T07:30:40.485837+0000","last_became_peered":"2024-02-05T07:30:40.485837+0000","last_unstale":"2024-02-05T07:33:57.566653+0000","last_undegraded":"2024-02-05T07:33:57.566653+0000","last_fullsized":"2024-02-05T07:33:57.566653+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:33:45.711399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:45.711399+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:00.359295+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":645,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558927+0000","last_change":"2024-02-05T07:30:40.498263+0000","last_active":"2024-02-05T07:33:57.558927+0000","last_peered":"2024-02-05T07:33:57.558927+0000","last_clean":"2024-02-05T07:33:57.558927+0000","last_became_active":"2024-02-05T07:30:40.497956+0000","last_became_peered":"2024-02-05T07:30:40.497956+0000","last_unstale":"2024-02-05T07:33:57.558927+0000","last_undegraded":"2024-02-05T07:33:57.558927+0000","last_fullsized":"2024-02-05T07:33:57.558927+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:33.183427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:30:33.183427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930756+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":355,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567626+0000","last_change":"2024-02-05T07:33:50.723381+0000","last_active":"2024-02-05T07:33:57.567626+0000","last_peered":"2024-02-05T07:33:57.567626+0000","last_clean":"2024-02-05T07:33:57.567626+0000","last_became_active":"2024-02-05T07:30:40.496076+0000","last_became_peered":"2024-02-05T07:30:40.496076+0000","last_unstale":"2024-02-05T07:33:57.567626+0000","last_undegraded":"2024-02-05T07:33:57.567626+0000","last_fullsized":"2024-02-05T07:33:57.567626+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_clean_scrub_stamp":"2024-02-05T07:33:50.723212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:09.045969+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":770,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567623+0000","last_change":"2024-02-05T07:32:48.164374+0000","last_active":"2024-02-05T07:33:57.567623+0000","last_peered":"2024-02-05T07:33:57.567623+0000","last_clean":"2024-02-05T07:33:57.567623+0000","last_became_active":"2024-02-05T07:30:40.487455+0000","last_became_peered":"2024-02-05T07:30:40.487455+0000","last_unstale":"2024-02-05T07:33:57.567623+0000","last_undegraded":"2024-02-05T07:33:57.567623+0000","last_fullsized":"2024-02-05T07:33:57.567623+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:48.164259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:32:48.164259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:01.924839+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":456,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.679674+0000","last_change":"2024-02-05T07:32:18.784193+0000","last_active":"2024-02-05T07:34:02.679674+0000","last_peered":"2024-02-05T07:34:02.679674+0000","last_clean":"2024-02-05T07:34:02.679674+0000","last_became_active":"2024-02-05T07:30:43.466854+0000","last_became_peered":"2024-02-05T07:30:43.466854+0000","last_unstale":"2024-02-05T07:34:02.679674+0000","last_undegraded":"2024-02-05T07:34:02.679674+0000","last_fullsized":"2024-02-05T07:34:02.679674+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:18.784071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:18.784071+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:07.049677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"326'19","reported_seq":704,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.564239+0000","last_change":"2024-02-05T07:30:40.503823+0000","last_active":"2024-02-05T07:33:57.564239+0000","last_peered":"2024-02-05T07:33:57.564239+0000","last_clean":"2024-02-05T07:33:57.564239+0000","last_became_active":"2024-02-05T07:30:40.483700+0000","last_became_peered":"2024-02-05T07:30:40.483700+0000","last_unstale":"2024-02-05T07:33:57.564239+0000","last_undegraded":"2024-02-05T07:33:57.564239+0000","last_fullsized":"2024-02-05T07:33:57.564239+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:30:35.138361+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:30:35.138361+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":518,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.568093+0000","last_change":"2024-02-05T07:30:40.476394+0000","last_active":"2024-02-05T07:33:57.568093+0000","last_peered":"2024-02-05T07:33:57.568093+0000","last_clean":"2024-02-05T07:33:57.568093+0000","last_became_active":"2024-02-05T07:30:40.476058+0000","last_became_peered":"2024-02-05T07:30:40.476058+0000","last_unstale":"2024-02-05T07:33:57.568093+0000","last_undegraded":"2024-02-05T07:33:57.568093+0000","last_fullsized":"2024-02-05T07:33:57.568093+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.314735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.314735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642268+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"365'11","reported_seq":444,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567084+0000","last_change":"2024-02-05T07:33:39.734203+0000","last_active":"2024-02-05T07:33:57.567084+0000","last_peered":"2024-02-05T07:33:57.567084+0000","last_clean":"2024-02-05T07:33:57.567084+0000","last_became_active":"2024-02-05T07:30:40.495708+0000","last_became_peered":"2024-02-05T07:30:40.495708+0000","last_unstale":"2024-02-05T07:33:57.567084+0000","last_undegraded":"2024-02-05T07:33:57.567084+0000","last_fullsized":"2024-02-05T07:33:57.567084+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:33:39.734129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:39.734129+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:00.775574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":762,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558405+0000","last_change":"2024-02-05T07:33:38.879276+0000","last_active":"2024-02-05T07:33:57.558405+0000","last_peered":"2024-02-05T07:33:57.558405+0000","last_clean":"2024-02-05T07:33:57.558405+0000","last_became_active":"2024-02-05T07:27:03.364334+0000","last_became_peered":"2024-02-05T07:27:03.364334+0000","last_unstale":"2024-02-05T07:33:57.558405+0000","last_undegraded":"2024-02-05T07:33:57.558405+0000","last_fullsized":"2024-02-05T07:33:57.558405+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:38.879177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:38.879177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:46.608368+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":599,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.563483+0000","last_change":"2024-02-05T07:28:41.013449+0000","last_active":"2024-02-05T07:33:57.563483+0000","last_peered":"2024-02-05T07:33:57.563483+0000","last_clean":"2024-02-05T07:33:57.563483+0000","last_became_active":"2024-02-05T07:28:19.302393+0000","last_became_peered":"2024-02-05T07:28:19.302393+0000","last_unstale":"2024-02-05T07:33:57.563483+0000","last_undegraded":"2024-02-05T07:33:57.563483+0000","last_fullsized":"2024-02-05T07:33:57.563483+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:28:41.013360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:28:41.013360+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.641866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"326'25","reported_seq":394,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567254+0000","last_change":"2024-02-05T07:30:40.512630+0000","last_active":"2024-02-05T07:33:57.567254+0000","last_peered":"2024-02-05T07:33:57.567254+0000","last_clean":"2024-02-05T07:33:57.567254+0000","last_became_active":"2024-02-05T07:30:40.486394+0000","last_became_peered":"2024-02-05T07:30:40.486394+0000","last_unstale":"2024-02-05T07:33:57.567254+0000","last_undegraded":"2024-02-05T07:33:57.567254+0000","last_fullsized":"2024-02-05T07:33:57.567254+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:30:33.959124+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:30:33.959124+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642297+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":139,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558807+0000","last_change":"2024-02-05T07:30:40.505910+0000","last_active":"2024-02-05T07:33:57.558807+0000","last_peered":"2024-02-05T07:33:57.558807+0000","last_clean":"2024-02-05T07:33:57.558807+0000","last_became_active":"2024-02-05T07:30:40.505600+0000","last_became_peered":"2024-02-05T07:30:40.505600+0000","last_unstale":"2024-02-05T07:33:57.558807+0000","last_undegraded":"2024-02-05T07:33:57.558807+0000","last_fullsized":"2024-02-05T07:33:57.558807+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:46.210169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:46.210169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"217'9","reported_seq":571,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.570092+0000","last_change":"2024-02-05T07:30:27.993196+0000","last_active":"2024-02-05T07:33:57.570092+0000","last_peered":"2024-02-05T07:33:57.570092+0000","last_clean":"2024-02-05T07:33:57.570092+0000","last_became_active":"2024-02-05T07:25:27.246616+0000","last_became_peered":"2024-02-05T07:25:27.246616+0000","last_unstale":"2024-02-05T07:33:57.570092+0000","last_undegraded":"2024-02-05T07:33:57.570092+0000","last_fullsized":"2024-02-05T07:33:57.570092+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9","last_scrub_stamp":"2024-02-05T07:30:27.993000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:27.993000+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:01.924543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":506,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.566564+0000","last_change":"2024-02-05T07:30:40.508206+0000","last_active":"2024-02-05T07:33:57.566564+0000","last_peered":"2024-02-05T07:33:57.566564+0000","last_clean":"2024-02-05T07:33:57.566564+0000","last_became_active":"2024-02-05T07:30:40.507831+0000","last_became_peered":"2024-02-05T07:30:40.507831+0000","last_unstale":"2024-02-05T07:33:57.566564+0000","last_undegraded":"2024-02-05T07:33:57.566564+0000","last_fullsized":"2024-02-05T07:33:57.566564+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:27:01.069970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:27:01.069970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.641712+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.16","version":"86'128","reported_seq":410,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.561395+0000","last_change":"2024-02-05T07:32:42.321154+0000","last_active":"2024-02-05T07:33:57.561395+0000","last_peered":"2024-02-05T07:33:57.561395+0000","last_clean":"2024-02-05T07:33:57.561395+0000","last_became_active":"2024-02-05T07:31:41.837877+0000","last_became_peered":"2024-02-05T07:31:41.837877+0000","last_unstale":"2024-02-05T07:33:57.561395+0000","last_undegraded":"2024-02-05T07:33:57.561395+0000","last_fullsized":"2024-02-05T07:33:57.561395+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:42.321045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:42.321045+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557675+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"326'19","reported_seq":822,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558899+0000","last_change":"2024-02-05T07:32:31.667966+0000","last_active":"2024-02-05T07:33:57.558899+0000","last_peered":"2024-02-05T07:33:57.558899+0000","last_clean":"2024-02-05T07:33:57.558899+0000","last_became_active":"2024-02-05T07:28:25.779886+0000","last_became_peered":"2024-02-05T07:28:25.779886+0000","last_unstale":"2024-02-05T07:33:57.558899+0000","last_undegraded":"2024-02-05T07:33:57.558899+0000","last_fullsized":"2024-02-05T07:33:57.558899+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:32:31.667882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:31.667882+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930960+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":775,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.562040+0000","last_change":"2024-02-05T07:30:40.510773+0000","last_active":"2024-02-05T07:33:57.562040+0000","last_peered":"2024-02-05T07:33:57.562040+0000","last_clean":"2024-02-05T07:33:57.562040+0000","last_became_active":"2024-02-05T07:30:40.510301+0000","last_became_peered":"2024-02-05T07:30:40.510301+0000","last_unstale":"2024-02-05T07:33:57.562040+0000","last_undegraded":"2024-02-05T07:33:57.562040+0000","last_fullsized":"2024-02-05T07:33:57.562040+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:36.798282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:29:36.798282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":783,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.561562+0000","last_change":"2024-02-05T07:32:48.346195+0000","last_active":"2024-02-05T07:33:57.561562+0000","last_peered":"2024-02-05T07:33:57.561562+0000","last_clean":"2024-02-05T07:33:57.561562+0000","last_became_active":"2024-02-05T07:30:40.487141+0000","last_became_peered":"2024-02-05T07:30:40.487141+0000","last_unstale":"2024-02-05T07:33:57.561562+0000","last_undegraded":"2024-02-05T07:33:57.561562+0000","last_fullsized":"2024-02-05T07:33:57.561562+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_clean_scrub_stamp":"2024-02-05T07:32:48.345965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:06.833776+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":347,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.563926+0000","last_change":"2024-02-05T07:33:37.735878+0000","last_active":"2024-02-05T07:33:57.563926+0000","last_peered":"2024-02-05T07:33:57.563926+0000","last_clean":"2024-02-05T07:33:57.563926+0000","last_became_active":"2024-02-05T07:27:17.705910+0000","last_became_peered":"2024-02-05T07:27:17.705910+0000","last_unstale":"2024-02-05T07:33:57.563926+0000","last_undegraded":"2024-02-05T07:33:57.563926+0000","last_fullsized":"2024-02-05T07:33:57.563926+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.735757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.735757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:51.255829+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"86'128","reported_seq":628,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.681013+0000","last_change":"2024-02-05T07:32:23.818975+0000","last_active":"2024-02-05T07:34:02.681013+0000","last_peered":"2024-02-05T07:34:02.681013+0000","last_clean":"2024-02-05T07:34:02.681013+0000","last_became_active":"2024-02-05T07:31:41.837612+0000","last_became_peered":"2024-02-05T07:31:41.837612+0000","last_unstale":"2024-02-05T07:34:02.681013+0000","last_undegraded":"2024-02-05T07:34:02.681013+0000","last_fullsized":"2024-02-05T07:34:02.681013+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:23.818901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:23.818901+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:07.049746+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"325'15","reported_seq":388,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559320+0000","last_change":"2024-02-05T07:29:57.775013+0000","last_active":"2024-02-05T07:33:57.559320+0000","last_peered":"2024-02-05T07:33:57.559320+0000","last_clean":"2024-02-05T07:33:57.559320+0000","last_became_active":"2024-02-05T07:28:26.589705+0000","last_became_peered":"2024-02-05T07:28:26.589705+0000","last_unstale":"2024-02-05T07:33:57.559320+0000","last_undegraded":"2024-02-05T07:33:57.559320+0000","last_fullsized":"2024-02-05T07:33:57.559320+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:29:57.774913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:57.774913+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":563,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.563905+0000","last_change":"2024-02-05T07:30:40.489183+0000","last_active":"2024-02-05T07:33:57.563905+0000","last_peered":"2024-02-05T07:33:57.563905+0000","last_clean":"2024-02-05T07:33:57.563905+0000","last_became_active":"2024-02-05T07:30:40.488789+0000","last_became_peered":"2024-02-05T07:30:40.488789+0000","last_unstale":"2024-02-05T07:33:57.563905+0000","last_undegraded":"2024-02-05T07:33:57.563905+0000","last_fullsized":"2024-02-05T07:33:57.563905+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.220841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.220841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642231+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":349,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.564874+0000","last_change":"2024-02-05T07:33:47.684589+0000","last_active":"2024-02-05T07:33:57.564874+0000","last_peered":"2024-02-05T07:33:57.564874+0000","last_clean":"2024-02-05T07:33:57.564874+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:33:57.564874+0000","last_undegraded":"2024-02-05T07:33:57.564874+0000","last_fullsized":"2024-02-05T07:33:57.564874+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:47.684508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.684508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:12.589901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":524,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559788+0000","last_change":"2024-02-05T07:33:26.419431+0000","last_active":"2024-02-05T07:33:57.559788+0000","last_peered":"2024-02-05T07:33:57.559788+0000","last_clean":"2024-02-05T07:33:57.559788+0000","last_became_active":"2024-02-05T07:28:26.587701+0000","last_became_peered":"2024-02-05T07:28:26.587701+0000","last_unstale":"2024-02-05T07:33:57.559788+0000","last_undegraded":"2024-02-05T07:33:57.559788+0000","last_fullsized":"2024-02-05T07:33:57.559788+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:26.419332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:33:26.419332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:54.619952+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":1010,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567041+0000","last_change":"2024-02-05T07:30:43.515841+0000","last_active":"2024-02-05T07:33:57.567041+0000","last_peered":"2024-02-05T07:33:57.567041+0000","last_clean":"2024-02-05T07:33:57.567041+0000","last_became_active":"2024-02-05T07:30:43.515488+0000","last_became_peered":"2024-02-05T07:30:43.515488+0000","last_unstale":"2024-02-05T07:33:57.567041+0000","last_undegraded":"2024-02-05T07:33:57.567041+0000","last_fullsized":"2024-02-05T07:33:57.567041+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:30:36.995330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.995330+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:01.924614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"326'19","reported_seq":394,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.564071+0000","last_change":"2024-02-05T07:30:40.906805+0000","last_active":"2024-02-05T07:33:57.564071+0000","last_peered":"2024-02-05T07:33:57.564071+0000","last_clean":"2024-02-05T07:33:57.564071+0000","last_became_active":"2024-02-05T07:30:40.501893+0000","last_became_peered":"2024-02-05T07:30:40.501893+0000","last_unstale":"2024-02-05T07:33:57.564071+0000","last_undegraded":"2024-02-05T07:33:57.564071+0000","last_fullsized":"2024-02-05T07:33:57.564071+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:30:40.906693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.906693+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642350+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":164,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.563277+0000","last_change":"2024-02-05T07:30:40.492005+0000","last_active":"2024-02-05T07:33:57.563277+0000","last_peered":"2024-02-05T07:33:57.563277+0000","last_clean":"2024-02-05T07:33:57.563277+0000","last_became_active":"2024-02-05T07:30:40.491686+0000","last_became_peered":"2024-02-05T07:30:40.491686+0000","last_unstale":"2024-02-05T07:33:57.563277+0000","last_undegraded":"2024-02-05T07:33:57.563277+0000","last_fullsized":"2024-02-05T07:33:57.563277+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.097532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.097532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":637,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.557434+0000","last_change":"2024-02-05T07:30:40.497509+0000","last_active":"2024-02-05T07:33:57.557434+0000","last_peered":"2024-02-05T07:33:57.557434+0000","last_clean":"2024-02-05T07:33:57.557434+0000","last_became_active":"2024-02-05T07:30:40.497190+0000","last_became_peered":"2024-02-05T07:30:40.497190+0000","last_unstale":"2024-02-05T07:33:57.557434+0000","last_undegraded":"2024-02-05T07:33:57.557434+0000","last_fullsized":"2024-02-05T07:33:57.557434+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:29:58.191815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930682+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":353,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.564518+0000","last_change":"2024-02-05T07:29:40.983327+0000","last_active":"2024-02-05T07:33:57.564518+0000","last_peered":"2024-02-05T07:33:57.564518+0000","last_clean":"2024-02-05T07:33:57.564518+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:33:57.564518+0000","last_undegraded":"2024-02-05T07:33:57.564518+0000","last_fullsized":"2024-02-05T07:33:57.564518+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:40.983227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.983227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642151+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"86'128","reported_seq":404,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559464+0000","last_change":"2024-02-05T07:32:43.318421+0000","last_active":"2024-02-05T07:33:57.559464+0000","last_peered":"2024-02-05T07:33:57.559464+0000","last_clean":"2024-02-05T07:33:57.559464+0000","last_became_active":"2024-02-05T07:31:41.843850+0000","last_became_peered":"2024-02-05T07:31:41.843850+0000","last_unstale":"2024-02-05T07:33:57.559464+0000","last_undegraded":"2024-02-05T07:33:57.559464+0000","last_fullsized":"2024-02-05T07:33:57.559464+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:43.318297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:43.318297+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.720359+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"325'8","reported_seq":365,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.570374+0000","last_change":"2024-02-05T07:29:53.958469+0000","last_active":"2024-02-05T07:33:57.570374+0000","last_peered":"2024-02-05T07:33:57.570374+0000","last_clean":"2024-02-05T07:33:57.570374+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:33:57.570374+0000","last_undegraded":"2024-02-05T07:33:57.570374+0000","last_fullsized":"2024-02-05T07:33:57.570374+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:29:53.958384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.958384+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":352,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567289+0000","last_change":"2024-02-05T07:28:46.056835+0000","last_active":"2024-02-05T07:33:57.567289+0000","last_peered":"2024-02-05T07:33:57.567289+0000","last_clean":"2024-02-05T07:33:57.567289+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:33:57.567289+0000","last_undegraded":"2024-02-05T07:33:57.567289+0000","last_fullsized":"2024-02-05T07:33:57.567289+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:46.056702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:28:46.056702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.641991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":618,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.680063+0000","last_change":"2024-02-05T07:33:30.216853+0000","last_active":"2024-02-05T07:34:02.680063+0000","last_peered":"2024-02-05T07:34:02.680063+0000","last_clean":"2024-02-05T07:34:02.680063+0000","last_became_active":"2024-02-05T07:30:40.500403+0000","last_became_peered":"2024-02-05T07:30:40.500403+0000","last_unstale":"2024-02-05T07:34:02.680063+0000","last_undegraded":"2024-02-05T07:34:02.680063+0000","last_fullsized":"2024-02-05T07:34:02.680063+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:30.216745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.216745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:48.415365+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":148,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567349+0000","last_change":"2024-02-05T07:30:45.926725+0000","last_active":"2024-02-05T07:33:57.567349+0000","last_peered":"2024-02-05T07:33:57.567349+0000","last_clean":"2024-02-05T07:33:57.567349+0000","last_became_active":"2024-02-05T07:30:40.489230+0000","last_became_peered":"2024-02-05T07:30:40.489230+0000","last_unstale":"2024-02-05T07:33:57.567349+0000","last_undegraded":"2024-02-05T07:33:57.567349+0000","last_fullsized":"2024-02-05T07:33:57.567349+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:45.926395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:30:45.926395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:01.924775+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1a","version":"86'128","reported_seq":476,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.557325+0000","last_change":"2024-02-05T07:32:25.652816+0000","last_active":"2024-02-05T07:33:57.557325+0000","last_peered":"2024-02-05T07:33:57.557325+0000","last_clean":"2024-02-05T07:33:57.557325+0000","last_became_active":"2024-02-05T07:31:41.820206+0000","last_became_peered":"2024-02-05T07:31:41.820206+0000","last_unstale":"2024-02-05T07:33:57.557325+0000","last_undegraded":"2024-02-05T07:33:57.557325+0000","last_fullsized":"2024-02-05T07:33:57.557325+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:25.652696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:25.652696+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930859+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"326'19","reported_seq":604,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.681252+0000","last_change":"2024-02-05T07:32:43.787447+0000","last_active":"2024-02-05T07:34:02.681252+0000","last_peered":"2024-02-05T07:34:02.681252+0000","last_clean":"2024-02-05T07:34:02.681252+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:34:02.681252+0000","last_undegraded":"2024-02-05T07:34:02.681252+0000","last_fullsized":"2024-02-05T07:34:02.681252+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:32:43.787327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:43.787327+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:13.482809+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":349,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.568612+0000","last_change":"2024-02-05T07:29:38.000417+0000","last_active":"2024-02-05T07:33:57.568612+0000","last_peered":"2024-02-05T07:33:57.568612+0000","last_clean":"2024-02-05T07:33:57.568612+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:33:57.568612+0000","last_undegraded":"2024-02-05T07:33:57.568612+0000","last_fullsized":"2024-02-05T07:33:57.568612+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:38.000345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:38.000345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":141,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.570442+0000","last_change":"2024-02-05T07:30:40.480962+0000","last_active":"2024-02-05T07:33:57.570442+0000","last_peered":"2024-02-05T07:33:57.570442+0000","last_clean":"2024-02-05T07:33:57.570442+0000","last_became_active":"2024-02-05T07:30:40.479162+0000","last_became_peered":"2024-02-05T07:30:40.479162+0000","last_unstale":"2024-02-05T07:33:57.570442+0000","last_undegraded":"2024-02-05T07:33:57.570442+0000","last_fullsized":"2024-02-05T07:33:57.570442+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:28.153881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:28.153881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:01.924709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":771,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.560239+0000","last_change":"2024-02-05T07:29:55.800941+0000","last_active":"2024-02-05T07:33:57.560239+0000","last_peered":"2024-02-05T07:33:57.560239+0000","last_clean":"2024-02-05T07:33:57.560239+0000","last_became_active":"2024-02-05T07:27:03.360468+0000","last_became_peered":"2024-02-05T07:27:03.360468+0000","last_unstale":"2024-02-05T07:33:57.560239+0000","last_undegraded":"2024-02-05T07:33:57.560239+0000","last_fullsized":"2024-02-05T07:33:57.560239+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:55.800824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.800824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:05.635268+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1b","version":"86'128","reported_seq":551,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.557512+0000","last_change":"2024-02-05T07:32:26.685749+0000","last_active":"2024-02-05T07:33:57.557512+0000","last_peered":"2024-02-05T07:33:57.557512+0000","last_clean":"2024-02-05T07:33:57.557512+0000","last_became_active":"2024-02-05T07:31:41.823891+0000","last_became_peered":"2024-02-05T07:31:41.823891+0000","last_unstale":"2024-02-05T07:33:57.557512+0000","last_undegraded":"2024-02-05T07:33:57.557512+0000","last_fullsized":"2024-02-05T07:33:57.557512+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:26.685655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:26.685655+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930893+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"326'20","reported_seq":516,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.566337+0000","last_change":"2024-02-05T07:30:40.534624+0000","last_active":"2024-02-05T07:33:57.566337+0000","last_peered":"2024-02-05T07:33:57.566337+0000","last_clean":"2024-02-05T07:33:57.566337+0000","last_became_active":"2024-02-05T07:30:40.495954+0000","last_became_peered":"2024-02-05T07:30:40.495954+0000","last_unstale":"2024-02-05T07:33:57.566337+0000","last_undegraded":"2024-02-05T07:33:57.566337+0000","last_fullsized":"2024-02-05T07:33:57.566337+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:29:40.154854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.154854+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":813,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.560365+0000","last_change":"2024-02-05T07:30:40.526955+0000","last_active":"2024-02-05T07:33:57.560365+0000","last_peered":"2024-02-05T07:33:57.560365+0000","last_clean":"2024-02-05T07:33:57.560365+0000","last_became_active":"2024-02-05T07:30:40.495135+0000","last_became_peered":"2024-02-05T07:30:40.495135+0000","last_unstale":"2024-02-05T07:33:57.560365+0000","last_undegraded":"2024-02-05T07:33:57.560365+0000","last_fullsized":"2024-02-05T07:33:57.560365+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:52.715259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:52.715259+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557133+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"365'11","reported_seq":783,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.681384+0000","last_change":"2024-02-05T07:33:31.254022+0000","last_active":"2024-02-05T07:34:02.681384+0000","last_peered":"2024-02-05T07:34:02.681384+0000","last_clean":"2024-02-05T07:34:02.681384+0000","last_became_active":"2024-02-05T07:30:40.497371+0000","last_became_peered":"2024-02-05T07:30:40.497371+0000","last_unstale":"2024-02-05T07:34:02.681384+0000","last_undegraded":"2024-02-05T07:34:02.681384+0000","last_fullsized":"2024-02-05T07:34:02.681384+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:33:31.253933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:31.253933+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:42.464942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":258,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567150+0000","last_change":"2024-02-05T07:30:41.950393+0000","last_active":"2024-02-05T07:33:57.567150+0000","last_peered":"2024-02-05T07:33:57.567150+0000","last_clean":"2024-02-05T07:33:57.567150+0000","last_became_active":"2024-02-05T07:30:40.486920+0000","last_became_peered":"2024-02-05T07:30:40.486920+0000","last_unstale":"2024-02-05T07:33:57.567150+0000","last_undegraded":"2024-02-05T07:33:57.567150+0000","last_fullsized":"2024-02-05T07:33:57.567150+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:30:41.950253+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:30:41.950253+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:01.924646+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"86'128","reported_seq":798,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.570571+0000","last_change":"2024-02-05T07:32:42.278242+0000","last_active":"2024-02-05T07:33:57.570571+0000","last_peered":"2024-02-05T07:33:57.570571+0000","last_clean":"2024-02-05T07:33:57.570571+0000","last_became_active":"2024-02-05T07:32:01.562409+0000","last_became_peered":"2024-02-05T07:32:01.562409+0000","last_unstale":"2024-02-05T07:33:57.570571+0000","last_undegraded":"2024-02-05T07:33:57.570571+0000","last_fullsized":"2024-02-05T07:33:57.570571+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:42.278112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:42.278112+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:03.855341+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"326'23","reported_seq":375,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.563718+0000","last_change":"2024-02-05T07:28:47.062765+0000","last_active":"2024-02-05T07:33:57.563718+0000","last_peered":"2024-02-05T07:33:57.563718+0000","last_clean":"2024-02-05T07:33:57.563718+0000","last_became_active":"2024-02-05T07:27:16.747998+0000","last_became_peered":"2024-02-05T07:27:16.747998+0000","last_unstale":"2024-02-05T07:33:57.563718+0000","last_undegraded":"2024-02-05T07:33:57.563718+0000","last_fullsized":"2024-02-05T07:33:57.563718+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'17","last_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.062618+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642024+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":451,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559883+0000","last_change":"2024-02-05T07:30:39.874170+0000","last_active":"2024-02-05T07:33:57.559883+0000","last_peered":"2024-02-05T07:33:57.559883+0000","last_clean":"2024-02-05T07:33:57.559883+0000","last_became_active":"2024-02-05T07:25:30.261037+0000","last_became_peered":"2024-02-05T07:25:30.261037+0000","last_unstale":"2024-02-05T07:33:57.559883+0000","last_undegraded":"2024-02-05T07:33:57.559883+0000","last_fullsized":"2024-02-05T07:33:57.559883+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.874081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":775,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.562237+0000","last_change":"2024-02-05T07:30:26.669849+0000","last_active":"2024-02-05T07:33:57.562237+0000","last_peered":"2024-02-05T07:33:57.562237+0000","last_clean":"2024-02-05T07:33:57.562237+0000","last_became_active":"2024-02-05T07:27:16.706777+0000","last_became_peered":"2024-02-05T07:27:16.706777+0000","last_unstale":"2024-02-05T07:33:57.562237+0000","last_undegraded":"2024-02-05T07:33:57.562237+0000","last_fullsized":"2024-02-05T07:33:57.562237+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:26.669576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:26.669576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930729+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":353,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567439+0000","last_change":"2024-02-05T07:28:53.138071+0000","last_active":"2024-02-05T07:33:57.567439+0000","last_peered":"2024-02-05T07:33:57.567439+0000","last_clean":"2024-02-05T07:33:57.567439+0000","last_became_active":"2024-02-05T07:28:26.603067+0000","last_became_peered":"2024-02-05T07:28:26.603067+0000","last_unstale":"2024-02-05T07:33:57.567439+0000","last_undegraded":"2024-02-05T07:33:57.567439+0000","last_fullsized":"2024-02-05T07:33:57.567439+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:53.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:53.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.641894+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.19","version":"86'128","reported_seq":873,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.563354+0000","last_change":"2024-02-05T07:32:44.746470+0000","last_active":"2024-02-05T07:33:57.563354+0000","last_peered":"2024-02-05T07:33:57.563354+0000","last_clean":"2024-02-05T07:33:57.563354+0000","last_became_active":"2024-02-05T07:31:41.858087+0000","last_became_peered":"2024-02-05T07:31:41.858087+0000","last_unstale":"2024-02-05T07:33:57.563354+0000","last_undegraded":"2024-02-05T07:33:57.563354+0000","last_fullsized":"2024-02-05T07:33:57.563354+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:44.746349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:44.746349+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:06.631214+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"326'27","reported_seq":859,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567304+0000","last_change":"2024-02-05T07:33:33.995153+0000","last_active":"2024-02-05T07:33:57.567304+0000","last_peered":"2024-02-05T07:33:57.567304+0000","last_clean":"2024-02-05T07:33:57.567304+0000","last_became_active":"2024-02-05T07:27:16.736296+0000","last_became_peered":"2024-02-05T07:27:16.736296+0000","last_unstale":"2024-02-05T07:33:57.567304+0000","last_undegraded":"2024-02-05T07:33:57.567304+0000","last_fullsized":"2024-02-05T07:33:57.567304+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:33:33.995051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.995051+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:40.380483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":519,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559274+0000","last_change":"2024-02-05T07:30:40.503947+0000","last_active":"2024-02-05T07:33:57.559274+0000","last_peered":"2024-02-05T07:33:57.559274+0000","last_clean":"2024-02-05T07:33:57.559274+0000","last_became_active":"2024-02-05T07:30:40.503627+0000","last_became_peered":"2024-02-05T07:30:40.503627+0000","last_unstale":"2024-02-05T07:33:57.559274+0000","last_undegraded":"2024-02-05T07:33:57.559274+0000","last_fullsized":"2024-02-05T07:33:57.559274+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:34.663987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:34.663987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":761,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.560088+0000","last_change":"2024-02-05T07:30:40.884273+0000","last_active":"2024-02-05T07:33:57.560088+0000","last_peered":"2024-02-05T07:33:57.560088+0000","last_clean":"2024-02-05T07:33:57.560088+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:33:57.560088+0000","last_undegraded":"2024-02-05T07:33:57.560088+0000","last_fullsized":"2024-02-05T07:33:57.560088+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:40.884197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.884197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:05.635335+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":774,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559104+0000","last_change":"2024-02-05T07:29:59.002788+0000","last_active":"2024-02-05T07:33:57.559104+0000","last_peered":"2024-02-05T07:33:57.559104+0000","last_clean":"2024-02-05T07:33:57.559104+0000","last_became_active":"2024-02-05T07:28:26.590448+0000","last_became_peered":"2024-02-05T07:28:26.590448+0000","last_unstale":"2024-02-05T07:33:57.559104+0000","last_undegraded":"2024-02-05T07:33:57.559104+0000","last_fullsized":"2024-02-05T07:33:57.559104+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.002677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.002677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930705+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"86'128","reported_seq":473,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.566104+0000","last_change":"2024-02-05T07:32:43.758495+0000","last_active":"2024-02-05T07:33:57.566104+0000","last_peered":"2024-02-05T07:33:57.566104+0000","last_clean":"2024-02-05T07:33:57.566104+0000","last_became_active":"2024-02-05T07:31:41.834645+0000","last_became_peered":"2024-02-05T07:31:41.834645+0000","last_unstale":"2024-02-05T07:33:57.566104+0000","last_undegraded":"2024-02-05T07:33:57.566104+0000","last_fullsized":"2024-02-05T07:33:57.566104+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:43.758385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:43.758385+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.642485+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"325'7","reported_seq":801,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558535+0000","last_change":"2024-02-05T07:32:40.597579+0000","last_active":"2024-02-05T07:33:57.558535+0000","last_peered":"2024-02-05T07:33:57.558535+0000","last_clean":"2024-02-05T07:33:57.558535+0000","last_became_active":"2024-02-05T07:27:16.745897+0000","last_became_peered":"2024-02-05T07:27:16.745897+0000","last_unstale":"2024-02-05T07:33:57.558535+0000","last_undegraded":"2024-02-05T07:33:57.558535+0000","last_fullsized":"2024-02-05T07:33:57.558535+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'7","last_scrub_stamp":"2024-02-05T07:32:40.597479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:40.597479+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.931159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":592,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.561900+0000","last_change":"2024-02-05T07:30:39.726603+0000","last_active":"2024-02-05T07:33:57.561900+0000","last_peered":"2024-02-05T07:33:57.561900+0000","last_clean":"2024-02-05T07:33:57.561900+0000","last_became_active":"2024-02-05T07:28:26.590322+0000","last_became_peered":"2024-02-05T07:28:26.590322+0000","last_unstale":"2024-02-05T07:33:57.561900+0000","last_undegraded":"2024-02-05T07:33:57.561900+0000","last_fullsized":"2024-02-05T07:33:57.561900+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.726525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.726525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":147,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558005+0000","last_change":"2024-02-05T07:33:44.867890+0000","last_active":"2024-02-05T07:33:57.558005+0000","last_peered":"2024-02-05T07:33:57.558005+0000","last_clean":"2024-02-05T07:33:57.558005+0000","last_became_active":"2024-02-05T07:30:40.504365+0000","last_became_peered":"2024-02-05T07:30:40.504365+0000","last_unstale":"2024-02-05T07:33:57.558005+0000","last_undegraded":"2024-02-05T07:33:57.558005+0000","last_fullsized":"2024-02-05T07:33:57.558005+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.867764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.867764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:55.577507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":567,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.561676+0000","last_change":"2024-02-05T07:29:50.742042+0000","last_active":"2024-02-05T07:33:57.561676+0000","last_peered":"2024-02-05T07:33:57.561676+0000","last_clean":"2024-02-05T07:33:57.561676+0000","last_became_active":"2024-02-05T07:27:16.769581+0000","last_became_peered":"2024-02-05T07:27:16.769581+0000","last_unstale":"2024-02-05T07:33:57.561676+0000","last_undegraded":"2024-02-05T07:33:57.561676+0000","last_fullsized":"2024-02-05T07:33:57.561676+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:50.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:00.557388+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"325'19","reported_seq":825,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.557814+0000","last_change":"2024-02-05T07:30:00.001976+0000","last_active":"2024-02-05T07:33:57.557814+0000","last_peered":"2024-02-05T07:33:57.557814+0000","last_clean":"2024-02-05T07:33:57.557814+0000","last_became_active":"2024-02-05T07:27:16.707036+0000","last_became_peered":"2024-02-05T07:27:16.707036+0000","last_unstale":"2024-02-05T07:33:57.557814+0000","last_undegraded":"2024-02-05T07:33:57.557814+0000","last_fullsized":"2024-02-05T07:33:57.557814+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'19","last_scrub_stamp":"2024-02-05T07:30:00.001863+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:30:00.001863+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930606+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":595,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558510+0000","last_change":"2024-02-05T07:29:53.022701+0000","last_active":"2024-02-05T07:33:57.558510+0000","last_peered":"2024-02-05T07:33:57.558510+0000","last_clean":"2024-02-05T07:33:57.558510+0000","last_became_active":"2024-02-05T07:28:26.616107+0000","last_became_peered":"2024-02-05T07:28:26.616107+0000","last_unstale":"2024-02-05T07:33:57.558510+0000","last_undegraded":"2024-02-05T07:33:57.558510+0000","last_fullsized":"2024-02-05T07:33:57.558510+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.022606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.022606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":141,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.558700+0000","last_change":"2024-02-05T07:30:40.491944+0000","last_active":"2024-02-05T07:33:57.558700+0000","last_peered":"2024-02-05T07:33:57.558700+0000","last_clean":"2024-02-05T07:33:57.558700+0000","last_became_active":"2024-02-05T07:30:40.491630+0000","last_became_peered":"2024-02-05T07:30:40.491630+0000","last_unstale":"2024-02-05T07:33:57.558700+0000","last_undegraded":"2024-02-05T07:33:57.558700+0000","last_fullsized":"2024-02-05T07:33:57.558700+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:54.207089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:54.207089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":527,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559637+0000","last_change":"2024-02-05T07:33:47.485038+0000","last_active":"2024-02-05T07:33:57.559637+0000","last_peered":"2024-02-05T07:33:57.559637+0000","last_clean":"2024-02-05T07:33:57.559637+0000","last_became_active":"2024-02-05T07:30:40.475535+0000","last_became_peered":"2024-02-05T07:30:40.475535+0000","last_unstale":"2024-02-05T07:33:57.559637+0000","last_undegraded":"2024-02-05T07:33:57.559637+0000","last_fullsized":"2024-02-05T07:33:57.559637+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:47.484894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.484894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:55.364815+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"86'128","reported_seq":410,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-05T07:34:02.680168+0000","last_change":"2024-02-05T07:32:22.866491+0000","last_active":"2024-02-05T07:34:02.680168+0000","last_peered":"2024-02-05T07:34:02.680168+0000","last_clean":"2024-02-05T07:34:02.680168+0000","last_became_active":"2024-02-05T07:31:41.837502+0000","last_became_peered":"2024-02-05T07:31:41.837502+0000","last_unstale":"2024-02-05T07:34:02.680168+0000","last_undegraded":"2024-02-05T07:34:02.680168+0000","last_fullsized":"2024-02-05T07:34:02.680168+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:22.866405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:22.866405+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:07.049811+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"326'27","reported_seq":383,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.564154+0000","last_change":"2024-02-05T07:33:25.674216+0000","last_active":"2024-02-05T07:33:57.564154+0000","last_peered":"2024-02-05T07:33:57.564154+0000","last_clean":"2024-02-05T07:33:57.564154+0000","last_became_active":"2024-02-05T07:27:16.751179+0000","last_became_peered":"2024-02-05T07:27:16.751179+0000","last_unstale":"2024-02-05T07:33:57.564154+0000","last_undegraded":"2024-02-05T07:33:57.564154+0000","last_fullsized":"2024-02-05T07:33:57.564154+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:33:25.674080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:25.674080+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:47.794161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":354,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.567856+0000","last_change":"2024-02-05T07:32:49.743375+0000","last_active":"2024-02-05T07:33:57.567856+0000","last_peered":"2024-02-05T07:33:57.567856+0000","last_clean":"2024-02-05T07:33:57.567856+0000","last_became_active":"2024-02-05T07:28:26.595627+0000","last_became_peered":"2024-02-05T07:28:26.595627+0000","last_unstale":"2024-02-05T07:33:57.567856+0000","last_undegraded":"2024-02-05T07:33:57.567856+0000","last_fullsized":"2024-02-05T07:33:57.567856+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:49.743255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:32:49.743255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:59.867887+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":777,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.557345+0000","last_change":"2024-02-05T07:33:46.937819+0000","last_active":"2024-02-05T07:33:57.557345+0000","last_peered":"2024-02-05T07:33:57.557345+0000","last_clean":"2024-02-05T07:33:57.557345+0000","last_became_active":"2024-02-05T07:28:25.778935+0000","last_became_peered":"2024-02-05T07:28:25.778935+0000","last_unstale":"2024-02-05T07:33:57.557345+0000","last_undegraded":"2024-02-05T07:33:57.557345+0000","last_fullsized":"2024-02-05T07:33:57.557345+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:46.937738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.937738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:47.605598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":359,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.564798+0000","last_change":"2024-02-05T07:30:40.496738+0000","last_active":"2024-02-05T07:33:57.564798+0000","last_peered":"2024-02-05T07:33:57.564798+0000","last_clean":"2024-02-05T07:33:57.564798+0000","last_became_active":"2024-02-05T07:30:40.493086+0000","last_became_peered":"2024-02-05T07:30:40.493086+0000","last_unstale":"2024-02-05T07:33:57.564798+0000","last_undegraded":"2024-02-05T07:33:57.564798+0000","last_fullsized":"2024-02-05T07:33:57.564798+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:44.050019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:44.050019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:58.641824+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"86'128","reported_seq":412,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.559680+0000","last_change":"2024-02-05T07:32:44.373808+0000","last_active":"2024-02-05T07:33:57.559680+0000","last_peered":"2024-02-05T07:33:57.559680+0000","last_clean":"2024-02-05T07:33:57.559680+0000","last_became_active":"2024-02-05T07:31:41.843342+0000","last_became_peered":"2024-02-05T07:31:41.843342+0000","last_unstale":"2024-02-05T07:33:57.559680+0000","last_undegraded":"2024-02-05T07:33:57.559680+0000","last_fullsized":"2024-02-05T07:33:57.559680+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:44.373683+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_clean_scrub_stamp":"2024-02-05T07:32:44.373683+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:12.613629+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"325'15","reported_seq":790,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.557648+0000","last_change":"2024-02-05T07:29:33.839515+0000","last_active":"2024-02-05T07:33:57.557648+0000","last_peered":"2024-02-05T07:33:57.557648+0000","last_clean":"2024-02-05T07:33:57.557648+0000","last_became_active":"2024-02-05T07:28:25.777815+0000","last_became_peered":"2024-02-05T07:28:25.777815+0000","last_unstale":"2024-02-05T07:33:57.557648+0000","last_undegraded":"2024-02-05T07:33:57.557648+0000","last_fullsized":"2024-02-05T07:33:57.557648+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:29:33.839402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:33.839402+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:33:57.930503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":540,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.569339+0000","last_change":"2024-02-05T07:33:44.117187+0000","last_active":"2024-02-05T07:33:57.569339+0000","last_peered":"2024-02-05T07:33:57.569339+0000","last_clean":"2024-02-05T07:33:57.569339+0000","last_became_active":"2024-02-05T07:28:26.588496+0000","last_became_peered":"2024-02-05T07:28:26.588496+0000","last_unstale":"2024-02-05T07:33:57.569339+0000","last_undegraded":"2024-02-05T07:33:57.569339+0000","last_fullsized":"2024-02-05T07:33:57.569339+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.117071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.117071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:49.356777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":773,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.561494+0000","last_change":"2024-02-05T07:33:28.377593+0000","last_active":"2024-02-05T07:33:57.561494+0000","last_peered":"2024-02-05T07:33:57.561494+0000","last_clean":"2024-02-05T07:33:57.561494+0000","last_became_active":"2024-02-05T07:30:40.495711+0000","last_became_peered":"2024-02-05T07:30:40.495711+0000","last_unstale":"2024-02-05T07:33:57.561494+0000","last_undegraded":"2024-02-05T07:33:57.561494+0000","last_fullsized":"2024-02-05T07:33:57.561494+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.377472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.377472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:55.011291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":768,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-05T07:33:57.557580+0000","last_change":"2024-02-05T07:32:43.608109+0000","last_active":"2024-02-05T07:33:57.557580+0000","last_peered":"2024-02-05T07:33:57.557580+0000","last_clean":"2024-02-05T07:33:57.557580+0000","last_became_active":"2024-02-05T07:30:40.490288+0000","last_became_peered":"2024-02-05T07:30:40.490288+0000","last_unstale":"2024-02-05T07:33:57.557580+0000","last_undegraded":"2024-02-05T07:33:57.557580+0000","last_fullsized":"2024-02-05T07:33:57.557580+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:43.608002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:32:43.608002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2150760448,"num_objects":32818,"num_object_clones":0,"num_object_copies":98454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32818,"num_write_kb":2100352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6348210176,"data_stored":6348210176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32818,"ondisk_log_size":32818,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":207,"num_read_kb":202,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3968,"ondisk_log_size":3968,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1107,"num_read_kb":864,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":6480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":11876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":738,"ondisk_log_size":738,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":257698037947,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":301,"seq":1292785156165,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1312356,"kb_used_data":981404,"kb_used_omap":5,"kb_used_meta":330938,"kb_avail":92432796,"statfs":{"total":95995035648,"available":94651183104,"internally_reserved":0,"allocated":1004957696,"data_stored":1004577057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":338881029},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:33:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.93999999999999995,"15min":1.798},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":1.714,"5min":14.218,"15min":14.218},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":1.0389999999999999,"15min":1.802},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.73599999999999999,"5min":14.175000000000001,"15min":14.175000000000001},"last":0.66800000000000004}]},{"osd":1,"last update":"Mon Feb 5 07:33:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":2.29,"15min":1.8979999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":9.4250000000000007,"5min":55.923000000000002,"15min":55.923000000000002},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.8380000000000001,"5min":2.8559999999999999,"15min":2.6899999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":10.503,"5min":61.103000000000002,"15min":61.103000000000002},"last":1.0169999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:33:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":1.056,"15min":2.3410000000000002},"min":{"1min":0.29799999999999999,"5min":0.249,"15min":0.249},"max":{"1min":2.1549999999999998,"5min":34.689999999999998,"15min":34.689999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.79600000000000004,"5min":1.361,"15min":2.8530000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":3.948,"5min":34.646999999999998,"15min":34.646999999999998},"last":0.82799999999999996}]},{"osd":3,"last update":"Mon Feb 5 07:33:35 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":3.3479999999999999,"15min":2.5750000000000002},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":5.8970000000000002,"5min":55.978999999999999,"15min":55.978999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.4039999999999999,"5min":3.4039999999999999,"15min":2.8149999999999999},"min":{"1min":0.38500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":9.2780000000000005,"5min":55.631,"15min":55.631},"last":0.64600000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:33:35 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.48899999999999999,"15min":0.67400000000000004},"min":{"1min":0.25800000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59599999999999997,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.47099999999999997,"15min":0.59499999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.92700000000000005,"5min":1.869,"15min":1.869},"last":0.874}]},{"osd":5,"last update":"Mon Feb 5 07:33:35 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.48899999999999999,"15min":0.57199999999999995},"min":{"1min":0.28399999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.871,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.5,"15min":0.56399999999999995},"min":{"1min":0.311,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68100000000000005,"5min":1.526,"15min":1.526},"last":0.51400000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:33:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":1.8440000000000001,"15min":4.3440000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.69999999999999996,"5min":84.158000000000001,"15min":84.158000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.499,"5min":1.9079999999999999,"15min":4.484},"min":{"1min":0.36599999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.69499999999999995,"5min":85.018000000000001,"15min":85.018000000000001},"last":0.39000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575065,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1319696,"kb_used_data":652680,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":92425456,"statfs":{"total":95995035648,"available":94643666944,"internally_reserved":0,"allocated":668344320,"data_stored":667913198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.4450000000000001,"15min":2.1419999999999999},"min":{"1min":0.248,"5min":0.20899999999999999,"15min":0.16800000000000001},"max":{"1min":3.3969999999999998,"5min":15.09,"15min":45.393000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.587,"5min":2.028,"15min":2.3109999999999999},"min":{"1min":0.28999999999999998,"5min":0.23799999999999999,"15min":0.20899999999999999},"max":{"1min":8.2609999999999992,"5min":14.313000000000001,"15min":45.031999999999996},"last":0.29299999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:33:09 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":2.0139999999999998,"15min":1.7849999999999999},"min":{"1min":0.23799999999999999,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":3.5840000000000001,"5min":15.137,"15min":39.259999999999998},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.8480000000000001,"15min":1.6990000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.7610000000000001,"5min":14.906000000000001,"15min":40.972000000000001},"last":0.23699999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:33:24 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.948,"15min":2.105},"min":{"1min":0.249,"5min":0.22900000000000001,"15min":0.20999999999999999},"max":{"1min":8.3089999999999993,"5min":46.777000000000001,"15min":46.777000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":2.1269999999999998,"5min":2.6240000000000001,"15min":2.1000000000000001},"min":{"1min":0.32900000000000001,"5min":0.24299999999999999,"15min":0.24199999999999999},"max":{"1min":6.4290000000000003,"5min":50.164999999999999,"15min":50.164999999999999},"last":0.307}]},{"osd":4,"last update":"Mon Feb 5 07:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.29,"15min":1.742},"min":{"1min":0.41299999999999998,"5min":0.32600000000000001,"15min":0.312},"max":{"1min":2.02,"5min":8.9939999999999998,"15min":61.722000000000001},"last":89.409000000000006},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.119,"15min":1.7},"min":{"1min":0.38900000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.4780000000000002,"5min":5.5590000000000002,"15min":61.786000000000001},"last":89.915000000000006}]},{"osd":5,"last update":"Mon Feb 5 07:33:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.9299999999999999,"15min":1.974},"min":{"1min":0.44700000000000001,"5min":0.33900000000000002,"15min":0.309},"max":{"1min":3.4350000000000001,"5min":46.398000000000003,"15min":106.672},"last":0.66300000000000003},{"interface":"front","average":{"1min":2.028,"5min":2.0379999999999998,"15min":2.0299999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":7.1210000000000004,"5min":20.085999999999999,"15min":107.348},"last":0.626}]},{"osd":6,"last update":"Mon Feb 5 07:33:30 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.1699999999999999,"15min":1.6259999999999999},"min":{"1min":0.33200000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.508,"5min":5.5990000000000002,"15min":5.5990000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.4299999999999999,"15min":1.913},"min":{"1min":0.46999999999999997,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":3.403,"5min":8.923,"15min":8.923},"last":0.51300000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:33:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1920000000000002,"5min":2.633,"15min":1.9119999999999999},"min":{"1min":0.38900000000000001,"5min":0.318,"15min":0.318},"max":{"1min":7.0899999999999999,"5min":46.351999999999997,"15min":66.703999999999994},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.246,"5min":1.6040000000000001,"15min":1.5529999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.33500000000000002},"max":{"1min":3.5339999999999998,"5min":11.263999999999999,"15min":61.594999999999999},"last":0.56899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738590,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1946896,"kb_used_data":1261832,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":91798256,"statfs":{"total":95995035648,"available":94001414144,"internally_reserved":0,"allocated":1292115968,"data_stored":1291684493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:33:41 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":2.4860000000000002,"15min":2.0619999999999998},"min":{"1min":0.21099999999999999,"5min":0.192,"15min":0.16500000000000001},"max":{"1min":8.5939999999999994,"5min":32.546999999999997,"15min":45.372},"last":0.317},{"interface":"front","average":{"1min":0.55000000000000004,"5min":2.3809999999999998,"15min":2.105},"min":{"1min":0.27100000000000002,"5min":0.217,"15min":0.186},"max":{"1min":2.0019999999999998,"5min":29.675999999999998,"15min":45.231999999999999},"last":0.35499999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.2609999999999999,"15min":1.7430000000000001},"min":{"1min":0.23899999999999999,"5min":0.191,"15min":0.191},"max":{"1min":5.9180000000000001,"5min":6.5140000000000002,"15min":65.903999999999996},"last":40.548000000000002},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.175,"15min":1.7090000000000001},"min":{"1min":0.32400000000000001,"5min":0.224,"15min":0.182},"max":{"1min":4.7389999999999999,"5min":4.8220000000000001,"15min":65.694000000000003},"last":40.643000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:33:20 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":2.1309999999999998,"15min":2.1120000000000001},"min":{"1min":0.30099999999999999,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":4.7679999999999998,"5min":29.629000000000001,"15min":65.155000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.444,"5min":2.4460000000000002,"15min":2.2989999999999999},"min":{"1min":0.27300000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.9000000000000004,"5min":29.463999999999999,"15min":65.114000000000004},"last":0.50800000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":1.2430000000000001,"15min":1.901},"min":{"1min":0.29099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.8130000000000002,"5min":42.901000000000003,"15min":65.188000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.76000000000000001,"5min":1.4079999999999999,"15min":1.968},"min":{"1min":0.36599999999999999,"5min":0.27100000000000002,"15min":0.23699999999999999},"max":{"1min":3.5059999999999998,"5min":42.960999999999999,"15min":64.873000000000005},"last":0.63900000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.96399999999999997,"15min":1.6919999999999999},"min":{"1min":0.41699999999999998,"5min":0.30599999999999999,"15min":0.27000000000000002},"max":{"1min":2.1259999999999999,"5min":5.4669999999999996,"15min":95.772999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.0089999999999999,"15min":1.7430000000000001},"min":{"1min":0.34699999999999998,"5min":0.29299999999999998,"15min":0.27600000000000002},"max":{"1min":3.5950000000000002,"5min":7.1929999999999996,"15min":96.028000000000006},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:33:35 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.92400000000000004,"15min":1.248},"min":{"1min":0.34399999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.9249999999999998,"5min":4.6929999999999996,"15min":4.6929999999999996},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.84199999999999997,"5min":1.0820000000000001,"15min":1.4930000000000001},"min":{"1min":0.41299999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.5590000000000002,"5min":7.1319999999999997,"15min":7.1319999999999997},"last":0.53600000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:33:55 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.6659999999999999,"15min":1.8160000000000001},"min":{"1min":0.42199999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.891,"5min":43.012,"15min":65.263000000000005},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.83799999999999997,"5min":1.8220000000000001,"15min":1.831},"min":{"1min":0.40799999999999997,"5min":0.34000000000000002,"15min":0.309},"max":{"1min":3.6280000000000001,"5min":43.512,"15min":64.906000000000006},"last":0.49399999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411540,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1645808,"kb_used_data":956580,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":92099344,"statfs":{"total":95995035648,"available":94309728256,"internally_reserved":0,"allocated":979537920,"data_stored":979143014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:33:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.2749999999999999,"15min":2.0179999999999998},"min":{"1min":0.32500000000000001,"5min":0.19600000000000001,"15min":0.182},"max":{"1min":7.2519999999999998,"5min":12.759,"15min":92.638000000000005},"last":0.748},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.3160000000000001,"15min":2.1459999999999999},"min":{"1min":0.23999999999999999,"5min":0.22,"15min":0.20699999999999999},"max":{"1min":5.1870000000000003,"5min":12.647,"15min":93.481999999999999},"last":5.6600000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:33:06 2024","interfaces":[{"interface":"back","average":{"1min":1.851,"5min":2.879,"15min":2.2970000000000002},"min":{"1min":0.307,"5min":0.215,"15min":0.19400000000000001},"max":{"1min":6.7140000000000004,"5min":63.823,"15min":93.504000000000005},"last":3.4009999999999998},{"interface":"front","average":{"1min":2.452,"5min":3.3500000000000001,"15min":2.5030000000000001},"min":{"1min":0.35099999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":7.1859999999999999,"5min":64.771000000000001,"15min":93.004000000000005},"last":1.992}]},{"osd":3,"last update":"Mon Feb 5 07:33:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":2.7610000000000001,"15min":2.0609999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.214},"max":{"1min":6.4450000000000003,"5min":63.784999999999997,"15min":63.784999999999997},"last":3.8239999999999998},{"interface":"front","average":{"1min":1.643,"5min":2.9540000000000002,"15min":2.0459999999999998},"min":{"1min":0.32400000000000001,"5min":0.22600000000000001,"15min":0.193},"max":{"1min":7.226,"5min":63.521000000000001,"15min":63.521000000000001},"last":0.78000000000000003}]},{"osd":4,"last update":"Mon Feb 5 07:33:49 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.93500000000000005,"15min":2.016},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.24099999999999999},"max":{"1min":2.5049999999999999,"5min":5.4829999999999997,"15min":168.11699999999999},"last":5.7050000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":0.90700000000000003,"15min":2.0009999999999999},"min":{"1min":0.36099999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.6950000000000003,"5min":5.6680000000000001,"15min":168.589},"last":1.9570000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:33:06 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":0.97799999999999998,"15min":1.2250000000000001},"min":{"1min":0.47199999999999998,"5min":0.315,"15min":0.315},"max":{"1min":4.742,"5min":5.6929999999999996,"15min":39.923000000000002},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.276,"5min":0.94899999999999995,"15min":1.173},"min":{"1min":0.36699999999999999,"5min":0.33000000000000002,"15min":0.26400000000000001},"max":{"1min":2.7490000000000001,"5min":4.0709999999999997,"15min":38.920999999999999},"last":3.7189999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:33:33 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":0.93000000000000005,"15min":1.1599999999999999},"min":{"1min":0.45600000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.7919999999999998,"5min":4.109,"15min":4.109},"last":3.762},{"interface":"front","average":{"1min":1.075,"5min":0.97799999999999998,"15min":1.3340000000000001},"min":{"1min":0.441,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.7779999999999996,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.89100000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:33:06 2024","interfaces":[{"interface":"back","average":{"1min":1.726,"5min":1.851,"15min":1.3999999999999999},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":11.311999999999999,"5min":37.573999999999998,"15min":37.573999999999998},"last":5.7850000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.581,"15min":1.2889999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.819,"5min":37.432000000000002,"15min":37.432000000000002},"last":3.7949999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149903,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1648396,"kb_used_data":905028,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92096756,"statfs":{"total":95995035648,"available":94307078144,"internally_reserved":0,"allocated":926748672,"data_stored":926321086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":761199047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:33:41 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.1399999999999999,"15min":2.206},"min":{"1min":0.24399999999999999,"5min":0.192,"15min":0.192},"max":{"1min":3.2469999999999999,"5min":6.1349999999999998,"15min":74.114999999999995},"last":0.19900000000000001},{"interface":"front","average":{"1min":1.014,"5min":2.2320000000000002,"15min":2.3359999999999999},"min":{"1min":0.28000000000000003,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.6699999999999999,"5min":99.989999999999995,"15min":99.989999999999995},"last":0.22700000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:33:41 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":3.423,"15min":2.569},"min":{"1min":0.28899999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.4489999999999998,"5min":102.20399999999999,"15min":102.20399999999999},"last":0.21299999999999999},{"interface":"front","average":{"1min":1.0089999999999999,"5min":3.2610000000000001,"15min":2.8290000000000002},"min":{"1min":0.318,"5min":0.255,"15min":0.185},"max":{"1min":3.399,"5min":102.286,"15min":102.286},"last":0.27300000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:33:41 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":2.2370000000000001,"15min":2.4910000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.3759999999999999,"5min":99.847999999999999,"15min":99.847999999999999},"last":59.085000000000001},{"interface":"front","average":{"1min":0.97499999999999998,"5min":2.194,"15min":2.036},"min":{"1min":0.29299999999999998,"5min":0.26800000000000002,"15min":0.22500000000000001},"max":{"1min":4.6950000000000003,"5min":99.950999999999993,"15min":99.950999999999993},"last":59.887999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:33:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":2.0859999999999999,"15min":1.5620000000000001},"min":{"1min":0.39600000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.589,"5min":100.024,"15min":100.024},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.877,"5min":2.4279999999999999,"15min":1.6419999999999999},"min":{"1min":0.48299999999999998,"5min":0.30399999999999999,"15min":0.28599999999999998},"max":{"1min":3.4249999999999998,"5min":99.859999999999999,"15min":99.859999999999999},"last":0.32400000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":2.2749999999999999,"15min":1.873},"min":{"1min":0.46000000000000002,"5min":0.32700000000000001,"15min":0.28499999999999998},"max":{"1min":3.3159999999999998,"5min":99.911000000000001,"15min":99.911000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":1.1919999999999999,"5min":2.2639999999999998,"15min":1.6040000000000001},"min":{"1min":0.45200000000000001,"5min":0.33000000000000002,"15min":0.27400000000000002},"max":{"1min":4.6369999999999996,"5min":100.053,"15min":100.053},"last":0.35099999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:33:41 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":2.169,"15min":1.343},"min":{"1min":0.44700000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.2879999999999998,"5min":99.807000000000002,"15min":99.807000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":2.1859999999999999,"15min":1.3959999999999999},"min":{"1min":0.44700000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.6050000000000004,"5min":101.489,"15min":101.489},"last":0.38200000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:33:58 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":3.177,"15min":2.3039999999999998},"min":{"1min":0.45200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.4620000000000002,"5min":99.893000000000001,"15min":99.893000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.81699999999999995,"5min":2.8660000000000001,"15min":2.5640000000000001},"min":{"1min":0.51600000000000001,"5min":0.375,"15min":0.375},"max":{"1min":2.206,"5min":99.879000000000005,"15min":99.879000000000005},"last":0.51600000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953674,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1864108,"kb_used_data":1319268,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":91881044,"statfs":{"total":95995035648,"available":94086189056,"internally_reserved":0,"allocated":1350930432,"data_stored":1350534198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.77600000000000002,"15min":2.0979999999999999},"min":{"1min":0.35299999999999998,"5min":0.219,"15min":0.20300000000000001},"max":{"1min":1.363,"5min":13.103999999999999,"15min":101.212},"last":2.3460000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.91500000000000004,"15min":2.194},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23300000000000001},"max":{"1min":2.492,"5min":13.169,"15min":102.07599999999999},"last":0.47499999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:33:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":2.4039999999999999,"15min":2.3109999999999999},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":6.9279999999999999,"5min":45.655999999999999,"15min":69.102000000000004},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.208,"5min":2.6179999999999999,"15min":2.5430000000000001},"min":{"1min":0.35299999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.79,"5min":46.531999999999996,"15min":68.108999999999995},"last":0.66200000000000003}]},{"osd":2,"last update":"Mon Feb 5 07:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.755,"15min":1.347},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.0369999999999999,"5min":4.266,"15min":67.144999999999996},"last":2.0859999999999999},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.0700000000000001,"15min":1.413},"min":{"1min":0.39000000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.9429999999999996,"5min":9.8209999999999997,"15min":66.959999999999994},"last":0.68300000000000005}]},{"osd":3,"last update":"Mon Feb 5 07:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":2.8940000000000001,"15min":1.8640000000000001},"min":{"1min":0.29499999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.8599999999999999,"5min":46.549999999999997,"15min":66.019999999999996},"last":1.7549999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":2.9750000000000001,"15min":2.085},"min":{"1min":0.45900000000000002,"5min":0.33300000000000002,"15min":0.26400000000000001},"max":{"1min":7.9939999999999998,"5min":46.003999999999998,"15min":101.001},"last":2.5840000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.63300000000000001,"15min":0.58699999999999997},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.73999999999999999,"5min":10.917999999999999,"15min":10.917999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.67700000000000005,"15min":0.60899999999999999},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.24199999999999999},"max":{"1min":0.68600000000000005,"5min":15.105,"15min":15.105},"last":0.77300000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.65800000000000003,"15min":1.131},"min":{"1min":0.316,"5min":0.245,"15min":0.245},"max":{"1min":0.73799999999999999,"5min":14.606,"15min":14.606},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.72899999999999998,"15min":1.371},"min":{"1min":0.26600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.76400000000000001,"5min":13.114000000000001,"15min":13.114000000000001},"last":0.63300000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:33:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57099999999999995,"15min":0.72199999999999998},"min":{"1min":0.32400000000000001,"5min":0.30599999999999999,"15min":0.25700000000000001},"max":{"1min":0.88300000000000001,"5min":2.8260000000000001,"15min":15.073},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57699999999999996,"15min":0.69299999999999995},"min":{"1min":0.34300000000000003,"5min":0.316,"15min":0.248},"max":{"1min":0.76800000000000002,"5min":3.0670000000000002,"15min":14.581},"last":0.71099999999999997}]}]},{"osd":5,"up_from":0,"seq":176093659333,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1301413888,"data_stored":1301413888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":682164224,"data_stored":682164224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1026752512,"data_stored":1026752512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":931856384,"data_stored":931856384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1389297664,"data_stored":1389297664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1016725504,"data_stored":1016725504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:34:07.038 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-02-05T07:34:07.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-02-05T07:34:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:07.039 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:07.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:07.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:07.195 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:07.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:07.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:07.329 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:07.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:07.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:07.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:07.461 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:07.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:07.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:07.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:07.606 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:07.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:07.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:07.747 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:07.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:07.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:07.765 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:34:07.764858+0000 min lat: 0.00131497 max lat: 0.307728 avg lat: 0.00759776 2024-02-05T07:34:07.765 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:34:07.765 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 20 16 42107 42091 131.513 97.5 0.00675275 0.00759776 2024-02-05T07:34:07.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:07 smithi090 ceph-mon[33213]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:34:07.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:07 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:34:07.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:07 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:34:07.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:07 smithi090 ceph-mon[33213]: osdmap e430: 8 total, 8 up, 6 in 2024-02-05T07:34:07.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:07.891 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:07.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:07.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:07.971 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:34:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:34:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:42447 10.1 10:83b61621:::benchmark_data_smithi007_157098_object42446:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-02-05T07:34:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:07.965329+0000", 2024-02-05T07:34:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0039303999999999997, 2024-02-05T07:34:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039554389999999998, 2024-02-05T07:34:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:34:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:34:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:34:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42447 2024-02-05T07:34:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965329+0000", 2024-02-05T07:34:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965329+0000", 2024-02-05T07:34:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965330+0000", 2024-02-05T07:34:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0100000000000005e-07 2024-02-05T07:34:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965388+0000", 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8075000000000002e-05 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965397+0000", 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8850000000000005e-06 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965404+0000", 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.345e-06 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965470+0000", 2024-02-05T07:34:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5667999999999995e-05 2024-02-05T07:34:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965587+0000", 2024-02-05T07:34:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116902 2024-02-05T07:34:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965716+0000", 2024-02-05T07:34:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012917899999999999 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:42449 10.1 10:899f5672:::benchmark_data_smithi007_157098_object42448:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:07.965816+0000", 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0034434800000000001, 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035193540000000001, 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42449 2024-02-05T07:34:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965816+0000", 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965816+0000", 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965817+0000", 2024-02-05T07:34:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6899999999999996e-07 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965849+0000", 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2155e-05 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965858+0000", 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.208e-06 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965865+0000", 2024-02-05T07:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6429999999999995e-06 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.968139+0000", 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022733850000000002 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.968264+0000", 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012521200000000001 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.968405+0000", 2024-02-05T07:34:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014108400000000001 2024-02-05T07:34:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25454.0:42443 10.9 e430/423 10:9306e6db:::benchmark_data_smithi007_157098_object42442:head v 430'2580, mlcod=430'2578)", 2024-02-05T07:34:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:07.965803+0000", 2024-02-05T07:34:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0034564800000000001, 2024-02-05T07:34:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003575019, 2024-02-05T07:34:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:34:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965803+0000", 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965803+0000", 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965804+0000", 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999997e-07 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965875+0000", 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1253000000000004e-05 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965886+0000", 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0113e-05 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965900+0000", 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4282e-05 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.968127+0000", 2024-02-05T07:34:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022273879999999999 2024-02-05T07:34:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.968148+0000", 2024-02-05T07:34:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1092000000000001e-05 2024-02-05T07:34:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:42450 10.6 10:6b06c2b8:::benchmark_data_smithi007_157098_object42449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-02-05T07:34:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:07.966986+0000", 2024-02-05T07:34:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.002274, 2024-02-05T07:34:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024285280000000001, 2024-02-05T07:34:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:34:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:34:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:34:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42450 2024-02-05T07:34:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.966986+0000", 2024-02-05T07:34:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.966986+0000", 2024-02-05T07:34:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.966987+0000", 2024-02-05T07:34:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2270000000000001e-06 2024-02-05T07:34:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.967048+0000", 2024-02-05T07:34:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0650999999999999e-05 2024-02-05T07:34:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.967060+0000", 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2573e-05 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.967069+0000", 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4109999999999996e-06 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.967137+0000", 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8145000000000006e-05 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.967265+0000", 2024-02-05T07:34:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012824700000000001 2024-02-05T07:34:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.967383+0000", 2024-02-05T07:34:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011751 2024-02-05T07:34:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.968912+0000", 2024-02-05T07:34:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001529325 2024-02-05T07:34:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:42435 10.4 10:2f881bc2:::benchmark_data_smithi007_157098_object42434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-02-05T07:34:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:07.957780+0000", 2024-02-05T07:34:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.011480262999999999, 2024-02-05T07:34:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011693119, 2024-02-05T07:34:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:34:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:34:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:34:07.990 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42435 2024-02-05T07:34:07.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.990 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:07.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:07.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.957780+0000", 2024-02-05T07:34:07.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:07.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:07.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.957780+0000", 2024-02-05T07:34:07.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:07.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:07.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.957781+0000", 2024-02-05T07:34:07.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.093e-06 2024-02-05T07:34:07.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:07.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.957841+0000", 2024-02-05T07:34:07.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9948000000000003e-05 2024-02-05T07:34:07.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:07.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.957854+0000", 2024-02-05T07:34:07.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3593e-05 2024-02-05T07:34:07.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:07.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.957862+0000", 2024-02-05T07:34:07.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2899999999999997e-06 2024-02-05T07:34:07.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.957911+0000", 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9317999999999998e-05 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.958124+0000", 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021346000000000001 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3", 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.958612+0000", 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048764000000000001 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.962181+0000", 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003569329 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.964846+0000", 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026652120000000001 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:42436 10.1 10:8d2ab921:::benchmark_data_smithi007_157098_object42435:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:07.957942+0000", 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.011318083, 2024-02-05T07:34:07.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011590082999999999, 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42436 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.957942+0000", 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.957942+0000", 2024-02-05T07:34:07.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:07.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:07.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.957943+0000", 2024-02-05T07:34:07.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3399999999999998e-07 2024-02-05T07:34:07.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:07.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.958003+0000", 2024-02-05T07:34:07.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0885999999999999e-05 2024-02-05T07:34:07.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:07.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.958011+0000", 2024-02-05T07:34:07.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1910000000000003e-06 2024-02-05T07:34:07.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:07.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.958017+0000", 2024-02-05T07:34:07.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4289999999999998e-06 2024-02-05T07:34:07.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:07.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.958311+0000", 2024-02-05T07:34:07.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000294207 2024-02-05T07:34:07.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:07.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.958469+0000", 2024-02-05T07:34:07.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015778600000000001 2024-02-05T07:34:07.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:07.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.958617+0000", 2024-02-05T07:34:07.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014778999999999999 2024-02-05T07:34:07.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:07.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.960529+0000", 2024-02-05T07:34:07.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001912001 2024-02-05T07:34:07.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:07.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.963818+0000", 2024-02-05T07:34:07.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032887110000000002 2024-02-05T07:34:07.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:07.999 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:07.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:07.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:07.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:07.999 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:42438 10.4 10:2c336e89:::benchmark_data_smithi007_157098_object42437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-02-05T07:34:07.999 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:07.959239+0000", 2024-02-05T07:34:07.999 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.010021271999999999, 2024-02-05T07:34:07.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010393832, 2024-02-05T07:34:08.000 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:08.000 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:34:08.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:08.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:34:08.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:34:08.000 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42438 2024-02-05T07:34:08.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.000 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:08.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:08.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.959239+0000", 2024-02-05T07:34:08.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:08.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:08.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.959239+0000", 2024-02-05T07:34:08.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:08.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:08.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.959240+0000", 2024-02-05T07:34:08.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0139999999999999e-06 2024-02-05T07:34:08.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:08.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.959304+0000", 2024-02-05T07:34:08.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4727999999999994e-05 2024-02-05T07:34:08.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:08.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.959314+0000", 2024-02-05T07:34:08.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7410000000000004e-06 2024-02-05T07:34:08.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:08.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.959321+0000", 2024-02-05T07:34:08.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.939e-06 2024-02-05T07:34:08.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.960397+0000", 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001075803 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.960669+0000", 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000272434 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3", 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.960811+0000", 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000141619 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:08.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.962188+0000", 2024-02-05T07:34:08.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013768750000000001 2024-02-05T07:34:08.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:08.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.964875+0000", 2024-02-05T07:34:08.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002686941 2024-02-05T07:34:08.004 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:08.004 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:08.004 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:08.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.004 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:42440 10.c 10:3439356a:::benchmark_data_smithi007_157098_object42439:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-02-05T07:34:08.004 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:07.960208+0000", 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0090514529999999992, 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095134060000000003, 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42440 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.960208+0000", 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:08.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.960208+0000", 2024-02-05T07:34:08.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:08.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:08.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.960209+0000", 2024-02-05T07:34:08.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.082e-06 2024-02-05T07:34:08.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:08.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.960284+0000", 2024-02-05T07:34:08.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4906000000000004e-05 2024-02-05T07:34:08.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:08.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.960296+0000", 2024-02-05T07:34:08.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1156e-05 2024-02-05T07:34:08.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:08.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.960304+0000", 2024-02-05T07:34:08.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7059999999999993e-06 2024-02-05T07:34:08.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:08.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.960585+0000", 2024-02-05T07:34:08.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028084399999999998 2024-02-05T07:34:08.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:08.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.960714+0000", 2024-02-05T07:34:08.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012846600000000001 2024-02-05T07:34:08.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3", 2024-02-05T07:34:08.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.960858+0000", 2024-02-05T07:34:08.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014444300000000001 2024-02-05T07:34:08.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.962273+0000", 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014151770000000001 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.964953+0000", 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026798239999999999 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:42446 10.c 10:31504634:::benchmark_data_smithi007_157098_object42445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:07.965206+0000", 2024-02-05T07:34:08.009 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0040538559999999998, 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045854829999999996, 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42446 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965206+0000", 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965206+0000", 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965207+0000", 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4729999999999999e-06 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965272+0000", 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4488999999999999e-05 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:08.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965295+0000", 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2665999999999999e-05 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965306+0000", 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1172000000000001e-05 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965378+0000", 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2356999999999999e-05 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:08.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965520+0000", 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014155 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3", 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.965659+0000", 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013883999999999999 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:07.967413+0000", 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017545429999999999 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:08.013 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-02-05T07:34:08.014 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 9 2024-02-05T07:34:08.014 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:34:08.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:08.055 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:08.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:08.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:08.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:07 smithi007 ceph-mon[29098]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:34:08.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:07 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:34:08.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:07 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:34:08.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:07 smithi007 ceph-mon[29098]: osdmap e430: 8 total, 8 up, 6 in 2024-02-05T07:34:08.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:07 smithi007 ceph-mon[37158]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:34:08.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:07 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:34:08.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:07 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:34:08.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:07 smithi007 ceph-mon[37158]: osdmap e430: 8 total, 8 up, 6 in 2024-02-05T07:34:08.190 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:08.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:08.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:08.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:08.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:08.342 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:08.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:08.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:08.476 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:08.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:08.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:08.578 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:08.608 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:08.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:08.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:08.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:08.728 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:08 smithi090 ceph-mon[33213]: pgmap v787: 178 pgs: 178 active+clean; 2.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-02-05T07:34:08.742 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:08.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:08.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:08.765 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 21 15 43775 43760 130.217 104.312 0.00899625 0.00767349 2024-02-05T07:34:08.843 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:08.887 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:08.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:08.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:08 smithi007 ceph-mon[29098]: pgmap v787: 178 pgs: 178 active+clean; 2.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-02-05T07:34:08.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:08 smithi007 ceph-mon[37158]: pgmap v787: 178 pgs: 178 active+clean; 2.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-02-05T07:34:08.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:09.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:09.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:09.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:09.165 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:09.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:09.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:09.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:09.303 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:09.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:09.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T07:34:09.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:09.438 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:09.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:09.540 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:09.593 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:09.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:09.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:09.658 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:34:09.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:09.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:09.728 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:09.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:09.765 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 22 16 45360 45344 128.798 99 0.0139044 0.00775864 2024-02-05T07:34:09.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:09.862 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:09.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:09.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:09.997 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:09.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:10.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:10.127 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:10.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:10.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:10.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:10.258 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:10.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:10.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:10.388 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:10.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:10.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:10.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:10.517 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:10.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:10.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:10.519 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:10.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:10.625 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:34:10.668 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:10.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:10.672 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:34:10.714 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:34:10.758 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:34:10.763 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:34:10.765 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 23 16 46581 46565 126.516 76.3125 0.0126369 0.0078955 2024-02-05T07:34:10.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:10.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:10.800 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:10.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:10.801 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:34:10.839 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:10 smithi090 ceph-mon[33213]: pgmap v788: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T07:34:10.839 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:10 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2124360602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:10.839 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:10 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2325787719' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:10.842 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:34:10.882 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:34:10.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:10 smithi007 ceph-mon[29098]: pgmap v788: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T07:34:10.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:10 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2124360602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:10.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:10 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2325787719' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:10.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:10 smithi007 ceph-mon[37158]: pgmap v788: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T07:34:10.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:10 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2124360602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:10.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:10 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2325787719' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:10.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:10.922 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:34:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:34:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:34:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:34:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:34:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:34:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:34:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:34:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:34:10.947 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:10.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:10.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:11.049 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:11.093 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:11.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:11.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:11.226 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:11.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:11.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:11.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:34:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:34:11.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:11.361 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:11.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:11.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:11.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:11.494 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:11.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:11.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:11.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:11.629 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:11.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:11.730 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:11.765 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 24 16 48093 48077 125.181 94.5 0.00766348 0.00798163 2024-02-05T07:34:11.769 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:11.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:11.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:11.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:11.903 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:11.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:11.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:11.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:11 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:34:11.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:11 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2325787719' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:34:11.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:11 smithi007 ceph-mon[29098]: osdmap e431: 8 total, 8 up, 6 in 2024-02-05T07:34:11.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:11 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:34:11.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:11 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2325787719' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:34:11.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:11 smithi007 ceph-mon[37158]: osdmap e431: 8 total, 8 up, 6 in 2024-02-05T07:34:12.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:11 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:34:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:11 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2325787719' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:34:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:11 smithi090 ceph-mon[33213]: osdmap e431: 8 total, 8 up, 6 in 2024-02-05T07:34:12.038 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:12.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:12.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:12.167 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:12.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:12.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:12.299 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:12.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:12.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:12.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:12.432 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:12.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:12.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:12.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:12.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:12.665 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:12.707 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:12.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:12.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:12.767 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 25 16 49418 49402 123.483 82.8125 0.00891621 0.00809358 2024-02-05T07:34:12.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:12.832 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:12.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:12 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:12 smithi007 ceph-mon[29098]: pgmap v790: 178 pgs: 178 active+clean; 2.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-02-05T07:34:12.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:12 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:12.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:12 smithi007 ceph-mon[37158]: pgmap v790: 178 pgs: 178 active+clean; 2.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-02-05T07:34:12.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:12.994 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:12 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:12.994 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:12 smithi090 ceph-mon[33213]: pgmap v790: 178 pgs: 178 active+clean; 2.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-02-05T07:34:13.095 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:13.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:13.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:13.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:13.227 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:13.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:13.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:13.329 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:13.369 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:13.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:13.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:13.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:13.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:13.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:13.599 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:34:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:34:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:34:13.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:13.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:13.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:34:13.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:13.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:13.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:13.766 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 26 16 50644 50628 121.68 76.625 0.011427 0.0082122 2024-02-05T07:34:13.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:13.888 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:13.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:13.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:14.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:14.019 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:14.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:14.117 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:14.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:14.149 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:14.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:14.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:14.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:14.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:14.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:14.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:14.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:14.407 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:14.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:14.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:14.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:14.543 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:14.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:14.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:14.687 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:14.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:14.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:14.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:34:14.760 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pg_num to 21 2024-02-05T07:34:14.766 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 27 16 51790 51774 119.825 71.625 0.0177416 0.00833898 2024-02-05T07:34:14.771 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:14 smithi007 ceph-mon[29098]: pgmap v791: 178 pgs: 178 active+clean; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-05T07:34:14.771 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:14 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3269334933' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:34:14.771 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:14 smithi007 ceph-mon[37158]: pgmap v791: 178 pgs: 178 active+clean; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-05T07:34:14.771 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:14 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3269334933' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:34:14.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:14.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:14.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:14.956 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:14.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:15.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:14 smithi090 ceph-mon[33213]: pgmap v791: 178 pgs: 178 active+clean; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-05T07:34:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:14 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3269334933' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:34:15.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:15.090 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:15.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:15.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:15.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:15.234 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:15.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:15.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:15.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:15.381 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:15.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:15.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:15.522 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:15.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:15.612 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:34:15.612 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:34:15.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:15.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:15.666 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:15.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:15.767 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 28 16 53041 53025 118.337 78.1875 0.0168014 0.0084437 2024-02-05T07:34:15.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:15.817 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:15.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:15.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:15 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3269334933' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T07:34:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:15 smithi007 ceph-mon[29098]: osdmap e432: 8 total, 8 up, 6 in 2024-02-05T07:34:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:34:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:15 smithi007 ceph-mon[29098]: pgmap v793: 178 pgs: 178 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T07:34:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:34:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:15 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:34:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:34:15.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:15 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3269334933' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T07:34:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:15 smithi007 ceph-mon[37158]: osdmap e432: 8 total, 8 up, 6 in 2024-02-05T07:34:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:34:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:15 smithi007 ceph-mon[37158]: pgmap v793: 178 pgs: 178 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T07:34:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:34:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:15 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:34:15.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:34:15.944 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:15.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:15.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:15 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3269334933' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T07:34:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:15 smithi090 ceph-mon[33213]: osdmap e432: 8 total, 8 up, 6 in 2024-02-05T07:34:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:34:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:15 smithi090 ceph-mon[33213]: pgmap v793: 178 pgs: 178 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T07:34:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:34:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:15 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:34:16.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:34:16.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:16.081 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:16.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:16.170 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:16.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:16.210 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:16.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:16.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:16.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:16.361 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:16.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:16.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:16.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:16.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:16.523 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:16.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:16.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:16.656 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:16.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:16.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:16.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:16.769 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 29 16 54133 54117 116.603 68.25 0.0245487 0.00856885 2024-02-05T07:34:16.803 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:16.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:16.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:16.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:16.932 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:16.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:16.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:16.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:16 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T07:34:16.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:16 smithi007 ceph-mon[29098]: osdmap e433: 8 total, 8 up, 6 in 2024-02-05T07:34:16.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:16 smithi007 ceph-mon[29098]: osdmap e434: 8 total, 8 up, 6 in 2024-02-05T07:34:16.971 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:16 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T07:34:16.971 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:16 smithi007 ceph-mon[37158]: osdmap e433: 8 total, 8 up, 6 in 2024-02-05T07:34:16.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:16 smithi007 ceph-mon[37158]: osdmap e434: 8 total, 8 up, 6 in 2024-02-05T07:34:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:16 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T07:34:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:16 smithi090 ceph-mon[33213]: osdmap e433: 8 total, 8 up, 6 in 2024-02-05T07:34:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:16 smithi090 ceph-mon[33213]: osdmap e434: 8 total, 8 up, 6 in 2024-02-05T07:34:17.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:17.064 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:17.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:17.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:17.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:17.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:17.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:17.323 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:17.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:17.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T07:34:17.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:17.500 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:17.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:17.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:17.644 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:17.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:17.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:17.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:17.770 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 30 16 55402 55386 115.353 79.3125 0.0128633 0.00866142 2024-02-05T07:34:17.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:17.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:17.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:17.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:17.940 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:17.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:18.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:18.056 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:17 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2870581519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:18.057 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:17 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:34:18.057 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:17 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:34:18.057 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:17 smithi090 ceph-mon[33213]: pgmap v796: 178 pgs: 2 unknown, 2 peering, 174 active+clean; 3.2 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-02-05T07:34:18.057 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:17 smithi090 ceph-mon[33213]: osdmap e435: 8 total, 8 up, 6 in 2024-02-05T07:34:18.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:17 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2870581519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:18.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:17 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:34:18.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:17 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:34:18.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:17 smithi007 ceph-mon[29098]: pgmap v796: 178 pgs: 2 unknown, 2 peering, 174 active+clean; 3.2 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-02-05T07:34:18.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:17 smithi007 ceph-mon[29098]: osdmap e435: 8 total, 8 up, 6 in 2024-02-05T07:34:18.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:17 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2870581519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:18.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:17 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:34:18.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:17 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:34:18.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:17 smithi007 ceph-mon[37158]: pgmap v796: 178 pgs: 2 unknown, 2 peering, 174 active+clean; 3.2 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-02-05T07:34:18.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:17 smithi007 ceph-mon[37158]: osdmap e435: 8 total, 8 up, 6 in 2024-02-05T07:34:18.088 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:18.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:18.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:18.213 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:34:18.213 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:34:18.213 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:34:18.213 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:34:18.213 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:34:18.213 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:34:18.213 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:34:18.213 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:34:18.213 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:34:18.213 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:34:18.213 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:34:18.214 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:34:18.214 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:34:18.214 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:34:18.214 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:34:18.214 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:34:18.240 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:18.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:18.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:18.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:18.385 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:18.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:18.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:18.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:18.532 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:18.533 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:34:18.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:18.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:18.635 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:18.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:18.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:18.770 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 31 15 56825 56810 114.503 89 0.00782538 0.00872804 2024-02-05T07:34:18.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:18.824 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:18.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:18.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:18.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:18.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:18.968 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:18.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:19.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:19.129 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:19.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:19.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:19.261 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:19.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:19.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:19.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:19.424 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:19.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:19.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:19.559 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:19.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:19.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:19.692 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:19.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:19.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:19.770 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 32 16 58183 58167 113.575 84.8125 0.0135674 0.00879912 2024-02-05T07:34:19.795 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:19.832 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:19.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:19.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:19.965 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:19.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:19 smithi090 ceph-mon[33213]: osdmap e436: 8 total, 8 up, 6 in 2024-02-05T07:34:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:19 smithi090 ceph-mon[33213]: pgmap v799: 178 pgs: 1 active+undersized+remapped, 2 unknown, 2 peering, 173 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-02-05T07:34:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:19 smithi090 ceph-mon[33213]: osdmap e437: 8 total, 8 up, 6 in 2024-02-05T07:34:20.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:20.098 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:20.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:20.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:20.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:19 smithi007 ceph-mon[29098]: osdmap e436: 8 total, 8 up, 6 in 2024-02-05T07:34:20.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:19 smithi007 ceph-mon[29098]: pgmap v799: 178 pgs: 1 active+undersized+remapped, 2 unknown, 2 peering, 173 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-02-05T07:34:20.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:19 smithi007 ceph-mon[29098]: osdmap e437: 8 total, 8 up, 6 in 2024-02-05T07:34:20.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:19 smithi007 ceph-mon[37158]: osdmap e436: 8 total, 8 up, 6 in 2024-02-05T07:34:20.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:19 smithi007 ceph-mon[37158]: pgmap v799: 178 pgs: 1 active+undersized+remapped, 2 unknown, 2 peering, 173 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-02-05T07:34:20.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:19 smithi007 ceph-mon[37158]: osdmap e437: 8 total, 8 up, 6 in 2024-02-05T07:34:20.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:20.229 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:20.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:20.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:20.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:20.361 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:20.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:20.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:20.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:20.495 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:20.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:20.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:20.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:20.619 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 3] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:34:20.620 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:34:20.620 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-05T07:34:20.620 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.544114 2024-02-05T07:34:20.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 1 0.5441143624073118 2024-02-05T07:34:20.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:20.640 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:20.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:20.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:20.770 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:20.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:20.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:20.771 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 33 16 59358 59342 112.359 73.4375 0.0192205 0.00889342 2024-02-05T07:34:20.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:20.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:34:20.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:20.927 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:20.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:21.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:21.051 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:21.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:21.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:21.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:21.182 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:21.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:21.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:34:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:34:21.283 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:21.340 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:21.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:21.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:21.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:21.485 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:21.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:21.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:21 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-05T07:34:21.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:21 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-05T07:34:21.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:21.602 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:21 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-05T07:34:21.602 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:21 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-05T07:34:21.602 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:21 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-05T07:34:21.602 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:21 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-05T07:34:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:21.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:21.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:21.735 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:21.771 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 34 16 60423 60407 111.012 66.5625 0.00688137 0.00900245 2024-02-05T07:34:21.771 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:34:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-02-05T07:34:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-02-05T07:34:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:34:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:131844 8.12 8:4b3718d0:::benchmark_data_smithi007_105755_object131843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:34:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:56.264093+0000", 2024-02-05T07:34:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 445.50305043200001, 2024-02-05T07:34:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.079539735, 2024-02-05T07:34:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:34:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:34:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131844 2024-02-05T07:34:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264093+0000", 2024-02-05T07:34:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264093+0000", 2024-02-05T07:34:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264094+0000", 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.443e-06 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264295+0000", 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020106599999999999 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264323+0000", 2024-02-05T07:34:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7719999999999999e-05 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264343+0000", 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0304000000000001e-05 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265133+0000", 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079006200000000001 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265141+0000", 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3299999999999999e-06 2024-02-05T07:34:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.337772+0000", 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0726301060000001 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.339454+0000", 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001682565 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343536+0000", 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040822530000000001 2024-02-05T07:34:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343632+0000", 2024-02-05T07:34:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5885999999999996e-05 2024-02-05T07:34:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:131867 8.12 8:4f7063db:::benchmark_data_smithi007_105755_object131866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:34:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:56.264377+0000", 2024-02-05T07:34:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 445.50276631600002, 2024-02-05T07:34:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.089266753, 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131867 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264377+0000", 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264377+0000", 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264377+0000", 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3799999999999996e-07 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264419+0000", 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1569000000000001e-05 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264426+0000", 2024-02-05T07:34:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1890000000000001e-06 2024-02-05T07:34:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264433+0000", 2024-02-05T07:34:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5039999999999999e-06 2024-02-05T07:34:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265185+0000", 2024-02-05T07:34:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075286500000000002 2024-02-05T07:34:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265194+0000", 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6389999999999998e-06 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.339734+0000", 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0745397999999999 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.339874+0000", 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014015699999999999 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2,5,7", 2024-02-05T07:34:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.340138+0000", 2024-02-05T07:34:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026435199999999998 2024-02-05T07:34:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343625+0000", 2024-02-05T07:34:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003486669 2024-02-05T07:34:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.348328+0000", 2024-02-05T07:34:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047024069999999996 2024-02-05T07:34:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.348374+0000", 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6669999999999999e-05 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.351568+0000", 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031942749999999999 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353402+0000", 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018333970000000001 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353476+0000", 2024-02-05T07:34:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4153000000000006e-05 2024-02-05T07:34:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353643+0000", 2024-02-05T07:34:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000167369 2024-02-05T07:34:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:131870 8.12 8:4c9bd4f8:::benchmark_data_smithi007_105755_object131869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:34:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:56.264452+0000", 2024-02-05T07:34:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 445.50269125699998, 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0894224100000001, 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131870 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264452+0000", 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264452+0000", 2024-02-05T07:34:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264452+0000", 2024-02-05T07:34:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7800000000000001e-07 2024-02-05T07:34:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264506+0000", 2024-02-05T07:34:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4051000000000001e-05 2024-02-05T07:34:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264513+0000", 2024-02-05T07:34:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8170000000000001e-06 2024-02-05T07:34:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264519+0000", 2024-02-05T07:34:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3029999999999996e-06 2024-02-05T07:34:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265227+0000", 2024-02-05T07:34:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070795899999999998 2024-02-05T07:34:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265234+0000", 2024-02-05T07:34:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7989999999999996e-06 2024-02-05T07:34:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.341221+0000", 2024-02-05T07:34:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.075986812 2024-02-05T07:34:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.341349+0000", 2024-02-05T07:34:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012759699999999999 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2,5,7", 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.341592+0000", 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024358299999999999 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343631+0000", 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020391020000000001 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.348414+0000", 2024-02-05T07:34:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047826880000000002 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.349943+0000", 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015293609999999999 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.351610+0000", 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001667099 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353687+0000", 2024-02-05T07:34:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00207632 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353750+0000", 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2897000000000004e-05 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353874+0000", 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124444 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:131873 8.12 8:4dbc2e4d:::benchmark_data_smithi007_105755_object131872:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-02-05T07:34:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:26:56.264529+0000", 2024-02-05T07:34:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 445.50261349900001, 2024-02-05T07:34:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0895603920000001, 2024-02-05T07:34:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:34:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131873 2024-02-05T07:34:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264529+0000", 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264529+0000", 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264530+0000", 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1900000000000002e-07 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264555+0000", 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5378e-05 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264565+0000", 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3559999999999999e-06 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.264571+0000", 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9100000000000002e-06 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265270+0000", 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069908399999999999 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:56.265277+0000", 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8519999999999998e-06 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.343935+0000", 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0786580370000001 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.344059+0000", 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124413 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2,5,7", 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.344449+0000", 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039028899999999999 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.348341+0000", 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038918870000000001 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.349845+0000", 2024-02-05T07:34:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015036400000000001 2024-02-05T07:34:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.349985+0000", 2024-02-05T07:34:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139598 2024-02-05T07:34:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.351684+0000", 2024-02-05T07:34:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001699267 2024-02-05T07:34:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353913+0000", 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022296479999999999 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.353968+0000", 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4391999999999997e-05 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:26:57.354090+0000", 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012202200000000001 2024-02-05T07:34:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:151895 8.2 8:46433b01:::benchmark_data_smithi007_105755_object15585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:27:15.629580+0000", 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 426.13756322, 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.341571794, 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151895 2024-02-05T07:34:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629580+0000", 2024-02-05T07:34:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629580+0000", 2024-02-05T07:34:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629581+0000", 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1650000000000001e-06 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629585+0000", 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4329999999999997e-06 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629593+0000", 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.14e-06 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629600+0000", 2024-02-05T07:34:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1820000000000003e-06 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629662+0000", 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2609999999999999e-05 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.629791+0000", 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128362 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,5", 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.630331+0000", 2024-02-05T07:34:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054002399999999995 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.634382+0000", 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040511710000000001 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.634627+0000", 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024509 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.691063+0000", 2024-02-05T07:34:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.056435762 2024-02-05T07:34:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:15.691077+0000", 2024-02-05T07:34:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4270999999999999e-05 2024-02-05T07:34:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.971083+0000", 2024-02-05T07:34:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2800058949999999 2024-02-05T07:34:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:27:16.971152+0000", 2024-02-05T07:34:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8689000000000006e-05 2024-02-05T07:34:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:224444 8.4 8:23e19df8:::benchmark_data_smithi007_105755_object88134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-02-05T07:34:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:45.765107+0000", 2024-02-05T07:34:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 336.00203567199998, 2024-02-05T07:34:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.987133759, 2024-02-05T07:34:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224444 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765107+0000", 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765107+0000", 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765108+0000", 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4600000000000001e-07 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765110+0000", 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9970000000000001e-06 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765113+0000", 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765116+0000", 2024-02-05T07:34:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0589999999999998e-06 2024-02-05T07:34:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765147+0000", 2024-02-05T07:34:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0747999999999998e-05 2024-02-05T07:34:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.765152+0000", 2024-02-05T07:34:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-02-05T07:34:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.742202+0000", 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.97704997400000004 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.742312+0000", 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110503 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.742585+0000", 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027314400000000001 2024-02-05T07:34:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.745284+0000", 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026988189999999999 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.746606+0000", 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013217039999999999 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.752149+0000", 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055428370000000001 2024-02-05T07:34:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.752208+0000", 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8742000000000001e-05 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.752241+0000", 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3494000000000002e-05 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:224465 8.4 8:21f25b55:::benchmark_data_smithi007_105755_object88155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:45.790686+0000", 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 335.97645684600002, 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.96721204100000002, 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224465 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790686+0000", 2024-02-05T07:34:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790686+0000", 2024-02-05T07:34:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790687+0000", 2024-02-05T07:34:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4799999999999997e-07 2024-02-05T07:34:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790691+0000", 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7340000000000002e-06 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790701+0000", 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7399999999999999e-06 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790708+0000", 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7649999999999993e-06 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790770+0000", 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1412000000000001e-05 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.790780+0000", 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9580000000000002e-06 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.743922+0000", 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95314281099999998 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.744022+0000", 2024-02-05T07:34:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9419000000000005e-05 2024-02-05T07:34:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:34:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.744187+0000", 2024-02-05T07:34:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016518699999999999 2024-02-05T07:34:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.746615+0000", 2024-02-05T07:34:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024280299999999999 2024-02-05T07:34:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.746674+0000", 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9366e-05 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.757803+0000", 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011128209999999999 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.757860+0000", 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7358000000000002e-05 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.757898+0000", 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8102999999999998e-05 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:224477 8.4 8:277e5eae:::benchmark_data_smithi007_105755_object88167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:45.800502+0000", 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 335.96664080400001, 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95752454200000003, 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224477 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800502+0000", 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800502+0000", 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800503+0000", 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.048e-06 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800508+0000", 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3370000000000001e-06 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800517+0000", 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0189999999999995e-06 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800524+0000", 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6029999999999999e-06 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800573+0000", 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8501999999999997e-05 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800581+0000", 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4249999999999998e-06 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.745336+0000", 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94475443999999997 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.745409+0000", 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3042999999999998e-05 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.745628+0000", 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021931300000000001 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.749129+0000", 2024-02-05T07:34:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:21.823 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:21.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035011999999999999 2024-02-05T07:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.749137+0000", 2024-02-05T07:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3610000000000001e-06 2024-02-05T07:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.757945+0000", 2024-02-05T07:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008807601 2024-02-05T07:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.757991+0000", 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5973999999999997e-05 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758027+0000", 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5676000000000002e-05 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:224478 8.4 8:2023b659:::benchmark_data_smithi007_105755_object88168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-02-05T07:34:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:45.800541+0000", 2024-02-05T07:34:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 335.96660185000002, 2024-02-05T07:34:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95762280600000005, 2024-02-05T07:34:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:34:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:34:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224478 2024-02-05T07:34:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800541+0000", 2024-02-05T07:34:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800541+0000", 2024-02-05T07:34:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800542+0000", 2024-02-05T07:34:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0299999999999999e-07 2024-02-05T07:34:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800544+0000", 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5210000000000001e-06 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800550+0000", 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9950000000000002e-06 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800556+0000", 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8459999999999996e-06 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800599+0000", 2024-02-05T07:34:21.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2948000000000003e-05 2024-02-05T07:34:21.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:45.800605+0000", 2024-02-05T07:34:21.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8320000000000002e-06 2024-02-05T07:34:21.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.746699+0000", 2024-02-05T07:34:21.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94609387 2024-02-05T07:34:21.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.746798+0000", 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8925999999999994e-05 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.746973+0000", 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017586699999999999 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.749175+0000", 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022016969999999999 2024-02-05T07:34:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.749325+0000", 2024-02-05T07:34:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015013200000000001 2024-02-05T07:34:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758080+0000", 2024-02-05T07:34:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087546289999999999 2024-02-05T07:34:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758129+0000", 2024-02-05T07:34:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8982e-05 2024-02-05T07:34:21.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:46.758164+0000", 2024-02-05T07:34:21.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5058000000000002e-05 2024-02-05T07:34:21.833 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.833 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.833 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.833 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:90079 9.e 9:71a255a5:::benchmark_data_smithi007_136200_object10234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-02-05T07:34:21.833 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:32:00.549204+0000", 2024-02-05T07:34:21.833 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 141.21793868699999, 2024-02-05T07:34:21.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0150999940000001, 2024-02-05T07:34:21.834 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.834 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:34:21.834 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.834 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:34:21.834 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:34:21.834 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90079 2024-02-05T07:34:21.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.834 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549204+0000", 2024-02-05T07:34:21.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549204+0000", 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549206+0000", 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3859999999999999e-06 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549227+0000", 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1117000000000001e-05 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549238+0000", 2024-02-05T07:34:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1507e-05 2024-02-05T07:34:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549245+0000", 2024-02-05T07:34:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0469999999999996e-06 2024-02-05T07:34:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549303+0000", 2024-02-05T07:34:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7494000000000002e-05 2024-02-05T07:34:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549437+0000", 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134525 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549532+0000", 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4881000000000002e-05 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.551286+0000", 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001753665 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.554566+0000", 2024-02-05T07:34:21.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032797939999999999 2024-02-05T07:34:21.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.554578+0000", 2024-02-05T07:34:21.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2604e-05 2024-02-05T07:34:21.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.564255+0000", 2024-02-05T07:34:21.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0096767259999999 2024-02-05T07:34:21.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.564304+0000", 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9248e-05 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:90088 9.e 9:7a4763be:::benchmark_data_smithi007_136200_object10243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:32:00.560900+0000", 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 141.20624333999999, 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0181687180000001, 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.839 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90088 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560900+0000", 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560900+0000", 2024-02-05T07:34:21.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560901+0000", 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.234e-06 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560905+0000", 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0840000000000002e-06 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560916+0000", 2024-02-05T07:34:21.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1270000000000001e-05 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560922+0000", 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9619999999999998e-06 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560982+0000", 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0009000000000001e-05 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560994+0000", 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2305e-05 2024-02-05T07:34:21.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.564648+0000", 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.003653269 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.564736+0000", 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8554000000000006e-05 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.564834+0000", 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7688000000000004e-05 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.569842+0000", 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050076779999999998 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.574424+0000", 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045827799999999998 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.579011+0000", 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045867130000000001 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.579043+0000", 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1834e-05 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.579068+0000", 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5338e-05 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:90091 9.e 9:73fec898:::benchmark_data_smithi007_136200_object10246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-02-05T07:34:21.845 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:32:00.571555+0000", 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 141.195588044, 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0075982779999999, 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90091 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571555+0000", 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571555+0000", 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571556+0000", 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.243e-06 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571561+0000", 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4630000000000003e-06 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571571+0000", 2024-02-05T07:34:21.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0285999999999999e-05 2024-02-05T07:34:21.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571576+0000", 2024-02-05T07:34:21.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1780000000000002e-06 2024-02-05T07:34:21.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571617+0000", 2024-02-05T07:34:21.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1325000000000002e-05 2024-02-05T07:34:21.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571626+0000", 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1859999999999992e-06 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.566392+0000", 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99476686599999997 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.566514+0000", 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121883 2024-02-05T07:34:21.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:21.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.566620+0000", 2024-02-05T07:34:21.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000105686 2024-02-05T07:34:21.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.571944+0000", 2024-02-05T07:34:21.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053239339999999998 2024-02-05T07:34:21.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.574471+0000", 2024-02-05T07:34:21.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002527446 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.579107+0000", 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046361029999999999 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.579134+0000", 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6461999999999998e-05 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.579153+0000", 2024-02-05T07:34:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9216999999999999e-05 2024-02-05T07:34:21.852 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.852 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.852 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.852 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:90094 9.e 9:7011d553:::benchmark_data_smithi007_136200_object10249:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-02-05T07:34:21.852 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:32:00.577316+0000", 2024-02-05T07:34:21.852 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 141.189827006, 2024-02-05T07:34:21.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0049051760000001, 2024-02-05T07:34:21.852 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.852 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.852 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.852 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:34:21.852 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:34:21.853 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90094 2024-02-05T07:34:21.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.853 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577316+0000", 2024-02-05T07:34:21.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577316+0000", 2024-02-05T07:34:21.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577317+0000", 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.068e-06 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577321+0000", 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8630000000000001e-06 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577332+0000", 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0740000000000001e-05 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577338+0000", 2024-02-05T07:34:21.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0680000000000002e-06 2024-02-05T07:34:21.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577382+0000", 2024-02-05T07:34:21.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4008000000000003e-05 2024-02-05T07:34:21.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577389+0000", 2024-02-05T07:34:21.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8340000000000001e-06 2024-02-05T07:34:21.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.568360+0000", 2024-02-05T07:34:21.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99097137400000002 2024-02-05T07:34:21.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.568512+0000", 2024-02-05T07:34:21.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015188499999999999 2024-02-05T07:34:21.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:21.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.568626+0000", 2024-02-05T07:34:21.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011432700000000001 2024-02-05T07:34:21.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.574381+0000", 2024-02-05T07:34:21.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0057548419999999996 2024-02-05T07:34:21.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.574517+0000", 2024-02-05T07:34:21.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136496 2024-02-05T07:34:21.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.582160+0000", 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0076425549999999997 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.582193+0000", 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2756999999999998e-05 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.582221+0000", 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8359000000000001e-05 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.858 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:90110 9.e 9:731cf300:::benchmark_data_smithi007_136200_object10265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:32:00.600929+0000", 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 141.166214102, 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99471859399999996, 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90110 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.600929+0000", 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.600929+0000", 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.600930+0000", 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.102e-06 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.600934+0000", 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2289999999999997e-06 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.600945+0000", 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0657e-05 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.600951+0000", 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7649999999999999e-06 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.601002+0000", 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1172999999999999e-05 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.601018+0000", 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6612999999999998e-05 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.569888+0000", 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.96886916199999995 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.570002+0000", 2024-02-05T07:34:21.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011484699999999999 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.570079+0000", 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6959999999999995e-05 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.574387+0000", 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043078680000000003 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.582267+0000", 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0078799030000000006 2024-02-05T07:34:21.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.595585+0000", 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013318194 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.595621+0000", 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5772000000000002e-05 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.595647+0000", 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6349000000000001e-05 2024-02-05T07:34:21.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:90126 9.e 9:7051948f:::benchmark_data_smithi007_136200_object10281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:32:00.630567+0000", 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 141.13657548500001, 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.967434503, 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90126 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.865 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630567+0000", 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630567+0000", 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630570+0000", 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3360000000000002e-06 2024-02-05T07:34:21.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630579+0000", 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6069999999999997e-06 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630607+0000", 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8087000000000001e-05 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630623+0000", 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5716e-05 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630756+0000", 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013253200000000001 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630773+0000", 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7391999999999999e-05 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.571034+0000", 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94026063100000001 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.571111+0000", 2024-02-05T07:34:21.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6854000000000007e-05 2024-02-05T07:34:21.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:21.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.571177+0000", 2024-02-05T07:34:21.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5925000000000003e-05 2024-02-05T07:34:21.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.574481+0000", 2024-02-05T07:34:21.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033049580000000002 2024-02-05T07:34:21.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.583698+0000", 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092162170000000005 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.597946+0000", 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014248475 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.597979+0000", 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3009999999999997e-05 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.598002+0000", 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2762999999999999e-05 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:58660 10.c 10:3e924969:::benchmark_data_smithi007_157098_object58659:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:20.105925+0000", 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.6612182199999999, 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.108488287, 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.871 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:34:21.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:34:21.872 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 58660 2024-02-05T07:34:21.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.872 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.105925+0000", 2024-02-05T07:34:21.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.105925+0000", 2024-02-05T07:34:21.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.105926+0000", 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8100000000000002e-07 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.105987+0000", 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1494000000000004e-05 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.105997+0000", 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.028e-05 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.106005+0000", 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7090000000000002e-06 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.106058+0000", 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3189000000000002e-05 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.106160+0000", 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101407 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3", 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.106250+0000", 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0418000000000001e-05 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.108049+0000", 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001799012 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.114815+0000", 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.006765699 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.214287+0000", 2024-02-05T07:34:21.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099472465999999996 2024-02-05T07:34:21.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.214371+0000", 2024-02-05T07:34:21.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3448000000000005e-05 2024-02-05T07:34:21.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.214413+0000", 2024-02-05T07:34:21.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2383999999999999e-05 2024-02-05T07:34:21.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.876 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:59559 10.e 10:7a01d2b9:::benchmark_data_smithi007_157098_object59558:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:20.929850+0000", 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.83729329600000002, 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.140516539, 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59559 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.929850+0000", 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.929850+0000", 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.929851+0000", 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0899999999999999e-06 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.929936+0000", 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4986000000000003e-05 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.929948+0000", 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2182e-05 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.929956+0000", 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2120000000000003e-06 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.930015+0000", 2024-02-05T07:34:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8672999999999999e-05 2024-02-05T07:34:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.930131+0000", 2024-02-05T07:34:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011641099999999999 2024-02-05T07:34:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:34:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.930269+0000", 2024-02-05T07:34:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013761900000000001 2024-02-05T07:34:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.932607+0000", 2024-02-05T07:34:21.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002338676 2024-02-05T07:34:21.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.932682+0000", 2024-02-05T07:34:21.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.47e-05 2024-02-05T07:34:21.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.068745+0000", 2024-02-05T07:34:21.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13606273099999999 2024-02-05T07:34:21.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.070326+0000", 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015808160000000001 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.070366+0000", 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0442999999999999e-05 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25454.0:59563 10.3 e437/423 10:cdb030c0:::benchmark_data_smithi007_157098_object59562:head v 437'3630, mlcod=437'3628)", 2024-02-05T07:34:21.882 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:20.954754+0000", 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.81238928099999996, 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12317678, 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.954754+0000", 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.954754+0000", 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.954754+0000", 2024-02-05T07:34:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999997e-07 2024-02-05T07:34:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.954795+0000", 2024-02-05T07:34:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0478000000000001e-05 2024-02-05T07:34:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.954803+0000", 2024-02-05T07:34:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1620000000000008e-06 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.954808+0000", 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6709999999999998e-06 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.954851+0000", 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3705999999999998e-05 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.954864+0000", 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2948999999999999e-05 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.077897+0000", 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.123032767 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.077930+0000", 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3346999999999998e-05 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25454.0:59565 10.9 e437/423 10:9f29d557:::benchmark_data_smithi007_157098_object59564:head v 437'3702, mlcod=437'3700)", 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:20.956192+0000", 2024-02-05T07:34:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.81095105899999997, 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11868912600000001, 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.956192+0000", 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.956192+0000", 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.956192+0000", 2024-02-05T07:34:21.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.82e-07 2024-02-05T07:34:21.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.956212+0000", 2024-02-05T07:34:21.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9975000000000001e-05 2024-02-05T07:34:21.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.068567+0000", 2024-02-05T07:34:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.112354258 2024-02-05T07:34:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.068574+0000", 2024-02-05T07:34:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7249999999999997e-06 2024-02-05T07:34:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.071527+0000", 2024-02-05T07:34:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029521170000000002 2024-02-05T07:34:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.071546+0000", 2024-02-05T07:34:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9928e-05 2024-02-05T07:34:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.074815+0000", 2024-02-05T07:34:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032683019999999998 2024-02-05T07:34:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.074881+0000", 2024-02-05T07:34:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6239000000000001e-05 2024-02-05T07:34:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:59576 10.6 10:68fb04d0:::benchmark_data_smithi007_157098_object59575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-02-05T07:34:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:20.961367+0000", 2024-02-05T07:34:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.80577573499999999, 2024-02-05T07:34:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11164393, 2024-02-05T07:34:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:34:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:34:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:34:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:34:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:34:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59576 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.961367+0000", 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.961367+0000", 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:34:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.961368+0000", 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9100000000000002e-07 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:20.961404+0000", 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5450000000000001e-05 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.068314+0000", 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.106910428 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:34:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.068326+0000", 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2179e-05 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.069127+0000", 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080037900000000004 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.069272+0000", 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000145308 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:34:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.069421+0000", 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014939400000000001 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.071665+0000", 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022438530000000001 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.072847+0000", 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011817869999999999 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:34:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.072898+0000", 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1144999999999999e-05 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.072974+0000", 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5666000000000004e-05 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:21.073011+0000", 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7450000000000002e-05 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:34:21.897 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:34:21.897 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:34:21.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:21.959 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:21.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:22.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:22.087 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:22.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:22.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:22.212 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:22.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:22.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:22.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:22.344 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:22.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:22.355 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:34:22.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:22.468 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:22.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:22.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:22.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:22.608 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:22.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:22.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:22 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:22 smithi007 ceph-mon[29098]: pgmap v801: 178 pgs: 1 active+undersized+remapped, 1 unknown, 1 peering, 175 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 4.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 74 KiB/s, 0 objects/s recovering 2024-02-05T07:34:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:22 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-05T07:34:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:22 smithi007 ceph-mon[29098]: osdmap e438: 8 total, 8 up, 6 in 2024-02-05T07:34:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:22 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2328991155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:34:22.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:22 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:22.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:22 smithi007 ceph-mon[37158]: pgmap v801: 178 pgs: 1 active+undersized+remapped, 1 unknown, 1 peering, 175 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 4.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 74 KiB/s, 0 objects/s recovering 2024-02-05T07:34:22.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:22 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-05T07:34:22.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:22 smithi007 ceph-mon[37158]: osdmap e438: 8 total, 8 up, 6 in 2024-02-05T07:34:22.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:22 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2328991155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:34:22.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:22.746 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:22.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:22.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:22.770 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 35 16 61491 61475 109.747 66.75 0.0153926 0.00910535 2024-02-05T07:34:22.773 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:22 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:22.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:22 smithi090 ceph-mon[33213]: pgmap v801: 178 pgs: 1 active+undersized+remapped, 1 unknown, 1 peering, 175 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 4.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 74 KiB/s, 0 objects/s recovering 2024-02-05T07:34:22.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:22 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-05T07:34:22.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:22 smithi090 ceph-mon[33213]: osdmap e438: 8 total, 8 up, 6 in 2024-02-05T07:34:22.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:22 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2328991155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:34:22.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:22.887 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:22.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:22.989 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:23.018 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:23.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:23.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:23.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:23.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:23.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:23.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:23.302 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:23.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:23.404 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:23.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:23.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:23.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:23.580 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:23.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:23.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:23 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:34:23.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:23 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2328991155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:34:23.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:23 smithi007 ceph-mon[29098]: osdmap e439: 8 total, 8 up, 6 in 2024-02-05T07:34:23.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:34:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:34:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:34:23.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:23 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:34:23.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:23 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2328991155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:34:23.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:23 smithi007 ceph-mon[37158]: osdmap e439: 8 total, 8 up, 6 in 2024-02-05T07:34:23.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:23.724 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:23.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:23.736 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:23 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:34:23.736 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:23 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2328991155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:34:23.736 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:23 smithi090 ceph-mon[33213]: osdmap e439: 8 total, 8 up, 6 in 2024-02-05T07:34:23.771 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 36 16 62825 62809 109.015 83.375 0.0159373 0.00916806 2024-02-05T07:34:23.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:23.867 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:23.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:23.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:23.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:24.058 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:24.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:24.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:24.159 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:24.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:24.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:24.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:24.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:24.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:24.452 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:24.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T07:34:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:24.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:24.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:24.636 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[29098]: 6.16 scrub starts 2024-02-05T07:34:24.637 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[29098]: 6.16 scrub ok 2024-02-05T07:34:24.637 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[29098]: 6.f scrub starts 2024-02-05T07:34:24.637 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[29098]: 6.f scrub ok 2024-02-05T07:34:24.637 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[29098]: 1.f scrub starts 2024-02-05T07:34:24.637 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[29098]: 1.f scrub ok 2024-02-05T07:34:24.637 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[29098]: pgmap v804: 178 pgs: 1 active+undersized+remapped, 177 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 5.3 KiB/s rd, 56 MiB/s wr, 903 op/s; 80 KiB/s, 0 objects/s recovering 2024-02-05T07:34:24.637 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[29098]: osdmap e440: 8 total, 8 up, 6 in 2024-02-05T07:34:24.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[37158]: 6.16 scrub starts 2024-02-05T07:34:24.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[37158]: 6.16 scrub ok 2024-02-05T07:34:24.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[37158]: 6.f scrub starts 2024-02-05T07:34:24.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[37158]: 6.f scrub ok 2024-02-05T07:34:24.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[37158]: 1.f scrub starts 2024-02-05T07:34:24.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[37158]: 1.f scrub ok 2024-02-05T07:34:24.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[37158]: pgmap v804: 178 pgs: 1 active+undersized+remapped, 177 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 5.3 KiB/s rd, 56 MiB/s wr, 903 op/s; 80 KiB/s, 0 objects/s recovering 2024-02-05T07:34:24.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:24 smithi007 ceph-mon[37158]: osdmap e440: 8 total, 8 up, 6 in 2024-02-05T07:34:24.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:24.647 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:24.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:24.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:24.758 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:24 smithi090 ceph-mon[33213]: 6.16 scrub starts 2024-02-05T07:34:24.759 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:24 smithi090 ceph-mon[33213]: 6.16 scrub ok 2024-02-05T07:34:24.759 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:24 smithi090 ceph-mon[33213]: 6.f scrub starts 2024-02-05T07:34:24.759 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:24 smithi090 ceph-mon[33213]: 6.f scrub ok 2024-02-05T07:34:24.759 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:24 smithi090 ceph-mon[33213]: 1.f scrub starts 2024-02-05T07:34:24.759 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:24 smithi090 ceph-mon[33213]: 1.f scrub ok 2024-02-05T07:34:24.759 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:24 smithi090 ceph-mon[33213]: pgmap v804: 178 pgs: 1 active+undersized+remapped, 177 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 5.3 KiB/s rd, 56 MiB/s wr, 903 op/s; 80 KiB/s, 0 objects/s recovering 2024-02-05T07:34:24.759 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:24 smithi090 ceph-mon[33213]: osdmap e440: 8 total, 8 up, 6 in 2024-02-05T07:34:24.773 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 37 16 64231 64215 108.436 87.875 0.0125573 0.0092171 2024-02-05T07:34:24.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:24.803 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:24.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:24.904 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:24.947 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:24.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:24.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:25.010 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:34:25.049 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:25.092 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:25.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:25.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:25.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:25.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:25.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:25.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:25.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:25.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:25.499 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:25.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:25.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:25.635 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:25.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:25.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:25.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[29098]: 1.1b scrub starts 2024-02-05T07:34:25.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[29098]: 1.1b scrub ok 2024-02-05T07:34:25.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[29098]: 1.15 scrub starts 2024-02-05T07:34:25.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[29098]: 1.15 scrub ok 2024-02-05T07:34:25.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[29098]: 1.1c scrub starts 2024-02-05T07:34:25.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[29098]: 1.1c scrub ok 2024-02-05T07:34:25.666 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[29098]: 1.1a scrub starts 2024-02-05T07:34:25.666 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[29098]: 1.1a scrub ok 2024-02-05T07:34:25.666 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[29098]: 1.16 scrub starts 2024-02-05T07:34:25.666 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[29098]: 1.16 scrub ok 2024-02-05T07:34:25.666 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[29098]: osdmap e441: 8 total, 8 up, 6 in 2024-02-05T07:34:25.666 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-05T07:34:25.666 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-05T07:34:25.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[37158]: 1.1b scrub starts 2024-02-05T07:34:25.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[37158]: 1.1b scrub ok 2024-02-05T07:34:25.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[37158]: 1.15 scrub starts 2024-02-05T07:34:25.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[37158]: 1.15 scrub ok 2024-02-05T07:34:25.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[37158]: 1.1c scrub starts 2024-02-05T07:34:25.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[37158]: 1.1c scrub ok 2024-02-05T07:34:25.667 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[37158]: 1.1a scrub starts 2024-02-05T07:34:25.667 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[37158]: 1.1a scrub ok 2024-02-05T07:34:25.667 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[37158]: 1.16 scrub starts 2024-02-05T07:34:25.667 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[37158]: 1.16 scrub ok 2024-02-05T07:34:25.667 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[37158]: osdmap e441: 8 total, 8 up, 6 in 2024-02-05T07:34:25.667 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-05T07:34:25.667 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:25 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-05T07:34:25.716 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:25 smithi090 ceph-mon[33213]: 1.1b scrub starts 2024-02-05T07:34:25.716 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:25 smithi090 ceph-mon[33213]: 1.1b scrub ok 2024-02-05T07:34:25.716 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:25 smithi090 ceph-mon[33213]: 1.15 scrub starts 2024-02-05T07:34:25.716 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:25 smithi090 ceph-mon[33213]: 1.15 scrub ok 2024-02-05T07:34:25.716 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:25 smithi090 ceph-mon[33213]: 1.1c scrub starts 2024-02-05T07:34:25.716 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:25 smithi090 ceph-mon[33213]: 1.1c scrub ok 2024-02-05T07:34:25.716 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:25 smithi090 ceph-mon[33213]: 1.1a scrub starts 2024-02-05T07:34:25.716 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:25 smithi090 ceph-mon[33213]: 1.1a scrub ok 2024-02-05T07:34:25.716 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:25 smithi090 ceph-mon[33213]: 1.16 scrub starts 2024-02-05T07:34:25.716 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:25 smithi090 ceph-mon[33213]: 1.16 scrub ok 2024-02-05T07:34:25.716 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:25 smithi090 ceph-mon[33213]: osdmap e441: 8 total, 8 up, 6 in 2024-02-05T07:34:25.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:25 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-05T07:34:25.717 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:25 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-05T07:34:25.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:25.770 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:25.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:25.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:25.773 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 38 16 65574 65558 107.791 83.9375 0.0106987 0.009272 2024-02-05T07:34:25.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:25.915 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:25.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:26.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:34:26.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:26.094 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:26.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:26.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:26.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:26.251 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:26.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:26.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:26.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:26.407 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:26.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:26.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:26.543 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:26.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:26.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:26.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:26.679 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:26.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:26.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:26.709 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:26 smithi090 ceph-mon[33213]: 1.1d scrub starts 2024-02-05T07:34:26.710 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:26 smithi090 ceph-mon[33213]: 1.1d scrub ok 2024-02-05T07:34:26.710 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:26 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:34:26.710 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:26 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:34:26.710 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:26 smithi090 ceph-mon[33213]: pgmap v807: 177 pgs: 177 active+clean; 3.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 4.2 KiB/s rd, 42 MiB/s wr, 676 op/s; 16 KiB/s, 0 objects/s recovering 2024-02-05T07:34:26.710 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:26 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-05T07:34:26.710 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:26 smithi090 ceph-mon[33213]: osdmap e442: 8 total, 8 up, 6 in 2024-02-05T07:34:26.710 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:26 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:34:26.710 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:26 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:34:26.773 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 39 15 66856 66841 107.083 80.1875 0.0164734 0.00933344 2024-02-05T07:34:26.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:26.821 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:26.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:26.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:26.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[37158]: 1.1d scrub starts 2024-02-05T07:34:26.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[37158]: 1.1d scrub ok 2024-02-05T07:34:26.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:34:26.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:34:26.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[37158]: pgmap v807: 177 pgs: 177 active+clean; 3.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 4.2 KiB/s rd, 42 MiB/s wr, 676 op/s; 16 KiB/s, 0 objects/s recovering 2024-02-05T07:34:26.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-05T07:34:26.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[37158]: osdmap e442: 8 total, 8 up, 6 in 2024-02-05T07:34:26.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:34:26.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:34:26.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:26.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[29098]: 1.1d scrub starts 2024-02-05T07:34:26.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[29098]: 1.1d scrub ok 2024-02-05T07:34:26.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:34:26.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:34:26.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[29098]: pgmap v807: 177 pgs: 177 active+clean; 3.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 4.2 KiB/s rd, 42 MiB/s wr, 676 op/s; 16 KiB/s, 0 objects/s recovering 2024-02-05T07:34:26.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-05T07:34:26.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[29098]: osdmap e442: 8 total, 8 up, 6 in 2024-02-05T07:34:26.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:34:26.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:26 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:34:26.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:26.959 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:26.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:27.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:27.103 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:27.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:27.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:27.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:27.243 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:27.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:27.344 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:27.383 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:27.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:27.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:27.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:27.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:27.548 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:27.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:27.635 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:27 smithi090 ceph-mon[33213]: 1.e scrub starts 2024-02-05T07:34:27.635 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:27 smithi090 ceph-mon[33213]: 1.e scrub ok 2024-02-05T07:34:27.635 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:27 smithi090 ceph-mon[33213]: 3.0 scrub starts 2024-02-05T07:34:27.635 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:27 smithi090 ceph-mon[33213]: osdmap e443: 8 total, 8 up, 6 in 2024-02-05T07:34:27.636 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:27 smithi090 ceph-mon[33213]: 3.0 scrub ok 2024-02-05T07:34:27.636 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:27 smithi090 ceph-mon[33213]: 6.6 scrub starts 2024-02-05T07:34:27.636 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:27 smithi090 ceph-mon[33213]: 6.6 scrub ok 2024-02-05T07:34:27.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:27.689 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:27.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:27.773 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:34:27.773909+0000 min lat: 0.00131497 max lat: 0.307728 avg lat: 0.00938052 2024-02-05T07:34:27.774 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:34:27.774 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 40 16 68225 68209 106.544 85.5 0.0132331 0.00938052 2024-02-05T07:34:27.791 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:27.836 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:27.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:27.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:27.838 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:34:27.838 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:34:27.838 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-02-05T07:34:27.839 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:34:27.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:27 smithi007 ceph-mon[29098]: 1.e scrub starts 2024-02-05T07:34:27.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:27 smithi007 ceph-mon[29098]: 1.e scrub ok 2024-02-05T07:34:27.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:27 smithi007 ceph-mon[29098]: 3.0 scrub starts 2024-02-05T07:34:27.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:27 smithi007 ceph-mon[29098]: osdmap e443: 8 total, 8 up, 6 in 2024-02-05T07:34:27.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:27 smithi007 ceph-mon[29098]: 3.0 scrub ok 2024-02-05T07:34:27.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:27 smithi007 ceph-mon[29098]: 6.6 scrub starts 2024-02-05T07:34:27.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:27 smithi007 ceph-mon[29098]: 6.6 scrub ok 2024-02-05T07:34:27.886 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:27 smithi007 ceph-mon[37158]: 1.e scrub starts 2024-02-05T07:34:27.886 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:27 smithi007 ceph-mon[37158]: 1.e scrub ok 2024-02-05T07:34:27.886 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:27 smithi007 ceph-mon[37158]: 3.0 scrub starts 2024-02-05T07:34:27.886 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:27 smithi007 ceph-mon[37158]: osdmap e443: 8 total, 8 up, 6 in 2024-02-05T07:34:27.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:27 smithi007 ceph-mon[37158]: 3.0 scrub ok 2024-02-05T07:34:27.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:27 smithi007 ceph-mon[37158]: 6.6 scrub starts 2024-02-05T07:34:27.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:27 smithi007 ceph-mon[37158]: 6.6 scrub ok 2024-02-05T07:34:27.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:28.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:28.013 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:28.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:28.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:28.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:28.251 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:28.283 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:28.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:28.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:28.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:28.434 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:28.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:28.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:28.577 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:28.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:28.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:28.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:28.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:28.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 41 16 69412 69396 105.755 74.1875 0.0375266 0.00944993 2024-02-05T07:34:28.844 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:28 smithi090 ceph-mon[33213]: pgmap v810: 177 pgs: 177 active+clean; 3.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-05T07:34:28.844 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:28 smithi090 ceph-mon[33213]: osdmap e444: 8 total, 8 up, 6 in 2024-02-05T07:34:28.844 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:28 smithi090 ceph-mon[33213]: 6.1b scrub starts 2024-02-05T07:34:28.844 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:28 smithi090 ceph-mon[33213]: 5.15 scrub starts 2024-02-05T07:34:28.844 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:28 smithi090 ceph-mon[33213]: 5.15 scrub ok 2024-02-05T07:34:28.844 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:28 smithi090 ceph-mon[33213]: 6.1b scrub ok 2024-02-05T07:34:28.844 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:28 smithi090 ceph-mon[33213]: 1.10 scrub starts 2024-02-05T07:34:28.844 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:28 smithi090 ceph-mon[33213]: 1.10 scrub ok 2024-02-05T07:34:28.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[29098]: pgmap v810: 177 pgs: 177 active+clean; 3.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-05T07:34:28.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[29098]: osdmap e444: 8 total, 8 up, 6 in 2024-02-05T07:34:28.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[29098]: 6.1b scrub starts 2024-02-05T07:34:28.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[29098]: 5.15 scrub starts 2024-02-05T07:34:28.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[29098]: 5.15 scrub ok 2024-02-05T07:34:28.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[29098]: 6.1b scrub ok 2024-02-05T07:34:28.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[29098]: 1.10 scrub starts 2024-02-05T07:34:28.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[29098]: 1.10 scrub ok 2024-02-05T07:34:28.849 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[37158]: pgmap v810: 177 pgs: 177 active+clean; 3.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-05T07:34:28.849 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[37158]: osdmap e444: 8 total, 8 up, 6 in 2024-02-05T07:34:28.849 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[37158]: 6.1b scrub starts 2024-02-05T07:34:28.850 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[37158]: 5.15 scrub starts 2024-02-05T07:34:28.850 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[37158]: 5.15 scrub ok 2024-02-05T07:34:28.850 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[37158]: 6.1b scrub ok 2024-02-05T07:34:28.850 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[37158]: 1.10 scrub starts 2024-02-05T07:34:28.850 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:28 smithi007 ceph-mon[37158]: 1.10 scrub ok 2024-02-05T07:34:28.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:28.890 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:28.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:28.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T07:34:28.975 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:28.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:29.065 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:29.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:29.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:29.097 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:34:29.150 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:34:29.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:29.191 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:34:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:29.222 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:29.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:29.240 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:34:29.280 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:34:29.322 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:34:29.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:29.362 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:34:29.390 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:29.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:29.407 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:34:29.416 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:34:29.416 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:34:29.416 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:34:29.416 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:34:29.416 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:34:29.417 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:34:29.417 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:34:29.417 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:34:29.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:29.542 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:29.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:29.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:29.640 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.1 primary-affinity to 0.544114 (8356592) 2024-02-05T07:34:29.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:29.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[37158]: osdmap e445: 8 total, 8 up, 6 in 2024-02-05T07:34:29.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[37158]: 1.4 scrub starts 2024-02-05T07:34:29.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[37158]: 1.4 scrub ok 2024-02-05T07:34:29.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[37158]: 4.9 scrub starts 2024-02-05T07:34:29.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[37158]: 4.9 scrub ok 2024-02-05T07:34:29.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3730426940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:29.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2704564569' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5441143624073118}]: dispatch 2024-02-05T07:34:29.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5441143624073118}]: dispatch 2024-02-05T07:34:29.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[37158]: 1.7 scrub starts 2024-02-05T07:34:29.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[37158]: 1.7 scrub ok 2024-02-05T07:34:29.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-05T07:34:29.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-05T07:34:29.671 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:34:29.693 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:29.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:29.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:29.774 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 42 16 70525 70509 104.893 69.5625 0.0137954 0.0095281 2024-02-05T07:34:29.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:29 smithi090 ceph-mon[33213]: osdmap e445: 8 total, 8 up, 6 in 2024-02-05T07:34:29.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:29 smithi090 ceph-mon[33213]: 1.4 scrub starts 2024-02-05T07:34:29.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:29 smithi090 ceph-mon[33213]: 1.4 scrub ok 2024-02-05T07:34:29.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:29 smithi090 ceph-mon[33213]: 4.9 scrub starts 2024-02-05T07:34:29.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:29 smithi090 ceph-mon[33213]: 4.9 scrub ok 2024-02-05T07:34:29.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:29 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3730426940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:29.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:29 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2704564569' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5441143624073118}]: dispatch 2024-02-05T07:34:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:29 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5441143624073118}]: dispatch 2024-02-05T07:34:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:29 smithi090 ceph-mon[33213]: 1.7 scrub starts 2024-02-05T07:34:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:29 smithi090 ceph-mon[33213]: 1.7 scrub ok 2024-02-05T07:34:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-05T07:34:29.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:29 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-05T07:34:29.795 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:29.836 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:29.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:29.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[29098]: osdmap e445: 8 total, 8 up, 6 in 2024-02-05T07:34:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[29098]: 1.4 scrub starts 2024-02-05T07:34:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[29098]: 1.4 scrub ok 2024-02-05T07:34:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[29098]: 4.9 scrub starts 2024-02-05T07:34:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[29098]: 4.9 scrub ok 2024-02-05T07:34:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3730426940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2704564569' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5441143624073118}]: dispatch 2024-02-05T07:34:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5441143624073118}]: dispatch 2024-02-05T07:34:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[29098]: 1.7 scrub starts 2024-02-05T07:34:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[29098]: 1.7 scrub ok 2024-02-05T07:34:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-05T07:34:29.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:29 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-05T07:34:29.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:29.982 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:29.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:29.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:30.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:30.132 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:30.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:30.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:30.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:30.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:30.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:30.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:30.436 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:30.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:30.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:30.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:30.585 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:30.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:30.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:30.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:30.737 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:30.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:30.774 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 43 16 71889 71873 104.435 85.25 0.00863468 0.00957016 2024-02-05T07:34:30.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:30.890 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:30.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:30.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:30.911 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:30 smithi090 ceph-mon[33213]: pgmap v813: 176 pgs: 176 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-02-05T07:34:30.912 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:30 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5441143624073118}]': finished 2024-02-05T07:34:30.912 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-05T07:34:30.912 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:30 smithi090 ceph-mon[33213]: osdmap e446: 8 total, 8 up, 6 in 2024-02-05T07:34:30.912 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:30.912 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:30.912 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:30 smithi090 ceph-mon[33213]: 5.d scrub starts 2024-02-05T07:34:30.912 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:30 smithi090 ceph-mon[33213]: 5.d scrub ok 2024-02-05T07:34:30.912 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:30 smithi090 ceph-mon[33213]: osdmap e447: 8 total, 8 up, 6 in 2024-02-05T07:34:30.912 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:34:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:34:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[29098]: pgmap v813: 176 pgs: 176 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-02-05T07:34:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5441143624073118}]': finished 2024-02-05T07:34:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-05T07:34:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[29098]: osdmap e446: 8 total, 8 up, 6 in 2024-02-05T07:34:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[29098]: 5.d scrub starts 2024-02-05T07:34:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[29098]: 5.d scrub ok 2024-02-05T07:34:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[29098]: osdmap e447: 8 total, 8 up, 6 in 2024-02-05T07:34:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[37158]: pgmap v813: 176 pgs: 176 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-02-05T07:34:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5441143624073118}]': finished 2024-02-05T07:34:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-05T07:34:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[37158]: osdmap e446: 8 total, 8 up, 6 in 2024-02-05T07:34:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[37158]: 5.d scrub starts 2024-02-05T07:34:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[37158]: 5.d scrub ok 2024-02-05T07:34:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:30 smithi007 ceph-mon[37158]: osdmap e447: 8 total, 8 up, 6 in 2024-02-05T07:34:30.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:31.033 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:31.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:31.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:31.183 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:31.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:31.285 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:31.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:31.317 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:31.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:31.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:31.455 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:31.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:31.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:31.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:31.589 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:31.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:31.594 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:34:31.594 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:34:31.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:31.722 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:31.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:31.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:31.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:31 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:34:31.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:31 smithi090 ceph-mon[33213]: 1.1b scrub starts 2024-02-05T07:34:31.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:31 smithi090 ceph-mon[33213]: 1.1b scrub ok 2024-02-05T07:34:31.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:31 smithi090 ceph-mon[33213]: 1.13 scrub starts 2024-02-05T07:34:31.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:31 smithi090 ceph-mon[33213]: 1.13 scrub ok 2024-02-05T07:34:31.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:31 smithi090 ceph-mon[33213]: 1.3 scrub starts 2024-02-05T07:34:31.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:31 smithi090 ceph-mon[33213]: 1.3 scrub ok 2024-02-05T07:34:31.774 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 44 16 73232 73216 103.969 83.9375 0.0159821 0.00961276 2024-02-05T07:34:31.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:31.871 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:31.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:31.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:31 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:34:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:31 smithi007 ceph-mon[29098]: 1.1b scrub starts 2024-02-05T07:34:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:31 smithi007 ceph-mon[29098]: 1.1b scrub ok 2024-02-05T07:34:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:31 smithi007 ceph-mon[29098]: 1.13 scrub starts 2024-02-05T07:34:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:31 smithi007 ceph-mon[29098]: 1.13 scrub ok 2024-02-05T07:34:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:31 smithi007 ceph-mon[29098]: 1.3 scrub starts 2024-02-05T07:34:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:31 smithi007 ceph-mon[29098]: 1.3 scrub ok 2024-02-05T07:34:31.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:31 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:34:31.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:31 smithi007 ceph-mon[37158]: 1.1b scrub starts 2024-02-05T07:34:31.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:31 smithi007 ceph-mon[37158]: 1.1b scrub ok 2024-02-05T07:34:31.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:31 smithi007 ceph-mon[37158]: 1.13 scrub starts 2024-02-05T07:34:31.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:31 smithi007 ceph-mon[37158]: 1.13 scrub ok 2024-02-05T07:34:31.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:31 smithi007 ceph-mon[37158]: 1.3 scrub starts 2024-02-05T07:34:31.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:31 smithi007 ceph-mon[37158]: 1.3 scrub ok 2024-02-05T07:34:31.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:32.005 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:32.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:32.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:32.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:32.137 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:32.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:32.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:32.276 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:32.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:32.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:32.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T07:34:32.378 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:32.409 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:32.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:32.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:32.545 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:32.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:32.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:32.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:32.675 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:34:32.683 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:32.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:32.774 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 45 16 74492 74476 103.409 78.75 0.00692041 0.00966549 2024-02-05T07:34:32.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:32.826 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:32.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[29098]: pgmap v816: 176 pgs: 11 peering, 165 active+clean; 4.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 93 MiB/s wr, 1.49k op/s; 3.3 KiB/s, 0 objects/s recovering 2024-02-05T07:34:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[29098]: Health check failed: Reduced data availability: 9 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-02-05T07:34:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[29098]: osdmap e448: 8 total, 8 up, 6 in 2024-02-05T07:34:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[29098]: 1.b scrub starts 2024-02-05T07:34:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[29098]: 1.b scrub ok 2024-02-05T07:34:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[29098]: 3.14 scrub starts 2024-02-05T07:34:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[29098]: 3.14 scrub ok 2024-02-05T07:34:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[37158]: pgmap v816: 176 pgs: 11 peering, 165 active+clean; 4.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 93 MiB/s wr, 1.49k op/s; 3.3 KiB/s, 0 objects/s recovering 2024-02-05T07:34:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[37158]: Health check failed: Reduced data availability: 9 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-02-05T07:34:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[37158]: osdmap e448: 8 total, 8 up, 6 in 2024-02-05T07:34:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[37158]: 1.b scrub starts 2024-02-05T07:34:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[37158]: 1.b scrub ok 2024-02-05T07:34:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[37158]: 3.14 scrub starts 2024-02-05T07:34:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:32 smithi007 ceph-mon[37158]: 3.14 scrub ok 2024-02-05T07:34:32.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:32.955 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:32 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:32.955 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:32 smithi090 ceph-mon[33213]: pgmap v816: 176 pgs: 11 peering, 165 active+clean; 4.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 93 MiB/s wr, 1.49k op/s; 3.3 KiB/s, 0 objects/s recovering 2024-02-05T07:34:32.955 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:32 smithi090 ceph-mon[33213]: Health check failed: Reduced data availability: 9 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-02-05T07:34:32.955 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:32 smithi090 ceph-mon[33213]: osdmap e448: 8 total, 8 up, 6 in 2024-02-05T07:34:32.955 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:32 smithi090 ceph-mon[33213]: 1.b scrub starts 2024-02-05T07:34:32.955 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:32 smithi090 ceph-mon[33213]: 1.b scrub ok 2024-02-05T07:34:32.955 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:32 smithi090 ceph-mon[33213]: 3.14 scrub starts 2024-02-05T07:34:32.955 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:32 smithi090 ceph-mon[33213]: 3.14 scrub ok 2024-02-05T07:34:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:32.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:33.066 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:33.117 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:33.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:33.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:33.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:33.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:33.264 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:33.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:33.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:33.397 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:33.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:33.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:33.499 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:33.531 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:33.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:33.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:33.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:33.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:33.678 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:33.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:33.694 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:34:33.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:33 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:34:33.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:33 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:34:33.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:33 smithi090 ceph-mon[33213]: osdmap e449: 8 total, 8 up, 6 in 2024-02-05T07:34:33.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:33 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1465190336' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:33.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:33 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:33.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:33 smithi090 ceph-mon[33213]: 3.13 scrub starts 2024-02-05T07:34:33.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:33 smithi090 ceph-mon[33213]: 3.13 scrub ok 2024-02-05T07:34:33.774 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 46 16 75852 75836 103.008 85 0.00936338 0.0097031 2024-02-05T07:34:33.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:33.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:34:33.813 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:34:33.813 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:34:33.813 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-mon[29098]: osdmap e449: 8 total, 8 up, 6 in 2024-02-05T07:34:33.814 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1465190336' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:33.814 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:33.814 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-mon[29098]: 3.13 scrub starts 2024-02-05T07:34:33.814 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-mon[29098]: 3.13 scrub ok 2024-02-05T07:34:33.814 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:34:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:34:33.814 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:34:33.815 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:34:33.815 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-mon[37158]: osdmap e449: 8 total, 8 up, 6 in 2024-02-05T07:34:33.815 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1465190336' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:33.815 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:33.815 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-mon[37158]: 3.13 scrub starts 2024-02-05T07:34:33.815 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:33 smithi007 ceph-mon[37158]: 3.13 scrub ok 2024-02-05T07:34:33.821 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:33.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:33.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:33.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:33.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:34.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:34.121 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:34.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:34.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:34.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:34.260 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:34.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:34.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:34.362 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:34.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:34.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:34.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:34.526 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:34.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:34.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:34.607 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:34:34.607 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:34:34.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:34.675 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:34.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:34.775 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 47 16 77126 77110 102.511 79.625 0.00818601 0.00974983 2024-02-05T07:34:34.776 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:34.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:34.821 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:34.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:34.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:34 smithi007 ceph-mon[29098]: pgmap v819: 175 pgs: 12 peering, 163 active+clean; 4.3 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 55 MiB/s wr, 879 op/s; 13 KiB/s, 0 objects/s recovering 2024-02-05T07:34:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:34 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:34:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:34 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:34:34.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:34 smithi007 ceph-mon[29098]: osdmap e450: 8 total, 8 up, 6 in 2024-02-05T07:34:34.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:34 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:34.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:34 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:34 smithi007 ceph-mon[37158]: pgmap v819: 175 pgs: 12 peering, 163 active+clean; 4.3 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 55 MiB/s wr, 879 op/s; 13 KiB/s, 0 objects/s recovering 2024-02-05T07:34:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:34 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:34:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:34 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:34:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:34 smithi007 ceph-mon[37158]: osdmap e450: 8 total, 8 up, 6 in 2024-02-05T07:34:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:34 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:34 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:34.959 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:34 smithi090 ceph-mon[33213]: pgmap v819: 175 pgs: 12 peering, 163 active+clean; 4.3 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 55 MiB/s wr, 879 op/s; 13 KiB/s, 0 objects/s recovering 2024-02-05T07:34:34.959 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:34 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:34:34.959 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:34 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:34:34.960 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:34 smithi090 ceph-mon[33213]: osdmap e450: 8 total, 8 up, 6 in 2024-02-05T07:34:34.960 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:34 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:34.960 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:34 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:34.971 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:34.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:34.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:35.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:35.142 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:35.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:35.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:35.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:35.262 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:35.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T07:34:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:35.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:35.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:35.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:35.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:35.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:35.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:35.622 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:34:35.623 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:35.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:35.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:35.756 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:35.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:35.774 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 48 16 78292 78276 101.893 72.875 0.0151237 0.00980818 2024-02-05T07:34:35.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:35.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:35.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:35.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:35 smithi090 ceph-mon[33213]: 3.4 scrub starts 2024-02-05T07:34:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:35 smithi090 ceph-mon[33213]: 3.4 scrub ok 2024-02-05T07:34:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:35 smithi090 ceph-mon[33213]: 4.18 deep-scrub starts 2024-02-05T07:34:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:35 smithi090 ceph-mon[33213]: 4.18 deep-scrub ok 2024-02-05T07:34:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:35 smithi090 ceph-mon[33213]: 6.3 deep-scrub starts 2024-02-05T07:34:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:35 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-05T07:34:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:35 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-05T07:34:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:35 smithi090 ceph-mon[33213]: 6.3 deep-scrub ok 2024-02-05T07:34:36.028 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:36.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:36.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:36.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:36.170 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:36.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:36.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[29098]: 3.4 scrub starts 2024-02-05T07:34:36.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[29098]: 3.4 scrub ok 2024-02-05T07:34:36.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[29098]: 4.18 deep-scrub starts 2024-02-05T07:34:36.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[29098]: 4.18 deep-scrub ok 2024-02-05T07:34:36.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[29098]: 6.3 deep-scrub starts 2024-02-05T07:34:36.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-05T07:34:36.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-05T07:34:36.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[29098]: 6.3 deep-scrub ok 2024-02-05T07:34:36.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[37158]: 3.4 scrub starts 2024-02-05T07:34:36.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[37158]: 3.4 scrub ok 2024-02-05T07:34:36.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[37158]: 4.18 deep-scrub starts 2024-02-05T07:34:36.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[37158]: 4.18 deep-scrub ok 2024-02-05T07:34:36.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[37158]: 6.3 deep-scrub starts 2024-02-05T07:34:36.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-05T07:34:36.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-05T07:34:36.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:35 smithi007 ceph-mon[37158]: 6.3 deep-scrub ok 2024-02-05T07:34:36.255 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 3] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:34:36.256 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:34:36.256 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-02-05T07:34:36.256 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-02-05T07:34:36.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:34:36.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:36.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:36.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:36.489 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:36.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:36.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:36.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:36.655 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:36.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:36.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:36.775 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 49 16 79565 79549 101.437 79.5625 0.00536036 0.00985358 2024-02-05T07:34:36.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:36.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:36.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:36.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:36.945 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:36.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:36.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:36.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[29098]: pgmap v821: 175 pgs: 3 peering, 172 active+clean; 4.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 61 MiB/s wr, 969 op/s 2024-02-05T07:34:36.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-05T07:34:36.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[29098]: osdmap e451: 8 total, 8 up, 6 in 2024-02-05T07:34:36.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:34:36.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:34:36.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:36.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[29098]: 1.11 scrub starts 2024-02-05T07:34:36.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[29098]: 1.11 scrub ok 2024-02-05T07:34:36.951 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[37158]: pgmap v821: 175 pgs: 3 peering, 172 active+clean; 4.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 61 MiB/s wr, 969 op/s 2024-02-05T07:34:36.951 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-05T07:34:36.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[37158]: osdmap e451: 8 total, 8 up, 6 in 2024-02-05T07:34:36.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:34:36.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:34:36.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:36.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[37158]: 1.11 scrub starts 2024-02-05T07:34:36.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:36 smithi007 ceph-mon[37158]: 1.11 scrub ok 2024-02-05T07:34:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:36 smithi090 ceph-mon[33213]: pgmap v821: 175 pgs: 3 peering, 172 active+clean; 4.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 61 MiB/s wr, 969 op/s 2024-02-05T07:34:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:36 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-05T07:34:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:36 smithi090 ceph-mon[33213]: osdmap e451: 8 total, 8 up, 6 in 2024-02-05T07:34:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:36 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:34:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:36 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:34:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:36 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:34:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:36 smithi090 ceph-mon[33213]: 1.11 scrub starts 2024-02-05T07:34:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:36 smithi090 ceph-mon[33213]: 1.11 scrub ok 2024-02-05T07:34:37.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:37.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:37.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:37.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:34:37.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:37.284 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:37.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:37.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:37.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:37.432 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:34:37.450 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:37.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:37.457 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:37.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:37.472 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:34:37.518 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:34:37.552 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:34:37.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:37.592 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:37.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:37.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:37.617 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:34:37.661 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:34:37.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:37.705 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:34:37.730 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:37.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:37.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:37.746 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:34:37.753 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:34:37.753 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:34:37.753 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:34:37.753 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:34:37.753 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:34:37.753 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:34:37.753 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:34:37.753 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:34:37.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:37 smithi090 ceph-mon[33213]: osdmap e452: 8 total, 8 up, 6 in 2024-02-05T07:34:37.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:37 smithi090 ceph-mon[33213]: 6.1f scrub starts 2024-02-05T07:34:37.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:37 smithi090 ceph-mon[33213]: 6.1f scrub ok 2024-02-05T07:34:37.775 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:37 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2109453206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:37.775 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 50 16 80706 80690 100.835 71.3125 0.00676659 0.0099121 2024-02-05T07:34:37.816 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:34:37.816 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:34:37.816 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-02-05T07:34:37.816 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:34:37.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:37.869 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:37.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:37.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:37.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:38.013 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:38.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:38.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:37 smithi007 ceph-mon[29098]: osdmap e452: 8 total, 8 up, 6 in 2024-02-05T07:34:38.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:37 smithi007 ceph-mon[29098]: 6.1f scrub starts 2024-02-05T07:34:38.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:37 smithi007 ceph-mon[29098]: 6.1f scrub ok 2024-02-05T07:34:38.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:37 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2109453206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:38.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:37 smithi007 ceph-mon[37158]: osdmap e452: 8 total, 8 up, 6 in 2024-02-05T07:34:38.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:37 smithi007 ceph-mon[37158]: 6.1f scrub starts 2024-02-05T07:34:38.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:37 smithi007 ceph-mon[37158]: 6.1f scrub ok 2024-02-05T07:34:38.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:37 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2109453206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:38.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:38.147 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:38.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:38.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:38.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:38.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:38.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:38.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:38.422 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:38.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:38.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:38.441 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:38.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:38.559 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:38.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:38.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:38.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:38.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:38.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:38.775 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 51 16 81883 81867 100.3 73.5625 0.0118801 0.00996503 2024-02-05T07:34:38.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:38.829 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:38.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:38.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:38.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T07:34:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[29098]: pgmap v824: 175 pgs: 175 active+clean; 4.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-02-05T07:34:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[29098]: 5.5 scrub starts 2024-02-05T07:34:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[29098]: 5.5 scrub ok 2024-02-05T07:34:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[29098]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-02-05T07:34:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:34:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[29098]: osdmap e453: 8 total, 8 up, 6 in 2024-02-05T07:34:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[29098]: 6.d scrub starts 2024-02-05T07:34:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[29098]: 6.b scrub starts 2024-02-05T07:34:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[29098]: 6.d scrub ok 2024-02-05T07:34:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[29098]: 6.b scrub ok 2024-02-05T07:34:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[37158]: pgmap v824: 175 pgs: 175 active+clean; 4.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-02-05T07:34:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[37158]: 5.5 scrub starts 2024-02-05T07:34:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[37158]: 5.5 scrub ok 2024-02-05T07:34:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[37158]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-02-05T07:34:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:34:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[37158]: osdmap e453: 8 total, 8 up, 6 in 2024-02-05T07:34:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[37158]: 6.d scrub starts 2024-02-05T07:34:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[37158]: 6.b scrub starts 2024-02-05T07:34:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[37158]: 6.d scrub ok 2024-02-05T07:34:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:38 smithi007 ceph-mon[37158]: 6.b scrub ok 2024-02-05T07:34:38.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:38.995 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:38.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:38 smithi090 ceph-mon[33213]: pgmap v824: 175 pgs: 175 active+clean; 4.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-02-05T07:34:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:38 smithi090 ceph-mon[33213]: 5.5 scrub starts 2024-02-05T07:34:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:38 smithi090 ceph-mon[33213]: 5.5 scrub ok 2024-02-05T07:34:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:38 smithi090 ceph-mon[33213]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-02-05T07:34:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:38 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:34:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:38 smithi090 ceph-mon[33213]: osdmap e453: 8 total, 8 up, 6 in 2024-02-05T07:34:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:38 smithi090 ceph-mon[33213]: 6.d scrub starts 2024-02-05T07:34:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:38 smithi090 ceph-mon[33213]: 6.b scrub starts 2024-02-05T07:34:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:38 smithi090 ceph-mon[33213]: 6.d scrub ok 2024-02-05T07:34:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:38 smithi090 ceph-mon[33213]: 6.b scrub ok 2024-02-05T07:34:39.096 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:39.131 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:39.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:39.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:39.233 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:39.265 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:39.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:39.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:39.278 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:34:39.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:39.399 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:39.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:39.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:39.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:39.533 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:39.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:39.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:39.635 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:39.682 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:39.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:39.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:39.775 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 52 16 83275 83259 100.044 87 0.0134388 0.00999074 2024-02-05T07:34:39.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:39.817 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:39.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:39.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:39.952 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:39.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:39.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[29098]: 3.19 scrub starts 2024-02-05T07:34:39.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[29098]: 3.19 scrub ok 2024-02-05T07:34:39.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[29098]: osdmap e454: 8 total, 8 up, 6 in 2024-02-05T07:34:39.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[29098]: 4.0 deep-scrub starts 2024-02-05T07:34:39.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[29098]: 4.0 deep-scrub ok 2024-02-05T07:34:39.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[29098]: 1.17 scrub starts 2024-02-05T07:34:39.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[29098]: 1.17 scrub ok 2024-02-05T07:34:39.968 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:34:39.968 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:34:39.968 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[37158]: 3.19 scrub starts 2024-02-05T07:34:39.968 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[37158]: 3.19 scrub ok 2024-02-05T07:34:39.968 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[37158]: osdmap e454: 8 total, 8 up, 6 in 2024-02-05T07:34:39.969 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[37158]: 4.0 deep-scrub starts 2024-02-05T07:34:39.969 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[37158]: 4.0 deep-scrub ok 2024-02-05T07:34:39.969 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[37158]: 1.17 scrub starts 2024-02-05T07:34:39.969 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[37158]: 1.17 scrub ok 2024-02-05T07:34:39.969 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:34:39.969 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:39 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:34:40.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:39 smithi090 ceph-mon[33213]: 3.19 scrub starts 2024-02-05T07:34:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:39 smithi090 ceph-mon[33213]: 3.19 scrub ok 2024-02-05T07:34:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:39 smithi090 ceph-mon[33213]: osdmap e454: 8 total, 8 up, 6 in 2024-02-05T07:34:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:39 smithi090 ceph-mon[33213]: 4.0 deep-scrub starts 2024-02-05T07:34:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:39 smithi090 ceph-mon[33213]: 4.0 deep-scrub ok 2024-02-05T07:34:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:39 smithi090 ceph-mon[33213]: 1.17 scrub starts 2024-02-05T07:34:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:39 smithi090 ceph-mon[33213]: 1.17 scrub ok 2024-02-05T07:34:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:39 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:34:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:39 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:34:40.054 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:40.095 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:40.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:40.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:40.229 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:40.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:40.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:40.330 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:40.372 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:40.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:40.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:40.474 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:40.513 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:40.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:40.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:40.660 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:40.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:40.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:40.775 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 53 16 84625 84609 99.7482 84.375 0.00865459 0.0100206 2024-02-05T07:34:40.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:40.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:40.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:40.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:40.941 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:40.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:40.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:34:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:34:40.988 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:40 smithi090 ceph-mon[33213]: pgmap v827: 174 pgs: 174 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-02-05T07:34:40.988 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:40 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:34:40.988 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:40 smithi090 ceph-mon[33213]: osdmap e455: 8 total, 8 up, 6 in 2024-02-05T07:34:40.988 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:40 smithi090 ceph-mon[33213]: 3.17 deep-scrub starts 2024-02-05T07:34:40.988 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:40 smithi090 ceph-mon[33213]: 3.17 deep-scrub ok 2024-02-05T07:34:40.988 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:40 smithi090 ceph-mon[33213]: 5.9 scrub starts 2024-02-05T07:34:40.988 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:40 smithi090 ceph-mon[33213]: 5.9 scrub ok 2024-02-05T07:34:40.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:34:41.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:40 smithi007 ceph-mon[29098]: pgmap v827: 174 pgs: 174 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-02-05T07:34:41.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:40 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:34:41.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:40 smithi007 ceph-mon[29098]: osdmap e455: 8 total, 8 up, 6 in 2024-02-05T07:34:41.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:40 smithi007 ceph-mon[29098]: 3.17 deep-scrub starts 2024-02-05T07:34:41.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:40 smithi007 ceph-mon[29098]: 3.17 deep-scrub ok 2024-02-05T07:34:41.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:40 smithi007 ceph-mon[29098]: 5.9 scrub starts 2024-02-05T07:34:41.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:40 smithi007 ceph-mon[29098]: 5.9 scrub ok 2024-02-05T07:34:41.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:40 smithi007 ceph-mon[37158]: pgmap v827: 174 pgs: 174 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-02-05T07:34:41.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:40 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:34:41.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:40 smithi007 ceph-mon[37158]: osdmap e455: 8 total, 8 up, 6 in 2024-02-05T07:34:41.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:40 smithi007 ceph-mon[37158]: 3.17 deep-scrub starts 2024-02-05T07:34:41.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:40 smithi007 ceph-mon[37158]: 3.17 deep-scrub ok 2024-02-05T07:34:41.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:40 smithi007 ceph-mon[37158]: 5.9 scrub starts 2024-02-05T07:34:41.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:40 smithi007 ceph-mon[37158]: 5.9 scrub ok 2024-02-05T07:34:41.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:41.092 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:41.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:41.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:41.164 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:34:41.164 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:34:41.164 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:34:41.164 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:34:41.164 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:34:41.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:41.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:41.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:41.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:41.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:41.414 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:41.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:41.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:41.569 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:41.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:41.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:41.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:41.739 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:41.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:41.773 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:41 smithi090 ceph-mon[33213]: osdmap e456: 8 total, 8 up, 6 in 2024-02-05T07:34:41.775 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 54 16 85815 85799 99.2782 74.375 0.015027 0.0100658 2024-02-05T07:34:41.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T07:34:41.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:41.841 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:41 smithi007 ceph-mon[29098]: osdmap e456: 8 total, 8 up, 6 in 2024-02-05T07:34:41.842 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:41 smithi007 ceph-mon[37158]: osdmap e456: 8 total, 8 up, 6 in 2024-02-05T07:34:41.905 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:41.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:41.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:41.982 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:42.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:42.034 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:42.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:42.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:42.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:42.149 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:34:42.192 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:42.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:42.216 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:34:42.227 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:34:42.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:42.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:42.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:42.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:42.453 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:42.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:42.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:42.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:42.584 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:42.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:42.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:42.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:42.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:42.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:34:42.775 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 55 16 86894 86878 98.6993 67.4375 0.0115164 0.0101256 2024-02-05T07:34:42.817 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:42.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:42.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:43.004 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:43.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:43.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:43.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:42 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:42 smithi090 ceph-mon[33213]: pgmap v830: 174 pgs: 2 peering, 172 active+clean; 5.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-02-05T07:34:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:42 smithi090 ceph-mon[33213]: osdmap e457: 8 total, 8 up, 6 in 2024-02-05T07:34:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:42 smithi090 ceph-mon[33213]: 4.1d scrub starts 2024-02-05T07:34:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:42 smithi090 ceph-mon[33213]: 4.1d scrub ok 2024-02-05T07:34:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:42 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1338475442' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:34:43.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:43.149 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:43.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:43.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:42 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:43.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:42 smithi007 ceph-mon[29098]: pgmap v830: 174 pgs: 2 peering, 172 active+clean; 5.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-02-05T07:34:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:42 smithi007 ceph-mon[29098]: osdmap e457: 8 total, 8 up, 6 in 2024-02-05T07:34:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:42 smithi007 ceph-mon[29098]: 4.1d scrub starts 2024-02-05T07:34:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:42 smithi007 ceph-mon[29098]: 4.1d scrub ok 2024-02-05T07:34:43.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:42 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1338475442' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:34:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:42 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:42 smithi007 ceph-mon[37158]: pgmap v830: 174 pgs: 2 peering, 172 active+clean; 5.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-02-05T07:34:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:42 smithi007 ceph-mon[37158]: osdmap e457: 8 total, 8 up, 6 in 2024-02-05T07:34:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:42 smithi007 ceph-mon[37158]: 4.1d scrub starts 2024-02-05T07:34:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:42 smithi007 ceph-mon[37158]: 4.1d scrub ok 2024-02-05T07:34:43.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:42 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1338475442' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:34:43.251 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:43.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:43.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:43.433 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:43.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:43.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:43.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:43.560 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:43.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:43.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:43.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:43.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:43.775 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 56 16 88241 88225 98.4401 84.1875 0.0330133 0.010153 2024-02-05T07:34:43.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:43.825 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:43.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:43.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[29098]: from='client.25517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:34:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[29098]: 3.6 scrub starts 2024-02-05T07:34:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[29098]: 3.6 scrub ok 2024-02-05T07:34:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:34:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1338475442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:34:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[29098]: osdmap e458: 8 total, 8 up, 6 in 2024-02-05T07:34:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[29098]: pgmap v833: 173 pgs: 1 peering, 172 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 972 op/s 2024-02-05T07:34:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-05T07:34:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-05T07:34:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[37158]: from='client.25517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:34:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[37158]: 3.6 scrub starts 2024-02-05T07:34:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[37158]: 3.6 scrub ok 2024-02-05T07:34:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:34:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1338475442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:34:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[37158]: osdmap e458: 8 total, 8 up, 6 in 2024-02-05T07:34:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[37158]: pgmap v833: 173 pgs: 1 peering, 172 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 972 op/s 2024-02-05T07:34:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-05T07:34:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-05T07:34:43.925 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:34:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:34:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:34:43.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:43.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:43 smithi090 ceph-mon[33213]: from='client.25517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:34:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:43 smithi090 ceph-mon[33213]: 3.6 scrub starts 2024-02-05T07:34:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:43 smithi090 ceph-mon[33213]: 3.6 scrub ok 2024-02-05T07:34:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:43 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:34:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:43 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1338475442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:34:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:43 smithi090 ceph-mon[33213]: osdmap e458: 8 total, 8 up, 6 in 2024-02-05T07:34:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:43 smithi090 ceph-mon[33213]: pgmap v833: 173 pgs: 1 peering, 172 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 972 op/s 2024-02-05T07:34:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:43 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-05T07:34:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:43 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-05T07:34:44.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:44.103 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:44.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:44.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:44.244 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:44.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:44.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:44.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:44.376 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:44.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:44.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:44.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:44.501 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:44.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:44.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:44.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:44.634 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:44.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:44.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:44.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:44.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 57 16 89198 89182 97.7624 59.8125 0.00986409 0.0102243 2024-02-05T07:34:44.777 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:44.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:44.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:44.879 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:44.905 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:44.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:44.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:45.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:45.034 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:45.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:45.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:45.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:45.159 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:45.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:45.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:44 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-05T07:34:45.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:44 smithi007 ceph-mon[29098]: osdmap e459: 8 total, 8 up, 6 in 2024-02-05T07:34:45.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:44 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-05T07:34:45.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:44 smithi007 ceph-mon[37158]: osdmap e459: 8 total, 8 up, 6 in 2024-02-05T07:34:45.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:45.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:44 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-05T07:34:45.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:44 smithi090 ceph-mon[33213]: osdmap e459: 8 total, 8 up, 6 in 2024-02-05T07:34:45.292 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:45.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:45.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:45.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:45.435 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:45.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:45.536 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:45.575 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:45.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:45.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:45.676 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:45.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:45.701 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:45.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:45.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 58 16 89821 89805 96.7483 38.9375 0.00830454 0.0103312 2024-02-05T07:34:45.801 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:45.852 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:45.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:45.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:45.990 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:45.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:45.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:46.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:45 smithi007 ceph-mon[37158]: osdmap e460: 8 total, 8 up, 6 in 2024-02-05T07:34:46.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:45 smithi007 ceph-mon[37158]: pgmap v836: 173 pgs: 173 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-05T07:34:46.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:45 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:34:46.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:45 smithi007 ceph-mon[29098]: osdmap e460: 8 total, 8 up, 6 in 2024-02-05T07:34:46.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:45 smithi007 ceph-mon[29098]: pgmap v836: 173 pgs: 173 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-05T07:34:46.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:45 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:34:46.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:46.132 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":831,"stamp":"2024-02-05T07:34:41.758269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5444475228,"num_objects":83265,"num_object_clones":0,"num_object_copies":249795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83265,"num_whiteouts":0,"num_read":1351,"num_read_kb":1103,"num_write":83789,"num_write_kb":5316333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":270260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60203,"ondisk_log_size":60203,"up":519,"acting":519,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":523,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":19807132,"kb_used_data":15874484,"kb_used_omap":13,"kb_used_meta":3932530,"kb_avail":542663780,"statfs":{"total":575970213888,"available":555687710720,"internally_reserved":0,"allocated":16255471616,"data_stored":16252973626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":4026911456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,3,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":402587648,"num_objects":6143,"num_object_clones":0,"num_object_copies":18429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6143,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6143,"num_write_kb":393152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.051730"},"pg_stats":[{"pgid":"10.2","version":"455'5236","reported_seq":5310,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.937579+0000","last_change":"2024-02-05T07:33:38.775613+0000","last_active":"2024-02-05T07:34:39.937579+0000","last_peered":"2024-02-05T07:34:39.937579+0000","last_clean":"2024-02-05T07:34:39.937579+0000","last_became_active":"2024-02-05T07:33:38.775201+0000","last_became_peered":"2024-02-05T07:33:38.775201+0000","last_unstale":"2024-02-05T07:34:39.937579+0000","last_undegraded":"2024-02-05T07:34:39.937579+0000","last_fullsized":"2024-02-05T07:34:39.937579+0000","mapping_epoch":423,"log_start":"428'1700","ondisk_log_start":"428'1700","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:45.287394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343146496,"num_objects":5236,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5236,"num_write_kb":335104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":922,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.709786+0000","last_change":"2024-02-05T07:32:42.818086+0000","last_active":"2024-02-05T07:34:36.709786+0000","last_peered":"2024-02-05T07:34:36.709786+0000","last_clean":"2024-02-05T07:34:36.709786+0000","last_became_active":"2024-02-05T07:31:36.857092+0000","last_became_peered":"2024-02-05T07:31:36.857092+0000","last_unstale":"2024-02-05T07:34:36.709786+0000","last_undegraded":"2024-02-05T07:34:36.709786+0000","last_fullsized":"2024-02-05T07:34:36.709786+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:42.817975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:42.817975+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"326'25","reported_seq":897,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721227+0000","last_change":"2024-02-05T07:32:27.643863+0000","last_active":"2024-02-05T07:34:39.721227+0000","last_peered":"2024-02-05T07:34:39.721227+0000","last_clean":"2024-02-05T07:34:39.721227+0000","last_became_active":"2024-02-05T07:28:26.605082+0000","last_became_peered":"2024-02-05T07:28:26.605082+0000","last_unstale":"2024-02-05T07:34:39.721227+0000","last_undegraded":"2024-02-05T07:34:39.721227+0000","last_fullsized":"2024-02-05T07:34:39.721227+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:32:27.643788+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:32:27.643788+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":398,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.710786+0000","last_change":"2024-02-05T07:33:34.736291+0000","last_active":"2024-02-05T07:34:36.710786+0000","last_peered":"2024-02-05T07:34:36.710786+0000","last_clean":"2024-02-05T07:34:36.710786+0000","last_became_active":"2024-02-05T07:27:16.716461+0000","last_became_peered":"2024-02-05T07:27:16.716461+0000","last_unstale":"2024-02-05T07:34:36.710786+0000","last_undegraded":"2024-02-05T07:34:36.710786+0000","last_fullsized":"2024-02-05T07:34:36.710786+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:34.736160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:34.736160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:00.625101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"365'11","reported_seq":1053,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.733244+0000","last_change":"2024-02-05T07:34:29.851597+0000","last_active":"2024-02-05T07:34:39.733244+0000","last_peered":"2024-02-05T07:34:39.733244+0000","last_clean":"2024-02-05T07:34:39.733244+0000","last_became_active":"2024-02-05T07:30:40.484389+0000","last_became_peered":"2024-02-05T07:30:40.484389+0000","last_unstale":"2024-02-05T07:34:39.733244+0000","last_undegraded":"2024-02-05T07:34:39.733244+0000","last_fullsized":"2024-02-05T07:34:39.733244+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:34:29.851477+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:34:29.851477+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:58.303044+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":809,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.722180+0000","last_change":"2024-02-05T07:32:30.708289+0000","last_active":"2024-02-05T07:34:39.722180+0000","last_peered":"2024-02-05T07:34:39.722180+0000","last_clean":"2024-02-05T07:34:39.722180+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:34:39.722180+0000","last_undegraded":"2024-02-05T07:34:39.722180+0000","last_fullsized":"2024-02-05T07:34:39.722180+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:30.708205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:32:30.708205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"455'5122","reported_seq":5196,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.941039+0000","last_change":"2024-02-05T07:33:38.616173+0000","last_active":"2024-02-05T07:34:39.941039+0000","last_peered":"2024-02-05T07:34:39.941039+0000","last_clean":"2024-02-05T07:34:39.941039+0000","last_became_active":"2024-02-05T07:33:38.615825+0000","last_became_peered":"2024-02-05T07:33:38.615825+0000","last_unstale":"2024-02-05T07:34:39.941039+0000","last_undegraded":"2024-02-05T07:34:39.941039+0000","last_fullsized":"2024-02-05T07:34:39.941039+0000","mapping_epoch":423,"log_start":"427'1400","ondisk_log_start":"427'1400","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3722,"log_dups_size":0,"ondisk_log_size":3722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:53.838130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335675392,"num_objects":5122,"num_object_clones":0,"num_object_copies":15366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":327808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":850,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.735987+0000","last_change":"2024-02-05T07:32:41.228497+0000","last_active":"2024-02-05T07:34:39.735987+0000","last_peered":"2024-02-05T07:34:39.735987+0000","last_clean":"2024-02-05T07:34:39.735987+0000","last_became_active":"2024-02-05T07:31:36.856842+0000","last_became_peered":"2024-02-05T07:31:36.856842+0000","last_unstale":"2024-02-05T07:34:39.735987+0000","last_undegraded":"2024-02-05T07:34:39.735987+0000","last_fullsized":"2024-02-05T07:34:39.735987+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:41.228420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:41.228420+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"326'31","reported_seq":876,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.722033+0000","last_change":"2024-02-05T07:33:30.750667+0000","last_active":"2024-02-05T07:34:39.722033+0000","last_peered":"2024-02-05T07:34:39.722033+0000","last_clean":"2024-02-05T07:34:39.722033+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:34:39.722033+0000","last_undegraded":"2024-02-05T07:34:39.722033+0000","last_fullsized":"2024-02-05T07:34:39.722033+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'31","last_scrub_stamp":"2024-02-05T07:33:30.750548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.750548+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":673,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.718591+0000","last_change":"2024-02-05T07:33:44.229582+0000","last_active":"2024-02-05T07:34:38.718591+0000","last_peered":"2024-02-05T07:34:38.718591+0000","last_clean":"2024-02-05T07:34:38.718591+0000","last_became_active":"2024-02-05T07:28:25.791005+0000","last_became_peered":"2024-02-05T07:28:25.791005+0000","last_unstale":"2024-02-05T07:34:38.718591+0000","last_undegraded":"2024-02-05T07:34:38.718591+0000","last_fullsized":"2024-02-05T07:34:38.718591+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.229489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.229489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.986179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":28,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721172+0000","last_change":"2024-02-05T07:34:30.678802+0000","last_active":"2024-02-05T07:34:39.721172+0000","last_peered":"2024-02-05T07:34:39.721172+0000","last_clean":"2024-02-05T07:34:39.721172+0000","last_became_active":"2024-02-05T07:34:30.678610+0000","last_became_peered":"2024-02-05T07:34:30.678610+0000","last_unstale":"2024-02-05T07:34:39.721172+0000","last_undegraded":"2024-02-05T07:34:39.721172+0000","last_fullsized":"2024-02-05T07:34:39.721172+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.674493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.674493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":833,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.732688+0000","last_change":"2024-02-05T07:34:23.161750+0000","last_active":"2024-02-05T07:34:39.732688+0000","last_peered":"2024-02-05T07:34:39.732688+0000","last_clean":"2024-02-05T07:34:39.732688+0000","last_became_active":"2024-02-05T07:27:16.708888+0000","last_became_peered":"2024-02-05T07:27:16.708888+0000","last_unstale":"2024-02-05T07:34:39.732688+0000","last_undegraded":"2024-02-05T07:34:39.732688+0000","last_fullsized":"2024-02-05T07:34:39.732688+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:23.161521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:23.161521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:46.775844+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"453'5066","reported_seq":5137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.716262+0000","last_change":"2024-02-05T07:33:38.613283+0000","last_active":"2024-02-05T07:34:38.716262+0000","last_peered":"2024-02-05T07:34:38.716262+0000","last_clean":"2024-02-05T07:34:38.716262+0000","last_became_active":"2024-02-05T07:33:38.613051+0000","last_became_peered":"2024-02-05T07:33:38.613051+0000","last_unstale":"2024-02-05T07:34:38.716262+0000","last_undegraded":"2024-02-05T07:34:38.716262+0000","last_fullsized":"2024-02-05T07:34:38.716262+0000","mapping_epoch":423,"log_start":"427'1500","ondisk_log_start":"427'1500","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3566,"log_dups_size":0,"ondisk_log_size":3566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:05.930275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332005376,"num_objects":5066,"num_object_clones":0,"num_object_copies":15198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5066,"num_write_kb":324224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":500,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.741951+0000","last_change":"2024-02-05T07:34:38.728836+0000","last_active":"2024-02-05T07:34:39.741951+0000","last_peered":"2024-02-05T07:34:39.741951+0000","last_clean":"2024-02-05T07:34:39.741951+0000","last_became_active":"2024-02-05T07:34:38.728456+0000","last_became_peered":"2024-02-05T07:34:38.728456+0000","last_unstale":"2024-02-05T07:34:39.741951+0000","last_undegraded":"2024-02-05T07:34:39.741951+0000","last_fullsized":"2024-02-05T07:34:39.741951+0000","mapping_epoch":446,"log_start":"86'128","ondisk_log_start":"86'128","created":291,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:31.816271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:31.816271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:32.017471+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"326'24","reported_seq":863,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.722043+0000","last_change":"2024-02-05T07:29:50.053697+0000","last_active":"2024-02-05T07:34:39.722043+0000","last_peered":"2024-02-05T07:34:39.722043+0000","last_clean":"2024-02-05T07:34:39.722043+0000","last_became_active":"2024-02-05T07:28:25.781007+0000","last_became_peered":"2024-02-05T07:28:25.781007+0000","last_unstale":"2024-02-05T07:34:39.722043+0000","last_undegraded":"2024-02-05T07:34:39.722043+0000","last_fullsized":"2024-02-05T07:34:39.722043+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:29:50.053583+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.053583+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":829,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720639+0000","last_change":"2024-02-05T07:32:28.670360+0000","last_active":"2024-02-05T07:34:39.720639+0000","last_peered":"2024-02-05T07:34:39.720639+0000","last_clean":"2024-02-05T07:34:39.720639+0000","last_became_active":"2024-02-05T07:30:40.486134+0000","last_became_peered":"2024-02-05T07:30:40.486134+0000","last_unstale":"2024-02-05T07:34:39.720639+0000","last_undegraded":"2024-02-05T07:34:39.720639+0000","last_fullsized":"2024-02-05T07:34:39.720639+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:28.670286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:32:28.670286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":649,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721492+0000","last_change":"2024-02-05T07:29:45.012958+0000","last_active":"2024-02-05T07:34:39.721492+0000","last_peered":"2024-02-05T07:34:39.721492+0000","last_clean":"2024-02-05T07:34:39.721492+0000","last_became_active":"2024-02-05T07:28:26.601713+0000","last_became_peered":"2024-02-05T07:28:26.601713+0000","last_unstale":"2024-02-05T07:34:39.721492+0000","last_undegraded":"2024-02-05T07:34:39.721492+0000","last_fullsized":"2024-02-05T07:34:39.721492+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.012821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":407,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.710878+0000","last_change":"2024-02-05T07:33:28.696549+0000","last_active":"2024-02-05T07:34:36.710878+0000","last_peered":"2024-02-05T07:34:36.710878+0000","last_clean":"2024-02-05T07:34:36.710878+0000","last_became_active":"2024-02-05T07:30:40.498529+0000","last_became_peered":"2024-02-05T07:30:40.498529+0000","last_unstale":"2024-02-05T07:34:36.710878+0000","last_undegraded":"2024-02-05T07:34:36.710878+0000","last_fullsized":"2024-02-05T07:34:36.710878+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.696462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.696462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"455'5223","reported_seq":5297,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:40.365869+0000","last_change":"2024-02-05T07:33:38.778383+0000","last_active":"2024-02-05T07:34:40.365869+0000","last_peered":"2024-02-05T07:34:40.365869+0000","last_clean":"2024-02-05T07:34:40.365869+0000","last_became_active":"2024-02-05T07:33:38.778162+0000","last_became_peered":"2024-02-05T07:33:38.778162+0000","last_unstale":"2024-02-05T07:34:40.365869+0000","last_undegraded":"2024-02-05T07:34:40.365869+0000","last_fullsized":"2024-02-05T07:34:40.365869+0000","mapping_epoch":423,"log_start":"428'1828","ondisk_log_start":"428'1828","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3395,"log_dups_size":0,"ondisk_log_size":3395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.483201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342294528,"num_objects":5223,"num_object_clones":0,"num_object_copies":15669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5223,"num_write_kb":334272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":822,"reported_epoch":454,"state":"peering","last_fresh":"2024-02-05T07:34:38.721428+0000","last_change":"2024-02-05T07:34:39.734312+0000","last_active":"2024-02-05T07:34:39.734312+0000","last_peered":"2024-02-05T07:34:38.721428+0000","last_clean":"2024-02-05T07:34:38.721428+0000","last_became_active":"2024-02-05T07:31:36.839888+0000","last_became_peered":"2024-02-05T07:31:36.839888+0000","last_unstale":"2024-02-05T07:34:38.721428+0000","last_undegraded":"2024-02-05T07:34:38.721428+0000","last_fullsized":"2024-02-05T07:34:38.721428+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:40.216973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:40.216973+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"326'34","reported_seq":913,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.723968+0000","last_change":"2024-02-05T07:30:30.006087+0000","last_active":"2024-02-05T07:34:39.723968+0000","last_peered":"2024-02-05T07:34:39.723968+0000","last_clean":"2024-02-05T07:34:39.723968+0000","last_became_active":"2024-02-05T07:27:16.768454+0000","last_became_peered":"2024-02-05T07:27:16.768454+0000","last_unstale":"2024-02-05T07:34:39.723968+0000","last_undegraded":"2024-02-05T07:34:39.723968+0000","last_fullsized":"2024-02-05T07:34:39.723968+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'34","last_scrub_stamp":"2024-02-05T07:30:30.005988+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.005988+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":404,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.709068+0000","last_change":"2024-02-05T07:30:40.508975+0000","last_active":"2024-02-05T07:34:36.709068+0000","last_peered":"2024-02-05T07:34:36.709068+0000","last_clean":"2024-02-05T07:34:36.709068+0000","last_became_active":"2024-02-05T07:30:40.508561+0000","last_became_peered":"2024-02-05T07:30:40.508561+0000","last_unstale":"2024-02-05T07:34:36.709068+0000","last_undegraded":"2024-02-05T07:34:36.709068+0000","last_fullsized":"2024-02-05T07:34:36.709068+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:30.982457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.982457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"257'9","reported_seq":1023,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.722173+0000","last_change":"2024-02-05T07:32:39.565923+0000","last_active":"2024-02-05T07:34:39.722173+0000","last_peered":"2024-02-05T07:34:39.722173+0000","last_clean":"2024-02-05T07:34:39.722173+0000","last_became_active":"2024-02-05T07:27:03.375195+0000","last_became_peered":"2024-02-05T07:27:03.375195+0000","last_unstale":"2024-02-05T07:34:39.722173+0000","last_undegraded":"2024-02-05T07:34:39.722173+0000","last_fullsized":"2024-02-05T07:34:39.722173+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'9","last_scrub_stamp":"2024-02-05T07:32:39.565833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:39.565833+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":39,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.732846+0000","last_change":"2024-02-05T07:34:38.025632+0000","last_active":"2024-02-05T07:34:39.732846+0000","last_peered":"2024-02-05T07:34:39.732846+0000","last_clean":"2024-02-05T07:34:39.732846+0000","last_became_active":"2024-02-05T07:34:30.685912+0000","last_became_peered":"2024-02-05T07:34:30.685912+0000","last_unstale":"2024-02-05T07:34:39.732846+0000","last_undegraded":"2024-02-05T07:34:39.732846+0000","last_fullsized":"2024-02-05T07:34:39.732846+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:38.025380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.025380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:40.345829+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.e","version":"455'5330","reported_seq":5404,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:40.378150+0000","last_change":"2024-02-05T07:33:38.613937+0000","last_active":"2024-02-05T07:34:40.378150+0000","last_peered":"2024-02-05T07:34:40.378150+0000","last_clean":"2024-02-05T07:34:40.378150+0000","last_became_active":"2024-02-05T07:33:38.613669+0000","last_became_peered":"2024-02-05T07:33:38.613669+0000","last_unstale":"2024-02-05T07:34:40.378150+0000","last_undegraded":"2024-02-05T07:34:40.378150+0000","last_fullsized":"2024-02-05T07:34:40.378150+0000","mapping_epoch":423,"log_start":"428'2000","ondisk_log_start":"428'2000","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3330,"log_dups_size":0,"ondisk_log_size":3330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:46.055957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349306880,"num_objects":5330,"num_object_clones":0,"num_object_copies":15990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5330,"num_write_kb":341120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":819,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721563+0000","last_change":"2024-02-05T07:32:24.656130+0000","last_active":"2024-02-05T07:34:39.721563+0000","last_peered":"2024-02-05T07:34:39.721563+0000","last_clean":"2024-02-05T07:34:39.721563+0000","last_became_active":"2024-02-05T07:31:36.758820+0000","last_became_peered":"2024-02-05T07:31:36.758820+0000","last_unstale":"2024-02-05T07:34:39.721563+0000","last_undegraded":"2024-02-05T07:34:39.721563+0000","last_fullsized":"2024-02-05T07:34:39.721563+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:24.656057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:24.656057+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":833,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720590+0000","last_change":"2024-02-05T07:32:36.623960+0000","last_active":"2024-02-05T07:34:39.720590+0000","last_peered":"2024-02-05T07:34:39.720590+0000","last_clean":"2024-02-05T07:34:39.720590+0000","last_became_active":"2024-02-05T07:30:40.489212+0000","last_became_peered":"2024-02-05T07:30:40.489212+0000","last_unstale":"2024-02-05T07:34:39.720590+0000","last_undegraded":"2024-02-05T07:34:39.720590+0000","last_fullsized":"2024-02-05T07:34:39.720590+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:36.623865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:36.623865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"326'30","reported_seq":458,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.708944+0000","last_change":"2024-02-05T07:30:40.543094+0000","last_active":"2024-02-05T07:34:36.708944+0000","last_peered":"2024-02-05T07:34:36.708944+0000","last_clean":"2024-02-05T07:34:36.708944+0000","last_became_active":"2024-02-05T07:30:40.496940+0000","last_became_peered":"2024-02-05T07:30:40.496940+0000","last_unstale":"2024-02-05T07:34:36.708944+0000","last_undegraded":"2024-02-05T07:34:36.708944+0000","last_fullsized":"2024-02-05T07:34:36.708944+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-02-05T07:28:45.029295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:28:45.029295+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":404,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.717863+0000","last_change":"2024-02-05T07:33:48.701618+0000","last_active":"2024-02-05T07:34:36.717863+0000","last_peered":"2024-02-05T07:34:36.717863+0000","last_clean":"2024-02-05T07:34:36.717863+0000","last_became_active":"2024-02-05T07:30:40.508607+0000","last_became_peered":"2024-02-05T07:30:40.508607+0000","last_unstale":"2024-02-05T07:34:36.717863+0000","last_undegraded":"2024-02-05T07:34:36.717863+0000","last_fullsized":"2024-02-05T07:34:36.717863+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:48.701399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:48.701399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:03.427068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":826,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721846+0000","last_change":"2024-02-05T07:33:45.908153+0000","last_active":"2024-02-05T07:34:39.721846+0000","last_peered":"2024-02-05T07:34:39.721846+0000","last_clean":"2024-02-05T07:34:39.721846+0000","last_became_active":"2024-02-05T07:27:16.706666+0000","last_became_peered":"2024-02-05T07:27:16.706666+0000","last_unstale":"2024-02-05T07:34:39.721846+0000","last_undegraded":"2024-02-05T07:34:39.721846+0000","last_fullsized":"2024-02-05T07:34:39.721846+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:45.908064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:33:45.908064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:09.071480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"453'5082","reported_seq":5152,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.650193+0000","last_change":"2024-02-05T07:33:38.614195+0000","last_active":"2024-02-05T07:34:38.650193+0000","last_peered":"2024-02-05T07:34:38.650193+0000","last_clean":"2024-02-05T07:34:38.650193+0000","last_became_active":"2024-02-05T07:33:38.613892+0000","last_became_peered":"2024-02-05T07:33:38.613892+0000","last_unstale":"2024-02-05T07:34:38.650193+0000","last_undegraded":"2024-02-05T07:34:38.650193+0000","last_fullsized":"2024-02-05T07:34:38.650193+0000","mapping_epoch":423,"log_start":"427'1500","ondisk_log_start":"427'1500","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3582,"log_dups_size":0,"ondisk_log_size":3582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.064473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333053952,"num_objects":5082,"num_object_clones":0,"num_object_copies":15246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5082,"num_write_kb":325248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":1076,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.711432+0000","last_change":"2024-02-05T07:34:28.874781+0000","last_active":"2024-02-05T07:34:36.711432+0000","last_peered":"2024-02-05T07:34:36.711432+0000","last_clean":"2024-02-05T07:34:36.711432+0000","last_became_active":"2024-02-05T07:30:43.524588+0000","last_became_peered":"2024-02-05T07:30:43.524588+0000","last_unstale":"2024-02-05T07:34:36.711432+0000","last_undegraded":"2024-02-05T07:34:36.711432+0000","last_fullsized":"2024-02-05T07:34:36.711432+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:28.874337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.874337+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:47.754286+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"266'10","reported_seq":490,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.709168+0000","last_change":"2024-02-05T07:33:42.715242+0000","last_active":"2024-02-05T07:34:36.709168+0000","last_peered":"2024-02-05T07:34:36.709168+0000","last_clean":"2024-02-05T07:34:36.709168+0000","last_became_active":"2024-02-05T07:27:16.800207+0000","last_became_peered":"2024-02-05T07:27:16.800207+0000","last_unstale":"2024-02-05T07:34:36.709168+0000","last_undegraded":"2024-02-05T07:34:36.709168+0000","last_fullsized":"2024-02-05T07:34:36.709168+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'10","last_scrub_stamp":"2024-02-05T07:33:42.715162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:42.715162+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:56.032172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"326'35","reported_seq":889,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720686+0000","last_change":"2024-02-05T07:29:45.972745+0000","last_active":"2024-02-05T07:34:39.720686+0000","last_peered":"2024-02-05T07:34:39.720686+0000","last_clean":"2024-02-05T07:34:39.720686+0000","last_became_active":"2024-02-05T07:28:26.598831+0000","last_became_peered":"2024-02-05T07:28:26.598831+0000","last_unstale":"2024-02-05T07:34:39.720686+0000","last_undegraded":"2024-02-05T07:34:39.720686+0000","last_fullsized":"2024-02-05T07:34:39.720686+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:29:45.972589+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.972589+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":576,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.725132+0000","last_change":"2024-02-05T07:29:41.587097+0000","last_active":"2024-02-05T07:34:39.725132+0000","last_peered":"2024-02-05T07:34:39.725132+0000","last_clean":"2024-02-05T07:34:39.725132+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:34:39.725132+0000","last_undegraded":"2024-02-05T07:34:39.725132+0000","last_fullsized":"2024-02-05T07:34:39.725132+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:29:41.587022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:41.587022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":693,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.718953+0000","last_change":"2024-02-05T07:34:35.339060+0000","last_active":"2024-02-05T07:34:38.718953+0000","last_peered":"2024-02-05T07:34:38.718953+0000","last_clean":"2024-02-05T07:34:38.718953+0000","last_became_active":"2024-02-05T07:30:40.489120+0000","last_became_peered":"2024-02-05T07:30:40.489120+0000","last_unstale":"2024-02-05T07:34:38.718953+0000","last_undegraded":"2024-02-05T07:34:38.718953+0000","last_fullsized":"2024-02-05T07:34:38.718953+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:35.338979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:34:35.338979+0000","last_clean_scrub_stamp":"2024-02-05T07:34:35.338979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:01.319474+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"455'5408","reported_seq":5482,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:40.370249+0000","last_change":"2024-02-05T07:33:38.775156+0000","last_active":"2024-02-05T07:34:40.370249+0000","last_peered":"2024-02-05T07:34:40.370249+0000","last_clean":"2024-02-05T07:34:40.370249+0000","last_became_active":"2024-02-05T07:33:38.774713+0000","last_became_peered":"2024-02-05T07:33:38.774713+0000","last_unstale":"2024-02-05T07:34:40.370249+0000","last_undegraded":"2024-02-05T07:34:40.370249+0000","last_fullsized":"2024-02-05T07:34:40.370249+0000","mapping_epoch":423,"log_start":"428'2100","ondisk_log_start":"428'2100","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3308,"log_dups_size":0,"ondisk_log_size":3308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:05.742929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354418688,"num_objects":5408,"num_object_clones":0,"num_object_copies":16224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5408,"num_write_kb":346112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":686,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-05T07:34:37.715734+0000","last_change":"2024-02-05T07:34:29.214764+0000","last_active":"2024-02-05T07:34:37.715734+0000","last_peered":"2024-02-05T07:34:37.715734+0000","last_clean":"2024-02-05T07:34:37.715734+0000","last_became_active":"2024-02-05T07:31:36.850075+0000","last_became_peered":"2024-02-05T07:31:36.850075+0000","last_unstale":"2024-02-05T07:34:37.715734+0000","last_undegraded":"2024-02-05T07:34:37.715734+0000","last_fullsized":"2024-02-05T07:34:37.715734+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:29.214474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:29.214474+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:33.064919+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":487,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.719157+0000","last_change":"2024-02-05T07:32:49.752237+0000","last_active":"2024-02-05T07:34:38.719157+0000","last_peered":"2024-02-05T07:34:38.719157+0000","last_clean":"2024-02-05T07:34:38.719157+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:34:38.719157+0000","last_undegraded":"2024-02-05T07:34:38.719157+0000","last_fullsized":"2024-02-05T07:34:38.719157+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:32:49.752163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:49.752163+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"326'25","reported_seq":881,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.724358+0000","last_change":"2024-02-05T07:30:36.908113+0000","last_active":"2024-02-05T07:34:39.724358+0000","last_peered":"2024-02-05T07:34:39.724358+0000","last_clean":"2024-02-05T07:34:39.724358+0000","last_became_active":"2024-02-05T07:27:03.361342+0000","last_became_peered":"2024-02-05T07:27:03.361342+0000","last_unstale":"2024-02-05T07:34:39.724358+0000","last_undegraded":"2024-02-05T07:34:39.724358+0000","last_fullsized":"2024-02-05T07:34:39.724358+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:30:36.907989+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.907989+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":401,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.711676+0000","last_change":"2024-02-05T07:29:44.015577+0000","last_active":"2024-02-05T07:34:36.711676+0000","last_peered":"2024-02-05T07:34:36.711676+0000","last_clean":"2024-02-05T07:34:36.711676+0000","last_became_active":"2024-02-05T07:28:26.596096+0000","last_became_peered":"2024-02-05T07:28:26.596096+0000","last_unstale":"2024-02-05T07:34:36.711676+0000","last_undegraded":"2024-02-05T07:34:36.711676+0000","last_fullsized":"2024-02-05T07:34:36.711676+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:44.015468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:29:44.015468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":103,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.724615+0000","last_change":"2024-02-05T07:33:24.419375+0000","last_active":"2024-02-05T07:34:39.724615+0000","last_peered":"2024-02-05T07:34:39.724615+0000","last_clean":"2024-02-05T07:34:39.724615+0000","last_became_active":"2024-02-05T07:32:58.598709+0000","last_became_peered":"2024-02-05T07:32:58.598709+0000","last_unstale":"2024-02-05T07:34:39.724615+0000","last_undegraded":"2024-02-05T07:34:39.724615+0000","last_fullsized":"2024-02-05T07:34:39.724615+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:24.419250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:24.419250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"453'5101","reported_seq":5172,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.718520+0000","last_change":"2024-02-05T07:33:38.775383+0000","last_active":"2024-02-05T07:34:38.718520+0000","last_peered":"2024-02-05T07:34:38.718520+0000","last_clean":"2024-02-05T07:34:38.718520+0000","last_became_active":"2024-02-05T07:33:38.775048+0000","last_became_peered":"2024-02-05T07:33:38.775048+0000","last_unstale":"2024-02-05T07:34:38.718520+0000","last_undegraded":"2024-02-05T07:34:38.718520+0000","last_fullsized":"2024-02-05T07:34:38.718520+0000","mapping_epoch":423,"log_start":"427'1500","ondisk_log_start":"427'1500","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:00.065375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334299136,"num_objects":5101,"num_object_clones":0,"num_object_copies":15303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5101,"num_write_kb":326464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":679,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.719978+0000","last_change":"2024-02-05T07:33:47.197319+0000","last_active":"2024-02-05T07:34:38.719978+0000","last_peered":"2024-02-05T07:34:38.719978+0000","last_clean":"2024-02-05T07:34:38.719978+0000","last_became_active":"2024-02-05T07:31:36.840534+0000","last_became_peered":"2024-02-05T07:31:36.840534+0000","last_unstale":"2024-02-05T07:34:38.719978+0000","last_undegraded":"2024-02-05T07:34:38.719978+0000","last_fullsized":"2024-02-05T07:34:38.719978+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:47.197225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.197225+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:01.748450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":661,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.717168+0000","last_change":"2024-02-05T07:33:33.212392+0000","last_active":"2024-02-05T07:34:38.717168+0000","last_peered":"2024-02-05T07:34:38.717168+0000","last_clean":"2024-02-05T07:34:38.717168+0000","last_became_active":"2024-02-05T07:28:26.589673+0000","last_became_peered":"2024-02-05T07:28:26.589673+0000","last_unstale":"2024-02-05T07:34:38.717168+0000","last_undegraded":"2024-02-05T07:34:38.717168+0000","last_fullsized":"2024-02-05T07:34:38.717168+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:33.212264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.212264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:58.757962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"326'29","reported_seq":449,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.725657+0000","last_change":"2024-02-05T07:30:40.516301+0000","last_active":"2024-02-05T07:34:39.725657+0000","last_peered":"2024-02-05T07:34:39.725657+0000","last_clean":"2024-02-05T07:34:39.725657+0000","last_became_active":"2024-02-05T07:30:40.481234+0000","last_became_peered":"2024-02-05T07:30:40.481234+0000","last_unstale":"2024-02-05T07:34:39.725657+0000","last_undegraded":"2024-02-05T07:34:39.725657+0000","last_fullsized":"2024-02-05T07:34:39.725657+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'29","last_scrub_stamp":"2024-02-05T07:30:37.106672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:37.106672+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":447,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.731600+0000","last_change":"2024-02-05T07:34:34.799486+0000","last_active":"2024-02-05T07:34:39.731600+0000","last_peered":"2024-02-05T07:34:39.731600+0000","last_clean":"2024-02-05T07:34:39.731600+0000","last_became_active":"2024-02-05T07:34:30.705994+0000","last_became_peered":"2024-02-05T07:34:30.705994+0000","last_unstale":"2024-02-05T07:34:39.731600+0000","last_undegraded":"2024-02-05T07:34:39.731600+0000","last_fullsized":"2024-02-05T07:34:39.731600+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:34.799346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:34:34.799346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:43.527345+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":821,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.724470+0000","last_change":"2024-02-05T07:33:33.257106+0000","last_active":"2024-02-05T07:34:39.724470+0000","last_peered":"2024-02-05T07:34:39.724470+0000","last_clean":"2024-02-05T07:34:39.724470+0000","last_became_active":"2024-02-05T07:28:25.840850+0000","last_became_peered":"2024-02-05T07:28:25.840850+0000","last_unstale":"2024-02-05T07:34:39.724470+0000","last_undegraded":"2024-02-05T07:34:39.724470+0000","last_fullsized":"2024-02-05T07:34:39.724470+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:33.257028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.257028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.513556+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"455'5288","reported_seq":5362,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.930218+0000","last_change":"2024-02-05T07:33:38.777030+0000","last_active":"2024-02-05T07:34:39.930218+0000","last_peered":"2024-02-05T07:34:39.930218+0000","last_clean":"2024-02-05T07:34:39.930218+0000","last_became_active":"2024-02-05T07:33:38.776442+0000","last_became_peered":"2024-02-05T07:33:38.776442+0000","last_unstale":"2024-02-05T07:34:39.930218+0000","last_undegraded":"2024-02-05T07:34:39.930218+0000","last_fullsized":"2024-02-05T07:34:39.930218+0000","mapping_epoch":423,"log_start":"428'1600","ondisk_log_start":"428'1600","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:46.941588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346554368,"num_objects":5288,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5288,"num_write_kb":338432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":338,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.719213+0000","last_change":"2024-02-05T07:34:31.185521+0000","last_active":"2024-02-05T07:34:38.719213+0000","last_peered":"2024-02-05T07:34:38.719213+0000","last_clean":"2024-02-05T07:34:38.719213+0000","last_became_active":"2024-02-05T07:34:30.653119+0000","last_became_peered":"2024-02-05T07:34:30.653119+0000","last_unstale":"2024-02-05T07:34:38.719213+0000","last_undegraded":"2024-02-05T07:34:38.719213+0000","last_fullsized":"2024-02-05T07:34:38.719213+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:31.185403+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:34:31.185403+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:32.852955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":421,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.724963+0000","last_change":"2024-02-05T07:32:46.399673+0000","last_active":"2024-02-05T07:34:39.724963+0000","last_peered":"2024-02-05T07:34:39.724963+0000","last_clean":"2024-02-05T07:34:39.724963+0000","last_became_active":"2024-02-05T07:27:03.370297+0000","last_became_peered":"2024-02-05T07:27:03.370297+0000","last_unstale":"2024-02-05T07:34:39.724963+0000","last_undegraded":"2024-02-05T07:34:39.724963+0000","last_fullsized":"2024-02-05T07:34:39.724963+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:46.399572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:46.399572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"326'24","reported_seq":381,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.732420+0000","last_change":"2024-02-05T07:34:30.678315+0000","last_active":"2024-02-05T07:34:39.732420+0000","last_peered":"2024-02-05T07:34:39.732420+0000","last_clean":"2024-02-05T07:34:39.732420+0000","last_became_active":"2024-02-05T07:34:30.677886+0000","last_became_peered":"2024-02-05T07:34:30.677886+0000","last_unstale":"2024-02-05T07:34:39.732420+0000","last_undegraded":"2024-02-05T07:34:39.732420+0000","last_fullsized":"2024-02-05T07:34:39.732420+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:33:49.499846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:49.499846+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:54.262066+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":477,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.725019+0000","last_change":"2024-02-05T07:30:40.520395+0000","last_active":"2024-02-05T07:34:39.725019+0000","last_peered":"2024-02-05T07:34:39.725019+0000","last_clean":"2024-02-05T07:34:39.725019+0000","last_became_active":"2024-02-05T07:30:40.493770+0000","last_became_peered":"2024-02-05T07:30:40.493770+0000","last_unstale":"2024-02-05T07:34:39.725019+0000","last_undegraded":"2024-02-05T07:34:39.725019+0000","last_fullsized":"2024-02-05T07:34:39.725019+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:29:51.291146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.291146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":245,"num_read_kb":240,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":795,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720732+0000","last_change":"2024-02-05T07:33:39.809630+0000","last_active":"2024-02-05T07:34:39.720732+0000","last_peered":"2024-02-05T07:34:39.720732+0000","last_clean":"2024-02-05T07:34:39.720732+0000","last_became_active":"2024-02-05T07:27:16.724280+0000","last_became_peered":"2024-02-05T07:27:16.724280+0000","last_unstale":"2024-02-05T07:34:39.720732+0000","last_undegraded":"2024-02-05T07:34:39.720732+0000","last_fullsized":"2024-02-05T07:34:39.720732+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:39.809544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:33:39.809544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:03.386365+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":849,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.724998+0000","last_change":"2024-02-05T07:33:41.379002+0000","last_active":"2024-02-05T07:34:39.724998+0000","last_peered":"2024-02-05T07:34:39.724998+0000","last_clean":"2024-02-05T07:34:39.724998+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:34:39.724998+0000","last_undegraded":"2024-02-05T07:34:39.724998+0000","last_fullsized":"2024-02-05T07:34:39.724998+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:33:41.378909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.378909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:42.516508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"453'5088","reported_seq":5158,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.718554+0000","last_change":"2024-02-05T07:33:38.775761+0000","last_active":"2024-02-05T07:34:38.718554+0000","last_peered":"2024-02-05T07:34:38.718554+0000","last_clean":"2024-02-05T07:34:38.718554+0000","last_became_active":"2024-02-05T07:33:38.775338+0000","last_became_peered":"2024-02-05T07:33:38.775338+0000","last_unstale":"2024-02-05T07:34:38.718554+0000","last_undegraded":"2024-02-05T07:34:38.718554+0000","last_fullsized":"2024-02-05T07:34:38.718554+0000","mapping_epoch":423,"log_start":"427'1500","ondisk_log_start":"427'1500","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3588,"log_dups_size":0,"ondisk_log_size":3588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:50.485312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333447168,"num_objects":5088,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5088,"num_write_kb":325632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":595,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.731944+0000","last_change":"2024-02-05T07:33:50.943755+0000","last_active":"2024-02-05T07:34:39.731944+0000","last_peered":"2024-02-05T07:34:39.731944+0000","last_clean":"2024-02-05T07:34:39.731944+0000","last_became_active":"2024-02-05T07:30:40.474169+0000","last_became_peered":"2024-02-05T07:30:40.474169+0000","last_unstale":"2024-02-05T07:34:39.731944+0000","last_undegraded":"2024-02-05T07:34:39.731944+0000","last_fullsized":"2024-02-05T07:34:39.731944+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:50.943640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:50.943640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:06.509391+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"365'31","reported_seq":899,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.718716+0000","last_change":"2024-02-05T07:33:46.200250+0000","last_active":"2024-02-05T07:34:38.718716+0000","last_peered":"2024-02-05T07:34:38.718716+0000","last_clean":"2024-02-05T07:34:38.718716+0000","last_became_active":"2024-02-05T07:30:40.508686+0000","last_became_peered":"2024-02-05T07:30:40.508686+0000","last_unstale":"2024-02-05T07:34:38.718716+0000","last_undegraded":"2024-02-05T07:34:38.718716+0000","last_fullsized":"2024-02-05T07:34:38.718716+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'31","last_scrub_stamp":"2024-02-05T07:33:46.200150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.200150+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.420503+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":401,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.710093+0000","last_change":"2024-02-05T07:29:43.000854+0000","last_active":"2024-02-05T07:34:36.710093+0000","last_peered":"2024-02-05T07:34:36.710093+0000","last_clean":"2024-02-05T07:34:36.710093+0000","last_became_active":"2024-02-05T07:28:26.595640+0000","last_became_peered":"2024-02-05T07:28:26.595640+0000","last_unstale":"2024-02-05T07:34:36.710093+0000","last_undegraded":"2024-02-05T07:34:36.710093+0000","last_fullsized":"2024-02-05T07:34:36.710093+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:43.000778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:43.000778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":524,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.716854+0000","last_change":"2024-02-05T07:33:48.221978+0000","last_active":"2024-02-05T07:34:38.716854+0000","last_peered":"2024-02-05T07:34:38.716854+0000","last_clean":"2024-02-05T07:34:38.716854+0000","last_became_active":"2024-02-05T07:30:43.516031+0000","last_became_peered":"2024-02-05T07:30:43.516031+0000","last_unstale":"2024-02-05T07:34:38.716854+0000","last_undegraded":"2024-02-05T07:34:38.716854+0000","last_fullsized":"2024-02-05T07:34:38.716854+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:33:48.221845+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:09.727599+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":398,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.717773+0000","last_change":"2024-02-05T07:33:46.760555+0000","last_active":"2024-02-05T07:34:36.717773+0000","last_peered":"2024-02-05T07:34:36.717773+0000","last_clean":"2024-02-05T07:34:36.717773+0000","last_became_active":"2024-02-05T07:27:16.751341+0000","last_became_peered":"2024-02-05T07:27:16.751341+0000","last_unstale":"2024-02-05T07:34:36.717773+0000","last_undegraded":"2024-02-05T07:34:36.717773+0000","last_fullsized":"2024-02-05T07:34:36.717773+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:46.760448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.760448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:01.541210+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"455'5215","reported_seq":5289,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.921826+0000","last_change":"2024-02-05T07:33:38.615853+0000","last_active":"2024-02-05T07:34:39.921826+0000","last_peered":"2024-02-05T07:34:39.921826+0000","last_clean":"2024-02-05T07:34:39.921826+0000","last_became_active":"2024-02-05T07:33:38.615574+0000","last_became_peered":"2024-02-05T07:33:38.615574+0000","last_unstale":"2024-02-05T07:34:39.921826+0000","last_undegraded":"2024-02-05T07:34:39.921826+0000","last_fullsized":"2024-02-05T07:34:39.921826+0000","mapping_epoch":423,"log_start":"427'1500","ondisk_log_start":"427'1500","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3715,"log_dups_size":0,"ondisk_log_size":3715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:58.336758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341770240,"num_objects":5215,"num_object_clones":0,"num_object_copies":15645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5215,"num_write_kb":333760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":829,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720668+0000","last_change":"2024-02-05T07:32:21.667678+0000","last_active":"2024-02-05T07:34:39.720668+0000","last_peered":"2024-02-05T07:34:39.720668+0000","last_clean":"2024-02-05T07:34:39.720668+0000","last_became_active":"2024-02-05T07:30:43.509733+0000","last_became_peered":"2024-02-05T07:30:43.509733+0000","last_unstale":"2024-02-05T07:34:39.720668+0000","last_undegraded":"2024-02-05T07:34:39.720668+0000","last_fullsized":"2024-02-05T07:34:39.720668+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:21.667581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:21.667581+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":835,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.732641+0000","last_change":"2024-02-05T07:33:23.118646+0000","last_active":"2024-02-05T07:34:39.732641+0000","last_peered":"2024-02-05T07:34:39.732641+0000","last_clean":"2024-02-05T07:34:39.732641+0000","last_became_active":"2024-02-05T07:30:40.503816+0000","last_became_peered":"2024-02-05T07:30:40.503816+0000","last_unstale":"2024-02-05T07:34:39.732641+0000","last_undegraded":"2024-02-05T07:34:39.732641+0000","last_fullsized":"2024-02-05T07:34:39.732641+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:23.118569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:23.118569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"326'22","reported_seq":894,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.724743+0000","last_change":"2024-02-05T07:33:30.327114+0000","last_active":"2024-02-05T07:34:39.724743+0000","last_peered":"2024-02-05T07:34:39.724743+0000","last_clean":"2024-02-05T07:34:39.724743+0000","last_became_active":"2024-02-05T07:30:40.494460+0000","last_became_peered":"2024-02-05T07:30:40.494460+0000","last_unstale":"2024-02-05T07:34:39.724743+0000","last_undegraded":"2024-02-05T07:34:39.724743+0000","last_fullsized":"2024-02-05T07:34:39.724743+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'22","last_scrub_stamp":"2024-02-05T07:33:30.327027+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.327027+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:47.402987+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":860,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.731704+0000","last_change":"2024-02-05T07:33:42.096031+0000","last_active":"2024-02-05T07:34:39.731704+0000","last_peered":"2024-02-05T07:34:39.731704+0000","last_clean":"2024-02-05T07:34:39.731704+0000","last_became_active":"2024-02-05T07:27:16.728869+0000","last_became_peered":"2024-02-05T07:27:16.728869+0000","last_unstale":"2024-02-05T07:34:39.731704+0000","last_undegraded":"2024-02-05T07:34:39.731704+0000","last_fullsized":"2024-02-05T07:34:39.731704+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:42.095953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:33:42.095953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:55.145283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":36,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.736220+0000","last_change":"2024-02-05T07:34:30.668997+0000","last_active":"2024-02-05T07:34:39.736220+0000","last_peered":"2024-02-05T07:34:39.736220+0000","last_clean":"2024-02-05T07:34:39.736220+0000","last_became_active":"2024-02-05T07:34:30.668558+0000","last_became_peered":"2024-02-05T07:34:30.668558+0000","last_unstale":"2024-02-05T07:34:39.736220+0000","last_undegraded":"2024-02-05T07:34:39.736220+0000","last_fullsized":"2024-02-05T07:34:39.736220+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:34:26.591385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:26.591385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:52.547419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":697,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.719292+0000","last_change":"2024-02-05T07:32:38.877762+0000","last_active":"2024-02-05T07:34:38.719292+0000","last_peered":"2024-02-05T07:34:38.719292+0000","last_clean":"2024-02-05T07:34:38.719292+0000","last_became_active":"2024-02-05T07:30:40.489314+0000","last_became_peered":"2024-02-05T07:30:40.489314+0000","last_unstale":"2024-02-05T07:34:38.719292+0000","last_undegraded":"2024-02-05T07:34:38.719292+0000","last_fullsized":"2024-02-05T07:34:38.719292+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:38.877685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:32:38.877685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"453'5136","reported_seq":5207,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.718345+0000","last_change":"2024-02-05T07:33:38.775531+0000","last_active":"2024-02-05T07:34:38.718345+0000","last_peered":"2024-02-05T07:34:38.718345+0000","last_clean":"2024-02-05T07:34:38.718345+0000","last_became_active":"2024-02-05T07:33:38.775221+0000","last_became_peered":"2024-02-05T07:33:38.775221+0000","last_unstale":"2024-02-05T07:34:38.718345+0000","last_undegraded":"2024-02-05T07:34:38.718345+0000","last_fullsized":"2024-02-05T07:34:38.718345+0000","mapping_epoch":423,"log_start":"427'1600","ondisk_log_start":"427'1600","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:44.600443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336592896,"num_objects":5136,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5136,"num_write_kb":328704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":832,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.731342+0000","last_change":"2024-02-05T07:33:41.082128+0000","last_active":"2024-02-05T07:34:39.731342+0000","last_peered":"2024-02-05T07:34:39.731342+0000","last_clean":"2024-02-05T07:34:39.731342+0000","last_became_active":"2024-02-05T07:28:14.305034+0000","last_became_peered":"2024-02-05T07:28:14.305034+0000","last_unstale":"2024-02-05T07:34:39.731342+0000","last_undegraded":"2024-02-05T07:34:39.731342+0000","last_fullsized":"2024-02-05T07:34:39.731342+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:41.082046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.082046+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:43.103693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":576,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.728003+0000","last_change":"2024-02-05T07:34:37.540422+0000","last_active":"2024-02-05T07:34:39.728003+0000","last_peered":"2024-02-05T07:34:39.728003+0000","last_clean":"2024-02-05T07:34:39.728003+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:34:39.728003+0000","last_undegraded":"2024-02-05T07:34:39.728003+0000","last_fullsized":"2024-02-05T07:34:39.728003+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:37.540304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:34:37.540304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:56.793282+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"326'44","reported_seq":758,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.731701+0000","last_change":"2024-02-05T07:30:36.009087+0000","last_active":"2024-02-05T07:34:39.731701+0000","last_peered":"2024-02-05T07:34:39.731701+0000","last_clean":"2024-02-05T07:34:39.731701+0000","last_became_active":"2024-02-05T07:27:16.708475+0000","last_became_peered":"2024-02-05T07:27:16.708475+0000","last_unstale":"2024-02-05T07:34:39.731701+0000","last_undegraded":"2024-02-05T07:34:39.731701+0000","last_fullsized":"2024-02-05T07:34:39.731701+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'44","last_scrub_stamp":"2024-02-05T07:30:36.008885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.008885+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":211,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.741868+0000","last_change":"2024-02-05T07:30:32.964745+0000","last_active":"2024-02-05T07:34:39.741868+0000","last_peered":"2024-02-05T07:34:39.741868+0000","last_clean":"2024-02-05T07:34:39.741868+0000","last_became_active":"2024-02-05T07:30:30.524566+0000","last_became_peered":"2024-02-05T07:30:30.524566+0000","last_unstale":"2024-02-05T07:34:39.741868+0000","last_undegraded":"2024-02-05T07:34:39.741868+0000","last_fullsized":"2024-02-05T07:34:39.741868+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:32.964622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:32.964622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":746,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.723952+0000","last_change":"2024-02-05T07:33:43.473208+0000","last_active":"2024-02-05T07:34:39.723952+0000","last_peered":"2024-02-05T07:34:39.723952+0000","last_clean":"2024-02-05T07:34:39.723952+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:34:39.723952+0000","last_undegraded":"2024-02-05T07:34:39.723952+0000","last_fullsized":"2024-02-05T07:34:39.723952+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:33:43.473107+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:33:43.473107+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:12.505533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":412,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.711132+0000","last_change":"2024-02-05T07:34:26.897153+0000","last_active":"2024-02-05T07:34:36.711132+0000","last_peered":"2024-02-05T07:34:36.711132+0000","last_clean":"2024-02-05T07:34:36.711132+0000","last_became_active":"2024-02-05T07:30:40.508071+0000","last_became_peered":"2024-02-05T07:30:40.508071+0000","last_unstale":"2024-02-05T07:34:36.711132+0000","last_undegraded":"2024-02-05T07:34:36.711132+0000","last_fullsized":"2024-02-05T07:34:36.711132+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:26.897028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:26.897028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:52.202110+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.5","version":"455'5126","reported_seq":5200,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.921399+0000","last_change":"2024-02-05T07:33:38.778391+0000","last_active":"2024-02-05T07:34:39.921399+0000","last_peered":"2024-02-05T07:34:39.921399+0000","last_clean":"2024-02-05T07:34:39.921399+0000","last_became_active":"2024-02-05T07:33:38.778086+0000","last_became_peered":"2024-02-05T07:33:38.778086+0000","last_unstale":"2024-02-05T07:34:39.921399+0000","last_undegraded":"2024-02-05T07:34:39.921399+0000","last_fullsized":"2024-02-05T07:34:39.921399+0000","mapping_epoch":423,"log_start":"427'1400","ondisk_log_start":"427'1400","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3726,"log_dups_size":0,"ondisk_log_size":3726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:41.700207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335937536,"num_objects":5126,"num_object_clones":0,"num_object_copies":15378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5126,"num_write_kb":328064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":772,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721898+0000","last_change":"2024-02-05T07:34:25.789090+0000","last_active":"2024-02-05T07:34:39.721898+0000","last_peered":"2024-02-05T07:34:39.721898+0000","last_clean":"2024-02-05T07:34:39.721898+0000","last_became_active":"2024-02-05T07:34:24.397559+0000","last_became_peered":"2024-02-05T07:34:24.397559+0000","last_unstale":"2024-02-05T07:34:39.721898+0000","last_undegraded":"2024-02-05T07:34:39.721898+0000","last_fullsized":"2024-02-05T07:34:39.721898+0000","mapping_epoch":370,"log_start":"86'128","ondisk_log_start":"86'128","created":291,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:25.788995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:25.788995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:51.894738+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"326'15","reported_seq":611,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.725887+0000","last_change":"2024-02-05T07:29:51.665907+0000","last_active":"2024-02-05T07:34:39.725887+0000","last_peered":"2024-02-05T07:34:39.725887+0000","last_clean":"2024-02-05T07:34:39.725887+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:34:39.725887+0000","last_undegraded":"2024-02-05T07:34:39.725887+0000","last_fullsized":"2024-02-05T07:34:39.725887+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:29:51.665808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.665808+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":401,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.712116+0000","last_change":"2024-02-05T07:30:38.998491+0000","last_active":"2024-02-05T07:34:36.712116+0000","last_peered":"2024-02-05T07:34:36.712116+0000","last_clean":"2024-02-05T07:34:36.712116+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:34:36.712116+0000","last_undegraded":"2024-02-05T07:34:36.712116+0000","last_fullsized":"2024-02-05T07:34:36.712116+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:38.995586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:38.995586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":835,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.736087+0000","last_change":"2024-02-05T07:32:47.161468+0000","last_active":"2024-02-05T07:34:39.736087+0000","last_peered":"2024-02-05T07:34:39.736087+0000","last_clean":"2024-02-05T07:34:39.736087+0000","last_became_active":"2024-02-05T07:27:16.708205+0000","last_became_peered":"2024-02-05T07:27:16.708205+0000","last_unstale":"2024-02-05T07:34:39.736087+0000","last_undegraded":"2024-02-05T07:34:39.736087+0000","last_fullsized":"2024-02-05T07:34:39.736087+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:47.161343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:47.161343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":203,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.731842+0000","last_change":"2024-02-05T07:32:46.198277+0000","last_active":"2024-02-05T07:34:39.731842+0000","last_peered":"2024-02-05T07:34:39.731842+0000","last_clean":"2024-02-05T07:34:39.731842+0000","last_became_active":"2024-02-05T07:30:40.486401+0000","last_became_peered":"2024-02-05T07:30:40.486401+0000","last_unstale":"2024-02-05T07:34:39.731842+0000","last_undegraded":"2024-02-05T07:34:39.731842+0000","last_fullsized":"2024-02-05T07:34:39.731842+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:46.198155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:32:46.198155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"455'5298","reported_seq":5372,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:40.368642+0000","last_change":"2024-02-05T07:33:38.776231+0000","last_active":"2024-02-05T07:34:40.368642+0000","last_peered":"2024-02-05T07:34:40.368642+0000","last_clean":"2024-02-05T07:34:40.368642+0000","last_became_active":"2024-02-05T07:33:38.775291+0000","last_became_peered":"2024-02-05T07:33:38.775291+0000","last_unstale":"2024-02-05T07:34:40.368642+0000","last_undegraded":"2024-02-05T07:34:40.368642+0000","last_fullsized":"2024-02-05T07:34:40.368642+0000","mapping_epoch":423,"log_start":"428'2000","ondisk_log_start":"428'2000","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:44.655160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347209728,"num_objects":5298,"num_object_clones":0,"num_object_copies":15894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5298,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":673,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-05T07:34:37.715343+0000","last_change":"2024-02-05T07:34:23.202713+0000","last_active":"2024-02-05T07:34:37.715343+0000","last_peered":"2024-02-05T07:34:37.715343+0000","last_clean":"2024-02-05T07:34:37.715343+0000","last_became_active":"2024-02-05T07:28:26.596711+0000","last_became_peered":"2024-02-05T07:28:26.596711+0000","last_unstale":"2024-02-05T07:34:37.715343+0000","last_undegraded":"2024-02-05T07:34:37.715343+0000","last_fullsized":"2024-02-05T07:34:37.715343+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:23.202589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:23.202589+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:33.394554+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"326'35","reported_seq":735,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.718239+0000","last_change":"2024-02-05T07:33:27.163756+0000","last_active":"2024-02-05T07:34:38.718239+0000","last_peered":"2024-02-05T07:34:38.718239+0000","last_clean":"2024-02-05T07:34:38.718239+0000","last_became_active":"2024-02-05T07:30:40.482753+0000","last_became_peered":"2024-02-05T07:30:40.482753+0000","last_unstale":"2024-02-05T07:34:38.718239+0000","last_undegraded":"2024-02-05T07:34:38.718239+0000","last_fullsized":"2024-02-05T07:34:38.718239+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:33:27.163551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:27.163551+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":708,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.718774+0000","last_change":"2024-02-05T07:32:39.793152+0000","last_active":"2024-02-05T07:34:38.718774+0000","last_peered":"2024-02-05T07:34:38.718774+0000","last_clean":"2024-02-05T07:34:38.718774+0000","last_became_active":"2024-02-05T07:30:40.507705+0000","last_became_peered":"2024-02-05T07:30:40.507705+0000","last_unstale":"2024-02-05T07:34:38.718774+0000","last_undegraded":"2024-02-05T07:34:38.718774+0000","last_fullsized":"2024-02-05T07:34:38.718774+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_clean_scrub_stamp":"2024-02-05T07:32:39.793075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":835,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.741966+0000","last_change":"2024-02-05T07:33:28.121535+0000","last_active":"2024-02-05T07:34:39.741966+0000","last_peered":"2024-02-05T07:34:39.741966+0000","last_clean":"2024-02-05T07:34:39.741966+0000","last_became_active":"2024-02-05T07:30:40.479288+0000","last_became_peered":"2024-02-05T07:30:40.479288+0000","last_unstale":"2024-02-05T07:34:39.741966+0000","last_undegraded":"2024-02-05T07:34:39.741966+0000","last_fullsized":"2024-02-05T07:34:39.741966+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.121427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.121427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":616,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.736317+0000","last_change":"2024-02-05T07:33:31.033044+0000","last_active":"2024-02-05T07:34:39.736317+0000","last_peered":"2024-02-05T07:34:39.736317+0000","last_clean":"2024-02-05T07:34:39.736317+0000","last_became_active":"2024-02-05T07:30:40.496822+0000","last_became_peered":"2024-02-05T07:30:40.496822+0000","last_unstale":"2024-02-05T07:34:39.736317+0000","last_undegraded":"2024-02-05T07:34:39.736317+0000","last_fullsized":"2024-02-05T07:34:39.736317+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:31.032967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_clean_scrub_stamp":"2024-02-05T07:33:31.032967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:58.673311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"453'5085","reported_seq":5155,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.647449+0000","last_change":"2024-02-05T07:33:38.614213+0000","last_active":"2024-02-05T07:34:38.647449+0000","last_peered":"2024-02-05T07:34:38.647449+0000","last_clean":"2024-02-05T07:34:38.647449+0000","last_became_active":"2024-02-05T07:33:38.613855+0000","last_became_peered":"2024-02-05T07:33:38.613855+0000","last_unstale":"2024-02-05T07:34:38.647449+0000","last_undegraded":"2024-02-05T07:34:38.647449+0000","last_fullsized":"2024-02-05T07:34:38.647449+0000","mapping_epoch":423,"log_start":"427'1500","ondisk_log_start":"427'1500","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3585,"log_dups_size":0,"ondisk_log_size":3585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:42.559822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333250560,"num_objects":5085,"num_object_clones":0,"num_object_copies":15255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":325440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":1088,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.710697+0000","last_change":"2024-02-05T07:34:32.693736+0000","last_active":"2024-02-05T07:34:36.710697+0000","last_peered":"2024-02-05T07:34:36.710697+0000","last_clean":"2024-02-05T07:34:36.710697+0000","last_became_active":"2024-02-05T07:34:32.693094+0000","last_became_peered":"2024-02-05T07:34:32.693094+0000","last_unstale":"2024-02-05T07:34:36.710697+0000","last_undegraded":"2024-02-05T07:34:36.710697+0000","last_fullsized":"2024-02-05T07:34:36.710697+0000","mapping_epoch":302,"log_start":"86'128","ondisk_log_start":"86'128","created":291,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:41.754605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:41.754605+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":831,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.736134+0000","last_change":"2024-02-05T07:33:29.082880+0000","last_active":"2024-02-05T07:34:39.736134+0000","last_peered":"2024-02-05T07:34:39.736134+0000","last_clean":"2024-02-05T07:34:39.736134+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:34:39.736134+0000","last_undegraded":"2024-02-05T07:34:39.736134+0000","last_fullsized":"2024-02-05T07:34:39.736134+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:29.082803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:29.082803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:43.891007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"325'18","reported_seq":919,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720834+0000","last_change":"2024-02-05T07:34:28.902927+0000","last_active":"2024-02-05T07:34:39.720834+0000","last_peered":"2024-02-05T07:34:39.720834+0000","last_clean":"2024-02-05T07:34:39.720834+0000","last_became_active":"2024-02-05T07:27:16.706871+0000","last_became_peered":"2024-02-05T07:27:16.706871+0000","last_unstale":"2024-02-05T07:34:39.720834+0000","last_undegraded":"2024-02-05T07:34:39.720834+0000","last_fullsized":"2024-02-05T07:34:39.720834+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'18","last_scrub_stamp":"2024-02-05T07:34:28.902809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.902809+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:52.320348+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":200,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721863+0000","last_change":"2024-02-05T07:33:33.822290+0000","last_active":"2024-02-05T07:34:39.721863+0000","last_peered":"2024-02-05T07:34:39.721863+0000","last_clean":"2024-02-05T07:34:39.721863+0000","last_became_active":"2024-02-05T07:30:40.518949+0000","last_became_peered":"2024-02-05T07:30:40.518949+0000","last_unstale":"2024-02-05T07:34:39.721863+0000","last_undegraded":"2024-02-05T07:34:39.721863+0000","last_fullsized":"2024-02-05T07:34:39.721863+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:33.822199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.822199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":404,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.710623+0000","last_change":"2024-02-05T07:30:40.486334+0000","last_active":"2024-02-05T07:34:36.710623+0000","last_peered":"2024-02-05T07:34:36.710623+0000","last_clean":"2024-02-05T07:34:36.710623+0000","last_became_active":"2024-02-05T07:30:40.485991+0000","last_became_peered":"2024-02-05T07:30:40.485991+0000","last_unstale":"2024-02-05T07:34:36.710623+0000","last_undegraded":"2024-02-05T07:34:36.710623+0000","last_fullsized":"2024-02-05T07:34:36.710623+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:48.088229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:48.088229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"455'5263","reported_seq":5337,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:40.386793+0000","last_change":"2024-02-05T07:33:38.775313+0000","last_active":"2024-02-05T07:34:40.386793+0000","last_peered":"2024-02-05T07:34:40.386793+0000","last_clean":"2024-02-05T07:34:40.386793+0000","last_became_active":"2024-02-05T07:33:38.774996+0000","last_became_peered":"2024-02-05T07:33:38.774996+0000","last_unstale":"2024-02-05T07:34:40.386793+0000","last_undegraded":"2024-02-05T07:34:40.386793+0000","last_fullsized":"2024-02-05T07:34:40.386793+0000","mapping_epoch":423,"log_start":"428'1900","ondisk_log_start":"428'1900","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3363,"log_dups_size":0,"ondisk_log_size":3363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344915968,"num_objects":5263,"num_object_clones":0,"num_object_copies":15789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5263,"num_write_kb":336832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":680,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721530+0000","last_change":"2024-02-05T07:34:28.628631+0000","last_active":"2024-02-05T07:34:39.721530+0000","last_peered":"2024-02-05T07:34:39.721530+0000","last_clean":"2024-02-05T07:34:39.721530+0000","last_became_active":"2024-02-05T07:34:28.628242+0000","last_became_peered":"2024-02-05T07:34:28.628242+0000","last_unstale":"2024-02-05T07:34:39.721530+0000","last_undegraded":"2024-02-05T07:34:39.721530+0000","last_fullsized":"2024-02-05T07:34:39.721530+0000","mapping_epoch":367,"log_start":"86'128","ondisk_log_start":"86'128","created":291,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:22.621608+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_clean_scrub_stamp":"2024-02-05T07:32:22.621608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"365'12","reported_seq":867,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.716775+0000","last_change":"2024-02-05T07:32:30.801254+0000","last_active":"2024-02-05T07:34:38.716775+0000","last_peered":"2024-02-05T07:34:38.716775+0000","last_clean":"2024-02-05T07:34:38.716775+0000","last_became_active":"2024-02-05T07:30:40.500113+0000","last_became_peered":"2024-02-05T07:30:40.500113+0000","last_unstale":"2024-02-05T07:34:38.716775+0000","last_undegraded":"2024-02-05T07:34:38.716775+0000","last_fullsized":"2024-02-05T07:34:38.716775+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'12","last_scrub_stamp":"2024-02-05T07:32:30.801134+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:32:30.801134+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"326'27","reported_seq":455,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.716781+0000","last_change":"2024-02-05T07:32:29.833732+0000","last_active":"2024-02-05T07:34:38.716781+0000","last_peered":"2024-02-05T07:34:38.716781+0000","last_clean":"2024-02-05T07:34:38.716781+0000","last_became_active":"2024-02-05T07:30:40.497982+0000","last_became_peered":"2024-02-05T07:30:40.497982+0000","last_unstale":"2024-02-05T07:34:38.716781+0000","last_undegraded":"2024-02-05T07:34:38.716781+0000","last_fullsized":"2024-02-05T07:34:38.716781+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:32:29.833636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:29.833636+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":216,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721466+0000","last_change":"2024-02-05T07:33:27.649686+0000","last_active":"2024-02-05T07:34:39.721466+0000","last_peered":"2024-02-05T07:34:39.721466+0000","last_clean":"2024-02-05T07:34:39.721466+0000","last_became_active":"2024-02-05T07:30:40.503903+0000","last_became_peered":"2024-02-05T07:30:40.503903+0000","last_unstale":"2024-02-05T07:34:39.721466+0000","last_undegraded":"2024-02-05T07:34:39.721466+0000","last_fullsized":"2024-02-05T07:34:39.721466+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:33:27.649596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:27.649596+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:43.748950+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":490,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.717052+0000","last_change":"2024-02-05T07:33:35.177937+0000","last_active":"2024-02-05T07:34:38.717052+0000","last_peered":"2024-02-05T07:34:38.717052+0000","last_clean":"2024-02-05T07:34:38.717052+0000","last_became_active":"2024-02-05T07:28:26.589671+0000","last_became_peered":"2024-02-05T07:28:26.589671+0000","last_unstale":"2024-02-05T07:34:38.717052+0000","last_undegraded":"2024-02-05T07:34:38.717052+0000","last_fullsized":"2024-02-05T07:34:38.717052+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:35.177804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:33:35.177804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:44.172989+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":681,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.723651+0000","last_change":"2024-02-05T07:33:44.492558+0000","last_active":"2024-02-05T07:34:39.723651+0000","last_peered":"2024-02-05T07:34:39.723651+0000","last_clean":"2024-02-05T07:34:39.723651+0000","last_became_active":"2024-02-05T07:30:43.523108+0000","last_became_peered":"2024-02-05T07:30:43.523108+0000","last_unstale":"2024-02-05T07:34:39.723651+0000","last_undegraded":"2024-02-05T07:34:39.723651+0000","last_fullsized":"2024-02-05T07:34:39.723651+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:44.492478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.492478+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:44.499792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"326'12","reported_seq":429,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.717723+0000","last_change":"2024-02-05T07:33:41.762486+0000","last_active":"2024-02-05T07:34:36.717723+0000","last_peered":"2024-02-05T07:34:36.717723+0000","last_clean":"2024-02-05T07:34:36.717723+0000","last_became_active":"2024-02-05T07:30:40.496980+0000","last_became_peered":"2024-02-05T07:30:40.496980+0000","last_unstale":"2024-02-05T07:34:36.717723+0000","last_undegraded":"2024-02-05T07:34:36.717723+0000","last_fullsized":"2024-02-05T07:34:36.717723+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'12","last_scrub_stamp":"2024-02-05T07:33:41.762404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.762404+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:00.099696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":634,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.716934+0000","last_change":"2024-02-05T07:33:51.105371+0000","last_active":"2024-02-05T07:34:38.716934+0000","last_peered":"2024-02-05T07:34:38.716934+0000","last_clean":"2024-02-05T07:34:38.716934+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:34:38.716934+0000","last_undegraded":"2024-02-05T07:34:38.716934+0000","last_fullsized":"2024-02-05T07:34:38.716934+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:33:51.105251+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:33:51.105251+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:18.353438+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":401,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.710859+0000","last_change":"2024-02-05T07:30:27.965535+0000","last_active":"2024-02-05T07:34:36.710859+0000","last_peered":"2024-02-05T07:34:36.710859+0000","last_clean":"2024-02-05T07:34:36.710859+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:34:36.710859+0000","last_undegraded":"2024-02-05T07:34:36.710859+0000","last_fullsized":"2024-02-05T07:34:36.710859+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:27.965449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:27.965449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":684,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.719051+0000","last_change":"2024-02-05T07:32:32.844370+0000","last_active":"2024-02-05T07:34:38.719051+0000","last_peered":"2024-02-05T07:34:38.719051+0000","last_clean":"2024-02-05T07:34:38.719051+0000","last_became_active":"2024-02-05T07:30:40.509216+0000","last_became_peered":"2024-02-05T07:30:40.509216+0000","last_unstale":"2024-02-05T07:34:38.719051+0000","last_undegraded":"2024-02-05T07:34:38.719051+0000","last_fullsized":"2024-02-05T07:34:38.719051+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:32.844264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:32:32.844264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":482,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.709711+0000","last_change":"2024-02-05T07:34:30.927994+0000","last_active":"2024-02-05T07:34:36.709711+0000","last_peered":"2024-02-05T07:34:36.709711+0000","last_clean":"2024-02-05T07:34:36.709711+0000","last_became_active":"2024-02-05T07:34:30.654622+0000","last_became_peered":"2024-02-05T07:34:30.654622+0000","last_unstale":"2024-02-05T07:34:36.709711+0000","last_undegraded":"2024-02-05T07:34:36.709711+0000","last_fullsized":"2024-02-05T07:34:36.709711+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:30.927865+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.702565+0000","last_clean_scrub_stamp":"2024-02-05T07:34:30.927865+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:48.546128+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"325'15","reported_seq":433,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.711078+0000","last_change":"2024-02-05T07:33:45.711524+0000","last_active":"2024-02-05T07:34:36.711078+0000","last_peered":"2024-02-05T07:34:36.711078+0000","last_clean":"2024-02-05T07:34:36.711078+0000","last_became_active":"2024-02-05T07:30:40.485837+0000","last_became_peered":"2024-02-05T07:30:40.485837+0000","last_unstale":"2024-02-05T07:34:36.711078+0000","last_undegraded":"2024-02-05T07:34:36.711078+0000","last_fullsized":"2024-02-05T07:34:36.711078+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:33:45.711399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:45.711399+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:00.359295+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":699,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720796+0000","last_change":"2024-02-05T07:30:40.498263+0000","last_active":"2024-02-05T07:34:39.720796+0000","last_peered":"2024-02-05T07:34:39.720796+0000","last_clean":"2024-02-05T07:34:39.720796+0000","last_became_active":"2024-02-05T07:30:40.497956+0000","last_became_peered":"2024-02-05T07:30:40.497956+0000","last_unstale":"2024-02-05T07:34:39.720796+0000","last_undegraded":"2024-02-05T07:34:39.720796+0000","last_fullsized":"2024-02-05T07:34:39.720796+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:33.183427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:30:33.183427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":404,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.717844+0000","last_change":"2024-02-05T07:33:50.723381+0000","last_active":"2024-02-05T07:34:36.717844+0000","last_peered":"2024-02-05T07:34:36.717844+0000","last_clean":"2024-02-05T07:34:36.717844+0000","last_became_active":"2024-02-05T07:30:40.496076+0000","last_became_peered":"2024-02-05T07:30:40.496076+0000","last_unstale":"2024-02-05T07:34:36.717844+0000","last_undegraded":"2024-02-05T07:34:36.717844+0000","last_fullsized":"2024-02-05T07:34:36.717844+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_clean_scrub_stamp":"2024-02-05T07:33:50.723212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:09.045969+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":824,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.731845+0000","last_change":"2024-02-05T07:32:48.164374+0000","last_active":"2024-02-05T07:34:39.731845+0000","last_peered":"2024-02-05T07:34:39.731845+0000","last_clean":"2024-02-05T07:34:39.731845+0000","last_became_active":"2024-02-05T07:30:40.487455+0000","last_became_peered":"2024-02-05T07:30:40.487455+0000","last_unstale":"2024-02-05T07:34:39.731845+0000","last_undegraded":"2024-02-05T07:34:39.731845+0000","last_fullsized":"2024-02-05T07:34:39.731845+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:48.164259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:32:48.164259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":524,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.716430+0000","last_change":"2024-02-05T07:34:28.166523+0000","last_active":"2024-02-05T07:34:38.716430+0000","last_peered":"2024-02-05T07:34:38.716430+0000","last_clean":"2024-02-05T07:34:38.716430+0000","last_became_active":"2024-02-05T07:30:43.466854+0000","last_became_peered":"2024-02-05T07:30:43.466854+0000","last_unstale":"2024-02-05T07:34:38.716430+0000","last_undegraded":"2024-02-05T07:34:38.716430+0000","last_fullsized":"2024-02-05T07:34:38.716430+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:28.166398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.166398+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:29.249590+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"326'19","reported_seq":752,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.711893+0000","last_change":"2024-02-05T07:30:40.503823+0000","last_active":"2024-02-05T07:34:36.711893+0000","last_peered":"2024-02-05T07:34:36.711893+0000","last_clean":"2024-02-05T07:34:36.711893+0000","last_became_active":"2024-02-05T07:30:40.483700+0000","last_became_peered":"2024-02-05T07:30:40.483700+0000","last_unstale":"2024-02-05T07:34:36.711893+0000","last_undegraded":"2024-02-05T07:34:36.711893+0000","last_fullsized":"2024-02-05T07:34:36.711893+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:30:35.138361+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:30:35.138361+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":566,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.709814+0000","last_change":"2024-02-05T07:30:40.476394+0000","last_active":"2024-02-05T07:34:36.709814+0000","last_peered":"2024-02-05T07:34:36.709814+0000","last_clean":"2024-02-05T07:34:36.709814+0000","last_became_active":"2024-02-05T07:30:40.476058+0000","last_became_peered":"2024-02-05T07:30:40.476058+0000","last_unstale":"2024-02-05T07:34:36.709814+0000","last_undegraded":"2024-02-05T07:34:36.709814+0000","last_fullsized":"2024-02-05T07:34:36.709814+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.314735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.314735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"365'11","reported_seq":500,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.711813+0000","last_change":"2024-02-05T07:33:39.734203+0000","last_active":"2024-02-05T07:34:36.711813+0000","last_peered":"2024-02-05T07:34:36.711813+0000","last_clean":"2024-02-05T07:34:36.711813+0000","last_became_active":"2024-02-05T07:30:40.495708+0000","last_became_peered":"2024-02-05T07:30:40.495708+0000","last_unstale":"2024-02-05T07:34:36.711813+0000","last_undegraded":"2024-02-05T07:34:36.711813+0000","last_fullsized":"2024-02-05T07:34:36.711813+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:33:39.734129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:39.734129+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:00.775574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":816,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721577+0000","last_change":"2024-02-05T07:33:38.879276+0000","last_active":"2024-02-05T07:34:39.721577+0000","last_peered":"2024-02-05T07:34:39.721577+0000","last_clean":"2024-02-05T07:34:39.721577+0000","last_became_active":"2024-02-05T07:27:03.364334+0000","last_became_peered":"2024-02-05T07:27:03.364334+0000","last_unstale":"2024-02-05T07:34:39.721577+0000","last_undegraded":"2024-02-05T07:34:39.721577+0000","last_fullsized":"2024-02-05T07:34:39.721577+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:38.879177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:38.879177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:46.608368+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":664,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.709569+0000","last_change":"2024-02-05T07:34:35.914394+0000","last_active":"2024-02-05T07:34:36.709569+0000","last_peered":"2024-02-05T07:34:36.709569+0000","last_clean":"2024-02-05T07:34:36.709569+0000","last_became_active":"2024-02-05T07:28:19.302393+0000","last_became_peered":"2024-02-05T07:28:19.302393+0000","last_unstale":"2024-02-05T07:34:36.709569+0000","last_undegraded":"2024-02-05T07:34:36.709569+0000","last_fullsized":"2024-02-05T07:34:36.709569+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:35.914275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:35.914275+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:38.526356+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"326'25","reported_seq":443,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.712031+0000","last_change":"2024-02-05T07:30:40.512630+0000","last_active":"2024-02-05T07:34:36.712031+0000","last_peered":"2024-02-05T07:34:36.712031+0000","last_clean":"2024-02-05T07:34:36.712031+0000","last_became_active":"2024-02-05T07:30:40.486394+0000","last_became_peered":"2024-02-05T07:30:40.486394+0000","last_unstale":"2024-02-05T07:34:36.712031+0000","last_undegraded":"2024-02-05T07:34:36.712031+0000","last_fullsized":"2024-02-05T07:34:36.712031+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:30:33.959124+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:30:33.959124+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":201,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720869+0000","last_change":"2024-02-05T07:34:32.854993+0000","last_active":"2024-02-05T07:34:39.720869+0000","last_peered":"2024-02-05T07:34:39.720869+0000","last_clean":"2024-02-05T07:34:39.720869+0000","last_became_active":"2024-02-05T07:30:40.505600+0000","last_became_peered":"2024-02-05T07:30:40.505600+0000","last_unstale":"2024-02-05T07:34:39.720869+0000","last_undegraded":"2024-02-05T07:34:39.720869+0000","last_fullsized":"2024-02-05T07:34:39.720869+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:32.854871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:32.854871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:59.631282+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"217'9","reported_seq":641,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.733091+0000","last_change":"2024-02-05T07:34:27.947088+0000","last_active":"2024-02-05T07:34:39.733091+0000","last_peered":"2024-02-05T07:34:39.733091+0000","last_clean":"2024-02-05T07:34:39.733091+0000","last_became_active":"2024-02-05T07:25:27.246616+0000","last_became_peered":"2024-02-05T07:25:27.246616+0000","last_unstale":"2024-02-05T07:34:39.733091+0000","last_undegraded":"2024-02-05T07:34:39.733091+0000","last_fullsized":"2024-02-05T07:34:39.733091+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9","last_scrub_stamp":"2024-02-05T07:34:27.946504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:34:27.946504+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:43.736690+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":562,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.711495+0000","last_change":"2024-02-05T07:34:22.810896+0000","last_active":"2024-02-05T07:34:36.711495+0000","last_peered":"2024-02-05T07:34:36.711495+0000","last_clean":"2024-02-05T07:34:36.711495+0000","last_became_active":"2024-02-05T07:30:40.507831+0000","last_became_peered":"2024-02-05T07:30:40.507831+0000","last_unstale":"2024-02-05T07:34:36.711495+0000","last_undegraded":"2024-02-05T07:34:36.711495+0000","last_fullsized":"2024-02-05T07:34:36.711495+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:22.810759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:22.810759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:48.571309+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.16","version":"86'128","reported_seq":382,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.717068+0000","last_change":"2024-02-05T07:34:24.144804+0000","last_active":"2024-02-05T07:34:38.717068+0000","last_peered":"2024-02-05T07:34:38.717068+0000","last_clean":"2024-02-05T07:34:38.717068+0000","last_became_active":"2024-02-05T07:34:19.559304+0000","last_became_peered":"2024-02-05T07:34:19.559304+0000","last_unstale":"2024-02-05T07:34:38.717068+0000","last_undegraded":"2024-02-05T07:34:38.717068+0000","last_fullsized":"2024-02-05T07:34:38.717068+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:24.144718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:24.144718+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:41.058893+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"326'19","reported_seq":876,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720936+0000","last_change":"2024-02-05T07:32:31.667966+0000","last_active":"2024-02-05T07:34:39.720936+0000","last_peered":"2024-02-05T07:34:39.720936+0000","last_clean":"2024-02-05T07:34:39.720936+0000","last_became_active":"2024-02-05T07:28:25.779886+0000","last_became_peered":"2024-02-05T07:28:25.779886+0000","last_unstale":"2024-02-05T07:34:39.720936+0000","last_undegraded":"2024-02-05T07:34:39.720936+0000","last_fullsized":"2024-02-05T07:34:39.720936+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:32:31.667882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:31.667882+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":837,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721053+0000","last_change":"2024-02-05T07:34:31.893068+0000","last_active":"2024-02-05T07:34:39.721053+0000","last_peered":"2024-02-05T07:34:39.721053+0000","last_clean":"2024-02-05T07:34:39.721053+0000","last_became_active":"2024-02-05T07:30:40.510301+0000","last_became_peered":"2024-02-05T07:30:40.510301+0000","last_unstale":"2024-02-05T07:34:39.721053+0000","last_undegraded":"2024-02-05T07:34:39.721053+0000","last_fullsized":"2024-02-05T07:34:39.721053+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:31.892895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:34:31.892895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:37.814244+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":837,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.724278+0000","last_change":"2024-02-05T07:32:48.346195+0000","last_active":"2024-02-05T07:34:39.724278+0000","last_peered":"2024-02-05T07:34:39.724278+0000","last_clean":"2024-02-05T07:34:39.724278+0000","last_became_active":"2024-02-05T07:30:40.487141+0000","last_became_peered":"2024-02-05T07:30:40.487141+0000","last_unstale":"2024-02-05T07:34:39.724278+0000","last_undegraded":"2024-02-05T07:34:39.724278+0000","last_fullsized":"2024-02-05T07:34:39.724278+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_clean_scrub_stamp":"2024-02-05T07:32:48.345965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":396,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.710023+0000","last_change":"2024-02-05T07:33:37.735878+0000","last_active":"2024-02-05T07:34:36.710023+0000","last_peered":"2024-02-05T07:34:36.710023+0000","last_clean":"2024-02-05T07:34:36.710023+0000","last_became_active":"2024-02-05T07:27:17.705910+0000","last_became_peered":"2024-02-05T07:27:17.705910+0000","last_unstale":"2024-02-05T07:34:36.710023+0000","last_undegraded":"2024-02-05T07:34:36.710023+0000","last_fullsized":"2024-02-05T07:34:36.710023+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.735757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.735757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:51.255829+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"86'128","reported_seq":694,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-05T07:34:37.715275+0000","last_change":"2024-02-05T07:34:19.567196+0000","last_active":"2024-02-05T07:34:37.715275+0000","last_peered":"2024-02-05T07:34:37.715275+0000","last_clean":"2024-02-05T07:34:37.715275+0000","last_became_active":"2024-02-05T07:34:19.566891+0000","last_became_peered":"2024-02-05T07:34:19.566891+0000","last_unstale":"2024-02-05T07:34:37.715275+0000","last_undegraded":"2024-02-05T07:34:37.715275+0000","last_fullsized":"2024-02-05T07:34:37.715275+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:23.818901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:23.818901+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"325'15","reported_seq":443,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.723483+0000","last_change":"2024-02-05T07:29:57.775013+0000","last_active":"2024-02-05T07:34:39.723483+0000","last_peered":"2024-02-05T07:34:39.723483+0000","last_clean":"2024-02-05T07:34:39.723483+0000","last_became_active":"2024-02-05T07:28:26.589705+0000","last_became_peered":"2024-02-05T07:28:26.589705+0000","last_unstale":"2024-02-05T07:34:39.723483+0000","last_undegraded":"2024-02-05T07:34:39.723483+0000","last_fullsized":"2024-02-05T07:34:39.723483+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:29:57.774913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:57.774913+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":611,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.711810+0000","last_change":"2024-02-05T07:30:40.489183+0000","last_active":"2024-02-05T07:34:36.711810+0000","last_peered":"2024-02-05T07:34:36.711810+0000","last_clean":"2024-02-05T07:34:36.711810+0000","last_became_active":"2024-02-05T07:30:40.488789+0000","last_became_peered":"2024-02-05T07:30:40.488789+0000","last_unstale":"2024-02-05T07:34:36.711810+0000","last_undegraded":"2024-02-05T07:34:36.711810+0000","last_fullsized":"2024-02-05T07:34:36.711810+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.220841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.220841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":398,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.710869+0000","last_change":"2024-02-05T07:33:47.684589+0000","last_active":"2024-02-05T07:34:36.710869+0000","last_peered":"2024-02-05T07:34:36.710869+0000","last_clean":"2024-02-05T07:34:36.710869+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:34:36.710869+0000","last_undegraded":"2024-02-05T07:34:36.710869+0000","last_fullsized":"2024-02-05T07:34:36.710869+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:47.684508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.684508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:12.589901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":280,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.716380+0000","last_change":"2024-02-05T07:34:30.670504+0000","last_active":"2024-02-05T07:34:38.716380+0000","last_peered":"2024-02-05T07:34:38.716380+0000","last_clean":"2024-02-05T07:34:38.716380+0000","last_became_active":"2024-02-05T07:34:30.670179+0000","last_became_peered":"2024-02-05T07:34:30.670179+0000","last_unstale":"2024-02-05T07:34:38.716380+0000","last_undegraded":"2024-02-05T07:34:38.716380+0000","last_fullsized":"2024-02-05T07:34:38.716380+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:26.419332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:33:26.419332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":1075,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.731392+0000","last_change":"2024-02-05T07:30:43.515841+0000","last_active":"2024-02-05T07:34:39.731392+0000","last_peered":"2024-02-05T07:34:39.731392+0000","last_clean":"2024-02-05T07:34:39.731392+0000","last_became_active":"2024-02-05T07:30:43.515488+0000","last_became_peered":"2024-02-05T07:30:43.515488+0000","last_unstale":"2024-02-05T07:34:39.731392+0000","last_undegraded":"2024-02-05T07:34:39.731392+0000","last_fullsized":"2024-02-05T07:34:39.731392+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:30:36.995330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.995330+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"326'19","reported_seq":443,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.710294+0000","last_change":"2024-02-05T07:30:40.906805+0000","last_active":"2024-02-05T07:34:36.710294+0000","last_peered":"2024-02-05T07:34:36.710294+0000","last_clean":"2024-02-05T07:34:36.710294+0000","last_became_active":"2024-02-05T07:30:40.501893+0000","last_became_peered":"2024-02-05T07:30:40.501893+0000","last_unstale":"2024-02-05T07:34:36.710294+0000","last_undegraded":"2024-02-05T07:34:36.710294+0000","last_fullsized":"2024-02-05T07:34:36.710294+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:30:40.906693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.906693+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":218,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.725272+0000","last_change":"2024-02-05T07:30:40.492005+0000","last_active":"2024-02-05T07:34:39.725272+0000","last_peered":"2024-02-05T07:34:39.725272+0000","last_clean":"2024-02-05T07:34:39.725272+0000","last_became_active":"2024-02-05T07:30:40.491686+0000","last_became_peered":"2024-02-05T07:30:40.491686+0000","last_unstale":"2024-02-05T07:34:39.725272+0000","last_undegraded":"2024-02-05T07:34:39.725272+0000","last_fullsized":"2024-02-05T07:34:39.725272+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.097532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.097532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":691,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720499+0000","last_change":"2024-02-05T07:30:40.497509+0000","last_active":"2024-02-05T07:34:39.720499+0000","last_peered":"2024-02-05T07:34:39.720499+0000","last_clean":"2024-02-05T07:34:39.720499+0000","last_became_active":"2024-02-05T07:30:40.497190+0000","last_became_peered":"2024-02-05T07:30:40.497190+0000","last_unstale":"2024-02-05T07:34:39.720499+0000","last_undegraded":"2024-02-05T07:34:39.720499+0000","last_fullsized":"2024-02-05T07:34:39.720499+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:29:58.191815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":402,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.710536+0000","last_change":"2024-02-05T07:29:40.983327+0000","last_active":"2024-02-05T07:34:36.710536+0000","last_peered":"2024-02-05T07:34:36.710536+0000","last_clean":"2024-02-05T07:34:36.710536+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:34:36.710536+0000","last_undegraded":"2024-02-05T07:34:36.710536+0000","last_fullsized":"2024-02-05T07:34:36.710536+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:40.983227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.983227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"86'128","reported_seq":373,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720945+0000","last_change":"2024-02-05T07:34:23.780004+0000","last_active":"2024-02-05T07:34:39.720945+0000","last_peered":"2024-02-05T07:34:39.720945+0000","last_clean":"2024-02-05T07:34:39.720945+0000","last_became_active":"2024-02-05T07:34:19.555715+0000","last_became_peered":"2024-02-05T07:34:19.555715+0000","last_unstale":"2024-02-05T07:34:39.720945+0000","last_undegraded":"2024-02-05T07:34:39.720945+0000","last_fullsized":"2024-02-05T07:34:39.720945+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:23.779897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:23.779897+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:44.803706+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"325'8","reported_seq":414,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.709261+0000","last_change":"2024-02-05T07:29:53.958469+0000","last_active":"2024-02-05T07:34:36.709261+0000","last_peered":"2024-02-05T07:34:36.709261+0000","last_clean":"2024-02-05T07:34:36.709261+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:34:36.709261+0000","last_undegraded":"2024-02-05T07:34:36.709261+0000","last_fullsized":"2024-02-05T07:34:36.709261+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:29:53.958384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.958384+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":401,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.717606+0000","last_change":"2024-02-05T07:28:46.056835+0000","last_active":"2024-02-05T07:34:36.717606+0000","last_peered":"2024-02-05T07:34:36.717606+0000","last_clean":"2024-02-05T07:34:36.717606+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:34:36.717606+0000","last_undegraded":"2024-02-05T07:34:36.717606+0000","last_fullsized":"2024-02-05T07:34:36.717606+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:46.056702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:28:46.056702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":668,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-05T07:34:38.716729+0000","last_change":"2024-02-05T07:33:30.216853+0000","last_active":"2024-02-05T07:34:38.716729+0000","last_peered":"2024-02-05T07:34:38.716729+0000","last_clean":"2024-02-05T07:34:38.716729+0000","last_became_active":"2024-02-05T07:30:40.500403+0000","last_became_peered":"2024-02-05T07:30:40.500403+0000","last_unstale":"2024-02-05T07:34:38.716729+0000","last_undegraded":"2024-02-05T07:34:38.716729+0000","last_fullsized":"2024-02-05T07:34:38.716729+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:30.216745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.216745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:48.415365+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":202,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.736436+0000","last_change":"2024-02-05T07:30:45.926725+0000","last_active":"2024-02-05T07:34:39.736436+0000","last_peered":"2024-02-05T07:34:39.736436+0000","last_clean":"2024-02-05T07:34:39.736436+0000","last_became_active":"2024-02-05T07:30:40.489230+0000","last_became_peered":"2024-02-05T07:30:40.489230+0000","last_unstale":"2024-02-05T07:34:39.736436+0000","last_undegraded":"2024-02-05T07:34:39.736436+0000","last_fullsized":"2024-02-05T07:34:39.736436+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:45.926395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:30:45.926395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"326'19","reported_seq":653,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-05T07:34:37.716232+0000","last_change":"2024-02-05T07:32:43.787447+0000","last_active":"2024-02-05T07:34:37.716232+0000","last_peered":"2024-02-05T07:34:37.716232+0000","last_clean":"2024-02-05T07:34:37.716232+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:34:37.716232+0000","last_undegraded":"2024-02-05T07:34:37.716232+0000","last_fullsized":"2024-02-05T07:34:37.716232+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:32:43.787327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:43.787327+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":398,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.708881+0000","last_change":"2024-02-05T07:29:38.000417+0000","last_active":"2024-02-05T07:34:36.708881+0000","last_peered":"2024-02-05T07:34:36.708881+0000","last_clean":"2024-02-05T07:34:36.708881+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:34:36.708881+0000","last_undegraded":"2024-02-05T07:34:36.708881+0000","last_fullsized":"2024-02-05T07:34:36.708881+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:38.000345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:38.000345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":195,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.732734+0000","last_change":"2024-02-05T07:30:40.480962+0000","last_active":"2024-02-05T07:34:39.732734+0000","last_peered":"2024-02-05T07:34:39.732734+0000","last_clean":"2024-02-05T07:34:39.732734+0000","last_became_active":"2024-02-05T07:30:40.479162+0000","last_became_peered":"2024-02-05T07:30:40.479162+0000","last_unstale":"2024-02-05T07:34:39.732734+0000","last_undegraded":"2024-02-05T07:34:39.732734+0000","last_fullsized":"2024-02-05T07:34:39.732734+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:28.153881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:28.153881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":435,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721724+0000","last_change":"2024-02-05T07:34:30.662610+0000","last_active":"2024-02-05T07:34:39.721724+0000","last_peered":"2024-02-05T07:34:39.721724+0000","last_clean":"2024-02-05T07:34:39.721724+0000","last_became_active":"2024-02-05T07:34:30.662349+0000","last_became_peered":"2024-02-05T07:34:30.662349+0000","last_unstale":"2024-02-05T07:34:39.721724+0000","last_undegraded":"2024-02-05T07:34:39.721724+0000","last_fullsized":"2024-02-05T07:34:39.721724+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:55.800824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.800824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"326'20","reported_seq":564,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.710702+0000","last_change":"2024-02-05T07:30:40.534624+0000","last_active":"2024-02-05T07:34:36.710702+0000","last_peered":"2024-02-05T07:34:36.710702+0000","last_clean":"2024-02-05T07:34:36.710702+0000","last_became_active":"2024-02-05T07:30:40.495954+0000","last_became_peered":"2024-02-05T07:30:40.495954+0000","last_unstale":"2024-02-05T07:34:36.710702+0000","last_undegraded":"2024-02-05T07:34:36.710702+0000","last_fullsized":"2024-02-05T07:34:36.710702+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:29:40.154854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.154854+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":875,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.724257+0000","last_change":"2024-02-05T07:34:38.558839+0000","last_active":"2024-02-05T07:34:39.724257+0000","last_peered":"2024-02-05T07:34:39.724257+0000","last_clean":"2024-02-05T07:34:39.724257+0000","last_became_active":"2024-02-05T07:30:40.495135+0000","last_became_peered":"2024-02-05T07:30:40.495135+0000","last_unstale":"2024-02-05T07:34:39.724257+0000","last_undegraded":"2024-02-05T07:34:39.724257+0000","last_fullsized":"2024-02-05T07:34:39.724257+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:34:38.558593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.558593+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:58.060722+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"365'11","reported_seq":839,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-05T07:34:37.715570+0000","last_change":"2024-02-05T07:33:31.254022+0000","last_active":"2024-02-05T07:34:37.715570+0000","last_peered":"2024-02-05T07:34:37.715570+0000","last_clean":"2024-02-05T07:34:37.715570+0000","last_became_active":"2024-02-05T07:30:40.497371+0000","last_became_peered":"2024-02-05T07:30:40.497371+0000","last_unstale":"2024-02-05T07:34:37.715570+0000","last_undegraded":"2024-02-05T07:34:37.715570+0000","last_fullsized":"2024-02-05T07:34:37.715570+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:33:31.253933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:31.253933+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:42.464942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":312,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.731496+0000","last_change":"2024-02-05T07:30:41.950393+0000","last_active":"2024-02-05T07:34:39.731496+0000","last_peered":"2024-02-05T07:34:39.731496+0000","last_clean":"2024-02-05T07:34:39.731496+0000","last_became_active":"2024-02-05T07:30:40.486920+0000","last_became_peered":"2024-02-05T07:30:40.486920+0000","last_unstale":"2024-02-05T07:34:39.731496+0000","last_undegraded":"2024-02-05T07:34:39.731496+0000","last_fullsized":"2024-02-05T07:34:39.731496+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:30:41.950253+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:30:41.950253+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"86'128","reported_seq":876,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.735750+0000","last_change":"2024-02-05T07:34:19.569752+0000","last_active":"2024-02-05T07:34:39.735750+0000","last_peered":"2024-02-05T07:34:39.735750+0000","last_clean":"2024-02-05T07:34:39.735750+0000","last_became_active":"2024-02-05T07:34:19.569186+0000","last_became_peered":"2024-02-05T07:34:19.569186+0000","last_unstale":"2024-02-05T07:34:39.735750+0000","last_undegraded":"2024-02-05T07:34:39.735750+0000","last_fullsized":"2024-02-05T07:34:39.735750+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:42.278112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:42.278112+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"326'23","reported_seq":423,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.711589+0000","last_change":"2024-02-05T07:28:47.062765+0000","last_active":"2024-02-05T07:34:36.711589+0000","last_peered":"2024-02-05T07:34:36.711589+0000","last_clean":"2024-02-05T07:34:36.711589+0000","last_became_active":"2024-02-05T07:27:16.747998+0000","last_became_peered":"2024-02-05T07:27:16.747998+0000","last_unstale":"2024-02-05T07:34:36.711589+0000","last_undegraded":"2024-02-05T07:34:36.711589+0000","last_fullsized":"2024-02-05T07:34:36.711589+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'17","last_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:28:47.062618+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":31,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.736204+0000","last_change":"2024-02-05T07:34:30.686108+0000","last_active":"2024-02-05T07:34:39.736204+0000","last_peered":"2024-02-05T07:34:39.736204+0000","last_clean":"2024-02-05T07:34:39.736204+0000","last_became_active":"2024-02-05T07:34:30.685805+0000","last_became_peered":"2024-02-05T07:34:30.685805+0000","last_unstale":"2024-02-05T07:34:39.736204+0000","last_undegraded":"2024-02-05T07:34:39.736204+0000","last_fullsized":"2024-02-05T07:34:39.736204+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.874081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":829,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721248+0000","last_change":"2024-02-05T07:30:26.669849+0000","last_active":"2024-02-05T07:34:39.721248+0000","last_peered":"2024-02-05T07:34:39.721248+0000","last_clean":"2024-02-05T07:34:39.721248+0000","last_became_active":"2024-02-05T07:27:16.706777+0000","last_became_peered":"2024-02-05T07:27:16.706777+0000","last_unstale":"2024-02-05T07:34:39.721248+0000","last_undegraded":"2024-02-05T07:34:39.721248+0000","last_fullsized":"2024-02-05T07:34:39.721248+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:26.669576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:26.669576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":401,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.717656+0000","last_change":"2024-02-05T07:28:53.138071+0000","last_active":"2024-02-05T07:34:36.717656+0000","last_peered":"2024-02-05T07:34:36.717656+0000","last_clean":"2024-02-05T07:34:36.717656+0000","last_became_active":"2024-02-05T07:28:26.603067+0000","last_became_peered":"2024-02-05T07:28:26.603067+0000","last_unstale":"2024-02-05T07:34:36.717656+0000","last_undegraded":"2024-02-05T07:34:36.717656+0000","last_fullsized":"2024-02-05T07:34:36.717656+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:28:53.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:28:53.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.19","version":"86'128","reported_seq":938,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.709392+0000","last_change":"2024-02-05T07:34:19.569993+0000","last_active":"2024-02-05T07:34:36.709392+0000","last_peered":"2024-02-05T07:34:36.709392+0000","last_clean":"2024-02-05T07:34:36.709392+0000","last_became_active":"2024-02-05T07:34:19.567208+0000","last_became_peered":"2024-02-05T07:34:19.567208+0000","last_unstale":"2024-02-05T07:34:36.709392+0000","last_undegraded":"2024-02-05T07:34:36.709392+0000","last_fullsized":"2024-02-05T07:34:36.709392+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:44.746349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:44.746349+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"326'27","reported_seq":913,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.732832+0000","last_change":"2024-02-05T07:33:33.995153+0000","last_active":"2024-02-05T07:34:39.732832+0000","last_peered":"2024-02-05T07:34:39.732832+0000","last_clean":"2024-02-05T07:34:39.732832+0000","last_became_active":"2024-02-05T07:27:16.736296+0000","last_became_peered":"2024-02-05T07:27:16.736296+0000","last_unstale":"2024-02-05T07:34:39.732832+0000","last_undegraded":"2024-02-05T07:34:39.732832+0000","last_fullsized":"2024-02-05T07:34:39.732832+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:33:33.995051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.995051+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:40.380483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":574,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720836+0000","last_change":"2024-02-05T07:30:40.503947+0000","last_active":"2024-02-05T07:34:39.720836+0000","last_peered":"2024-02-05T07:34:39.720836+0000","last_clean":"2024-02-05T07:34:39.720836+0000","last_became_active":"2024-02-05T07:30:40.503627+0000","last_became_peered":"2024-02-05T07:30:40.503627+0000","last_unstale":"2024-02-05T07:34:39.720836+0000","last_undegraded":"2024-02-05T07:34:39.720836+0000","last_fullsized":"2024-02-05T07:34:39.720836+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:34.663987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:34.663987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":815,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.727896+0000","last_change":"2024-02-05T07:30:40.884273+0000","last_active":"2024-02-05T07:34:39.727896+0000","last_peered":"2024-02-05T07:34:39.727896+0000","last_clean":"2024-02-05T07:34:39.727896+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:34:39.727896+0000","last_undegraded":"2024-02-05T07:34:39.727896+0000","last_fullsized":"2024-02-05T07:34:39.727896+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:40.884197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.884197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":828,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.722122+0000","last_change":"2024-02-05T07:29:59.002788+0000","last_active":"2024-02-05T07:34:39.722122+0000","last_peered":"2024-02-05T07:34:39.722122+0000","last_clean":"2024-02-05T07:34:39.722122+0000","last_became_active":"2024-02-05T07:28:26.590448+0000","last_became_peered":"2024-02-05T07:28:26.590448+0000","last_unstale":"2024-02-05T07:34:39.722122+0000","last_undegraded":"2024-02-05T07:34:39.722122+0000","last_fullsized":"2024-02-05T07:34:39.722122+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.002677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.002677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"325'7","reported_seq":855,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720754+0000","last_change":"2024-02-05T07:32:40.597579+0000","last_active":"2024-02-05T07:34:39.720754+0000","last_peered":"2024-02-05T07:34:39.720754+0000","last_clean":"2024-02-05T07:34:39.720754+0000","last_became_active":"2024-02-05T07:27:16.745897+0000","last_became_peered":"2024-02-05T07:27:16.745897+0000","last_unstale":"2024-02-05T07:34:39.720754+0000","last_undegraded":"2024-02-05T07:34:39.720754+0000","last_fullsized":"2024-02-05T07:34:39.720754+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'7","last_scrub_stamp":"2024-02-05T07:32:40.597479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:40.597479+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":646,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720971+0000","last_change":"2024-02-05T07:30:39.726603+0000","last_active":"2024-02-05T07:34:39.720971+0000","last_peered":"2024-02-05T07:34:39.720971+0000","last_clean":"2024-02-05T07:34:39.720971+0000","last_became_active":"2024-02-05T07:28:26.590322+0000","last_became_peered":"2024-02-05T07:28:26.590322+0000","last_unstale":"2024-02-05T07:34:39.720971+0000","last_undegraded":"2024-02-05T07:34:39.720971+0000","last_fullsized":"2024-02-05T07:34:39.720971+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.726525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.726525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":201,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.722014+0000","last_change":"2024-02-05T07:33:44.867890+0000","last_active":"2024-02-05T07:34:39.722014+0000","last_peered":"2024-02-05T07:34:39.722014+0000","last_clean":"2024-02-05T07:34:39.722014+0000","last_became_active":"2024-02-05T07:30:40.504365+0000","last_became_peered":"2024-02-05T07:30:40.504365+0000","last_unstale":"2024-02-05T07:34:39.722014+0000","last_undegraded":"2024-02-05T07:34:39.722014+0000","last_fullsized":"2024-02-05T07:34:39.722014+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.867764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.867764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:55.577507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":622,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.724567+0000","last_change":"2024-02-05T07:29:50.742042+0000","last_active":"2024-02-05T07:34:39.724567+0000","last_peered":"2024-02-05T07:34:39.724567+0000","last_clean":"2024-02-05T07:34:39.724567+0000","last_became_active":"2024-02-05T07:27:16.769581+0000","last_became_peered":"2024-02-05T07:27:16.769581+0000","last_unstale":"2024-02-05T07:34:39.724567+0000","last_undegraded":"2024-02-05T07:34:39.724567+0000","last_fullsized":"2024-02-05T07:34:39.724567+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:50.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"325'19","reported_seq":879,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721897+0000","last_change":"2024-02-05T07:30:00.001976+0000","last_active":"2024-02-05T07:34:39.721897+0000","last_peered":"2024-02-05T07:34:39.721897+0000","last_clean":"2024-02-05T07:34:39.721897+0000","last_became_active":"2024-02-05T07:27:16.707036+0000","last_became_peered":"2024-02-05T07:27:16.707036+0000","last_unstale":"2024-02-05T07:34:39.721897+0000","last_undegraded":"2024-02-05T07:34:39.721897+0000","last_fullsized":"2024-02-05T07:34:39.721897+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'19","last_scrub_stamp":"2024-02-05T07:30:00.001863+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:30:00.001863+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":649,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721853+0000","last_change":"2024-02-05T07:29:53.022701+0000","last_active":"2024-02-05T07:34:39.721853+0000","last_peered":"2024-02-05T07:34:39.721853+0000","last_clean":"2024-02-05T07:34:39.721853+0000","last_became_active":"2024-02-05T07:28:26.616107+0000","last_became_peered":"2024-02-05T07:28:26.616107+0000","last_unstale":"2024-02-05T07:34:39.721853+0000","last_undegraded":"2024-02-05T07:34:39.721853+0000","last_fullsized":"2024-02-05T07:34:39.721853+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.022606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.022606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":195,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720662+0000","last_change":"2024-02-05T07:30:40.491944+0000","last_active":"2024-02-05T07:34:39.720662+0000","last_peered":"2024-02-05T07:34:39.720662+0000","last_clean":"2024-02-05T07:34:39.720662+0000","last_became_active":"2024-02-05T07:30:40.491630+0000","last_became_peered":"2024-02-05T07:30:40.491630+0000","last_unstale":"2024-02-05T07:34:39.720662+0000","last_undegraded":"2024-02-05T07:34:39.720662+0000","last_fullsized":"2024-02-05T07:34:39.720662+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:54.207089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:54.207089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":411,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.736042+0000","last_change":"2024-02-05T07:34:30.676878+0000","last_active":"2024-02-05T07:34:39.736042+0000","last_peered":"2024-02-05T07:34:39.736042+0000","last_clean":"2024-02-05T07:34:39.736042+0000","last_became_active":"2024-02-05T07:34:30.676596+0000","last_became_peered":"2024-02-05T07:34:30.676596+0000","last_unstale":"2024-02-05T07:34:39.736042+0000","last_undegraded":"2024-02-05T07:34:39.736042+0000","last_fullsized":"2024-02-05T07:34:39.736042+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:47.484894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.484894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:11.445490+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"326'27","reported_seq":432,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.710502+0000","last_change":"2024-02-05T07:33:25.674216+0000","last_active":"2024-02-05T07:34:36.710502+0000","last_peered":"2024-02-05T07:34:36.710502+0000","last_clean":"2024-02-05T07:34:36.710502+0000","last_became_active":"2024-02-05T07:27:16.751179+0000","last_became_peered":"2024-02-05T07:27:16.751179+0000","last_unstale":"2024-02-05T07:34:36.710502+0000","last_undegraded":"2024-02-05T07:34:36.710502+0000","last_fullsized":"2024-02-05T07:34:36.710502+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:33:25.674080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:25.674080+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:47.794161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":402,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.711411+0000","last_change":"2024-02-05T07:32:49.743375+0000","last_active":"2024-02-05T07:34:36.711411+0000","last_peered":"2024-02-05T07:34:36.711411+0000","last_clean":"2024-02-05T07:34:36.711411+0000","last_became_active":"2024-02-05T07:28:26.595627+0000","last_became_peered":"2024-02-05T07:28:26.595627+0000","last_unstale":"2024-02-05T07:34:36.711411+0000","last_undegraded":"2024-02-05T07:34:36.711411+0000","last_fullsized":"2024-02-05T07:34:36.711411+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:49.743255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:32:49.743255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":831,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720492+0000","last_change":"2024-02-05T07:33:46.937819+0000","last_active":"2024-02-05T07:34:39.720492+0000","last_peered":"2024-02-05T07:34:39.720492+0000","last_clean":"2024-02-05T07:34:39.720492+0000","last_became_active":"2024-02-05T07:28:25.778935+0000","last_became_peered":"2024-02-05T07:28:25.778935+0000","last_unstale":"2024-02-05T07:34:39.720492+0000","last_undegraded":"2024-02-05T07:34:39.720492+0000","last_fullsized":"2024-02-05T07:34:39.720492+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:46.937738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.937738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:47.605598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":415,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-02-05T07:34:36.710815+0000","last_change":"2024-02-05T07:34:28.009081+0000","last_active":"2024-02-05T07:34:36.710815+0000","last_peered":"2024-02-05T07:34:36.710815+0000","last_clean":"2024-02-05T07:34:36.710815+0000","last_became_active":"2024-02-05T07:30:40.493086+0000","last_became_peered":"2024-02-05T07:30:40.493086+0000","last_unstale":"2024-02-05T07:34:36.710815+0000","last_undegraded":"2024-02-05T07:34:36.710815+0000","last_fullsized":"2024-02-05T07:34:36.710815+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:28.008966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.008966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:29.986309+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"450'19","reported_seq":856,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.720658+0000","last_change":"2024-02-05T07:34:34.853204+0000","last_active":"2024-02-05T07:34:39.720658+0000","last_peered":"2024-02-05T07:34:39.720658+0000","last_clean":"2024-02-05T07:34:39.720658+0000","last_became_active":"2024-02-05T07:28:25.777815+0000","last_became_peered":"2024-02-05T07:28:25.777815+0000","last_unstale":"2024-02-05T07:34:39.720658+0000","last_undegraded":"2024-02-05T07:34:39.720658+0000","last_fullsized":"2024-02-05T07:34:39.720658+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'19","last_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_deep_scrub":"450'19","last_deep_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_clean_scrub_stamp":"2024-02-05T07:34:34.853068+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:55.869683+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":594,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.732527+0000","last_change":"2024-02-05T07:33:44.117187+0000","last_active":"2024-02-05T07:34:39.732527+0000","last_peered":"2024-02-05T07:34:39.732527+0000","last_clean":"2024-02-05T07:34:39.732527+0000","last_became_active":"2024-02-05T07:28:26.588496+0000","last_became_peered":"2024-02-05T07:28:26.588496+0000","last_unstale":"2024-02-05T07:34:39.732527+0000","last_undegraded":"2024-02-05T07:34:39.732527+0000","last_fullsized":"2024-02-05T07:34:39.732527+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.117071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.117071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:49.356777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":656,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.731103+0000","last_change":"2024-02-05T07:34:30.668015+0000","last_active":"2024-02-05T07:34:39.731103+0000","last_peered":"2024-02-05T07:34:39.731103+0000","last_clean":"2024-02-05T07:34:39.731103+0000","last_became_active":"2024-02-05T07:34:30.667680+0000","last_became_peered":"2024-02-05T07:34:30.667680+0000","last_unstale":"2024-02-05T07:34:39.731103+0000","last_undegraded":"2024-02-05T07:34:39.731103+0000","last_fullsized":"2024-02-05T07:34:39.731103+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.377472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.377472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:47.301673+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":822,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-02-05T07:34:39.721723+0000","last_change":"2024-02-05T07:32:43.608109+0000","last_active":"2024-02-05T07:34:39.721723+0000","last_peered":"2024-02-05T07:34:39.721723+0000","last_clean":"2024-02-05T07:34:39.721723+0000","last_became_active":"2024-02-05T07:30:40.490288+0000","last_became_peered":"2024-02-05T07:30:40.490288+0000","last_unstale":"2024-02-05T07:34:39.721723+0000","last_undegraded":"2024-02-05T07:34:39.721723+0000","last_fullsized":"2024-02-05T07:34:39.721723+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:43.608002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:32:43.608002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5443878912,"num_objects":83067,"num_object_clones":0,"num_object_copies":249201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83067,"num_write_kb":5316288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16348413952,"data_stored":16348413952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56539,"ondisk_log_size":56539,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":240,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":26,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2816,"ondisk_log_size":2816,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":6480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":11876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":742,"ondisk_log_size":742,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":257698037955,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":301,"seq":1292785156172,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2673444,"kb_used_data":2275860,"kb_used_omap":5,"kb_used_meta":397562,"kb_avail":91071708,"statfs":{"total":95995035648,"available":93257428992,"internally_reserved":0,"allocated":2330480640,"data_stored":2330099893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":407104005},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:34:36 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.65700000000000003,"15min":1.7030000000000001},"min":{"1min":0.22800000000000001,"5min":0.192,"15min":0.192},"max":{"1min":2.391,"5min":4.4119999999999999,"15min":14.218},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.208,"5min":0.84299999999999997,"15min":1.7370000000000001},"min":{"1min":0.34899999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":14.175000000000001},"last":2.1930000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:34:36 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":2.133,"15min":1.8460000000000001},"min":{"1min":0.26600000000000001,"5min":0.217,"15min":0.217},"max":{"1min":2.8500000000000001,"5min":55.923000000000002,"15min":55.923000000000002},"last":1.931},{"interface":"front","average":{"1min":0.82299999999999995,"5min":2.4990000000000001,"15min":2.5710000000000002},"min":{"1min":0.29499999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.6219999999999999,"5min":61.103000000000002,"15min":61.103000000000002},"last":2.6219999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:34:36 2024","interfaces":[{"interface":"back","average":{"1min":3.395,"5min":1.1379999999999999,"15min":2.3679999999999999},"min":{"1min":0.318,"5min":0.249,"15min":0.249},"max":{"1min":32.103000000000002,"5min":32.103000000000002,"15min":34.689999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":3.2650000000000001,"5min":1.294,"15min":2.831},"min":{"1min":0.35599999999999998,"5min":0.245,"15min":0.245},"max":{"1min":32.149999999999999,"5min":32.149999999999999,"15min":34.646999999999998},"last":0.42899999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:34:36 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":3.0739999999999998,"15min":2.484},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.5019999999999998,"5min":55.978999999999999,"15min":55.978999999999999},"last":1.21},{"interface":"front","average":{"1min":0.89200000000000002,"5min":3.0779999999999998,"15min":2.706},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.9859999999999998,"5min":55.631,"15min":55.631},"last":2.54}]},{"osd":4,"last update":"Mon Feb 5 07:34:36 2024","interfaces":[{"interface":"back","average":{"1min":7.4859999999999998,"5min":1.833,"15min":1.1220000000000001},"min":{"1min":0.29299999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":151.02099999999999,"5min":151.02099999999999,"15min":151.02099999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":7.6029999999999998,"5min":1.8600000000000001,"15min":1.0580000000000001},"min":{"1min":0.29999999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":149.21799999999999,"5min":149.21799999999999,"15min":149.21799999999999},"last":0.50700000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:34:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.5,"15min":0.57599999999999996},"min":{"1min":0.34100000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.8919999999999999,"5min":2.8919999999999999,"15min":2.8919999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47799999999999998,"15min":0.55600000000000005},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.72999999999999998,"5min":0.83899999999999997,"15min":1.526},"last":0.44600000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:34:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.83199999999999996,"15min":4.0069999999999997},"min":{"1min":0.35699999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.999,"5min":30.998999999999999,"15min":84.158000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.86499999999999999,"15min":4.1360000000000001},"min":{"1min":0.37,"5min":0.312,"15min":0.312},"max":{"1min":1.0129999999999999,"5min":31.067,"15min":85.018000000000001},"last":0.378}]}]},{"osd":1,"up_from":13,"seq":55834575073,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2350364,"kb_used_data":1655568,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":91394788,"statfs":{"total":95995035648,"available":93588262912,"internally_reserved":0,"allocated":1695301632,"data_stored":1694867072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.4450000000000001,"15min":2.1419999999999999},"min":{"1min":0.248,"5min":0.20899999999999999,"15min":0.16800000000000001},"max":{"1min":3.3969999999999998,"5min":15.09,"15min":45.393000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.587,"5min":2.028,"15min":2.3109999999999999},"min":{"1min":0.28999999999999998,"5min":0.23799999999999999,"15min":0.20899999999999999},"max":{"1min":8.2609999999999992,"5min":14.313000000000001,"15min":45.031999999999996},"last":4.0439999999999996}]},{"osd":2,"last update":"Mon Feb 5 07:34:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.3979999999999999,"15min":1.806},"min":{"1min":0.254,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":2.8119999999999998,"5min":15.137,"15min":39.259999999999998},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.5329999999999999,"15min":1.7230000000000001},"min":{"1min":0.23699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.9489999999999998,"5min":14.906000000000001,"15min":40.972000000000001},"last":3.6869999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:34:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":2.5510000000000002,"15min":2.145},"min":{"1min":0.245,"5min":0.22900000000000001,"15min":0.20999999999999999},"max":{"1min":3.5590000000000002,"5min":46.777000000000001,"15min":46.777000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.103,"5min":2.452,"15min":2.1400000000000001},"min":{"1min":0.28899999999999998,"5min":0.24299999999999999,"15min":0.24199999999999999},"max":{"1min":4.9790000000000001,"5min":50.164999999999999,"15min":50.164999999999999},"last":3.597}]},{"osd":4,"last update":"Mon Feb 5 07:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.29,"15min":1.742},"min":{"1min":0.41299999999999998,"5min":0.32600000000000001,"15min":0.312},"max":{"1min":2.02,"5min":8.9939999999999998,"15min":61.722000000000001},"last":3.6299999999999999},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.119,"15min":1.7},"min":{"1min":0.38900000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.4780000000000002,"5min":5.5590000000000002,"15min":61.786000000000001},"last":0.48599999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.7569999999999999,"15min":1.994},"min":{"1min":0.433,"5min":0.33900000000000002,"15min":0.309},"max":{"1min":4.9169999999999998,"5min":46.398000000000003,"15min":106.672},"last":3.5369999999999999},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.653,"15min":2.04},"min":{"1min":0.438,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.3370000000000002,"5min":8.5410000000000004,"15min":107.348},"last":0.88700000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:34:31 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.98199999999999998,"15min":1.5629999999999999},"min":{"1min":0.34399999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.9379999999999999,"5min":3.5259999999999998,"15min":5.5990000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.294,"5min":1.258,"15min":1.8560000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":4.9530000000000003,"5min":8.923,"15min":8.923},"last":3.6589999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:34:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":2.1949999999999998,"15min":1.929},"min":{"1min":0.46000000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.9710000000000001,"5min":46.351999999999997,"15min":66.703999999999994},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.77200000000000002,"5min":1.149,"15min":1.5649999999999999},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5820000000000001,"5min":5.6539999999999999,"15min":61.594999999999999},"last":0.54700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738598,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4132220,"kb_used_data":3402728,"kb_used_omap":1,"kb_used_meta":729470,"kb_avail":89612932,"statfs":{"total":95995035648,"available":91763642368,"internally_reserved":0,"allocated":3484393472,"data_stored":3483948445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":746977733},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:33:41 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":2.4860000000000002,"15min":2.0619999999999998},"min":{"1min":0.21099999999999999,"5min":0.192,"15min":0.16500000000000001},"max":{"1min":8.5939999999999994,"5min":32.546999999999997,"15min":45.372},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":2.3809999999999998,"15min":2.105},"min":{"1min":0.27100000000000002,"5min":0.217,"15min":0.186},"max":{"1min":2.0019999999999998,"5min":29.675999999999998,"15min":45.231999999999999},"last":0.54300000000000004}]},{"osd":2,"last update":"Mon Feb 5 07:34:12 2024","interfaces":[{"interface":"back","average":{"1min":2.9289999999999998,"5min":1.5549999999999999,"15min":1.9079999999999999},"min":{"1min":0.26400000000000001,"5min":0.191,"15min":0.191},"max":{"1min":40.548000000000002,"5min":40.548000000000002,"15min":65.903999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":2.6699999999999999,"5min":1.458,"15min":1.857},"min":{"1min":0.246,"5min":0.224,"15min":0.182},"max":{"1min":40.643000000000001,"5min":40.643000000000001,"15min":65.694000000000003},"last":2.831}]},{"osd":3,"last update":"Mon Feb 5 07:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":2.024,"15min":2.133},"min":{"1min":0.27600000000000002,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":3.109,"5min":29.629000000000001,"15min":65.155000000000001},"last":2.919},{"interface":"front","average":{"1min":1.125,"5min":2.306,"15min":2.347},"min":{"1min":0.26600000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.6509999999999998,"5min":29.463999999999999,"15min":65.114000000000004},"last":0.505}]},{"osd":4,"last update":"Mon Feb 5 07:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":1.2430000000000001,"15min":1.901},"min":{"1min":0.29099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.8130000000000002,"5min":42.901000000000003,"15min":65.188000000000002},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":1.4079999999999999,"15min":1.968},"min":{"1min":0.36599999999999999,"5min":0.27100000000000002,"15min":0.23699999999999999},"max":{"1min":3.5059999999999998,"5min":42.960999999999999,"15min":64.873000000000005},"last":0.41199999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.83899999999999997,"15min":1.7050000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.27000000000000002},"max":{"1min":2.27,"5min":4.6959999999999997,"15min":95.772999999999996},"last":2.8900000000000001},{"interface":"front","average":{"1min":1.296,"5min":1.0029999999999999,"15min":1.79},"min":{"1min":0.377,"5min":0.29299999999999998,"15min":0.27600000000000002},"max":{"1min":5.9340000000000002,"5min":7.1929999999999996,"15min":96.028000000000006},"last":0.48699999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":0.94799999999999995,"15min":1.256},"min":{"1min":0.38600000000000001,"5min":0.308,"15min":0.26300000000000001},"max":{"1min":3.7610000000000001,"5min":3.7610000000000001,"15min":4.6929999999999996},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.889,"5min":1.1200000000000001,"15min":1.506},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":8.718,"5min":8.718,"15min":8.718},"last":1.393}]},{"osd":7,"last update":"Mon Feb 5 07:33:55 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.6659999999999999,"15min":1.8160000000000001},"min":{"1min":0.42199999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.891,"5min":43.012,"15min":65.263000000000005},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.83799999999999997,"5min":1.8220000000000001,"15min":1.831},"min":{"1min":0.40799999999999997,"5min":0.34000000000000002,"15min":0.309},"max":{"1min":3.6280000000000001,"5min":43.512,"15min":64.906000000000006},"last":1.3480000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411548,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3381520,"kb_used_data":2664508,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":90363632,"statfs":{"total":95995035648,"available":92532359168,"internally_reserved":0,"allocated":2728456192,"data_stored":2728049420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":1.081,"15min":2.0259999999999998},"min":{"1min":0.218,"5min":0.19600000000000001,"15min":0.182},"max":{"1min":2.1699999999999999,"5min":12.759,"15min":92.638000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.532,"5min":1.4610000000000001,"15min":2.2189999999999999},"min":{"1min":0.26700000000000002,"5min":0.22,"15min":0.20699999999999999},"max":{"1min":11.606999999999999,"5min":12.647,"15min":93.481999999999999},"last":5.452}]},{"osd":1,"last update":"Mon Feb 5 07:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":2.8290000000000002,"15min":2.339},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.19400000000000001},"max":{"1min":3.7469999999999999,"5min":63.823,"15min":93.504000000000005},"last":1.3360000000000001},{"interface":"front","average":{"1min":0.746,"5min":3.0979999999999999,"15min":2.5209999999999999},"min":{"1min":0.26200000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.133,"5min":64.771000000000001,"15min":93.004000000000005},"last":0.76600000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:34:28 2024","interfaces":[{"interface":"back","average":{"1min":1.8819999999999999,"5min":2.7599999999999998,"15min":2.1509999999999998},"min":{"1min":0.28599999999999998,"5min":0.252,"15min":0.214},"max":{"1min":4.5789999999999997,"5min":63.784999999999997,"15min":63.784999999999997},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.22,"5min":2.9540000000000002,"15min":2.093},"min":{"1min":0.376,"5min":0.28100000000000003,"15min":0.193},"max":{"1min":5.1989999999999998,"5min":63.521000000000001,"15min":63.521000000000001},"last":0.78900000000000003}]},{"osd":4,"last update":"Mon Feb 5 07:33:49 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.93500000000000005,"15min":2.016},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.24099999999999999},"max":{"1min":2.5049999999999999,"5min":5.4829999999999997,"15min":168.11699999999999},"last":2.9550000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":0.90700000000000003,"15min":2.0009999999999999},"min":{"1min":0.36099999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.6950000000000003,"5min":5.6680000000000001,"15min":168.589},"last":0.72799999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.85999999999999999,"15min":1.2390000000000001},"min":{"1min":0.41599999999999998,"5min":0.315,"15min":0.315},"max":{"1min":2.8610000000000002,"5min":4.742,"15min":39.923000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.085,"5min":0.94499999999999995,"15min":1.2050000000000001},"min":{"1min":0.48599999999999999,"5min":0.33000000000000002,"15min":0.26400000000000001},"max":{"1min":3.7189999999999999,"5min":4.0709999999999997,"15min":38.920999999999999},"last":1.387}]},{"osd":6,"last update":"Mon Feb 5 07:34:34 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.01,"15min":1.1859999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.5250000000000004,"5min":4.5250000000000004,"15min":4.5250000000000004},"last":1.653},{"interface":"front","average":{"1min":1.2769999999999999,"5min":0.93100000000000005,"15min":1.3180000000000001},"min":{"1min":0.46500000000000002,"5min":0.379,"15min":0.33100000000000002},"max":{"1min":4.6020000000000003,"5min":4.7779999999999996,"15min":5.7240000000000002},"last":0.52300000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.3500000000000001,"15min":1.4299999999999999},"min":{"1min":0.41299999999999998,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":5.7850000000000001,"5min":11.311999999999999,"15min":37.573999999999998},"last":2.992},{"interface":"front","average":{"1min":1.181,"5min":1.0900000000000001,"15min":1.325},"min":{"1min":0.44700000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.9689999999999999,"5min":8.7349999999999994,"15min":37.432000000000002},"last":1.4179999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149910,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2943552,"kb_used_data":2194608,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":90801600,"statfs":{"total":95995035648,"available":92980838400,"internally_reserved":0,"allocated":2247278592,"data_stored":2246855808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":766900679},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:33:41 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.1399999999999999,"15min":2.206},"min":{"1min":0.24399999999999999,"5min":0.192,"15min":0.192},"max":{"1min":3.2469999999999999,"5min":6.1349999999999998,"15min":74.114999999999995},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.014,"5min":2.2320000000000002,"15min":2.3359999999999999},"min":{"1min":0.28000000000000003,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.6699999999999999,"5min":99.989999999999995,"15min":99.989999999999995},"last":7.7450000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:33:41 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":3.423,"15min":2.569},"min":{"1min":0.28899999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.4489999999999998,"5min":102.20399999999999,"15min":102.20399999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.0089999999999999,"5min":3.2610000000000001,"15min":2.8290000000000002},"min":{"1min":0.318,"5min":0.255,"15min":0.185},"max":{"1min":3.399,"5min":102.286,"15min":102.286},"last":0.55000000000000004}]},{"osd":2,"last update":"Mon Feb 5 07:33:41 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":2.2370000000000001,"15min":2.4910000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.3759999999999999,"5min":99.847999999999999,"15min":99.847999999999999},"last":1.5960000000000001},{"interface":"front","average":{"1min":0.97499999999999998,"5min":2.194,"15min":2.036},"min":{"1min":0.29299999999999998,"5min":0.26800000000000002,"15min":0.22500000000000001},"max":{"1min":4.6950000000000003,"5min":99.950999999999993,"15min":99.950999999999993},"last":0.20200000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:33:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":2.0859999999999999,"15min":1.5620000000000001},"min":{"1min":0.39600000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.589,"5min":100.024,"15min":100.024},"last":1.248},{"interface":"front","average":{"1min":0.877,"5min":2.4279999999999999,"15min":1.6419999999999999},"min":{"1min":0.48299999999999998,"5min":0.30399999999999999,"15min":0.28599999999999998},"max":{"1min":3.4249999999999998,"5min":99.859999999999999,"15min":99.859999999999999},"last":0.73399999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":2.234,"15min":1.897},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.28499999999999998},"max":{"1min":2.9670000000000001,"5min":99.911000000000001,"15min":99.911000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.81799999999999995,"5min":2.1880000000000002,"15min":1.619},"min":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.27400000000000002},"max":{"1min":2.8879999999999999,"5min":100.053,"15min":100.053},"last":1.3140000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:33:41 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":2.169,"15min":1.343},"min":{"1min":0.44700000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.2879999999999998,"5min":99.807000000000002,"15min":99.807000000000002},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.0800000000000001,"5min":2.1859999999999999,"15min":1.3959999999999999},"min":{"1min":0.44700000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.6050000000000004,"5min":101.489,"15min":101.489},"last":1.288}]},{"osd":7,"last update":"Mon Feb 5 07:33:58 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":3.177,"15min":2.3039999999999998},"min":{"1min":0.45200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.4620000000000002,"5min":99.893000000000001,"15min":99.893000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.81699999999999995,"5min":2.8660000000000001,"15min":2.5640000000000001},"min":{"1min":0.51600000000000001,"5min":0.375,"15min":0.375},"max":{"1min":2.206,"5min":99.879000000000005,"15min":99.879000000000005},"last":0.67000000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953682,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4326032,"kb_used_data":3681212,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":89419120,"statfs":{"total":95995035648,"available":91565178880,"internally_reserved":0,"allocated":3769561088,"data_stored":3769152988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.77600000000000002,"15min":2.0979999999999999},"min":{"1min":0.35299999999999998,"5min":0.219,"15min":0.20300000000000001},"max":{"1min":1.363,"5min":13.103999999999999,"15min":101.212},"last":1.7430000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.91500000000000004,"15min":2.194},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23300000000000001},"max":{"1min":2.492,"5min":13.169,"15min":102.07599999999999},"last":0.57799999999999996}]},{"osd":1,"last update":"Mon Feb 5 07:33:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":2.4039999999999999,"15min":2.3109999999999999},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":6.9279999999999999,"5min":45.655999999999999,"15min":69.102000000000004},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.208,"5min":2.6179999999999999,"15min":2.5430000000000001},"min":{"1min":0.35299999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.79,"5min":46.531999999999996,"15min":68.108999999999995},"last":1.133}]},{"osd":2,"last update":"Mon Feb 5 07:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.755,"15min":1.347},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.0369999999999999,"5min":4.266,"15min":67.144999999999996},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.0700000000000001,"15min":1.413},"min":{"1min":0.39000000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.9429999999999996,"5min":9.8209999999999997,"15min":66.959999999999994},"last":0.28199999999999997}]},{"osd":3,"last update":"Mon Feb 5 07:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":2.8940000000000001,"15min":1.8640000000000001},"min":{"1min":0.29499999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.8599999999999999,"5min":46.549999999999997,"15min":66.019999999999996},"last":0.68700000000000006},{"interface":"front","average":{"1min":1.1879999999999999,"5min":2.9750000000000001,"15min":2.085},"min":{"1min":0.45900000000000002,"5min":0.33300000000000002,"15min":0.26400000000000001},"max":{"1min":7.9939999999999998,"5min":46.003999999999998,"15min":101.001},"last":1.6299999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.47799999999999998,"15min":0.59799999999999998},"min":{"1min":0.36899999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.69999999999999996,"5min":1.907,"15min":10.917999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.47399999999999998,"15min":0.626},"min":{"1min":0.30599999999999999,"5min":0.27700000000000002,"15min":0.24199999999999999},"max":{"1min":1.3939999999999999,"5min":2.2410000000000001,"15min":15.105},"last":0.26700000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.50900000000000001,"15min":1.081},"min":{"1min":0.34699999999999998,"5min":0.25600000000000001,"15min":0.245},"max":{"1min":1.236,"5min":1.2470000000000001,"15min":14.606},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.51900000000000002,"15min":1.3009999999999999},"min":{"1min":0.44600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":13.114000000000001},"last":0.32100000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:33:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57099999999999995,"15min":0.72199999999999998},"min":{"1min":0.32400000000000001,"5min":0.30599999999999999,"15min":0.25700000000000001},"max":{"1min":0.88300000000000001,"5min":2.8260000000000001,"15min":15.073},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57699999999999996,"15min":0.69299999999999995},"min":{"1min":0.34300000000000003,"5min":0.316,"15min":0.248},"max":{"1min":0.76800000000000002,"5min":3.0670000000000002,"15min":14.581},"last":0.34000000000000002}]}]},{"osd":5,"up_from":0,"seq":176093659341,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3492478976,"data_stored":3492478976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1704394752,"data_stored":1704394752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2731671552,"data_stored":2731671552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2278293504,"data_stored":2278293504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3798466560,"data_stored":3798466560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2343108608,"data_stored":2343108608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:34:46.140 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-02-05T07:34:46.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-02-05T07:34:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:46.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:46.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:46.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:45 smithi090 ceph-mon[33213]: osdmap e460: 8 total, 8 up, 6 in 2024-02-05T07:34:46.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:45 smithi090 ceph-mon[33213]: pgmap v836: 173 pgs: 173 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-05T07:34:46.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:45 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:34:46.277 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:46.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:46.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:46.404 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:46.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:46.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:46.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:46.535 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:46.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:46.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:46.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:46.691 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:46.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:46.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 59 16 90914 90898 96.2663 68.3125 0.00992685 0.010383 2024-02-05T07:34:46.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:46.834 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:46.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:46.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:46.936 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:46.978 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:46.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:46.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:47.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:47.113 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:47.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:47.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:46 smithi007 ceph-mon[29098]: osdmap e461: 8 total, 8 up, 6 in 2024-02-05T07:34:47.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:46 smithi007 ceph-mon[37158]: osdmap e461: 8 total, 8 up, 6 in 2024-02-05T07:34:47.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:47.251 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:47.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:47.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:47.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:46 smithi090 ceph-mon[33213]: osdmap e461: 8 total, 8 up, 6 in 2024-02-05T07:34:47.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:47.377 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:47.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:47.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:47.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:47.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:47.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:47.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:47.642 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:47.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:47.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:47.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:47.771 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:47.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:47.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:34:47.776372+0000 min lat: 0.00131497 max lat: 0.307728 avg lat: 0.0104263 2024-02-05T07:34:47.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:34:47.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 60 16 92059 92043 95.8545 71.5625 0.00968462 0.0104263 2024-02-05T07:34:47.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:47.915 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:47.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:48.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:48.048 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:48.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:48.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:48.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:48.165 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:47 smithi090 ceph-mon[33213]: osdmap e462: 8 total, 8 up, 6 in 2024-02-05T07:34:48.165 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:47 smithi090 ceph-mon[33213]: pgmap v839: 172 pgs: 1 peering, 171 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T07:34:48.165 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:47 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-05T07:34:48.165 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:47 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-05T07:34:48.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:47 smithi007 ceph-mon[29098]: osdmap e462: 8 total, 8 up, 6 in 2024-02-05T07:34:48.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:47 smithi007 ceph-mon[29098]: pgmap v839: 172 pgs: 1 peering, 171 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T07:34:48.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:47 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-05T07:34:48.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:47 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-05T07:34:48.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:47 smithi007 ceph-mon[37158]: osdmap e462: 8 total, 8 up, 6 in 2024-02-05T07:34:48.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:47 smithi007 ceph-mon[37158]: pgmap v839: 172 pgs: 1 peering, 171 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T07:34:48.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:47 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-05T07:34:48.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:47 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-05T07:34:48.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:48.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:48.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:48.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:48.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:48.441 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:48.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:48.543 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:48.567 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:48.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:48.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:48.709 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:48.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:48.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:48.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 61 16 93111 93095 95.361 65.75 0.0176088 0.0104817 2024-02-05T07:34:48.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:48.838 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:48.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:48.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:48.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:48.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:49.064 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:49.100 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:49.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:48 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-05T07:34:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:48 smithi007 ceph-mon[29098]: osdmap e463: 8 total, 8 up, 6 in 2024-02-05T07:34:49.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:48 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-05T07:34:49.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:48 smithi007 ceph-mon[37158]: osdmap e463: 8 total, 8 up, 6 in 2024-02-05T07:34:49.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:49.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:49.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:49.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:48 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-05T07:34:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:48 smithi090 ceph-mon[33213]: osdmap e463: 8 total, 8 up, 6 in 2024-02-05T07:34:49.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:49.352 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:49.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:49.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:49.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:49.501 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:49.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:49.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:49.625 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:49.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:49.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:49.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:49.756 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:49.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:49.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:49.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 62 16 94016 94000 94.7352 56.5625 0.0119629 0.0105503 2024-02-05T07:34:49.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:34:49.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:49.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:49.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:49.988 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:50.013 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:50.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:50.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:49 smithi007 ceph-mon[29098]: osdmap e464: 8 total, 8 up, 6 in 2024-02-05T07:34:50.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:49 smithi007 ceph-mon[29098]: pgmap v842: 172 pgs: 1 peering, 171 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 55 MiB/s wr, 879 op/s 2024-02-05T07:34:50.020 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:49 smithi007 ceph-mon[37158]: osdmap e464: 8 total, 8 up, 6 in 2024-02-05T07:34:50.020 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:49 smithi007 ceph-mon[37158]: pgmap v842: 172 pgs: 1 peering, 171 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 55 MiB/s wr, 879 op/s 2024-02-05T07:34:50.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:50.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:50.244 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:50.245 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:34:50.271 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:50.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:50.272 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:34:50.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:49 smithi090 ceph-mon[33213]: osdmap e464: 8 total, 8 up, 6 in 2024-02-05T07:34:50.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:49 smithi090 ceph-mon[33213]: pgmap v842: 172 pgs: 1 peering, 171 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 55 MiB/s wr, 879 op/s 2024-02-05T07:34:50.313 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:34:50.361 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:34:50.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:50.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:50.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:50.404 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:34:50.443 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:34:50.482 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:34:50.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:50.520 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:34:50.533 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:34:50.533 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:34:50.533 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:34:50.533 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:34:50.533 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:34:50.533 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:34:50.533 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:34:50.533 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:34:50.535 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:50.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:50.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:50.668 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:50.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:50.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:50.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 63 16 95181 95165 94.3872 72.8125 0.00680643 0.0105896 2024-02-05T07:34:50.800 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:50.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:50.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:50.944 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:50.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:51.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:34:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:34:51.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:50 smithi090 ceph-mon[33213]: osdmap e465: 8 total, 8 up, 6 in 2024-02-05T07:34:51.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:50 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1258708017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:51.045 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:51.082 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:51.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:51.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:51.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:50 smithi007 ceph-mon[29098]: osdmap e465: 8 total, 8 up, 6 in 2024-02-05T07:34:51.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:50 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1258708017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:51.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:50 smithi007 ceph-mon[37158]: osdmap e465: 8 total, 8 up, 6 in 2024-02-05T07:34:51.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:50 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1258708017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:51.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:51.216 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:51.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:51.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:51.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:51.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:51.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:51.478 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:51.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:51.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:51.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:51.614 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:51.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:51.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:51.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:51.746 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:34:51.746 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:34:51.757 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:51.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:51.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 64 16 96549 96533 94.2482 85.5 0.00798941 0.0106057 2024-02-05T07:34:51.859 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:51.881 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:51.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:51.918 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:51.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:52.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:52.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:51 smithi090 ceph-mon[33213]: osdmap e466: 8 total, 8 up, 6 in 2024-02-05T07:34:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:51 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:51 smithi090 ceph-mon[33213]: pgmap v845: 171 pgs: 2 peering, 169 active+clean; 5.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T07:34:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:51 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-05T07:34:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:51 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-05T07:34:52.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:51 smithi007 ceph-mon[37158]: osdmap e466: 8 total, 8 up, 6 in 2024-02-05T07:34:52.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:51 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:52.038 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:51 smithi007 ceph-mon[37158]: pgmap v845: 171 pgs: 2 peering, 169 active+clean; 5.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T07:34:52.038 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:51 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-05T07:34:52.038 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:51 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-05T07:34:52.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:51 smithi007 ceph-mon[29098]: osdmap e466: 8 total, 8 up, 6 in 2024-02-05T07:34:52.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:51 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:34:52.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:51 smithi007 ceph-mon[29098]: pgmap v845: 171 pgs: 2 peering, 169 active+clean; 5.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T07:34:52.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:51 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-05T07:34:52.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:51 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-05T07:34:52.068 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:52.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:52.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:52.201 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:52.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:52.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:52.332 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:52.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:52.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:52.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T07:34:52.477 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:52.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:52.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:52.579 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:52.620 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:52.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:52.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:52.765 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:34:52.768 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:52.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:52.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 65 16 97790 97774 93.9914 77.5625 0.0260434 0.0106339 2024-02-05T07:34:52.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:52.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:52.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:52.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:53.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:53.067 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:53.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:53.168 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:52 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-05T07:34:53.169 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:52 smithi090 ceph-mon[33213]: osdmap e467: 8 total, 8 up, 6 in 2024-02-05T07:34:53.169 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:53.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:52 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-05T07:34:53.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:52 smithi007 ceph-mon[29098]: osdmap e467: 8 total, 8 up, 6 in 2024-02-05T07:34:53.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:52 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-05T07:34:53.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:52 smithi007 ceph-mon[37158]: osdmap e467: 8 total, 8 up, 6 in 2024-02-05T07:34:53.206 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:53.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:53.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:53.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:53.339 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:53.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:53.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:53.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:34:53.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:53.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:53.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:53.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:53.618 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:53.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:53.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:53.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 66 16 99012 98996 93.7245 76.375 0.0125692 0.0106644 2024-02-05T07:34:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:53.800 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:53.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:53.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:53.902 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:34:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:34:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:34:53.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:53 smithi007 ceph-mon[29098]: osdmap e468: 8 total, 8 up, 6 in 2024-02-05T07:34:53.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:53 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3840736351' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T07:34:53.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:53 smithi007 ceph-mon[29098]: pgmap v848: 171 pgs: 1 peering, 170 active+clean; 5.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 793 op/s 2024-02-05T07:34:53.937 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pg_num to 11 2024-02-05T07:34:53.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:53.957 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:53.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:53.992 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:53 smithi090 ceph-mon[33213]: osdmap e468: 8 total, 8 up, 6 in 2024-02-05T07:34:53.992 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:53 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3840736351' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T07:34:53.992 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:53 smithi090 ceph-mon[33213]: pgmap v848: 171 pgs: 1 peering, 170 active+clean; 5.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 793 op/s 2024-02-05T07:34:54.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:54.106 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:54.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:54.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:54.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:53 smithi007 ceph-mon[37158]: osdmap e468: 8 total, 8 up, 6 in 2024-02-05T07:34:54.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:53 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3840736351' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T07:34:54.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:53 smithi007 ceph-mon[37158]: pgmap v848: 171 pgs: 1 peering, 170 active+clean; 5.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 793 op/s 2024-02-05T07:34:54.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:54.241 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:54.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:54.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:54.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:54.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:54.481 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:54.522 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:54.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:54.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:54.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:54.673 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:34:54.673 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:34:54.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:54.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 67 16 100308 100292 93.5345 81 0.016265 0.0106868 2024-02-05T07:34:54.799 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:54.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:54.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:54.876 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:34:54.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:54.934 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:54.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:54.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:55.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:54 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3840736351' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-05T07:34:55.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:54 smithi090 ceph-mon[33213]: osdmap e469: 8 total, 8 up, 6 in 2024-02-05T07:34:55.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:54 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:34:55.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:54 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:34:55.036 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:55.083 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:55.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:54 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3840736351' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-05T07:34:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:54 smithi007 ceph-mon[29098]: osdmap e469: 8 total, 8 up, 6 in 2024-02-05T07:34:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:54 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:34:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:54 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:34:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:54 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3840736351' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-05T07:34:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:54 smithi007 ceph-mon[37158]: osdmap e469: 8 total, 8 up, 6 in 2024-02-05T07:34:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:54 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:34:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:54 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:34:55.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:55.229 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:55.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:55.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:55.363 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:55.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:55.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T07:34:55.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:55.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:55.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:55.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:55.630 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:55.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:55.732 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:55.773 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:55.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:55.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:55.777 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 68 16 101524 101508 93.2765 76 0.0128595 0.0107157 2024-02-05T07:34:55.852 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:34:55.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:55.912 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:55.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:55.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:55.974 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:34:56.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:56.043 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:56.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:56.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:56.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:56.180 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:56.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:56.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:55 smithi090 ceph-mon[33213]: osdmap e470: 8 total, 8 up, 6 in 2024-02-05T07:34:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:55 smithi090 ceph-mon[33213]: pgmap v851: 170 pgs: 1 clean+premerge+peered, 169 active+clean; 6.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-05T07:34:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:55 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-05T07:34:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:55 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-05T07:34:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:55 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/582241736' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:55 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:56.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:34:56.324 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:56.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:56.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:34:56.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:55 smithi007 ceph-mon[29098]: osdmap e470: 8 total, 8 up, 6 in 2024-02-05T07:34:56.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:55 smithi007 ceph-mon[29098]: pgmap v851: 170 pgs: 1 clean+premerge+peered, 169 active+clean; 6.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-05T07:34:56.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:55 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-05T07:34:56.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:55 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-05T07:34:56.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:55 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/582241736' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:56.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:55 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:56.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:55 smithi007 ceph-mon[37158]: osdmap e470: 8 total, 8 up, 6 in 2024-02-05T07:34:56.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:55 smithi007 ceph-mon[37158]: pgmap v851: 170 pgs: 1 clean+premerge+peered, 169 active+clean; 6.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-05T07:34:56.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:55 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-05T07:34:56.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:55 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-05T07:34:56.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:55 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/582241736' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:56.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:55 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:34:56.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:56.466 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:56.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:56.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:56.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:56.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:56.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:56.740 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:56.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:56.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:56.777 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 69 16 102737 102721 93.0234 75.8125 0.00453796 0.0107453 2024-02-05T07:34:56.842 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:56.879 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:56.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:56.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:57.023 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:57.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:57.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:57.149 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:57.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:57.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:57.266 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:56 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:34:57.266 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:56 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-05T07:34:57.266 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:56 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:34:57.266 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:56 smithi090 ceph-mon[33213]: osdmap e471: 8 total, 8 up, 6 in 2024-02-05T07:34:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:57.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:57.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:57.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:56 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:34:57.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:56 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-05T07:34:57.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:56 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:34:57.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:56 smithi007 ceph-mon[29098]: osdmap e471: 8 total, 8 up, 6 in 2024-02-05T07:34:57.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:56 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:34:57.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:56 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-05T07:34:57.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:56 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:34:57.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:56 smithi007 ceph-mon[37158]: osdmap e471: 8 total, 8 up, 6 in 2024-02-05T07:34:57.426 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:57.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:57.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:57.559 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:57.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:57.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:57.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:57.693 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:57.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:57.777 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 70 16 103948 103932 92.7757 75.6875 0.0129733 0.0107742 2024-02-05T07:34:57.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:57.827 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:57.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:57.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:57.963 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:57.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:58.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:58.107 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:58.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:58.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:58.122 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:57 smithi007 ceph-mon[29098]: osdmap e472: 8 total, 8 up, 6 in 2024-02-05T07:34:58.122 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:57 smithi007 ceph-mon[29098]: pgmap v854: 170 pgs: 1 clean+premerge+peered, 169 active+clean; 6.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-02-05T07:34:58.122 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:57 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T07:34:58.123 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:57 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T07:34:58.123 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:57 smithi007 ceph-mon[37158]: osdmap e472: 8 total, 8 up, 6 in 2024-02-05T07:34:58.123 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:57 smithi007 ceph-mon[37158]: pgmap v854: 170 pgs: 1 clean+premerge+peered, 169 active+clean; 6.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-02-05T07:34:58.123 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:57 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T07:34:58.123 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:57 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T07:34:58.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:34:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:58.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:58.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:34:58.270 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:57 smithi090 ceph-mon[33213]: osdmap e472: 8 total, 8 up, 6 in 2024-02-05T07:34:58.270 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:57 smithi090 ceph-mon[33213]: pgmap v854: 170 pgs: 1 clean+premerge+peered, 169 active+clean; 6.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-02-05T07:34:58.270 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:57 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T07:34:58.270 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:57 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T07:34:58.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:58.386 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:58.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:58.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:58.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:58.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:58.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:58.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:58.677 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:58.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:58.777 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 71 16 105246 105230 92.6115 81.125 0.00876998 0.010793 2024-02-05T07:34:58.779 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:58.792 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:34:58.816 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:58.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:58.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:58.834 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:34:58.873 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:34:58.915 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:34:58.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:34:58.956 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:34:58.962 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:58.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:34:59.036 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:34:59.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:59.076 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:34:59.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:59.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:59.119 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:34:59.129 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:34:59.129 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:34:59.129 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:34:59.129 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:34:59.129 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:34:59.129 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:34:59.129 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:34:59.129 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:34:59.197 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:59 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-05T07:34:59.197 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:59 smithi090 ceph-mon[33213]: osdmap e473: 8 total, 8 up, 6 in 2024-02-05T07:34:59.197 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:34:59 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4152631482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:59.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:34:59.206 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:59 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-05T07:34:59.207 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:59 smithi007 ceph-mon[37158]: osdmap e473: 8 total, 8 up, 6 in 2024-02-05T07:34:59.207 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:34:59 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4152631482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:59.209 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:59 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-05T07:34:59.209 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:59 smithi007 ceph-mon[29098]: osdmap e473: 8 total, 8 up, 6 in 2024-02-05T07:34:59.209 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:34:59 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4152631482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:34:59.240 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:59.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:59.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:34:59.288 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:34:59.288 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:34:59.288 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:34:59.288 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:34:59.289 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:34:59.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:34:59.380 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:59.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:34:59.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:59.527 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:59.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:59.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:34:59.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:59.663 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:59.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:34:59.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:34:59.779 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 72 16 106500 106484 92.4118 78.375 0.00932704 0.0108166 2024-02-05T07:34:59.807 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:34:59.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:59.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:34:59.901 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 3] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:34:59.902 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:34:59.902 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-02-05T07:34:59.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:34:59.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:34:59.940 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:34:59.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:34:59.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:34:59.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T07:35:00.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:00.090 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:00.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:00.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[29098]: osdmap e474: 8 total, 8 up, 6 in 2024-02-05T07:35:00.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[29098]: pgmap v857: 169 pgs: 3 unknown, 3 peering, 163 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-02-05T07:35:00.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:35:00.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:35:00.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T07:35:00.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[29098]: osdmap e475: 8 total, 8 up, 6 in 2024-02-05T07:35:00.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[29098]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:00.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[29098]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:00.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[29098]: 1.b starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:00.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[29098]: 1.c starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:00.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:35:00.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[37158]: osdmap e474: 8 total, 8 up, 6 in 2024-02-05T07:35:00.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[37158]: pgmap v857: 169 pgs: 3 unknown, 3 peering, 163 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-02-05T07:35:00.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:35:00.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:35:00.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T07:35:00.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[37158]: osdmap e475: 8 total, 8 up, 6 in 2024-02-05T07:35:00.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[37158]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:00.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[37158]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:00.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[37158]: 1.b starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:00.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:00 smithi007 ceph-mon[37158]: 1.c starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:00.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:00.193 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:00 smithi090 ceph-mon[33213]: osdmap e474: 8 total, 8 up, 6 in 2024-02-05T07:35:00.193 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:00 smithi090 ceph-mon[33213]: pgmap v857: 169 pgs: 3 unknown, 3 peering, 163 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-02-05T07:35:00.193 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:00 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:35:00.193 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:00 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:35:00.193 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:00 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T07:35:00.193 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:00 smithi090 ceph-mon[33213]: osdmap e475: 8 total, 8 up, 6 in 2024-02-05T07:35:00.193 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:00 smithi090 ceph-mon[33213]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:00.193 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:00 smithi090 ceph-mon[33213]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:00.193 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:00 smithi090 ceph-mon[33213]: 1.b starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:00.193 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:00 smithi090 ceph-mon[33213]: 1.c starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:00.226 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:00.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:00.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:00.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:00.356 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:35:00.372 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:35:00.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:00.378 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:00.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:00.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:00.510 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:00.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:00.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:00.612 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:00.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:00.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:00.781 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 73 16 107735 107719 92.2005 77.1875 0.0079618 0.0108415 2024-02-05T07:35:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:00.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:00.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:00.944 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:00.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:01.009 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:35:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:35:01.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:01.077 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:01.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:01.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:01.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:01 smithi007 ceph-mon[29098]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:01.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:01 smithi007 ceph-mon[29098]: 1.e starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:01.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:01 smithi007 ceph-mon[29098]: osdmap e476: 8 total, 8 up, 6 in 2024-02-05T07:35:01.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:01 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:35:01.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:01 smithi007 ceph-mon[37158]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:01.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:01 smithi007 ceph-mon[37158]: 1.e starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:01.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:01 smithi007 ceph-mon[37158]: osdmap e476: 8 total, 8 up, 6 in 2024-02-05T07:35:01.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:01 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:35:01.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:01.210 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:01.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:01.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:01 smithi090 ceph-mon[33213]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:01.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:01 smithi090 ceph-mon[33213]: 1.e starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:35:01.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:01 smithi090 ceph-mon[33213]: osdmap e476: 8 total, 8 up, 6 in 2024-02-05T07:35:01.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:01 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:35:01.331 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:01.364 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:01.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:01.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:01.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:01.508 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:01.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:01.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:01.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:01.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:01.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:01.753 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:01.781 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 74 16 109111 109095 92.116 86 0.012409 0.0108513 2024-02-05T07:35:01.796 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:01.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:01.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:01.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:01.937 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:01.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:01.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:01.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:35:02.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:02.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:02.072 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:02.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:02.115 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:35:02.173 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-02-05T07:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-02-05T07:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:210982 8.5 8.3d282b45 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e301)", 2024-02-05T07:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:24.576942+0000", 2024-02-05T07:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 397.63456752399998, 2024-02-05T07:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1733628220000001, 2024-02-05T07:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 210982 2024-02-05T07:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.576942+0000", 2024-02-05T07:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.576942+0000", 2024-02-05T07:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.576943+0000", 2024-02-05T07:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7659999999999999e-06 2024-02-05T07:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.576949+0000", 2024-02-05T07:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3510000000000001e-06 2024-02-05T07:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.576958+0000", 2024-02-05T07:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7780000000000002e-06 2024-02-05T07:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.576970+0000", 2024-02-05T07:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253999999999999e-05 2024-02-05T07:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.579239+0000", 2024-02-05T07:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022689939999999999 2024-02-05T07:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.579251+0000", 2024-02-05T07:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2125e-05 2024-02-05T07:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.750291+0000", 2024-02-05T07:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1710403359999999 2024-02-05T07:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.750304+0000", 2024-02-05T07:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3217999999999999e-05 2024-02-05T07:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24356.0:2078 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 93862608739072 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-02-05T07:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:24.587417+0000", 2024-02-05T07:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 397.62409221899998, 2024-02-05T07:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2858643619999999, 2024-02-05T07:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24356", 2024-02-05T07:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2716729371", 2024-02-05T07:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2078 2024-02-05T07:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587417+0000", 2024-02-05T07:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587417+0000", 2024-02-05T07:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587418+0000", 2024-02-05T07:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.083e-06 2024-02-05T07:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587421+0000", 2024-02-05T07:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9119999999999998e-06 2024-02-05T07:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587428+0000", 2024-02-05T07:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8379999999999996e-06 2024-02-05T07:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587435+0000", 2024-02-05T07:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9210000000000003e-06 2024-02-05T07:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587488+0000", 2024-02-05T07:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3013999999999998e-05 2024-02-05T07:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587498+0000", 2024-02-05T07:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9599999999999995e-06 2024-02-05T07:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.844368+0000", 2024-02-05T07:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2568704500000001 2024-02-05T07:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.844526+0000", 2024-02-05T07:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015806899999999999 2024-02-05T07:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.873281+0000", 2024-02-05T07:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028755115000000001 2024-02-05T07:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24356.0:2079 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 93862613159296 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-02-05T07:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:24.587448+0000", 2024-02-05T07:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 397.62406110699999, 2024-02-05T07:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4188860320000001, 2024-02-05T07:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24356", 2024-02-05T07:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2716729371", 2024-02-05T07:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2079 2024-02-05T07:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587448+0000", 2024-02-05T07:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587448+0000", 2024-02-05T07:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587448+0000", 2024-02-05T07:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.91e-07 2024-02-05T07:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587450+0000", 2024-02-05T07:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.691e-06 2024-02-05T07:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587466+0000", 2024-02-05T07:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5739e-05 2024-02-05T07:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587471+0000", 2024-02-05T07:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0479999999999998e-06 2024-02-05T07:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587505+0000", 2024-02-05T07:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4044000000000001e-05 2024-02-05T07:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:24.587512+0000", 2024-02-05T07:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2370000000000003e-06 2024-02-05T07:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.868905+0000", 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2813931219999999 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.869089+0000", 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018377199999999999 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:26.006334+0000", 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13724508799999999 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24356.0:2081 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 93862608734464 gen 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-02-05T07:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:25.742864+0000", 2024-02-05T07:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 396.46864521200001, 2024-02-05T07:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87074127499999998, 2024-02-05T07:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24356", 2024-02-05T07:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2716729371", 2024-02-05T07:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2081 2024-02-05T07:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.742864+0000", 2024-02-05T07:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.742864+0000", 2024-02-05T07:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.742865+0000", 2024-02-05T07:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4190000000000001e-06 2024-02-05T07:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.742876+0000", 2024-02-05T07:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1001e-05 2024-02-05T07:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.742886+0000", 2024-02-05T07:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8519999999999999e-06 2024-02-05T07:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.742900+0000", 2024-02-05T07:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3908e-05 2024-02-05T07:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.748077+0000", 2024-02-05T07:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051767239999999997 2024-02-05T07:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:25.748087+0000", 2024-02-05T07:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0609e-05 2024-02-05T07:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:26.597167+0000", 2024-02-05T07:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.84907927599999999 2024-02-05T07:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:26.597325+0000", 2024-02-05T07:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015865599999999999 2024-02-05T07:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:26.613605+0000", 2024-02-05T07:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016279829999999999 2024-02-05T07:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:218870 8.5 8:a7634133:::benchmark_data_smithi007_105755_object82560:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-02-05T07:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:37.819605+0000", 2024-02-05T07:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 384.39190394100001, 2024-02-05T07:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.88666498199999999, 2024-02-05T07:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218870 2024-02-05T07:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819605+0000", 2024-02-05T07:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819605+0000", 2024-02-05T07:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819607+0000", 2024-02-05T07:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5069999999999999e-06 2024-02-05T07:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819612+0000", 2024-02-05T07:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0130000000000001e-06 2024-02-05T07:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819625+0000", 2024-02-05T07:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3754000000000001e-05 2024-02-05T07:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819633+0000", 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9869999999999999e-06 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819696+0000", 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2669999999999995e-05 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.819706+0000", 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7049999999999994e-06 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.700281+0000", 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.88057515200000003 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.700417+0000", 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013600400000000001 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.700693+0000", 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027553400000000002 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.702831+0000", 2024-02-05T07:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021382390000000001 2024-02-05T07:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.706057+0000", 2024-02-05T07:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032261880000000001 2024-02-05T07:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.706167+0000", 2024-02-05T07:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110256 2024-02-05T07:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.706232+0000", 2024-02-05T07:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5079000000000005e-05 2024-02-05T07:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.706270+0000", 2024-02-05T07:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7894e-05 2024-02-05T07:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:218941 8.5 8:a24ccea1:::benchmark_data_smithi007_105755_object82631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-02-05T07:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:37.877696+0000", 2024-02-05T07:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 384.33381344899999, 2024-02-05T07:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.83040171799999996, 2024-02-05T07:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218941 2024-02-05T07:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877696+0000", 2024-02-05T07:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877696+0000", 2024-02-05T07:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877697+0000", 2024-02-05T07:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6750000000000001e-06 2024-02-05T07:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877703+0000", 2024-02-05T07:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7450000000000001e-06 2024-02-05T07:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877714+0000", 2024-02-05T07:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1246e-05 2024-02-05T07:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877722+0000", 2024-02-05T07:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5780000000000001e-06 2024-02-05T07:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877950+0000", 2024-02-05T07:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022855 2024-02-05T07:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.877962+0000", 2024-02-05T07:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1847e-05 2024-02-05T07:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.702860+0000", 2024-02-05T07:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82489743699999996 2024-02-05T07:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.702981+0000", 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120763 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.703200+0000", 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021935999999999999 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.707893+0000", 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046932520000000002 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.707931+0000", 2024-02-05T07:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.756e-05 2024-02-05T07:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.707973+0000", 2024-02-05T07:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1836999999999999e-05 2024-02-05T07:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708059+0000", 2024-02-05T07:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6238999999999999e-05 2024-02-05T07:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708097+0000", 2024-02-05T07:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8628999999999999e-05 2024-02-05T07:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:218996 8.5 8:a17532da:::benchmark_data_smithi007_105755_object82686:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-02-05T07:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:37.922624+0000", 2024-02-05T07:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 384.28888463599998, 2024-02-05T07:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.78597759899999997, 2024-02-05T07:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218996 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.922624+0000", 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.922624+0000", 2024-02-05T07:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.922626+0000", 2024-02-05T07:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9130000000000001e-06 2024-02-05T07:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.922631+0000", 2024-02-05T07:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6450000000000004e-06 2024-02-05T07:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.922645+0000", 2024-02-05T07:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3974e-05 2024-02-05T07:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.922653+0000", 2024-02-05T07:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9330000000000001e-06 2024-02-05T07:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.923228+0000", 2024-02-05T07:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057506799999999998 2024-02-05T07:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.923236+0000", 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3049999999999993e-06 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.704144+0000", 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.78090789400000005 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.704253+0000", 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108874 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-02-05T07:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.704445+0000", 2024-02-05T07:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192177 2024-02-05T07:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.707941+0000", 2024-02-05T07:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034960429999999999 2024-02-05T07:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708161+0000", 2024-02-05T07:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000220168 2024-02-05T07:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708210+0000", 2024-02-05T07:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8848999999999999e-05 2024-02-05T07:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708555+0000", 2024-02-05T07:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034451700000000002 2024-02-05T07:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708602+0000", 2024-02-05T07:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7239e-05 2024-02-05T07:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:219029 8.5 8:a492476d:::benchmark_data_smithi007_105755_object82719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-02-05T07:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:37.952103+0000", 2024-02-05T07:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 384.259405738, 2024-02-05T07:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.75848268500000005, 2024-02-05T07:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:35:02.261 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 219029 2024-02-05T07:35:02.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.261 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952103+0000", 2024-02-05T07:35:02.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952103+0000", 2024-02-05T07:35:02.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952105+0000", 2024-02-05T07:35:02.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3969999999999999e-06 2024-02-05T07:35:02.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952109+0000", 2024-02-05T07:35:02.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.464e-06 2024-02-05T07:35:02.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952122+0000", 2024-02-05T07:35:02.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3064999999999999e-05 2024-02-05T07:35:02.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952131+0000", 2024-02-05T07:35:02.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5879999999999998e-06 2024-02-05T07:35:02.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952181+0000", 2024-02-05T07:35:02.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9895000000000003e-05 2024-02-05T07:35:02.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:35:02.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:37.952190+0000", 2024-02-05T07:35:02.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8799999999999997e-06 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.706326+0000", 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.75413592699999998 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.706428+0000", 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102396 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.706617+0000", 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018896900000000001 2024-02-05T07:35:02.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.710420+0000", 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038025670000000002 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.710430+0000", 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.024e-05 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.710471+0000", 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1331000000000001e-05 2024-02-05T07:35:02.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.710528+0000", 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6611999999999999e-05 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.710586+0000", 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8353999999999997e-05 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.266 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:219073 8.5 8:a5d9854b:::benchmark_data_smithi007_105755_object82763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:28:38.040105+0000", 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 384.17140374500002, 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.672581702, 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 219073 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040105+0000", 2024-02-05T07:35:02.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040105+0000", 2024-02-05T07:35:02.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040107+0000", 2024-02-05T07:35:02.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.153e-06 2024-02-05T07:35:02.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040111+0000", 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8899999999999998e-06 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040123+0000", 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1719000000000001e-05 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040130+0000", 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2024-02-05T07:35:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040200+0000", 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0444999999999994e-05 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.040209+0000", 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6780000000000006e-06 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708697+0000", 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66848853500000005 2024-02-05T07:35:02.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.708831+0000", 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133588 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.709080+0000", 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024862199999999999 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.710602+0000", 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015226879999999999 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.711434+0000", 2024-02-05T07:35:02.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083224299999999998 2024-02-05T07:35:02.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.712628+0000", 2024-02-05T07:35:02.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011935959999999999 2024-02-05T07:35:02.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:35:02.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.712667+0000", 2024-02-05T07:35:02.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8506e-05 2024-02-05T07:35:02.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:28:38.712687+0000", 2024-02-05T07:35:02.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0537999999999999e-05 2024-02-05T07:35:02.273 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.273 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.273 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:02.275 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:50655 9.b 9:df37fbe4:::benchmark_data_smithi007_136200_object50654:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-02-05T07:35:02.275 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:30:56.230078+0000", 2024-02-05T07:35:02.275 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 245.98143117000001, 2024-02-05T07:35:02.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36011623100000001, 2024-02-05T07:35:02.275 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.275 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:02.276 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.276 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:35:02.276 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:35:02.276 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 50655 2024-02-05T07:35:02.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.276 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:56.230078+0000", 2024-02-05T07:35:02.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:56.230078+0000", 2024-02-05T07:35:02.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:56.230079+0000", 2024-02-05T07:35:02.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9199999999999999e-07 2024-02-05T07:35:02.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:56.230137+0000", 2024-02-05T07:35:02.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8183000000000002e-05 2024-02-05T07:35:02.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:56.230147+0000", 2024-02-05T07:35:02.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0335e-05 2024-02-05T07:35:02.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:56.230153+0000", 2024-02-05T07:35:02.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5509999999999999e-06 2024-02-05T07:35:02.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:56.230201+0000", 2024-02-05T07:35:02.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7856999999999999e-05 2024-02-05T07:35:02.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:56.230335+0000", 2024-02-05T07:35:02.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013384000000000001 2024-02-05T07:35:02.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2024-02-05T07:35:02.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:56.230489+0000", 2024-02-05T07:35:02.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000153924 2024-02-05T07:35:02.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:35:02.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:56.231896+0000", 2024-02-05T07:35:02.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001407118 2024-02-05T07:35:02.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:56.233484+0000", 2024-02-05T07:35:02.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015883990000000001 2024-02-05T07:35:02.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:56.590094+0000", 2024-02-05T07:35:02.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35661011300000001 2024-02-05T07:35:02.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:35:02.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:56.590151+0000", 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6938000000000003e-05 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:30:56.590194+0000", 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3081000000000003e-05 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:34802 10.f 10:f0517665:::benchmark_data_smithi007_157098_object34801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:03.162187+0000", 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.049321955000003, 2024-02-05T07:35:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.212103137, 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 34802 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162187+0000", 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162187+0000", 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162188+0000", 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2699999999999998e-07 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162220+0000", 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2243000000000003e-05 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162229+0000", 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7679999999999997e-06 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162234+0000", 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6290000000000001e-06 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162291+0000", 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.787e-05 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162415+0000", 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000123752 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162555+0000", 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139367 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.165023+0000", 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024686959999999998 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.168590+0000", 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035666000000000001 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.374178+0000", 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205588454 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.374234+0000", 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5562000000000001e-05 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.374290+0000", 2024-02-05T07:35:02.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6369000000000002e-05 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:34803 10.7 10:e6d3c41b:::benchmark_data_smithi007_157098_object34802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:03.162245+0000", 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.049264590999996, 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.212018399, 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 34803 2024-02-05T07:35:02.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.288 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162245+0000", 2024-02-05T07:35:02.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162245+0000", 2024-02-05T07:35:02.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162245+0000", 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9399999999999995e-07 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162288+0000", 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2502000000000001e-05 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162293+0000", 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0710000000000002e-06 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162297+0000", 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1420000000000003e-06 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162340+0000", 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3588999999999999e-05 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162476+0000", 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013537299999999999 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2024-02-05T07:35:02.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.162571+0000", 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5562000000000004e-05 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.163886+0000", 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013148820000000001 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.168621+0000", 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047343740000000004 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.374142+0000", 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205521905 2024-02-05T07:35:02.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:35:02.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.374213+0000", 2024-02-05T07:35:02.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0815999999999994e-05 2024-02-05T07:35:02.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.374263+0000", 2024-02-05T07:35:02.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9688999999999998e-05 2024-02-05T07:35:02.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.292 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:34807 10.f 10:f8313a23:::benchmark_data_smithi007_157098_object34806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:03.163170+0000", 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.04833928, 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21584429999999999, 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 34807 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.163170+0000", 2024-02-05T07:35:02.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.163170+0000", 2024-02-05T07:35:02.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.163171+0000", 2024-02-05T07:35:02.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.85e-07 2024-02-05T07:35:02.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.163242+0000", 2024-02-05T07:35:02.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1507000000000005e-05 2024-02-05T07:35:02.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.163272+0000", 2024-02-05T07:35:02.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9478000000000001e-05 2024-02-05T07:35:02.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.163280+0000", 2024-02-05T07:35:02.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7390000000000005e-06 2024-02-05T07:35:02.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.163892+0000", 2024-02-05T07:35:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061190599999999995 2024-02-05T07:35:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.164028+0000", 2024-02-05T07:35:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136138 2024-02-05T07:35:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2024-02-05T07:35:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.164172+0000", 2024-02-05T07:35:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014391499999999999 2024-02-05T07:35:02.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:35:02.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.165130+0000", 2024-02-05T07:35:02.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000957533 2024-02-05T07:35:02.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.168740+0000", 2024-02-05T07:35:02.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036102489999999998 2024-02-05T07:35:02.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.378904+0000", 2024-02-05T07:35:02.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21016346899999999 2024-02-05T07:35:02.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:35:02.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.378970+0000", 2024-02-05T07:35:02.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6414000000000005e-05 2024-02-05T07:35:02.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:03.379014+0000", 2024-02-05T07:35:02.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4066999999999997e-05 2024-02-05T07:35:02.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.298 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:88977 10.7 10:e4650fef:::benchmark_data_smithi007_157098_object88976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e459)", 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:44.384533+0000", 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.826976107, 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22858235299999999, 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 88977 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.384533+0000", 2024-02-05T07:35:02.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.384533+0000", 2024-02-05T07:35:02.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.384534+0000", 2024-02-05T07:35:02.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9800000000000002e-07 2024-02-05T07:35:02.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.384576+0000", 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1748000000000001e-05 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.384586+0000", 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7699999999999996e-06 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.384593+0000", 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2819999999999999e-06 2024-02-05T07:35:02.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.384644+0000", 2024-02-05T07:35:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0955999999999998e-05 2024-02-05T07:35:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.384742+0000", 2024-02-05T07:35:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8511000000000006e-05 2024-02-05T07:35:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2024-02-05T07:35:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.384848+0000", 2024-02-05T07:35:02.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000105826 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.386498+0000", 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016498190000000001 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.390500+0000", 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040016299999999999 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.613051+0000", 2024-02-05T07:35:02.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22255146000000001 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.613088+0000", 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6720000000000001e-05 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.613115+0000", 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7633000000000001e-05 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:88979 10.0 10:08030c32:::benchmark_data_smithi007_157098_object88978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e459)", 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:44.390822+0000", 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.820686860999999, 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23050763699999999, 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 88979 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.390822+0000", 2024-02-05T07:35:02.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.390822+0000", 2024-02-05T07:35:02.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.390823+0000", 2024-02-05T07:35:02.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1100000000000004e-07 2024-02-05T07:35:02.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.390846+0000", 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3323e-05 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.390855+0000", 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8310000000000007e-06 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.390861+0000", 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5589999999999996e-06 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.390906+0000", 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5547000000000003e-05 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.391001+0000", 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4816999999999997e-05 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.391090+0000", 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8349000000000004e-05 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.614443+0000", 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.223353311 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.614466+0000", 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2685999999999999e-05 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.621215+0000", 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0067495350000000001 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.621288+0000", 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2947999999999993e-05 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.621330+0000", 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1820000000000003e-05 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:88981 10.0 10:013b601e:::benchmark_data_smithi007_157098_object88980:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e459)", 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:44.391363+0000", 2024-02-05T07:35:02.310 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.820146254000001, 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23021472600000001, 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 88981 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.391363+0000", 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.391363+0000", 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.391364+0000", 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1900000000000001e-07 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.391405+0000", 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1e-05 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.391414+0000", 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8349999999999993e-06 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.391420+0000", 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8900000000000001e-06 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392684+0000", 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001263712 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392766+0000", 2024-02-05T07:35:02.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1743999999999995e-05 2024-02-05T07:35:02.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:35:02.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.612691+0000", 2024-02-05T07:35:02.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21992520300000001 2024-02-05T07:35:02.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:35:02.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.614594+0000", 2024-02-05T07:35:02.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001902749 2024-02-05T07:35:02.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.615298+0000", 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070436000000000001 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.621484+0000", 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061861160000000002 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.621539+0000", 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4398000000000003e-05 2024-02-05T07:35:02.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.621578+0000", 2024-02-05T07:35:02.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8899999999999997e-05 2024-02-05T07:35:02.316 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.316 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.316 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.316 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:88982 10.7 10:e21261d4:::benchmark_data_smithi007_157098_object88981:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e459)", 2024-02-05T07:35:02.316 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:44.392007+0000", 2024-02-05T07:35:02.316 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.819502228000001, 2024-02-05T07:35:02.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22121817499999999, 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 88982 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392007+0000", 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392007+0000", 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392008+0000", 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5199999999999996e-07 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392083+0000", 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5442e-05 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392091+0000", 2024-02-05T07:35:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2230000000000007e-06 2024-02-05T07:35:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392098+0000", 2024-02-05T07:35:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4099999999999996e-06 2024-02-05T07:35:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392168+0000", 2024-02-05T07:35:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0393000000000002e-05 2024-02-05T07:35:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392257+0000", 2024-02-05T07:35:02.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9334999999999998e-05 2024-02-05T07:35:02.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2024-02-05T07:35:02.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392362+0000", 2024-02-05T07:35:02.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104701 2024-02-05T07:35:02.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.613153+0000", 2024-02-05T07:35:02.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22079057599999999 2024-02-05T07:35:02.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.613176+0000", 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3180000000000002e-05 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.613183+0000", 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.708e-06 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.613207+0000", 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4627e-05 2024-02-05T07:35:02.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.613225+0000", 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7927999999999998e-05 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.25454.0:88977 10.7 e459/423 ondisk, result = 0)", 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:44.392307+0000", 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.819202418, 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22081515700000001, 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.322 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392307+0000", 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392307+0000", 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392307+0000", 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2499999999999995e-07 2024-02-05T07:35:02.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392310+0000", 2024-02-05T07:35:02.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8439999999999998e-06 2024-02-05T07:35:02.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392319+0000", 2024-02-05T07:35:02.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4970000000000001e-06 2024-02-05T07:35:02.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.392324+0000", 2024-02-05T07:35:02.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7080000000000002e-06 2024-02-05T07:35:02.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.613040+0000", 2024-02-05T07:35:02.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22071611399999999 2024-02-05T07:35:02.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.613047+0000", 2024-02-05T07:35:02.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5740000000000001e-06 2024-02-05T07:35:02.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.613122+0000", 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4894999999999994e-05 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.25454.0:88979 10.0 e459/423 ondisk, result = 0)", 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:44.393739+0000", 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.817770119999999, 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22070993999999999, 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.393739+0000", 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.393739+0000", 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.393740+0000", 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.06e-07 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.393742+0000", 2024-02-05T07:35:02.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9050000000000001e-06 2024-02-05T07:35:02.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.612649+0000", 2024-02-05T07:35:02.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218906341 2024-02-05T07:35:02.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.612665+0000", 2024-02-05T07:35:02.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6487999999999999e-05 2024-02-05T07:35:02.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.614434+0000", 2024-02-05T07:35:02.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017688199999999999 2024-02-05T07:35:02.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.614439+0000", 2024-02-05T07:35:02.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0499999999999999e-06 2024-02-05T07:35:02.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.614449+0000", 2024-02-05T07:35:02.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8300000000000008e-06 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.25454.0:88982 10.7 e459/423 ondisk, result = 0)", 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:34:44.394495+0000", 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.817013753000001, 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218731334, 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.394495+0000", 2024-02-05T07:35:02.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.394495+0000", 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.394496+0000", 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7100000000000002e-07 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.394499+0000", 2024-02-05T07:35:02.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.841e-06 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.612922+0000", 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21842315800000001 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.612933+0000", 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1121000000000001e-05 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.613170+0000", 2024-02-05T07:35:02.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000236963 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.613173+0000", 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2119999999999999e-06 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:34:44.613227+0000", 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3467999999999997e-05 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:35:02.333 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:35:02.334 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:35:02.334 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:02.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:02.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:02.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:02.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:02.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:02.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:02.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:02.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:02.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:02.744 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:02.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:02.781 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 75 15 110330 110315 91.9043 76.25 0.00602663 0.0108763 2024-02-05T07:35:02.845 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:02 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:02.845 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:02 smithi090 ceph-mon[33213]: pgmap v860: 169 pgs: 2 active+undersized+remapped, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 unknown, 5 peering, 159 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-02-05T07:35:02.845 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:02 smithi090 ceph-mon[33213]: osdmap e477: 8 total, 8 up, 6 in 2024-02-05T07:35:02.845 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:02 smithi090 ceph-mon[33213]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:02.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:02.889 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:02.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:02.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:02.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:02 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:02.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:02 smithi007 ceph-mon[29098]: pgmap v860: 169 pgs: 2 active+undersized+remapped, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 unknown, 5 peering, 159 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-02-05T07:35:02.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:02 smithi007 ceph-mon[29098]: osdmap e477: 8 total, 8 up, 6 in 2024-02-05T07:35:02.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:02 smithi007 ceph-mon[29098]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:02.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:02 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:02.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:02 smithi007 ceph-mon[37158]: pgmap v860: 169 pgs: 2 active+undersized+remapped, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 unknown, 5 peering, 159 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-02-05T07:35:02.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:02 smithi007 ceph-mon[37158]: osdmap e477: 8 total, 8 up, 6 in 2024-02-05T07:35:02.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:02 smithi007 ceph-mon[37158]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:02.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:03.021 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:03.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:03.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:03.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:03.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:03.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:03.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T07:35:03.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:03.301 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:03.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:03.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:03.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:03.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:03.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:03.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:03.563 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:03.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:03.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:03.665 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:03.715 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:03.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:03.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:03.782 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 76 16 111517 111501 91.6704 74.125 0.0148765 0.0109038 2024-02-05T07:35:03.817 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:03.858 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:03.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:03.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:03.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:35:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:35:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:35:03.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:03 smithi007 ceph-mon[29098]: osdmap e478: 8 total, 8 up, 6 in 2024-02-05T07:35:03.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:03 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-05T07:35:03.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:03 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-05T07:35:03.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:03 smithi007 ceph-mon[37158]: osdmap e478: 8 total, 8 up, 6 in 2024-02-05T07:35:03.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:03 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-05T07:35:03.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:03 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-05T07:35:03.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:03.974 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:35:03.989 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:35:03.999 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:35:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:04.018 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:04.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:04.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:03 smithi090 ceph-mon[33213]: osdmap e478: 8 total, 8 up, 6 in 2024-02-05T07:35:04.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:03 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-05T07:35:04.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:03 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-05T07:35:04.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:04.150 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:04.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:04.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:04.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:04.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:04.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:04.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:04.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:04.528 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:04.577 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:04.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:04.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:04.702 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:04.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:04.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:04.782 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 77 16 112694 112678 91.4352 73.5625 0.0162476 0.0109319 2024-02-05T07:35:04.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:04.858 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:04.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:04.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:04 smithi007 ceph-mon[29098]: pgmap v863: 168 pgs: 1 active+undersized+remapped, 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 unknown, 4 peering, 159 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 978 op/s; 1/319524 objects misplaced (0.000%) 2024-02-05T07:35:04.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:04 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-05T07:35:04.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:04 smithi007 ceph-mon[29098]: osdmap e479: 8 total, 8 up, 6 in 2024-02-05T07:35:04.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:04 smithi007 ceph-mon[29098]: osdmap e480: 8 total, 8 up, 6 in 2024-02-05T07:35:04.886 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:04 smithi007 ceph-mon[37158]: pgmap v863: 168 pgs: 1 active+undersized+remapped, 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 unknown, 4 peering, 159 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 978 op/s; 1/319524 objects misplaced (0.000%) 2024-02-05T07:35:04.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:04 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-05T07:35:04.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:04 smithi007 ceph-mon[37158]: osdmap e479: 8 total, 8 up, 6 in 2024-02-05T07:35:04.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:04 smithi007 ceph-mon[37158]: osdmap e480: 8 total, 8 up, 6 in 2024-02-05T07:35:04.958 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:05.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:05.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:05.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:05.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:04 smithi090 ceph-mon[33213]: pgmap v863: 168 pgs: 1 active+undersized+remapped, 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 unknown, 4 peering, 159 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 978 op/s; 1/319524 objects misplaced (0.000%) 2024-02-05T07:35:05.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:04 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-05T07:35:05.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:04 smithi090 ceph-mon[33213]: osdmap e479: 8 total, 8 up, 6 in 2024-02-05T07:35:05.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:04 smithi090 ceph-mon[33213]: osdmap e480: 8 total, 8 up, 6 in 2024-02-05T07:35:05.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:05.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:05.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:05.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:05.278 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:05.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:05.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:05.409 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:05.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:05.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:05.543 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:05.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:05.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:05.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:05.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:05 smithi007 ceph-mon[29098]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:35:05.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:05 smithi007 ceph-mon[29098]: osdmap e481: 8 total, 8 up, 6 in 2024-02-05T07:35:05.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:05 smithi007 ceph-mon[37158]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:35:05.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:05 smithi007 ceph-mon[37158]: osdmap e481: 8 total, 8 up, 6 in 2024-02-05T07:35:05.686 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:05.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:05.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:05.782 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 78 16 114053 114037 91.3518 84.9375 0.0078018 0.0109424 2024-02-05T07:35:05.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:05.828 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:05.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:05.870 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":864,"stamp":"2024-02-05T07:35:03.591726+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6967728476,"num_objects":106508,"num_object_clones":0,"num_object_copies":319524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":106508,"num_whiteouts":0,"num_read":1371,"num_read_kb":1123,"num_write":107032,"num_write_kb":6803885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":270260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59128,"ondisk_log_size":59128,"up":504,"acting":503,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":511,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":24573692,"kb_used_data":20524460,"kb_used_omap":13,"kb_used_meta":4049138,"kb_avail":537897220,"statfs":{"total":575970213888,"available":550806753280,"internally_reserved":0,"allocated":21017047040,"data_stored":21014356782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":4146318048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":256245760,"num_objects":3910,"num_object_clones":0,"num_object_copies":11730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3910,"num_write_kb":250240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030180"},"pg_stats":[{"pgid":"10.2","version":"475'6691","reported_seq":6805,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.937936+0000","last_change":"2024-02-05T07:33:38.775613+0000","last_active":"2024-02-05T07:34:59.937936+0000","last_peered":"2024-02-05T07:34:59.937936+0000","last_clean":"2024-02-05T07:34:59.937936+0000","last_became_active":"2024-02-05T07:33:38.775201+0000","last_became_peered":"2024-02-05T07:33:38.775201+0000","last_unstale":"2024-02-05T07:34:59.937936+0000","last_undegraded":"2024-02-05T07:34:59.937936+0000","last_fullsized":"2024-02-05T07:34:59.937936+0000","mapping_epoch":423,"log_start":"431'3200","ondisk_log_start":"431'3200","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3491,"log_dups_size":0,"ondisk_log_size":3491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438501376,"num_objects":6691,"num_object_clones":0,"num_object_copies":20073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6691,"num_write_kb":428224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":996,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.574110+0000","last_change":"2024-02-05T07:34:46.868819+0000","last_active":"2024-02-05T07:35:01.574110+0000","last_peered":"2024-02-05T07:35:01.574110+0000","last_clean":"2024-02-05T07:35:01.574110+0000","last_became_active":"2024-02-05T07:34:46.868295+0000","last_became_peered":"2024-02-05T07:34:46.868295+0000","last_unstale":"2024-02-05T07:35:01.574110+0000","last_undegraded":"2024-02-05T07:35:01.574110+0000","last_fullsized":"2024-02-05T07:35:01.574110+0000","mapping_epoch":367,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:42.817975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:42.817975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"326'25","reported_seq":937,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.567015+0000","last_change":"2024-02-05T07:32:27.643863+0000","last_active":"2024-02-05T07:34:59.567015+0000","last_peered":"2024-02-05T07:34:59.567015+0000","last_clean":"2024-02-05T07:34:59.567015+0000","last_became_active":"2024-02-05T07:28:26.605082+0000","last_became_peered":"2024-02-05T07:28:26.605082+0000","last_unstale":"2024-02-05T07:34:59.567015+0000","last_undegraded":"2024-02-05T07:34:59.567015+0000","last_fullsized":"2024-02-05T07:34:59.567015+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:32:27.643788+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:32:27.643788+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951419+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":448,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.574028+0000","last_change":"2024-02-05T07:33:34.736291+0000","last_active":"2024-02-05T07:35:01.574028+0000","last_peered":"2024-02-05T07:35:01.574028+0000","last_clean":"2024-02-05T07:35:01.574028+0000","last_became_active":"2024-02-05T07:27:16.716461+0000","last_became_peered":"2024-02-05T07:27:16.716461+0000","last_unstale":"2024-02-05T07:35:01.574028+0000","last_undegraded":"2024-02-05T07:35:01.574028+0000","last_fullsized":"2024-02-05T07:35:01.574028+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:34.736160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:34.736160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"365'11","reported_seq":1098,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.573017+0000","last_change":"2024-02-05T07:34:29.851597+0000","last_active":"2024-02-05T07:34:59.573017+0000","last_peered":"2024-02-05T07:34:59.573017+0000","last_clean":"2024-02-05T07:34:59.573017+0000","last_became_active":"2024-02-05T07:30:40.484389+0000","last_became_peered":"2024-02-05T07:30:40.484389+0000","last_unstale":"2024-02-05T07:34:59.573017+0000","last_undegraded":"2024-02-05T07:34:59.573017+0000","last_fullsized":"2024-02-05T07:34:59.573017+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:34:29.851477+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:34:29.851477+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:58.303044+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":850,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.561960+0000","last_change":"2024-02-05T07:32:30.708289+0000","last_active":"2024-02-05T07:34:59.561960+0000","last_peered":"2024-02-05T07:34:59.561960+0000","last_clean":"2024-02-05T07:34:59.561960+0000","last_became_active":"2024-02-05T07:18:55.466464+0000","last_became_peered":"2024-02-05T07:18:55.466464+0000","last_unstale":"2024-02-05T07:34:59.561960+0000","last_undegraded":"2024-02-05T07:34:59.561960+0000","last_fullsized":"2024-02-05T07:34:59.561960+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:30.708205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:32:30.708205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951436+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"475'6609","reported_seq":6722,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:35:00.019810+0000","last_change":"2024-02-05T07:33:38.616173+0000","last_active":"2024-02-05T07:35:00.019810+0000","last_peered":"2024-02-05T07:35:00.019810+0000","last_clean":"2024-02-05T07:35:00.019810+0000","last_became_active":"2024-02-05T07:33:38.615825+0000","last_became_peered":"2024-02-05T07:33:38.615825+0000","last_unstale":"2024-02-05T07:35:00.019810+0000","last_undegraded":"2024-02-05T07:35:00.019810+0000","last_fullsized":"2024-02-05T07:35:00.019810+0000","mapping_epoch":423,"log_start":"431'2864","ondisk_log_start":"431'2864","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3745,"log_dups_size":0,"ondisk_log_size":3745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433127424,"num_objects":6609,"num_object_clones":0,"num_object_copies":19827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6609,"num_write_kb":422976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":910,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568214+0000","last_change":"2024-02-05T07:34:50.890143+0000","last_active":"2024-02-05T07:34:59.568214+0000","last_peered":"2024-02-05T07:34:59.568214+0000","last_clean":"2024-02-05T07:34:59.568214+0000","last_became_active":"2024-02-05T07:34:50.889535+0000","last_became_peered":"2024-02-05T07:34:50.889535+0000","last_unstale":"2024-02-05T07:34:59.568214+0000","last_undegraded":"2024-02-05T07:34:59.568214+0000","last_fullsized":"2024-02-05T07:34:59.568214+0000","mapping_epoch":302,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:41.228420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:41.228420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":69,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"326'31","reported_seq":916,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.566125+0000","last_change":"2024-02-05T07:33:30.750667+0000","last_active":"2024-02-05T07:34:59.566125+0000","last_peered":"2024-02-05T07:34:59.566125+0000","last_clean":"2024-02-05T07:34:59.566125+0000","last_became_active":"2024-02-05T07:18:51.447880+0000","last_became_peered":"2024-02-05T07:18:51.447880+0000","last_unstale":"2024-02-05T07:34:59.566125+0000","last_undegraded":"2024-02-05T07:34:59.566125+0000","last_fullsized":"2024-02-05T07:34:59.566125+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'31","last_scrub_stamp":"2024-02-05T07:33:30.750548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.750548+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951590+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":712,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.974887+0000","last_change":"2024-02-05T07:33:44.229582+0000","last_active":"2024-02-05T07:34:57.974887+0000","last_peered":"2024-02-05T07:34:57.974887+0000","last_clean":"2024-02-05T07:34:57.974887+0000","last_became_active":"2024-02-05T07:28:25.791005+0000","last_became_peered":"2024-02-05T07:28:25.791005+0000","last_unstale":"2024-02-05T07:34:57.974887+0000","last_undegraded":"2024-02-05T07:34:57.974887+0000","last_fullsized":"2024-02-05T07:34:57.974887+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.229489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.229489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.986179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":68,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.566929+0000","last_change":"2024-02-05T07:34:30.678802+0000","last_active":"2024-02-05T07:34:59.566929+0000","last_peered":"2024-02-05T07:34:59.566929+0000","last_clean":"2024-02-05T07:34:59.566929+0000","last_became_active":"2024-02-05T07:34:30.678610+0000","last_became_peered":"2024-02-05T07:34:30.678610+0000","last_unstale":"2024-02-05T07:34:59.566929+0000","last_undegraded":"2024-02-05T07:34:59.566929+0000","last_fullsized":"2024-02-05T07:34:59.566929+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.674493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.674493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":874,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.573480+0000","last_change":"2024-02-05T07:34:23.161750+0000","last_active":"2024-02-05T07:34:59.573480+0000","last_peered":"2024-02-05T07:34:59.573480+0000","last_clean":"2024-02-05T07:34:59.573480+0000","last_became_active":"2024-02-05T07:27:16.708888+0000","last_became_peered":"2024-02-05T07:27:16.708888+0000","last_unstale":"2024-02-05T07:34:59.573480+0000","last_undegraded":"2024-02-05T07:34:59.573480+0000","last_fullsized":"2024-02-05T07:34:59.573480+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:23.161521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:23.161521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:46.775844+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"473'6528","reported_seq":6638,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:58.691791+0000","last_change":"2024-02-05T07:33:38.613283+0000","last_active":"2024-02-05T07:34:58.691791+0000","last_peered":"2024-02-05T07:34:58.691791+0000","last_clean":"2024-02-05T07:34:58.691791+0000","last_became_active":"2024-02-05T07:33:38.613051+0000","last_became_peered":"2024-02-05T07:33:38.613051+0000","last_unstale":"2024-02-05T07:34:58.691791+0000","last_undegraded":"2024-02-05T07:34:58.691791+0000","last_fullsized":"2024-02-05T07:34:58.691791+0000","mapping_epoch":423,"log_start":"430'2860","ondisk_log_start":"430'2860","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3668,"log_dups_size":0,"ondisk_log_size":3668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:05.930275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427819008,"num_objects":6528,"num_object_clones":0,"num_object_copies":19584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6528,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":1,"reported_epoch":473,"state":"unknown","last_fresh":"2024-02-05T07:34:57.974952+0000","last_change":"2024-02-05T07:34:58.731001+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T07:34:57.974952+0000","last_undegraded":"2024-02-05T07:34:57.974952+0000","last_fullsized":"2024-02-05T07:34:57.974952+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:31.816271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:31.816271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"326'24","reported_seq":903,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.561659+0000","last_change":"2024-02-05T07:29:50.053697+0000","last_active":"2024-02-05T07:34:59.561659+0000","last_peered":"2024-02-05T07:34:59.561659+0000","last_clean":"2024-02-05T07:34:59.561659+0000","last_became_active":"2024-02-05T07:28:25.781007+0000","last_became_peered":"2024-02-05T07:28:25.781007+0000","last_unstale":"2024-02-05T07:34:59.561659+0000","last_undegraded":"2024-02-05T07:34:59.561659+0000","last_fullsized":"2024-02-05T07:34:59.561659+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:29:50.053583+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.053583+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951042+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":869,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.560133+0000","last_change":"2024-02-05T07:32:28.670360+0000","last_active":"2024-02-05T07:34:59.560133+0000","last_peered":"2024-02-05T07:34:59.560133+0000","last_clean":"2024-02-05T07:34:59.560133+0000","last_became_active":"2024-02-05T07:30:40.486134+0000","last_became_peered":"2024-02-05T07:30:40.486134+0000","last_unstale":"2024-02-05T07:34:59.560133+0000","last_undegraded":"2024-02-05T07:34:59.560133+0000","last_fullsized":"2024-02-05T07:34:59.560133+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:28.670286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:32:28.670286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951376+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":690,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.561371+0000","last_change":"2024-02-05T07:29:45.012958+0000","last_active":"2024-02-05T07:34:59.561371+0000","last_peered":"2024-02-05T07:34:59.561371+0000","last_clean":"2024-02-05T07:34:59.561371+0000","last_became_active":"2024-02-05T07:28:26.601713+0000","last_became_peered":"2024-02-05T07:28:26.601713+0000","last_unstale":"2024-02-05T07:34:59.561371+0000","last_undegraded":"2024-02-05T07:34:59.561371+0000","last_fullsized":"2024-02-05T07:34:59.561371+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.012821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951060+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":457,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.575460+0000","last_change":"2024-02-05T07:33:28.696549+0000","last_active":"2024-02-05T07:35:01.575460+0000","last_peered":"2024-02-05T07:35:01.575460+0000","last_clean":"2024-02-05T07:35:01.575460+0000","last_became_active":"2024-02-05T07:30:40.498529+0000","last_became_peered":"2024-02-05T07:30:40.498529+0000","last_unstale":"2024-02-05T07:35:01.575460+0000","last_undegraded":"2024-02-05T07:35:01.575460+0000","last_fullsized":"2024-02-05T07:35:01.575460+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.696462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.696462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.065451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"475'6684","reported_seq":6798,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:35:00.382357+0000","last_change":"2024-02-05T07:33:38.778383+0000","last_active":"2024-02-05T07:35:00.382357+0000","last_peered":"2024-02-05T07:35:00.382357+0000","last_clean":"2024-02-05T07:35:00.382357+0000","last_became_active":"2024-02-05T07:33:38.778162+0000","last_became_peered":"2024-02-05T07:33:38.778162+0000","last_unstale":"2024-02-05T07:35:00.382357+0000","last_undegraded":"2024-02-05T07:35:00.382357+0000","last_fullsized":"2024-02-05T07:35:00.382357+0000","mapping_epoch":423,"log_start":"434'3430","ondisk_log_start":"434'3430","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3254,"log_dups_size":0,"ondisk_log_size":3254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.483201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438042624,"num_objects":6684,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6684,"num_write_kb":427776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":885,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568631+0000","last_change":"2024-02-05T07:34:42.772751+0000","last_active":"2024-02-05T07:34:59.568631+0000","last_peered":"2024-02-05T07:34:59.568631+0000","last_clean":"2024-02-05T07:34:59.568631+0000","last_became_active":"2024-02-05T07:34:42.772349+0000","last_became_peered":"2024-02-05T07:34:42.772349+0000","last_unstale":"2024-02-05T07:34:59.568631+0000","last_undegraded":"2024-02-05T07:34:59.568631+0000","last_fullsized":"2024-02-05T07:34:59.568631+0000","mapping_epoch":302,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:40.216973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:40.216973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"326'34","reported_seq":954,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568558+0000","last_change":"2024-02-05T07:30:30.006087+0000","last_active":"2024-02-05T07:34:59.568558+0000","last_peered":"2024-02-05T07:34:59.568558+0000","last_clean":"2024-02-05T07:34:59.568558+0000","last_became_active":"2024-02-05T07:27:16.768454+0000","last_became_peered":"2024-02-05T07:27:16.768454+0000","last_unstale":"2024-02-05T07:34:59.568558+0000","last_undegraded":"2024-02-05T07:34:59.568558+0000","last_fullsized":"2024-02-05T07:34:59.568558+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'34","last_scrub_stamp":"2024-02-05T07:30:30.005988+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.005988+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":454,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.569120+0000","last_change":"2024-02-05T07:30:40.508975+0000","last_active":"2024-02-05T07:35:01.569120+0000","last_peered":"2024-02-05T07:35:01.569120+0000","last_clean":"2024-02-05T07:35:01.569120+0000","last_became_active":"2024-02-05T07:30:40.508561+0000","last_became_peered":"2024-02-05T07:30:40.508561+0000","last_unstale":"2024-02-05T07:35:01.569120+0000","last_undegraded":"2024-02-05T07:35:01.569120+0000","last_fullsized":"2024-02-05T07:35:01.569120+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:30.982457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.982457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.065218+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"257'9","reported_seq":1067,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.563220+0000","last_change":"2024-02-05T07:32:39.565923+0000","last_active":"2024-02-05T07:34:59.563220+0000","last_peered":"2024-02-05T07:34:59.563220+0000","last_clean":"2024-02-05T07:34:59.563220+0000","last_became_active":"2024-02-05T07:27:03.375195+0000","last_became_peered":"2024-02-05T07:27:03.375195+0000","last_unstale":"2024-02-05T07:34:59.563220+0000","last_undegraded":"2024-02-05T07:34:59.563220+0000","last_fullsized":"2024-02-05T07:34:59.563220+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'9","last_scrub_stamp":"2024-02-05T07:32:39.565833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:39.565833+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951553+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":79,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.572026+0000","last_change":"2024-02-05T07:34:38.025632+0000","last_active":"2024-02-05T07:34:59.572026+0000","last_peered":"2024-02-05T07:34:59.572026+0000","last_clean":"2024-02-05T07:34:59.572026+0000","last_became_active":"2024-02-05T07:34:30.685912+0000","last_became_peered":"2024-02-05T07:34:30.685912+0000","last_unstale":"2024-02-05T07:34:59.572026+0000","last_undegraded":"2024-02-05T07:34:59.572026+0000","last_fullsized":"2024-02-05T07:34:59.572026+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:38.025380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.025380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:40.345829+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.e","version":"475'6764","reported_seq":6878,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:35:00.333307+0000","last_change":"2024-02-05T07:33:38.613937+0000","last_active":"2024-02-05T07:35:00.333307+0000","last_peered":"2024-02-05T07:35:00.333307+0000","last_clean":"2024-02-05T07:35:00.333307+0000","last_became_active":"2024-02-05T07:33:38.613669+0000","last_became_peered":"2024-02-05T07:33:38.613669+0000","last_unstale":"2024-02-05T07:35:00.333307+0000","last_undegraded":"2024-02-05T07:35:00.333307+0000","last_fullsized":"2024-02-05T07:35:00.333307+0000","mapping_epoch":423,"log_start":"435'3508","ondisk_log_start":"435'3508","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3256,"log_dups_size":0,"ondisk_log_size":3256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.048196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443285504,"num_objects":6764,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6764,"num_write_kb":432896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":867,"reported_epoch":474,"state":"peering","last_fresh":"2024-02-05T07:34:58.987263+0000","last_change":"2024-02-05T07:34:59.565958+0000","last_active":"2024-02-05T07:34:59.565958+0000","last_peered":"2024-02-05T07:34:58.987263+0000","last_clean":"2024-02-05T07:34:58.987263+0000","last_became_active":"2024-02-05T07:31:36.758820+0000","last_became_peered":"2024-02-05T07:31:36.758820+0000","last_unstale":"2024-02-05T07:34:58.987263+0000","last_undegraded":"2024-02-05T07:34:58.987263+0000","last_fullsized":"2024-02-05T07:34:58.987263+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:24.656057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:24.656057+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":873,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.560114+0000","last_change":"2024-02-05T07:32:36.623960+0000","last_active":"2024-02-05T07:34:59.560114+0000","last_peered":"2024-02-05T07:34:59.560114+0000","last_clean":"2024-02-05T07:34:59.560114+0000","last_became_active":"2024-02-05T07:30:40.489212+0000","last_became_peered":"2024-02-05T07:30:40.489212+0000","last_unstale":"2024-02-05T07:34:59.560114+0000","last_undegraded":"2024-02-05T07:34:59.560114+0000","last_fullsized":"2024-02-05T07:34:59.560114+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:36.623865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:36.623865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951527+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"454'38","reported_seq":524,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.569034+0000","last_change":"2024-02-05T07:34:38.943008+0000","last_active":"2024-02-05T07:35:01.569034+0000","last_peered":"2024-02-05T07:35:01.569034+0000","last_clean":"2024-02-05T07:35:01.569034+0000","last_became_active":"2024-02-05T07:30:40.496940+0000","last_became_peered":"2024-02-05T07:30:40.496940+0000","last_unstale":"2024-02-05T07:35:01.569034+0000","last_undegraded":"2024-02-05T07:35:01.569034+0000","last_fullsized":"2024-02-05T07:35:01.569034+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'38","last_scrub_stamp":"2024-02-05T07:34:38.942830+0000","last_deep_scrub":"454'38","last_deep_scrub_stamp":"2024-02-05T07:34:38.942830+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.942830+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:53.297060+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":454,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.585188+0000","last_change":"2024-02-05T07:33:48.701618+0000","last_active":"2024-02-05T07:35:01.585188+0000","last_peered":"2024-02-05T07:35:01.585188+0000","last_clean":"2024-02-05T07:35:01.585188+0000","last_became_active":"2024-02-05T07:30:40.508607+0000","last_became_peered":"2024-02-05T07:30:40.508607+0000","last_unstale":"2024-02-05T07:35:01.585188+0000","last_undegraded":"2024-02-05T07:35:01.585188+0000","last_fullsized":"2024-02-05T07:35:01.585188+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:48.701399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:48.701399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:03.427068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":866,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.562777+0000","last_change":"2024-02-05T07:33:45.908153+0000","last_active":"2024-02-05T07:34:59.562777+0000","last_peered":"2024-02-05T07:34:59.562777+0000","last_clean":"2024-02-05T07:34:59.562777+0000","last_became_active":"2024-02-05T07:27:16.706666+0000","last_became_peered":"2024-02-05T07:27:16.706666+0000","last_unstale":"2024-02-05T07:34:59.562777+0000","last_undegraded":"2024-02-05T07:34:59.562777+0000","last_fullsized":"2024-02-05T07:34:59.562777+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:45.908064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:33:45.908064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:09.071480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"473'6542","reported_seq":6652,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:58.691192+0000","last_change":"2024-02-05T07:33:38.614195+0000","last_active":"2024-02-05T07:34:58.691192+0000","last_peered":"2024-02-05T07:34:58.691192+0000","last_clean":"2024-02-05T07:34:58.691192+0000","last_became_active":"2024-02-05T07:33:38.613892+0000","last_became_peered":"2024-02-05T07:33:38.613892+0000","last_unstale":"2024-02-05T07:34:58.691192+0000","last_undegraded":"2024-02-05T07:34:58.691192+0000","last_fullsized":"2024-02-05T07:34:58.691192+0000","mapping_epoch":423,"log_start":"430'2873","ondisk_log_start":"430'2873","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.064473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428736512,"num_objects":6542,"num_object_clones":0,"num_object_copies":19626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6542,"num_write_kb":418688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":1139,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.575690+0000","last_change":"2024-02-05T07:34:28.874781+0000","last_active":"2024-02-05T07:35:01.575690+0000","last_peered":"2024-02-05T07:35:01.575690+0000","last_clean":"2024-02-05T07:35:01.575690+0000","last_became_active":"2024-02-05T07:30:43.524588+0000","last_became_peered":"2024-02-05T07:30:43.524588+0000","last_unstale":"2024-02-05T07:35:01.575690+0000","last_undegraded":"2024-02-05T07:35:01.575690+0000","last_fullsized":"2024-02-05T07:35:01.575690+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:28.874337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.874337+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:50.255972+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"266'10","reported_seq":545,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.569196+0000","last_change":"2024-02-05T07:33:42.715242+0000","last_active":"2024-02-05T07:35:01.569196+0000","last_peered":"2024-02-05T07:35:01.569196+0000","last_clean":"2024-02-05T07:35:01.569196+0000","last_became_active":"2024-02-05T07:27:16.800207+0000","last_became_peered":"2024-02-05T07:27:16.800207+0000","last_unstale":"2024-02-05T07:35:01.569196+0000","last_undegraded":"2024-02-05T07:35:01.569196+0000","last_fullsized":"2024-02-05T07:35:01.569196+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'10","last_scrub_stamp":"2024-02-05T07:33:42.715162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:42.715162+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.065510+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"326'35","reported_seq":929,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.560242+0000","last_change":"2024-02-05T07:29:45.972745+0000","last_active":"2024-02-05T07:34:59.560242+0000","last_peered":"2024-02-05T07:34:59.560242+0000","last_clean":"2024-02-05T07:34:59.560242+0000","last_became_active":"2024-02-05T07:28:26.598831+0000","last_became_peered":"2024-02-05T07:28:26.598831+0000","last_unstale":"2024-02-05T07:34:59.560242+0000","last_undegraded":"2024-02-05T07:34:59.560242+0000","last_fullsized":"2024-02-05T07:34:59.560242+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:29:45.972589+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:29:45.972589+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951120+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":624,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.571069+0000","last_change":"2024-02-05T07:34:42.579613+0000","last_active":"2024-02-05T07:34:59.571069+0000","last_peered":"2024-02-05T07:34:59.571069+0000","last_clean":"2024-02-05T07:34:59.571069+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:34:59.571069+0000","last_undegraded":"2024-02-05T07:34:59.571069+0000","last_fullsized":"2024-02-05T07:34:59.571069+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:34:42.579486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:42.579486+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:01.974512+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":732,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.972904+0000","last_change":"2024-02-05T07:34:35.339060+0000","last_active":"2024-02-05T07:34:57.972904+0000","last_peered":"2024-02-05T07:34:57.972904+0000","last_clean":"2024-02-05T07:34:57.972904+0000","last_became_active":"2024-02-05T07:30:40.489120+0000","last_became_peered":"2024-02-05T07:30:40.489120+0000","last_unstale":"2024-02-05T07:34:57.972904+0000","last_undegraded":"2024-02-05T07:34:57.972904+0000","last_fullsized":"2024-02-05T07:34:57.972904+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:35.338979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:34:35.338979+0000","last_clean_scrub_stamp":"2024-02-05T07:34:35.338979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:01.319474+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"475'6884","reported_seq":6998,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:35:00.368039+0000","last_change":"2024-02-05T07:33:38.775156+0000","last_active":"2024-02-05T07:35:00.368039+0000","last_peered":"2024-02-05T07:35:00.368039+0000","last_clean":"2024-02-05T07:35:00.368039+0000","last_became_active":"2024-02-05T07:33:38.774713+0000","last_became_peered":"2024-02-05T07:33:38.774713+0000","last_unstale":"2024-02-05T07:35:00.368039+0000","last_undegraded":"2024-02-05T07:35:00.368039+0000","last_fullsized":"2024-02-05T07:35:00.368039+0000","mapping_epoch":423,"log_start":"436'3632","ondisk_log_start":"436'3632","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3252,"log_dups_size":0,"ondisk_log_size":3252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:05.742929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451149824,"num_objects":6884,"num_object_clones":0,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6884,"num_write_kb":440576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":747,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.978125+0000","last_change":"2024-02-05T07:34:54.952193+0000","last_active":"2024-02-05T07:34:57.978125+0000","last_peered":"2024-02-05T07:34:57.978125+0000","last_clean":"2024-02-05T07:34:57.978125+0000","last_became_active":"2024-02-05T07:34:54.951858+0000","last_became_peered":"2024-02-05T07:34:54.951858+0000","last_unstale":"2024-02-05T07:34:57.978125+0000","last_undegraded":"2024-02-05T07:34:57.978125+0000","last_fullsized":"2024-02-05T07:34:57.978125+0000","mapping_epoch":367,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:29.214474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:29.214474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:37.103396+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":529,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.973071+0000","last_change":"2024-02-05T07:32:49.752237+0000","last_active":"2024-02-05T07:34:57.973071+0000","last_peered":"2024-02-05T07:34:57.973071+0000","last_clean":"2024-02-05T07:34:57.973071+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:34:57.973071+0000","last_undegraded":"2024-02-05T07:34:57.973071+0000","last_fullsized":"2024-02-05T07:34:57.973071+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:32:49.752163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:49.752163+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"326'25","reported_seq":921,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.571793+0000","last_change":"2024-02-05T07:30:36.908113+0000","last_active":"2024-02-05T07:34:59.571793+0000","last_peered":"2024-02-05T07:34:59.571793+0000","last_clean":"2024-02-05T07:34:59.571793+0000","last_became_active":"2024-02-05T07:27:03.361342+0000","last_became_peered":"2024-02-05T07:27:03.361342+0000","last_unstale":"2024-02-05T07:34:59.571793+0000","last_undegraded":"2024-02-05T07:34:59.571793+0000","last_fullsized":"2024-02-05T07:34:59.571793+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:30:36.907989+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.907989+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":451,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.574719+0000","last_change":"2024-02-05T07:29:44.015577+0000","last_active":"2024-02-05T07:35:01.574719+0000","last_peered":"2024-02-05T07:35:01.574719+0000","last_clean":"2024-02-05T07:35:01.574719+0000","last_became_active":"2024-02-05T07:28:26.596096+0000","last_became_peered":"2024-02-05T07:28:26.596096+0000","last_unstale":"2024-02-05T07:35:01.574719+0000","last_undegraded":"2024-02-05T07:35:01.574719+0000","last_fullsized":"2024-02-05T07:35:01.574719+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:44.015468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:29:44.015468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.065069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":143,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568545+0000","last_change":"2024-02-05T07:33:24.419375+0000","last_active":"2024-02-05T07:34:59.568545+0000","last_peered":"2024-02-05T07:34:59.568545+0000","last_clean":"2024-02-05T07:34:59.568545+0000","last_became_active":"2024-02-05T07:32:58.598709+0000","last_became_peered":"2024-02-05T07:32:58.598709+0000","last_unstale":"2024-02-05T07:34:59.568545+0000","last_undegraded":"2024-02-05T07:34:59.568545+0000","last_fullsized":"2024-02-05T07:34:59.568545+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:24.419250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:24.419250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"473'6508","reported_seq":6618,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:58.726155+0000","last_change":"2024-02-05T07:33:38.775383+0000","last_active":"2024-02-05T07:34:58.726155+0000","last_peered":"2024-02-05T07:34:58.726155+0000","last_clean":"2024-02-05T07:34:58.726155+0000","last_became_active":"2024-02-05T07:33:38.775048+0000","last_became_peered":"2024-02-05T07:33:38.775048+0000","last_unstale":"2024-02-05T07:34:58.726155+0000","last_undegraded":"2024-02-05T07:34:58.726155+0000","last_fullsized":"2024-02-05T07:34:58.726155+0000","mapping_epoch":423,"log_start":"430'2837","ondisk_log_start":"430'2837","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3671,"log_dups_size":0,"ondisk_log_size":3671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:00.065375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426508288,"num_objects":6508,"num_object_clones":0,"num_object_copies":19524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6508,"num_write_kb":416512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":731,"reported_epoch":472,"state":"peering","last_fresh":"2024-02-05T07:34:56.972358+0000","last_change":"2024-02-05T07:34:57.975911+0000","last_active":"2024-02-05T07:34:55.949543+0000","last_peered":"2024-02-05T07:34:56.972358+0000","last_clean":"2024-02-05T07:34:56.972358+0000","last_became_active":"2024-02-05T07:31:36.840534+0000","last_became_peered":"2024-02-05T07:34:56.971851+0000","last_unstale":"2024-02-05T07:34:56.972358+0000","last_undegraded":"2024-02-05T07:34:56.972358+0000","last_fullsized":"2024-02-05T07:34:56.972358+0000","mapping_epoch":302,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:47.197225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.197225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":699,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.974704+0000","last_change":"2024-02-05T07:33:33.212392+0000","last_active":"2024-02-05T07:34:57.974704+0000","last_peered":"2024-02-05T07:34:57.974704+0000","last_clean":"2024-02-05T07:34:57.974704+0000","last_became_active":"2024-02-05T07:28:26.589673+0000","last_became_peered":"2024-02-05T07:28:26.589673+0000","last_unstale":"2024-02-05T07:34:57.974704+0000","last_undegraded":"2024-02-05T07:34:57.974704+0000","last_fullsized":"2024-02-05T07:34:57.974704+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:33.212264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.212264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:58.757962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"326'29","reported_seq":490,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.567976+0000","last_change":"2024-02-05T07:30:40.516301+0000","last_active":"2024-02-05T07:34:59.567976+0000","last_peered":"2024-02-05T07:34:59.567976+0000","last_clean":"2024-02-05T07:34:59.567976+0000","last_became_active":"2024-02-05T07:30:40.481234+0000","last_became_peered":"2024-02-05T07:30:40.481234+0000","last_unstale":"2024-02-05T07:34:59.567976+0000","last_undegraded":"2024-02-05T07:34:59.567976+0000","last_fullsized":"2024-02-05T07:34:59.567976+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'29","last_scrub_stamp":"2024-02-05T07:30:37.106672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:37.106672+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":487,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.570828+0000","last_change":"2024-02-05T07:34:34.799486+0000","last_active":"2024-02-05T07:34:59.570828+0000","last_peered":"2024-02-05T07:34:59.570828+0000","last_clean":"2024-02-05T07:34:59.570828+0000","last_became_active":"2024-02-05T07:34:30.705994+0000","last_became_peered":"2024-02-05T07:34:30.705994+0000","last_unstale":"2024-02-05T07:34:59.570828+0000","last_undegraded":"2024-02-05T07:34:59.570828+0000","last_fullsized":"2024-02-05T07:34:59.570828+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:34.799346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:34:34.799346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:43.527345+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":861,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.569361+0000","last_change":"2024-02-05T07:33:33.257106+0000","last_active":"2024-02-05T07:34:59.569361+0000","last_peered":"2024-02-05T07:34:59.569361+0000","last_clean":"2024-02-05T07:34:59.569361+0000","last_became_active":"2024-02-05T07:28:25.840850+0000","last_became_peered":"2024-02-05T07:28:25.840850+0000","last_unstale":"2024-02-05T07:34:59.569361+0000","last_undegraded":"2024-02-05T07:34:59.569361+0000","last_fullsized":"2024-02-05T07:34:59.569361+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:33.257028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.257028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.513556+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"475'6663","reported_seq":6777,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:35:00.015519+0000","last_change":"2024-02-05T07:33:38.777030+0000","last_active":"2024-02-05T07:35:00.015519+0000","last_peered":"2024-02-05T07:35:00.015519+0000","last_clean":"2024-02-05T07:35:00.015519+0000","last_became_active":"2024-02-05T07:33:38.776442+0000","last_became_peered":"2024-02-05T07:33:38.776442+0000","last_unstale":"2024-02-05T07:35:00.015519+0000","last_undegraded":"2024-02-05T07:35:00.015519+0000","last_fullsized":"2024-02-05T07:35:00.015519+0000","mapping_epoch":423,"log_start":"431'2930","ondisk_log_start":"431'2930","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3733,"log_dups_size":0,"ondisk_log_size":3733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436666368,"num_objects":6663,"num_object_clones":0,"num_object_copies":19989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6663,"num_write_kb":426432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":386,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.973253+0000","last_change":"2024-02-05T07:34:31.185521+0000","last_active":"2024-02-05T07:34:57.973253+0000","last_peered":"2024-02-05T07:34:57.973253+0000","last_clean":"2024-02-05T07:34:57.973253+0000","last_became_active":"2024-02-05T07:34:30.653119+0000","last_became_peered":"2024-02-05T07:34:30.653119+0000","last_unstale":"2024-02-05T07:34:57.973253+0000","last_undegraded":"2024-02-05T07:34:57.973253+0000","last_fullsized":"2024-02-05T07:34:57.973253+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:31.185403+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:34:31.185403+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:48.418757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":461,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.569037+0000","last_change":"2024-02-05T07:32:46.399673+0000","last_active":"2024-02-05T07:34:59.569037+0000","last_peered":"2024-02-05T07:34:59.569037+0000","last_clean":"2024-02-05T07:34:59.569037+0000","last_became_active":"2024-02-05T07:27:03.370297+0000","last_became_peered":"2024-02-05T07:27:03.370297+0000","last_unstale":"2024-02-05T07:34:59.569037+0000","last_undegraded":"2024-02-05T07:34:59.569037+0000","last_fullsized":"2024-02-05T07:34:59.569037+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:46.399572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:46.399572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"326'24","reported_seq":421,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.572389+0000","last_change":"2024-02-05T07:34:30.678315+0000","last_active":"2024-02-05T07:34:59.572389+0000","last_peered":"2024-02-05T07:34:59.572389+0000","last_clean":"2024-02-05T07:34:59.572389+0000","last_became_active":"2024-02-05T07:34:30.677886+0000","last_became_peered":"2024-02-05T07:34:30.677886+0000","last_unstale":"2024-02-05T07:34:59.572389+0000","last_undegraded":"2024-02-05T07:34:59.572389+0000","last_fullsized":"2024-02-05T07:34:59.572389+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:33:49.499846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:49.499846+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":538,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568992+0000","last_change":"2024-02-05T07:30:40.520395+0000","last_active":"2024-02-05T07:34:59.568992+0000","last_peered":"2024-02-05T07:34:59.568992+0000","last_clean":"2024-02-05T07:34:59.568992+0000","last_became_active":"2024-02-05T07:30:40.493770+0000","last_became_peered":"2024-02-05T07:30:40.493770+0000","last_unstale":"2024-02-05T07:34:59.568992+0000","last_undegraded":"2024-02-05T07:34:59.568992+0000","last_fullsized":"2024-02-05T07:34:59.568992+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:29:51.291146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.291146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":265,"num_read_kb":260,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":836,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.560254+0000","last_change":"2024-02-05T07:33:39.809630+0000","last_active":"2024-02-05T07:34:59.560254+0000","last_peered":"2024-02-05T07:34:59.560254+0000","last_clean":"2024-02-05T07:34:59.560254+0000","last_became_active":"2024-02-05T07:27:16.724280+0000","last_became_peered":"2024-02-05T07:27:16.724280+0000","last_unstale":"2024-02-05T07:34:59.560254+0000","last_undegraded":"2024-02-05T07:34:59.560254+0000","last_fullsized":"2024-02-05T07:34:59.560254+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:39.809544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:33:39.809544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:03.386365+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":889,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.569193+0000","last_change":"2024-02-05T07:33:41.379002+0000","last_active":"2024-02-05T07:34:59.569193+0000","last_peered":"2024-02-05T07:34:59.569193+0000","last_clean":"2024-02-05T07:34:59.569193+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:34:59.569193+0000","last_undegraded":"2024-02-05T07:34:59.569193+0000","last_fullsized":"2024-02-05T07:34:59.569193+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:33:41.378909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.378909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"473'6553","reported_seq":6663,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:58.714141+0000","last_change":"2024-02-05T07:33:38.775761+0000","last_active":"2024-02-05T07:34:58.714141+0000","last_peered":"2024-02-05T07:34:58.714141+0000","last_clean":"2024-02-05T07:34:58.714141+0000","last_became_active":"2024-02-05T07:33:38.775338+0000","last_became_peered":"2024-02-05T07:33:38.775338+0000","last_unstale":"2024-02-05T07:34:58.714141+0000","last_undegraded":"2024-02-05T07:34:58.714141+0000","last_fullsized":"2024-02-05T07:34:58.714141+0000","mapping_epoch":423,"log_start":"430'2876","ondisk_log_start":"430'2876","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3677,"log_dups_size":0,"ondisk_log_size":3677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:03.250327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429457408,"num_objects":6553,"num_object_clones":0,"num_object_copies":19659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":419392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":636,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.571958+0000","last_change":"2024-02-05T07:33:50.943755+0000","last_active":"2024-02-05T07:34:59.571958+0000","last_peered":"2024-02-05T07:34:59.571958+0000","last_clean":"2024-02-05T07:34:59.571958+0000","last_became_active":"2024-02-05T07:30:40.474169+0000","last_became_peered":"2024-02-05T07:30:40.474169+0000","last_unstale":"2024-02-05T07:34:59.571958+0000","last_undegraded":"2024-02-05T07:34:59.571958+0000","last_fullsized":"2024-02-05T07:34:59.571958+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:50.943640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:50.943640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:06.509391+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"365'31","reported_seq":941,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.975937+0000","last_change":"2024-02-05T07:33:46.200250+0000","last_active":"2024-02-05T07:34:57.975937+0000","last_peered":"2024-02-05T07:34:57.975937+0000","last_clean":"2024-02-05T07:34:57.975937+0000","last_became_active":"2024-02-05T07:30:40.508686+0000","last_became_peered":"2024-02-05T07:30:40.508686+0000","last_unstale":"2024-02-05T07:34:57.975937+0000","last_undegraded":"2024-02-05T07:34:57.975937+0000","last_fullsized":"2024-02-05T07:34:57.975937+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'31","last_scrub_stamp":"2024-02-05T07:33:46.200150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.200150+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.420503+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":451,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.573913+0000","last_change":"2024-02-05T07:29:43.000854+0000","last_active":"2024-02-05T07:35:01.573913+0000","last_peered":"2024-02-05T07:35:01.573913+0000","last_clean":"2024-02-05T07:35:01.573913+0000","last_became_active":"2024-02-05T07:28:26.595640+0000","last_became_peered":"2024-02-05T07:28:26.595640+0000","last_unstale":"2024-02-05T07:35:01.573913+0000","last_undegraded":"2024-02-05T07:35:01.573913+0000","last_fullsized":"2024-02-05T07:35:01.573913+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:43.000778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:43.000778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.064828+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":573,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.973255+0000","last_change":"2024-02-05T07:33:48.221978+0000","last_active":"2024-02-05T07:34:57.973255+0000","last_peered":"2024-02-05T07:34:57.973255+0000","last_clean":"2024-02-05T07:34:57.973255+0000","last_became_active":"2024-02-05T07:30:43.516031+0000","last_became_peered":"2024-02-05T07:30:43.516031+0000","last_unstale":"2024-02-05T07:34:57.973255+0000","last_undegraded":"2024-02-05T07:34:57.973255+0000","last_fullsized":"2024-02-05T07:34:57.973255+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:33:48.221845+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":448,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.585097+0000","last_change":"2024-02-05T07:33:46.760555+0000","last_active":"2024-02-05T07:35:01.585097+0000","last_peered":"2024-02-05T07:35:01.585097+0000","last_clean":"2024-02-05T07:35:01.585097+0000","last_became_active":"2024-02-05T07:27:16.751341+0000","last_became_peered":"2024-02-05T07:27:16.751341+0000","last_unstale":"2024-02-05T07:35:01.585097+0000","last_undegraded":"2024-02-05T07:35:01.585097+0000","last_fullsized":"2024-02-05T07:35:01.585097+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:46.760448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.760448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"475'6654","reported_seq":6768,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:35:00.002360+0000","last_change":"2024-02-05T07:33:38.615853+0000","last_active":"2024-02-05T07:35:00.002360+0000","last_peered":"2024-02-05T07:35:00.002360+0000","last_clean":"2024-02-05T07:35:00.002360+0000","last_became_active":"2024-02-05T07:33:38.615574+0000","last_became_peered":"2024-02-05T07:33:38.615574+0000","last_unstale":"2024-02-05T07:35:00.002360+0000","last_undegraded":"2024-02-05T07:35:00.002360+0000","last_fullsized":"2024-02-05T07:35:00.002360+0000","mapping_epoch":423,"log_start":"431'2921","ondisk_log_start":"431'2921","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3733,"log_dups_size":0,"ondisk_log_size":3733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436076544,"num_objects":6654,"num_object_clones":0,"num_object_copies":19962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":425856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":880,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.563335+0000","last_change":"2024-02-05T07:32:21.667678+0000","last_active":"2024-02-05T07:34:59.563335+0000","last_peered":"2024-02-05T07:34:59.563335+0000","last_clean":"2024-02-05T07:34:59.563335+0000","last_became_active":"2024-02-05T07:30:43.509733+0000","last_became_peered":"2024-02-05T07:30:43.509733+0000","last_unstale":"2024-02-05T07:34:59.563335+0000","last_undegraded":"2024-02-05T07:34:59.563335+0000","last_fullsized":"2024-02-05T07:34:59.563335+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:21.667581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:21.667581+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":875,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.572723+0000","last_change":"2024-02-05T07:33:23.118646+0000","last_active":"2024-02-05T07:34:59.572723+0000","last_peered":"2024-02-05T07:34:59.572723+0000","last_clean":"2024-02-05T07:34:59.572723+0000","last_became_active":"2024-02-05T07:30:40.503816+0000","last_became_peered":"2024-02-05T07:30:40.503816+0000","last_unstale":"2024-02-05T07:34:59.572723+0000","last_undegraded":"2024-02-05T07:34:59.572723+0000","last_fullsized":"2024-02-05T07:34:59.572723+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:23.118569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:23.118569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"326'22","reported_seq":934,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568814+0000","last_change":"2024-02-05T07:33:30.327114+0000","last_active":"2024-02-05T07:34:59.568814+0000","last_peered":"2024-02-05T07:34:59.568814+0000","last_clean":"2024-02-05T07:34:59.568814+0000","last_became_active":"2024-02-05T07:30:40.494460+0000","last_became_peered":"2024-02-05T07:30:40.494460+0000","last_unstale":"2024-02-05T07:34:59.568814+0000","last_undegraded":"2024-02-05T07:34:59.568814+0000","last_fullsized":"2024-02-05T07:34:59.568814+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'22","last_scrub_stamp":"2024-02-05T07:33:30.327027+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.327027+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":900,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568752+0000","last_change":"2024-02-05T07:33:42.096031+0000","last_active":"2024-02-05T07:34:59.568752+0000","last_peered":"2024-02-05T07:34:59.568752+0000","last_clean":"2024-02-05T07:34:59.568752+0000","last_became_active":"2024-02-05T07:27:16.728869+0000","last_became_peered":"2024-02-05T07:27:16.728869+0000","last_unstale":"2024-02-05T07:34:59.568752+0000","last_undegraded":"2024-02-05T07:34:59.568752+0000","last_fullsized":"2024-02-05T07:34:59.568752+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:42.095953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:33:42.095953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":76,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568517+0000","last_change":"2024-02-05T07:34:30.668997+0000","last_active":"2024-02-05T07:34:59.568517+0000","last_peered":"2024-02-05T07:34:59.568517+0000","last_clean":"2024-02-05T07:34:59.568517+0000","last_became_active":"2024-02-05T07:34:30.668558+0000","last_became_peered":"2024-02-05T07:34:30.668558+0000","last_unstale":"2024-02-05T07:34:59.568517+0000","last_undegraded":"2024-02-05T07:34:59.568517+0000","last_fullsized":"2024-02-05T07:34:59.568517+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:34:26.591385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:26.591385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:52.547419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":736,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.975752+0000","last_change":"2024-02-05T07:32:38.877762+0000","last_active":"2024-02-05T07:34:57.975752+0000","last_peered":"2024-02-05T07:34:57.975752+0000","last_clean":"2024-02-05T07:34:57.975752+0000","last_became_active":"2024-02-05T07:30:40.489314+0000","last_became_peered":"2024-02-05T07:30:40.489314+0000","last_unstale":"2024-02-05T07:34:57.975752+0000","last_undegraded":"2024-02-05T07:34:57.975752+0000","last_fullsized":"2024-02-05T07:34:57.975752+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:38.877685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:32:38.877685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"473'6605","reported_seq":6715,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:58.703220+0000","last_change":"2024-02-05T07:33:38.775531+0000","last_active":"2024-02-05T07:34:58.703220+0000","last_peered":"2024-02-05T07:34:58.703220+0000","last_clean":"2024-02-05T07:34:58.703220+0000","last_became_active":"2024-02-05T07:33:38.775221+0000","last_became_peered":"2024-02-05T07:33:38.775221+0000","last_unstale":"2024-02-05T07:34:58.703220+0000","last_undegraded":"2024-02-05T07:34:58.703220+0000","last_fullsized":"2024-02-05T07:34:58.703220+0000","mapping_epoch":423,"log_start":"430'2911","ondisk_log_start":"430'2911","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:03.250255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432865280,"num_objects":6605,"num_object_clones":0,"num_object_copies":19815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6605,"num_write_kb":422720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":882,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568191+0000","last_change":"2024-02-05T07:33:41.082128+0000","last_active":"2024-02-05T07:34:59.568191+0000","last_peered":"2024-02-05T07:34:59.568191+0000","last_clean":"2024-02-05T07:34:59.568191+0000","last_became_active":"2024-02-05T07:28:14.305034+0000","last_became_peered":"2024-02-05T07:28:14.305034+0000","last_unstale":"2024-02-05T07:34:59.568191+0000","last_undegraded":"2024-02-05T07:34:59.568191+0000","last_fullsized":"2024-02-05T07:34:59.568191+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:41.082046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.082046+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":616,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568592+0000","last_change":"2024-02-05T07:34:37.540422+0000","last_active":"2024-02-05T07:34:59.568592+0000","last_peered":"2024-02-05T07:34:59.568592+0000","last_clean":"2024-02-05T07:34:59.568592+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:34:59.568592+0000","last_undegraded":"2024-02-05T07:34:59.568592+0000","last_fullsized":"2024-02-05T07:34:59.568592+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:37.540304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:34:37.540304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:56.793282+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"326'44","reported_seq":802,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.571079+0000","last_change":"2024-02-05T07:30:36.009087+0000","last_active":"2024-02-05T07:34:59.571079+0000","last_peered":"2024-02-05T07:34:59.571079+0000","last_clean":"2024-02-05T07:34:59.571079+0000","last_became_active":"2024-02-05T07:27:16.708475+0000","last_became_peered":"2024-02-05T07:27:16.708475+0000","last_unstale":"2024-02-05T07:34:59.571079+0000","last_undegraded":"2024-02-05T07:34:59.571079+0000","last_fullsized":"2024-02-05T07:34:59.571079+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'44","last_scrub_stamp":"2024-02-05T07:30:36.008885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.008885+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":251,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.573405+0000","last_change":"2024-02-05T07:30:32.964745+0000","last_active":"2024-02-05T07:34:59.573405+0000","last_peered":"2024-02-05T07:34:59.573405+0000","last_clean":"2024-02-05T07:34:59.573405+0000","last_became_active":"2024-02-05T07:30:30.524566+0000","last_became_peered":"2024-02-05T07:30:30.524566+0000","last_unstale":"2024-02-05T07:34:59.573405+0000","last_undegraded":"2024-02-05T07:34:59.573405+0000","last_fullsized":"2024-02-05T07:34:59.573405+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:32.964622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:32.964622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":786,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.569284+0000","last_change":"2024-02-05T07:33:43.473208+0000","last_active":"2024-02-05T07:34:59.569284+0000","last_peered":"2024-02-05T07:34:59.569284+0000","last_clean":"2024-02-05T07:34:59.569284+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:34:59.569284+0000","last_undegraded":"2024-02-05T07:34:59.569284+0000","last_fullsized":"2024-02-05T07:34:59.569284+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:33:43.473107+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:33:43.473107+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:12.505533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":462,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.574934+0000","last_change":"2024-02-05T07:34:26.897153+0000","last_active":"2024-02-05T07:35:01.574934+0000","last_peered":"2024-02-05T07:35:01.574934+0000","last_clean":"2024-02-05T07:35:01.574934+0000","last_became_active":"2024-02-05T07:30:40.508071+0000","last_became_peered":"2024-02-05T07:30:40.508071+0000","last_unstale":"2024-02-05T07:35:01.574934+0000","last_undegraded":"2024-02-05T07:35:01.574934+0000","last_fullsized":"2024-02-05T07:35:01.574934+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:26.897028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:26.897028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:52.202110+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.5","version":"475'6644","reported_seq":6757,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:35:00.024516+0000","last_change":"2024-02-05T07:33:38.778391+0000","last_active":"2024-02-05T07:35:00.024516+0000","last_peered":"2024-02-05T07:35:00.024516+0000","last_clean":"2024-02-05T07:35:00.024516+0000","last_became_active":"2024-02-05T07:33:38.778086+0000","last_became_peered":"2024-02-05T07:33:38.778086+0000","last_unstale":"2024-02-05T07:35:00.024516+0000","last_undegraded":"2024-02-05T07:35:00.024516+0000","last_fullsized":"2024-02-05T07:35:00.024516+0000","mapping_epoch":423,"log_start":"431'2890","ondisk_log_start":"431'2890","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3754,"log_dups_size":0,"ondisk_log_size":3754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435421184,"num_objects":6644,"num_object_clones":0,"num_object_copies":19932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6644,"num_write_kb":425216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":551,"reported_epoch":475,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:34:59.591084+0000","last_change":"2024-02-05T07:34:59.590968+0000","last_active":"2024-02-05T07:34:59.591084+0000","last_peered":"2024-02-05T07:34:59.591084+0000","last_clean":"2024-02-05T07:30:38.596366+0000","last_became_active":"2024-02-05T07:34:59.589375+0000","last_became_peered":"2024-02-05T07:34:59.589375+0000","last_unstale":"2024-02-05T07:34:59.591084+0000","last_undegraded":"2024-02-05T07:34:59.591084+0000","last_fullsized":"2024-02-05T07:34:59.591084+0000","mapping_epoch":474,"log_start":"86'128","ondisk_log_start":"86'128","created":291,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:25.788995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:25.788995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[2,3,4],"avail_no_missing":["2","0","3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"326'15","reported_seq":651,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.572777+0000","last_change":"2024-02-05T07:29:51.665907+0000","last_active":"2024-02-05T07:34:59.572777+0000","last_peered":"2024-02-05T07:34:59.572777+0000","last_clean":"2024-02-05T07:34:59.572777+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:34:59.572777+0000","last_undegraded":"2024-02-05T07:34:59.572777+0000","last_fullsized":"2024-02-05T07:34:59.572777+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:29:51.665808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:51.665808+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":451,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.576000+0000","last_change":"2024-02-05T07:30:38.998491+0000","last_active":"2024-02-05T07:35:01.576000+0000","last_peered":"2024-02-05T07:35:01.576000+0000","last_clean":"2024-02-05T07:35:01.576000+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:35:01.576000+0000","last_undegraded":"2024-02-05T07:35:01.576000+0000","last_fullsized":"2024-02-05T07:35:01.576000+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:38.995586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:38.995586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.065276+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":875,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.571505+0000","last_change":"2024-02-05T07:32:47.161468+0000","last_active":"2024-02-05T07:34:59.571505+0000","last_peered":"2024-02-05T07:34:59.571505+0000","last_clean":"2024-02-05T07:34:59.571505+0000","last_became_active":"2024-02-05T07:27:16.708205+0000","last_became_peered":"2024-02-05T07:27:16.708205+0000","last_unstale":"2024-02-05T07:34:59.571505+0000","last_undegraded":"2024-02-05T07:34:59.571505+0000","last_fullsized":"2024-02-05T07:34:59.571505+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:47.161343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:47.161343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":244,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.573676+0000","last_change":"2024-02-05T07:32:46.198277+0000","last_active":"2024-02-05T07:34:59.573676+0000","last_peered":"2024-02-05T07:34:59.573676+0000","last_clean":"2024-02-05T07:34:59.573676+0000","last_became_active":"2024-02-05T07:30:40.486401+0000","last_became_peered":"2024-02-05T07:30:40.486401+0000","last_unstale":"2024-02-05T07:34:59.573676+0000","last_undegraded":"2024-02-05T07:34:59.573676+0000","last_fullsized":"2024-02-05T07:34:59.573676+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:46.198155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:32:46.198155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"475'6765","reported_seq":6879,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:35:00.363414+0000","last_change":"2024-02-05T07:33:38.776231+0000","last_active":"2024-02-05T07:35:00.363414+0000","last_peered":"2024-02-05T07:35:00.363414+0000","last_clean":"2024-02-05T07:35:00.363414+0000","last_became_active":"2024-02-05T07:33:38.775291+0000","last_became_peered":"2024-02-05T07:33:38.775291+0000","last_unstale":"2024-02-05T07:35:00.363414+0000","last_undegraded":"2024-02-05T07:35:00.363414+0000","last_fullsized":"2024-02-05T07:35:00.363414+0000","mapping_epoch":423,"log_start":"435'3536","ondisk_log_start":"435'3536","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3229,"log_dups_size":0,"ondisk_log_size":3229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.048147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443351040,"num_objects":6765,"num_object_clones":0,"num_object_copies":20295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6765,"num_write_kb":432960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":721,"reported_epoch":472,"state":"peering","last_fresh":"2024-02-05T07:34:56.957944+0000","last_change":"2024-02-05T07:34:57.971041+0000","last_active":"2024-02-05T07:34:57.971041+0000","last_peered":"2024-02-05T07:34:56.957944+0000","last_clean":"2024-02-05T07:34:56.957944+0000","last_became_active":"2024-02-05T07:28:26.596711+0000","last_became_peered":"2024-02-05T07:28:26.596711+0000","last_unstale":"2024-02-05T07:34:56.957944+0000","last_undegraded":"2024-02-05T07:34:56.957944+0000","last_fullsized":"2024-02-05T07:34:56.957944+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:23.202589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:23.202589+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"326'35","reported_seq":774,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.974712+0000","last_change":"2024-02-05T07:33:27.163756+0000","last_active":"2024-02-05T07:34:57.974712+0000","last_peered":"2024-02-05T07:34:57.974712+0000","last_clean":"2024-02-05T07:34:57.974712+0000","last_became_active":"2024-02-05T07:30:40.482753+0000","last_became_peered":"2024-02-05T07:30:40.482753+0000","last_unstale":"2024-02-05T07:34:57.974712+0000","last_undegraded":"2024-02-05T07:34:57.974712+0000","last_fullsized":"2024-02-05T07:34:57.974712+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:33:27.163551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:27.163551+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":746,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.975516+0000","last_change":"2024-02-05T07:32:39.793152+0000","last_active":"2024-02-05T07:34:57.975516+0000","last_peered":"2024-02-05T07:34:57.975516+0000","last_clean":"2024-02-05T07:34:57.975516+0000","last_became_active":"2024-02-05T07:30:40.507705+0000","last_became_peered":"2024-02-05T07:30:40.507705+0000","last_unstale":"2024-02-05T07:34:57.975516+0000","last_undegraded":"2024-02-05T07:34:57.975516+0000","last_fullsized":"2024-02-05T07:34:57.975516+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_clean_scrub_stamp":"2024-02-05T07:32:39.793075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":875,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.573621+0000","last_change":"2024-02-05T07:33:28.121535+0000","last_active":"2024-02-05T07:34:59.573621+0000","last_peered":"2024-02-05T07:34:59.573621+0000","last_clean":"2024-02-05T07:34:59.573621+0000","last_became_active":"2024-02-05T07:30:40.479288+0000","last_became_peered":"2024-02-05T07:30:40.479288+0000","last_unstale":"2024-02-05T07:34:59.573621+0000","last_undegraded":"2024-02-05T07:34:59.573621+0000","last_fullsized":"2024-02-05T07:34:59.573621+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.121427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.121427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":657,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568654+0000","last_change":"2024-02-05T07:33:31.033044+0000","last_active":"2024-02-05T07:34:59.568654+0000","last_peered":"2024-02-05T07:34:59.568654+0000","last_clean":"2024-02-05T07:34:59.568654+0000","last_became_active":"2024-02-05T07:30:40.496822+0000","last_became_peered":"2024-02-05T07:30:40.496822+0000","last_unstale":"2024-02-05T07:34:59.568654+0000","last_undegraded":"2024-02-05T07:34:59.568654+0000","last_fullsized":"2024-02-05T07:34:59.568654+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:31.032967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_clean_scrub_stamp":"2024-02-05T07:33:31.032967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"473'6545","reported_seq":6655,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:58.714291+0000","last_change":"2024-02-05T07:33:38.614213+0000","last_active":"2024-02-05T07:34:58.714291+0000","last_peered":"2024-02-05T07:34:58.714291+0000","last_clean":"2024-02-05T07:34:58.714291+0000","last_became_active":"2024-02-05T07:33:38.613855+0000","last_became_peered":"2024-02-05T07:33:38.613855+0000","last_unstale":"2024-02-05T07:34:58.714291+0000","last_undegraded":"2024-02-05T07:34:58.714291+0000","last_fullsized":"2024-02-05T07:34:58.714291+0000","mapping_epoch":423,"log_start":"430'2880","ondisk_log_start":"430'2880","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3665,"log_dups_size":0,"ondisk_log_size":3665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:03.246294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428933120,"num_objects":6545,"num_object_clones":0,"num_object_copies":19635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6545,"num_write_kb":418880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":1157,"reported_epoch":477,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:35:01.575679+0000","last_change":"2024-02-05T07:34:59.610810+0000","last_active":"2024-02-05T07:35:01.575679+0000","last_peered":"2024-02-05T07:35:01.575679+0000","last_clean":"2024-02-05T07:34:56.960306+0000","last_became_active":"2024-02-05T07:34:59.604309+0000","last_became_peered":"2024-02-05T07:34:59.604309+0000","last_unstale":"2024-02-05T07:35:01.575679+0000","last_undegraded":"2024-02-05T07:35:01.575679+0000","last_fullsized":"2024-02-05T07:35:01.575679+0000","mapping_epoch":474,"log_start":"86'128","ondisk_log_start":"86'128","created":291,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:41.754605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:41.754605+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,0],"avail_no_missing":["3","0","6"],"object_location_counts":[{"shards":"0,3,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":871,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568431+0000","last_change":"2024-02-05T07:33:29.082880+0000","last_active":"2024-02-05T07:34:59.568431+0000","last_peered":"2024-02-05T07:34:59.568431+0000","last_clean":"2024-02-05T07:34:59.568431+0000","last_became_active":"2024-02-05T07:18:53.460756+0000","last_became_peered":"2024-02-05T07:18:53.460756+0000","last_unstale":"2024-02-05T07:34:59.568431+0000","last_undegraded":"2024-02-05T07:34:59.568431+0000","last_fullsized":"2024-02-05T07:34:59.568431+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:29.082803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:29.082803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"325'18","reported_seq":959,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.560332+0000","last_change":"2024-02-05T07:34:28.902927+0000","last_active":"2024-02-05T07:34:59.560332+0000","last_peered":"2024-02-05T07:34:59.560332+0000","last_clean":"2024-02-05T07:34:59.560332+0000","last_became_active":"2024-02-05T07:27:16.706871+0000","last_became_peered":"2024-02-05T07:27:16.706871+0000","last_unstale":"2024-02-05T07:34:59.560332+0000","last_undegraded":"2024-02-05T07:34:59.560332+0000","last_fullsized":"2024-02-05T07:34:59.560332+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'18","last_scrub_stamp":"2024-02-05T07:34:28.902809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.902809+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:52.320348+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":240,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.561523+0000","last_change":"2024-02-05T07:33:33.822290+0000","last_active":"2024-02-05T07:34:59.561523+0000","last_peered":"2024-02-05T07:34:59.561523+0000","last_clean":"2024-02-05T07:34:59.561523+0000","last_became_active":"2024-02-05T07:30:40.518949+0000","last_became_peered":"2024-02-05T07:30:40.518949+0000","last_unstale":"2024-02-05T07:34:59.561523+0000","last_undegraded":"2024-02-05T07:34:59.561523+0000","last_fullsized":"2024-02-05T07:34:59.561523+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:33.822199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.822199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":462,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.573722+0000","last_change":"2024-02-05T07:34:38.057821+0000","last_active":"2024-02-05T07:35:01.573722+0000","last_peered":"2024-02-05T07:35:01.573722+0000","last_clean":"2024-02-05T07:35:01.573722+0000","last_became_active":"2024-02-05T07:30:40.485991+0000","last_became_peered":"2024-02-05T07:30:40.485991+0000","last_unstale":"2024-02-05T07:35:01.573722+0000","last_undegraded":"2024-02-05T07:35:01.573722+0000","last_fullsized":"2024-02-05T07:35:01.573722+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:38.057598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.057598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:46.270675+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"475'6671","reported_seq":6785,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:35:00.370801+0000","last_change":"2024-02-05T07:33:38.775313+0000","last_active":"2024-02-05T07:35:00.370801+0000","last_peered":"2024-02-05T07:35:00.370801+0000","last_clean":"2024-02-05T07:35:00.370801+0000","last_became_active":"2024-02-05T07:33:38.774996+0000","last_became_peered":"2024-02-05T07:33:38.774996+0000","last_unstale":"2024-02-05T07:35:00.370801+0000","last_undegraded":"2024-02-05T07:35:00.370801+0000","last_fullsized":"2024-02-05T07:35:00.370801+0000","mapping_epoch":423,"log_start":"433'3426","ondisk_log_start":"433'3426","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3245,"log_dups_size":0,"ondisk_log_size":3245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437190656,"num_objects":6671,"num_object_clones":0,"num_object_copies":20013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6671,"num_write_kb":426944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":737,"reported_epoch":475,"state":"active+clean+remapped","last_fresh":"2024-02-05T07:34:59.589483+0000","last_change":"2024-02-05T07:34:59.589483+0000","last_active":"2024-02-05T07:34:59.589483+0000","last_peered":"2024-02-05T07:34:59.589483+0000","last_clean":"2024-02-05T07:34:59.589483+0000","last_became_active":"2024-02-05T07:34:59.583120+0000","last_became_peered":"2024-02-05T07:34:59.583120+0000","last_unstale":"2024-02-05T07:34:59.589483+0000","last_undegraded":"2024-02-05T07:34:59.589483+0000","last_fullsized":"2024-02-05T07:34:59.589483+0000","mapping_epoch":474,"log_start":"86'128","ondisk_log_start":"86'128","created":291,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:22.621608+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.145047+0000","last_clean_scrub_stamp":"2024-02-05T07:32:22.621608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,1],"avail_no_missing":["4","1","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"365'12","reported_seq":917,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.972279+0000","last_change":"2024-02-05T07:34:40.283207+0000","last_active":"2024-02-05T07:34:57.972279+0000","last_peered":"2024-02-05T07:34:57.972279+0000","last_clean":"2024-02-05T07:34:57.972279+0000","last_became_active":"2024-02-05T07:30:40.500113+0000","last_became_peered":"2024-02-05T07:30:40.500113+0000","last_unstale":"2024-02-05T07:34:57.972279+0000","last_undegraded":"2024-02-05T07:34:57.972279+0000","last_fullsized":"2024-02-05T07:34:57.972279+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'12","last_scrub_stamp":"2024-02-05T07:34:40.283087+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:34:40.283087+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:40.434853+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"326'27","reported_seq":493,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.973036+0000","last_change":"2024-02-05T07:32:29.833732+0000","last_active":"2024-02-05T07:34:57.973036+0000","last_peered":"2024-02-05T07:34:57.973036+0000","last_clean":"2024-02-05T07:34:57.973036+0000","last_became_active":"2024-02-05T07:30:40.497982+0000","last_became_peered":"2024-02-05T07:30:40.497982+0000","last_unstale":"2024-02-05T07:34:57.973036+0000","last_undegraded":"2024-02-05T07:34:57.973036+0000","last_fullsized":"2024-02-05T07:34:57.973036+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:32:29.833636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:29.833636+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":256,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.560839+0000","last_change":"2024-02-05T07:33:27.649686+0000","last_active":"2024-02-05T07:34:59.560839+0000","last_peered":"2024-02-05T07:34:59.560839+0000","last_clean":"2024-02-05T07:34:59.560839+0000","last_became_active":"2024-02-05T07:30:40.503903+0000","last_became_peered":"2024-02-05T07:30:40.503903+0000","last_unstale":"2024-02-05T07:34:59.560839+0000","last_undegraded":"2024-02-05T07:34:59.560839+0000","last_fullsized":"2024-02-05T07:34:59.560839+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:33:27.649596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:27.649596+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.048113+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":529,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.973821+0000","last_change":"2024-02-05T07:33:35.177937+0000","last_active":"2024-02-05T07:34:57.973821+0000","last_peered":"2024-02-05T07:34:57.973821+0000","last_clean":"2024-02-05T07:34:57.973821+0000","last_became_active":"2024-02-05T07:28:26.589671+0000","last_became_peered":"2024-02-05T07:28:26.589671+0000","last_unstale":"2024-02-05T07:34:57.973821+0000","last_undegraded":"2024-02-05T07:34:57.973821+0000","last_fullsized":"2024-02-05T07:34:57.973821+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:35.177804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:33:35.177804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":718,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-02-05T07:34:54.935833+0000","last_change":"2024-02-05T07:33:44.492558+0000","last_active":"2024-02-05T07:34:54.935833+0000","last_peered":"2024-02-05T07:34:54.935833+0000","last_clean":"2024-02-05T07:34:54.935833+0000","last_became_active":"2024-02-05T07:30:43.523108+0000","last_became_peered":"2024-02-05T07:30:43.523108+0000","last_unstale":"2024-02-05T07:34:54.935833+0000","last_undegraded":"2024-02-05T07:34:54.935833+0000","last_fullsized":"2024-02-05T07:34:54.935833+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:44.492478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.492478+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:08.262980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"326'12","reported_seq":479,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.584921+0000","last_change":"2024-02-05T07:33:41.762486+0000","last_active":"2024-02-05T07:35:01.584921+0000","last_peered":"2024-02-05T07:35:01.584921+0000","last_clean":"2024-02-05T07:35:01.584921+0000","last_became_active":"2024-02-05T07:30:40.496980+0000","last_became_peered":"2024-02-05T07:30:40.496980+0000","last_unstale":"2024-02-05T07:35:01.584921+0000","last_undegraded":"2024-02-05T07:35:01.584921+0000","last_fullsized":"2024-02-05T07:35:01.584921+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'12","last_scrub_stamp":"2024-02-05T07:33:41.762404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.762404+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":672,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.973426+0000","last_change":"2024-02-05T07:33:51.105371+0000","last_active":"2024-02-05T07:34:57.973426+0000","last_peered":"2024-02-05T07:34:57.973426+0000","last_clean":"2024-02-05T07:34:57.973426+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:34:57.973426+0000","last_undegraded":"2024-02-05T07:34:57.973426+0000","last_fullsized":"2024-02-05T07:34:57.973426+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:33:51.105251+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:33:51.105251+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:18.353438+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":451,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.574688+0000","last_change":"2024-02-05T07:30:27.965535+0000","last_active":"2024-02-05T07:35:01.574688+0000","last_peered":"2024-02-05T07:35:01.574688+0000","last_clean":"2024-02-05T07:35:01.574688+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:35:01.574688+0000","last_undegraded":"2024-02-05T07:35:01.574688+0000","last_fullsized":"2024-02-05T07:35:01.574688+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:27.965449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:27.965449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.065304+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":722,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.976113+0000","last_change":"2024-02-05T07:32:32.844370+0000","last_active":"2024-02-05T07:34:57.976113+0000","last_peered":"2024-02-05T07:34:57.976113+0000","last_clean":"2024-02-05T07:34:57.976113+0000","last_became_active":"2024-02-05T07:30:40.509216+0000","last_became_peered":"2024-02-05T07:30:40.509216+0000","last_unstale":"2024-02-05T07:34:57.976113+0000","last_undegraded":"2024-02-05T07:34:57.976113+0000","last_fullsized":"2024-02-05T07:34:57.976113+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:32.844264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:32:32.844264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":798,"reported_epoch":475,"state":"active+undersized+remapped","last_fresh":"2024-02-05T07:34:59.594124+0000","last_change":"2024-02-05T07:34:59.594029+0000","last_active":"2024-02-05T07:34:59.594124+0000","last_peered":"2024-02-05T07:34:59.594124+0000","last_clean":"2024-02-05T07:34:28.619743+0000","last_became_active":"2024-02-05T07:34:59.594029+0000","last_became_peered":"2024-02-05T07:34:59.594029+0000","last_unstale":"2024-02-05T07:34:59.594124+0000","last_undegraded":"2024-02-05T07:34:59.594124+0000","last_fullsized":"2024-02-05T07:34:59.567104+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:30.927865+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:28:45.702565+0000","last_clean_scrub_stamp":"2024-02-05T07:34:30.927865+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[1,4],"avail_no_missing":["1","3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"325'15","reported_seq":483,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.574936+0000","last_change":"2024-02-05T07:33:45.711524+0000","last_active":"2024-02-05T07:35:01.574936+0000","last_peered":"2024-02-05T07:35:01.574936+0000","last_clean":"2024-02-05T07:35:01.574936+0000","last_became_active":"2024-02-05T07:30:40.485837+0000","last_became_peered":"2024-02-05T07:30:40.485837+0000","last_unstale":"2024-02-05T07:35:01.574936+0000","last_undegraded":"2024-02-05T07:35:01.574936+0000","last_fullsized":"2024-02-05T07:35:01.574936+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:33:45.711399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:45.711399+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":739,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.560299+0000","last_change":"2024-02-05T07:30:40.498263+0000","last_active":"2024-02-05T07:34:59.560299+0000","last_peered":"2024-02-05T07:34:59.560299+0000","last_clean":"2024-02-05T07:34:59.560299+0000","last_became_active":"2024-02-05T07:30:40.497956+0000","last_became_peered":"2024-02-05T07:30:40.497956+0000","last_unstale":"2024-02-05T07:34:59.560299+0000","last_undegraded":"2024-02-05T07:34:59.560299+0000","last_fullsized":"2024-02-05T07:34:59.560299+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:33.183427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:30:33.183427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951273+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":454,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.585014+0000","last_change":"2024-02-05T07:33:50.723381+0000","last_active":"2024-02-05T07:35:01.585014+0000","last_peered":"2024-02-05T07:35:01.585014+0000","last_clean":"2024-02-05T07:35:01.585014+0000","last_became_active":"2024-02-05T07:30:40.496076+0000","last_became_peered":"2024-02-05T07:30:40.496076+0000","last_unstale":"2024-02-05T07:35:01.585014+0000","last_undegraded":"2024-02-05T07:35:01.585014+0000","last_fullsized":"2024-02-05T07:35:01.585014+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_clean_scrub_stamp":"2024-02-05T07:33:50.723212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:09.045969+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":864,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.571828+0000","last_change":"2024-02-05T07:32:48.164374+0000","last_active":"2024-02-05T07:34:59.571828+0000","last_peered":"2024-02-05T07:34:59.571828+0000","last_clean":"2024-02-05T07:34:59.571828+0000","last_became_active":"2024-02-05T07:30:40.487455+0000","last_became_peered":"2024-02-05T07:30:40.487455+0000","last_unstale":"2024-02-05T07:34:59.571828+0000","last_undegraded":"2024-02-05T07:34:59.571828+0000","last_fullsized":"2024-02-05T07:34:59.571828+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:48.164259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:32:48.164259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":570,"reported_epoch":472,"state":"peering","last_fresh":"2024-02-05T07:34:56.956997+0000","last_change":"2024-02-05T07:34:57.970674+0000","last_active":"2024-02-05T07:34:57.970674+0000","last_peered":"2024-02-05T07:34:56.956997+0000","last_clean":"2024-02-05T07:34:56.956997+0000","last_became_active":"2024-02-05T07:30:43.466854+0000","last_became_peered":"2024-02-05T07:30:43.466854+0000","last_unstale":"2024-02-05T07:34:56.956997+0000","last_undegraded":"2024-02-05T07:34:56.956997+0000","last_fullsized":"2024-02-05T07:34:56.956997+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:28.166398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.166398+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"326'19","reported_seq":802,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.574968+0000","last_change":"2024-02-05T07:30:40.503823+0000","last_active":"2024-02-05T07:35:01.574968+0000","last_peered":"2024-02-05T07:35:01.574968+0000","last_clean":"2024-02-05T07:35:01.574968+0000","last_became_active":"2024-02-05T07:30:40.483700+0000","last_became_peered":"2024-02-05T07:30:40.483700+0000","last_unstale":"2024-02-05T07:35:01.574968+0000","last_undegraded":"2024-02-05T07:35:01.574968+0000","last_fullsized":"2024-02-05T07:35:01.574968+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:30:35.138361+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:30:35.138361+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.065335+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":616,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.573633+0000","last_change":"2024-02-05T07:30:40.476394+0000","last_active":"2024-02-05T07:35:01.573633+0000","last_peered":"2024-02-05T07:35:01.573633+0000","last_clean":"2024-02-05T07:35:01.573633+0000","last_became_active":"2024-02-05T07:30:40.476058+0000","last_became_peered":"2024-02-05T07:30:40.476058+0000","last_unstale":"2024-02-05T07:35:01.573633+0000","last_undegraded":"2024-02-05T07:35:01.573633+0000","last_fullsized":"2024-02-05T07:35:01.573633+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:49.314735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:49.314735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.065159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"365'11","reported_seq":555,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.575855+0000","last_change":"2024-02-05T07:33:39.734203+0000","last_active":"2024-02-05T07:35:01.575855+0000","last_peered":"2024-02-05T07:35:01.575855+0000","last_clean":"2024-02-05T07:35:01.575855+0000","last_became_active":"2024-02-05T07:30:40.495708+0000","last_became_peered":"2024-02-05T07:30:40.495708+0000","last_unstale":"2024-02-05T07:35:01.575855+0000","last_undegraded":"2024-02-05T07:35:01.575855+0000","last_fullsized":"2024-02-05T07:35:01.575855+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:33:39.734129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:39.734129+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":856,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.561477+0000","last_change":"2024-02-05T07:33:38.879276+0000","last_active":"2024-02-05T07:34:59.561477+0000","last_peered":"2024-02-05T07:34:59.561477+0000","last_clean":"2024-02-05T07:34:59.561477+0000","last_became_active":"2024-02-05T07:27:03.364334+0000","last_became_peered":"2024-02-05T07:27:03.364334+0000","last_unstale":"2024-02-05T07:34:59.561477+0000","last_undegraded":"2024-02-05T07:34:59.561477+0000","last_fullsized":"2024-02-05T07:34:59.561477+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:38.879177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:38.879177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:04.048224+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":712,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T07:34:55.954519+0000","last_change":"2024-02-05T07:34:35.914394+0000","last_active":"2024-02-05T07:34:55.954519+0000","last_peered":"2024-02-05T07:34:55.954519+0000","last_clean":"2024-02-05T07:34:55.954519+0000","last_became_active":"2024-02-05T07:28:19.302393+0000","last_became_peered":"2024-02-05T07:28:19.302393+0000","last_unstale":"2024-02-05T07:34:55.954519+0000","last_undegraded":"2024-02-05T07:34:55.954519+0000","last_fullsized":"2024-02-05T07:34:55.954519+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:35.914275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:35.914275+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:49.046072+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"326'25","reported_seq":493,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.575966+0000","last_change":"2024-02-05T07:30:40.512630+0000","last_active":"2024-02-05T07:35:01.575966+0000","last_peered":"2024-02-05T07:35:01.575966+0000","last_clean":"2024-02-05T07:35:01.575966+0000","last_became_active":"2024-02-05T07:30:40.486394+0000","last_became_peered":"2024-02-05T07:30:40.486394+0000","last_unstale":"2024-02-05T07:35:01.575966+0000","last_undegraded":"2024-02-05T07:35:01.575966+0000","last_fullsized":"2024-02-05T07:35:01.575966+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:30:33.959124+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:30:33.959124+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.065191+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":241,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.560710+0000","last_change":"2024-02-05T07:34:32.854993+0000","last_active":"2024-02-05T07:34:59.560710+0000","last_peered":"2024-02-05T07:34:59.560710+0000","last_clean":"2024-02-05T07:34:59.560710+0000","last_became_active":"2024-02-05T07:30:40.505600+0000","last_became_peered":"2024-02-05T07:30:40.505600+0000","last_unstale":"2024-02-05T07:34:59.560710+0000","last_undegraded":"2024-02-05T07:34:59.560710+0000","last_fullsized":"2024-02-05T07:34:59.560710+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:32.854871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:32.854871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:59.631282+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"217'9","reported_seq":685,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.572790+0000","last_change":"2024-02-05T07:34:27.947088+0000","last_active":"2024-02-05T07:34:59.572790+0000","last_peered":"2024-02-05T07:34:59.572790+0000","last_clean":"2024-02-05T07:34:59.572790+0000","last_became_active":"2024-02-05T07:25:27.246616+0000","last_became_peered":"2024-02-05T07:25:27.246616+0000","last_unstale":"2024-02-05T07:34:59.572790+0000","last_undegraded":"2024-02-05T07:34:59.572790+0000","last_fullsized":"2024-02-05T07:34:59.572790+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9","last_scrub_stamp":"2024-02-05T07:34:27.946504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:34:27.946504+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:43.736690+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":612,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.574832+0000","last_change":"2024-02-05T07:34:22.810896+0000","last_active":"2024-02-05T07:35:01.574832+0000","last_peered":"2024-02-05T07:35:01.574832+0000","last_clean":"2024-02-05T07:35:01.574832+0000","last_became_active":"2024-02-05T07:30:40.507831+0000","last_became_peered":"2024-02-05T07:30:40.507831+0000","last_unstale":"2024-02-05T07:35:01.574832+0000","last_undegraded":"2024-02-05T07:35:01.574832+0000","last_fullsized":"2024-02-05T07:35:01.574832+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:22.810759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:22.810759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:48.571309+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"326'19","reported_seq":916,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.560856+0000","last_change":"2024-02-05T07:32:31.667966+0000","last_active":"2024-02-05T07:34:59.560856+0000","last_peered":"2024-02-05T07:34:59.560856+0000","last_clean":"2024-02-05T07:34:59.560856+0000","last_became_active":"2024-02-05T07:28:25.779886+0000","last_became_peered":"2024-02-05T07:28:25.779886+0000","last_unstale":"2024-02-05T07:34:59.560856+0000","last_undegraded":"2024-02-05T07:34:59.560856+0000","last_fullsized":"2024-02-05T07:34:59.560856+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:32:31.667882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:31.667882+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951400+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":877,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.565970+0000","last_change":"2024-02-05T07:34:31.893068+0000","last_active":"2024-02-05T07:34:59.565970+0000","last_peered":"2024-02-05T07:34:59.565970+0000","last_clean":"2024-02-05T07:34:59.565970+0000","last_became_active":"2024-02-05T07:30:40.510301+0000","last_became_peered":"2024-02-05T07:30:40.510301+0000","last_unstale":"2024-02-05T07:34:59.565970+0000","last_undegraded":"2024-02-05T07:34:59.565970+0000","last_fullsized":"2024-02-05T07:34:59.565970+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:31.892895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:34:31.892895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:37.814244+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":877,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.571707+0000","last_change":"2024-02-05T07:32:48.346195+0000","last_active":"2024-02-05T07:34:59.571707+0000","last_peered":"2024-02-05T07:34:59.571707+0000","last_clean":"2024-02-05T07:34:59.571707+0000","last_became_active":"2024-02-05T07:30:40.487141+0000","last_became_peered":"2024-02-05T07:30:40.487141+0000","last_unstale":"2024-02-05T07:34:59.571707+0000","last_undegraded":"2024-02-05T07:34:59.571707+0000","last_fullsized":"2024-02-05T07:34:59.571707+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_clean_scrub_stamp":"2024-02-05T07:32:48.345965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":446,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.574280+0000","last_change":"2024-02-05T07:33:37.735878+0000","last_active":"2024-02-05T07:35:01.574280+0000","last_peered":"2024-02-05T07:35:01.574280+0000","last_clean":"2024-02-05T07:35:01.574280+0000","last_became_active":"2024-02-05T07:27:17.705910+0000","last_became_peered":"2024-02-05T07:27:17.705910+0000","last_unstale":"2024-02-05T07:35:01.574280+0000","last_undegraded":"2024-02-05T07:35:01.574280+0000","last_fullsized":"2024-02-05T07:35:01.574280+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.735757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.735757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.065479+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"325'15","reported_seq":483,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.567224+0000","last_change":"2024-02-05T07:29:57.775013+0000","last_active":"2024-02-05T07:34:59.567224+0000","last_peered":"2024-02-05T07:34:59.567224+0000","last_clean":"2024-02-05T07:34:59.567224+0000","last_became_active":"2024-02-05T07:28:26.589705+0000","last_became_peered":"2024-02-05T07:28:26.589705+0000","last_unstale":"2024-02-05T07:34:59.567224+0000","last_undegraded":"2024-02-05T07:34:59.567224+0000","last_fullsized":"2024-02-05T07:34:59.567224+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:29:57.774913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:57.774913+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":661,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.574864+0000","last_change":"2024-02-05T07:30:40.489183+0000","last_active":"2024-02-05T07:35:01.574864+0000","last_peered":"2024-02-05T07:35:01.574864+0000","last_clean":"2024-02-05T07:35:01.574864+0000","last_became_active":"2024-02-05T07:30:40.488789+0000","last_became_peered":"2024-02-05T07:30:40.488789+0000","last_unstale":"2024-02-05T07:35:01.574864+0000","last_undegraded":"2024-02-05T07:35:01.574864+0000","last_fullsized":"2024-02-05T07:35:01.574864+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.220841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.220841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.065132+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":448,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.573950+0000","last_change":"2024-02-05T07:33:47.684589+0000","last_active":"2024-02-05T07:35:01.573950+0000","last_peered":"2024-02-05T07:35:01.573950+0000","last_clean":"2024-02-05T07:35:01.573950+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:35:01.573950+0000","last_undegraded":"2024-02-05T07:35:01.573950+0000","last_fullsized":"2024-02-05T07:35:01.573950+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:47.684508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.684508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:12.589901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":318,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.971624+0000","last_change":"2024-02-05T07:34:30.670504+0000","last_active":"2024-02-05T07:34:57.971624+0000","last_peered":"2024-02-05T07:34:57.971624+0000","last_clean":"2024-02-05T07:34:57.971624+0000","last_became_active":"2024-02-05T07:34:30.670179+0000","last_became_peered":"2024-02-05T07:34:30.670179+0000","last_unstale":"2024-02-05T07:34:57.971624+0000","last_undegraded":"2024-02-05T07:34:57.971624+0000","last_fullsized":"2024-02-05T07:34:57.971624+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:26.419332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:33:26.419332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":884,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.590897+0000","last_change":"2024-02-05T07:35:01.577675+0000","last_active":"2024-02-05T07:35:01.590897+0000","last_peered":"2024-02-05T07:35:01.590897+0000","last_clean":"2024-02-05T07:35:01.590897+0000","last_became_active":"2024-02-05T07:35:01.577001+0000","last_became_peered":"2024-02-05T07:35:01.577001+0000","last_unstale":"2024-02-05T07:35:01.590897+0000","last_undegraded":"2024-02-05T07:35:01.590897+0000","last_fullsized":"2024-02-05T07:35:01.590897+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:30:36.995330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.995330+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"326'19","reported_seq":493,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.574383+0000","last_change":"2024-02-05T07:30:40.906805+0000","last_active":"2024-02-05T07:35:01.574383+0000","last_peered":"2024-02-05T07:35:01.574383+0000","last_clean":"2024-02-05T07:35:01.574383+0000","last_became_active":"2024-02-05T07:30:40.501893+0000","last_became_peered":"2024-02-05T07:30:40.501893+0000","last_unstale":"2024-02-05T07:35:01.574383+0000","last_undegraded":"2024-02-05T07:35:01.574383+0000","last_fullsized":"2024-02-05T07:35:01.574383+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:30:40.906693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.906693+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.065249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":258,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.571561+0000","last_change":"2024-02-05T07:30:40.492005+0000","last_active":"2024-02-05T07:34:59.571561+0000","last_peered":"2024-02-05T07:34:59.571561+0000","last_clean":"2024-02-05T07:34:59.571561+0000","last_became_active":"2024-02-05T07:30:40.491686+0000","last_became_peered":"2024-02-05T07:30:40.491686+0000","last_unstale":"2024-02-05T07:34:59.571561+0000","last_undegraded":"2024-02-05T07:34:59.571561+0000","last_fullsized":"2024-02-05T07:34:59.571561+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.097532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.097532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":731,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.563287+0000","last_change":"2024-02-05T07:30:40.497509+0000","last_active":"2024-02-05T07:34:59.563287+0000","last_peered":"2024-02-05T07:34:59.563287+0000","last_clean":"2024-02-05T07:34:59.563287+0000","last_became_active":"2024-02-05T07:30:40.497190+0000","last_became_peered":"2024-02-05T07:30:40.497190+0000","last_unstale":"2024-02-05T07:34:59.563287+0000","last_undegraded":"2024-02-05T07:34:59.563287+0000","last_fullsized":"2024-02-05T07:34:59.563287+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:29:58.191815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":452,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.573686+0000","last_change":"2024-02-05T07:29:40.983327+0000","last_active":"2024-02-05T07:35:01.573686+0000","last_peered":"2024-02-05T07:35:01.573686+0000","last_clean":"2024-02-05T07:35:01.573686+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:35:01.573686+0000","last_undegraded":"2024-02-05T07:35:01.573686+0000","last_fullsized":"2024-02-05T07:35:01.573686+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:40.983227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.983227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.064890+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"325'8","reported_seq":464,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.569281+0000","last_change":"2024-02-05T07:29:53.958469+0000","last_active":"2024-02-05T07:35:01.569281+0000","last_peered":"2024-02-05T07:35:01.569281+0000","last_clean":"2024-02-05T07:35:01.569281+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:35:01.569281+0000","last_undegraded":"2024-02-05T07:35:01.569281+0000","last_fullsized":"2024-02-05T07:35:01.569281+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:29:53.958384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.958384+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.065104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":459,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.584660+0000","last_change":"2024-02-05T07:34:39.980293+0000","last_active":"2024-02-05T07:35:01.584660+0000","last_peered":"2024-02-05T07:35:01.584660+0000","last_clean":"2024-02-05T07:35:01.584660+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:35:01.584660+0000","last_undegraded":"2024-02-05T07:35:01.584660+0000","last_fullsized":"2024-02-05T07:35:01.584660+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:39.980067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:34:39.980067+0000","last_clean_scrub_stamp":"2024-02-05T07:34:39.980067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:03.081159+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":706,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.972250+0000","last_change":"2024-02-05T07:33:30.216853+0000","last_active":"2024-02-05T07:34:57.972250+0000","last_peered":"2024-02-05T07:34:57.972250+0000","last_clean":"2024-02-05T07:34:57.972250+0000","last_became_active":"2024-02-05T07:30:40.500403+0000","last_became_peered":"2024-02-05T07:30:40.500403+0000","last_unstale":"2024-02-05T07:34:57.972250+0000","last_undegraded":"2024-02-05T07:34:57.972250+0000","last_fullsized":"2024-02-05T07:34:57.972250+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:30.216745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.216745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":242,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.571410+0000","last_change":"2024-02-05T07:30:45.926725+0000","last_active":"2024-02-05T07:34:59.571410+0000","last_peered":"2024-02-05T07:34:59.571410+0000","last_clean":"2024-02-05T07:34:59.571410+0000","last_became_active":"2024-02-05T07:30:40.489230+0000","last_became_peered":"2024-02-05T07:30:40.489230+0000","last_unstale":"2024-02-05T07:34:59.571410+0000","last_undegraded":"2024-02-05T07:34:59.571410+0000","last_fullsized":"2024-02-05T07:34:59.571410+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:45.926395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:30:45.926395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"326'19","reported_seq":693,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.977657+0000","last_change":"2024-02-05T07:32:43.787447+0000","last_active":"2024-02-05T07:34:57.977657+0000","last_peered":"2024-02-05T07:34:57.977657+0000","last_clean":"2024-02-05T07:34:57.977657+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:34:57.977657+0000","last_undegraded":"2024-02-05T07:34:57.977657+0000","last_fullsized":"2024-02-05T07:34:57.977657+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:32:43.787327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:43.787327+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":448,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.568950+0000","last_change":"2024-02-05T07:29:38.000417+0000","last_active":"2024-02-05T07:35:01.568950+0000","last_peered":"2024-02-05T07:35:01.568950+0000","last_clean":"2024-02-05T07:35:01.568950+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:35:01.568950+0000","last_undegraded":"2024-02-05T07:35:01.568950+0000","last_fullsized":"2024-02-05T07:35:01.568950+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:38.000345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:38.000345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.064795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":236,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.572820+0000","last_change":"2024-02-05T07:30:40.480962+0000","last_active":"2024-02-05T07:34:59.572820+0000","last_peered":"2024-02-05T07:34:59.572820+0000","last_clean":"2024-02-05T07:34:59.572820+0000","last_became_active":"2024-02-05T07:30:40.479162+0000","last_became_peered":"2024-02-05T07:30:40.479162+0000","last_unstale":"2024-02-05T07:34:59.572820+0000","last_undegraded":"2024-02-05T07:34:59.572820+0000","last_fullsized":"2024-02-05T07:34:59.572820+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:28.153881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:28.153881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":475,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.565812+0000","last_change":"2024-02-05T07:34:30.662610+0000","last_active":"2024-02-05T07:34:59.565812+0000","last_peered":"2024-02-05T07:34:59.565812+0000","last_clean":"2024-02-05T07:34:59.565812+0000","last_became_active":"2024-02-05T07:34:30.662349+0000","last_became_peered":"2024-02-05T07:34:30.662349+0000","last_unstale":"2024-02-05T07:34:59.565812+0000","last_undegraded":"2024-02-05T07:34:59.565812+0000","last_fullsized":"2024-02-05T07:34:59.565812+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:55.800824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:55.800824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951169+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"326'20","reported_seq":615,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.574589+0000","last_change":"2024-02-05T07:30:40.534624+0000","last_active":"2024-02-05T07:35:01.574589+0000","last_peered":"2024-02-05T07:35:01.574589+0000","last_clean":"2024-02-05T07:35:01.574589+0000","last_became_active":"2024-02-05T07:30:40.495954+0000","last_became_peered":"2024-02-05T07:30:40.495954+0000","last_unstale":"2024-02-05T07:35:01.574589+0000","last_undegraded":"2024-02-05T07:35:01.574589+0000","last_fullsized":"2024-02-05T07:35:01.574589+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:29:40.154854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:29:40.154854+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.064861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":915,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568843+0000","last_change":"2024-02-05T07:34:38.558839+0000","last_active":"2024-02-05T07:34:59.568843+0000","last_peered":"2024-02-05T07:34:59.568843+0000","last_clean":"2024-02-05T07:34:59.568843+0000","last_became_active":"2024-02-05T07:30:40.495135+0000","last_became_peered":"2024-02-05T07:30:40.495135+0000","last_unstale":"2024-02-05T07:34:59.568843+0000","last_undegraded":"2024-02-05T07:34:59.568843+0000","last_fullsized":"2024-02-05T07:34:59.568843+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:34:38.558593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.558593+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:58.060722+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"365'11","reported_seq":883,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-05T07:34:57.977787+0000","last_change":"2024-02-05T07:33:31.254022+0000","last_active":"2024-02-05T07:34:57.977787+0000","last_peered":"2024-02-05T07:34:57.977787+0000","last_clean":"2024-02-05T07:34:57.977787+0000","last_became_active":"2024-02-05T07:30:40.497371+0000","last_became_peered":"2024-02-05T07:30:40.497371+0000","last_unstale":"2024-02-05T07:34:57.977787+0000","last_undegraded":"2024-02-05T07:34:57.977787+0000","last_fullsized":"2024-02-05T07:34:57.977787+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:33:31.253933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:31.253933+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":352,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.570716+0000","last_change":"2024-02-05T07:30:41.950393+0000","last_active":"2024-02-05T07:34:59.570716+0000","last_peered":"2024-02-05T07:34:59.570716+0000","last_clean":"2024-02-05T07:34:59.570716+0000","last_became_active":"2024-02-05T07:30:40.486920+0000","last_became_peered":"2024-02-05T07:30:40.486920+0000","last_unstale":"2024-02-05T07:34:59.570716+0000","last_undegraded":"2024-02-05T07:34:59.570716+0000","last_fullsized":"2024-02-05T07:34:59.570716+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:30:41.950253+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:30:41.950253+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"326'23","reported_seq":482,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.574622+0000","last_change":"2024-02-05T07:34:41.935675+0000","last_active":"2024-02-05T07:35:01.574622+0000","last_peered":"2024-02-05T07:35:01.574622+0000","last_clean":"2024-02-05T07:35:01.574622+0000","last_became_active":"2024-02-05T07:27:16.747998+0000","last_became_peered":"2024-02-05T07:27:16.747998+0000","last_unstale":"2024-02-05T07:35:01.574622+0000","last_undegraded":"2024-02-05T07:35:01.574622+0000","last_fullsized":"2024-02-05T07:35:01.574622+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'23","last_scrub_stamp":"2024-02-05T07:34:41.934886+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:34:41.934886+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:07.837680+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":71,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.571170+0000","last_change":"2024-02-05T07:34:30.686108+0000","last_active":"2024-02-05T07:34:59.571170+0000","last_peered":"2024-02-05T07:34:59.571170+0000","last_clean":"2024-02-05T07:34:59.571170+0000","last_became_active":"2024-02-05T07:34:30.685805+0000","last_became_peered":"2024-02-05T07:34:30.685805+0000","last_unstale":"2024-02-05T07:34:59.571170+0000","last_undegraded":"2024-02-05T07:34:59.571170+0000","last_fullsized":"2024-02-05T07:34:59.571170+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.874081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":869,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.567103+0000","last_change":"2024-02-05T07:30:26.669849+0000","last_active":"2024-02-05T07:34:59.567103+0000","last_peered":"2024-02-05T07:34:59.567103+0000","last_clean":"2024-02-05T07:34:59.567103+0000","last_became_active":"2024-02-05T07:27:16.706777+0000","last_became_peered":"2024-02-05T07:27:16.706777+0000","last_unstale":"2024-02-05T07:34:59.567103+0000","last_undegraded":"2024-02-05T07:34:59.567103+0000","last_fullsized":"2024-02-05T07:34:59.567103+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:26.669576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:26.669576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951246+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":459,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.584824+0000","last_change":"2024-02-05T07:34:37.126181+0000","last_active":"2024-02-05T07:35:01.584824+0000","last_peered":"2024-02-05T07:35:01.584824+0000","last_clean":"2024-02-05T07:35:01.584824+0000","last_became_active":"2024-02-05T07:28:26.603067+0000","last_became_peered":"2024-02-05T07:28:26.603067+0000","last_unstale":"2024-02-05T07:35:01.584824+0000","last_undegraded":"2024-02-05T07:35:01.584824+0000","last_fullsized":"2024-02-05T07:35:01.584824+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:37.125938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:37.125938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:58.352547+0000","scrub_duration":217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"326'27","reported_seq":953,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.570929+0000","last_change":"2024-02-05T07:33:33.995153+0000","last_active":"2024-02-05T07:34:59.570929+0000","last_peered":"2024-02-05T07:34:59.570929+0000","last_clean":"2024-02-05T07:34:59.570929+0000","last_became_active":"2024-02-05T07:27:16.736296+0000","last_became_peered":"2024-02-05T07:27:16.736296+0000","last_unstale":"2024-02-05T07:34:59.570929+0000","last_undegraded":"2024-02-05T07:34:59.570929+0000","last_fullsized":"2024-02-05T07:34:59.570929+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:33:33.995051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.995051+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":614,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.560625+0000","last_change":"2024-02-05T07:30:40.503947+0000","last_active":"2024-02-05T07:34:59.560625+0000","last_peered":"2024-02-05T07:34:59.560625+0000","last_clean":"2024-02-05T07:34:59.560625+0000","last_became_active":"2024-02-05T07:30:40.503627+0000","last_became_peered":"2024-02-05T07:30:40.503627+0000","last_unstale":"2024-02-05T07:34:59.560625+0000","last_undegraded":"2024-02-05T07:34:59.560625+0000","last_fullsized":"2024-02-05T07:34:59.560625+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:34.663987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:34.663987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":855,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568502+0000","last_change":"2024-02-05T07:30:40.884273+0000","last_active":"2024-02-05T07:34:59.568502+0000","last_peered":"2024-02-05T07:34:59.568502+0000","last_clean":"2024-02-05T07:34:59.568502+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:34:59.568502+0000","last_undegraded":"2024-02-05T07:34:59.568502+0000","last_fullsized":"2024-02-05T07:34:59.568502+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:40.884197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.884197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":869,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.561907+0000","last_change":"2024-02-05T07:29:59.002788+0000","last_active":"2024-02-05T07:34:59.561907+0000","last_peered":"2024-02-05T07:34:59.561907+0000","last_clean":"2024-02-05T07:34:59.561907+0000","last_became_active":"2024-02-05T07:28:26.590448+0000","last_became_peered":"2024-02-05T07:28:26.590448+0000","last_unstale":"2024-02-05T07:34:59.561907+0000","last_undegraded":"2024-02-05T07:34:59.561907+0000","last_fullsized":"2024-02-05T07:34:59.561907+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:59.002677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:59.002677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"325'7","reported_seq":895,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.560512+0000","last_change":"2024-02-05T07:32:40.597579+0000","last_active":"2024-02-05T07:34:59.560512+0000","last_peered":"2024-02-05T07:34:59.560512+0000","last_clean":"2024-02-05T07:34:59.560512+0000","last_became_active":"2024-02-05T07:27:16.745897+0000","last_became_peered":"2024-02-05T07:27:16.745897+0000","last_unstale":"2024-02-05T07:34:59.560512+0000","last_undegraded":"2024-02-05T07:34:59.560512+0000","last_fullsized":"2024-02-05T07:34:59.560512+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'7","last_scrub_stamp":"2024-02-05T07:32:40.597479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:40.597479+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":686,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.561799+0000","last_change":"2024-02-05T07:30:39.726603+0000","last_active":"2024-02-05T07:34:59.561799+0000","last_peered":"2024-02-05T07:34:59.561799+0000","last_clean":"2024-02-05T07:34:59.561799+0000","last_became_active":"2024-02-05T07:28:26.590322+0000","last_became_peered":"2024-02-05T07:28:26.590322+0000","last_unstale":"2024-02-05T07:34:59.561799+0000","last_undegraded":"2024-02-05T07:34:59.561799+0000","last_fullsized":"2024-02-05T07:34:59.561799+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.726525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.726525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951338+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":241,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.563119+0000","last_change":"2024-02-05T07:33:44.867890+0000","last_active":"2024-02-05T07:34:59.563119+0000","last_peered":"2024-02-05T07:34:59.563119+0000","last_clean":"2024-02-05T07:34:59.563119+0000","last_became_active":"2024-02-05T07:30:40.504365+0000","last_became_peered":"2024-02-05T07:30:40.504365+0000","last_unstale":"2024-02-05T07:34:59.563119+0000","last_undegraded":"2024-02-05T07:34:59.563119+0000","last_fullsized":"2024-02-05T07:34:59.563119+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.867764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.867764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:01.989695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":662,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.569450+0000","last_change":"2024-02-05T07:29:50.742042+0000","last_active":"2024-02-05T07:34:59.569450+0000","last_peered":"2024-02-05T07:34:59.569450+0000","last_clean":"2024-02-05T07:34:59.569450+0000","last_became_active":"2024-02-05T07:27:16.769581+0000","last_became_peered":"2024-02-05T07:27:16.769581+0000","last_unstale":"2024-02-05T07:34:59.569450+0000","last_undegraded":"2024-02-05T07:34:59.569450+0000","last_fullsized":"2024-02-05T07:34:59.569450+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:50.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:29:50.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"325'19","reported_seq":919,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.562940+0000","last_change":"2024-02-05T07:30:00.001976+0000","last_active":"2024-02-05T07:34:59.562940+0000","last_peered":"2024-02-05T07:34:59.562940+0000","last_clean":"2024-02-05T07:34:59.562940+0000","last_became_active":"2024-02-05T07:27:16.707036+0000","last_became_peered":"2024-02-05T07:27:16.707036+0000","last_unstale":"2024-02-05T07:34:59.562940+0000","last_undegraded":"2024-02-05T07:34:59.562940+0000","last_fullsized":"2024-02-05T07:34:59.562940+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'19","last_scrub_stamp":"2024-02-05T07:30:00.001863+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:30:00.001863+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951100+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":689,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.566106+0000","last_change":"2024-02-05T07:29:53.022701+0000","last_active":"2024-02-05T07:34:59.566106+0000","last_peered":"2024-02-05T07:34:59.566106+0000","last_clean":"2024-02-05T07:34:59.566106+0000","last_became_active":"2024-02-05T07:28:26.616107+0000","last_became_peered":"2024-02-05T07:28:26.616107+0000","last_unstale":"2024-02-05T07:34:59.566106+0000","last_undegraded":"2024-02-05T07:34:59.566106+0000","last_fullsized":"2024-02-05T07:34:59.566106+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:53.022606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:29:53.022606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951137+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":235,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.560351+0000","last_change":"2024-02-05T07:30:40.491944+0000","last_active":"2024-02-05T07:34:59.560351+0000","last_peered":"2024-02-05T07:34:59.560351+0000","last_clean":"2024-02-05T07:34:59.560351+0000","last_became_active":"2024-02-05T07:30:40.491630+0000","last_became_peered":"2024-02-05T07:30:40.491630+0000","last_unstale":"2024-02-05T07:34:59.560351+0000","last_undegraded":"2024-02-05T07:34:59.560351+0000","last_fullsized":"2024-02-05T07:34:59.560351+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:29:54.207089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:29:54.207089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":451,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.568814+0000","last_change":"2024-02-05T07:34:30.676878+0000","last_active":"2024-02-05T07:34:59.568814+0000","last_peered":"2024-02-05T07:34:59.568814+0000","last_clean":"2024-02-05T07:34:59.568814+0000","last_became_active":"2024-02-05T07:34:30.676596+0000","last_became_peered":"2024-02-05T07:34:30.676596+0000","last_unstale":"2024-02-05T07:34:59.568814+0000","last_undegraded":"2024-02-05T07:34:59.568814+0000","last_fullsized":"2024-02-05T07:34:59.568814+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:47.484894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.484894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:11.445490+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"326'27","reported_seq":482,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.574437+0000","last_change":"2024-02-05T07:33:25.674216+0000","last_active":"2024-02-05T07:35:01.574437+0000","last_peered":"2024-02-05T07:35:01.574437+0000","last_clean":"2024-02-05T07:35:01.574437+0000","last_became_active":"2024-02-05T07:27:16.751179+0000","last_became_peered":"2024-02-05T07:27:16.751179+0000","last_unstale":"2024-02-05T07:35:01.574437+0000","last_undegraded":"2024-02-05T07:35:01.574437+0000","last_fullsized":"2024-02-05T07:35:01.574437+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:33:25.674080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:25.674080+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:05.003101+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":452,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.575123+0000","last_change":"2024-02-05T07:32:49.743375+0000","last_active":"2024-02-05T07:35:01.575123+0000","last_peered":"2024-02-05T07:35:01.575123+0000","last_clean":"2024-02-05T07:35:01.575123+0000","last_became_active":"2024-02-05T07:28:26.595627+0000","last_became_peered":"2024-02-05T07:28:26.595627+0000","last_unstale":"2024-02-05T07:35:01.575123+0000","last_undegraded":"2024-02-05T07:35:01.575123+0000","last_fullsized":"2024-02-05T07:35:01.575123+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:49.743255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:32:49.743255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:02.065393+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":871,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.563191+0000","last_change":"2024-02-05T07:33:46.937819+0000","last_active":"2024-02-05T07:34:59.563191+0000","last_peered":"2024-02-05T07:34:59.563191+0000","last_clean":"2024-02-05T07:34:59.563191+0000","last_became_active":"2024-02-05T07:28:25.778935+0000","last_became_peered":"2024-02-05T07:28:25.778935+0000","last_unstale":"2024-02-05T07:34:59.563191+0000","last_undegraded":"2024-02-05T07:34:59.563191+0000","last_fullsized":"2024-02-05T07:34:59.563191+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:46.937738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.937738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":466,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-02-05T07:35:01.573847+0000","last_change":"2024-02-05T07:34:28.009081+0000","last_active":"2024-02-05T07:35:01.573847+0000","last_peered":"2024-02-05T07:35:01.573847+0000","last_clean":"2024-02-05T07:35:01.573847+0000","last_became_active":"2024-02-05T07:30:40.493086+0000","last_became_peered":"2024-02-05T07:30:40.493086+0000","last_unstale":"2024-02-05T07:35:01.573847+0000","last_undegraded":"2024-02-05T07:35:01.573847+0000","last_fullsized":"2024-02-05T07:35:01.573847+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:28.008966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.008966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:29.986309+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"450'19","reported_seq":896,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.563443+0000","last_change":"2024-02-05T07:34:34.853204+0000","last_active":"2024-02-05T07:34:59.563443+0000","last_peered":"2024-02-05T07:34:59.563443+0000","last_clean":"2024-02-05T07:34:59.563443+0000","last_became_active":"2024-02-05T07:28:25.777815+0000","last_became_peered":"2024-02-05T07:28:25.777815+0000","last_unstale":"2024-02-05T07:34:59.563443+0000","last_undegraded":"2024-02-05T07:34:59.563443+0000","last_fullsized":"2024-02-05T07:34:59.563443+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'19","last_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_deep_scrub":"450'19","last_deep_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_clean_scrub_stamp":"2024-02-05T07:34:34.853068+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:35:55.869683+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":635,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.573362+0000","last_change":"2024-02-05T07:33:44.117187+0000","last_active":"2024-02-05T07:34:59.573362+0000","last_peered":"2024-02-05T07:34:59.573362+0000","last_clean":"2024-02-05T07:34:59.573362+0000","last_became_active":"2024-02-05T07:28:26.588496+0000","last_became_peered":"2024-02-05T07:28:26.588496+0000","last_unstale":"2024-02-05T07:34:59.573362+0000","last_undegraded":"2024-02-05T07:34:59.573362+0000","last_fullsized":"2024-02-05T07:34:59.573362+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.117071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.117071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":696,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.567847+0000","last_change":"2024-02-05T07:34:30.668015+0000","last_active":"2024-02-05T07:34:59.567847+0000","last_peered":"2024-02-05T07:34:59.567847+0000","last_clean":"2024-02-05T07:34:59.567847+0000","last_became_active":"2024-02-05T07:34:30.667680+0000","last_became_peered":"2024-02-05T07:34:30.667680+0000","last_unstale":"2024-02-05T07:34:59.567847+0000","last_undegraded":"2024-02-05T07:34:59.567847+0000","last_fullsized":"2024-02-05T07:34:59.567847+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.377472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.377472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":862,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-02-05T07:34:59.562682+0000","last_change":"2024-02-05T07:32:43.608109+0000","last_active":"2024-02-05T07:34:59.562682+0000","last_peered":"2024-02-05T07:34:59.562682+0000","last_clean":"2024-02-05T07:34:59.562682+0000","last_became_active":"2024-02-05T07:30:40.490288+0000","last_became_peered":"2024-02-05T07:30:40.490288+0000","last_unstale":"2024-02-05T07:34:59.562682+0000","last_undegraded":"2024-02-05T07:34:59.562682+0000","last_fullsized":"2024-02-05T07:34:59.562682+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:43.608002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:32:43.608002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:34:59.951512+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":6967132160,"num_objects":106310,"num_object_clones":0,"num_object_copies":318930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106310,"num_write_kb":6803840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21094662144,"data_stored":21094662144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56736,"ondisk_log_size":56736,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":265,"num_read_kb":260,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1536,"ondisk_log_size":1536,"up":63,"acting":62,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":6480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":11876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":750,"ondisk_log_size":750,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":257698037959,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":301,"seq":1292785156176,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3327016,"kb_used_data":2923868,"kb_used_omap":5,"kb_used_meta":403130,"kb_avail":90418136,"statfs":{"total":95995035648,"available":92588171264,"internally_reserved":0,"allocated":2994040832,"data_stored":2993632185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":412805637},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:34:36 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.65700000000000003,"15min":1.7030000000000001},"min":{"1min":0.22800000000000001,"5min":0.192,"15min":0.192},"max":{"1min":2.391,"5min":4.4119999999999999,"15min":14.218},"last":1.01},{"interface":"front","average":{"1min":1.208,"5min":0.84299999999999997,"15min":1.7370000000000001},"min":{"1min":0.34899999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":14.175000000000001},"last":0.29199999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:34:36 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":2.133,"15min":1.8460000000000001},"min":{"1min":0.26600000000000001,"5min":0.217,"15min":0.217},"max":{"1min":2.8500000000000001,"5min":55.923000000000002,"15min":55.923000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":2.4990000000000001,"15min":2.5710000000000002},"min":{"1min":0.29499999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.6219999999999999,"5min":61.103000000000002,"15min":61.103000000000002},"last":0.34200000000000003}]},{"osd":2,"last update":"Mon Feb 5 07:34:36 2024","interfaces":[{"interface":"back","average":{"1min":3.395,"5min":1.1379999999999999,"15min":2.3679999999999999},"min":{"1min":0.318,"5min":0.249,"15min":0.249},"max":{"1min":32.103000000000002,"5min":32.103000000000002,"15min":34.689999999999998},"last":1.478},{"interface":"front","average":{"1min":3.2650000000000001,"5min":1.294,"15min":2.831},"min":{"1min":0.35599999999999998,"5min":0.245,"15min":0.245},"max":{"1min":32.149999999999999,"5min":32.149999999999999,"15min":34.646999999999998},"last":0.75900000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:34:36 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":3.0739999999999998,"15min":2.484},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.5019999999999998,"5min":55.978999999999999,"15min":55.978999999999999},"last":2.218},{"interface":"front","average":{"1min":0.89200000000000002,"5min":3.0779999999999998,"15min":2.706},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.9859999999999998,"5min":55.631,"15min":55.631},"last":0.32100000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:34:36 2024","interfaces":[{"interface":"back","average":{"1min":7.4859999999999998,"5min":1.833,"15min":1.1220000000000001},"min":{"1min":0.29299999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":151.02099999999999,"5min":151.02099999999999,"15min":151.02099999999999},"last":0.23999999999999999},{"interface":"front","average":{"1min":7.6029999999999998,"5min":1.8600000000000001,"15min":1.0580000000000001},"min":{"1min":0.29999999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":149.21799999999999,"5min":149.21799999999999,"15min":149.21799999999999},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:34:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.5,"15min":0.57599999999999996},"min":{"1min":0.34100000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.8919999999999999,"5min":2.8919999999999999,"15min":2.8919999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47799999999999998,"15min":0.55600000000000005},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.72999999999999998,"5min":0.83899999999999997,"15min":1.526},"last":0.38700000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:34:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.83199999999999996,"15min":4.0069999999999997},"min":{"1min":0.35699999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.999,"5min":30.998999999999999,"15min":84.158000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.86499999999999999,"15min":4.1360000000000001},"min":{"1min":0.37,"5min":0.312,"15min":0.312},"max":{"1min":1.0129999999999999,"5min":31.067,"15min":85.018000000000001},"last":0.48899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575077,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2828400,"kb_used_data":2121124,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":90916752,"statfs":{"total":95995035648,"available":93098754048,"internally_reserved":0,"allocated":2172030976,"data_stored":2171564630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:34:50 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.2130000000000001,"15min":2.2200000000000002},"min":{"1min":0.27700000000000002,"5min":0.20899999999999999,"15min":0.16800000000000001},"max":{"1min":5.0449999999999999,"5min":8.8819999999999997,"15min":45.393000000000001},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.9630000000000001,"5min":1.9530000000000001,"15min":2.4140000000000001},"min":{"1min":0.29299999999999998,"5min":0.26500000000000001,"15min":0.20899999999999999},"max":{"1min":5.6399999999999997,"5min":14.313000000000001,"15min":45.031999999999996},"last":0.45500000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:34:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.3979999999999999,"15min":1.806},"min":{"1min":0.254,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":2.8119999999999998,"5min":15.137,"15min":39.259999999999998},"last":1.361},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.5329999999999999,"15min":1.7230000000000001},"min":{"1min":0.23699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.9489999999999998,"5min":14.906000000000001,"15min":40.972000000000001},"last":1.282}]},{"osd":3,"last update":"Mon Feb 5 07:34:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":2.5510000000000002,"15min":2.145},"min":{"1min":0.245,"5min":0.22900000000000001,"15min":0.20999999999999999},"max":{"1min":3.5590000000000002,"5min":46.777000000000001,"15min":46.777000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.103,"5min":2.452,"15min":2.1400000000000001},"min":{"1min":0.28899999999999998,"5min":0.24299999999999999,"15min":0.24199999999999999},"max":{"1min":4.9790000000000001,"5min":50.164999999999999,"15min":50.164999999999999},"last":0.72599999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:34:50 2024","interfaces":[{"interface":"back","average":{"1min":7.3940000000000001,"5min":2.4390000000000001,"15min":2.1960000000000002},"min":{"1min":0.41499999999999998,"5min":0.32600000000000001,"15min":0.312},"max":{"1min":89.409000000000006,"5min":89.409000000000006,"15min":89.409000000000006},"last":0.47899999999999998},{"interface":"front","average":{"1min":6.9249999999999998,"5min":2.133,"15min":2.1219999999999999},"min":{"1min":0.47199999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":89.915000000000006,"5min":89.915000000000006,"15min":89.915000000000006},"last":0.28699999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.7569999999999999,"15min":1.994},"min":{"1min":0.433,"5min":0.33900000000000002,"15min":0.309},"max":{"1min":4.9169999999999998,"5min":46.398000000000003,"15min":106.672},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.653,"15min":2.04},"min":{"1min":0.438,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.3370000000000002,"5min":8.5410000000000004,"15min":107.348},"last":0.54200000000000004}]},{"osd":6,"last update":"Mon Feb 5 07:34:31 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.98199999999999998,"15min":1.5629999999999999},"min":{"1min":0.34399999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.9379999999999999,"5min":3.5259999999999998,"15min":5.5990000000000002},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.258,"15min":1.8560000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":4.9530000000000003,"5min":8.923,"15min":8.923},"last":1.3420000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:34:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":2.1949999999999998,"15min":1.929},"min":{"1min":0.46000000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.9710000000000001,"5min":46.351999999999997,"15min":66.703999999999994},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.77200000000000002,"5min":1.149,"15min":1.5649999999999999},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5820000000000001,"5min":5.6539999999999999,"15min":61.594999999999999},"last":0.79800000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738602,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5095936,"kb_used_data":4334512,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":88649216,"statfs":{"total":95995035648,"available":90776797184,"internally_reserved":0,"allocated":4438540288,"data_stored":4438068609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:34:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.9450000000000001,"15min":2.121},"min":{"1min":0.27500000000000002,"5min":0.192,"15min":0.16500000000000001},"max":{"1min":3.8519999999999999,"5min":25.277999999999999,"15min":45.372},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.8049999999999999,"5min":1.95,"15min":2.2010000000000001},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.186},"max":{"1min":8.6460000000000008,"5min":24.652999999999999,"15min":45.231999999999999},"last":2.8820000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:34:12 2024","interfaces":[{"interface":"back","average":{"1min":2.9289999999999998,"5min":1.5549999999999999,"15min":1.9079999999999999},"min":{"1min":0.26400000000000001,"5min":0.191,"15min":0.191},"max":{"1min":40.548000000000002,"5min":40.548000000000002,"15min":65.903999999999996},"last":3.0870000000000002},{"interface":"front","average":{"1min":2.6699999999999999,"5min":1.458,"15min":1.857},"min":{"1min":0.246,"5min":0.224,"15min":0.182},"max":{"1min":40.643000000000001,"5min":40.643000000000001,"15min":65.694000000000003},"last":2.9340000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":2.024,"15min":2.133},"min":{"1min":0.27600000000000002,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":3.109,"5min":29.629000000000001,"15min":65.155000000000001},"last":2.5489999999999999},{"interface":"front","average":{"1min":1.125,"5min":2.306,"15min":2.347},"min":{"1min":0.26600000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.6509999999999998,"5min":29.463999999999999,"15min":65.114000000000004},"last":3.3999999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:34:47 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":0.85799999999999998,"15min":1.956},"min":{"1min":0.308,"5min":0.27200000000000002,"15min":0.26700000000000002},"max":{"1min":3.7149999999999999,"5min":3.7149999999999999,"15min":65.188000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.9610000000000001,"5min":1.052,"15min":2.0619999999999998},"min":{"1min":0.30499999999999999,"5min":0.29699999999999999,"15min":0.23699999999999999},"max":{"1min":8.6899999999999995,"5min":8.6899999999999995,"15min":64.873000000000005},"last":2.7530000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.83899999999999997,"15min":1.7050000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.27000000000000002},"max":{"1min":2.27,"5min":4.6959999999999997,"15min":95.772999999999996},"last":2.5059999999999998},{"interface":"front","average":{"1min":1.296,"5min":1.0029999999999999,"15min":1.79},"min":{"1min":0.377,"5min":0.29299999999999998,"15min":0.27600000000000002},"max":{"1min":5.9340000000000002,"5min":7.1929999999999996,"15min":96.028000000000006},"last":2.9140000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":0.94799999999999995,"15min":1.256},"min":{"1min":0.38600000000000001,"5min":0.308,"15min":0.26300000000000001},"max":{"1min":3.7610000000000001,"5min":3.7610000000000001,"15min":4.6929999999999996},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.889,"5min":1.1200000000000001,"15min":1.506},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":8.718,"5min":8.718,"15min":8.718},"last":2.7919999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:34:58 2024","interfaces":[{"interface":"back","average":{"1min":1.897,"5min":1.377,"15min":1.9019999999999999},"min":{"1min":0.46400000000000002,"5min":0.34899999999999998,"15min":0.28599999999999998},"max":{"1min":8.3900000000000006,"5min":31.890999999999998,"15min":65.263000000000005},"last":0.621},{"interface":"front","average":{"1min":2.5790000000000002,"5min":1.597,"15min":1.96},"min":{"1min":0.49399999999999999,"5min":0.36699999999999999,"15min":0.309},"max":{"1min":8.7910000000000004,"5min":32.209000000000003,"15min":64.906000000000006},"last":3.044}]}]},{"osd":2,"up_from":18,"seq":77309411552,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4143124,"kb_used_data":3403908,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":89602028,"statfs":{"total":95995035648,"available":91752476672,"internally_reserved":0,"allocated":3485601792,"data_stored":3485168110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":1.081,"15min":2.0259999999999998},"min":{"1min":0.218,"5min":0.19600000000000001,"15min":0.182},"max":{"1min":2.1699999999999999,"5min":12.759,"15min":92.638000000000005},"last":3.0390000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.4610000000000001,"15min":2.2189999999999999},"min":{"1min":0.26700000000000002,"5min":0.22,"15min":0.20699999999999999},"max":{"1min":11.606999999999999,"5min":12.647,"15min":93.481999999999999},"last":4.6360000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":2.8290000000000002,"15min":2.339},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.19400000000000001},"max":{"1min":3.7469999999999999,"5min":63.823,"15min":93.504000000000005},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.746,"5min":3.0979999999999999,"15min":2.5209999999999999},"min":{"1min":0.26200000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.133,"5min":64.771000000000001,"15min":93.004000000000005},"last":0.51800000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:34:28 2024","interfaces":[{"interface":"back","average":{"1min":1.8819999999999999,"5min":2.7599999999999998,"15min":2.1509999999999998},"min":{"1min":0.28599999999999998,"5min":0.252,"15min":0.214},"max":{"1min":4.5789999999999997,"5min":63.784999999999997,"15min":63.784999999999997},"last":2.5699999999999998},{"interface":"front","average":{"1min":1.22,"5min":2.9540000000000002,"15min":2.093},"min":{"1min":0.376,"5min":0.28100000000000003,"15min":0.193},"max":{"1min":5.1989999999999998,"5min":63.521000000000001,"15min":63.521000000000001},"last":2.8340000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:34:51 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":0.92300000000000004,"15min":2.0710000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24099999999999999},"max":{"1min":5.7050000000000001,"5min":5.7050000000000001,"15min":168.11699999999999},"last":4.2699999999999996},{"interface":"front","average":{"1min":1.2889999999999999,"5min":0.91900000000000004,"15min":2.044},"min":{"1min":0.44900000000000001,"5min":0.33900000000000002,"15min":0.27200000000000002},"max":{"1min":4.5599999999999996,"5min":4.6950000000000003,"15min":168.589},"last":2.976}]},{"osd":5,"last update":"Mon Feb 5 07:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.85999999999999999,"15min":1.2390000000000001},"min":{"1min":0.41599999999999998,"5min":0.315,"15min":0.315},"max":{"1min":2.8610000000000002,"5min":4.742,"15min":39.923000000000002},"last":3.0070000000000001},{"interface":"front","average":{"1min":1.085,"5min":0.94499999999999995,"15min":1.2050000000000001},"min":{"1min":0.48599999999999999,"5min":0.33000000000000002,"15min":0.26400000000000001},"max":{"1min":3.7189999999999999,"5min":4.0709999999999997,"15min":38.920999999999999},"last":2.5049999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:34:34 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.01,"15min":1.1859999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.5250000000000004,"5min":4.5250000000000004,"15min":4.5250000000000004},"last":2.5419999999999998},{"interface":"front","average":{"1min":1.2769999999999999,"5min":0.93100000000000005,"15min":1.3180000000000001},"min":{"1min":0.46500000000000002,"5min":0.379,"15min":0.33100000000000002},"max":{"1min":4.6020000000000003,"5min":4.7779999999999996,"15min":5.7240000000000002},"last":2.9380000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:34:07 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.3500000000000001,"15min":1.4299999999999999},"min":{"1min":0.41299999999999998,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":5.7850000000000001,"5min":11.311999999999999,"15min":37.573999999999998},"last":4.5970000000000004},{"interface":"front","average":{"1min":1.181,"5min":1.0900000000000001,"15min":1.325},"min":{"1min":0.44700000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.9689999999999999,"5min":8.7349999999999994,"15min":37.432000000000002},"last":2.5939999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149915,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3812668,"kb_used_data":3033204,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":89932484,"statfs":{"total":95995035648,"available":92090863616,"internally_reserved":0,"allocated":3106000896,"data_stored":3105526002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":798161351},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:34:45 2024","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.1839999999999999,"15min":2.2919999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":5.4470000000000001,"5min":6.1239999999999997,"15min":74.114999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.429,"5min":2.2050000000000001,"15min":2.403},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.7450000000000001,"5min":99.989999999999995,"15min":99.989999999999995},"last":0.40799999999999997}]},{"osd":1,"last update":"Mon Feb 5 07:34:45 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":2.5019999999999998,"15min":2.6099999999999999},"min":{"1min":0.21299999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.6019999999999999,"5min":102.20399999999999,"15min":102.20399999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.163,"5min":2.5339999999999998,"15min":2.8769999999999998},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.185},"max":{"1min":3.2029999999999998,"5min":102.286,"15min":102.286},"last":0.50800000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:34:45 2024","interfaces":[{"interface":"back","average":{"1min":4.2380000000000004,"5min":2.7919999999999998,"15min":2.7450000000000001},"min":{"1min":0.34899999999999998,"5min":0.245,"15min":0.22700000000000001},"max":{"1min":59.085000000000001,"5min":99.847999999999999,"15min":99.847999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":4.2329999999999997,"5min":2.794,"15min":2.29},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":59.887999999999998,"5min":99.950999999999993,"15min":99.950999999999993},"last":0.52300000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:34:56 2024","interfaces":[{"interface":"back","average":{"1min":3.4900000000000002,"5min":2.5990000000000002,"15min":1.758},"min":{"1min":0.30499999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":31.655999999999999,"5min":100.024,"15min":100.024},"last":0.60399999999999998},{"interface":"front","average":{"1min":4.3440000000000003,"5min":3.0190000000000001,"15min":1.8879999999999999},"min":{"1min":0.32000000000000001,"5min":0.30399999999999999,"15min":0.28599999999999998},"max":{"1min":37.049999999999997,"5min":99.859999999999999,"15min":99.859999999999999},"last":0.67300000000000004}]},{"osd":5,"last update":"Mon Feb 5 07:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":2.234,"15min":1.897},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.28499999999999998},"max":{"1min":2.9670000000000001,"5min":99.911000000000001,"15min":99.911000000000001},"last":0.63},{"interface":"front","average":{"1min":0.81799999999999995,"5min":2.1880000000000002,"15min":1.619},"min":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.27400000000000002},"max":{"1min":2.8879999999999999,"5min":100.053,"15min":100.053},"last":0.68999999999999995}]},{"osd":6,"last update":"Mon Feb 5 07:34:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":2.246,"15min":1.3680000000000001},"min":{"1min":0.33100000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.7850000000000001,"5min":99.807000000000002,"15min":99.807000000000002},"last":23.18},{"interface":"front","average":{"1min":2.1400000000000001,"5min":2.4129999999999998,"15min":1.472},"min":{"1min":0.38200000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":17.84,"5min":101.489,"15min":101.489},"last":19.187999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:33:58 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":3.177,"15min":2.3039999999999998},"min":{"1min":0.45200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.4620000000000002,"5min":99.893000000000001,"15min":99.893000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.81699999999999995,"5min":2.8660000000000001,"15min":2.5640000000000001},"min":{"1min":0.51600000000000001,"5min":0.375,"15min":0.375},"max":{"1min":2.206,"5min":99.879000000000005,"15min":99.879000000000005},"last":0.73199999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953686,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5366548,"kb_used_data":4707844,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":88378604,"statfs":{"total":95995035648,"available":90499690496,"internally_reserved":0,"allocated":4820832256,"data_stored":4820397246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.65700000000000003,"15min":2.125},"min":{"1min":0.22700000000000001,"5min":0.219,"15min":0.20300000000000001},"max":{"1min":2.3460000000000001,"5min":3.7610000000000001,"15min":101.212},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.72899999999999998,"15min":2.2250000000000001},"min":{"1min":0.23599999999999999,"5min":0.23499999999999999,"15min":0.23300000000000001},"max":{"1min":2.496,"5min":4.0659999999999998,"15min":102.07599999999999},"last":0.33400000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30099999999999999,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":1.542,"5min":45.655999999999999,"15min":69.102000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":2.1360000000000001,"15min":2.548},"min":{"1min":0.30299999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.133,"5min":46.531999999999996,"15min":68.108999999999995},"last":0.63200000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:34:51 2024","interfaces":[{"interface":"back","average":{"1min":12.347,"5min":3.0190000000000001,"15min":2.1379999999999999},"min":{"1min":0.30399999999999999,"5min":0.249,"15min":0.249},"max":{"1min":180.40199999999999,"5min":180.40199999999999,"15min":180.40199999999999},"last":1.3500000000000001},{"interface":"front","average":{"1min":11.576000000000001,"5min":3.1419999999999999,"15min":2.1499999999999999},"min":{"1min":0.28199999999999997,"5min":0.28000000000000003,"15min":0.24299999999999999},"max":{"1min":180.381,"5min":180.381,"15min":180.381},"last":1.0009999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:34:51 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":2.633,"15min":1.8979999999999999},"min":{"1min":0.308,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.673,"5min":46.549999999999997,"15min":66.019999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.5389999999999999,"5min":2.851,"15min":2.1539999999999999},"min":{"1min":0.433,"5min":0.33300000000000002,"15min":0.26400000000000001},"max":{"1min":3.6589999999999998,"5min":46.003999999999998,"15min":101.001},"last":0.66200000000000003}]},{"osd":5,"last update":"Mon Feb 5 07:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.47799999999999998,"15min":0.59799999999999998},"min":{"1min":0.36899999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.69999999999999996,"5min":1.907,"15min":10.917999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.47399999999999998,"15min":0.626},"min":{"1min":0.30599999999999999,"5min":0.27700000000000002,"15min":0.24199999999999999},"max":{"1min":1.3939999999999999,"5min":2.2410000000000001,"15min":15.105},"last":0.51900000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.50900000000000001,"15min":1.081},"min":{"1min":0.34699999999999998,"5min":0.25600000000000001,"15min":0.245},"max":{"1min":1.236,"5min":1.2470000000000001,"15min":14.606},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.51900000000000002,"15min":1.3009999999999999},"min":{"1min":0.44600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":13.114000000000001},"last":0.38}]},{"osd":7,"last update":"Mon Feb 5 07:34:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.52200000000000002,"15min":0.73099999999999998},"min":{"1min":0.38300000000000001,"5min":0.30599999999999999,"15min":0.25700000000000001},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":15.073},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53500000000000003,"15min":0.69599999999999995},"min":{"1min":0.34000000000000002,"5min":0.316,"15min":0.248},"max":{"1min":0.85599999999999998,"5min":1.359,"15min":14.581},"last":0.58099999999999996}]}]},{"osd":5,"up_from":0,"seq":176093659345,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4440915968,"data_stored":4440915968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2179465216,"data_stored":2179465216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3494576128,"data_stored":3494576128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3134521344,"data_stored":3134521344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4834983936,"data_stored":4834983936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3010199552,"data_stored":3010199552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:35:05.876 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-05T07:35:05.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set unique_pool_3 pg_num 26 2024-02-05T07:35:05.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:06.006 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:06.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:06.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:05 smithi090 ceph-mon[33213]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:35:06.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:05 smithi090 ceph-mon[33213]: osdmap e481: 8 total, 8 up, 6 in 2024-02-05T07:35:06.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:06.135 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:06.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:06.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:06.263 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:06.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:06.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:06.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:06.394 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:06.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:06.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:06.527 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:06.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:06.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:06.657 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:06.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:06.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:06.772 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:35:06.787 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 79 16 115276 115260 91.1575 76.4375 0.0208072 0.0109649 2024-02-05T07:35:06.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:06.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:06.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:06.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:06 smithi007 ceph-mon[29098]: pgmap v866: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 peering, 164 active+clean; 6.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 7.0 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 1/332379 objects misplaced (0.000%); 33 KiB/s, 0 objects/s recovering 2024-02-05T07:35:06.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:06 smithi007 ceph-mon[29098]: osdmap e482: 8 total, 8 up, 6 in 2024-02-05T07:35:06.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:06 smithi007 ceph-mon[37158]: pgmap v866: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 peering, 164 active+clean; 6.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 7.0 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 1/332379 objects misplaced (0.000%); 33 KiB/s, 0 objects/s recovering 2024-02-05T07:35:06.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:06 smithi007 ceph-mon[37158]: osdmap e482: 8 total, 8 up, 6 in 2024-02-05T07:35:06.948 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:06.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:06.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:07.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:06 smithi090 ceph-mon[33213]: pgmap v866: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 peering, 164 active+clean; 6.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 7.0 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 1/332379 objects misplaced (0.000%); 33 KiB/s, 0 objects/s recovering 2024-02-05T07:35:07.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:06 smithi090 ceph-mon[33213]: osdmap e482: 8 total, 8 up, 6 in 2024-02-05T07:35:07.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:07.078 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:07.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:07.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:07.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:07.210 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:07.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:07.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:07.351 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:07.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:07.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:07.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:07.487 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:07.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:07.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:07.618 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:07.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:07.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:07.648 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:35:07.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:07.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:07.752 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:07.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:07.786 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:35:07.787000+0000 min lat: 0.00131497 max lat: 0.307728 avg lat: 0.0109854 2024-02-05T07:35:07.787 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:35:07.787 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 80 16 116525 116509 90.9938 78.0625 0.0147029 0.0109854 2024-02-05T07:35:07.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:07.893 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:07.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:07.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:07 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4273064561' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:35:07.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:07 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:35:07.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-05T07:35:07.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:07 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-05T07:35:07.910 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:07 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4273064561' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:35:07.910 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:07 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:35:07.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-05T07:35:07.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:07 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-05T07:35:07.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:08.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:08.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:07 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4273064561' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:35:08.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:07 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:35:08.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-05T07:35:08.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:07 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-05T07:35:08.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:08.159 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:08.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:08.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:08.291 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:08.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:08.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:08.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:08.423 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:08.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:08.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:08.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:08.556 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:08.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:08.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:08.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:08.705 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:08.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:08.787 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 81 16 117819 117803 90.8688 80.875 0.0168042 0.0110002 2024-02-05T07:35:08.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:08.846 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:08.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:08.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:08.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[29098]: pgmap v869: 167 pgs: 2 active+remapped+backfill_toofull, 165 active+clean; 6.9 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 7.0 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 1/338346 objects misplaced (0.000%); 33 KiB/s, 0 objects/s recovering 2024-02-05T07:35:08.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:35:08.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:35:08.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-05T07:35:08.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[29098]: osdmap e483: 8 total, 8 up, 6 in 2024-02-05T07:35:08.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[29098]: 1.11 scrub starts 2024-02-05T07:35:08.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[29098]: 1.11 scrub ok 2024-02-05T07:35:08.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[29098]: 1.9 scrub starts 2024-02-05T07:35:08.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[29098]: 1.9 scrub ok 2024-02-05T07:35:08.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[37158]: pgmap v869: 167 pgs: 2 active+remapped+backfill_toofull, 165 active+clean; 6.9 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 7.0 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 1/338346 objects misplaced (0.000%); 33 KiB/s, 0 objects/s recovering 2024-02-05T07:35:08.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:35:08.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:35:08.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-05T07:35:08.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[37158]: osdmap e483: 8 total, 8 up, 6 in 2024-02-05T07:35:08.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[37158]: 1.11 scrub starts 2024-02-05T07:35:08.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[37158]: 1.11 scrub ok 2024-02-05T07:35:08.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[37158]: 1.9 scrub starts 2024-02-05T07:35:08.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:08 smithi007 ceph-mon[37158]: 1.9 scrub ok 2024-02-05T07:35:08.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:08.981 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:08.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:08 smithi090 ceph-mon[33213]: pgmap v869: 167 pgs: 2 active+remapped+backfill_toofull, 165 active+clean; 6.9 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 7.0 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 1/338346 objects misplaced (0.000%); 33 KiB/s, 0 objects/s recovering 2024-02-05T07:35:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:08 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:35:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:08 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:35:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:08 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-05T07:35:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:08 smithi090 ceph-mon[33213]: osdmap e483: 8 total, 8 up, 6 in 2024-02-05T07:35:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:08 smithi090 ceph-mon[33213]: 1.11 scrub starts 2024-02-05T07:35:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:08 smithi090 ceph-mon[33213]: 1.11 scrub ok 2024-02-05T07:35:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:08 smithi090 ceph-mon[33213]: 1.9 scrub starts 2024-02-05T07:35:09.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:08 smithi090 ceph-mon[33213]: 1.9 scrub ok 2024-02-05T07:35:09.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:09.114 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:09.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:09.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:09.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:09.259 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:09.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:09.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:09.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:09.397 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:09.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:09.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:09.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:09.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:09.543 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:09.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:09.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:09.691 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:09.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:09 smithi090 ceph-mon[33213]: 1.b scrub starts 2024-02-05T07:35:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:09 smithi090 ceph-mon[33213]: 1.b scrub ok 2024-02-05T07:35:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:09 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:35:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:09 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:35:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:09 smithi090 ceph-mon[33213]: osdmap e484: 8 total, 8 up, 6 in 2024-02-05T07:35:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:09 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:09.787 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 82 16 119162 119146 90.7842 83.9375 0.00730045 0.0110106 2024-02-05T07:35:09.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:09.824 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:09.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:09.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:09.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:09 smithi007 ceph-mon[29098]: 1.b scrub starts 2024-02-05T07:35:09.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:09 smithi007 ceph-mon[29098]: 1.b scrub ok 2024-02-05T07:35:09.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:09 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:35:09.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:09 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:35:09.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:09 smithi007 ceph-mon[29098]: osdmap e484: 8 total, 8 up, 6 in 2024-02-05T07:35:09.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:09 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:09 smithi007 ceph-mon[37158]: 1.b scrub starts 2024-02-05T07:35:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:09 smithi007 ceph-mon[37158]: 1.b scrub ok 2024-02-05T07:35:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:09 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:35:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:09 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:35:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:09 smithi007 ceph-mon[37158]: osdmap e484: 8 total, 8 up, 6 in 2024-02-05T07:35:09.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:09 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:09.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:09.966 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:09.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:10.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:10.099 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:10.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:10.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:10.200 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:10.244 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:10.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:10.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:10.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:10.376 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:10.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:10.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:10.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:10.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:10.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:10.633 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:10.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:10.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:10.734 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:10.765 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:10.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:10.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:10.787 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 83 16 120331 120315 90.5706 73.0625 0.0143704 0.0110364 2024-02-05T07:35:10.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:10.914 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:10.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:10.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:10 smithi007 ceph-mon[29098]: 1.2 scrub starts 2024-02-05T07:35:10.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:10 smithi007 ceph-mon[29098]: 1.2 scrub ok 2024-02-05T07:35:10.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:10 smithi007 ceph-mon[29098]: pgmap v872: 167 pgs: 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 163 active+clean; 7.0 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 67 MiB/s wr, 1.07k op/s; 1/345234 objects misplaced (0.000%) 2024-02-05T07:35:10.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:10 smithi007 ceph-mon[29098]: osdmap e485: 8 total, 8 up, 6 in 2024-02-05T07:35:10.941 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:10 smithi007 ceph-mon[37158]: 1.2 scrub starts 2024-02-05T07:35:10.941 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:10 smithi007 ceph-mon[37158]: 1.2 scrub ok 2024-02-05T07:35:10.941 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:10 smithi007 ceph-mon[37158]: pgmap v872: 167 pgs: 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 163 active+clean; 7.0 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 67 MiB/s wr, 1.07k op/s; 1/345234 objects misplaced (0.000%) 2024-02-05T07:35:10.941 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:10 smithi007 ceph-mon[37158]: osdmap e485: 8 total, 8 up, 6 in 2024-02-05T07:35:11.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:11.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:35:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:35:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:10 smithi090 ceph-mon[33213]: 1.2 scrub starts 2024-02-05T07:35:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:10 smithi090 ceph-mon[33213]: 1.2 scrub ok 2024-02-05T07:35:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:10 smithi090 ceph-mon[33213]: pgmap v872: 167 pgs: 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 163 active+clean; 7.0 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 67 MiB/s wr, 1.07k op/s; 1/345234 objects misplaced (0.000%) 2024-02-05T07:35:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:10 smithi090 ceph-mon[33213]: osdmap e485: 8 total, 8 up, 6 in 2024-02-05T07:35:11.043 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:11.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:11.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:11.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:11.173 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:11.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:11.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:11.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:11.321 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:11.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:11.380 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:35:11.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:11.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:11.485 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:11.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:11.518 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:35:11.570 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:35:11.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:11.612 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:35:11.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:11.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:11.654 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:35:11.698 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:35:11.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:11.740 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:35:11.768 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:11.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:11.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:11.787 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 84 16 121562 121546 90.4083 76.9375 0.0134722 0.0110564 2024-02-05T07:35:11.794 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:35:11.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[29098]: osdmap e486: 8 total, 8 up, 6 in 2024-02-05T07:35:11.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[29098]: 2.0 scrub starts 2024-02-05T07:35:11.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[29098]: 2.0 scrub ok 2024-02-05T07:35:11.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[29098]: 3.18 scrub starts 2024-02-05T07:35:11.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[29098]: 3.18 scrub ok 2024-02-05T07:35:11.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-05T07:35:11.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-05T07:35:11.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/658618397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:35:11.795 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[37158]: osdmap e486: 8 total, 8 up, 6 in 2024-02-05T07:35:11.795 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[37158]: 2.0 scrub starts 2024-02-05T07:35:11.795 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[37158]: 2.0 scrub ok 2024-02-05T07:35:11.795 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[37158]: 3.18 scrub starts 2024-02-05T07:35:11.796 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[37158]: 3.18 scrub ok 2024-02-05T07:35:11.796 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-05T07:35:11.796 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-05T07:35:11.796 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:11 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/658618397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:35:11.834 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:35:11.843 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:35:11.843 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:35:11.843 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:35:11.843 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:35:11.843 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:35:11.843 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:35:11.843 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:35:11.844 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:35:11.870 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:11.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:11.898 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:11.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:11.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:12.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:11 smithi090 ceph-mon[33213]: osdmap e486: 8 total, 8 up, 6 in 2024-02-05T07:35:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:11 smithi090 ceph-mon[33213]: 2.0 scrub starts 2024-02-05T07:35:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:11 smithi090 ceph-mon[33213]: 2.0 scrub ok 2024-02-05T07:35:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:11 smithi090 ceph-mon[33213]: 3.18 scrub starts 2024-02-05T07:35:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:11 smithi090 ceph-mon[33213]: 3.18 scrub ok 2024-02-05T07:35:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:11 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-05T07:35:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:11 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-05T07:35:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:11 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/658618397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:35:12.043 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:12.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:12.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:12.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:12.175 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:12.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:12.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:12.277 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:12.317 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:12.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:12.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:12.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:12.549 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:12.593 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:12.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:12.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:12.744 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:12.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:12.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:12.790 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 85 16 122889 122873 90.3203 82.9375 0.00568599 0.0110672 2024-02-05T07:35:12.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:12.887 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:12.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:12.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:12.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:12.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[29098]: pgmap v875: 166 pgs: 1 clean+premerge+peered, 2 active+remapped+backfill_toofull, 163 active+clean; 7.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 100 MiB/s wr, 1.60k op/s; 1/357612 objects misplaced (0.000%) 2024-02-05T07:35:12.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-05T07:35:12.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[29098]: osdmap e487: 8 total, 8 up, 6 in 2024-02-05T07:35:12.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[29098]: 4.12 scrub starts 2024-02-05T07:35:12.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[29098]: 4.12 scrub ok 2024-02-05T07:35:12.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[29098]: 4.8 scrub starts 2024-02-05T07:35:12.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[29098]: 4.8 scrub ok 2024-02-05T07:35:12.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[29098]: 3.2 scrub starts 2024-02-05T07:35:12.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[29098]: 3.2 scrub ok 2024-02-05T07:35:12.988 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:12.988 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[37158]: pgmap v875: 166 pgs: 1 clean+premerge+peered, 2 active+remapped+backfill_toofull, 163 active+clean; 7.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 100 MiB/s wr, 1.60k op/s; 1/357612 objects misplaced (0.000%) 2024-02-05T07:35:12.988 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-05T07:35:12.988 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[37158]: osdmap e487: 8 total, 8 up, 6 in 2024-02-05T07:35:12.988 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[37158]: 4.12 scrub starts 2024-02-05T07:35:12.988 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[37158]: 4.12 scrub ok 2024-02-05T07:35:12.989 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[37158]: 4.8 scrub starts 2024-02-05T07:35:12.989 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[37158]: 4.8 scrub ok 2024-02-05T07:35:12.989 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[37158]: 3.2 scrub starts 2024-02-05T07:35:12.989 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:12 smithi007 ceph-mon[37158]: 3.2 scrub ok 2024-02-05T07:35:12.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:12.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:35:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:12 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:12 smithi090 ceph-mon[33213]: pgmap v875: 166 pgs: 1 clean+premerge+peered, 2 active+remapped+backfill_toofull, 163 active+clean; 7.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 100 MiB/s wr, 1.60k op/s; 1/357612 objects misplaced (0.000%) 2024-02-05T07:35:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:12 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-05T07:35:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:12 smithi090 ceph-mon[33213]: osdmap e487: 8 total, 8 up, 6 in 2024-02-05T07:35:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:12 smithi090 ceph-mon[33213]: 4.12 scrub starts 2024-02-05T07:35:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:12 smithi090 ceph-mon[33213]: 4.12 scrub ok 2024-02-05T07:35:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:12 smithi090 ceph-mon[33213]: 4.8 scrub starts 2024-02-05T07:35:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:12 smithi090 ceph-mon[33213]: 4.8 scrub ok 2024-02-05T07:35:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:12 smithi090 ceph-mon[33213]: 3.2 scrub starts 2024-02-05T07:35:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:12 smithi090 ceph-mon[33213]: 3.2 scrub ok 2024-02-05T07:35:13.031 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:13.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:13.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:13.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:13.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:13.263 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:13.294 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:13.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:13.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:13.450 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:13.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:13.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:13.581 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:13.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:13.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:13.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:13.728 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:13.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:13.787 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 86 16 124026 124010 90.0964 71.0625 0.00902497 0.0110944 2024-02-05T07:35:13.820 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:35:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:35:13.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[29098]: osdmap e488: 8 total, 8 up, 6 in 2024-02-05T07:35:13.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[29098]: 4.b scrub starts 2024-02-05T07:35:13.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[29098]: 4.b scrub ok 2024-02-05T07:35:13.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[29098]: 6.15 scrub starts 2024-02-05T07:35:13.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[29098]: 4.1e scrub starts 2024-02-05T07:35:13.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[29098]: 4.1e scrub ok 2024-02-05T07:35:13.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[29098]: 6.15 scrub ok 2024-02-05T07:35:13.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[29098]: osdmap e489: 8 total, 8 up, 6 in 2024-02-05T07:35:13.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[37158]: osdmap e488: 8 total, 8 up, 6 in 2024-02-05T07:35:13.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[37158]: 4.b scrub starts 2024-02-05T07:35:13.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[37158]: 4.b scrub ok 2024-02-05T07:35:13.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[37158]: 6.15 scrub starts 2024-02-05T07:35:13.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[37158]: 4.1e scrub starts 2024-02-05T07:35:13.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[37158]: 4.1e scrub ok 2024-02-05T07:35:13.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[37158]: 6.15 scrub ok 2024-02-05T07:35:13.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:13 smithi007 ceph-mon[37158]: osdmap e489: 8 total, 8 up, 6 in 2024-02-05T07:35:13.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:13.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:13.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:13.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:14.008 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:14.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:14.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:14.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:13 smithi090 ceph-mon[33213]: osdmap e488: 8 total, 8 up, 6 in 2024-02-05T07:35:14.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:13 smithi090 ceph-mon[33213]: 4.b scrub starts 2024-02-05T07:35:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:13 smithi090 ceph-mon[33213]: 4.b scrub ok 2024-02-05T07:35:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:13 smithi090 ceph-mon[33213]: 6.15 scrub starts 2024-02-05T07:35:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:13 smithi090 ceph-mon[33213]: 4.1e scrub starts 2024-02-05T07:35:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:13 smithi090 ceph-mon[33213]: 4.1e scrub ok 2024-02-05T07:35:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:13 smithi090 ceph-mon[33213]: 6.15 scrub ok 2024-02-05T07:35:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:13 smithi090 ceph-mon[33213]: osdmap e489: 8 total, 8 up, 6 in 2024-02-05T07:35:14.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:14.155 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:14.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:14.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:14.306 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:14.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:14.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:14.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:14.438 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:14.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:14.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:14.571 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:14.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:14.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:14.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:14.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:14.793 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 87 16 125318 125302 89.9829 80.75 0.0030442 0.0111083 2024-02-05T07:35:14.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:14.863 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:14.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:14.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:14.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:14.988 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:35:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:14.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:15.007 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:14 smithi007 ceph-mon[29098]: 4.e scrub starts 2024-02-05T07:35:15.007 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:14 smithi007 ceph-mon[29098]: 4.e scrub ok 2024-02-05T07:35:15.007 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:14 smithi007 ceph-mon[29098]: pgmap v878: 166 pgs: 1 clean+premerge+peered, 2 active+remapped+backfill_toofull, 163 active+clean; 7.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 64 MiB/s wr, 1.03k op/s; 1/357612 objects misplaced (0.000%) 2024-02-05T07:35:15.007 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:14 smithi007 ceph-mon[29098]: osdmap e490: 8 total, 8 up, 6 in 2024-02-05T07:35:15.007 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:14 smithi007 ceph-mon[37158]: 4.e scrub starts 2024-02-05T07:35:15.007 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:14 smithi007 ceph-mon[37158]: 4.e scrub ok 2024-02-05T07:35:15.007 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:14 smithi007 ceph-mon[37158]: pgmap v878: 166 pgs: 1 clean+premerge+peered, 2 active+remapped+backfill_toofull, 163 active+clean; 7.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 64 MiB/s wr, 1.03k op/s; 1/357612 objects misplaced (0.000%) 2024-02-05T07:35:15.008 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:14 smithi007 ceph-mon[37158]: osdmap e490: 8 total, 8 up, 6 in 2024-02-05T07:35:15.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:14 smithi090 ceph-mon[33213]: 4.e scrub starts 2024-02-05T07:35:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:14 smithi090 ceph-mon[33213]: 4.e scrub ok 2024-02-05T07:35:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:14 smithi090 ceph-mon[33213]: pgmap v878: 166 pgs: 1 clean+premerge+peered, 2 active+remapped+backfill_toofull, 163 active+clean; 7.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 64 MiB/s wr, 1.03k op/s; 1/357612 objects misplaced (0.000%) 2024-02-05T07:35:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:14 smithi090 ceph-mon[33213]: osdmap e490: 8 total, 8 up, 6 in 2024-02-05T07:35:15.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:15.122 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:15.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:15.224 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:15.231 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:35:15.248 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:15.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:15.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:15.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:15.362 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:35:15.363 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:35:15.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:15.405 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:15.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:15.505 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:15.545 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:15.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:15.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:15.687 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:15.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:15.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:15.751 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 10 pg_num to 26 2024-02-05T07:35:15.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:15.793 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 88 16 126384 126368 89.7175 66.625 0.0263061 0.0111409 2024-02-05T07:35:15.818 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:15.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:15.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:15.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:15.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:15 smithi090 ceph-mon[33213]: 5.f scrub starts 2024-02-05T07:35:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:15 smithi090 ceph-mon[33213]: 5.f scrub ok 2024-02-05T07:35:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:15 smithi090 ceph-mon[33213]: 6.13 scrub starts 2024-02-05T07:35:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:15 smithi090 ceph-mon[33213]: 6.1c scrub starts 2024-02-05T07:35:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:15 smithi090 ceph-mon[33213]: 6.1c scrub ok 2024-02-05T07:35:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:15 smithi090 ceph-mon[33213]: 6.13 scrub ok 2024-02-05T07:35:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-05T07:35:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:15 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-05T07:35:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:15 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2284011302' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:35:16.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:16.057 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[29098]: 5.f scrub starts 2024-02-05T07:35:16.057 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[29098]: 5.f scrub ok 2024-02-05T07:35:16.057 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[29098]: 6.13 scrub starts 2024-02-05T07:35:16.057 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[29098]: 6.1c scrub starts 2024-02-05T07:35:16.057 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[29098]: 6.1c scrub ok 2024-02-05T07:35:16.057 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[29098]: 6.13 scrub ok 2024-02-05T07:35:16.057 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-05T07:35:16.057 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-05T07:35:16.057 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2284011302' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:35:16.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[37158]: 5.f scrub starts 2024-02-05T07:35:16.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[37158]: 5.f scrub ok 2024-02-05T07:35:16.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[37158]: 6.13 scrub starts 2024-02-05T07:35:16.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[37158]: 6.1c scrub starts 2024-02-05T07:35:16.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[37158]: 6.1c scrub ok 2024-02-05T07:35:16.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[37158]: 6.13 scrub ok 2024-02-05T07:35:16.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-05T07:35:16.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-05T07:35:16.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:15 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2284011302' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:35:16.082 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:16.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:16.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:16.212 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:16.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:16.314 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:16.356 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:16.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:16.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:16.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:16.502 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:16.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:16.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:16.642 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:16.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:16.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:16.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:16.775 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:16.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:16.793 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 89 16 127702 127686 89.6349 82.375 0.00830744 0.0111518 2024-02-05T07:35:16.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:16.914 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[29098]: pgmap v881: 165 pgs: 2 active+remapped+backfill_toofull, 163 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 1/370203 objects misplaced (0.000%) 2024-02-05T07:35:16.914 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-05T07:35:16.915 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2284011302' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-02-05T07:35:16.915 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[29098]: osdmap e491: 8 total, 8 up, 6 in 2024-02-05T07:35:16.915 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:16.915 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:35:16.915 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[29098]: 6.19 scrub starts 2024-02-05T07:35:16.915 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[29098]: 3.d scrub starts 2024-02-05T07:35:16.915 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[29098]: 3.d scrub ok 2024-02-05T07:35:16.915 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[29098]: 6.19 scrub ok 2024-02-05T07:35:16.915 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[37158]: pgmap v881: 165 pgs: 2 active+remapped+backfill_toofull, 163 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 1/370203 objects misplaced (0.000%) 2024-02-05T07:35:16.916 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-05T07:35:16.916 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2284011302' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-02-05T07:35:16.916 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[37158]: osdmap e491: 8 total, 8 up, 6 in 2024-02-05T07:35:16.916 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:16.916 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:35:16.916 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[37158]: 6.19 scrub starts 2024-02-05T07:35:16.916 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[37158]: 3.d scrub starts 2024-02-05T07:35:16.916 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[37158]: 3.d scrub ok 2024-02-05T07:35:16.916 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:16 smithi007 ceph-mon[37158]: 6.19 scrub ok 2024-02-05T07:35:16.918 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:16.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:16.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:16.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T07:35:17.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:16 smithi090 ceph-mon[33213]: pgmap v881: 165 pgs: 2 active+remapped+backfill_toofull, 163 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 1/370203 objects misplaced (0.000%) 2024-02-05T07:35:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:16 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-05T07:35:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:16 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2284011302' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-02-05T07:35:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:16 smithi090 ceph-mon[33213]: osdmap e491: 8 total, 8 up, 6 in 2024-02-05T07:35:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:16 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:16 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:35:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:16 smithi090 ceph-mon[33213]: 6.19 scrub starts 2024-02-05T07:35:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:16 smithi090 ceph-mon[33213]: 3.d scrub starts 2024-02-05T07:35:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:16 smithi090 ceph-mon[33213]: 3.d scrub ok 2024-02-05T07:35:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:16 smithi090 ceph-mon[33213]: 6.19 scrub ok 2024-02-05T07:35:17.083 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:17.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:17.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:17.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:17.216 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:17.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:17.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:17.372 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:17.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:17.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:17.503 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:17.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:17.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:17.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:17.652 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:17.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:17.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:17.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:17.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:17.793 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total time run: 90.0091 2024-02-05T07:35:17.793 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total writes made: 128927 2024-02-05T07:35:17.793 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Write size: 65536 2024-02-05T07:35:17.793 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object size: 65536 2024-02-05T07:35:17.793 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Bandwidth (MB/sec): 89.5236 2024-02-05T07:35:17.793 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Bandwidth: 30.4729 2024-02-05T07:35:17.794 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max bandwidth (MB/sec): 222.188 2024-02-05T07:35:17.794 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min bandwidth (MB/sec): 38.9375 2024-02-05T07:35:17.794 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average IOPS: 1432 2024-02-05T07:35:17.794 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev IOPS: 487.567 2024-02-05T07:35:17.794 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max IOPS: 3555 2024-02-05T07:35:17.794 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min IOPS: 623 2024-02-05T07:35:17.794 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average Latency(s): 0.0111662 2024-02-05T07:35:17.794 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Latency(s): 0.0101171 2024-02-05T07:35:17.794 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max latency(s): 0.307728 2024-02-05T07:35:17.794 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min latency(s): 0.00131497 2024-02-05T07:35:17.893 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:17.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:17.924 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:17.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:17.981 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T07:35:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:17 smithi090 ceph-mon[33213]: osdmap e492: 8 total, 8 up, 6 in 2024-02-05T07:35:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:17 smithi090 ceph-mon[33213]: 5.1b scrub starts 2024-02-05T07:35:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:17 smithi090 ceph-mon[33213]: 5.1b scrub ok 2024-02-05T07:35:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:17 smithi090 ceph-mon[33213]: 3.5 scrub starts 2024-02-05T07:35:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:17 smithi090 ceph-mon[33213]: 3.5 scrub ok 2024-02-05T07:35:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:17 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:35:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:17 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:35:18.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:18.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:18.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:18.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:18.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:17 smithi007 ceph-mon[29098]: osdmap e492: 8 total, 8 up, 6 in 2024-02-05T07:35:18.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:17 smithi007 ceph-mon[29098]: 5.1b scrub starts 2024-02-05T07:35:18.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:17 smithi007 ceph-mon[29098]: 5.1b scrub ok 2024-02-05T07:35:18.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:17 smithi007 ceph-mon[29098]: 3.5 scrub starts 2024-02-05T07:35:18.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:17 smithi007 ceph-mon[29098]: 3.5 scrub ok 2024-02-05T07:35:18.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:17 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:35:18.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:17 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:35:18.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:17 smithi007 ceph-mon[37158]: osdmap e492: 8 total, 8 up, 6 in 2024-02-05T07:35:18.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:17 smithi007 ceph-mon[37158]: 5.1b scrub starts 2024-02-05T07:35:18.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:17 smithi007 ceph-mon[37158]: 5.1b scrub ok 2024-02-05T07:35:18.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:17 smithi007 ceph-mon[37158]: 3.5 scrub starts 2024-02-05T07:35:18.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:17 smithi007 ceph-mon[37158]: 3.5 scrub ok 2024-02-05T07:35:18.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:17 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:35:18.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:17 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:35:18.174 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:18.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:18.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:18.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:18.299 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:18.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:18.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:18.401 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:18.425 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:18.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:18.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:18.550 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:18.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:18.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:18.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:18.677 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:18.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:18.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:18.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:18.809 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:18.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:18.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:18.942 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:18.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:18.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:18 smithi090 ceph-mon[33213]: pgmap v884: 165 pgs: 2 active+remapped+backfill_toofull, 163 active+clean; 7.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 1/375756 objects misplaced (0.000%) 2024-02-05T07:35:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:18 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:35:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:18 smithi090 ceph-mon[33213]: osdmap e493: 8 total, 8 up, 6 in 2024-02-05T07:35:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:18 smithi090 ceph-mon[33213]: 6.5 scrub starts 2024-02-05T07:35:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:18 smithi090 ceph-mon[33213]: 4.1a scrub starts 2024-02-05T07:35:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:18 smithi090 ceph-mon[33213]: 4.1a scrub ok 2024-02-05T07:35:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:18 smithi090 ceph-mon[33213]: 6.5 scrub ok 2024-02-05T07:35:19.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:19.078 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:19.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:19.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:18 smithi007 ceph-mon[29098]: pgmap v884: 165 pgs: 2 active+remapped+backfill_toofull, 163 active+clean; 7.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 1/375756 objects misplaced (0.000%) 2024-02-05T07:35:19.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:18 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:35:19.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:18 smithi007 ceph-mon[29098]: osdmap e493: 8 total, 8 up, 6 in 2024-02-05T07:35:19.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:18 smithi007 ceph-mon[29098]: 6.5 scrub starts 2024-02-05T07:35:19.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:18 smithi007 ceph-mon[29098]: 4.1a scrub starts 2024-02-05T07:35:19.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:18 smithi007 ceph-mon[29098]: 4.1a scrub ok 2024-02-05T07:35:19.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:18 smithi007 ceph-mon[29098]: 6.5 scrub ok 2024-02-05T07:35:19.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:18 smithi007 ceph-mon[37158]: pgmap v884: 165 pgs: 2 active+remapped+backfill_toofull, 163 active+clean; 7.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 1/375756 objects misplaced (0.000%) 2024-02-05T07:35:19.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:18 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:35:19.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:18 smithi007 ceph-mon[37158]: osdmap e493: 8 total, 8 up, 6 in 2024-02-05T07:35:19.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:18 smithi007 ceph-mon[37158]: 6.5 scrub starts 2024-02-05T07:35:19.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:18 smithi007 ceph-mon[37158]: 4.1a scrub starts 2024-02-05T07:35:19.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:18 smithi007 ceph-mon[37158]: 4.1a scrub ok 2024-02-05T07:35:19.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:18 smithi007 ceph-mon[37158]: 6.5 scrub ok 2024-02-05T07:35:19.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:19.206 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:19.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:19.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:19.348 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:19.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:19.449 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:19.485 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:19.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:19.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:19.629 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:19.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:19.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:19.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:19.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:19.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:19.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:19.894 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:19.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:19.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:19.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:20.032 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:20.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:20.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:20.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:20.163 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:20.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:20.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:20.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[29098]: osdmap e494: 8 total, 8 up, 6 in 2024-02-05T07:35:20.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[29098]: 10.7 scrub starts 2024-02-05T07:35:20.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[29098]: 10.1 scrub starts 2024-02-05T07:35:20.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[29098]: 10.3 scrub starts 2024-02-05T07:35:20.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[29098]: pgmap v887: 174 pgs: 2 peering, 10 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 255 B/s rd, 59 MiB/s wr, 939 op/s; 1/381459 objects misplaced (0.000%) 2024-02-05T07:35:20.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-05T07:35:20.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:35:20.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-05T07:35:20.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:35:20.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[37158]: osdmap e494: 8 total, 8 up, 6 in 2024-02-05T07:35:20.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[37158]: 10.7 scrub starts 2024-02-05T07:35:20.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[37158]: 10.1 scrub starts 2024-02-05T07:35:20.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[37158]: 10.3 scrub starts 2024-02-05T07:35:20.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[37158]: pgmap v887: 174 pgs: 2 peering, 10 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 255 B/s rd, 59 MiB/s wr, 939 op/s; 1/381459 objects misplaced (0.000%) 2024-02-05T07:35:20.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-05T07:35:20.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:35:20.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-05T07:35:20.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:19 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:35:20.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:19 smithi090 ceph-mon[33213]: osdmap e494: 8 total, 8 up, 6 in 2024-02-05T07:35:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:19 smithi090 ceph-mon[33213]: 10.7 scrub starts 2024-02-05T07:35:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:19 smithi090 ceph-mon[33213]: 10.1 scrub starts 2024-02-05T07:35:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:19 smithi090 ceph-mon[33213]: 10.3 scrub starts 2024-02-05T07:35:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:19 smithi090 ceph-mon[33213]: pgmap v887: 174 pgs: 2 peering, 10 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 255 B/s rd, 59 MiB/s wr, 939 op/s; 1/381459 objects misplaced (0.000%) 2024-02-05T07:35:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:19 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-05T07:35:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:19 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:35:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:19 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-05T07:35:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:19 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:35:20.293 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:20.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:20.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:20.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:20.422 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:20.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:20.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:20.550 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:20.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:20.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:20.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:20.675 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:20.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:20.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:20.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:20.804 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:20.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:20.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:20.930 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:20.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:20.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:21.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:21.045 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-05T07:35:21.045 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T07:35:21.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[29098]: osdmap e495: 8 total, 8 up, 6 in 2024-02-05T07:35:21.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[29098]: 10.2 scrub starts 2024-02-05T07:35:21.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[29098]: 10.0 scrub starts 2024-02-05T07:35:21.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[29098]: 10.1 scrub ok 2024-02-05T07:35:21.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[29098]: 10.4 scrub starts 2024-02-05T07:35:21.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[29098]: 10.7 scrub ok 2024-02-05T07:35:21.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[29098]: 10.3 scrub ok 2024-02-05T07:35:21.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[29098]: 10.8 scrub starts 2024-02-05T07:35:21.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[29098]: 10.0 scrub ok 2024-02-05T07:35:21.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[29098]: 10.4 scrub ok 2024-02-05T07:35:21.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-05T07:35:21.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T07:35:21.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[37158]: osdmap e495: 8 total, 8 up, 6 in 2024-02-05T07:35:21.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[37158]: 10.2 scrub starts 2024-02-05T07:35:21.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[37158]: 10.0 scrub starts 2024-02-05T07:35:21.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[37158]: 10.1 scrub ok 2024-02-05T07:35:21.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[37158]: 10.4 scrub starts 2024-02-05T07:35:21.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[37158]: 10.7 scrub ok 2024-02-05T07:35:21.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[37158]: 10.3 scrub ok 2024-02-05T07:35:21.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[37158]: 10.8 scrub starts 2024-02-05T07:35:21.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[37158]: 10.0 scrub ok 2024-02-05T07:35:21.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:20 smithi007 ceph-mon[37158]: 10.4 scrub ok 2024-02-05T07:35:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:21.056 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:21.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:21.157 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:21.182 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:21.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:21.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:35:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:35:21.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:20 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-05T07:35:21.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:20 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T07:35:21.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:20 smithi090 ceph-mon[33213]: osdmap e495: 8 total, 8 up, 6 in 2024-02-05T07:35:21.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:20 smithi090 ceph-mon[33213]: 10.2 scrub starts 2024-02-05T07:35:21.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:20 smithi090 ceph-mon[33213]: 10.0 scrub starts 2024-02-05T07:35:21.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:20 smithi090 ceph-mon[33213]: 10.1 scrub ok 2024-02-05T07:35:21.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:20 smithi090 ceph-mon[33213]: 10.4 scrub starts 2024-02-05T07:35:21.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:20 smithi090 ceph-mon[33213]: 10.7 scrub ok 2024-02-05T07:35:21.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:20 smithi090 ceph-mon[33213]: 10.3 scrub ok 2024-02-05T07:35:21.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:20 smithi090 ceph-mon[33213]: 10.8 scrub starts 2024-02-05T07:35:21.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:20 smithi090 ceph-mon[33213]: 10.0 scrub ok 2024-02-05T07:35:21.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:20 smithi090 ceph-mon[33213]: 10.4 scrub ok 2024-02-05T07:35:21.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:21.311 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:21.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:21.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:21.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:21.436 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:21.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:21.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:21.562 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:21.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:21.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:21.588 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:35:21.588 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:35:21.664 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:21.687 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:21.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:21.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:21.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:21.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:21.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:21.974 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:21.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:21.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:21.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[29098]: 10.2 scrub ok 2024-02-05T07:35:21.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[29098]: 10.8 scrub ok 2024-02-05T07:35:21.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[29098]: osdmap e496: 8 total, 8 up, 6 in 2024-02-05T07:35:21.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:21.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[29098]: 10.6 scrub starts 2024-02-05T07:35:21.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[29098]: 10.9 scrub starts 2024-02-05T07:35:21.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[29098]: pgmap v890: 174 pgs: 4 remapped+peering, 3 active+clean+scrubbing, 2 peering, 6 unknown, 2 active+remapped+backfill_toofull, 157 active+clean; 7.1 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 1/348675 objects misplaced (0.000%) 2024-02-05T07:35:21.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[29098]: 10.6 scrub ok 2024-02-05T07:35:21.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[29098]: osdmap e497: 8 total, 8 up, 6 in 2024-02-05T07:35:21.995 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[37158]: 10.2 scrub ok 2024-02-05T07:35:21.995 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[37158]: 10.8 scrub ok 2024-02-05T07:35:21.995 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[37158]: osdmap e496: 8 total, 8 up, 6 in 2024-02-05T07:35:21.995 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:21.995 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[37158]: 10.6 scrub starts 2024-02-05T07:35:21.995 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[37158]: 10.9 scrub starts 2024-02-05T07:35:21.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[37158]: pgmap v890: 174 pgs: 4 remapped+peering, 3 active+clean+scrubbing, 2 peering, 6 unknown, 2 active+remapped+backfill_toofull, 157 active+clean; 7.1 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 1/348675 objects misplaced (0.000%) 2024-02-05T07:35:21.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[37158]: 10.6 scrub ok 2024-02-05T07:35:21.996 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:21 smithi007 ceph-mon[37158]: osdmap e497: 8 total, 8 up, 6 in 2024-02-05T07:35:22.076 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:22.120 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:22.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:22.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:22.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:22.250 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:22.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:22.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:22.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:21 smithi090 ceph-mon[33213]: 10.2 scrub ok 2024-02-05T07:35:22.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:21 smithi090 ceph-mon[33213]: 10.8 scrub ok 2024-02-05T07:35:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:21 smithi090 ceph-mon[33213]: osdmap e496: 8 total, 8 up, 6 in 2024-02-05T07:35:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:21 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:21 smithi090 ceph-mon[33213]: 10.6 scrub starts 2024-02-05T07:35:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:21 smithi090 ceph-mon[33213]: 10.9 scrub starts 2024-02-05T07:35:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:21 smithi090 ceph-mon[33213]: pgmap v890: 174 pgs: 4 remapped+peering, 3 active+clean+scrubbing, 2 peering, 6 unknown, 2 active+remapped+backfill_toofull, 157 active+clean; 7.1 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 1/348675 objects misplaced (0.000%) 2024-02-05T07:35:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:21 smithi090 ceph-mon[33213]: 10.6 scrub ok 2024-02-05T07:35:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:21 smithi090 ceph-mon[33213]: osdmap e497: 8 total, 8 up, 6 in 2024-02-05T07:35:22.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:22.379 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:22.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:22.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:22.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:22.514 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:22.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:22.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:22.643 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:22.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:22.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:22.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:22.776 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:22.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:22.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:22.808 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:35:22.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:22.921 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:22.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:22.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:23.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:23.051 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:23.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:23.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.9 scrub ok 2024-02-05T07:35:23.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.10 starting backfill to osd.0 from (0'0,0'0] MAX to 494'8034 2024-02-05T07:35:23.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8016 2024-02-05T07:35:23.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8093 2024-02-05T07:35:23.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8051 2024-02-05T07:35:23.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 494'8054 2024-02-05T07:35:23.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 494'8155 2024-02-05T07:35:23.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.11 starting backfill to osd.2 from (0'0,0'0] MAX to 494'8146 2024-02-05T07:35:23.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 494'8093 2024-02-05T07:35:23.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8140 2024-02-05T07:35:23.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 494'8140 2024-02-05T07:35:23.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.12 starting backfill to osd.3 from (0'0,0'0] MAX to 494'8155 2024-02-05T07:35:23.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8108 2024-02-05T07:35:23.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3229786117' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:35:23.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:35:23.063 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 494'8146 2024-02-05T07:35:23.063 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 494'8155 2024-02-05T07:35:23.063 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 4.3 scrub starts 2024-02-05T07:35:23.063 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 494'8104 2024-02-05T07:35:23.063 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 4.3 scrub ok 2024-02-05T07:35:23.063 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 5.3 scrub starts 2024-02-05T07:35:23.063 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: 5.3 scrub ok 2024-02-05T07:35:23.063 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:35:23.063 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:35:23.063 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[29098]: osdmap e498: 8 total, 8 up, 6 in 2024-02-05T07:35:23.063 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.9 scrub ok 2024-02-05T07:35:23.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.10 starting backfill to osd.0 from (0'0,0'0] MAX to 494'8034 2024-02-05T07:35:23.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8016 2024-02-05T07:35:23.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8093 2024-02-05T07:35:23.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8051 2024-02-05T07:35:23.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 494'8054 2024-02-05T07:35:23.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 494'8155 2024-02-05T07:35:23.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.11 starting backfill to osd.2 from (0'0,0'0] MAX to 494'8146 2024-02-05T07:35:23.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 494'8093 2024-02-05T07:35:23.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8140 2024-02-05T07:35:23.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 494'8140 2024-02-05T07:35:23.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.12 starting backfill to osd.3 from (0'0,0'0] MAX to 494'8155 2024-02-05T07:35:23.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8108 2024-02-05T07:35:23.064 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3229786117' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:35:23.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:35:23.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 494'8146 2024-02-05T07:35:23.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 494'8155 2024-02-05T07:35:23.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 4.3 scrub starts 2024-02-05T07:35:23.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 494'8104 2024-02-05T07:35:23.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 4.3 scrub ok 2024-02-05T07:35:23.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 5.3 scrub starts 2024-02-05T07:35:23.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: 5.3 scrub ok 2024-02-05T07:35:23.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:35:23.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:35:23.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:22 smithi007 ceph-mon[37158]: osdmap e498: 8 total, 8 up, 6 in 2024-02-05T07:35:23.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:23.181 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:23.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:23.274 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=sqlstore.transactions t=2024-02-05T07:35:22.798312662Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-05T07:35:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.9 scrub ok 2024-02-05T07:35:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.10 starting backfill to osd.0 from (0'0,0'0] MAX to 494'8034 2024-02-05T07:35:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8016 2024-02-05T07:35:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8093 2024-02-05T07:35:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8051 2024-02-05T07:35:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 494'8054 2024-02-05T07:35:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 494'8155 2024-02-05T07:35:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.11 starting backfill to osd.2 from (0'0,0'0] MAX to 494'8146 2024-02-05T07:35:23.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 494'8093 2024-02-05T07:35:23.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8140 2024-02-05T07:35:23.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 494'8140 2024-02-05T07:35:23.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.12 starting backfill to osd.3 from (0'0,0'0] MAX to 494'8155 2024-02-05T07:35:23.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 494'8108 2024-02-05T07:35:23.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3229786117' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:35:23.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:35:23.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 494'8146 2024-02-05T07:35:23.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 494'8155 2024-02-05T07:35:23.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 4.3 scrub starts 2024-02-05T07:35:23.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 494'8104 2024-02-05T07:35:23.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 4.3 scrub ok 2024-02-05T07:35:23.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 5.3 scrub starts 2024-02-05T07:35:23.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: 5.3 scrub ok 2024-02-05T07:35:23.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:35:23.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:35:23.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:22 smithi090 ceph-mon[33213]: osdmap e498: 8 total, 8 up, 6 in 2024-02-05T07:35:23.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:23.321 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:23.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:23.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:23.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:23.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:23.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:23.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:23.595 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:23.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:23.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:23.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:35:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:35:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:35:23.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:23.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:23.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:23.843 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:23.872 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:23.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:23.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:24.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:24.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:24.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:23 smithi007 ceph-mon[29098]: pgmap v893: 173 pgs: 1 activating+remapped, 4 remapped+peering, 3 active+clean+scrubbing, 2 peering, 5 unknown, 2 active+remapped+backfill_toofull, 156 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 3921/360435 objects misplaced (1.088%) 2024-02-05T07:35:24.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:23 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-05T07:35:24.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:23 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-05T07:35:24.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:23 smithi007 ceph-mon[37158]: pgmap v893: 173 pgs: 1 activating+remapped, 4 remapped+peering, 3 active+clean+scrubbing, 2 peering, 5 unknown, 2 active+remapped+backfill_toofull, 156 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 3921/360435 objects misplaced (1.088%) 2024-02-05T07:35:24.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:23 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-05T07:35:24.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:23 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-05T07:35:24.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:24.146 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:24.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:24.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:24.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:24.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:23 smithi090 ceph-mon[33213]: pgmap v893: 173 pgs: 1 activating+remapped, 4 remapped+peering, 3 active+clean+scrubbing, 2 peering, 5 unknown, 2 active+remapped+backfill_toofull, 156 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 3921/360435 objects misplaced (1.088%) 2024-02-05T07:35:24.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:23 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-05T07:35:24.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:23 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-05T07:35:24.294 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:24.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:24.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:24.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:24.439 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:24.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:24.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:24.594 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:24.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:24.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:24.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:24.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:24.825 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:24.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:24.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:24.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:25.002 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:25.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:25.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:25.130 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:25.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:25.139 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:24 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-02-05T07:35:25.139 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:24 smithi007 ceph-mon[29098]: osdmap e499: 8 total, 8 up, 6 in 2024-02-05T07:35:25.139 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:24 smithi007 ceph-mon[29098]: osdmap e500: 8 total, 8 up, 6 in 2024-02-05T07:35:25.139 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:24 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-02-05T07:35:25.139 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:24 smithi007 ceph-mon[37158]: osdmap e499: 8 total, 8 up, 6 in 2024-02-05T07:35:25.139 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:24 smithi007 ceph-mon[37158]: osdmap e500: 8 total, 8 up, 6 in 2024-02-05T07:35:25.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:25.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:24 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-02-05T07:35:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:24 smithi090 ceph-mon[33213]: osdmap e499: 8 total, 8 up, 6 in 2024-02-05T07:35:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:24 smithi090 ceph-mon[33213]: osdmap e500: 8 total, 8 up, 6 in 2024-02-05T07:35:25.276 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:25.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:25.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:25.378 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:25.407 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:25.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:25.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:25.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:25.536 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:25.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:25.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:25.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:25.665 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:25.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:25.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:25.795 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:25.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:25.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:25.838 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:35:25.838 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:35:25.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:25.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:25.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:26.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:26.049 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 3] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:35:26.049 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:35:26.049 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-05T07:35:26.049 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-02-05T07:35:26.050 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 0 0 2024-02-05T07:35:26.069 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:26.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:26.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:26.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:26.235 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:26.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:26.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:26.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:26.364 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:26.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:26.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:26.495 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:26.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:26.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:26.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:26.627 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:26.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:26.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:26.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:26.758 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:26.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:26.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:26.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:26 smithi007 ceph-mon[29098]: pgmap v896: 173 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 1 activating+remapped, 4 remapped+peering, 1 active+clean+scrubbing, 1 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 22083/377544 objects misplaced (5.849%); 17 MiB/s, 266 objects/s recovering 2024-02-05T07:35:26.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:26 smithi007 ceph-mon[29098]: osdmap e501: 8 total, 8 up, 6 in 2024-02-05T07:35:26.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:26 smithi007 ceph-mon[37158]: pgmap v896: 173 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 1 activating+remapped, 4 remapped+peering, 1 active+clean+scrubbing, 1 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 22083/377544 objects misplaced (5.849%); 17 MiB/s, 266 objects/s recovering 2024-02-05T07:35:26.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:26 smithi007 ceph-mon[37158]: osdmap e501: 8 total, 8 up, 6 in 2024-02-05T07:35:26.917 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:35:26.928 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:26.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:26.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:26.961 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:35:27.003 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:35:27.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:26 smithi090 ceph-mon[33213]: pgmap v896: 173 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 1 activating+remapped, 4 remapped+peering, 1 active+clean+scrubbing, 1 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 22083/377544 objects misplaced (5.849%); 17 MiB/s, 266 objects/s recovering 2024-02-05T07:35:27.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:26 smithi090 ceph-mon[33213]: osdmap e501: 8 total, 8 up, 6 in 2024-02-05T07:35:27.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:27.047 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:35:27.058 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:27.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:27.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:27.080 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:35:27.095 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:35:27.146 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:35:27.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:27.187 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:35:27.188 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:27.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:27.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:27.235 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:35:27.246 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:35:27.246 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:35:27.246 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:35:27.246 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:35:27.246 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:35:27.246 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:35:27.247 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:35:27.247 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:35:27.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:27.316 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:27.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:27.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:27.471 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:27.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:27.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:27.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:27.604 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:27.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:27.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:27.676 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T07:35:27.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:27.779 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:27.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:27.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:27.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:27.912 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:27.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:27.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:27.921 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:27 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:35:27.921 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:27 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:35:27.921 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:27 smithi090 ceph-mon[33213]: osdmap e502: 8 total, 8 up, 6 in 2024-02-05T07:35:27.921 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:27 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2072469531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:35:27.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:27 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:35:27.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:27 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:35:27.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:27 smithi007 ceph-mon[29098]: osdmap e502: 8 total, 8 up, 6 in 2024-02-05T07:35:27.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:27 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2072469531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:35:27.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:27 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:35:27.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:27 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:35:27.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:27 smithi007 ceph-mon[37158]: osdmap e502: 8 total, 8 up, 6 in 2024-02-05T07:35:27.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:27 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2072469531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:35:28.014 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:28.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:28.071 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:28.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:28.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:28.214 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:28.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:28.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:28.316 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:28.354 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:28.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:28.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:28.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:28.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:28.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:28.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:28.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:28.633 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:28.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:28.734 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:28.737 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:35:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:35:28.761 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:28.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:28.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:28.791 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:35:28.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:28.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:28 smithi007 ceph-mon[29098]: pgmap v899: 172 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 160 active+clean; 7.5 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 55039/371208 objects misplaced (14.827%); 37 MiB/s, 596 objects/s recovering 2024-02-05T07:35:28.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:28 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:35:28.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:28 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:35:28.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:28 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:28.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:28 smithi007 ceph-mon[29098]: osdmap e503: 8 total, 8 up, 6 in 2024-02-05T07:35:28.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:28 smithi007 ceph-mon[37158]: pgmap v899: 172 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 160 active+clean; 7.5 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 55039/371208 objects misplaced (14.827%); 37 MiB/s, 596 objects/s recovering 2024-02-05T07:35:28.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:28 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:35:28.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:28 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:35:28.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:28 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:28.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:28 smithi007 ceph-mon[37158]: osdmap e503: 8 total, 8 up, 6 in 2024-02-05T07:35:28.894 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:28.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:28.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:29.019 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:29.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:29.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:28 smithi090 ceph-mon[33213]: pgmap v899: 172 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 160 active+clean; 7.5 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 55039/371208 objects misplaced (14.827%); 37 MiB/s, 596 objects/s recovering 2024-02-05T07:35:29.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:28 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:35:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:28 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:35:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:28 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:28 smithi090 ceph-mon[33213]: osdmap e503: 8 total, 8 up, 6 in 2024-02-05T07:35:29.121 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:29.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:29.160 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:29.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:29.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:29.291 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:29.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:29.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:29.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:29.424 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:29.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:29.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:29.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:29.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:29.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:29.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:29.685 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:29.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:29.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:29.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:29.843 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:29.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:29.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:29.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:29.989 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:29.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:29.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:30.091 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:30.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:30.122 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:30.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:30.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:30.253 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:30.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:30.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:30.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:30.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:30.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:30.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:30.526 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:30.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:30.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:30.656 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:30.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:30.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:30.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:35:30.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:30 smithi007 ceph-mon[29098]: pgmap v901: 172 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 160 active+clean; 7.4 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 866 B/s rd, 0 op/s; 55039/365346 objects misplaced (15.065%); 32 MiB/s, 505 objects/s recovering 2024-02-05T07:35:30.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:30 smithi007 ceph-mon[29098]: osdmap e504: 8 total, 8 up, 6 in 2024-02-05T07:35:30.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:30 smithi007 ceph-mon[37158]: pgmap v901: 172 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 160 active+clean; 7.4 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 866 B/s rd, 0 op/s; 55039/365346 objects misplaced (15.065%); 32 MiB/s, 505 objects/s recovering 2024-02-05T07:35:30.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:30 smithi007 ceph-mon[37158]: osdmap e504: 8 total, 8 up, 6 in 2024-02-05T07:35:30.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:30.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:30.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:30.923 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:30 smithi090 ceph-mon[33213]: pgmap v901: 172 pgs: 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 160 active+clean; 7.4 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 866 B/s rd, 0 op/s; 55039/365346 objects misplaced (15.065%); 32 MiB/s, 505 objects/s recovering 2024-02-05T07:35:30.923 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:30 smithi090 ceph-mon[33213]: osdmap e504: 8 total, 8 up, 6 in 2024-02-05T07:35:30.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:30.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:30.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:31.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:31.065 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:31.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:31.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:31.198 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:31.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:31.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:31.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:35:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:35:31.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:31.339 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:31.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:31.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:31.466 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:31.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:31.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:31.608 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:31.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:31.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:31.734 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:31.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:31.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:31.879 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:31.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:31.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:31 smithi007 ceph-mon[29098]: osdmap e505: 8 total, 8 up, 6 in 2024-02-05T07:35:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:31 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:31 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:35:31.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:31 smithi007 ceph-mon[37158]: osdmap e505: 8 total, 8 up, 6 in 2024-02-05T07:35:31.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:31 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:31.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:31 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:35:31.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:32.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:32.006 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:32.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:31 smithi090 ceph-mon[33213]: osdmap e505: 8 total, 8 up, 6 in 2024-02-05T07:35:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:31 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:31 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:35:32.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:32.136 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:32.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:32.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:32.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:32.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:32.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:32.393 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:32.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:32.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:32.525 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:32.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:32.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:32.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:32.670 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:32.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:32.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:32.772 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:32.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:32.820 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:32.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:32.921 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:32 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:32 smithi007 ceph-mon[29098]: pgmap v904: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 7.2 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.28k op/s; 50102/353703 objects misplaced (14.165%); 41 MiB/s, 652 objects/s recovering 2024-02-05T07:35:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:32 smithi007 ceph-mon[29098]: osdmap e506: 8 total, 8 up, 6 in 2024-02-05T07:35:32.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:32 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:32 smithi007 ceph-mon[37158]: pgmap v904: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 7.2 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.28k op/s; 50102/353703 objects misplaced (14.165%); 41 MiB/s, 652 objects/s recovering 2024-02-05T07:35:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:32 smithi007 ceph-mon[37158]: osdmap e506: 8 total, 8 up, 6 in 2024-02-05T07:35:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:32.953 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:32.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:33.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:32 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:33.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:32 smithi090 ceph-mon[33213]: pgmap v904: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 7.2 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.28k op/s; 50102/353703 objects misplaced (14.165%); 41 MiB/s, 652 objects/s recovering 2024-02-05T07:35:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:32 smithi090 ceph-mon[33213]: osdmap e506: 8 total, 8 up, 6 in 2024-02-05T07:35:33.054 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:33.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:33.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:33.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:33.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:33.217 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:33.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:33.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:33.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:33.359 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:33.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:33.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:33.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:33.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:33.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:33.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:33.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:33.633 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:33.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:33.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:33.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:33.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:33.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:33.908 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:33.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:33.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:35:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:35:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:35:33.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:33 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:35:33.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:33 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:35:33.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:33 smithi007 ceph-mon[29098]: osdmap e507: 8 total, 8 up, 6 in 2024-02-05T07:35:33.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:33 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:35:33.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:33 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:35:33.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:33 smithi007 ceph-mon[37158]: osdmap e507: 8 total, 8 up, 6 in 2024-02-05T07:35:34.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:34.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:33 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:35:34.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:33 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:35:34.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:33 smithi090 ceph-mon[33213]: osdmap e507: 8 total, 8 up, 6 in 2024-02-05T07:35:34.052 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:34.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:34.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:34.184 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:34.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:34.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:34.326 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:34.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:34.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:34.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:34.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:34.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:34.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:34.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:34.739 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:34.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:34.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:34.883 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:34.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:34.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:34 smithi007 ceph-mon[29098]: pgmap v907: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 7.2 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 49299/353193 objects misplaced (13.958%); 48 MiB/s, 774 objects/s recovering 2024-02-05T07:35:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:34 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:34 smithi007 ceph-mon[29098]: osdmap e508: 8 total, 8 up, 6 in 2024-02-05T07:35:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:34 smithi007 ceph-mon[37158]: pgmap v907: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 7.2 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 49299/353193 objects misplaced (13.958%); 48 MiB/s, 774 objects/s recovering 2024-02-05T07:35:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:34 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:34.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:34 smithi007 ceph-mon[37158]: osdmap e508: 8 total, 8 up, 6 in 2024-02-05T07:35:34.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:35.014 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:35.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:35.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:35.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:34 smithi090 ceph-mon[33213]: pgmap v907: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 7.2 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 49299/353193 objects misplaced (13.958%); 48 MiB/s, 774 objects/s recovering 2024-02-05T07:35:35.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:34 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:34 smithi090 ceph-mon[33213]: osdmap e508: 8 total, 8 up, 6 in 2024-02-05T07:35:35.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:35.139 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:35.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:35.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:35.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:35.275 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:35.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:35.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:35.406 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:35.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:35.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:35.508 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:35.537 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:35.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:35.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:35.639 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:35.669 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:35.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:35.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:35.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:35.813 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:35.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:35.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:35.944 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:35.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:36.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:36.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:36.086 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:36.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:36.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:36.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:36.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:36.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:36.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:36.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:36.485 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:36.511 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:36.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:36.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:36.639 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:36.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:36.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:36.768 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:36.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:36.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:36.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:36.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:36 smithi007 ceph-mon[29098]: pgmap v909: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 7.0 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 906 op/s; 47189/343251 objects misplaced (13.748%); 39 MiB/s, 628 objects/s recovering 2024-02-05T07:35:36.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:36 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:35:36.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:36 smithi007 ceph-mon[37158]: pgmap v909: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 7.0 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 906 op/s; 47189/343251 objects misplaced (13.748%); 39 MiB/s, 628 objects/s recovering 2024-02-05T07:35:36.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:36 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:35:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:36.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:36.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:37.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:36 smithi090 ceph-mon[33213]: pgmap v909: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 7.0 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 906 op/s; 47189/343251 objects misplaced (13.748%); 39 MiB/s, 628 objects/s recovering 2024-02-05T07:35:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:36 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:35:37.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:37.084 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:37.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:37.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:37.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:37.254 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:37.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:37.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:37.356 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:37.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:37.403 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:37.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:37.504 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:37.554 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:37.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:37.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:37.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:37.707 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:37.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:37.808 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:37.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:37.848 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:37.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:37.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:37.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:37.973 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:37.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:38.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:38.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:38.108 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:38.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:38.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:38.255 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:38.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:38.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:38.356 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:38.400 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:38.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:38.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:38.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:38 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:38.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:38 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:38.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:38 smithi007 ceph-mon[29098]: pgmap v910: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 956 op/s; 44313/336492 objects misplaced (13.169%); 37 MiB/s, 587 objects/s recovering 2024-02-05T07:35:38.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:38 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:38.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:38 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:38.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:38 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:38.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:38 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:38.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:38 smithi007 ceph-mon[37158]: pgmap v910: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 956 op/s; 44313/336492 objects misplaced (13.169%); 37 MiB/s, 587 objects/s recovering 2024-02-05T07:35:38.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:38 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:38.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:38 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:38.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:38.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:38 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:38.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:38 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:38.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:38 smithi090 ceph-mon[33213]: pgmap v910: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 956 op/s; 44313/336492 objects misplaced (13.169%); 37 MiB/s, 587 objects/s recovering 2024-02-05T07:35:38.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:38 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:38.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:38 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:38.535 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:38.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:38.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:38.665 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:38.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:38.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:38.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:38.829 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:38.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:38.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:38.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:38.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:38.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:39.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:39.088 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:39.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:39.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:39.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:39.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:39.219 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:39.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:39.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:39.347 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:39.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:39.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:39.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:39 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:35:39.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:39 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:35:39.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:39 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:39.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:39 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:35:39.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:39 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:35:39.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:39 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:39.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:39.491 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:39.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:39.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:39.523 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:39 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:35:39.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:39 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:35:39.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:39 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:35:39.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:39.631 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:39.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:39.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:39.733 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:39.762 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:39.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:39.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:39.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:39.908 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:39.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:39.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:40.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:40.038 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:40.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:40.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:40.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:40.170 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:40.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:40.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:40.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:40.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:40.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:40.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:40.425 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:40.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:40.527 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:40.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:40.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:40.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:40.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:40.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:40.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:40.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:40.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:40.810 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:40.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:40.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:40.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:40 smithi007 ceph-mon[29098]: pgmap v911: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 666 B/s rd, 0 B/s wr, 960 op/s; 44313/331581 objects misplaced (13.364%); 29 MiB/s, 459 objects/s recovering 2024-02-05T07:35:40.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:40 smithi007 ceph-mon[37158]: pgmap v911: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 666 B/s rd, 0 B/s wr, 960 op/s; 44313/331581 objects misplaced (13.364%); 29 MiB/s, 459 objects/s recovering 2024-02-05T07:35:40.933 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:40 smithi090 ceph-mon[33213]: pgmap v911: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 666 B/s rd, 0 B/s wr, 960 op/s; 44313/331581 objects misplaced (13.364%); 29 MiB/s, 459 objects/s recovering 2024-02-05T07:35:40.939 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:40.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:40.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:41.041 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:41.071 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:41.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:41.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:41.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:41.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:41.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:41.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:41.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:35:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:35:41.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:41.335 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:41.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:41.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:41.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:41.477 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:41.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:41.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:41.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:41.605 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:41.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:41.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:41.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:41.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:41.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:41.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:41.869 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:41.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:41.970 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:42.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:42.010 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:42.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:42.087 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:35:42.087 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-02-05T07:35:42.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-02-05T07:35:42.087 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:35:42.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.087 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151878 8.18 8:1b9df18c:::benchmark_data_smithi007_105755_object15568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:35:42.087 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:15.617665+0000", 2024-02-05T07:35:42.088 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 505.96077644500002, 2024-02-05T07:35:42.088 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.218317409, 2024-02-05T07:35:42.088 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.088 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:35:42.088 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:35:42.088 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:35:42.088 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:35:42.088 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151878 2024-02-05T07:35:42.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.088 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.088 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.088 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.617665+0000", 2024-02-05T07:35:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.617665+0000", 2024-02-05T07:35:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.617667+0000", 2024-02-05T07:35:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.567e-06 2024-02-05T07:35:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.090 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.617673+0000", 2024-02-05T07:35:42.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2530000000000001e-06 2024-02-05T07:35:42.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.090 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.090 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.617684+0000", 2024-02-05T07:35:42.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1e-05 2024-02-05T07:35:42.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.091 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.091 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.617692+0000", 2024-02-05T07:35:42.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9240000000000007e-06 2024-02-05T07:35:42.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.091 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.091 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.618154+0000", 2024-02-05T07:35:42.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00046236600000000002 2024-02-05T07:35:42.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.618283+0000", 2024-02-05T07:35:42.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012861 2024-02-05T07:35:42.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,5,7", 2024-02-05T07:35:42.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.618581+0000", 2024-02-05T07:35:42.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029788500000000001 2024-02-05T07:35:42.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:35:42.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.619902+0000", 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001321521 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621361+0000", 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001459019 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.704079+0000", 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.082718268999999997 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.736869+0000", 2024-02-05T07:35:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.032790080999999999 2024-02-05T07:35:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:35:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.736883+0000", 2024-02-05T07:35:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3345e-05 2024-02-05T07:35:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.835934+0000", 2024-02-05T07:35:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.099051094 2024-02-05T07:35:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.835982+0000", 2024-02-05T07:35:42.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8474999999999999e-05 2024-02-05T07:35:42.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.095 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.095 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151885 8.18 8:18501e19:::benchmark_data_smithi007_105755_object15575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:35:42.095 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:15.621172+0000", 2024-02-05T07:35:42.095 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 505.95726979199998, 2024-02-05T07:35:42.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.375381483, 2024-02-05T07:35:42.095 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.095 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:35:42.095 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151885 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621172+0000", 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621172+0000", 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621173+0000", 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3069999999999999e-06 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621211+0000", 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7966000000000002e-05 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621228+0000", 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.685e-05 2024-02-05T07:35:42.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621236+0000", 2024-02-05T07:35:42.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3129999999999999e-06 2024-02-05T07:35:42.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621393+0000", 2024-02-05T07:35:42.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015687599999999999 2024-02-05T07:35:42.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.621526+0000", 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013271100000000001 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,5,7", 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.702361+0000", 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.080835844000000004 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.705711+0000", 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033500320000000002 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.736929+0000", 2024-02-05T07:35:42.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.03121765 2024-02-05T07:35:42.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:35:42.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.736939+0000", 2024-02-05T07:35:42.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7610000000000002e-06 2024-02-05T07:35:42.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.888509+0000", 2024-02-05T07:35:42.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1515699399999999 2024-02-05T07:35:42.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for missing object", 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.888700+0000", 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000190903 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.996413+0000", 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10771344099999999 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.996553+0000", 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000139889 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.101 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.102 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151889 8.9 8:97173e06:::benchmark_data_smithi007_105755_object15579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:35:42.102 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:15.708625+0000", 2024-02-05T07:35:42.102 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 505.86981614699999, 2024-02-05T07:35:42.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1227700549999999, 2024-02-05T07:35:42.102 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.102 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.102 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:35:42.102 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:35:42.102 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:35:42.102 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151889 2024-02-05T07:35:42.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.103 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.103 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.103 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708625+0000", 2024-02-05T07:35:42.103 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.103 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.103 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708625+0000", 2024-02-05T07:35:42.103 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.103 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.103 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708627+0000", 2024-02-05T07:35:42.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6530000000000001e-06 2024-02-05T07:35:42.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708630+0000", 2024-02-05T07:35:42.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7609999999999999e-06 2024-02-05T07:35:42.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708638+0000", 2024-02-05T07:35:42.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3839999999999997e-06 2024-02-05T07:35:42.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708645+0000", 2024-02-05T07:35:42.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9190000000000002e-06 2024-02-05T07:35:42.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.721400+0000", 2024-02-05T07:35:42.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012754557999999999 2024-02-05T07:35:42.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:35:42.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.721411+0000", 2024-02-05T07:35:42.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1426999999999999e-05 2024-02-05T07:35:42.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.826103+0000", 2024-02-05T07:35:42.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.104692137 2024-02-05T07:35:42.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.826226+0000", 2024-02-05T07:35:42.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012312 2024-02-05T07:35:42.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,5", 2024-02-05T07:35:42.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.826614+0000", 2024-02-05T07:35:42.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00038772999999999998 2024-02-05T07:35:42.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:35:42.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.828167+0000", 2024-02-05T07:35:42.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001553432 2024-02-05T07:35:42.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.828629+0000", 2024-02-05T07:35:42.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00046157499999999998 2024-02-05T07:35:42.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.831136+0000", 2024-02-05T07:35:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025074310000000001 2024-02-05T07:35:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.831256+0000", 2024-02-05T07:35:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000119261 2024-02-05T07:35:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.831395+0000", 2024-02-05T07:35:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000139667 2024-02-05T07:35:42.108 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.109 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.109 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.109 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151899 8.16 8:69ded5dd:::benchmark_data_smithi007_105755_object15589:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-02-05T07:35:42.109 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:15.708654+0000", 2024-02-05T07:35:42.109 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 505.869787143, 2024-02-05T07:35:42.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1801824700000001, 2024-02-05T07:35:42.109 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.109 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.109 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:35:42.109 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:35:42.109 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:35:42.109 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151899 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708654+0000", 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708654+0000", 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708655+0000", 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5299999999999999e-07 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708657+0000", 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5979999999999999e-06 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708665+0000", 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4510000000000003e-06 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.708672+0000", 2024-02-05T07:35:42.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8510000000000001e-06 2024-02-05T07:35:42.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.728610+0000", 2024-02-05T07:35:42.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.019938852 2024-02-05T07:35:42.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:35:42.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.728624+0000", 2024-02-05T07:35:42.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.367e-05 2024-02-05T07:35:42.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.783950+0000", 2024-02-05T07:35:42.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0553263230000001 2024-02-05T07:35:42.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.784094+0000", 2024-02-05T07:35:42.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000143125 2024-02-05T07:35:42.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,3,7", 2024-02-05T07:35:42.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.784438+0000", 2024-02-05T07:35:42.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000344684 2024-02-05T07:35:42.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:35:42.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.787789+0000", 2024-02-05T07:35:42.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003350853 2024-02-05T07:35:42.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.794404+0000", 2024-02-05T07:35:42.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0066144910000000001 2024-02-05T07:35:42.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.796128+0000", 2024-02-05T07:35:42.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00172437 2024-02-05T07:35:42.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.888636+0000", 2024-02-05T07:35:42.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.092508198999999999 2024-02-05T07:35:42.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.888696+0000", 2024-02-05T07:35:42.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9998999999999999e-05 2024-02-05T07:35:42.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.888837+0000", 2024-02-05T07:35:42.116 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014055099999999999 2024-02-05T07:35:42.116 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.116 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.116 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.116 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:151911 8.16 8:6ed6f866:::benchmark_data_smithi007_105755_object15601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-02-05T07:35:42.116 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:27:15.723119+0000", 2024-02-05T07:35:42.116 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 505.85532201500001, 2024-02-05T07:35:42.116 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1659391960000001, 2024-02-05T07:35:42.116 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.116 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.117 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:35:42.117 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:35:42.117 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:35:42.117 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 151911 2024-02-05T07:35:42.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.117 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.117 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.723119+0000", 2024-02-05T07:35:42.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.723119+0000", 2024-02-05T07:35:42.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.723121+0000", 2024-02-05T07:35:42.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.395e-06 2024-02-05T07:35:42.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.723125+0000", 2024-02-05T07:35:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9469999999999996e-06 2024-02-05T07:35:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.723134+0000", 2024-02-05T07:35:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3139999999999993e-06 2024-02-05T07:35:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.723143+0000", 2024-02-05T07:35:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7320000000000004e-06 2024-02-05T07:35:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.728656+0000", 2024-02-05T07:35:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0055137060000000002 2024-02-05T07:35:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:35:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:15.728662+0000", 2024-02-05T07:35:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9109999999999998e-06 2024-02-05T07:35:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.785892+0000", 2024-02-05T07:35:42.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.057230085 2024-02-05T07:35:42.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.121 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.786022+0000", 2024-02-05T07:35:42.121 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012957499999999999 2024-02-05T07:35:42.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.121 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,3,7", 2024-02-05T07:35:42.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.786266+0000", 2024-02-05T07:35:42.121 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00024366200000000001 2024-02-05T07:35:42.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.121 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:35:42.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.787796+0000", 2024-02-05T07:35:42.122 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001529835 2024-02-05T07:35:42.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.122 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.122 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.122 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.794445+0000", 2024-02-05T07:35:42.122 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0066497789999999998 2024-02-05T07:35:42.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.122 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.122 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.122 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.796168+0000", 2024-02-05T07:35:42.122 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00172317 2024-02-05T07:35:42.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.122 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.123 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.123 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.888881+0000", 2024-02-05T07:35:42.123 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.092712185000000003 2024-02-05T07:35:42.123 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.123 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.123 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.888938+0000", 2024-02-05T07:35:42.123 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7420000000000003e-05 2024-02-05T07:35:42.123 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.123 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.123 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:27:16.889059+0000", 2024-02-05T07:35:42.123 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012048 2024-02-05T07:35:42.124 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.124 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.124 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.124 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:142776 10.12 10:4ee87194:::benchmark_data_smithi007_157098_object13846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-02-05T07:35:42.124 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:31.907061+0000", 2024-02-05T07:35:42.124 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.6713799720000004, 2024-02-05T07:35:42.124 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.15728530499999999, 2024-02-05T07:35:42.124 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.124 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.124 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:35:42.125 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:35:42.125 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:35:42.125 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 142776 2024-02-05T07:35:42.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.125 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.125 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.125 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907061+0000", 2024-02-05T07:35:42.125 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.125 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.125 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907061+0000", 2024-02-05T07:35:42.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907063+0000", 2024-02-05T07:35:42.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6020000000000001e-06 2024-02-05T07:35:42.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907068+0000", 2024-02-05T07:35:42.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.434e-06 2024-02-05T07:35:42.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.127 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907080+0000", 2024-02-05T07:35:42.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1977000000000001e-05 2024-02-05T07:35:42.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.127 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907089+0000", 2024-02-05T07:35:42.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1389999999999995e-06 2024-02-05T07:35:42.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.127 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907798+0000", 2024-02-05T07:35:42.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00070967800000000004 2024-02-05T07:35:42.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.128 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.128 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907937+0000", 2024-02-05T07:35:42.128 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013838899999999999 2024-02-05T07:35:42.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.128 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,3,4,6", 2024-02-05T07:35:42.128 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.908227+0000", 2024-02-05T07:35:42.128 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028989700000000001 2024-02-05T07:35:42.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.128 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:35:42.128 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.909450+0000", 2024-02-05T07:35:42.128 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012234520000000001 2024-02-05T07:35:42.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.129 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.129 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.913246+0000", 2024-02-05T07:35:42.129 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037962099999999999 2024-02-05T07:35:42.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.129 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.129 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.913297+0000", 2024-02-05T07:35:42.129 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0652999999999998e-05 2024-02-05T07:35:42.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.129 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.129 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.913341+0000", 2024-02-05T07:35:42.130 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4206999999999999e-05 2024-02-05T07:35:42.130 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.130 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.130 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.130 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.913805+0000", 2024-02-05T07:35:42.130 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00046370599999999998 2024-02-05T07:35:42.130 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.130 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.130 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.130 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.063525+0000", 2024-02-05T07:35:42.130 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14971975700000001 2024-02-05T07:35:42.130 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.130 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.131 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.131 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.064187+0000", 2024-02-05T07:35:42.131 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00066257000000000002 2024-02-05T07:35:42.131 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.131 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.131 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.131 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.064347+0000", 2024-02-05T07:35:42.131 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000159634 2024-02-05T07:35:42.131 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.131 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.131 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.131 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.131 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.131 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:142777 10.12 10:4f2b3996:::benchmark_data_smithi007_157098_object13847:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-02-05T07:35:42.132 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:31.910713+0000", 2024-02-05T07:35:42.132 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.6677287659999998, 2024-02-05T07:35:42.132 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16056696200000001, 2024-02-05T07:35:42.132 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.132 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.132 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:35:42.132 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:35:42.132 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:35:42.132 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 142777 2024-02-05T07:35:42.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.132 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.132 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.910713+0000", 2024-02-05T07:35:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.910713+0000", 2024-02-05T07:35:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.910714+0000", 2024-02-05T07:35:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.378e-06 2024-02-05T07:35:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.910721+0000", 2024-02-05T07:35:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8279999999999997e-06 2024-02-05T07:35:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.910733+0000", 2024-02-05T07:35:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1854e-05 2024-02-05T07:35:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.910755+0000", 2024-02-05T07:35:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1997999999999998e-05 2024-02-05T07:35:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.910838+0000", 2024-02-05T07:35:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3765999999999997e-05 2024-02-05T07:35:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.911102+0000", 2024-02-05T07:35:42.136 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000263396 2024-02-05T07:35:42.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.136 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.136 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,3,4,6", 2024-02-05T07:35:42.136 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.911410+0000", 2024-02-05T07:35:42.136 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030852000000000001 2024-02-05T07:35:42.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.136 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.136 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.136 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.913445+0000", 2024-02-05T07:35:42.136 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020343739999999998 2024-02-05T07:35:42.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.137 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.137 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.913845+0000", 2024-02-05T07:35:42.137 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000400216 2024-02-05T07:35:42.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.137 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.137 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.913886+0000", 2024-02-05T07:35:42.137 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1032999999999998e-05 2024-02-05T07:35:42.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.138 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.138 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.917094+0000", 2024-02-05T07:35:42.138 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032076700000000001 2024-02-05T07:35:42.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.138 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:35:42.138 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.917112+0000", 2024-02-05T07:35:42.138 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8170000000000001e-05 2024-02-05T07:35:42.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.138 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.138 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.071032+0000", 2024-02-05T07:35:42.138 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.15391991399999999 2024-02-05T07:35:42.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.139 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.139 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.139 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.071138+0000", 2024-02-05T07:35:42.139 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010628600000000001 2024-02-05T07:35:42.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.139 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.139 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.139 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.071280+0000", 2024-02-05T07:35:42.139 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000141559 2024-02-05T07:35:42.139 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.140 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.140 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.140 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.140 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:142781 10.12 10:4dd68cc0:::benchmark_data_smithi007_157098_object13851:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-02-05T07:35:42.140 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:31.912235+0000", 2024-02-05T07:35:42.140 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.6662061470000005, 2024-02-05T07:35:42.140 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.15995722300000001, 2024-02-05T07:35:42.140 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.140 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.140 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:35:42.141 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:35:42.141 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:35:42.141 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 142781 2024-02-05T07:35:42.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.141 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.141 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.141 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.912235+0000", 2024-02-05T07:35:42.141 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.141 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.142 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.912235+0000", 2024-02-05T07:35:42.142 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.142 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.142 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.142 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.912236+0000", 2024-02-05T07:35:42.142 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.15e-06 2024-02-05T07:35:42.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.142 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.142 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.142 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.912241+0000", 2024-02-05T07:35:42.142 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7199999999999997e-06 2024-02-05T07:35:42.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.142 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.912252+0000", 2024-02-05T07:35:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1028e-05 2024-02-05T07:35:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.912260+0000", 2024-02-05T07:35:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6839999999999996e-06 2024-02-05T07:35:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.914094+0000", 2024-02-05T07:35:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0018345620000000001 2024-02-05T07:35:42.143 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.914218+0000", 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012376600000000001 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,3,4,6", 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.914480+0000", 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026203000000000001 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.917221+0000", 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002740503 2024-02-05T07:35:42.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.145 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.145 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.917384+0000", 2024-02-05T07:35:42.145 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016310399999999999 2024-02-05T07:35:42.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.145 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.145 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.917991+0000", 2024-02-05T07:35:42.145 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00060672500000000002 2024-02-05T07:35:42.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.145 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.145 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.918319+0000", 2024-02-05T07:35:42.145 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032889600000000002 2024-02-05T07:35:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.918360+0000", 2024-02-05T07:35:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0249e-05 2024-02-05T07:35:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.071995+0000", 2024-02-05T07:35:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.153635825 2024-02-05T07:35:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.072058+0000", 2024-02-05T07:35:42.147 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2402000000000003e-05 2024-02-05T07:35:42.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.147 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.147 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.072192+0000", 2024-02-05T07:35:42.147 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013457899999999999 2024-02-05T07:35:42.147 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.147 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.147 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.147 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:142793 10.d e506/423 10:b95366dc:::benchmark_data_smithi007_157098_object13863:head v 506'8875, mlcod=506'8873)", 2024-02-05T07:35:42.147 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:31.920032+0000", 2024-02-05T07:35:42.147 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.6584089409999994, 2024-02-05T07:35:42.148 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14482795200000001, 2024-02-05T07:35:42.148 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.148 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.148 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.148 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.148 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920032+0000", 2024-02-05T07:35:42.148 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.148 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.148 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920032+0000", 2024-02-05T07:35:42.149 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.149 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.149 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.149 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.149 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920033+0000", 2024-02-05T07:35:42.149 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1800000000000002e-07 2024-02-05T07:35:42.149 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.149 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.149 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.149 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920037+0000", 2024-02-05T07:35:42.149 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1579999999999998e-06 2024-02-05T07:35:42.149 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.149 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.149 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.150 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920045+0000", 2024-02-05T07:35:42.150 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5299999999999996e-06 2024-02-05T07:35:42.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.150 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.150 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920052+0000", 2024-02-05T07:35:42.150 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9979999999999997e-06 2024-02-05T07:35:42.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.150 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.150 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920106+0000", 2024-02-05T07:35:42.150 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3356000000000002e-05 2024-02-05T07:35:42.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920127+0000", 2024-02-05T07:35:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0766000000000001e-05 2024-02-05T07:35:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.064820+0000", 2024-02-05T07:35:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14469389699999999 2024-02-05T07:35:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.064860+0000", 2024-02-05T07:35:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9928999999999997e-05 2024-02-05T07:35:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "MOSDPGPushReply(10.16 506/496 [PushReplyOp(10:6b6bf244:::benchmark_data_smithi007_157098_object58135:head)])", 2024-02-05T07:35:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:31.920064+0000", 2024-02-05T07:35:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.6583773770000008, 2024-02-05T07:35:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14048513200000001, 2024-02-05T07:35:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:35:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.153 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.153 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920064+0000", 2024-02-05T07:35:42.153 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.153 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.153 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.153 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920064+0000", 2024-02-05T07:35:42.153 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.155 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920064+0000", 2024-02-05T07:35:42.156 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1100000000000002e-07 2024-02-05T07:35:42.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.156 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.156 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920067+0000", 2024-02-05T07:35:42.156 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1750000000000001e-06 2024-02-05T07:35:42.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.156 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.156 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920075+0000", 2024-02-05T07:35:42.156 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.238e-06 2024-02-05T07:35:42.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.157 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.157 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920081+0000", 2024-02-05T07:35:42.157 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2369999999999997e-06 2024-02-05T07:35:42.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.157 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.157 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920120+0000", 2024-02-05T07:35:42.157 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8860999999999999e-05 2024-02-05T07:35:42.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.157 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.158 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.060549+0000", 2024-02-05T07:35:42.158 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14042931 2024-02-05T07:35:42.158 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.158 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.158 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.158 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:142796 10.10 e506/496 10:08cf4633:::benchmark_data_smithi007_157098_object13866:head v 506'8437, mlcod=506'8435)", 2024-02-05T07:35:42.158 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:31.920963+0000", 2024-02-05T07:35:42.158 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.6574786229999994, 2024-02-05T07:35:42.158 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14140070699999999, 2024-02-05T07:35:42.158 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920963+0000", 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920963+0000", 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.159 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920963+0000", 2024-02-05T07:35:42.160 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4799999999999998e-07 2024-02-05T07:35:42.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.160 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.160 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920970+0000", 2024-02-05T07:35:42.160 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.81e-06 2024-02-05T07:35:42.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.160 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.160 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.060394+0000", 2024-02-05T07:35:42.160 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.13942379699999999 2024-02-05T07:35:42.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.060407+0000", 2024-02-05T07:35:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3054000000000001e-05 2024-02-05T07:35:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.060525+0000", 2024-02-05T07:35:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000118078 2024-02-05T07:35:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.161 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.060550+0000", 2024-02-05T07:35:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5344e-05 2024-02-05T07:35:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.062318+0000", 2024-02-05T07:35:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017671570000000001 2024-02-05T07:35:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.062363+0000", 2024-02-05T07:35:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5819000000000003e-05 2024-02-05T07:35:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:143165 10.14 10:28773d8d:::benchmark_data_smithi007_157098_object14235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-02-05T07:35:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:32.333483+0000", 2024-02-05T07:35:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.2449580190000002, 2024-02-05T07:35:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.117106743, 2024-02-05T07:35:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:35:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:35:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:35:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 143165 2024-02-05T07:35:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.333483+0000", 2024-02-05T07:35:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.333483+0000", 2024-02-05T07:35:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.333485+0000", 2024-02-05T07:35:42.164 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5370000000000001e-06 2024-02-05T07:35:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.333490+0000", 2024-02-05T07:35:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0839999999999999e-06 2024-02-05T07:35:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.333500+0000", 2024-02-05T07:35:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0448999999999999e-05 2024-02-05T07:35:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.165 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.333508+0000", 2024-02-05T07:35:42.166 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3300000000000001e-06 2024-02-05T07:35:42.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.166 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.166 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.333562+0000", 2024-02-05T07:35:42.166 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3884000000000001e-05 2024-02-05T07:35:42.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.166 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.166 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.333696+0000", 2024-02-05T07:35:42.166 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000134153 2024-02-05T07:35:42.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.167 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,3,4", 2024-02-05T07:35:42.167 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.333962+0000", 2024-02-05T07:35:42.167 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026640300000000002 2024-02-05T07:35:42.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.167 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.167 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.344608+0000", 2024-02-05T07:35:42.167 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010645738 2024-02-05T07:35:42.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.167 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.167 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.346197+0000", 2024-02-05T07:35:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0015893369999999999 2024-02-05T07:35:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.346242+0000", 2024-02-05T07:35:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4835e-05 2024-02-05T07:35:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:35:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.445044+0000", 2024-02-05T07:35:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.098801951999999998 2024-02-05T07:35:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.450185+0000", 2024-02-05T07:35:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0051408799999999996 2024-02-05T07:35:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.450462+0000", 2024-02-05T07:35:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00027722199999999998 2024-02-05T07:35:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.450590+0000", 2024-02-05T07:35:42.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012793900000000001 2024-02-05T07:35:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:143171 10.12 10:4df2cfc1:::benchmark_data_smithi007_157098_object14241:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-02-05T07:35:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:32.337386+0000", 2024-02-05T07:35:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.2410555809999995, 2024-02-05T07:35:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.107953194, 2024-02-05T07:35:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:35:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:35:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:35:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 143171 2024-02-05T07:35:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.337386+0000", 2024-02-05T07:35:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.337386+0000", 2024-02-05T07:35:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.337387+0000", 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1500000000000003e-07 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.337391+0000", 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4089999999999996e-06 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.337402+0000", 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0481e-05 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.337409+0000", 2024-02-05T07:35:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6190000000000002e-06 2024-02-05T07:35:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.337467+0000", 2024-02-05T07:35:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7451999999999998e-05 2024-02-05T07:35:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.337605+0000", 2024-02-05T07:35:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013794799999999999 2024-02-05T07:35:42.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.174 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,3,4,6", 2024-02-05T07:35:42.174 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.338054+0000", 2024-02-05T07:35:42.174 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00044913499999999997 2024-02-05T07:35:42.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.174 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.174 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.343918+0000", 2024-02-05T07:35:42.174 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0058644489999999999 2024-02-05T07:35:42.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.174 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:35:42.174 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.343933+0000", 2024-02-05T07:35:42.175 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4708999999999999e-05 2024-02-05T07:35:42.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.175 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.175 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.343973+0000", 2024-02-05T07:35:42.175 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0000999999999999e-05 2024-02-05T07:35:42.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.175 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.175 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.344019+0000", 2024-02-05T07:35:42.175 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6108e-05 2024-02-05T07:35:42.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.175 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.176 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.344064+0000", 2024-02-05T07:35:42.176 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4901999999999998e-05 2024-02-05T07:35:42.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.176 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.176 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.445133+0000", 2024-02-05T07:35:42.176 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10106955199999999 2024-02-05T07:35:42.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.176 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.176 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.445202+0000", 2024-02-05T07:35:42.176 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8107000000000004e-05 2024-02-05T07:35:42.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.177 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.177 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.445339+0000", 2024-02-05T07:35:42.177 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013740700000000001 2024-02-05T07:35:42.177 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.177 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.177 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.177 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:143170 10.c e506/423 10:30d38161:::benchmark_data_smithi007_157098_object14240:head v 506'9096, mlcod=506'9092)", 2024-02-05T07:35:42.177 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:32.338853+0000", 2024-02-05T07:35:42.177 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.2395885399999997, 2024-02-05T07:35:42.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.107519853, 2024-02-05T07:35:42.178 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.178 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.178 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.338853+0000", 2024-02-05T07:35:42.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.338853+0000", 2024-02-05T07:35:42.179 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.179 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.179 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.338854+0000", 2024-02-05T07:35:42.179 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7300000000000005e-07 2024-02-05T07:35:42.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.179 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.179 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.338860+0000", 2024-02-05T07:35:42.179 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3620000000000002e-06 2024-02-05T07:35:42.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.179 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.180 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.338908+0000", 2024-02-05T07:35:42.180 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8384000000000003e-05 2024-02-05T07:35:42.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.180 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.180 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.338916+0000", 2024-02-05T07:35:42.180 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7230000000000004e-06 2024-02-05T07:35:42.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.180 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.180 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.339404+0000", 2024-02-05T07:35:42.180 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00048768800000000001 2024-02-05T07:35:42.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.181 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.181 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.339423+0000", 2024-02-05T07:35:42.181 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9309000000000001e-05 2024-02-05T07:35:42.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.181 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.181 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.446338+0000", 2024-02-05T07:35:42.181 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.106915257 2024-02-05T07:35:42.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.181 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.182 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.446373+0000", 2024-02-05T07:35:42.182 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4357000000000003e-05 2024-02-05T07:35:42.182 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.182 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.182 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.182 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.182 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.182 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:143174 10.14 10:293b553f:::benchmark_data_smithi007_157098_object14244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-02-05T07:35:42.182 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:32.340956+0000", 2024-02-05T07:35:42.182 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.237485178, 2024-02-05T07:35:42.182 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.11849124599999999, 2024-02-05T07:35:42.182 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.182 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.183 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:35:42.183 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:35:42.183 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:35:42.183 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 143174 2024-02-05T07:35:42.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.183 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.183 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.183 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.183 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.340956+0000", 2024-02-05T07:35:42.183 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.183 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.340956+0000", 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.340958+0000", 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.99e-06 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.340964+0000", 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5049999999999998e-06 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.340975+0000", 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0883000000000001e-05 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.340982+0000", 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8970000000000008e-06 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.446429+0000", 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.105446102 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.446563+0000", 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013442900000000001 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,3,4", 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.446839+0000", 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00027634600000000001 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.450673+0000", 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038338090000000001 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:35:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.450687+0000", 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3359e-05 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.457033+0000", 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0063466870000000002 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.457085+0000", 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1332999999999998e-05 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.459239+0000", 2024-02-05T07:35:42.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021548560000000001 2024-02-05T07:35:42.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.459313+0000", 2024-02-05T07:35:42.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3559000000000005e-05 2024-02-05T07:35:42.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.459447+0000", 2024-02-05T07:35:42.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013449100000000001 2024-02-05T07:35:42.188 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.188 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.188 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.189 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:143177 10.c e506/423 10:3b1b10d1:::benchmark_data_smithi007_157098_object14247:head v 506'9097, mlcod=506'9092)", 2024-02-05T07:35:42.189 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:32.341089+0000", 2024-02-05T07:35:42.189 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.2373527210000006, 2024-02-05T07:35:42.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10530833000000001, 2024-02-05T07:35:42.189 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.189 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.189 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.341089+0000", 2024-02-05T07:35:42.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.341089+0000", 2024-02-05T07:35:42.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.341089+0000", 2024-02-05T07:35:42.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8599999999999999e-07 2024-02-05T07:35:42.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.341098+0000", 2024-02-05T07:35:42.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4740000000000005e-06 2024-02-05T07:35:42.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.191 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.191 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.341104+0000", 2024-02-05T07:35:42.191 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4200000000000004e-06 2024-02-05T07:35:42.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.191 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.191 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.341109+0000", 2024-02-05T07:35:42.191 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9690000000000002e-06 2024-02-05T07:35:42.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.191 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.191 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.346253+0000", 2024-02-05T07:35:42.192 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0051439220000000004 2024-02-05T07:35:42.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.192 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.192 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.346272+0000", 2024-02-05T07:35:42.192 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9352999999999999e-05 2024-02-05T07:35:42.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.192 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.192 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.446375+0000", 2024-02-05T07:35:42.192 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10010314200000001 2024-02-05T07:35:42.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.192 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.446397+0000", 2024-02-05T07:35:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1664000000000001e-05 2024-02-05T07:35:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:143178 10.c e506/423 10:3e17f1d7:::benchmark_data_smithi007_157098_object14248:head v 506'9098, mlcod=506'9092)", 2024-02-05T07:35:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:32.343397+0000", 2024-02-05T07:35:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.2350444740000004, 2024-02-05T07:35:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.121655178, 2024-02-05T07:35:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.193 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.343397+0000", 2024-02-05T07:35:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.343397+0000", 2024-02-05T07:35:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.343397+0000", 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0999999999999999e-07 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.343405+0000", 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1910000000000003e-06 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.343412+0000", 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6699999999999994e-06 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.343418+0000", 2024-02-05T07:35:42.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1290000000000001e-06 2024-02-05T07:35:42.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.447361+0000", 2024-02-05T07:35:42.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.103942875 2024-02-05T07:35:42.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.450896+0000", 2024-02-05T07:35:42.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003534218 2024-02-05T07:35:42.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.464378+0000", 2024-02-05T07:35:42.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.01348282 2024-02-05T07:35:42.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.465052+0000", 2024-02-05T07:35:42.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000673765 2024-02-05T07:35:42.197 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.197 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.197 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.198 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:143182 10.4 e506/493 10:26359f51:::benchmark_data_smithi007_157098_object14252:head v 506'8587, mlcod=506'8585)", 2024-02-05T07:35:42.198 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:32.345611+0000", 2024-02-05T07:35:42.198 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.2328307560000002, 2024-02-05T07:35:42.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.105020144, 2024-02-05T07:35:42.198 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.198 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.198 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.198 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.198 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.345611+0000", 2024-02-05T07:35:42.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.198 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.345611+0000", 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.345611+0000", 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3300000000000001e-07 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.345619+0000", 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.909e-06 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.345629+0000", 2024-02-05T07:35:42.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3300000000000005e-06 2024-02-05T07:35:42.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.200 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.200 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.345636+0000", 2024-02-05T07:35:42.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9070000000000001e-06 2024-02-05T07:35:42.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.200 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.200 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.445077+0000", 2024-02-05T07:35:42.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.099441019000000005 2024-02-05T07:35:42.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.445098+0000", 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1829000000000001e-05 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.450597+0000", 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054991440000000001 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.450631+0000", 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3272999999999998e-05 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.201 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.202 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.202 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:143183 10.11 10:8a619113:::benchmark_data_smithi007_157098_object14253:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-02-05T07:35:42.202 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:32.346838+0000", 2024-02-05T07:35:42.202 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.2316033389999994, 2024-02-05T07:35:42.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.112697937, 2024-02-05T07:35:42.202 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.202 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.202 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:35:42.202 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:35:42.202 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:35:42.202 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 143183 2024-02-05T07:35:42.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.203 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.203 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.346838+0000", 2024-02-05T07:35:42.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.203 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.346838+0000", 2024-02-05T07:35:42.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.204 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.346840+0000", 2024-02-05T07:35:42.204 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8700000000000001e-06 2024-02-05T07:35:42.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.204 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.204 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.346845+0000", 2024-02-05T07:35:42.204 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3159999999999996e-06 2024-02-05T07:35:42.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.204 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.204 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.446351+0000", 2024-02-05T07:35:42.204 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.099505892999999998 2024-02-05T07:35:42.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.446364+0000", 2024-02-05T07:35:42.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3215e-05 2024-02-05T07:35:42.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.446495+0000", 2024-02-05T07:35:42.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013103400000000001 2024-02-05T07:35:42.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.446657+0000", 2024-02-05T07:35:42.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016147199999999999 2024-02-05T07:35:42.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4,6", 2024-02-05T07:35:42.206 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.446932+0000", 2024-02-05T07:35:42.206 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000275457 2024-02-05T07:35:42.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.206 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.452491+0000", 2024-02-05T07:35:42.206 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0055588039999999997 2024-02-05T07:35:42.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:35:42.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.452777+0000", 2024-02-05T07:35:42.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028545099999999999 2024-02-05T07:35:42.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.207 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.453014+0000", 2024-02-05T07:35:42.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000237932 2024-02-05T07:35:42.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.207 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.453197+0000", 2024-02-05T07:35:42.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018281699999999999 2024-02-05T07:35:42.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.208 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:35:42.208 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.459364+0000", 2024-02-05T07:35:42.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0061664600000000003 2024-02-05T07:35:42.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.208 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.208 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.459418+0000", 2024-02-05T07:35:42.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4734000000000001e-05 2024-02-05T07:35:42.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.208 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.208 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.459536+0000", 2024-02-05T07:35:42.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000117482 2024-02-05T07:35:42.209 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.209 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.209 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.209 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:143184 10.b e506/423 10:db9ca725:::benchmark_data_smithi007_157098_object14254:head v 506'8931, mlcod=506'8928)", 2024-02-05T07:35:42.209 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:32.347317+0000", 2024-02-05T07:35:42.209 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.2311243639999994, 2024-02-05T07:35:42.209 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.104312057, 2024-02-05T07:35:42.209 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:35:42.209 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:35:42.209 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:35:42.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.209 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.347317+0000", 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.347317+0000", 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.347318+0000", 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9599999999999999e-07 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:35:42.211 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.347324+0000", 2024-02-05T07:35:42.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0310000000000004e-06 2024-02-05T07:35:42.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.211 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:35:42.211 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.447335+0000", 2024-02-05T07:35:42.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10001117900000001 2024-02-05T07:35:42.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.211 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:35:42.211 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.447345+0000", 2024-02-05T07:35:42.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8079999999999999e-06 2024-02-05T07:35:42.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.212 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:35:42.212 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.447818+0000", 2024-02-05T07:35:42.212 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00047288400000000002 2024-02-05T07:35:42.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.212 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:35:42.212 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.449065+0000", 2024-02-05T07:35:42.212 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001247784 2024-02-05T07:35:42.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.213 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:35:42.213 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.451591+0000", 2024-02-05T07:35:42.213 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025254359999999998 2024-02-05T07:35:42.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:35:42.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:35:42.213 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:35:42.213 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.451629+0000", 2024-02-05T07:35:42.213 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8339e-05 2024-02-05T07:35:42.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.213 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.213 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:35:42.213 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:35:42.213 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:35:42.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:42.243 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:42.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:42.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:42.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:42.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:42.477 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:42.503 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:42.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:42.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:42.633 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:42.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:42.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:42.766 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:42.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:42.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:42.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:42.896 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:42.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:42.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:42 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:42 smithi007 ceph-mon[29098]: pgmap v912: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.5 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 40236/321501 objects misplaced (12.515%); 37 MiB/s, 591 objects/s recovering 2024-02-05T07:35:42.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:42 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:42.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:42 smithi007 ceph-mon[37158]: pgmap v912: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.5 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 40236/321501 objects misplaced (12.515%); 37 MiB/s, 591 objects/s recovering 2024-02-05T07:35:42.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:42 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:42 smithi090 ceph-mon[33213]: pgmap v912: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.5 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 40236/321501 objects misplaced (12.515%); 37 MiB/s, 591 objects/s recovering 2024-02-05T07:35:43.028 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:43.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:43.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:43.157 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:43.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:43.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:43.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:43.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:43.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:43.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:43.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:43.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:43.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:43.546 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:43.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:43.573 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:35:43.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:43.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:43.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:43.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:43.836 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:43.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:43.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:43 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:35:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:43 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:35:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:43 smithi007 ceph-mon[29098]: osdmap e509: 8 total, 8 up, 6 in 2024-02-05T07:35:43.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:35:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:35:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:35:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:43 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:35:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:43 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:35:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:43 smithi007 ceph-mon[37158]: osdmap e509: 8 total, 8 up, 6 in 2024-02-05T07:35:43.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:43.967 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:43.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:43.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:44.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:43 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:35:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:43 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:35:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:43 smithi090 ceph-mon[33213]: osdmap e509: 8 total, 8 up, 6 in 2024-02-05T07:35:44.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:44.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:44.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:44.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:44.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:44.226 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:44.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:44.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:44.366 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:44.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:44.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:44.517 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:44.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:44.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:44.648 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:44.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:44.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:44.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:44.775 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:44.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:44.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:44.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:44.915 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:44.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:44.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:44 smithi007 ceph-mon[29098]: pgmap v914: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.5 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 39503/321021 objects misplaced (12.305%); 41 MiB/s, 653 objects/s recovering 2024-02-05T07:35:44.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:44 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:35:44.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:44 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:35:44.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:44 smithi007 ceph-mon[29098]: osdmap e510: 8 total, 8 up, 6 in 2024-02-05T07:35:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:44 smithi007 ceph-mon[37158]: pgmap v914: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.5 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 39503/321021 objects misplaced (12.305%); 41 MiB/s, 653 objects/s recovering 2024-02-05T07:35:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:44 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:35:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:44 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:35:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:44 smithi007 ceph-mon[37158]: osdmap e510: 8 total, 8 up, 6 in 2024-02-05T07:35:45.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:45.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:44 smithi090 ceph-mon[33213]: pgmap v914: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.5 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 39503/321021 objects misplaced (12.305%); 41 MiB/s, 653 objects/s recovering 2024-02-05T07:35:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:44 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:35:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:44 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:35:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:44 smithi090 ceph-mon[33213]: osdmap e510: 8 total, 8 up, 6 in 2024-02-05T07:35:45.042 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:45.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:45.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:45.186 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:45.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:45.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:45.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:45.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:45.331 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:45.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:45.432 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:45.472 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:45.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:45.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:45.615 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:45.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:45.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:45.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:45.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:45.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:45.872 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:45.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:45.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:45.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:46.013 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:46.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:46.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:46.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:46.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:46.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:46.274 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:46.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:46.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:46.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:46.410 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:46.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:46.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:46.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:46.538 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:46.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:46.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:46.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:46.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:46.663 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:46.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:46.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:46.794 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:46.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:46.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:46.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:46.920 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:46.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:46.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:46.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:46 smithi007 ceph-mon[29098]: pgmap v916: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.3 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 37646/311709 objects misplaced (12.077%); 33 MiB/s, 520 objects/s recovering 2024-02-05T07:35:46.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:46 smithi007 ceph-mon[29098]: 4.1f scrub starts 2024-02-05T07:35:46.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:46 smithi007 ceph-mon[29098]: 4.1f scrub ok 2024-02-05T07:35:46.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:46 smithi007 ceph-mon[29098]: osdmap e511: 8 total, 8 up, 6 in 2024-02-05T07:35:46.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:35:46.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:46 smithi007 ceph-mon[37158]: pgmap v916: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.3 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 37646/311709 objects misplaced (12.077%); 33 MiB/s, 520 objects/s recovering 2024-02-05T07:35:46.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:46 smithi007 ceph-mon[37158]: 4.1f scrub starts 2024-02-05T07:35:46.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:46 smithi007 ceph-mon[37158]: 4.1f scrub ok 2024-02-05T07:35:46.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:46 smithi007 ceph-mon[37158]: osdmap e511: 8 total, 8 up, 6 in 2024-02-05T07:35:46.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:35:47.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:47.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:46 smithi090 ceph-mon[33213]: pgmap v916: 172 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.3 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 37646/311709 objects misplaced (12.077%); 33 MiB/s, 520 objects/s recovering 2024-02-05T07:35:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:46 smithi090 ceph-mon[33213]: 4.1f scrub starts 2024-02-05T07:35:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:46 smithi090 ceph-mon[33213]: 4.1f scrub ok 2024-02-05T07:35:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:46 smithi090 ceph-mon[33213]: osdmap e511: 8 total, 8 up, 6 in 2024-02-05T07:35:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:35:47.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:47.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:47.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:47.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:47.182 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:47.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:47.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:47.317 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:47.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:47.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:47.446 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:47.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:47.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:47.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:47.581 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:47.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:47.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:47.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:47.707 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:47.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:47.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:47.808 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:47.847 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:47.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:47.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:47 smithi007 ceph-mon[29098]: 4.a scrub starts 2024-02-05T07:35:47.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:47 smithi007 ceph-mon[29098]: 4.a scrub ok 2024-02-05T07:35:47.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:47 smithi007 ceph-mon[29098]: osdmap e512: 8 total, 8 up, 6 in 2024-02-05T07:35:47.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:47 smithi007 ceph-mon[37158]: 4.a scrub starts 2024-02-05T07:35:47.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:47 smithi007 ceph-mon[37158]: 4.a scrub ok 2024-02-05T07:35:47.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:47 smithi007 ceph-mon[37158]: osdmap e512: 8 total, 8 up, 6 in 2024-02-05T07:35:47.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:47.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:47.981 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:47.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:48.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:47 smithi090 ceph-mon[33213]: 4.a scrub starts 2024-02-05T07:35:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:47 smithi090 ceph-mon[33213]: 4.a scrub ok 2024-02-05T07:35:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:47 smithi090 ceph-mon[33213]: osdmap e512: 8 total, 8 up, 6 in 2024-02-05T07:35:48.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:48.108 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:48.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:48.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:48.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:48.254 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:48.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:48.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:48.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:48.383 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:48.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:48.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:48.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:48.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:48.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:48.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:48.663 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:48.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:48.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:48.788 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:48.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:48.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:48 smithi007 ceph-mon[29098]: 4.10 scrub starts 2024-02-05T07:35:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:48 smithi007 ceph-mon[29098]: 4.10 scrub ok 2024-02-05T07:35:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:48 smithi007 ceph-mon[29098]: pgmap v919: 172 pgs: 1 peering, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s; 35558/305568 objects misplaced (11.637%); 28 MiB/s, 444 objects/s recovering 2024-02-05T07:35:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:48 smithi007 ceph-mon[29098]: 10.16 scrub starts 2024-02-05T07:35:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:48 smithi007 ceph-mon[29098]: osdmap e513: 8 total, 8 up, 6 in 2024-02-05T07:35:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:48 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3154172103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:35:48.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:48 smithi007 ceph-mon[37158]: 4.10 scrub starts 2024-02-05T07:35:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:48 smithi007 ceph-mon[37158]: 4.10 scrub ok 2024-02-05T07:35:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:48 smithi007 ceph-mon[37158]: pgmap v919: 172 pgs: 1 peering, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s; 35558/305568 objects misplaced (11.637%); 28 MiB/s, 444 objects/s recovering 2024-02-05T07:35:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:48 smithi007 ceph-mon[37158]: 10.16 scrub starts 2024-02-05T07:35:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:48 smithi007 ceph-mon[37158]: osdmap e513: 8 total, 8 up, 6 in 2024-02-05T07:35:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:48 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3154172103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:35:48.934 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:48.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:48.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:48 smithi090 ceph-mon[33213]: 4.10 scrub starts 2024-02-05T07:35:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:48 smithi090 ceph-mon[33213]: 4.10 scrub ok 2024-02-05T07:35:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:48 smithi090 ceph-mon[33213]: pgmap v919: 172 pgs: 1 peering, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s; 35558/305568 objects misplaced (11.637%); 28 MiB/s, 444 objects/s recovering 2024-02-05T07:35:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:48 smithi090 ceph-mon[33213]: 10.16 scrub starts 2024-02-05T07:35:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:48 smithi090 ceph-mon[33213]: osdmap e513: 8 total, 8 up, 6 in 2024-02-05T07:35:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:48 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3154172103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:35:49.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:49.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:49.099 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:35:49.169 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:49.210 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:49.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:49.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:49.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:49.352 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:49.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:49.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:49.455 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:49.478 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:49.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:49.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:49.580 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:49.626 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:49.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:49.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:49.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:49.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:49.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:49.879 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:49.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:49.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:49 smithi007 ceph-mon[29098]: 10.17 scrub starts 2024-02-05T07:35:49.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:49 smithi007 ceph-mon[29098]: 10.15 scrub starts 2024-02-05T07:35:49.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:49 smithi007 ceph-mon[29098]: 10.16 scrub ok 2024-02-05T07:35:49.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:49 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:35:49.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:49 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3154172103' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:35:49.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:49 smithi007 ceph-mon[29098]: osdmap e514: 8 total, 8 up, 6 in 2024-02-05T07:35:49.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:49 smithi007 ceph-mon[37158]: 10.17 scrub starts 2024-02-05T07:35:49.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:49 smithi007 ceph-mon[37158]: 10.15 scrub starts 2024-02-05T07:35:49.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:49 smithi007 ceph-mon[37158]: 10.16 scrub ok 2024-02-05T07:35:49.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:49 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:35:49.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:49 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3154172103' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:35:49.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:49 smithi007 ceph-mon[37158]: osdmap e514: 8 total, 8 up, 6 in 2024-02-05T07:35:49.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:50.005 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:50.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:50.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:49 smithi090 ceph-mon[33213]: 10.17 scrub starts 2024-02-05T07:35:50.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:49 smithi090 ceph-mon[33213]: 10.15 scrub starts 2024-02-05T07:35:50.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:49 smithi090 ceph-mon[33213]: 10.16 scrub ok 2024-02-05T07:35:50.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:49 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:35:50.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:49 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3154172103' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:35:50.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:49 smithi090 ceph-mon[33213]: osdmap e514: 8 total, 8 up, 6 in 2024-02-05T07:35:50.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:50.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:50.136 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:50.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:50.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:50.266 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:50.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:50.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:50.368 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:50.413 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:50.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:50.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:50.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:50.546 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:50.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:50.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:50.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:50.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:50.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:50.813 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:50.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:50.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:50.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:50 smithi007 ceph-mon[29098]: pgmap v922: 172 pgs: 1 peering, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 162 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 34962/301215 objects misplaced (11.607%) 2024-02-05T07:35:50.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:50 smithi007 ceph-mon[37158]: pgmap v922: 172 pgs: 1 peering, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 162 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 34962/301215 objects misplaced (11.607%) 2024-02-05T07:35:50.943 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:50 smithi090 ceph-mon[33213]: pgmap v922: 172 pgs: 1 peering, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 162 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 34962/301215 objects misplaced (11.607%) 2024-02-05T07:35:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:50.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:50.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:51.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:51.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:51.200 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:51.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:51.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:51.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:35:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:35:51.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:51.370 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:51.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:51.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:51.516 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:51.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:51.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:51.656 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:51.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:51.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:51.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:51.787 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:51.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:51.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:51.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:51.919 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:51.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:51.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:52.021 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:51 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:52.053 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:52.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:52.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:52.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:51 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:52.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:51 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:52.197 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:52.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:52.299 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:52.332 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:52.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:52.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:52.459 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:52.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:52.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:52.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:52.605 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:52.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:52.706 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:52.736 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:52.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:52.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:52.869 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:52.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:52.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:53.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:53.001 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:53.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:52 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:52 smithi090 ceph-mon[33213]: pgmap v923: 172 pgs: 1 peering, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 163 active+clean; 6.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 900 B/s rd, 0 op/s; 30959/294186 objects misplaced (10.524%); 721 KiB/s, 11 objects/s recovering 2024-02-05T07:35:53.102 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:53.131 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:53.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:53.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:53.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:52 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:53.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:52 smithi007 ceph-mon[29098]: pgmap v923: 172 pgs: 1 peering, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 163 active+clean; 6.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 900 B/s rd, 0 op/s; 30959/294186 objects misplaced (10.524%); 721 KiB/s, 11 objects/s recovering 2024-02-05T07:35:53.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:52 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:35:53.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:52 smithi007 ceph-mon[37158]: pgmap v923: 172 pgs: 1 peering, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 163 active+clean; 6.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 900 B/s rd, 0 op/s; 30959/294186 objects misplaced (10.524%); 721 KiB/s, 11 objects/s recovering 2024-02-05T07:35:53.233 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:53.261 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:53.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:53.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:53.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:53.395 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:53.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:53.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:53.527 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:53.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:53.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:53.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:53.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:35:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:35:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:35:53.673 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:53.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:53.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:53.723 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.0 primary-affinity to 0 (802) 2024-02-05T07:35:53.775 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:53.804 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:53.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:53.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:53.905 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:53.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:54.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:53 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3073275677' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-05T07:35:54.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:53 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-05T07:35:54.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:54.087 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:54.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:54.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:54.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:53 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3073275677' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-05T07:35:54.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:53 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-05T07:35:54.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:53 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3073275677' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-05T07:35:54.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:53 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-05T07:35:54.188 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:54.227 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:54.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:54.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:54.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:54.357 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:54.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:54.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:54.495 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:54.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:54.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:54.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:54.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:54.643 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:54.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:54.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:54.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:54.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:54.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:54.875 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:54.899 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:54.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:54.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:55.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:55.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:54 smithi090 ceph-mon[33213]: pgmap v924: 172 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 164 active+clean; 6.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 766 B/s rd, 0 op/s; 30959/293811 objects misplaced (10.537%); 613 KiB/s, 9 objects/s recovering 2024-02-05T07:35:55.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:54 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-02-05T07:35:55.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:54 smithi090 ceph-mon[33213]: osdmap e515: 8 total, 8 up, 6 in 2024-02-05T07:35:55.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:54 smithi090 ceph-mon[33213]: osdmap e516: 8 total, 8 up, 6 in 2024-02-05T07:35:55.041 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:55.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:55.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:55.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:54 smithi007 ceph-mon[29098]: pgmap v924: 172 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 164 active+clean; 6.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 766 B/s rd, 0 op/s; 30959/293811 objects misplaced (10.537%); 613 KiB/s, 9 objects/s recovering 2024-02-05T07:35:55.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:54 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-02-05T07:35:55.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:54 smithi007 ceph-mon[29098]: osdmap e515: 8 total, 8 up, 6 in 2024-02-05T07:35:55.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:54 smithi007 ceph-mon[29098]: osdmap e516: 8 total, 8 up, 6 in 2024-02-05T07:35:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:54 smithi007 ceph-mon[37158]: pgmap v924: 172 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 164 active+clean; 6.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 766 B/s rd, 0 op/s; 30959/293811 objects misplaced (10.537%); 613 KiB/s, 9 objects/s recovering 2024-02-05T07:35:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:54 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-02-05T07:35:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:54 smithi007 ceph-mon[37158]: osdmap e515: 8 total, 8 up, 6 in 2024-02-05T07:35:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:54 smithi007 ceph-mon[37158]: osdmap e516: 8 total, 8 up, 6 in 2024-02-05T07:35:55.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:55.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:55.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:55.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:55.333 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:55.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:55.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:55.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:55.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:55.464 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:55.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:55.565 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:55.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:55.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:55.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:55.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:55.728 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:55.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:55.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:55.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:55.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T07:35:55.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:55.860 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:55.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:55.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:55.990 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:55.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:55.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:56.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:55 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:35:56.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:55 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:35:56.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:56.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:56.160 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:56.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:56.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:55 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:35:56.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:55 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:35:56.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:55 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:35:56.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:55 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:35:56.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:56.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:56.293 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:56.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:56.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:56.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:56.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:56.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:56.548 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:56.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:56.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:35:56.673 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:56.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:35:56.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:56.817 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:56.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:56.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:56.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:56.948 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:56.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:56.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:57.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:56 smithi090 ceph-mon[33213]: pgmap v927: 172 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 164 active+clean; 5.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 340 op/s; 30137/283833 objects misplaced (10.618%) 2024-02-05T07:35:57.050 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:57.089 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:57.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:57.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:57.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:56 smithi007 ceph-mon[29098]: pgmap v927: 172 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 164 active+clean; 5.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 340 op/s; 30137/283833 objects misplaced (10.618%) 2024-02-05T07:35:57.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:56 smithi007 ceph-mon[37158]: pgmap v927: 172 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 164 active+clean; 5.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 340 op/s; 30137/283833 objects misplaced (10.618%) 2024-02-05T07:35:57.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:57.218 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:57.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:57.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:57.349 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:57.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:57.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:57.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:57.481 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:57.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:57.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:57.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:57.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:57.642 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:57.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:57.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:57.773 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:57.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:57.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:35:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:57.901 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:57.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:35:58.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:58.025 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:58.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:58.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:35:58.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:58.168 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:58.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:58.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:57 smithi007 ceph-mon[29098]: pgmap v928: 172 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 164 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 27629/277053 objects misplaced (9.972%); 27 MiB/s, 430 objects/s recovering 2024-02-05T07:35:58.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:57 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:58.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:57 smithi007 ceph-mon[37158]: pgmap v928: 172 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 164 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 27629/277053 objects misplaced (9.972%); 27 MiB/s, 430 objects/s recovering 2024-02-05T07:35:58.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:57 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:58.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:57 smithi090 ceph-mon[33213]: pgmap v928: 172 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 164 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 27629/277053 objects misplaced (9.972%); 27 MiB/s, 430 objects/s recovering 2024-02-05T07:35:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:57 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:35:58.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:58.297 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:58.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:58.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:58.429 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:58.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:58.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:58.558 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:58.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:58.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:58.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:35:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:58.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:58.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:35:58.802 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:58.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:58.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:58.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:35:58.975 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:58.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:58.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:35:59.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:59.117 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:59.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:59.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:59.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:35:59.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:59.254 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:35:59.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:35:59.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:59.382 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:59.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:59.485 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:59.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:59.612 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:35:59.643 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:59.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:35:59.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:35:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:59.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:35:59.876 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:59 smithi007 ceph-mon[29098]: osdmap e517: 8 total, 8 up, 6 in 2024-02-05T07:35:59.876 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:59 smithi007 ceph-mon[29098]: pgmap v930: 172 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 164 active+clean; 5.5 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.25k op/s; 27629/271407 objects misplaced (10.180%); 12 MiB/s, 199 objects/s recovering 2024-02-05T07:35:59.876 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:35:59 smithi007 ceph-mon[29098]: osdmap e518: 8 total, 8 up, 6 in 2024-02-05T07:35:59.877 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:59 smithi007 ceph-mon[37158]: osdmap e517: 8 total, 8 up, 6 in 2024-02-05T07:35:59.877 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:59 smithi007 ceph-mon[37158]: pgmap v930: 172 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 164 active+clean; 5.5 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.25k op/s; 27629/271407 objects misplaced (10.180%); 12 MiB/s, 199 objects/s recovering 2024-02-05T07:35:59.877 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:35:59 smithi007 ceph-mon[37158]: osdmap e518: 8 total, 8 up, 6 in 2024-02-05T07:35:59.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:35:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:35:59.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:35:59.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:00.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:00.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:59 smithi090 ceph-mon[33213]: osdmap e517: 8 total, 8 up, 6 in 2024-02-05T07:36:00.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:59 smithi090 ceph-mon[33213]: pgmap v930: 172 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 164 active+clean; 5.5 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.25k op/s; 27629/271407 objects misplaced (10.180%); 12 MiB/s, 199 objects/s recovering 2024-02-05T07:36:00.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:35:59 smithi090 ceph-mon[33213]: osdmap e518: 8 total, 8 up, 6 in 2024-02-05T07:36:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:00.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:00.147 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:00.185 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:00.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:00.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:00.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:00.317 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:00.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:00.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:00.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:00.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:00.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:00.562 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:00.566 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:00.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:00.614 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:00.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:00.715 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:00.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:00.772 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:00.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:00.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:00.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:00.906 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:00.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:01.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:01.052 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:01.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:01.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:01.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:01.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:01.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:01.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:36:00 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:36:01.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:01.329 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:01.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:01.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:01.463 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:01.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:01.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:01.494 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:36:01.547 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:36:01.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:01.591 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:36:01.606 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:01.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:01.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:36:01.636 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:36:01.678 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:36:01.708 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:01.728 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:36:01.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:01.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:01.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:01.770 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:36:01.815 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:36:01.820 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:36:01.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:01.902 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:01.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:01.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:02.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:02.037 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:02.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:02.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:02.133 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:01 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:36:02.133 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:01 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:36:02.133 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:02.133 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:01 smithi090 ceph-mon[33213]: pgmap v932: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 5.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.31k op/s; 23725/261567 objects misplaced (9.070%); 27 MiB/s, 427 objects/s recovering 2024-02-05T07:36:02.133 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:01 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/314471294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:36:02.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:02.171 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:02.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:02.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:02.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:01 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:36:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:01 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:36:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:01 smithi007 ceph-mon[29098]: pgmap v932: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 5.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.31k op/s; 23725/261567 objects misplaced (9.070%); 27 MiB/s, 427 objects/s recovering 2024-02-05T07:36:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:01 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/314471294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:36:02.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:01 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:36:02.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:01 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:36:02.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:02.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:01 smithi007 ceph-mon[37158]: pgmap v932: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 5.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.31k op/s; 23725/261567 objects misplaced (9.070%); 27 MiB/s, 427 objects/s recovering 2024-02-05T07:36:02.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:01 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/314471294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:36:02.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:02.316 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:02.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:02.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:02.448 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:02.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:02.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:02.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:02.604 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:02.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:02.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:02.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:02.763 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:02.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:02.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:02.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:02.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:02.898 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:02.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:02.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:03.046 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:03.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:03.147 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:03.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:03.212 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:03.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:03.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:03.358 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:03.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:03.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:03.460 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:03.490 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:03.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:03.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:03.635 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:03.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:03.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:03.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:03.767 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:03.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:03.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:03.897 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:03.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:03.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:36:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:36:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:36:03.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:04.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:04.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:04.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:04.178 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:04.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:04.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:04.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:04.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:04.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:04.445 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:04.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:04.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:04.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:04.591 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:04.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:04.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:04 smithi007 ceph-mon[29098]: pgmap v933: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 5.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 950 op/s; 23725/261033 objects misplaced (9.089%); 19 MiB/s, 310 objects/s recovering 2024-02-05T07:36:04.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:04 smithi007 ceph-mon[37158]: pgmap v933: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 5.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 950 op/s; 23725/261033 objects misplaced (9.089%); 19 MiB/s, 310 objects/s recovering 2024-02-05T07:36:04.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:04.722 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:04.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:04.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:04.773 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:04 smithi090 ceph-mon[33213]: pgmap v933: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 5.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 950 op/s; 23725/261033 objects misplaced (9.089%); 19 MiB/s, 310 objects/s recovering 2024-02-05T07:36:04.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:04.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:04.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:04.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:04.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:04.987 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:04.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:05.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:05.130 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:05.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:05.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:05.169 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:36:05.169 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:36:05.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:36:05.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:05.242 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 3] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:36:05.242 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:36:05.243 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 4, 6, 3] 2024-02-05T07:36:05.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd out 2 2024-02-05T07:36:05.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:05.291 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:05.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:05.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:05.421 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:05.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:05.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:05.571 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:05.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:05.663 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:05.674 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:05.737 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:05.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:05.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:05.869 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:05.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:05.970 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:06.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:06.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:06.116 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:06.145 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:06.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:06.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:06.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:06.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T07:36:06.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:06.305 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:06.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:06.345 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:06.358 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:06.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:06.434 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:06.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:06.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:06.561 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:06.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:06.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:06.663 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:06.695 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:06.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:06.763 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:06 smithi007 ceph-mon[29098]: pgmap v934: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 5.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 23327/250671 objects misplaced (9.306%); 13 MiB/s, 209 objects/s recovering 2024-02-05T07:36:06.765 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:06 smithi007 ceph-mon[37158]: pgmap v934: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 5.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 23327/250671 objects misplaced (9.306%); 13 MiB/s, 209 objects/s recovering 2024-02-05T07:36:06.796 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:06.835 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:06.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:06.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:06.985 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:06.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:06.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:07.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:06 smithi090 ceph-mon[33213]: pgmap v934: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 5.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 23327/250671 objects misplaced (9.306%); 13 MiB/s, 209 objects/s recovering 2024-02-05T07:36:07.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:07.116 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:07.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:07.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:07.218 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:07.260 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:07.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:07.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:07.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:07.408 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:07.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:07.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:07.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:07.551 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:07.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:07.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:07.692 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:07.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:07.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:07.826 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:07.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:07.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:07.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:07.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:08.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:08.099 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:08.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:08.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:08.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:08.229 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:08.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:08.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:08.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:08.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:08.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:08.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:08.515 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:08.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:08.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:08.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:08.661 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:08.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:08.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:08 smithi007 ceph-mon[29098]: pgmap v935: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 4.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 20766/243426 objects misplaced (8.531%); 20 MiB/s, 317 objects/s recovering 2024-02-05T07:36:08.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:08 smithi007 ceph-mon[37158]: pgmap v935: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 4.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 20766/243426 objects misplaced (8.531%); 20 MiB/s, 317 objects/s recovering 2024-02-05T07:36:08.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:08.792 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:08.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:08.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:08.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:08.926 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:08.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:08.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:09.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:08 smithi090 ceph-mon[33213]: pgmap v935: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 4.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 20766/243426 objects misplaced (8.531%); 20 MiB/s, 317 objects/s recovering 2024-02-05T07:36:09.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:09.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:09.073 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:09.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:09.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:09.204 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:09.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:09.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:09.335 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:09.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:09.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:09.466 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:09.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:09.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:09.597 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:09.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:09.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:09.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:09.728 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:09.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:09.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:09.860 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:09.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:09.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:09.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:10.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:10.006 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:10.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:10.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:10.136 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:10.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:10.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:10.267 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:10.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:10.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:10.410 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:10.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:10.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:10.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:10.559 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:10.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:10.660 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:10.690 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:10.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:10.796 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:10.837 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:10.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:10.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:10 smithi007 ceph-mon[29098]: pgmap v936: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 4.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 20766/237693 objects misplaced (8.736%); 17 MiB/s, 270 objects/s recovering 2024-02-05T07:36:10.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:10 smithi007 ceph-mon[37158]: pgmap v936: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 4.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 20766/237693 objects misplaced (8.736%); 17 MiB/s, 270 objects/s recovering 2024-02-05T07:36:10.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:10.968 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:10 smithi090 ceph-mon[33213]: pgmap v936: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 4.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 20766/237693 objects misplaced (8.736%); 17 MiB/s, 270 objects/s recovering 2024-02-05T07:36:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:10.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:11.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:11.101 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:11.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:11.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:11.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:11.234 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:11.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:11.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:36:10 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:36:11.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:11.370 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:11.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:11.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:11.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:11.503 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:11.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:11.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:11.647 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:11.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:11.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:11.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:11.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:11.786 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:11.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:11.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:11.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:11.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:12.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:12.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:12.060 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:12.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:12.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:12.186 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:12.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:12.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:12.288 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:12.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:12.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:12.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:12.438 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:12.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:12.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:12.540 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:12.568 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:12.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:12.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:12.701 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:12.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:12.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:12.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:12 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:12.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:12 smithi007 ceph-mon[29098]: pgmap v937: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 4.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.28k op/s; 17873/226374 objects misplaced (7.895%); 20 MiB/s, 316 objects/s recovering 2024-02-05T07:36:12.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:12 smithi007 ceph-mon[29098]: osdmap e519: 8 total, 8 up, 6 in 2024-02-05T07:36:12.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:12 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:12.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:12 smithi007 ceph-mon[37158]: pgmap v937: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 4.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.28k op/s; 17873/226374 objects misplaced (7.895%); 20 MiB/s, 316 objects/s recovering 2024-02-05T07:36:12.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:12 smithi007 ceph-mon[37158]: osdmap e519: 8 total, 8 up, 6 in 2024-02-05T07:36:12.843 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:12.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:12.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:12.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:12.972 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:12.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:12.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:13.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:12 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:12 smithi090 ceph-mon[33213]: pgmap v937: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 4.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.28k op/s; 17873/226374 objects misplaced (7.895%); 20 MiB/s, 316 objects/s recovering 2024-02-05T07:36:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:12 smithi090 ceph-mon[33213]: osdmap e519: 8 total, 8 up, 6 in 2024-02-05T07:36:13.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:13.103 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:13.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:13.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:13.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:13.229 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:13.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:13.331 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:13.353 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:13.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:13.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:13.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:13.482 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:13.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:13.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:13.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:13.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:13.651 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:36:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:36:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:36:13.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:13.747 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:13.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:13.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:13.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:13.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:13.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:13 smithi007 ceph-mon[29098]: osdmap e520: 8 total, 8 up, 6 in 2024-02-05T07:36:13.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:13 smithi007 ceph-mon[29098]: osdmap e521: 8 total, 8 up, 6 in 2024-02-05T07:36:13.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:13 smithi007 ceph-mon[37158]: osdmap e520: 8 total, 8 up, 6 in 2024-02-05T07:36:13.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:13 smithi007 ceph-mon[37158]: osdmap e521: 8 total, 8 up, 6 in 2024-02-05T07:36:13.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:14.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:13 smithi090 ceph-mon[33213]: osdmap e520: 8 total, 8 up, 6 in 2024-02-05T07:36:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:13 smithi090 ceph-mon[33213]: osdmap e521: 8 total, 8 up, 6 in 2024-02-05T07:36:14.031 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:14.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:14.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:14.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:14.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:14.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:14.307 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:14.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:14.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:14.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:14.461 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:14.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:14.562 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:14.592 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:14.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:14.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:14.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:14.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:14.736 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:14.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:14.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:14.867 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:14.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:14.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:14 smithi007 ceph-mon[29098]: pgmap v940: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 4.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s; 17873/225786 objects misplaced (7.916%); 16 MiB/s, 254 objects/s recovering 2024-02-05T07:36:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:14 smithi007 ceph-mon[29098]: osdmap e522: 8 total, 8 up, 6 in 2024-02-05T07:36:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:14 smithi007 ceph-mon[37158]: pgmap v940: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 4.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s; 17873/225786 objects misplaced (7.916%); 16 MiB/s, 254 objects/s recovering 2024-02-05T07:36:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:14 smithi007 ceph-mon[37158]: osdmap e522: 8 total, 8 up, 6 in 2024-02-05T07:36:14.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:14.999 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:15.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:15.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:15.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:14 smithi090 ceph-mon[33213]: pgmap v940: 172 pgs: 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 165 active+clean; 4.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s; 17873/225786 objects misplaced (7.916%); 16 MiB/s, 254 objects/s recovering 2024-02-05T07:36:15.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:14 smithi090 ceph-mon[33213]: osdmap e522: 8 total, 8 up, 6 in 2024-02-05T07:36:15.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:15.130 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:15.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:15.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:15.231 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:15.271 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:15.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:15.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:15.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:15.415 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:15.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:15.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:15.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:15.559 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:15.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:15.660 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:15.689 INFO:teuthology.orchestra.run.smithi007.stderr:marked out osd.2. 2024-02-05T07:36:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:15.691 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:15.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:15.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:15.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:15.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:15.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:15 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a[29094]: 2024-02-05T07:36:15.612+0000 7fe3564d3700 -1 mon.a@0(leader).osd e522 definitely_dead 0 2024-02-05T07:36:15.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:15 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1466191491' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-05T07:36:15.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:15 smithi007 ceph-mon[29098]: Client client.admin marked osd.2 out, while it was still marked up 2024-02-05T07:36:15.900 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:15 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1466191491' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-05T07:36:15.900 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:15 smithi007 ceph-mon[37158]: Client client.admin marked osd.2 out, while it was still marked up 2024-02-05T07:36:15.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:15.944 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:15.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:15.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:16.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:15 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1466191491' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-05T07:36:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:15 smithi090 ceph-mon[33213]: Client client.admin marked osd.2 out, while it was still marked up 2024-02-05T07:36:16.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:16.070 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:16.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:16.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:16.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:16.195 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:16.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:16.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:16.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:16.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:16.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:16.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:16.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:16.450 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:16.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:16.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:16.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:16.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:16.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:16.704 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:36:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:16.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:16.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:16.848 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:16.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:16.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:16 smithi007 ceph-mon[29098]: pgmap v943: 172 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 166 active+clean; 4.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 767 op/s; 16650/217176 objects misplaced (7.667%); 7.1 MiB/s, 113 objects/s recovering 2024-02-05T07:36:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:16 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1466191491' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-05T07:36:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:16 smithi007 ceph-mon[29098]: osdmap e523: 8 total, 8 up, 5 in 2024-02-05T07:36:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:16 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:36:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:16 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1094649402' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:36:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:16 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:36:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:16 smithi007 ceph-mon[37158]: pgmap v943: 172 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 166 active+clean; 4.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 767 op/s; 16650/217176 objects misplaced (7.667%); 7.1 MiB/s, 113 objects/s recovering 2024-02-05T07:36:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:16 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1466191491' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-05T07:36:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:16 smithi007 ceph-mon[37158]: osdmap e523: 8 total, 8 up, 5 in 2024-02-05T07:36:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:16 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:36:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:16 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1094649402' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:36:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:16 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:36:16.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:16.974 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:16.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:16.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:17.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:16 smithi090 ceph-mon[33213]: pgmap v943: 172 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 166 active+clean; 4.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 767 op/s; 16650/217176 objects misplaced (7.667%); 7.1 MiB/s, 113 objects/s recovering 2024-02-05T07:36:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:16 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1466191491' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-05T07:36:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:16 smithi090 ceph-mon[33213]: osdmap e523: 8 total, 8 up, 5 in 2024-02-05T07:36:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:16 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:36:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:16 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1094649402' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:36:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:16 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:36:17.076 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:17.102 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:17.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:17.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:17.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:17.227 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:17.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:17.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:17.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:17.356 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:17.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:17.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:17.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:17.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:17.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:17.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:17.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:17.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:17.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:17.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:17.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:17.894 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:17.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:17.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:17.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:17 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:36:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:17 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:36:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:17 smithi090 ceph-mon[33213]: 10.11 starting backfill to osd.3 from (0'0,0'0] MAX to 522'10028 2024-02-05T07:36:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:17 smithi090 ceph-mon[33213]: osdmap e524: 8 total, 8 up, 5 in 2024-02-05T07:36:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:17 smithi090 ceph-mon[33213]: 10.18 continuing backfill to osd.1 from (446'4495,522'9884] MIN to 522'9884 2024-02-05T07:36:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:17 smithi090 ceph-mon[33213]: 10.18 continuing backfill to osd.3 from (446'4495,522'9884] MIN to 522'9884 2024-02-05T07:36:18.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:17 smithi090 ceph-mon[33213]: 10.18 starting backfill to osd.6 from (0'0,0'0] MAX to 522'9884 2024-02-05T07:36:18.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:17 smithi090 ceph-mon[33213]: 10.11 continuing backfill to osd.6 from (456'5287,522'10028] MIN to 522'10028 2024-02-05T07:36:18.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:17 smithi090 ceph-mon[33213]: 10.12 continuing backfill to osd.3 from (455'5293,522'9981] MIN to 522'9981 2024-02-05T07:36:18.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:17 smithi090 ceph-mon[33213]: 10.12 continuing backfill to osd.6 from (455'5293,522'9981] MIN to 522'9981 2024-02-05T07:36:18.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:17 smithi090 ceph-mon[33213]: osdmap e525: 8 total, 8 up, 5 in 2024-02-05T07:36:18.037 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:18.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:18.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:18.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:18.164 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:18.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:18.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:18.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:36:18.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:36:18.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[29098]: 10.11 starting backfill to osd.3 from (0'0,0'0] MAX to 522'10028 2024-02-05T07:36:18.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[29098]: osdmap e524: 8 total, 8 up, 5 in 2024-02-05T07:36:18.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[29098]: 10.18 continuing backfill to osd.1 from (446'4495,522'9884] MIN to 522'9884 2024-02-05T07:36:18.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[29098]: 10.18 continuing backfill to osd.3 from (446'4495,522'9884] MIN to 522'9884 2024-02-05T07:36:18.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[29098]: 10.18 starting backfill to osd.6 from (0'0,0'0] MAX to 522'9884 2024-02-05T07:36:18.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[29098]: 10.11 continuing backfill to osd.6 from (456'5287,522'10028] MIN to 522'10028 2024-02-05T07:36:18.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[29098]: 10.12 continuing backfill to osd.3 from (455'5293,522'9981] MIN to 522'9981 2024-02-05T07:36:18.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[29098]: 10.12 continuing backfill to osd.6 from (455'5293,522'9981] MIN to 522'9981 2024-02-05T07:36:18.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[29098]: osdmap e525: 8 total, 8 up, 5 in 2024-02-05T07:36:18.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:36:18.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:36:18.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[37158]: 10.11 starting backfill to osd.3 from (0'0,0'0] MAX to 522'10028 2024-02-05T07:36:18.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[37158]: osdmap e524: 8 total, 8 up, 5 in 2024-02-05T07:36:18.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[37158]: 10.18 continuing backfill to osd.1 from (446'4495,522'9884] MIN to 522'9884 2024-02-05T07:36:18.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[37158]: 10.18 continuing backfill to osd.3 from (446'4495,522'9884] MIN to 522'9884 2024-02-05T07:36:18.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[37158]: 10.18 starting backfill to osd.6 from (0'0,0'0] MAX to 522'9884 2024-02-05T07:36:18.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[37158]: 10.11 continuing backfill to osd.6 from (456'5287,522'10028] MIN to 522'10028 2024-02-05T07:36:18.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[37158]: 10.12 continuing backfill to osd.3 from (455'5293,522'9981] MIN to 522'9981 2024-02-05T07:36:18.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[37158]: 10.12 continuing backfill to osd.6 from (455'5293,522'9981] MIN to 522'9981 2024-02-05T07:36:18.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:17 smithi007 ceph-mon[37158]: osdmap e525: 8 total, 8 up, 5 in 2024-02-05T07:36:18.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:18.291 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:18.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:18.309 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:18.381 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:36:18.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:18.420 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:36:18.427 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:18.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:18.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:18.454 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:36:18.489 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:36:18.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:18.529 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:36:18.558 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:18.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:18.570 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:36:18.615 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:36:18.654 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:36:18.657 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:36:18.657 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:36:18.657 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:36:18.657 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:36:18.657 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:36:18.657 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:36:18.657 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:36:18.657 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:36:18.660 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:18.685 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:18.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:18.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:18.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:18.818 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:18.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:18.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:18.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:18.949 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:18.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: pgmap v946: 172 pgs: 8 remapped+peering, 2 active+remapped+backfill_toofull, 162 active+clean; 4.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 4.0 KiB/s rd, 27 KiB/s wr, 1.18k op/s; 4424/211662 objects misplaced (2.090%); 8.6 MiB/s, 146 objects/s recovering 2024-02-05T07:36:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 10.14 starting backfill to osd.3 from (0'0,0'0] MAX to 522'9955 2024-02-05T07:36:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 10.a starting backfill to osd.3 from (0'0,0'0] MAX to 522'11815 2024-02-05T07:36:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 10.9 starting backfill to osd.0 from (0'0,0'0] MAX to 522'9906 2024-02-05T07:36:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 10.8 starting backfill to osd.3 from (0'0,0'0] MAX to 522'9986 2024-02-05T07:36:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 1.1 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 1.2 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 522'11906 2024-02-05T07:36:19.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 1.a starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 1.9 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 1.8 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 10.15 starting backfill to osd.0 from (442'3990,511'8884] MIN to 522'9823 2024-02-05T07:36:19.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 522'11796 2024-02-05T07:36:19.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 10.13 starting backfill to osd.0 from (0'0,0'0] MAX to 522'9786 2024-02-05T07:36:19.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 1.d starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 1.5 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 10.7 starting backfill to osd.4 from (0'0,0'0] MAX to 522'9871 2024-02-05T07:36:19.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:36:19.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:19.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 10.13 continuing backfill to osd.1 from (445'4245,522'9786] 10:cdfe3c22:::benchmark_data_smithi007_157098_object46573:head to 522'9786 2024-02-05T07:36:19.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 10.3 starting backfill to osd.0 from (0'0,0'0] MAX to 522'9880 2024-02-05T07:36:19.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 10.5 starting backfill to osd.4 from (0'0,0'0] MAX to 522'9710 2024-02-05T07:36:19.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: 1.7 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:36:19.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:36:19.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:18 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2891769612' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:36:19.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:19.090 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:19.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:19.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:19.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: pgmap v946: 172 pgs: 8 remapped+peering, 2 active+remapped+backfill_toofull, 162 active+clean; 4.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 4.0 KiB/s rd, 27 KiB/s wr, 1.18k op/s; 4424/211662 objects misplaced (2.090%); 8.6 MiB/s, 146 objects/s recovering 2024-02-05T07:36:19.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 10.14 starting backfill to osd.3 from (0'0,0'0] MAX to 522'9955 2024-02-05T07:36:19.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 10.a starting backfill to osd.3 from (0'0,0'0] MAX to 522'11815 2024-02-05T07:36:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 10.9 starting backfill to osd.0 from (0'0,0'0] MAX to 522'9906 2024-02-05T07:36:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 10.8 starting backfill to osd.3 from (0'0,0'0] MAX to 522'9986 2024-02-05T07:36:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 1.1 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 1.2 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 522'11906 2024-02-05T07:36:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 1.a starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 1.9 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 1.8 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 10.15 starting backfill to osd.0 from (442'3990,511'8884] MIN to 522'9823 2024-02-05T07:36:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 522'11796 2024-02-05T07:36:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 10.13 starting backfill to osd.0 from (0'0,0'0] MAX to 522'9786 2024-02-05T07:36:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 1.d starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 1.5 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 10.7 starting backfill to osd.4 from (0'0,0'0] MAX to 522'9871 2024-02-05T07:36:19.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:36:19.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:19.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 10.13 continuing backfill to osd.1 from (445'4245,522'9786] 10:cdfe3c22:::benchmark_data_smithi007_157098_object46573:head to 522'9786 2024-02-05T07:36:19.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 10.3 starting backfill to osd.0 from (0'0,0'0] MAX to 522'9880 2024-02-05T07:36:19.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 10.5 starting backfill to osd.4 from (0'0,0'0] MAX to 522'9710 2024-02-05T07:36:19.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: 1.7 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:36:19.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:36:19.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2891769612' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:36:19.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: pgmap v946: 172 pgs: 8 remapped+peering, 2 active+remapped+backfill_toofull, 162 active+clean; 4.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 4.0 KiB/s rd, 27 KiB/s wr, 1.18k op/s; 4424/211662 objects misplaced (2.090%); 8.6 MiB/s, 146 objects/s recovering 2024-02-05T07:36:19.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 10.14 starting backfill to osd.3 from (0'0,0'0] MAX to 522'9955 2024-02-05T07:36:19.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 10.a starting backfill to osd.3 from (0'0,0'0] MAX to 522'11815 2024-02-05T07:36:19.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 10.9 starting backfill to osd.0 from (0'0,0'0] MAX to 522'9906 2024-02-05T07:36:19.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 10.8 starting backfill to osd.3 from (0'0,0'0] MAX to 522'9986 2024-02-05T07:36:19.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 1.1 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 1.2 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 522'11906 2024-02-05T07:36:19.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 1.a starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 1.9 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 1.8 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 10.15 starting backfill to osd.0 from (442'3990,511'8884] MIN to 522'9823 2024-02-05T07:36:19.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 522'11796 2024-02-05T07:36:19.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 10.13 starting backfill to osd.0 from (0'0,0'0] MAX to 522'9786 2024-02-05T07:36:19.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 1.d starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 1.5 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 10.7 starting backfill to osd.4 from (0'0,0'0] MAX to 522'9871 2024-02-05T07:36:19.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:36:19.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:19.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 10.13 continuing backfill to osd.1 from (445'4245,522'9786] 10:cdfe3c22:::benchmark_data_smithi007_157098_object46573:head to 522'9786 2024-02-05T07:36:19.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 10.3 starting backfill to osd.0 from (0'0,0'0] MAX to 522'9880 2024-02-05T07:36:19.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 10.5 starting backfill to osd.4 from (0'0,0'0] MAX to 522'9710 2024-02-05T07:36:19.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: 1.7 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:19.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:36:19.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:36:19.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:18 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2891769612' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:36:19.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:19.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:19.223 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:19.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:19.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:19.379 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:19.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:19.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:19.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:19.511 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:19.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:19.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:19.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:19.643 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:19.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:19.734 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:19 smithi007 ceph-mon[29098]: osdmap e526: 8 total, 8 up, 5 in 2024-02-05T07:36:19.734 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:19 smithi007 ceph-mon[29098]: osdmap e527: 8 total, 8 up, 5 in 2024-02-05T07:36:19.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:19 smithi007 ceph-mon[37158]: osdmap e526: 8 total, 8 up, 5 in 2024-02-05T07:36:19.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:19.773 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:19.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:19.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:19.875 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:19.908 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:19.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:19.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:20.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:20.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:19 smithi090 ceph-mon[33213]: osdmap e526: 8 total, 8 up, 5 in 2024-02-05T07:36:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:19 smithi090 ceph-mon[33213]: osdmap e527: 8 total, 8 up, 5 in 2024-02-05T07:36:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:20.043 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:20.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:20.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:20.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:19 smithi007 ceph-mon[37158]: osdmap e527: 8 total, 8 up, 5 in 2024-02-05T07:36:20.175 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:20.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:20.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:20.277 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:20.321 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:20.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:20.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:20.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:20.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:20.457 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:20.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:20.558 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:20.588 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:20.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:20.690 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:20.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:20.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:20.774 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:36:20.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:20 smithi007 ceph-mon[29098]: pgmap v949: 172 pgs: 3 peering, 8 remapped+peering, 4 active+remapped+backfill_toofull, 157 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 6.0 KiB/s rd, 54 KiB/s wr, 709 op/s; 15114/208755 objects misplaced (7.240%); 1.5 MiB/s, 35 objects/s recovering 2024-02-05T07:36:20.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:20 smithi007 ceph-mon[37158]: pgmap v949: 172 pgs: 3 peering, 8 remapped+peering, 4 active+remapped+backfill_toofull, 157 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 6.0 KiB/s rd, 54 KiB/s wr, 709 op/s; 15114/208755 objects misplaced (7.240%); 1.5 MiB/s, 35 objects/s recovering 2024-02-05T07:36:20.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:20.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:20.915 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:20.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:21.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:21.023 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:36:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:36:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:20 smithi090 ceph-mon[33213]: pgmap v949: 172 pgs: 3 peering, 8 remapped+peering, 4 active+remapped+backfill_toofull, 157 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 6.0 KiB/s rd, 54 KiB/s wr, 709 op/s; 15114/208755 objects misplaced (7.240%); 1.5 MiB/s, 35 objects/s recovering 2024-02-05T07:36:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:21.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:21.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:21.197 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:21.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:21.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:21.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:21.331 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:21.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:21.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:21.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:36:21.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:21.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:21.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:21.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:21.592 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:21.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:21.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:21.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:21.739 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:21.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:21.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:21.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:21.869 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:21.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:21.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:21.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:22.013 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:22.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:22.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:22.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:22.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:22.146 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:22.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:22.189 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6, 3] out_osds: [5, 7, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:36:22.189 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:36:22.189 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-02-05T07:36:22.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd in 2 2024-02-05T07:36:22.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:22.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:22.276 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:22.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:22.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:22.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:22.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:22.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:22.543 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:22.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:22.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:22.650 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:36:22.650 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:36:22.650 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-02-05T07:36:22.650 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-02-05T07:36:22.650 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:36:22.675 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:22.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:22.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:22.812 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:22.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:22.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:22.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:22 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:36:22.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:22 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:36:22.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:22 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:22.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:22 smithi007 ceph-mon[29098]: pgmap v951: 172 pgs: 1 active+remapped+backfilling, 3 peering, 6 remapped+peering, 9 active+remapped+backfill_toofull, 153 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 9.3 KiB/s rd, 71 KiB/s wr, 542 op/s; 33738/204774 objects misplaced (16.476%); 15 MiB/s, 260 objects/s recovering 2024-02-05T07:36:22.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:22 smithi007 ceph-mon[29098]: osdmap e528: 8 total, 8 up, 5 in 2024-02-05T07:36:22.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:22 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:36:22.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:22 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:36:22.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:22 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:22.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:22 smithi007 ceph-mon[37158]: pgmap v951: 172 pgs: 1 active+remapped+backfilling, 3 peering, 6 remapped+peering, 9 active+remapped+backfill_toofull, 153 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 9.3 KiB/s rd, 71 KiB/s wr, 542 op/s; 33738/204774 objects misplaced (16.476%); 15 MiB/s, 260 objects/s recovering 2024-02-05T07:36:22.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:22 smithi007 ceph-mon[37158]: osdmap e528: 8 total, 8 up, 5 in 2024-02-05T07:36:22.945 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:22.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:22.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:22 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:36:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:22 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:36:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:22 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:22 smithi090 ceph-mon[33213]: pgmap v951: 172 pgs: 1 active+remapped+backfilling, 3 peering, 6 remapped+peering, 9 active+remapped+backfill_toofull, 153 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 9.3 KiB/s rd, 71 KiB/s wr, 542 op/s; 33738/204774 objects misplaced (16.476%); 15 MiB/s, 260 objects/s recovering 2024-02-05T07:36:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:22 smithi090 ceph-mon[33213]: osdmap e528: 8 total, 8 up, 5 in 2024-02-05T07:36:23.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:23.108 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:23.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:23.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:23.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:23.268 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:23.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:23.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:23.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:23.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:23.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:23.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:23.535 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:23.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:23.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:23.670 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:23.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:23.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:23.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:23.796 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:23.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:23.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:23.800 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:36:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:36:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:36:23.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:23.936 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:23.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:24.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:24.078 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:24.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:24.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:23 smithi007 ceph-mon[29098]: pgmap v953: 172 pgs: 3 active+remapped+backfilling, 3 peering, 11 active+remapped+backfill_toofull, 155 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 5.7 KiB/s rd, 39 KiB/s wr, 490 op/s; 41196/203484 objects misplaced (20.245%); 30 MiB/s, 490 objects/s recovering 2024-02-05T07:36:24.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:23 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:24.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:23 smithi007 ceph-mon[29098]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-05T07:36:24.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:23 smithi007 ceph-mon[37158]: pgmap v953: 172 pgs: 3 active+remapped+backfilling, 3 peering, 11 active+remapped+backfill_toofull, 155 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 5.7 KiB/s rd, 39 KiB/s wr, 490 op/s; 41196/203484 objects misplaced (20.245%); 30 MiB/s, 490 objects/s recovering 2024-02-05T07:36:24.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:23 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:24.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:23 smithi007 ceph-mon[37158]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-05T07:36:24.180 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:24.220 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:24.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:24.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:23 smithi090 ceph-mon[33213]: pgmap v953: 172 pgs: 3 active+remapped+backfilling, 3 peering, 11 active+remapped+backfill_toofull, 155 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 5.7 KiB/s rd, 39 KiB/s wr, 490 op/s; 41196/203484 objects misplaced (20.245%); 30 MiB/s, 490 objects/s recovering 2024-02-05T07:36:24.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:23 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:24.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:23 smithi090 ceph-mon[33213]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-05T07:36:24.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:24.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:24.365 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:24.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:24.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:24.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:24.510 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:24.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:24.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:24.635 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:24.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:24.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:24.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:24.791 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:24.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:24.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:24.924 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:24.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:25.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:25.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:25.069 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:25.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:25.104 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:25.114 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T07:36:25.167 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:25.169 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:25.179 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:25.197 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:25.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:25.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:25.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:25.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:25.380 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:25.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:25.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:25.509 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:25.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:25.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:25.643 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:25.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:25.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:25 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-05T07:36:25.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:25 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-05T07:36:25.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:25 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-05T07:36:25.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:25 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-05T07:36:25.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:25.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:25.775 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:25.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:25.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:25.909 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:25.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:25.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:26.011 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:26.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:25 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-05T07:36:26.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:25 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-05T07:36:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:26.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:26.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:26.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:26.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:26.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:26.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:26.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:26.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:26.336 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:26.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:26.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:26.476 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:26.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:26.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:26.578 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:26.604 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:26.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:26.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:26.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:26.752 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:26.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:26.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:26.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:26.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:26.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:26 smithi007 ceph-mon[29098]: pgmap v954: 172 pgs: 3 active+remapped+backfilling, 11 active+remapped+backfill_toofull, 158 active+clean; 4.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 5.6 KiB/s rd, 33 KiB/s wr, 767 op/s; 40217/196575 objects misplaced (20.459%); 26 MiB/s, 418 objects/s recovering 2024-02-05T07:36:26.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:26 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-02-05T07:36:26.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:26 smithi007 ceph-mon[29098]: osdmap e529: 8 total, 8 up, 5 in 2024-02-05T07:36:26.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:26 smithi007 ceph-mon[37158]: pgmap v954: 172 pgs: 3 active+remapped+backfilling, 11 active+remapped+backfill_toofull, 158 active+clean; 4.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 5.6 KiB/s rd, 33 KiB/s wr, 767 op/s; 40217/196575 objects misplaced (20.459%); 26 MiB/s, 418 objects/s recovering 2024-02-05T07:36:26.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:26 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-02-05T07:36:26.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:26 smithi007 ceph-mon[37158]: osdmap e529: 8 total, 8 up, 5 in 2024-02-05T07:36:26.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:27.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:26 smithi090 ceph-mon[33213]: pgmap v954: 172 pgs: 3 active+remapped+backfilling, 11 active+remapped+backfill_toofull, 158 active+clean; 4.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 5.6 KiB/s rd, 33 KiB/s wr, 767 op/s; 40217/196575 objects misplaced (20.459%); 26 MiB/s, 418 objects/s recovering 2024-02-05T07:36:27.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:26 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-02-05T07:36:27.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:26 smithi090 ceph-mon[33213]: osdmap e529: 8 total, 8 up, 5 in 2024-02-05T07:36:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:27.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:27.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:27.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:27.171 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:27.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:27.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:27.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:27.304 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:27.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:27.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:27.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:27.438 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:27.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:27.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:27.570 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:27.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:27.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:27.715 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:27.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:27.817 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:27.857 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:27.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:27.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:27.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:27 smithi007 ceph-mon[29098]: osdmap e530: 8 total, 8 up, 5 in 2024-02-05T07:36:27.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:27 smithi007 ceph-mon[37158]: osdmap e530: 8 total, 8 up, 5 in 2024-02-05T07:36:27.924 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:27 smithi090 ceph-mon[33213]: osdmap e530: 8 total, 8 up, 5 in 2024-02-05T07:36:27.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:27.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:27.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:28.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:28.138 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:28.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:28.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:28.240 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:28.271 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:28.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:28.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:28.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:28.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:28.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:28.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:28.536 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:28.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:28.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:28.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:28.671 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:28.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:28.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:28.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:28.815 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:28.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:28.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:28.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:28 smithi007 ceph-mon[29098]: pgmap v957: 172 pgs: 3 active+remapped+backfilling, 11 active+remapped+backfill_toofull, 158 active+clean; 3.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 992 op/s; 35927/186918 objects misplaced (19.221%); 33 MiB/s, 521 objects/s recovering 2024-02-05T07:36:28.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:28 smithi007 ceph-mon[29098]: osdmap e531: 8 total, 8 up, 5 in 2024-02-05T07:36:28.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:28 smithi007 ceph-mon[37158]: pgmap v957: 172 pgs: 3 active+remapped+backfilling, 11 active+remapped+backfill_toofull, 158 active+clean; 3.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 992 op/s; 35927/186918 objects misplaced (19.221%); 33 MiB/s, 521 objects/s recovering 2024-02-05T07:36:28.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:28 smithi007 ceph-mon[37158]: osdmap e531: 8 total, 8 up, 5 in 2024-02-05T07:36:28.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:28.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:28.949 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:28.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:28 smithi090 ceph-mon[33213]: pgmap v957: 172 pgs: 3 active+remapped+backfilling, 11 active+remapped+backfill_toofull, 158 active+clean; 3.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 992 op/s; 35927/186918 objects misplaced (19.221%); 33 MiB/s, 521 objects/s recovering 2024-02-05T07:36:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:28 smithi090 ceph-mon[33213]: osdmap e531: 8 total, 8 up, 5 in 2024-02-05T07:36:29.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:29.081 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:29.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:29.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:29.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:29.211 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:29.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:29.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:29.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:29.343 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:29.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:29.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:29.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:29.476 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:29.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:29.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:29.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:29.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:29.623 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:29.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:29.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:29.755 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:29.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:29.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:29.886 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:29.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:29.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:29 smithi007 ceph-mon[37158]: osdmap e532: 8 total, 8 up, 5 in 2024-02-05T07:36:29.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-05T07:36:29.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:29 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-05T07:36:29.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:29 smithi007 ceph-mon[29098]: osdmap e532: 8 total, 8 up, 5 in 2024-02-05T07:36:29.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-05T07:36:29.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:29 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-05T07:36:29.987 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:29 smithi090 ceph-mon[33213]: osdmap e532: 8 total, 8 up, 5 in 2024-02-05T07:36:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-05T07:36:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:29 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-05T07:36:30.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:30.045 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:30.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:30.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:30.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:30.197 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:30.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:30.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:30.329 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:30.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:30.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:30.471 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:30.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:30.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:30.603 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:30.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:30.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:30.705 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:30.759 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:30.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:30.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:30.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:30 smithi007 ceph-mon[29098]: pgmap v960: 171 pgs: 3 active+remapped+backfilling, 11 active+remapped+backfill_toofull, 157 active+clean; 3.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 34993/180804 objects misplaced (19.354%); 24 MiB/s, 383 objects/s recovering 2024-02-05T07:36:30.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-05T07:36:30.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:30 smithi007 ceph-mon[29098]: osdmap e533: 8 total, 8 up, 5 in 2024-02-05T07:36:30.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:30 smithi007 ceph-mon[37158]: pgmap v960: 171 pgs: 3 active+remapped+backfilling, 11 active+remapped+backfill_toofull, 157 active+clean; 3.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 34993/180804 objects misplaced (19.354%); 24 MiB/s, 383 objects/s recovering 2024-02-05T07:36:30.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-05T07:36:30.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:30 smithi007 ceph-mon[37158]: osdmap e533: 8 total, 8 up, 5 in 2024-02-05T07:36:30.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:30.903 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:30.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:30.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:30.990 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:30 smithi090 ceph-mon[33213]: pgmap v960: 171 pgs: 3 active+remapped+backfilling, 11 active+remapped+backfill_toofull, 157 active+clean; 3.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 34993/180804 objects misplaced (19.354%); 24 MiB/s, 383 objects/s recovering 2024-02-05T07:36:30.990 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-05T07:36:30.990 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:30 smithi090 ceph-mon[33213]: osdmap e533: 8 total, 8 up, 5 in 2024-02-05T07:36:31.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:31.031 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:31.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:31.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:31.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:31.171 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:31.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:31.232 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:31.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:31.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:36:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:36:31.303 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:31.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:31.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:31.312 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:36:31.341 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:36:31.364 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a[29094]: 2024-02-05T07:36:31.188+0000 7fe3564d3700 -1 mon.a@0(leader).osd e534 definitely_dead 0 2024-02-05T07:36:31.369 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:36:31.399 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:36:31.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:31.426 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:36:31.434 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:31.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:31.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:31.452 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:36:31.481 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:36:31.508 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:36:31.510 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:36:31.510 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:36:31.510 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:36:31.510 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:36:31.510 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:36:31.510 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:36:31.510 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:36:31.510 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:36:31.536 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:31.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:31.566 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:31.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:31.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:31.695 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:36:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:marked in osd.2. 2024-02-05T07:36:31.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:31.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:31.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:31.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:31.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-mon[29098]: osdmap e534: 8 total, 8 up, 5 in 2024-02-05T07:36:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:36:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1723911912' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:36:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:36:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2701293037' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-05T07:36:31.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-05T07:36:31.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1004220443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:36:31.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-mon[37158]: osdmap e534: 8 total, 8 up, 5 in 2024-02-05T07:36:31.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:36:31.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1723911912' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:36:31.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:36:31.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2701293037' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-05T07:36:31.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-05T07:36:31.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:31 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1004220443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:36:31.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:31.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:31.958 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:31.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:31 smithi090 ceph-mon[33213]: osdmap e534: 8 total, 8 up, 5 in 2024-02-05T07:36:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:31 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:36:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:31 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1723911912' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:36:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:31 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:36:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:31 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2701293037' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-05T07:36:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:31 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-05T07:36:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:31 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1004220443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:36:32.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:32.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:32.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:32.214 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:32.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:32.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:32.344 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:32.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:32.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:32.349 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-02-05T07:36:32.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:32.470 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:32.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:32.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:32.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:32.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:32.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:32.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:32.734 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:32.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:32.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:32.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:32.861 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:32.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:32 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:32 smithi007 ceph-mon[29098]: pgmap v963: 171 pgs: 2 peering, 3 active+remapped+backfilling, 11 active+remapped+backfill_toofull, 155 active+clean; 3.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 32102/170691 objects misplaced (18.807%); 7.3 MiB/s, 116 objects/s recovering 2024-02-05T07:36:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:32 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:36:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:32 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:36:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:32 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-05T07:36:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:32 smithi007 ceph-mon[29098]: osdmap e535: 8 total, 8 up, 6 in 2024-02-05T07:36:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:32 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:32 smithi007 ceph-mon[37158]: pgmap v963: 171 pgs: 2 peering, 3 active+remapped+backfilling, 11 active+remapped+backfill_toofull, 155 active+clean; 3.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 32102/170691 objects misplaced (18.807%); 7.3 MiB/s, 116 objects/s recovering 2024-02-05T07:36:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:32 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:36:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:32 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:36:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:32 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-05T07:36:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:32 smithi007 ceph-mon[37158]: osdmap e535: 8 total, 8 up, 6 in 2024-02-05T07:36:32.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:32.991 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:32.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:32.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:32 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:32 smithi090 ceph-mon[33213]: pgmap v963: 171 pgs: 2 peering, 3 active+remapped+backfilling, 11 active+remapped+backfill_toofull, 155 active+clean; 3.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 32102/170691 objects misplaced (18.807%); 7.3 MiB/s, 116 objects/s recovering 2024-02-05T07:36:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:32 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:36:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:32 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:36:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:32 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-05T07:36:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:32 smithi090 ceph-mon[33213]: osdmap e535: 8 total, 8 up, 6 in 2024-02-05T07:36:33.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:33.126 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:33.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:33.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:33.228 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:33.257 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:33.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:33.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:33.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:33.384 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:33.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:33.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:33.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:33.514 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:33.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:33.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:33.638 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:33.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:33.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:33.740 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:33.768 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:33.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:33.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:33.794 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:36:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:36:33.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[29098]: 3.15 scrub starts 2024-02-05T07:36:33.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[29098]: 3.15 scrub ok 2024-02-05T07:36:33.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[29098]: 1.6 scrub starts 2024-02-05T07:36:33.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[29098]: 1.6 scrub ok 2024-02-05T07:36:33.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[29098]: osdmap e536: 8 total, 8 up, 6 in 2024-02-05T07:36:33.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[29098]: 10.18 continuing backfill to osd.1 from (446'4495,534'10329] MIN to 534'10329 2024-02-05T07:36:33.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[29098]: 10.18 continuing backfill to osd.3 from (446'4495,534'10329] MIN to 534'10329 2024-02-05T07:36:33.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[29098]: 10.12 starting backfill to osd.2 from (455'5293,522'9981] MIN to 534'10401 2024-02-05T07:36:33.796 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[29098]: 10.12 continuing backfill to osd.3 from (480'7096,534'10401] 10:48053da9:::benchmark_data_smithi007_157098_object119686:head to 534'10401 2024-02-05T07:36:33.796 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[29098]: 10.12 continuing backfill to osd.6 from (480'7096,534'10401] 10:48053da9:::benchmark_data_smithi007_157098_object119686:head to 534'10401 2024-02-05T07:36:33.796 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[29098]: 10.19 scrub starts 2024-02-05T07:36:33.796 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[29098]: 10.19 scrub ok 2024-02-05T07:36:33.796 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-05T07:36:33.796 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-05T07:36:33.796 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[37158]: 3.15 scrub starts 2024-02-05T07:36:33.796 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[37158]: 3.15 scrub ok 2024-02-05T07:36:33.796 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[37158]: 1.6 scrub starts 2024-02-05T07:36:33.796 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[37158]: 1.6 scrub ok 2024-02-05T07:36:33.796 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[37158]: osdmap e536: 8 total, 8 up, 6 in 2024-02-05T07:36:33.797 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[37158]: 10.18 continuing backfill to osd.1 from (446'4495,534'10329] MIN to 534'10329 2024-02-05T07:36:33.797 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[37158]: 10.18 continuing backfill to osd.3 from (446'4495,534'10329] MIN to 534'10329 2024-02-05T07:36:33.797 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[37158]: 10.12 starting backfill to osd.2 from (455'5293,522'9981] MIN to 534'10401 2024-02-05T07:36:33.797 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[37158]: 10.12 continuing backfill to osd.3 from (480'7096,534'10401] 10:48053da9:::benchmark_data_smithi007_157098_object119686:head to 534'10401 2024-02-05T07:36:33.797 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[37158]: 10.12 continuing backfill to osd.6 from (480'7096,534'10401] 10:48053da9:::benchmark_data_smithi007_157098_object119686:head to 534'10401 2024-02-05T07:36:33.797 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[37158]: 10.19 scrub starts 2024-02-05T07:36:33.797 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[37158]: 10.19 scrub ok 2024-02-05T07:36:33.797 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-05T07:36:33.797 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:33 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-05T07:36:33.870 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:33.898 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:33.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:33.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:34.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:33 smithi090 ceph-mon[33213]: 3.15 scrub starts 2024-02-05T07:36:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:33 smithi090 ceph-mon[33213]: 3.15 scrub ok 2024-02-05T07:36:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:33 smithi090 ceph-mon[33213]: 1.6 scrub starts 2024-02-05T07:36:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:33 smithi090 ceph-mon[33213]: 1.6 scrub ok 2024-02-05T07:36:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:33 smithi090 ceph-mon[33213]: osdmap e536: 8 total, 8 up, 6 in 2024-02-05T07:36:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:33 smithi090 ceph-mon[33213]: 10.18 continuing backfill to osd.1 from (446'4495,534'10329] MIN to 534'10329 2024-02-05T07:36:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:33 smithi090 ceph-mon[33213]: 10.18 continuing backfill to osd.3 from (446'4495,534'10329] MIN to 534'10329 2024-02-05T07:36:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:33 smithi090 ceph-mon[33213]: 10.12 starting backfill to osd.2 from (455'5293,522'9981] MIN to 534'10401 2024-02-05T07:36:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:33 smithi090 ceph-mon[33213]: 10.12 continuing backfill to osd.3 from (480'7096,534'10401] 10:48053da9:::benchmark_data_smithi007_157098_object119686:head to 534'10401 2024-02-05T07:36:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:33 smithi090 ceph-mon[33213]: 10.12 continuing backfill to osd.6 from (480'7096,534'10401] 10:48053da9:::benchmark_data_smithi007_157098_object119686:head to 534'10401 2024-02-05T07:36:34.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:33 smithi090 ceph-mon[33213]: 10.19 scrub starts 2024-02-05T07:36:34.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:33 smithi090 ceph-mon[33213]: 10.19 scrub ok 2024-02-05T07:36:34.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:33 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-05T07:36:34.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:33 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-05T07:36:34.028 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:34.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:34.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:34.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:34.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:34.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:34.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:34.305 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:34.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:34.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:34.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:34.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:34.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:34.569 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:34.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:34.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:34.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:34.701 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:34.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:34.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:34.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:34.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:34.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:34.988 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:34.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:34.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:35.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:36:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: pgmap v966: 170 pgs: 2 remapped+peering, 11 peering, 2 active+remapped+backfilling, 9 active+remapped+backfill_toofull, 146 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 967 op/s; 26826/169200 objects misplaced (15.855%); 17 MiB/s, 277 objects/s recovering 2024-02-05T07:36:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: 10.10 scrub starts 2024-02-05T07:36:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: 10.10 scrub ok 2024-02-05T07:36:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-02-05T07:36:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: 10.11 starting backfill to osd.2 from (0'0,0'0] MAX to 534'10483 2024-02-05T07:36:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: 1.1 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: 1.7 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: 1.a starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: 1.5 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: osdmap e537: 8 total, 8 up, 6 in 2024-02-05T07:36:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: 1.2 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: 1.8 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: 1.9 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: 10.13 continuing backfill to osd.1 from (469'6157,534'10214] 10:cf6f950d:::benchmark_data_smithi007_157098_object123847:head to 534'10214 2024-02-05T07:36:35.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: 1.a scrub starts 2024-02-05T07:36:35.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: 1.a scrub ok 2024-02-05T07:36:35.026 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:34 smithi090 ceph-mon[33213]: osdmap e538: 8 total, 8 up, 6 in 2024-02-05T07:36:35.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: pgmap v966: 170 pgs: 2 remapped+peering, 11 peering, 2 active+remapped+backfilling, 9 active+remapped+backfill_toofull, 146 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 967 op/s; 26826/169200 objects misplaced (15.855%); 17 MiB/s, 277 objects/s recovering 2024-02-05T07:36:35.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: 10.10 scrub starts 2024-02-05T07:36:35.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: 10.10 scrub ok 2024-02-05T07:36:35.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:35.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-02-05T07:36:35.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: 10.11 starting backfill to osd.2 from (0'0,0'0] MAX to 534'10483 2024-02-05T07:36:35.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: 1.1 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: 1.7 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: 1.a starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: 1.5 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: osdmap e537: 8 total, 8 up, 6 in 2024-02-05T07:36:35.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: 1.2 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: 1.8 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: 1.9 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: 10.13 continuing backfill to osd.1 from (469'6157,534'10214] 10:cf6f950d:::benchmark_data_smithi007_157098_object123847:head to 534'10214 2024-02-05T07:36:35.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: 1.a scrub starts 2024-02-05T07:36:35.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: 1.a scrub ok 2024-02-05T07:36:35.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[37158]: osdmap e538: 8 total, 8 up, 6 in 2024-02-05T07:36:35.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: pgmap v966: 170 pgs: 2 remapped+peering, 11 peering, 2 active+remapped+backfilling, 9 active+remapped+backfill_toofull, 146 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 967 op/s; 26826/169200 objects misplaced (15.855%); 17 MiB/s, 277 objects/s recovering 2024-02-05T07:36:35.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: 10.10 scrub starts 2024-02-05T07:36:35.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: 10.10 scrub ok 2024-02-05T07:36:35.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:35.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-02-05T07:36:35.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: 10.11 starting backfill to osd.2 from (0'0,0'0] MAX to 534'10483 2024-02-05T07:36:35.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: 1.1 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: 1.7 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: 1.a starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: 1.5 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: osdmap e537: 8 total, 8 up, 6 in 2024-02-05T07:36:35.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: 1.2 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: 1.8 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: 1.9 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:36:35.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: 10.13 continuing backfill to osd.1 from (469'6157,534'10214] 10:cf6f950d:::benchmark_data_smithi007_157098_object123847:head to 534'10214 2024-02-05T07:36:35.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: 1.a scrub starts 2024-02-05T07:36:35.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: 1.a scrub ok 2024-02-05T07:36:35.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:34 smithi007 ceph-mon[29098]: osdmap e538: 8 total, 8 up, 6 in 2024-02-05T07:36:35.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:35.150 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:35.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:35.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:35.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:35.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:35.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:35.411 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:35.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:35.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:35.545 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:35.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:35.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:35.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:35.676 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:35.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:35.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:35.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:35.808 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:35.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:35.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:35.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:35.944 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:35.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:36.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:35 smithi090 ceph-mon[33213]: 6.1d scrub starts 2024-02-05T07:36:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:35 smithi090 ceph-mon[33213]: 6.1d scrub ok 2024-02-05T07:36:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:35 smithi090 ceph-mon[33213]: osdmap e539: 8 total, 8 up, 6 in 2024-02-05T07:36:36.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:36.071 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:36.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:36.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:35 smithi007 ceph-mon[29098]: 6.1d scrub starts 2024-02-05T07:36:36.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:35 smithi007 ceph-mon[29098]: 6.1d scrub ok 2024-02-05T07:36:36.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:35 smithi007 ceph-mon[29098]: osdmap e539: 8 total, 8 up, 6 in 2024-02-05T07:36:36.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:35 smithi007 ceph-mon[37158]: 6.1d scrub starts 2024-02-05T07:36:36.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:35 smithi007 ceph-mon[37158]: 6.1d scrub ok 2024-02-05T07:36:36.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:35 smithi007 ceph-mon[37158]: osdmap e539: 8 total, 8 up, 6 in 2024-02-05T07:36:36.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:36.203 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:36.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:36.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:36.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:36.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:36.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:36.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:36.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:36.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:36.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:36.587 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:36.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:36.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:36.724 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:36.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:36.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:36.849 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:36.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:36.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:36.980 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:36.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:37.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:36 smithi090 ceph-mon[33213]: pgmap v969: 170 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 2 remapped+peering, 11 peering, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 149 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 17818/165540 objects misplaced (10.764%) 2024-02-05T07:36:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:36 smithi090 ceph-mon[33213]: 4.1 scrub starts 2024-02-05T07:36:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:36 smithi090 ceph-mon[33213]: 4.1 scrub ok 2024-02-05T07:36:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:36 smithi090 ceph-mon[33213]: osdmap e540: 8 total, 8 up, 6 in 2024-02-05T07:36:37.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:37.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:36:37.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:37.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:37.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:37.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:36 smithi007 ceph-mon[29098]: pgmap v969: 170 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 2 remapped+peering, 11 peering, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 149 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 17818/165540 objects misplaced (10.764%) 2024-02-05T07:36:37.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:36 smithi007 ceph-mon[29098]: 4.1 scrub starts 2024-02-05T07:36:37.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:36 smithi007 ceph-mon[29098]: 4.1 scrub ok 2024-02-05T07:36:37.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:36 smithi007 ceph-mon[29098]: osdmap e540: 8 total, 8 up, 6 in 2024-02-05T07:36:37.171 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:36 smithi007 ceph-mon[37158]: pgmap v969: 170 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 2 remapped+peering, 11 peering, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 149 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 17818/165540 objects misplaced (10.764%) 2024-02-05T07:36:37.171 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:36 smithi007 ceph-mon[37158]: 4.1 scrub starts 2024-02-05T07:36:37.171 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:36 smithi007 ceph-mon[37158]: 4.1 scrub ok 2024-02-05T07:36:37.171 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:36 smithi007 ceph-mon[37158]: osdmap e540: 8 total, 8 up, 6 in 2024-02-05T07:36:37.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:37.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:37.254 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:37.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:37.349 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6, 3, 2] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:36:37.350 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:36:37.350 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-02-05T07:36:37.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:36:37.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:37.380 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:37.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:37.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:37.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:37.543 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:37.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:37.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:37.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:37.648 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:37.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:37.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:37.712 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:37 smithi007 ceph-mon[29098]: 10.17 scrub starts 2024-02-05T07:36:37.712 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:37 smithi007 ceph-mon[29098]: 10.17 scrub ok 2024-02-05T07:36:37.713 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:37 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T07:36:37.713 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:37 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T07:36:37.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:37.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:37.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:37.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:37.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:37.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:37.974 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:37 smithi007 ceph-mon[37158]: 10.17 scrub starts 2024-02-05T07:36:37.974 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:37 smithi007 ceph-mon[37158]: 10.17 scrub ok 2024-02-05T07:36:37.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:37 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T07:36:37.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:37 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T07:36:38.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:37 smithi090 ceph-mon[33213]: 10.17 scrub starts 2024-02-05T07:36:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:37 smithi090 ceph-mon[33213]: 10.17 scrub ok 2024-02-05T07:36:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:37 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T07:36:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:37 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T07:36:38.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:38.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:38.068 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:38.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:38.169 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:38.226 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:38.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:38.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:38.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:38.357 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:38.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:38.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:38.488 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:36:38.488 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:36:38.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:38.497 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:38.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:38.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:38.639 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:38.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:38.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:38.781 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:38.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:38.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:38.909 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:38.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:38.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:38 smithi007 ceph-mon[29098]: pgmap v972: 169 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped, 4 peering, 2 active+remapped+backfill_toofull, 159 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 4.5 KiB/s rd, 3 op/s; 9078/160428 objects misplaced (5.659%) 2024-02-05T07:36:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:38 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-05T07:36:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:38 smithi007 ceph-mon[29098]: osdmap e541: 8 total, 8 up, 6 in 2024-02-05T07:36:38.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:38 smithi007 ceph-mon[37158]: pgmap v972: 169 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped, 4 peering, 2 active+remapped+backfill_toofull, 159 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 4.5 KiB/s rd, 3 op/s; 9078/160428 objects misplaced (5.659%) 2024-02-05T07:36:38.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:38 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-05T07:36:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:38 smithi007 ceph-mon[37158]: osdmap e541: 8 total, 8 up, 6 in 2024-02-05T07:36:39.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:39.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:38 smithi090 ceph-mon[33213]: pgmap v972: 169 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped, 4 peering, 2 active+remapped+backfill_toofull, 159 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 4.5 KiB/s rd, 3 op/s; 9078/160428 objects misplaced (5.659%) 2024-02-05T07:36:39.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:38 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-05T07:36:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:38 smithi090 ceph-mon[33213]: osdmap e541: 8 total, 8 up, 6 in 2024-02-05T07:36:39.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:39.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:39.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:39.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:39.181 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:39.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:39.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:39.326 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:39.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:39.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:39.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:39.523 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:39.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:39.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:39.657 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:39.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:39.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:39.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:39.790 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:39.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:39.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:39.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:39.924 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:39.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:39.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:39 smithi090 ceph-mon[33213]: osdmap e542: 8 total, 8 up, 6 in 2024-02-05T07:36:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:39 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:36:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:39 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:40.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:40.041 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:40.087 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:40.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:40.156 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:39 smithi007 ceph-mon[29098]: osdmap e542: 8 total, 8 up, 6 in 2024-02-05T07:36:40.157 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:39 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:36:40.157 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:39 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:40.157 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:39 smithi007 ceph-mon[37158]: osdmap e542: 8 total, 8 up, 6 in 2024-02-05T07:36:40.157 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:39 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:36:40.157 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:39 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:40.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:40.220 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:40.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:40.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:40.322 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:40.365 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:40.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:40.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:40.467 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:40.496 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:40.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:40.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:40.579 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T07:36:40.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:40.665 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:40.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:40.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:40.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:40.780 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:40.794 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:40.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:40.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:40.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:40 smithi007 ceph-mon[29098]: pgmap v975: 169 pgs: 2 active+remapped+backfilling, 2 clean+premerge+peered, 4 peering, 3 active+remapped+backfill_toofull, 158 active+clean; 3.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 3.2 KiB/s rd, 2 op/s; 8900/154086 objects misplaced (5.776%) 2024-02-05T07:36:40.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:40 smithi007 ceph-mon[29098]: 3.1d scrub starts 2024-02-05T07:36:40.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:40 smithi007 ceph-mon[29098]: 3.1d scrub ok 2024-02-05T07:36:40.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:40 smithi007 ceph-mon[29098]: osdmap e543: 8 total, 8 up, 6 in 2024-02-05T07:36:40.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:40 smithi007 ceph-mon[37158]: pgmap v975: 169 pgs: 2 active+remapped+backfilling, 2 clean+premerge+peered, 4 peering, 3 active+remapped+backfill_toofull, 158 active+clean; 3.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 3.2 KiB/s rd, 2 op/s; 8900/154086 objects misplaced (5.776%) 2024-02-05T07:36:40.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:40 smithi007 ceph-mon[37158]: 3.1d scrub starts 2024-02-05T07:36:40.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:40 smithi007 ceph-mon[37158]: 3.1d scrub ok 2024-02-05T07:36:40.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:40 smithi007 ceph-mon[37158]: osdmap e543: 8 total, 8 up, 6 in 2024-02-05T07:36:40.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:40.924 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:40.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:41.001 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:40 smithi090 ceph-mon[33213]: pgmap v975: 169 pgs: 2 active+remapped+backfilling, 2 clean+premerge+peered, 4 peering, 3 active+remapped+backfill_toofull, 158 active+clean; 3.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 3.2 KiB/s rd, 2 op/s; 8900/154086 objects misplaced (5.776%) 2024-02-05T07:36:41.001 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:40 smithi090 ceph-mon[33213]: 3.1d scrub starts 2024-02-05T07:36:41.001 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:40 smithi090 ceph-mon[33213]: 3.1d scrub ok 2024-02-05T07:36:41.001 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:40 smithi090 ceph-mon[33213]: osdmap e543: 8 total, 8 up, 6 in 2024-02-05T07:36:41.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:41.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:41.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:41.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:41.180 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:41.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:41.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:36:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:36:41.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:41.312 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:41.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:41.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:41.446 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:41.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:41.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:41.583 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:41.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:41.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:41.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:41.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:41.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:41.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:41.817 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:41.853 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:41.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:41.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:41.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:41.985 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:41.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:42.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:41 smithi007 ceph-mon[29098]: osdmap e544: 8 total, 8 up, 6 in 2024-02-05T07:36:42.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:41 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:36:42.002 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:41 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:36:42.002 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:41 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:36:42.002 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:41 smithi007 ceph-mon[29098]: 5.2 scrub starts 2024-02-05T07:36:42.002 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:41 smithi007 ceph-mon[29098]: 5.2 scrub ok 2024-02-05T07:36:42.002 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:41 smithi007 ceph-mon[37158]: osdmap e544: 8 total, 8 up, 6 in 2024-02-05T07:36:42.002 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:41 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:36:42.002 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:41 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:36:42.002 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:41 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:36:42.002 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:41 smithi007 ceph-mon[37158]: 5.2 scrub starts 2024-02-05T07:36:42.002 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:41 smithi007 ceph-mon[37158]: 5.2 scrub ok 2024-02-05T07:36:42.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:41 smithi090 ceph-mon[33213]: osdmap e544: 8 total, 8 up, 6 in 2024-02-05T07:36:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:41 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:36:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:41 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:36:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:41 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:36:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:41 smithi090 ceph-mon[33213]: 5.2 scrub starts 2024-02-05T07:36:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:41 smithi090 ceph-mon[33213]: 5.2 scrub ok 2024-02-05T07:36:42.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:42.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:42.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:42.292 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:42.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:42.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:42.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:42.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:42.553 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:42.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:42.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:42.655 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:42.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:42.679 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:42.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:42.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:42.819 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:42.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:42.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:42.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:42.949 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:42.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:42.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:42 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:42 smithi090 ceph-mon[33213]: pgmap v978: 168 pgs: 1 active+remapped+backfilling, 1 clean+premerge+peered, 3 active+remapped+backfill_toofull, 163 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 6802/144336 objects misplaced (4.713%) 2024-02-05T07:36:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:42 smithi090 ceph-mon[33213]: 5.c scrub starts 2024-02-05T07:36:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:42 smithi090 ceph-mon[33213]: 5.c scrub ok 2024-02-05T07:36:43.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:43.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:43.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:43.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:42 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:43.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:42 smithi007 ceph-mon[29098]: pgmap v978: 168 pgs: 1 active+remapped+backfilling, 1 clean+premerge+peered, 3 active+remapped+backfill_toofull, 163 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 6802/144336 objects misplaced (4.713%) 2024-02-05T07:36:43.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:42 smithi007 ceph-mon[29098]: 5.c scrub starts 2024-02-05T07:36:43.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:42 smithi007 ceph-mon[29098]: 5.c scrub ok 2024-02-05T07:36:43.153 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:42 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:43.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:42 smithi007 ceph-mon[37158]: pgmap v978: 168 pgs: 1 active+remapped+backfilling, 1 clean+premerge+peered, 3 active+remapped+backfill_toofull, 163 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 6802/144336 objects misplaced (4.713%) 2024-02-05T07:36:43.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:42 smithi007 ceph-mon[37158]: 5.c scrub starts 2024-02-05T07:36:43.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:42 smithi007 ceph-mon[37158]: 5.c scrub ok 2024-02-05T07:36:43.181 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:43.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:43.209 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:43.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:43.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:43.338 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:43.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:43.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:43.483 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:43.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:43.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:43.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:43.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:43.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:36:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:36:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:36:43.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:43.757 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:43.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:43.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:43.899 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:43.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:43.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:44.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:43 smithi090 ceph-mon[33213]: 5.10 scrub starts 2024-02-05T07:36:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:43 smithi090 ceph-mon[33213]: 5.10 scrub ok 2024-02-05T07:36:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:43 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1477539279' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:36:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:43 smithi090 ceph-mon[33213]: 6.10 scrub starts 2024-02-05T07:36:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:43 smithi090 ceph-mon[33213]: 6.10 scrub ok 2024-02-05T07:36:44.041 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:44.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:44.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:44.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:43 smithi007 ceph-mon[29098]: 5.10 scrub starts 2024-02-05T07:36:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:43 smithi007 ceph-mon[29098]: 5.10 scrub ok 2024-02-05T07:36:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:43 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1477539279' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:36:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:43 smithi007 ceph-mon[29098]: 6.10 scrub starts 2024-02-05T07:36:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:43 smithi007 ceph-mon[29098]: 6.10 scrub ok 2024-02-05T07:36:44.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:43 smithi007 ceph-mon[37158]: 5.10 scrub starts 2024-02-05T07:36:44.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:43 smithi007 ceph-mon[37158]: 5.10 scrub ok 2024-02-05T07:36:44.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:43 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1477539279' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:36:44.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:43 smithi007 ceph-mon[37158]: 6.10 scrub starts 2024-02-05T07:36:44.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:43 smithi007 ceph-mon[37158]: 6.10 scrub ok 2024-02-05T07:36:44.180 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:44.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:44.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:44.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:44.313 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:44.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:44.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:44.455 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:44.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:44.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:44.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:44.584 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:44.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:44.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:44.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:44.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:44.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:44.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:44.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:44.846 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:44.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:44.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:44 smithi007 ceph-mon[37158]: pgmap v979: 168 pgs: 1 active+remapped+backfilling, 1 clean+premerge+peered, 3 active+remapped+backfill_toofull, 163 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 910 B/s rd, 0 op/s; 6802/143736 objects misplaced (4.732%) 2024-02-05T07:36:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:44 smithi007 ceph-mon[37158]: 6.1e scrub starts 2024-02-05T07:36:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:44 smithi007 ceph-mon[37158]: 6.1e scrub ok 2024-02-05T07:36:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:44 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:36:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:44 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:36:44.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:44 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:44 smithi007 ceph-mon[29098]: pgmap v979: 168 pgs: 1 active+remapped+backfilling, 1 clean+premerge+peered, 3 active+remapped+backfill_toofull, 163 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 910 B/s rd, 0 op/s; 6802/143736 objects misplaced (4.732%) 2024-02-05T07:36:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:44 smithi007 ceph-mon[29098]: 6.1e scrub starts 2024-02-05T07:36:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:44 smithi007 ceph-mon[29098]: 6.1e scrub ok 2024-02-05T07:36:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:44 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:36:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:44 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:36:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:44 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:44.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:44.979 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:44.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:44.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:44 smithi090 ceph-mon[33213]: pgmap v979: 168 pgs: 1 active+remapped+backfilling, 1 clean+premerge+peered, 3 active+remapped+backfill_toofull, 163 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 910 B/s rd, 0 op/s; 6802/143736 objects misplaced (4.732%) 2024-02-05T07:36:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:44 smithi090 ceph-mon[33213]: 6.1e scrub starts 2024-02-05T07:36:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:44 smithi090 ceph-mon[33213]: 6.1e scrub ok 2024-02-05T07:36:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:44 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:36:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:44 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:36:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:44 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:36:45.081 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:45.128 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:45.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:45.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:45.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:45.257 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:45.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:45.359 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:45.399 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:45.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:45.450 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:36:45.450 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:36:45.450 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:36:45.450 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:36:45.450 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:36:45.450 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:36:45.450 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:36:45.450 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:36:45.451 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:36:45.451 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:36:45.451 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:36:45.451 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:36:45.451 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:36:45.451 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:36:45.451 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:36:45.451 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:36:45.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:45.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:45.524 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:45.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:45.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:45.661 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:45.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:45.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:45.786 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:45.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:45.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:45.926 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:45.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:45.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:45 smithi090 ceph-mon[33213]: osdmap e545: 8 total, 8 up, 6 in 2024-02-05T07:36:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:45 smithi090 ceph-mon[33213]: 3.11 scrub starts 2024-02-05T07:36:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:45 smithi090 ceph-mon[33213]: 3.11 scrub ok 2024-02-05T07:36:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:45 smithi090 ceph-mon[33213]: 10.f scrub starts 2024-02-05T07:36:46.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:46.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:46.165 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:46.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:45 smithi007 ceph-mon[29098]: osdmap e545: 8 total, 8 up, 6 in 2024-02-05T07:36:46.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:45 smithi007 ceph-mon[29098]: 3.11 scrub starts 2024-02-05T07:36:46.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:45 smithi007 ceph-mon[29098]: 3.11 scrub ok 2024-02-05T07:36:46.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:45 smithi007 ceph-mon[29098]: 10.f scrub starts 2024-02-05T07:36:46.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:45 smithi007 ceph-mon[37158]: osdmap e545: 8 total, 8 up, 6 in 2024-02-05T07:36:46.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:45 smithi007 ceph-mon[37158]: 3.11 scrub starts 2024-02-05T07:36:46.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:45 smithi007 ceph-mon[37158]: 3.11 scrub ok 2024-02-05T07:36:46.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:45 smithi007 ceph-mon[37158]: 10.f scrub starts 2024-02-05T07:36:46.192 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:46.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:46.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:46.319 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:46.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:46.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:46.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:46.444 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:46.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:46.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:46.572 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:46.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:46.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:46.674 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:46.700 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:46.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:46.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:46.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:46.832 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:46.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:46.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:46.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:46.963 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:46.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:46.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:46 smithi090 ceph-mon[33213]: pgmap v981: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s; 6598/134961 objects misplaced (4.889%) 2024-02-05T07:36:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:46 smithi090 ceph-mon[33213]: osdmap e546: 8 total, 8 up, 6 in 2024-02-05T07:36:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:36:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:46 smithi090 ceph-mon[33213]: 10.f scrub ok 2024-02-05T07:36:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:46 smithi090 ceph-mon[33213]: 5.1e scrub starts 2024-02-05T07:36:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:46 smithi090 ceph-mon[33213]: 5.1e scrub ok 2024-02-05T07:36:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:46 smithi090 ceph-mon[33213]: 3.3 scrub starts 2024-02-05T07:36:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:46 smithi090 ceph-mon[33213]: 3.3 scrub ok 2024-02-05T07:36:47.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:47.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:47.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[29098]: pgmap v981: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s; 6598/134961 objects misplaced (4.889%) 2024-02-05T07:36:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[29098]: osdmap e546: 8 total, 8 up, 6 in 2024-02-05T07:36:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:36:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[29098]: 10.f scrub ok 2024-02-05T07:36:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[29098]: 5.1e scrub starts 2024-02-05T07:36:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[29098]: 5.1e scrub ok 2024-02-05T07:36:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[29098]: 3.3 scrub starts 2024-02-05T07:36:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[29098]: 3.3 scrub ok 2024-02-05T07:36:47.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[37158]: pgmap v981: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s; 6598/134961 objects misplaced (4.889%) 2024-02-05T07:36:47.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[37158]: osdmap e546: 8 total, 8 up, 6 in 2024-02-05T07:36:47.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:36:47.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[37158]: 10.f scrub ok 2024-02-05T07:36:47.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[37158]: 5.1e scrub starts 2024-02-05T07:36:47.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[37158]: 5.1e scrub ok 2024-02-05T07:36:47.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[37158]: 3.3 scrub starts 2024-02-05T07:36:47.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:46 smithi007 ceph-mon[37158]: 3.3 scrub ok 2024-02-05T07:36:47.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:47.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:47.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:47.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:47.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:47.406 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:47.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:47.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:47.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:47.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:47.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:47.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:47.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:47.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:47.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:47.812 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:47.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:47.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:47.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:47.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[29098]: 3.12 scrub starts 2024-02-05T07:36:47.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[29098]: 3.12 scrub ok 2024-02-05T07:36:47.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[29098]: 4.f scrub starts 2024-02-05T07:36:47.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[29098]: 4.f scrub ok 2024-02-05T07:36:47.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[29098]: osdmap e547: 8 total, 8 up, 6 in 2024-02-05T07:36:47.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[29098]: 10.13 scrub starts 2024-02-05T07:36:47.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:36:47.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:36:47.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[29098]: 5.1c scrub starts 2024-02-05T07:36:47.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[29098]: 5.1c scrub ok 2024-02-05T07:36:47.949 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[37158]: 3.12 scrub starts 2024-02-05T07:36:47.949 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[37158]: 3.12 scrub ok 2024-02-05T07:36:47.949 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[37158]: 4.f scrub starts 2024-02-05T07:36:47.949 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[37158]: 4.f scrub ok 2024-02-05T07:36:47.949 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[37158]: osdmap e547: 8 total, 8 up, 6 in 2024-02-05T07:36:47.949 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[37158]: 10.13 scrub starts 2024-02-05T07:36:47.949 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:36:47.949 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:36:47.950 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[37158]: 5.1c scrub starts 2024-02-05T07:36:47.950 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:47 smithi007 ceph-mon[37158]: 5.1c scrub ok 2024-02-05T07:36:47.953 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:47.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:47 smithi090 ceph-mon[33213]: 3.12 scrub starts 2024-02-05T07:36:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:47 smithi090 ceph-mon[33213]: 3.12 scrub ok 2024-02-05T07:36:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:47 smithi090 ceph-mon[33213]: 4.f scrub starts 2024-02-05T07:36:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:47 smithi090 ceph-mon[33213]: 4.f scrub ok 2024-02-05T07:36:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:47 smithi090 ceph-mon[33213]: osdmap e547: 8 total, 8 up, 6 in 2024-02-05T07:36:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:47 smithi090 ceph-mon[33213]: 10.13 scrub starts 2024-02-05T07:36:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:47 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:36:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:47 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:36:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:47 smithi090 ceph-mon[33213]: 5.1c scrub starts 2024-02-05T07:36:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:47 smithi090 ceph-mon[33213]: 5.1c scrub ok 2024-02-05T07:36:48.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:48.086 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:48.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:48.188 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:48.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:48.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:48.352 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:48.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:48.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:48.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:48.494 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:48.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:48.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:48.635 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:48.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:48.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:48.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:48.768 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:48.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:48.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:48.870 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:48.898 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:48.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:48.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:49.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:48 smithi090 ceph-mon[33213]: 3.8 scrub starts 2024-02-05T07:36:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:48 smithi090 ceph-mon[33213]: 3.8 scrub ok 2024-02-05T07:36:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:48 smithi090 ceph-mon[33213]: pgmap v984: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 5307/124968 objects misplaced (4.247%); 3.8 MiB/s, 60 objects/s recovering 2024-02-05T07:36:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:48 smithi090 ceph-mon[33213]: osdmap e548: 8 total, 8 up, 6 in 2024-02-05T07:36:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:48 smithi090 ceph-mon[33213]: 10.13 scrub ok 2024-02-05T07:36:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:48 smithi090 ceph-mon[33213]: 10.5 scrub starts 2024-02-05T07:36:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:48 smithi090 ceph-mon[33213]: 3.1b scrub starts 2024-02-05T07:36:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:48 smithi090 ceph-mon[33213]: 3.1b scrub ok 2024-02-05T07:36:49.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:49.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:49.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:49.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[29098]: 3.8 scrub starts 2024-02-05T07:36:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[29098]: 3.8 scrub ok 2024-02-05T07:36:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[29098]: pgmap v984: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 5307/124968 objects misplaced (4.247%); 3.8 MiB/s, 60 objects/s recovering 2024-02-05T07:36:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[29098]: osdmap e548: 8 total, 8 up, 6 in 2024-02-05T07:36:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[29098]: 10.13 scrub ok 2024-02-05T07:36:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[29098]: 10.5 scrub starts 2024-02-05T07:36:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[29098]: 3.1b scrub starts 2024-02-05T07:36:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[29098]: 3.1b scrub ok 2024-02-05T07:36:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[37158]: 3.8 scrub starts 2024-02-05T07:36:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[37158]: 3.8 scrub ok 2024-02-05T07:36:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[37158]: pgmap v984: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 5307/124968 objects misplaced (4.247%); 3.8 MiB/s, 60 objects/s recovering 2024-02-05T07:36:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[37158]: osdmap e548: 8 total, 8 up, 6 in 2024-02-05T07:36:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[37158]: 10.13 scrub ok 2024-02-05T07:36:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[37158]: 10.5 scrub starts 2024-02-05T07:36:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[37158]: 3.1b scrub starts 2024-02-05T07:36:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:48 smithi007 ceph-mon[37158]: 3.1b scrub ok 2024-02-05T07:36:49.180 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:49.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:49.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:49.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:49.329 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:49.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:49.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:49.479 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:49.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:49.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:49.610 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:49.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:49.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:49.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:49.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:49.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:49.875 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:49.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:49.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:49.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:49 smithi007 ceph-mon[29098]: 4.11 scrub starts 2024-02-05T07:36:49.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:49 smithi007 ceph-mon[29098]: 4.11 scrub ok 2024-02-05T07:36:49.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:49 smithi007 ceph-mon[29098]: 10.5 scrub ok 2024-02-05T07:36:49.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:49 smithi007 ceph-mon[29098]: 10.18 scrub starts 2024-02-05T07:36:49.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:49 smithi007 ceph-mon[29098]: 10.b scrub starts 2024-02-05T07:36:49.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:49 smithi007 ceph-mon[37158]: 4.11 scrub starts 2024-02-05T07:36:49.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:49 smithi007 ceph-mon[37158]: 4.11 scrub ok 2024-02-05T07:36:49.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:49 smithi007 ceph-mon[37158]: 10.5 scrub ok 2024-02-05T07:36:49.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:49 smithi007 ceph-mon[37158]: 10.18 scrub starts 2024-02-05T07:36:49.898 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:49 smithi007 ceph-mon[37158]: 10.b scrub starts 2024-02-05T07:36:49.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:50.022 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:50.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:50.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:50.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:50.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:50.172 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:50.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:50.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:49 smithi090 ceph-mon[33213]: 4.11 scrub starts 2024-02-05T07:36:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:49 smithi090 ceph-mon[33213]: 4.11 scrub ok 2024-02-05T07:36:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:49 smithi090 ceph-mon[33213]: 10.5 scrub ok 2024-02-05T07:36:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:49 smithi090 ceph-mon[33213]: 10.18 scrub starts 2024-02-05T07:36:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:49 smithi090 ceph-mon[33213]: 10.b scrub starts 2024-02-05T07:36:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:50.324 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:50.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:50.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:50.452 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:50.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:50.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:50.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:50.597 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:50.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:50.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:50.699 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:50.736 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:50.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:50.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:50.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:50.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:50.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:50.934 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:36:50.945 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:50.949 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:36:50.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:51.034 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:51.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:51.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:51.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[29098]: 10.15 scrub starts 2024-02-05T07:36:51.176 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[29098]: pgmap v986: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s; 5126/118728 objects misplaced (4.317%); 4.9 MiB/s, 78 objects/s recovering 2024-02-05T07:36:51.176 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[29098]: 10.15 scrub ok 2024-02-05T07:36:51.176 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[29098]: 5.1d scrub starts 2024-02-05T07:36:51.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[29098]: 5.1d scrub ok 2024-02-05T07:36:51.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[29098]: 10.18 scrub ok 2024-02-05T07:36:51.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[29098]: 10.b scrub ok 2024-02-05T07:36:51.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[29098]: 5.11 scrub starts 2024-02-05T07:36:51.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[29098]: 5.11 scrub ok 2024-02-05T07:36:51.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[37158]: 10.15 scrub starts 2024-02-05T07:36:51.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[37158]: pgmap v986: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s; 5126/118728 objects misplaced (4.317%); 4.9 MiB/s, 78 objects/s recovering 2024-02-05T07:36:51.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[37158]: 10.15 scrub ok 2024-02-05T07:36:51.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[37158]: 5.1d scrub starts 2024-02-05T07:36:51.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[37158]: 5.1d scrub ok 2024-02-05T07:36:51.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[37158]: 10.18 scrub ok 2024-02-05T07:36:51.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[37158]: 10.b scrub ok 2024-02-05T07:36:51.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[37158]: 5.11 scrub starts 2024-02-05T07:36:51.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:50 smithi007 ceph-mon[37158]: 5.11 scrub ok 2024-02-05T07:36:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:51.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:51.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:36:51 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:36:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:50 smithi090 ceph-mon[33213]: 10.15 scrub starts 2024-02-05T07:36:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:50 smithi090 ceph-mon[33213]: pgmap v986: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 2.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s; 5126/118728 objects misplaced (4.317%); 4.9 MiB/s, 78 objects/s recovering 2024-02-05T07:36:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:50 smithi090 ceph-mon[33213]: 10.15 scrub ok 2024-02-05T07:36:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:50 smithi090 ceph-mon[33213]: 5.1d scrub starts 2024-02-05T07:36:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:50 smithi090 ceph-mon[33213]: 5.1d scrub ok 2024-02-05T07:36:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:50 smithi090 ceph-mon[33213]: 10.18 scrub ok 2024-02-05T07:36:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:50 smithi090 ceph-mon[33213]: 10.b scrub ok 2024-02-05T07:36:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:50 smithi090 ceph-mon[33213]: 5.11 scrub starts 2024-02-05T07:36:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:50 smithi090 ceph-mon[33213]: 5.11 scrub ok 2024-02-05T07:36:51.283 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:51.323 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:51.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:51.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:51.425 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:51.458 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:51.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:51.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:51.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:51.603 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:51.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:51.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:51.734 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:51.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:51.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:51.824 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:36:51.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:51.864 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:51.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:51.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:51.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:52.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:52.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:52.113 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:52.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:52.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:51 smithi007 ceph-mon[29098]: from='client.15723 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:36:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:51 smithi007 ceph-mon[29098]: 10.14 scrub starts 2024-02-05T07:36:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:51 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3261660494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:36:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:51 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:36:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:51 smithi007 ceph-mon[29098]: 10.14 scrub ok 2024-02-05T07:36:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:51 smithi007 ceph-mon[29098]: 1.a scrub starts 2024-02-05T07:36:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:51 smithi007 ceph-mon[29098]: 1.a scrub ok 2024-02-05T07:36:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:51 smithi007 ceph-mon[37158]: from='client.15723 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:36:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:51 smithi007 ceph-mon[37158]: 10.14 scrub starts 2024-02-05T07:36:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:51 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3261660494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:36:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:51 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:36:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:51 smithi007 ceph-mon[37158]: 10.14 scrub ok 2024-02-05T07:36:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:51 smithi007 ceph-mon[37158]: 1.a scrub starts 2024-02-05T07:36:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:51 smithi007 ceph-mon[37158]: 1.a scrub ok 2024-02-05T07:36:52.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:52.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:51 smithi090 ceph-mon[33213]: from='client.15723 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:36:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:51 smithi090 ceph-mon[33213]: 10.14 scrub starts 2024-02-05T07:36:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:51 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3261660494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:36:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:51 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:36:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:51 smithi090 ceph-mon[33213]: 10.14 scrub ok 2024-02-05T07:36:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:51 smithi090 ceph-mon[33213]: 1.a scrub starts 2024-02-05T07:36:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:51 smithi090 ceph-mon[33213]: 1.a scrub ok 2024-02-05T07:36:52.286 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:52.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:52.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:52.414 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:52.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:52.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:52.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:52.545 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:52.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:52.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:52.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:52.690 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:52.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:52.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:52.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:36:52.821 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:52.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:52.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:52.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:52 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:52 smithi007 ceph-mon[29098]: pgmap v987: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 op/s; 4454/108522 objects misplaced (4.104%) 2024-02-05T07:36:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:52 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:36:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:52 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:36:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:52 smithi007 ceph-mon[29098]: osdmap e549: 8 total, 8 up, 6 in 2024-02-05T07:36:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:52 smithi007 ceph-mon[29098]: 1.8 scrub starts 2024-02-05T07:36:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:52 smithi007 ceph-mon[29098]: 1.8 scrub ok 2024-02-05T07:36:52.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:52 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:52.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:52 smithi007 ceph-mon[37158]: pgmap v987: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 op/s; 4454/108522 objects misplaced (4.104%) 2024-02-05T07:36:52.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:52 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:36:52.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:52 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:36:52.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:52 smithi007 ceph-mon[37158]: osdmap e549: 8 total, 8 up, 6 in 2024-02-05T07:36:52.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:52 smithi007 ceph-mon[37158]: 1.8 scrub starts 2024-02-05T07:36:52.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:52 smithi007 ceph-mon[37158]: 1.8 scrub ok 2024-02-05T07:36:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:52.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:53.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:53.082 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:53.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:53.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:53.184 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:53.209 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:53.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:53.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:52 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:36:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:52 smithi090 ceph-mon[33213]: pgmap v987: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 op/s; 4454/108522 objects misplaced (4.104%) 2024-02-05T07:36:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:52 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:36:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:52 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:36:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:52 smithi090 ceph-mon[33213]: osdmap e549: 8 total, 8 up, 6 in 2024-02-05T07:36:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:52 smithi090 ceph-mon[33213]: 1.8 scrub starts 2024-02-05T07:36:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:52 smithi090 ceph-mon[33213]: 1.8 scrub ok 2024-02-05T07:36:53.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:53.342 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:53.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:53.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:53.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:53.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:53.590 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:53.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:53.624 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:53.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:53.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:53.770 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:53.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:53.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:53.804 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:36:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:36:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:36:53.814 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":986,"stamp":"2024-02-05T07:36:49.333191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2581183679,"num_objects":39576,"num_object_clones":0,"num_object_copies":118728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5126,"num_objects_unfound":0,"num_objects_dirty":39576,"num_whiteouts":0,"num_read":1510,"num_read_kb":1279,"num_write":158681,"num_write_kb":4904228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16267,"num_bytes_recovered":1038687666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100583,"ondisk_log_size":100583,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":520,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":13447900,"kb_used_data":8775052,"kb_used_omap":13,"kb_used_meta":4672498,"kb_avail":549023012,"statfs":{"total":575970213888,"available":562199564288,"internally_reserved":0,"allocated":8985653248,"data_stored":8980914934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":4784638688},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,10,7,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-392560640,"num_objects":-5990,"num_object_clones":0,"num_object_copies":-17970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1472,"num_objects_unfound":0,"num_objects_dirty":-5990,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5990,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":23658496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.574424"},"pg_stats":[{"pgid":"10.2","version":"545'10861","reported_seq":11173,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.959734+0000","last_change":"2024-02-05T07:35:20.520500+0000","last_active":"2024-02-05T07:36:44.959734+0000","last_peered":"2024-02-05T07:36:44.959734+0000","last_clean":"2024-02-05T07:36:44.959734+0000","last_became_active":"2024-02-05T07:35:18.799363+0000","last_became_peered":"2024-02-05T07:35:18.799363+0000","last_unstale":"2024-02-05T07:36:44.959734+0000","last_undegraded":"2024-02-05T07:36:44.959734+0000","last_fullsized":"2024-02-05T07:36:44.959734+0000","mapping_epoch":423,"log_start":"481'7172","ondisk_log_start":"481'7172","created":423,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8150","last_scrub_stamp":"2024-02-05T07:35:20.520372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:20.520372+0000","objects_scrubbed":4003,"log_size":3689,"log_dups_size":0,"ondisk_log_size":3689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":737,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2712,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":1191,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.819581+0000","last_change":"2024-02-05T07:36:35.619311+0000","last_active":"2024-02-05T07:36:46.819581+0000","last_peered":"2024-02-05T07:36:46.819581+0000","last_clean":"2024-02-05T07:36:46.819581+0000","last_became_active":"2024-02-05T07:36:35.618848+0000","last_became_peered":"2024-02-05T07:36:35.618848+0000","last_unstale":"2024-02-05T07:36:46.819581+0000","last_undegraded":"2024-02-05T07:36:46.819581+0000","last_fullsized":"2024-02-05T07:36:46.819581+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:35:08.100173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:35:08.100173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"326'25","reported_seq":1077,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.760030+0000","last_change":"2024-02-05T07:32:27.643863+0000","last_active":"2024-02-05T07:36:44.760030+0000","last_peered":"2024-02-05T07:36:44.760030+0000","last_clean":"2024-02-05T07:36:44.760030+0000","last_became_active":"2024-02-05T07:28:26.605082+0000","last_became_peered":"2024-02-05T07:28:26.605082+0000","last_unstale":"2024-02-05T07:36:44.760030+0000","last_undegraded":"2024-02-05T07:36:44.760030+0000","last_fullsized":"2024-02-05T07:36:44.760030+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:32:27.643788+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:32:27.643788+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":589,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.819874+0000","last_change":"2024-02-05T07:33:34.736291+0000","last_active":"2024-02-05T07:36:46.819874+0000","last_peered":"2024-02-05T07:36:46.819874+0000","last_clean":"2024-02-05T07:36:46.819874+0000","last_became_active":"2024-02-05T07:27:16.716461+0000","last_became_peered":"2024-02-05T07:27:16.716461+0000","last_unstale":"2024-02-05T07:36:46.819874+0000","last_undegraded":"2024-02-05T07:36:46.819874+0000","last_fullsized":"2024-02-05T07:36:46.819874+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:34.736160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:34.736160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"538'13","reported_seq":36,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.771592+0000","last_change":"2024-02-05T07:36:32.822462+0000","last_active":"2024-02-05T07:36:44.771592+0000","last_peered":"2024-02-05T07:36:44.771592+0000","last_clean":"2024-02-05T07:36:44.771592+0000","last_became_active":"2024-02-05T07:36:32.743613+0000","last_became_peered":"2024-02-05T07:36:32.743613+0000","last_unstale":"2024-02-05T07:36:44.771592+0000","last_undegraded":"2024-02-05T07:36:44.771592+0000","last_fullsized":"2024-02-05T07:36:44.771592+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:34:29.851477+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:34:29.851477+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1002,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.760670+0000","last_change":"2024-02-05T07:36:32.711145+0000","last_active":"2024-02-05T07:36:44.760670+0000","last_peered":"2024-02-05T07:36:44.760670+0000","last_clean":"2024-02-05T07:36:44.760670+0000","last_became_active":"2024-02-05T07:36:32.710651+0000","last_became_peered":"2024-02-05T07:36:32.710651+0000","last_unstale":"2024-02-05T07:36:44.760670+0000","last_undegraded":"2024-02-05T07:36:44.760670+0000","last_fullsized":"2024-02-05T07:36:44.760670+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:30.708205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:32:30.708205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"545'10787","reported_seq":11109,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:45.357930+0000","last_change":"2024-02-05T07:36:33.730223+0000","last_active":"2024-02-05T07:36:45.357930+0000","last_peered":"2024-02-05T07:36:45.357930+0000","last_clean":"2024-02-05T07:36:45.357930+0000","last_became_active":"2024-02-05T07:36:33.707045+0000","last_became_peered":"2024-02-05T07:36:33.707045+0000","last_unstale":"2024-02-05T07:36:45.357930+0000","last_undegraded":"2024-02-05T07:36:45.357930+0000","last_fullsized":"2024-02-05T07:36:45.357930+0000","mapping_epoch":536,"log_start":"476'6679","ondisk_log_start":"476'6679","created":423,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8061","last_scrub_stamp":"2024-02-05T07:35:20.146489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:20.146489+0000","objects_scrubbed":4072,"log_size":4108,"log_dups_size":0,"ondisk_log_size":4108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87752704,"num_objects":1339,"num_object_clones":0,"num_object_copies":4017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2733,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":482,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.766155+0000","last_change":"2024-02-05T07:36:35.627948+0000","last_active":"2024-02-05T07:36:44.766155+0000","last_peered":"2024-02-05T07:36:44.766155+0000","last_clean":"2024-02-05T07:36:44.766155+0000","last_became_active":"2024-02-05T07:36:35.627588+0000","last_became_peered":"2024-02-05T07:36:35.627588+0000","last_unstale":"2024-02-05T07:36:44.766155+0000","last_undegraded":"2024-02-05T07:36:44.766155+0000","last_fullsized":"2024-02-05T07:36:44.766155+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:41.228420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:41.228420+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":69,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:45.160016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289523,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":868568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"326'31","reported_seq":1104,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759879+0000","last_change":"2024-02-05T07:36:32.772355+0000","last_active":"2024-02-05T07:36:44.759879+0000","last_peered":"2024-02-05T07:36:44.759879+0000","last_clean":"2024-02-05T07:36:44.759879+0000","last_became_active":"2024-02-05T07:36:32.717742+0000","last_became_peered":"2024-02-05T07:36:32.717742+0000","last_unstale":"2024-02-05T07:36:44.759879+0000","last_undegraded":"2024-02-05T07:36:44.759879+0000","last_fullsized":"2024-02-05T07:36:44.759879+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'31","last_scrub_stamp":"2024-02-05T07:33:30.750548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.750548+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":862,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830964+0000","last_change":"2024-02-05T07:33:44.229582+0000","last_active":"2024-02-05T07:36:47.830964+0000","last_peered":"2024-02-05T07:36:47.830964+0000","last_clean":"2024-02-05T07:36:47.830964+0000","last_became_active":"2024-02-05T07:28:25.791005+0000","last_became_peered":"2024-02-05T07:28:25.791005+0000","last_unstale":"2024-02-05T07:36:47.830964+0000","last_undegraded":"2024-02-05T07:36:47.830964+0000","last_fullsized":"2024-02-05T07:36:47.830964+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.229489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.229489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":217,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759982+0000","last_change":"2024-02-05T07:36:41.530864+0000","last_active":"2024-02-05T07:36:44.759982+0000","last_peered":"2024-02-05T07:36:44.759982+0000","last_clean":"2024-02-05T07:36:44.759982+0000","last_became_active":"2024-02-05T07:34:30.678610+0000","last_became_peered":"2024-02-05T07:34:30.678610+0000","last_unstale":"2024-02-05T07:36:44.759982+0000","last_undegraded":"2024-02-05T07:36:44.759982+0000","last_fullsized":"2024-02-05T07:36:44.759982+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:41.530749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:41.530749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:00.868644+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.766250+0000","last_change":"2024-02-05T07:36:32.724079+0000","last_active":"2024-02-05T07:36:44.766250+0000","last_peered":"2024-02-05T07:36:44.766250+0000","last_clean":"2024-02-05T07:36:44.766250+0000","last_became_active":"2024-02-05T07:36:32.723796+0000","last_became_peered":"2024-02-05T07:36:32.723796+0000","last_unstale":"2024-02-05T07:36:44.766250+0000","last_undegraded":"2024-02-05T07:36:44.766250+0000","last_fullsized":"2024-02-05T07:36:44.766250+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:23.161521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:23.161521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"548'10836","reported_seq":11153,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:48.786335+0000","last_change":"2024-02-05T07:35:20.368840+0000","last_active":"2024-02-05T07:36:48.786335+0000","last_peered":"2024-02-05T07:36:48.786335+0000","last_clean":"2024-02-05T07:36:48.786335+0000","last_became_active":"2024-02-05T07:35:18.791210+0000","last_became_peered":"2024-02-05T07:35:18.791210+0000","last_unstale":"2024-02-05T07:36:48.786335+0000","last_undegraded":"2024-02-05T07:36:48.786335+0000","last_fullsized":"2024-02-05T07:36:48.786335+0000","mapping_epoch":423,"log_start":"475'6596","ondisk_log_start":"475'6596","created":423,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8038","last_scrub_stamp":"2024-02-05T07:35:20.368714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:20.368714+0000","objects_scrubbed":3967,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":477,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76611584,"num_objects":1169,"num_object_clones":0,"num_object_copies":3507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2798,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"326'24","reported_seq":1051,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.760512+0000","last_change":"2024-02-05T07:35:13.047805+0000","last_active":"2024-02-05T07:36:44.760512+0000","last_peered":"2024-02-05T07:36:44.760512+0000","last_clean":"2024-02-05T07:36:44.760512+0000","last_became_active":"2024-02-05T07:28:25.781007+0000","last_became_peered":"2024-02-05T07:28:25.781007+0000","last_unstale":"2024-02-05T07:36:44.760512+0000","last_undegraded":"2024-02-05T07:36:44.760512+0000","last_fullsized":"2024-02-05T07:36:44.760512+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:35:13.047702+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:35:13.047702+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1022,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759087+0000","last_change":"2024-02-05T07:36:32.703504+0000","last_active":"2024-02-05T07:36:44.759087+0000","last_peered":"2024-02-05T07:36:44.759087+0000","last_clean":"2024-02-05T07:36:44.759087+0000","last_became_active":"2024-02-05T07:36:32.702469+0000","last_became_peered":"2024-02-05T07:36:32.702469+0000","last_unstale":"2024-02-05T07:36:44.759087+0000","last_undegraded":"2024-02-05T07:36:44.759087+0000","last_fullsized":"2024-02-05T07:36:44.759087+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:28.670286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:32:28.670286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":850,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.760538+0000","last_change":"2024-02-05T07:36:32.712676+0000","last_active":"2024-02-05T07:36:44.760538+0000","last_peered":"2024-02-05T07:36:44.760538+0000","last_clean":"2024-02-05T07:36:44.760538+0000","last_became_active":"2024-02-05T07:36:32.712332+0000","last_became_peered":"2024-02-05T07:36:32.712332+0000","last_unstale":"2024-02-05T07:36:44.760538+0000","last_undegraded":"2024-02-05T07:36:44.760538+0000","last_fullsized":"2024-02-05T07:36:44.760538+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:14.031714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:35:14.031714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":598,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.820317+0000","last_change":"2024-02-05T07:33:28.696549+0000","last_active":"2024-02-05T07:36:46.820317+0000","last_peered":"2024-02-05T07:36:46.820317+0000","last_clean":"2024-02-05T07:36:46.820317+0000","last_became_active":"2024-02-05T07:30:40.498529+0000","last_became_peered":"2024-02-05T07:30:40.498529+0000","last_unstale":"2024-02-05T07:36:46.820317+0000","last_undegraded":"2024-02-05T07:36:46.820317+0000","last_fullsized":"2024-02-05T07:36:46.820317+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.696462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.696462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"545'10800","reported_seq":11110,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:45.417795+0000","last_change":"2024-02-05T07:35:19.907738+0000","last_active":"2024-02-05T07:36:45.417795+0000","last_peered":"2024-02-05T07:36:45.417795+0000","last_clean":"2024-02-05T07:36:45.417795+0000","last_became_active":"2024-02-05T07:35:18.797257+0000","last_became_peered":"2024-02-05T07:35:18.797257+0000","last_unstale":"2024-02-05T07:36:45.417795+0000","last_undegraded":"2024-02-05T07:36:45.417795+0000","last_fullsized":"2024-02-05T07:36:45.417795+0000","mapping_epoch":423,"log_start":"482'7249","ondisk_log_start":"482'7249","created":423,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8152","last_scrub_stamp":"2024-02-05T07:35:19.907566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:19.907566+0000","objects_scrubbed":3999,"log_size":3551,"log_dups_size":0,"ondisk_log_size":3551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87359488,"num_objects":1333,"num_object_clones":0,"num_object_copies":3999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2666,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":350,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.770848+0000","last_change":"2024-02-05T07:36:35.630640+0000","last_active":"2024-02-05T07:36:44.770848+0000","last_peered":"2024-02-05T07:36:44.770848+0000","last_clean":"2024-02-05T07:36:44.770848+0000","last_became_active":"2024-02-05T07:36:35.627308+0000","last_became_peered":"2024-02-05T07:36:35.627308+0000","last_unstale":"2024-02-05T07:36:44.770848+0000","last_undegraded":"2024-02-05T07:36:44.770848+0000","last_fullsized":"2024-02-05T07:36:44.770848+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:36:34.077864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:36:34.077864+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:45.159991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"326'34","reported_seq":1094,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.773411+0000","last_change":"2024-02-05T07:30:30.006087+0000","last_active":"2024-02-05T07:36:44.773411+0000","last_peered":"2024-02-05T07:36:44.773411+0000","last_clean":"2024-02-05T07:36:44.773411+0000","last_became_active":"2024-02-05T07:27:16.768454+0000","last_became_peered":"2024-02-05T07:27:16.768454+0000","last_unstale":"2024-02-05T07:36:44.773411+0000","last_undegraded":"2024-02-05T07:36:44.773411+0000","last_fullsized":"2024-02-05T07:36:44.773411+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'34","last_scrub_stamp":"2024-02-05T07:30:30.005988+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.005988+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":595,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821158+0000","last_change":"2024-02-05T07:30:40.508975+0000","last_active":"2024-02-05T07:36:46.821158+0000","last_peered":"2024-02-05T07:36:46.821158+0000","last_clean":"2024-02-05T07:36:46.821158+0000","last_became_active":"2024-02-05T07:30:40.508561+0000","last_became_peered":"2024-02-05T07:30:40.508561+0000","last_unstale":"2024-02-05T07:36:46.821158+0000","last_undegraded":"2024-02-05T07:36:46.821158+0000","last_fullsized":"2024-02-05T07:36:46.821158+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:30.982457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:30.982457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"538'11","reported_seq":1248,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.760623+0000","last_change":"2024-02-05T07:36:32.743569+0000","last_active":"2024-02-05T07:36:44.760623+0000","last_peered":"2024-02-05T07:36:44.760623+0000","last_clean":"2024-02-05T07:36:44.760623+0000","last_became_active":"2024-02-05T07:36:32.709942+0000","last_became_peered":"2024-02-05T07:36:32.709942+0000","last_unstale":"2024-02-05T07:36:44.760623+0000","last_undegraded":"2024-02-05T07:36:44.760623+0000","last_fullsized":"2024-02-05T07:36:44.760623+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'9","last_scrub_stamp":"2024-02-05T07:32:39.565833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:39.565833+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.765237+0000","last_change":"2024-02-05T07:36:32.744571+0000","last_active":"2024-02-05T07:36:44.765237+0000","last_peered":"2024-02-05T07:36:44.765237+0000","last_clean":"2024-02-05T07:36:44.765237+0000","last_became_active":"2024-02-05T07:36:32.744070+0000","last_became_peered":"2024-02-05T07:36:32.744070+0000","last_unstale":"2024-02-05T07:36:44.765237+0000","last_undegraded":"2024-02-05T07:36:44.765237+0000","last_fullsized":"2024-02-05T07:36:44.765237+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:38.025380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.025380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.e","version":"545'13726","reported_seq":13996,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:45.422265+0000","last_change":"2024-02-05T07:36:32.736314+0000","last_active":"2024-02-05T07:36:45.422265+0000","last_peered":"2024-02-05T07:36:45.422265+0000","last_clean":"2024-02-05T07:36:45.422265+0000","last_became_active":"2024-02-05T07:36:32.709815+0000","last_became_peered":"2024-02-05T07:36:32.709815+0000","last_unstale":"2024-02-05T07:36:45.422265+0000","last_undegraded":"2024-02-05T07:36:45.422265+0000","last_fullsized":"2024-02-05T07:36:45.422265+0000","mapping_epoch":535,"log_start":"515'10428","ondisk_log_start":"515'10428","created":423,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":1221,"ondisk_log_size":3298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164888576,"num_objects":2516,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13726,"num_write_kb":519744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":1073,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759535+0000","last_change":"2024-02-05T07:36:35.629574+0000","last_active":"2024-02-05T07:36:44.759535+0000","last_peered":"2024-02-05T07:36:44.759535+0000","last_clean":"2024-02-05T07:36:44.759535+0000","last_became_active":"2024-02-05T07:36:35.628935+0000","last_became_peered":"2024-02-05T07:36:35.628935+0000","last_unstale":"2024-02-05T07:36:44.759535+0000","last_undegraded":"2024-02-05T07:36:44.759535+0000","last_fullsized":"2024-02-05T07:36:44.759535+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:32:24.656057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:32:24.656057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1014,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.758983+0000","last_change":"2024-02-05T07:32:36.623960+0000","last_active":"2024-02-05T07:36:44.758983+0000","last_peered":"2024-02-05T07:36:44.758983+0000","last_clean":"2024-02-05T07:36:44.758983+0000","last_became_active":"2024-02-05T07:30:40.489212+0000","last_became_peered":"2024-02-05T07:30:40.489212+0000","last_unstale":"2024-02-05T07:36:44.758983+0000","last_undegraded":"2024-02-05T07:36:44.758983+0000","last_fullsized":"2024-02-05T07:36:44.758983+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:36.623865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:36.623865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"454'38","reported_seq":665,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821057+0000","last_change":"2024-02-05T07:34:38.943008+0000","last_active":"2024-02-05T07:36:46.821057+0000","last_peered":"2024-02-05T07:36:46.821057+0000","last_clean":"2024-02-05T07:36:46.821057+0000","last_became_active":"2024-02-05T07:30:40.496940+0000","last_became_peered":"2024-02-05T07:30:40.496940+0000","last_unstale":"2024-02-05T07:36:46.821057+0000","last_undegraded":"2024-02-05T07:36:46.821057+0000","last_fullsized":"2024-02-05T07:36:46.821057+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'38","last_scrub_stamp":"2024-02-05T07:34:38.942830+0000","last_deep_scrub":"454'38","last_deep_scrub_stamp":"2024-02-05T07:34:38.942830+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.942830+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":595,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821961+0000","last_change":"2024-02-05T07:33:48.701618+0000","last_active":"2024-02-05T07:36:46.821961+0000","last_peered":"2024-02-05T07:36:46.821961+0000","last_clean":"2024-02-05T07:36:46.821961+0000","last_became_active":"2024-02-05T07:30:40.508607+0000","last_became_peered":"2024-02-05T07:30:40.508607+0000","last_unstale":"2024-02-05T07:36:46.821961+0000","last_undegraded":"2024-02-05T07:36:46.821961+0000","last_fullsized":"2024-02-05T07:36:46.821961+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:48.701399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:48.701399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1019,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759395+0000","last_change":"2024-02-05T07:36:32.709346+0000","last_active":"2024-02-05T07:36:44.759395+0000","last_peered":"2024-02-05T07:36:44.759395+0000","last_clean":"2024-02-05T07:36:44.759395+0000","last_became_active":"2024-02-05T07:36:32.708979+0000","last_became_peered":"2024-02-05T07:36:32.708979+0000","last_unstale":"2024-02-05T07:36:44.759395+0000","last_undegraded":"2024-02-05T07:36:44.759395+0000","last_fullsized":"2024-02-05T07:36:44.759395+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:45.908064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:33:45.908064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"548'13751","reported_seq":14061,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:48.784740+0000","last_change":"2024-02-05T07:36:46.213073+0000","last_active":"2024-02-05T07:36:48.784740+0000","last_peered":"2024-02-05T07:36:48.784740+0000","last_clean":"2024-02-05T07:36:48.784740+0000","last_became_active":"2024-02-05T07:36:32.717068+0000","last_became_peered":"2024-02-05T07:36:32.717068+0000","last_unstale":"2024-02-05T07:36:48.784740+0000","last_undegraded":"2024-02-05T07:36:48.784740+0000","last_fullsized":"2024-02-05T07:36:48.784740+0000","mapping_epoch":535,"log_start":"514'10026","ondisk_log_start":"514'10026","created":423,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"546'13589","last_scrub_stamp":"2024-02-05T07:36:46.212874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.212874+0000","objects_scrubbed":2519,"log_size":3725,"log_dups_size":203,"ondisk_log_size":3725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:12.382989+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153026560,"num_objects":2335,"num_object_clones":0,"num_object_copies":7005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13751,"num_write_kb":514752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":1320,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.820448+0000","last_change":"2024-02-05T07:36:36.612245+0000","last_active":"2024-02-05T07:36:46.820448+0000","last_peered":"2024-02-05T07:36:46.820448+0000","last_clean":"2024-02-05T07:36:46.820448+0000","last_became_active":"2024-02-05T07:36:36.611863+0000","last_became_peered":"2024-02-05T07:36:36.611863+0000","last_unstale":"2024-02-05T07:36:46.820448+0000","last_undegraded":"2024-02-05T07:36:46.820448+0000","last_fullsized":"2024-02-05T07:36:46.820448+0000","mapping_epoch":367,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:28.874337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.874337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"266'10","reported_seq":707,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821254+0000","last_change":"2024-02-05T07:33:42.715242+0000","last_active":"2024-02-05T07:36:46.821254+0000","last_peered":"2024-02-05T07:36:46.821254+0000","last_clean":"2024-02-05T07:36:46.821254+0000","last_became_active":"2024-02-05T07:27:16.800207+0000","last_became_peered":"2024-02-05T07:27:16.800207+0000","last_unstale":"2024-02-05T07:36:46.821254+0000","last_undegraded":"2024-02-05T07:36:46.821254+0000","last_fullsized":"2024-02-05T07:36:46.821254+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'10","last_scrub_stamp":"2024-02-05T07:33:42.715162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:42.715162+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"326'35","reported_seq":1078,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759135+0000","last_change":"2024-02-05T07:36:35.442778+0000","last_active":"2024-02-05T07:36:44.759135+0000","last_peered":"2024-02-05T07:36:44.759135+0000","last_clean":"2024-02-05T07:36:44.759135+0000","last_became_active":"2024-02-05T07:28:26.598831+0000","last_became_peered":"2024-02-05T07:28:26.598831+0000","last_unstale":"2024-02-05T07:36:44.759135+0000","last_undegraded":"2024-02-05T07:36:44.759135+0000","last_fullsized":"2024-02-05T07:36:44.759135+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:36:35.442664+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:36:35.442664+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:55.060557+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":765,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.773183+0000","last_change":"2024-02-05T07:34:42.579613+0000","last_active":"2024-02-05T07:36:44.773183+0000","last_peered":"2024-02-05T07:36:44.773183+0000","last_clean":"2024-02-05T07:36:44.773183+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:36:44.773183+0000","last_undegraded":"2024-02-05T07:36:44.773183+0000","last_fullsized":"2024-02-05T07:36:44.773183+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:34:42.579486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:42.579486+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":894,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830127+0000","last_change":"2024-02-05T07:36:32.701554+0000","last_active":"2024-02-05T07:36:47.830127+0000","last_peered":"2024-02-05T07:36:47.830127+0000","last_clean":"2024-02-05T07:36:47.830127+0000","last_became_active":"2024-02-05T07:36:32.701203+0000","last_became_peered":"2024-02-05T07:36:32.701203+0000","last_unstale":"2024-02-05T07:36:47.830127+0000","last_undegraded":"2024-02-05T07:36:47.830127+0000","last_fullsized":"2024-02-05T07:36:47.830127+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:35.338979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:34:35.338979+0000","last_clean_scrub_stamp":"2024-02-05T07:34:35.338979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"545'13883","reported_seq":14141,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:45.415270+0000","last_change":"2024-02-05T07:33:38.775156+0000","last_active":"2024-02-05T07:36:45.415270+0000","last_peered":"2024-02-05T07:36:45.415270+0000","last_clean":"2024-02-05T07:36:45.415270+0000","last_became_active":"2024-02-05T07:33:38.774713+0000","last_became_peered":"2024-02-05T07:33:38.774713+0000","last_unstale":"2024-02-05T07:36:45.415270+0000","last_undegraded":"2024-02-05T07:36:45.415270+0000","last_fullsized":"2024-02-05T07:36:45.415270+0000","mapping_epoch":423,"log_start":"516'10600","ondisk_log_start":"516'10600","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3283,"log_dups_size":1221,"ondisk_log_size":3283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165216256,"num_objects":2521,"num_object_clones":0,"num_object_copies":7563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13883,"num_write_kb":524928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":956,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.831426+0000","last_change":"2024-02-05T07:36:35.619497+0000","last_active":"2024-02-05T07:36:47.831426+0000","last_peered":"2024-02-05T07:36:47.831426+0000","last_clean":"2024-02-05T07:36:47.831426+0000","last_became_active":"2024-02-05T07:36:35.619195+0000","last_became_peered":"2024-02-05T07:36:35.619195+0000","last_unstale":"2024-02-05T07:36:47.831426+0000","last_undegraded":"2024-02-05T07:36:47.831426+0000","last_fullsized":"2024-02-05T07:36:47.831426+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:29.214474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:29.214474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":710,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830272+0000","last_change":"2024-02-05T07:35:21.887728+0000","last_active":"2024-02-05T07:36:47.830272+0000","last_peered":"2024-02-05T07:36:47.830272+0000","last_clean":"2024-02-05T07:36:47.830272+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:36:47.830272+0000","last_undegraded":"2024-02-05T07:36:47.830272+0000","last_fullsized":"2024-02-05T07:36:47.830272+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:35:21.887636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:35:21.887636+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"326'25","reported_seq":1062,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.774303+0000","last_change":"2024-02-05T07:30:36.908113+0000","last_active":"2024-02-05T07:36:44.774303+0000","last_peered":"2024-02-05T07:36:44.774303+0000","last_clean":"2024-02-05T07:36:44.774303+0000","last_became_active":"2024-02-05T07:27:03.361342+0000","last_became_peered":"2024-02-05T07:27:03.361342+0000","last_unstale":"2024-02-05T07:36:44.774303+0000","last_undegraded":"2024-02-05T07:36:44.774303+0000","last_fullsized":"2024-02-05T07:36:44.774303+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:30:36.907989+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.907989+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":612,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.820879+0000","last_change":"2024-02-05T07:36:32.741872+0000","last_active":"2024-02-05T07:36:46.820879+0000","last_peered":"2024-02-05T07:36:46.820879+0000","last_clean":"2024-02-05T07:36:46.820879+0000","last_became_active":"2024-02-05T07:36:32.741377+0000","last_became_peered":"2024-02-05T07:36:32.741377+0000","last_unstale":"2024-02-05T07:36:46.820879+0000","last_undegraded":"2024-02-05T07:36:46.820879+0000","last_fullsized":"2024-02-05T07:36:46.820879+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:17.067204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:35:17.067204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":284,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.774503+0000","last_change":"2024-02-05T07:33:24.419375+0000","last_active":"2024-02-05T07:36:44.774503+0000","last_peered":"2024-02-05T07:36:44.774503+0000","last_clean":"2024-02-05T07:36:44.774503+0000","last_became_active":"2024-02-05T07:32:58.598709+0000","last_became_peered":"2024-02-05T07:32:58.598709+0000","last_unstale":"2024-02-05T07:36:44.774503+0000","last_undegraded":"2024-02-05T07:36:44.774503+0000","last_fullsized":"2024-02-05T07:36:44.774503+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:24.419250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:24.419250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"548'13718","reported_seq":13983,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:48.747122+0000","last_change":"2024-02-05T07:33:38.775383+0000","last_active":"2024-02-05T07:36:48.747122+0000","last_peered":"2024-02-05T07:36:48.747122+0000","last_clean":"2024-02-05T07:36:48.747122+0000","last_became_active":"2024-02-05T07:33:38.775048+0000","last_became_peered":"2024-02-05T07:33:38.775048+0000","last_unstale":"2024-02-05T07:36:48.747122+0000","last_undegraded":"2024-02-05T07:36:48.747122+0000","last_fullsized":"2024-02-05T07:36:48.747122+0000","mapping_epoch":423,"log_start":"514'9994","ondisk_log_start":"514'9994","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3724,"log_dups_size":174,"ondisk_log_size":3724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150798360,"num_objects":2302,"num_object_clones":0,"num_object_copies":6906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13718,"num_write_kb":512577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":928,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830676+0000","last_change":"2024-02-05T07:36:32.497773+0000","last_active":"2024-02-05T07:36:47.830676+0000","last_peered":"2024-02-05T07:36:47.830676+0000","last_clean":"2024-02-05T07:36:47.830676+0000","last_became_active":"2024-02-05T07:36:28.658583+0000","last_became_peered":"2024-02-05T07:36:28.658583+0000","last_unstale":"2024-02-05T07:36:47.830676+0000","last_undegraded":"2024-02-05T07:36:47.830676+0000","last_fullsized":"2024-02-05T07:36:47.830676+0000","mapping_epoch":302,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:36:32.497664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:36:32.497664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:39.115870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":870,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830423+0000","last_change":"2024-02-05T07:36:41.525416+0000","last_active":"2024-02-05T07:36:47.830423+0000","last_peered":"2024-02-05T07:36:47.830423+0000","last_clean":"2024-02-05T07:36:47.830423+0000","last_became_active":"2024-02-05T07:36:32.709025+0000","last_became_peered":"2024-02-05T07:36:32.709025+0000","last_unstale":"2024-02-05T07:36:47.830423+0000","last_undegraded":"2024-02-05T07:36:47.830423+0000","last_fullsized":"2024-02-05T07:36:47.830423+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:41.525301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:36:41.525301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:05.462600+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"326'29","reported_seq":686,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.776731+0000","last_change":"2024-02-05T07:36:32.797322+0000","last_active":"2024-02-05T07:36:44.776731+0000","last_peered":"2024-02-05T07:36:44.776731+0000","last_clean":"2024-02-05T07:36:44.776731+0000","last_became_active":"2024-02-05T07:36:32.712346+0000","last_became_peered":"2024-02-05T07:36:32.712346+0000","last_unstale":"2024-02-05T07:36:44.776731+0000","last_undegraded":"2024-02-05T07:36:44.776731+0000","last_fullsized":"2024-02-05T07:36:44.776731+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'29","last_scrub_stamp":"2024-02-05T07:35:21.877054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:21.877054+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:48.738378+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.764953+0000","last_change":"2024-02-05T07:36:32.733360+0000","last_active":"2024-02-05T07:36:44.764953+0000","last_peered":"2024-02-05T07:36:44.764953+0000","last_clean":"2024-02-05T07:36:44.764953+0000","last_became_active":"2024-02-05T07:36:32.732781+0000","last_became_peered":"2024-02-05T07:36:32.732781+0000","last_unstale":"2024-02-05T07:36:44.764953+0000","last_undegraded":"2024-02-05T07:36:44.764953+0000","last_fullsized":"2024-02-05T07:36:44.764953+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:34.799346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:34:34.799346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1014,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.775351+0000","last_change":"2024-02-05T07:36:32.710252+0000","last_active":"2024-02-05T07:36:44.775351+0000","last_peered":"2024-02-05T07:36:44.775351+0000","last_clean":"2024-02-05T07:36:44.775351+0000","last_became_active":"2024-02-05T07:36:32.709872+0000","last_became_peered":"2024-02-05T07:36:32.709872+0000","last_unstale":"2024-02-05T07:36:44.775351+0000","last_undegraded":"2024-02-05T07:36:44.775351+0000","last_fullsized":"2024-02-05T07:36:44.775351+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:33.257028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.257028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"545'10918","reported_seq":11240,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:45.425306+0000","last_change":"2024-02-05T07:36:33.722770+0000","last_active":"2024-02-05T07:36:45.425306+0000","last_peered":"2024-02-05T07:36:45.425306+0000","last_clean":"2024-02-05T07:36:45.425306+0000","last_became_active":"2024-02-05T07:36:33.698507+0000","last_became_peered":"2024-02-05T07:36:33.698507+0000","last_unstale":"2024-02-05T07:36:45.425306+0000","last_undegraded":"2024-02-05T07:36:45.425306+0000","last_fullsized":"2024-02-05T07:36:45.425306+0000","mapping_epoch":536,"log_start":"486'7516","ondisk_log_start":"486'7516","created":423,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8110","last_scrub_stamp":"2024-02-05T07:35:20.687974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:20.687974+0000","objects_scrubbed":4038,"log_size":3402,"log_dups_size":27,"ondisk_log_size":3402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80609280,"num_objects":1230,"num_object_clones":0,"num_object_copies":3690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2808,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":579,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830372+0000","last_change":"2024-02-05T07:36:40.752646+0000","last_active":"2024-02-05T07:36:47.830372+0000","last_peered":"2024-02-05T07:36:47.830372+0000","last_clean":"2024-02-05T07:36:47.830372+0000","last_became_active":"2024-02-05T07:36:40.752358+0000","last_became_peered":"2024-02-05T07:36:40.752358+0000","last_unstale":"2024-02-05T07:36:47.830372+0000","last_undegraded":"2024-02-05T07:36:47.830372+0000","last_fullsized":"2024-02-05T07:36:47.830372+0000","mapping_epoch":446,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:31.185403+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:34:31.185403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":602,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.773865+0000","last_change":"2024-02-05T07:32:46.399673+0000","last_active":"2024-02-05T07:36:44.773865+0000","last_peered":"2024-02-05T07:36:44.773865+0000","last_clean":"2024-02-05T07:36:44.773865+0000","last_became_active":"2024-02-05T07:27:03.370297+0000","last_became_peered":"2024-02-05T07:27:03.370297+0000","last_unstale":"2024-02-05T07:36:44.773865+0000","last_undegraded":"2024-02-05T07:36:44.773865+0000","last_fullsized":"2024-02-05T07:36:44.773865+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:46.399572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:46.399572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"326'24","reported_seq":43,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.765579+0000","last_change":"2024-02-05T07:36:32.847697+0000","last_active":"2024-02-05T07:36:44.765579+0000","last_peered":"2024-02-05T07:36:44.765579+0000","last_clean":"2024-02-05T07:36:44.765579+0000","last_became_active":"2024-02-05T07:36:32.753752+0000","last_became_peered":"2024-02-05T07:36:32.753752+0000","last_unstale":"2024-02-05T07:36:44.765579+0000","last_undegraded":"2024-02-05T07:36:44.765579+0000","last_fullsized":"2024-02-05T07:36:44.765579+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:33:49.499846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:49.499846+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":791,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.775063+0000","last_change":"2024-02-05T07:35:10.736128+0000","last_active":"2024-02-05T07:36:44.775063+0000","last_peered":"2024-02-05T07:36:44.775063+0000","last_clean":"2024-02-05T07:36:44.775063+0000","last_became_active":"2024-02-05T07:30:40.493770+0000","last_became_peered":"2024-02-05T07:30:40.493770+0000","last_unstale":"2024-02-05T07:36:44.775063+0000","last_undegraded":"2024-02-05T07:36:44.775063+0000","last_fullsized":"2024-02-05T07:36:44.775063+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:35:10.736006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:35:10.736006+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":369,"num_read_kb":364,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":976,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759229+0000","last_change":"2024-02-05T07:33:39.809630+0000","last_active":"2024-02-05T07:36:44.759229+0000","last_peered":"2024-02-05T07:36:44.759229+0000","last_clean":"2024-02-05T07:36:44.759229+0000","last_became_active":"2024-02-05T07:27:16.724280+0000","last_became_peered":"2024-02-05T07:27:16.724280+0000","last_unstale":"2024-02-05T07:36:44.759229+0000","last_undegraded":"2024-02-05T07:36:44.759229+0000","last_fullsized":"2024-02-05T07:36:44.759229+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:39.809544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:33:39.809544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":1030,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.775415+0000","last_change":"2024-02-05T07:33:41.379002+0000","last_active":"2024-02-05T07:36:44.775415+0000","last_peered":"2024-02-05T07:36:44.775415+0000","last_clean":"2024-02-05T07:36:44.775415+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:36:44.775415+0000","last_undegraded":"2024-02-05T07:36:44.775415+0000","last_fullsized":"2024-02-05T07:36:44.775415+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:33:41.378909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.378909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"548'13736","reported_seq":14000,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:48.777522+0000","last_change":"2024-02-05T07:33:38.775761+0000","last_active":"2024-02-05T07:36:48.777522+0000","last_peered":"2024-02-05T07:36:48.777522+0000","last_clean":"2024-02-05T07:36:48.777522+0000","last_became_active":"2024-02-05T07:33:38.775338+0000","last_became_peered":"2024-02-05T07:33:38.775338+0000","last_unstale":"2024-02-05T07:36:48.777522+0000","last_undegraded":"2024-02-05T07:36:48.777522+0000","last_fullsized":"2024-02-05T07:36:48.777522+0000","mapping_epoch":423,"log_start":"514'9951","ondisk_log_start":"514'9951","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3785,"log_dups_size":174,"ondisk_log_size":3785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152436736,"num_objects":2326,"num_object_clones":0,"num_object_copies":6978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13736,"num_write_kb":513984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.765950+0000","last_change":"2024-02-05T07:36:32.733109+0000","last_active":"2024-02-05T07:36:44.765950+0000","last_peered":"2024-02-05T07:36:44.765950+0000","last_clean":"2024-02-05T07:36:44.765950+0000","last_became_active":"2024-02-05T07:36:32.732334+0000","last_became_peered":"2024-02-05T07:36:32.732334+0000","last_unstale":"2024-02-05T07:36:44.765950+0000","last_undegraded":"2024-02-05T07:36:44.765950+0000","last_fullsized":"2024-02-05T07:36:44.765950+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:50.943640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:50.943640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"365'31","reported_seq":1114,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830396+0000","last_change":"2024-02-05T07:33:46.200250+0000","last_active":"2024-02-05T07:36:47.830396+0000","last_peered":"2024-02-05T07:36:47.830396+0000","last_clean":"2024-02-05T07:36:47.830396+0000","last_became_active":"2024-02-05T07:30:40.508686+0000","last_became_peered":"2024-02-05T07:30:40.508686+0000","last_unstale":"2024-02-05T07:36:47.830396+0000","last_undegraded":"2024-02-05T07:36:47.830396+0000","last_fullsized":"2024-02-05T07:36:47.830396+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'31","last_scrub_stamp":"2024-02-05T07:33:46.200150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.200150+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":600,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.819868+0000","last_change":"2024-02-05T07:35:12.093470+0000","last_active":"2024-02-05T07:36:46.819868+0000","last_peered":"2024-02-05T07:36:46.819868+0000","last_clean":"2024-02-05T07:36:46.819868+0000","last_became_active":"2024-02-05T07:28:26.595640+0000","last_became_peered":"2024-02-05T07:28:26.595640+0000","last_unstale":"2024-02-05T07:36:46.819868+0000","last_undegraded":"2024-02-05T07:36:46.819868+0000","last_fullsized":"2024-02-05T07:36:46.819868+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:12.093350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:35:12.093350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":756,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.829897+0000","last_change":"2024-02-05T07:35:22.815134+0000","last_active":"2024-02-05T07:36:47.829897+0000","last_peered":"2024-02-05T07:36:47.829897+0000","last_clean":"2024-02-05T07:36:47.829897+0000","last_became_active":"2024-02-05T07:35:22.814884+0000","last_became_peered":"2024-02-05T07:35:22.814884+0000","last_unstale":"2024-02-05T07:36:47.829897+0000","last_undegraded":"2024-02-05T07:36:47.829897+0000","last_fullsized":"2024-02-05T07:36:47.829897+0000","mapping_epoch":367,"log_start":"86'128","ondisk_log_start":"86'128","created":20,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:33:48.221845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":589,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821889+0000","last_change":"2024-02-05T07:33:46.760555+0000","last_active":"2024-02-05T07:36:46.821889+0000","last_peered":"2024-02-05T07:36:46.821889+0000","last_clean":"2024-02-05T07:36:46.821889+0000","last_became_active":"2024-02-05T07:27:16.751341+0000","last_became_peered":"2024-02-05T07:27:16.751341+0000","last_unstale":"2024-02-05T07:36:46.821889+0000","last_undegraded":"2024-02-05T07:36:46.821889+0000","last_fullsized":"2024-02-05T07:36:46.821889+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:46.760448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.760448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"545'10830","reported_seq":11151,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:45.369547+0000","last_change":"2024-02-05T07:36:33.750052+0000","last_active":"2024-02-05T07:36:45.369547+0000","last_peered":"2024-02-05T07:36:45.369547+0000","last_clean":"2024-02-05T07:36:45.369547+0000","last_became_active":"2024-02-05T07:36:33.702298+0000","last_became_peered":"2024-02-05T07:36:33.702298+0000","last_unstale":"2024-02-05T07:36:45.369547+0000","last_undegraded":"2024-02-05T07:36:45.369547+0000","last_fullsized":"2024-02-05T07:36:45.369547+0000","mapping_epoch":536,"log_start":"483'7303","ondisk_log_start":"483'7303","created":423,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'8102","last_scrub_stamp":"2024-02-05T07:35:21.544679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:21.544679+0000","objects_scrubbed":3995,"log_size":3527,"log_dups_size":0,"ondisk_log_size":3527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":420,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83034112,"num_objects":1267,"num_object_clones":0,"num_object_copies":3801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2728,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":1086,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759046+0000","last_change":"2024-02-05T07:36:35.629332+0000","last_active":"2024-02-05T07:36:44.759046+0000","last_peered":"2024-02-05T07:36:44.759046+0000","last_clean":"2024-02-05T07:36:44.759046+0000","last_became_active":"2024-02-05T07:36:35.628967+0000","last_became_peered":"2024-02-05T07:36:35.628967+0000","last_unstale":"2024-02-05T07:36:44.759046+0000","last_undegraded":"2024-02-05T07:36:44.759046+0000","last_fullsized":"2024-02-05T07:36:44.759046+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:35:09.073175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:35:09.073175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.765491+0000","last_change":"2024-02-05T07:36:32.754330+0000","last_active":"2024-02-05T07:36:44.765491+0000","last_peered":"2024-02-05T07:36:44.765491+0000","last_clean":"2024-02-05T07:36:44.765491+0000","last_became_active":"2024-02-05T07:36:32.753264+0000","last_became_peered":"2024-02-05T07:36:32.753264+0000","last_unstale":"2024-02-05T07:36:44.765491+0000","last_undegraded":"2024-02-05T07:36:44.765491+0000","last_fullsized":"2024-02-05T07:36:44.765491+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:23.118569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:23.118569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"326'22","reported_seq":1075,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.773643+0000","last_change":"2024-02-05T07:33:30.327114+0000","last_active":"2024-02-05T07:36:44.773643+0000","last_peered":"2024-02-05T07:36:44.773643+0000","last_clean":"2024-02-05T07:36:44.773643+0000","last_became_active":"2024-02-05T07:30:40.494460+0000","last_became_peered":"2024-02-05T07:30:40.494460+0000","last_unstale":"2024-02-05T07:36:44.773643+0000","last_undegraded":"2024-02-05T07:36:44.773643+0000","last_fullsized":"2024-02-05T07:36:44.773643+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'22","last_scrub_stamp":"2024-02-05T07:33:30.327027+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.327027+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.765436+0000","last_change":"2024-02-05T07:36:32.734528+0000","last_active":"2024-02-05T07:36:44.765436+0000","last_peered":"2024-02-05T07:36:44.765436+0000","last_clean":"2024-02-05T07:36:44.765436+0000","last_became_active":"2024-02-05T07:36:32.733822+0000","last_became_peered":"2024-02-05T07:36:32.733822+0000","last_unstale":"2024-02-05T07:36:44.765436+0000","last_undegraded":"2024-02-05T07:36:44.765436+0000","last_fullsized":"2024-02-05T07:36:44.765436+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:42.095953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:33:42.095953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":33,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.766055+0000","last_change":"2024-02-05T07:36:32.849750+0000","last_active":"2024-02-05T07:36:44.766055+0000","last_peered":"2024-02-05T07:36:44.766055+0000","last_clean":"2024-02-05T07:36:44.766055+0000","last_became_active":"2024-02-05T07:36:32.781195+0000","last_became_peered":"2024-02-05T07:36:32.781195+0000","last_unstale":"2024-02-05T07:36:44.766055+0000","last_undegraded":"2024-02-05T07:36:44.766055+0000","last_fullsized":"2024-02-05T07:36:44.766055+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:34:26.591385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:26.591385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":906,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830562+0000","last_change":"2024-02-05T07:36:32.719020+0000","last_active":"2024-02-05T07:36:47.830562+0000","last_peered":"2024-02-05T07:36:47.830562+0000","last_clean":"2024-02-05T07:36:47.830562+0000","last_became_active":"2024-02-05T07:36:32.718684+0000","last_became_peered":"2024-02-05T07:36:32.718684+0000","last_unstale":"2024-02-05T07:36:47.830562+0000","last_undegraded":"2024-02-05T07:36:47.830562+0000","last_fullsized":"2024-02-05T07:36:47.830562+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:18.035057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:35:18.035057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"548'13838","reported_seq":14114,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:48.756229+0000","last_change":"2024-02-05T07:36:32.732619+0000","last_active":"2024-02-05T07:36:48.756229+0000","last_peered":"2024-02-05T07:36:48.756229+0000","last_clean":"2024-02-05T07:36:48.756229+0000","last_became_active":"2024-02-05T07:36:32.709197+0000","last_became_peered":"2024-02-05T07:36:32.709197+0000","last_unstale":"2024-02-05T07:36:48.756229+0000","last_undegraded":"2024-02-05T07:36:48.756229+0000","last_fullsized":"2024-02-05T07:36:48.756229+0000","mapping_epoch":535,"log_start":"514'10045","ondisk_log_start":"514'10045","created":423,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3793,"log_dups_size":203,"ondisk_log_size":3793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150732800,"num_objects":2300,"num_object_clones":0,"num_object_copies":6900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13838,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":345,"reported_epoch":546,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:36:45.798423+0000","last_change":"2024-02-05T07:36:33.751959+0000","last_active":"2024-02-05T07:36:45.798423+0000","last_peered":"2024-02-05T07:36:45.798423+0000","last_clean":"2024-02-05T07:19:30.999323+0000","last_became_active":"2024-02-05T07:36:33.699620+0000","last_became_peered":"2024-02-05T07:36:33.699620+0000","last_unstale":"2024-02-05T07:36:45.798423+0000","last_undegraded":"2024-02-05T07:36:45.798423+0000","last_fullsized":"2024-02-05T07:36:45.798423+0000","mapping_epoch":536,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:41.082046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.082046+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[0,4,6],"avail_no_missing":["0","4","6"],"object_location_counts":[{"shards":"0,4,6","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":757,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.773599+0000","last_change":"2024-02-05T07:34:37.540422+0000","last_active":"2024-02-05T07:36:44.773599+0000","last_peered":"2024-02-05T07:36:44.773599+0000","last_clean":"2024-02-05T07:36:44.773599+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:36:44.773599+0000","last_undegraded":"2024-02-05T07:36:44.773599+0000","last_fullsized":"2024-02-05T07:36:44.773599+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:37.540304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:34:37.540304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"538'46","reported_seq":54,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.765120+0000","last_change":"2024-02-05T07:36:32.852166+0000","last_active":"2024-02-05T07:36:44.765120+0000","last_peered":"2024-02-05T07:36:44.765120+0000","last_clean":"2024-02-05T07:36:44.765120+0000","last_became_active":"2024-02-05T07:36:32.738270+0000","last_became_peered":"2024-02-05T07:36:32.738270+0000","last_unstale":"2024-02-05T07:36:44.765120+0000","last_undegraded":"2024-02-05T07:36:44.765120+0000","last_fullsized":"2024-02-05T07:36:44.765120+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'44","last_scrub_stamp":"2024-02-05T07:30:36.008885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:36.008885+0000","objects_scrubbed":10,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":53,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759054+0000","last_change":"2024-02-05T07:36:16.722155+0000","last_active":"2024-02-05T07:36:44.759054+0000","last_peered":"2024-02-05T07:36:44.759054+0000","last_clean":"2024-02-05T07:36:44.759054+0000","last_became_active":"2024-02-05T07:36:16.721809+0000","last_became_peered":"2024-02-05T07:36:16.721809+0000","last_unstale":"2024-02-05T07:36:44.759054+0000","last_undegraded":"2024-02-05T07:36:44.759054+0000","last_fullsized":"2024-02-05T07:36:44.759054+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:32.964622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:32.964622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":927,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.773486+0000","last_change":"2024-02-05T07:33:43.473208+0000","last_active":"2024-02-05T07:36:44.773486+0000","last_peered":"2024-02-05T07:36:44.773486+0000","last_clean":"2024-02-05T07:36:44.773486+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:36:44.773486+0000","last_undegraded":"2024-02-05T07:36:44.773486+0000","last_fullsized":"2024-02-05T07:36:44.773486+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:33:43.473107+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:33:43.473107+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":603,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821230+0000","last_change":"2024-02-05T07:34:26.897153+0000","last_active":"2024-02-05T07:36:46.821230+0000","last_peered":"2024-02-05T07:36:46.821230+0000","last_clean":"2024-02-05T07:36:46.821230+0000","last_became_active":"2024-02-05T07:30:40.508071+0000","last_became_peered":"2024-02-05T07:30:40.508071+0000","last_unstale":"2024-02-05T07:36:46.821230+0000","last_undegraded":"2024-02-05T07:36:46.821230+0000","last_fullsized":"2024-02-05T07:36:46.821230+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:26.897028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:26.897028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.5","version":"545'10629","reported_seq":14923,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:45.395095+0000","last_change":"2024-02-05T07:36:32.868666+0000","last_active":"2024-02-05T07:36:45.395095+0000","last_peered":"2024-02-05T07:36:45.395095+0000","last_clean":"2024-02-05T07:36:45.395095+0000","last_became_active":"2024-02-05T07:36:32.767092+0000","last_became_peered":"2024-02-05T07:36:32.767092+0000","last_unstale":"2024-02-05T07:36:45.395095+0000","last_undegraded":"2024-02-05T07:36:45.395095+0000","last_fullsized":"2024-02-05T07:36:45.395095+0000","mapping_epoch":535,"log_start":"470'6266","ondisk_log_start":"470'6266","created":423,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":4363,"log_dups_size":0,"ondisk_log_size":4363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:48.179056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85622784,"num_objects":1306,"num_object_clones":0,"num_object_copies":3918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6650,"num_write_kb":254624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2008,"num_bytes_recovered":131530752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"326'15","reported_seq":800,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.776875+0000","last_change":"2024-02-05T07:35:12.777278+0000","last_active":"2024-02-05T07:36:44.776875+0000","last_peered":"2024-02-05T07:36:44.776875+0000","last_clean":"2024-02-05T07:36:44.776875+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:36:44.776875+0000","last_undegraded":"2024-02-05T07:36:44.776875+0000","last_fullsized":"2024-02-05T07:36:44.776875+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:35:12.776983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:12.776983+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":592,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.820764+0000","last_change":"2024-02-05T07:30:38.998491+0000","last_active":"2024-02-05T07:36:46.820764+0000","last_peered":"2024-02-05T07:36:46.820764+0000","last_clean":"2024-02-05T07:36:46.820764+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:36:46.820764+0000","last_undegraded":"2024-02-05T07:36:46.820764+0000","last_fullsized":"2024-02-05T07:36:46.820764+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:38.995586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:38.995586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.770737+0000","last_change":"2024-02-05T07:36:32.723290+0000","last_active":"2024-02-05T07:36:44.770737+0000","last_peered":"2024-02-05T07:36:44.770737+0000","last_clean":"2024-02-05T07:36:44.770737+0000","last_became_active":"2024-02-05T07:36:32.722610+0000","last_became_peered":"2024-02-05T07:36:32.722610+0000","last_unstale":"2024-02-05T07:36:44.770737+0000","last_undegraded":"2024-02-05T07:36:44.770737+0000","last_fullsized":"2024-02-05T07:36:44.770737+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:47.161343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:47.161343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.764159+0000","last_change":"2024-02-05T07:36:32.734750+0000","last_active":"2024-02-05T07:36:44.764159+0000","last_peered":"2024-02-05T07:36:44.764159+0000","last_clean":"2024-02-05T07:36:44.764159+0000","last_became_active":"2024-02-05T07:36:32.734230+0000","last_became_peered":"2024-02-05T07:36:32.734230+0000","last_unstale":"2024-02-05T07:36:44.764159+0000","last_undegraded":"2024-02-05T07:36:44.764159+0000","last_fullsized":"2024-02-05T07:36:44.764159+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:46.198155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:32:46.198155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"545'10905","reported_seq":11216,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:45.392700+0000","last_change":"2024-02-05T07:35:20.550097+0000","last_active":"2024-02-05T07:36:45.392700+0000","last_peered":"2024-02-05T07:36:45.392700+0000","last_clean":"2024-02-05T07:36:45.392700+0000","last_became_active":"2024-02-05T07:35:18.800109+0000","last_became_peered":"2024-02-05T07:35:18.800109+0000","last_unstale":"2024-02-05T07:36:45.392700+0000","last_undegraded":"2024-02-05T07:36:45.392700+0000","last_fullsized":"2024-02-05T07:36:45.392700+0000","mapping_epoch":423,"log_start":"484'7404","ondisk_log_start":"484'7404","created":423,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'8142","last_scrub_stamp":"2024-02-05T07:35:20.549958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:20.549958+0000","objects_scrubbed":3994,"log_size":3501,"log_dups_size":0,"ondisk_log_size":3501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80674816,"num_objects":1231,"num_object_clones":0,"num_object_copies":3693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2763,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"326'35","reported_seq":958,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830653+0000","last_change":"2024-02-05T07:36:16.812273+0000","last_active":"2024-02-05T07:36:47.830653+0000","last_peered":"2024-02-05T07:36:47.830653+0000","last_clean":"2024-02-05T07:36:47.830653+0000","last_became_active":"2024-02-05T07:36:16.736226+0000","last_became_peered":"2024-02-05T07:36:16.736226+0000","last_unstale":"2024-02-05T07:36:47.830653+0000","last_undegraded":"2024-02-05T07:36:47.830653+0000","last_fullsized":"2024-02-05T07:36:47.830653+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:35:46.529276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:46.529276+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":905,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830497+0000","last_change":"2024-02-05T07:35:15.914525+0000","last_active":"2024-02-05T07:36:47.830497+0000","last_peered":"2024-02-05T07:36:47.830497+0000","last_clean":"2024-02-05T07:36:47.830497+0000","last_became_active":"2024-02-05T07:30:40.507705+0000","last_became_peered":"2024-02-05T07:30:40.507705+0000","last_unstale":"2024-02-05T07:36:47.830497+0000","last_undegraded":"2024-02-05T07:36:47.830497+0000","last_fullsized":"2024-02-05T07:36:47.830497+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:15.914445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_clean_scrub_stamp":"2024-02-05T07:35:15.914445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.764942+0000","last_change":"2024-02-05T07:36:32.733809+0000","last_active":"2024-02-05T07:36:44.764942+0000","last_peered":"2024-02-05T07:36:44.764942+0000","last_clean":"2024-02-05T07:36:44.764942+0000","last_became_active":"2024-02-05T07:36:32.733494+0000","last_became_peered":"2024-02-05T07:36:32.733494+0000","last_unstale":"2024-02-05T07:36:44.764942+0000","last_undegraded":"2024-02-05T07:36:44.764942+0000","last_fullsized":"2024-02-05T07:36:44.764942+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.121427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.121427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.766435+0000","last_change":"2024-02-05T07:36:32.716609+0000","last_active":"2024-02-05T07:36:44.766435+0000","last_peered":"2024-02-05T07:36:44.766435+0000","last_clean":"2024-02-05T07:36:44.766435+0000","last_became_active":"2024-02-05T07:36:32.715956+0000","last_became_peered":"2024-02-05T07:36:32.715956+0000","last_unstale":"2024-02-05T07:36:44.766435+0000","last_undegraded":"2024-02-05T07:36:44.766435+0000","last_fullsized":"2024-02-05T07:36:44.766435+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:31.032967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_clean_scrub_stamp":"2024-02-05T07:33:31.032967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"548'10855","reported_seq":11186,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:48.535958+0000","last_change":"2024-02-05T07:36:32.761603+0000","last_active":"2024-02-05T07:36:48.535958+0000","last_peered":"2024-02-05T07:36:48.535958+0000","last_clean":"2024-02-05T07:36:48.535958+0000","last_became_active":"2024-02-05T07:36:32.750071+0000","last_became_peered":"2024-02-05T07:36:32.750071+0000","last_unstale":"2024-02-05T07:36:48.535958+0000","last_undegraded":"2024-02-05T07:36:48.535958+0000","last_fullsized":"2024-02-05T07:36:48.535958+0000","mapping_epoch":535,"log_start":"475'6649","ondisk_log_start":"475'6649","created":423,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8069","last_scrub_stamp":"2024-02-05T07:35:20.035412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:20.035412+0000","objects_scrubbed":3980,"log_size":4206,"log_dups_size":0,"ondisk_log_size":4206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76611584,"num_objects":1169,"num_object_clones":0,"num_object_copies":3507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2811,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.766251+0000","last_change":"2024-02-05T07:36:32.737113+0000","last_active":"2024-02-05T07:36:44.766251+0000","last_peered":"2024-02-05T07:36:44.766251+0000","last_clean":"2024-02-05T07:36:44.766251+0000","last_became_active":"2024-02-05T07:36:32.733003+0000","last_became_peered":"2024-02-05T07:36:32.733003+0000","last_unstale":"2024-02-05T07:36:44.766251+0000","last_undegraded":"2024-02-05T07:36:44.766251+0000","last_fullsized":"2024-02-05T07:36:44.766251+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:29.082803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:29.082803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"325'18","reported_seq":1118,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759316+0000","last_change":"2024-02-05T07:36:16.769406+0000","last_active":"2024-02-05T07:36:44.759316+0000","last_peered":"2024-02-05T07:36:44.759316+0000","last_clean":"2024-02-05T07:36:44.759316+0000","last_became_active":"2024-02-05T07:36:16.741080+0000","last_became_peered":"2024-02-05T07:36:16.741080+0000","last_unstale":"2024-02-05T07:36:44.759316+0000","last_undegraded":"2024-02-05T07:36:44.759316+0000","last_fullsized":"2024-02-05T07:36:44.759316+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'18","last_scrub_stamp":"2024-02-05T07:34:28.902809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.902809+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":381,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.760403+0000","last_change":"2024-02-05T07:33:33.822290+0000","last_active":"2024-02-05T07:36:44.760403+0000","last_peered":"2024-02-05T07:36:44.760403+0000","last_clean":"2024-02-05T07:36:44.760403+0000","last_became_active":"2024-02-05T07:30:40.518949+0000","last_became_peered":"2024-02-05T07:30:40.518949+0000","last_unstale":"2024-02-05T07:36:44.760403+0000","last_undegraded":"2024-02-05T07:36:44.760403+0000","last_fullsized":"2024-02-05T07:36:44.760403+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:33.822199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.822199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":603,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.819626+0000","last_change":"2024-02-05T07:34:38.057821+0000","last_active":"2024-02-05T07:36:46.819626+0000","last_peered":"2024-02-05T07:36:46.819626+0000","last_clean":"2024-02-05T07:36:46.819626+0000","last_became_active":"2024-02-05T07:30:40.485991+0000","last_became_peered":"2024-02-05T07:30:40.485991+0000","last_unstale":"2024-02-05T07:36:46.819626+0000","last_undegraded":"2024-02-05T07:36:46.819626+0000","last_fullsized":"2024-02-05T07:36:46.819626+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:38.057598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.057598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"545'10831","reported_seq":11143,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:45.319967+0000","last_change":"2024-02-05T07:35:21.478756+0000","last_active":"2024-02-05T07:36:45.319967+0000","last_peered":"2024-02-05T07:36:45.319967+0000","last_clean":"2024-02-05T07:36:45.319967+0000","last_became_active":"2024-02-05T07:35:18.802299+0000","last_became_peered":"2024-02-05T07:35:18.802299+0000","last_unstale":"2024-02-05T07:36:45.319967+0000","last_undegraded":"2024-02-05T07:36:45.319967+0000","last_fullsized":"2024-02-05T07:36:45.319967+0000","mapping_epoch":423,"log_start":"484'7382","ondisk_log_start":"484'7382","created":423,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8111","last_scrub_stamp":"2024-02-05T07:35:21.478656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:21.478656+0000","objects_scrubbed":4005,"log_size":3449,"log_dups_size":0,"ondisk_log_size":3449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":465,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84213760,"num_objects":1285,"num_object_clones":0,"num_object_copies":3855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"365'12","reported_seq":1090,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830046+0000","last_change":"2024-02-05T07:34:40.283207+0000","last_active":"2024-02-05T07:36:47.830046+0000","last_peered":"2024-02-05T07:36:47.830046+0000","last_clean":"2024-02-05T07:36:47.830046+0000","last_became_active":"2024-02-05T07:30:40.500113+0000","last_became_peered":"2024-02-05T07:30:40.500113+0000","last_unstale":"2024-02-05T07:36:47.830046+0000","last_undegraded":"2024-02-05T07:36:47.830046+0000","last_fullsized":"2024-02-05T07:36:47.830046+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'12","last_scrub_stamp":"2024-02-05T07:34:40.283087+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:34:40.283087+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"326'27","reported_seq":652,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.829772+0000","last_change":"2024-02-05T07:35:12.026975+0000","last_active":"2024-02-05T07:36:47.829772+0000","last_peered":"2024-02-05T07:36:47.829772+0000","last_clean":"2024-02-05T07:36:47.829772+0000","last_became_active":"2024-02-05T07:30:40.497982+0000","last_became_peered":"2024-02-05T07:30:40.497982+0000","last_unstale":"2024-02-05T07:36:47.829772+0000","last_undegraded":"2024-02-05T07:36:47.829772+0000","last_fullsized":"2024-02-05T07:36:47.829772+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:35:12.026888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:12.026888+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":397,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.760035+0000","last_change":"2024-02-05T07:33:27.649686+0000","last_active":"2024-02-05T07:36:44.760035+0000","last_peered":"2024-02-05T07:36:44.760035+0000","last_clean":"2024-02-05T07:36:44.760035+0000","last_became_active":"2024-02-05T07:30:40.503903+0000","last_became_peered":"2024-02-05T07:30:40.503903+0000","last_unstale":"2024-02-05T07:36:44.760035+0000","last_undegraded":"2024-02-05T07:36:44.760035+0000","last_fullsized":"2024-02-05T07:36:44.760035+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:33:27.649596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:27.649596+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":691,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830329+0000","last_change":"2024-02-05T07:36:32.701872+0000","last_active":"2024-02-05T07:36:47.830329+0000","last_peered":"2024-02-05T07:36:47.830329+0000","last_clean":"2024-02-05T07:36:47.830329+0000","last_became_active":"2024-02-05T07:36:32.701535+0000","last_became_peered":"2024-02-05T07:36:32.701535+0000","last_unstale":"2024-02-05T07:36:47.830329+0000","last_undegraded":"2024-02-05T07:36:47.830329+0000","last_fullsized":"2024-02-05T07:36:47.830329+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:35.177804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:33:35.177804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"545'10775","reported_seq":17647,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:45.012338+0000","last_change":"2024-02-05T07:36:33.148823+0000","last_active":"2024-02-05T07:36:45.012338+0000","last_peered":"2024-02-05T07:36:45.012338+0000","last_clean":"2024-02-05T07:36:45.012338+0000","last_became_active":"2024-02-05T07:35:59.586194+0000","last_became_peered":"2024-02-05T07:35:59.586194+0000","last_unstale":"2024-02-05T07:36:45.012338+0000","last_undegraded":"2024-02-05T07:36:45.012338+0000","last_fullsized":"2024-02-05T07:36:45.012338+0000","mapping_epoch":517,"log_start":"481'7105","ondisk_log_start":"481'7105","created":493,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":5,"last_scrub":"536'10378","last_scrub_stamp":"2024-02-05T07:36:33.148656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:33.148656+0000","objects_scrubbed":1696,"log_size":3670,"log_dups_size":0,"ondisk_log_size":3670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:41.044865+0000","scrub_duration":317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85131264,"num_objects":1299,"num_object_clones":0,"num_object_copies":3897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":397,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"326'12","reported_seq":647,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821792+0000","last_change":"2024-02-05T07:36:32.779445+0000","last_active":"2024-02-05T07:36:46.821792+0000","last_peered":"2024-02-05T07:36:46.821792+0000","last_clean":"2024-02-05T07:36:46.821792+0000","last_became_active":"2024-02-05T07:36:32.742483+0000","last_became_peered":"2024-02-05T07:36:32.742483+0000","last_unstale":"2024-02-05T07:36:46.821792+0000","last_undegraded":"2024-02-05T07:36:46.821792+0000","last_fullsized":"2024-02-05T07:36:46.821792+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'12","last_scrub_stamp":"2024-02-05T07:33:41.762404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.762404+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":823,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.829969+0000","last_change":"2024-02-05T07:33:51.105371+0000","last_active":"2024-02-05T07:36:47.829969+0000","last_peered":"2024-02-05T07:36:47.829969+0000","last_clean":"2024-02-05T07:36:47.829969+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:36:47.829969+0000","last_undegraded":"2024-02-05T07:36:47.829969+0000","last_fullsized":"2024-02-05T07:36:47.829969+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:33:51.105251+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:33:51.105251+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":591,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.820905+0000","last_change":"2024-02-05T07:30:27.965535+0000","last_active":"2024-02-05T07:36:46.820905+0000","last_peered":"2024-02-05T07:36:46.820905+0000","last_clean":"2024-02-05T07:36:46.820905+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:36:46.820905+0000","last_undegraded":"2024-02-05T07:36:46.820905+0000","last_fullsized":"2024-02-05T07:36:46.820905+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:27.965449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:27.965449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":881,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830673+0000","last_change":"2024-02-05T07:35:13.118423+0000","last_active":"2024-02-05T07:36:47.830673+0000","last_peered":"2024-02-05T07:36:47.830673+0000","last_clean":"2024-02-05T07:36:47.830673+0000","last_became_active":"2024-02-05T07:30:40.509216+0000","last_became_peered":"2024-02-05T07:30:40.509216+0000","last_unstale":"2024-02-05T07:36:47.830673+0000","last_undegraded":"2024-02-05T07:36:47.830673+0000","last_fullsized":"2024-02-05T07:36:47.830673+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:13.118311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:35:13.118311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.18","version":"545'10775","reported_seq":15671,"reported_epoch":545,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T07:36:45.450325+0000","last_change":"2024-02-05T07:36:32.852959+0000","last_active":"2024-02-05T07:36:45.450325+0000","last_peered":"2024-02-05T07:36:45.450325+0000","last_clean":"2024-02-05T07:35:19.830442+0000","last_became_active":"2024-02-05T07:36:32.731751+0000","last_became_peered":"2024-02-05T07:36:32.731751+0000","last_unstale":"2024-02-05T07:36:45.450325+0000","last_undegraded":"2024-02-05T07:36:45.450325+0000","last_fullsized":"2024-02-05T07:36:45.450325+0000","mapping_epoch":535,"log_start":"461'5708","ondisk_log_start":"461'5708","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":5067,"log_dups_size":0,"ondisk_log_size":5067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85393408,"num_objects":1303,"num_object_clones":0,"num_object_copies":3909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":170,"num_objects_unfound":0,"num_objects_dirty":1303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6749,"num_write_kb":257664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1537,"num_bytes_recovered":100728832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,0,4],"avail_no_missing":["2","0","4"],"object_location_counts":[{"shards":"0,2,4","objects":1303}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"325'15","reported_seq":655,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821103+0000","last_change":"2024-02-05T07:36:32.807878+0000","last_active":"2024-02-05T07:36:46.821103+0000","last_peered":"2024-02-05T07:36:46.821103+0000","last_clean":"2024-02-05T07:36:46.821103+0000","last_became_active":"2024-02-05T07:36:32.747221+0000","last_became_peered":"2024-02-05T07:36:32.747221+0000","last_unstale":"2024-02-05T07:36:46.821103+0000","last_undegraded":"2024-02-05T07:36:46.821103+0000","last_fullsized":"2024-02-05T07:36:46.821103+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:33:45.711399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:45.711399+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":880,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759291+0000","last_change":"2024-02-05T07:30:40.498263+0000","last_active":"2024-02-05T07:36:44.759291+0000","last_peered":"2024-02-05T07:36:44.759291+0000","last_clean":"2024-02-05T07:36:44.759291+0000","last_became_active":"2024-02-05T07:30:40.497956+0000","last_became_peered":"2024-02-05T07:30:40.497956+0000","last_unstale":"2024-02-05T07:36:44.759291+0000","last_undegraded":"2024-02-05T07:36:44.759291+0000","last_fullsized":"2024-02-05T07:36:44.759291+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:33.183427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:30:33.183427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":594,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821858+0000","last_change":"2024-02-05T07:33:50.723381+0000","last_active":"2024-02-05T07:36:46.821858+0000","last_peered":"2024-02-05T07:36:46.821858+0000","last_clean":"2024-02-05T07:36:46.821858+0000","last_became_active":"2024-02-05T07:30:40.496076+0000","last_became_peered":"2024-02-05T07:30:40.496076+0000","last_unstale":"2024-02-05T07:36:46.821858+0000","last_undegraded":"2024-02-05T07:36:46.821858+0000","last_fullsized":"2024-02-05T07:36:46.821858+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_clean_scrub_stamp":"2024-02-05T07:33:50.723212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.766072+0000","last_change":"2024-02-05T07:36:32.739391+0000","last_active":"2024-02-05T07:36:44.766072+0000","last_peered":"2024-02-05T07:36:44.766072+0000","last_clean":"2024-02-05T07:36:44.766072+0000","last_became_active":"2024-02-05T07:36:32.739115+0000","last_became_peered":"2024-02-05T07:36:32.739115+0000","last_unstale":"2024-02-05T07:36:44.766072+0000","last_undegraded":"2024-02-05T07:36:44.766072+0000","last_fullsized":"2024-02-05T07:36:44.766072+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:48.164259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:32:48.164259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"326'19","reported_seq":943,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821242+0000","last_change":"2024-02-05T07:30:40.503823+0000","last_active":"2024-02-05T07:36:46.821242+0000","last_peered":"2024-02-05T07:36:46.821242+0000","last_clean":"2024-02-05T07:36:46.821242+0000","last_became_active":"2024-02-05T07:30:40.483700+0000","last_became_peered":"2024-02-05T07:30:40.483700+0000","last_unstale":"2024-02-05T07:36:46.821242+0000","last_undegraded":"2024-02-05T07:36:46.821242+0000","last_fullsized":"2024-02-05T07:36:46.821242+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:30:35.138361+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:30:35.138361+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":765,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.819581+0000","last_change":"2024-02-05T07:36:46.059245+0000","last_active":"2024-02-05T07:36:46.819581+0000","last_peered":"2024-02-05T07:36:46.819581+0000","last_clean":"2024-02-05T07:36:46.819581+0000","last_became_active":"2024-02-05T07:30:40.476058+0000","last_became_peered":"2024-02-05T07:30:40.476058+0000","last_unstale":"2024-02-05T07:36:46.819581+0000","last_undegraded":"2024-02-05T07:36:46.819581+0000","last_fullsized":"2024-02-05T07:36:46.819581+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:46.059129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.059129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:48.902775+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"538'13","reported_seq":737,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.820565+0000","last_change":"2024-02-05T07:36:32.782551+0000","last_active":"2024-02-05T07:36:46.820565+0000","last_peered":"2024-02-05T07:36:46.820565+0000","last_clean":"2024-02-05T07:36:46.820565+0000","last_became_active":"2024-02-05T07:36:32.746270+0000","last_became_peered":"2024-02-05T07:36:32.746270+0000","last_unstale":"2024-02-05T07:36:46.820565+0000","last_undegraded":"2024-02-05T07:36:46.820565+0000","last_fullsized":"2024-02-05T07:36:46.820565+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:33:39.734129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:39.734129+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1009,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.760342+0000","last_change":"2024-02-05T07:36:32.710166+0000","last_active":"2024-02-05T07:36:44.760342+0000","last_peered":"2024-02-05T07:36:44.760342+0000","last_clean":"2024-02-05T07:36:44.760342+0000","last_became_active":"2024-02-05T07:36:32.709654+0000","last_became_peered":"2024-02-05T07:36:32.709654+0000","last_unstale":"2024-02-05T07:36:44.760342+0000","last_undegraded":"2024-02-05T07:36:44.760342+0000","last_fullsized":"2024-02-05T07:36:44.760342+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:38.879177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:38.879177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"326'25","reported_seq":670,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.820683+0000","last_change":"2024-02-05T07:36:32.817266+0000","last_active":"2024-02-05T07:36:46.820683+0000","last_peered":"2024-02-05T07:36:46.820683+0000","last_clean":"2024-02-05T07:36:46.820683+0000","last_became_active":"2024-02-05T07:36:32.746757+0000","last_became_peered":"2024-02-05T07:36:32.746757+0000","last_unstale":"2024-02-05T07:36:46.820683+0000","last_undegraded":"2024-02-05T07:36:46.820683+0000","last_fullsized":"2024-02-05T07:36:46.820683+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:30:33.959124+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:30:33.959124+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":394,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759244+0000","last_change":"2024-02-05T07:36:32.702570+0000","last_active":"2024-02-05T07:36:44.759244+0000","last_peered":"2024-02-05T07:36:44.759244+0000","last_clean":"2024-02-05T07:36:44.759244+0000","last_became_active":"2024-02-05T07:36:32.702150+0000","last_became_peered":"2024-02-05T07:36:32.702150+0000","last_unstale":"2024-02-05T07:36:44.759244+0000","last_undegraded":"2024-02-05T07:36:44.759244+0000","last_fullsized":"2024-02-05T07:36:44.759244+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:32.854871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:32.854871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"538'11","reported_seq":36,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.765075+0000","last_change":"2024-02-05T07:36:32.883755+0000","last_active":"2024-02-05T07:36:44.765075+0000","last_peered":"2024-02-05T07:36:44.765075+0000","last_clean":"2024-02-05T07:36:44.765075+0000","last_became_active":"2024-02-05T07:36:32.745294+0000","last_became_peered":"2024-02-05T07:36:32.745294+0000","last_unstale":"2024-02-05T07:36:44.765075+0000","last_undegraded":"2024-02-05T07:36:44.765075+0000","last_fullsized":"2024-02-05T07:36:44.765075+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9","last_scrub_stamp":"2024-02-05T07:34:27.946504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:34:27.946504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":753,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821084+0000","last_change":"2024-02-05T07:34:22.810896+0000","last_active":"2024-02-05T07:36:46.821084+0000","last_peered":"2024-02-05T07:36:46.821084+0000","last_clean":"2024-02-05T07:36:46.821084+0000","last_became_active":"2024-02-05T07:30:40.507831+0000","last_became_peered":"2024-02-05T07:30:40.507831+0000","last_unstale":"2024-02-05T07:36:46.821084+0000","last_undegraded":"2024-02-05T07:36:46.821084+0000","last_fullsized":"2024-02-05T07:36:46.821084+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:22.810759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:22.810759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"326'19","reported_seq":1075,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759302+0000","last_change":"2024-02-05T07:36:16.751485+0000","last_active":"2024-02-05T07:36:44.759302+0000","last_peered":"2024-02-05T07:36:44.759302+0000","last_clean":"2024-02-05T07:36:44.759302+0000","last_became_active":"2024-02-05T07:36:16.701055+0000","last_became_peered":"2024-02-05T07:36:16.701055+0000","last_unstale":"2024-02-05T07:36:44.759302+0000","last_undegraded":"2024-02-05T07:36:44.759302+0000","last_fullsized":"2024-02-05T07:36:44.759302+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:32:31.667882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:31.667882+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1030,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759809+0000","last_change":"2024-02-05T07:36:32.702949+0000","last_active":"2024-02-05T07:36:44.759809+0000","last_peered":"2024-02-05T07:36:44.759809+0000","last_clean":"2024-02-05T07:36:44.759809+0000","last_became_active":"2024-02-05T07:36:32.702585+0000","last_became_peered":"2024-02-05T07:36:32.702585+0000","last_unstale":"2024-02-05T07:36:44.759809+0000","last_undegraded":"2024-02-05T07:36:44.759809+0000","last_fullsized":"2024-02-05T07:36:44.759809+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:31.892895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:34:31.892895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1018,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.774216+0000","last_change":"2024-02-05T07:32:48.346195+0000","last_active":"2024-02-05T07:36:44.774216+0000","last_peered":"2024-02-05T07:36:44.774216+0000","last_clean":"2024-02-05T07:36:44.774216+0000","last_became_active":"2024-02-05T07:30:40.487141+0000","last_became_peered":"2024-02-05T07:30:40.487141+0000","last_unstale":"2024-02-05T07:36:44.774216+0000","last_undegraded":"2024-02-05T07:36:44.774216+0000","last_fullsized":"2024-02-05T07:36:44.774216+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_clean_scrub_stamp":"2024-02-05T07:32:48.345965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":599,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.819829+0000","last_change":"2024-02-05T07:36:32.742746+0000","last_active":"2024-02-05T07:36:46.819829+0000","last_peered":"2024-02-05T07:36:46.819829+0000","last_clean":"2024-02-05T07:36:46.819829+0000","last_became_active":"2024-02-05T07:36:32.742150+0000","last_became_peered":"2024-02-05T07:36:32.742150+0000","last_unstale":"2024-02-05T07:36:46.819829+0000","last_undegraded":"2024-02-05T07:36:46.819829+0000","last_fullsized":"2024-02-05T07:36:46.819829+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.735757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.735757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"325'15","reported_seq":664,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.772665+0000","last_change":"2024-02-05T07:36:32.815601+0000","last_active":"2024-02-05T07:36:44.772665+0000","last_peered":"2024-02-05T07:36:44.772665+0000","last_clean":"2024-02-05T07:36:44.772665+0000","last_became_active":"2024-02-05T07:36:32.710090+0000","last_became_peered":"2024-02-05T07:36:32.710090+0000","last_unstale":"2024-02-05T07:36:44.772665+0000","last_undegraded":"2024-02-05T07:36:44.772665+0000","last_fullsized":"2024-02-05T07:36:44.772665+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:35:11.749389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:11.749389+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":810,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821247+0000","last_change":"2024-02-05T07:36:32.120486+0000","last_active":"2024-02-05T07:36:46.821247+0000","last_peered":"2024-02-05T07:36:46.821247+0000","last_clean":"2024-02-05T07:36:46.821247+0000","last_became_active":"2024-02-05T07:30:40.488789+0000","last_became_peered":"2024-02-05T07:30:40.488789+0000","last_unstale":"2024-02-05T07:36:46.821247+0000","last_undegraded":"2024-02-05T07:36:46.821247+0000","last_fullsized":"2024-02-05T07:36:46.821247+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:32.120410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:32.120410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:44.650252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":589,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.820378+0000","last_change":"2024-02-05T07:33:47.684589+0000","last_active":"2024-02-05T07:36:46.820378+0000","last_peered":"2024-02-05T07:36:46.820378+0000","last_clean":"2024-02-05T07:36:46.820378+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:36:46.820378+0000","last_undegraded":"2024-02-05T07:36:46.820378+0000","last_fullsized":"2024-02-05T07:36:46.820378+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:47.684508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.684508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":476,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.829641+0000","last_change":"2024-02-05T07:36:43.520812+0000","last_active":"2024-02-05T07:36:47.829641+0000","last_peered":"2024-02-05T07:36:47.829641+0000","last_clean":"2024-02-05T07:36:47.829641+0000","last_became_active":"2024-02-05T07:34:30.670179+0000","last_became_peered":"2024-02-05T07:34:30.670179+0000","last_unstale":"2024-02-05T07:36:47.829641+0000","last_undegraded":"2024-02-05T07:36:47.829641+0000","last_fullsized":"2024-02-05T07:36:47.829641+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:43.520666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:36:43.520666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:00.289525+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"326'19","reported_seq":634,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.820163+0000","last_change":"2024-02-05T07:30:40.906805+0000","last_active":"2024-02-05T07:36:46.820163+0000","last_peered":"2024-02-05T07:36:46.820163+0000","last_clean":"2024-02-05T07:36:46.820163+0000","last_became_active":"2024-02-05T07:30:40.501893+0000","last_became_peered":"2024-02-05T07:30:40.501893+0000","last_unstale":"2024-02-05T07:36:46.820163+0000","last_undegraded":"2024-02-05T07:36:46.820163+0000","last_fullsized":"2024-02-05T07:36:46.820163+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:30:40.906693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.906693+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":411,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.773231+0000","last_change":"2024-02-05T07:36:32.712937+0000","last_active":"2024-02-05T07:36:44.773231+0000","last_peered":"2024-02-05T07:36:44.773231+0000","last_clean":"2024-02-05T07:36:44.773231+0000","last_became_active":"2024-02-05T07:36:32.712612+0000","last_became_peered":"2024-02-05T07:36:32.712612+0000","last_unstale":"2024-02-05T07:36:44.773231+0000","last_undegraded":"2024-02-05T07:36:44.773231+0000","last_fullsized":"2024-02-05T07:36:44.773231+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.097532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.097532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":880,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.758880+0000","last_change":"2024-02-05T07:36:42.529702+0000","last_active":"2024-02-05T07:36:44.758880+0000","last_peered":"2024-02-05T07:36:44.758880+0000","last_clean":"2024-02-05T07:36:44.758880+0000","last_became_active":"2024-02-05T07:30:40.497190+0000","last_became_peered":"2024-02-05T07:30:40.497190+0000","last_unstale":"2024-02-05T07:36:44.758880+0000","last_undegraded":"2024-02-05T07:36:44.758880+0000","last_fullsized":"2024-02-05T07:36:44.758880+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:42.529597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:36:42.529597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:05.970823+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":601,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.819520+0000","last_change":"2024-02-05T07:35:15.273201+0000","last_active":"2024-02-05T07:36:46.819520+0000","last_peered":"2024-02-05T07:36:46.819520+0000","last_clean":"2024-02-05T07:36:46.819520+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:36:46.819520+0000","last_undegraded":"2024-02-05T07:36:46.819520+0000","last_fullsized":"2024-02-05T07:36:46.819520+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:15.272861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:35:15.272861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"325'8","reported_seq":613,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821257+0000","last_change":"2024-02-05T07:35:47.070886+0000","last_active":"2024-02-05T07:36:46.821257+0000","last_peered":"2024-02-05T07:36:46.821257+0000","last_clean":"2024-02-05T07:36:46.821257+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:36:46.821257+0000","last_undegraded":"2024-02-05T07:36:46.821257+0000","last_fullsized":"2024-02-05T07:36:46.821257+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:35:47.070805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:47.070805+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:47.632484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":599,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821673+0000","last_change":"2024-02-05T07:34:39.980293+0000","last_active":"2024-02-05T07:36:46.821673+0000","last_peered":"2024-02-05T07:36:46.821673+0000","last_clean":"2024-02-05T07:36:46.821673+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:36:46.821673+0000","last_undegraded":"2024-02-05T07:36:46.821673+0000","last_fullsized":"2024-02-05T07:36:46.821673+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:39.980067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:34:39.980067+0000","last_clean_scrub_stamp":"2024-02-05T07:34:39.980067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":857,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830032+0000","last_change":"2024-02-05T07:33:30.216853+0000","last_active":"2024-02-05T07:36:47.830032+0000","last_peered":"2024-02-05T07:36:47.830032+0000","last_clean":"2024-02-05T07:36:47.830032+0000","last_became_active":"2024-02-05T07:30:40.500403+0000","last_became_peered":"2024-02-05T07:30:40.500403+0000","last_unstale":"2024-02-05T07:36:47.830032+0000","last_undegraded":"2024-02-05T07:36:47.830032+0000","last_fullsized":"2024-02-05T07:36:47.830032+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:30.216745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.216745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.771036+0000","last_change":"2024-02-05T07:36:32.724135+0000","last_active":"2024-02-05T07:36:44.771036+0000","last_peered":"2024-02-05T07:36:44.771036+0000","last_clean":"2024-02-05T07:36:44.771036+0000","last_became_active":"2024-02-05T07:36:32.723725+0000","last_became_peered":"2024-02-05T07:36:32.723725+0000","last_unstale":"2024-02-05T07:36:44.771036+0000","last_undegraded":"2024-02-05T07:36:44.771036+0000","last_fullsized":"2024-02-05T07:36:44.771036+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:45.926395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:30:45.926395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.11","version":"548'11049","reported_seq":17368,"reported_epoch":548,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:36:48.731153+0000","last_change":"2024-02-05T07:36:33.789339+0000","last_active":"2024-02-05T07:36:48.731153+0000","last_peered":"2024-02-05T07:36:48.731153+0000","last_clean":"2024-02-05T07:36:31.298373+0000","last_became_active":"2024-02-05T07:36:33.754876+0000","last_became_peered":"2024-02-05T07:36:33.754876+0000","last_unstale":"2024-02-05T07:36:48.731153+0000","last_undegraded":"2024-02-05T07:36:48.731153+0000","last_fullsized":"2024-02-05T07:36:48.731153+0000","mapping_epoch":536,"log_start":"481'7192","ondisk_log_start":"481'7192","created":493,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3857,"log_dups_size":0,"ondisk_log_size":3857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73039872,"num_objects":1115,"num_object_clones":0,"num_object_copies":3345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1115,"num_objects_unfound":0,"num_objects_dirty":1115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6995,"num_write_kb":259488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2008,"num_bytes_recovered":131596288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,1],"avail_no_missing":["6","0","1","3","4"],"object_location_counts":[{"shards":"0,1,6","objects":1115}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"326'19","reported_seq":852,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830846+0000","last_change":"2024-02-05T07:35:45.529783+0000","last_active":"2024-02-05T07:36:47.830846+0000","last_peered":"2024-02-05T07:36:47.830846+0000","last_clean":"2024-02-05T07:36:47.830846+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:36:47.830846+0000","last_undegraded":"2024-02-05T07:36:47.830846+0000","last_fullsized":"2024-02-05T07:36:47.830846+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:35:45.529663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:45.529663+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:57.950039+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":596,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821051+0000","last_change":"2024-02-05T07:35:11.098562+0000","last_active":"2024-02-05T07:36:46.821051+0000","last_peered":"2024-02-05T07:36:46.821051+0000","last_clean":"2024-02-05T07:36:46.821051+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:36:46.821051+0000","last_undegraded":"2024-02-05T07:36:46.821051+0000","last_fullsized":"2024-02-05T07:36:46.821051+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:11.098430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:35:11.098430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.765403+0000","last_change":"2024-02-05T07:36:32.754122+0000","last_active":"2024-02-05T07:36:44.765403+0000","last_peered":"2024-02-05T07:36:44.765403+0000","last_clean":"2024-02-05T07:36:44.765403+0000","last_became_active":"2024-02-05T07:36:32.752879+0000","last_became_peered":"2024-02-05T07:36:32.752879+0000","last_unstale":"2024-02-05T07:36:44.765403+0000","last_undegraded":"2024-02-05T07:36:44.765403+0000","last_fullsized":"2024-02-05T07:36:44.765403+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:28.153881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:28.153881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":635,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759629+0000","last_change":"2024-02-05T07:36:34.513089+0000","last_active":"2024-02-05T07:36:44.759629+0000","last_peered":"2024-02-05T07:36:44.759629+0000","last_clean":"2024-02-05T07:36:44.759629+0000","last_became_active":"2024-02-05T07:36:32.719681+0000","last_became_peered":"2024-02-05T07:36:32.719681+0000","last_unstale":"2024-02-05T07:36:44.759629+0000","last_undegraded":"2024-02-05T07:36:44.759629+0000","last_fullsized":"2024-02-05T07:36:44.759629+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:34.512979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:36:34.512979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:44.724212+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.10","version":"545'10730","reported_seq":16222,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:45.388958+0000","last_change":"2024-02-05T07:36:33.609194+0000","last_active":"2024-02-05T07:36:45.388958+0000","last_peered":"2024-02-05T07:36:45.388958+0000","last_clean":"2024-02-05T07:36:45.388958+0000","last_became_active":"2024-02-05T07:36:12.645812+0000","last_became_peered":"2024-02-05T07:36:12.645812+0000","last_unstale":"2024-02-05T07:36:45.388958+0000","last_undegraded":"2024-02-05T07:36:45.388958+0000","last_fullsized":"2024-02-05T07:36:45.388958+0000","mapping_epoch":519,"log_start":"483'7204","ondisk_log_start":"483'7204","created":493,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"536'10309","last_scrub_stamp":"2024-02-05T07:36:33.609080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:33.609080+0000","objects_scrubbed":1656,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:37.442825+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80936960,"num_objects":1235,"num_object_clones":0,"num_object_copies":3705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":421,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"326'20","reported_seq":764,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.820789+0000","last_change":"2024-02-05T07:35:13.094823+0000","last_active":"2024-02-05T07:36:46.820789+0000","last_peered":"2024-02-05T07:36:46.820789+0000","last_clean":"2024-02-05T07:36:46.820789+0000","last_became_active":"2024-02-05T07:30:40.495954+0000","last_became_peered":"2024-02-05T07:30:40.495954+0000","last_unstale":"2024-02-05T07:36:46.820789+0000","last_undegraded":"2024-02-05T07:36:46.820789+0000","last_fullsized":"2024-02-05T07:36:46.820789+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:35:13.094694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:13.094694+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":1056,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.775129+0000","last_change":"2024-02-05T07:34:38.558839+0000","last_active":"2024-02-05T07:36:44.775129+0000","last_peered":"2024-02-05T07:36:44.775129+0000","last_clean":"2024-02-05T07:36:44.775129+0000","last_became_active":"2024-02-05T07:30:40.495135+0000","last_became_peered":"2024-02-05T07:30:40.495135+0000","last_unstale":"2024-02-05T07:36:44.775129+0000","last_undegraded":"2024-02-05T07:36:44.775129+0000","last_fullsized":"2024-02-05T07:36:44.775129+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:34:38.558593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.558593+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"538'13","reported_seq":1076,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:47.830968+0000","last_change":"2024-02-05T07:36:32.761937+0000","last_active":"2024-02-05T07:36:47.830968+0000","last_peered":"2024-02-05T07:36:47.830968+0000","last_clean":"2024-02-05T07:36:47.830968+0000","last_became_active":"2024-02-05T07:36:32.717060+0000","last_became_peered":"2024-02-05T07:36:32.717060+0000","last_unstale":"2024-02-05T07:36:47.830968+0000","last_undegraded":"2024-02-05T07:36:47.830968+0000","last_fullsized":"2024-02-05T07:36:47.830968+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:33:31.253933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:31.253933+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":33,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.765769+0000","last_change":"2024-02-05T07:36:32.766920+0000","last_active":"2024-02-05T07:36:44.765769+0000","last_peered":"2024-02-05T07:36:44.765769+0000","last_clean":"2024-02-05T07:36:44.765769+0000","last_became_active":"2024-02-05T07:36:32.722692+0000","last_became_peered":"2024-02-05T07:36:32.722692+0000","last_unstale":"2024-02-05T07:36:44.765769+0000","last_undegraded":"2024-02-05T07:36:44.765769+0000","last_fullsized":"2024-02-05T07:36:44.765769+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:35:15.112415+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:35:15.112415+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"545'10685","reported_seq":19160,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:45.418461+0000","last_change":"2024-02-05T07:36:36.720616+0000","last_active":"2024-02-05T07:36:45.418461+0000","last_peered":"2024-02-05T07:36:45.418461+0000","last_clean":"2024-02-05T07:36:45.418461+0000","last_became_active":"2024-02-05T07:36:36.623778+0000","last_became_peered":"2024-02-05T07:36:36.623778+0000","last_unstale":"2024-02-05T07:36:45.418461+0000","last_undegraded":"2024-02-05T07:36:45.418461+0000","last_fullsized":"2024-02-05T07:36:45.418461+0000","mapping_epoch":539,"log_start":"469'6157","ondisk_log_start":"469'6157","created":493,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:47.223695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87261184,"num_objects":1332,"num_object_clones":0,"num_object_copies":3996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6680,"num_write_kb":256352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4080,"num_bytes_recovered":267321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"326'23","reported_seq":622,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.820281+0000","last_change":"2024-02-05T07:34:41.935675+0000","last_active":"2024-02-05T07:36:46.820281+0000","last_peered":"2024-02-05T07:36:46.820281+0000","last_clean":"2024-02-05T07:36:46.820281+0000","last_became_active":"2024-02-05T07:27:16.747998+0000","last_became_peered":"2024-02-05T07:27:16.747998+0000","last_unstale":"2024-02-05T07:36:46.820281+0000","last_undegraded":"2024-02-05T07:36:46.820281+0000","last_fullsized":"2024-02-05T07:36:46.820281+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'23","last_scrub_stamp":"2024-02-05T07:34:41.934886+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:34:41.934886+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.771135+0000","last_change":"2024-02-05T07:36:32.749299+0000","last_active":"2024-02-05T07:36:44.771135+0000","last_peered":"2024-02-05T07:36:44.771135+0000","last_clean":"2024-02-05T07:36:44.771135+0000","last_became_active":"2024-02-05T07:36:32.748986+0000","last_became_peered":"2024-02-05T07:36:32.748986+0000","last_unstale":"2024-02-05T07:36:44.771135+0000","last_undegraded":"2024-02-05T07:36:44.771135+0000","last_fullsized":"2024-02-05T07:36:44.771135+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.874081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1022,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.760327+0000","last_change":"2024-02-05T07:36:32.702723+0000","last_active":"2024-02-05T07:36:44.760327+0000","last_peered":"2024-02-05T07:36:44.760327+0000","last_clean":"2024-02-05T07:36:44.760327+0000","last_became_active":"2024-02-05T07:36:32.702187+0000","last_became_peered":"2024-02-05T07:36:32.702187+0000","last_unstale":"2024-02-05T07:36:44.760327+0000","last_undegraded":"2024-02-05T07:36:44.760327+0000","last_fullsized":"2024-02-05T07:36:44.760327+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:26.669576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:26.669576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":600,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821766+0000","last_change":"2024-02-05T07:34:37.126181+0000","last_active":"2024-02-05T07:36:46.821766+0000","last_peered":"2024-02-05T07:36:46.821766+0000","last_clean":"2024-02-05T07:36:46.821766+0000","last_became_active":"2024-02-05T07:28:26.603067+0000","last_became_peered":"2024-02-05T07:28:26.603067+0000","last_unstale":"2024-02-05T07:36:46.821766+0000","last_undegraded":"2024-02-05T07:36:46.821766+0000","last_fullsized":"2024-02-05T07:36:46.821766+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:37.125938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:37.125938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"546'10874","reported_seq":11182,"reported_epoch":546,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:36:46.195825+0000","last_change":"2024-02-05T07:36:32.840219+0000","last_active":"2024-02-05T07:36:46.195825+0000","last_peered":"2024-02-05T07:36:46.195825+0000","last_clean":"2024-02-05T07:35:19.777163+0000","last_became_active":"2024-02-05T07:36:32.807995+0000","last_became_peered":"2024-02-05T07:36:32.807995+0000","last_unstale":"2024-02-05T07:36:46.195825+0000","last_undegraded":"2024-02-05T07:36:46.195825+0000","last_fullsized":"2024-02-05T07:36:46.195825+0000","mapping_epoch":535,"log_start":"480'7096","ondisk_log_start":"480'7096","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84017152,"num_objects":1282,"num_object_clones":0,"num_object_copies":3846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3840,"num_objects_unfound":0,"num_objects_dirty":1282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6822,"num_write_kb":259328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[0,1,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":1282}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"326'27","reported_seq":45,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.764189+0000","last_change":"2024-02-05T07:36:32.820618+0000","last_active":"2024-02-05T07:36:44.764189+0000","last_peered":"2024-02-05T07:36:44.764189+0000","last_clean":"2024-02-05T07:36:44.764189+0000","last_became_active":"2024-02-05T07:36:32.744301+0000","last_became_peered":"2024-02-05T07:36:32.744301+0000","last_unstale":"2024-02-05T07:36:44.764189+0000","last_undegraded":"2024-02-05T07:36:44.764189+0000","last_fullsized":"2024-02-05T07:36:44.764189+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:33:33.995051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.995051+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":754,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759180+0000","last_change":"2024-02-05T07:30:40.503947+0000","last_active":"2024-02-05T07:36:44.759180+0000","last_peered":"2024-02-05T07:36:44.759180+0000","last_clean":"2024-02-05T07:36:44.759180+0000","last_became_active":"2024-02-05T07:30:40.503627+0000","last_became_peered":"2024-02-05T07:30:40.503627+0000","last_unstale":"2024-02-05T07:36:44.759180+0000","last_undegraded":"2024-02-05T07:36:44.759180+0000","last_fullsized":"2024-02-05T07:36:44.759180+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:34.663987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:34.663987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":996,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.773170+0000","last_change":"2024-02-05T07:30:40.884273+0000","last_active":"2024-02-05T07:36:44.773170+0000","last_peered":"2024-02-05T07:36:44.773170+0000","last_clean":"2024-02-05T07:36:44.773170+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:36:44.773170+0000","last_undegraded":"2024-02-05T07:36:44.773170+0000","last_fullsized":"2024-02-05T07:36:44.773170+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:40.884197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:30:40.884197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1017,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.760520+0000","last_change":"2024-02-05T07:36:43.532770+0000","last_active":"2024-02-05T07:36:44.760520+0000","last_peered":"2024-02-05T07:36:44.760520+0000","last_clean":"2024-02-05T07:36:44.760520+0000","last_became_active":"2024-02-05T07:28:26.590448+0000","last_became_peered":"2024-02-05T07:28:26.590448+0000","last_unstale":"2024-02-05T07:36:44.760520+0000","last_undegraded":"2024-02-05T07:36:44.760520+0000","last_fullsized":"2024-02-05T07:36:44.760520+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:43.532678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:36:43.532678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:44.061138+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.15","version":"547'10747","reported_seq":18038,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821062+0000","last_change":"2024-02-05T07:36:32.771431+0000","last_active":"2024-02-05T07:36:46.821062+0000","last_peered":"2024-02-05T07:36:46.821062+0000","last_clean":"2024-02-05T07:36:46.821062+0000","last_became_active":"2024-02-05T07:36:32.740682+0000","last_became_peered":"2024-02-05T07:36:32.740682+0000","last_unstale":"2024-02-05T07:36:46.821062+0000","last_undegraded":"2024-02-05T07:36:46.821062+0000","last_fullsized":"2024-02-05T07:36:46.821062+0000","mapping_epoch":535,"log_start":"473'6520","ondisk_log_start":"473'6520","created":493,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":1299,"log_size":4227,"log_dups_size":0,"ondisk_log_size":4227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:49.104935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77889536,"num_objects":1189,"num_object_clones":0,"num_object_copies":3567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6769,"num_write_kb":254624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3491,"num_bytes_recovered":228786176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"325'7","reported_seq":1036,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759152+0000","last_change":"2024-02-05T07:32:40.597579+0000","last_active":"2024-02-05T07:36:44.759152+0000","last_peered":"2024-02-05T07:36:44.759152+0000","last_clean":"2024-02-05T07:36:44.759152+0000","last_became_active":"2024-02-05T07:27:16.745897+0000","last_became_peered":"2024-02-05T07:27:16.745897+0000","last_unstale":"2024-02-05T07:36:44.759152+0000","last_undegraded":"2024-02-05T07:36:44.759152+0000","last_fullsized":"2024-02-05T07:36:44.759152+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'7","last_scrub_stamp":"2024-02-05T07:32:40.597479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:40.597479+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":827,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759645+0000","last_change":"2024-02-05T07:30:39.726603+0000","last_active":"2024-02-05T07:36:44.759645+0000","last_peered":"2024-02-05T07:36:44.759645+0000","last_clean":"2024-02-05T07:36:44.759645+0000","last_became_active":"2024-02-05T07:28:26.590322+0000","last_became_peered":"2024-02-05T07:28:26.590322+0000","last_unstale":"2024-02-05T07:36:44.759645+0000","last_undegraded":"2024-02-05T07:36:44.759645+0000","last_fullsized":"2024-02-05T07:36:44.759645+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:30:39.726525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:30:39.726525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":382,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759899+0000","last_change":"2024-02-05T07:33:44.867890+0000","last_active":"2024-02-05T07:36:44.759899+0000","last_peered":"2024-02-05T07:36:44.759899+0000","last_clean":"2024-02-05T07:36:44.759899+0000","last_became_active":"2024-02-05T07:30:40.504365+0000","last_became_peered":"2024-02-05T07:30:40.504365+0000","last_unstale":"2024-02-05T07:36:44.759899+0000","last_undegraded":"2024-02-05T07:36:44.759899+0000","last_fullsized":"2024-02-05T07:36:44.759899+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.867764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.867764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":822,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.775437+0000","last_change":"2024-02-05T07:36:32.710890+0000","last_active":"2024-02-05T07:36:44.775437+0000","last_peered":"2024-02-05T07:36:44.775437+0000","last_clean":"2024-02-05T07:36:44.775437+0000","last_became_active":"2024-02-05T07:36:32.710575+0000","last_became_peered":"2024-02-05T07:36:32.710575+0000","last_unstale":"2024-02-05T07:36:44.775437+0000","last_undegraded":"2024-02-05T07:36:44.775437+0000","last_fullsized":"2024-02-05T07:36:44.775437+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:15.923221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:35:15.923221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.14","version":"545'10846","reported_seq":19230,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:45.008217+0000","last_change":"2024-02-05T07:36:33.729322+0000","last_active":"2024-02-05T07:36:45.008217+0000","last_peered":"2024-02-05T07:36:45.008217+0000","last_clean":"2024-02-05T07:36:45.008217+0000","last_became_active":"2024-02-05T07:36:33.699658+0000","last_became_peered":"2024-02-05T07:36:33.699658+0000","last_unstale":"2024-02-05T07:36:45.008217+0000","last_undegraded":"2024-02-05T07:36:45.008217+0000","last_fullsized":"2024-02-05T07:36:45.008217+0000","mapping_epoch":536,"log_start":"481'7192","ondisk_log_start":"481'7192","created":493,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3654,"log_dups_size":0,"ondisk_log_size":3654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:49.692469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85655552,"num_objects":1307,"num_object_clones":0,"num_object_copies":3921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6795,"num_write_kb":259264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2700,"num_bytes_recovered":176947200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"325'19","reported_seq":1068,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759562+0000","last_change":"2024-02-05T07:35:18.020758+0000","last_active":"2024-02-05T07:36:44.759562+0000","last_peered":"2024-02-05T07:36:44.759562+0000","last_clean":"2024-02-05T07:36:44.759562+0000","last_became_active":"2024-02-05T07:27:16.707036+0000","last_became_peered":"2024-02-05T07:27:16.707036+0000","last_unstale":"2024-02-05T07:36:44.759562+0000","last_undegraded":"2024-02-05T07:36:44.759562+0000","last_fullsized":"2024-02-05T07:36:44.759562+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'19","last_scrub_stamp":"2024-02-05T07:35:18.020665+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:35:18.020665+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":838,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.760322+0000","last_change":"2024-02-05T07:36:39.547357+0000","last_active":"2024-02-05T07:36:44.760322+0000","last_peered":"2024-02-05T07:36:44.760322+0000","last_clean":"2024-02-05T07:36:44.760322+0000","last_became_active":"2024-02-05T07:28:26.616107+0000","last_became_peered":"2024-02-05T07:28:26.616107+0000","last_unstale":"2024-02-05T07:36:44.760322+0000","last_undegraded":"2024-02-05T07:36:44.760322+0000","last_fullsized":"2024-02-05T07:36:44.760322+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:39.547282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:39.547282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:49.619046+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":384,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759029+0000","last_change":"2024-02-05T07:35:17.049239+0000","last_active":"2024-02-05T07:36:44.759029+0000","last_peered":"2024-02-05T07:36:44.759029+0000","last_clean":"2024-02-05T07:36:44.759029+0000","last_became_active":"2024-02-05T07:30:40.491630+0000","last_became_peered":"2024-02-05T07:30:40.491630+0000","last_unstale":"2024-02-05T07:36:44.759029+0000","last_undegraded":"2024-02-05T07:36:44.759029+0000","last_fullsized":"2024-02-05T07:36:44.759029+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:17.049130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:35:17.049130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.766343+0000","last_change":"2024-02-05T07:36:32.737398+0000","last_active":"2024-02-05T07:36:44.766343+0000","last_peered":"2024-02-05T07:36:44.766343+0000","last_clean":"2024-02-05T07:36:44.766343+0000","last_became_active":"2024-02-05T07:36:32.733405+0000","last_became_peered":"2024-02-05T07:36:32.733405+0000","last_unstale":"2024-02-05T07:36:44.766343+0000","last_undegraded":"2024-02-05T07:36:44.766343+0000","last_fullsized":"2024-02-05T07:36:44.766343+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:47.484894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.484894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.17","version":"545'10762","reported_seq":18055,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:45.401925+0000","last_change":"2024-02-05T07:36:36.626867+0000","last_active":"2024-02-05T07:36:45.401925+0000","last_peered":"2024-02-05T07:36:45.401925+0000","last_clean":"2024-02-05T07:36:45.401925+0000","last_became_active":"2024-02-05T07:35:47.718921+0000","last_became_peered":"2024-02-05T07:35:47.718921+0000","last_unstale":"2024-02-05T07:36:45.401925+0000","last_undegraded":"2024-02-05T07:36:45.401925+0000","last_fullsized":"2024-02-05T07:36:45.401925+0000","mapping_epoch":512,"log_start":"482'7236","ondisk_log_start":"482'7236","created":493,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":5,"last_scrub":"539'10435","last_scrub_stamp":"2024-02-05T07:36:36.626755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:36.626755+0000","objects_scrubbed":1577,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:54.222932+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81920000,"num_objects":1250,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":327,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"326'27","reported_seq":666,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.820354+0000","last_change":"2024-02-05T07:36:32.847108+0000","last_active":"2024-02-05T07:36:46.820354+0000","last_peered":"2024-02-05T07:36:46.820354+0000","last_clean":"2024-02-05T07:36:46.820354+0000","last_became_active":"2024-02-05T07:36:32.741691+0000","last_became_peered":"2024-02-05T07:36:32.741691+0000","last_unstale":"2024-02-05T07:36:46.820354+0000","last_undegraded":"2024-02-05T07:36:46.820354+0000","last_fullsized":"2024-02-05T07:36:46.820354+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:33:25.674080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:25.674080+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":605,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.821391+0000","last_change":"2024-02-05T07:36:32.747614+0000","last_active":"2024-02-05T07:36:46.821391+0000","last_peered":"2024-02-05T07:36:46.821391+0000","last_clean":"2024-02-05T07:36:46.821391+0000","last_became_active":"2024-02-05T07:36:32.746731+0000","last_became_peered":"2024-02-05T07:36:32.746731+0000","last_unstale":"2024-02-05T07:36:46.821391+0000","last_undegraded":"2024-02-05T07:36:46.821391+0000","last_fullsized":"2024-02-05T07:36:46.821391+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:49.743255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:32:49.743255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1012,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.758837+0000","last_change":"2024-02-05T07:33:46.937819+0000","last_active":"2024-02-05T07:36:44.758837+0000","last_peered":"2024-02-05T07:36:44.758837+0000","last_clean":"2024-02-05T07:36:44.758837+0000","last_became_active":"2024-02-05T07:28:25.778935+0000","last_became_peered":"2024-02-05T07:28:25.778935+0000","last_unstale":"2024-02-05T07:36:44.758837+0000","last_undegraded":"2024-02-05T07:36:44.758837+0000","last_fullsized":"2024-02-05T07:36:44.758837+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:46.937738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.937738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":606,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-05T07:36:46.819995+0000","last_change":"2024-02-05T07:34:28.009081+0000","last_active":"2024-02-05T07:36:46.819995+0000","last_peered":"2024-02-05T07:36:46.819995+0000","last_clean":"2024-02-05T07:36:46.819995+0000","last_became_active":"2024-02-05T07:30:40.493086+0000","last_became_peered":"2024-02-05T07:30:40.493086+0000","last_unstale":"2024-02-05T07:36:46.819995+0000","last_undegraded":"2024-02-05T07:36:46.819995+0000","last_fullsized":"2024-02-05T07:36:46.819995+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:28.008966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.008966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.16","version":"548'10913","reported_seq":18282,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-05T07:36:48.739745+0000","last_change":"2024-02-05T07:35:48.199339+0000","last_active":"2024-02-05T07:36:48.739745+0000","last_peered":"2024-02-05T07:36:48.739745+0000","last_clean":"2024-02-05T07:36:48.739745+0000","last_became_active":"2024-02-05T07:35:46.685503+0000","last_became_peered":"2024-02-05T07:35:46.685503+0000","last_unstale":"2024-02-05T07:36:48.739745+0000","last_undegraded":"2024-02-05T07:36:48.739745+0000","last_fullsized":"2024-02-05T07:36:48.739745+0000","mapping_epoch":511,"log_start":"475'6675","ondisk_log_start":"475'6675","created":493,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":5,"last_scrub":"513'8989","last_scrub_stamp":"2024-02-05T07:35:48.199192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:48.199192+0000","objects_scrubbed":3075,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:36:59.539016+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75038720,"num_objects":1145,"num_object_clones":0,"num_object_copies":3435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1930,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"450'19","reported_seq":1037,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759053+0000","last_change":"2024-02-05T07:34:34.853204+0000","last_active":"2024-02-05T07:36:44.759053+0000","last_peered":"2024-02-05T07:36:44.759053+0000","last_clean":"2024-02-05T07:36:44.759053+0000","last_became_active":"2024-02-05T07:28:25.777815+0000","last_became_peered":"2024-02-05T07:28:25.777815+0000","last_unstale":"2024-02-05T07:36:44.759053+0000","last_undegraded":"2024-02-05T07:36:44.759053+0000","last_fullsized":"2024-02-05T07:36:44.759053+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'19","last_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_deep_scrub":"450'19","last_deep_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_clean_scrub_stamp":"2024-02-05T07:34:34.853068+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.766334+0000","last_change":"2024-02-05T07:36:32.734803+0000","last_active":"2024-02-05T07:36:44.766334+0000","last_peered":"2024-02-05T07:36:44.766334+0000","last_clean":"2024-02-05T07:36:44.766334+0000","last_became_active":"2024-02-05T07:36:32.734502+0000","last_became_peered":"2024-02-05T07:36:32.734502+0000","last_unstale":"2024-02-05T07:36:44.766334+0000","last_undegraded":"2024-02-05T07:36:44.766334+0000","last_fullsized":"2024-02-05T07:36:44.766334+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.117071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.117071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":29,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.765157+0000","last_change":"2024-02-05T07:36:32.709307+0000","last_active":"2024-02-05T07:36:44.765157+0000","last_peered":"2024-02-05T07:36:44.765157+0000","last_clean":"2024-02-05T07:36:44.765157+0000","last_became_active":"2024-02-05T07:36:32.708917+0000","last_became_peered":"2024-02-05T07:36:32.708917+0000","last_unstale":"2024-02-05T07:36:44.765157+0000","last_undegraded":"2024-02-05T07:36:44.765157+0000","last_fullsized":"2024-02-05T07:36:44.765157+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.377472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.377472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1003,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-05T07:36:44.759302+0000","last_change":"2024-02-05T07:32:43.608109+0000","last_active":"2024-02-05T07:36:44.759302+0000","last_peered":"2024-02-05T07:36:44.759302+0000","last_clean":"2024-02-05T07:36:44.759302+0000","last_became_active":"2024-02-05T07:30:40.490288+0000","last_became_peered":"2024-02-05T07:30:40.490288+0000","last_unstale":"2024-02-05T07:36:44.759302+0000","last_undegraded":"2024-02-05T07:36:44.759302+0000","last_fullsized":"2024-02-05T07:36:44.759302+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:43.608002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:32:43.608002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":2580480024,"num_objects":39377,"num_object_clones":0,"num_object_copies":118131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5125,"num_objects_unfound":0,"num_objects_dirty":39377,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157926,"num_write_kb":4903745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15829,"num_bytes_recovered":1037238272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8869392384,"data_stored":8869380168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99715,"ondisk_log_size":99715,"up":78,"acting":78,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":369,"num_read_kb":364,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697707,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":53,"num_write":33,"num_write_kb":438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1437762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1104,"num_read_kb":861,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":8928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":11876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":752,"ondisk_log_size":752,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1146,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":257698037980,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":301,"seq":1292785156198,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1725432,"kb_used_data":1201508,"kb_used_omap":5,"kb_used_meta":523898,"kb_avail":92019720,"statfs":{"total":95995035648,"available":94228193280,"internally_reserved":0,"allocated":1230344192,"data_stored":1229569836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":536472069},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":1.859,"5min":1.147,"15min":1.6419999999999999},"min":{"1min":0.19600000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.0990000000000002,"5min":13.976000000000001,"15min":14.218},"last":1.9670000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.147,"15min":1.6539999999999999},"min":{"1min":0.23599999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.0270000000000001,"5min":13.962999999999999,"15min":14.175000000000001},"last":3.1469999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.54,"15min":1.806},"min":{"1min":0.26700000000000002,"5min":0.217,"15min":0.217},"max":{"1min":8.8640000000000008,"5min":19.861000000000001,"15min":55.923000000000002},"last":1.742},{"interface":"front","average":{"1min":1.498,"5min":1.7549999999999999,"15min":2.4049999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":11.467000000000001,"5min":27.518000000000001,"15min":61.103000000000002},"last":1.1160000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":1.8899999999999999,"5min":1.704,"15min":2.2240000000000002},"min":{"1min":0.45600000000000002,"5min":0.27400000000000002,"15min":0.249},"max":{"1min":11.157,"5min":32.103000000000002,"15min":34.689999999999998},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.7290000000000001,"5min":1.704,"15min":2.577},"min":{"1min":0.26300000000000001,"5min":0.245,"15min":0.245},"max":{"1min":10.314,"5min":32.149999999999999,"15min":34.646999999999998},"last":1.694}]},{"osd":3,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":1.929,"5min":2.028,"15min":2.4220000000000002},"min":{"1min":0.30399999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":11.430999999999999,"5min":36.090000000000003,"15min":55.978999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":2.391,"5min":2.1120000000000001,"15min":2.629},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":11.332000000000001,"5min":35.161000000000001,"15min":55.631},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":2.0529999999999999,"15min":1.153},"min":{"1min":0.28299999999999997,"5min":0.224,"15min":0.224},"max":{"1min":1.3149999999999999,"5min":151.02099999999999,"15min":151.02099999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.79100000000000004,"5min":2.1899999999999999,"15min":1.141},"min":{"1min":0.27800000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.907,"5min":149.21799999999999,"15min":149.21799999999999},"last":0.80000000000000004}]},{"osd":5,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.59399999999999997,"15min":0.58899999999999997},"min":{"1min":0.317,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.915,"5min":2.8919999999999999,"15min":2.8919999999999999},"last":2.431},{"interface":"front","average":{"1min":0.502,"5min":0.47499999999999998,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.97099999999999997,"5min":1.0669999999999999,"15min":1.526},"last":0.46899999999999997}]},{"osd":7,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.50600000000000001,"15min":3.3359999999999999},"min":{"1min":0.34000000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":84.158000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.72499999999999998,"15min":3.508},"min":{"1min":0.317,"5min":0.312,"15min":0.312},"max":{"1min":0.874,"5min":23.055,"15min":85.018000000000001},"last":1.0600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575098,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2088632,"kb_used_data":1245296,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":91656520,"statfs":{"total":95995035648,"available":93856276480,"internally_reserved":0,"allocated":1275183104,"data_stored":1274369183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:35:53 2024","interfaces":[{"interface":"back","average":{"1min":5.5670000000000002,"5min":2.2120000000000002,"15min":2.569},"min":{"1min":0.32600000000000001,"5min":0.215,"15min":0.16800000000000001},"max":{"1min":29.449999999999999,"5min":29.449999999999999,"15min":45.393000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":6.0010000000000003,"5min":2.9020000000000001,"15min":2.7869999999999999},"min":{"1min":0.39300000000000002,"5min":0.28000000000000003,"15min":0.20899999999999999},"max":{"1min":29.268999999999998,"5min":29.268999999999998,"15min":45.031999999999996},"last":0.755}]},{"osd":2,"last update":"Mon Feb 5 07:36:20 2024","interfaces":[{"interface":"back","average":{"1min":6.0149999999999997,"5min":2.3330000000000002,"15min":2.2949999999999999},"min":{"1min":0.373,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":35.057000000000002,"5min":35.057000000000002,"15min":39.259999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":5.2569999999999997,"5min":2.2869999999999999,"15min":2.1699999999999999},"min":{"1min":0.29599999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":31.879000000000001,"5min":31.879000000000001,"15min":40.972000000000001},"last":2.4750000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:36:32 2024","interfaces":[{"interface":"back","average":{"1min":3.1389999999999998,"5min":2.6629999999999998,"15min":2.4399999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.20999999999999999},"max":{"1min":8.3919999999999995,"5min":46.777000000000001,"15min":46.777000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":2.7080000000000002,"5min":2.5350000000000001,"15min":2.379},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.3350000000000009,"5min":50.164999999999999,"15min":50.164999999999999},"last":0.97699999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":2.464,"15min":2.2240000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.312},"max":{"1min":3.0339999999999998,"5min":89.409000000000006,"15min":89.409000000000006},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.002,"5min":2.1619999999999999,"15min":2.1539999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":2.827,"5min":89.915000000000006,"15min":89.915000000000006},"last":0.44}]},{"osd":5,"last update":"Mon Feb 5 07:36:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.8240000000000001,"15min":2.1219999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.309},"max":{"1min":6.7610000000000001,"5min":46.398000000000003,"15min":106.672},"last":0.95199999999999996},{"interface":"front","average":{"1min":2.0499999999999998,"5min":1.7010000000000001,"15min":2.1659999999999999},"min":{"1min":0.39200000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":5.6219999999999999,"5min":8.5410000000000004,"15min":107.348},"last":0.82099999999999995}]},{"osd":6,"last update":"Mon Feb 5 07:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.1140000000000001,"15min":1.4850000000000001},"min":{"1min":0.314,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.7080000000000002,"5min":4.8209999999999997,"15min":5.5990000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.454,"15min":1.7909999999999999},"min":{"1min":0.42499999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":6.8179999999999996,"5min":8.923,"15min":8.923},"last":0.92800000000000005}]},{"osd":7,"last update":"Mon Feb 5 07:36:20 2024","interfaces":[{"interface":"back","average":{"1min":1.9570000000000001,"5min":2.2730000000000001,"15min":2.0710000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.318},"max":{"1min":5.7110000000000003,"5min":46.351999999999997,"15min":66.703999999999994},"last":0.877},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1599999999999999,"15min":1.6519999999999999},"min":{"1min":0.40500000000000003,"5min":0.39500000000000002,"15min":0.33500000000000002},"max":{"1min":2.8599999999999999,"5min":4.7510000000000003,"15min":61.594999999999999},"last":0.57099999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738623,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2676940,"kb_used_data":1829436,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":91068212,"statfs":{"total":95995035648,"available":93253849088,"internally_reserved":0,"allocated":1873342464,"data_stored":1872529648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":867826117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:35:47 2024","interfaces":[{"interface":"back","average":{"1min":3.1459999999999999,"5min":2.5070000000000001,"15min":2.3050000000000002},"min":{"1min":0.189,"5min":0.189,"15min":0.16500000000000001},"max":{"1min":11.311,"5min":25.277999999999999,"15min":45.372},"last":0.48099999999999998},{"interface":"front","average":{"1min":2.145,"5min":2.2989999999999999,"15min":2.3170000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.186},"max":{"1min":5.7619999999999996,"5min":24.652999999999999,"15min":45.231999999999999},"last":2.0859999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:36:14 2024","interfaces":[{"interface":"back","average":{"1min":6.0919999999999996,"5min":3.069,"15min":2.4540000000000002},"min":{"1min":0.22900000000000001,"5min":0.191,"15min":0.191},"max":{"1min":47.137999999999998,"5min":47.137999999999998,"15min":65.903999999999996},"last":2.5089999999999999},{"interface":"front","average":{"1min":6.351,"5min":2.9540000000000002,"15min":2.3940000000000001},"min":{"1min":0.58799999999999997,"5min":0.246,"15min":0.186},"max":{"1min":47.170999999999999,"5min":47.170999999999999,"15min":65.694000000000003},"last":2.4590000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:36:27 2024","interfaces":[{"interface":"back","average":{"1min":3.399,"5min":2.0230000000000001,"15min":2.3919999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.21299999999999999},"max":{"1min":26.43,"5min":26.43,"15min":65.155000000000001},"last":2.1709999999999998},{"interface":"front","average":{"1min":6.202,"5min":3.0550000000000002,"15min":2.8610000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.24299999999999999},"max":{"1min":25.337,"5min":25.337,"15min":65.114000000000004},"last":0.63600000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:35:47 2024","interfaces":[{"interface":"back","average":{"1min":2.948,"5min":1.335,"15min":2.1139999999999999},"min":{"1min":0.46400000000000002,"5min":0.27200000000000002,"15min":0.26700000000000002},"max":{"1min":11.417999999999999,"5min":11.417999999999999,"15min":65.188000000000002},"last":0.314},{"interface":"front","average":{"1min":1.8220000000000001,"5min":1.28,"15min":2.113},"min":{"1min":0.32200000000000001,"5min":0.29699999999999999,"15min":0.23699999999999999},"max":{"1min":4.6589999999999998,"5min":8.6899999999999995,"15min":64.873000000000005},"last":1.625}]},{"osd":5,"last update":"Mon Feb 5 07:36:27 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.1220000000000001,"15min":1.806},"min":{"1min":0.38900000000000001,"5min":0.318,"15min":0.27000000000000002},"max":{"1min":4.4409999999999998,"5min":4.6959999999999997,"15min":95.772999999999996},"last":2.1440000000000001},{"interface":"front","average":{"1min":1.7170000000000001,"5min":1.48,"15min":1.97},"min":{"1min":0.35799999999999998,"5min":0.32900000000000001,"15min":0.27600000000000002},"max":{"1min":4.4320000000000004,"5min":8.7599999999999998,"15min":96.028000000000006},"last":1.702}]},{"osd":6,"last update":"Mon Feb 5 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":2.9849999999999999,"5min":1.8200000000000001,"15min":1.452},"min":{"1min":0.39700000000000002,"5min":0.33700000000000002,"15min":0.26300000000000001},"max":{"1min":12.670999999999999,"5min":12.670999999999999,"15min":12.670999999999999},"last":0.379},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.4570000000000001,"15min":1.526},"min":{"1min":0.49399999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":4.3010000000000002,"5min":8.718,"15min":8.718},"last":1.573}]},{"osd":7,"last update":"Mon Feb 5 07:35:58 2024","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":1.548,"15min":2.0609999999999999},"min":{"1min":0.42899999999999999,"5min":0.34899999999999998,"15min":0.28599999999999998},"max":{"1min":11.456,"5min":11.456,"15min":65.263000000000005},"last":0.433},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.427,"15min":2.0249999999999999},"min":{"1min":0.39100000000000001,"5min":0.36699999999999999,"15min":0.309},"max":{"1min":4.5179999999999998,"5min":8.7910000000000004,"15min":64.906000000000006},"last":1.6679999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411573,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2207368,"kb_used_data":1399932,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":91537784,"statfs":{"total":95995035648,"available":93734690816,"internally_reserved":0,"allocated":1433530368,"data_stored":1432775819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:36:14 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":2.4249999999999998,"15min":2.6040000000000001},"min":{"1min":0.36299999999999999,"5min":0.20100000000000001,"15min":0.182},"max":{"1min":80.427000000000007,"5min":80.427000000000007,"15min":92.638000000000005},"last":1.042},{"interface":"front","average":{"1min":8.4060000000000006,"5min":3.1299999999999999,"15min":2.9009999999999998},"min":{"1min":0.40300000000000002,"5min":0.22700000000000001,"15min":0.20699999999999999},"max":{"1min":80.939999999999998,"5min":80.939999999999998,"15min":93.481999999999999},"last":2.7170000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:36:14 2024","interfaces":[{"interface":"back","average":{"1min":11.689,"5min":4.9340000000000002,"15min":3.169},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":110.917,"5min":110.917,"15min":110.917},"last":4.9119999999999999},{"interface":"front","average":{"1min":1.385,"5min":3.0600000000000001,"15min":2.6640000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":5.75,"5min":64.771000000000001,"15min":93.004000000000005},"last":3.1619999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:36:30 2024","interfaces":[{"interface":"back","average":{"1min":10.382,"5min":4.298,"15min":2.9689999999999999},"min":{"1min":0.253,"5min":0.252,"15min":0.214},"max":{"1min":110.773,"5min":110.773,"15min":110.773},"last":5.0350000000000001},{"interface":"front","average":{"1min":3.0329999999999999,"5min":2.5760000000000001,"15min":2.3410000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.193},"max":{"1min":47.368000000000002,"5min":63.521000000000001,"15min":63.521000000000001},"last":3.367}]},{"osd":4,"last update":"Mon Feb 5 07:35:53 2024","interfaces":[{"interface":"back","average":{"1min":2.282,"5min":1.276,"15min":2.1880000000000002},"min":{"1min":0.27700000000000002,"5min":0.24299999999999999,"15min":0.24099999999999999},"max":{"1min":6.6150000000000002,"5min":6.6150000000000002,"15min":168.11699999999999},"last":2.02},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.0820000000000001,"15min":2.105},"min":{"1min":0.42999999999999999,"5min":0.34399999999999997,"15min":0.27200000000000002},"max":{"1min":9.3539999999999992,"5min":9.3539999999999992,"15min":168.589},"last":1.1619999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.98899999999999999,"15min":1.3080000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.315},"max":{"1min":1.246,"5min":4.742,"15min":39.923000000000002},"last":1.2290000000000001},{"interface":"front","average":{"1min":5.8140000000000001,"5min":2.089,"15min":1.6120000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.26400000000000001},"max":{"1min":47.761000000000003,"5min":47.761000000000003,"15min":47.761000000000003},"last":4.8029999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:36:40 2024","interfaces":[{"interface":"back","average":{"1min":5.1420000000000003,"5min":2.335,"15min":1.5580000000000001},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":47.726999999999997,"5min":47.726999999999997,"15min":47.726999999999997},"last":4.8550000000000004},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.153,"15min":1.2889999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.33100000000000002},"max":{"1min":4.4550000000000001,"5min":9.2899999999999991,"15min":9.2899999999999991},"last":1.198}]},{"osd":7,"last update":"Mon Feb 5 07:36:14 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.494,"15min":1.5569999999999999},"min":{"1min":0.35699999999999998,"5min":0.35199999999999998,"15min":0.29299999999999998},"max":{"1min":6.4669999999999996,"5min":11.311999999999999,"15min":37.573999999999998},"last":2.0600000000000001},{"interface":"front","average":{"1min":10.957000000000001,"5min":3.1669999999999998,"15min":2.0720000000000001},"min":{"1min":0.43099999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":110.86799999999999,"5min":110.86799999999999,"15min":110.86799999999999},"last":4.8840000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149936,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2092172,"kb_used_data":1226620,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":91652980,"statfs":{"total":95995035648,"available":93852651520,"internally_reserved":0,"allocated":1256058880,"data_stored":1255246921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":886307271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:35:49 2024","interfaces":[{"interface":"back","average":{"1min":3.081,"5min":1.7150000000000001,"15min":2.472},"min":{"1min":0.28699999999999998,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":30.849,"5min":30.849,"15min":74.114999999999995},"last":1.7310000000000001},{"interface":"front","average":{"1min":4.4089999999999998,"5min":3.016,"15min":2.6699999999999999},"min":{"1min":0.223,"5min":0.185,"15min":0.185},"max":{"1min":36.847999999999999,"5min":99.989999999999995,"15min":99.989999999999995},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Feb 5 07:35:49 2024","interfaces":[{"interface":"back","average":{"1min":2.875,"5min":2.9710000000000001,"15min":2.7749999999999999},"min":{"1min":0.32700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":26.329000000000001,"5min":102.20399999999999,"15min":102.20399999999999},"last":1.216},{"interface":"front","average":{"1min":2.8610000000000002,"5min":2.9809999999999999,"15min":3.0390000000000001},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.185},"max":{"1min":26.867000000000001,"5min":102.286,"15min":102.286},"last":1.2949999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:35:49 2024","interfaces":[{"interface":"back","average":{"1min":8.8580000000000005,"5min":4.4450000000000003,"15min":3.3069999999999999},"min":{"1min":0.34999999999999998,"5min":0.253,"15min":0.22700000000000001},"max":{"1min":145.94999999999999,"5min":145.94999999999999,"15min":145.94999999999999},"last":2.1200000000000001},{"interface":"front","average":{"1min":9.8059999999999992,"5min":4.6680000000000001,"15min":2.915},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":146.006,"5min":146.006,"15min":146.006},"last":0.52900000000000003}]},{"osd":4,"last update":"Mon Feb 5 07:35:56 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":2.6800000000000002,"15min":1.794},"min":{"1min":0.30499999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":8.1289999999999996,"5min":100.024,"15min":100.024},"last":0.499},{"interface":"front","average":{"1min":1.1220000000000001,"5min":3.1160000000000001,"15min":1.925},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.28599999999999998},"max":{"1min":4.359,"5min":99.859999999999999,"15min":99.859999999999999},"last":0.376}]},{"osd":5,"last update":"Mon Feb 5 07:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1819999999999999,"15min":1.952},"min":{"1min":0.308,"5min":0.308,"15min":0.28499999999999998},"max":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":99.911000000000001},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.202,"15min":1.7150000000000001},"min":{"1min":0.36799999999999999,"5min":0.33000000000000002,"15min":0.27400000000000002},"max":{"1min":8.173,"5min":8.173,"15min":100.053},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Feb 5 07:35:49 2024","interfaces":[{"interface":"back","average":{"1min":3.5819999999999999,"5min":2.8180000000000001,"15min":1.5449999999999999},"min":{"1min":0.38500000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":32.505000000000003,"5min":99.807000000000002,"15min":99.807000000000002},"last":1.3660000000000001},{"interface":"front","average":{"1min":3.3410000000000002,"5min":2.9660000000000002,"15min":1.6279999999999999},"min":{"1min":0.35699999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":31.084,"5min":101.489,"15min":101.489},"last":0.55000000000000004}]},{"osd":7,"last update":"Mon Feb 5 07:36:02 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.359,"15min":2.3820000000000001},"min":{"1min":0.38700000000000001,"5min":0.34499999999999997,"15min":0.29499999999999998},"max":{"1min":4.4199999999999999,"5min":6.7649999999999997,"15min":99.893000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.5309999999999999,"15min":2.71},"min":{"1min":0.47499999999999998,"5min":0.375,"15min":0.375},"max":{"1min":6.726,"5min":17.611000000000001,"15min":99.879000000000005},"last":1.3939999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953707,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2657356,"kb_used_data":1872260,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":91087796,"statfs":{"total":95995035648,"available":93273903104,"internally_reserved":0,"allocated":1917194240,"data_stored":1916423527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:35:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":0.80100000000000005,"15min":2.181},"min":{"1min":0.27000000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.1950000000000003,"5min":4.1950000000000003,"15min":101.212},"last":1.337},{"interface":"front","average":{"1min":1.222,"5min":0.82599999999999996,"15min":2.2719999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.0779999999999998,"5min":4.0659999999999998,"15min":102.07599999999999},"last":2.2519999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:35:52 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":2.0699999999999998,"15min":2.3650000000000002},"min":{"1min":0.28899999999999998,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":3.5979999999999999,"5min":45.655999999999999,"15min":69.102000000000004},"last":2.6320000000000001},{"interface":"front","average":{"1min":1.111,"5min":2.1629999999999998,"15min":2.589},"min":{"1min":0.27600000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.2480000000000002,"5min":46.531999999999996,"15min":68.108999999999995},"last":2.2850000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:35:52 2024","interfaces":[{"interface":"back","average":{"1min":6.4969999999999999,"5min":4.1550000000000002,"15min":2.5379999999999998},"min":{"1min":0.25900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":93.075000000000003,"5min":180.40199999999999,"15min":180.40199999999999},"last":0.26000000000000001},{"interface":"front","average":{"1min":6.0410000000000004,"5min":4.1459999999999999,"15min":2.5169999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.24299999999999999},"max":{"1min":92.600999999999999,"5min":180.381,"15min":180.381},"last":2.5179999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":2.5720000000000001,"15min":1.911},"min":{"1min":0.29899999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.5149999999999999,"5min":46.549999999999997,"15min":66.019999999999996},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":2.8359999999999999,"15min":2.177},"min":{"1min":0.317,"5min":0.317,"15min":0.26400000000000001},"max":{"1min":2.3780000000000001,"5min":46.003999999999998,"15min":101.001},"last":0.47799999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:36:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.46899999999999997,"15min":0.59899999999999998},"min":{"1min":0.309,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1719999999999999,"5min":1.4299999999999999,"15min":10.917999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.625},"min":{"1min":0.315,"5min":0.26700000000000002,"15min":0.24199999999999999},"max":{"1min":0.89500000000000002,"5min":1.3939999999999999,"15min":15.105},"last":0.437}]},{"osd":6,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55600000000000005,"15min":0.98099999999999998},"min":{"1min":0.28699999999999998,"5min":0.25600000000000001,"15min":0.245},"max":{"1min":2.2829999999999999,"5min":3.5670000000000002,"15min":14.606},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.52600000000000002,"15min":1.1439999999999999},"min":{"1min":0.33000000000000002,"5min":0.254,"15min":0.254},"max":{"1min":1.069,"5min":1.377,"15min":13.114000000000001},"last":0.35199999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:35:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.53500000000000003,"15min":0.73199999999999998},"min":{"1min":0.42299999999999999,"5min":0.30599999999999999,"15min":0.25700000000000001},"max":{"1min":1.008,"5min":1.4350000000000001,"15min":15.073},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.52900000000000003,"15min":0.69899999999999995},"min":{"1min":0.33500000000000002,"5min":0.316,"15min":0.248},"max":{"1min":2.0699999999999998,"5min":2.0699999999999998,"15min":14.581},"last":0.373}]}]},{"osd":5,"up_from":0,"seq":176093659366,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1869549568,"data_stored":1869545496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1261502464,"data_stored":1261502464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1422983168,"data_stored":1422983168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1246101504,"data_stored":1246101504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1857687552,"data_stored":1857683480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1211568128,"data_stored":1211564056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:36:53.822 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-05T07:36:53.822 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-02-05T07:36:53.872 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:53.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:53.906 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:53.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:54.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:54.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:54.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:54.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:54.162 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:53 smithi007 ceph-mon[29098]: pgmap v989: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 784 B/s rd, 0 op/s; 4454/108045 objects misplaced (4.122%) 2024-02-05T07:36:54.162 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:53 smithi007 ceph-mon[29098]: 10.a scrub starts 2024-02-05T07:36:54.163 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:53 smithi007 ceph-mon[37158]: pgmap v989: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 784 B/s rd, 0 op/s; 4454/108045 objects misplaced (4.122%) 2024-02-05T07:36:54.163 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:53 smithi007 ceph-mon[37158]: 10.a scrub starts 2024-02-05T07:36:54.169 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:54.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:54.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:54.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:54.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:53 smithi090 ceph-mon[33213]: pgmap v989: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 784 B/s rd, 0 op/s; 4454/108045 objects misplaced (4.122%) 2024-02-05T07:36:54.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:53 smithi090 ceph-mon[33213]: 10.a scrub starts 2024-02-05T07:36:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:54.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:54.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:54.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:54.421 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:54.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:54.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:54.549 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:54.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:54.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:54.689 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:54.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:54.791 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:54.819 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:54.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:54.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:54.958 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:54.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:55.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:55.115 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:55.138 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:55.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:55.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:36:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:36:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[29098]: osdmap e550: 8 total, 8 up, 6 in 2024-02-05T07:36:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[29098]: 3.c deep-scrub starts 2024-02-05T07:36:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[29098]: 3.c deep-scrub ok 2024-02-05T07:36:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[29098]: 10.a scrub ok 2024-02-05T07:36:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[29098]: 3.1c scrub starts 2024-02-05T07:36:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[29098]: 3.1c scrub ok 2024-02-05T07:36:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:36:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:36:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[37158]: osdmap e550: 8 total, 8 up, 6 in 2024-02-05T07:36:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[37158]: 3.c deep-scrub starts 2024-02-05T07:36:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[37158]: 3.c deep-scrub ok 2024-02-05T07:36:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[37158]: 10.a scrub ok 2024-02-05T07:36:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[37158]: 3.1c scrub starts 2024-02-05T07:36:55.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:54 smithi007 ceph-mon[37158]: 3.1c scrub ok 2024-02-05T07:36:55.240 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:55.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:55.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:54 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:36:55.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:54 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:36:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:54 smithi090 ceph-mon[33213]: osdmap e550: 8 total, 8 up, 6 in 2024-02-05T07:36:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:54 smithi090 ceph-mon[33213]: 3.c deep-scrub starts 2024-02-05T07:36:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:54 smithi090 ceph-mon[33213]: 3.c deep-scrub ok 2024-02-05T07:36:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:54 smithi090 ceph-mon[33213]: 10.a scrub ok 2024-02-05T07:36:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:54 smithi090 ceph-mon[33213]: 3.1c scrub starts 2024-02-05T07:36:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:54 smithi090 ceph-mon[33213]: 3.1c scrub ok 2024-02-05T07:36:55.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:55.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:55.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:55.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:55.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:55.533 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:55.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:55.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:55.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:55.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:55.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:55.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:55.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:55.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:55.855 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:36:55.856 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:36:55.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.856 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:229223 10.8 10:166d9784:::benchmark_data_smithi007_157098_object100293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-02-05T07:36:55.856 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:55.836973+0000", 2024-02-05T07:36:55.856 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.016254175999999999, 2024-02-05T07:36:55.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016302621999999999, 2024-02-05T07:36:55.856 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:36:55.856 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:36:55.856 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:36:55.856 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:36:55.856 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:36:55.856 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 229223 2024-02-05T07:36:55.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.856 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:36:55.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:36:55.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.836973+0000", 2024-02-05T07:36:55.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:36:55.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:36:55.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.836973+0000", 2024-02-05T07:36:55.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:36:55.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:36:55.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.836974+0000", 2024-02-05T07:36:55.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.432e-06 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.836979+0000", 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2529999999999998e-06 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.836991+0000", 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.24e-05 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.836999+0000", 2024-02-05T07:36:55.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9389999999999997e-06 2024-02-05T07:36:55.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:36:55.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.837046+0000", 2024-02-05T07:36:55.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7475999999999997e-05 2024-02-05T07:36:55.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:36:55.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.837187+0000", 2024-02-05T07:36:55.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014032699999999999 2024-02-05T07:36:55.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,4", 2024-02-05T07:36:55.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.837287+0000", 2024-02-05T07:36:55.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000100505 2024-02-05T07:36:55.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.860 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:36:55.860 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.839544+0000", 2024-02-05T07:36:55.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0022565910000000001 2024-02-05T07:36:55.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.860 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:36:55.860 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.840611+0000", 2024-02-05T07:36:55.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001066769 2024-02-05T07:36:55.860 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:36:55.860 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:36:55.860 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:229230 10.a e550/535 10:5b50e270:::benchmark_data_smithi007_157098_object100300:head v 550'14361, mlcod=550'14358)", 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:55.845578+0000", 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0076492169999999998, 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0078323500000000001, 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.845578+0000", 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.845578+0000", 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.845579+0000", 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7199999999999998e-07 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.845586+0000", 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.79e-06 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:36:55.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.845596+0000", 2024-02-05T07:36:55.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6360000000000006e-06 2024-02-05T07:36:55.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:36:55.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.845603+0000", 2024-02-05T07:36:55.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7939999999999997e-06 2024-02-05T07:36:55.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:36:55.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.847109+0000", 2024-02-05T07:36:55.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001506463 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.847128+0000", 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8300000000000001e-05 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.853253+0000", 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0061248209999999999 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.865 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:229232 10.5 10:a2174fa5:::benchmark_data_smithi007_157098_object100302:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-02-05T07:36:55.865 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:55.846075+0000", 2024-02-05T07:36:55.865 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0071518579999999997, 2024-02-05T07:36:55.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0074404709999999997, 2024-02-05T07:36:55.865 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:36:55.865 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:36:55.865 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:36:55.865 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:36:55.865 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:36:55.865 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 229232 2024-02-05T07:36:55.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.865 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:36:55.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:36:55.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846075+0000", 2024-02-05T07:36:55.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:36:55.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:36:55.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846075+0000", 2024-02-05T07:36:55.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:36:55.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:36:55.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846077+0000", 2024-02-05T07:36:55.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.564e-06 2024-02-05T07:36:55.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:36:55.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846082+0000", 2024-02-05T07:36:55.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0459999999999996e-06 2024-02-05T07:36:55.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:36:55.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846097+0000", 2024-02-05T07:36:55.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5435999999999999e-05 2024-02-05T07:36:55.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:36:55.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846106+0000", 2024-02-05T07:36:55.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3450000000000006e-06 2024-02-05T07:36:55.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.868 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:36:55.868 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846265+0000", 2024-02-05T07:36:55.868 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015939700000000001 2024-02-05T07:36:55.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.868 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:36:55.868 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846496+0000", 2024-02-05T07:36:55.868 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023114699999999999 2024-02-05T07:36:55.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.868 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,3", 2024-02-05T07:36:55.868 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.852454+0000", 2024-02-05T07:36:55.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059579009999999998 2024-02-05T07:36:55.869 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:36:55.869 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:36:55.869 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:36:55.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.869 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:229233 10.9 10:93fd0dd5:::benchmark_data_smithi007_157098_object100303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-02-05T07:36:55.869 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:55.846484+0000", 2024-02-05T07:36:55.869 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0067431640000000003, 2024-02-05T07:36:55.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0071246590000000002, 2024-02-05T07:36:55.869 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:36:55.869 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-02-05T07:36:55.869 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:36:55.869 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:36:55.870 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:36:55.870 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 229233 2024-02-05T07:36:55.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.870 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:36:55.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:36:55.870 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846484+0000", 2024-02-05T07:36:55.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:36:55.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:36:55.870 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846484+0000", 2024-02-05T07:36:55.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:36:55.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846485+0000", 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8000000000000004e-07 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846489+0000", 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4700000000000004e-06 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846500+0000", 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0557000000000001e-05 2024-02-05T07:36:55.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:36:55.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846507+0000", 2024-02-05T07:36:55.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5830000000000001e-06 2024-02-05T07:36:55.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:36:55.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846561+0000", 2024-02-05T07:36:55.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3324e-05 2024-02-05T07:36:55.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:36:55.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846691+0000", 2024-02-05T07:36:55.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000130185 2024-02-05T07:36:55.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-02-05T07:36:55.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.846816+0000", 2024-02-05T07:36:55.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012464900000000001 2024-02-05T07:36:55.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:36:55.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.852470+0000", 2024-02-05T07:36:55.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0056542789999999999 2024-02-05T07:36:55.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.852483+0000", 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3117e-05 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.25454.0:229223 10.8 e550/536 ondisk, result = 0)", 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:55.853478+0000", 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 4294967295.9997487, 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029514999999999999, 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:36:55.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.875 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:36:55.875 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.853478+0000", 2024-02-05T07:36:55.875 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:36:55.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.875 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:36:55.875 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.853478+0000", 2024-02-05T07:36:55.875 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:36:55.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.875 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:36:55.875 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.853479+0000", 2024-02-05T07:36:55.875 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0400000000000005e-07 2024-02-05T07:36:55.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:36:55.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.853484+0000", 2024-02-05T07:36:55.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2599999999999999e-06 2024-02-05T07:36:55.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:36:55.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.853494+0000", 2024-02-05T07:36:55.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0818e-05 2024-02-05T07:36:55.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:36:55.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.853502+0000", 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6699999999999994e-06 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.853560+0000", 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7404000000000001e-05 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:55.853568+0000", 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.303e-06 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:36:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:36:55.878 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:36:55.878 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 5 2024-02-05T07:36:55.878 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:36:55.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:55.931 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:55.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:56.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:56.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:55 smithi007 ceph-mon[29098]: pgmap v991: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 2.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s; 4277/99330 objects misplaced (4.306%) 2024-02-05T07:36:56.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:55 smithi007 ceph-mon[37158]: pgmap v991: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 2.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s; 4277/99330 objects misplaced (4.306%) 2024-02-05T07:36:56.075 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:56.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:56.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:56.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:56.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:55 smithi090 ceph-mon[33213]: pgmap v991: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 2.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s; 4277/99330 objects misplaced (4.306%) 2024-02-05T07:36:56.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:36:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:56.352 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:56.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:36:56.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:56.493 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:56.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:56.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:56.579 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:36:56.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:56.696 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:56.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:56.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:56.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:56.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:56.849 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:56.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:56.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:56.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:57.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:57.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:57.131 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:56 smithi007 ceph-mon[29098]: 4.4 deep-scrub starts 2024-02-05T07:36:57.131 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:56 smithi007 ceph-mon[29098]: 4.4 deep-scrub ok 2024-02-05T07:36:57.131 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:56 smithi007 ceph-mon[37158]: 4.4 deep-scrub starts 2024-02-05T07:36:57.131 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:56 smithi007 ceph-mon[37158]: 4.4 deep-scrub ok 2024-02-05T07:36:57.131 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:57.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:57.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:57.174 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:56 smithi090 ceph-mon[33213]: 4.4 deep-scrub starts 2024-02-05T07:36:57.174 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:56 smithi090 ceph-mon[33213]: 4.4 deep-scrub ok 2024-02-05T07:36:57.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T07:36:57.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:57.238 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:57.278 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:57.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:57.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:57.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:57.410 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:57.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:57.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:57.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:57.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:57.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:57.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:57.676 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:57.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:57.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:57.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:57.807 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:57.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:57.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:57.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:36:57.935 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:36:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:57.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:57.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:36:58.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:58.108 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:58.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:58.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:58.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:57 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:36:58.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:57 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:36:58.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:57 smithi007 ceph-mon[29098]: pgmap v992: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 3650/89709 objects misplaced (4.069%) 2024-02-05T07:36:58.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:57 smithi007 ceph-mon[29098]: 5.1f scrub starts 2024-02-05T07:36:58.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:57 smithi007 ceph-mon[29098]: 5.1f scrub ok 2024-02-05T07:36:58.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:57 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:36:58.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:57 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:36:58.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:57 smithi007 ceph-mon[37158]: pgmap v992: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 3650/89709 objects misplaced (4.069%) 2024-02-05T07:36:58.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:57 smithi007 ceph-mon[37158]: 5.1f scrub starts 2024-02-05T07:36:58.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:57 smithi007 ceph-mon[37158]: 5.1f scrub ok 2024-02-05T07:36:58.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:58.237 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:58.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:58.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:57 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:36:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:57 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:36:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:57 smithi090 ceph-mon[33213]: pgmap v992: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 3650/89709 objects misplaced (4.069%) 2024-02-05T07:36:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:57 smithi090 ceph-mon[33213]: 5.1f scrub starts 2024-02-05T07:36:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:57 smithi090 ceph-mon[33213]: 5.1f scrub ok 2024-02-05T07:36:58.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:58.378 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:58.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:58.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:58.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:58.514 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:58.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:58.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:58.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:36:58.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:58.649 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:58.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:36:58.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:58.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:58.797 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:58.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:58.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:36:58.937 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:58.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:36:59.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:36:59.070 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:59.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:59.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:36:59.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:59.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[29098]: osdmap e551: 8 total, 8 up, 6 in 2024-02-05T07:36:59.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[29098]: 6.12 deep-scrub starts 2024-02-05T07:36:59.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[29098]: 6.12 deep-scrub ok 2024-02-05T07:36:59.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[29098]: 10.d scrub starts 2024-02-05T07:36:59.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[29098]: 1.5 scrub starts 2024-02-05T07:36:59.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[29098]: 1.5 scrub ok 2024-02-05T07:36:59.176 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:36:59.176 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:36:59.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[37158]: osdmap e551: 8 total, 8 up, 6 in 2024-02-05T07:36:59.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[37158]: 6.12 deep-scrub starts 2024-02-05T07:36:59.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[37158]: 6.12 deep-scrub ok 2024-02-05T07:36:59.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[37158]: 10.d scrub starts 2024-02-05T07:36:59.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[37158]: 1.5 scrub starts 2024-02-05T07:36:59.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[37158]: 1.5 scrub ok 2024-02-05T07:36:59.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:36:59.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:58 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:36:59.219 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:59.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:59.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:36:59.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:58 smithi090 ceph-mon[33213]: osdmap e551: 8 total, 8 up, 6 in 2024-02-05T07:36:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:58 smithi090 ceph-mon[33213]: 6.12 deep-scrub starts 2024-02-05T07:36:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:58 smithi090 ceph-mon[33213]: 6.12 deep-scrub ok 2024-02-05T07:36:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:58 smithi090 ceph-mon[33213]: 10.d scrub starts 2024-02-05T07:36:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:58 smithi090 ceph-mon[33213]: 1.5 scrub starts 2024-02-05T07:36:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:58 smithi090 ceph-mon[33213]: 1.5 scrub ok 2024-02-05T07:36:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:58 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:36:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:58 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:36:59.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:36:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:36:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:59.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:36:59.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:59.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:59.492 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:59.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:59.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:59.642 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:59.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:59.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:59.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:36:59.784 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:59.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:36:59.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:36:59.923 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:36:59.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:36:59.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:00.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:00.069 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:00.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:00.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[29098]: osdmap e552: 8 total, 8 up, 6 in 2024-02-05T07:37:00.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[29098]: 10.d scrub ok 2024-02-05T07:37:00.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[29098]: 4.14 scrub starts 2024-02-05T07:37:00.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[29098]: 4.14 scrub ok 2024-02-05T07:37:00.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[29098]: 3.1a scrub starts 2024-02-05T07:37:00.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[29098]: 3.1a scrub ok 2024-02-05T07:37:00.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[29098]: pgmap v995: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 1.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 227 op/s; 3453/83058 objects misplaced (4.157%) 2024-02-05T07:37:00.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[29098]: 5.e scrub starts 2024-02-05T07:37:00.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[29098]: 5.e scrub ok 2024-02-05T07:37:00.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[37158]: osdmap e552: 8 total, 8 up, 6 in 2024-02-05T07:37:00.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[37158]: 10.d scrub ok 2024-02-05T07:37:00.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[37158]: 4.14 scrub starts 2024-02-05T07:37:00.099 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[37158]: 4.14 scrub ok 2024-02-05T07:37:00.099 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[37158]: 3.1a scrub starts 2024-02-05T07:37:00.099 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[37158]: 3.1a scrub ok 2024-02-05T07:37:00.099 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[37158]: pgmap v995: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 1.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 227 op/s; 3453/83058 objects misplaced (4.157%) 2024-02-05T07:37:00.099 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[37158]: 5.e scrub starts 2024-02-05T07:37:00.099 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:36:59 smithi007 ceph-mon[37158]: 5.e scrub ok 2024-02-05T07:37:00.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:00.211 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:00.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:00.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:00.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:59 smithi090 ceph-mon[33213]: osdmap e552: 8 total, 8 up, 6 in 2024-02-05T07:37:00.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:59 smithi090 ceph-mon[33213]: 10.d scrub ok 2024-02-05T07:37:00.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:59 smithi090 ceph-mon[33213]: 4.14 scrub starts 2024-02-05T07:37:00.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:59 smithi090 ceph-mon[33213]: 4.14 scrub ok 2024-02-05T07:37:00.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:59 smithi090 ceph-mon[33213]: 3.1a scrub starts 2024-02-05T07:37:00.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:59 smithi090 ceph-mon[33213]: 3.1a scrub ok 2024-02-05T07:37:00.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:59 smithi090 ceph-mon[33213]: pgmap v995: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 1.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 227 op/s; 3453/83058 objects misplaced (4.157%) 2024-02-05T07:37:00.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:59 smithi090 ceph-mon[33213]: 5.e scrub starts 2024-02-05T07:37:00.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:36:59 smithi090 ceph-mon[33213]: 5.e scrub ok 2024-02-05T07:37:00.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:00.357 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:00.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:00.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:00.488 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:00.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:00.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:00.590 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:00.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:00.616 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:00.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:00.717 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:00.757 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:00.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:00.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:00.890 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:00.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:00.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:00.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:01.020 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:01.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:01.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:01.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:01.153 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:01.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:01.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:01.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:00 smithi007 ceph-mon[29098]: 6.9 scrub starts 2024-02-05T07:37:01.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:00 smithi007 ceph-mon[29098]: 6.9 scrub ok 2024-02-05T07:37:01.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:00 smithi007 ceph-mon[37158]: 6.9 scrub starts 2024-02-05T07:37:01.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:00 smithi007 ceph-mon[37158]: 6.9 scrub ok 2024-02-05T07:37:01.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:01.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:37:01 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:37:01.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:00 smithi090 ceph-mon[33213]: 6.9 scrub starts 2024-02-05T07:37:01.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:00 smithi090 ceph-mon[33213]: 6.9 scrub ok 2024-02-05T07:37:01.283 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:01.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:01.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:01.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:01.411 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:01.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:01.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:01.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:01.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:01.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:01.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:01.693 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:01.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:01.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:01.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:01.823 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:01.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:01.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:01.829 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:37:01.878 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:37:01.920 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:37:01.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:01.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:01 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:37:01.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:01 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:37:01.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:01.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:01 smithi007 ceph-mon[29098]: 5.16 scrub starts 2024-02-05T07:37:01.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:01 smithi007 ceph-mon[29098]: 5.16 scrub ok 2024-02-05T07:37:01.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:01 smithi007 ceph-mon[29098]: pgmap v996: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 1.4 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 386 op/s; 2786/71364 objects misplaced (3.904%) 2024-02-05T07:37:01.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:01 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1437712897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:01.945 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:01 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:37:01.945 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:01 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:37:01.945 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:01.952 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:01.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:01.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:01.964 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:37:02.004 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:37:02.046 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:37:02.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:02.083 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:37:02.085 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:02.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:02.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:02.122 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:37:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:37:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:37:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:37:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:37:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:37:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:37:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:37:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:37:02.131 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:01 smithi007 ceph-mon[37158]: 5.16 scrub starts 2024-02-05T07:37:02.132 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:01 smithi007 ceph-mon[37158]: 5.16 scrub ok 2024-02-05T07:37:02.132 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:01 smithi007 ceph-mon[37158]: pgmap v996: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 1.4 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 386 op/s; 2786/71364 objects misplaced (3.904%) 2024-02-05T07:37:02.132 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:01 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1437712897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:02.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:02.215 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:02.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:02.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:01 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:37:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:01 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:37:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:01 smithi090 ceph-mon[33213]: 5.16 scrub starts 2024-02-05T07:37:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:01 smithi090 ceph-mon[33213]: 5.16 scrub ok 2024-02-05T07:37:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:01 smithi090 ceph-mon[33213]: pgmap v996: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 1.4 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 386 op/s; 2786/71364 objects misplaced (3.904%) 2024-02-05T07:37:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:01 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1437712897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:02.318 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:02.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:02.370 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:02.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:02.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:02.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:02.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:02.630 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:02.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:02.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:02.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:02.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:02.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:02.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:02.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:03.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:03.039 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:03.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:03.141 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:03.192 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:03.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:03.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:03.269 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:02 smithi007 ceph-mon[29098]: 6.a scrub starts 2024-02-05T07:37:03.269 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:02 smithi007 ceph-mon[29098]: 6.a scrub ok 2024-02-05T07:37:03.269 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:02 smithi007 ceph-mon[29098]: 4.c scrub starts 2024-02-05T07:37:03.269 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:02 smithi007 ceph-mon[29098]: 4.c scrub ok 2024-02-05T07:37:03.269 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:02 smithi007 ceph-mon[37158]: 6.a scrub starts 2024-02-05T07:37:03.269 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:02 smithi007 ceph-mon[37158]: 6.a scrub ok 2024-02-05T07:37:03.269 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:02 smithi007 ceph-mon[37158]: 4.c scrub starts 2024-02-05T07:37:03.270 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:02 smithi007 ceph-mon[37158]: 4.c scrub ok 2024-02-05T07:37:03.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:02 smithi090 ceph-mon[33213]: 6.a scrub starts 2024-02-05T07:37:03.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:02 smithi090 ceph-mon[33213]: 6.a scrub ok 2024-02-05T07:37:03.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:02 smithi090 ceph-mon[33213]: 4.c scrub starts 2024-02-05T07:37:03.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:02 smithi090 ceph-mon[33213]: 4.c scrub ok 2024-02-05T07:37:03.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:03.323 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:03.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:03.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:03.425 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:03.454 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:03.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:03.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:03.580 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:03.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:03.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:03.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:03.722 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:03.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:03.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:03.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:03.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:03.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:37:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:37:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:37:03.923 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:03.953 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:04.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:04.005 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:04.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:04.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:04.137 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:04.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:04.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:04.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:04.257 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:03 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:37:04.257 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:03 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:37:04.257 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:03 smithi007 ceph-mon[29098]: pgmap v997: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 1.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.19k op/s; 2786/70806 objects misplaced (3.935%) 2024-02-05T07:37:04.257 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:03 smithi007 ceph-mon[29098]: 3.9 scrub starts 2024-02-05T07:37:04.257 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:03 smithi007 ceph-mon[29098]: 3.9 scrub ok 2024-02-05T07:37:04.257 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:03 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:37:04.258 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:03 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:37:04.258 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:03 smithi007 ceph-mon[37158]: pgmap v997: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 1.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.19k op/s; 2786/70806 objects misplaced (3.935%) 2024-02-05T07:37:04.258 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:03 smithi007 ceph-mon[37158]: 3.9 scrub starts 2024-02-05T07:37:04.258 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:03 smithi007 ceph-mon[37158]: 3.9 scrub ok 2024-02-05T07:37:04.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:03 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:37:04.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:03 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:37:04.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:03 smithi090 ceph-mon[33213]: pgmap v997: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 164 active+clean; 1.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.19k op/s; 2786/70806 objects misplaced (3.935%) 2024-02-05T07:37:04.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:03 smithi090 ceph-mon[33213]: 3.9 scrub starts 2024-02-05T07:37:04.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:03 smithi090 ceph-mon[33213]: 3.9 scrub ok 2024-02-05T07:37:04.287 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:04.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:04.389 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:04.439 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:04.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:04.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:04.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:04.582 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:04.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:04.616 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pg_num to 21 2024-02-05T07:37:04.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:04.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:04.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:04.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:04.856 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:04.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:04.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:04.992 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:04.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:05.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:37:05.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:05.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:05.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:05.241 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:05.267 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:05.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:05.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:05.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:04 smithi090 ceph-mon[33213]: osdmap e553: 8 total, 8 up, 6 in 2024-02-05T07:37:05.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:04 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1888326667' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:37:05.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:04 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1888326667' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T07:37:05.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:04 smithi090 ceph-mon[33213]: osdmap e554: 8 total, 8 up, 6 in 2024-02-05T07:37:05.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:05.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:05.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:05.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:04 smithi007 ceph-mon[37158]: osdmap e553: 8 total, 8 up, 6 in 2024-02-05T07:37:05.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:04 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1888326667' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:37:05.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:04 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1888326667' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T07:37:05.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:04 smithi007 ceph-mon[37158]: osdmap e554: 8 total, 8 up, 6 in 2024-02-05T07:37:05.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:04 smithi007 ceph-mon[29098]: osdmap e553: 8 total, 8 up, 6 in 2024-02-05T07:37:05.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:04 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1888326667' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:37:05.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:04 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1888326667' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T07:37:05.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:04 smithi007 ceph-mon[29098]: osdmap e554: 8 total, 8 up, 6 in 2024-02-05T07:37:05.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:05.566 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:05.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:05.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:05.713 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:05.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:05.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:05.843 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:05.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:05.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:05.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:05.990 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:05.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:06.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:06.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:06.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:06.224 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:05 smithi007 ceph-mon[29098]: pgmap v1000: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.08k op/s; 2585/62055 objects misplaced (4.166%) 2024-02-05T07:37:06.224 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:05 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:37:06.224 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:05 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:37:06.224 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:05 smithi007 ceph-mon[37158]: pgmap v1000: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.08k op/s; 2585/62055 objects misplaced (4.166%) 2024-02-05T07:37:06.224 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:05 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:37:06.225 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:05 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:37:06.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:06.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:05 smithi090 ceph-mon[33213]: pgmap v1000: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.08k op/s; 2585/62055 objects misplaced (4.166%) 2024-02-05T07:37:06.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:05 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:37:06.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:05 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:37:06.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:06.316 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:06.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:06.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:06.446 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:06.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:06.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:06.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:06.588 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:06.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:06.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:06.743 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:06.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:06.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:06.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:06.892 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:06.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:06.993 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:07.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:07.038 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:07.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:07.139 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:07.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:07.192 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:07.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:07.215 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:07.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:06 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:37:07.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:06 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T07:37:07.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:06 smithi090 ceph-mon[33213]: osdmap e555: 8 total, 8 up, 6 in 2024-02-05T07:37:07.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:06 smithi090 ceph-mon[33213]: 4.15 scrub starts 2024-02-05T07:37:07.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:06 smithi090 ceph-mon[33213]: 4.15 scrub ok 2024-02-05T07:37:07.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:06 smithi090 ceph-mon[33213]: 4.1b scrub starts 2024-02-05T07:37:07.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:06 smithi090 ceph-mon[33213]: 4.1b scrub ok 2024-02-05T07:37:07.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:06 smithi090 ceph-mon[33213]: osdmap e556: 8 total, 8 up, 6 in 2024-02-05T07:37:07.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:07.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:37:07.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T07:37:07.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[29098]: osdmap e555: 8 total, 8 up, 6 in 2024-02-05T07:37:07.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[29098]: 4.15 scrub starts 2024-02-05T07:37:07.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[29098]: 4.15 scrub ok 2024-02-05T07:37:07.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[29098]: 4.1b scrub starts 2024-02-05T07:37:07.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[29098]: 4.1b scrub ok 2024-02-05T07:37:07.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[29098]: osdmap e556: 8 total, 8 up, 6 in 2024-02-05T07:37:07.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:37:07.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T07:37:07.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[37158]: osdmap e555: 8 total, 8 up, 6 in 2024-02-05T07:37:07.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[37158]: 4.15 scrub starts 2024-02-05T07:37:07.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[37158]: 4.15 scrub ok 2024-02-05T07:37:07.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[37158]: 4.1b scrub starts 2024-02-05T07:37:07.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[37158]: 4.1b scrub ok 2024-02-05T07:37:07.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:06 smithi007 ceph-mon[37158]: osdmap e556: 8 total, 8 up, 6 in 2024-02-05T07:37:07.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:07.330 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:07.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:07.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:07.464 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:07.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:07.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:07.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:07.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:07.760 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:07.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:07.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:07.866 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:07.918 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:07.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:07.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:08.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:08.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:08.064 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:08.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:08.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:07 smithi007 ceph-mon[29098]: pgmap v1003: 178 pgs: 1 active+remapped+backfill_toofull, 1 peering, 10 unknown, 1 active+remapped+backfilling, 165 active+clean; 1.1 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 2062/52479 objects misplaced (3.929%) 2024-02-05T07:37:08.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:07 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:37:08.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:07 smithi007 ceph-mon[29098]: 1.4 scrub starts 2024-02-05T07:37:08.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:07 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:37:08.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:07 smithi007 ceph-mon[29098]: 1.4 scrub ok 2024-02-05T07:37:08.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:07 smithi007 ceph-mon[29098]: 1.5 scrub starts 2024-02-05T07:37:08.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:07 smithi007 ceph-mon[29098]: 1.5 scrub ok 2024-02-05T07:37:08.113 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:07 smithi007 ceph-mon[37158]: pgmap v1003: 178 pgs: 1 active+remapped+backfill_toofull, 1 peering, 10 unknown, 1 active+remapped+backfilling, 165 active+clean; 1.1 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 2062/52479 objects misplaced (3.929%) 2024-02-05T07:37:08.113 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:07 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:37:08.113 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:07 smithi007 ceph-mon[37158]: 1.4 scrub starts 2024-02-05T07:37:08.113 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:07 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:37:08.113 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:07 smithi007 ceph-mon[37158]: 1.4 scrub ok 2024-02-05T07:37:08.113 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:07 smithi007 ceph-mon[37158]: 1.5 scrub starts 2024-02-05T07:37:08.113 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:07 smithi007 ceph-mon[37158]: 1.5 scrub ok 2024-02-05T07:37:08.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:08.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:08.205 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:08.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:07 smithi090 ceph-mon[33213]: pgmap v1003: 178 pgs: 1 active+remapped+backfill_toofull, 1 peering, 10 unknown, 1 active+remapped+backfilling, 165 active+clean; 1.1 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 2062/52479 objects misplaced (3.929%) 2024-02-05T07:37:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:07 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:37:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:07 smithi090 ceph-mon[33213]: 1.4 scrub starts 2024-02-05T07:37:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:07 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:37:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:07 smithi090 ceph-mon[33213]: 1.4 scrub ok 2024-02-05T07:37:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:07 smithi090 ceph-mon[33213]: 1.5 scrub starts 2024-02-05T07:37:08.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:07 smithi090 ceph-mon[33213]: 1.5 scrub ok 2024-02-05T07:37:08.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:08.335 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:08.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:08.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:08.463 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:08.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:08.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:08.590 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:08.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:08.630 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:37:08.630 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:37:08.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:08.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:08.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:08.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:08.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:08.839 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:08.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:08.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:08.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:08.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:08.965 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:08.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:08.989 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:37:09.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:09.098 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:09.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:09.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:09.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:09.239 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:09.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:08 smithi090 ceph-mon[33213]: 3.16 scrub starts 2024-02-05T07:37:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:08 smithi090 ceph-mon[33213]: 3.16 scrub ok 2024-02-05T07:37:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:08 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T07:37:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:08 smithi090 ceph-mon[33213]: osdmap e557: 8 total, 8 up, 6 in 2024-02-05T07:37:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:08 smithi090 ceph-mon[33213]: 1.1 scrub starts 2024-02-05T07:37:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:08 smithi090 ceph-mon[33213]: 1.1 scrub ok 2024-02-05T07:37:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:08 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3852850230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:37:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:08 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:37:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:08 smithi090 ceph-mon[33213]: 1.0 scrub starts 2024-02-05T07:37:09.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:08 smithi090 ceph-mon[33213]: 1.0 scrub ok 2024-02-05T07:37:09.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:08 smithi090 ceph-mon[33213]: 1.2 scrub starts 2024-02-05T07:37:09.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:08 smithi090 ceph-mon[33213]: 1.2 scrub ok 2024-02-05T07:37:09.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:09.364 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[29098]: 3.16 scrub starts 2024-02-05T07:37:09.364 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[29098]: 3.16 scrub ok 2024-02-05T07:37:09.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T07:37:09.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[29098]: osdmap e557: 8 total, 8 up, 6 in 2024-02-05T07:37:09.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[29098]: 1.1 scrub starts 2024-02-05T07:37:09.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[29098]: 1.1 scrub ok 2024-02-05T07:37:09.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3852850230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:37:09.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:37:09.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[29098]: 1.0 scrub starts 2024-02-05T07:37:09.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[29098]: 1.0 scrub ok 2024-02-05T07:37:09.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[29098]: 1.2 scrub starts 2024-02-05T07:37:09.367 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[29098]: 1.2 scrub ok 2024-02-05T07:37:09.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[37158]: 3.16 scrub starts 2024-02-05T07:37:09.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[37158]: 3.16 scrub ok 2024-02-05T07:37:09.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T07:37:09.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[37158]: osdmap e557: 8 total, 8 up, 6 in 2024-02-05T07:37:09.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[37158]: 1.1 scrub starts 2024-02-05T07:37:09.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[37158]: 1.1 scrub ok 2024-02-05T07:37:09.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3852850230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:37:09.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:37:09.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[37158]: 1.0 scrub starts 2024-02-05T07:37:09.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[37158]: 1.0 scrub ok 2024-02-05T07:37:09.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[37158]: 1.2 scrub starts 2024-02-05T07:37:09.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:08 smithi007 ceph-mon[37158]: 1.2 scrub ok 2024-02-05T07:37:09.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:09.373 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:09.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:09.474 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:09.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:09.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:09.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:09.663 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:09.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:09.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:09.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:09.792 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:09.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:09.893 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:09.924 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:09.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:10.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:10.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:10.056 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:10.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:10.157 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:10.185 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:10.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:10.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:10.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:09 smithi090 ceph-mon[33213]: 4.2 scrub starts 2024-02-05T07:37:10.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:09 smithi090 ceph-mon[33213]: 4.2 scrub ok 2024-02-05T07:37:10.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:09 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:37:10.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:09 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:37:10.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:09 smithi090 ceph-mon[33213]: osdmap e558: 8 total, 8 up, 6 in 2024-02-05T07:37:10.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:09 smithi090 ceph-mon[33213]: pgmap v1006: 178 pgs: 1 active+remapped+backfill_toofull, 2 peering, 10 unknown, 165 active+clean; 943 MiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.35k op/s; 1522/45822 objects misplaced (3.322%); 7.6 MiB/s, 121 objects/s recovering 2024-02-05T07:37:10.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:10.313 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:09 smithi007 ceph-mon[29098]: 4.2 scrub starts 2024-02-05T07:37:10.313 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:09 smithi007 ceph-mon[29098]: 4.2 scrub ok 2024-02-05T07:37:10.313 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:09 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:37:10.313 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:09 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:37:10.313 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:09 smithi007 ceph-mon[29098]: osdmap e558: 8 total, 8 up, 6 in 2024-02-05T07:37:10.313 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:09 smithi007 ceph-mon[29098]: pgmap v1006: 178 pgs: 1 active+remapped+backfill_toofull, 2 peering, 10 unknown, 165 active+clean; 943 MiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.35k op/s; 1522/45822 objects misplaced (3.322%); 7.6 MiB/s, 121 objects/s recovering 2024-02-05T07:37:10.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:09 smithi007 ceph-mon[37158]: 4.2 scrub starts 2024-02-05T07:37:10.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:09 smithi007 ceph-mon[37158]: 4.2 scrub ok 2024-02-05T07:37:10.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:09 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:37:10.314 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:09 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:37:10.314 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:09 smithi007 ceph-mon[37158]: osdmap e558: 8 total, 8 up, 6 in 2024-02-05T07:37:10.314 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:09 smithi007 ceph-mon[37158]: pgmap v1006: 178 pgs: 1 active+remapped+backfill_toofull, 2 peering, 10 unknown, 165 active+clean; 943 MiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.35k op/s; 1522/45822 objects misplaced (3.322%); 7.6 MiB/s, 121 objects/s recovering 2024-02-05T07:37:10.321 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:10.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:10.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:10.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:10.470 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:10.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:10.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:10.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:10.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:10.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:10.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:10.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:10.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:10.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:10.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:10.894 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:10.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:10.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T07:37:10.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:11.025 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:11.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:11.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:11.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:11.154 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:11.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:11.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:10 smithi007 ceph-mon[29098]: 1.d starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: 1.10 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: osdmap e559: 8 total, 8 up, 6 in 2024-02-05T07:37:11.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: 1.f starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: 1.c starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: 1.12 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: 1.11 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: 1.12 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: 1.11 starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: 1.e starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: 1.e starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: 1.13 starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: 1.b starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: 1.14 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.176 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: 1.14 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.176 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[29098]: 1.b starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.d starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.10 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: osdmap e559: 8 total, 8 up, 6 in 2024-02-05T07:37:11.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.f starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.c starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.12 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.11 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.12 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.11 starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.e starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.e starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.13 starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.b starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.14 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.14 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:11 smithi007 ceph-mon[37158]: 1.b starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:11.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:37:11.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:10 smithi090 ceph-mon[33213]: 1.d starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:10 smithi090 ceph-mon[33213]: 1.10 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-mon[33213]: osdmap e559: 8 total, 8 up, 6 in 2024-02-05T07:37:11.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-mon[33213]: 1.f starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-mon[33213]: 1.c starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-mon[33213]: 1.12 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-mon[33213]: 1.11 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-mon[33213]: 1.12 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-mon[33213]: 1.11 starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-mon[33213]: 1.e starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-mon[33213]: 1.e starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-mon[33213]: 1.13 starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-mon[33213]: 1.b starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-mon[33213]: 1.14 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-mon[33213]: 1.14 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.276 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:11 smithi090 ceph-mon[33213]: 1.b starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:11.280 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6, 3, 2] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:37:11.280 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:37:11.280 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-02-05T07:37:11.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:37:11.298 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:37:11.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:11.319 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:11.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:11.345 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:37:11.388 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:37:11.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:11.437 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:37:11.479 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:11.480 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:37:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:11.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:11.523 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:37:11.569 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:37:11.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:11.612 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:37:11.614 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:11.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:11.628 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:37:11.628 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:37:11.629 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:37:11.629 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:37:11.629 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:37:11.629 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:37:11.629 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:37:11.629 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:37:11.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:11.760 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:11.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:11.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:11.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:11.899 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:11.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:12.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:12.028 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:12.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:12.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:12.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:12.158 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:12.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:12.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:12 smithi007 ceph-mon[29098]: osdmap e560: 8 total, 8 up, 6 in 2024-02-05T07:37:12.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:12 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:12.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:12 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1592806340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:12.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:12 smithi007 ceph-mon[29098]: pgmap v1009: 178 pgs: 2 active+clean+remapped, 4 active+remapped+backfill_toofull, 2 peering, 5 unknown, 165 active+clean; 720 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 1.44k op/s; 901/35118 objects misplaced (2.566%); 7.6 MiB/s, 120 objects/s recovering 2024-02-05T07:37:12.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:12 smithi007 ceph-mon[37158]: osdmap e560: 8 total, 8 up, 6 in 2024-02-05T07:37:12.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:12 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:12.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:12 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1592806340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:12.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:12 smithi007 ceph-mon[37158]: pgmap v1009: 178 pgs: 2 active+clean+remapped, 4 active+remapped+backfill_toofull, 2 peering, 5 unknown, 165 active+clean; 720 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 1.44k op/s; 901/35118 objects misplaced (2.566%); 7.6 MiB/s, 120 objects/s recovering 2024-02-05T07:37:12.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:12.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:12 smithi090 ceph-mon[33213]: osdmap e560: 8 total, 8 up, 6 in 2024-02-05T07:37:12.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:12 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:12.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:12 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1592806340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:12.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:12 smithi090 ceph-mon[33213]: pgmap v1009: 178 pgs: 2 active+clean+remapped, 4 active+remapped+backfill_toofull, 2 peering, 5 unknown, 165 active+clean; 720 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 1.44k op/s; 901/35118 objects misplaced (2.566%); 7.6 MiB/s, 120 objects/s recovering 2024-02-05T07:37:12.297 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:12.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:12.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:12.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:12.431 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:12.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:12.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:12.598 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:12.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:12.599 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:12.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:12.730 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:12.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:12.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:12.876 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:12.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:12.920 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:12.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:13.032 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:13.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:13.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:13.164 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:13.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:13.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:13.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:13 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:37:13.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:13 smithi090 ceph-mon[33213]: osdmap e561: 8 total, 8 up, 6 in 2024-02-05T07:37:13.310 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:13.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:13.412 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:13.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:13 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:37:13.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:13 smithi007 ceph-mon[29098]: osdmap e561: 8 total, 8 up, 6 in 2024-02-05T07:37:13.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:13 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:37:13.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:13 smithi007 ceph-mon[37158]: osdmap e561: 8 total, 8 up, 6 in 2024-02-05T07:37:13.468 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:13.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:13.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:13.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:13.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:13.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:13.741 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:13.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:13.785 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:37:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:37:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:37:13.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:37:13.842 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:13.927 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:13.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:13.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:14.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:14.079 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:14.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:14.181 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:14.208 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:14.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:14.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:14.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:14.339 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:14.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:14.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:14.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:14 smithi007 ceph-mon[29098]: pgmap v1011: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 4 peering, 3 unknown, 166 active+clean; 709 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 857 op/s; 901/34605 objects misplaced (2.604%); 49 KiB/s, 0 objects/s recovering 2024-02-05T07:37:14.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:14 smithi007 ceph-mon[37158]: pgmap v1011: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 4 peering, 3 unknown, 166 active+clean; 709 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 857 op/s; 901/34605 objects misplaced (2.604%); 49 KiB/s, 0 objects/s recovering 2024-02-05T07:37:14.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:14.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:14.471 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:14.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:14.523 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:14 smithi090 ceph-mon[33213]: pgmap v1011: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 4 peering, 3 unknown, 166 active+clean; 709 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 857 op/s; 901/34605 objects misplaced (2.604%); 49 KiB/s, 0 objects/s recovering 2024-02-05T07:37:14.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:37:14.572 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:14.600 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:14.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:14.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:14.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:14.732 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:14.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:14.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:14.874 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:14.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:14.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:15.007 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:15.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:15.110 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:15.150 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:15.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:15.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:15.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:15.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:15.421 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:15.466 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:15.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:15.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:15.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:15.598 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:15.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:15.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:15.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:15.730 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:15.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:15.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:15.863 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:15.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:15 smithi007 ceph-mon[29098]: osdmap e562: 8 total, 8 up, 6 in 2024-02-05T07:37:15.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:15 smithi007 ceph-mon[37158]: osdmap e562: 8 total, 8 up, 6 in 2024-02-05T07:37:15.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:16.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:16.009 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:16.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:16.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:15 smithi090 ceph-mon[33213]: osdmap e562: 8 total, 8 up, 6 in 2024-02-05T07:37:16.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:16.140 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:16.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:16.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:16.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:16.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:16.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:16.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:16.404 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:16.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:16.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:16.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:16.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:16.538 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:16.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:16.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:16.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:16.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:16.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:16.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:16.827 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:16.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:16 smithi007 ceph-mon[29098]: pgmap v1013: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 3 peering, 1 unknown, 169 active+clean; 516 MiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 1.27k op/s; 901/25323 objects misplaced (3.558%); 40 KiB/s, 0 objects/s recovering 2024-02-05T07:37:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:16 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:37:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:16 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:37:16.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:16 smithi007 ceph-mon[37158]: pgmap v1013: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 3 peering, 1 unknown, 169 active+clean; 516 MiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 1.27k op/s; 901/25323 objects misplaced (3.558%); 40 KiB/s, 0 objects/s recovering 2024-02-05T07:37:16.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:16 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:37:16.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:16 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:37:16.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:16.975 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:16.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:16.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:17.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:16 smithi090 ceph-mon[33213]: pgmap v1013: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 3 peering, 1 unknown, 169 active+clean; 516 MiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 1.27k op/s; 901/25323 objects misplaced (3.558%); 40 KiB/s, 0 objects/s recovering 2024-02-05T07:37:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:16 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:37:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:16 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:37:17.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:17.113 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:17.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:17.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:17.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:17.246 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:17.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:17.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:17.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:17.379 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:17.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:17.481 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:17.525 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:17.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:17.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:17.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:17.693 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:17.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:17.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:17.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:17.825 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:17.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:17.862 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:37:17.863 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-02-05T07:37:17.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-02-05T07:37:17.863 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:37:17.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.863 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:190462 10.15 e527/524 10:abd78c7a:::benchmark_data_smithi007_157098_object61532:head v 527'9875, mlcod=526'9873)", 2024-02-05T07:37:17.863 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:19.777482+0000", 2024-02-05T07:37:17.863 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 57.368504452000003, 2024-02-05T07:37:17.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.43455628000000002, 2024-02-05T07:37:17.863 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.863 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.863 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777482+0000", 2024-02-05T07:37:17.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777482+0000", 2024-02-05T07:37:17.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777483+0000", 2024-02-05T07:37:17.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2249999999999999e-06 2024-02-05T07:37:17.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.865 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777498+0000", 2024-02-05T07:37:17.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5310000000000001e-05 2024-02-05T07:37:17.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.865 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777510+0000", 2024-02-05T07:37:17.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1960999999999999e-05 2024-02-05T07:37:17.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777519+0000", 2024-02-05T07:37:17.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7809999999999994e-06 2024-02-05T07:37:17.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777849+0000", 2024-02-05T07:37:17.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032970900000000002 2024-02-05T07:37:17.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777871+0000", 2024-02-05T07:37:17.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2288000000000001e-05 2024-02-05T07:37:17.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:20.211883+0000", 2024-02-05T07:37:17.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.43401152500000001 2024-02-05T07:37:17.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:20.212038+0000", 2024-02-05T07:37:17.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015548100000000001 2024-02-05T07:37:17.867 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.868 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.868 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.868 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:190484 10.15 e527/524 10:a84f9610:::benchmark_data_smithi007_157098_object61554:head v 527'9876, mlcod=526'9873)", 2024-02-05T07:37:17.868 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:19.822798+0000", 2024-02-05T07:37:17.868 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 57.323188533, 2024-02-05T07:37:17.868 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.38933499199999999, 2024-02-05T07:37:17.868 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.868 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.868 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.868 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.822798+0000", 2024-02-05T07:37:17.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.822798+0000", 2024-02-05T07:37:17.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.822799+0000", 2024-02-05T07:37:17.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.649e-06 2024-02-05T07:37:17.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.870 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.822809+0000", 2024-02-05T07:37:17.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.414e-06 2024-02-05T07:37:17.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.870 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.822829+0000", 2024-02-05T07:37:17.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0523999999999999e-05 2024-02-05T07:37:17.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.822841+0000", 2024-02-05T07:37:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1194e-05 2024-02-05T07:37:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.822934+0000", 2024-02-05T07:37:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3120000000000003e-05 2024-02-05T07:37:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.822958+0000", 2024-02-05T07:37:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4488999999999999e-05 2024-02-05T07:37:17.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:20.212048+0000", 2024-02-05T07:37:17.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.38908970999999998 2024-02-05T07:37:17.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:20.212133+0000", 2024-02-05T07:37:17.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4892e-05 2024-02-05T07:37:17.872 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.872 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.872 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.873 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:190489 10.15 e527/524 10:aab5776f:::benchmark_data_smithi007_157098_object61559:head v 527'9877, mlcod=526'9873)", 2024-02-05T07:37:17.873 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:19.843163+0000", 2024-02-05T07:37:17.873 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 57.302823519, 2024-02-05T07:37:17.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.36900828099999999, 2024-02-05T07:37:17.873 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.873 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.873 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.843163+0000", 2024-02-05T07:37:17.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.874 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.843163+0000", 2024-02-05T07:37:17.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.874 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.874 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.843164+0000", 2024-02-05T07:37:17.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8100000000000001e-07 2024-02-05T07:37:17.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.874 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.874 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.843173+0000", 2024-02-05T07:37:17.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.969e-06 2024-02-05T07:37:17.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.875 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.875 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.843192+0000", 2024-02-05T07:37:17.875 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9304e-05 2024-02-05T07:37:17.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.875 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.875 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.843202+0000", 2024-02-05T07:37:17.875 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0217000000000001e-05 2024-02-05T07:37:17.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.875 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.875 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.843285+0000", 2024-02-05T07:37:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2956999999999999e-05 2024-02-05T07:37:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.843310+0000", 2024-02-05T07:37:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4649000000000001e-05 2024-02-05T07:37:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:20.212147+0000", 2024-02-05T07:37:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.36883744000000002 2024-02-05T07:37:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:20.212171+0000", 2024-02-05T07:37:17.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3764000000000002e-05 2024-02-05T07:37:17.877 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.877 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.877 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.877 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:190501 10.5 e527/524 10:a5d1d51a:::benchmark_data_smithi007_157098_object61571:head v 527'9769, mlcod=526'9767)", 2024-02-05T07:37:17.877 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:19.891889+0000", 2024-02-05T07:37:17.877 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 57.254097823999999, 2024-02-05T07:37:17.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.320414007, 2024-02-05T07:37:17.877 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.877 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.877 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.891889+0000", 2024-02-05T07:37:17.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.891889+0000", 2024-02-05T07:37:17.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.891890+0000", 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0640000000000001e-06 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.891916+0000", 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6035000000000001e-05 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.891945+0000", 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9577000000000002e-05 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.891956+0000", 2024-02-05T07:37:17.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1049e-05 2024-02-05T07:37:17.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.880 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.880 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.892055+0000", 2024-02-05T07:37:17.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8801999999999994e-05 2024-02-05T07:37:17.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.880 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.880 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.892082+0000", 2024-02-05T07:37:17.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6676999999999999e-05 2024-02-05T07:37:17.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.881 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.881 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:20.212181+0000", 2024-02-05T07:37:17.881 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.32009888399999997 2024-02-05T07:37:17.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.881 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.881 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:20.212303+0000", 2024-02-05T07:37:17.881 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121919 2024-02-05T07:37:17.881 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.881 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.881 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.881 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217805 10.18 10:1ea1ad8a:::benchmark_data_smithi007_157098_object88875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.882 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:45.879289+0000", 2024-02-05T07:37:17.882 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.266697439000001, 2024-02-05T07:37:17.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96299109100000002, 2024-02-05T07:37:17.882 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.882 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.882 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.882 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.882 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.882 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217805 2024-02-05T07:37:17.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.882 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879289+0000", 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879289+0000", 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879290+0000", 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3510000000000001e-06 2024-02-05T07:37:17.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879295+0000", 2024-02-05T07:37:17.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6920000000000001e-06 2024-02-05T07:37:17.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879318+0000", 2024-02-05T07:37:17.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3315999999999998e-05 2024-02-05T07:37:17.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879327+0000", 2024-02-05T07:37:17.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8149999999999995e-06 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879380+0000", 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3075999999999998e-05 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879399+0000", 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8719000000000002e-05 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.835865+0000", 2024-02-05T07:37:17.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.956466339 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.835977+0000", 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011222000000000001 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.836346+0000", 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00036861100000000003 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.840534+0000", 2024-02-05T07:37:17.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041881720000000004 2024-02-05T07:37:17.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.887 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.842031+0000", 2024-02-05T07:37:17.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001497246 2024-02-05T07:37:17.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.887 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.842205+0000", 2024-02-05T07:37:17.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000173382 2024-02-05T07:37:17.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.842254+0000", 2024-02-05T07:37:17.888 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9375000000000001e-05 2024-02-05T07:37:17.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.842280+0000", 2024-02-05T07:37:17.888 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5777000000000001e-05 2024-02-05T07:37:17.888 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.888 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.888 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217840 10.18 10:19c295f5:::benchmark_data_smithi007_157098_object88910:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:45.903257+0000", 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.242729257000001, 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.94821095200000005, 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217840 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903257+0000", 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903257+0000", 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903258+0000", 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4600000000000003e-07 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903263+0000", 2024-02-05T07:37:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9490000000000004e-06 2024-02-05T07:37:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903274+0000", 2024-02-05T07:37:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1116e-05 2024-02-05T07:37:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903282+0000", 2024-02-05T07:37:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1480000000000006e-06 2024-02-05T07:37:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903364+0000", 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2136000000000001e-05 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903373+0000", 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3950000000000001e-06 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.838826+0000", 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.93545278899999995 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.838978+0000", 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000152325 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.839306+0000", 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032797799999999998 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.844005+0000", 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0046995270000000002 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.847770+0000", 2024-02-05T07:37:17.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003764117 2024-02-05T07:37:17.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851395+0000", 2024-02-05T07:37:17.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0036254 2024-02-05T07:37:17.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851439+0000", 2024-02-05T07:37:17.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4477000000000001e-05 2024-02-05T07:37:17.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851468+0000", 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8649e-05 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217841 10.18 10:1fd606f1:::benchmark_data_smithi007_157098_object88911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:45.905042+0000", 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.240944636999998, 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.946656622, 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.895 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.896 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.896 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.896 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217841 2024-02-05T07:37:17.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.896 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.896 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905042+0000", 2024-02-05T07:37:17.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.896 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905042+0000", 2024-02-05T07:37:17.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905043+0000", 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2720000000000001e-06 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905048+0000", 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7419999999999997e-06 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905059+0000", 2024-02-05T07:37:17.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1039000000000001e-05 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905066+0000", 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7309999999999993e-06 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905159+0000", 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2286000000000004e-05 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905167+0000", 2024-02-05T07:37:17.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7290000000000006e-06 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.840574+0000", 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.93540616200000004 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.840650+0000", 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6518000000000002e-05 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.840854+0000", 2024-02-05T07:37:17.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020390100000000001 2024-02-05T07:37:17.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.844013+0000", 2024-02-05T07:37:17.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0031590780000000001 2024-02-05T07:37:17.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.847836+0000", 2024-02-05T07:37:17.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038230949999999999 2024-02-05T07:37:17.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851594+0000", 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037579369999999998 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851656+0000", 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2031000000000003e-05 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851698+0000", 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2101000000000002e-05 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217870 10.18 10:1ed08b88:::benchmark_data_smithi007_157098_object88940:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:45.960539+0000", 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.18544717, 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.89133640199999997, 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217870 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.902 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960539+0000", 2024-02-05T07:37:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960539+0000", 2024-02-05T07:37:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960541+0000", 2024-02-05T07:37:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4869999999999999e-06 2024-02-05T07:37:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960545+0000", 2024-02-05T07:37:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8359999999999999e-06 2024-02-05T07:37:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960557+0000", 2024-02-05T07:37:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1575e-05 2024-02-05T07:37:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960565+0000", 2024-02-05T07:37:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0819999999999999e-06 2024-02-05T07:37:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960624+0000", 2024-02-05T07:37:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9039000000000002e-05 2024-02-05T07:37:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960634+0000", 2024-02-05T07:37:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8770000000000005e-06 2024-02-05T07:37:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.842355+0000", 2024-02-05T07:37:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.88172097500000002 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.842453+0000", 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8101000000000001e-05 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.842665+0000", 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00021142200000000001 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.847852+0000", 2024-02-05T07:37:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0051875929999999999 2024-02-05T07:37:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.847885+0000", 2024-02-05T07:37:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2909e-05 2024-02-05T07:37:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851777+0000", 2024-02-05T07:37:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038922800000000001 2024-02-05T07:37:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851835+0000", 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7788999999999998e-05 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851876+0000", 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0436999999999999e-05 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217871 10.18 10:1967d2fc:::benchmark_data_smithi007_157098_object88941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:45.960911+0000", 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.185075701999999, 2024-02-05T07:37:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.89109762400000003, 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217871 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960911+0000", 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960911+0000", 2024-02-05T07:37:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960912+0000", 2024-02-05T07:37:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7499999999999999e-07 2024-02-05T07:37:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960916+0000", 2024-02-05T07:37:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5630000000000002e-06 2024-02-05T07:37:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960925+0000", 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2869999999999993e-06 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960933+0000", 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.695e-06 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.961021+0000", 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7545000000000003e-05 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.961029+0000", 2024-02-05T07:37:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9079999999999995e-06 2024-02-05T07:37:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.844064+0000", 2024-02-05T07:37:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.883035668 2024-02-05T07:37:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.844182+0000", 2024-02-05T07:37:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011744900000000001 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.844465+0000", 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028307199999999999 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.847860+0000", 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033948450000000001 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851335+0000", 2024-02-05T07:37:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003475634 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851935+0000", 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000599831 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851983+0000", 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8065000000000001e-05 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852008+0000", 2024-02-05T07:37:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5187000000000001e-05 2024-02-05T07:37:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217905 10.18 10:1b875e8f:::benchmark_data_smithi007_157098_object88975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:45.983757+0000", 2024-02-05T07:37:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.162228892000002, 2024-02-05T07:37:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.86843629099999997, 2024-02-05T07:37:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.916 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217905 2024-02-05T07:37:17.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.916 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.916 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.916 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.916 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983757+0000", 2024-02-05T07:37:17.916 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.916 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.916 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.916 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983757+0000", 2024-02-05T07:37:17.916 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983759+0000", 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1489999999999999e-06 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983762+0000", 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.428e-06 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983770+0000", 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4930000000000001e-06 2024-02-05T07:37:17.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983777+0000", 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9979999999999997e-06 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983837+0000", 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0634000000000003e-05 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983846+0000", 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0499999999999997e-06 2024-02-05T07:37:17.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.846009+0000", 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.86216245199999997 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.846089+0000", 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0739000000000001e-05 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.846345+0000", 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00025539399999999999 2024-02-05T07:37:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.920 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.920 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.920 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.849745+0000", 2024-02-05T07:37:17.920 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0034002979999999999 2024-02-05T07:37:17.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.920 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.920 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.920 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851533+0000", 2024-02-05T07:37:17.920 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001787935 2024-02-05T07:37:17.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.920 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.920 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.920 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852120+0000", 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00058712100000000002 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852157+0000", 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6579000000000003e-05 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852194+0000", 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7020999999999997e-05 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.922 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217931 10.18 10:1ecf49c3:::benchmark_data_smithi007_157098_object89001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.922 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:45.998689+0000", 2024-02-05T07:37:17.922 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.147296932, 2024-02-05T07:37:17.922 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.85374700999999997, 2024-02-05T07:37:17.922 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.922 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.922 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.922 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.922 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.922 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217931 2024-02-05T07:37:17.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.922 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.923 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.923 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.998689+0000", 2024-02-05T07:37:17.923 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.923 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.923 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.923 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.998689+0000", 2024-02-05T07:37:17.923 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.923 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.923 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.923 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.998690+0000", 2024-02-05T07:37:17.923 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8700000000000004e-07 2024-02-05T07:37:17.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.924 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.924 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.998694+0000", 2024-02-05T07:37:17.924 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6009999999999999e-06 2024-02-05T07:37:17.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.924 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.924 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.998704+0000", 2024-02-05T07:37:17.924 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0013000000000001e-05 2024-02-05T07:37:17.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.924 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.924 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.998711+0000", 2024-02-05T07:37:17.925 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3130000000000001e-06 2024-02-05T07:37:17.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.925 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.925 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.999722+0000", 2024-02-05T07:37:17.925 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0010110830000000001 2024-02-05T07:37:17.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.925 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.925 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.999731+0000", 2024-02-05T07:37:17.925 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8759999999999994e-06 2024-02-05T07:37:17.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.925 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.926 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.847942+0000", 2024-02-05T07:37:17.926 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.84821022800000001 2024-02-05T07:37:17.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.926 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.926 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.926 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.848067+0000", 2024-02-05T07:37:17.926 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000125079 2024-02-05T07:37:17.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.926 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.926 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.926 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.848370+0000", 2024-02-05T07:37:17.926 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030292200000000001 2024-02-05T07:37:17.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.927 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.927 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851552+0000", 2024-02-05T07:37:17.927 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0031822859999999999 2024-02-05T07:37:17.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.927 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.927 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852061+0000", 2024-02-05T07:37:17.927 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00050924800000000001 2024-02-05T07:37:17.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.927 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.927 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852347+0000", 2024-02-05T07:37:17.927 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028602899999999998 2024-02-05T07:37:17.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.928 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.928 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.928 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852400+0000", 2024-02-05T07:37:17.928 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2794999999999999e-05 2024-02-05T07:37:17.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.928 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.928 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.928 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852436+0000", 2024-02-05T07:37:17.928 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6550000000000001e-05 2024-02-05T07:37:17.928 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.928 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.928 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.929 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.929 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217938 10.18 10:1d8d2565:::benchmark_data_smithi007_157098_object89008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.929 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.003702+0000", 2024-02-05T07:37:17.929 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.142284411999999, 2024-02-05T07:37:17.929 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.85442914000000003, 2024-02-05T07:37:17.929 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.929 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.929 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.929 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.929 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.929 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217938 2024-02-05T07:37:17.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.930 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.930 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.930 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.930 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003702+0000", 2024-02-05T07:37:17.930 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.930 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.930 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.930 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003702+0000", 2024-02-05T07:37:17.930 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.930 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.930 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.930 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003718+0000", 2024-02-05T07:37:17.931 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5654e-05 2024-02-05T07:37:17.931 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.931 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.931 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.931 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003724+0000", 2024-02-05T07:37:17.931 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2360000000000001e-06 2024-02-05T07:37:17.931 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.931 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.931 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.931 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003736+0000", 2024-02-05T07:37:17.931 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.221e-05 2024-02-05T07:37:17.931 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.931 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.932 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.932 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003745+0000", 2024-02-05T07:37:17.932 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9139999999999997e-06 2024-02-05T07:37:17.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.932 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.932 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003812+0000", 2024-02-05T07:37:17.932 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6923999999999998e-05 2024-02-05T07:37:17.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.932 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.932 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003821+0000", 2024-02-05T07:37:17.932 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6229999999999992e-06 2024-02-05T07:37:17.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.933 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.933 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.849791+0000", 2024-02-05T07:37:17.933 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.84596975200000002 2024-02-05T07:37:17.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.933 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.933 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.849887+0000", 2024-02-05T07:37:17.933 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5978999999999997e-05 2024-02-05T07:37:17.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.933 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.933 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.850108+0000", 2024-02-05T07:37:17.933 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00022115700000000001 2024-02-05T07:37:17.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.934 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851713+0000", 2024-02-05T07:37:17.934 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001604926 2024-02-05T07:37:17.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.934 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852282+0000", 2024-02-05T07:37:17.934 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00056861800000000001 2024-02-05T07:37:17.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.934 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.858030+0000", 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0057477659999999996 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.858088+0000", 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8162000000000003e-05 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.858131+0000", 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3219000000000001e-05 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.936 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217953 10.18 10:1831d604:::benchmark_data_smithi007_157098_object89023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.936 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.012541+0000", 2024-02-05T07:37:17.936 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.13344485, 2024-02-05T07:37:17.936 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.85552399199999996, 2024-02-05T07:37:17.936 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.936 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.936 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.936 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.936 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.936 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217953 2024-02-05T07:37:17.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.936 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012541+0000", 2024-02-05T07:37:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012541+0000", 2024-02-05T07:37:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012543+0000", 2024-02-05T07:37:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0189999999999999e-06 2024-02-05T07:37:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012546+0000", 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8099999999999999e-06 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012555+0000", 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0480000000000004e-06 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012562+0000", 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4749999999999998e-06 2024-02-05T07:37:17.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.939 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.939 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012615+0000", 2024-02-05T07:37:17.939 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3467999999999997e-05 2024-02-05T07:37:17.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.939 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.939 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012625+0000", 2024-02-05T07:37:17.939 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0131e-05 2024-02-05T07:37:17.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.939 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.939 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852479+0000", 2024-02-05T07:37:17.939 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.83985321499999999 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852593+0000", 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000114287 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852937+0000", 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00034402199999999998 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.857954+0000", 2024-02-05T07:37:17.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0050172649999999999 2024-02-05T07:37:17.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.941 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.867855+0000", 2024-02-05T07:37:17.941 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0099010669999999995 2024-02-05T07:37:17.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.941 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.867964+0000", 2024-02-05T07:37:17.941 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010865 2024-02-05T07:37:17.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.941 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868022+0000", 2024-02-05T07:37:17.942 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8421000000000002e-05 2024-02-05T07:37:17.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.942 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.942 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.942 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868065+0000", 2024-02-05T07:37:17.942 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3114000000000001e-05 2024-02-05T07:37:17.942 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.942 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.942 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.942 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.942 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217954 10.18 10:190cb164:::benchmark_data_smithi007_157098_object89024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.942 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.012763+0000", 2024-02-05T07:37:17.942 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.133222994, 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.85552692900000005, 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217954 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012763+0000", 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.943 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.944 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.944 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012763+0000", 2024-02-05T07:37:17.944 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.944 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.944 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.944 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012764+0000", 2024-02-05T07:37:17.944 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1299999999999999e-07 2024-02-05T07:37:17.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.944 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.944 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.944 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012768+0000", 2024-02-05T07:37:17.944 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1479999999999999e-06 2024-02-05T07:37:17.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.945 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.945 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012778+0000", 2024-02-05T07:37:17.945 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7399999999999999e-06 2024-02-05T07:37:17.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.945 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.945 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012786+0000", 2024-02-05T07:37:17.945 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8399999999999995e-06 2024-02-05T07:37:17.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.945 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.945 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012836+0000", 2024-02-05T07:37:17.945 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0311e-05 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012844+0000", 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.678e-06 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.855466+0000", 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.84262185000000001 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.855597+0000", 2024-02-05T07:37:17.946 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013140600000000001 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.855985+0000", 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00038769599999999998 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.867870+0000", 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011885578000000001 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868141+0000", 2024-02-05T07:37:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00027049099999999998 2024-02-05T07:37:17.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.948 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.948 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868205+0000", 2024-02-05T07:37:17.948 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3759999999999999e-05 2024-02-05T07:37:17.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.948 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.948 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868254+0000", 2024-02-05T07:37:17.948 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9206999999999998e-05 2024-02-05T07:37:17.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.948 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.948 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868290+0000", 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6411000000000001e-05 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217964 10.18 10:1bd88409:::benchmark_data_smithi007_157098_object89034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.019472+0000", 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.126514333999999, 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.84900089400000001, 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.950 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217964 2024-02-05T07:37:17.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.950 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.950 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.950 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.950 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019472+0000", 2024-02-05T07:37:17.950 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.950 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.950 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.950 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019472+0000", 2024-02-05T07:37:17.950 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.950 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.951 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.951 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019473+0000", 2024-02-05T07:37:17.951 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.248e-06 2024-02-05T07:37:17.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.951 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.951 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.951 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019478+0000", 2024-02-05T07:37:17.951 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5360000000000003e-06 2024-02-05T07:37:17.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.951 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.951 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.951 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019487+0000", 2024-02-05T07:37:17.951 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4180000000000003e-06 2024-02-05T07:37:17.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.952 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.952 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.952 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019493+0000", 2024-02-05T07:37:17.952 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0599999999999996e-06 2024-02-05T07:37:17.952 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.952 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.952 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.952 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019544+0000", 2024-02-05T07:37:17.952 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0637000000000003e-05 2024-02-05T07:37:17.952 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.952 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.952 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.952 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019554+0000", 2024-02-05T07:37:17.952 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0282000000000001e-05 2024-02-05T07:37:17.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.953 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.953 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.858164+0000", 2024-02-05T07:37:17.953 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.83860974300000002 2024-02-05T07:37:17.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.953 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.953 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.858280+0000", 2024-02-05T07:37:17.953 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011600899999999999 2024-02-05T07:37:17.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.953 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.953 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.858648+0000", 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000368296 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.867876+0000", 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0092275440000000007 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868334+0000", 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00045833199999999999 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.954 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868378+0000", 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3856000000000002e-05 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868437+0000", 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9039999999999997e-05 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868473+0000", 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5892999999999997e-05 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.956 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.956 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217998 10.18 10:1a72f410:::benchmark_data_smithi007_157098_object89068:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.956 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.052088+0000", 2024-02-05T07:37:17.956 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.09389784, 2024-02-05T07:37:17.956 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.82240571100000004, 2024-02-05T07:37:17.956 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.956 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.956 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.956 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.956 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.956 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217998 2024-02-05T07:37:17.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.956 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.956 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.957 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.957 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052088+0000", 2024-02-05T07:37:17.957 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.957 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.957 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.957 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052088+0000", 2024-02-05T07:37:17.957 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.957 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.957 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.957 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052090+0000", 2024-02-05T07:37:17.957 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.198e-06 2024-02-05T07:37:17.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.958 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.958 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.958 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052095+0000", 2024-02-05T07:37:17.958 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.079e-06 2024-02-05T07:37:17.958 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.958 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.958 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.958 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052109+0000", 2024-02-05T07:37:17.958 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4113e-05 2024-02-05T07:37:17.958 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.958 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.958 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.958 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052118+0000", 2024-02-05T07:37:17.959 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0990000000000005e-06 2024-02-05T07:37:17.959 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.959 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.960 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.960 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052249+0000", 2024-02-05T07:37:17.960 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013130600000000001 2024-02-05T07:37:17.960 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.960 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.960 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.960 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052259+0000", 2024-02-05T07:37:17.960 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2189999999999993e-06 2024-02-05T07:37:17.960 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.960 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.960 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.961 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.861087+0000", 2024-02-05T07:37:17.961 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.80882802600000003 2024-02-05T07:37:17.961 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.961 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.961 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.961 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.861213+0000", 2024-02-05T07:37:17.961 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012615400000000001 2024-02-05T07:37:17.961 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.961 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.961 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.961 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.861671+0000", 2024-02-05T07:37:17.961 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00045862600000000001 2024-02-05T07:37:17.961 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.961 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868525+0000", 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0068532360000000004 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868566+0000", 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1046e-05 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.874398+0000", 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.005832556 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.963 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.963 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.874449+0000", 2024-02-05T07:37:17.963 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1103000000000002e-05 2024-02-05T07:37:17.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.963 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.963 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.963 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.874494+0000", 2024-02-05T07:37:17.963 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4950000000000002e-05 2024-02-05T07:37:17.963 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.963 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.963 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.963 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.963 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:218069 10.18 10:1d5b1d6a:::benchmark_data_smithi007_157098_object89139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.964 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.127842+0000", 2024-02-05T07:37:17.964 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.018143941999998, 2024-02-05T07:37:17.964 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.75143445600000003, 2024-02-05T07:37:17.964 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.964 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.964 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.964 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.964 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.964 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 218069 2024-02-05T07:37:17.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.964 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.964 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.964 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.964 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127842+0000", 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127842+0000", 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127844+0000", 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2410000000000001e-06 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.965 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127849+0000", 2024-02-05T07:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4380000000000003e-06 2024-02-05T07:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127859+0000", 2024-02-05T07:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0084e-05 2024-02-05T07:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127867+0000", 2024-02-05T07:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2849999999999995e-06 2024-02-05T07:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127948+0000", 2024-02-05T07:37:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0420999999999995e-05 2024-02-05T07:37:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127958+0000", 2024-02-05T07:37:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.049e-05 2024-02-05T07:37:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.865082+0000", 2024-02-05T07:37:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.73712396099999999 2024-02-05T07:37:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.968 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.968 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.865207+0000", 2024-02-05T07:37:17.968 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000124919 2024-02-05T07:37:17.968 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.968 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.968 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.865760+0000", 2024-02-05T07:37:17.968 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00055247599999999999 2024-02-05T07:37:17.968 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.968 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.968 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879087+0000", 2024-02-05T07:37:17.969 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013327105000000001 2024-02-05T07:37:17.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.969 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.969 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879137+0000", 2024-02-05T07:37:17.969 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9985999999999999e-05 2024-02-05T07:37:17.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.969 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.969 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879185+0000", 2024-02-05T07:37:17.969 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8028000000000002e-05 2024-02-05T07:37:17.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879236+0000", 2024-02-05T07:37:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1190000000000003e-05 2024-02-05T07:37:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879277+0000", 2024-02-05T07:37:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0831999999999998e-05 2024-02-05T07:37:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.970 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:218139 10.18 10:1cee83cf:::benchmark_data_smithi007_157098_object89209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.971 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.208744+0000", 2024-02-05T07:37:17.971 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 30.937241994000001, 2024-02-05T07:37:17.971 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.67071878399999996, 2024-02-05T07:37:17.971 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.971 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.971 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.971 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.971 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.971 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 218139 2024-02-05T07:37:17.971 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.971 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.971 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.971 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.971 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208744+0000", 2024-02-05T07:37:17.972 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.972 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.972 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.972 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.972 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208744+0000", 2024-02-05T07:37:17.972 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.972 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.972 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.972 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.972 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208746+0000", 2024-02-05T07:37:17.972 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1710000000000001e-06 2024-02-05T07:37:17.972 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.972 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.972 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208750+0000", 2024-02-05T07:37:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4619999999999998e-06 2024-02-05T07:37:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208762+0000", 2024-02-05T07:37:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1678999999999999e-05 2024-02-05T07:37:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208768+0000", 2024-02-05T07:37:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.494e-06 2024-02-05T07:37:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.974 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.974 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208814+0000", 2024-02-05T07:37:17.974 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5813000000000003e-05 2024-02-05T07:37:17.974 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.974 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.974 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.974 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208823+0000", 2024-02-05T07:37:17.974 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8659999999999995e-06 2024-02-05T07:37:17.974 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.974 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.974 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.974 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868596+0000", 2024-02-05T07:37:17.974 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.65977350499999998 2024-02-05T07:37:17.974 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.975 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.975 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868709+0000", 2024-02-05T07:37:17.975 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011272700000000001 2024-02-05T07:37:17.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.975 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.975 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.869138+0000", 2024-02-05T07:37:17.975 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00042847799999999999 2024-02-05T07:37:17.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.976 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.976 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879094+0000", 2024-02-05T07:37:17.976 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0099561029999999991 2024-02-05T07:37:17.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.976 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.976 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.976 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879335+0000", 2024-02-05T07:37:17.976 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00024086100000000001 2024-02-05T07:37:17.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.976 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.976 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.976 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879380+0000", 2024-02-05T07:37:17.976 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5540000000000001e-05 2024-02-05T07:37:17.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.977 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.977 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879426+0000", 2024-02-05T07:37:17.977 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6338999999999998e-05 2024-02-05T07:37:17.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.977 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.977 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879463+0000", 2024-02-05T07:37:17.977 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6745999999999997e-05 2024-02-05T07:37:17.977 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.977 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.977 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.978 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:218157 10.18 10:198d6fce:::benchmark_data_smithi007_157098_object89227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.978 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.234341+0000", 2024-02-05T07:37:17.978 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 30.911645418999999, 2024-02-05T07:37:17.978 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.64906166700000001, 2024-02-05T07:37:17.978 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.978 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.978 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.978 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.978 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.978 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 218157 2024-02-05T07:37:17.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.978 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.978 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.978 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234341+0000", 2024-02-05T07:37:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234341+0000", 2024-02-05T07:37:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234342+0000", 2024-02-05T07:37:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3200000000000004e-07 2024-02-05T07:37:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.980 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.980 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234345+0000", 2024-02-05T07:37:17.980 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.63e-06 2024-02-05T07:37:17.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.980 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.980 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234356+0000", 2024-02-05T07:37:17.980 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0413e-05 2024-02-05T07:37:17.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.980 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.980 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234362+0000", 2024-02-05T07:37:17.980 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8189999999999997e-06 2024-02-05T07:37:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234412+0000", 2024-02-05T07:37:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0738000000000001e-05 2024-02-05T07:37:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234422+0000", 2024-02-05T07:37:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3230000000000004e-06 2024-02-05T07:37:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.871037+0000", 2024-02-05T07:37:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.63661556100000005 2024-02-05T07:37:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.871162+0000", 2024-02-05T07:37:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012524300000000001 2024-02-05T07:37:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.872236+0000", 2024-02-05T07:37:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0010734119999999999 2024-02-05T07:37:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879149+0000", 2024-02-05T07:37:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0069134670000000004 2024-02-05T07:37:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879505+0000", 2024-02-05T07:37:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00035600399999999999 2024-02-05T07:37:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.883309+0000", 2024-02-05T07:37:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038041389999999998 2024-02-05T07:37:17.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.883361+0000", 2024-02-05T07:37:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1496000000000003e-05 2024-02-05T07:37:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.883403+0000", 2024-02-05T07:37:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1690000000000002e-05 2024-02-05T07:37:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.985 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:218164 10.18 10:1a060940:::benchmark_data_smithi007_157098_object89234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:37:17.985 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.251100+0000", 2024-02-05T07:37:17.985 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 30.894886602, 2024-02-05T07:37:17.985 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.63564669699999998, 2024-02-05T07:37:17.985 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:17.985 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 218164 2024-02-05T07:37:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251100+0000", 2024-02-05T07:37:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251100+0000", 2024-02-05T07:37:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251101+0000", 2024-02-05T07:37:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3179999999999999e-06 2024-02-05T07:37:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:17.987 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251105+0000", 2024-02-05T07:37:17.988 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3699999999999997e-06 2024-02-05T07:37:17.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.988 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:17.988 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251116+0000", 2024-02-05T07:37:17.988 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0961999999999999e-05 2024-02-05T07:37:17.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.988 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:17.988 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251123+0000", 2024-02-05T07:37:17.988 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5200000000000003e-06 2024-02-05T07:37:17.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.988 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251169+0000", 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5741999999999997e-05 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251176+0000", 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8900000000000001e-06 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.874530+0000", 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.62335404500000002 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.989 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.874646+0000", 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000116889 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.875018+0000", 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00037153499999999999 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879566+0000", 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0045484200000000001 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.990 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.883271+0000", 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037046729999999999 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.884149+0000", 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00087832700000000004 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.884261+0000", 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000111209 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:17.991 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:17.992 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.886746+0000", 2024-02-05T07:37:17.992 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024857970000000001 2024-02-05T07:37:17.992 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.992 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.992 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.992 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:17.992 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:17.992 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:37:17.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:18.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:18.037 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:18.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:18.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:18.146 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:18.194 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:18.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:18.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:18.273 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Removed 128927 objects 2024-02-05T07:37:18.273 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Clean up completed and total clean up time :120.292 2024-02-05T07:37:18.296 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:18.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:18.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:18.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:18.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:18.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:18.446 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:18.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:18.540 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T07:37:18.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:18.572 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:18.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:18.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:18.696 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:18.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:18.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:18.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:18 smithi007 ceph-mon[29098]: pgmap v1014: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 289 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 5.2 KiB/s rd, 35 KiB/s wr, 1.12k op/s; 231/14460 objects misplaced (1.598%); 58 KiB/s, 0 objects/s recovering 2024-02-05T07:37:18.806 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:18 smithi007 ceph-mon[37158]: pgmap v1014: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 289 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 5.2 KiB/s rd, 35 KiB/s wr, 1.12k op/s; 231/14460 objects misplaced (1.598%); 58 KiB/s, 0 objects/s recovering 2024-02-05T07:37:18.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:18.822 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:18.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:18.829 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:18.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:18.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:18.948 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:18.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:19.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:18 smithi090 ceph-mon[33213]: pgmap v1014: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 289 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 5.2 KiB/s rd, 35 KiB/s wr, 1.12k op/s; 231/14460 objects misplaced (1.598%); 58 KiB/s, 0 objects/s recovering 2024-02-05T07:37:19.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:19.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:19.074 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:19.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:19.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:19.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:19.201 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:19.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:19.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:19.326 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:19.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:19.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:19.620 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:19.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:19.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:19.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:19.747 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:19.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:19.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:19.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:19.873 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:19.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:19.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:19.999 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:20.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:20.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:20.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:20.126 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:20.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:20.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:20.253 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:20.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:20.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:20.384 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:37:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:20.388 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:20.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:20.388 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:37:20.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:20.515 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:20.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:20.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:20.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:20.642 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:20.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:20.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:20.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:20.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:20.770 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:20.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:20.871 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:20.894 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:20.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:20.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:20 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:37:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:20 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:37:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:20 smithi007 ceph-mon[29098]: pgmap v1015: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 148 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 2.4 KiB/s rd, 14 KiB/s wr, 1.15k op/s; 231/7689 objects misplaced (3.004%); 36 KiB/s, 0 objects/s recovering 2024-02-05T07:37:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:20 smithi007 ceph-mon[29098]: osdmap e563: 8 total, 8 up, 6 in 2024-02-05T07:37:20.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:20 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:37:20.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:20 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:37:20.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:20 smithi007 ceph-mon[37158]: pgmap v1015: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 148 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 2.4 KiB/s rd, 14 KiB/s wr, 1.15k op/s; 231/7689 objects misplaced (3.004%); 36 KiB/s, 0 objects/s recovering 2024-02-05T07:37:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:20 smithi007 ceph-mon[37158]: osdmap e563: 8 total, 8 up, 6 in 2024-02-05T07:37:20.996 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:21.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:20 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:37:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:20 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:37:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:20 smithi090 ceph-mon[33213]: pgmap v1015: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 148 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 2.4 KiB/s rd, 14 KiB/s wr, 1.15k op/s; 231/7689 objects misplaced (3.004%); 36 KiB/s, 0 objects/s recovering 2024-02-05T07:37:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:20 smithi090 ceph-mon[33213]: osdmap e563: 8 total, 8 up, 6 in 2024-02-05T07:37:21.025 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:21.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:21.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:21.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:21.152 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:21.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:21.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:21.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:21.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:21.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:21.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:21.407 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:21.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:21.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:21.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:21.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:37:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:37:21.534 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:21.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:21.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:21.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:21.660 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:21.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:21.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:21.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:21.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:21.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:21.917 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:21.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:21.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:21.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:21 smithi007 ceph-mon[29098]: from='client.25655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:37:21.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:21 smithi007 ceph-mon[37158]: from='client.25655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:37:22.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:22.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:21 smithi090 ceph-mon[33213]: from='client.25655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:37:22.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:22.046 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:22.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:22.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:22.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:22.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:22.303 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:22.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:22.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:22.428 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:22.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:22.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:22.559 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:22.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:22.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:22.684 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:22.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:22.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:22.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:22.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:22.816 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:22.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:22.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:22.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:22 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:22.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:22 smithi007 ceph-mon[29098]: pgmap v1017: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 10 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 231/1053 objects misplaced (21.937%) 2024-02-05T07:37:22.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:22 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:22.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:22 smithi007 ceph-mon[37158]: pgmap v1017: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 10 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 231/1053 objects misplaced (21.937%) 2024-02-05T07:37:22.941 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:22.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:22.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:23.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:22 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:22 smithi090 ceph-mon[33213]: pgmap v1017: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 10 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 231/1053 objects misplaced (21.937%) 2024-02-05T07:37:23.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:23.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:23.072 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:23.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:23.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:23.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:23.202 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:23.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:23.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:23.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:23.329 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:23.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:23.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:23.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:23.455 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:23.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:23.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:23.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:23.584 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:23.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:23.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:37:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:37:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:37:23.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:23.713 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:23.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:23.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:23.840 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:23.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:23.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:23.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:23.969 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:23.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:24.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:24.096 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:24.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:24.197 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:24.224 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:24.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:24.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:24.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:24.355 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:24.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:24.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:24.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:24.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:24.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:24.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:24.715 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:24.743 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:24.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:24.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:24.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:24.871 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:24.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:24.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:24.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:24 smithi007 ceph-mon[29098]: pgmap v1018: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T07:37:24.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:24 smithi007 ceph-mon[29098]: osdmap e564: 8 total, 8 up, 6 in 2024-02-05T07:37:24.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:24 smithi007 ceph-mon[37158]: pgmap v1018: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T07:37:24.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:24 smithi007 ceph-mon[37158]: osdmap e564: 8 total, 8 up, 6 in 2024-02-05T07:37:24.973 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:24.998 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:24.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:24.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:25.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:24 smithi090 ceph-mon[33213]: pgmap v1018: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T07:37:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:24 smithi090 ceph-mon[33213]: osdmap e564: 8 total, 8 up, 6 in 2024-02-05T07:37:25.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:25.126 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:25.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:25.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:25.254 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:25.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:25.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:25.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:25.379 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:25.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:25.480 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:25.507 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:25.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:25.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:25.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:25.633 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:25.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:25.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:25.735 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:25.758 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:25.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:25.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:25.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:25.864 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1016,"stamp":"2024-02-05T07:37:19.635618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":155467865,"num_objects":2563,"num_object_clones":0,"num_object_copies":7689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":231,"num_objects_unfound":0,"num_objects_dirty":2563,"num_whiteouts":0,"num_read":1523,"num_read_kb":1284,"num_write":161350,"num_write_kb":3621482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2934,"num_bytes_recovered":164427704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89265,"ondisk_log_size":89265,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":551,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":5778396,"kb_used_data":930848,"kb_used_omap":13,"kb_used_meta":4847410,"kb_avail":556692516,"statfs":{"total":575970213888,"available":570053136384,"internally_reserved":0,"allocated":953188352,"data_stored":947953594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":4963748576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,5,10,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-587890712,"num_objects":-8972,"num_object_clones":0,"num_object_copies":-26916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-670,"num_objects_unfound":0,"num_objects_dirty":-8972,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9002,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.615682"},"pg_stats":[{"pgid":"10.2","version":"562'12000","reported_seq":12346,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.010905+0000","last_change":"2024-02-05T07:35:20.520500+0000","last_active":"2024-02-05T07:37:15.010905+0000","last_peered":"2024-02-05T07:37:15.010905+0000","last_clean":"2024-02-05T07:37:15.010905+0000","last_became_active":"2024-02-05T07:35:18.799363+0000","last_became_peered":"2024-02-05T07:35:18.799363+0000","last_unstale":"2024-02-05T07:37:15.010905+0000","last_undegraded":"2024-02-05T07:37:15.010905+0000","last_fullsized":"2024-02-05T07:37:15.010905+0000","mapping_epoch":423,"log_start":"508'8898","ondisk_log_start":"508'8898","created":423,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8150","last_scrub_stamp":"2024-02-05T07:35:20.520372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:20.520372+0000","objects_scrubbed":4003,"log_size":3102,"log_dups_size":0,"ondisk_log_size":3102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.697304+0000","scrub_duration":737,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9961472,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3851,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":1225,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618321+0000","last_change":"2024-02-05T07:36:35.619311+0000","last_active":"2024-02-05T07:37:14.618321+0000","last_peered":"2024-02-05T07:37:14.618321+0000","last_clean":"2024-02-05T07:37:14.618321+0000","last_became_active":"2024-02-05T07:36:35.618848+0000","last_became_peered":"2024-02-05T07:36:35.618848+0000","last_unstale":"2024-02-05T07:37:14.618321+0000","last_undegraded":"2024-02-05T07:37:14.618321+0000","last_fullsized":"2024-02-05T07:37:14.618321+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:35:08.100173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:35:08.100173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306475+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"326'25","reported_seq":1120,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614791+0000","last_change":"2024-02-05T07:37:02.567815+0000","last_active":"2024-02-05T07:37:14.614791+0000","last_peered":"2024-02-05T07:37:14.614791+0000","last_clean":"2024-02-05T07:37:14.614791+0000","last_became_active":"2024-02-05T07:28:26.605082+0000","last_became_peered":"2024-02-05T07:28:26.605082+0000","last_unstale":"2024-02-05T07:37:14.614791+0000","last_undegraded":"2024-02-05T07:37:14.614791+0000","last_fullsized":"2024-02-05T07:37:14.614791+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:37:02.567716+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:37:02.567716+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:06.986293+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":619,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.620596+0000","last_change":"2024-02-05T07:33:34.736291+0000","last_active":"2024-02-05T07:37:14.620596+0000","last_peered":"2024-02-05T07:37:14.620596+0000","last_clean":"2024-02-05T07:37:14.620596+0000","last_became_active":"2024-02-05T07:27:16.716461+0000","last_became_peered":"2024-02-05T07:27:16.716461+0000","last_unstale":"2024-02-05T07:37:14.620596+0000","last_undegraded":"2024-02-05T07:37:14.620596+0000","last_fullsized":"2024-02-05T07:37:14.620596+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:34.736160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:34.736160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"538'13","reported_seq":76,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.627895+0000","last_change":"2024-02-05T07:36:32.822462+0000","last_active":"2024-02-05T07:37:14.627895+0000","last_peered":"2024-02-05T07:37:14.627895+0000","last_clean":"2024-02-05T07:37:14.627895+0000","last_became_active":"2024-02-05T07:36:32.743613+0000","last_became_peered":"2024-02-05T07:36:32.743613+0000","last_unstale":"2024-02-05T07:37:14.627895+0000","last_undegraded":"2024-02-05T07:37:14.627895+0000","last_fullsized":"2024-02-05T07:37:14.627895+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:34:29.851477+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:34:29.851477+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1037,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.616114+0000","last_change":"2024-02-05T07:36:32.711145+0000","last_active":"2024-02-05T07:37:14.616114+0000","last_peered":"2024-02-05T07:37:14.616114+0000","last_clean":"2024-02-05T07:37:14.616114+0000","last_became_active":"2024-02-05T07:36:32.710651+0000","last_became_peered":"2024-02-05T07:36:32.710651+0000","last_unstale":"2024-02-05T07:37:14.616114+0000","last_undegraded":"2024-02-05T07:37:14.616114+0000","last_fullsized":"2024-02-05T07:37:14.616114+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:30.708205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:32:30.708205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.553561+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"562'11994","reported_seq":12350,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.447726+0000","last_change":"2024-02-05T07:36:33.730223+0000","last_active":"2024-02-05T07:37:15.447726+0000","last_peered":"2024-02-05T07:37:15.447726+0000","last_clean":"2024-02-05T07:37:15.447726+0000","last_became_active":"2024-02-05T07:36:33.707045+0000","last_became_peered":"2024-02-05T07:36:33.707045+0000","last_unstale":"2024-02-05T07:37:15.447726+0000","last_undegraded":"2024-02-05T07:37:15.447726+0000","last_fullsized":"2024-02-05T07:37:15.447726+0000","mapping_epoch":536,"log_start":"502'8248","ondisk_log_start":"502'8248","created":423,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8061","last_scrub_stamp":"2024-02-05T07:35:20.146489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:20.146489+0000","objects_scrubbed":4072,"log_size":3746,"log_dups_size":0,"ondisk_log_size":3746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8650752,"num_objects":132,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3940,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":527,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.624385+0000","last_change":"2024-02-05T07:36:52.193219+0000","last_active":"2024-02-05T07:37:14.624385+0000","last_peered":"2024-02-05T07:37:14.624385+0000","last_clean":"2024-02-05T07:37:14.624385+0000","last_became_active":"2024-02-05T07:36:35.627588+0000","last_became_peered":"2024-02-05T07:36:35.627588+0000","last_unstale":"2024-02-05T07:37:14.624385+0000","last_undegraded":"2024-02-05T07:37:14.624385+0000","last_fullsized":"2024-02-05T07:37:14.624385+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:36:52.192975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:36:52.192975+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":69,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:03.854602+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"326'31","reported_seq":1139,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.615498+0000","last_change":"2024-02-05T07:36:32.772355+0000","last_active":"2024-02-05T07:37:14.615498+0000","last_peered":"2024-02-05T07:37:14.615498+0000","last_clean":"2024-02-05T07:37:14.615498+0000","last_became_active":"2024-02-05T07:36:32.717742+0000","last_became_peered":"2024-02-05T07:36:32.717742+0000","last_unstale":"2024-02-05T07:37:14.615498+0000","last_undegraded":"2024-02-05T07:37:14.615498+0000","last_fullsized":"2024-02-05T07:37:14.615498+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'31","last_scrub_stamp":"2024-02-05T07:33:30.750548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.750548+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.553667+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":891,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618790+0000","last_change":"2024-02-05T07:33:44.229582+0000","last_active":"2024-02-05T07:37:14.618790+0000","last_peered":"2024-02-05T07:37:14.618790+0000","last_clean":"2024-02-05T07:37:14.618790+0000","last_became_active":"2024-02-05T07:28:25.791005+0000","last_became_peered":"2024-02-05T07:28:25.791005+0000","last_unstale":"2024-02-05T07:37:14.618790+0000","last_undegraded":"2024-02-05T07:37:14.618790+0000","last_fullsized":"2024-02-05T07:37:14.618790+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.229489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.229489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.527041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":251,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614729+0000","last_change":"2024-02-05T07:36:41.530864+0000","last_active":"2024-02-05T07:37:14.614729+0000","last_peered":"2024-02-05T07:37:14.614729+0000","last_clean":"2024-02-05T07:37:14.614729+0000","last_became_active":"2024-02-05T07:34:30.678610+0000","last_became_peered":"2024-02-05T07:34:30.678610+0000","last_unstale":"2024-02-05T07:37:14.614729+0000","last_undegraded":"2024-02-05T07:37:14.614729+0000","last_fullsized":"2024-02-05T07:37:14.614729+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:41.530749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:41.530749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:00.868644+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":63,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.627211+0000","last_change":"2024-02-05T07:36:32.724079+0000","last_active":"2024-02-05T07:37:14.627211+0000","last_peered":"2024-02-05T07:37:14.627211+0000","last_clean":"2024-02-05T07:37:14.627211+0000","last_became_active":"2024-02-05T07:36:32.723796+0000","last_became_peered":"2024-02-05T07:36:32.723796+0000","last_unstale":"2024-02-05T07:37:14.627211+0000","last_undegraded":"2024-02-05T07:37:14.627211+0000","last_fullsized":"2024-02-05T07:37:14.627211+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:23.161521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:23.161521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298910+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":814,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.627660+0000","last_change":"2024-02-05T07:37:12.046423+0000","last_active":"2024-02-05T07:37:14.627660+0000","last_peered":"2024-02-05T07:37:14.627660+0000","last_clean":"2024-02-05T07:37:14.627660+0000","last_became_active":"2024-02-05T07:37:12.045435+0000","last_became_peered":"2024-02-05T07:37:12.045435+0000","last_unstale":"2024-02-05T07:37:14.627660+0000","last_undegraded":"2024-02-05T07:37:14.627660+0000","last_fullsized":"2024-02-05T07:37:14.627660+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:31.185403+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:34:31.185403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:17.329184+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"562'12005","reported_seq":12350,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:18.249314+0000","last_change":"2024-02-05T07:35:20.368840+0000","last_active":"2024-02-05T07:37:18.249314+0000","last_peered":"2024-02-05T07:37:18.249314+0000","last_clean":"2024-02-05T07:37:18.249314+0000","last_became_active":"2024-02-05T07:35:18.791210+0000","last_became_peered":"2024-02-05T07:35:18.791210+0000","last_unstale":"2024-02-05T07:37:18.249314+0000","last_undegraded":"2024-02-05T07:37:18.249314+0000","last_fullsized":"2024-02-05T07:37:18.249314+0000","mapping_epoch":423,"log_start":"507'8505","ondisk_log_start":"507'8505","created":423,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8038","last_scrub_stamp":"2024-02-05T07:35:20.368714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:20.368714+0000","objects_scrubbed":3967,"log_size":3500,"log_dups_size":0,"ondisk_log_size":3500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:20.648721+0000","scrub_duration":477,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3967,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"326'24","reported_seq":1086,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.615942+0000","last_change":"2024-02-05T07:35:13.047805+0000","last_active":"2024-02-05T07:37:14.615942+0000","last_peered":"2024-02-05T07:37:14.615942+0000","last_clean":"2024-02-05T07:37:14.615942+0000","last_became_active":"2024-02-05T07:28:25.781007+0000","last_became_peered":"2024-02-05T07:28:25.781007+0000","last_unstale":"2024-02-05T07:37:14.615942+0000","last_undegraded":"2024-02-05T07:37:14.615942+0000","last_fullsized":"2024-02-05T07:37:14.615942+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:35:13.047702+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:35:13.047702+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.555392+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1064,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.613942+0000","last_change":"2024-02-05T07:37:03.618764+0000","last_active":"2024-02-05T07:37:14.613942+0000","last_peered":"2024-02-05T07:37:14.613942+0000","last_clean":"2024-02-05T07:37:14.613942+0000","last_became_active":"2024-02-05T07:36:32.702469+0000","last_became_peered":"2024-02-05T07:36:32.702469+0000","last_unstale":"2024-02-05T07:37:14.613942+0000","last_undegraded":"2024-02-05T07:37:14.613942+0000","last_fullsized":"2024-02-05T07:37:14.613942+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:03.618665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:37:03.618665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:25.633418+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":885,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.615559+0000","last_change":"2024-02-05T07:36:32.712676+0000","last_active":"2024-02-05T07:37:14.615559+0000","last_peered":"2024-02-05T07:37:14.615559+0000","last_clean":"2024-02-05T07:37:14.615559+0000","last_became_active":"2024-02-05T07:36:32.712332+0000","last_became_peered":"2024-02-05T07:36:32.712332+0000","last_unstale":"2024-02-05T07:37:14.615559+0000","last_undegraded":"2024-02-05T07:37:14.615559+0000","last_fullsized":"2024-02-05T07:37:14.615559+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:14.031714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:35:14.031714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.555324+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":628,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.624707+0000","last_change":"2024-02-05T07:33:28.696549+0000","last_active":"2024-02-05T07:37:14.624707+0000","last_peered":"2024-02-05T07:37:14.624707+0000","last_clean":"2024-02-05T07:37:14.624707+0000","last_became_active":"2024-02-05T07:30:40.498529+0000","last_became_peered":"2024-02-05T07:30:40.498529+0000","last_unstale":"2024-02-05T07:37:14.624707+0000","last_undegraded":"2024-02-05T07:37:14.624707+0000","last_fullsized":"2024-02-05T07:37:14.624707+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.696462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.696462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.305926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"562'11996","reported_seq":12341,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.361885+0000","last_change":"2024-02-05T07:35:19.907738+0000","last_active":"2024-02-05T07:37:15.361885+0000","last_peered":"2024-02-05T07:37:15.361885+0000","last_clean":"2024-02-05T07:37:15.361885+0000","last_became_active":"2024-02-05T07:35:18.797257+0000","last_became_peered":"2024-02-05T07:35:18.797257+0000","last_unstale":"2024-02-05T07:37:15.361885+0000","last_undegraded":"2024-02-05T07:37:15.361885+0000","last_fullsized":"2024-02-05T07:37:15.361885+0000","mapping_epoch":423,"log_start":"508'8838","ondisk_log_start":"508'8838","created":423,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8152","last_scrub_stamp":"2024-02-05T07:35:19.907566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:19.907566+0000","objects_scrubbed":3999,"log_size":3158,"log_dups_size":0,"ondisk_log_size":3158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.555358+0000","scrub_duration":929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8978432,"num_objects":137,"num_object_clones":0,"num_object_copies":411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3862,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":395,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.630978+0000","last_change":"2024-02-05T07:36:51.217816+0000","last_active":"2024-02-05T07:37:14.630978+0000","last_peered":"2024-02-05T07:37:14.630978+0000","last_clean":"2024-02-05T07:37:14.630978+0000","last_became_active":"2024-02-05T07:36:35.627308+0000","last_became_peered":"2024-02-05T07:36:35.627308+0000","last_unstale":"2024-02-05T07:37:14.630978+0000","last_undegraded":"2024-02-05T07:37:14.630978+0000","last_fullsized":"2024-02-05T07:37:14.630978+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:36:51.217709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:36:51.217709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:18.233690+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"326'34","reported_seq":1137,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.617830+0000","last_change":"2024-02-05T07:36:46.676874+0000","last_active":"2024-02-05T07:37:14.617830+0000","last_peered":"2024-02-05T07:37:14.617830+0000","last_clean":"2024-02-05T07:37:14.617830+0000","last_became_active":"2024-02-05T07:27:16.768454+0000","last_became_peered":"2024-02-05T07:27:16.768454+0000","last_unstale":"2024-02-05T07:37:14.617830+0000","last_undegraded":"2024-02-05T07:37:14.617830+0000","last_fullsized":"2024-02-05T07:37:14.617830+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'34","last_scrub_stamp":"2024-02-05T07:36:46.676770+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.676770+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:48.015918+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":633,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.620608+0000","last_change":"2024-02-05T07:36:47.116839+0000","last_active":"2024-02-05T07:37:14.620608+0000","last_peered":"2024-02-05T07:37:14.620608+0000","last_clean":"2024-02-05T07:37:14.620608+0000","last_became_active":"2024-02-05T07:30:40.508561+0000","last_became_peered":"2024-02-05T07:30:40.508561+0000","last_unstale":"2024-02-05T07:37:14.620608+0000","last_undegraded":"2024-02-05T07:37:14.620608+0000","last_fullsized":"2024-02-05T07:37:14.620608+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:47.116727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:47.116727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:55.608652+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"538'11","reported_seq":1297,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.616102+0000","last_change":"2024-02-05T07:36:59.523533+0000","last_active":"2024-02-05T07:37:14.616102+0000","last_peered":"2024-02-05T07:37:14.616102+0000","last_clean":"2024-02-05T07:37:14.616102+0000","last_became_active":"2024-02-05T07:36:32.709942+0000","last_became_peered":"2024-02-05T07:36:32.709942+0000","last_unstale":"2024-02-05T07:37:14.616102+0000","last_undegraded":"2024-02-05T07:37:14.616102+0000","last_fullsized":"2024-02-05T07:37:14.616102+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'11","last_scrub_stamp":"2024-02-05T07:36:59.523421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.523421+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:13.722164+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":63,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.627763+0000","last_change":"2024-02-05T07:36:32.744571+0000","last_active":"2024-02-05T07:37:14.627763+0000","last_peered":"2024-02-05T07:37:14.627763+0000","last_clean":"2024-02-05T07:37:14.627763+0000","last_became_active":"2024-02-05T07:36:32.744070+0000","last_became_peered":"2024-02-05T07:36:32.744070+0000","last_unstale":"2024-02-05T07:37:14.627763+0000","last_undegraded":"2024-02-05T07:37:14.627763+0000","last_fullsized":"2024-02-05T07:37:14.627763+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:38.025380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.025380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.e","version":"562'16003","reported_seq":16307,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.427788+0000","last_change":"2024-02-05T07:36:32.736314+0000","last_active":"2024-02-05T07:37:15.427788+0000","last_peered":"2024-02-05T07:37:15.427788+0000","last_clean":"2024-02-05T07:37:15.427788+0000","last_became_active":"2024-02-05T07:36:32.709815+0000","last_became_peered":"2024-02-05T07:36:32.709815+0000","last_unstale":"2024-02-05T07:37:15.427788+0000","last_undegraded":"2024-02-05T07:37:15.427788+0000","last_fullsized":"2024-02-05T07:37:15.427788+0000","mapping_epoch":535,"log_start":"534'12829","ondisk_log_start":"534'12829","created":423,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3174,"log_dups_size":1221,"ondisk_log_size":3174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.553685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15663104,"num_objects":239,"num_object_clones":0,"num_object_copies":717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16003,"num_write_kb":519744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":1131,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614789+0000","last_change":"2024-02-05T07:37:07.528655+0000","last_active":"2024-02-05T07:37:14.614789+0000","last_peered":"2024-02-05T07:37:14.614789+0000","last_clean":"2024-02-05T07:37:14.614789+0000","last_became_active":"2024-02-05T07:37:06.995057+0000","last_became_peered":"2024-02-05T07:37:06.995057+0000","last_unstale":"2024-02-05T07:37:14.614789+0000","last_undegraded":"2024-02-05T07:37:14.614789+0000","last_fullsized":"2024-02-05T07:37:14.614789+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:07.528587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:07.528587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:16.024478+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1048,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.613915+0000","last_change":"2024-02-05T07:32:36.623960+0000","last_active":"2024-02-05T07:37:14.613915+0000","last_peered":"2024-02-05T07:37:14.613915+0000","last_clean":"2024-02-05T07:37:14.613915+0000","last_became_active":"2024-02-05T07:30:40.489212+0000","last_became_peered":"2024-02-05T07:30:40.489212+0000","last_unstale":"2024-02-05T07:37:14.613915+0000","last_undegraded":"2024-02-05T07:37:14.613915+0000","last_fullsized":"2024-02-05T07:37:14.613915+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:36.623865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:36.623865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.553598+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"454'38","reported_seq":695,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.620495+0000","last_change":"2024-02-05T07:34:38.943008+0000","last_active":"2024-02-05T07:37:14.620495+0000","last_peered":"2024-02-05T07:37:14.620495+0000","last_clean":"2024-02-05T07:37:14.620495+0000","last_became_active":"2024-02-05T07:30:40.496940+0000","last_became_peered":"2024-02-05T07:30:40.496940+0000","last_unstale":"2024-02-05T07:37:14.620495+0000","last_undegraded":"2024-02-05T07:37:14.620495+0000","last_fullsized":"2024-02-05T07:37:14.620495+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'38","last_scrub_stamp":"2024-02-05T07:34:38.942830+0000","last_deep_scrub":"454'38","last_deep_scrub_stamp":"2024-02-05T07:34:38.942830+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.942830+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306340+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":625,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.621178+0000","last_change":"2024-02-05T07:33:48.701618+0000","last_active":"2024-02-05T07:37:14.621178+0000","last_peered":"2024-02-05T07:37:14.621178+0000","last_clean":"2024-02-05T07:37:14.621178+0000","last_became_active":"2024-02-05T07:30:40.508607+0000","last_became_peered":"2024-02-05T07:30:40.508607+0000","last_unstale":"2024-02-05T07:37:14.621178+0000","last_undegraded":"2024-02-05T07:37:14.621178+0000","last_fullsized":"2024-02-05T07:37:14.621178+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:48.701399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:48.701399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306091+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1053,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614362+0000","last_change":"2024-02-05T07:36:32.709346+0000","last_active":"2024-02-05T07:37:14.614362+0000","last_peered":"2024-02-05T07:37:14.614362+0000","last_clean":"2024-02-05T07:37:14.614362+0000","last_became_active":"2024-02-05T07:36:32.708979+0000","last_became_peered":"2024-02-05T07:36:32.708979+0000","last_unstale":"2024-02-05T07:37:14.614362+0000","last_undegraded":"2024-02-05T07:37:14.614362+0000","last_fullsized":"2024-02-05T07:37:14.614362+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:45.908064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:33:45.908064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.553729+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"562'16086","reported_seq":16424,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:18.264503+0000","last_change":"2024-02-05T07:36:46.213073+0000","last_active":"2024-02-05T07:37:18.264503+0000","last_peered":"2024-02-05T07:37:18.264503+0000","last_clean":"2024-02-05T07:37:18.264503+0000","last_became_active":"2024-02-05T07:36:32.717068+0000","last_became_peered":"2024-02-05T07:36:32.717068+0000","last_unstale":"2024-02-05T07:37:18.264503+0000","last_undegraded":"2024-02-05T07:37:18.264503+0000","last_fullsized":"2024-02-05T07:37:18.264503+0000","mapping_epoch":535,"log_start":"532'12518","ondisk_log_start":"532'12518","created":423,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"546'13589","last_scrub_stamp":"2024-02-05T07:36:46.212874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.212874+0000","objects_scrubbed":2519,"log_size":3568,"log_dups_size":203,"ondisk_log_size":3568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:12.382989+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16086,"num_write_kb":514752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":1366,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.624876+0000","last_change":"2024-02-05T07:37:07.354642+0000","last_active":"2024-02-05T07:37:14.624876+0000","last_peered":"2024-02-05T07:37:14.624876+0000","last_clean":"2024-02-05T07:37:14.624876+0000","last_became_active":"2024-02-05T07:37:06.978821+0000","last_became_peered":"2024-02-05T07:37:06.978821+0000","last_unstale":"2024-02-05T07:37:14.624876+0000","last_undegraded":"2024-02-05T07:37:14.624876+0000","last_fullsized":"2024-02-05T07:37:14.624876+0000","mapping_epoch":367,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:07.353770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:07.353770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:23.371584+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"266'10","reported_seq":743,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.620767+0000","last_change":"2024-02-05T07:33:42.715242+0000","last_active":"2024-02-05T07:37:14.620767+0000","last_peered":"2024-02-05T07:37:14.620767+0000","last_clean":"2024-02-05T07:37:14.620767+0000","last_became_active":"2024-02-05T07:27:16.800207+0000","last_became_peered":"2024-02-05T07:27:16.800207+0000","last_unstale":"2024-02-05T07:37:14.620767+0000","last_undegraded":"2024-02-05T07:37:14.620767+0000","last_fullsized":"2024-02-05T07:37:14.620767+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'10","last_scrub_stamp":"2024-02-05T07:33:42.715162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:42.715162+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.305983+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"326'35","reported_seq":1112,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614025+0000","last_change":"2024-02-05T07:36:35.442778+0000","last_active":"2024-02-05T07:37:14.614025+0000","last_peered":"2024-02-05T07:37:14.614025+0000","last_clean":"2024-02-05T07:37:14.614025+0000","last_became_active":"2024-02-05T07:28:26.598831+0000","last_became_peered":"2024-02-05T07:28:26.598831+0000","last_unstale":"2024-02-05T07:37:14.614025+0000","last_undegraded":"2024-02-05T07:37:14.614025+0000","last_fullsized":"2024-02-05T07:37:14.614025+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:36:35.442664+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:36:35.442664+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:55.060557+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":799,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618005+0000","last_change":"2024-02-05T07:34:42.579613+0000","last_active":"2024-02-05T07:37:14.618005+0000","last_peered":"2024-02-05T07:37:14.618005+0000","last_clean":"2024-02-05T07:37:14.618005+0000","last_became_active":"2024-02-05T07:24:34.238438+0000","last_became_peered":"2024-02-05T07:24:34.238438+0000","last_unstale":"2024-02-05T07:37:14.618005+0000","last_undegraded":"2024-02-05T07:37:14.618005+0000","last_fullsized":"2024-02-05T07:37:14.618005+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:34:42.579486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:42.579486+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.697245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":923,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.620070+0000","last_change":"2024-02-05T07:36:32.701554+0000","last_active":"2024-02-05T07:37:14.620070+0000","last_peered":"2024-02-05T07:37:14.620070+0000","last_clean":"2024-02-05T07:37:14.620070+0000","last_became_active":"2024-02-05T07:36:32.701203+0000","last_became_peered":"2024-02-05T07:36:32.701203+0000","last_unstale":"2024-02-05T07:37:14.620070+0000","last_undegraded":"2024-02-05T07:37:14.620070+0000","last_fullsized":"2024-02-05T07:37:14.620070+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:35.338979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:34:35.338979+0000","last_clean_scrub_stamp":"2024-02-05T07:34:35.338979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.527090+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"562'16191","reported_seq":16483,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.411624+0000","last_change":"2024-02-05T07:33:38.775156+0000","last_active":"2024-02-05T07:37:15.411624+0000","last_peered":"2024-02-05T07:37:15.411624+0000","last_clean":"2024-02-05T07:37:15.411624+0000","last_became_active":"2024-02-05T07:33:38.774713+0000","last_became_peered":"2024-02-05T07:33:38.774713+0000","last_unstale":"2024-02-05T07:37:15.411624+0000","last_undegraded":"2024-02-05T07:37:15.411624+0000","last_fullsized":"2024-02-05T07:37:15.411624+0000","mapping_epoch":423,"log_start":"538'13037","ondisk_log_start":"538'13037","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3154,"log_dups_size":1221,"ondisk_log_size":3154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.555223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13959168,"num_objects":213,"num_object_clones":0,"num_object_copies":639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16191,"num_write_kb":524928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":993,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618297+0000","last_change":"2024-02-05T07:37:06.985232+0000","last_active":"2024-02-05T07:37:14.618297+0000","last_peered":"2024-02-05T07:37:14.618297+0000","last_clean":"2024-02-05T07:37:14.618297+0000","last_became_active":"2024-02-05T07:37:06.984856+0000","last_became_peered":"2024-02-05T07:37:06.984856+0000","last_unstale":"2024-02-05T07:37:14.618297+0000","last_undegraded":"2024-02-05T07:37:14.618297+0000","last_fullsized":"2024-02-05T07:37:14.618297+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:29.214474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:29.214474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.609950+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":744,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.620189+0000","last_change":"2024-02-05T07:35:21.887728+0000","last_active":"2024-02-05T07:37:14.620189+0000","last_peered":"2024-02-05T07:37:14.620189+0000","last_clean":"2024-02-05T07:37:14.620189+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:37:14.620189+0000","last_undegraded":"2024-02-05T07:37:14.620189+0000","last_fullsized":"2024-02-05T07:37:14.620189+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:35:21.887636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:35:21.887636+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.527189+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"326'25","reported_seq":1104,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.616492+0000","last_change":"2024-02-05T07:37:08.715017+0000","last_active":"2024-02-05T07:37:14.616492+0000","last_peered":"2024-02-05T07:37:14.616492+0000","last_clean":"2024-02-05T07:37:14.616492+0000","last_became_active":"2024-02-05T07:27:03.361342+0000","last_became_peered":"2024-02-05T07:27:03.361342+0000","last_unstale":"2024-02-05T07:37:14.616492+0000","last_undegraded":"2024-02-05T07:37:14.616492+0000","last_fullsized":"2024-02-05T07:37:14.616492+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:37:08.714926+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:37:08.714926+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:10.266975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":642,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.619815+0000","last_change":"2024-02-05T07:36:32.741872+0000","last_active":"2024-02-05T07:37:14.619815+0000","last_peered":"2024-02-05T07:37:14.619815+0000","last_clean":"2024-02-05T07:37:14.619815+0000","last_became_active":"2024-02-05T07:36:32.741377+0000","last_became_peered":"2024-02-05T07:36:32.741377+0000","last_unstale":"2024-02-05T07:37:14.619815+0000","last_undegraded":"2024-02-05T07:37:14.619815+0000","last_fullsized":"2024-02-05T07:37:14.619815+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:17.067204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:35:17.067204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306600+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":319,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.615451+0000","last_change":"2024-02-05T07:33:24.419375+0000","last_active":"2024-02-05T07:37:14.615451+0000","last_peered":"2024-02-05T07:37:14.615451+0000","last_clean":"2024-02-05T07:37:14.615451+0000","last_became_active":"2024-02-05T07:32:58.598709+0000","last_became_peered":"2024-02-05T07:32:58.598709+0000","last_unstale":"2024-02-05T07:37:14.615451+0000","last_undegraded":"2024-02-05T07:37:14.615451+0000","last_fullsized":"2024-02-05T07:37:14.615451+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:24.419250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:24.419250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.697054+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"562'16020","reported_seq":16338,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:18.269511+0000","last_change":"2024-02-05T07:36:58.875574+0000","last_active":"2024-02-05T07:37:18.269511+0000","last_peered":"2024-02-05T07:37:18.269511+0000","last_clean":"2024-02-05T07:37:18.269511+0000","last_became_active":"2024-02-05T07:33:38.775048+0000","last_became_peered":"2024-02-05T07:33:38.775048+0000","last_unstale":"2024-02-05T07:37:18.269511+0000","last_undegraded":"2024-02-05T07:37:18.269511+0000","last_fullsized":"2024-02-05T07:37:18.269511+0000","mapping_epoch":423,"log_start":"532'12476","ondisk_log_start":"532'12476","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"551'14458","last_scrub_stamp":"2024-02-05T07:36:58.875424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:58.875424+0000","objects_scrubbed":1586,"log_size":3544,"log_dups_size":174,"ondisk_log_size":3544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:26.207932+0000","scrub_duration":413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16020,"num_write_kb":512577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":965,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618209+0000","last_change":"2024-02-05T07:37:06.985401+0000","last_active":"2024-02-05T07:37:14.618209+0000","last_peered":"2024-02-05T07:37:14.618209+0000","last_clean":"2024-02-05T07:37:14.618209+0000","last_became_active":"2024-02-05T07:37:06.985145+0000","last_became_peered":"2024-02-05T07:37:06.985145+0000","last_unstale":"2024-02-05T07:37:14.618209+0000","last_undegraded":"2024-02-05T07:37:14.618209+0000","last_fullsized":"2024-02-05T07:37:14.618209+0000","mapping_epoch":302,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:36:32.497664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:36:32.497664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.609984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":898,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618366+0000","last_change":"2024-02-05T07:36:41.525416+0000","last_active":"2024-02-05T07:37:14.618366+0000","last_peered":"2024-02-05T07:37:14.618366+0000","last_clean":"2024-02-05T07:37:14.618366+0000","last_became_active":"2024-02-05T07:36:32.709025+0000","last_became_peered":"2024-02-05T07:36:32.709025+0000","last_unstale":"2024-02-05T07:37:14.618366+0000","last_undegraded":"2024-02-05T07:37:14.618366+0000","last_fullsized":"2024-02-05T07:37:14.618366+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:41.525301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:36:41.525301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:05.462600+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"326'29","reported_seq":721,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.617347+0000","last_change":"2024-02-05T07:36:32.797322+0000","last_active":"2024-02-05T07:37:14.617347+0000","last_peered":"2024-02-05T07:37:14.617347+0000","last_clean":"2024-02-05T07:37:14.617347+0000","last_became_active":"2024-02-05T07:36:32.712346+0000","last_became_peered":"2024-02-05T07:36:32.712346+0000","last_unstale":"2024-02-05T07:37:14.617347+0000","last_undegraded":"2024-02-05T07:37:14.617347+0000","last_fullsized":"2024-02-05T07:37:14.617347+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'29","last_scrub_stamp":"2024-02-05T07:35:21.877054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:21.877054+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.697421+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":63,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.624792+0000","last_change":"2024-02-05T07:36:32.733360+0000","last_active":"2024-02-05T07:37:14.624792+0000","last_peered":"2024-02-05T07:37:14.624792+0000","last_clean":"2024-02-05T07:37:14.624792+0000","last_became_active":"2024-02-05T07:36:32.732781+0000","last_became_peered":"2024-02-05T07:36:32.732781+0000","last_unstale":"2024-02-05T07:37:14.624792+0000","last_undegraded":"2024-02-05T07:37:14.624792+0000","last_fullsized":"2024-02-05T07:37:14.624792+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:34.799346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:34:34.799346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.299022+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1048,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.616598+0000","last_change":"2024-02-05T07:36:32.710252+0000","last_active":"2024-02-05T07:37:14.616598+0000","last_peered":"2024-02-05T07:37:14.616598+0000","last_clean":"2024-02-05T07:37:14.616598+0000","last_became_active":"2024-02-05T07:36:32.709872+0000","last_became_peered":"2024-02-05T07:36:32.709872+0000","last_unstale":"2024-02-05T07:37:14.616598+0000","last_undegraded":"2024-02-05T07:37:14.616598+0000","last_fullsized":"2024-02-05T07:37:14.616598+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:33.257028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.257028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.697025+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"562'12029","reported_seq":12385,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.465975+0000","last_change":"2024-02-05T07:36:33.722770+0000","last_active":"2024-02-05T07:37:15.465975+0000","last_peered":"2024-02-05T07:37:15.465975+0000","last_clean":"2024-02-05T07:37:15.465975+0000","last_became_active":"2024-02-05T07:36:33.698507+0000","last_became_peered":"2024-02-05T07:36:33.698507+0000","last_unstale":"2024-02-05T07:37:15.465975+0000","last_undegraded":"2024-02-05T07:37:15.465975+0000","last_fullsized":"2024-02-05T07:37:15.465975+0000","mapping_epoch":536,"log_start":"502'8325","ondisk_log_start":"502'8325","created":423,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8110","last_scrub_stamp":"2024-02-05T07:35:20.687974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:20.687974+0000","objects_scrubbed":4038,"log_size":3704,"log_dups_size":27,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7798784,"num_objects":119,"num_object_clones":0,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3919,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":615,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.620192+0000","last_change":"2024-02-05T07:37:06.970652+0000","last_active":"2024-02-05T07:37:14.620192+0000","last_peered":"2024-02-05T07:37:14.620192+0000","last_clean":"2024-02-05T07:37:14.620192+0000","last_became_active":"2024-02-05T07:37:06.970233+0000","last_became_peered":"2024-02-05T07:37:06.970233+0000","last_unstale":"2024-02-05T07:37:14.620192+0000","last_undegraded":"2024-02-05T07:37:14.620192+0000","last_fullsized":"2024-02-05T07:37:14.620192+0000","mapping_epoch":446,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:31.185403+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:34:31.185403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.609910+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":636,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618460+0000","last_change":"2024-02-05T07:32:46.399673+0000","last_active":"2024-02-05T07:37:14.618460+0000","last_peered":"2024-02-05T07:37:14.618460+0000","last_clean":"2024-02-05T07:37:14.618460+0000","last_became_active":"2024-02-05T07:27:03.370297+0000","last_became_peered":"2024-02-05T07:27:03.370297+0000","last_unstale":"2024-02-05T07:37:14.618460+0000","last_undegraded":"2024-02-05T07:37:14.618460+0000","last_fullsized":"2024-02-05T07:37:14.618460+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:46.399572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:46.399572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.696972+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"326'24","reported_seq":77,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.630236+0000","last_change":"2024-02-05T07:36:32.847697+0000","last_active":"2024-02-05T07:37:14.630236+0000","last_peered":"2024-02-05T07:37:14.630236+0000","last_clean":"2024-02-05T07:37:14.630236+0000","last_became_active":"2024-02-05T07:36:32.753752+0000","last_became_peered":"2024-02-05T07:36:32.753752+0000","last_unstale":"2024-02-05T07:37:14.630236+0000","last_undegraded":"2024-02-05T07:37:14.630236+0000","last_fullsized":"2024-02-05T07:37:14.630236+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:33:49.499846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:49.499846+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298847+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":855,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.615833+0000","last_change":"2024-02-05T07:35:10.736128+0000","last_active":"2024-02-05T07:37:14.615833+0000","last_peered":"2024-02-05T07:37:14.615833+0000","last_clean":"2024-02-05T07:37:14.615833+0000","last_became_active":"2024-02-05T07:30:40.493770+0000","last_became_peered":"2024-02-05T07:30:40.493770+0000","last_unstale":"2024-02-05T07:37:14.615833+0000","last_undegraded":"2024-02-05T07:37:14.615833+0000","last_fullsized":"2024-02-05T07:37:14.615833+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:35:10.736006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:35:10.736006+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.697274+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":399,"num_read_kb":394,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1011,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614245+0000","last_change":"2024-02-05T07:33:39.809630+0000","last_active":"2024-02-05T07:37:14.614245+0000","last_peered":"2024-02-05T07:37:14.614245+0000","last_clean":"2024-02-05T07:37:14.614245+0000","last_became_active":"2024-02-05T07:27:16.724280+0000","last_became_peered":"2024-02-05T07:27:16.724280+0000","last_unstale":"2024-02-05T07:37:14.614245+0000","last_undegraded":"2024-02-05T07:37:14.614245+0000","last_fullsized":"2024-02-05T07:37:14.614245+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:39.809544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:33:39.809544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.555239+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":1064,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.619091+0000","last_change":"2024-02-05T07:33:41.379002+0000","last_active":"2024-02-05T07:37:14.619091+0000","last_peered":"2024-02-05T07:37:14.619091+0000","last_clean":"2024-02-05T07:37:14.619091+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:37:14.619091+0000","last_undegraded":"2024-02-05T07:37:14.619091+0000","last_fullsized":"2024-02-05T07:37:14.619091+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:33:41.378909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.378909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.697101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"562'16062","reported_seq":16385,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:18.244270+0000","last_change":"2024-02-05T07:36:50.001729+0000","last_active":"2024-02-05T07:37:18.244270+0000","last_peered":"2024-02-05T07:37:18.244270+0000","last_clean":"2024-02-05T07:37:18.244270+0000","last_became_active":"2024-02-05T07:33:38.775338+0000","last_became_peered":"2024-02-05T07:33:38.775338+0000","last_unstale":"2024-02-05T07:37:18.244270+0000","last_undegraded":"2024-02-05T07:37:18.244270+0000","last_fullsized":"2024-02-05T07:37:18.244270+0000","mapping_epoch":423,"log_start":"532'12516","ondisk_log_start":"532'12516","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'13830","last_scrub_stamp":"2024-02-05T07:36:50.001581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:50.001581+0000","objects_scrubbed":2254,"log_size":3546,"log_dups_size":174,"ondisk_log_size":3546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:01.206941+0000","scrub_duration":582,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16062,"num_write_kb":513984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":63,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.627126+0000","last_change":"2024-02-05T07:36:32.733109+0000","last_active":"2024-02-05T07:37:14.627126+0000","last_peered":"2024-02-05T07:37:14.627126+0000","last_clean":"2024-02-05T07:37:14.627126+0000","last_became_active":"2024-02-05T07:36:32.732334+0000","last_became_peered":"2024-02-05T07:36:32.732334+0000","last_unstale":"2024-02-05T07:37:14.627126+0000","last_undegraded":"2024-02-05T07:37:14.627126+0000","last_fullsized":"2024-02-05T07:37:14.627126+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:50.943640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:50.943640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298737+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"365'31","reported_seq":1148,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618044+0000","last_change":"2024-02-05T07:33:46.200250+0000","last_active":"2024-02-05T07:37:14.618044+0000","last_peered":"2024-02-05T07:37:14.618044+0000","last_clean":"2024-02-05T07:37:14.618044+0000","last_became_active":"2024-02-05T07:30:40.508686+0000","last_became_peered":"2024-02-05T07:30:40.508686+0000","last_unstale":"2024-02-05T07:37:14.618044+0000","last_undegraded":"2024-02-05T07:37:14.618044+0000","last_fullsized":"2024-02-05T07:37:14.618044+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'31","last_scrub_stamp":"2024-02-05T07:33:46.200150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.200150+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.527025+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":630,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.619924+0000","last_change":"2024-02-05T07:35:12.093470+0000","last_active":"2024-02-05T07:37:14.619924+0000","last_peered":"2024-02-05T07:37:14.619924+0000","last_clean":"2024-02-05T07:37:14.619924+0000","last_became_active":"2024-02-05T07:28:26.595640+0000","last_became_peered":"2024-02-05T07:28:26.595640+0000","last_unstale":"2024-02-05T07:37:14.619924+0000","last_undegraded":"2024-02-05T07:37:14.619924+0000","last_fullsized":"2024-02-05T07:37:14.619924+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:12.093350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:35:12.093350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306572+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":800,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.619768+0000","last_change":"2024-02-05T07:37:08.472969+0000","last_active":"2024-02-05T07:37:14.619768+0000","last_peered":"2024-02-05T07:37:14.619768+0000","last_clean":"2024-02-05T07:37:14.619768+0000","last_became_active":"2024-02-05T07:37:06.977158+0000","last_became_peered":"2024-02-05T07:37:06.977158+0000","last_unstale":"2024-02-05T07:37:14.619768+0000","last_undegraded":"2024-02-05T07:37:14.619768+0000","last_fullsized":"2024-02-05T07:37:14.619768+0000","mapping_epoch":367,"log_start":"86'128","ondisk_log_start":"86'128","created":20,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:08.472857+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:37:08.472857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:28.947408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":619,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.621098+0000","last_change":"2024-02-05T07:33:46.760555+0000","last_active":"2024-02-05T07:37:14.621098+0000","last_peered":"2024-02-05T07:37:14.621098+0000","last_clean":"2024-02-05T07:37:14.621098+0000","last_became_active":"2024-02-05T07:27:16.751341+0000","last_became_peered":"2024-02-05T07:27:16.751341+0000","last_unstale":"2024-02-05T07:37:14.621098+0000","last_undegraded":"2024-02-05T07:37:14.621098+0000","last_fullsized":"2024-02-05T07:37:14.621098+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:46.760448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.760448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306065+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"562'11974","reported_seq":12329,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.449464+0000","last_change":"2024-02-05T07:36:33.750052+0000","last_active":"2024-02-05T07:37:15.449464+0000","last_peered":"2024-02-05T07:37:15.449464+0000","last_clean":"2024-02-05T07:37:15.449464+0000","last_became_active":"2024-02-05T07:36:33.702298+0000","last_became_peered":"2024-02-05T07:36:33.702298+0000","last_unstale":"2024-02-05T07:37:15.449464+0000","last_undegraded":"2024-02-05T07:37:15.449464+0000","last_fullsized":"2024-02-05T07:37:15.449464+0000","mapping_epoch":536,"log_start":"501'8276","ondisk_log_start":"501'8276","created":423,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'8102","last_scrub_stamp":"2024-02-05T07:35:21.544679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:21.544679+0000","objects_scrubbed":3995,"log_size":3698,"log_dups_size":0,"ondisk_log_size":3698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":420,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8060928,"num_objects":123,"num_object_clones":0,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3872,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":1137,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614188+0000","last_change":"2024-02-05T07:37:08.476697+0000","last_active":"2024-02-05T07:37:14.614188+0000","last_peered":"2024-02-05T07:37:14.614188+0000","last_clean":"2024-02-05T07:37:14.614188+0000","last_became_active":"2024-02-05T07:37:06.996589+0000","last_became_peered":"2024-02-05T07:37:06.996589+0000","last_unstale":"2024-02-05T07:37:14.614188+0000","last_undegraded":"2024-02-05T07:37:14.614188+0000","last_fullsized":"2024-02-05T07:37:14.614188+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:08.476588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:08.476588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:09.252202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":63,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.630153+0000","last_change":"2024-02-05T07:36:32.754330+0000","last_active":"2024-02-05T07:37:14.630153+0000","last_peered":"2024-02-05T07:37:14.630153+0000","last_clean":"2024-02-05T07:37:14.630153+0000","last_became_active":"2024-02-05T07:36:32.753264+0000","last_became_peered":"2024-02-05T07:36:32.753264+0000","last_unstale":"2024-02-05T07:37:14.630153+0000","last_undegraded":"2024-02-05T07:37:14.630153+0000","last_fullsized":"2024-02-05T07:37:14.630153+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:23.118569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:23.118569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"326'22","reported_seq":1109,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618256+0000","last_change":"2024-02-05T07:33:30.327114+0000","last_active":"2024-02-05T07:37:14.618256+0000","last_peered":"2024-02-05T07:37:14.618256+0000","last_clean":"2024-02-05T07:37:14.618256+0000","last_became_active":"2024-02-05T07:30:40.494460+0000","last_became_peered":"2024-02-05T07:30:40.494460+0000","last_unstale":"2024-02-05T07:37:14.618256+0000","last_undegraded":"2024-02-05T07:37:14.618256+0000","last_fullsized":"2024-02-05T07:37:14.618256+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'22","last_scrub_stamp":"2024-02-05T07:33:30.327027+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.327027+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.697132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":63,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.631455+0000","last_change":"2024-02-05T07:36:32.734528+0000","last_active":"2024-02-05T07:37:14.631455+0000","last_peered":"2024-02-05T07:37:14.631455+0000","last_clean":"2024-02-05T07:37:14.631455+0000","last_became_active":"2024-02-05T07:36:32.733822+0000","last_became_peered":"2024-02-05T07:36:32.733822+0000","last_unstale":"2024-02-05T07:37:14.631455+0000","last_undegraded":"2024-02-05T07:37:14.631455+0000","last_fullsized":"2024-02-05T07:37:14.631455+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:42.095953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:33:42.095953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":67,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.624269+0000","last_change":"2024-02-05T07:36:32.849750+0000","last_active":"2024-02-05T07:37:14.624269+0000","last_peered":"2024-02-05T07:37:14.624269+0000","last_clean":"2024-02-05T07:37:14.624269+0000","last_became_active":"2024-02-05T07:36:32.781195+0000","last_became_peered":"2024-02-05T07:36:32.781195+0000","last_unstale":"2024-02-05T07:37:14.624269+0000","last_undegraded":"2024-02-05T07:37:14.624269+0000","last_fullsized":"2024-02-05T07:37:14.624269+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:34:26.591385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:26.591385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":935,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618206+0000","last_change":"2024-02-05T07:36:32.719020+0000","last_active":"2024-02-05T07:37:14.618206+0000","last_peered":"2024-02-05T07:37:14.618206+0000","last_clean":"2024-02-05T07:37:14.618206+0000","last_became_active":"2024-02-05T07:36:32.718684+0000","last_became_peered":"2024-02-05T07:36:32.718684+0000","last_unstale":"2024-02-05T07:37:14.618206+0000","last_undegraded":"2024-02-05T07:37:14.618206+0000","last_fullsized":"2024-02-05T07:37:14.618206+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:18.035057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:35:18.035057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.527205+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"562'16138","reported_seq":16472,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:18.218327+0000","last_change":"2024-02-05T07:36:54.188309+0000","last_active":"2024-02-05T07:37:18.218327+0000","last_peered":"2024-02-05T07:37:18.218327+0000","last_clean":"2024-02-05T07:37:18.218327+0000","last_became_active":"2024-02-05T07:36:32.709197+0000","last_became_peered":"2024-02-05T07:36:32.709197+0000","last_unstale":"2024-02-05T07:37:18.218327+0000","last_undegraded":"2024-02-05T07:37:18.218327+0000","last_fullsized":"2024-02-05T07:37:18.218327+0000","mapping_epoch":535,"log_start":"531'12551","ondisk_log_start":"531'12551","created":423,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'14245","last_scrub_stamp":"2024-02-05T07:36:54.188147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:54.188147+0000","objects_scrubbed":1922,"log_size":3587,"log_dups_size":203,"ondisk_log_size":3587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:12.783013+0000","scrub_duration":748,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16138,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":402,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.631210+0000","last_change":"2024-02-05T07:37:08.283685+0000","last_active":"2024-02-05T07:37:14.631210+0000","last_peered":"2024-02-05T07:37:14.631210+0000","last_clean":"2024-02-05T07:37:14.631210+0000","last_became_active":"2024-02-05T07:37:06.985122+0000","last_became_peered":"2024-02-05T07:37:06.985122+0000","last_unstale":"2024-02-05T07:37:14.631210+0000","last_undegraded":"2024-02-05T07:37:14.631210+0000","last_fullsized":"2024-02-05T07:37:14.631210+0000","mapping_epoch":553,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:08.283492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:08.283492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:11.651281+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":792,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.626037+0000","last_change":"2024-02-05T07:34:37.540422+0000","last_active":"2024-02-05T07:37:14.626037+0000","last_peered":"2024-02-05T07:37:14.626037+0000","last_clean":"2024-02-05T07:37:14.626037+0000","last_became_active":"2024-02-05T07:24:34.234322+0000","last_became_peered":"2024-02-05T07:24:34.234322+0000","last_unstale":"2024-02-05T07:37:14.626037+0000","last_undegraded":"2024-02-05T07:37:14.626037+0000","last_fullsized":"2024-02-05T07:37:14.626037+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:37.540304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:34:37.540304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.697187+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"550'56","reported_seq":112,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.626915+0000","last_change":"2024-02-05T07:36:56.264458+0000","last_active":"2024-02-05T07:37:14.626915+0000","last_peered":"2024-02-05T07:37:14.626915+0000","last_clean":"2024-02-05T07:37:14.626915+0000","last_became_active":"2024-02-05T07:36:32.738270+0000","last_became_peered":"2024-02-05T07:36:32.738270+0000","last_unstale":"2024-02-05T07:37:14.626915+0000","last_undegraded":"2024-02-05T07:37:14.626915+0000","last_fullsized":"2024-02-05T07:37:14.626915+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'56","last_scrub_stamp":"2024-02-05T07:36:56.264300+0000","last_deep_scrub":"550'56","last_deep_scrub_stamp":"2024-02-05T07:36:56.264300+0000","last_clean_scrub_stamp":"2024-02-05T07:36:56.264300+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:13.976837+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":95,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614138+0000","last_change":"2024-02-05T07:36:46.455943+0000","last_active":"2024-02-05T07:37:14.614138+0000","last_peered":"2024-02-05T07:37:14.614138+0000","last_clean":"2024-02-05T07:37:14.614138+0000","last_became_active":"2024-02-05T07:36:16.721809+0000","last_became_peered":"2024-02-05T07:36:16.721809+0000","last_unstale":"2024-02-05T07:37:14.614138+0000","last_undegraded":"2024-02-05T07:37:14.614138+0000","last_fullsized":"2024-02-05T07:37:14.614138+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:46.455867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.455867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:11.056797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":961,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.615712+0000","last_change":"2024-02-05T07:33:43.473208+0000","last_active":"2024-02-05T07:37:14.615712+0000","last_peered":"2024-02-05T07:37:14.615712+0000","last_clean":"2024-02-05T07:37:14.615712+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:37:14.615712+0000","last_undegraded":"2024-02-05T07:37:14.615712+0000","last_fullsized":"2024-02-05T07:37:14.615712+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:33:43.473107+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:33:43.473107+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.697162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":633,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618631+0000","last_change":"2024-02-05T07:34:26.897153+0000","last_active":"2024-02-05T07:37:14.618631+0000","last_peered":"2024-02-05T07:37:14.618631+0000","last_clean":"2024-02-05T07:37:14.618631+0000","last_became_active":"2024-02-05T07:30:40.508071+0000","last_became_peered":"2024-02-05T07:30:40.508071+0000","last_unstale":"2024-02-05T07:37:14.618631+0000","last_undegraded":"2024-02-05T07:37:14.618631+0000","last_fullsized":"2024-02-05T07:37:14.618631+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:26.897028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:26.897028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306311+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":330,"reported_epoch":562,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:37:14.628257+0000","last_change":"2024-02-05T07:37:10.027341+0000","last_active":"2024-02-05T07:37:14.628257+0000","last_peered":"2024-02-05T07:37:14.628257+0000","last_clean":"2024-02-05T07:19:30.999323+0000","last_became_active":"2024-02-05T07:37:10.017622+0000","last_became_peered":"2024-02-05T07:37:10.017622+0000","last_unstale":"2024-02-05T07:37:14.628257+0000","last_undegraded":"2024-02-05T07:37:14.628257+0000","last_fullsized":"2024-02-05T07:37:14.628257+0000","mapping_epoch":558,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:36:32.497664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:36:32.497664+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":4,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[0,3,6],"avail_no_missing":["0","2","3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"562'11778","reported_seq":16126,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.472527+0000","last_change":"2024-02-05T07:36:48.809415+0000","last_active":"2024-02-05T07:37:15.472527+0000","last_peered":"2024-02-05T07:37:15.472527+0000","last_clean":"2024-02-05T07:37:15.472527+0000","last_became_active":"2024-02-05T07:36:32.767092+0000","last_became_peered":"2024-02-05T07:36:32.767092+0000","last_unstale":"2024-02-05T07:37:15.472527+0000","last_undegraded":"2024-02-05T07:37:15.472527+0000","last_fullsized":"2024-02-05T07:37:15.472527+0000","mapping_epoch":535,"log_start":"499'8084","ondisk_log_start":"499'8084","created":423,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'10722","last_scrub_stamp":"2024-02-05T07:36:48.809151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:48.809151+0000","objects_scrubbed":1168,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:09.630030+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7012352,"num_objects":107,"num_object_clones":0,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1061,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"326'15","reported_seq":834,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.617545+0000","last_change":"2024-02-05T07:35:12.777278+0000","last_active":"2024-02-05T07:37:14.617545+0000","last_peered":"2024-02-05T07:37:14.617545+0000","last_clean":"2024-02-05T07:37:14.617545+0000","last_became_active":"2024-02-05T07:24:34.250146+0000","last_became_peered":"2024-02-05T07:24:34.250146+0000","last_unstale":"2024-02-05T07:37:14.617545+0000","last_undegraded":"2024-02-05T07:37:14.617545+0000","last_fullsized":"2024-02-05T07:37:14.617545+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:35:12.776983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:12.776983+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.697360+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":630,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.625389+0000","last_change":"2024-02-05T07:36:54.146303+0000","last_active":"2024-02-05T07:37:14.625389+0000","last_peered":"2024-02-05T07:37:14.625389+0000","last_clean":"2024-02-05T07:37:14.625389+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:37:14.625389+0000","last_undegraded":"2024-02-05T07:37:14.625389+0000","last_fullsized":"2024-02-05T07:37:14.625389+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:54.146194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:54.146194+0000","last_clean_scrub_stamp":"2024-02-05T07:36:54.146194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:23.763260+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":63,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.630880+0000","last_change":"2024-02-05T07:36:32.723290+0000","last_active":"2024-02-05T07:37:14.630880+0000","last_peered":"2024-02-05T07:37:14.630880+0000","last_clean":"2024-02-05T07:37:14.630880+0000","last_became_active":"2024-02-05T07:36:32.722610+0000","last_became_peered":"2024-02-05T07:36:32.722610+0000","last_unstale":"2024-02-05T07:37:14.630880+0000","last_undegraded":"2024-02-05T07:37:14.630880+0000","last_fullsized":"2024-02-05T07:37:14.630880+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:47.161343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:32:47.161343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":71,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.631110+0000","last_change":"2024-02-05T07:37:00.310446+0000","last_active":"2024-02-05T07:37:14.631110+0000","last_peered":"2024-02-05T07:37:14.631110+0000","last_clean":"2024-02-05T07:37:14.631110+0000","last_became_active":"2024-02-05T07:36:32.734230+0000","last_became_peered":"2024-02-05T07:36:32.734230+0000","last_unstale":"2024-02-05T07:37:14.631110+0000","last_undegraded":"2024-02-05T07:37:14.631110+0000","last_fullsized":"2024-02-05T07:37:14.631110+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:00.309850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:00.309850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:19.733704+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":1007,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.617720+0000","last_change":"2024-02-05T07:37:12.029853+0000","last_active":"2024-02-05T07:37:14.617720+0000","last_peered":"2024-02-05T07:37:14.617720+0000","last_clean":"2024-02-05T07:37:14.617720+0000","last_became_active":"2024-02-05T07:37:12.029331+0000","last_became_peered":"2024-02-05T07:37:12.029331+0000","last_unstale":"2024-02-05T07:37:14.617720+0000","last_undegraded":"2024-02-05T07:37:14.617720+0000","last_fullsized":"2024-02-05T07:37:14.617720+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:29.214474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:29.214474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:17.557594+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.4","version":"562'12027","reported_seq":12372,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.370261+0000","last_change":"2024-02-05T07:35:20.550097+0000","last_active":"2024-02-05T07:37:15.370261+0000","last_peered":"2024-02-05T07:37:15.370261+0000","last_clean":"2024-02-05T07:37:15.370261+0000","last_became_active":"2024-02-05T07:35:18.800109+0000","last_became_peered":"2024-02-05T07:35:18.800109+0000","last_unstale":"2024-02-05T07:37:15.370261+0000","last_undegraded":"2024-02-05T07:37:15.370261+0000","last_fullsized":"2024-02-05T07:37:15.370261+0000","mapping_epoch":423,"log_start":"508'8916","ondisk_log_start":"508'8916","created":423,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'8142","last_scrub_stamp":"2024-02-05T07:35:20.549958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:20.549958+0000","objects_scrubbed":3994,"log_size":3111,"log_dups_size":0,"ondisk_log_size":3111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.555374+0000","scrub_duration":565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3885,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"326'35","reported_seq":987,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618471+0000","last_change":"2024-02-05T07:36:16.812273+0000","last_active":"2024-02-05T07:37:14.618471+0000","last_peered":"2024-02-05T07:37:14.618471+0000","last_clean":"2024-02-05T07:37:14.618471+0000","last_became_active":"2024-02-05T07:36:16.736226+0000","last_became_peered":"2024-02-05T07:36:16.736226+0000","last_unstale":"2024-02-05T07:37:14.618471+0000","last_undegraded":"2024-02-05T07:37:14.618471+0000","last_fullsized":"2024-02-05T07:37:14.618471+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:35:46.529276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:46.529276+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.527223+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":933,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618126+0000","last_change":"2024-02-05T07:35:15.914525+0000","last_active":"2024-02-05T07:37:14.618126+0000","last_peered":"2024-02-05T07:37:14.618126+0000","last_clean":"2024-02-05T07:37:14.618126+0000","last_became_active":"2024-02-05T07:30:40.507705+0000","last_became_peered":"2024-02-05T07:30:40.507705+0000","last_unstale":"2024-02-05T07:37:14.618126+0000","last_undegraded":"2024-02-05T07:37:14.618126+0000","last_fullsized":"2024-02-05T07:37:14.618126+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:15.914445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_clean_scrub_stamp":"2024-02-05T07:35:15.914445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.527140+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":63,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.627911+0000","last_change":"2024-02-05T07:36:32.733809+0000","last_active":"2024-02-05T07:37:14.627911+0000","last_peered":"2024-02-05T07:37:14.627911+0000","last_clean":"2024-02-05T07:37:14.627911+0000","last_became_active":"2024-02-05T07:36:32.733494+0000","last_became_peered":"2024-02-05T07:36:32.733494+0000","last_unstale":"2024-02-05T07:37:14.627911+0000","last_undegraded":"2024-02-05T07:37:14.627911+0000","last_fullsized":"2024-02-05T07:37:14.627911+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.121427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.121427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298629+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":63,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.624682+0000","last_change":"2024-02-05T07:36:32.716609+0000","last_active":"2024-02-05T07:37:14.624682+0000","last_peered":"2024-02-05T07:37:14.624682+0000","last_clean":"2024-02-05T07:37:14.624682+0000","last_became_active":"2024-02-05T07:36:32.715956+0000","last_became_peered":"2024-02-05T07:36:32.715956+0000","last_unstale":"2024-02-05T07:37:14.624682+0000","last_undegraded":"2024-02-05T07:37:14.624682+0000","last_fullsized":"2024-02-05T07:37:14.624682+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:31.032967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:48.262537+0000","last_clean_scrub_stamp":"2024-02-05T07:33:31.032967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":1374,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.624419+0000","last_change":"2024-02-05T07:37:12.037281+0000","last_active":"2024-02-05T07:37:14.624419+0000","last_peered":"2024-02-05T07:37:14.624419+0000","last_clean":"2024-02-05T07:37:14.624419+0000","last_became_active":"2024-02-05T07:37:12.035736+0000","last_became_peered":"2024-02-05T07:37:12.035736+0000","last_unstale":"2024-02-05T07:37:14.624419+0000","last_undegraded":"2024-02-05T07:37:14.624419+0000","last_fullsized":"2024-02-05T07:37:14.624419+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:28.874337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.874337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:17.261581+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":436907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.7","version":"562'12024","reported_seq":12383,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:18.251184+0000","last_change":"2024-02-05T07:36:32.761603+0000","last_active":"2024-02-05T07:37:18.251184+0000","last_peered":"2024-02-05T07:37:18.251184+0000","last_clean":"2024-02-05T07:37:18.251184+0000","last_became_active":"2024-02-05T07:36:32.750071+0000","last_became_peered":"2024-02-05T07:36:32.750071+0000","last_unstale":"2024-02-05T07:37:18.251184+0000","last_undegraded":"2024-02-05T07:37:18.251184+0000","last_fullsized":"2024-02-05T07:37:18.251184+0000","mapping_epoch":535,"log_start":"507'8523","ondisk_log_start":"507'8523","created":423,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8069","last_scrub_stamp":"2024-02-05T07:35:20.035412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:20.035412+0000","objects_scrubbed":3980,"log_size":3501,"log_dups_size":0,"ondisk_log_size":3501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:20.648671+0000","scrub_duration":1162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3980,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":63,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.624486+0000","last_change":"2024-02-05T07:36:32.737113+0000","last_active":"2024-02-05T07:37:14.624486+0000","last_peered":"2024-02-05T07:37:14.624486+0000","last_clean":"2024-02-05T07:37:14.624486+0000","last_became_active":"2024-02-05T07:36:32.733003+0000","last_became_peered":"2024-02-05T07:36:32.733003+0000","last_unstale":"2024-02-05T07:37:14.624486+0000","last_undegraded":"2024-02-05T07:37:14.624486+0000","last_fullsized":"2024-02-05T07:37:14.624486+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:29.082803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:29.082803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"325'18","reported_seq":1152,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614595+0000","last_change":"2024-02-05T07:36:16.769406+0000","last_active":"2024-02-05T07:37:14.614595+0000","last_peered":"2024-02-05T07:37:14.614595+0000","last_clean":"2024-02-05T07:37:14.614595+0000","last_became_active":"2024-02-05T07:36:16.741080+0000","last_became_peered":"2024-02-05T07:36:16.741080+0000","last_unstale":"2024-02-05T07:37:14.614595+0000","last_undegraded":"2024-02-05T07:37:14.614595+0000","last_fullsized":"2024-02-05T07:37:14.614595+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'18","last_scrub_stamp":"2024-02-05T07:34:28.902809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.902809+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.555273+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":415,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.615885+0000","last_change":"2024-02-05T07:33:33.822290+0000","last_active":"2024-02-05T07:37:14.615885+0000","last_peered":"2024-02-05T07:37:14.615885+0000","last_clean":"2024-02-05T07:37:14.615885+0000","last_became_active":"2024-02-05T07:30:40.518949+0000","last_became_peered":"2024-02-05T07:30:40.518949+0000","last_unstale":"2024-02-05T07:37:14.615885+0000","last_undegraded":"2024-02-05T07:37:14.615885+0000","last_fullsized":"2024-02-05T07:37:14.615885+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:33.822199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.822199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.553614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":633,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.619864+0000","last_change":"2024-02-05T07:34:38.057821+0000","last_active":"2024-02-05T07:37:14.619864+0000","last_peered":"2024-02-05T07:37:14.619864+0000","last_clean":"2024-02-05T07:37:14.619864+0000","last_became_active":"2024-02-05T07:30:40.485991+0000","last_became_peered":"2024-02-05T07:30:40.485991+0000","last_unstale":"2024-02-05T07:37:14.619864+0000","last_undegraded":"2024-02-05T07:37:14.619864+0000","last_fullsized":"2024-02-05T07:37:14.619864+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:38.057598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.057598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306258+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":1129,"reported_epoch":562,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:37:14.614622+0000","last_change":"2024-02-05T07:37:10.028178+0000","last_active":"2024-02-05T07:37:14.614622+0000","last_peered":"2024-02-05T07:37:14.614622+0000","last_clean":"2024-02-05T07:37:06.997565+0000","last_became_active":"2024-02-05T07:37:10.019568+0000","last_became_peered":"2024-02-05T07:37:10.019568+0000","last_unstale":"2024-02-05T07:37:14.614622+0000","last_undegraded":"2024-02-05T07:37:14.614622+0000","last_fullsized":"2024-02-05T07:37:14.614622+0000","mapping_epoch":558,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:36:58.509342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:36:58.509342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,2,6],"avail_no_missing":["4","1","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.6","version":"562'11997","reported_seq":12343,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.425707+0000","last_change":"2024-02-05T07:35:21.478756+0000","last_active":"2024-02-05T07:37:15.425707+0000","last_peered":"2024-02-05T07:37:15.425707+0000","last_clean":"2024-02-05T07:37:15.425707+0000","last_became_active":"2024-02-05T07:35:18.802299+0000","last_became_peered":"2024-02-05T07:35:18.802299+0000","last_unstale":"2024-02-05T07:37:15.425707+0000","last_undegraded":"2024-02-05T07:37:15.425707+0000","last_fullsized":"2024-02-05T07:37:15.425707+0000","mapping_epoch":423,"log_start":"508'8865","ondisk_log_start":"508'8865","created":423,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8111","last_scrub_stamp":"2024-02-05T07:35:21.478656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:21.478656+0000","objects_scrubbed":4005,"log_size":3132,"log_dups_size":0,"ondisk_log_size":3132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.555410+0000","scrub_duration":465,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7798784,"num_objects":119,"num_object_clones":0,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3886,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"365'12","reported_seq":1125,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.617796+0000","last_change":"2024-02-05T07:34:40.283207+0000","last_active":"2024-02-05T07:37:14.617796+0000","last_peered":"2024-02-05T07:37:14.617796+0000","last_clean":"2024-02-05T07:37:14.617796+0000","last_became_active":"2024-02-05T07:30:40.500113+0000","last_became_peered":"2024-02-05T07:30:40.500113+0000","last_unstale":"2024-02-05T07:37:14.617796+0000","last_undegraded":"2024-02-05T07:37:14.617796+0000","last_fullsized":"2024-02-05T07:37:14.617796+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'12","last_scrub_stamp":"2024-02-05T07:34:40.283087+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:34:40.283087+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.527075+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"326'27","reported_seq":681,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.619039+0000","last_change":"2024-02-05T07:35:12.026975+0000","last_active":"2024-02-05T07:37:14.619039+0000","last_peered":"2024-02-05T07:37:14.619039+0000","last_clean":"2024-02-05T07:37:14.619039+0000","last_became_active":"2024-02-05T07:30:40.497982+0000","last_became_peered":"2024-02-05T07:30:40.497982+0000","last_unstale":"2024-02-05T07:37:14.619039+0000","last_undegraded":"2024-02-05T07:37:14.619039+0000","last_fullsized":"2024-02-05T07:37:14.619039+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:35:12.026888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:12.026888+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.527107+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":431,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.615548+0000","last_change":"2024-02-05T07:33:27.649686+0000","last_active":"2024-02-05T07:37:14.615548+0000","last_peered":"2024-02-05T07:37:14.615548+0000","last_clean":"2024-02-05T07:37:14.615548+0000","last_became_active":"2024-02-05T07:30:40.503903+0000","last_became_peered":"2024-02-05T07:30:40.503903+0000","last_unstale":"2024-02-05T07:37:14.615548+0000","last_undegraded":"2024-02-05T07:37:14.615548+0000","last_fullsized":"2024-02-05T07:37:14.615548+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:33:27.649596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:27.649596+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.553632+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":728,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618202+0000","last_change":"2024-02-05T07:37:02.526520+0000","last_active":"2024-02-05T07:37:14.618202+0000","last_peered":"2024-02-05T07:37:14.618202+0000","last_clean":"2024-02-05T07:37:14.618202+0000","last_became_active":"2024-02-05T07:36:32.701535+0000","last_became_peered":"2024-02-05T07:36:32.701535+0000","last_unstale":"2024-02-05T07:37:14.618202+0000","last_undegraded":"2024-02-05T07:37:14.618202+0000","last_fullsized":"2024-02-05T07:37:14.618202+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:02.526362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:37:02.526362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:07.050683+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":322,"reported_epoch":562,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:37:14.631672+0000","last_change":"2024-02-05T07:37:10.035545+0000","last_active":"2024-02-05T07:37:14.631672+0000","last_peered":"2024-02-05T07:37:14.631672+0000","last_clean":"2024-02-05T07:19:30.999323+0000","last_became_active":"2024-02-05T07:37:10.032758+0000","last_became_peered":"2024-02-05T07:37:10.032758+0000","last_unstale":"2024-02-05T07:37:14.631672+0000","last_undegraded":"2024-02-05T07:37:14.631672+0000","last_fullsized":"2024-02-05T07:37:14.631672+0000","mapping_epoch":558,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:35:09.073175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:35:09.073175+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":4,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[0,2,4],"avail_no_missing":["0","1","2","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.19","version":"562'11918","reported_seq":18824,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.048525+0000","last_change":"2024-02-05T07:36:33.148823+0000","last_active":"2024-02-05T07:37:15.048525+0000","last_peered":"2024-02-05T07:37:15.048525+0000","last_clean":"2024-02-05T07:37:15.048525+0000","last_became_active":"2024-02-05T07:35:59.586194+0000","last_became_peered":"2024-02-05T07:35:59.586194+0000","last_unstale":"2024-02-05T07:37:15.048525+0000","last_undegraded":"2024-02-05T07:37:15.048525+0000","last_fullsized":"2024-02-05T07:37:15.048525+0000","mapping_epoch":517,"log_start":"508'8798","ondisk_log_start":"508'8798","created":493,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":5,"last_scrub":"536'10378","last_scrub_stamp":"2024-02-05T07:36:33.148656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:33.148656+0000","objects_scrubbed":1696,"log_size":3120,"log_dups_size":0,"ondisk_log_size":3120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:41.044865+0000","scrub_duration":317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10223616,"num_objects":156,"num_object_clones":0,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1540,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"326'12","reported_seq":678,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.620865+0000","last_change":"2024-02-05T07:36:32.779445+0000","last_active":"2024-02-05T07:37:14.620865+0000","last_peered":"2024-02-05T07:37:14.620865+0000","last_clean":"2024-02-05T07:37:14.620865+0000","last_became_active":"2024-02-05T07:36:32.742483+0000","last_became_peered":"2024-02-05T07:36:32.742483+0000","last_unstale":"2024-02-05T07:37:14.620865+0000","last_undegraded":"2024-02-05T07:37:14.620865+0000","last_fullsized":"2024-02-05T07:37:14.620865+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'12","last_scrub_stamp":"2024-02-05T07:33:41.762404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.762404+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":851,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.619785+0000","last_change":"2024-02-05T07:33:51.105371+0000","last_active":"2024-02-05T07:37:14.619785+0000","last_peered":"2024-02-05T07:37:14.619785+0000","last_clean":"2024-02-05T07:37:14.619785+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:37:14.619785+0000","last_undegraded":"2024-02-05T07:37:14.619785+0000","last_fullsized":"2024-02-05T07:37:14.619785+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:33:51.105251+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:33:51.105251+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.527058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":630,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618249+0000","last_change":"2024-02-05T07:37:01.178948+0000","last_active":"2024-02-05T07:37:14.618249+0000","last_peered":"2024-02-05T07:37:14.618249+0000","last_clean":"2024-02-05T07:37:14.618249+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:37:14.618249+0000","last_undegraded":"2024-02-05T07:37:14.618249+0000","last_fullsized":"2024-02-05T07:37:14.618249+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:01.178816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:01.178816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:08.546954+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":909,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618419+0000","last_change":"2024-02-05T07:35:13.118423+0000","last_active":"2024-02-05T07:37:14.618419+0000","last_peered":"2024-02-05T07:37:14.618419+0000","last_clean":"2024-02-05T07:37:14.618419+0000","last_became_active":"2024-02-05T07:30:40.509216+0000","last_became_peered":"2024-02-05T07:30:40.509216+0000","last_unstale":"2024-02-05T07:37:14.618419+0000","last_undegraded":"2024-02-05T07:37:14.618419+0000","last_fullsized":"2024-02-05T07:37:14.618419+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:13.118311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:35:13.118311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.527155+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":813,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.619499+0000","last_change":"2024-02-05T07:37:12.043234+0000","last_active":"2024-02-05T07:37:14.619499+0000","last_peered":"2024-02-05T07:37:14.619499+0000","last_clean":"2024-02-05T07:37:14.619499+0000","last_became_active":"2024-02-05T07:37:12.042178+0000","last_became_peered":"2024-02-05T07:37:12.042178+0000","last_unstale":"2024-02-05T07:37:14.619499+0000","last_undegraded":"2024-02-05T07:37:14.619499+0000","last_fullsized":"2024-02-05T07:37:14.619499+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:31.185403+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:34:31.185403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:17.261644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.18","version":"562'11883","reported_seq":16886,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.424723+0000","last_change":"2024-02-05T07:36:49.741000+0000","last_active":"2024-02-05T07:37:15.424723+0000","last_peered":"2024-02-05T07:37:15.424723+0000","last_clean":"2024-02-05T07:37:15.424723+0000","last_became_active":"2024-02-05T07:36:46.821811+0000","last_became_peered":"2024-02-05T07:36:46.821811+0000","last_unstale":"2024-02-05T07:37:15.424723+0000","last_undegraded":"2024-02-05T07:37:15.424723+0000","last_fullsized":"2024-02-05T07:37:15.424723+0000","mapping_epoch":546,"log_start":"498'8187","ondisk_log_start":"498'8187","created":493,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":5,"last_scrub":"548'10909","last_scrub_stamp":"2024-02-05T07:36:49.740861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:49.740861+0000","objects_scrubbed":1104,"log_size":3696,"log_dups_size":0,"ondisk_log_size":3696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:07.756136+0000","scrub_duration":492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8192000,"num_objects":125,"num_object_clones":0,"num_object_copies":375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":979,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"325'15","reported_seq":686,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618534+0000","last_change":"2024-02-05T07:36:32.807878+0000","last_active":"2024-02-05T07:37:14.618534+0000","last_peered":"2024-02-05T07:37:14.618534+0000","last_clean":"2024-02-05T07:37:14.618534+0000","last_became_active":"2024-02-05T07:36:32.747221+0000","last_became_peered":"2024-02-05T07:36:32.747221+0000","last_unstale":"2024-02-05T07:37:14.618534+0000","last_undegraded":"2024-02-05T07:37:14.618534+0000","last_fullsized":"2024-02-05T07:37:14.618534+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:33:45.711399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:45.711399+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306205+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":922,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614345+0000","last_change":"2024-02-05T07:36:45.476354+0000","last_active":"2024-02-05T07:37:14.614345+0000","last_peered":"2024-02-05T07:37:14.614345+0000","last_clean":"2024-02-05T07:37:14.614345+0000","last_became_active":"2024-02-05T07:30:40.497956+0000","last_became_peered":"2024-02-05T07:30:40.497956+0000","last_unstale":"2024-02-05T07:37:14.614345+0000","last_undegraded":"2024-02-05T07:37:14.614345+0000","last_fullsized":"2024-02-05T07:37:14.614345+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:45.476243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:36:45.476243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:03.697650+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":625,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.621012+0000","last_change":"2024-02-05T07:33:50.723381+0000","last_active":"2024-02-05T07:37:14.621012+0000","last_peered":"2024-02-05T07:37:14.621012+0000","last_clean":"2024-02-05T07:37:14.621012+0000","last_became_active":"2024-02-05T07:30:40.496076+0000","last_became_peered":"2024-02-05T07:30:40.496076+0000","last_unstale":"2024-02-05T07:37:14.621012+0000","last_undegraded":"2024-02-05T07:37:14.621012+0000","last_fullsized":"2024-02-05T07:37:14.621012+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_clean_scrub_stamp":"2024-02-05T07:33:50.723212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306147+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":63,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.627206+0000","last_change":"2024-02-05T07:36:32.739391+0000","last_active":"2024-02-05T07:37:14.627206+0000","last_peered":"2024-02-05T07:37:14.627206+0000","last_clean":"2024-02-05T07:37:14.627206+0000","last_became_active":"2024-02-05T07:36:32.739115+0000","last_became_peered":"2024-02-05T07:36:32.739115+0000","last_unstale":"2024-02-05T07:37:14.627206+0000","last_undegraded":"2024-02-05T07:37:14.627206+0000","last_fullsized":"2024-02-05T07:37:14.627206+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:48.164259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:32:48.164259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298550+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":806,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.617198+0000","last_change":"2024-02-05T07:37:12.038750+0000","last_active":"2024-02-05T07:37:14.617198+0000","last_peered":"2024-02-05T07:37:14.617198+0000","last_clean":"2024-02-05T07:37:14.617198+0000","last_became_active":"2024-02-05T07:37:12.038301+0000","last_became_peered":"2024-02-05T07:37:12.038301+0000","last_unstale":"2024-02-05T07:37:14.617198+0000","last_undegraded":"2024-02-05T07:37:14.617198+0000","last_fullsized":"2024-02-05T07:37:14.617198+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:33:48.221845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:17.557625+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"326'19","reported_seq":981,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.624046+0000","last_change":"2024-02-05T07:37:06.319292+0000","last_active":"2024-02-05T07:37:14.624046+0000","last_peered":"2024-02-05T07:37:14.624046+0000","last_clean":"2024-02-05T07:37:14.624046+0000","last_became_active":"2024-02-05T07:30:40.483700+0000","last_became_peered":"2024-02-05T07:30:40.483700+0000","last_unstale":"2024-02-05T07:37:14.624046+0000","last_undegraded":"2024-02-05T07:37:14.624046+0000","last_fullsized":"2024-02-05T07:37:14.624046+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:37:06.318619+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:37:06.318619+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:27.781398+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":795,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.619627+0000","last_change":"2024-02-05T07:36:46.059245+0000","last_active":"2024-02-05T07:37:14.619627+0000","last_peered":"2024-02-05T07:37:14.619627+0000","last_clean":"2024-02-05T07:37:14.619627+0000","last_became_active":"2024-02-05T07:30:40.476058+0000","last_became_peered":"2024-02-05T07:30:40.476058+0000","last_unstale":"2024-02-05T07:37:14.619627+0000","last_undegraded":"2024-02-05T07:37:14.619627+0000","last_fullsized":"2024-02-05T07:37:14.619627+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:46.059129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.059129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:48.902775+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"538'13","reported_seq":773,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.625118+0000","last_change":"2024-02-05T07:36:32.782551+0000","last_active":"2024-02-05T07:37:14.625118+0000","last_peered":"2024-02-05T07:37:14.625118+0000","last_clean":"2024-02-05T07:37:14.625118+0000","last_became_active":"2024-02-05T07:36:32.746270+0000","last_became_peered":"2024-02-05T07:36:32.746270+0000","last_unstale":"2024-02-05T07:37:14.625118+0000","last_undegraded":"2024-02-05T07:37:14.625118+0000","last_fullsized":"2024-02-05T07:37:14.625118+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:33:39.734129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:39.734129+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1043,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.615614+0000","last_change":"2024-02-05T07:36:32.710166+0000","last_active":"2024-02-05T07:37:14.615614+0000","last_peered":"2024-02-05T07:37:14.615614+0000","last_clean":"2024-02-05T07:37:14.615614+0000","last_became_active":"2024-02-05T07:36:32.709654+0000","last_became_peered":"2024-02-05T07:36:32.709654+0000","last_unstale":"2024-02-05T07:37:14.615614+0000","last_undegraded":"2024-02-05T07:37:14.615614+0000","last_fullsized":"2024-02-05T07:37:14.615614+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:38.879177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:38.879177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.553715+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":388,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618005+0000","last_change":"2024-02-05T07:37:12.040789+0000","last_active":"2024-02-05T07:37:14.618005+0000","last_peered":"2024-02-05T07:37:14.618005+0000","last_clean":"2024-02-05T07:37:14.618005+0000","last_became_active":"2024-02-05T07:37:12.040170+0000","last_became_peered":"2024-02-05T07:37:12.040170+0000","last_unstale":"2024-02-05T07:37:14.618005+0000","last_undegraded":"2024-02-05T07:37:14.618005+0000","last_fullsized":"2024-02-05T07:37:14.618005+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:33:41.082046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.082046+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:17.261614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":9,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"326'25","reported_seq":708,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.625283+0000","last_change":"2024-02-05T07:36:59.117388+0000","last_active":"2024-02-05T07:37:14.625283+0000","last_peered":"2024-02-05T07:37:14.625283+0000","last_clean":"2024-02-05T07:37:14.625283+0000","last_became_active":"2024-02-05T07:36:32.746757+0000","last_became_peered":"2024-02-05T07:36:32.746757+0000","last_unstale":"2024-02-05T07:37:14.625283+0000","last_undegraded":"2024-02-05T07:37:14.625283+0000","last_fullsized":"2024-02-05T07:37:14.625283+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:36:59.117218+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.117218+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:10.216074+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":428,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614533+0000","last_change":"2024-02-05T07:36:32.702570+0000","last_active":"2024-02-05T07:37:14.614533+0000","last_peered":"2024-02-05T07:37:14.614533+0000","last_clean":"2024-02-05T07:37:14.614533+0000","last_became_active":"2024-02-05T07:36:32.702150+0000","last_became_peered":"2024-02-05T07:36:32.702150+0000","last_unstale":"2024-02-05T07:37:14.614533+0000","last_undegraded":"2024-02-05T07:37:14.614533+0000","last_fullsized":"2024-02-05T07:37:14.614533+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:32.854871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:32.854871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.555255+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"538'11","reported_seq":76,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.627658+0000","last_change":"2024-02-05T07:36:32.883755+0000","last_active":"2024-02-05T07:37:14.627658+0000","last_peered":"2024-02-05T07:37:14.627658+0000","last_clean":"2024-02-05T07:37:14.627658+0000","last_became_active":"2024-02-05T07:36:32.745294+0000","last_became_peered":"2024-02-05T07:36:32.745294+0000","last_unstale":"2024-02-05T07:37:14.627658+0000","last_undegraded":"2024-02-05T07:37:14.627658+0000","last_fullsized":"2024-02-05T07:37:14.627658+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9","last_scrub_stamp":"2024-02-05T07:34:27.946504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:34:27.946504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":783,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618394+0000","last_change":"2024-02-05T07:34:22.810896+0000","last_active":"2024-02-05T07:37:14.618394+0000","last_peered":"2024-02-05T07:37:14.618394+0000","last_clean":"2024-02-05T07:37:14.618394+0000","last_became_active":"2024-02-05T07:30:40.507831+0000","last_became_peered":"2024-02-05T07:30:40.507831+0000","last_unstale":"2024-02-05T07:37:14.618394+0000","last_undegraded":"2024-02-05T07:37:14.618394+0000","last_fullsized":"2024-02-05T07:37:14.618394+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:22.810759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:22.810759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306285+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"326'19","reported_seq":1109,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614552+0000","last_change":"2024-02-05T07:36:16.751485+0000","last_active":"2024-02-05T07:37:14.614552+0000","last_peered":"2024-02-05T07:37:14.614552+0000","last_clean":"2024-02-05T07:37:14.614552+0000","last_became_active":"2024-02-05T07:36:16.701055+0000","last_became_peered":"2024-02-05T07:36:16.701055+0000","last_unstale":"2024-02-05T07:37:14.614552+0000","last_undegraded":"2024-02-05T07:37:14.614552+0000","last_fullsized":"2024-02-05T07:37:14.614552+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:32:31.667882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:32:31.667882+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.553524+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1064,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614611+0000","last_change":"2024-02-05T07:36:32.702949+0000","last_active":"2024-02-05T07:37:14.614611+0000","last_peered":"2024-02-05T07:37:14.614611+0000","last_clean":"2024-02-05T07:37:14.614611+0000","last_became_active":"2024-02-05T07:36:32.702585+0000","last_became_peered":"2024-02-05T07:36:32.702585+0000","last_unstale":"2024-02-05T07:37:14.614611+0000","last_undegraded":"2024-02-05T07:37:14.614611+0000","last_fullsized":"2024-02-05T07:37:14.614611+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:31.892895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:34:31.892895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.555307+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1052,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.616326+0000","last_change":"2024-02-05T07:32:48.346195+0000","last_active":"2024-02-05T07:37:14.616326+0000","last_peered":"2024-02-05T07:37:14.616326+0000","last_clean":"2024-02-05T07:37:14.616326+0000","last_became_active":"2024-02-05T07:30:40.487141+0000","last_became_peered":"2024-02-05T07:30:40.487141+0000","last_unstale":"2024-02-05T07:37:14.616326+0000","last_undegraded":"2024-02-05T07:37:14.616326+0000","last_fullsized":"2024-02-05T07:37:14.616326+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_clean_scrub_stamp":"2024-02-05T07:32:48.345965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.696999+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":629,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618551+0000","last_change":"2024-02-05T07:36:32.742746+0000","last_active":"2024-02-05T07:37:14.618551+0000","last_peered":"2024-02-05T07:37:14.618551+0000","last_clean":"2024-02-05T07:37:14.618551+0000","last_became_active":"2024-02-05T07:36:32.742150+0000","last_became_peered":"2024-02-05T07:36:32.742150+0000","last_unstale":"2024-02-05T07:37:14.618551+0000","last_undegraded":"2024-02-05T07:37:14.618551+0000","last_fullsized":"2024-02-05T07:37:14.618551+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.735757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.735757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.305953+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"325'15","reported_seq":698,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.615488+0000","last_change":"2024-02-05T07:36:32.815601+0000","last_active":"2024-02-05T07:37:14.615488+0000","last_peered":"2024-02-05T07:37:14.615488+0000","last_clean":"2024-02-05T07:37:14.615488+0000","last_became_active":"2024-02-05T07:36:32.710090+0000","last_became_peered":"2024-02-05T07:36:32.710090+0000","last_unstale":"2024-02-05T07:37:14.615488+0000","last_undegraded":"2024-02-05T07:37:14.615488+0000","last_fullsized":"2024-02-05T07:37:14.615488+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:35:11.749389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:11.749389+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.697390+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":840,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.621127+0000","last_change":"2024-02-05T07:36:32.120486+0000","last_active":"2024-02-05T07:37:14.621127+0000","last_peered":"2024-02-05T07:37:14.621127+0000","last_clean":"2024-02-05T07:37:14.621127+0000","last_became_active":"2024-02-05T07:30:40.488789+0000","last_became_peered":"2024-02-05T07:30:40.488789+0000","last_unstale":"2024-02-05T07:37:14.621127+0000","last_undegraded":"2024-02-05T07:37:14.621127+0000","last_fullsized":"2024-02-05T07:37:14.621127+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:32.120410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:32.120410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:44.650252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":619,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.620796+0000","last_change":"2024-02-05T07:33:47.684589+0000","last_active":"2024-02-05T07:37:14.620796+0000","last_peered":"2024-02-05T07:37:14.620796+0000","last_clean":"2024-02-05T07:37:14.620796+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:37:14.620796+0000","last_undegraded":"2024-02-05T07:37:14.620796+0000","last_fullsized":"2024-02-05T07:37:14.620796+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:47.684508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.684508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":505,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618835+0000","last_change":"2024-02-05T07:36:43.520812+0000","last_active":"2024-02-05T07:37:14.618835+0000","last_peered":"2024-02-05T07:37:14.618835+0000","last_clean":"2024-02-05T07:37:14.618835+0000","last_became_active":"2024-02-05T07:34:30.670179+0000","last_became_peered":"2024-02-05T07:34:30.670179+0000","last_unstale":"2024-02-05T07:37:14.618835+0000","last_undegraded":"2024-02-05T07:37:14.618835+0000","last_fullsized":"2024-02-05T07:37:14.618835+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:43.520666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:36:43.520666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:00.289525+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":1370,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.624539+0000","last_change":"2024-02-05T07:37:12.043490+0000","last_active":"2024-02-05T07:37:14.624539+0000","last_peered":"2024-02-05T07:37:14.624539+0000","last_clean":"2024-02-05T07:37:14.624539+0000","last_became_active":"2024-02-05T07:37:12.043091+0000","last_became_peered":"2024-02-05T07:37:12.043091+0000","last_unstale":"2024-02-05T07:37:14.624539+0000","last_undegraded":"2024-02-05T07:37:14.624539+0000","last_fullsized":"2024-02-05T07:37:14.624539+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:28.874337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.874337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:17.329145+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":109227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"326'19","reported_seq":672,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618822+0000","last_change":"2024-02-05T07:36:48.088544+0000","last_active":"2024-02-05T07:37:14.618822+0000","last_peered":"2024-02-05T07:37:14.618822+0000","last_clean":"2024-02-05T07:37:14.618822+0000","last_became_active":"2024-02-05T07:30:40.501893+0000","last_became_peered":"2024-02-05T07:30:40.501893+0000","last_unstale":"2024-02-05T07:37:14.618822+0000","last_undegraded":"2024-02-05T07:37:14.618822+0000","last_fullsized":"2024-02-05T07:37:14.618822+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:36:48.088406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:36:48.088406+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:52.926044+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":453,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618097+0000","last_change":"2024-02-05T07:37:07.757536+0000","last_active":"2024-02-05T07:37:14.618097+0000","last_peered":"2024-02-05T07:37:14.618097+0000","last_clean":"2024-02-05T07:37:14.618097+0000","last_became_active":"2024-02-05T07:36:32.712612+0000","last_became_peered":"2024-02-05T07:36:32.712612+0000","last_unstale":"2024-02-05T07:37:14.618097+0000","last_undegraded":"2024-02-05T07:37:14.618097+0000","last_fullsized":"2024-02-05T07:37:14.618097+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:07.757307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:37:07.757307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:28.872563+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":914,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.613831+0000","last_change":"2024-02-05T07:36:42.529702+0000","last_active":"2024-02-05T07:37:14.613831+0000","last_peered":"2024-02-05T07:37:14.613831+0000","last_clean":"2024-02-05T07:37:14.613831+0000","last_became_active":"2024-02-05T07:30:40.497190+0000","last_became_peered":"2024-02-05T07:30:40.497190+0000","last_unstale":"2024-02-05T07:37:14.613831+0000","last_undegraded":"2024-02-05T07:37:14.613831+0000","last_fullsized":"2024-02-05T07:37:14.613831+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:42.529597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:36:42.529597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:05.970823+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":631,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.619774+0000","last_change":"2024-02-05T07:35:15.273201+0000","last_active":"2024-02-05T07:37:14.619774+0000","last_peered":"2024-02-05T07:37:14.619774+0000","last_clean":"2024-02-05T07:37:14.619774+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:37:14.619774+0000","last_undegraded":"2024-02-05T07:37:14.619774+0000","last_fullsized":"2024-02-05T07:37:14.619774+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:15.272861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:35:15.272861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306501+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"325'8","reported_seq":643,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.620804+0000","last_change":"2024-02-05T07:35:47.070886+0000","last_active":"2024-02-05T07:37:14.620804+0000","last_peered":"2024-02-05T07:37:14.620804+0000","last_clean":"2024-02-05T07:37:14.620804+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:37:14.620804+0000","last_undegraded":"2024-02-05T07:37:14.620804+0000","last_fullsized":"2024-02-05T07:37:14.620804+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:35:47.070805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:47.070805+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":630,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.620459+0000","last_change":"2024-02-05T07:34:39.980293+0000","last_active":"2024-02-05T07:37:14.620459+0000","last_peered":"2024-02-05T07:37:14.620459+0000","last_clean":"2024-02-05T07:37:14.620459+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:37:14.620459+0000","last_undegraded":"2024-02-05T07:37:14.620459+0000","last_fullsized":"2024-02-05T07:37:14.620459+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:39.980067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:34:39.980067+0000","last_clean_scrub_stamp":"2024-02-05T07:34:39.980067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306390+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":893,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.617659+0000","last_change":"2024-02-05T07:36:50.465196+0000","last_active":"2024-02-05T07:37:14.617659+0000","last_peered":"2024-02-05T07:37:14.617659+0000","last_clean":"2024-02-05T07:37:14.617659+0000","last_became_active":"2024-02-05T07:30:40.500403+0000","last_became_peered":"2024-02-05T07:30:40.500403+0000","last_unstale":"2024-02-05T07:37:14.617659+0000","last_undegraded":"2024-02-05T07:37:14.617659+0000","last_fullsized":"2024-02-05T07:37:14.617659+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:50.465128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:36:50.465128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:59.691916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":71,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.631139+0000","last_change":"2024-02-05T07:36:58.296703+0000","last_active":"2024-02-05T07:37:14.631139+0000","last_peered":"2024-02-05T07:37:14.631139+0000","last_clean":"2024-02-05T07:37:14.631139+0000","last_became_active":"2024-02-05T07:36:32.723725+0000","last_became_peered":"2024-02-05T07:36:32.723725+0000","last_unstale":"2024-02-05T07:37:14.631139+0000","last_undegraded":"2024-02-05T07:37:14.631139+0000","last_fullsized":"2024-02-05T07:37:14.631139+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:58.296469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:58.296469+0000","last_clean_scrub_stamp":"2024-02-05T07:36:58.296469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:06.288351+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.11","version":"562'12194","reported_seq":19523,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:18.163385+0000","last_change":"2024-02-05T07:37:09.068340+0000","last_active":"2024-02-05T07:37:18.163385+0000","last_peered":"2024-02-05T07:37:18.163385+0000","last_clean":"2024-02-05T07:37:18.163385+0000","last_became_active":"2024-02-05T07:37:09.041644+0000","last_became_peered":"2024-02-05T07:37:09.041644+0000","last_unstale":"2024-02-05T07:37:18.163385+0000","last_undegraded":"2024-02-05T07:37:18.163385+0000","last_fullsized":"2024-02-05T07:37:18.163385+0000","mapping_epoch":557,"log_start":"508'8690","ondisk_log_start":"508'8690","created":493,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3504,"log_dups_size":0,"ondisk_log_size":3504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.610019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8140,"num_write_kb":259488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2493,"num_bytes_recovered":163381248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"326'19","reported_seq":880,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618052+0000","last_change":"2024-02-05T07:35:45.529783+0000","last_active":"2024-02-05T07:37:14.618052+0000","last_peered":"2024-02-05T07:37:14.618052+0000","last_clean":"2024-02-05T07:37:14.618052+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:37:14.618052+0000","last_undegraded":"2024-02-05T07:37:14.618052+0000","last_fullsized":"2024-02-05T07:37:14.618052+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:35:45.529663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:45.529663+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.527239+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":627,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.620404+0000","last_change":"2024-02-05T07:35:11.098562+0000","last_active":"2024-02-05T07:37:14.620404+0000","last_peered":"2024-02-05T07:37:14.620404+0000","last_clean":"2024-02-05T07:37:14.620404+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:37:14.620404+0000","last_undegraded":"2024-02-05T07:37:14.620404+0000","last_fullsized":"2024-02-05T07:37:14.620404+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:11.098430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:35:11.098430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306445+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":71,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.630062+0000","last_change":"2024-02-05T07:36:46.264042+0000","last_active":"2024-02-05T07:37:14.630062+0000","last_peered":"2024-02-05T07:37:14.630062+0000","last_clean":"2024-02-05T07:37:14.630062+0000","last_became_active":"2024-02-05T07:36:32.752879+0000","last_became_peered":"2024-02-05T07:36:32.752879+0000","last_unstale":"2024-02-05T07:37:14.630062+0000","last_undegraded":"2024-02-05T07:37:14.630062+0000","last_fullsized":"2024-02-05T07:37:14.630062+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:46.263956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.263956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:54.523479+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":670,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614893+0000","last_change":"2024-02-05T07:36:34.513089+0000","last_active":"2024-02-05T07:37:14.614893+0000","last_peered":"2024-02-05T07:37:14.614893+0000","last_clean":"2024-02-05T07:37:14.614893+0000","last_became_active":"2024-02-05T07:36:32.719681+0000","last_became_peered":"2024-02-05T07:36:32.719681+0000","last_unstale":"2024-02-05T07:37:14.614893+0000","last_undegraded":"2024-02-05T07:37:14.614893+0000","last_fullsized":"2024-02-05T07:37:14.614893+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:34.512979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:36:34.512979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:44.724212+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.10","version":"562'11846","reported_seq":17373,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.439038+0000","last_change":"2024-02-05T07:36:33.609194+0000","last_active":"2024-02-05T07:37:15.439038+0000","last_peered":"2024-02-05T07:37:15.439038+0000","last_clean":"2024-02-05T07:37:15.439038+0000","last_became_active":"2024-02-05T07:36:12.645812+0000","last_became_peered":"2024-02-05T07:36:12.645812+0000","last_unstale":"2024-02-05T07:37:15.439038+0000","last_undegraded":"2024-02-05T07:37:15.439038+0000","last_fullsized":"2024-02-05T07:37:15.439038+0000","mapping_epoch":519,"log_start":"508'8707","ondisk_log_start":"508'8707","created":493,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"536'10309","last_scrub_stamp":"2024-02-05T07:36:33.609080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:33.609080+0000","objects_scrubbed":1656,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:37.442825+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7798784,"num_objects":119,"num_object_clones":0,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1537,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"326'20","reported_seq":794,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618147+0000","last_change":"2024-02-05T07:35:13.094823+0000","last_active":"2024-02-05T07:37:14.618147+0000","last_peered":"2024-02-05T07:37:14.618147+0000","last_clean":"2024-02-05T07:37:14.618147+0000","last_became_active":"2024-02-05T07:30:40.495954+0000","last_became_peered":"2024-02-05T07:30:40.495954+0000","last_unstale":"2024-02-05T07:37:14.618147+0000","last_undegraded":"2024-02-05T07:37:14.618147+0000","last_fullsized":"2024-02-05T07:37:14.618147+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:35:13.094694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:13.094694+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306531+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":1090,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.616381+0000","last_change":"2024-02-05T07:34:38.558839+0000","last_active":"2024-02-05T07:37:14.616381+0000","last_peered":"2024-02-05T07:37:14.616381+0000","last_clean":"2024-02-05T07:37:14.616381+0000","last_became_active":"2024-02-05T07:30:40.495135+0000","last_became_peered":"2024-02-05T07:30:40.495135+0000","last_unstale":"2024-02-05T07:37:14.616381+0000","last_undegraded":"2024-02-05T07:37:14.616381+0000","last_fullsized":"2024-02-05T07:37:14.616381+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:34:38.558593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.558593+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.697216+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"538'13","reported_seq":1118,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618194+0000","last_change":"2024-02-05T07:36:57.488022+0000","last_active":"2024-02-05T07:37:14.618194+0000","last_peered":"2024-02-05T07:37:14.618194+0000","last_clean":"2024-02-05T07:37:14.618194+0000","last_became_active":"2024-02-05T07:36:32.717060+0000","last_became_peered":"2024-02-05T07:36:32.717060+0000","last_unstale":"2024-02-05T07:37:14.618194+0000","last_undegraded":"2024-02-05T07:37:14.618194+0000","last_fullsized":"2024-02-05T07:37:14.618194+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'13","last_scrub_stamp":"2024-02-05T07:36:57.487913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:57.487913+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:11.818158+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":67,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.626978+0000","last_change":"2024-02-05T07:36:32.766920+0000","last_active":"2024-02-05T07:37:14.626978+0000","last_peered":"2024-02-05T07:37:14.626978+0000","last_clean":"2024-02-05T07:37:14.626978+0000","last_became_active":"2024-02-05T07:36:32.722692+0000","last_became_peered":"2024-02-05T07:36:32.722692+0000","last_unstale":"2024-02-05T07:37:14.626978+0000","last_undegraded":"2024-02-05T07:37:14.626978+0000","last_fullsized":"2024-02-05T07:37:14.626978+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:35:15.112415+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:35:15.112415+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.299194+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"562'11806","reported_seq":20335,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.484398+0000","last_change":"2024-02-05T07:36:47.882842+0000","last_active":"2024-02-05T07:37:15.484398+0000","last_peered":"2024-02-05T07:37:15.484398+0000","last_clean":"2024-02-05T07:37:15.484398+0000","last_became_active":"2024-02-05T07:36:36.623778+0000","last_became_peered":"2024-02-05T07:36:36.623778+0000","last_unstale":"2024-02-05T07:37:15.484398+0000","last_undegraded":"2024-02-05T07:37:15.484398+0000","last_fullsized":"2024-02-05T07:37:15.484398+0000","mapping_epoch":539,"log_start":"498'8116","ondisk_log_start":"498'8116","created":493,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":5,"last_scrub":"548'10747","last_scrub_stamp":"2024-02-05T07:36:47.882492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:47.882492+0000","objects_scrubbed":1168,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:51.036911+0000","scrub_duration":601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1067,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"326'23","reported_seq":652,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.619713+0000","last_change":"2024-02-05T07:34:41.935675+0000","last_active":"2024-02-05T07:37:14.619713+0000","last_peered":"2024-02-05T07:37:14.619713+0000","last_clean":"2024-02-05T07:37:14.619713+0000","last_became_active":"2024-02-05T07:27:16.747998+0000","last_became_peered":"2024-02-05T07:27:16.747998+0000","last_unstale":"2024-02-05T07:37:14.619713+0000","last_undegraded":"2024-02-05T07:37:14.619713+0000","last_fullsized":"2024-02-05T07:37:14.619713+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'23","last_scrub_stamp":"2024-02-05T07:34:41.934886+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:34:41.934886+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306420+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":71,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.631225+0000","last_change":"2024-02-05T07:36:59.269602+0000","last_active":"2024-02-05T07:37:14.631225+0000","last_peered":"2024-02-05T07:37:14.631225+0000","last_clean":"2024-02-05T07:37:14.631225+0000","last_became_active":"2024-02-05T07:36:32.748986+0000","last_became_peered":"2024-02-05T07:36:32.748986+0000","last_unstale":"2024-02-05T07:37:14.631225+0000","last_undegraded":"2024-02-05T07:37:14.631225+0000","last_fullsized":"2024-02-05T07:37:14.631225+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:59.269484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.269484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:18.654743+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1064,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614830+0000","last_change":"2024-02-05T07:36:47.437547+0000","last_active":"2024-02-05T07:37:14.614830+0000","last_peered":"2024-02-05T07:37:14.614830+0000","last_clean":"2024-02-05T07:37:14.614830+0000","last_became_active":"2024-02-05T07:36:32.702187+0000","last_became_peered":"2024-02-05T07:36:32.702187+0000","last_unstale":"2024-02-05T07:37:14.614830+0000","last_undegraded":"2024-02-05T07:37:14.614830+0000","last_fullsized":"2024-02-05T07:37:14.614830+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:47.437414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:47.437414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:15.669142+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":630,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.620663+0000","last_change":"2024-02-05T07:34:37.126181+0000","last_active":"2024-02-05T07:37:14.620663+0000","last_peered":"2024-02-05T07:37:14.620663+0000","last_clean":"2024-02-05T07:37:14.620663+0000","last_became_active":"2024-02-05T07:28:26.603067+0000","last_became_peered":"2024-02-05T07:28:26.603067+0000","last_unstale":"2024-02-05T07:37:14.620663+0000","last_undegraded":"2024-02-05T07:37:14.620663+0000","last_fullsized":"2024-02-05T07:37:14.620663+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:37.125938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:37.125938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306365+0000","scrub_duration":217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"562'12079","reported_seq":12423,"reported_epoch":562,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:37:16.295724+0000","last_change":"2024-02-05T07:37:02.845889+0000","last_active":"2024-02-05T07:37:16.295724+0000","last_peered":"2024-02-05T07:37:16.295724+0000","last_clean":"2024-02-05T07:35:19.777163+0000","last_became_active":"2024-02-05T07:36:32.807995+0000","last_became_peered":"2024-02-05T07:36:32.807995+0000","last_unstale":"2024-02-05T07:37:16.295724+0000","last_undegraded":"2024-02-05T07:37:16.295724+0000","last_fullsized":"2024-02-05T07:37:16.295724+0000","mapping_epoch":535,"log_start":"509'8901","ondisk_log_start":"509'8901","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:33:37.591098+0000","objects_scrubbed":0,"log_size":3178,"log_dups_size":0,"ondisk_log_size":3178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5046272,"num_objects":77,"num_object_clones":0,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":231,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8027,"num_write_kb":259328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[0,1,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":77}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"326'27","reported_seq":79,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.624704+0000","last_change":"2024-02-05T07:36:32.820618+0000","last_active":"2024-02-05T07:37:14.624704+0000","last_peered":"2024-02-05T07:37:14.624704+0000","last_clean":"2024-02-05T07:37:14.624704+0000","last_became_active":"2024-02-05T07:36:32.744301+0000","last_became_peered":"2024-02-05T07:36:32.744301+0000","last_unstale":"2024-02-05T07:37:14.624704+0000","last_undegraded":"2024-02-05T07:37:14.624704+0000","last_fullsized":"2024-02-05T07:37:14.624704+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:33:33.995051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:33.995051+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":797,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614437+0000","last_change":"2024-02-05T07:36:48.410254+0000","last_active":"2024-02-05T07:37:14.614437+0000","last_peered":"2024-02-05T07:37:14.614437+0000","last_clean":"2024-02-05T07:37:14.614437+0000","last_became_active":"2024-02-05T07:30:40.503627+0000","last_became_peered":"2024-02-05T07:30:40.503627+0000","last_unstale":"2024-02-05T07:37:14.614437+0000","last_undegraded":"2024-02-05T07:37:14.614437+0000","last_fullsized":"2024-02-05T07:37:14.614437+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:48.410161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:48.410161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:56.673771+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1038,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.625941+0000","last_change":"2024-02-05T07:36:49.645272+0000","last_active":"2024-02-05T07:37:14.625941+0000","last_peered":"2024-02-05T07:37:14.625941+0000","last_clean":"2024-02-05T07:37:14.625941+0000","last_became_active":"2024-02-05T07:18:53.461409+0000","last_became_peered":"2024-02-05T07:18:53.461409+0000","last_unstale":"2024-02-05T07:37:14.625941+0000","last_undegraded":"2024-02-05T07:37:14.625941+0000","last_fullsized":"2024-02-05T07:37:14.625941+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:49.645159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:49.645159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:01.985453+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1052,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.615973+0000","last_change":"2024-02-05T07:36:43.532770+0000","last_active":"2024-02-05T07:37:14.615973+0000","last_peered":"2024-02-05T07:37:14.615973+0000","last_clean":"2024-02-05T07:37:14.615973+0000","last_became_active":"2024-02-05T07:28:26.590448+0000","last_became_peered":"2024-02-05T07:28:26.590448+0000","last_unstale":"2024-02-05T07:37:14.615973+0000","last_undegraded":"2024-02-05T07:37:14.615973+0000","last_fullsized":"2024-02-05T07:37:14.615973+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:43.532678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:36:43.532678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:44.061138+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.15","version":"562'11911","reported_seq":19251,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:16.827012+0000","last_change":"2024-02-05T07:36:49.541529+0000","last_active":"2024-02-05T07:37:16.827012+0000","last_peered":"2024-02-05T07:37:16.827012+0000","last_clean":"2024-02-05T07:37:16.827012+0000","last_became_active":"2024-02-05T07:36:32.740682+0000","last_became_peered":"2024-02-05T07:36:32.740682+0000","last_unstale":"2024-02-05T07:37:16.827012+0000","last_undegraded":"2024-02-05T07:37:16.827012+0000","last_fullsized":"2024-02-05T07:37:16.827012+0000","mapping_epoch":535,"log_start":"508'8508","ondisk_log_start":"508'8508","created":493,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"548'10863","last_scrub_stamp":"2024-02-05T07:36:49.541356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:49.541356+0000","objects_scrubbed":1129,"log_size":3403,"log_dups_size":0,"ondisk_log_size":3403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:12.751424+0000","scrub_duration":417,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4915200,"num_objects":75,"num_object_clones":0,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1054,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"325'7","reported_seq":1078,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614432+0000","last_change":"2024-02-05T07:37:06.495293+0000","last_active":"2024-02-05T07:37:14.614432+0000","last_peered":"2024-02-05T07:37:14.614432+0000","last_clean":"2024-02-05T07:37:14.614432+0000","last_became_active":"2024-02-05T07:27:16.745897+0000","last_became_peered":"2024-02-05T07:27:16.745897+0000","last_unstale":"2024-02-05T07:37:14.614432+0000","last_undegraded":"2024-02-05T07:37:14.614432+0000","last_fullsized":"2024-02-05T07:37:14.614432+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'7","last_scrub_stamp":"2024-02-05T07:37:06.495165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:06.495165+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.549830+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":869,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614512+0000","last_change":"2024-02-05T07:36:54.574175+0000","last_active":"2024-02-05T07:37:14.614512+0000","last_peered":"2024-02-05T07:37:14.614512+0000","last_clean":"2024-02-05T07:37:14.614512+0000","last_became_active":"2024-02-05T07:28:26.590322+0000","last_became_peered":"2024-02-05T07:28:26.590322+0000","last_unstale":"2024-02-05T07:37:14.614512+0000","last_undegraded":"2024-02-05T07:37:14.614512+0000","last_fullsized":"2024-02-05T07:37:14.614512+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:54.574076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:54.574076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:19.258832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":416,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614539+0000","last_change":"2024-02-05T07:33:44.867890+0000","last_active":"2024-02-05T07:37:14.614539+0000","last_peered":"2024-02-05T07:37:14.614539+0000","last_clean":"2024-02-05T07:37:14.614539+0000","last_became_active":"2024-02-05T07:30:40.504365+0000","last_became_peered":"2024-02-05T07:30:40.504365+0000","last_unstale":"2024-02-05T07:37:14.614539+0000","last_undegraded":"2024-02-05T07:37:14.614539+0000","last_fullsized":"2024-02-05T07:37:14.614539+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.867764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.867764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.553699+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":857,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.616695+0000","last_change":"2024-02-05T07:36:32.710890+0000","last_active":"2024-02-05T07:37:14.616695+0000","last_peered":"2024-02-05T07:37:14.616695+0000","last_clean":"2024-02-05T07:37:14.616695+0000","last_became_active":"2024-02-05T07:36:32.710575+0000","last_became_peered":"2024-02-05T07:36:32.710575+0000","last_unstale":"2024-02-05T07:37:14.616695+0000","last_undegraded":"2024-02-05T07:37:14.616695+0000","last_fullsized":"2024-02-05T07:37:14.616695+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:15.923221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:35:15.923221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.697330+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.14","version":"562'12028","reported_seq":20466,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.956711+0000","last_change":"2024-02-05T07:36:51.112075+0000","last_active":"2024-02-05T07:37:14.956711+0000","last_peered":"2024-02-05T07:37:14.956711+0000","last_clean":"2024-02-05T07:37:14.956711+0000","last_became_active":"2024-02-05T07:36:33.699658+0000","last_became_peered":"2024-02-05T07:36:33.699658+0000","last_unstale":"2024-02-05T07:37:14.956711+0000","last_undegraded":"2024-02-05T07:37:14.956711+0000","last_fullsized":"2024-02-05T07:37:14.956711+0000","mapping_epoch":536,"log_start":"510'8909","ondisk_log_start":"510'8909","created":493,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":5,"last_scrub":"548'11028","last_scrub_stamp":"2024-02-05T07:36:51.111906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:51.111906+0000","objects_scrubbed":1153,"log_size":3119,"log_dups_size":0,"ondisk_log_size":3119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:10.899041+0000","scrub_duration":489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9306112,"num_objects":142,"num_object_clones":0,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1011,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"325'19","reported_seq":1102,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614445+0000","last_change":"2024-02-05T07:35:18.020758+0000","last_active":"2024-02-05T07:37:14.614445+0000","last_peered":"2024-02-05T07:37:14.614445+0000","last_clean":"2024-02-05T07:37:14.614445+0000","last_became_active":"2024-02-05T07:27:16.707036+0000","last_became_peered":"2024-02-05T07:27:16.707036+0000","last_unstale":"2024-02-05T07:37:14.614445+0000","last_undegraded":"2024-02-05T07:37:14.614445+0000","last_fullsized":"2024-02-05T07:37:14.614445+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'19","last_scrub_stamp":"2024-02-05T07:35:18.020665+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:35:18.020665+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.555428+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":872,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.615138+0000","last_change":"2024-02-05T07:36:39.547357+0000","last_active":"2024-02-05T07:37:14.615138+0000","last_peered":"2024-02-05T07:37:14.615138+0000","last_clean":"2024-02-05T07:37:14.615138+0000","last_became_active":"2024-02-05T07:28:26.616107+0000","last_became_peered":"2024-02-05T07:28:26.616107+0000","last_unstale":"2024-02-05T07:37:14.615138+0000","last_undegraded":"2024-02-05T07:37:14.615138+0000","last_fullsized":"2024-02-05T07:37:14.615138+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:39.547282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:39.547282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:49.619046+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":418,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614260+0000","last_change":"2024-02-05T07:35:17.049239+0000","last_active":"2024-02-05T07:37:14.614260+0000","last_peered":"2024-02-05T07:37:14.614260+0000","last_clean":"2024-02-05T07:37:14.614260+0000","last_became_active":"2024-02-05T07:30:40.491630+0000","last_became_peered":"2024-02-05T07:30:40.491630+0000","last_unstale":"2024-02-05T07:37:14.614260+0000","last_undegraded":"2024-02-05T07:37:14.614260+0000","last_fullsized":"2024-02-05T07:37:14.614260+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:17.049130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:35:17.049130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.555340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":63,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.624578+0000","last_change":"2024-02-05T07:36:32.737398+0000","last_active":"2024-02-05T07:37:14.624578+0000","last_peered":"2024-02-05T07:37:14.624578+0000","last_clean":"2024-02-05T07:37:14.624578+0000","last_became_active":"2024-02-05T07:36:32.733405+0000","last_became_peered":"2024-02-05T07:36:32.733405+0000","last_unstale":"2024-02-05T07:37:14.624578+0000","last_undegraded":"2024-02-05T07:37:14.624578+0000","last_fullsized":"2024-02-05T07:37:14.624578+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:47.484894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.484894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298817+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.17","version":"562'11893","reported_seq":19220,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:15.374998+0000","last_change":"2024-02-05T07:36:36.626867+0000","last_active":"2024-02-05T07:37:15.374998+0000","last_peered":"2024-02-05T07:37:15.374998+0000","last_clean":"2024-02-05T07:37:15.374998+0000","last_became_active":"2024-02-05T07:35:47.718921+0000","last_became_peered":"2024-02-05T07:35:47.718921+0000","last_unstale":"2024-02-05T07:37:15.374998+0000","last_undegraded":"2024-02-05T07:37:15.374998+0000","last_fullsized":"2024-02-05T07:37:15.374998+0000","mapping_epoch":512,"log_start":"508'8789","ondisk_log_start":"508'8789","created":493,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":5,"last_scrub":"539'10435","last_scrub_stamp":"2024-02-05T07:36:36.626755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:36:36.626755+0000","objects_scrubbed":1577,"log_size":3104,"log_dups_size":0,"ondisk_log_size":3104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:54.222932+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7798784,"num_objects":119,"num_object_clones":0,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1458,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"326'27","reported_seq":697,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.624302+0000","last_change":"2024-02-05T07:36:32.847108+0000","last_active":"2024-02-05T07:37:14.624302+0000","last_peered":"2024-02-05T07:37:14.624302+0000","last_clean":"2024-02-05T07:37:14.624302+0000","last_became_active":"2024-02-05T07:36:32.741691+0000","last_became_peered":"2024-02-05T07:36:32.741691+0000","last_unstale":"2024-02-05T07:37:14.624302+0000","last_undegraded":"2024-02-05T07:37:14.624302+0000","last_fullsized":"2024-02-05T07:37:14.624302+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:33:25.674080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:25.674080+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.305881+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":635,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.618836+0000","last_change":"2024-02-05T07:36:32.747614+0000","last_active":"2024-02-05T07:37:14.618836+0000","last_peered":"2024-02-05T07:37:14.618836+0000","last_clean":"2024-02-05T07:37:14.618836+0000","last_became_active":"2024-02-05T07:36:32.746731+0000","last_became_peered":"2024-02-05T07:36:32.746731+0000","last_unstale":"2024-02-05T07:37:14.618836+0000","last_undegraded":"2024-02-05T07:37:14.618836+0000","last_fullsized":"2024-02-05T07:37:14.618836+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:49.743255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:32:49.743255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.305847+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1046,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.613816+0000","last_change":"2024-02-05T07:33:46.937819+0000","last_active":"2024-02-05T07:37:14.613816+0000","last_peered":"2024-02-05T07:37:14.613816+0000","last_clean":"2024-02-05T07:37:14.613816+0000","last_became_active":"2024-02-05T07:28:25.778935+0000","last_became_peered":"2024-02-05T07:28:25.778935+0000","last_unstale":"2024-02-05T07:37:14.613816+0000","last_undegraded":"2024-02-05T07:37:14.613816+0000","last_fullsized":"2024-02-05T07:37:14.613816+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:46.937738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.937738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.553649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":636,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.620687+0000","last_change":"2024-02-05T07:34:28.009081+0000","last_active":"2024-02-05T07:37:14.620687+0000","last_peered":"2024-02-05T07:37:14.620687+0000","last_clean":"2024-02-05T07:37:14.620687+0000","last_became_active":"2024-02-05T07:30:40.493086+0000","last_became_peered":"2024-02-05T07:30:40.493086+0000","last_unstale":"2024-02-05T07:37:14.620687+0000","last_undegraded":"2024-02-05T07:37:14.620687+0000","last_fullsized":"2024-02-05T07:37:14.620687+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:28.008966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.008966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.306233+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.16","version":"562'12058","reported_seq":19455,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:18.269501+0000","last_change":"2024-02-05T07:35:48.199339+0000","last_active":"2024-02-05T07:37:18.269501+0000","last_peered":"2024-02-05T07:37:18.269501+0000","last_clean":"2024-02-05T07:37:18.269501+0000","last_became_active":"2024-02-05T07:35:46.685503+0000","last_became_peered":"2024-02-05T07:35:46.685503+0000","last_unstale":"2024-02-05T07:37:18.269501+0000","last_undegraded":"2024-02-05T07:37:18.269501+0000","last_fullsized":"2024-02-05T07:37:18.269501+0000","mapping_epoch":511,"log_start":"507'8543","ondisk_log_start":"507'8543","created":493,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":5,"last_scrub":"513'8989","last_scrub_stamp":"2024-02-05T07:35:48.199192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:37.591098+0000","last_clean_scrub_stamp":"2024-02-05T07:35:48.199192+0000","objects_scrubbed":3075,"log_size":3515,"log_dups_size":0,"ondisk_log_size":3515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:20.648812+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3075,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"450'19","reported_seq":1071,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.613988+0000","last_change":"2024-02-05T07:34:34.853204+0000","last_active":"2024-02-05T07:37:14.613988+0000","last_peered":"2024-02-05T07:37:14.613988+0000","last_clean":"2024-02-05T07:37:14.613988+0000","last_became_active":"2024-02-05T07:28:25.777815+0000","last_became_peered":"2024-02-05T07:28:25.777815+0000","last_unstale":"2024-02-05T07:37:14.613988+0000","last_undegraded":"2024-02-05T07:37:14.613988+0000","last_fullsized":"2024-02-05T07:37:14.613988+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'19","last_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_deep_scrub":"450'19","last_deep_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_clean_scrub_stamp":"2024-02-05T07:34:34.853068+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.555292+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":63,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.627307+0000","last_change":"2024-02-05T07:36:32.734803+0000","last_active":"2024-02-05T07:37:14.627307+0000","last_peered":"2024-02-05T07:37:14.627307+0000","last_clean":"2024-02-05T07:37:14.627307+0000","last_became_active":"2024-02-05T07:36:32.734502+0000","last_became_peered":"2024-02-05T07:36:32.734502+0000","last_unstale":"2024-02-05T07:37:14.627307+0000","last_undegraded":"2024-02-05T07:37:14.627307+0000","last_fullsized":"2024-02-05T07:37:14.627307+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.117071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.117071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":63,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.631312+0000","last_change":"2024-02-05T07:36:32.709307+0000","last_active":"2024-02-05T07:37:14.631312+0000","last_peered":"2024-02-05T07:37:14.631312+0000","last_clean":"2024-02-05T07:37:14.631312+0000","last_became_active":"2024-02-05T07:36:32.708917+0000","last_became_peered":"2024-02-05T07:36:32.708917+0000","last_unstale":"2024-02-05T07:37:14.631312+0000","last_undegraded":"2024-02-05T07:37:14.631312+0000","last_fullsized":"2024-02-05T07:37:14.631312+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:28.377472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:28.377472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:15.298577+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1037,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-05T07:37:14.614308+0000","last_change":"2024-02-05T07:32:43.608109+0000","last_active":"2024-02-05T07:37:14.614308+0000","last_peered":"2024-02-05T07:37:14.614308+0000","last_clean":"2024-02-05T07:37:14.614308+0000","last_became_active":"2024-02-05T07:30:40.490288+0000","last_became_peered":"2024-02-05T07:30:40.490288+0000","last_unstale":"2024-02-05T07:37:14.614308+0000","last_undegraded":"2024-02-05T07:37:14.614308+0000","last_fullsized":"2024-02-05T07:37:14.614308+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:32:43.608002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:32:43.608002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:37:19.553544+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":154927104,"num_objects":2364,"num_object_clones":0,"num_object_copies":7092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":231,"num_objects_unfound":0,"num_objects_dirty":2364,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160611,"num_write_kb":3621217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2498,"num_bytes_recovered":163708928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":839393280,"data_stored":839385136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88387,"ondisk_log_size":88387,"up":78,"acting":78,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":399,"num_read_kb":394,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":534813,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":707144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1104,"num_read_kb":861,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":8928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":11876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":762,"ondisk_log_size":762,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1146,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":257698037986,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":301,"seq":1292785156204,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":540368,"kb_used_data":2568,"kb_used_omap":5,"kb_used_meta":537786,"kb_avail":93204784,"statfs":{"total":95995035648,"available":95441698816,"internally_reserved":0,"allocated":2629632,"data_stored":1788124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":550693381},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":1.859,"5min":1.147,"15min":1.6419999999999999},"min":{"1min":0.19600000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.0990000000000002,"5min":13.976000000000001,"15min":14.218},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.147,"15min":1.6539999999999999},"min":{"1min":0.23599999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.0270000000000001,"5min":13.962999999999999,"15min":14.175000000000001},"last":0.80000000000000004}]},{"osd":1,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.54,"15min":1.806},"min":{"1min":0.26700000000000002,"5min":0.217,"15min":0.217},"max":{"1min":8.8640000000000008,"5min":19.861000000000001,"15min":55.923000000000002},"last":1.843},{"interface":"front","average":{"1min":1.498,"5min":1.7549999999999999,"15min":2.4049999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":11.467000000000001,"5min":27.518000000000001,"15min":61.103000000000002},"last":0.625}]},{"osd":2,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":1.8899999999999999,"5min":1.704,"15min":2.2240000000000002},"min":{"1min":0.45600000000000002,"5min":0.27400000000000002,"15min":0.249},"max":{"1min":11.157,"5min":32.103000000000002,"15min":34.689999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.7290000000000001,"5min":1.704,"15min":2.577},"min":{"1min":0.26300000000000001,"5min":0.245,"15min":0.245},"max":{"1min":10.314,"5min":32.149999999999999,"15min":34.646999999999998},"last":1.819}]},{"osd":3,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":1.929,"5min":2.028,"15min":2.4220000000000002},"min":{"1min":0.30399999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":11.430999999999999,"5min":36.090000000000003,"15min":55.978999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":2.391,"5min":2.1120000000000001,"15min":2.629},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":11.332000000000001,"5min":35.161000000000001,"15min":55.631},"last":0.312}]},{"osd":4,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":2.0529999999999999,"15min":1.153},"min":{"1min":0.28299999999999997,"5min":0.224,"15min":0.224},"max":{"1min":1.3149999999999999,"5min":151.02099999999999,"15min":151.02099999999999},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.79100000000000004,"5min":2.1899999999999999,"15min":1.141},"min":{"1min":0.27800000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.907,"5min":149.21799999999999,"15min":149.21799999999999},"last":1.7470000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.59399999999999997,"15min":0.58899999999999997},"min":{"1min":0.317,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.915,"5min":2.8919999999999999,"15min":2.8919999999999999},"last":1.792},{"interface":"front","average":{"1min":0.502,"5min":0.47499999999999998,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.97099999999999997,"5min":1.0669999999999999,"15min":1.526},"last":0.438}]},{"osd":7,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.50600000000000001,"15min":3.3359999999999999},"min":{"1min":0.34000000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":84.158000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.72499999999999998,"15min":3.508},"min":{"1min":0.317,"5min":0.312,"15min":0.312},"max":{"1min":0.874,"5min":23.055,"15min":85.018000000000001},"last":0.53500000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575104,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1040852,"kb_used_data":171080,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92704300,"statfs":{"total":95995035648,"available":94929203200,"internally_reserved":0,"allocated":175185920,"data_stored":174279893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":890567109},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:36:56 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.4340000000000002,"15min":2.6720000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":10.35,"5min":29.449999999999999,"15min":45.393000000000001},"last":4.6740000000000004},{"interface":"front","average":{"1min":2.6560000000000001,"5min":2.7589999999999999,"15min":2.9129999999999998},"min":{"1min":0.34300000000000003,"5min":0.28000000000000003,"15min":0.20899999999999999},"max":{"1min":8.3610000000000007,"5min":29.268999999999998,"15min":45.031999999999996},"last":5.5629999999999997}]},{"osd":2,"last update":"Mon Feb 5 07:36:20 2024","interfaces":[{"interface":"back","average":{"1min":6.0149999999999997,"5min":2.3330000000000002,"15min":2.2949999999999999},"min":{"1min":0.373,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":35.057000000000002,"5min":35.057000000000002,"15min":39.259999999999998},"last":1.966},{"interface":"front","average":{"1min":5.2569999999999997,"5min":2.2869999999999999,"15min":2.1699999999999999},"min":{"1min":0.29599999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":31.879000000000001,"5min":31.879000000000001,"15min":40.972000000000001},"last":1.8300000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:36:32 2024","interfaces":[{"interface":"back","average":{"1min":3.1389999999999998,"5min":2.6629999999999998,"15min":2.4399999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.20999999999999999},"max":{"1min":8.3919999999999995,"5min":46.777000000000001,"15min":46.777000000000001},"last":1.1519999999999999},{"interface":"front","average":{"1min":2.7080000000000002,"5min":2.5350000000000001,"15min":2.379},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.3350000000000009,"5min":50.164999999999999,"15min":50.164999999999999},"last":1.696}]},{"osd":4,"last update":"Mon Feb 5 07:36:56 2024","interfaces":[{"interface":"back","average":{"1min":1.8460000000000001,"5min":2.4969999999999999,"15min":2.302},"min":{"1min":0.38500000000000001,"5min":0.33400000000000002,"15min":0.312},"max":{"1min":6.7329999999999997,"5min":89.409000000000006,"15min":89.409000000000006},"last":1.802},{"interface":"front","average":{"1min":1.429,"5min":2.2639999999999998,"15min":2.2010000000000001},"min":{"1min":0.29399999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.3890000000000002,"5min":89.915000000000006,"15min":89.915000000000006},"last":1.6499999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:36:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.8240000000000001,"15min":2.1219999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.309},"max":{"1min":6.7610000000000001,"5min":46.398000000000003,"15min":106.672},"last":1.74},{"interface":"front","average":{"1min":2.0499999999999998,"5min":1.7010000000000001,"15min":2.1659999999999999},"min":{"1min":0.39200000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":5.6219999999999999,"5min":8.5410000000000004,"15min":107.348},"last":0.81499999999999995}]},{"osd":6,"last update":"Mon Feb 5 07:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.1140000000000001,"15min":1.4850000000000001},"min":{"1min":0.314,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.7080000000000002,"5min":4.8209999999999997,"15min":5.5990000000000002},"last":1.6819999999999999},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.454,"15min":1.7909999999999999},"min":{"1min":0.42499999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":6.8179999999999996,"5min":8.923,"15min":8.923},"last":1.7669999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:36:20 2024","interfaces":[{"interface":"back","average":{"1min":1.9570000000000001,"5min":2.2730000000000001,"15min":2.0710000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.318},"max":{"1min":5.7110000000000003,"5min":46.351999999999997,"15min":66.703999999999994},"last":0.877},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1599999999999999,"15min":1.6519999999999999},"min":{"1min":0.40500000000000003,"5min":0.39500000000000002,"15min":0.33500000000000002},"max":{"1min":2.8599999999999999,"5min":4.7510000000000003,"15min":61.594999999999999},"last":1.845}]}]},{"osd":0,"up_from":8,"seq":34359738629,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1203464,"kb_used_data":283772,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":92541688,"statfs":{"total":95995035648,"available":94762688512,"internally_reserved":0,"allocated":290582528,"data_stored":289688020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:36:47 2024","interfaces":[{"interface":"back","average":{"1min":3.0960000000000001,"5min":2.7210000000000001,"15min":2.4809999999999999},"min":{"1min":0.32300000000000001,"5min":0.189,"15min":0.16500000000000001},"max":{"1min":12.308,"5min":25.277999999999999,"15min":45.372},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.994,"5min":2.3410000000000002,"15min":2.419},"min":{"1min":0.32600000000000001,"5min":0.192,"15min":0.186},"max":{"1min":6.3220000000000001,"5min":24.652999999999999,"15min":45.231999999999999},"last":0.46400000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:36:14 2024","interfaces":[{"interface":"back","average":{"1min":6.0919999999999996,"5min":3.069,"15min":2.4540000000000002},"min":{"1min":0.22900000000000001,"5min":0.191,"15min":0.191},"max":{"1min":47.137999999999998,"5min":47.137999999999998,"15min":65.903999999999996},"last":0.60499999999999998},{"interface":"front","average":{"1min":6.351,"5min":2.9540000000000002,"15min":2.3940000000000001},"min":{"1min":0.58799999999999997,"5min":0.246,"15min":0.186},"max":{"1min":47.170999999999999,"5min":47.170999999999999,"15min":65.694000000000003},"last":0.79900000000000004}]},{"osd":3,"last update":"Mon Feb 5 07:36:27 2024","interfaces":[{"interface":"back","average":{"1min":3.399,"5min":2.0230000000000001,"15min":2.3919999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.21299999999999999},"max":{"1min":26.43,"5min":26.43,"15min":65.155000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":6.202,"5min":3.0550000000000002,"15min":2.8610000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.24299999999999999},"max":{"1min":25.337,"5min":25.337,"15min":65.114000000000004},"last":0.20899999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:36:47 2024","interfaces":[{"interface":"back","average":{"1min":2.9670000000000001,"5min":1.776,"15min":2.2669999999999999},"min":{"1min":0.314,"5min":0.27200000000000002,"15min":0.26700000000000002},"max":{"1min":12.718999999999999,"5min":12.718999999999999,"15min":65.188000000000002},"last":0.308},{"interface":"front","average":{"1min":1.512,"5min":1.373,"15min":2.1560000000000001},"min":{"1min":0.46400000000000002,"5min":0.30499999999999999,"15min":0.23699999999999999},"max":{"1min":4.2519999999999998,"5min":8.6899999999999995,"15min":64.873000000000005},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:36:27 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.1220000000000001,"15min":1.806},"min":{"1min":0.38900000000000001,"5min":0.318,"15min":0.27000000000000002},"max":{"1min":4.4409999999999998,"5min":4.6959999999999997,"15min":95.772999999999996},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.7170000000000001,"5min":1.48,"15min":1.97},"min":{"1min":0.35799999999999998,"5min":0.32900000000000001,"15min":0.27600000000000002},"max":{"1min":4.4320000000000004,"5min":8.7599999999999998,"15min":96.028000000000006},"last":0.628}]},{"osd":6,"last update":"Mon Feb 5 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":2.9849999999999999,"5min":1.8200000000000001,"15min":1.452},"min":{"1min":0.39700000000000002,"5min":0.33700000000000002,"15min":0.26300000000000001},"max":{"1min":12.670999999999999,"5min":12.670999999999999,"15min":12.670999999999999},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.4570000000000001,"15min":1.526},"min":{"1min":0.49399999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":4.3010000000000002,"5min":8.718,"15min":8.718},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Feb 5 07:37:01 2024","interfaces":[{"interface":"back","average":{"1min":2.8929999999999998,"5min":1.946,"15min":2.1600000000000001},"min":{"1min":0.433,"5min":0.34899999999999998,"15min":0.28599999999999998},"max":{"1min":12.766999999999999,"5min":12.766999999999999,"15min":65.263000000000005},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.9339999999999999,"5min":1.53,"15min":2.0590000000000002},"min":{"1min":0.435,"5min":0.373,"15min":0.309},"max":{"1min":4.3700000000000001,"5min":8.7910000000000004,"15min":64.906000000000006},"last":0.58099999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411579,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1023572,"kb_used_data":194120,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92721580,"statfs":{"total":95995035648,"available":94946897920,"internally_reserved":0,"allocated":198778880,"data_stored":197937405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":849344965},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:36:14 2024","interfaces":[{"interface":"back","average":{"1min":7.609,"5min":2.4249999999999998,"15min":2.6040000000000001},"min":{"1min":0.36299999999999999,"5min":0.20100000000000001,"15min":0.182},"max":{"1min":80.427000000000007,"5min":80.427000000000007,"15min":92.638000000000005},"last":0.48299999999999998},{"interface":"front","average":{"1min":8.4060000000000006,"5min":3.1299999999999999,"15min":2.9009999999999998},"min":{"1min":0.40300000000000002,"5min":0.22700000000000001,"15min":0.20699999999999999},"max":{"1min":80.939999999999998,"5min":80.939999999999998,"15min":93.481999999999999},"last":2.25}]},{"osd":1,"last update":"Mon Feb 5 07:36:14 2024","interfaces":[{"interface":"back","average":{"1min":11.689,"5min":4.9340000000000002,"15min":3.169},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":110.917,"5min":110.917,"15min":110.917},"last":2.6440000000000001},{"interface":"front","average":{"1min":1.385,"5min":3.0600000000000001,"15min":2.6640000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":5.75,"5min":64.771000000000001,"15min":93.004000000000005},"last":3.4260000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:36:30 2024","interfaces":[{"interface":"back","average":{"1min":10.382,"5min":4.298,"15min":2.9689999999999999},"min":{"1min":0.253,"5min":0.252,"15min":0.214},"max":{"1min":110.773,"5min":110.773,"15min":110.773},"last":2.7160000000000002},{"interface":"front","average":{"1min":3.0329999999999999,"5min":2.5760000000000001,"15min":2.3410000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.193},"max":{"1min":47.368000000000002,"5min":63.521000000000001,"15min":63.521000000000001},"last":0.44700000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:36:55 2024","interfaces":[{"interface":"back","average":{"1min":3.3730000000000002,"5min":1.796,"15min":2.375},"min":{"1min":0.29199999999999998,"5min":0.24299999999999999,"15min":0.24099999999999999},"max":{"1min":46.521999999999998,"5min":46.521999999999998,"15min":168.11699999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.04,"5min":1.149,"15min":2.1219999999999999},"min":{"1min":0.36599999999999999,"5min":0.34399999999999997,"15min":0.27200000000000002},"max":{"1min":4.5629999999999997,"5min":9.3539999999999992,"15min":168.589},"last":0.59399999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.98899999999999999,"15min":1.3080000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.315},"max":{"1min":1.246,"5min":4.742,"15min":39.923000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":5.8140000000000001,"5min":2.089,"15min":1.6120000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.26400000000000001},"max":{"1min":47.761000000000003,"5min":47.761000000000003,"15min":47.761000000000003},"last":2.7410000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:36:40 2024","interfaces":[{"interface":"back","average":{"1min":5.1420000000000003,"5min":2.335,"15min":1.5580000000000001},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":47.726999999999997,"5min":47.726999999999997,"15min":47.726999999999997},"last":2.6840000000000002},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.153,"15min":1.2889999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.33100000000000002},"max":{"1min":4.4550000000000001,"5min":9.2899999999999991,"15min":9.2899999999999991},"last":0.51600000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:36:14 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.494,"15min":1.5569999999999999},"min":{"1min":0.35699999999999998,"5min":0.35199999999999998,"15min":0.29299999999999998},"max":{"1min":6.4669999999999996,"5min":11.311999999999999,"15min":37.573999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":10.957000000000001,"5min":3.1669999999999998,"15min":2.0720000000000001},"min":{"1min":0.43099999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":110.86799999999999,"5min":110.86799999999999,"15min":110.86799999999999},"last":2.7679999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149942,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":955204,"kb_used_data":75772,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92789948,"statfs":{"total":95995035648,"available":95016906752,"internally_reserved":0,"allocated":77590528,"data_stored":76696877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":900528583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:36:50 2024","interfaces":[{"interface":"back","average":{"1min":4.4189999999999996,"5min":2.278,"15min":2.738},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":48.688000000000002,"5min":48.688000000000002,"15min":74.114999999999995},"last":0.94199999999999995},{"interface":"front","average":{"1min":5.4299999999999997,"5min":2.7280000000000002,"15min":2.996},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":48.573,"5min":48.573,"15min":99.989999999999995},"last":0.48099999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.651,"15min":2.8420000000000001},"min":{"1min":0.24299999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.3889999999999993,"5min":26.329000000000001,"15min":102.20399999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.6970000000000001,"15min":3.1000000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.185},"max":{"1min":5.4880000000000004,"5min":26.867000000000001,"15min":102.286},"last":0.85699999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:36:50 2024","interfaces":[{"interface":"back","average":{"1min":2.1579999999999999,"5min":3.5019999999999998,"15min":3.4119999999999999},"min":{"1min":0.318,"5min":0.26200000000000001,"15min":0.22700000000000001},"max":{"1min":6.9560000000000004,"5min":145.94999999999999,"15min":145.94999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":2.726,"5min":3.8740000000000001,"15min":3.0609999999999999},"min":{"1min":0.29199999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":10.695,"5min":146.006,"15min":146.006},"last":1.9279999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:36:59 2024","interfaces":[{"interface":"back","average":{"1min":2.3650000000000002,"5min":1.8149999999999999,"15min":1.901},"min":{"1min":0.29399999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":9.2639999999999993,"5min":31.655999999999999,"15min":100.024},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.99,"5min":1.9770000000000001,"15min":2.0190000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":16.253,"5min":37.049999999999997,"15min":99.859999999999999},"last":0.92100000000000004}]},{"osd":5,"last update":"Mon Feb 5 07:36:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1819999999999999,"15min":1.952},"min":{"1min":0.308,"5min":0.308,"15min":0.28499999999999998},"max":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":99.911000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.423,"5min":1.202,"15min":1.7150000000000001},"min":{"1min":0.36799999999999999,"5min":0.33000000000000002,"15min":0.27400000000000002},"max":{"1min":8.173,"5min":8.173,"15min":100.053},"last":0.68700000000000006}]},{"osd":6,"last update":"Mon Feb 5 07:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.7989999999999999,"15min":1.5880000000000001},"min":{"1min":0.36599999999999999,"5min":0.33100000000000002,"15min":0.29299999999999998},"max":{"1min":9.5199999999999996,"5min":32.505000000000003,"15min":99.807000000000002},"last":0.82199999999999995},{"interface":"front","average":{"1min":2.3330000000000002,"5min":2.012,"15min":1.7170000000000001},"min":{"1min":0.33600000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":10.209,"5min":31.084,"15min":101.489},"last":0.79600000000000004}]},{"osd":7,"last update":"Mon Feb 5 07:37:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.2669999999999999,"15min":2.4220000000000002},"min":{"1min":0.40500000000000003,"5min":0.38700000000000001,"15min":0.29499999999999998},"max":{"1min":8.4329999999999998,"5min":8.4329999999999998,"15min":99.893000000000001},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3899999999999999,"15min":2.7170000000000001},"min":{"1min":0.41099999999999998,"5min":0.375,"15min":0.375},"max":{"1min":3.0939999999999999,"5min":17.611000000000001,"15min":99.879000000000005},"last":0.90400000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953713,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1014936,"kb_used_data":203536,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92730216,"statfs":{"total":95995035648,"available":94955741184,"internally_reserved":0,"allocated":208420864,"data_stored":207563275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:36:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":0.98399999999999999,"15min":2.2400000000000002},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":10.513,"5min":10.513,"15min":101.212},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.1319999999999999,"5min":0.94999999999999996,"15min":2.2970000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":4.5629999999999997,"5min":4.5629999999999997,"15min":102.07599999999999},"last":0.89100000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:36:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":1.2629999999999999,"15min":2.371},"min":{"1min":0.28999999999999998,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":2.6320000000000001,"5min":23.335000000000001,"15min":69.102000000000004},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.264,"5min":1.3440000000000001,"15min":2.6339999999999999},"min":{"1min":0.27300000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":5.0369999999999999,"5min":23.346,"15min":68.108999999999995},"last":0.79300000000000004}]},{"osd":2,"last update":"Mon Feb 5 07:36:57 2024","interfaces":[{"interface":"back","average":{"1min":2.2810000000000001,"5min":4.4699999999999998,"15min":2.6400000000000001},"min":{"1min":0.26000000000000001,"5min":0.249,"15min":0.249},"max":{"1min":28.236000000000001,"5min":180.40199999999999,"15min":180.40199999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":2.7210000000000001,"5min":4.4459999999999997,"15min":2.6459999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":28.172000000000001,"5min":180.381,"15min":180.381},"last":0.34000000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:36:57 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.4550000000000001,"15min":1.9710000000000001},"min":{"1min":0.30099999999999999,"5min":0.27600000000000002,"15min":0.23799999999999999},"max":{"1min":19.777999999999999,"5min":26.899999999999999,"15min":66.019999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.81100000000000005,"5min":1.5269999999999999,"15min":2.1419999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.26400000000000001},"max":{"1min":2.7690000000000001,"5min":24.518999999999998,"15min":101.001},"last":0.77600000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:36:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.46899999999999997,"15min":0.59899999999999998},"min":{"1min":0.309,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1719999999999999,"5min":1.4299999999999999,"15min":10.917999999999999},"last":0.37},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.625},"min":{"1min":0.315,"5min":0.26700000000000002,"15min":0.24199999999999999},"max":{"1min":0.89500000000000002,"5min":1.3939999999999999,"15min":15.105},"last":0.81299999999999994}]},{"osd":6,"last update":"Mon Feb 5 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55600000000000005,"15min":0.98099999999999998},"min":{"1min":0.28699999999999998,"5min":0.25600000000000001,"15min":0.245},"max":{"1min":2.2829999999999999,"5min":3.5670000000000002,"15min":14.606},"last":0.249},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.52600000000000002,"15min":1.1439999999999999},"min":{"1min":0.33000000000000002,"5min":0.254,"15min":0.254},"max":{"1min":1.069,"5min":1.377,"15min":13.114000000000001},"last":0.57999999999999996}]},{"osd":7,"last update":"Mon Feb 5 07:37:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.72899999999999998},"min":{"1min":0.33500000000000002,"5min":0.32400000000000001,"15min":0.25700000000000001},"max":{"1min":0.998,"5min":1.4350000000000001,"15min":15.073},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55000000000000004,"15min":0.70299999999999996},"min":{"1min":0.34000000000000002,"5min":0.33500000000000002,"15min":0.248},"max":{"1min":2.246,"5min":2.246,"15min":14.581},"last":0.64800000000000002}]}]},{"osd":5,"up_from":0,"seq":176093659372,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":281677824,"data_stored":281673752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":157286400,"data_stored":157286400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190119936,"data_stored":190119936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50855936,"data_stored":50855936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":159387648,"data_stored":159383576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:37:25.873 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-05T07:37:25.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pg_num 31 2024-02-05T07:37:25.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:25.885 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:25.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:25.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:26.010 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:26.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:26.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:26.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:26.141 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:26.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:26.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:26.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:26.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:26.271 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:26.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:26.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:26.401 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:26.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:26.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:26.514 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:26.533 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:26.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:26.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:26.661 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:26.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:26.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:26.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:26.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:26.788 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:26.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:26.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:26.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:26 smithi007 ceph-mon[29098]: pgmap v1020: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 582 op/s 2024-02-05T07:37:26.897 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:26 smithi007 ceph-mon[37158]: pgmap v1020: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 582 op/s 2024-02-05T07:37:26.913 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:26.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:26.996 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:27.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:27.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:26 smithi090 ceph-mon[33213]: pgmap v1020: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 582 op/s 2024-02-05T07:37:27.026 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:37:27.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:27.041 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:27.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:27.046 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:37:27.066 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:37:27.086 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:37:27.106 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:37:27.126 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:37:27.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:27.147 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:37:27.166 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:27.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:27.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:27.167 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:37:27.169 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:37:27.169 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:37:27.169 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:37:27.169 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:37:27.169 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:37:27.169 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:37:27.169 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:37:27.169 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:37:27.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:27.296 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:27.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:27.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:27.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:27.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:27.427 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:27.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:27.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:27.556 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:27.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:27.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:27.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:27.658 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:37:27.686 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:27.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:27.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:27.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:27.813 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:27.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:27.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:27.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:27.922 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:27 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1469196589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:37:27.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:27 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1472735174' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:27.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:27 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1469196589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:37:27.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:27 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1472735174' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:27.925 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:27 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1469196589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:37:27.925 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:27 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1472735174' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:27.943 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:27.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:27.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:28.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:28.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:28.070 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:28.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:28.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:28.194 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:28.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:28.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:28.296 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:28.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:28.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:28.421 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:28.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:28.446 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:28.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:28.546 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:28.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:28.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:28.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:28.701 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:28.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:28.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:28.826 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:28.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:28.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:28 smithi007 ceph-mon[29098]: pgmap v1021: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 297 op/s 2024-02-05T07:37:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:28 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:37:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:28 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1469196589' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:37:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:28 smithi007 ceph-mon[29098]: osdmap e565: 8 total, 8 up, 6 in 2024-02-05T07:37:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:28 smithi007 ceph-mon[37158]: pgmap v1021: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 297 op/s 2024-02-05T07:37:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:28 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:37:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:28 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1469196589' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:37:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:28 smithi007 ceph-mon[37158]: osdmap e565: 8 total, 8 up, 6 in 2024-02-05T07:37:28.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:28.955 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:28.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:28 smithi090 ceph-mon[33213]: pgmap v1021: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 297 op/s 2024-02-05T07:37:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:28 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:37:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:28 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1469196589' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:37:29.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:28 smithi090 ceph-mon[33213]: osdmap e565: 8 total, 8 up, 6 in 2024-02-05T07:37:29.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:29.081 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:29.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:29.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:29.209 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:29.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:29.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:29.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:29.336 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:29.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:29.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:29.460 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:29.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:29.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:29.590 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:29.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:29.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:29.716 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:29.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:29.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:29.817 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:29.841 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:29.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:29.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:29.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:37:29.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:29.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:29.973 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:29.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:30.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:30.098 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:30.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:30.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:30.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:30.325 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:30.349 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:30.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:30.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:30.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:30.474 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:30.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:30.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:30.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:30.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:30.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:30.727 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:30.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:30.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:30.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:30.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:30.859 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:30.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:30 smithi007 ceph-mon[29098]: pgmap v1023: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 20 op/s 2024-02-05T07:37:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:30 smithi007 ceph-mon[29098]: osdmap e566: 8 total, 8 up, 6 in 2024-02-05T07:37:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2024-02-05T07:37:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-05T07:37:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2024-02-05T07:37:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-05T07:37:30.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:30 smithi007 ceph-mon[37158]: pgmap v1023: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 20 op/s 2024-02-05T07:37:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:30 smithi007 ceph-mon[37158]: osdmap e566: 8 total, 8 up, 6 in 2024-02-05T07:37:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2024-02-05T07:37:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-05T07:37:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2024-02-05T07:37:30.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-05T07:37:30.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:37:30.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:31.022 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:31.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:30 smithi090 ceph-mon[33213]: pgmap v1023: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 20 op/s 2024-02-05T07:37:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:30 smithi090 ceph-mon[33213]: osdmap e566: 8 total, 8 up, 6 in 2024-02-05T07:37:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2024-02-05T07:37:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-05T07:37:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2024-02-05T07:37:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-05T07:37:31.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:31.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:31.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:31.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:31.279 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:31.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:31.320 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:31.344 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:37:31.344 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:37:31.381 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:31.406 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:31.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:31.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:31.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:37:31 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:37:31.535 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:31.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:31.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:31.663 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:31.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:31.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:31.788 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:31.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:31.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:31.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:31.913 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:31.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:31.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:31.922 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:31 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]': finished 2024-02-05T07:37:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:31 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-05T07:37:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:31 smithi007 ceph-mon[29098]: osdmap e567: 8 total, 8 up, 6 in 2024-02-05T07:37:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:31 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:37:31.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:31 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]': finished 2024-02-05T07:37:31.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:31 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-05T07:37:31.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:31 smithi007 ceph-mon[37158]: osdmap e567: 8 total, 8 up, 6 in 2024-02-05T07:37:31.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:31 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:37:32.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:32.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:31 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]': finished 2024-02-05T07:37:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:31 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-05T07:37:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:31 smithi090 ceph-mon[33213]: osdmap e567: 8 total, 8 up, 6 in 2024-02-05T07:37:32.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:31 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:37:32.044 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:32.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:32.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:32.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:32.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:32.170 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:32.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:32.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:32.299 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:32.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:32.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:32.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:32.429 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:32.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:32.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:32.556 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:32.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:32.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:32.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:32.682 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:32.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:32.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:32.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T07:37:32.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:32.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:32.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:32 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:32 smithi007 ceph-mon[29098]: pgmap v1026: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:37:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:32 smithi007 ceph-mon[29098]: osdmap e568: 8 total, 8 up, 6 in 2024-02-05T07:37:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:32 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:32 smithi007 ceph-mon[37158]: pgmap v1026: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:37:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:32 smithi007 ceph-mon[37158]: osdmap e568: 8 total, 8 up, 6 in 2024-02-05T07:37:32.938 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:32.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:32.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:33.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:32 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:32 smithi090 ceph-mon[33213]: pgmap v1026: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:37:33.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:32 smithi090 ceph-mon[33213]: osdmap e568: 8 total, 8 up, 6 in 2024-02-05T07:37:33.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:33.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:33.069 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:33.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:33.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:33.196 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:33.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:33.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:33.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:33.325 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:33.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:33.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:33.454 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:33.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:33.486 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:33.512 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:33.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:33.584 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:33.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:33.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:33.613 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:37:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:37:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:37:33.673 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pg_num to 31 2024-02-05T07:37:33.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:33.713 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:33.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:33.815 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:33.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:33.839 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:33.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:33.865 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:33 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2138224703' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-02-05T07:37:33.865 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:33 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2138224703' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-02-05T07:37:33.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:33.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:33.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:34.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:33 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2138224703' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-02-05T07:37:34.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:34.100 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:34.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:34.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:34.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:34.226 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:34.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:34.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:34.356 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:34.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:34.395 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-02-05T07:37:34.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-02-05T07:37:34.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:34.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:34.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:34.613 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:34.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:34.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:34.715 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:34.739 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:34.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:34.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:34.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:34.865 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:34.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:34.917 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:34 smithi007 ceph-mon[29098]: pgmap v1028: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 898 B/s rd, 0 op/s 2024-02-05T07:37:34.917 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:34 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2138224703' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-02-05T07:37:34.917 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:34 smithi007 ceph-mon[29098]: osdmap e569: 8 total, 8 up, 6 in 2024-02-05T07:37:34.917 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:34 smithi007 ceph-mon[29098]: osdmap e570: 8 total, 8 up, 6 in 2024-02-05T07:37:34.917 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:34 smithi007 ceph-mon[37158]: pgmap v1028: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 898 B/s rd, 0 op/s 2024-02-05T07:37:34.918 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:34 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2138224703' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-02-05T07:37:34.918 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:34 smithi007 ceph-mon[37158]: osdmap e569: 8 total, 8 up, 6 in 2024-02-05T07:37:34.918 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:34 smithi007 ceph-mon[37158]: osdmap e570: 8 total, 8 up, 6 in 2024-02-05T07:37:34.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:34.991 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:34.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:34 smithi090 ceph-mon[33213]: pgmap v1028: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 898 B/s rd, 0 op/s 2024-02-05T07:37:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:34 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2138224703' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-02-05T07:37:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:34 smithi090 ceph-mon[33213]: osdmap e569: 8 total, 8 up, 6 in 2024-02-05T07:37:35.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:34 smithi090 ceph-mon[33213]: osdmap e570: 8 total, 8 up, 6 in 2024-02-05T07:37:35.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:35.116 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:35.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:35.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:35.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:35.246 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:35.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:35.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:35.375 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:35.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:35.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:35.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:35.501 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:35.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:35.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:35.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:35.653 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:35.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:35.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:35.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:35.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:35.780 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:35.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:35.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:35.909 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:35.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:35.914 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:35 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-05T07:37:35.915 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:35 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-05T07:37:35.915 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:35 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-05T07:37:35.915 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:35 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-05T07:37:36.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:36.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:35 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-05T07:37:36.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:35 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-05T07:37:36.055 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:36.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:36.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:36.157 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:36.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:36.182 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:36.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:36.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:36.312 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:36.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:36.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:36.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:36.441 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:36.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:36.442 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:37:36.461 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:37:36.480 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:37:36.500 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:37:36.521 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:37:36.541 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:37:36.543 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:36.562 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:37:36.569 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:36.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:36.584 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:37:36.585 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:37:36.586 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:37:36.586 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:37:36.586 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:37:36.586 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:37:36.586 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:37:36.586 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:37:36.586 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:37:36.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:36.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:36.705 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:36.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:36.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:36 smithi007 ceph-mon[29098]: pgmap v1031: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:37:36.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:36 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-05T07:37:36.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:36 smithi007 ceph-mon[29098]: osdmap e571: 8 total, 8 up, 6 in 2024-02-05T07:37:36.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:36 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3187839436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:36.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:36 smithi007 ceph-mon[37158]: pgmap v1031: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:37:36.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:36 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-05T07:37:36.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:36 smithi007 ceph-mon[37158]: osdmap e571: 8 total, 8 up, 6 in 2024-02-05T07:37:36.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:36 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3187839436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:36.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:36.831 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:36.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:36.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:36.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:36.958 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:36.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:36.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:37.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:36 smithi090 ceph-mon[33213]: pgmap v1031: 178 pgs: 4 active+remapped+backfill_toofull, 174 active+clean; 528 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:37:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:36 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-05T07:37:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:36 smithi090 ceph-mon[33213]: osdmap e571: 8 total, 8 up, 6 in 2024-02-05T07:37:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:36 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3187839436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:37.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:37.084 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:37.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:37.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:37.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:37.209 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:37.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:37.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:37.336 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:37.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:37.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:37.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:37.464 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:37.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:37.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:37.592 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:37.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:37.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:37.703 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:37:37.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:37.724 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:37.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:37.825 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:37.850 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:37.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:37.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:37 smithi007 ceph-mon[29098]: osdmap e572: 8 total, 8 up, 6 in 2024-02-05T07:37:37.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:37 smithi007 ceph-mon[29098]: 1.d continuing backfill to osd.1 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:37.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:37 smithi007 ceph-mon[29098]: 1.e continuing backfill to osd.2 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:37.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:37 smithi007 ceph-mon[29098]: 1.e continuing backfill to osd.4 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:37.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:37 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4224269894' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:37:37.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:37 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-05T07:37:37.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:37 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-05T07:37:37.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:37 smithi007 ceph-mon[37158]: osdmap e572: 8 total, 8 up, 6 in 2024-02-05T07:37:37.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:37 smithi007 ceph-mon[37158]: 1.d continuing backfill to osd.1 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:37.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:37 smithi007 ceph-mon[37158]: 1.e continuing backfill to osd.2 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:37.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:37 smithi007 ceph-mon[37158]: 1.e continuing backfill to osd.4 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:37.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:37 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4224269894' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:37:37.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:37 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-05T07:37:37.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:37 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-05T07:37:37.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:37.975 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:37.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:37 smithi090 ceph-mon[33213]: osdmap e572: 8 total, 8 up, 6 in 2024-02-05T07:37:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:37 smithi090 ceph-mon[33213]: 1.d continuing backfill to osd.1 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:37 smithi090 ceph-mon[33213]: 1.e continuing backfill to osd.2 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:37 smithi090 ceph-mon[33213]: 1.e continuing backfill to osd.4 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:37 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4224269894' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:37:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:37 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-05T07:37:38.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:37 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-05T07:37:38.026 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:37:38.026 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:37:38.027 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-02-05T07:37:38.027 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-02-05T07:37:38.027 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:37:38.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:38.104 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:38.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:38.207 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:38.235 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:38.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:38.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:38.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:38.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:38.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:38.490 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:38.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:38.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:38.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:38.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:38.617 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:38.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:38.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:38.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:38.745 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:38.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:38.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:38.872 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:38.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:38.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:38 smithi007 ceph-mon[29098]: pgmap v1034: 188 pgs: 1 remapped+peering, 8 unknown, 3 active+remapped+backfill_toofull, 176 active+clean; 522 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:37:38.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:38 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:37:38.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:38 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:37:38.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:38 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4224269894' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:37:38.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:38 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-05T07:37:38.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:38 smithi007 ceph-mon[29098]: osdmap e573: 8 total, 8 up, 6 in 2024-02-05T07:37:38.889 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:38 smithi007 ceph-mon[37158]: pgmap v1034: 188 pgs: 1 remapped+peering, 8 unknown, 3 active+remapped+backfill_toofull, 176 active+clean; 522 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:37:38.889 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:38 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:37:38.889 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:38 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:37:38.889 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:38 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4224269894' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:37:38.889 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:38 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-05T07:37:38.889 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:38 smithi007 ceph-mon[37158]: osdmap e573: 8 total, 8 up, 6 in 2024-02-05T07:37:38.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:39.000 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:39.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:39.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:38 smithi090 ceph-mon[33213]: pgmap v1034: 188 pgs: 1 remapped+peering, 8 unknown, 3 active+remapped+backfill_toofull, 176 active+clean; 522 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:37:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:38 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:37:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:38 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:37:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:38 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4224269894' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:37:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:38 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-05T07:37:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:38 smithi090 ceph-mon[33213]: osdmap e573: 8 total, 8 up, 6 in 2024-02-05T07:37:39.102 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:39.127 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:39.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:39.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:39.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:39.253 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:39.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:39.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:39.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:39.381 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:39.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:39.382 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:39.395 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6, 3, 2] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:37:39.395 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:37:39.396 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-05T07:37:39.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T07:37:39.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:39.506 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:39.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:39.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:39.644 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:39.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:39.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:39.769 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:39.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:39.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:39.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T07:37:39.872 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:39.898 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:39.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:39.955 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[29098]: 1.10 scrub starts 2024-02-05T07:37:39.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[29098]: 1.10 scrub ok 2024-02-05T07:37:39.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[29098]: 1.5 scrub starts 2024-02-05T07:37:39.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[29098]: 1.5 scrub ok 2024-02-05T07:37:39.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[29098]: 1.13 scrub starts 2024-02-05T07:37:39.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[29098]: 1.13 scrub ok 2024-02-05T07:37:39.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[29098]: 1.8 scrub starts 2024-02-05T07:37:39.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[29098]: 1.8 scrub ok 2024-02-05T07:37:39.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[29098]: osdmap e574: 8 total, 8 up, 6 in 2024-02-05T07:37:39.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[29098]: 1.1d continuing backfill to osd.1 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:39.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[29098]: 1.1d starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:39.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[29098]: osdmap e575: 8 total, 8 up, 6 in 2024-02-05T07:37:39.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[37158]: 1.10 scrub starts 2024-02-05T07:37:39.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[37158]: 1.10 scrub ok 2024-02-05T07:37:39.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[37158]: 1.5 scrub starts 2024-02-05T07:37:39.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[37158]: 1.5 scrub ok 2024-02-05T07:37:39.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[37158]: 1.13 scrub starts 2024-02-05T07:37:39.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[37158]: 1.13 scrub ok 2024-02-05T07:37:39.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[37158]: 1.8 scrub starts 2024-02-05T07:37:39.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[37158]: 1.8 scrub ok 2024-02-05T07:37:39.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[37158]: osdmap e574: 8 total, 8 up, 6 in 2024-02-05T07:37:39.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[37158]: 1.1d continuing backfill to osd.1 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:39.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[37158]: 1.1d starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:39.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:39 smithi007 ceph-mon[37158]: osdmap e575: 8 total, 8 up, 6 in 2024-02-05T07:37:40.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:39 smithi090 ceph-mon[33213]: 1.10 scrub starts 2024-02-05T07:37:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:39 smithi090 ceph-mon[33213]: 1.10 scrub ok 2024-02-05T07:37:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:39 smithi090 ceph-mon[33213]: 1.5 scrub starts 2024-02-05T07:37:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:39 smithi090 ceph-mon[33213]: 1.5 scrub ok 2024-02-05T07:37:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:39 smithi090 ceph-mon[33213]: 1.13 scrub starts 2024-02-05T07:37:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:39 smithi090 ceph-mon[33213]: 1.13 scrub ok 2024-02-05T07:37:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:39 smithi090 ceph-mon[33213]: 1.8 scrub starts 2024-02-05T07:37:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:39 smithi090 ceph-mon[33213]: 1.8 scrub ok 2024-02-05T07:37:40.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:39 smithi090 ceph-mon[33213]: osdmap e574: 8 total, 8 up, 6 in 2024-02-05T07:37:40.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:39 smithi090 ceph-mon[33213]: 1.1d continuing backfill to osd.1 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:40.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:39 smithi090 ceph-mon[33213]: 1.1d starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:39 smithi090 ceph-mon[33213]: osdmap e575: 8 total, 8 up, 6 in 2024-02-05T07:37:40.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:40.026 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:40.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:40.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:40.154 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:40.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:40.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:40.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:40.283 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:40.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:40.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:40.410 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:40.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:40.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:40.541 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:40.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:40.643 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:40.668 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:40.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:40.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:40.770 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:40.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:40.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:40.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:40.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:40.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:37:40.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.3 scrub starts 2024-02-05T07:37:40.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.3 scrub ok 2024-02-05T07:37:40.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.d scrub starts 2024-02-05T07:37:40.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.d scrub ok 2024-02-05T07:37:40.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.c scrub starts 2024-02-05T07:37:40.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.c scrub ok 2024-02-05T07:37:40.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: pgmap v1037: 188 pgs: 3 remapped+peering, 7 unknown, 3 active+remapped+backfill_toofull, 175 active+clean; 473 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:40.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.b scrub starts 2024-02-05T07:37:40.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.b scrub ok 2024-02-05T07:37:40.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.1b starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.1a starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.16 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.1c starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.17 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.18 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.18 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.1e continuing backfill to osd.4 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:40.919 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.19 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.920 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.15 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.920 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.15 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.920 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: 1.15 starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.920 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:37:40.920 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:37:40.920 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:37:40.920 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[29098]: osdmap e576: 8 total, 8 up, 6 in 2024-02-05T07:37:40.920 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.3 scrub starts 2024-02-05T07:37:40.920 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.3 scrub ok 2024-02-05T07:37:40.920 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.d scrub starts 2024-02-05T07:37:40.921 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.d scrub ok 2024-02-05T07:37:40.921 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.c scrub starts 2024-02-05T07:37:40.921 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.c scrub ok 2024-02-05T07:37:40.921 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: pgmap v1037: 188 pgs: 3 remapped+peering, 7 unknown, 3 active+remapped+backfill_toofull, 175 active+clean; 473 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:40.921 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.b scrub starts 2024-02-05T07:37:40.921 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.b scrub ok 2024-02-05T07:37:40.921 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.1b starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.921 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.1a starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.921 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.16 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.921 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.1c starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.921 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.17 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.921 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.18 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.921 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.18 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.921 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.1e continuing backfill to osd.4 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:40.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.19 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.15 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.15 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: 1.15 starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:40.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:37:40.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:37:40.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:37:40.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:40 smithi007 ceph-mon[37158]: osdmap e576: 8 total, 8 up, 6 in 2024-02-05T07:37:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:40.923 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:40.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.3 scrub starts 2024-02-05T07:37:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.3 scrub ok 2024-02-05T07:37:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.d scrub starts 2024-02-05T07:37:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.d scrub ok 2024-02-05T07:37:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.c scrub starts 2024-02-05T07:37:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.c scrub ok 2024-02-05T07:37:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: pgmap v1037: 188 pgs: 3 remapped+peering, 7 unknown, 3 active+remapped+backfill_toofull, 175 active+clean; 473 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.b scrub starts 2024-02-05T07:37:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.b scrub ok 2024-02-05T07:37:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.1b starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.1a starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.16 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.1c starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.17 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.18 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.18 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.1e continuing backfill to osd.4 from (86'128,86'128] MIN to 86'128 2024-02-05T07:37:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.19 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.15 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.15 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: 1.15 starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:37:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:37:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:37:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:37:41.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:40 smithi090 ceph-mon[33213]: osdmap e576: 8 total, 8 up, 6 in 2024-02-05T07:37:41.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:41.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:41.051 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:41.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:41.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:41.177 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:41.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:41.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:41.302 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:41.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:41.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:41.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:37:41.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:41.428 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:41.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:41.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:37:41.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:41.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:41.557 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:41.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:41.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:41.682 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:41.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:41.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:41.774 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[29098]: 1.6 scrub starts 2024-02-05T07:37:41.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[29098]: 1.6 scrub ok 2024-02-05T07:37:41.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[29098]: 1.e scrub starts 2024-02-05T07:37:41.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[29098]: 1.e scrub ok 2024-02-05T07:37:41.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[29098]: 1.11 scrub starts 2024-02-05T07:37:41.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[29098]: 1.11 scrub ok 2024-02-05T07:37:41.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[29098]: 1.a scrub starts 2024-02-05T07:37:41.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[29098]: 1.a scrub ok 2024-02-05T07:37:41.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[29098]: 10.11 scrub starts 2024-02-05T07:37:41.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[29098]: 10.11 scrub ok 2024-02-05T07:37:41.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:37:41.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[29098]: 1.1d scrub starts 2024-02-05T07:37:41.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[29098]: 1.1d scrub ok 2024-02-05T07:37:41.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[29098]: osdmap e577: 8 total, 8 up, 6 in 2024-02-05T07:37:41.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[37158]: 1.6 scrub starts 2024-02-05T07:37:41.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[37158]: 1.6 scrub ok 2024-02-05T07:37:41.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[37158]: 1.e scrub starts 2024-02-05T07:37:41.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[37158]: 1.e scrub ok 2024-02-05T07:37:41.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[37158]: 1.11 scrub starts 2024-02-05T07:37:41.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[37158]: 1.11 scrub ok 2024-02-05T07:37:41.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[37158]: 1.a scrub starts 2024-02-05T07:37:41.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[37158]: 1.a scrub ok 2024-02-05T07:37:41.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[37158]: 10.11 scrub starts 2024-02-05T07:37:41.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[37158]: 10.11 scrub ok 2024-02-05T07:37:41.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:37:41.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[37158]: 1.1d scrub starts 2024-02-05T07:37:41.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[37158]: 1.1d scrub ok 2024-02-05T07:37:41.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:41 smithi007 ceph-mon[37158]: osdmap e577: 8 total, 8 up, 6 in 2024-02-05T07:37:41.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:41.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:41.811 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:41.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:41.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:41.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:41.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:42.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-mon[33213]: 1.6 scrub starts 2024-02-05T07:37:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-mon[33213]: 1.6 scrub ok 2024-02-05T07:37:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-mon[33213]: 1.e scrub starts 2024-02-05T07:37:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-mon[33213]: 1.e scrub ok 2024-02-05T07:37:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-mon[33213]: 1.11 scrub starts 2024-02-05T07:37:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-mon[33213]: 1.11 scrub ok 2024-02-05T07:37:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-mon[33213]: 1.a scrub starts 2024-02-05T07:37:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-mon[33213]: 1.a scrub ok 2024-02-05T07:37:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-mon[33213]: 10.11 scrub starts 2024-02-05T07:37:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-mon[33213]: 10.11 scrub ok 2024-02-05T07:37:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:37:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-mon[33213]: 1.1d scrub starts 2024-02-05T07:37:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-mon[33213]: 1.1d scrub ok 2024-02-05T07:37:42.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:41 smithi090 ceph-mon[33213]: osdmap e577: 8 total, 8 up, 6 in 2024-02-05T07:37:42.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:42.062 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:42.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:42.157 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:42.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:42.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:42.193 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:42.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:42.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:42.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:42.319 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:42.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:42.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:42.444 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:42.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:42.546 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:42.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:42.570 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:42.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:42.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:42.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:42.697 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:42.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:42.710 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_3' removed 2024-02-05T07:37:42.722 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:42.722 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: 1.9 scrub starts 2024-02-05T07:37:42.722 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: 1.9 scrub ok 2024-02-05T07:37:42.722 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: pgmap v1040: 188 pgs: 1 peering, 1 active+clean+remapped, 2 remapped+peering, 3 unknown, 4 active+remapped+backfill_toofull, 177 active+clean; 351 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 3.0 KiB/s rd, 12 KiB/s wr, 3 op/s; 19 KiB/s, 0 objects/s recovering 2024-02-05T07:37:42.722 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: 1.14 scrub starts 2024-02-05T07:37:42.722 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: 1.14 scrub ok 2024-02-05T07:37:42.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: 1.f scrub starts 2024-02-05T07:37:42.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: 1.f scrub ok 2024-02-05T07:37:42.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1524169493' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:37:42.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:37:42.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:37:42.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:37:42.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:37:42.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: 1.12 scrub starts 2024-02-05T07:37:42.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[29098]: 1.12 scrub ok 2024-02-05T07:37:42.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:42.828 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:42.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:42.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:42.961 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:42.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:42.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: 1.9 scrub starts 2024-02-05T07:37:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: 1.9 scrub ok 2024-02-05T07:37:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: pgmap v1040: 188 pgs: 1 peering, 1 active+clean+remapped, 2 remapped+peering, 3 unknown, 4 active+remapped+backfill_toofull, 177 active+clean; 351 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 3.0 KiB/s rd, 12 KiB/s wr, 3 op/s; 19 KiB/s, 0 objects/s recovering 2024-02-05T07:37:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: 1.14 scrub starts 2024-02-05T07:37:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: 1.14 scrub ok 2024-02-05T07:37:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: 1.f scrub starts 2024-02-05T07:37:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: 1.f scrub ok 2024-02-05T07:37:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1524169493' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:37:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:37:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:37:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:37:43.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:37:43.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: 1.12 scrub starts 2024-02-05T07:37:43.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:42 smithi090 ceph-mon[33213]: 1.12 scrub ok 2024-02-05T07:37:43.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:43.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:43.088 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:43.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:43.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:43.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: 1.9 scrub starts 2024-02-05T07:37:43.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: 1.9 scrub ok 2024-02-05T07:37:43.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: pgmap v1040: 188 pgs: 1 peering, 1 active+clean+remapped, 2 remapped+peering, 3 unknown, 4 active+remapped+backfill_toofull, 177 active+clean; 351 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 3.0 KiB/s rd, 12 KiB/s wr, 3 op/s; 19 KiB/s, 0 objects/s recovering 2024-02-05T07:37:43.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: 1.14 scrub starts 2024-02-05T07:37:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: 1.14 scrub ok 2024-02-05T07:37:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: 1.f scrub starts 2024-02-05T07:37:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: 1.f scrub ok 2024-02-05T07:37:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1524169493' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:37:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:37:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:37:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:37:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:37:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: 1.12 scrub starts 2024-02-05T07:37:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:42 smithi007 ceph-mon[37158]: 1.12 scrub ok 2024-02-05T07:37:43.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:43.221 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:43.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:43.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:43.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:43.350 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:43.361 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:43.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:43.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:43.376 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:37:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-02-05T07:37:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-02-05T07:37:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:37:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:200399 8.c 8:369e3f1b:::benchmark_data_smithi007_105755_object64089:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e290)", 2024-02-05T07:37:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:28:12.480723+0000", 2024-02-05T07:37:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 570.89166083099997, 2024-02-05T07:37:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.072542493, 2024-02-05T07:37:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:37:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:37:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:37:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200399 2024-02-05T07:37:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480723+0000", 2024-02-05T07:37:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480723+0000", 2024-02-05T07:37:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480724+0000", 2024-02-05T07:37:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3799999999999996e-07 2024-02-05T07:37:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480727+0000", 2024-02-05T07:37:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6239999999999999e-06 2024-02-05T07:37:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480735+0000", 2024-02-05T07:37:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.357e-06 2024-02-05T07:37:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480742+0000", 2024-02-05T07:37:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.097e-06 2024-02-05T07:37:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480875+0000", 2024-02-05T07:37:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013302 2024-02-05T07:37:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-02-05T07:37:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480883+0000", 2024-02-05T07:37:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.661e-06 2024-02-05T07:37:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.490966+0000", 2024-02-05T07:37:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010083844999999999 2024-02-05T07:37:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.490976+0000", 2024-02-05T07:37:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4399999999999994e-06 2024-02-05T07:37:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.553187+0000", 2024-02-05T07:37:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0622112269999999 2024-02-05T07:37:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.553266+0000", 2024-02-05T07:37:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8484000000000003e-05 2024-02-05T07:37:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:200403 8.c 8:36ae280d:::benchmark_data_smithi007_105755_object64093:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e290)", 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:28:12.480754+0000", 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 570.89163033399996, 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.073251822, 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200403 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480754+0000", 2024-02-05T07:37:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480754+0000", 2024-02-05T07:37:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480754+0000", 2024-02-05T07:37:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8000000000000001e-07 2024-02-05T07:37:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480756+0000", 2024-02-05T07:37:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9530000000000002e-06 2024-02-05T07:37:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480761+0000", 2024-02-05T07:37:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2469999999999999e-06 2024-02-05T07:37:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480766+0000", 2024-02-05T07:37:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1699999999999996e-06 2024-02-05T07:37:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480990+0000", 2024-02-05T07:37:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00022330000000000001 2024-02-05T07:37:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting_for_map not empty", 2024-02-05T07:37:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.480995+0000", 2024-02-05T07:37:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5199999999999997e-06 2024-02-05T07:37:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.491000+0000", 2024-02-05T07:37:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010004677 2024-02-05T07:37:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.491006+0000", 2024-02-05T07:37:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1020000000000002e-06 2024-02-05T07:37:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.553938+0000", 2024-02-05T07:37:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0629322329999999 2024-02-05T07:37:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.554005+0000", 2024-02-05T07:37:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.724e-05 2024-02-05T07:37:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:200429 8.c 8:39182b98:::benchmark_data_smithi007_105755_object64119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-02-05T07:37:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:28:12.503389+0000", 2024-02-05T07:37:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 570.868995154, 2024-02-05T07:37:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0624662739999999, 2024-02-05T07:37:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:37:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:37:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200429 2024-02-05T07:37:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503389+0000", 2024-02-05T07:37:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503389+0000", 2024-02-05T07:37:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503390+0000", 2024-02-05T07:37:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3e-06 2024-02-05T07:37:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503395+0000", 2024-02-05T07:37:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2469999999999999e-06 2024-02-05T07:37:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503406+0000", 2024-02-05T07:37:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0305e-05 2024-02-05T07:37:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503413+0000", 2024-02-05T07:37:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5390000000000002e-06 2024-02-05T07:37:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503474+0000", 2024-02-05T07:37:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0556000000000001e-05 2024-02-05T07:37:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.503483+0000", 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8699999999999998e-06 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.554034+0000", 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0505512459999999 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.554148+0000", 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113948 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:37:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.554485+0000", 2024-02-05T07:37:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00033712500000000002 2024-02-05T07:37:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.561433+0000", 2024-02-05T07:37:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0069477769999999996 2024-02-05T07:37:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.561478+0000", 2024-02-05T07:37:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5457999999999997e-05 2024-02-05T07:37:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.565745+0000", 2024-02-05T07:37:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.004267374 2024-02-05T07:37:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.565805+0000", 2024-02-05T07:37:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9747999999999998e-05 2024-02-05T07:37:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.565855+0000", 2024-02-05T07:37:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9781000000000003e-05 2024-02-05T07:37:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:200448 8.c 8:3b8b152a:::benchmark_data_smithi007_105755_object64138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-02-05T07:37:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:28:12.525915+0000", 2024-02-05T07:37:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 570.84646856500001, 2024-02-05T07:37:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0402964589999999, 2024-02-05T07:37:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:37:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:37:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200448 2024-02-05T07:37:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.525915+0000", 2024-02-05T07:37:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.525915+0000", 2024-02-05T07:37:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.525917+0000", 2024-02-05T07:37:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8530000000000001e-06 2024-02-05T07:37:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.525925+0000", 2024-02-05T07:37:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7379999999999994e-06 2024-02-05T07:37:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.525946+0000", 2024-02-05T07:37:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1277e-05 2024-02-05T07:37:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.525957+0000", 2024-02-05T07:37:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0308e-05 2024-02-05T07:37:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.526027+0000", 2024-02-05T07:37:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0742999999999997e-05 2024-02-05T07:37:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:12.526038+0000", 2024-02-05T07:37:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.094e-05 2024-02-05T07:37:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.559347+0000", 2024-02-05T07:37:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0333087940000001 2024-02-05T07:37:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.559469+0000", 2024-02-05T07:37:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012193600000000001 2024-02-05T07:37:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.559585+0000", 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011568799999999999 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.565685+0000", 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0061002809999999999 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.565915+0000", 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023005000000000001 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.566126+0000", 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00021152399999999999 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.566177+0000", 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0293000000000001e-05 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:28:13.566212+0000", 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5033999999999997e-05 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:271911 8.13 8:ce51e4df:::benchmark_data_smithi007_105755_object135601:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:37:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.609391+0000", 2024-02-05T07:37:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 495.762992876, 2024-02-05T07:37:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.103982032, 2024-02-05T07:37:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:37:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:37:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:37:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 271911 2024-02-05T07:37:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609391+0000", 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609391+0000", 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609392+0000", 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.252e-06 2024-02-05T07:37:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609396+0000", 2024-02-05T07:37:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7419999999999999e-06 2024-02-05T07:37:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609405+0000", 2024-02-05T07:37:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0149999999999992e-06 2024-02-05T07:37:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609413+0000", 2024-02-05T07:37:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6399999999999997e-06 2024-02-05T07:37:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609478+0000", 2024-02-05T07:37:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4993999999999995e-05 2024-02-05T07:37:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-02-05T07:37:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.609485+0000", 2024-02-05T07:37:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4630000000000004e-06 2024-02-05T07:37:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.618256+0000", 2024-02-05T07:37:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0087711200000000003 2024-02-05T07:37:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.618265+0000", 2024-02-05T07:37:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7169999999999997e-06 2024-02-05T07:37:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.683143+0000", 2024-02-05T07:37:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0648781140000001 2024-02-05T07:37:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for missing object", 2024-02-05T07:37:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.683264+0000", 2024-02-05T07:37:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121237 2024-02-05T07:37:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.713247+0000", 2024-02-05T07:37:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.029982775999999999 2024-02-05T07:37:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.713373+0000", 2024-02-05T07:37:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000125962 2024-02-05T07:37:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:271929 8.8 8:15173cd4:::benchmark_data_smithi007_105755_object135619:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:37:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.624345+0000", 2024-02-05T07:37:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 495.748039205, 2024-02-05T07:37:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0390269809999999, 2024-02-05T07:37:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:37:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 271929 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624345+0000", 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624345+0000", 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624345+0000", 2024-02-05T07:37:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4000000000000001e-07 2024-02-05T07:37:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624349+0000", 2024-02-05T07:37:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9779999999999999e-06 2024-02-05T07:37:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624359+0000", 2024-02-05T07:37:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6309999999999998e-06 2024-02-05T07:37:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624366+0000", 2024-02-05T07:37:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0729999999999999e-06 2024-02-05T07:37:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624778+0000", 2024-02-05T07:37:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00041162499999999999 2024-02-05T07:37:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.624786+0000", 2024-02-05T07:37:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5339999999999999e-06 2024-02-05T07:37:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.653311+0000", 2024-02-05T07:37:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0285250699999999 2024-02-05T07:37:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.653431+0000", 2024-02-05T07:37:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000119527 2024-02-05T07:37:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:37:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.653812+0000", 2024-02-05T07:37:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00038111800000000001 2024-02-05T07:37:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.659043+0000", 2024-02-05T07:37:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0052313819999999997 2024-02-05T07:37:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.663156+0000", 2024-02-05T07:37:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041125220000000004 2024-02-05T07:37:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.663222+0000", 2024-02-05T07:37:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5782000000000001e-05 2024-02-05T07:37:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.663321+0000", 2024-02-05T07:37:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9026999999999999e-05 2024-02-05T07:37:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.663372+0000", 2024-02-05T07:37:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0971999999999999e-05 2024-02-05T07:37:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:271931 8.8 8:10a7a263:::benchmark_data_smithi007_105755_object135621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:37:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.630044+0000", 2024-02-05T07:37:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 495.74233958399998, 2024-02-05T07:37:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.038097405, 2024-02-05T07:37:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:37:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:37:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 271931 2024-02-05T07:37:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630044+0000", 2024-02-05T07:37:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630044+0000", 2024-02-05T07:37:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630046+0000", 2024-02-05T07:37:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3990000000000001e-06 2024-02-05T07:37:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630049+0000", 2024-02-05T07:37:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3840000000000001e-06 2024-02-05T07:37:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630057+0000", 2024-02-05T07:37:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8429999999999993e-06 2024-02-05T07:37:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630063+0000", 2024-02-05T07:37:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4949999999999996e-06 2024-02-05T07:37:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630175+0000", 2024-02-05T07:37:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000111443 2024-02-05T07:37:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.630184+0000", 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1260000000000004e-06 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.659103+0000", 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.028919361 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.659235+0000", 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013129 2024-02-05T07:37:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-02-05T07:37:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.659391+0000", 2024-02-05T07:37:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015600100000000001 2024-02-05T07:37:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.663439+0000", 2024-02-05T07:37:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0040486960000000001 2024-02-05T07:37:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.663567+0000", 2024-02-05T07:37:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012772799999999999 2024-02-05T07:37:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.668046+0000", 2024-02-05T07:37:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044785299999999997 2024-02-05T07:37:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.668100+0000", 2024-02-05T07:37:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4119999999999997e-05 2024-02-05T07:37:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.668142+0000", 2024-02-05T07:37:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1989000000000001e-05 2024-02-05T07:37:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:271936 8.13 8:c86abfbb:::benchmark_data_smithi007_105755_object135626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:37:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.631907+0000", 2024-02-05T07:37:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 495.74047737500001, 2024-02-05T07:37:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0919809620000001, 2024-02-05T07:37:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:37:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:37:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 271936 2024-02-05T07:37:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.631907+0000", 2024-02-05T07:37:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.631907+0000", 2024-02-05T07:37:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.631907+0000", 2024-02-05T07:37:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1200000000000002e-07 2024-02-05T07:37:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.631911+0000", 2024-02-05T07:37:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5559999999999999e-06 2024-02-05T07:37:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.631920+0000", 2024-02-05T07:37:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8240000000000006e-06 2024-02-05T07:37:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.631927+0000", 2024-02-05T07:37:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2690000000000002e-06 2024-02-05T07:37:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.635499+0000", 2024-02-05T07:37:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003572062 2024-02-05T07:37:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.635509+0000", 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7799999999999995e-06 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.683531+0000", 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.048021976 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.683661+0000", 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013036600000000001 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:37:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.684156+0000", 2024-02-05T07:37:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00049498499999999998 2024-02-05T07:37:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722733+0000", 2024-02-05T07:37:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.038577127000000003 2024-02-05T07:37:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722799+0000", 2024-02-05T07:37:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6266999999999994e-05 2024-02-05T07:37:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722945+0000", 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014543100000000001 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.723147+0000", 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020186100000000001 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.723887+0000", 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00074074600000000003 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:271951 8.13 8:cc87e9b1:::benchmark_data_smithi007_105755_object135641:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.641980+0000", 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 495.73040356400003, 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0821172720000001, 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 271951 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.641980+0000", 2024-02-05T07:37:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.641980+0000", 2024-02-05T07:37:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.641981+0000", 2024-02-05T07:37:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.105e-06 2024-02-05T07:37:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.641986+0000", 2024-02-05T07:37:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4909999999999999e-06 2024-02-05T07:37:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.642000+0000", 2024-02-05T07:37:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4235e-05 2024-02-05T07:37:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.642008+0000", 2024-02-05T07:37:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6320000000000008e-06 2024-02-05T07:37:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.643114+0000", 2024-02-05T07:37:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001106459 2024-02-05T07:37:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.643125+0000", 2024-02-05T07:37:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0823e-05 2024-02-05T07:37:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.706220+0000", 2024-02-05T07:37:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.063094722 2024-02-05T07:37:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.706363+0000", 2024-02-05T07:37:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000142714 2024-02-05T07:37:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:37:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.706511+0000", 2024-02-05T07:37:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014864699999999999 2024-02-05T07:37:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722741+0000", 2024-02-05T07:37:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016229389 2024-02-05T07:37:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722884+0000", 2024-02-05T07:37:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014346699999999999 2024-02-05T07:37:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.723945+0000", 2024-02-05T07:37:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001060851 2024-02-05T07:37:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.723994+0000", 2024-02-05T07:37:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9403000000000001e-05 2024-02-05T07:37:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724098+0000", 2024-02-05T07:37:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103334 2024-02-05T07:37:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:271972 8.13 8:ca91f266:::benchmark_data_smithi007_105755_object135662:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.666032+0000", 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 495.70635227999998, 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0584022369999999, 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 271972 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666032+0000", 2024-02-05T07:37:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666032+0000", 2024-02-05T07:37:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666032+0000", 2024-02-05T07:37:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5799999999999999e-07 2024-02-05T07:37:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666036+0000", 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7079999999999999e-06 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666045+0000", 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7960000000000001e-06 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666052+0000", 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9480000000000002e-06 2024-02-05T07:37:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666191+0000", 2024-02-05T07:37:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013942699999999999 2024-02-05T07:37:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.666201+0000", 2024-02-05T07:37:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8309999999999996e-06 2024-02-05T07:37:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.708519+0000", 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.042318219 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.708648+0000", 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012891899999999999 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.708787+0000", 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013839200000000001 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722746+0000", 2024-02-05T07:37:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013959651 2024-02-05T07:37:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724143+0000", 2024-02-05T07:37:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0013966180000000001 2024-02-05T07:37:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724188+0000", 2024-02-05T07:37:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5382e-05 2024-02-05T07:37:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724284+0000", 2024-02-05T07:37:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5748999999999994e-05 2024-02-05T07:37:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724434+0000", 2024-02-05T07:37:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000149939 2024-02-05T07:37:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:272009 8.13 8:ce2194ad:::benchmark_data_smithi007_105755_object135699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:37:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.712912+0000", 2024-02-05T07:37:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 495.65947145600001, 2024-02-05T07:37:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0118669250000001, 2024-02-05T07:37:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:37:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:37:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 272009 2024-02-05T07:37:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.712912+0000", 2024-02-05T07:37:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.712912+0000", 2024-02-05T07:37:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.712914+0000", 2024-02-05T07:37:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3260000000000001e-06 2024-02-05T07:37:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.712919+0000", 2024-02-05T07:37:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9760000000000003e-06 2024-02-05T07:37:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.712935+0000", 2024-02-05T07:37:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5977e-05 2024-02-05T07:37:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.712943+0000", 2024-02-05T07:37:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.551e-06 2024-02-05T07:37:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.713018+0000", 2024-02-05T07:37:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4529000000000004e-05 2024-02-05T07:37:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.713029+0000", 2024-02-05T07:37:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1487e-05 2024-02-05T07:37:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.710502+0000", 2024-02-05T07:37:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99747268200000005 2024-02-05T07:37:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.710622+0000", 2024-02-05T07:37:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000120048 2024-02-05T07:37:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:37:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.710753+0000", 2024-02-05T07:37:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013055200000000001 2024-02-05T07:37:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722752+0000", 2024-02-05T07:37:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011999157999999999 2024-02-05T07:37:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724485+0000", 2024-02-05T07:37:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017332140000000001 2024-02-05T07:37:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724578+0000", 2024-02-05T07:37:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3110999999999997e-05 2024-02-05T07:37:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724671+0000", 2024-02-05T07:37:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2610999999999998e-05 2024-02-05T07:37:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724779+0000", 2024-02-05T07:37:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010870300000000001 2024-02-05T07:37:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24977.0:272018 8.13 8:ce136c69:::benchmark_data_smithi007_105755_object135708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:37:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:29:27.726075+0000", 2024-02-05T07:37:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 495.646308852, 2024-02-05T07:37:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99892678000000001, 2024-02-05T07:37:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24977", 2024-02-05T07:37:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:37:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 272018 2024-02-05T07:37:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726075+0000", 2024-02-05T07:37:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726075+0000", 2024-02-05T07:37:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726076+0000", 2024-02-05T07:37:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.235e-06 2024-02-05T07:37:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726081+0000", 2024-02-05T07:37:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8099999999999997e-06 2024-02-05T07:37:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726092+0000", 2024-02-05T07:37:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0865e-05 2024-02-05T07:37:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726100+0000", 2024-02-05T07:37:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9969999999999998e-06 2024-02-05T07:37:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726173+0000", 2024-02-05T07:37:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3363999999999998e-05 2024-02-05T07:37:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:27.726185+0000", 2024-02-05T07:37:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1368e-05 2024-02-05T07:37:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.713401+0000", 2024-02-05T07:37:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98721674500000001 2024-02-05T07:37:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.713510+0000", 2024-02-05T07:37:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108312 2024-02-05T07:37:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:37:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.713648+0000", 2024-02-05T07:37:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000137837 2024-02-05T07:37:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.722757+0000", 2024-02-05T07:37:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0091097469999999996 2024-02-05T07:37:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724532+0000", 2024-02-05T07:37:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017743100000000001 2024-02-05T07:37:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724827+0000", 2024-02-05T07:37:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029515699999999999 2024-02-05T07:37:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.724880+0000", 2024-02-05T07:37:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3392e-05 2024-02-05T07:37:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:29:28.725002+0000", 2024-02-05T07:37:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121641 2024-02-05T07:37:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:125089 9.7 9:e4c9f568:::benchmark_data_smithi007_136200_object45244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-02-05T07:37:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:39.293800+0000", 2024-02-05T07:37:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 304.07858356100002, 2024-02-05T07:37:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0385608829999999, 2024-02-05T07:37:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:37:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:37:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:37:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 125089 2024-02-05T07:37:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.293800+0000", 2024-02-05T07:37:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.293800+0000", 2024-02-05T07:37:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.460 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.460 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.293801+0000", 2024-02-05T07:37:43.460 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.13e-06 2024-02-05T07:37:43.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.460 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.460 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.460 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.293805+0000", 2024-02-05T07:37:43.460 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.63e-06 2024-02-05T07:37:43.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.460 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.460 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.460 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.293813+0000", 2024-02-05T07:37:43.460 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0560000000000005e-06 2024-02-05T07:37:43.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.461 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.293820+0000", 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5729999999999996e-06 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.293869+0000", 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8967000000000001e-05 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.294134+0000", 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026571800000000002 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:37:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.294253+0000", 2024-02-05T07:37:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011818 2024-02-05T07:37:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.310233+0000", 2024-02-05T07:37:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.015980755999999999 2024-02-05T07:37:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.310248+0000", 2024-02-05T07:37:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4812e-05 2024-02-05T07:37:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.326702+0000", 2024-02-05T07:37:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016454116000000001 2024-02-05T07:37:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.326712+0000", 2024-02-05T07:37:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0074e-05 2024-02-05T07:37:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.332273+0000", 2024-02-05T07:37:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.005561138 2024-02-05T07:37:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.332361+0000", 2024-02-05T07:37:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7732999999999995e-05 2024-02-05T07:37:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:125118 9.7 9:ec3ff33e:::benchmark_data_smithi007_136200_object45273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e404)", 2024-02-05T07:37:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:39.335732+0000", 2024-02-05T07:37:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 304.03665167100002, 2024-02-05T07:37:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.004064861, 2024-02-05T07:37:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:37:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:37:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 125118 2024-02-05T07:37:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335732+0000", 2024-02-05T07:37:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335732+0000", 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335733+0000", 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1650000000000001e-06 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335737+0000", 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3890000000000001e-06 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335747+0000", 2024-02-05T07:37:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0152e-05 2024-02-05T07:37:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335754+0000", 2024-02-05T07:37:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5379999999999997e-06 2024-02-05T07:37:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335831+0000", 2024-02-05T07:37:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6379999999999997e-05 2024-02-05T07:37:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:39.335857+0000", 2024-02-05T07:37:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5741e-05 2024-02-05T07:37:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.332771+0000", 2024-02-05T07:37:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99691459800000004 2024-02-05T07:37:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.332894+0000", 2024-02-05T07:37:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012235300000000001 2024-02-05T07:37:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:37:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.333042+0000", 2024-02-05T07:37:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014887799999999999 2024-02-05T07:37:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.339562+0000", 2024-02-05T07:37:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0065198979999999997 2024-02-05T07:37:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.339629+0000", 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6812999999999999e-05 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.339708+0000", 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9349000000000002e-05 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.339761+0000", 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2426999999999998e-05 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.339797+0000", 2024-02-05T07:37:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6180000000000003e-05 2024-02-05T07:37:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:125390 9.8 9:1235e142:::benchmark_data_smithi007_136200_object45545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-02-05T07:37:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:40.336843+0000", 2024-02-05T07:37:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 303.03554118900001, 2024-02-05T07:37:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0096065670000001, 2024-02-05T07:37:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 125390 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.336843+0000", 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.336843+0000", 2024-02-05T07:37:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.336844+0000", 2024-02-05T07:37:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.198e-06 2024-02-05T07:37:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.336848+0000", 2024-02-05T07:37:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0609999999999997e-06 2024-02-05T07:37:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.336859+0000", 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0975000000000001e-05 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.336867+0000", 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3140000000000004e-06 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.336997+0000", 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012979699999999999 2024-02-05T07:37:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.337013+0000", 2024-02-05T07:37:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6231000000000001e-05 2024-02-05T07:37:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.341645+0000", 2024-02-05T07:37:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0046315459999999 2024-02-05T07:37:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.341802+0000", 2024-02-05T07:37:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015739000000000001 2024-02-05T07:37:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:37:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.341952+0000", 2024-02-05T07:37:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015017599999999999 2024-02-05T07:37:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.344427+0000", 2024-02-05T07:37:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024746460000000001 2024-02-05T07:37:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.346280+0000", 2024-02-05T07:37:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0018532310000000001 2024-02-05T07:37:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.346348+0000", 2024-02-05T07:37:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7379000000000007e-05 2024-02-05T07:37:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.346405+0000", 2024-02-05T07:37:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7061000000000001e-05 2024-02-05T07:37:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.346449+0000", 2024-02-05T07:37:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4561999999999998e-05 2024-02-05T07:37:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:125406 9.8 9:19e233ec:::benchmark_data_smithi007_136200_object45561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-02-05T07:37:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:40.358368+0000", 2024-02-05T07:37:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 303.01401567300002, 2024-02-05T07:37:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99250793900000001, 2024-02-05T07:37:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:37:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:37:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 125406 2024-02-05T07:37:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358368+0000", 2024-02-05T07:37:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358368+0000", 2024-02-05T07:37:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358369+0000", 2024-02-05T07:37:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.111e-06 2024-02-05T07:37:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358373+0000", 2024-02-05T07:37:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8489999999999999e-06 2024-02-05T07:37:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358385+0000", 2024-02-05T07:37:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1976e-05 2024-02-05T07:37:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358395+0000", 2024-02-05T07:37:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6900000000000004e-06 2024-02-05T07:37:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358702+0000", 2024-02-05T07:37:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030728500000000002 2024-02-05T07:37:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.358712+0000", 2024-02-05T07:37:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7149999999999993e-06 2024-02-05T07:37:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.344459+0000", 2024-02-05T07:37:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98574705799999995 2024-02-05T07:37:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.344597+0000", 2024-02-05T07:37:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013843500000000001 2024-02-05T07:37:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:37:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.344688+0000", 2024-02-05T07:37:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0884999999999995e-05 2024-02-05T07:37:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.350688+0000", 2024-02-05T07:37:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0060001389999999998 2024-02-05T07:37:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.350752+0000", 2024-02-05T07:37:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4022000000000005e-05 2024-02-05T07:37:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.350781+0000", 2024-02-05T07:37:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.813e-05 2024-02-05T07:37:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.350831+0000", 2024-02-05T07:37:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0778000000000001e-05 2024-02-05T07:37:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.350876+0000", 2024-02-05T07:37:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4866e-05 2024-02-05T07:37:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:125412 9.8 9:1a6cf359:::benchmark_data_smithi007_136200_object45567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-02-05T07:37:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:40.362668+0000", 2024-02-05T07:37:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 303.009716389, 2024-02-05T07:37:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99224961300000003, 2024-02-05T07:37:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:37:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:37:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 125412 2024-02-05T07:37:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362668+0000", 2024-02-05T07:37:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362668+0000", 2024-02-05T07:37:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362669+0000", 2024-02-05T07:37:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.263e-06 2024-02-05T07:37:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362672+0000", 2024-02-05T07:37:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3000000000000002e-06 2024-02-05T07:37:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362680+0000", 2024-02-05T07:37:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6510000000000001e-06 2024-02-05T07:37:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362686+0000", 2024-02-05T07:37:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2979999999999997e-06 2024-02-05T07:37:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362836+0000", 2024-02-05T07:37:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014964899999999999 2024-02-05T07:37:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.362846+0000", 2024-02-05T07:37:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0237e-05 2024-02-05T07:37:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.346606+0000", 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98375975400000004 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.346719+0000", 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011328999999999999 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.346808+0000", 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8731000000000001e-05 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.350945+0000", 2024-02-05T07:37:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041377760000000001 2024-02-05T07:37:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.350963+0000", 2024-02-05T07:37:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.753e-05 2024-02-05T07:37:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.354797+0000", 2024-02-05T07:37:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003834456 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.354860+0000", 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2292000000000006e-05 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.354917+0000", 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7386000000000002e-05 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:125415 9.8 9:1d0f3dec:::benchmark_data_smithi007_136200_object45570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-02-05T07:37:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:40.368752+0000", 2024-02-05T07:37:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 303.00363213600002, 2024-02-05T07:37:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98631972800000001, 2024-02-05T07:37:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:37:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:37:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 125415 2024-02-05T07:37:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368752+0000", 2024-02-05T07:37:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368752+0000", 2024-02-05T07:37:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368754+0000", 2024-02-05T07:37:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9489999999999999e-06 2024-02-05T07:37:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368759+0000", 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4770000000000002e-06 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368777+0000", 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7572999999999999e-05 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368786+0000", 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7839999999999992e-06 2024-02-05T07:37:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368885+0000", 2024-02-05T07:37:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9445999999999996e-05 2024-02-05T07:37:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:40.368895+0000", 2024-02-05T07:37:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0104e-05 2024-02-05T07:37:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.348592+0000", 2024-02-05T07:37:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97969647900000001 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.348695+0000", 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103731 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.348801+0000", 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000105932 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.351003+0000", 2024-02-05T07:37:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0022017790000000001 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.351014+0000", 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0637e-05 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.354982+0000", 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039685950000000001 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.355034+0000", 2024-02-05T07:37:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1740000000000003e-05 2024-02-05T07:37:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:41.355071+0000", 2024-02-05T07:37:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7502000000000001e-05 2024-02-05T07:37:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:158058 10.15 10:a899e25f:::benchmark_data_smithi007_157098_object29128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-02-05T07:37:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:46.722616+0000", 2024-02-05T07:37:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 116.64976787099999, 2024-02-05T07:37:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.93586535000000004, 2024-02-05T07:37:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 158058 2024-02-05T07:37:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.722616+0000", 2024-02-05T07:37:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.722616+0000", 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.722617+0000", 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4699999999999999e-06 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.722622+0000", 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9760000000000003e-06 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.722635+0000", 2024-02-05T07:37:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2414e-05 2024-02-05T07:37:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.722644+0000", 2024-02-05T07:37:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8850000000000005e-06 2024-02-05T07:37:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.722711+0000", 2024-02-05T07:37:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7224000000000006e-05 2024-02-05T07:37:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.722720+0000", 2024-02-05T07:37:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2310000000000002e-06 2024-02-05T07:37:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.650483+0000", 2024-02-05T07:37:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.92776255799999996 2024-02-05T07:37:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.650611+0000", 2024-02-05T07:37:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000127751 2024-02-05T07:37:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:37:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.650942+0000", 2024-02-05T07:37:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00033108200000000001 2024-02-05T07:37:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.655424+0000", 2024-02-05T07:37:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044823149999999997 2024-02-05T07:37:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.655510+0000", 2024-02-05T07:37:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6026999999999995e-05 2024-02-05T07:37:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.658325+0000", 2024-02-05T07:37:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0028150710000000002 2024-02-05T07:37:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.658438+0000", 2024-02-05T07:37:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0001127 2024-02-05T07:37:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.658481+0000", 2024-02-05T07:37:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3646000000000002e-05 2024-02-05T07:37:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:158069 10.15 10:abf97c57:::benchmark_data_smithi007_157098_object29139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-02-05T07:37:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:46.733317+0000", 2024-02-05T07:37:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 116.639066684, 2024-02-05T07:37:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.92539375199999996, 2024-02-05T07:37:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:37:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:37:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:37:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:37:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:37:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 158069 2024-02-05T07:37:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:37:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:37:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.733317+0000", 2024-02-05T07:37:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:37:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.733317+0000", 2024-02-05T07:37:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:37:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:37:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.733319+0000", 2024-02-05T07:37:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6810000000000001e-06 2024-02-05T07:37:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:37:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.733324+0000", 2024-02-05T07:37:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3589999999999999e-06 2024-02-05T07:37:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:37:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.733340+0000", 2024-02-05T07:37:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5656999999999999e-05 2024-02-05T07:37:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:37:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.733348+0000", 2024-02-05T07:37:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2109999999999998e-06 2024-02-05T07:37:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.733440+0000", 2024-02-05T07:37:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2031e-05 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:46.733449+0000", 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1500000000000005e-06 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.652553+0000", 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.91910343900000002 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.652635+0000", 2024-02-05T07:37:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2587000000000001e-05 2024-02-05T07:37:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:37:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.652820+0000", 2024-02-05T07:37:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018459999999999999 2024-02-05T07:37:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:37:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.655522+0000", 2024-02-05T07:37:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0027023239999999999 2024-02-05T07:37:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.658563+0000", 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003040255 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.658621+0000", 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8867999999999999e-05 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.658674+0000", 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2873e-05 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:37:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:47.658711+0000", 2024-02-05T07:37:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6717000000000001e-05 2024-02-05T07:37:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:43.514 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:37:43.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:43.538 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:43.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:43.639 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:43.664 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:43.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:43.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:43.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:43.792 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:43.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:43.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:43.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:43.906 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:37:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:37:43.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[29098]: 1.1b scrub starts 2024-02-05T07:37:43.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[29098]: 1.1b scrub ok 2024-02-05T07:37:43.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[29098]: 1.19 scrub starts 2024-02-05T07:37:43.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[29098]: 1.19 scrub ok 2024-02-05T07:37:43.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[29098]: 1.18 scrub starts 2024-02-05T07:37:43.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[29098]: 1.18 scrub ok 2024-02-05T07:37:43.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:37:43.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[29098]: osdmap e578: 8 total, 8 up, 6 in 2024-02-05T07:37:43.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[29098]: 1.16 scrub starts 2024-02-05T07:37:43.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[29098]: 1.16 scrub ok 2024-02-05T07:37:43.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[37158]: 1.1b scrub starts 2024-02-05T07:37:43.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[37158]: 1.1b scrub ok 2024-02-05T07:37:43.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[37158]: 1.19 scrub starts 2024-02-05T07:37:43.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[37158]: 1.19 scrub ok 2024-02-05T07:37:43.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[37158]: 1.18 scrub starts 2024-02-05T07:37:43.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[37158]: 1.18 scrub ok 2024-02-05T07:37:43.908 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:37:43.908 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[37158]: osdmap e578: 8 total, 8 up, 6 in 2024-02-05T07:37:43.908 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[37158]: 1.16 scrub starts 2024-02-05T07:37:43.908 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:43 smithi007 ceph-mon[37158]: 1.16 scrub ok 2024-02-05T07:37:43.921 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:43.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:43.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:43.940 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T07:37:43.941 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T07:37:43.941 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-02-05T07:37:43.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-02-05T07:37:44.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:43 smithi090 ceph-mon[33213]: 1.1b scrub starts 2024-02-05T07:37:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:43 smithi090 ceph-mon[33213]: 1.1b scrub ok 2024-02-05T07:37:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:43 smithi090 ceph-mon[33213]: 1.19 scrub starts 2024-02-05T07:37:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:43 smithi090 ceph-mon[33213]: 1.19 scrub ok 2024-02-05T07:37:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:43 smithi090 ceph-mon[33213]: 1.18 scrub starts 2024-02-05T07:37:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:43 smithi090 ceph-mon[33213]: 1.18 scrub ok 2024-02-05T07:37:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:43 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:37:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:43 smithi090 ceph-mon[33213]: osdmap e578: 8 total, 8 up, 6 in 2024-02-05T07:37:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:43 smithi090 ceph-mon[33213]: 1.16 scrub starts 2024-02-05T07:37:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:43 smithi090 ceph-mon[33213]: 1.16 scrub ok 2024-02-05T07:37:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:44.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:44.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:44.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:44.178 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:44.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:44.280 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:44.303 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:44.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:44.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:44.433 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:44.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:44.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:44.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:44.563 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:44.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:44.665 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:44.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:44.692 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:44.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:44.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:44.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:44.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:44.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:44.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:44.945 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:44.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:44.970 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[29098]: 1.7 scrub starts 2024-02-05T07:37:44.970 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[29098]: 1.7 scrub ok 2024-02-05T07:37:44.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[29098]: 3.f scrub starts 2024-02-05T07:37:44.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[29098]: 3.f scrub ok 2024-02-05T07:37:44.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[29098]: pgmap v1043: 162 pgs: 1 peering, 1 active+clean+remapped, 2 remapped+peering, 2 unknown, 3 active+remapped+backfill_toofull, 153 active+clean; 601 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:37:44.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[29098]: 1.1e scrub starts 2024-02-05T07:37:44.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[29098]: 1.1e scrub ok 2024-02-05T07:37:44.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[29098]: 1.1c scrub starts 2024-02-05T07:37:44.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[29098]: 1.1c scrub ok 2024-02-05T07:37:44.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:37:44.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[29098]: 5.7 scrub starts 2024-02-05T07:37:44.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[29098]: 5.7 scrub ok 2024-02-05T07:37:44.971 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[37158]: 1.7 scrub starts 2024-02-05T07:37:44.971 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[37158]: 1.7 scrub ok 2024-02-05T07:37:44.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[37158]: 3.f scrub starts 2024-02-05T07:37:44.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[37158]: 3.f scrub ok 2024-02-05T07:37:44.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[37158]: pgmap v1043: 162 pgs: 1 peering, 1 active+clean+remapped, 2 remapped+peering, 2 unknown, 3 active+remapped+backfill_toofull, 153 active+clean; 601 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:37:44.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[37158]: 1.1e scrub starts 2024-02-05T07:37:44.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[37158]: 1.1e scrub ok 2024-02-05T07:37:44.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[37158]: 1.1c scrub starts 2024-02-05T07:37:44.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[37158]: 1.1c scrub ok 2024-02-05T07:37:44.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:37:44.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[37158]: 5.7 scrub starts 2024-02-05T07:37:44.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:44 smithi007 ceph-mon[37158]: 5.7 scrub ok 2024-02-05T07:37:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:44 smithi090 ceph-mon[33213]: 1.7 scrub starts 2024-02-05T07:37:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:44 smithi090 ceph-mon[33213]: 1.7 scrub ok 2024-02-05T07:37:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:44 smithi090 ceph-mon[33213]: 3.f scrub starts 2024-02-05T07:37:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:44 smithi090 ceph-mon[33213]: 3.f scrub ok 2024-02-05T07:37:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:44 smithi090 ceph-mon[33213]: pgmap v1043: 162 pgs: 1 peering, 1 active+clean+remapped, 2 remapped+peering, 2 unknown, 3 active+remapped+backfill_toofull, 153 active+clean; 601 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:37:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:44 smithi090 ceph-mon[33213]: 1.1e scrub starts 2024-02-05T07:37:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:44 smithi090 ceph-mon[33213]: 1.1e scrub ok 2024-02-05T07:37:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:44 smithi090 ceph-mon[33213]: 1.1c scrub starts 2024-02-05T07:37:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:44 smithi090 ceph-mon[33213]: 1.1c scrub ok 2024-02-05T07:37:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:44 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:37:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:44 smithi090 ceph-mon[33213]: 5.7 scrub starts 2024-02-05T07:37:45.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:44 smithi090 ceph-mon[33213]: 5.7 scrub ok 2024-02-05T07:37:45.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:45.074 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:45.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:45.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:45.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:45.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:45.202 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:45.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:45.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:45.324 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:45.328 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:45.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:45.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:45.456 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:45.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:45.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:45.581 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:45.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:45.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:45.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:45.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:45.834 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:45.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:45.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:45.936 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:45.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:45.961 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:45.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:45.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:45 smithi007 ceph-mon[29098]: 3.1e scrub starts 2024-02-05T07:37:45.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:45 smithi007 ceph-mon[29098]: 3.1e scrub ok 2024-02-05T07:37:45.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:45 smithi007 ceph-mon[37158]: 3.1e scrub starts 2024-02-05T07:37:45.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:45 smithi007 ceph-mon[37158]: 3.1e scrub ok 2024-02-05T07:37:46.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T07:37:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:45 smithi090 ceph-mon[33213]: 3.1e scrub starts 2024-02-05T07:37:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:45 smithi090 ceph-mon[33213]: 3.1e scrub ok 2024-02-05T07:37:46.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:46.086 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:46.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:46.188 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:46.212 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:46.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:46.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:46.267 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:37:46.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:46.339 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:46.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:46.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:46.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:46.464 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:46.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:46.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:46.590 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:46.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:46.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:46.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:46.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:46.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:46.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[37158]: 5.18 scrub starts 2024-02-05T07:37:46.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[37158]: 5.18 scrub ok 2024-02-05T07:37:46.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[37158]: pgmap v1044: 162 pgs: 1 active+clean+remapped, 1 unknown, 3 active+remapped+backfill_toofull, 157 active+clean; 584 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:46.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[37158]: 5.a scrub starts 2024-02-05T07:37:46.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[37158]: 5.a scrub ok 2024-02-05T07:37:46.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[37158]: 4.5 scrub starts 2024-02-05T07:37:46.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[37158]: 4.5 scrub ok 2024-02-05T07:37:46.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[37158]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-02-05T07:37:46.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:37:46.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[29098]: 5.18 scrub starts 2024-02-05T07:37:46.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[29098]: 5.18 scrub ok 2024-02-05T07:37:46.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[29098]: pgmap v1044: 162 pgs: 1 active+clean+remapped, 1 unknown, 3 active+remapped+backfill_toofull, 157 active+clean; 584 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:46.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[29098]: 5.a scrub starts 2024-02-05T07:37:46.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[29098]: 5.a scrub ok 2024-02-05T07:37:46.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[29098]: 4.5 scrub starts 2024-02-05T07:37:46.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[29098]: 4.5 scrub ok 2024-02-05T07:37:46.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[29098]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-02-05T07:37:46.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:37:46.817 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:46.832 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:46.841 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:46.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:46.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:46.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:46.969 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:46.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:46 smithi090 ceph-mon[33213]: 5.18 scrub starts 2024-02-05T07:37:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:46 smithi090 ceph-mon[33213]: 5.18 scrub ok 2024-02-05T07:37:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:46 smithi090 ceph-mon[33213]: pgmap v1044: 162 pgs: 1 active+clean+remapped, 1 unknown, 3 active+remapped+backfill_toofull, 157 active+clean; 584 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:46 smithi090 ceph-mon[33213]: 5.a scrub starts 2024-02-05T07:37:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:46 smithi090 ceph-mon[33213]: 5.a scrub ok 2024-02-05T07:37:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:46 smithi090 ceph-mon[33213]: 4.5 scrub starts 2024-02-05T07:37:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:46 smithi090 ceph-mon[33213]: 4.5 scrub ok 2024-02-05T07:37:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:46 smithi090 ceph-mon[33213]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-02-05T07:37:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:37:47.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:47.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:47.095 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:47.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:47.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:47.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:47.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:47.347 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:47.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:47.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:47.473 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:47.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:47.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:47.600 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:47.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:47.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:47.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:47.727 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:47.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:47.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:47.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:47.854 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:47.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:47.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:47.958 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:47 smithi007 ceph-mon[29098]: 3.a scrub starts 2024-02-05T07:37:47.958 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:47 smithi007 ceph-mon[29098]: 3.a scrub ok 2024-02-05T07:37:47.958 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:47 smithi007 ceph-mon[37158]: 3.a scrub starts 2024-02-05T07:37:47.958 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:47 smithi007 ceph-mon[37158]: 3.a scrub ok 2024-02-05T07:37:47.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:47.981 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:47.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:47.990 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:47 smithi090 ceph-mon[33213]: 3.a scrub starts 2024-02-05T07:37:47.990 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:47 smithi090 ceph-mon[33213]: 3.a scrub ok 2024-02-05T07:37:48.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:48.106 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:48.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:48.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:48.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:48.232 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:48.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:48.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:48.323 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:37:48.323 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:37:48.324 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-02-05T07:37:48.324 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:37:48.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:48.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:48.361 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:48.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:48.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:48.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:48.488 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:48.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:48.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:48.616 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:48.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:48.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:48.744 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:48.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:48.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:48.871 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:48.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:48.877 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:48 smithi007 ceph-mon[29098]: 6.14 scrub starts 2024-02-05T07:37:48.877 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:48 smithi007 ceph-mon[29098]: 6.14 scrub ok 2024-02-05T07:37:48.877 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:48 smithi007 ceph-mon[29098]: pgmap v1045: 162 pgs: 3 active+remapped+backfill_toofull, 159 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:48.877 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:48 smithi007 ceph-mon[29098]: 3.10 scrub starts 2024-02-05T07:37:48.877 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:48 smithi007 ceph-mon[29098]: 3.10 scrub ok 2024-02-05T07:37:48.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:48 smithi007 ceph-mon[37158]: 6.14 scrub starts 2024-02-05T07:37:48.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:48 smithi007 ceph-mon[37158]: 6.14 scrub ok 2024-02-05T07:37:48.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:48 smithi007 ceph-mon[37158]: pgmap v1045: 162 pgs: 3 active+remapped+backfill_toofull, 159 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:48.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:48 smithi007 ceph-mon[37158]: 3.10 scrub starts 2024-02-05T07:37:48.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:48 smithi007 ceph-mon[37158]: 3.10 scrub ok 2024-02-05T07:37:48.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:49.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:49.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:49.002 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:48 smithi090 ceph-mon[33213]: 6.14 scrub starts 2024-02-05T07:37:49.002 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:48 smithi090 ceph-mon[33213]: 6.14 scrub ok 2024-02-05T07:37:49.002 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:48 smithi090 ceph-mon[33213]: pgmap v1045: 162 pgs: 3 active+remapped+backfill_toofull, 159 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:49.002 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:48 smithi090 ceph-mon[33213]: 3.10 scrub starts 2024-02-05T07:37:49.003 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:48 smithi090 ceph-mon[33213]: 3.10 scrub ok 2024-02-05T07:37:49.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T07:37:49.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:49.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:49.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:49.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:49.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:49.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:49.254 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:49.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:37:49.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:49.381 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:49.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:49.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:49.507 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:49.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:49.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:49.635 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:49.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:49.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:49.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:49.762 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:49.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:49.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:49.887 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:49.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:49.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:49 smithi007 ceph-mon[29098]: 4.13 scrub starts 2024-02-05T07:37:49.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:49 smithi007 ceph-mon[29098]: 4.13 scrub ok 2024-02-05T07:37:49.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:49 smithi007 ceph-mon[37158]: 4.13 scrub starts 2024-02-05T07:37:49.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:49 smithi007 ceph-mon[37158]: 4.13 scrub ok 2024-02-05T07:37:49.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:50.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:50.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:50.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:49 smithi090 ceph-mon[33213]: 4.13 scrub starts 2024-02-05T07:37:50.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:49 smithi090 ceph-mon[33213]: 4.13 scrub ok 2024-02-05T07:37:50.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:50.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:50.140 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:50.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:50.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:50.266 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:50.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:50.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:50.392 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:50.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:50.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:50.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:50.518 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:50.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:50.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:50.631 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:37:50.631 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:37:50.631 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 578, 2024-02-05T07:37:50.631 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "e3e7059e-c3f5-11ee-95b6-87774f69a715", 2024-02-05T07:37:50.631 INFO:teuthology.orchestra.run.smithi007.stdout: "created": "2024-02-05T07:13:14.603885+0000", 2024-02-05T07:37:50.631 INFO:teuthology.orchestra.run.smithi007.stdout: "modified": "2024-02-05T07:37:42.700254+0000", 2024-02-05T07:37:50.632 INFO:teuthology.orchestra.run.smithi007.stdout: "last_up_change": "2024-02-05T07:28:24.555976+0000", 2024-02-05T07:37:50.632 INFO:teuthology.orchestra.run.smithi007.stdout: "last_in_change": "2024-02-05T07:36:31.668197+0000", 2024-02-05T07:37:50.632 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T07:37:50.632 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_num": 5804032, 2024-02-05T07:37:50.632 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_set": [ 2024-02-05T07:37:50.632 INFO:teuthology.orchestra.run.smithi007.stdout: "nodeep-scrub", 2024-02-05T07:37:50.632 INFO:teuthology.orchestra.run.smithi007.stdout: "pglog_hardlimit", 2024-02-05T07:37:50.632 INFO:teuthology.orchestra.run.smithi007.stdout: "purged_snapdirs", 2024-02-05T07:37:50.632 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_deletes", 2024-02-05T07:37:50.632 INFO:teuthology.orchestra.run.smithi007.stdout: "sortbitwise" 2024-02-05T07:37:50.632 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:37:50.632 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_version": 19, 2024-02-05T07:37:50.632 INFO:teuthology.orchestra.run.smithi007.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T07:37:50.632 INFO:teuthology.orchestra.run.smithi007.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T07:37:50.633 INFO:teuthology.orchestra.run.smithi007.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T07:37:50.633 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_snapshot": "", 2024-02-05T07:37:50.633 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_max": 10, 2024-02-05T07:37:50.633 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd": 8, 2024-02-05T07:37:50.633 INFO:teuthology.orchestra.run.smithi007.stdout: "require_min_compat_client": "luminous", 2024-02-05T07:37:50.633 INFO:teuthology.orchestra.run.smithi007.stdout: "min_compat_client": "luminous", 2024-02-05T07:37:50.633 INFO:teuthology.orchestra.run.smithi007.stdout: "require_osd_release": "squid", 2024-02-05T07:37:50.633 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_crimson": false, 2024-02-05T07:37:50.633 INFO:teuthology.orchestra.run.smithi007.stdout: "pools": [ 2024-02-05T07:37:50.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.633 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 1, 2024-02-05T07:37:50.633 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".mgr", 2024-02-05T07:37:50.633 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:16:32.505567+0000", 2024-02-05T07:37:50.633 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:37:50.634 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:37:50.634 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:37:50.634 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:37:50.634 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:37:50.634 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:37:50.634 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:37:50.634 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:37:50.634 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:37:50.634 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:37:50.634 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:37:50.634 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:37:50.634 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 31, 2024-02-05T07:37:50.634 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 31, 2024-02-05T07:37:50.635 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 31, 2024-02-05T07:37:50.635 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 31, 2024-02-05T07:37:50.635 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 31, 2024-02-05T07:37:50.635 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:37:50.635 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "1.b", 2024-02-05T07:37:50.635 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 543, 2024-02-05T07:37:50.635 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 542, 2024-02-05T07:37:50.635 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 542, 2024-02-05T07:37:50.635 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "86'128", 2024-02-05T07:37:50.635 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "86'128" 2024-02-05T07:37:50.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.635 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "573", 2024-02-05T07:37:50.635 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:37:50.635 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "543", 2024-02-05T07:37:50.636 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "571", 2024-02-05T07:37:50.636 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:37:50.636 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:37:50.636 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:37:50.636 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:37:50.636 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:37:50.636 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:37:50.636 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:37:50.636 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:37:50.636 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:37:50.636 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:37:50.636 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:37:50.636 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:37:50.636 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:37:50.637 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:37:50.637 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:37:50.637 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:37:50.637 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:37:50.637 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:37:50.637 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:37:50.637 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:37:50.637 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:37:50.637 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:37:50.637 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:37:50.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.637 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:37:50.637 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:37:50.637 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:37:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:37:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:37:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:37:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:37:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:37:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:37:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:37:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:37:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-02-05T07:37:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_max": 32, 2024-02-05T07:37:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_min": 1 2024-02-05T07:37:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:37:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": {} 2024-02-05T07:37:50.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.639 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:37:50.639 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:37:50.639 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.7999999523162842, 2024-02-05T07:37:50.639 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.7999999523162842, 2024-02-05T07:37:50.639 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:37:50.639 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.5499999523162842, 2024-02-05T07:37:50.639 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.5499999523162842, 2024-02-05T07:37:50.639 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.7850494384765625, 2024-02-05T07:37:50.639 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.89999997615814209, 2024-02-05T07:37:50.639 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-02-05T07:37:50.639 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.640 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 2, 2024-02-05T07:37:50.640 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "datapool", 2024-02-05T07:37:50.640 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:47.259462+0000", 2024-02-05T07:37:50.640 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 8193, 2024-02-05T07:37:50.640 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T07:37:50.640 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:37:50.640 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:37:50.640 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:37:50.644 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:37:50.644 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:37:50.644 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:37:50.644 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:37:50.644 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:37:50.644 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:37:50.644 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:37:50.644 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 3, 2024-02-05T07:37:50.644 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 3, 2024-02-05T07:37:50.644 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 3, 2024-02-05T07:37:50.644 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 3, 2024-02-05T07:37:50.644 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 3, 2024-02-05T07:37:50.644 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "71", 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 2, 2024-02-05T07:37:50.645 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 71, 2024-02-05T07:37:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:37:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:37:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:37:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:37:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:37:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:37:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:37:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:37:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:37:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:37:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:37:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:37:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:37:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:37:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:37:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:37:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:37:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:37:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:37:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd": {} 2024-02-05T07:37:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:37:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:37:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 4.690000057220459, 2024-02-05T07:37:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 4.690000057220459, 2024-02-05T07:37:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.70999997854232788, 2024-02-05T07:37:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 3.3299999237060547, 2024-02-05T07:37:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 3.3299999237060547, 2024-02-05T07:37:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.7850494384765625, 2024-02-05T07:37:50.649 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.89999997615814209, 2024-02-05T07:37:50.649 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-02-05T07:37:50.649 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.649 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.649 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 3, 2024-02-05T07:37:50.649 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".rgw.root", 2024-02-05T07:37:50.649 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:47.762336+0000", 2024-02-05T07:37:50.649 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:37:50.649 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:37:50.649 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:37:50.649 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:37:50.649 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:37:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:37:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:37:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:37:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:37:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:37:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:37:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:37:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:37:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:37:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:37:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:37:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:37:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:37:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:37:50.651 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:37:50.651 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:37:50.652 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:37:50.652 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:37:50.652 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:37:50.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.652 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "69", 2024-02-05T07:37:50.652 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:37:50.652 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:37:50.652 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:37:50.652 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:37:50.652 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:37:50.652 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:37:50.652 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:37:50.652 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:37:50.653 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:37:50.653 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:37:50.653 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:37:50.653 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:37:50.653 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:37:50.653 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:37:50.653 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:37:50.653 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:37:50.653 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:37:50.653 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:37:50.653 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:37:50.653 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:37:50.653 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:37:50.653 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:37:50.654 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:37:50.655 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:37:50.655 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:37:50.655 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:37:50.655 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:37:50.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.655 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:37:50.655 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:37:50.655 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.4000000953674316, 2024-02-05T07:37:50.655 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.4000000953674316, 2024-02-05T07:37:50.655 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:37:50.655 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.059999942779541, 2024-02-05T07:37:50.655 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.059999942779541, 2024-02-05T07:37:50.655 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.7850494384765625, 2024-02-05T07:37:50.656 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.89999997615814209, 2024-02-05T07:37:50.656 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-02-05T07:37:50.656 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.656 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.656 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 4, 2024-02-05T07:37:50.656 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.log", 2024-02-05T07:37:50.656 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:49.736599+0000", 2024-02-05T07:37:50.656 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:37:50.656 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:37:50.656 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:37:50.656 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:37:50.656 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:37:50.656 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:37:50.657 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:37:50.657 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:37:50.657 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:37:50.657 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:37:50.657 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:37:50.657 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:37:50.657 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:37:50.657 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:37:50.657 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "71", 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:37:50.658 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:37:50.659 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:37:50.659 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:37:50.659 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:37:50.659 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:37:50.659 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:37:50.659 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:37:50.659 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:37:50.659 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:37:50.659 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:37:50.659 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:37:50.659 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:37:50.659 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:37:50.659 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:37:50.660 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:37:50.660 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:37:50.660 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:37:50.660 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:37:50.660 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:37:50.660 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:37:50.660 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:37:50.660 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:37:50.660 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:37:50.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.660 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:37:50.660 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:37:50.660 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:37:50.661 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.190000057220459, 2024-02-05T07:37:50.662 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.190000057220459, 2024-02-05T07:37:50.662 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:37:50.662 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.8799999952316284, 2024-02-05T07:37:50.662 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.8799999952316284, 2024-02-05T07:37:50.662 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.7850494384765625, 2024-02-05T07:37:50.662 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.89999997615814209, 2024-02-05T07:37:50.662 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-02-05T07:37:50.662 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.662 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.662 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 5, 2024-02-05T07:37:50.662 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.control", 2024-02-05T07:37:50.662 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:51.696948+0000", 2024-02-05T07:37:50.662 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:37:50.663 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:37:50.663 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:37:50.663 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:37:50.663 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:37:50.663 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:37:50.663 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:37:50.663 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:37:50.663 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:37:50.663 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:37:50.663 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:37:50.663 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:37:50.663 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:37:50.663 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:37:50.663 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:37:50.664 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:37:50.664 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:37:50.664 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:37:50.664 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:37:50.664 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:37:50.664 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:37:50.664 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:37:50.664 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:37:50.664 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:37:50.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.664 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "73", 2024-02-05T07:37:50.664 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:37:50.664 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:37:50.664 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:37:50.665 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:37:50.666 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9700000286102295, 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9700000286102295, 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.690000057220459, 2024-02-05T07:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.690000057220459, 2024-02-05T07:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.7850494384765625, 2024-02-05T07:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.89999997615814209, 2024-02-05T07:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-02-05T07:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 6, 2024-02-05T07:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.meta", 2024-02-05T07:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:53.768555+0000", 2024-02-05T07:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:37:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:37:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:37:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:37:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:37:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:37:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:37:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:37:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:37:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:37:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:37:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:37:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:37:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:37:50.669 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:37:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:37:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:37:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:37:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:37:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:37:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:37:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:37:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "75", 2024-02-05T07:37:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:37:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:37:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:37:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:37:50.670 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:37:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:37:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:37:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:37:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:37:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:37:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:37:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:37:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:37:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:37:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:37:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:37:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:37:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:37:50.671 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:37:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:37:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:37:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:37:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:37:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:37:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:37:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:37:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:37:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:37:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:37:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:37:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:37:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:37:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:37:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:37:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:37:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:37:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:37:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-02-05T07:37:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_bias": 4 2024-02-05T07:37:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:37:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:37:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:37:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:37:50.673 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9700000286102295, 2024-02-05T07:37:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9700000286102295, 2024-02-05T07:37:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-02-05T07:37:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.690000057220459, 2024-02-05T07:37:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.690000057220459, 2024-02-05T07:37:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.7850494384765625, 2024-02-05T07:37:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.89999997615814209, 2024-02-05T07:37:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-02-05T07:37:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:37:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:37:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-02-05T07:37:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "703c3948-8003-4015-9dd8-21019cf0eeb8", 2024-02-05T07:37:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:37:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:37:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:37:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0, 2024-02-05T07:37:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:37:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:37:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 8, 2024-02-05T07:37:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 575, 2024-02-05T07:37:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:37:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:37:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:37:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.676 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-02-05T07:37:50.676 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:37:50.676 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.676 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.676 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:37:50.676 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.676 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.676 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.676 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6802", 2024-02-05T07:37:50.676 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:37:50.676 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.676 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.677 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:37:50.677 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.677 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.677 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6804", 2024-02-05T07:37:50.677 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:37:50.677 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.677 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.677 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:37:50.677 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.677 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.677 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6803", 2024-02-05T07:37:50.678 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:37:50.678 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.678 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.678 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6801/2069016689", 2024-02-05T07:37:50.678 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6802/2069016689", 2024-02-05T07:37:50.678 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6804/2069016689", 2024-02-05T07:37:50.678 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6803/2069016689", 2024-02-05T07:37:50.678 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:37:50.678 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:37:50.678 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:37:50.678 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.678 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "5c3e7d1e-a61d-44b9-b21e-f91c99a096bc", 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0.5441131591796875, 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 13, 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 576, 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.680 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.680 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6805", 2024-02-05T07:37:50.680 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:37:50.680 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.680 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.680 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:37:50.680 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.680 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.680 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6806", 2024-02-05T07:37:50.680 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:37:50.680 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.680 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6808", 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.681 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6807", 2024-02-05T07:37:50.682 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:37:50.682 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.682 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.682 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6805/865486532", 2024-02-05T07:37:50.682 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6806/865486532", 2024-02-05T07:37:50.682 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6808/865486532", 2024-02-05T07:37:50.682 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6807/865486532", 2024-02-05T07:37:50.682 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:37:50.682 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:37:50.682 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:37:50.682 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.682 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "f4fd930f-7bb2-46ca-9964-98235bd47b3a", 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 18, 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 576, 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.683 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6809", 2024-02-05T07:37:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:37:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:37:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6810", 2024-02-05T07:37:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:37:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.685 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:37:50.685 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.685 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.685 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.685 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6812", 2024-02-05T07:37:50.685 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:37:50.685 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.685 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.685 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:37:50.685 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.685 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6811", 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6809/904871370", 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6810/904871370", 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6812/904871370", 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6811/904871370", 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "149ed0d6-b049-4478-9819-cafc446e94d3", 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 26, 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 576, 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:37:50.687 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.688 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.688 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.688 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6813", 2024-02-05T07:37:50.688 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:37:50.688 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.688 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.688 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.688 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:37:50.688 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.688 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.688 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.688 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6814", 2024-02-05T07:37:50.688 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6816", 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.689 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.690 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.690 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6815", 2024-02-05T07:37:50.690 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:37:50.690 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.690 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.690 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.690 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6813/2758586242", 2024-02-05T07:37:50.690 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6814/2758586242", 2024-02-05T07:37:50.690 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6816/2758586242", 2024-02-05T07:37:50.690 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6815/2758586242", 2024-02-05T07:37:50.690 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:37:50.690 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:37:50.690 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:37:50.691 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.691 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.691 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.691 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-02-05T07:37:50.691 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d", 2024-02-05T07:37:50.691 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:37:50.691 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:37:50.691 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:37:50.691 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:37:50.691 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:37:50.691 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:37:50.691 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 32, 2024-02-05T07:37:50.691 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 576, 2024-02-05T07:37:50.691 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:37:50.692 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:37:50.692 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:37:50.692 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.692 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.692 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6800", 2024-02-05T07:37:50.692 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:37:50.692 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.692 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.692 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:37:50.692 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.692 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.693 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6801", 2024-02-05T07:37:50.693 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:37:50.693 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.693 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.693 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:37:50.693 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.693 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.693 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6803", 2024-02-05T07:37:50.693 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:37:50.693 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.693 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6802", 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6800/1154929470", 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6801/1154929470", 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6803/1154929470", 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6802/1154929470", 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:37:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:37:50.695 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:37:50.695 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.695 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-02-05T07:37:50.695 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "244a7930-c22c-413e-86e1-11e0749f7367", 2024-02-05T07:37:50.695 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:37:50.695 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-02-05T07:37:50.695 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-02-05T07:37:50.695 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0.7362823486328125, 2024-02-05T07:37:50.695 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:37:50.695 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:37:50.695 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 41, 2024-02-05T07:37:50.695 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 336, 2024-02-05T07:37:50.696 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:37:50.696 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:37:50.696 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:37:50.696 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.696 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.696 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.696 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6804", 2024-02-05T07:37:50.696 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:37:50.696 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.696 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.696 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:37:50.696 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.696 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.697 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.697 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6805", 2024-02-05T07:37:50.697 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:37:50.697 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.697 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.697 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:37:50.697 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.697 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.697 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.697 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6807", 2024-02-05T07:37:50.697 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:37:50.697 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.697 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6806", 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6804/398113577", 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6805/398113577", 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6807/398113577", 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6806/398113577", 2024-02-05T07:37:50.698 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7", 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 52, 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 230, 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 301, 2024-02-05T07:37:50.699 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 576, 2024-02-05T07:37:50.700 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 231, 2024-02-05T07:37:50.700 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:37:50.700 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:37:50.700 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.700 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.700 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.700 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6808", 2024-02-05T07:37:50.700 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:37:50.700 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.700 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.700 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.700 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:37:50.700 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.700 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6809", 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6811", 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6810", 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6808/313196533", 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6809/313196533", 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6811/313196533", 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6810/313196533", 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:37:50.702 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:37:50.703 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:37:50.703 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.703 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-02-05T07:37:50.703 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9", 2024-02-05T07:37:50.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:50.704 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:50.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:50.705 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:37:50.705 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-02-05T07:37:50.705 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-02-05T07:37:50.705 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:37:50.705 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:37:50.705 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:37:50.705 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 60, 2024-02-05T07:37:50.705 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 349, 2024-02-05T07:37:50.705 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:37:50.705 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:37:50.705 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:37:50.705 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.705 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.705 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.706 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6812", 2024-02-05T07:37:50.706 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:37:50.706 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.706 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.706 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:37:50.706 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.706 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.706 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6813", 2024-02-05T07:37:50.706 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:37:50.706 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.706 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.707 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:37:50.707 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.707 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.707 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6815", 2024-02-05T07:37:50.707 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:37:50.707 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.707 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.707 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:37:50.707 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:37:50.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.707 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:37:50.708 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6814", 2024-02-05T07:37:50.708 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:37:50.708 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.708 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.708 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6812/2983605980", 2024-02-05T07:37:50.708 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6813/2983605980", 2024-02-05T07:37:50.708 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2983605980", 2024-02-05T07:37:50.708 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6814/2983605980", 2024-02-05T07:37:50.708 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:37:50.708 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:37:50.708 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:37:50.708 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.708 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:37:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_xinfo": [ 2024-02-05T07:37:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-02-05T07:37:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:37:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:37:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:37:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:37:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:37:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:37:03.028265+0000", 2024-02-05T07:37:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:37:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-02-05T07:37:50.710 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:37:50.710 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:37:50.710 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:37:50.710 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:37:50.710 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:37:50.710 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:36:58.589115+0000", 2024-02-05T07:37:50.710 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:37:50.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.710 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-02-05T07:37:50.710 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:37:50.710 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:37:50.710 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:37:50.710 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:37:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:37:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:37:19.331253+0000", 2024-02-05T07:37:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:37:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-02-05T07:37:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:37:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:37:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:37:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:37:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:37:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:36:57.040965+0000", 2024-02-05T07:37:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:37:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.712 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-02-05T07:37:50.712 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:37:50.712 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:37:50.712 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:37:50.712 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:37:50.712 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:37:50.712 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:37:39.693451+0000", 2024-02-05T07:37:50.712 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:37:50.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.712 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-02-05T07:37:50.712 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:36:53.373644+0000", 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:37:50.713 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:36:47.359474+0000", 2024-02-05T07:37:50.714 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 230 2024-02-05T07:37:50.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.714 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-02-05T07:37:50.714 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:37:50.714 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:37:50.714 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:37:50.714 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:37:50.714 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-02-05T07:37:50.714 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:36:43.764350+0000", 2024-02-05T07:37:50.714 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:37:50.714 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.714 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:37:50.714 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap": [ 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.d", 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: 4, 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: 7, 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: 1 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "5.15", 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: 2, 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: 0, 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: 1 2024-02-05T07:37:50.715 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.716 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "6.0", 2024-02-05T07:37:50.716 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:37:50.716 INFO:teuthology.orchestra.run.smithi007.stdout: 1, 2024-02-05T07:37:50.716 INFO:teuthology.orchestra.run.smithi007.stdout: 0, 2024-02-05T07:37:50.716 INFO:teuthology.orchestra.run.smithi007.stdout: 6 2024-02-05T07:37:50.716 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.716 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.716 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:37:50.716 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_items": [ 2024-02-05T07:37:50.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.716 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.1a", 2024-02-05T07:37:50.716 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 4, 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 0 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.1b", 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 0, 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.718 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.4", 2024-02-05T07:37:50.718 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:37:50.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.718 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:37:50.718 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 0 2024-02-05T07:37:50.718 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.718 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.718 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.7", 2024-02-05T07:37:50.718 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:37:50.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.718 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_primaries": [], 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_temp": [ 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "1.15", 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: 2, 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: 4, 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: 6 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "1.17", 2024-02-05T07:37:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:37:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: 6, 2024-02-05T07:37:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: 3, 2024-02-05T07:37:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: 2 2024-02-05T07:37:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:37:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "1.1a", 2024-02-05T07:37:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:37:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: 4, 2024-02-05T07:37:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: 2, 2024-02-05T07:37:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: 6 2024-02-05T07:37:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:37:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:37:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_temp": [], 2024-02-05T07:37:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklist": { 2024-02-05T07:37:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2180087772": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:37:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2066685086": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:37:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/4100020626": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:37:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1953351182": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:37:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3203189162": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:37:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3418781614": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:37:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1518592084": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:37:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/3380493069": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:37:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2208590106": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:37:50.722 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/223579766": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:37:50.722 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/2634895574": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:37:50.722 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1462812719": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:37:50.722 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2823976078": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:37:50.722 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1905520582": "2024-02-06T07:13:47.004765+0000" 2024-02-05T07:37:50.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.722 INFO:teuthology.orchestra.run.smithi007.stdout: "range_blocklist": {}, 2024-02-05T07:37:50.722 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profiles": { 2024-02-05T07:37:50.722 INFO:teuthology.orchestra.run.smithi007.stdout: "default": { 2024-02-05T07:37:50.722 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-failure-domain": "osd", 2024-02-05T07:37:50.722 INFO:teuthology.orchestra.run.smithi007.stdout: "k": "2", 2024-02-05T07:37:50.722 INFO:teuthology.orchestra.run.smithi007.stdout: "m": "1", 2024-02-05T07:37:50.722 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin": "jerasure", 2024-02-05T07:37:50.722 INFO:teuthology.orchestra.run.smithi007.stdout: "technique": "reed_sol_van" 2024-02-05T07:37:50.723 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:37:50.723 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps_queue": [], 2024-02-05T07:37:50.723 INFO:teuthology.orchestra.run.smithi007.stdout: "new_removed_snaps": [], 2024-02-05T07:37:50.723 INFO:teuthology.orchestra.run.smithi007.stdout: "new_purged_snaps": [], 2024-02-05T07:37:50.723 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_node_flags": {}, 2024-02-05T07:37:50.723 INFO:teuthology.orchestra.run.smithi007.stdout: "device_class_flags": {}, 2024-02-05T07:37:50.723 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode": { 2024-02-05T07:37:50.723 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_enabled": false, 2024-02-05T07:37:50.723 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_bucket_count": 0, 2024-02-05T07:37:50.723 INFO:teuthology.orchestra.run.smithi007.stdout: "degraded_stretch_mode": 0, 2024-02-05T07:37:50.723 INFO:teuthology.orchestra.run.smithi007.stdout: "recovering_stretch_mode": 0, 2024-02-05T07:37:50.723 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_bucket": 0 2024-02-05T07:37:50.723 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:37:50.724 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:37:50.773 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:50 smithi090 ceph-mon[33213]: 6.1a scrub starts 2024-02-05T07:37:50.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:50 smithi090 ceph-mon[33213]: 6.1a scrub ok 2024-02-05T07:37:50.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:50 smithi090 ceph-mon[33213]: pgmap v1046: 162 pgs: 3 active+remapped+backfill_toofull, 159 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:50.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:50 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1340516239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:37:50.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:50.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:50.830 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:50.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:50.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:50 smithi007 ceph-mon[29098]: 6.1a scrub starts 2024-02-05T07:37:50.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:50 smithi007 ceph-mon[29098]: 6.1a scrub ok 2024-02-05T07:37:50.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:50 smithi007 ceph-mon[29098]: pgmap v1046: 162 pgs: 3 active+remapped+backfill_toofull, 159 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:50.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:50 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1340516239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:37:50.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:50 smithi007 ceph-mon[37158]: 6.1a scrub starts 2024-02-05T07:37:50.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:50 smithi007 ceph-mon[37158]: 6.1a scrub ok 2024-02-05T07:37:50.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:50 smithi007 ceph-mon[37158]: pgmap v1046: 162 pgs: 3 active+remapped+backfill_toofull, 159 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:50.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:50 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1340516239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:37:50.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:50.955 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:50.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:50.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:51.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:51.076 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:37:51.076 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:37:51.076 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:37:51.077 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:37:51.077 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:37:51.081 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:51.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:51.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:51.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:51.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:51.209 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:51.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:51.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:51.334 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:51.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:51.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:51.436 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:51.460 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:51.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:51.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:51.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:37:51 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:37:51.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:51.586 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:51.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:51.676 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T07:37:51.688 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:51.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:51.712 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:51.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:51.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:51.814 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_4' created 2024-02-05T07:37:51.839 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:51.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:51.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:51.919 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:37:51.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:51.965 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:51.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:51.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:51 smithi090 ceph-mon[33213]: 6.e scrub starts 2024-02-05T07:37:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:51 smithi090 ceph-mon[33213]: 6.e scrub ok 2024-02-05T07:37:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:51 smithi090 ceph-mon[33213]: 5.19 scrub starts 2024-02-05T07:37:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:51 smithi090 ceph-mon[33213]: 5.19 scrub ok 2024-02-05T07:37:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:51 smithi090 ceph-mon[33213]: 4.19 scrub starts 2024-02-05T07:37:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:51 smithi090 ceph-mon[33213]: 4.19 scrub ok 2024-02-05T07:37:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:51 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1478400172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-05T07:37:52.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:52.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:52.091 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:52.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:51 smithi007 ceph-mon[29098]: 6.e scrub starts 2024-02-05T07:37:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:51 smithi007 ceph-mon[29098]: 6.e scrub ok 2024-02-05T07:37:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:51 smithi007 ceph-mon[29098]: 5.19 scrub starts 2024-02-05T07:37:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:51 smithi007 ceph-mon[29098]: 5.19 scrub ok 2024-02-05T07:37:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:51 smithi007 ceph-mon[29098]: 4.19 scrub starts 2024-02-05T07:37:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:51 smithi007 ceph-mon[29098]: 4.19 scrub ok 2024-02-05T07:37:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:51 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1478400172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-05T07:37:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:51 smithi007 ceph-mon[37158]: 6.e scrub starts 2024-02-05T07:37:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:51 smithi007 ceph-mon[37158]: 6.e scrub ok 2024-02-05T07:37:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:51 smithi007 ceph-mon[37158]: 5.19 scrub starts 2024-02-05T07:37:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:51 smithi007 ceph-mon[37158]: 5.19 scrub ok 2024-02-05T07:37:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:51 smithi007 ceph-mon[37158]: 4.19 scrub starts 2024-02-05T07:37:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:51 smithi007 ceph-mon[37158]: 4.19 scrub ok 2024-02-05T07:37:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:51 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1478400172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-05T07:37:52.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:52.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:52.217 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:52.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:52.261 INFO:tasks.thrashosds.thrasher:j is {'epoch': 578, 'fsid': 'e3e7059e-c3f5-11ee-95b6-87774f69a715', 'created': '2024-02-05T07:13:14.603885+0000', 'modified': '2024-02-05T07:37:42.700254+0000', 'last_up_change': '2024-02-05T07:28:24.555976+0000', 'last_in_change': '2024-02-05T07:36:31.668197+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T07:16:32.505567+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 543, 'last_epoch_started': 542, 'last_epoch_clean': 542, 'source_version': "86'128", 'target_version': "86'128"}, 'last_change': '573', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '543', 'last_force_op_resend_preluminous': '571', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7999999523162842, 'score_stable': 1.7999999523162842, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.5499999523162842, 'raw_score_stable': 1.5499999523162842, 'primary_affinity_weighted': 0.7850494384765625, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T07:18:47.259462+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.690000057220459, 'score_stable': 4.690000057220459, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 3.3299999237060547, 'raw_score_stable': 3.3299999237060547, 'primary_affinity_weighted': 0.7850494384765625, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T07:18:47.762336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4000000953674316, 'score_stable': 2.4000000953674316, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.7850494384765625, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T07:18:49.736599+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.7850494384765625, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T07:18:51.696948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.7850494384765625, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T07:18:53.768555+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.7850494384765625, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}], 'osds': [{'osd': 0, 'uuid': '703c3948-8003-4015-9dd8-21019cf0eeb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 575, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6801', 'nonce': 2069016689}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6802', 'nonce': 2069016689}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6804', 'nonce': 2069016689}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6803', 'nonce': 2069016689}]}, 'public_addr': '172.21.15.7:6801/2069016689', 'cluster_addr': '172.21.15.7:6802/2069016689', 'heartbeat_back_addr': '172.21.15.7:6804/2069016689', 'heartbeat_front_addr': '172.21.15.7:6803/2069016689', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '5c3e7d1e-a61d-44b9-b21e-f91c99a096bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5441131591796875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 576, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6805', 'nonce': 865486532}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6806', 'nonce': 865486532}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6808', 'nonce': 865486532}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6807', 'nonce': 865486532}]}, 'public_addr': '172.21.15.7:6805/865486532', 'cluster_addr': '172.21.15.7:6806/865486532', 'heartbeat_back_addr': '172.21.15.7:6808/865486532', 'heartbeat_front_addr': '172.21.15.7:6807/865486532', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4fd930f-7bb2-46ca-9964-98235bd47b3a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 576, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6809', 'nonce': 904871370}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6810', 'nonce': 904871370}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6812', 'nonce': 904871370}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6811', 'nonce': 904871370}]}, 'public_addr': '172.21.15.7:6809/904871370', 'cluster_addr': '172.21.15.7:6810/904871370', 'heartbeat_back_addr': '172.21.15.7:6812/904871370', 'heartbeat_front_addr': '172.21.15.7:6811/904871370', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '149ed0d6-b049-4478-9819-cafc446e94d3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 576, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6813', 'nonce': 2758586242}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6814', 'nonce': 2758586242}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6816', 'nonce': 2758586242}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6815', 'nonce': 2758586242}]}, 'public_addr': '172.21.15.7:6813/2758586242', 'cluster_addr': '172.21.15.7:6814/2758586242', 'heartbeat_back_addr': '172.21.15.7:6816/2758586242', 'heartbeat_front_addr': '172.21.15.7:6815/2758586242', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 576, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 1154929470}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6801', 'nonce': 1154929470}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6803', 'nonce': 1154929470}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 1154929470}]}, 'public_addr': '172.21.15.90:6800/1154929470', 'cluster_addr': '172.21.15.90:6801/1154929470', 'heartbeat_back_addr': '172.21.15.90:6803/1154929470', 'heartbeat_front_addr': '172.21.15.90:6802/1154929470', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '244a7930-c22c-413e-86e1-11e0749f7367', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.7362823486328125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 336, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 398113577}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6805', 'nonce': 398113577}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6807', 'nonce': 398113577}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 398113577}]}, 'public_addr': '172.21.15.90:6804/398113577', 'cluster_addr': '172.21.15.90:6805/398113577', 'heartbeat_back_addr': '172.21.15.90:6807/398113577', 'heartbeat_front_addr': '172.21.15.90:6806/398113577', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0adbb562-3012-4f60-b5ec-02550a10b2c7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 52, 'last_clean_end': 230, 'up_from': 301, 'up_thru': 576, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 313196533}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 313196533}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6811', 'nonce': 313196533}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6810', 'nonce': 313196533}]}, 'public_addr': '172.21.15.90:6808/313196533', 'cluster_addr': '172.21.15.90:6809/313196533', 'heartbeat_back_addr': '172.21.15.90:6811/313196533', 'heartbeat_front_addr': '172.21.15.90:6810/313196533', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dbc26c36-a0e9-4f21-9e98-739270cea6d9', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 349, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 2983605980}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 2983605980}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6815', 'nonce': 2983605980}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6814', 'nonce': 2983605980}]}, 'public_addr': '172.21.15.90:6812/2983605980', 'cluster_addr': '172.21.15.90:6813/2983605980', 'heartbeat_back_addr': '172.21.15.90:6815/2983605980', 'heartbeat_front_addr': '172.21.15.90:6814/2983605980', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:37:03.028265+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:36:58.589115+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:37:19.331253+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:36:57.040965+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:37:39.693451+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T07:36:53.373644+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:36:47.359474+0000', 'dead_epoch': 230}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T07:36:43.764350+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.d', 'osds': [4, 7, 1]}, {'pgid': '5.15', 'osds': [2, 0, 1]}, {'pgid': '6.0', 'osds': [1, 0, 6]}], 'pg_upmap_items': [{'pgid': '3.1a', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '3.1b', 'mappings': [{'from': 0, 'to': 2}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.15', 'osds': [2, 4, 6]}, {'pgid': '1.17', 'osds': [6, 3, 2]}, {'pgid': '1.1a', 'osds': [4, 2, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.7:0/2180087772': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/2066685086': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/4100020626': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/1953351182': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/3203189162': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/3418781614': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:6800/1518592084': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:6800/3380493069': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/2208590106': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:0/223579766': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:6800/2634895574': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/1462812719': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/2823976078': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/1905520582': '2024-02-06T07:13:47.004765+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T07:37:52.261 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 5.15 2024-02-05T07:37:52.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd rm-pg-upmap 5.15 2024-02-05T07:37:52.317 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:52.368 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:52.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:52.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:52.497 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:52.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:52.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:52.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:52.623 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:52.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:52.725 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:52.751 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:52.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:52.811 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:37:52.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:52.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[29098]: 5.1 scrub starts 2024-02-05T07:37:52.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[29098]: 5.1 scrub ok 2024-02-05T07:37:52.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[29098]: 5.8 scrub starts 2024-02-05T07:37:52.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[29098]: 5.8 scrub ok 2024-02-05T07:37:52.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[29098]: pgmap v1047: 162 pgs: 3 active+remapped+backfill_toofull, 159 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:52.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-05T07:37:52.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[29098]: 5.12 scrub starts 2024-02-05T07:37:52.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[29098]: 5.12 scrub ok 2024-02-05T07:37:52.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-05T07:37:52.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[29098]: osdmap e579: 8 total, 8 up, 6 in 2024-02-05T07:37:52.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2429802791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:37:52.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:52.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[37158]: 5.1 scrub starts 2024-02-05T07:37:52.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[37158]: 5.1 scrub ok 2024-02-05T07:37:52.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[37158]: 5.8 scrub starts 2024-02-05T07:37:52.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[37158]: 5.8 scrub ok 2024-02-05T07:37:52.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[37158]: pgmap v1047: 162 pgs: 3 active+remapped+backfill_toofull, 159 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:52.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-05T07:37:52.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[37158]: 5.12 scrub starts 2024-02-05T07:37:52.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[37158]: 5.12 scrub ok 2024-02-05T07:37:52.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-05T07:37:52.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[37158]: osdmap e579: 8 total, 8 up, 6 in 2024-02-05T07:37:52.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:52 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2429802791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:37:52.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:52.876 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:52.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:52.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:52.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:53.002 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:37:53.004 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:53.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:53.022 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:37:53.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:52 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:37:53.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:52 smithi090 ceph-mon[33213]: 5.1 scrub starts 2024-02-05T07:37:53.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:52 smithi090 ceph-mon[33213]: 5.1 scrub ok 2024-02-05T07:37:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:52 smithi090 ceph-mon[33213]: 5.8 scrub starts 2024-02-05T07:37:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:52 smithi090 ceph-mon[33213]: 5.8 scrub ok 2024-02-05T07:37:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:52 smithi090 ceph-mon[33213]: pgmap v1047: 162 pgs: 3 active+remapped+backfill_toofull, 159 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail 2024-02-05T07:37:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:52 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-05T07:37:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:52 smithi090 ceph-mon[33213]: 5.12 scrub starts 2024-02-05T07:37:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:52 smithi090 ceph-mon[33213]: 5.12 scrub ok 2024-02-05T07:37:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:52 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-05T07:37:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:52 smithi090 ceph-mon[33213]: osdmap e579: 8 total, 8 up, 6 in 2024-02-05T07:37:53.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:52 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2429802791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:37:53.041 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:37:53.060 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:37:53.080 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:37:53.106 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:37:53.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:53.125 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:37:53.130 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:53.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:53.145 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:37:53.147 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:37:53.147 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:37:53.147 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:37:53.147 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:37:53.147 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:37:53.147 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:37:53.147 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:37:53.147 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:37:53.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:53.256 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:53.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:53.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:53.381 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:53.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:53.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:53.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:53.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:53.512 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:53.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:53.598 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:37:53.598 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:37:53.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:53.637 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:53.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:53.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:53.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:53.756 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:37:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:37:53.763 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:53.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:53.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:53.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:53.891 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:53.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:53.902 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:53 smithi090 ceph-mon[33213]: 6.c scrub starts 2024-02-05T07:37:53.902 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:53 smithi090 ceph-mon[33213]: 6.c scrub ok 2024-02-05T07:37:53.902 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:53 smithi090 ceph-mon[33213]: 6.1 scrub starts 2024-02-05T07:37:53.903 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:53 smithi090 ceph-mon[33213]: 6.1 scrub ok 2024-02-05T07:37:53.903 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:53 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:37:53.903 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:53 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2429802791' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:37:53.903 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:53 smithi090 ceph-mon[33213]: osdmap e580: 8 total, 8 up, 6 in 2024-02-05T07:37:53.903 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:53 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/368893873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:53.903 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:53 smithi090 ceph-mon[33213]: 5.9 scrub starts 2024-02-05T07:37:53.903 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:53 smithi090 ceph-mon[33213]: 5.9 scrub ok 2024-02-05T07:37:53.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:54.016 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:54.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:54.118 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:54.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:54.142 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:54.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:54.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[29098]: 6.c scrub starts 2024-02-05T07:37:54.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[29098]: 6.c scrub ok 2024-02-05T07:37:54.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[29098]: 6.1 scrub starts 2024-02-05T07:37:54.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[29098]: 6.1 scrub ok 2024-02-05T07:37:54.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:37:54.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2429802791' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:37:54.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[29098]: osdmap e580: 8 total, 8 up, 6 in 2024-02-05T07:37:54.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/368893873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:54.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[29098]: 5.9 scrub starts 2024-02-05T07:37:54.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[29098]: 5.9 scrub ok 2024-02-05T07:37:54.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[37158]: 6.c scrub starts 2024-02-05T07:37:54.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[37158]: 6.c scrub ok 2024-02-05T07:37:54.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[37158]: 6.1 scrub starts 2024-02-05T07:37:54.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[37158]: 6.1 scrub ok 2024-02-05T07:37:54.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:37:54.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2429802791' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:37:54.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[37158]: osdmap e580: 8 total, 8 up, 6 in 2024-02-05T07:37:54.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/368893873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:37:54.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[37158]: 5.9 scrub starts 2024-02-05T07:37:54.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:53 smithi007 ceph-mon[37158]: 5.9 scrub ok 2024-02-05T07:37:54.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T07:37:54.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:54.305 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:54.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:54.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:54.431 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:54.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:54.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:54.478 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:37:54.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:54.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:54.557 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:54.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:54.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-02-05T07:37:54.649 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:54.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:54.687 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:54.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:54.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:54.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:54.815 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:54.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:54.875 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:54 smithi090 ceph-mon[33213]: 3.e scrub starts 2024-02-05T07:37:54.875 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:54 smithi090 ceph-mon[33213]: 3.e scrub ok 2024-02-05T07:37:54.875 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:54 smithi090 ceph-mon[33213]: pgmap v1050: 178 pgs: 5 creating+peering, 11 unknown, 3 active+remapped+backfill_toofull, 159 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:37:54.876 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:54 smithi090 ceph-mon[33213]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:37:54.877 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:54 smithi090 ceph-mon[33213]: osdmap e581: 8 total, 8 up, 6 in 2024-02-05T07:37:54.877 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:54 smithi090 ceph-mon[33213]: 5.b scrub starts 2024-02-05T07:37:54.877 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:54 smithi090 ceph-mon[33213]: 5.b scrub ok 2024-02-05T07:37:54.877 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:54 smithi090 ceph-mon[33213]: osdmap e582: 8 total, 8 up, 6 in 2024-02-05T07:37:54.877 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:54 smithi090 ceph-mon[33213]: 6.0 scrub starts 2024-02-05T07:37:54.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:54.922 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[29098]: 3.e scrub starts 2024-02-05T07:37:54.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[29098]: 3.e scrub ok 2024-02-05T07:37:54.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[29098]: pgmap v1050: 178 pgs: 5 creating+peering, 11 unknown, 3 active+remapped+backfill_toofull, 159 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:37:54.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[29098]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:37:54.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[29098]: osdmap e581: 8 total, 8 up, 6 in 2024-02-05T07:37:54.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[29098]: 5.b scrub starts 2024-02-05T07:37:54.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[29098]: 5.b scrub ok 2024-02-05T07:37:54.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[29098]: osdmap e582: 8 total, 8 up, 6 in 2024-02-05T07:37:54.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[29098]: 6.0 scrub starts 2024-02-05T07:37:54.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[37158]: 3.e scrub starts 2024-02-05T07:37:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[37158]: 3.e scrub ok 2024-02-05T07:37:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[37158]: pgmap v1050: 178 pgs: 5 creating+peering, 11 unknown, 3 active+remapped+backfill_toofull, 159 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:37:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[37158]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T07:37:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[37158]: osdmap e581: 8 total, 8 up, 6 in 2024-02-05T07:37:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[37158]: 5.b scrub starts 2024-02-05T07:37:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[37158]: 5.b scrub ok 2024-02-05T07:37:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[37158]: osdmap e582: 8 total, 8 up, 6 in 2024-02-05T07:37:54.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:54 smithi007 ceph-mon[37158]: 6.0 scrub starts 2024-02-05T07:37:54.939 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:54.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:55.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:55.065 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:55.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:55.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:55.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:55.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:55.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:55.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:55.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:55.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:55.318 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:55.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:55.419 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:55.443 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:55.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:55.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:55.545 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:55.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:55.570 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:55.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:55.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:55.694 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:55.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:55.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:55.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:55.822 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:55.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:55.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:55.936 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:37:55.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:55.948 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:55.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:56.016 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:55 smithi090 ceph-mon[33213]: 4.d scrub starts 2024-02-05T07:37:56.016 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:55 smithi090 ceph-mon[33213]: 4.d scrub ok 2024-02-05T07:37:56.016 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:55 smithi090 ceph-mon[33213]: 6.11 scrub starts 2024-02-05T07:37:56.016 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:55 smithi090 ceph-mon[33213]: 6.11 scrub ok 2024-02-05T07:37:56.016 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:55 smithi090 ceph-mon[33213]: 6.0 scrub ok 2024-02-05T07:37:56.049 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:56.072 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:56.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:56.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:55 smithi007 ceph-mon[29098]: 4.d scrub starts 2024-02-05T07:37:56.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:55 smithi007 ceph-mon[29098]: 4.d scrub ok 2024-02-05T07:37:56.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:55 smithi007 ceph-mon[29098]: 6.11 scrub starts 2024-02-05T07:37:56.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:55 smithi007 ceph-mon[29098]: 6.11 scrub ok 2024-02-05T07:37:56.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:55 smithi007 ceph-mon[29098]: 6.0 scrub ok 2024-02-05T07:37:56.095 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:55 smithi007 ceph-mon[37158]: 4.d scrub starts 2024-02-05T07:37:56.095 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:55 smithi007 ceph-mon[37158]: 4.d scrub ok 2024-02-05T07:37:56.095 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:55 smithi007 ceph-mon[37158]: 6.11 scrub starts 2024-02-05T07:37:56.095 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:55 smithi007 ceph-mon[37158]: 6.11 scrub ok 2024-02-05T07:37:56.095 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:55 smithi007 ceph-mon[37158]: 6.0 scrub ok 2024-02-05T07:37:56.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:56.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:56.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:56.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:56.324 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:56.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:56.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:56.403 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:37:56.403 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:37:56.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:56.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:56.450 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:56.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:56.551 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:56.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:56.575 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:56.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:56.676 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:56.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:56.701 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:56.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:56.802 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:56.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:56.826 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:56.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:56.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:56.962 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:56.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:57.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:56 smithi090 ceph-mon[33213]: 5.1a scrub starts 2024-02-05T07:37:57.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:56 smithi090 ceph-mon[33213]: 5.1a scrub ok 2024-02-05T07:37:57.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:56 smithi090 ceph-mon[33213]: pgmap v1053: 178 pgs: 5 creating+peering, 5 unknown, 3 active+remapped+backfill_toofull, 165 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:37:57.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:56 smithi090 ceph-mon[33213]: 6.4 scrub starts 2024-02-05T07:37:57.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:56 smithi090 ceph-mon[33213]: 6.4 scrub ok 2024-02-05T07:37:57.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T07:37:57.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:57.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:56 smithi007 ceph-mon[29098]: 5.1a scrub starts 2024-02-05T07:37:57.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:56 smithi007 ceph-mon[29098]: 5.1a scrub ok 2024-02-05T07:37:57.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:56 smithi007 ceph-mon[29098]: pgmap v1053: 178 pgs: 5 creating+peering, 5 unknown, 3 active+remapped+backfill_toofull, 165 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:37:57.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:56 smithi007 ceph-mon[29098]: 6.4 scrub starts 2024-02-05T07:37:57.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:56 smithi007 ceph-mon[29098]: 6.4 scrub ok 2024-02-05T07:37:57.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:56 smithi007 ceph-mon[37158]: 5.1a scrub starts 2024-02-05T07:37:57.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:56 smithi007 ceph-mon[37158]: 5.1a scrub ok 2024-02-05T07:37:57.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:56 smithi007 ceph-mon[37158]: pgmap v1053: 178 pgs: 5 creating+peering, 5 unknown, 3 active+remapped+backfill_toofull, 165 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:37:57.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:56 smithi007 ceph-mon[37158]: 6.4 scrub starts 2024-02-05T07:37:57.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:56 smithi007 ceph-mon[37158]: 6.4 scrub ok 2024-02-05T07:37:57.102 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:57.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:57.119 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:37:57.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:57.229 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:57.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:57.274 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:37:57.330 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:57.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:57.357 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:57.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:57.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:57.482 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:57.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:57.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:57.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:37:57.609 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:57.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:57.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:37:57.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:57.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:57.736 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:57.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:clear 5.15 pg_upmap mapping 2024-02-05T07:37:57.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:57.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:57.862 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:57.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:57.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:57.981 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:57.987 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:57.987 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:37:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:57.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:58.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:57 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4268604325' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.15"}]: dispatch 2024-02-05T07:37:58.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:57 smithi090 ceph-mon[33213]: 4.1c scrub starts 2024-02-05T07:37:58.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:57 smithi090 ceph-mon[33213]: 4.1c scrub ok 2024-02-05T07:37:58.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:57 smithi090 ceph-mon[33213]: 2.2 scrub starts 2024-02-05T07:37:58.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:57 smithi090 ceph-mon[33213]: 2.2 scrub ok 2024-02-05T07:37:58.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:57 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4268604325' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.15"}]: dispatch 2024-02-05T07:37:58.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:57 smithi007 ceph-mon[37158]: 4.1c scrub starts 2024-02-05T07:37:58.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:57 smithi007 ceph-mon[37158]: 4.1c scrub ok 2024-02-05T07:37:58.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:57 smithi007 ceph-mon[37158]: 2.2 scrub starts 2024-02-05T07:37:58.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:57 smithi007 ceph-mon[37158]: 2.2 scrub ok 2024-02-05T07:37:58.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:57 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4268604325' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.15"}]: dispatch 2024-02-05T07:37:58.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:57 smithi007 ceph-mon[29098]: 4.1c scrub starts 2024-02-05T07:37:58.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:57 smithi007 ceph-mon[29098]: 4.1c scrub ok 2024-02-05T07:37:58.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:57 smithi007 ceph-mon[29098]: 2.2 scrub starts 2024-02-05T07:37:58.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:57 smithi007 ceph-mon[29098]: 2.2 scrub ok 2024-02-05T07:37:58.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:58.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:58.113 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:58.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:58.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:58.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:58.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:58.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:58.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:58.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:58.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:58.491 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:58.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:58.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:58.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:58.616 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:58.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:58.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:58.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:37:58.742 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:58.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:58.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:37:58.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:58.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:58.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:58.962 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:37:58.962 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:37:58.962 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:58 smithi090 ceph-mon[33213]: pgmap v1054: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-02-05T07:37:58.963 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:58 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4268604325' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.15"}]': finished 2024-02-05T07:37:58.963 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:58 smithi090 ceph-mon[33213]: osdmap e583: 8 total, 8 up, 6 in 2024-02-05T07:37:58.963 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:58 smithi090 ceph-mon[33213]: 5.5 scrub starts 2024-02-05T07:37:58.963 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:58 smithi090 ceph-mon[33213]: 5.5 scrub ok 2024-02-05T07:37:58.963 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:58 smithi090 ceph-mon[33213]: 6.3 deep-scrub starts 2024-02-05T07:37:58.963 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:58 smithi090 ceph-mon[33213]: 6.3 deep-scrub ok 2024-02-05T07:37:58.969 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:37:58.993 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:58.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:58.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:37:59.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:37:59.118 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:59.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:37:59.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:58 smithi007 ceph-mon[37158]: pgmap v1054: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-02-05T07:37:59.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:58 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4268604325' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.15"}]': finished 2024-02-05T07:37:59.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:58 smithi007 ceph-mon[37158]: osdmap e583: 8 total, 8 up, 6 in 2024-02-05T07:37:59.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:58 smithi007 ceph-mon[37158]: 5.5 scrub starts 2024-02-05T07:37:59.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:58 smithi007 ceph-mon[37158]: 5.5 scrub ok 2024-02-05T07:37:59.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:58 smithi007 ceph-mon[37158]: 6.3 deep-scrub starts 2024-02-05T07:37:59.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:58 smithi007 ceph-mon[37158]: 6.3 deep-scrub ok 2024-02-05T07:37:59.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:58 smithi007 ceph-mon[29098]: pgmap v1054: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-02-05T07:37:59.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:58 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4268604325' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.15"}]': finished 2024-02-05T07:37:59.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:58 smithi007 ceph-mon[29098]: osdmap e583: 8 total, 8 up, 6 in 2024-02-05T07:37:59.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:58 smithi007 ceph-mon[29098]: 5.5 scrub starts 2024-02-05T07:37:59.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:58 smithi007 ceph-mon[29098]: 5.5 scrub ok 2024-02-05T07:37:59.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:58 smithi007 ceph-mon[29098]: 6.3 deep-scrub starts 2024-02-05T07:37:59.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:58 smithi007 ceph-mon[29098]: 6.3 deep-scrub ok 2024-02-05T07:37:59.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:37:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:59.245 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:59.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:37:59.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:37:59.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:59.371 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:37:59.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:37:59.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:59.498 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:59.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:59.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:59.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:59.625 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:59.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:59.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:37:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:59.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:37:59.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T07:37:59.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:37:59.891 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:37:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:37:59.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:37:59.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:59 smithi007 ceph-mon[29098]: 6.17 scrub starts 2024-02-05T07:37:59.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:59 smithi007 ceph-mon[29098]: 6.17 scrub ok 2024-02-05T07:37:59.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:59 smithi007 ceph-mon[29098]: osdmap e584: 8 total, 8 up, 6 in 2024-02-05T07:37:59.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:59 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:37:59.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:37:59 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:37:59.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:59 smithi007 ceph-mon[37158]: 6.17 scrub starts 2024-02-05T07:37:59.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:59 smithi007 ceph-mon[37158]: 6.17 scrub ok 2024-02-05T07:37:59.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:59 smithi007 ceph-mon[37158]: osdmap e584: 8 total, 8 up, 6 in 2024-02-05T07:37:59.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:59 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:37:59.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:37:59 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:37:59.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:00.019 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:00.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:00.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:00.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:59 smithi090 ceph-mon[33213]: 6.17 scrub starts 2024-02-05T07:38:00.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:59 smithi090 ceph-mon[33213]: 6.17 scrub ok 2024-02-05T07:38:00.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:59 smithi090 ceph-mon[33213]: osdmap e584: 8 total, 8 up, 6 in 2024-02-05T07:38:00.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:59 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:38:00.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:37:59 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:38:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:38:00.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:00.145 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:00.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:00.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:00.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:00.271 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:00.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:00.372 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:00.397 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:00.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:00.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:00.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:00.522 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:00.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:00.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:00.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:00.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:00.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:00.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:00.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:00.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:00.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:00.901 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:00.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:01.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:01.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:01.028 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:01.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:01.058 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[29098]: pgmap v1057: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-02-05T07:38:01.058 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[29098]: 5.0 scrub starts 2024-02-05T07:38:01.058 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[29098]: 5.0 scrub ok 2024-02-05T07:38:01.059 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:38:01.059 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:38:01.059 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[29098]: osdmap e585: 8 total, 8 up, 6 in 2024-02-05T07:38:01.059 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[29098]: 3.19 deep-scrub starts 2024-02-05T07:38:01.059 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[29098]: 3.19 deep-scrub ok 2024-02-05T07:38:01.059 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[29098]: 4.8 scrub starts 2024-02-05T07:38:01.059 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[29098]: 4.8 scrub ok 2024-02-05T07:38:01.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[37158]: pgmap v1057: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-02-05T07:38:01.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[37158]: 5.0 scrub starts 2024-02-05T07:38:01.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[37158]: 5.0 scrub ok 2024-02-05T07:38:01.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:38:01.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:38:01.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[37158]: osdmap e585: 8 total, 8 up, 6 in 2024-02-05T07:38:01.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[37158]: 3.19 deep-scrub starts 2024-02-05T07:38:01.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[37158]: 3.19 deep-scrub ok 2024-02-05T07:38:01.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[37158]: 4.8 scrub starts 2024-02-05T07:38:01.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:00 smithi007 ceph-mon[37158]: 4.8 scrub ok 2024-02-05T07:38:01.072 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:00 smithi090 ceph-mon[33213]: pgmap v1057: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-02-05T07:38:01.072 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:00 smithi090 ceph-mon[33213]: 5.0 scrub starts 2024-02-05T07:38:01.072 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:00 smithi090 ceph-mon[33213]: 5.0 scrub ok 2024-02-05T07:38:01.072 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:00 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:38:01.072 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:00 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:38:01.072 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:00 smithi090 ceph-mon[33213]: osdmap e585: 8 total, 8 up, 6 in 2024-02-05T07:38:01.073 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:00 smithi090 ceph-mon[33213]: 3.19 deep-scrub starts 2024-02-05T07:38:01.073 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:00 smithi090 ceph-mon[33213]: 3.19 deep-scrub ok 2024-02-05T07:38:01.073 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:00 smithi090 ceph-mon[33213]: 4.8 scrub starts 2024-02-05T07:38:01.073 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:00 smithi090 ceph-mon[33213]: 4.8 scrub ok 2024-02-05T07:38:01.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:01.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:01.154 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:01.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:01.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:01.278 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:01.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:01.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:01.405 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:01.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:01.471 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:38:01 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:38:01.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:01.531 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:01.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:01.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:01.658 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:01.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:01.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:01.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:01.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:01.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:01.910 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:01.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:02.011 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:02.025 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-02-05T07:38:02.026 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:38:02.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:02.037 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:02.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:02.073 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:38:02.073 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:38:02.074 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:02 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:38:02.074 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:02 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:38:02.074 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:02 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:02.074 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:02 smithi090 ceph-mon[33213]: 6.2 scrub starts 2024-02-05T07:38:02.074 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:02 smithi090 ceph-mon[33213]: 6.2 scrub ok 2024-02-05T07:38:02.074 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:02 smithi090 ceph-mon[33213]: pgmap v1059: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:38:02.074 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:02 smithi090 ceph-mon[33213]: 6.8 deep-scrub starts 2024-02-05T07:38:02.074 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:02 smithi090 ceph-mon[33213]: 6.8 deep-scrub ok 2024-02-05T07:38:02.074 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:02 smithi090 ceph-mon[33213]: 3.6 scrub starts 2024-02-05T07:38:02.074 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:02 smithi090 ceph-mon[33213]: 3.6 scrub ok 2024-02-05T07:38:02.074 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:02 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1786362162' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:38:02.074 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:02 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/146901427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:38:02.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:02.163 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:02.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:02.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:02.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:02.289 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:02.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:02.390 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:02.414 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:02.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:02.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:38:02.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:38:02.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:02.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[37158]: 6.2 scrub starts 2024-02-05T07:38:02.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[37158]: 6.2 scrub ok 2024-02-05T07:38:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[37158]: pgmap v1059: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:38:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[37158]: 6.8 deep-scrub starts 2024-02-05T07:38:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[37158]: 6.8 deep-scrub ok 2024-02-05T07:38:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[37158]: 3.6 scrub starts 2024-02-05T07:38:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[37158]: 3.6 scrub ok 2024-02-05T07:38:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1786362162' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:38:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/146901427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:38:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:38:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:38:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[29098]: 6.2 scrub starts 2024-02-05T07:38:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[29098]: 6.2 scrub ok 2024-02-05T07:38:02.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[29098]: pgmap v1059: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:38:02.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[29098]: 6.8 deep-scrub starts 2024-02-05T07:38:02.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[29098]: 6.8 deep-scrub ok 2024-02-05T07:38:02.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[29098]: 3.6 scrub starts 2024-02-05T07:38:02.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[29098]: 3.6 scrub ok 2024-02-05T07:38:02.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1786362162' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T07:38:02.425 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:02 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/146901427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:38:02.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:02.544 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:02.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:02.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:02.671 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:02.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:02.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T07:38:02.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:02.798 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:02.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:02.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:02.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:02.927 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:02.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:02.935 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:38:03.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:03.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:03.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:03.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:03.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:03.180 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:03.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:03.209 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:03 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:38:03.209 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:03 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1786362162' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:38:03.209 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:03 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/146901427' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:38:03.209 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:03 smithi090 ceph-mon[33213]: osdmap e586: 8 total, 8 up, 6 in 2024-02-05T07:38:03.209 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:03 smithi090 ceph-mon[33213]: 3.d scrub starts 2024-02-05T07:38:03.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:03.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:03.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:03.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:03.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:03 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:38:03.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:03 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1786362162' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:38:03.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:03 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/146901427' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:38:03.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:03 smithi007 ceph-mon[29098]: osdmap e586: 8 total, 8 up, 6 in 2024-02-05T07:38:03.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:03 smithi007 ceph-mon[29098]: 3.d scrub starts 2024-02-05T07:38:03.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:03 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:38:03.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:03 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1786362162' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T07:38:03.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:03 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/146901427' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:38:03.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:03 smithi007 ceph-mon[37158]: osdmap e586: 8 total, 8 up, 6 in 2024-02-05T07:38:03.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:03 smithi007 ceph-mon[37158]: 3.d scrub starts 2024-02-05T07:38:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:03.434 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:03.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:03.522 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6, 3, 2] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:38:03.522 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:38:03.523 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 4, 6, 3, 2] 2024-02-05T07:38:03.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd out 4 2024-02-05T07:38:03.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:03.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:03.560 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:03.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:03.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:03.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:03.686 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:03.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:03.702 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:38:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:38:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:38:03.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:03.810 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:03.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:03.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:03.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:03.930 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:38:03.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:03.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:03.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:03.951 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:38:03.971 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:38:03.986 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:03.994 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:38:04.015 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:38:04.037 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:38:04.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:04.064 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:38:04.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:04 smithi007 ceph-mon[29098]: pgmap v1061: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 915 B/s rd, 0 op/s 2024-02-05T07:38:04.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:04 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3279557696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:04.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:04 smithi007 ceph-mon[37158]: pgmap v1061: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 915 B/s rd, 0 op/s 2024-02-05T07:38:04.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:04 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3279557696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:04.066 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:04.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:04.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:04.085 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:38:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:38:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:38:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:38:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:38:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:38:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:38:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:38:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:38:04.125 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:04 smithi090 ceph-mon[33213]: pgmap v1061: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 595 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 915 B/s rd, 0 op/s 2024-02-05T07:38:04.125 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:04 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3279557696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:04.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:04.192 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:04.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:04.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:04.319 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:04.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:04.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:04.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:04.445 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:04.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:04.479 DEBUG:teuthology.orchestra.run.smithi007:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-02-05T07:38:04.488 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T07:38:04.488 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T07:38:04.536 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:hints = 1 2024-02-05T07:38:04.537 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T07:38:04.537 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object prefix: benchmark_data_smithi007_183880 2024-02-05T07:38:04.537 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:38:04.537 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 0 0 0 0 0 0 - 0 2024-02-05T07:38:04.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:04.597 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:04.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:04.693 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:38:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:38:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-02-05T07:38:04.694 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:38:04.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:04.731 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:04.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:04.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:04.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:04.861 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:04.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:04.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:04.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:05.007 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:05.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:05.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:05.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:38:05.110 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:05.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:05.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:05.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:05.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:05 smithi090 ceph-mon[33213]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:38:05.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:05 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:38:05.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:05 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:38:05.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:05.286 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:05.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:05.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:05 smithi007 ceph-mon[29098]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:38:05.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:05 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:38:05.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:05 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:38:05.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:05 smithi007 ceph-mon[37158]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T07:38:05.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:05 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:38:05.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:05 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:38:05.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:05.414 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:05.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:05.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:05.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:05.527 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T07:38:05.537 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 1 16 2518 2502 156.367 156.375 0.00217884 0.00634711 2024-02-05T07:38:05.551 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:05.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:05.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:05.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:05.694 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:05.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:05.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:05.796 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:05.819 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:38:05.837 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:05.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:05.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:05.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:05.968 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:05.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:06.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:06.098 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:06.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:06.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:06.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:06.209 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:06 smithi090 ceph-mon[33213]: osdmap e587: 8 total, 8 up, 6 in 2024-02-05T07:38:06.209 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:06 smithi090 ceph-mon[33213]: pgmap v1063: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 12 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 1.9 MiB/s wr, 32 op/s 2024-02-05T07:38:06.230 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:06.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:06.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:06.359 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:06.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:06.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:06 smithi007 ceph-mon[29098]: osdmap e587: 8 total, 8 up, 6 in 2024-02-05T07:38:06.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:06 smithi007 ceph-mon[29098]: pgmap v1063: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 12 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 1.9 MiB/s wr, 32 op/s 2024-02-05T07:38:06.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:06 smithi007 ceph-mon[37158]: osdmap e587: 8 total, 8 up, 6 in 2024-02-05T07:38:06.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:06 smithi007 ceph-mon[37158]: pgmap v1063: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 12 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 1.9 MiB/s wr, 32 op/s 2024-02-05T07:38:06.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:06.489 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:06.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:06.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:06.537 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 2 15 4951 4936 154.221 152.125 0.00683346 0.00646916 2024-02-05T07:38:06.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:06.617 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:06.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:06.632 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:06.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:06.746 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:06.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:06.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:06.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:06.892 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:06.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:06.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:06.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:07.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:07.022 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:07.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:07.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:38:07.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:07.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:07.152 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:07.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:07.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:07.304 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:07.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:07.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:07.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:07.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:07.538 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 3 16 7118 7102 147.935 135.375 0.00328824 0.00674353 2024-02-05T07:38:07.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:07.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:07.595 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:07.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:07.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:07.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:07.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:07.823 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:38:07.823 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:38:07.823 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:38:07.823 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:38:07.823 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:38:07.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:07.883 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:07.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:07.985 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:08.025 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:08.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:08.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:08.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:08.156 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:08.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:08.218 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:07 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a[29094]: 2024-02-05T07:38:07.940+0000 7fe3564d3700 -1 mon.a@0(leader).osd e587 definitely_dead 0 2024-02-05T07:38:08.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:08.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:08.284 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:08.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:08.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:08.415 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:08.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:08.435 INFO:teuthology.orchestra.run.smithi007.stderr:marked out osd.4. 2024-02-05T07:38:08.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:08.537 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 4 16 9111 9095 142.09 124.562 0.00136729 0.0069004 2024-02-05T07:38:08.540 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:08.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:08.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T07:38:08.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:08.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:08.666 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:08.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:08.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:08 smithi007 ceph-mon[29098]: pgmap v1064: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 173 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 23 MiB/s wr, 365 op/s 2024-02-05T07:38:08.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:08 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3543923339' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-05T07:38:08.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:08 smithi007 ceph-mon[29098]: Client client.admin marked osd.4 out, while it was still marked up 2024-02-05T07:38:08.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:08 smithi007 ceph-mon[37158]: pgmap v1064: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 173 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 23 MiB/s wr, 365 op/s 2024-02-05T07:38:08.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:08 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3543923339' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-05T07:38:08.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:08 smithi007 ceph-mon[37158]: Client client.admin marked osd.4 out, while it was still marked up 2024-02-05T07:38:08.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:08.773 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:08 smithi090 ceph-mon[33213]: pgmap v1064: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 173 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 23 MiB/s wr, 365 op/s 2024-02-05T07:38:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:08 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3543923339' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-05T07:38:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:08 smithi090 ceph-mon[33213]: Client client.admin marked osd.4 out, while it was still marked up 2024-02-05T07:38:08.787 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:38:08.792 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:08.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:08.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:08.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:08.918 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:08.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:09.012 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:09.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:09.048 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:09.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:09.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:09.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:09.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:09.175 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:09.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:09.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:09.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:09.301 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:09.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:09.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:09.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:09.429 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:09.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:09.523 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:09 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3543923339' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-05T07:38:09.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:09 smithi090 ceph-mon[33213]: osdmap e588: 8 total, 8 up, 5 in 2024-02-05T07:38:09.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:09.537 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 5 16 9111 9095 113.673 0 - 0.0069004 2024-02-05T07:38:09.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:09.554 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:09.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:09.655 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:09.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:09 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3543923339' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-05T07:38:09.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:09 smithi007 ceph-mon[29098]: osdmap e588: 8 total, 8 up, 5 in 2024-02-05T07:38:09.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:09 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3543923339' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-05T07:38:09.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:09 smithi007 ceph-mon[37158]: osdmap e588: 8 total, 8 up, 5 in 2024-02-05T07:38:09.695 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:09.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:09.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:09.826 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:09.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:09.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:09.969 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:09.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:10.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:10.099 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:10.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:10.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:10.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:10.239 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:10.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:10.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:10.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:10.371 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:10.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:10.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:10.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:10.512 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:10.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:10.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: pgmap v1066: 178 pgs: 26 peering, 3 active+remapped+backfill_toofull, 149 active+clean; 315 MiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 278 B/s rd, 43 MiB/s wr, 685 op/s 2024-02-05T07:38:10.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: Health check failed: Reduced data availability: 11 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-02-05T07:38:10.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.1a continuing backfill to osd.1 from (86'128,86'128] MIN to 86'128 2024-02-05T07:38:10.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: osdmap e589: 8 total, 8 up, 5 in 2024-02-05T07:38:10.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.10 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.4 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: osdmap e590: 8 total, 8 up, 5 in 2024-02-05T07:38:10.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.f starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.3 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.1d starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.e starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.1e starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.13 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.1b starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.1 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.5 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.a starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.14 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:10 smithi090 ceph-mon[33213]: 1.2 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.538 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 6 16 10546 10530 109.675 44.8438 0.00824191 0.00910385 2024-02-05T07:38:10.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:10.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:10.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:10.756 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:10.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: pgmap v1066: 178 pgs: 26 peering, 3 active+remapped+backfill_toofull, 149 active+clean; 315 MiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 278 B/s rd, 43 MiB/s wr, 685 op/s 2024-02-05T07:38:10.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: Health check failed: Reduced data availability: 11 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-02-05T07:38:10.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.1a continuing backfill to osd.1 from (86'128,86'128] MIN to 86'128 2024-02-05T07:38:10.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: osdmap e589: 8 total, 8 up, 5 in 2024-02-05T07:38:10.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.10 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.4 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: osdmap e590: 8 total, 8 up, 5 in 2024-02-05T07:38:10.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.f starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.3 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.1d starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.e starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.1e starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.13 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.1b starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.1 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.5 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.a starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.14 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[29098]: 1.2 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.784 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: pgmap v1066: 178 pgs: 26 peering, 3 active+remapped+backfill_toofull, 149 active+clean; 315 MiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 278 B/s rd, 43 MiB/s wr, 685 op/s 2024-02-05T07:38:10.784 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: Health check failed: Reduced data availability: 11 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-02-05T07:38:10.784 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.1a continuing backfill to osd.1 from (86'128,86'128] MIN to 86'128 2024-02-05T07:38:10.784 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: osdmap e589: 8 total, 8 up, 5 in 2024-02-05T07:38:10.784 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.10 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.784 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.4 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.784 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: osdmap e590: 8 total, 8 up, 5 in 2024-02-05T07:38:10.784 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.f starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.784 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.3 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.784 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.1d starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.784 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.e starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.784 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.1e starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.785 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.13 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.785 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.1b starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.785 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.1 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.785 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.5 starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.785 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.785 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.a starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.785 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.14 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.785 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:10 smithi007 ceph-mon[37158]: 1.2 starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:10.787 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:10.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:10.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:10.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:10.891 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:38:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-02-05T07:38:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-02-05T07:38:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:38:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:271957 8.17 8:e915d766:::benchmark_data_smithi007_105755_object135647:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:38:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:29:27.647635+0000", 2024-02-05T07:38:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 523.24076943499995, 2024-02-05T07:38:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98865452799999998, 2024-02-05T07:38:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:38:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:38:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:38:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:38:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:38:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 271957 2024-02-05T07:38:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:38:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:38:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647635+0000", 2024-02-05T07:38:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:38:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:38:10.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647635+0000", 2024-02-05T07:38:10.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:38:10.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:38:10.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647636+0000", 2024-02-05T07:38:10.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3880000000000001e-06 2024-02-05T07:38:10.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:38:10.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647642+0000", 2024-02-05T07:38:10.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8270000000000003e-06 2024-02-05T07:38:10.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:38:10.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647655+0000", 2024-02-05T07:38:10.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.332e-05 2024-02-05T07:38:10.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:38:10.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647664+0000", 2024-02-05T07:38:10.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2099999999999999e-06 2024-02-05T07:38:10.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:38:10.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647726+0000", 2024-02-05T07:38:10.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1384999999999996e-05 2024-02-05T07:38:10.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:38:10.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.647735+0000", 2024-02-05T07:38:10.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2024-02-05T07:38:10.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:38:10.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.631569+0000", 2024-02-05T07:38:10.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98383460599999994 2024-02-05T07:38:10.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:38:10.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.631666+0000", 2024-02-05T07:38:10.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6846000000000001e-05 2024-02-05T07:38:10.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,6", 2024-02-05T07:38:10.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.631751+0000", 2024-02-05T07:38:10.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5303000000000006e-05 2024-02-05T07:38:10.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:38:10.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.635981+0000", 2024-02-05T07:38:10.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042294289999999998 2024-02-05T07:38:10.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:38:10.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636138+0000", 2024-02-05T07:38:10.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156988 2024-02-05T07:38:10.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:38:10.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636232+0000", 2024-02-05T07:38:10.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4400000000000004e-05 2024-02-05T07:38:10.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:38:10.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636266+0000", 2024-02-05T07:38:10.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3318999999999997e-05 2024-02-05T07:38:10.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:38:10.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636289+0000", 2024-02-05T07:38:10.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3609999999999999e-05 2024-02-05T07:38:10.898 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:38:10.898 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:38:10.898 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:38:10.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.899 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:271993 8.17 8:eb340f93:::benchmark_data_smithi007_105755_object135683:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:38:10.899 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:29:27.688048+0000", 2024-02-05T07:38:10.899 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 523.20035575300005, 2024-02-05T07:38:10.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94901903399999998, 2024-02-05T07:38:10.899 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:38:10.899 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:38:10.899 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:38:10.899 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:38:10.899 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:38:10.899 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 271993 2024-02-05T07:38:10.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.899 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:38:10.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:38:10.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688048+0000", 2024-02-05T07:38:10.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:38:10.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:38:10.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688048+0000", 2024-02-05T07:38:10.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:38:10.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:38:10.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688049+0000", 2024-02-05T07:38:10.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.096e-06 2024-02-05T07:38:10.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688054+0000", 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3980000000000001e-06 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688064+0000", 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0614999999999999e-05 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688072+0000", 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5299999999999999e-06 2024-02-05T07:38:10.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:38:10.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688162+0000", 2024-02-05T07:38:10.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0325e-05 2024-02-05T07:38:10.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:38:10.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.688171+0000", 2024-02-05T07:38:10.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1689999999999998e-06 2024-02-05T07:38:10.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:38:10.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.633668+0000", 2024-02-05T07:38:10.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94549656900000001 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.633783+0000", 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115307 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,6", 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.633905+0000", 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121409 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636196+0000", 2024-02-05T07:38:10.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022911559999999999 2024-02-05T07:38:10.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:38:10.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636210+0000", 2024-02-05T07:38:10.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3899e-05 2024-02-05T07:38:10.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:38:10.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.637021+0000", 2024-02-05T07:38:10.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081141799999999995 2024-02-05T07:38:10.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:38:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.637047+0000", 2024-02-05T07:38:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5973e-05 2024-02-05T07:38:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:38:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.637067+0000", 2024-02-05T07:38:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0169999999999998e-05 2024-02-05T07:38:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:38:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:38:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:38:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24977.0:272015 8.17 8:ed63ad4e:::benchmark_data_smithi007_105755_object135705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-02-05T07:38:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:29:27.722017+0000", 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 523.16638731800003, 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.91513220399999995, 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24977", 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/4242267857", 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 272015 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722017+0000", 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:38:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:38:10.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722017+0000", 2024-02-05T07:38:10.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:38:10.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:38:10.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722018+0000", 2024-02-05T07:38:10.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7600000000000006e-07 2024-02-05T07:38:10.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:38:10.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722021+0000", 2024-02-05T07:38:10.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2024-02-05T07:38:10.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:38:10.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722034+0000", 2024-02-05T07:38:10.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2901999999999999e-05 2024-02-05T07:38:10.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:38:10.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722039+0000", 2024-02-05T07:38:10.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5430000000000001e-06 2024-02-05T07:38:10.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:38:10.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722093+0000", 2024-02-05T07:38:10.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4280999999999997e-05 2024-02-05T07:38:10.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:38:10.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:27.722102+0000", 2024-02-05T07:38:10.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8189999999999998e-06 2024-02-05T07:38:10.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:38:10.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.634969+0000", 2024-02-05T07:38:10.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.91286651699999999 2024-02-05T07:38:10.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:38:10.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.635072+0000", 2024-02-05T07:38:10.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103607 2024-02-05T07:38:10.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,6", 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.635152+0000", 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9505000000000005e-05 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636325+0000", 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011735739999999999 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.636339+0000", 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3173000000000001e-05 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.637105+0000", 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076605499999999997 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.637128+0000", 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3546000000000001e-05 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:29:28.637149+0000", 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0764e-05 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:38:10.912 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:38:10.913 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:38:10.913 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:38:10.913 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:38:10.935 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:10.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:10.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:11.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:11.064 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:11.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:11.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:11.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:11.192 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:11.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:11.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:11.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:38:11 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:38:11.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:11.321 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:11.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:11.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:11.449 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:11.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:11.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:11.538 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 7 16 12289 12273 109.568 108.938 0.0128452 0.00911528 2024-02-05T07:38:11.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:11.592 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:11.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:11.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:11.723 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:11.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:11.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:11.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:11.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T07:38:11.856 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:11.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:11.881 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:11 smithi090 ceph-mon[33213]: osdmap e591: 8 total, 8 up, 5 in 2024-02-05T07:38:11.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:11 smithi007 ceph-mon[29098]: osdmap e591: 8 total, 8 up, 5 in 2024-02-05T07:38:11.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:11 smithi007 ceph-mon[37158]: osdmap e591: 8 total, 8 up, 5 in 2024-02-05T07:38:11.958 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:12.003 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:12.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:12.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:12.105 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:12.135 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:12.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:12.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:12.143 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:38:12.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:12.289 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:12.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:12.390 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:12.419 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:12.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:12.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:12.538 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 8 16 13799 13783 107.659 94.375 0.018133 0.00927492 2024-02-05T07:38:12.562 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:12.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:12.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:12.664 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:12.673 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:38:12.678 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:38:12.690 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:12.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:12.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:12.743 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:38:12.785 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:38:12.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:12.833 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:12.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:12.837 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:38:12.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:12 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:12.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:12 smithi007 ceph-mon[29098]: pgmap v1070: 178 pgs: 3 active+recovering+undersized+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 26 peering, 3 active+remapped+backfill_toofull, 137 active+clean; 535 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 511 B/s rd, 90 MiB/s wr, 1.45k op/s; 2/26244 objects degraded (0.008%); 2081/26244 objects misplaced (7.929%); 3.3 MiB/s, 56 objects/s recovering 2024-02-05T07:38:12.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:12 smithi007 ceph-mon[29098]: Health check failed: Degraded data redundancy: 2/26244 objects degraded (0.008%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:38:12.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:12 smithi007 ceph-mon[29098]: osdmap e592: 8 total, 8 up, 5 in 2024-02-05T07:38:12.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:12 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1001018107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:38:12.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:12 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:38:12.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:12 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1709382038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:12.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:12 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:12.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:12 smithi007 ceph-mon[37158]: pgmap v1070: 178 pgs: 3 active+recovering+undersized+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 26 peering, 3 active+remapped+backfill_toofull, 137 active+clean; 535 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 511 B/s rd, 90 MiB/s wr, 1.45k op/s; 2/26244 objects degraded (0.008%); 2081/26244 objects misplaced (7.929%); 3.3 MiB/s, 56 objects/s recovering 2024-02-05T07:38:12.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:12 smithi007 ceph-mon[37158]: Health check failed: Degraded data redundancy: 2/26244 objects degraded (0.008%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:38:12.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:12 smithi007 ceph-mon[37158]: osdmap e592: 8 total, 8 up, 5 in 2024-02-05T07:38:12.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:12 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1001018107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:38:12.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:12 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:38:12.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:12 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1709382038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:12.876 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:38:12.917 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:38:12.934 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:12.960 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:38:12.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:12.981 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:12.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:13.000 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:38:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:38:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:38:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:38:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:38:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:38:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:38:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:38:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:38:13.004 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:12 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:13.004 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:12 smithi090 ceph-mon[33213]: pgmap v1070: 178 pgs: 3 active+recovering+undersized+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 26 peering, 3 active+remapped+backfill_toofull, 137 active+clean; 535 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 511 B/s rd, 90 MiB/s wr, 1.45k op/s; 2/26244 objects degraded (0.008%); 2081/26244 objects misplaced (7.929%); 3.3 MiB/s, 56 objects/s recovering 2024-02-05T07:38:13.004 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:12 smithi090 ceph-mon[33213]: Health check failed: Degraded data redundancy: 2/26244 objects degraded (0.008%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:38:13.005 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:12 smithi090 ceph-mon[33213]: osdmap e592: 8 total, 8 up, 5 in 2024-02-05T07:38:13.005 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:12 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1001018107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:38:13.005 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:12 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:38:13.005 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:12 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1709382038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:13.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:13.126 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:13.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:13.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:13.228 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:13.272 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:13.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:13.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:13.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:13.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:13.530 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:13.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:13.538 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 9 16 15397 15381 106.793 99.875 0.00736197 0.00935097 2024-02-05T07:38:13.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:13.675 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:13.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:13.776 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:13.805 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:13.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:13.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:13.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:13.913 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:38:13.913 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:38:13.913 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:13 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:38:13.913 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:13 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:38:13.913 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:13 smithi090 ceph-mon[33213]: osdmap e593: 8 total, 8 up, 5 in 2024-02-05T07:38:13.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:38:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:38:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:38:13.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:13 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:38:13.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:13 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:38:13.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:13 smithi007 ceph-mon[37158]: osdmap e593: 8 total, 8 up, 5 in 2024-02-05T07:38:13.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:13 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:38:13.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:13 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:38:13.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:13 smithi007 ceph-mon[29098]: osdmap e593: 8 total, 8 up, 5 in 2024-02-05T07:38:13.951 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:13.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:13.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:14.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:14.093 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:14.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:14.195 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:14.221 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:14.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:14.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:14.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:14.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:14.435 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 6, 3, 2] out_osds: [5, 7, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:38:14.435 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:38:14.435 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-02-05T07:38:14.435 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-02-05T07:38:14.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:38:14.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:14.525 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:14.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:14.539 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 10 16 17046 17030 106.419 103.062 0.00415629 0.00938983 2024-02-05T07:38:14.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T07:38:14.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:14.656 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:14.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:14.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:14.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:14.783 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:14.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:14.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:14.845 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:38:14.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:14.908 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:14.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:14.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:14 smithi007 ceph-mon[29098]: pgmap v1073: 178 pgs: 3 active+recovering+undersized+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 26 peering, 6 active+remapped+backfill_toofull, 134 active+clean; 674 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.8 KiB/s rd, 91 MiB/s wr, 1.46k op/s; 2/32907 objects degraded (0.006%); 2081/32907 objects misplaced (6.324%); 3.4 MiB/s, 62 objects/s recovering 2024-02-05T07:38:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:14 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:38:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:14 smithi007 ceph-mon[29098]: osdmap e594: 8 total, 8 up, 5 in 2024-02-05T07:38:14.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:14 smithi007 ceph-mon[37158]: pgmap v1073: 178 pgs: 3 active+recovering+undersized+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 26 peering, 6 active+remapped+backfill_toofull, 134 active+clean; 674 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.8 KiB/s rd, 91 MiB/s wr, 1.46k op/s; 2/32907 objects degraded (0.006%); 2081/32907 objects misplaced (6.324%); 3.4 MiB/s, 62 objects/s recovering 2024-02-05T07:38:14.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:14 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:38:14.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:14 smithi007 ceph-mon[37158]: osdmap e594: 8 total, 8 up, 5 in 2024-02-05T07:38:14.967 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:14 smithi090 ceph-mon[33213]: pgmap v1073: 178 pgs: 3 active+recovering+undersized+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 26 peering, 6 active+remapped+backfill_toofull, 134 active+clean; 674 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.8 KiB/s rd, 91 MiB/s wr, 1.46k op/s; 2/32907 objects degraded (0.006%); 2081/32907 objects misplaced (6.324%); 3.4 MiB/s, 62 objects/s recovering 2024-02-05T07:38:14.967 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:14 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:38:14.967 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:14 smithi090 ceph-mon[33213]: osdmap e594: 8 total, 8 up, 5 in 2024-02-05T07:38:15.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:15.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:15.035 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:15.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:15.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:15.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:15.161 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:15.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:15.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:15.286 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:15.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:15.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:15.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:15.416 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:15.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:15.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:15.539 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 11 16 17294 17278 98.1545 15.5 0.00325843 0.0093919 2024-02-05T07:38:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:15.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:15.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:38:15.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:15.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:15.685 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:15.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:15.786 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:15.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:15.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:15.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:15.922 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:15 smithi007 ceph-mon[29098]: osdmap e595: 8 total, 8 up, 5 in 2024-02-05T07:38:15.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:15 smithi007 ceph-mon[29098]: osdmap e596: 8 total, 8 up, 5 in 2024-02-05T07:38:15.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:15 smithi007 ceph-mon[37158]: osdmap e595: 8 total, 8 up, 5 in 2024-02-05T07:38:15.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:15 smithi007 ceph-mon[37158]: osdmap e596: 8 total, 8 up, 5 in 2024-02-05T07:38:15.937 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:15.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:15.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:15 smithi090 ceph-mon[33213]: osdmap e595: 8 total, 8 up, 5 in 2024-02-05T07:38:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:15 smithi090 ceph-mon[33213]: osdmap e596: 8 total, 8 up, 5 in 2024-02-05T07:38:16.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:16.062 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:16.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:16.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:16.189 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:16.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:16.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:16.315 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:16.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:16.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:16.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:16.444 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:16.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:16.539 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 12 16 17446 17430 90.7667 9.5 0.00206938 0.0102543 2024-02-05T07:38:16.544 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:16.572 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:16.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:16.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:16.699 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:16.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:16.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:38:16.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:16.829 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:16.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:16.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:16.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:16.955 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:16.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:16 smithi090 ceph-mon[33213]: pgmap v1076: 178 pgs: 3 peering, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 1 active+undersized+remapped, 4 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 161 active+clean; 870 MiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 503/42327 objects misplaced (1.188%); 25 MiB/s, 408 objects/s recovering 2024-02-05T07:38:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:16 smithi090 ceph-mon[33213]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 15 pgs peering) 2024-02-05T07:38:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:16 smithi090 ceph-mon[33213]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/32907 objects degraded (0.006%), 1 pg degraded) 2024-02-05T07:38:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:16 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:38:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:16 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:38:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:16 smithi090 ceph-mon[33213]: osdmap e597: 8 total, 8 up, 5 in 2024-02-05T07:38:17.056 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:16 smithi007 ceph-mon[29098]: pgmap v1076: 178 pgs: 3 peering, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 1 active+undersized+remapped, 4 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 161 active+clean; 870 MiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 503/42327 objects misplaced (1.188%); 25 MiB/s, 408 objects/s recovering 2024-02-05T07:38:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:16 smithi007 ceph-mon[29098]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 15 pgs peering) 2024-02-05T07:38:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:16 smithi007 ceph-mon[29098]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/32907 objects degraded (0.006%), 1 pg degraded) 2024-02-05T07:38:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:16 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:38:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:16 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:38:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:16 smithi007 ceph-mon[29098]: osdmap e597: 8 total, 8 up, 5 in 2024-02-05T07:38:17.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:16 smithi007 ceph-mon[37158]: pgmap v1076: 178 pgs: 3 peering, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 1 active+undersized+remapped, 4 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 161 active+clean; 870 MiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 503/42327 objects misplaced (1.188%); 25 MiB/s, 408 objects/s recovering 2024-02-05T07:38:17.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:16 smithi007 ceph-mon[37158]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 15 pgs peering) 2024-02-05T07:38:17.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:16 smithi007 ceph-mon[37158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/32907 objects degraded (0.006%), 1 pg degraded) 2024-02-05T07:38:17.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:16 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:38:17.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:16 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:38:17.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:16 smithi007 ceph-mon[37158]: osdmap e597: 8 total, 8 up, 5 in 2024-02-05T07:38:17.080 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:17.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:17.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:17.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:17.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:17.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:17.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:17.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:17.475 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:17.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:17.539 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 13 16 18314 18298 87.9571 54.25 0.00212267 0.0112897 2024-02-05T07:38:17.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:17.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:17.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:17.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:17.748 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:17.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:17.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:17.895 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:17.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:17.996 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:18.023 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:18.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:18.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:18.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:18.152 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:18.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:18.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:18.280 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:18.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:18.382 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:18.423 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:18.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:18.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:18.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:18.539 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 14 16 19950 19934 88.9773 102.25 0.00518138 0.0112308 2024-02-05T07:38:18.551 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:18.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:18.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:18.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:18.693 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:18.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:18.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:18.795 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:18.835 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:18.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:18.934 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:18 smithi090 ceph-mon[33213]: pgmap v1079: 178 pgs: 6 peering, 2 active+undersized+remapped, 6 active+remapped+backfill_toofull, 164 active+clean; 1.0 GiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 33 MiB/s, 530 objects/s recovering 2024-02-05T07:38:18.934 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:18 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:38:18.934 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:18 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:38:18.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:18.964 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:18.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:18.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:18.978 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:38:18.978 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:38:19.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:19.094 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:19.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:19.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:19.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:18 smithi007 ceph-mon[29098]: pgmap v1079: 178 pgs: 6 peering, 2 active+undersized+remapped, 6 active+remapped+backfill_toofull, 164 active+clean; 1.0 GiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 33 MiB/s, 530 objects/s recovering 2024-02-05T07:38:19.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:18 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:38:19.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:18 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:38:19.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:18 smithi007 ceph-mon[37158]: pgmap v1079: 178 pgs: 6 peering, 2 active+undersized+remapped, 6 active+remapped+backfill_toofull, 164 active+clean; 1.0 GiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 33 MiB/s, 530 objects/s recovering 2024-02-05T07:38:19.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:18 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:38:19.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:18 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:38:19.195 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:19.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:19.236 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:19.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:19.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:19.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:19.363 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:19.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:19.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:19.504 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:19.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:19.539 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 15 16 21556 21540 89.7365 100.375 0.00412703 0.0111381 2024-02-05T07:38:19.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:19.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:19.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:19.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:19.773 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:19 smithi090 ceph-mon[33213]: osdmap e598: 8 total, 8 up, 5 in 2024-02-05T07:38:19.775 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:19.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:19.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:19.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T07:38:19.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:19.929 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:19.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:19.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:19.972 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:19.995 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:20.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:20.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:20.087 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:20.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:20.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:19 smithi007 ceph-mon[29098]: osdmap e598: 8 total, 8 up, 5 in 2024-02-05T07:38:20.115 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:19 smithi007 ceph-mon[37158]: osdmap e598: 8 total, 8 up, 5 in 2024-02-05T07:38:20.152 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:38:20.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:20.215 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:20.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:20.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:20.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:20.342 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:20.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:20.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:20.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:20.486 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:20.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:20.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:20.539 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 16 16 23221 23205 90.6313 104.062 0.0109991 0.0110259 2024-02-05T07:38:20.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:20.632 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:20.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:20.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:20.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:20.759 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:20.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:20.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:20.886 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:20.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:20.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:20.988 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:21.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:20 smithi090 ceph-mon[33213]: pgmap v1081: 178 pgs: 6 peering, 2 active+undersized+remapped, 6 active+remapped+backfill_toofull, 164 active+clean; 1.1 GiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 28 MiB/s, 440 objects/s recovering 2024-02-05T07:38:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:20 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:38:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:20 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:38:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:20 smithi090 ceph-mon[33213]: osdmap e599: 8 total, 8 up, 5 in 2024-02-05T07:38:21.030 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:21.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:21.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:20 smithi007 ceph-mon[29098]: pgmap v1081: 178 pgs: 6 peering, 2 active+undersized+remapped, 6 active+remapped+backfill_toofull, 164 active+clean; 1.1 GiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 28 MiB/s, 440 objects/s recovering 2024-02-05T07:38:21.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:20 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:38:21.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:20 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:38:21.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:20 smithi007 ceph-mon[29098]: osdmap e599: 8 total, 8 up, 5 in 2024-02-05T07:38:21.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:20 smithi007 ceph-mon[37158]: pgmap v1081: 178 pgs: 6 peering, 2 active+undersized+remapped, 6 active+remapped+backfill_toofull, 164 active+clean; 1.1 GiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 28 MiB/s, 440 objects/s recovering 2024-02-05T07:38:21.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:20 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:38:21.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:20 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:38:21.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:20 smithi007 ceph-mon[37158]: osdmap e599: 8 total, 8 up, 5 in 2024-02-05T07:38:21.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:21.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:21.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:21.284 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:21.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:21.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:21.437 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:21.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:21.493 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:38:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:38:21.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:21.545 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 17 16 24713 24697 90.7782 93.25 0.0083718 0.0110062 2024-02-05T07:38:21.578 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:21.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:21.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:21.707 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:38:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:21.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:38:21.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:21.862 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:21.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:21.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:21.994 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:21.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:21.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:22.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:22.111 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-02-05T07:38:22.112 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:38:22.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:22.140 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:22.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:22.241 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:22.268 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:22.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:22.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:22.397 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:22.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:22.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:22.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:22.541 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:22.541 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 18 16 26312 26296 91.2829 99.9375 0.00636049 0.0109479 2024-02-05T07:38:22.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:22.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:22.643 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:22.669 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:22.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:22.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:22.809 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:22.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:22.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:22.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:22.944 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:22.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:22 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:22 smithi090 ceph-mon[33213]: pgmap v1083: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 1.3 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 891 B/s rd, 56 MiB/s wr, 889 op/s; 11 KiB/s, 0 objects/s recovering 2024-02-05T07:38:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:22 smithi090 ceph-mon[33213]: from='client.25561 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:38:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:22 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:38:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:22 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:38:23.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:23.075 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:23.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:23.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:23.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:22 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:23.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:22 smithi007 ceph-mon[29098]: pgmap v1083: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 1.3 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 891 B/s rd, 56 MiB/s wr, 889 op/s; 11 KiB/s, 0 objects/s recovering 2024-02-05T07:38:23.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:22 smithi007 ceph-mon[29098]: from='client.25561 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:38:23.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:22 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:38:23.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:22 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:38:23.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:22 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:23.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:22 smithi007 ceph-mon[37158]: pgmap v1083: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 1.3 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 891 B/s rd, 56 MiB/s wr, 889 op/s; 11 KiB/s, 0 objects/s recovering 2024-02-05T07:38:23.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:22 smithi007 ceph-mon[37158]: from='client.25561 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:38:23.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:22 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:38:23.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:22 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:38:23.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:23.200 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:23.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:23.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:23.328 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:23.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:23.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:23.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:23.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:23.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:23.542 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 19 16 27644 27628 90.8598 83.25 0.00761468 0.0109985 2024-02-05T07:38:23.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:23.584 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:23.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:23.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:23.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:23.710 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:23.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:23.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:23.856 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:23.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:23.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:23.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:23 smithi007 ceph-mon[29098]: osdmap e600: 8 total, 8 up, 5 in 2024-02-05T07:38:23.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:38:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:38:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:38:23.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:23 smithi007 ceph-mon[37158]: osdmap e600: 8 total, 8 up, 5 in 2024-02-05T07:38:23.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:23.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:23.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:24.004 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:23 smithi090 ceph-mon[33213]: osdmap e600: 8 total, 8 up, 5 in 2024-02-05T07:38:24.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T07:38:24.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:24.146 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:24.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:24.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:24.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:24.276 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:24.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:24.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:24.413 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:24.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:24.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:24.541 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:24.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:24.542 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:38:24.542088+0000 min lat: 0.00118746 max lat: 1.24847 avg lat: 0.0110431 2024-02-05T07:38:24.542 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:38:24.542 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 20 16 28981 28965 90.4946 83.5625 0.00484044 0.0110431 2024-02-05T07:38:24.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:24.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:24.695 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:24.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:24.796 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:24.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:24.838 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:24.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:24.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:24.981 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:24.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:25.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:24 smithi090 ceph-mon[33213]: pgmap v1085: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 1.4 GiB data, 8.2 GiB used, 439 GiB / 447 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s; 11 KiB/s, 0 objects/s recovering 2024-02-05T07:38:25.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:25.123 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:25.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:25.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:24 smithi007 ceph-mon[29098]: pgmap v1085: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 1.4 GiB data, 8.2 GiB used, 439 GiB / 447 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s; 11 KiB/s, 0 objects/s recovering 2024-02-05T07:38:25.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:24 smithi007 ceph-mon[37158]: pgmap v1085: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 1.4 GiB data, 8.2 GiB used, 439 GiB / 447 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s; 11 KiB/s, 0 objects/s recovering 2024-02-05T07:38:25.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:25.251 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:25.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:25.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:25.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:25.377 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:25.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:25.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:25.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:25.542 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 21 16 30273 30257 90.0303 80.75 0.00450203 0.0110969 2024-02-05T07:38:25.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:25.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:25.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:25.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:25.790 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:25.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:25.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:25.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:25.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:25.931 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:25.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:26.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:26.056 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:26.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:26.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:26.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:26.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:26.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:26.313 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:26.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:26.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:26.439 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:26.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:26.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:26.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:26.542 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 22 16 31691 31675 89.966 88.625 0.00966526 0.0111092 2024-02-05T07:38:26.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:26.569 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:26.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:26.670 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:26.711 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:26.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:26.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:26.855 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:26.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:26.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:26.982 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:26.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:27.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:26 smithi090 ceph-mon[33213]: pgmap v1086: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 1.6 GiB data, 8.8 GiB used, 438 GiB / 447 GiB avail; 1.5 KiB/s rd, 88 MiB/s wr, 1.40k op/s; 9.6 KiB/s, 0 objects/s recovering 2024-02-05T07:38:27.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:27.095 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:26 smithi007 ceph-mon[37158]: pgmap v1086: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 1.6 GiB data, 8.8 GiB used, 438 GiB / 447 GiB avail; 1.5 KiB/s rd, 88 MiB/s wr, 1.40k op/s; 9.6 KiB/s, 0 objects/s recovering 2024-02-05T07:38:27.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:26 smithi007 ceph-mon[29098]: pgmap v1086: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 1.6 GiB data, 8.8 GiB used, 438 GiB / 447 GiB avail; 1.5 KiB/s rd, 88 MiB/s wr, 1.40k op/s; 9.6 KiB/s, 0 objects/s recovering 2024-02-05T07:38:27.109 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:27.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:27.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:27.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:27.265 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:27.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:27.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:27.407 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:27.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:27.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:27.534 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:27.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:27.542 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 23 16 33219 33203 90.2062 95.5 0.0100588 0.0110796 2024-02-05T07:38:27.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:27.661 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:27.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:27.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:27.713 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1083,"stamp":"2024-02-05T07:38:21.376136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1351972116,"num_objects":20818,"num_object_clones":0,"num_object_copies":62454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20818,"num_whiteouts":0,"num_read":1569,"num_read_kb":1325,"num_write":21349,"num_write_kb":1319821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2797,"num_bytes_recovered":150488798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21505,"ondisk_log_size":21505,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":8059052,"kb_used_data":3838316,"kb_used_omap":11,"kb_used_meta":4220660,"kb_avail":460666708,"statfs":{"total":479975178240,"available":471722708992,"internally_reserved":0,"allocated":3930435584,"data_stored":3925096985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12005,"internal_metadata":4321956123},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":334692352,"num_objects":5107,"num_object_clones":0,"num_object_copies":15321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5107,"num_write_kb":326848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.745145"},"pg_stats":[{"pgid":"11.3","version":"598'1291","reported_seq":1337,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.798052+0000","last_change":"2024-02-05T07:37:52.811299+0000","last_active":"2024-02-05T07:38:18.798052+0000","last_peered":"2024-02-05T07:38:18.798052+0000","last_clean":"2024-02-05T07:38:18.798052+0000","last_became_active":"2024-02-05T07:37:52.811051+0000","last_became_peered":"2024-02-05T07:37:52.811051+0000","last_unstale":"2024-02-05T07:38:18.798052+0000","last_undegraded":"2024-02-05T07:38:18.798052+0000","last_fullsized":"2024-02-05T07:38:18.798052+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1291,"log_dups_size":0,"ondisk_log_size":1291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:06.881933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1291,"num_write_kb":82624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":1311,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.649090+0000","last_change":"2024-02-05T07:37:41.337312+0000","last_active":"2024-02-05T07:38:16.649090+0000","last_peered":"2024-02-05T07:38:16.649090+0000","last_clean":"2024-02-05T07:38:16.649090+0000","last_became_active":"2024-02-05T07:37:36.688127+0000","last_became_peered":"2024-02-05T07:37:36.688127+0000","last_unstale":"2024-02-05T07:38:16.649090+0000","last_undegraded":"2024-02-05T07:38:16.649090+0000","last_fullsized":"2024-02-05T07:38:16.649090+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.337196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.337196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:03.725482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"326'25","reported_seq":666,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.649570+0000","last_change":"2024-02-05T07:38:09.523613+0000","last_active":"2024-02-05T07:38:16.649570+0000","last_peered":"2024-02-05T07:38:16.649570+0000","last_clean":"2024-02-05T07:38:16.649570+0000","last_became_active":"2024-02-05T07:38:09.493734+0000","last_became_peered":"2024-02-05T07:38:09.493734+0000","last_unstale":"2024-02-05T07:38:16.649570+0000","last_undegraded":"2024-02-05T07:38:16.649570+0000","last_fullsized":"2024-02-05T07:38:16.649570+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:37:02.567716+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:37:02.567716+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.392546+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":695,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.651676+0000","last_change":"2024-02-05T07:38:09.498695+0000","last_active":"2024-02-05T07:38:16.651676+0000","last_peered":"2024-02-05T07:38:16.651676+0000","last_clean":"2024-02-05T07:38:16.651676+0000","last_became_active":"2024-02-05T07:38:09.496706+0000","last_became_peered":"2024-02-05T07:38:09.496706+0000","last_unstale":"2024-02-05T07:38:16.651676+0000","last_undegraded":"2024-02-05T07:38:16.651676+0000","last_fullsized":"2024-02-05T07:38:16.651676+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:34.736160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:34.736160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"538'13","reported_seq":161,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694863+0000","last_change":"2024-02-05T07:36:32.822462+0000","last_active":"2024-02-05T07:38:18.694863+0000","last_peered":"2024-02-05T07:38:18.694863+0000","last_clean":"2024-02-05T07:38:18.694863+0000","last_became_active":"2024-02-05T07:36:32.743613+0000","last_became_peered":"2024-02-05T07:36:32.743613+0000","last_unstale":"2024-02-05T07:38:18.694863+0000","last_undegraded":"2024-02-05T07:38:18.694863+0000","last_fullsized":"2024-02-05T07:38:18.694863+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:34:29.851477+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:34:29.851477+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.318046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1005,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.699846+0000","last_change":"2024-02-05T07:38:09.495987+0000","last_active":"2024-02-05T07:38:18.699846+0000","last_peered":"2024-02-05T07:38:18.699846+0000","last_clean":"2024-02-05T07:38:18.699846+0000","last_became_active":"2024-02-05T07:38:09.495645+0000","last_became_peered":"2024-02-05T07:38:09.495645+0000","last_unstale":"2024-02-05T07:38:18.699846+0000","last_undegraded":"2024-02-05T07:38:18.699846+0000","last_fullsized":"2024-02-05T07:38:18.699846+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:50.243379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:37:50.243379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:08.869868+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2","version":"598'1377","reported_seq":1425,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:20.500732+0000","last_change":"2024-02-05T07:37:53.089553+0000","last_active":"2024-02-05T07:38:20.500732+0000","last_peered":"2024-02-05T07:38:20.500732+0000","last_clean":"2024-02-05T07:38:20.500732+0000","last_became_active":"2024-02-05T07:37:53.089215+0000","last_became_peered":"2024-02-05T07:37:53.089215+0000","last_unstale":"2024-02-05T07:38:20.500732+0000","last_undegraded":"2024-02-05T07:38:20.500732+0000","last_fullsized":"2024-02-05T07:38:20.500732+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1377,"log_dups_size":0,"ondisk_log_size":1377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:52.206114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90243072,"num_objects":1377,"num_object_clones":0,"num_object_copies":4131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1377,"num_write_kb":88128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":615,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.691152+0000","last_change":"2024-02-05T07:37:38.390073+0000","last_active":"2024-02-05T07:38:18.691152+0000","last_peered":"2024-02-05T07:38:18.691152+0000","last_clean":"2024-02-05T07:38:18.691152+0000","last_became_active":"2024-02-05T07:37:36.688448+0000","last_became_peered":"2024-02-05T07:37:36.688448+0000","last_unstale":"2024-02-05T07:38:18.691152+0000","last_undegraded":"2024-02-05T07:38:18.691152+0000","last_fullsized":"2024-02-05T07:38:18.691152+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.389984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.389984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":69,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:51.027955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"326'31","reported_seq":1107,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694935+0000","last_change":"2024-02-05T07:38:09.530860+0000","last_active":"2024-02-05T07:38:18.694935+0000","last_peered":"2024-02-05T07:38:18.694935+0000","last_clean":"2024-02-05T07:38:18.694935+0000","last_became_active":"2024-02-05T07:38:09.503330+0000","last_became_peered":"2024-02-05T07:38:09.503330+0000","last_unstale":"2024-02-05T07:38:18.694935+0000","last_undegraded":"2024-02-05T07:38:18.694935+0000","last_fullsized":"2024-02-05T07:38:18.694935+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'31","last_scrub_stamp":"2024-02-05T07:37:54.045831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.045831+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:03.367249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":977,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693905+0000","last_change":"2024-02-05T07:38:09.471668+0000","last_active":"2024-02-05T07:38:18.693905+0000","last_peered":"2024-02-05T07:38:18.693905+0000","last_clean":"2024-02-05T07:38:18.693905+0000","last_became_active":"2024-02-05T07:38:09.471049+0000","last_became_peered":"2024-02-05T07:38:09.471049+0000","last_unstale":"2024-02-05T07:38:18.693905+0000","last_undegraded":"2024-02-05T07:38:18.693905+0000","last_fullsized":"2024-02-05T07:38:18.693905+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:46.657168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:37:46.657168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:47.714295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":29,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693846+0000","last_change":"2024-02-05T07:38:09.464556+0000","last_active":"2024-02-05T07:38:18.693846+0000","last_peered":"2024-02-05T07:38:18.693846+0000","last_clean":"2024-02-05T07:38:18.693846+0000","last_became_active":"2024-02-05T07:38:09.464116+0000","last_became_peered":"2024-02-05T07:38:09.464116+0000","last_unstale":"2024-02-05T07:38:18.693846+0000","last_undegraded":"2024-02-05T07:38:18.693846+0000","last_fullsized":"2024-02-05T07:38:18.693846+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:41.530749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:41.530749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.742085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":141,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.696065+0000","last_change":"2024-02-05T07:38:09.483305+0000","last_active":"2024-02-05T07:38:18.696065+0000","last_peered":"2024-02-05T07:38:18.696065+0000","last_clean":"2024-02-05T07:38:18.696065+0000","last_became_active":"2024-02-05T07:38:09.482289+0000","last_became_peered":"2024-02-05T07:38:09.482289+0000","last_unstale":"2024-02-05T07:38:18.696065+0000","last_undegraded":"2024-02-05T07:38:18.696065+0000","last_fullsized":"2024-02-05T07:38:18.696065+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:23.161521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:23.161521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.318012+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"598'1263","reported_seq":1311,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.798502+0000","last_change":"2024-02-05T07:37:53.089618+0000","last_active":"2024-02-05T07:38:18.798502+0000","last_peered":"2024-02-05T07:38:18.798502+0000","last_clean":"2024-02-05T07:38:18.798502+0000","last_became_active":"2024-02-05T07:37:53.089261+0000","last_became_peered":"2024-02-05T07:37:53.089261+0000","last_unstale":"2024-02-05T07:38:18.798502+0000","last_undegraded":"2024-02-05T07:38:18.798502+0000","last_fullsized":"2024-02-05T07:38:18.798502+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1263,"log_dups_size":0,"ondisk_log_size":1263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:18.714101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82771968,"num_objects":1263,"num_object_clones":0,"num_object_copies":3789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1263,"num_write_kb":80832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":902,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.691852+0000","last_change":"2024-02-05T07:37:39.417183+0000","last_active":"2024-02-05T07:38:18.691852+0000","last_peered":"2024-02-05T07:38:18.691852+0000","last_clean":"2024-02-05T07:38:18.691852+0000","last_became_active":"2024-02-05T07:37:36.693385+0000","last_became_peered":"2024-02-05T07:37:36.693385+0000","last_unstale":"2024-02-05T07:38:18.691852+0000","last_undegraded":"2024-02-05T07:38:18.691852+0000","last_fullsized":"2024-02-05T07:38:18.691852+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.417059+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.417059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:04.358015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"326'24","reported_seq":635,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694443+0000","last_change":"2024-02-05T07:38:09.520379+0000","last_active":"2024-02-05T07:38:18.694443+0000","last_peered":"2024-02-05T07:38:18.694443+0000","last_clean":"2024-02-05T07:38:18.694443+0000","last_became_active":"2024-02-05T07:38:09.471120+0000","last_became_peered":"2024-02-05T07:38:09.471120+0000","last_unstale":"2024-02-05T07:38:18.694443+0000","last_undegraded":"2024-02-05T07:38:18.694443+0000","last_fullsized":"2024-02-05T07:38:18.694443+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:35:13.047702+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:35:13.047702+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.741615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1025,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693241+0000","last_change":"2024-02-05T07:38:09.490148+0000","last_active":"2024-02-05T07:38:18.693241+0000","last_peered":"2024-02-05T07:38:18.693241+0000","last_clean":"2024-02-05T07:38:18.693241+0000","last_became_active":"2024-02-05T07:38:09.489807+0000","last_became_peered":"2024-02-05T07:38:09.489807+0000","last_unstale":"2024-02-05T07:38:18.693241+0000","last_undegraded":"2024-02-05T07:38:18.693241+0000","last_fullsized":"2024-02-05T07:38:18.693241+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:03.618665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:37:03.618665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.318186+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":853,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694769+0000","last_change":"2024-02-05T07:38:09.472871+0000","last_active":"2024-02-05T07:38:18.694769+0000","last_peered":"2024-02-05T07:38:18.694769+0000","last_clean":"2024-02-05T07:38:18.694769+0000","last_became_active":"2024-02-05T07:38:09.472546+0000","last_became_peered":"2024-02-05T07:38:09.472546+0000","last_unstale":"2024-02-05T07:38:18.694769+0000","last_undegraded":"2024-02-05T07:38:18.694769+0000","last_fullsized":"2024-02-05T07:38:18.694769+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:14.031714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:35:14.031714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.741646+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":706,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.648545+0000","last_change":"2024-02-05T07:37:52.481462+0000","last_active":"2024-02-05T07:38:16.648545+0000","last_peered":"2024-02-05T07:38:16.648545+0000","last_clean":"2024-02-05T07:38:16.648545+0000","last_became_active":"2024-02-05T07:30:40.498529+0000","last_became_peered":"2024-02-05T07:30:40.498529+0000","last_unstale":"2024-02-05T07:38:16.648545+0000","last_undegraded":"2024-02-05T07:38:16.648545+0000","last_fullsized":"2024-02-05T07:38:16.648545+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:52.481255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:52.481255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:20.603837+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"597'1093","reported_seq":1139,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.830083+0000","last_change":"2024-02-05T07:37:52.817253+0000","last_active":"2024-02-05T07:38:16.830083+0000","last_peered":"2024-02-05T07:38:16.830083+0000","last_clean":"2024-02-05T07:38:16.830083+0000","last_became_active":"2024-02-05T07:37:52.816932+0000","last_became_peered":"2024-02-05T07:37:52.816932+0000","last_unstale":"2024-02-05T07:38:16.830083+0000","last_undegraded":"2024-02-05T07:38:16.830083+0000","last_fullsized":"2024-02-05T07:38:16.830083+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1093,"log_dups_size":0,"ondisk_log_size":1093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:19.281227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71630848,"num_objects":1093,"num_object_clones":0,"num_object_copies":3279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1093,"num_write_kb":69952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":498,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.691359+0000","last_change":"2024-02-05T07:38:11.671455+0000","last_active":"2024-02-05T07:38:18.691359+0000","last_peered":"2024-02-05T07:38:18.691359+0000","last_clean":"2024-02-05T07:38:18.691359+0000","last_became_active":"2024-02-05T07:38:11.671087+0000","last_became_peered":"2024-02-05T07:38:11.671087+0000","last_unstale":"2024-02-05T07:38:18.691359+0000","last_undegraded":"2024-02-05T07:38:18.691359+0000","last_fullsized":"2024-02-05T07:38:18.691359+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:40.411931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:40.411931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:41.823264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"326'34","reported_seq":1231,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.703262+0000","last_change":"2024-02-05T07:38:09.528549+0000","last_active":"2024-02-05T07:38:18.703262+0000","last_peered":"2024-02-05T07:38:18.703262+0000","last_clean":"2024-02-05T07:38:18.703262+0000","last_became_active":"2024-02-05T07:38:09.500221+0000","last_became_peered":"2024-02-05T07:38:09.500221+0000","last_unstale":"2024-02-05T07:38:18.703262+0000","last_undegraded":"2024-02-05T07:38:18.703262+0000","last_fullsized":"2024-02-05T07:38:18.703262+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'34","last_scrub_stamp":"2024-02-05T07:36:46.676770+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.676770+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.691649+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":710,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.649092+0000","last_change":"2024-02-05T07:38:09.485474+0000","last_active":"2024-02-05T07:38:16.649092+0000","last_peered":"2024-02-05T07:38:16.649092+0000","last_clean":"2024-02-05T07:38:16.649092+0000","last_became_active":"2024-02-05T07:38:09.485164+0000","last_became_peered":"2024-02-05T07:38:09.485164+0000","last_unstale":"2024-02-05T07:38:16.649092+0000","last_undegraded":"2024-02-05T07:38:16.649092+0000","last_fullsized":"2024-02-05T07:38:16.649092+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:47.116727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:47.116727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"589'12","reported_seq":1252,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.696429+0000","last_change":"2024-02-05T07:38:09.514237+0000","last_active":"2024-02-05T07:38:18.696429+0000","last_peered":"2024-02-05T07:38:18.696429+0000","last_clean":"2024-02-05T07:38:18.696429+0000","last_became_active":"2024-02-05T07:38:09.488856+0000","last_became_peered":"2024-02-05T07:38:09.488856+0000","last_unstale":"2024-02-05T07:38:18.696429+0000","last_undegraded":"2024-02-05T07:38:18.696429+0000","last_fullsized":"2024-02-05T07:38:18.696429+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'11","last_scrub_stamp":"2024-02-05T07:36:59.523421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.523421+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.318203+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":135,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693526+0000","last_change":"2024-02-05T07:36:32.744571+0000","last_active":"2024-02-05T07:38:18.693526+0000","last_peered":"2024-02-05T07:38:18.693526+0000","last_clean":"2024-02-05T07:38:18.693526+0000","last_became_active":"2024-02-05T07:36:32.744070+0000","last_became_peered":"2024-02-05T07:36:32.744070+0000","last_unstale":"2024-02-05T07:38:18.693526+0000","last_undegraded":"2024-02-05T07:38:18.693526+0000","last_fullsized":"2024-02-05T07:38:18.693526+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:38.025380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.025380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.318063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"598'1435","reported_seq":1481,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:20.511684+0000","last_change":"2024-02-05T07:37:53.089610+0000","last_active":"2024-02-05T07:38:20.511684+0000","last_peered":"2024-02-05T07:38:20.511684+0000","last_clean":"2024-02-05T07:38:20.511684+0000","last_became_active":"2024-02-05T07:37:53.089296+0000","last_became_peered":"2024-02-05T07:37:53.089296+0000","last_unstale":"2024-02-05T07:38:20.511684+0000","last_undegraded":"2024-02-05T07:38:20.511684+0000","last_fullsized":"2024-02-05T07:38:20.511684+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1435,"log_dups_size":0,"ondisk_log_size":1435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:56.640199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94044160,"num_objects":1435,"num_object_clones":0,"num_object_copies":4305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1435,"num_write_kb":91840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":1084,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693263+0000","last_change":"2024-02-05T07:38:13.714836+0000","last_active":"2024-02-05T07:38:18.693263+0000","last_peered":"2024-02-05T07:38:18.693263+0000","last_clean":"2024-02-05T07:38:18.693263+0000","last_became_active":"2024-02-05T07:38:13.714462+0000","last_became_peered":"2024-02-05T07:38:13.714462+0000","last_unstale":"2024-02-05T07:38:18.693263+0000","last_undegraded":"2024-02-05T07:38:18.693263+0000","last_fullsized":"2024-02-05T07:38:18.693263+0000","mapping_epoch":593,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.203320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.203320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:19.343865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":662,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.633757+0000","last_change":"2024-02-05T07:38:09.479255+0000","last_active":"2024-02-05T07:38:16.633757+0000","last_peered":"2024-02-05T07:38:16.633757+0000","last_clean":"2024-02-05T07:38:16.633757+0000","last_became_active":"2024-02-05T07:38:09.478485+0000","last_became_peered":"2024-02-05T07:38:09.478485+0000","last_unstale":"2024-02-05T07:38:16.633757+0000","last_undegraded":"2024-02-05T07:38:16.633757+0000","last_fullsized":"2024-02-05T07:38:16.633757+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.155552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.155552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:52.680972+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"454'38","reported_seq":766,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.633636+0000","last_change":"2024-02-05T07:34:38.943008+0000","last_active":"2024-02-05T07:38:16.633636+0000","last_peered":"2024-02-05T07:38:16.633636+0000","last_clean":"2024-02-05T07:38:16.633636+0000","last_became_active":"2024-02-05T07:30:40.496940+0000","last_became_peered":"2024-02-05T07:30:40.496940+0000","last_unstale":"2024-02-05T07:38:16.633636+0000","last_undegraded":"2024-02-05T07:38:16.633636+0000","last_fullsized":"2024-02-05T07:38:16.633636+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'38","last_scrub_stamp":"2024-02-05T07:34:38.942830+0000","last_deep_scrub":"454'38","last_deep_scrub_stamp":"2024-02-05T07:34:38.942830+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.942830+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":702,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.652222+0000","last_change":"2024-02-05T07:38:09.490257+0000","last_active":"2024-02-05T07:38:16.652222+0000","last_peered":"2024-02-05T07:38:16.652222+0000","last_clean":"2024-02-05T07:38:16.652222+0000","last_became_active":"2024-02-05T07:38:09.488092+0000","last_became_peered":"2024-02-05T07:38:09.488092+0000","last_unstale":"2024-02-05T07:38:16.652222+0000","last_undegraded":"2024-02-05T07:38:16.652222+0000","last_fullsized":"2024-02-05T07:38:16.652222+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:48.701399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:33:48.701399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1020,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.648991+0000","last_change":"2024-02-05T07:38:09.480292+0000","last_active":"2024-02-05T07:38:16.648991+0000","last_peered":"2024-02-05T07:38:16.648991+0000","last_clean":"2024-02-05T07:38:16.648991+0000","last_became_active":"2024-02-05T07:38:09.479403+0000","last_became_peered":"2024-02-05T07:38:09.479403+0000","last_unstale":"2024-02-05T07:38:16.648991+0000","last_undegraded":"2024-02-05T07:38:16.648991+0000","last_fullsized":"2024-02-05T07:38:16.648991+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:01.252507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:38:01.252507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:05.928451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"598'1473","reported_seq":2710,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:20.104785+0000","last_change":"2024-02-05T07:38:16.659362+0000","last_active":"2024-02-05T07:38:20.104785+0000","last_peered":"2024-02-05T07:38:20.104785+0000","last_clean":"2024-02-05T07:38:20.104785+0000","last_became_active":"2024-02-05T07:38:16.656778+0000","last_became_peered":"2024-02-05T07:38:16.656778+0000","last_unstale":"2024-02-05T07:38:20.104785+0000","last_undegraded":"2024-02-05T07:38:20.104785+0000","last_fullsized":"2024-02-05T07:38:20.104785+0000","mapping_epoch":596,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:00.828689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96534528,"num_objects":1473,"num_object_clones":0,"num_object_copies":4419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1473,"num_write_kb":94272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":589,"num_bytes_recovered":38600704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":1455,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.648702+0000","last_change":"2024-02-05T07:38:11.676831+0000","last_active":"2024-02-05T07:38:16.648702+0000","last_peered":"2024-02-05T07:38:16.648702+0000","last_clean":"2024-02-05T07:38:16.648702+0000","last_became_active":"2024-02-05T07:38:11.676482+0000","last_became_peered":"2024-02-05T07:38:11.676482+0000","last_unstale":"2024-02-05T07:38:16.648702+0000","last_undegraded":"2024-02-05T07:38:16.648702+0000","last_fullsized":"2024-02-05T07:38:16.648702+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:07.353770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:07.353770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:33.183952+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"589'11","reported_seq":845,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.649411+0000","last_change":"2024-02-05T07:38:09.522356+0000","last_active":"2024-02-05T07:38:16.649411+0000","last_peered":"2024-02-05T07:38:16.649411+0000","last_clean":"2024-02-05T07:38:16.649411+0000","last_became_active":"2024-02-05T07:38:09.495697+0000","last_became_peered":"2024-02-05T07:38:09.495697+0000","last_unstale":"2024-02-05T07:38:16.649411+0000","last_undegraded":"2024-02-05T07:38:16.649411+0000","last_fullsized":"2024-02-05T07:38:16.649411+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'10","last_scrub_stamp":"2024-02-05T07:37:59.362892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:59.362892+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:01.515806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"326'35","reported_seq":664,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693056+0000","last_change":"2024-02-05T07:38:09.531968+0000","last_active":"2024-02-05T07:38:18.693056+0000","last_peered":"2024-02-05T07:38:18.693056+0000","last_clean":"2024-02-05T07:38:18.693056+0000","last_became_active":"2024-02-05T07:38:09.480688+0000","last_became_peered":"2024-02-05T07:38:09.480688+0000","last_unstale":"2024-02-05T07:38:18.693056+0000","last_undegraded":"2024-02-05T07:38:18.693056+0000","last_fullsized":"2024-02-05T07:38:18.693056+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:36:35.442664+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:36:35.442664+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.742056+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":890,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.700047+0000","last_change":"2024-02-05T07:38:09.503849+0000","last_active":"2024-02-05T07:38:18.700047+0000","last_peered":"2024-02-05T07:38:18.700047+0000","last_clean":"2024-02-05T07:38:18.700047+0000","last_became_active":"2024-02-05T07:38:09.467535+0000","last_became_peered":"2024-02-05T07:38:09.467535+0000","last_unstale":"2024-02-05T07:38:18.700047+0000","last_undegraded":"2024-02-05T07:38:18.700047+0000","last_fullsized":"2024-02-05T07:38:18.700047+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:38:01.592772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:01.592772+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:22.094679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1003,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693592+0000","last_change":"2024-02-05T07:37:58.737127+0000","last_active":"2024-02-05T07:38:18.693592+0000","last_peered":"2024-02-05T07:38:18.693592+0000","last_clean":"2024-02-05T07:38:18.693592+0000","last_became_active":"2024-02-05T07:36:32.701203+0000","last_became_peered":"2024-02-05T07:36:32.701203+0000","last_unstale":"2024-02-05T07:38:18.693592+0000","last_undegraded":"2024-02-05T07:38:18.693592+0000","last_fullsized":"2024-02-05T07:38:18.693592+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:58.737033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:58.737033+0000","last_clean_scrub_stamp":"2024-02-05T07:37:58.737033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:13.241276+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"598'1478","reported_seq":2723,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:20.489810+0000","last_change":"2024-02-05T07:38:16.745449+0000","last_active":"2024-02-05T07:38:20.489810+0000","last_peered":"2024-02-05T07:38:20.489810+0000","last_clean":"2024-02-05T07:38:20.489810+0000","last_became_active":"2024-02-05T07:38:16.742262+0000","last_became_peered":"2024-02-05T07:38:16.742262+0000","last_unstale":"2024-02-05T07:38:20.489810+0000","last_undegraded":"2024-02-05T07:38:20.489810+0000","last_fullsized":"2024-02-05T07:38:20.489810+0000","mapping_epoch":596,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1478,"log_dups_size":0,"ondisk_log_size":1478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:12.278833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96862208,"num_objects":1478,"num_object_clones":0,"num_object_copies":4434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1478,"num_write_kb":94592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":593,"num_bytes_recovered":38862848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":1081,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694909+0000","last_change":"2024-02-05T07:37:42.825173+0000","last_active":"2024-02-05T07:38:18.694909+0000","last_peered":"2024-02-05T07:38:18.694909+0000","last_clean":"2024-02-05T07:38:18.694909+0000","last_became_active":"2024-02-05T07:37:36.688626+0000","last_became_peered":"2024-02-05T07:37:36.688626+0000","last_unstale":"2024-02-05T07:38:18.694909+0000","last_undegraded":"2024-02-05T07:38:18.694909+0000","last_fullsized":"2024-02-05T07:38:18.694909+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:42.822474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:42.822474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:10.833740+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":828,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693700+0000","last_change":"2024-02-05T07:35:21.887728+0000","last_active":"2024-02-05T07:38:18.693700+0000","last_peered":"2024-02-05T07:38:18.693700+0000","last_clean":"2024-02-05T07:38:18.693700+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:38:18.693700+0000","last_undegraded":"2024-02-05T07:38:18.693700+0000","last_fullsized":"2024-02-05T07:38:18.693700+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:35:21.887636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:35:21.887636+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.741709+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"326'25","reported_seq":1195,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.698959+0000","last_change":"2024-02-05T07:38:09.521342+0000","last_active":"2024-02-05T07:38:18.698959+0000","last_peered":"2024-02-05T07:38:18.698959+0000","last_clean":"2024-02-05T07:38:18.698959+0000","last_became_active":"2024-02-05T07:38:09.496732+0000","last_became_peered":"2024-02-05T07:38:09.496732+0000","last_unstale":"2024-02-05T07:38:18.698959+0000","last_undegraded":"2024-02-05T07:38:18.698959+0000","last_fullsized":"2024-02-05T07:38:18.698959+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:37:08.714926+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:37:08.714926+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.691614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":713,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.649662+0000","last_change":"2024-02-05T07:36:32.741872+0000","last_active":"2024-02-05T07:38:16.649662+0000","last_peered":"2024-02-05T07:38:16.649662+0000","last_clean":"2024-02-05T07:38:16.649662+0000","last_became_active":"2024-02-05T07:36:32.741377+0000","last_became_peered":"2024-02-05T07:36:32.741377+0000","last_unstale":"2024-02-05T07:38:16.649662+0000","last_undegraded":"2024-02-05T07:38:16.649662+0000","last_fullsized":"2024-02-05T07:38:16.649662+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:17.067204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:35:17.067204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":399,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.695842+0000","last_change":"2024-02-05T07:37:54.698715+0000","last_active":"2024-02-05T07:38:18.695842+0000","last_peered":"2024-02-05T07:38:18.695842+0000","last_clean":"2024-02-05T07:38:18.695842+0000","last_became_active":"2024-02-05T07:32:58.598709+0000","last_became_peered":"2024-02-05T07:32:58.598709+0000","last_unstale":"2024-02-05T07:38:18.695842+0000","last_undegraded":"2024-02-05T07:38:18.695842+0000","last_fullsized":"2024-02-05T07:38:18.695842+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:54.698600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.698600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:09.839793+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"598'1428","reported_seq":2642,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:20.087119+0000","last_change":"2024-02-05T07:38:15.649296+0000","last_active":"2024-02-05T07:38:20.087119+0000","last_peered":"2024-02-05T07:38:20.087119+0000","last_clean":"2024-02-05T07:38:20.087119+0000","last_became_active":"2024-02-05T07:38:15.646279+0000","last_became_peered":"2024-02-05T07:38:15.646279+0000","last_unstale":"2024-02-05T07:38:20.087119+0000","last_undegraded":"2024-02-05T07:38:20.087119+0000","last_fullsized":"2024-02-05T07:38:20.087119+0000","mapping_epoch":595,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1428,"log_dups_size":0,"ondisk_log_size":1428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:56.784300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93585408,"num_objects":1428,"num_object_clones":0,"num_object_copies":4284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1428,"num_write_kb":91392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":577,"num_bytes_recovered":37814272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":1053,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694284+0000","last_change":"2024-02-05T07:37:39.746542+0000","last_active":"2024-02-05T07:38:18.694284+0000","last_peered":"2024-02-05T07:38:18.694284+0000","last_clean":"2024-02-05T07:38:18.694284+0000","last_became_active":"2024-02-05T07:37:36.693264+0000","last_became_peered":"2024-02-05T07:37:36.693264+0000","last_unstale":"2024-02-05T07:38:18.694284+0000","last_undegraded":"2024-02-05T07:38:18.694284+0000","last_fullsized":"2024-02-05T07:38:18.694284+0000","mapping_epoch":302,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.746435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.746435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:40.674447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":970,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693706+0000","last_change":"2024-02-05T07:36:41.525416+0000","last_active":"2024-02-05T07:38:18.693706+0000","last_peered":"2024-02-05T07:38:18.693706+0000","last_clean":"2024-02-05T07:38:18.693706+0000","last_became_active":"2024-02-05T07:36:32.709025+0000","last_became_peered":"2024-02-05T07:36:32.709025+0000","last_unstale":"2024-02-05T07:38:18.693706+0000","last_undegraded":"2024-02-05T07:38:18.693706+0000","last_fullsized":"2024-02-05T07:38:18.693706+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:41.525301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:36:41.525301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.742172+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"326'29","reported_seq":793,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.700590+0000","last_change":"2024-02-05T07:36:32.797322+0000","last_active":"2024-02-05T07:38:18.700590+0000","last_peered":"2024-02-05T07:38:18.700590+0000","last_clean":"2024-02-05T07:38:18.700590+0000","last_became_active":"2024-02-05T07:36:32.712346+0000","last_became_peered":"2024-02-05T07:36:32.712346+0000","last_unstale":"2024-02-05T07:38:18.700590+0000","last_undegraded":"2024-02-05T07:38:18.700590+0000","last_fullsized":"2024-02-05T07:38:18.700590+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'29","last_scrub_stamp":"2024-02-05T07:35:21.877054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:21.877054+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.691547+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":135,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694662+0000","last_change":"2024-02-05T07:36:32.733360+0000","last_active":"2024-02-05T07:38:18.694662+0000","last_peered":"2024-02-05T07:38:18.694662+0000","last_clean":"2024-02-05T07:38:18.694662+0000","last_became_active":"2024-02-05T07:36:32.732781+0000","last_became_peered":"2024-02-05T07:36:32.732781+0000","last_unstale":"2024-02-05T07:38:18.694662+0000","last_undegraded":"2024-02-05T07:38:18.694662+0000","last_fullsized":"2024-02-05T07:38:18.694662+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:34.799346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:34:34.799346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.318094+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1129,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.697980+0000","last_change":"2024-02-05T07:37:52.721302+0000","last_active":"2024-02-05T07:38:18.697980+0000","last_peered":"2024-02-05T07:38:18.697980+0000","last_clean":"2024-02-05T07:38:18.697980+0000","last_became_active":"2024-02-05T07:36:32.709872+0000","last_became_peered":"2024-02-05T07:36:32.709872+0000","last_unstale":"2024-02-05T07:38:18.697980+0000","last_undegraded":"2024-02-05T07:38:18.697980+0000","last_fullsized":"2024-02-05T07:38:18.697980+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:52.721133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:52.721133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:20.480397+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"597'1038","reported_seq":1084,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.824457+0000","last_change":"2024-02-05T07:37:53.093098+0000","last_active":"2024-02-05T07:38:16.824457+0000","last_peered":"2024-02-05T07:38:16.824457+0000","last_clean":"2024-02-05T07:38:16.824457+0000","last_became_active":"2024-02-05T07:37:53.091347+0000","last_became_peered":"2024-02-05T07:37:53.091347+0000","last_unstale":"2024-02-05T07:38:16.824457+0000","last_undegraded":"2024-02-05T07:38:16.824457+0000","last_fullsized":"2024-02-05T07:38:16.824457+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:17.120261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68026368,"num_objects":1038,"num_object_clones":0,"num_object_copies":3114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1038,"num_write_kb":66432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":713,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693992+0000","last_change":"2024-02-05T07:38:11.676690+0000","last_active":"2024-02-05T07:38:18.693992+0000","last_peered":"2024-02-05T07:38:18.693992+0000","last_clean":"2024-02-05T07:38:18.693992+0000","last_became_active":"2024-02-05T07:38:11.676446+0000","last_became_peered":"2024-02-05T07:38:11.676446+0000","last_unstale":"2024-02-05T07:38:18.693992+0000","last_undegraded":"2024-02-05T07:38:18.693992+0000","last_fullsized":"2024-02-05T07:38:18.693992+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.778695+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.778695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:52.659561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":723,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.703777+0000","last_change":"2024-02-05T07:38:09.500549+0000","last_active":"2024-02-05T07:38:18.703777+0000","last_peered":"2024-02-05T07:38:18.703777+0000","last_clean":"2024-02-05T07:38:18.703777+0000","last_became_active":"2024-02-05T07:38:09.496845+0000","last_became_peered":"2024-02-05T07:38:09.496845+0000","last_unstale":"2024-02-05T07:38:18.703777+0000","last_undegraded":"2024-02-05T07:38:18.703777+0000","last_fullsized":"2024-02-05T07:38:18.703777+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:44.604094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:44.604094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:02.054747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"326'24","reported_seq":149,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.696637+0000","last_change":"2024-02-05T07:36:32.847697+0000","last_active":"2024-02-05T07:38:18.696637+0000","last_peered":"2024-02-05T07:38:18.696637+0000","last_clean":"2024-02-05T07:38:18.696637+0000","last_became_active":"2024-02-05T07:36:32.753752+0000","last_became_peered":"2024-02-05T07:36:32.753752+0000","last_unstale":"2024-02-05T07:38:18.696637+0000","last_undegraded":"2024-02-05T07:38:18.696637+0000","last_fullsized":"2024-02-05T07:38:18.696637+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:33:49.499846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:49.499846+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.317997+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":992,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.696354+0000","last_change":"2024-02-05T07:35:10.736128+0000","last_active":"2024-02-05T07:38:18.696354+0000","last_peered":"2024-02-05T07:38:18.696354+0000","last_clean":"2024-02-05T07:38:18.696354+0000","last_became_active":"2024-02-05T07:30:40.493770+0000","last_became_peered":"2024-02-05T07:30:40.493770+0000","last_unstale":"2024-02-05T07:38:18.696354+0000","last_undegraded":"2024-02-05T07:38:18.696354+0000","last_fullsized":"2024-02-05T07:38:18.696354+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:35:10.736006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:35:10.736006+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.691410+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":464,"num_read_kb":459,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":421,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.633940+0000","last_change":"2024-02-05T07:38:09.494199+0000","last_active":"2024-02-05T07:38:16.633940+0000","last_peered":"2024-02-05T07:38:16.633940+0000","last_clean":"2024-02-05T07:38:16.633940+0000","last_became_active":"2024-02-05T07:38:09.493827+0000","last_became_peered":"2024-02-05T07:38:09.493827+0000","last_unstale":"2024-02-05T07:38:16.633940+0000","last_undegraded":"2024-02-05T07:38:16.633940+0000","last_fullsized":"2024-02-05T07:38:16.633940+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:39.809544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:33:39.809544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":1145,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.701571+0000","last_change":"2024-02-05T07:37:56.569432+0000","last_active":"2024-02-05T07:38:18.701571+0000","last_peered":"2024-02-05T07:38:18.701571+0000","last_clean":"2024-02-05T07:38:18.701571+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:38:18.701571+0000","last_undegraded":"2024-02-05T07:38:18.701571+0000","last_fullsized":"2024-02-05T07:38:18.701571+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:37:56.569258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:56.569258+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:00.998796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"597'1143","reported_seq":1189,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.996928+0000","last_change":"2024-02-05T07:37:52.817223+0000","last_active":"2024-02-05T07:38:16.996928+0000","last_peered":"2024-02-05T07:38:16.996928+0000","last_clean":"2024-02-05T07:38:16.996928+0000","last_became_active":"2024-02-05T07:37:52.816946+0000","last_became_peered":"2024-02-05T07:37:52.816946+0000","last_unstale":"2024-02-05T07:38:16.996928+0000","last_undegraded":"2024-02-05T07:38:16.996928+0000","last_fullsized":"2024-02-05T07:38:16.996928+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1143,"log_dups_size":0,"ondisk_log_size":1143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:56.888561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74907648,"num_objects":1143,"num_object_clones":0,"num_object_copies":3429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1143,"num_write_kb":73152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":135,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.695841+0000","last_change":"2024-02-05T07:36:32.733109+0000","last_active":"2024-02-05T07:38:18.695841+0000","last_peered":"2024-02-05T07:38:18.695841+0000","last_clean":"2024-02-05T07:38:18.695841+0000","last_became_active":"2024-02-05T07:36:32.732334+0000","last_became_peered":"2024-02-05T07:36:32.732334+0000","last_unstale":"2024-02-05T07:38:18.695841+0000","last_undegraded":"2024-02-05T07:38:18.695841+0000","last_fullsized":"2024-02-05T07:38:18.695841+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:50.943640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:50.943640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.317933+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"589'32","reported_seq":1265,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694699+0000","last_change":"2024-02-05T07:38:09.529174+0000","last_active":"2024-02-05T07:38:18.694699+0000","last_peered":"2024-02-05T07:38:18.694699+0000","last_clean":"2024-02-05T07:38:18.694699+0000","last_became_active":"2024-02-05T07:38:09.479528+0000","last_became_peered":"2024-02-05T07:38:09.479528+0000","last_unstale":"2024-02-05T07:38:18.694699+0000","last_undegraded":"2024-02-05T07:38:18.694699+0000","last_fullsized":"2024-02-05T07:38:18.694699+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'31","last_scrub_stamp":"2024-02-05T07:37:45.644963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.644963+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:04.014316+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":707,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.649314+0000","last_change":"2024-02-05T07:38:09.489766+0000","last_active":"2024-02-05T07:38:16.649314+0000","last_peered":"2024-02-05T07:38:16.649314+0000","last_clean":"2024-02-05T07:38:16.649314+0000","last_became_active":"2024-02-05T07:38:09.488354+0000","last_became_peered":"2024-02-05T07:38:09.488354+0000","last_unstale":"2024-02-05T07:38:16.649314+0000","last_undegraded":"2024-02-05T07:38:16.649314+0000","last_fullsized":"2024-02-05T07:38:16.649314+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:12.093350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:35:12.093350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":875,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693088+0000","last_change":"2024-02-05T07:37:08.472969+0000","last_active":"2024-02-05T07:38:18.693088+0000","last_peered":"2024-02-05T07:38:18.693088+0000","last_clean":"2024-02-05T07:38:18.693088+0000","last_became_active":"2024-02-05T07:37:06.977158+0000","last_became_peered":"2024-02-05T07:37:06.977158+0000","last_unstale":"2024-02-05T07:38:18.693088+0000","last_undegraded":"2024-02-05T07:38:18.693088+0000","last_fullsized":"2024-02-05T07:38:18.693088+0000","mapping_epoch":367,"log_start":"86'128","ondisk_log_start":"86'128","created":20,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:08.472857+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:37:08.472857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:26.970779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":696,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.652177+0000","last_change":"2024-02-05T07:38:09.489635+0000","last_active":"2024-02-05T07:38:16.652177+0000","last_peered":"2024-02-05T07:38:16.652177+0000","last_clean":"2024-02-05T07:38:16.652177+0000","last_became_active":"2024-02-05T07:38:09.488951+0000","last_became_peered":"2024-02-05T07:38:09.488951+0000","last_unstale":"2024-02-05T07:38:16.652177+0000","last_undegraded":"2024-02-05T07:38:16.652177+0000","last_fullsized":"2024-02-05T07:38:16.652177+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:46.760448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:46.760448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"597'1078","reported_seq":1123,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.994174+0000","last_change":"2024-02-05T07:37:53.090431+0000","last_active":"2024-02-05T07:38:16.994174+0000","last_peered":"2024-02-05T07:38:16.994174+0000","last_clean":"2024-02-05T07:38:16.994174+0000","last_became_active":"2024-02-05T07:37:53.090153+0000","last_became_peered":"2024-02-05T07:37:53.090153+0000","last_unstale":"2024-02-05T07:38:16.994174+0000","last_undegraded":"2024-02-05T07:38:16.994174+0000","last_fullsized":"2024-02-05T07:38:16.994174+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:54.731975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70647808,"num_objects":1078,"num_object_clones":0,"num_object_copies":3234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1078,"num_write_kb":68992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":1097,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.691390+0000","last_change":"2024-02-05T07:38:11.675335+0000","last_active":"2024-02-05T07:38:18.691390+0000","last_peered":"2024-02-05T07:38:18.691390+0000","last_clean":"2024-02-05T07:38:18.691390+0000","last_became_active":"2024-02-05T07:38:11.674897+0000","last_became_peered":"2024-02-05T07:38:11.674897+0000","last_unstale":"2024-02-05T07:38:18.691390+0000","last_undegraded":"2024-02-05T07:38:18.691390+0000","last_fullsized":"2024-02-05T07:38:18.691390+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:08.476588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:08.476588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:22.388178+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":141,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.696560+0000","last_change":"2024-02-05T07:38:09.501558+0000","last_active":"2024-02-05T07:38:18.696560+0000","last_peered":"2024-02-05T07:38:18.696560+0000","last_clean":"2024-02-05T07:38:18.696560+0000","last_became_active":"2024-02-05T07:38:09.500885+0000","last_became_peered":"2024-02-05T07:38:09.500885+0000","last_unstale":"2024-02-05T07:38:18.696560+0000","last_undegraded":"2024-02-05T07:38:18.696560+0000","last_fullsized":"2024-02-05T07:38:18.696560+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:23.118569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:23.118569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.317878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"326'22","reported_seq":83,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.695516+0000","last_change":"2024-02-05T07:37:31.677003+0000","last_active":"2024-02-05T07:38:18.695516+0000","last_peered":"2024-02-05T07:38:18.695516+0000","last_clean":"2024-02-05T07:38:18.695516+0000","last_became_active":"2024-02-05T07:37:31.671370+0000","last_became_peered":"2024-02-05T07:37:31.671370+0000","last_unstale":"2024-02-05T07:38:18.695516+0000","last_undegraded":"2024-02-05T07:38:18.695516+0000","last_fullsized":"2024-02-05T07:38:18.695516+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'22","last_scrub_stamp":"2024-02-05T07:33:30.327027+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:33:30.327027+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.317909+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693783+0000","last_change":"2024-02-05T07:36:32.734528+0000","last_active":"2024-02-05T07:38:18.693783+0000","last_peered":"2024-02-05T07:38:18.693783+0000","last_clean":"2024-02-05T07:38:18.693783+0000","last_became_active":"2024-02-05T07:36:32.733822+0000","last_became_peered":"2024-02-05T07:36:32.733822+0000","last_unstale":"2024-02-05T07:38:18.693783+0000","last_undegraded":"2024-02-05T07:38:18.693783+0000","last_fullsized":"2024-02-05T07:38:18.693783+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:42.095953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:33:42.095953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.317949+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":139,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.691058+0000","last_change":"2024-02-05T07:36:32.849750+0000","last_active":"2024-02-05T07:38:18.691058+0000","last_peered":"2024-02-05T07:38:18.691058+0000","last_clean":"2024-02-05T07:38:18.691058+0000","last_became_active":"2024-02-05T07:36:32.781195+0000","last_became_peered":"2024-02-05T07:36:32.781195+0000","last_unstale":"2024-02-05T07:38:18.691058+0000","last_undegraded":"2024-02-05T07:38:18.691058+0000","last_fullsized":"2024-02-05T07:38:18.691058+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:34:26.591385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:26.591385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.318028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1007,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694547+0000","last_change":"2024-02-05T07:36:32.719020+0000","last_active":"2024-02-05T07:38:18.694547+0000","last_peered":"2024-02-05T07:38:18.694547+0000","last_clean":"2024-02-05T07:38:18.694547+0000","last_became_active":"2024-02-05T07:36:32.718684+0000","last_became_peered":"2024-02-05T07:36:32.718684+0000","last_unstale":"2024-02-05T07:38:18.694547+0000","last_undegraded":"2024-02-05T07:38:18.694547+0000","last_fullsized":"2024-02-05T07:38:18.694547+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:18.035057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:35:18.035057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.741845+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"597'1158","reported_seq":1204,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.836406+0000","last_change":"2024-02-05T07:37:53.089611+0000","last_active":"2024-02-05T07:38:16.836406+0000","last_peered":"2024-02-05T07:38:16.836406+0000","last_clean":"2024-02-05T07:38:16.836406+0000","last_became_active":"2024-02-05T07:37:53.089298+0000","last_became_peered":"2024-02-05T07:37:53.089298+0000","last_unstale":"2024-02-05T07:38:16.836406+0000","last_undegraded":"2024-02-05T07:38:16.836406+0000","last_fullsized":"2024-02-05T07:38:16.836406+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1158,"log_dups_size":0,"ondisk_log_size":1158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:11.880709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75890688,"num_objects":1158,"num_object_clones":0,"num_object_copies":3474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1158,"num_write_kb":74112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":492,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693513+0000","last_change":"2024-02-05T07:38:15.643159+0000","last_active":"2024-02-05T07:38:18.693513+0000","last_peered":"2024-02-05T07:38:18.693513+0000","last_clean":"2024-02-05T07:38:18.693513+0000","last_became_active":"2024-02-05T07:38:15.642779+0000","last_became_peered":"2024-02-05T07:38:15.642779+0000","last_unstale":"2024-02-05T07:38:18.693513+0000","last_undegraded":"2024-02-05T07:38:18.693513+0000","last_fullsized":"2024-02-05T07:38:18.693513+0000","mapping_epoch":595,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:08.283492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:08.283492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:29.546083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":878,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.700183+0000","last_change":"2024-02-05T07:38:09.464095+0000","last_active":"2024-02-05T07:38:18.700183+0000","last_peered":"2024-02-05T07:38:18.700183+0000","last_clean":"2024-02-05T07:38:18.700183+0000","last_became_active":"2024-02-05T07:38:09.463763+0000","last_became_peered":"2024-02-05T07:38:09.463763+0000","last_unstale":"2024-02-05T07:38:18.700183+0000","last_undegraded":"2024-02-05T07:38:18.700183+0000","last_fullsized":"2024-02-05T07:38:18.700183+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:58.567980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:58.567980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:25.435230+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"550'56","reported_seq":197,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694856+0000","last_change":"2024-02-05T07:36:56.264458+0000","last_active":"2024-02-05T07:38:18.694856+0000","last_peered":"2024-02-05T07:38:18.694856+0000","last_clean":"2024-02-05T07:38:18.694856+0000","last_became_active":"2024-02-05T07:36:32.738270+0000","last_became_peered":"2024-02-05T07:36:32.738270+0000","last_unstale":"2024-02-05T07:38:18.694856+0000","last_undegraded":"2024-02-05T07:38:18.694856+0000","last_fullsized":"2024-02-05T07:38:18.694856+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'56","last_scrub_stamp":"2024-02-05T07:36:56.264300+0000","last_deep_scrub":"550'56","last_deep_scrub_stamp":"2024-02-05T07:36:56.264300+0000","last_clean_scrub_stamp":"2024-02-05T07:36:56.264300+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.318220+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":536,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694104+0000","last_change":"2024-02-05T07:38:09.484740+0000","last_active":"2024-02-05T07:38:18.694104+0000","last_peered":"2024-02-05T07:38:18.694104+0000","last_clean":"2024-02-05T07:38:18.694104+0000","last_became_active":"2024-02-05T07:38:09.484410+0000","last_became_peered":"2024-02-05T07:38:09.484410+0000","last_unstale":"2024-02-05T07:38:18.694104+0000","last_undegraded":"2024-02-05T07:38:18.694104+0000","last_fullsized":"2024-02-05T07:38:18.694104+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:46.455867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.455867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.742241+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":1042,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.698552+0000","last_change":"2024-02-05T07:37:57.550023+0000","last_active":"2024-02-05T07:38:18.698552+0000","last_peered":"2024-02-05T07:38:18.698552+0000","last_clean":"2024-02-05T07:38:18.698552+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:38:18.698552+0000","last_undegraded":"2024-02-05T07:38:18.698552+0000","last_fullsized":"2024-02-05T07:38:18.698552+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:37:57.549938+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:37:57.549938+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:21.836629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":710,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.651767+0000","last_change":"2024-02-05T07:38:09.487179+0000","last_active":"2024-02-05T07:38:16.651767+0000","last_peered":"2024-02-05T07:38:16.651767+0000","last_clean":"2024-02-05T07:38:16.651767+0000","last_became_active":"2024-02-05T07:38:09.486846+0000","last_became_peered":"2024-02-05T07:38:09.486846+0000","last_unstale":"2024-02-05T07:38:16.651767+0000","last_undegraded":"2024-02-05T07:38:16.651767+0000","last_fullsized":"2024-02-05T07:38:16.651767+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:26.897028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:26.897028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"598'1283","reported_seq":1331,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.808472+0000","last_change":"2024-02-05T07:37:52.813698+0000","last_active":"2024-02-05T07:38:18.808472+0000","last_peered":"2024-02-05T07:38:18.808472+0000","last_clean":"2024-02-05T07:38:18.808472+0000","last_became_active":"2024-02-05T07:37:52.813405+0000","last_became_peered":"2024-02-05T07:37:52.813405+0000","last_unstale":"2024-02-05T07:38:18.808472+0000","last_undegraded":"2024-02-05T07:38:18.808472+0000","last_fullsized":"2024-02-05T07:38:18.808472+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1283,"log_dups_size":0,"ondisk_log_size":1283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:17.246028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84082688,"num_objects":1283,"num_object_clones":0,"num_object_copies":3849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1283,"num_write_kb":82112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":419,"reported_epoch":597,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:38:16.653444+0000","last_change":"2024-02-05T07:38:09.982478+0000","last_active":"2024-02-05T07:38:16.653444+0000","last_peered":"2024-02-05T07:38:16.653444+0000","last_clean":"2024-02-05T07:28:23.750841+0000","last_became_active":"2024-02-05T07:38:09.977818+0000","last_became_peered":"2024-02-05T07:38:09.977818+0000","last_unstale":"2024-02-05T07:38:16.653444+0000","last_undegraded":"2024-02-05T07:38:16.653444+0000","last_fullsized":"2024-02-05T07:38:16.653444+0000","mapping_epoch":589,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:40.192857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:40.192857+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10063,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":30189,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,4],"avail_no_missing":["3","1","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"326'15","reported_seq":923,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.700432+0000","last_change":"2024-02-05T07:38:09.509126+0000","last_active":"2024-02-05T07:38:18.700432+0000","last_peered":"2024-02-05T07:38:18.700432+0000","last_clean":"2024-02-05T07:38:18.700432+0000","last_became_active":"2024-02-05T07:38:09.479890+0000","last_became_peered":"2024-02-05T07:38:09.479890+0000","last_unstale":"2024-02-05T07:38:18.700432+0000","last_undegraded":"2024-02-05T07:38:18.700432+0000","last_fullsized":"2024-02-05T07:38:18.700432+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:35:12.776983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:12.776983+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.691477+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":700,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.649800+0000","last_change":"2024-02-05T07:36:54.146303+0000","last_active":"2024-02-05T07:38:16.649800+0000","last_peered":"2024-02-05T07:38:16.649800+0000","last_clean":"2024-02-05T07:38:16.649800+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:38:16.649800+0000","last_undegraded":"2024-02-05T07:38:16.649800+0000","last_fullsized":"2024-02-05T07:38:16.649800+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:54.146194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:54.146194+0000","last_clean_scrub_stamp":"2024-02-05T07:36:54.146194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:23.763260+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":149,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.691178+0000","last_change":"2024-02-05T07:38:09.492094+0000","last_active":"2024-02-05T07:38:18.691178+0000","last_peered":"2024-02-05T07:38:18.691178+0000","last_clean":"2024-02-05T07:38:18.691178+0000","last_became_active":"2024-02-05T07:38:09.491746+0000","last_became_peered":"2024-02-05T07:38:09.491746+0000","last_unstale":"2024-02-05T07:38:18.691178+0000","last_undegraded":"2024-02-05T07:38:18.691178+0000","last_fullsized":"2024-02-05T07:38:18.691178+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:45.362685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.362685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:52.297763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":143,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693409+0000","last_change":"2024-02-05T07:37:00.310446+0000","last_active":"2024-02-05T07:38:18.693409+0000","last_peered":"2024-02-05T07:38:18.693409+0000","last_clean":"2024-02-05T07:38:18.693409+0000","last_became_active":"2024-02-05T07:36:32.734230+0000","last_became_peered":"2024-02-05T07:36:32.734230+0000","last_unstale":"2024-02-05T07:38:18.693409+0000","last_undegraded":"2024-02-05T07:38:18.693409+0000","last_fullsized":"2024-02-05T07:38:18.693409+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:00.309850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:00.309850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:19.733704+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"598'1230","reported_seq":1278,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.803797+0000","last_change":"2024-02-05T07:37:53.089683+0000","last_active":"2024-02-05T07:38:18.803797+0000","last_peered":"2024-02-05T07:38:18.803797+0000","last_clean":"2024-02-05T07:38:18.803797+0000","last_became_active":"2024-02-05T07:37:53.089386+0000","last_became_peered":"2024-02-05T07:37:53.089386+0000","last_unstale":"2024-02-05T07:38:18.803797+0000","last_undegraded":"2024-02-05T07:38:18.803797+0000","last_fullsized":"2024-02-05T07:38:18.803797+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1230,"log_dups_size":0,"ondisk_log_size":1230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:16.725367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80609280,"num_objects":1230,"num_object_clones":0,"num_object_copies":3690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1230,"num_write_kb":78720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":1105,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694619+0000","last_change":"2024-02-05T07:38:11.660956+0000","last_active":"2024-02-05T07:38:18.694619+0000","last_peered":"2024-02-05T07:38:18.694619+0000","last_clean":"2024-02-05T07:38:18.694619+0000","last_became_active":"2024-02-05T07:38:11.660622+0000","last_became_peered":"2024-02-05T07:38:11.660622+0000","last_unstale":"2024-02-05T07:38:18.694619+0000","last_undegraded":"2024-02-05T07:38:18.694619+0000","last_fullsized":"2024-02-05T07:38:18.694619+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.730473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.730473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:04.402857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"326'35","reported_seq":1059,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693792+0000","last_change":"2024-02-05T07:36:16.812273+0000","last_active":"2024-02-05T07:38:18.693792+0000","last_peered":"2024-02-05T07:38:18.693792+0000","last_clean":"2024-02-05T07:38:18.693792+0000","last_became_active":"2024-02-05T07:36:16.736226+0000","last_became_peered":"2024-02-05T07:36:16.736226+0000","last_unstale":"2024-02-05T07:38:18.693792+0000","last_undegraded":"2024-02-05T07:38:18.693792+0000","last_fullsized":"2024-02-05T07:38:18.693792+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:35:46.529276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:46.529276+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.741895+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1015,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694609+0000","last_change":"2024-02-05T07:38:09.480281+0000","last_active":"2024-02-05T07:38:18.694609+0000","last_peered":"2024-02-05T07:38:18.694609+0000","last_clean":"2024-02-05T07:38:18.694609+0000","last_became_active":"2024-02-05T07:38:09.479410+0000","last_became_peered":"2024-02-05T07:38:09.479410+0000","last_unstale":"2024-02-05T07:38:18.694609+0000","last_undegraded":"2024-02-05T07:38:18.694609+0000","last_fullsized":"2024-02-05T07:38:18.694609+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:15.914445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_clean_scrub_stamp":"2024-02-05T07:35:15.914445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.741810+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":143,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.692310+0000","last_change":"2024-02-05T07:37:54.312733+0000","last_active":"2024-02-05T07:38:18.692310+0000","last_peered":"2024-02-05T07:38:18.692310+0000","last_clean":"2024-02-05T07:38:18.692310+0000","last_became_active":"2024-02-05T07:36:32.733494+0000","last_became_peered":"2024-02-05T07:36:32.733494+0000","last_unstale":"2024-02-05T07:38:18.692310+0000","last_undegraded":"2024-02-05T07:38:18.692310+0000","last_fullsized":"2024-02-05T07:38:18.692310+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:54.312642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.312642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:04.803941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":143,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693312+0000","last_change":"2024-02-05T07:38:01.478633+0000","last_active":"2024-02-05T07:38:18.693312+0000","last_peered":"2024-02-05T07:38:18.693312+0000","last_clean":"2024-02-05T07:38:18.693312+0000","last_became_active":"2024-02-05T07:36:32.715956+0000","last_became_peered":"2024-02-05T07:36:32.715956+0000","last_unstale":"2024-02-05T07:38:18.693312+0000","last_undegraded":"2024-02-05T07:38:18.693312+0000","last_fullsized":"2024-02-05T07:38:18.693312+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:01.478555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:38:01.478555+0000","last_clean_scrub_stamp":"2024-02-05T07:38:01.478555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:11.378011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.6","version":"598'1458","reported_seq":1506,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:20.501841+0000","last_change":"2024-02-05T07:37:53.090459+0000","last_active":"2024-02-05T07:38:20.501841+0000","last_peered":"2024-02-05T07:38:20.501841+0000","last_clean":"2024-02-05T07:38:20.501841+0000","last_became_active":"2024-02-05T07:37:53.090173+0000","last_became_peered":"2024-02-05T07:37:53.090173+0000","last_unstale":"2024-02-05T07:38:20.501841+0000","last_undegraded":"2024-02-05T07:38:20.501841+0000","last_fullsized":"2024-02-05T07:38:20.501841+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1458,"log_dups_size":0,"ondisk_log_size":1458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:16.230043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95551488,"num_objects":1458,"num_object_clones":0,"num_object_copies":4374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1458,"num_write_kb":93312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":1460,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.648374+0000","last_change":"2024-02-05T07:37:39.295414+0000","last_active":"2024-02-05T07:38:16.648374+0000","last_peered":"2024-02-05T07:38:16.648374+0000","last_clean":"2024-02-05T07:38:16.648374+0000","last_became_active":"2024-02-05T07:37:36.695548+0000","last_became_peered":"2024-02-05T07:37:36.695548+0000","last_unstale":"2024-02-05T07:38:16.648374+0000","last_undegraded":"2024-02-05T07:38:16.648374+0000","last_fullsized":"2024-02-05T07:38:16.648374+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.295329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.295329+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:45.596598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":143,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.691248+0000","last_change":"2024-02-05T07:37:51.288853+0000","last_active":"2024-02-05T07:38:18.691248+0000","last_peered":"2024-02-05T07:38:18.691248+0000","last_clean":"2024-02-05T07:38:18.691248+0000","last_became_active":"2024-02-05T07:36:32.733003+0000","last_became_peered":"2024-02-05T07:36:32.733003+0000","last_unstale":"2024-02-05T07:38:18.691248+0000","last_undegraded":"2024-02-05T07:38:18.691248+0000","last_fullsized":"2024-02-05T07:38:18.691248+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.288772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.288772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:03.240106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"325'18","reported_seq":1097,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.697628+0000","last_change":"2024-02-05T07:38:09.513467+0000","last_active":"2024-02-05T07:38:18.697628+0000","last_peered":"2024-02-05T07:38:18.697628+0000","last_clean":"2024-02-05T07:38:18.697628+0000","last_became_active":"2024-02-05T07:38:09.480893+0000","last_became_peered":"2024-02-05T07:38:09.480893+0000","last_unstale":"2024-02-05T07:38:18.697628+0000","last_undegraded":"2024-02-05T07:38:18.697628+0000","last_fullsized":"2024-02-05T07:38:18.697628+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'18","last_scrub_stamp":"2024-02-05T07:34:28.902809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.902809+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.691372+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.700280+0000","last_change":"2024-02-05T07:38:09.486267+0000","last_active":"2024-02-05T07:38:18.700280+0000","last_peered":"2024-02-05T07:38:18.700280+0000","last_clean":"2024-02-05T07:38:18.700280+0000","last_became_active":"2024-02-05T07:38:09.485926+0000","last_became_peered":"2024-02-05T07:38:09.485926+0000","last_unstale":"2024-02-05T07:38:18.700280+0000","last_undegraded":"2024-02-05T07:38:18.700280+0000","last_fullsized":"2024-02-05T07:38:18.700280+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:53.092642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:37:53.092642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:22.363132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":703,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.651466+0000","last_change":"2024-02-05T07:34:38.057821+0000","last_active":"2024-02-05T07:38:16.651466+0000","last_peered":"2024-02-05T07:38:16.651466+0000","last_clean":"2024-02-05T07:38:16.651466+0000","last_became_active":"2024-02-05T07:30:40.485991+0000","last_became_peered":"2024-02-05T07:30:40.485991+0000","last_unstale":"2024-02-05T07:38:16.651466+0000","last_undegraded":"2024-02-05T07:38:16.651466+0000","last_fullsized":"2024-02-05T07:38:16.651466+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:38.057598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:38.057598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"598'1394","reported_seq":2524,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:20.107439+0000","last_change":"2024-02-05T07:38:15.669115+0000","last_active":"2024-02-05T07:38:20.107439+0000","last_peered":"2024-02-05T07:38:20.107439+0000","last_clean":"2024-02-05T07:38:20.107439+0000","last_became_active":"2024-02-05T07:38:15.650851+0000","last_became_peered":"2024-02-05T07:38:15.650851+0000","last_unstale":"2024-02-05T07:38:20.107439+0000","last_undegraded":"2024-02-05T07:38:20.107439+0000","last_fullsized":"2024-02-05T07:38:20.107439+0000","mapping_epoch":595,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":1394,"log_dups_size":0,"ondisk_log_size":1394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:18.108415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91357184,"num_objects":1394,"num_object_clones":0,"num_object_copies":4182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1394,"num_write_kb":89216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":536,"num_bytes_recovered":35127296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":1084,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.692602+0000","last_change":"2024-02-05T07:38:15.651199+0000","last_active":"2024-02-05T07:38:18.692602+0000","last_peered":"2024-02-05T07:38:18.692602+0000","last_clean":"2024-02-05T07:38:18.692602+0000","last_became_active":"2024-02-05T07:38:15.650809+0000","last_became_peered":"2024-02-05T07:38:15.650809+0000","last_unstale":"2024-02-05T07:38:18.692602+0000","last_undegraded":"2024-02-05T07:38:18.692602+0000","last_fullsized":"2024-02-05T07:38:18.692602+0000","mapping_epoch":595,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.180648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.180648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.352444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"589'13","reported_seq":1228,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693175+0000","last_change":"2024-02-05T07:38:09.523382+0000","last_active":"2024-02-05T07:38:18.693175+0000","last_peered":"2024-02-05T07:38:18.693175+0000","last_clean":"2024-02-05T07:38:18.693175+0000","last_became_active":"2024-02-05T07:38:09.471365+0000","last_became_peered":"2024-02-05T07:38:09.471365+0000","last_unstale":"2024-02-05T07:38:18.693175+0000","last_undegraded":"2024-02-05T07:38:18.693175+0000","last_fullsized":"2024-02-05T07:38:18.693175+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'12","last_scrub_stamp":"2024-02-05T07:37:53.638258+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:37:53.638258+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:02.590707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"326'27","reported_seq":783,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.692940+0000","last_change":"2024-02-05T07:38:09.513629+0000","last_active":"2024-02-05T07:38:18.692940+0000","last_peered":"2024-02-05T07:38:18.692940+0000","last_clean":"2024-02-05T07:38:18.692940+0000","last_became_active":"2024-02-05T07:38:09.471133+0000","last_became_peered":"2024-02-05T07:38:09.471133+0000","last_unstale":"2024-02-05T07:38:18.692940+0000","last_undegraded":"2024-02-05T07:38:18.692940+0000","last_fullsized":"2024-02-05T07:38:18.692940+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:38:00.687570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:00.687570+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:19.607140+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":48,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.703104+0000","last_change":"2024-02-05T07:38:09.518512+0000","last_active":"2024-02-05T07:38:18.703104+0000","last_peered":"2024-02-05T07:38:18.703104+0000","last_clean":"2024-02-05T07:38:18.703104+0000","last_became_active":"2024-02-05T07:38:09.482757+0000","last_became_peered":"2024-02-05T07:38:09.482757+0000","last_unstale":"2024-02-05T07:38:18.703104+0000","last_undegraded":"2024-02-05T07:38:18.703104+0000","last_fullsized":"2024-02-05T07:38:18.703104+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-02-05T07:37:43.210581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:37:43.210581+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:53.036534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":800,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693593+0000","last_change":"2024-02-05T07:37:02.526520+0000","last_active":"2024-02-05T07:38:18.693593+0000","last_peered":"2024-02-05T07:38:18.693593+0000","last_clean":"2024-02-05T07:38:18.693593+0000","last_became_active":"2024-02-05T07:36:32.701535+0000","last_became_peered":"2024-02-05T07:36:32.701535+0000","last_unstale":"2024-02-05T07:38:18.693593+0000","last_undegraded":"2024-02-05T07:38:18.693593+0000","last_fullsized":"2024-02-05T07:38:18.693593+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:02.526362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:37:02.526362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.742274+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":415,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.696498+0000","last_change":"2024-02-05T07:37:42.592248+0000","last_active":"2024-02-05T07:38:18.696498+0000","last_peered":"2024-02-05T07:38:18.696498+0000","last_clean":"2024-02-05T07:38:18.696498+0000","last_became_active":"2024-02-05T07:37:41.643328+0000","last_became_peered":"2024-02-05T07:37:41.643328+0000","last_unstale":"2024-02-05T07:38:18.696498+0000","last_undegraded":"2024-02-05T07:38:18.696498+0000","last_fullsized":"2024-02-05T07:38:18.696498+0000","mapping_epoch":576,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:42.592127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:42.592127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:10.114471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"326'12","reported_seq":748,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.652123+0000","last_change":"2024-02-05T07:36:32.779445+0000","last_active":"2024-02-05T07:38:16.652123+0000","last_peered":"2024-02-05T07:38:16.652123+0000","last_clean":"2024-02-05T07:38:16.652123+0000","last_became_active":"2024-02-05T07:36:32.742483+0000","last_became_peered":"2024-02-05T07:36:32.742483+0000","last_unstale":"2024-02-05T07:38:16.652123+0000","last_undegraded":"2024-02-05T07:38:16.652123+0000","last_fullsized":"2024-02-05T07:38:16.652123+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'12","last_scrub_stamp":"2024-02-05T07:33:41.762404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:41.762404+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":932,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693228+0000","last_change":"2024-02-05T07:37:47.639730+0000","last_active":"2024-02-05T07:38:18.693228+0000","last_peered":"2024-02-05T07:38:18.693228+0000","last_clean":"2024-02-05T07:38:18.693228+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:38:18.693228+0000","last_undegraded":"2024-02-05T07:38:18.693228+0000","last_fullsized":"2024-02-05T07:38:18.693228+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:37:47.639652+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:37:47.639652+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:53.761039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":700,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.652258+0000","last_change":"2024-02-05T07:37:01.178948+0000","last_active":"2024-02-05T07:38:16.652258+0000","last_peered":"2024-02-05T07:38:16.652258+0000","last_clean":"2024-02-05T07:38:16.652258+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:38:16.652258+0000","last_undegraded":"2024-02-05T07:38:16.652258+0000","last_fullsized":"2024-02-05T07:38:16.652258+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:01.178816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:01.178816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":988,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694413+0000","last_change":"2024-02-05T07:38:09.480543+0000","last_active":"2024-02-05T07:38:18.694413+0000","last_peered":"2024-02-05T07:38:18.694413+0000","last_clean":"2024-02-05T07:38:18.694413+0000","last_became_active":"2024-02-05T07:38:09.480228+0000","last_became_peered":"2024-02-05T07:38:09.480228+0000","last_unstale":"2024-02-05T07:38:18.694413+0000","last_undegraded":"2024-02-05T07:38:18.694413+0000","last_fullsized":"2024-02-05T07:38:18.694413+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:13.118311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:35:13.118311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.741675+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":901,"reported_epoch":597,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:38:16.649331+0000","last_change":"2024-02-05T07:38:09.761875+0000","last_active":"2024-02-05T07:38:16.649331+0000","last_peered":"2024-02-05T07:38:16.649331+0000","last_clean":"2024-02-05T07:38:05.051036+0000","last_became_active":"2024-02-05T07:38:09.753712+0000","last_became_peered":"2024-02-05T07:38:09.753712+0000","last_unstale":"2024-02-05T07:38:16.649331+0000","last_undegraded":"2024-02-05T07:38:16.649331+0000","last_fullsized":"2024-02-05T07:38:16.649331+0000","mapping_epoch":589,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.260984+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.260984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,4],"avail_no_missing":["3","0","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"325'15","reported_seq":756,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.651871+0000","last_change":"2024-02-05T07:36:32.807878+0000","last_active":"2024-02-05T07:38:16.651871+0000","last_peered":"2024-02-05T07:38:16.651871+0000","last_clean":"2024-02-05T07:38:16.651871+0000","last_became_active":"2024-02-05T07:36:32.747221+0000","last_became_peered":"2024-02-05T07:36:32.747221+0000","last_unstale":"2024-02-05T07:38:16.651871+0000","last_undegraded":"2024-02-05T07:38:16.651871+0000","last_fullsized":"2024-02-05T07:38:16.651871+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:33:45.711399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:33:45.711399+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":539,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693239+0000","last_change":"2024-02-05T07:38:09.479485+0000","last_active":"2024-02-05T07:38:18.693239+0000","last_peered":"2024-02-05T07:38:18.693239+0000","last_clean":"2024-02-05T07:38:18.693239+0000","last_became_active":"2024-02-05T07:38:09.479116+0000","last_became_peered":"2024-02-05T07:38:09.479116+0000","last_unstale":"2024-02-05T07:38:18.693239+0000","last_undegraded":"2024-02-05T07:38:18.693239+0000","last_fullsized":"2024-02-05T07:38:18.693239+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:45.476243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:36:45.476243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.742019+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":695,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.651312+0000","last_change":"2024-02-05T07:33:50.723381+0000","last_active":"2024-02-05T07:38:16.651312+0000","last_peered":"2024-02-05T07:38:16.651312+0000","last_clean":"2024-02-05T07:38:16.651312+0000","last_became_active":"2024-02-05T07:30:40.496076+0000","last_became_peered":"2024-02-05T07:30:40.496076+0000","last_unstale":"2024-02-05T07:38:16.651312+0000","last_undegraded":"2024-02-05T07:38:16.651312+0000","last_fullsized":"2024-02-05T07:38:16.651312+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_clean_scrub_stamp":"2024-02-05T07:33:50.723212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":149,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.695962+0000","last_change":"2024-02-05T07:38:09.489174+0000","last_active":"2024-02-05T07:38:18.695962+0000","last_peered":"2024-02-05T07:38:18.695962+0000","last_clean":"2024-02-05T07:38:18.695962+0000","last_became_active":"2024-02-05T07:38:09.488755+0000","last_became_peered":"2024-02-05T07:38:09.488755+0000","last_unstale":"2024-02-05T07:38:18.695962+0000","last_undegraded":"2024-02-05T07:38:18.695962+0000","last_fullsized":"2024-02-05T07:38:18.695962+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:47.330693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:37:47.330693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:08.363155+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":896,"reported_epoch":598,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:38:18.692535+0000","last_change":"2024-02-05T07:38:09.758537+0000","last_active":"2024-02-05T07:38:18.692535+0000","last_peered":"2024-02-05T07:38:18.692535+0000","last_clean":"2024-02-05T07:38:05.043923+0000","last_became_active":"2024-02-05T07:38:09.756415+0000","last_became_peered":"2024-02-05T07:38:09.756415+0000","last_unstale":"2024-02-05T07:38:18.692535+0000","last_undegraded":"2024-02-05T07:38:18.692535+0000","last_fullsized":"2024-02-05T07:38:18.692535+0000","mapping_epoch":589,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:37.743034+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:37:37.743034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,4],"avail_no_missing":["6","0","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"326'19","reported_seq":1052,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.651458+0000","last_change":"2024-02-05T07:37:06.319292+0000","last_active":"2024-02-05T07:38:16.651458+0000","last_peered":"2024-02-05T07:38:16.651458+0000","last_clean":"2024-02-05T07:38:16.651458+0000","last_became_active":"2024-02-05T07:30:40.483700+0000","last_became_peered":"2024-02-05T07:30:40.483700+0000","last_unstale":"2024-02-05T07:38:16.651458+0000","last_undegraded":"2024-02-05T07:38:16.651458+0000","last_fullsized":"2024-02-05T07:38:16.651458+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:37:06.318619+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:37:06.318619+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:27.781398+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":866,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.634079+0000","last_change":"2024-02-05T07:36:46.059245+0000","last_active":"2024-02-05T07:38:16.634079+0000","last_peered":"2024-02-05T07:38:16.634079+0000","last_clean":"2024-02-05T07:38:16.634079+0000","last_became_active":"2024-02-05T07:30:40.476058+0000","last_became_peered":"2024-02-05T07:30:40.476058+0000","last_unstale":"2024-02-05T07:38:16.634079+0000","last_undegraded":"2024-02-05T07:38:16.634079+0000","last_fullsized":"2024-02-05T07:38:16.634079+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:46.059129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.059129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"538'13","reported_seq":855,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.648941+0000","last_change":"2024-02-05T07:36:32.782551+0000","last_active":"2024-02-05T07:38:16.648941+0000","last_peered":"2024-02-05T07:38:16.648941+0000","last_clean":"2024-02-05T07:38:16.648941+0000","last_became_active":"2024-02-05T07:36:32.746270+0000","last_became_peered":"2024-02-05T07:36:32.746270+0000","last_unstale":"2024-02-05T07:38:16.648941+0000","last_undegraded":"2024-02-05T07:38:16.648941+0000","last_fullsized":"2024-02-05T07:38:16.648941+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'11","last_scrub_stamp":"2024-02-05T07:33:39.734129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:33:39.734129+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1012,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.695831+0000","last_change":"2024-02-05T07:38:09.504184+0000","last_active":"2024-02-05T07:38:18.695831+0000","last_peered":"2024-02-05T07:38:18.695831+0000","last_clean":"2024-02-05T07:38:18.695831+0000","last_became_active":"2024-02-05T07:38:09.503255+0000","last_became_peered":"2024-02-05T07:38:09.503255+0000","last_unstale":"2024-02-05T07:38:18.695831+0000","last_undegraded":"2024-02-05T07:38:18.695831+0000","last_fullsized":"2024-02-05T07:38:18.695831+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:58.277028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:58.277028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:27.690676+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":469,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.653698+0000","last_change":"2024-02-05T07:37:40.289255+0000","last_active":"2024-02-05T07:38:16.653698+0000","last_peered":"2024-02-05T07:38:16.653698+0000","last_clean":"2024-02-05T07:38:16.653698+0000","last_became_active":"2024-02-05T07:37:12.040170+0000","last_became_peered":"2024-02-05T07:37:12.040170+0000","last_unstale":"2024-02-05T07:38:16.653698+0000","last_undegraded":"2024-02-05T07:38:16.653698+0000","last_fullsized":"2024-02-05T07:38:16.653698+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:40.289174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:40.289174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:47.896509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"326'25","reported_seq":778,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.649687+0000","last_change":"2024-02-05T07:36:59.117388+0000","last_active":"2024-02-05T07:38:16.649687+0000","last_peered":"2024-02-05T07:38:16.649687+0000","last_clean":"2024-02-05T07:38:16.649687+0000","last_became_active":"2024-02-05T07:36:32.746757+0000","last_became_peered":"2024-02-05T07:36:32.746757+0000","last_unstale":"2024-02-05T07:38:16.649687+0000","last_undegraded":"2024-02-05T07:38:16.649687+0000","last_fullsized":"2024-02-05T07:38:16.649687+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:36:59.117218+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.117218+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":397,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.692105+0000","last_change":"2024-02-05T07:38:09.495374+0000","last_active":"2024-02-05T07:38:18.692105+0000","last_peered":"2024-02-05T07:38:18.692105+0000","last_clean":"2024-02-05T07:38:18.692105+0000","last_became_active":"2024-02-05T07:38:09.495040+0000","last_became_peered":"2024-02-05T07:38:09.495040+0000","last_unstale":"2024-02-05T07:38:18.692105+0000","last_undegraded":"2024-02-05T07:38:18.692105+0000","last_fullsized":"2024-02-05T07:38:18.692105+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:32.854871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:34:32.854871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.318079+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"584'12","reported_seq":45,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.649315+0000","last_change":"2024-02-05T07:37:58.783503+0000","last_active":"2024-02-05T07:38:16.649315+0000","last_peered":"2024-02-05T07:38:16.649315+0000","last_clean":"2024-02-05T07:38:16.649315+0000","last_became_active":"2024-02-05T07:37:58.781263+0000","last_became_peered":"2024-02-05T07:37:58.781263+0000","last_unstale":"2024-02-05T07:38:16.649315+0000","last_undegraded":"2024-02-05T07:38:16.649315+0000","last_fullsized":"2024-02-05T07:38:16.649315+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9","last_scrub_stamp":"2024-02-05T07:34:27.946504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:34:27.946504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":860,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.652015+0000","last_change":"2024-02-05T07:38:09.488116+0000","last_active":"2024-02-05T07:38:16.652015+0000","last_peered":"2024-02-05T07:38:16.652015+0000","last_clean":"2024-02-05T07:38:16.652015+0000","last_became_active":"2024-02-05T07:38:09.487782+0000","last_became_peered":"2024-02-05T07:38:09.487782+0000","last_unstale":"2024-02-05T07:38:16.652015+0000","last_undegraded":"2024-02-05T07:38:16.652015+0000","last_fullsized":"2024-02-05T07:38:16.652015+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:22.810759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:22.810759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.16","version":"86'128","reported_seq":382,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.699634+0000","last_change":"2024-02-05T07:37:43.586635+0000","last_active":"2024-02-05T07:38:18.699634+0000","last_peered":"2024-02-05T07:38:18.699634+0000","last_clean":"2024-02-05T07:38:18.699634+0000","last_became_active":"2024-02-05T07:37:41.647262+0000","last_became_peered":"2024-02-05T07:37:41.647262+0000","last_unstale":"2024-02-05T07:38:18.699634+0000","last_undegraded":"2024-02-05T07:38:18.699634+0000","last_fullsized":"2024-02-05T07:38:18.699634+0000","mapping_epoch":576,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:43.586550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:43.586550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:00.556310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"326'19","reported_seq":648,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693626+0000","last_change":"2024-02-05T07:38:09.532591+0000","last_active":"2024-02-05T07:38:18.693626+0000","last_peered":"2024-02-05T07:38:18.693626+0000","last_clean":"2024-02-05T07:38:18.693626+0000","last_became_active":"2024-02-05T07:38:09.484252+0000","last_became_peered":"2024-02-05T07:38:09.484252+0000","last_unstale":"2024-02-05T07:38:18.693626+0000","last_undegraded":"2024-02-05T07:38:18.693626+0000","last_fullsized":"2024-02-05T07:38:18.693626+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:37:48.219951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:48.219951+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:51.707593+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1033,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694250+0000","last_change":"2024-02-05T07:38:09.472344+0000","last_active":"2024-02-05T07:38:18.694250+0000","last_peered":"2024-02-05T07:38:18.694250+0000","last_clean":"2024-02-05T07:38:18.694250+0000","last_became_active":"2024-02-05T07:38:09.472022+0000","last_became_peered":"2024-02-05T07:38:09.472022+0000","last_unstale":"2024-02-05T07:38:18.694250+0000","last_undegraded":"2024-02-05T07:38:18.694250+0000","last_fullsized":"2024-02-05T07:38:18.694250+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:31.892895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:34:31.892895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.741511+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1133,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.698864+0000","last_change":"2024-02-05T07:37:51.700650+0000","last_active":"2024-02-05T07:38:18.698864+0000","last_peered":"2024-02-05T07:38:18.698864+0000","last_clean":"2024-02-05T07:38:18.698864+0000","last_became_active":"2024-02-05T07:30:40.487141+0000","last_became_peered":"2024-02-05T07:30:40.487141+0000","last_unstale":"2024-02-05T07:38:18.698864+0000","last_undegraded":"2024-02-05T07:38:18.698864+0000","last_fullsized":"2024-02-05T07:38:18.698864+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.700545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.700545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:13.517250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":707,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.634038+0000","last_change":"2024-02-05T07:37:54.469820+0000","last_active":"2024-02-05T07:38:16.634038+0000","last_peered":"2024-02-05T07:38:16.634038+0000","last_clean":"2024-02-05T07:38:16.634038+0000","last_became_active":"2024-02-05T07:36:32.742150+0000","last_became_peered":"2024-02-05T07:36:32.742150+0000","last_unstale":"2024-02-05T07:38:16.634038+0000","last_undegraded":"2024-02-05T07:38:16.634038+0000","last_fullsized":"2024-02-05T07:38:16.634038+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:54.469701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.469701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:15.409936+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"86'128","reported_seq":1091,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694652+0000","last_change":"2024-02-05T07:38:11.665661+0000","last_active":"2024-02-05T07:38:18.694652+0000","last_peered":"2024-02-05T07:38:18.694652+0000","last_clean":"2024-02-05T07:38:18.694652+0000","last_became_active":"2024-02-05T07:38:11.665385+0000","last_became_peered":"2024-02-05T07:38:11.665385+0000","last_unstale":"2024-02-05T07:38:18.694652+0000","last_undegraded":"2024-02-05T07:38:18.694652+0000","last_fullsized":"2024-02-05T07:38:18.694652+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:34:29.214474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:34:29.214474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:22.758440+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"325'15","reported_seq":770,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.696603+0000","last_change":"2024-02-05T07:36:32.815601+0000","last_active":"2024-02-05T07:38:18.696603+0000","last_peered":"2024-02-05T07:38:18.696603+0000","last_clean":"2024-02-05T07:38:18.696603+0000","last_became_active":"2024-02-05T07:36:32.710090+0000","last_became_peered":"2024-02-05T07:36:32.710090+0000","last_unstale":"2024-02-05T07:38:18.696603+0000","last_undegraded":"2024-02-05T07:38:18.696603+0000","last_fullsized":"2024-02-05T07:38:18.696603+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:35:11.749389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:11.749389+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.691511+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":917,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.649948+0000","last_change":"2024-02-05T07:38:09.473515+0000","last_active":"2024-02-05T07:38:16.649948+0000","last_peered":"2024-02-05T07:38:16.649948+0000","last_clean":"2024-02-05T07:38:16.649948+0000","last_became_active":"2024-02-05T07:38:09.472668+0000","last_became_peered":"2024-02-05T07:38:09.472668+0000","last_unstale":"2024-02-05T07:38:16.649948+0000","last_undegraded":"2024-02-05T07:38:16.649948+0000","last_fullsized":"2024-02-05T07:38:16.649948+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:32.120410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:32.120410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":689,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.651871+0000","last_change":"2024-02-05T07:33:47.684589+0000","last_active":"2024-02-05T07:38:16.651871+0000","last_peered":"2024-02-05T07:38:16.651871+0000","last_clean":"2024-02-05T07:38:16.651871+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:38:16.651871+0000","last_undegraded":"2024-02-05T07:38:16.651871+0000","last_fullsized":"2024-02-05T07:38:16.651871+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:47.684508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.684508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":577,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.692647+0000","last_change":"2024-02-05T07:36:43.520812+0000","last_active":"2024-02-05T07:38:18.692647+0000","last_peered":"2024-02-05T07:38:18.692647+0000","last_clean":"2024-02-05T07:38:18.692647+0000","last_became_active":"2024-02-05T07:34:30.670179+0000","last_became_peered":"2024-02-05T07:34:30.670179+0000","last_unstale":"2024-02-05T07:38:18.692647+0000","last_undegraded":"2024-02-05T07:38:18.692647+0000","last_fullsized":"2024-02-05T07:38:18.692647+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:43.520666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:36:43.520666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.742113+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":1460,"reported_epoch":598,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:38:18.694149+0000","last_change":"2024-02-05T07:38:09.737580+0000","last_active":"2024-02-05T07:38:18.694149+0000","last_peered":"2024-02-05T07:38:18.694149+0000","last_clean":"2024-02-05T07:38:05.047416+0000","last_became_active":"2024-02-05T07:38:09.724932+0000","last_became_peered":"2024-02-05T07:38:09.724932+0000","last_unstale":"2024-02-05T07:38:18.694149+0000","last_undegraded":"2024-02-05T07:38:18.694149+0000","last_fullsized":"2024-02-05T07:38:18.694149+0000","mapping_epoch":589,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.395638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.395638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,4],"avail_no_missing":["2","1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"326'19","reported_seq":742,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.633845+0000","last_change":"2024-02-05T07:36:48.088544+0000","last_active":"2024-02-05T07:38:16.633845+0000","last_peered":"2024-02-05T07:38:16.633845+0000","last_clean":"2024-02-05T07:38:16.633845+0000","last_became_active":"2024-02-05T07:30:40.501893+0000","last_became_peered":"2024-02-05T07:30:40.501893+0000","last_unstale":"2024-02-05T07:38:16.633845+0000","last_undegraded":"2024-02-05T07:38:16.633845+0000","last_fullsized":"2024-02-05T07:38:16.633845+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:36:48.088406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:36:48.088406+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":526,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.700140+0000","last_change":"2024-02-05T07:37:07.757536+0000","last_active":"2024-02-05T07:38:18.700140+0000","last_peered":"2024-02-05T07:38:18.700140+0000","last_clean":"2024-02-05T07:38:18.700140+0000","last_became_active":"2024-02-05T07:36:32.712612+0000","last_became_peered":"2024-02-05T07:36:32.712612+0000","last_unstale":"2024-02-05T07:38:18.700140+0000","last_undegraded":"2024-02-05T07:38:18.700140+0000","last_fullsized":"2024-02-05T07:38:18.700140+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:07.757307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:37:07.757307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:28.872563+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":531,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693152+0000","last_change":"2024-02-05T07:38:09.480888+0000","last_active":"2024-02-05T07:38:18.693152+0000","last_peered":"2024-02-05T07:38:18.693152+0000","last_clean":"2024-02-05T07:38:18.693152+0000","last_became_active":"2024-02-05T07:38:09.480566+0000","last_became_peered":"2024-02-05T07:38:09.480566+0000","last_unstale":"2024-02-05T07:38:18.693152+0000","last_undegraded":"2024-02-05T07:38:18.693152+0000","last_fullsized":"2024-02-05T07:38:18.693152+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:42.529597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:36:42.529597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.741989+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":701,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.651162+0000","last_change":"2024-02-05T07:35:15.273201+0000","last_active":"2024-02-05T07:38:16.651162+0000","last_peered":"2024-02-05T07:38:16.651162+0000","last_clean":"2024-02-05T07:38:16.651162+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:38:16.651162+0000","last_undegraded":"2024-02-05T07:38:16.651162+0000","last_fullsized":"2024-02-05T07:38:16.651162+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:15.272861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:35:15.272861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"86'128","reported_seq":1117,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.699545+0000","last_change":"2024-02-05T07:38:11.679051+0000","last_active":"2024-02-05T07:38:18.699545+0000","last_peered":"2024-02-05T07:38:18.699545+0000","last_clean":"2024-02-05T07:38:18.699545+0000","last_became_active":"2024-02-05T07:38:11.678664+0000","last_became_peered":"2024-02-05T07:38:11.678664+0000","last_unstale":"2024-02-05T07:38:18.699545+0000","last_undegraded":"2024-02-05T07:38:18.699545+0000","last_fullsized":"2024-02-05T07:38:18.699545+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:07.528587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:07.528587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.692619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"325'8","reported_seq":713,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.649629+0000","last_change":"2024-02-05T07:35:47.070886+0000","last_active":"2024-02-05T07:38:16.649629+0000","last_peered":"2024-02-05T07:38:16.649629+0000","last_clean":"2024-02-05T07:38:16.649629+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:38:16.649629+0000","last_undegraded":"2024-02-05T07:38:16.649629+0000","last_fullsized":"2024-02-05T07:38:16.649629+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:35:47.070805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:47.070805+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":700,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.651952+0000","last_change":"2024-02-05T07:34:39.980293+0000","last_active":"2024-02-05T07:38:16.651952+0000","last_peered":"2024-02-05T07:38:16.651952+0000","last_clean":"2024-02-05T07:38:16.651952+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:38:16.651952+0000","last_undegraded":"2024-02-05T07:38:16.651952+0000","last_fullsized":"2024-02-05T07:38:16.651952+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:39.980067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:34:39.980067+0000","last_clean_scrub_stamp":"2024-02-05T07:34:39.980067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":972,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693162+0000","last_change":"2024-02-05T07:38:09.464714+0000","last_active":"2024-02-05T07:38:18.693162+0000","last_peered":"2024-02-05T07:38:18.693162+0000","last_clean":"2024-02-05T07:38:18.693162+0000","last_became_active":"2024-02-05T07:38:09.464446+0000","last_became_peered":"2024-02-05T07:38:09.464446+0000","last_unstale":"2024-02-05T07:38:18.693162+0000","last_undegraded":"2024-02-05T07:38:18.693162+0000","last_fullsized":"2024-02-05T07:38:18.693162+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:50.465128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:36:50.465128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.742143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":149,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.691466+0000","last_change":"2024-02-05T07:38:09.504998+0000","last_active":"2024-02-05T07:38:18.691466+0000","last_peered":"2024-02-05T07:38:18.691466+0000","last_clean":"2024-02-05T07:38:18.691466+0000","last_became_active":"2024-02-05T07:38:09.503369+0000","last_became_peered":"2024-02-05T07:38:09.503369+0000","last_unstale":"2024-02-05T07:38:18.691466+0000","last_undegraded":"2024-02-05T07:38:18.691466+0000","last_fullsized":"2024-02-05T07:38:18.691466+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:58.296469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:58.296469+0000","last_clean_scrub_stamp":"2024-02-05T07:36:58.296469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.318172+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1a","version":"86'128","reported_seq":438,"reported_epoch":598,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:38:19.095597+0000","last_change":"2024-02-05T07:38:09.498407+0000","last_active":"2024-02-05T07:38:19.095597+0000","last_peered":"2024-02-05T07:38:19.095597+0000","last_clean":"2024-02-05T07:25:57.244953+0000","last_became_active":"2024-02-05T07:38:09.480023+0000","last_became_peered":"2024-02-05T07:38:09.480023+0000","last_unstale":"2024-02-05T07:38:19.095597+0000","last_undegraded":"2024-02-05T07:38:19.095597+0000","last_fullsized":"2024-02-05T07:38:19.095597+0000","mapping_epoch":588,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:36:51.217709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:36:51.217709+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":4,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[4,2,6],"avail_no_missing":["4","1","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"326'19","reported_seq":952,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694754+0000","last_change":"2024-02-05T07:35:45.529783+0000","last_active":"2024-02-05T07:38:18.694754+0000","last_peered":"2024-02-05T07:38:18.694754+0000","last_clean":"2024-02-05T07:38:18.694754+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:38:18.694754+0000","last_undegraded":"2024-02-05T07:38:18.694754+0000","last_fullsized":"2024-02-05T07:38:18.694754+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:35:45.529663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:45.529663+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.741928+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":697,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.633580+0000","last_change":"2024-02-05T07:35:11.098562+0000","last_active":"2024-02-05T07:38:16.633580+0000","last_peered":"2024-02-05T07:38:16.633580+0000","last_clean":"2024-02-05T07:38:16.633580+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:38:16.633580+0000","last_undegraded":"2024-02-05T07:38:16.633580+0000","last_fullsized":"2024-02-05T07:38:16.633580+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:11.098430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:35:11.098430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":143,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.696478+0000","last_change":"2024-02-05T07:36:46.264042+0000","last_active":"2024-02-05T07:38:18.696478+0000","last_peered":"2024-02-05T07:38:18.696478+0000","last_clean":"2024-02-05T07:38:18.696478+0000","last_became_active":"2024-02-05T07:36:32.752879+0000","last_became_peered":"2024-02-05T07:36:32.752879+0000","last_unstale":"2024-02-05T07:38:18.696478+0000","last_undegraded":"2024-02-05T07:38:18.696478+0000","last_fullsized":"2024-02-05T07:38:18.696478+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:46.263956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.263956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.318126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":630,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693368+0000","last_change":"2024-02-05T07:38:09.504345+0000","last_active":"2024-02-05T07:38:18.693368+0000","last_peered":"2024-02-05T07:38:18.693368+0000","last_clean":"2024-02-05T07:38:18.693368+0000","last_became_active":"2024-02-05T07:38:09.503835+0000","last_became_peered":"2024-02-05T07:38:09.503835+0000","last_unstale":"2024-02-05T07:38:18.693368+0000","last_undegraded":"2024-02-05T07:38:18.693368+0000","last_fullsized":"2024-02-05T07:38:18.693368+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:34.512979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:36:34.512979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.318141+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1b","version":"86'128","reported_seq":878,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.692401+0000","last_change":"2024-02-05T07:38:15.645275+0000","last_active":"2024-02-05T07:38:18.692401+0000","last_peered":"2024-02-05T07:38:18.692401+0000","last_clean":"2024-02-05T07:38:18.692401+0000","last_became_active":"2024-02-05T07:38:15.642871+0000","last_became_peered":"2024-02-05T07:38:15.642871+0000","last_unstale":"2024-02-05T07:38:18.692401+0000","last_undegraded":"2024-02-05T07:38:18.692401+0000","last_fullsized":"2024-02-05T07:38:18.692401+0000","mapping_epoch":595,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:42.201171+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:42.201171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.352425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"326'20","reported_seq":864,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.653318+0000","last_change":"2024-02-05T07:35:13.094823+0000","last_active":"2024-02-05T07:38:16.653318+0000","last_peered":"2024-02-05T07:38:16.653318+0000","last_clean":"2024-02-05T07:38:16.653318+0000","last_became_active":"2024-02-05T07:30:40.495954+0000","last_became_peered":"2024-02-05T07:30:40.495954+0000","last_unstale":"2024-02-05T07:38:16.653318+0000","last_undegraded":"2024-02-05T07:38:16.653318+0000","last_fullsized":"2024-02-05T07:38:16.653318+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:35:13.094694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:13.094694+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"585'2","reported_seq":1182,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.697866+0000","last_change":"2024-02-05T07:38:09.518275+0000","last_active":"2024-02-05T07:38:18.697866+0000","last_peered":"2024-02-05T07:38:18.697866+0000","last_clean":"2024-02-05T07:38:18.697866+0000","last_became_active":"2024-02-05T07:38:09.481401+0000","last_became_peered":"2024-02-05T07:38:09.481401+0000","last_unstale":"2024-02-05T07:38:18.697866+0000","last_undegraded":"2024-02-05T07:38:18.697866+0000","last_fullsized":"2024-02-05T07:38:18.697866+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'2","last_scrub_stamp":"2024-02-05T07:38:00.631235+0000","last_deep_scrub":"585'2","last_deep_scrub_stamp":"2024-02-05T07:38:00.631235+0000","last_clean_scrub_stamp":"2024-02-05T07:38:00.631235+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:25.815592+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"589'14","reported_seq":1213,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694853+0000","last_change":"2024-02-05T07:38:09.524876+0000","last_active":"2024-02-05T07:38:18.694853+0000","last_peered":"2024-02-05T07:38:18.694853+0000","last_clean":"2024-02-05T07:38:18.694853+0000","last_became_active":"2024-02-05T07:38:09.472910+0000","last_became_peered":"2024-02-05T07:38:09.472910+0000","last_unstale":"2024-02-05T07:38:18.694853+0000","last_undegraded":"2024-02-05T07:38:18.694853+0000","last_fullsized":"2024-02-05T07:38:18.694853+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'13","last_scrub_stamp":"2024-02-05T07:36:57.487913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:57.487913+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.074044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":149,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694933+0000","last_change":"2024-02-05T07:38:09.497609+0000","last_active":"2024-02-05T07:38:18.694933+0000","last_peered":"2024-02-05T07:38:18.694933+0000","last_clean":"2024-02-05T07:38:18.694933+0000","last_became_active":"2024-02-05T07:38:09.482346+0000","last_became_peered":"2024-02-05T07:38:09.482346+0000","last_unstale":"2024-02-05T07:38:18.694933+0000","last_undegraded":"2024-02-05T07:38:18.694933+0000","last_fullsized":"2024-02-05T07:38:18.694933+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:35:15.112415+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:35:15.112415+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.318111+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"86'128","reported_seq":633,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.690964+0000","last_change":"2024-02-05T07:37:42.405367+0000","last_active":"2024-02-05T07:38:18.690964+0000","last_peered":"2024-02-05T07:38:18.690964+0000","last_clean":"2024-02-05T07:38:18.690964+0000","last_became_active":"2024-02-05T07:37:41.644081+0000","last_became_peered":"2024-02-05T07:37:41.644081+0000","last_unstale":"2024-02-05T07:38:18.690964+0000","last_undegraded":"2024-02-05T07:38:18.690964+0000","last_fullsized":"2024-02-05T07:38:18.690964+0000","mapping_epoch":576,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:42.405286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:42.405286+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":69,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:10.272046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"326'23","reported_seq":741,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.649557+0000","last_change":"2024-02-05T07:38:09.521703+0000","last_active":"2024-02-05T07:38:16.649557+0000","last_peered":"2024-02-05T07:38:16.649557+0000","last_clean":"2024-02-05T07:38:16.649557+0000","last_became_active":"2024-02-05T07:38:09.486225+0000","last_became_peered":"2024-02-05T07:38:09.486225+0000","last_unstale":"2024-02-05T07:38:16.649557+0000","last_undegraded":"2024-02-05T07:38:16.649557+0000","last_fullsized":"2024-02-05T07:38:16.649557+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'23","last_scrub_stamp":"2024-02-05T07:34:41.934886+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:34:41.934886+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":149,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.691485+0000","last_change":"2024-02-05T07:38:09.489856+0000","last_active":"2024-02-05T07:38:18.691485+0000","last_peered":"2024-02-05T07:38:18.691485+0000","last_clean":"2024-02-05T07:38:18.691485+0000","last_became_active":"2024-02-05T07:38:09.489555+0000","last_became_peered":"2024-02-05T07:38:09.489555+0000","last_unstale":"2024-02-05T07:38:18.691485+0000","last_undegraded":"2024-02-05T07:38:18.691485+0000","last_fullsized":"2024-02-05T07:38:18.691485+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:59.269484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.269484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:29.200326+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1023,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.649469+0000","last_change":"2024-02-05T07:38:09.484586+0000","last_active":"2024-02-05T07:38:16.649469+0000","last_peered":"2024-02-05T07:38:16.649469+0000","last_clean":"2024-02-05T07:38:16.649469+0000","last_became_active":"2024-02-05T07:38:09.484261+0000","last_became_peered":"2024-02-05T07:38:09.484261+0000","last_unstale":"2024-02-05T07:38:16.649469+0000","last_undegraded":"2024-02-05T07:38:16.649469+0000","last_fullsized":"2024-02-05T07:38:16.649469+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:47.437414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:47.437414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":701,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.652061+0000","last_change":"2024-02-05T07:34:37.126181+0000","last_active":"2024-02-05T07:38:16.652061+0000","last_peered":"2024-02-05T07:38:16.652061+0000","last_clean":"2024-02-05T07:38:16.652061+0000","last_became_active":"2024-02-05T07:28:26.603067+0000","last_became_peered":"2024-02-05T07:28:26.603067+0000","last_unstale":"2024-02-05T07:38:16.652061+0000","last_undegraded":"2024-02-05T07:38:16.652061+0000","last_fullsized":"2024-02-05T07:38:16.652061+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:37.125938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:37.125938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.19","version":"86'128","reported_seq":1325,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.652158+0000","last_change":"2024-02-05T07:37:42.340756+0000","last_active":"2024-02-05T07:38:16.652158+0000","last_peered":"2024-02-05T07:38:16.652158+0000","last_clean":"2024-02-05T07:38:16.652158+0000","last_became_active":"2024-02-05T07:37:41.647791+0000","last_became_peered":"2024-02-05T07:37:41.647791+0000","last_unstale":"2024-02-05T07:38:16.652158+0000","last_undegraded":"2024-02-05T07:38:16.652158+0000","last_fullsized":"2024-02-05T07:38:16.652158+0000","mapping_epoch":576,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:42.340630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:42.340630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:08.312703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"326'27","reported_seq":159,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694577+0000","last_change":"2024-02-05T07:37:57.440904+0000","last_active":"2024-02-05T07:38:18.694577+0000","last_peered":"2024-02-05T07:38:18.694577+0000","last_clean":"2024-02-05T07:38:18.694577+0000","last_became_active":"2024-02-05T07:36:32.744301+0000","last_became_peered":"2024-02-05T07:36:32.744301+0000","last_unstale":"2024-02-05T07:38:18.694577+0000","last_undegraded":"2024-02-05T07:38:18.694577+0000","last_fullsized":"2024-02-05T07:38:18.694577+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:37:57.440806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:57.440806+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:09.436391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":77,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.691954+0000","last_change":"2024-02-05T07:38:09.501225+0000","last_active":"2024-02-05T07:38:18.691954+0000","last_peered":"2024-02-05T07:38:18.691954+0000","last_clean":"2024-02-05T07:38:18.691954+0000","last_became_active":"2024-02-05T07:38:09.500871+0000","last_became_peered":"2024-02-05T07:38:09.500871+0000","last_unstale":"2024-02-05T07:38:18.691954+0000","last_undegraded":"2024-02-05T07:38:18.691954+0000","last_fullsized":"2024-02-05T07:38:18.691954+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:48.410161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:48.410161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.318157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1116,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.700102+0000","last_change":"2024-02-05T07:38:09.477962+0000","last_active":"2024-02-05T07:38:18.700102+0000","last_peered":"2024-02-05T07:38:18.700102+0000","last_clean":"2024-02-05T07:38:18.700102+0000","last_became_active":"2024-02-05T07:38:09.477634+0000","last_became_peered":"2024-02-05T07:38:09.477634+0000","last_unstale":"2024-02-05T07:38:18.700102+0000","last_undegraded":"2024-02-05T07:38:18.700102+0000","last_fullsized":"2024-02-05T07:38:18.700102+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:49.645159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:49.645159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":535,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694379+0000","last_change":"2024-02-05T07:38:09.471205+0000","last_active":"2024-02-05T07:38:18.694379+0000","last_peered":"2024-02-05T07:38:18.694379+0000","last_clean":"2024-02-05T07:38:18.694379+0000","last_became_active":"2024-02-05T07:38:09.470792+0000","last_became_peered":"2024-02-05T07:38:09.470792+0000","last_unstale":"2024-02-05T07:38:18.694379+0000","last_undegraded":"2024-02-05T07:38:18.694379+0000","last_fullsized":"2024-02-05T07:38:18.694379+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:43.532678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:36:43.532678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.742300+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1e","version":"86'128","reported_seq":471,"reported_epoch":597,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T07:38:16.653796+0000","last_change":"2024-02-05T07:38:09.981431+0000","last_active":"2024-02-05T07:38:16.653796+0000","last_peered":"2024-02-05T07:38:16.653796+0000","last_clean":"2024-02-05T07:38:05.055641+0000","last_became_active":"2024-02-05T07:38:09.975696+0000","last_became_peered":"2024-02-05T07:38:09.975696+0000","last_unstale":"2024-02-05T07:38:16.653796+0000","last_undegraded":"2024-02-05T07:38:16.653796+0000","last_fullsized":"2024-02-05T07:38:16.653796+0000","mapping_epoch":589,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:43.389010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:43.389010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,4],"avail_no_missing":["3","0","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"325'7","reported_seq":617,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.651572+0000","last_change":"2024-02-05T07:38:09.527356+0000","last_active":"2024-02-05T07:38:16.651572+0000","last_peered":"2024-02-05T07:38:16.651572+0000","last_clean":"2024-02-05T07:38:16.651572+0000","last_became_active":"2024-02-05T07:38:09.496642+0000","last_became_peered":"2024-02-05T07:38:09.496642+0000","last_unstale":"2024-02-05T07:38:16.651572+0000","last_undegraded":"2024-02-05T07:38:16.651572+0000","last_fullsized":"2024-02-05T07:38:16.651572+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'7","last_scrub_stamp":"2024-02-05T07:37:06.495165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:06.495165+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:32.730738+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":352,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.701496+0000","last_change":"2024-02-05T07:38:09.495876+0000","last_active":"2024-02-05T07:38:18.701496+0000","last_peered":"2024-02-05T07:38:18.701496+0000","last_clean":"2024-02-05T07:38:18.701496+0000","last_became_active":"2024-02-05T07:38:09.495573+0000","last_became_peered":"2024-02-05T07:38:09.495573+0000","last_unstale":"2024-02-05T07:38:18.701496+0000","last_undegraded":"2024-02-05T07:38:18.701496+0000","last_fullsized":"2024-02-05T07:38:18.701496+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:54.574076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:54.574076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.691579+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":33,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.698697+0000","last_change":"2024-02-05T07:38:09.462374+0000","last_active":"2024-02-05T07:38:18.698697+0000","last_peered":"2024-02-05T07:38:18.698697+0000","last_clean":"2024-02-05T07:38:18.698697+0000","last_became_active":"2024-02-05T07:38:09.462036+0000","last_became_peered":"2024-02-05T07:38:09.462036+0000","last_unstale":"2024-02-05T07:38:18.698697+0000","last_undegraded":"2024-02-05T07:38:18.698697+0000","last_fullsized":"2024-02-05T07:38:18.698697+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:55.064845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:55.064845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:00.444477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":929,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.698063+0000","last_change":"2024-02-05T07:36:32.710890+0000","last_active":"2024-02-05T07:38:18.698063+0000","last_peered":"2024-02-05T07:38:18.698063+0000","last_clean":"2024-02-05T07:38:18.698063+0000","last_became_active":"2024-02-05T07:36:32.710575+0000","last_became_peered":"2024-02-05T07:36:32.710575+0000","last_unstale":"2024-02-05T07:38:18.698063+0000","last_undegraded":"2024-02-05T07:38:18.698063+0000","last_fullsized":"2024-02-05T07:38:18.698063+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:15.923221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:35:15.923221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.691442+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"325'19","reported_seq":645,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.647988+0000","last_change":"2024-02-05T07:38:09.506020+0000","last_active":"2024-02-05T07:38:16.647988+0000","last_peered":"2024-02-05T07:38:16.647988+0000","last_clean":"2024-02-05T07:38:16.647988+0000","last_became_active":"2024-02-05T07:38:09.479937+0000","last_became_peered":"2024-02-05T07:38:09.479937+0000","last_unstale":"2024-02-05T07:38:16.647988+0000","last_undegraded":"2024-02-05T07:38:16.647988+0000","last_fullsized":"2024-02-05T07:38:16.647988+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'19","last_scrub_stamp":"2024-02-05T07:35:18.020665+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:35:18.020665+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":355,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.694518+0000","last_change":"2024-02-05T07:38:09.485212+0000","last_active":"2024-02-05T07:38:18.694518+0000","last_peered":"2024-02-05T07:38:18.694518+0000","last_clean":"2024-02-05T07:38:18.694518+0000","last_became_active":"2024-02-05T07:38:09.484875+0000","last_became_peered":"2024-02-05T07:38:09.484875+0000","last_unstale":"2024-02-05T07:38:18.694518+0000","last_undegraded":"2024-02-05T07:38:18.694518+0000","last_fullsized":"2024-02-05T07:38:18.694518+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:39.547282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:39.547282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.741958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":35,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693424+0000","last_change":"2024-02-05T07:38:09.471374+0000","last_active":"2024-02-05T07:38:18.693424+0000","last_peered":"2024-02-05T07:38:18.693424+0000","last_clean":"2024-02-05T07:38:18.693424+0000","last_became_active":"2024-02-05T07:38:09.471013+0000","last_became_peered":"2024-02-05T07:38:09.471013+0000","last_unstale":"2024-02-05T07:38:18.693424+0000","last_undegraded":"2024-02-05T07:38:18.693424+0000","last_fullsized":"2024-02-05T07:38:18.693424+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:17.049130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:35:17.049130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.741746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":135,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693861+0000","last_change":"2024-02-05T07:36:32.737398+0000","last_active":"2024-02-05T07:38:18.693861+0000","last_peered":"2024-02-05T07:38:18.693861+0000","last_clean":"2024-02-05T07:38:18.693861+0000","last_became_active":"2024-02-05T07:36:32.733405+0000","last_became_peered":"2024-02-05T07:36:32.733405+0000","last_unstale":"2024-02-05T07:38:18.693861+0000","last_undegraded":"2024-02-05T07:38:18.693861+0000","last_fullsized":"2024-02-05T07:38:18.693861+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:47.484894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:33:47.484894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.317980+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"86'128","reported_seq":378,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693904+0000","last_change":"2024-02-05T07:37:43.698413+0000","last_active":"2024-02-05T07:38:18.693904+0000","last_peered":"2024-02-05T07:38:18.693904+0000","last_clean":"2024-02-05T07:38:18.693904+0000","last_became_active":"2024-02-05T07:37:41.643917+0000","last_became_peered":"2024-02-05T07:37:41.643917+0000","last_unstale":"2024-02-05T07:38:18.693904+0000","last_undegraded":"2024-02-05T07:38:18.693904+0000","last_fullsized":"2024-02-05T07:38:18.693904+0000","mapping_epoch":576,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:43.698299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:43.698299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:55.064030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"326'27","reported_seq":775,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.633554+0000","last_change":"2024-02-05T07:37:50.422750+0000","last_active":"2024-02-05T07:38:16.633554+0000","last_peered":"2024-02-05T07:38:16.633554+0000","last_clean":"2024-02-05T07:38:16.633554+0000","last_became_active":"2024-02-05T07:36:32.741691+0000","last_became_peered":"2024-02-05T07:36:32.741691+0000","last_unstale":"2024-02-05T07:38:16.633554+0000","last_undegraded":"2024-02-05T07:38:16.633554+0000","last_fullsized":"2024-02-05T07:38:16.633554+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:37:50.422661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:50.422661+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:55.331441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":714,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.651510+0000","last_change":"2024-02-05T07:37:44.381379+0000","last_active":"2024-02-05T07:38:16.651510+0000","last_peered":"2024-02-05T07:38:16.651510+0000","last_clean":"2024-02-05T07:38:16.651510+0000","last_became_active":"2024-02-05T07:36:32.746731+0000","last_became_peered":"2024-02-05T07:36:32.746731+0000","last_unstale":"2024-02-05T07:38:16.651510+0000","last_undegraded":"2024-02-05T07:38:16.651510+0000","last_fullsized":"2024-02-05T07:38:16.651510+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:44.381300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:37:44.381300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:50.586247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":527,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.692990+0000","last_change":"2024-02-05T07:38:09.473135+0000","last_active":"2024-02-05T07:38:18.692990+0000","last_peered":"2024-02-05T07:38:18.692990+0000","last_clean":"2024-02-05T07:38:18.692990+0000","last_became_active":"2024-02-05T07:38:09.472733+0000","last_became_peered":"2024-02-05T07:38:09.472733+0000","last_unstale":"2024-02-05T07:38:18.692990+0000","last_undegraded":"2024-02-05T07:38:18.692990+0000","last_fullsized":"2024-02-05T07:38:18.692990+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:45.170539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.170539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:07.485628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":707,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-05T07:38:16.651768+0000","last_change":"2024-02-05T07:34:28.009081+0000","last_active":"2024-02-05T07:38:16.651768+0000","last_peered":"2024-02-05T07:38:16.651768+0000","last_clean":"2024-02-05T07:38:16.651768+0000","last_became_active":"2024-02-05T07:30:40.493086+0000","last_became_peered":"2024-02-05T07:30:40.493086+0000","last_unstale":"2024-02-05T07:38:16.651768+0000","last_undegraded":"2024-02-05T07:38:16.651768+0000","last_fullsized":"2024-02-05T07:38:16.651768+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:34:28.008966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:34:28.008966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"86'128","reported_seq":1238,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.699766+0000","last_change":"2024-02-05T07:38:15.647867+0000","last_active":"2024-02-05T07:38:18.699766+0000","last_peered":"2024-02-05T07:38:18.699766+0000","last_clean":"2024-02-05T07:38:18.699766+0000","last_became_active":"2024-02-05T07:38:15.647473+0000","last_became_peered":"2024-02-05T07:38:15.647473+0000","last_unstale":"2024-02-05T07:38:18.699766+0000","last_undegraded":"2024-02-05T07:38:18.699766+0000","last_fullsized":"2024-02-05T07:38:18.699766+0000","mapping_epoch":595,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.590393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.590393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:07.220353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"450'19","reported_seq":877,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.692820+0000","last_change":"2024-02-05T07:38:09.515026+0000","last_active":"2024-02-05T07:38:18.692820+0000","last_peered":"2024-02-05T07:38:18.692820+0000","last_clean":"2024-02-05T07:38:18.692820+0000","last_became_active":"2024-02-05T07:38:09.470878+0000","last_became_peered":"2024-02-05T07:38:09.470878+0000","last_unstale":"2024-02-05T07:38:18.692820+0000","last_undegraded":"2024-02-05T07:38:18.692820+0000","last_fullsized":"2024-02-05T07:38:18.692820+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'19","last_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_deep_scrub":"450'19","last_deep_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_clean_scrub_stamp":"2024-02-05T07:34:34.853068+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:21.741571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":135,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.696221+0000","last_change":"2024-02-05T07:36:32.734803+0000","last_active":"2024-02-05T07:38:18.696221+0000","last_peered":"2024-02-05T07:38:18.696221+0000","last_clean":"2024-02-05T07:38:18.696221+0000","last_became_active":"2024-02-05T07:36:32.734502+0000","last_became_peered":"2024-02-05T07:36:32.734502+0000","last_unstale":"2024-02-05T07:38:18.696221+0000","last_undegraded":"2024-02-05T07:38:18.696221+0000","last_fullsized":"2024-02-05T07:38:18.696221+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:33:44.117071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:33:44.117071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:20.317965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":143,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693608+0000","last_change":"2024-02-05T07:37:50.267485+0000","last_active":"2024-02-05T07:38:18.693608+0000","last_peered":"2024-02-05T07:38:18.693608+0000","last_clean":"2024-02-05T07:38:18.693608+0000","last_became_active":"2024-02-05T07:36:32.708917+0000","last_became_peered":"2024-02-05T07:36:32.708917+0000","last_unstale":"2024-02-05T07:38:18.693608+0000","last_undegraded":"2024-02-05T07:38:18.693608+0000","last_fullsized":"2024-02-05T07:38:18.693608+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:50.267364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:50.267364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:39:06.913273+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":653,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-05T07:38:18.693301+0000","last_change":"2024-02-05T07:38:09.480452+0000","last_active":"2024-02-05T07:38:18.693301+0000","last_peered":"2024-02-05T07:38:18.693301+0000","last_clean":"2024-02-05T07:38:18.693301+0000","last_became_active":"2024-02-05T07:38:09.480027+0000","last_became_peered":"2024-02-05T07:38:09.480027+0000","last_unstale":"2024-02-05T07:38:18.693301+0000","last_undegraded":"2024-02-05T07:38:18.693301+0000","last_fullsized":"2024-02-05T07:38:18.693301+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:49.186617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:49.186617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:38:49.896888+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1351352320,"num_objects":20620,"num_object_clones":0,"num_object_copies":61860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20620,"num_write_kb":1319680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2295,"num_bytes_recovered":150405120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4108124160,"data_stored":4108124160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20620,"ondisk_log_size":20620,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":464,"num_read_kb":459,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":613848,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":12,"num_write":7,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":70441,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":4552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1097,"num_read_kb":854,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":10072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":503808,"data_stored":13460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":763,"ondisk_log_size":763,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":257698037999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":301,"seq":1292785156216,"num_pgs":109,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1455976,"kb_used_data":890392,"kb_used_omap":5,"kb_used_meta":565562,"kb_avail":92289176,"statfs":{"total":95995035648,"available":94504116224,"internally_reserved":0,"allocated":911761408,"data_stored":910709624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":579136005},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:37:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":1.095,"15min":1.5329999999999999},"min":{"1min":0.27600000000000002,"5min":0.192,"15min":0.192},"max":{"1min":1.9670000000000001,"5min":13.976000000000001,"15min":14.218},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.032,"5min":1.1579999999999999,"15min":1.577},"min":{"1min":0.41399999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.5919999999999996,"5min":13.962999999999999,"15min":14.175000000000001},"last":0.44600000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:37:46 2024","interfaces":[{"interface":"back","average":{"1min":3.544,"5min":1.7130000000000001,"15min":1.9279999999999999},"min":{"1min":0.26600000000000001,"5min":0.217,"15min":0.217},"max":{"1min":46.261000000000003,"5min":46.261000000000003,"15min":55.923000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":3.5179999999999998,"5min":1.778,"15min":2.4649999999999999},"min":{"1min":0.35099999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":46.415999999999997,"5min":46.415999999999997,"15min":61.103000000000002},"last":0.88800000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:37:46 2024","interfaces":[{"interface":"back","average":{"1min":3.569,"5min":2.2669999999999999,"15min":2.2629999999999999},"min":{"1min":0.374,"5min":0.29799999999999999,"15min":0.249},"max":{"1min":46.927,"5min":46.927,"15min":46.927},"last":0.51300000000000001},{"interface":"front","average":{"1min":3.351,"5min":2.161,"15min":2.5609999999999999},"min":{"1min":0.33300000000000002,"5min":0.245,"15min":0.245},"max":{"1min":46.506,"5min":46.506,"15min":46.506},"last":0.35499999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:37:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":1.2330000000000001,"15min":2.3250000000000002},"min":{"1min":0.315,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.2999999999999998,"5min":14.006,"15min":55.978999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.373,"15min":2.5070000000000001},"min":{"1min":0.312,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.1989999999999998,"5min":15.032999999999999,"15min":55.631},"last":0.32800000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:37:46 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.0680000000000001,"15min":1.131},"min":{"1min":0.252,"5min":0.224,"15min":0.224},"max":{"1min":1.133,"5min":151.02099999999999,"15min":151.02099999999999},"last":0.375},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.2360000000000002,"15min":1.135},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.7470000000000001,"5min":149.21799999999999,"15min":149.21799999999999},"last":0.46400000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:37:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.65800000000000003,"15min":0.59499999999999997},"min":{"1min":0.30599999999999999,"5min":0.28399999999999997,"15min":0.26800000000000002},"max":{"1min":2.431,"5min":2.8919999999999999,"15min":2.8919999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49099999999999999,"15min":0.53700000000000003},"min":{"1min":0.33800000000000002,"5min":0.311,"15min":0.28100000000000003},"max":{"1min":0.83399999999999996,"5min":1.0669999999999999,"15min":1.526},"last":0.53100000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:37:46 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52200000000000002,"15min":2.996},"min":{"1min":0.315,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.876,"5min":2.0550000000000002,"15min":84.158000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.745,"15min":3.1600000000000001},"min":{"1min":0.32200000000000001,"5min":0.317,"15min":0.312},"max":{"1min":1.097,"5min":23.055,"15min":85.018000000000001},"last":0.55200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575117,"num_pgs":119,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1939908,"kb_used_data":979960,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":91805244,"statfs":{"total":95995035648,"available":94008569856,"internally_reserved":0,"allocated":1003479040,"data_stored":1002395362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":982972869},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:38:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":2.3220000000000001,"15min":2.2509999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.16800000000000001},"max":{"1min":4.6740000000000004,"5min":29.449999999999999,"15min":30.536999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.24,"5min":2.6890000000000001,"15min":2.5390000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.20899999999999999},"max":{"1min":5.5629999999999997,"5min":29.268999999999998,"15min":31.853999999999999},"last":4.7590000000000003}]},{"osd":2,"last update":"Mon Feb 5 07:37:21 2024","interfaces":[{"interface":"back","average":{"1min":4.665,"5min":3.024,"15min":2.5510000000000002},"min":{"1min":0.28399999999999997,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":42.823999999999998,"5min":42.823999999999998,"15min":42.823999999999998},"last":0.99299999999999999},{"interface":"front","average":{"1min":4.3979999999999997,"5min":2.8300000000000001,"15min":2.4089999999999998},"min":{"1min":0.29999999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":42.426000000000002,"5min":42.426000000000002,"15min":42.426000000000002},"last":0.64500000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:37:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":2.0169999999999999,"15min":2.2509999999999999},"min":{"1min":0.27800000000000002,"5min":0.24299999999999999,"15min":0.20999999999999999},"max":{"1min":7.0380000000000003,"5min":8.3919999999999995,"15min":46.777000000000001},"last":1.73},{"interface":"front","average":{"1min":1.954,"5min":1.9299999999999999,"15min":2.2389999999999999},"min":{"1min":0.33400000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.5419999999999998,"5min":9.5419999999999998,"15min":50.164999999999999},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:38:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.3380000000000001,"15min":2.2320000000000002},"min":{"1min":0.441,"5min":0.33400000000000002,"15min":0.312},"max":{"1min":1.9470000000000001,"5min":89.409000000000006,"15min":89.409000000000006},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.599,"5min":2.3809999999999998,"15min":2.1499999999999999},"min":{"1min":0.371,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":14.629,"5min":89.915000000000006,"15min":89.915000000000006},"last":0.95599999999999996}]},{"osd":5,"last update":"Mon Feb 5 07:37:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3009999999999999,"15min":2.145},"min":{"1min":0.47299999999999998,"5min":0.33600000000000002,"15min":0.309},"max":{"1min":6.5140000000000002,"5min":6.7610000000000001,"15min":106.672},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.462,"15min":2.1909999999999998},"min":{"1min":0.44400000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.4740000000000002,"5min":7.1210000000000004,"15min":107.348},"last":1.587}]},{"osd":6,"last update":"Mon Feb 5 07:37:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.1339999999999999,"15min":1.4419999999999999},"min":{"1min":0.42299999999999999,"5min":0.314,"15min":0.29199999999999998},"max":{"1min":4.4169999999999998,"5min":4.8209999999999997,"15min":5.5990000000000002},"last":1.03},{"interface":"front","average":{"1min":1.389,"5min":1.4099999999999999,"15min":1.74},"min":{"1min":0.40100000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":6.484,"5min":6.8179999999999996,"15min":8.923},"last":0.67900000000000005}]},{"osd":7,"last update":"Mon Feb 5 07:37:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.524,"15min":2.113},"min":{"1min":0.378,"5min":0.36599999999999999,"15min":0.318},"max":{"1min":4.4980000000000002,"5min":7.0899999999999999,"15min":66.703999999999994},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.1990000000000001,"15min":1.6910000000000001},"min":{"1min":0.42799999999999999,"5min":0.39500000000000002,"15min":0.33500000000000002},"max":{"1min":4.4409999999999998,"5min":4.7510000000000003,"15min":61.594999999999999},"last":1.0649999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738641,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1634856,"kb_used_data":692952,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92110296,"statfs":{"total":95995035648,"available":94320943104,"internally_reserved":0,"allocated":709582848,"data_stored":708517059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":964491717},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:37:49 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.8979999999999999,"15min":2.3159999999999998},"min":{"1min":0.214,"5min":0.189,"15min":0.16500000000000001},"max":{"1min":3.153,"5min":12.308,"15min":45.372},"last":1.228},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.46,"15min":2.161},"min":{"1min":0.218,"5min":0.192,"15min":0.186},"max":{"1min":4.2889999999999997,"5min":8.6460000000000008,"15min":45.231999999999999},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:37:18 2024","interfaces":[{"interface":"back","average":{"1min":1.98,"5min":3.1019999999999999,"15min":2.4569999999999999},"min":{"1min":0.36499999999999999,"5min":0.22900000000000001,"15min":0.191},"max":{"1min":4.359,"5min":47.137999999999998,"15min":65.903999999999996},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.4770000000000001,"5min":2.895,"15min":2.3839999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.186},"max":{"1min":4.9199999999999999,"5min":47.170999999999999,"15min":65.694000000000003},"last":0.67300000000000004}]},{"osd":3,"last update":"Mon Feb 5 07:37:27 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.609,"15min":2.1629999999999998},"min":{"1min":0.28899999999999998,"5min":0.25600000000000001,"15min":0.21299999999999999},"max":{"1min":5.2690000000000001,"5min":26.43,"15min":65.155000000000001},"last":1.085},{"interface":"front","average":{"1min":1.4570000000000001,"5min":2.516,"15min":2.5979999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.258,"5min":25.337,"15min":65.114000000000004},"last":2.589}]},{"osd":4,"last update":"Mon Feb 5 07:37:49 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.77,"15min":2.1269999999999998},"min":{"1min":0.308,"5min":0.29099999999999998,"15min":0.26700000000000002},"max":{"1min":2.9049999999999998,"5min":12.718999999999999,"15min":65.188000000000002},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.3879999999999999,"15min":2.04},"min":{"1min":0.40300000000000002,"5min":0.30499999999999999,"15min":0.23699999999999999},"max":{"1min":3.6659999999999999,"5min":8.6899999999999995,"15min":64.873000000000005},"last":0.61599999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:37:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.099,"15min":1.78},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":4.9690000000000003,"5min":4.9690000000000003,"15min":95.772999999999996},"last":0.63},{"interface":"front","average":{"1min":1.121,"5min":1.4550000000000001,"15min":1.9410000000000001},"min":{"1min":0.52800000000000002,"5min":0.34699999999999998,"15min":0.27600000000000002},"max":{"1min":3.7120000000000002,"5min":8.7599999999999998,"15min":96.028000000000006},"last":0.65200000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:37:43 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":1.8149999999999999,"15min":1.4159999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26300000000000001},"max":{"1min":2.8260000000000001,"5min":12.670999999999999,"15min":12.670999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.012,"5min":1.49,"15min":1.4810000000000001},"min":{"1min":0.441,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":3.7789999999999999,"5min":8.718,"15min":8.718},"last":0.70999999999999996}]},{"osd":7,"last update":"Mon Feb 5 07:38:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":1.8819999999999999,"15min":2.0070000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.28599999999999998},"max":{"1min":2.0710000000000002,"5min":12.766999999999999,"15min":65.263000000000005},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.753,"5min":1.532,"15min":1.9430000000000001},"min":{"1min":0.44500000000000001,"5min":0.39100000000000001,"15min":0.309},"max":{"1min":3.7450000000000001,"5min":8.7910000000000004,"15min":64.906000000000006},"last":0.57999999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411592,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1498208,"kb_used_data":646552,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92246944,"statfs":{"total":95995035648,"available":94460870656,"internally_reserved":0,"allocated":662069248,"data_stored":661017800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:37:19 2024","interfaces":[{"interface":"back","average":{"1min":1.8480000000000001,"5min":2.6429999999999998,"15min":2.6139999999999999},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.182},"max":{"1min":5.96,"5min":80.427000000000007,"15min":92.638000000000005},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.681,"5min":3.1520000000000001,"15min":2.927},"min":{"1min":0.24399999999999999,"5min":0.23999999999999999,"15min":0.20699999999999999},"max":{"1min":3.6219999999999999,"5min":80.939999999999998,"15min":93.481999999999999},"last":2.0449999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:37:19 2024","interfaces":[{"interface":"back","average":{"1min":2.4590000000000001,"5min":3.7320000000000002,"15min":3.2549999999999999},"min":{"1min":0.28299999999999997,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":4.9240000000000004,"5min":110.917,"15min":110.917},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.637,"15min":2.7069999999999999},"min":{"1min":0.312,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":5.8449999999999998,"5min":7.1859999999999999,"15min":93.004000000000005},"last":0.432}]},{"osd":3,"last update":"Mon Feb 5 07:37:32 2024","interfaces":[{"interface":"back","average":{"1min":2.4369999999999998,"5min":3.8140000000000001,"15min":2.9409999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":7.1639999999999997,"5min":110.773,"15min":110.773},"last":1.3220000000000001},{"interface":"front","average":{"1min":2.2669999999999999,"5min":1.972,"15min":2.2850000000000001},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.193},"max":{"1min":10.598000000000001,"5min":47.368000000000002,"15min":63.521000000000001},"last":1.137}]},{"osd":4,"last update":"Mon Feb 5 07:37:56 2024","interfaces":[{"interface":"back","average":{"1min":3.0430000000000001,"5min":2.1800000000000002,"15min":2.4550000000000001},"min":{"1min":0.32800000000000001,"5min":0.24299999999999999,"15min":0.24099999999999999},"max":{"1min":40.436,"5min":46.521999999999998,"15min":168.11699999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":3.649,"5min":1.7050000000000001,"15min":2.214},"min":{"1min":0.443,"5min":0.36099999999999999,"15min":0.27200000000000002},"max":{"1min":40.329000000000001,"5min":40.329000000000001,"15min":168.589},"last":0.46899999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:37:19 2024","interfaces":[{"interface":"back","average":{"1min":1.7390000000000001,"5min":1.1910000000000001,"15min":1.323},"min":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.315},"max":{"1min":6.0129999999999999,"5min":6.0129999999999999,"15min":39.923000000000002},"last":0.65000000000000002},{"interface":"front","average":{"1min":2.2530000000000001,"5min":2.3889999999999998,"15min":1.669},"min":{"1min":0.36199999999999999,"5min":0.30499999999999999,"15min":0.26400000000000001},"max":{"1min":4.8819999999999997,"5min":47.761000000000003,"15min":47.761000000000003},"last":1.2749999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:37:45 2024","interfaces":[{"interface":"back","average":{"1min":3.843,"5min":2.9449999999999998,"15min":1.7290000000000001},"min":{"1min":0.309,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":40.828000000000003,"5min":47.726999999999997,"15min":47.726999999999997},"last":1.1799999999999999},{"interface":"front","average":{"1min":3.7770000000000001,"5min":1.7450000000000001,"15min":1.4399999999999999},"min":{"1min":0.437,"5min":0.372,"15min":0.33100000000000002},"max":{"1min":40.872999999999998,"5min":40.872999999999998,"15min":40.872999999999998},"last":0.71499999999999997}]},{"osd":7,"last update":"Mon Feb 5 07:37:19 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.452,"15min":1.585},"min":{"1min":0.35699999999999998,"5min":0.35199999999999998,"15min":0.29299999999999998},"max":{"1min":3.5830000000000002,"5min":11.311999999999999,"15min":37.573999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":2.286,"5min":3.4489999999999998,"15min":2.1480000000000001},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":4.9169999999999998,"5min":110.86799999999999,"15min":110.86799999999999},"last":1.214}]}]},{"osd":3,"up_from":26,"seq":111669149954,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1530104,"kb_used_data":628460,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92215048,"statfs":{"total":95995035648,"available":94428209152,"internally_reserved":0,"allocated":643543040,"data_stored":642457140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":923269575},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:37:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":2.2440000000000002,"15min":2.133},"min":{"1min":0.20899999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.0519999999999996,"5min":48.688000000000002,"15min":48.688000000000002},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.79700000000000004,"5min":2.6150000000000002,"15min":2.7130000000000001},"min":{"1min":0.22500000000000001,"5min":0.185,"15min":0.185},"max":{"1min":3.5939999999999999,"5min":48.573,"15min":99.989999999999995},"last":2.391}]},{"osd":1,"last update":"Mon Feb 5 07:37:53 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.6259999999999999,"15min":2.6280000000000001},"min":{"1min":0.23999999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":6.1150000000000002,"5min":26.329000000000001,"15min":102.20399999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.22,"5min":1.546,"15min":2.6469999999999998},"min":{"1min":0.27800000000000002,"5min":0.20999999999999999,"15min":0.185},"max":{"1min":6.0220000000000002,"5min":26.867000000000001,"15min":102.286},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Feb 5 07:37:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":3.504,"15min":3.0609999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22700000000000001},"max":{"1min":8.1240000000000006,"5min":145.94999999999999,"15min":145.94999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.1060000000000001,"5min":3.7690000000000001,"15min":2.9889999999999999},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.6289999999999996,"5min":146.006,"15min":146.006},"last":1.4510000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:38:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":1.7110000000000001,"15min":1.867},"min":{"1min":0.29599999999999999,"5min":0.29399999999999998,"15min":0.27000000000000002},"max":{"1min":3.54,"5min":31.655999999999999,"15min":100.024},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.98,"15min":2.0470000000000002},"min":{"1min":0.374,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.7050000000000001,"5min":37.049999999999997,"15min":99.859999999999999},"last":1.6399999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:37:22 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.083,"15min":1.9410000000000001},"min":{"1min":0.38700000000000001,"5min":0.308,"15min":0.28499999999999998},"max":{"1min":3.0699999999999998,"5min":6.6680000000000001,"15min":99.911000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.587,"5min":1.252,"15min":1.7470000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.27400000000000002},"max":{"1min":8.4350000000000005,"5min":8.4350000000000005,"15min":100.053},"last":0.39900000000000002}]},{"osd":6,"last update":"Mon Feb 5 07:37:53 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.651,"15min":1.583},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.29299999999999998},"max":{"1min":3.036,"5min":32.505000000000003,"15min":99.807000000000002},"last":0.442},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.948,"15min":1.706},"min":{"1min":0.371,"5min":0.33600000000000002,"15min":0.30499999999999999},"max":{"1min":3.5089999999999999,"5min":31.084,"15min":101.489},"last":0.36899999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:38:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1699999999999999,"15min":2.403},"min":{"1min":0.42899999999999999,"5min":0.38700000000000001,"15min":0.29499999999999998},"max":{"1min":5.024,"5min":8.4329999999999998,"15min":99.893000000000001},"last":1.526},{"interface":"front","average":{"1min":0.86899999999999999,"5min":1.2569999999999999,"15min":2.355},"min":{"1min":0.41799999999999998,"5min":0.41099999999999998,"15min":0.375},"max":{"1min":2.948,"5min":17.611000000000001,"15min":99.879000000000005},"last":0.51000000000000001}]}]},{"osd":4,"up_from":0,"seq":137438953726,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":176093659385,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":706543616,"data_stored":706543616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1026228224,"data_stored":1026228224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":666173440,"data_stored":666173440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":648347648,"data_stored":648347648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":150011904,"data_stored":150011904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":910819328,"data_stored":910819328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:38:27.722 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-02-05T07:38:27.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-02-05T07:38:27.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:27.832 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:27.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:27.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:27.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:27.959 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:27.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:28.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:28.086 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:28.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:28.188 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:28.214 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:28.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:28.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:28.369 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:28.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:28.470 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:28.509 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:28.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:28.543 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 24 16 34511 34495 89.812 80.75 0.0153974 0.0111275 2024-02-05T07:38:28.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:28.638 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:28.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:28.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:28.790 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:28.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:28.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:28.870 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:28.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:28.916 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:28.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:28.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:28.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:28 smithi007 ceph-mon[29098]: pgmap v1087: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 1.8 GiB data, 9.5 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-02-05T07:38:28.997 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:28 smithi007 ceph-mon[37158]: pgmap v1087: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 1.8 GiB data, 9.5 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-02-05T07:38:29.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:29.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:28 smithi090 ceph-mon[33213]: pgmap v1087: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 1.8 GiB data, 9.5 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-02-05T07:38:29.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:29.071 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:29.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:29.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:29.225 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:29.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:29.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:29.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:29.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:29.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:29.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:29.479 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:29.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:29.542 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 25 16 35705 35689 89.2043 74.625 0.0053562 0.0112026 2024-02-05T07:38:29.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:29.605 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:29.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:29.621 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:38:29.666 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:38:29.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:29.714 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:38:29.731 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:29.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:29.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:29.755 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:38:29.785 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:38:29.794 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:38:29.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:29.842 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:38:29.858 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:29.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:29.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:29.881 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:38:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:29 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4152461093' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:38:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:29 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:38:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:29 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/915227203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:29.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:29 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4152461093' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:38:29.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:29 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:38:29.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:29 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/915227203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:29.927 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:38:29.935 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:38:29.935 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:38:29.935 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:38:29.935 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:38:29.935 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:38:29.936 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:38:29.936 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:38:29.936 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:38:29.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:29.984 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:29.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:29.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:29 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4152461093' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:38:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:29 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:38:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:29 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/915227203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:30.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:30.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:30.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:30.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:30.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:30.253 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:30.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:30.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:30.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:30.379 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:30.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:30.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:30.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:30.507 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:30.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:30.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:30.542 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 26 16 36936 36920 88.7323 76.9375 0.00614474 0.0112632 2024-02-05T07:38:30.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:30.633 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:30.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:30.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:30.761 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:30.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:30.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:30.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:30.888 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:30.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:30.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:30.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:31.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:31.031 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:31.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:31.102 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: pgmap v1088: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.0 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 585 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-02-05T07:38:31.103 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:38:31.103 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:38:31.103 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: osdmap e601: 8 total, 8 up, 5 in 2024-02-05T07:38:31.103 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: 1.0 scrub starts 2024-02-05T07:38:31.103 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: 1.0 scrub ok 2024-02-05T07:38:31.103 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-02-05T07:38:31.103 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-02-05T07:38:31.103 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [6, 2]}]: dispatch 2024-02-05T07:38:31.103 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]: dispatch 2024-02-05T07:38:31.103 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [6, 2]}]: dispatch 2024-02-05T07:38:31.103 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [2, 0]}]: dispatch 2024-02-05T07:38:31.103 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]: dispatch 2024-02-05T07:38:31.103 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [6, 2]}]: dispatch 2024-02-05T07:38:31.104 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]}]: dispatch 2024-02-05T07:38:31.104 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [2, 0]}]: dispatch 2024-02-05T07:38:31.104 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [6, 2]}]: dispatch 2024-02-05T07:38:31.104 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]}]: dispatch 2024-02-05T07:38:31.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:31.155 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:31.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:31.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: pgmap v1088: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.0 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 585 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-02-05T07:38:31.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:38:31.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:38:31.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: osdmap e601: 8 total, 8 up, 5 in 2024-02-05T07:38:31.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: 1.0 scrub starts 2024-02-05T07:38:31.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: 1.0 scrub ok 2024-02-05T07:38:31.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-02-05T07:38:31.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-02-05T07:38:31.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [6, 2]}]: dispatch 2024-02-05T07:38:31.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]: dispatch 2024-02-05T07:38:31.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [6, 2]}]: dispatch 2024-02-05T07:38:31.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [2, 0]}]: dispatch 2024-02-05T07:38:31.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]: dispatch 2024-02-05T07:38:31.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [6, 2]}]: dispatch 2024-02-05T07:38:31.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]}]: dispatch 2024-02-05T07:38:31.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [2, 0]}]: dispatch 2024-02-05T07:38:31.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [6, 2]}]: dispatch 2024-02-05T07:38:31.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]}]: dispatch 2024-02-05T07:38:31.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: pgmap v1088: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.0 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 585 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-02-05T07:38:31.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:38:31.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:38:31.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: osdmap e601: 8 total, 8 up, 5 in 2024-02-05T07:38:31.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: 1.0 scrub starts 2024-02-05T07:38:31.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: 1.0 scrub ok 2024-02-05T07:38:31.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-02-05T07:38:31.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-02-05T07:38:31.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [6, 2]}]: dispatch 2024-02-05T07:38:31.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]: dispatch 2024-02-05T07:38:31.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [6, 2]}]: dispatch 2024-02-05T07:38:31.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [2, 0]}]: dispatch 2024-02-05T07:38:31.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]: dispatch 2024-02-05T07:38:31.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [6, 2]}]: dispatch 2024-02-05T07:38:31.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]}]: dispatch 2024-02-05T07:38:31.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [2, 0]}]: dispatch 2024-02-05T07:38:31.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [6, 2]}]: dispatch 2024-02-05T07:38:31.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]}]: dispatch 2024-02-05T07:38:31.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:31.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:31.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:31.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:31.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:31.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:31.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:31.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:38:31.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:31.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:31.542 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 27 16 38142 38126 88.2373 75.375 0.00902845 0.0113174 2024-02-05T07:38:31.639 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:31.663 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:31.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:31.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:31.805 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:31.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:31.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:31.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:31.950 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:31.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:32.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:32.078 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:32.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:32.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]': finished 2024-02-05T07:38:32.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [6, 2]}]': finished 2024-02-05T07:38:32.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-02-05T07:38:32.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [2, 0]}]': finished 2024-02-05T07:38:32.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [6, 2]}]': finished 2024-02-05T07:38:32.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]}]': finished 2024-02-05T07:38:32.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[29098]: osdmap e602: 8 total, 8 up, 5 in 2024-02-05T07:38:32.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[29098]: 4.9 scrub starts 2024-02-05T07:38:32.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[29098]: 4.9 scrub ok 2024-02-05T07:38:32.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:38:32.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[29098]: 1.1b scrub starts 2024-02-05T07:38:32.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[29098]: 1.1b scrub ok 2024-02-05T07:38:32.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[29098]: 2.0 scrub starts 2024-02-05T07:38:32.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[29098]: osdmap e603: 8 total, 8 up, 5 in 2024-02-05T07:38:32.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]': finished 2024-02-05T07:38:32.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [6, 2]}]': finished 2024-02-05T07:38:32.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-02-05T07:38:32.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [2, 0]}]': finished 2024-02-05T07:38:32.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [6, 2]}]': finished 2024-02-05T07:38:32.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]}]': finished 2024-02-05T07:38:32.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[37158]: osdmap e602: 8 total, 8 up, 5 in 2024-02-05T07:38:32.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[37158]: 4.9 scrub starts 2024-02-05T07:38:32.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[37158]: 4.9 scrub ok 2024-02-05T07:38:32.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:38:32.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[37158]: 1.1b scrub starts 2024-02-05T07:38:32.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[37158]: 1.1b scrub ok 2024-02-05T07:38:32.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[37158]: 2.0 scrub starts 2024-02-05T07:38:32.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:31 smithi007 ceph-mon[37158]: osdmap e603: 8 total, 8 up, 5 in 2024-02-05T07:38:32.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:32.206 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:32.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:32.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]': finished 2024-02-05T07:38:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [6, 2]}]': finished 2024-02-05T07:38:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-02-05T07:38:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [2, 0]}]': finished 2024-02-05T07:38:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [6, 2]}]': finished 2024-02-05T07:38:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]}]': finished 2024-02-05T07:38:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-mon[33213]: osdmap e602: 8 total, 8 up, 5 in 2024-02-05T07:38:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-mon[33213]: 4.9 scrub starts 2024-02-05T07:38:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-mon[33213]: 4.9 scrub ok 2024-02-05T07:38:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:38:32.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-mon[33213]: 1.1b scrub starts 2024-02-05T07:38:32.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-mon[33213]: 1.1b scrub ok 2024-02-05T07:38:32.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-mon[33213]: 2.0 scrub starts 2024-02-05T07:38:32.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:31 smithi090 ceph-mon[33213]: osdmap e603: 8 total, 8 up, 5 in 2024-02-05T07:38:32.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:32.357 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:32.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:32.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:32.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:32.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:32.504 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:32.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:32.542 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 28 15 39225 39210 87.5054 67.75 0.0136628 0.0114215 2024-02-05T07:38:32.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:32.647 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:32.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:32.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:32.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:32.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:32.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:32.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:32.902 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:32.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:33.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:33.030 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:33.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:33.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:33.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:33.159 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:33.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:33.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:32 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:33.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:32 smithi007 ceph-mon[29098]: pgmap v1091: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.1 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-02-05T07:38:33.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:32 smithi007 ceph-mon[29098]: 2.0 scrub ok 2024-02-05T07:38:33.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:32 smithi007 ceph-mon[29098]: 1.2 scrub starts 2024-02-05T07:38:33.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:32 smithi007 ceph-mon[29098]: 1.2 scrub ok 2024-02-05T07:38:33.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:32 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:33.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:32 smithi007 ceph-mon[37158]: pgmap v1091: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.1 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-02-05T07:38:33.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:32 smithi007 ceph-mon[37158]: 2.0 scrub ok 2024-02-05T07:38:33.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:32 smithi007 ceph-mon[37158]: 1.2 scrub starts 2024-02-05T07:38:33.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:32 smithi007 ceph-mon[37158]: 1.2 scrub ok 2024-02-05T07:38:33.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:33.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:32 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:33.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:32 smithi090 ceph-mon[33213]: pgmap v1091: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.1 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-02-05T07:38:33.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:32 smithi090 ceph-mon[33213]: 2.0 scrub ok 2024-02-05T07:38:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:32 smithi090 ceph-mon[33213]: 1.2 scrub starts 2024-02-05T07:38:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:32 smithi090 ceph-mon[33213]: 1.2 scrub ok 2024-02-05T07:38:33.307 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:33.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:33.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:33.452 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:33.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:33.542 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 29 16 40634 40618 87.5223 88 0.00624056 0.0114185 2024-02-05T07:38:33.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:33.601 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:33.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:33.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:33.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:33.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:33.800 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:38:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:38:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:38:33.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:33.884 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:33.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:33.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:33.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:34.026 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:34.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:34.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:34.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:33 smithi007 ceph-mon[29098]: 1.15 scrub starts 2024-02-05T07:38:34.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:33 smithi007 ceph-mon[29098]: 1.15 scrub ok 2024-02-05T07:38:34.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:33 smithi007 ceph-mon[29098]: 3.14 scrub starts 2024-02-05T07:38:34.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:33 smithi007 ceph-mon[29098]: 3.14 scrub ok 2024-02-05T07:38:34.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:33 smithi007 ceph-mon[29098]: 1.d scrub starts 2024-02-05T07:38:34.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:33 smithi007 ceph-mon[29098]: 1.d scrub ok 2024-02-05T07:38:34.102 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:33 smithi007 ceph-mon[37158]: 1.15 scrub starts 2024-02-05T07:38:34.102 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:33 smithi007 ceph-mon[37158]: 1.15 scrub ok 2024-02-05T07:38:34.102 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:33 smithi007 ceph-mon[37158]: 3.14 scrub starts 2024-02-05T07:38:34.102 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:33 smithi007 ceph-mon[37158]: 3.14 scrub ok 2024-02-05T07:38:34.102 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:33 smithi007 ceph-mon[37158]: 1.d scrub starts 2024-02-05T07:38:34.102 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:33 smithi007 ceph-mon[37158]: 1.d scrub ok 2024-02-05T07:38:34.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:34.155 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:34.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:34.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:34.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:33 smithi090 ceph-mon[33213]: 1.15 scrub starts 2024-02-05T07:38:34.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:33 smithi090 ceph-mon[33213]: 1.15 scrub ok 2024-02-05T07:38:34.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:33 smithi090 ceph-mon[33213]: 3.14 scrub starts 2024-02-05T07:38:34.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:33 smithi090 ceph-mon[33213]: 3.14 scrub ok 2024-02-05T07:38:34.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:33 smithi090 ceph-mon[33213]: 1.d scrub starts 2024-02-05T07:38:34.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:33 smithi090 ceph-mon[33213]: 1.d scrub ok 2024-02-05T07:38:34.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:34.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:34.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:34.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:34.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:34.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:34.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:34.511 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:34.542 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 30 16 41953 41937 87.3526 82.4375 0.00920486 0.0114416 2024-02-05T07:38:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:34.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:34.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:34.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:34.679 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:34.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:34.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:34.806 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:34.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:34.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:34.934 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:34.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:35.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:35.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:35.062 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:35.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:35.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:35.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[29098]: pgmap v1093: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T07:38:35.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[29098]: 6.19 scrub starts 2024-02-05T07:38:35.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[29098]: 4.18 scrub starts 2024-02-05T07:38:35.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[29098]: 4.18 scrub ok 2024-02-05T07:38:35.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[29098]: 6.19 scrub ok 2024-02-05T07:38:35.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[29098]: 1.5 scrub starts 2024-02-05T07:38:35.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[29098]: 1.5 scrub ok 2024-02-05T07:38:35.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[29098]: osdmap e604: 8 total, 8 up, 5 in 2024-02-05T07:38:35.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[37158]: pgmap v1093: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T07:38:35.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[37158]: 6.19 scrub starts 2024-02-05T07:38:35.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[37158]: 4.18 scrub starts 2024-02-05T07:38:35.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[37158]: 4.18 scrub ok 2024-02-05T07:38:35.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[37158]: 6.19 scrub ok 2024-02-05T07:38:35.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[37158]: 1.5 scrub starts 2024-02-05T07:38:35.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[37158]: 1.5 scrub ok 2024-02-05T07:38:35.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:34 smithi007 ceph-mon[37158]: osdmap e604: 8 total, 8 up, 5 in 2024-02-05T07:38:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:35.222 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:35.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:34 smithi090 ceph-mon[33213]: pgmap v1093: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T07:38:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:34 smithi090 ceph-mon[33213]: 6.19 scrub starts 2024-02-05T07:38:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:34 smithi090 ceph-mon[33213]: 4.18 scrub starts 2024-02-05T07:38:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:34 smithi090 ceph-mon[33213]: 4.18 scrub ok 2024-02-05T07:38:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:34 smithi090 ceph-mon[33213]: 6.19 scrub ok 2024-02-05T07:38:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:34 smithi090 ceph-mon[33213]: 1.5 scrub starts 2024-02-05T07:38:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:34 smithi090 ceph-mon[33213]: 1.5 scrub ok 2024-02-05T07:38:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:34 smithi090 ceph-mon[33213]: osdmap e604: 8 total, 8 up, 5 in 2024-02-05T07:38:35.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:38:35.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:35.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:35.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:35.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:35.543 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 31 16 43221 43205 87.091 79.25 0.0141363 0.011475 2024-02-05T07:38:35.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:35.566 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:35.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:35.667 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:35.692 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:35.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:35.795 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:35.840 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:35.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:35.852 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:38:35.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:36.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:36.092 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:35 smithi007 ceph-mon[29098]: 1.17 scrub starts 2024-02-05T07:38:36.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:35 smithi007 ceph-mon[29098]: 1.17 scrub ok 2024-02-05T07:38:36.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:35 smithi007 ceph-mon[29098]: 5.6 scrub starts 2024-02-05T07:38:36.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:35 smithi007 ceph-mon[29098]: 5.6 scrub ok 2024-02-05T07:38:36.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:35 smithi007 ceph-mon[29098]: 6.1f scrub starts 2024-02-05T07:38:36.093 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:35 smithi007 ceph-mon[37158]: 1.17 scrub starts 2024-02-05T07:38:36.093 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:35 smithi007 ceph-mon[37158]: 1.17 scrub ok 2024-02-05T07:38:36.093 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:35 smithi007 ceph-mon[37158]: 5.6 scrub starts 2024-02-05T07:38:36.093 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:35 smithi007 ceph-mon[37158]: 5.6 scrub ok 2024-02-05T07:38:36.093 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:35 smithi007 ceph-mon[37158]: 6.1f scrub starts 2024-02-05T07:38:36.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:36.150 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:36.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:36.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:36.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:35 smithi090 ceph-mon[33213]: 1.17 scrub starts 2024-02-05T07:38:36.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:35 smithi090 ceph-mon[33213]: 1.17 scrub ok 2024-02-05T07:38:36.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:35 smithi090 ceph-mon[33213]: 5.6 scrub starts 2024-02-05T07:38:36.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:35 smithi090 ceph-mon[33213]: 5.6 scrub ok 2024-02-05T07:38:36.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:35 smithi090 ceph-mon[33213]: 6.1f scrub starts 2024-02-05T07:38:36.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:36.304 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:36.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:36.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:36.430 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:36.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:36.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:36.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:36.544 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 32 16 44371 44355 86.6125 71.875 0.0107401 0.0115395 2024-02-05T07:38:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:36.560 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:36.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:36.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:36.714 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:38:36.714 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:38:36.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:38:36.715 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15846.0:44537 11.5 e604/579 11:a0d2d154:::benchmark_data_smithi007_183880_object44536:head v 604'2727, mlcod=604'2725)", 2024-02-05T07:38:36.715 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:36.710220+0000", 2024-02-05T07:38:36.715 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00073055500000000003, 2024-02-05T07:38:36.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00076049400000000001, 2024-02-05T07:38:36.715 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:38:36.715 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-02-05T07:38:36.715 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:38:36.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:38:36.715 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:38:36.715 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:36.710220+0000", 2024-02-05T07:38:36.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:38:36.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:38:36.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:38:36.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:38:36.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:36.710220+0000", 2024-02-05T07:38:36.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:38:36.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:38:36.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:38:36.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:38:36.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:36.710221+0000", 2024-02-05T07:38:36.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0100000000000004e-07 2024-02-05T07:38:36.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:38:36.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:38:36.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:38:36.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:36.710251+0000", 2024-02-05T07:38:36.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9961e-05 2024-02-05T07:38:36.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:38:36.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:38:36.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:38:36.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:36.710266+0000", 2024-02-05T07:38:36.717 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4808e-05 2024-02-05T07:38:36.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:38:36.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:38:36.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:38:36.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:36.710274+0000", 2024-02-05T07:38:36.717 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4819999999999994e-06 2024-02-05T07:38:36.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:38:36.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:38:36.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:38:36.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:36.710331+0000", 2024-02-05T07:38:36.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6577999999999999e-05 2024-02-05T07:38:36.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:38:36.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:38:36.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:38:36.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:36.710352+0000", 2024-02-05T07:38:36.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1169e-05 2024-02-05T07:38:36.718 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:38:36.718 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:38:36.718 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:38:36.718 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:38:36.718 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:38:36.718 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 1 2024-02-05T07:38:36.718 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:38:36.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:36.719 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:36.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:36.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:36.846 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:36.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:36.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:36.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:36.990 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:37.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:37.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:36 smithi007 ceph-mon[29098]: pgmap v1095: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.8 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-02-05T07:38:37.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:36 smithi007 ceph-mon[29098]: 4.b scrub starts 2024-02-05T07:38:37.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:36 smithi007 ceph-mon[29098]: 4.b scrub ok 2024-02-05T07:38:37.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:36 smithi007 ceph-mon[29098]: 6.1f scrub ok 2024-02-05T07:38:37.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:36 smithi007 ceph-mon[29098]: 4.e scrub starts 2024-02-05T07:38:37.178 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:36 smithi007 ceph-mon[29098]: 4.e scrub ok 2024-02-05T07:38:37.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:36 smithi007 ceph-mon[37158]: pgmap v1095: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.8 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-02-05T07:38:37.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:36 smithi007 ceph-mon[37158]: 4.b scrub starts 2024-02-05T07:38:37.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:36 smithi007 ceph-mon[37158]: 4.b scrub ok 2024-02-05T07:38:37.179 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:36 smithi007 ceph-mon[37158]: 6.1f scrub ok 2024-02-05T07:38:37.179 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:36 smithi007 ceph-mon[37158]: 4.e scrub starts 2024-02-05T07:38:37.179 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:36 smithi007 ceph-mon[37158]: 4.e scrub ok 2024-02-05T07:38:37.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:37.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:37.219 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:37.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:36 smithi090 ceph-mon[33213]: pgmap v1095: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.8 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-02-05T07:38:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:36 smithi090 ceph-mon[33213]: 4.b scrub starts 2024-02-05T07:38:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:36 smithi090 ceph-mon[33213]: 4.b scrub ok 2024-02-05T07:38:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:36 smithi090 ceph-mon[33213]: 6.1f scrub ok 2024-02-05T07:38:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:36 smithi090 ceph-mon[33213]: 4.e scrub starts 2024-02-05T07:38:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:36 smithi090 ceph-mon[33213]: 4.e scrub ok 2024-02-05T07:38:37.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:37.363 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:37.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:37.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:37.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:37.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:37.544 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 33 16 45642 45626 86.3949 79.4375 0.0093477 0.0115692 2024-02-05T07:38:37.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:37.618 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:37.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:37.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:37.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:37.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:37.747 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:37.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:37.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:37.884 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:37.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:37.986 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:38.012 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:38.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:38.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:38.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:38.156 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:38.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:38.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:37 smithi007 ceph-mon[29098]: pgmap v1096: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 774 B/s rd, 78 MiB/s wr, 1.25k op/s; 6 B/s, 0 objects/s recovering 2024-02-05T07:38:38.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:37 smithi007 ceph-mon[29098]: 5.f scrub starts 2024-02-05T07:38:38.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:37 smithi007 ceph-mon[29098]: 5.f scrub ok 2024-02-05T07:38:38.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:37 smithi007 ceph-mon[37158]: pgmap v1096: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 774 B/s rd, 78 MiB/s wr, 1.25k op/s; 6 B/s, 0 objects/s recovering 2024-02-05T07:38:38.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:37 smithi007 ceph-mon[37158]: 5.f scrub starts 2024-02-05T07:38:38.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:37 smithi007 ceph-mon[37158]: 5.f scrub ok 2024-02-05T07:38:38.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:38.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:37 smithi090 ceph-mon[33213]: pgmap v1096: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 774 B/s rd, 78 MiB/s wr, 1.25k op/s; 6 B/s, 0 objects/s recovering 2024-02-05T07:38:38.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:37 smithi090 ceph-mon[33213]: 5.f scrub starts 2024-02-05T07:38:38.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:37 smithi090 ceph-mon[33213]: 5.f scrub ok 2024-02-05T07:38:38.280 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:38.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:38.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:38.408 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:38.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:38.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:38.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:38.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:38.545 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 34 16 46639 46623 85.6865 62.3125 0.0141372 0.0116621 2024-02-05T07:38:38.547 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:38.547 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:38.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:38.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:38.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:38.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:38.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:38.816 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:38.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:38.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:38.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:38.943 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:38.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:39.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:39.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:39.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:38 smithi007 ceph-mon[29098]: 3.b scrub starts 2024-02-05T07:38:39.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:38 smithi007 ceph-mon[29098]: 3.b scrub ok 2024-02-05T07:38:39.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:38 smithi007 ceph-mon[29098]: 4.12 scrub starts 2024-02-05T07:38:39.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:38 smithi007 ceph-mon[29098]: 4.12 scrub ok 2024-02-05T07:38:39.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:38 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:38:39.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:38 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:38:39.097 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T07:38:39.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:38 smithi007 ceph-mon[37158]: 3.b scrub starts 2024-02-05T07:38:39.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:38 smithi007 ceph-mon[37158]: 3.b scrub ok 2024-02-05T07:38:39.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:38 smithi007 ceph-mon[37158]: 4.12 scrub starts 2024-02-05T07:38:39.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:38 smithi007 ceph-mon[37158]: 4.12 scrub ok 2024-02-05T07:38:39.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:38 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:38:39.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:38 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:38:39.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:39.195 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:39.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:39.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:38 smithi090 ceph-mon[33213]: 3.b scrub starts 2024-02-05T07:38:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:38 smithi090 ceph-mon[33213]: 3.b scrub ok 2024-02-05T07:38:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:38 smithi090 ceph-mon[33213]: 4.12 scrub starts 2024-02-05T07:38:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:38 smithi090 ceph-mon[33213]: 4.12 scrub ok 2024-02-05T07:38:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:38 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:38:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:38 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:38:39.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:39.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:39.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:39.517 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:39.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:39.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:39.548 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 35 16 47837 47821 85.372 74.875 0.00812036 0.0117074 2024-02-05T07:38:39.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:39.645 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:39.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:39.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:39.747 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:39.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:39.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:39.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:39.876 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:39.914 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:39.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:39.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:40.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:40.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:40.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:40.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:39 smithi007 ceph-mon[29098]: osdmap e605: 8 total, 8 up, 5 in 2024-02-05T07:38:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:39 smithi007 ceph-mon[29098]: pgmap v1098: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 672 B/s rd, 80 MiB/s wr, 1.28k op/s; 6 B/s, 0 objects/s recovering 2024-02-05T07:38:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:39 smithi007 ceph-mon[29098]: 4.7 scrub starts 2024-02-05T07:38:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:39 smithi007 ceph-mon[29098]: 4.7 scrub ok 2024-02-05T07:38:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:39 smithi007 ceph-mon[29098]: osdmap e606: 8 total, 8 up, 5 in 2024-02-05T07:38:40.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:39 smithi007 ceph-mon[37158]: osdmap e605: 8 total, 8 up, 5 in 2024-02-05T07:38:40.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:39 smithi007 ceph-mon[37158]: pgmap v1098: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 672 B/s rd, 80 MiB/s wr, 1.28k op/s; 6 B/s, 0 objects/s recovering 2024-02-05T07:38:40.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:39 smithi007 ceph-mon[37158]: 4.7 scrub starts 2024-02-05T07:38:40.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:39 smithi007 ceph-mon[37158]: 4.7 scrub ok 2024-02-05T07:38:40.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:39 smithi007 ceph-mon[37158]: osdmap e606: 8 total, 8 up, 5 in 2024-02-05T07:38:40.183 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:40.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:40.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:40.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:39 smithi090 ceph-mon[33213]: osdmap e605: 8 total, 8 up, 5 in 2024-02-05T07:38:40.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:39 smithi090 ceph-mon[33213]: pgmap v1098: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 672 B/s rd, 80 MiB/s wr, 1.28k op/s; 6 B/s, 0 objects/s recovering 2024-02-05T07:38:40.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:39 smithi090 ceph-mon[33213]: 4.7 scrub starts 2024-02-05T07:38:40.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:39 smithi090 ceph-mon[33213]: 4.7 scrub ok 2024-02-05T07:38:40.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:39 smithi090 ceph-mon[33213]: osdmap e606: 8 total, 8 up, 5 in 2024-02-05T07:38:40.285 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:40.336 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:40.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:40.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:40.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:40.465 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:40.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:40.546 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 36 16 49035 49019 85.0803 74.875 0.0189604 0.0117464 2024-02-05T07:38:40.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:40.591 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:40.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:40.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:40.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:40.718 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:40.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:40.820 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:40.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:40.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:41.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:41.016 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:41.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:41.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:41.143 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:41.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:41.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:41.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:41.269 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:41.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:41.371 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:41.412 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:41.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:41.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:41.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:41.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:38:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:38:41.546 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 37 16 50164 50148 84.6878 70.5625 0.0142311 0.0118016 2024-02-05T07:38:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:41.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:41.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:41.648 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pg_num to 21 2024-02-05T07:38:41.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:41.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:41.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:41.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:41.808 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:41.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:41.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:41.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:41.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:41 smithi007 ceph-mon[29098]: osdmap e607: 8 total, 8 up, 5 in 2024-02-05T07:38:41.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:41 smithi007 ceph-mon[29098]: 1.1a scrub starts 2024-02-05T07:38:41.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:41 smithi007 ceph-mon[29098]: 1.1a scrub ok 2024-02-05T07:38:41.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:41 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2608894473' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:38:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:41 smithi007 ceph-mon[37158]: osdmap e607: 8 total, 8 up, 5 in 2024-02-05T07:38:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:41 smithi007 ceph-mon[37158]: 1.1a scrub starts 2024-02-05T07:38:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:41 smithi007 ceph-mon[37158]: 1.1a scrub ok 2024-02-05T07:38:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:41 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2608894473' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:38:41.935 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:41.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:42.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:41 smithi090 ceph-mon[33213]: osdmap e607: 8 total, 8 up, 5 in 2024-02-05T07:38:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:41 smithi090 ceph-mon[33213]: 1.1a scrub starts 2024-02-05T07:38:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:41 smithi090 ceph-mon[33213]: 1.1a scrub ok 2024-02-05T07:38:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:41 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2608894473' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T07:38:42.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:42.064 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:42.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:42.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:42.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:42.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:42.343 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:42.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:42.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:42.470 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:42.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:42.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:42.547 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 38 16 51035 51019 83.8917 54.4375 0.0126506 0.0119107 2024-02-05T07:38:42.572 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:42.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:42.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:42.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:42.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:42.748 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:42.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:42.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:42.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:42.877 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:42.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[29098]: 3.1 scrub starts 2024-02-05T07:38:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[29098]: 3.1 scrub ok 2024-02-05T07:38:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[29098]: pgmap v1101: 178 pgs: 1 peering, 5 active+remapped+backfill_toofull, 172 active+clean; 2.9 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 80 MiB/s wr, 1.27k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:38:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[29098]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:38:42.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:38:42.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2608894473' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T07:38:42.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[29098]: 6.15 scrub starts 2024-02-05T07:38:42.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[29098]: osdmap e608: 8 total, 8 up, 5 in 2024-02-05T07:38:42.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:38:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[37158]: 3.1 scrub starts 2024-02-05T07:38:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[37158]: 3.1 scrub ok 2024-02-05T07:38:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[37158]: pgmap v1101: 178 pgs: 1 peering, 5 active+remapped+backfill_toofull, 172 active+clean; 2.9 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 80 MiB/s wr, 1.27k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:38:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[37158]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:38:42.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:38:42.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2608894473' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T07:38:42.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[37158]: 6.15 scrub starts 2024-02-05T07:38:42.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[37158]: osdmap e608: 8 total, 8 up, 5 in 2024-02-05T07:38:42.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:42 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:38:42.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:42 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:42 smithi090 ceph-mon[33213]: 3.1 scrub starts 2024-02-05T07:38:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:42 smithi090 ceph-mon[33213]: 3.1 scrub ok 2024-02-05T07:38:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:42 smithi090 ceph-mon[33213]: pgmap v1101: 178 pgs: 1 peering, 5 active+remapped+backfill_toofull, 172 active+clean; 2.9 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 80 MiB/s wr, 1.27k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:38:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:42 smithi090 ceph-mon[33213]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-05T07:38:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:42 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:38:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:42 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2608894473' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T07:38:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:42 smithi090 ceph-mon[33213]: 6.15 scrub starts 2024-02-05T07:38:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:42 smithi090 ceph-mon[33213]: osdmap e608: 8 total, 8 up, 5 in 2024-02-05T07:38:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:42 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:38:43.028 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:43.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:43.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:43.156 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:43.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:43.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:43.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:43.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:43.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:43.430 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:43.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:43.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:43.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:43.547 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 39 16 52089 52073 83.4297 65.875 0.00902394 0.0119799 2024-02-05T07:38:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:43.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:43.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:43.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:43.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:43.841 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:43.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:43.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:38:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:38:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[29098]: 4.3 scrub starts 2024-02-05T07:38:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[29098]: 4.3 scrub ok 2024-02-05T07:38:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[29098]: 6.15 scrub ok 2024-02-05T07:38:43.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[29098]: 5.3 scrub starts 2024-02-05T07:38:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[29098]: 5.3 scrub ok 2024-02-05T07:38:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:38:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:38:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:38:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:38:43.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:38:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[37158]: 4.3 scrub starts 2024-02-05T07:38:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[37158]: 4.3 scrub ok 2024-02-05T07:38:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[37158]: 6.15 scrub ok 2024-02-05T07:38:43.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[37158]: 5.3 scrub starts 2024-02-05T07:38:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[37158]: 5.3 scrub ok 2024-02-05T07:38:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:38:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:38:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:38:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:38:43.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:43 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:38:43.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:43.969 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:43.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:43.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:44.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:43 smithi090 ceph-mon[33213]: 4.3 scrub starts 2024-02-05T07:38:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:43 smithi090 ceph-mon[33213]: 4.3 scrub ok 2024-02-05T07:38:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:43 smithi090 ceph-mon[33213]: 6.15 scrub ok 2024-02-05T07:38:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:43 smithi090 ceph-mon[33213]: 5.3 scrub starts 2024-02-05T07:38:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:43 smithi090 ceph-mon[33213]: 5.3 scrub ok 2024-02-05T07:38:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:43 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:38:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:43 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:38:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:43 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:38:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:43 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:38:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:43 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T07:38:44.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:44.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:44.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:44.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:44.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:44.325 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:44.371 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:44.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:44.473 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:44.497 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:44.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:44.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:44.547 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:38:44.547909+0000 min lat: 0.00118746 max lat: 1.24847 avg lat: 0.0120206 2024-02-05T07:38:44.548 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:38:44.548 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 40 15 53247 53232 83.1532 72.4375 0.0146461 0.0120206 2024-02-05T07:38:44.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:44.625 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:44.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:44.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:44.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:44.753 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:44.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:44.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:44.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:44.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:44.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:44 smithi007 ceph-mon[37158]: pgmap v1103: 178 pgs: 1 peering, 3 active+remapped+backfill_toofull, 174 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 58 MiB/s wr, 926 op/s 2024-02-05T07:38:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:44 smithi007 ceph-mon[37158]: 5.4 scrub starts 2024-02-05T07:38:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:44 smithi007 ceph-mon[37158]: 5.4 scrub ok 2024-02-05T07:38:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:44 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T07:38:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:44 smithi007 ceph-mon[37158]: osdmap e609: 8 total, 8 up, 5 in 2024-02-05T07:38:44.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:44 smithi007 ceph-mon[37158]: osdmap e610: 8 total, 8 up, 5 in 2024-02-05T07:38:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:44 smithi007 ceph-mon[29098]: pgmap v1103: 178 pgs: 1 peering, 3 active+remapped+backfill_toofull, 174 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 58 MiB/s wr, 926 op/s 2024-02-05T07:38:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:44 smithi007 ceph-mon[29098]: 5.4 scrub starts 2024-02-05T07:38:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:44 smithi007 ceph-mon[29098]: 5.4 scrub ok 2024-02-05T07:38:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:44 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T07:38:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:44 smithi007 ceph-mon[29098]: osdmap e609: 8 total, 8 up, 5 in 2024-02-05T07:38:44.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:44 smithi007 ceph-mon[29098]: osdmap e610: 8 total, 8 up, 5 in 2024-02-05T07:38:44.983 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:45.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:44 smithi090 ceph-mon[33213]: pgmap v1103: 178 pgs: 1 peering, 3 active+remapped+backfill_toofull, 174 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 58 MiB/s wr, 926 op/s 2024-02-05T07:38:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:44 smithi090 ceph-mon[33213]: 5.4 scrub starts 2024-02-05T07:38:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:44 smithi090 ceph-mon[33213]: 5.4 scrub ok 2024-02-05T07:38:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:44 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T07:38:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:44 smithi090 ceph-mon[33213]: osdmap e609: 8 total, 8 up, 5 in 2024-02-05T07:38:45.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:44 smithi090 ceph-mon[33213]: osdmap e610: 8 total, 8 up, 5 in 2024-02-05T07:38:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:45.034 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:45.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:45.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:45.161 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:45.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:45.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:45.307 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:45.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:45.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:45.453 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:45.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:45.548 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 41 16 54474 54458 82.9939 76.625 0.00813308 0.0120436 2024-02-05T07:38:45.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:45.598 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:45.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:45.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:45.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:45.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:45.752 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:45.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:45.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:45.903 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:45.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:45.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:46.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:46.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:46.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:46.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:46.204 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:46.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:46.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:46.313 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:46.347 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:38:46.363 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:46.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:46.408 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:38:46.460 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:38:46.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:46.517 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:38:46.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:46.518 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:46.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:46.548 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 42 16 55506 55490 82.5534 64.5 0.00936929 0.0121061 2024-02-05T07:38:46.565 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:38:46.613 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:38:46.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:46.645 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:46.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:46.668 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:38:46.717 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:38:46.717 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: pgmap v1106: 178 pgs: 1 peering, 2 active+remapped+backfill_toofull, 175 active+clean; 3.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.1 KiB/s rd, 54 MiB/s wr, 863 op/s 2024-02-05T07:38:46.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 2.1 scrub starts 2024-02-05T07:38:46.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 2.1 scrub ok 2024-02-05T07:38:46.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.18 starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.18 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.1a starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.17 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.1b starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: osdmap e611: 8 total, 8 up, 5 in 2024-02-05T07:38:46.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.1d starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.1c starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.19 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.15 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.15 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.1e starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.1e starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.16 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:38:46.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:38:46.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.16 scrub starts 2024-02-05T07:38:46.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: 1.16 scrub ok 2024-02-05T07:38:46.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3302455047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:46.719 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: pgmap v1106: 178 pgs: 1 peering, 2 active+remapped+backfill_toofull, 175 active+clean; 3.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.1 KiB/s rd, 54 MiB/s wr, 863 op/s 2024-02-05T07:38:46.719 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 2.1 scrub starts 2024-02-05T07:38:46.719 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 2.1 scrub ok 2024-02-05T07:38:46.719 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.18 starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.18 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.1a starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.17 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.1b starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: osdmap e611: 8 total, 8 up, 5 in 2024-02-05T07:38:46.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.1d starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.1c starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.19 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.15 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.15 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.1e starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.1e starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.16 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:46.721 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:38:46.721 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:38:46.721 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.16 scrub starts 2024-02-05T07:38:46.721 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: 1.16 scrub ok 2024-02-05T07:38:46.721 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:46 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3302455047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:46.721 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:38:46.721 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:38:46.721 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:38:46.721 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:38:46.721 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:38:46.722 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:38:46.722 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:38:46.722 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:38:46.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:46.772 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:46.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:46.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:46.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:46.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:46.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:47.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: pgmap v1106: 178 pgs: 1 peering, 2 active+remapped+backfill_toofull, 175 active+clean; 3.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.1 KiB/s rd, 54 MiB/s wr, 863 op/s 2024-02-05T07:38:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 2.1 scrub starts 2024-02-05T07:38:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 2.1 scrub ok 2024-02-05T07:38:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.18 starting backfill to osd.3 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.18 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.1a starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.17 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:47.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.1b starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:47.027 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: osdmap e611: 8 total, 8 up, 5 in 2024-02-05T07:38:47.027 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.1d starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:47.027 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.1c starting backfill to osd.0 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:47.027 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.19 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:47.027 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.15 starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:47.027 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.15 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:47.027 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.1e starting backfill to osd.1 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:47.027 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.1e starting backfill to osd.2 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:47.028 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.16 starting backfill to osd.6 from (0'0,0'0] MAX to 86'128 2024-02-05T07:38:47.028 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:38:47.028 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:38:47.028 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.16 scrub starts 2024-02-05T07:38:47.029 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: 1.16 scrub ok 2024-02-05T07:38:47.029 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:46 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3302455047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:47.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:47.044 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:47.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:47.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:47.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:47.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:47.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:47.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:47.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:47.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:47.455 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:47.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:47.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:47.548 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 43 16 56747 56731 82.4372 77.5625 0.00898587 0.012125 2024-02-05T07:38:47.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:47.583 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:47.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:47.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:47.680 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:38:47.684 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:47.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:47.709 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:47.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:47.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:47.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:47.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:47.867 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:47 smithi007 ceph-mon[29098]: 1.1c scrub starts 2024-02-05T07:38:47.867 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:47 smithi007 ceph-mon[29098]: 1.1c scrub ok 2024-02-05T07:38:47.868 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:47 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2632372386' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:38:47.868 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:47 smithi007 ceph-mon[29098]: osdmap e612: 8 total, 8 up, 5 in 2024-02-05T07:38:47.868 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:47 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:38:47.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:47 smithi007 ceph-mon[37158]: 1.1c scrub starts 2024-02-05T07:38:47.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:47 smithi007 ceph-mon[37158]: 1.1c scrub ok 2024-02-05T07:38:47.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:47 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2632372386' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:38:47.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:47 smithi007 ceph-mon[37158]: osdmap e612: 8 total, 8 up, 5 in 2024-02-05T07:38:47.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:47 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:38:47.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:47.979 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:47.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:48.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:47 smithi090 ceph-mon[33213]: 1.1c scrub starts 2024-02-05T07:38:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:47 smithi090 ceph-mon[33213]: 1.1c scrub ok 2024-02-05T07:38:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:47 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2632372386' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:38:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:47 smithi090 ceph-mon[33213]: osdmap e612: 8 total, 8 up, 5 in 2024-02-05T07:38:48.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:47 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:38:48.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:48.108 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:48.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:48.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:48.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:48.251 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:48.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:48.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:48.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:48.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:48.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:48.496 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:48.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:48.548 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 44 16 58031 58015 82.3874 80.25 0.0108093 0.0121318 2024-02-05T07:38:48.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:48.692 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:48.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:48.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:48.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:48.820 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:48.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:48.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:48.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:48.948 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:48.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:49.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:48 smithi090 ceph-mon[33213]: pgmap v1109: 178 pgs: 2 peering, 1 active+clean+remapped, 4 remapped+peering, 4 active+remapped+backfill_toofull, 167 active+clean; 3.3 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T07:38:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:48 smithi090 ceph-mon[33213]: 3.1f scrub starts 2024-02-05T07:38:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:48 smithi090 ceph-mon[33213]: 3.1f scrub ok 2024-02-05T07:38:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:48 smithi090 ceph-mon[33213]: 5.1b scrub starts 2024-02-05T07:38:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:48 smithi090 ceph-mon[33213]: 5.1b scrub ok 2024-02-05T07:38:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:48 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:38:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:48 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:38:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:48 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:38:49.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:48 smithi090 ceph-mon[33213]: osdmap e613: 8 total, 8 up, 5 in 2024-02-05T07:38:49.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:49.076 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:49.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:49.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[29098]: pgmap v1109: 178 pgs: 2 peering, 1 active+clean+remapped, 4 remapped+peering, 4 active+remapped+backfill_toofull, 167 active+clean; 3.3 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T07:38:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[29098]: 3.1f scrub starts 2024-02-05T07:38:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[29098]: 3.1f scrub ok 2024-02-05T07:38:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[29098]: 5.1b scrub starts 2024-02-05T07:38:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[29098]: 5.1b scrub ok 2024-02-05T07:38:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:38:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:38:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:38:49.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[29098]: osdmap e613: 8 total, 8 up, 5 in 2024-02-05T07:38:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[37158]: pgmap v1109: 178 pgs: 2 peering, 1 active+clean+remapped, 4 remapped+peering, 4 active+remapped+backfill_toofull, 167 active+clean; 3.3 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T07:38:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[37158]: 3.1f scrub starts 2024-02-05T07:38:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[37158]: 3.1f scrub ok 2024-02-05T07:38:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[37158]: 5.1b scrub starts 2024-02-05T07:38:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[37158]: 5.1b scrub ok 2024-02-05T07:38:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:38:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:38:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:38:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:48 smithi007 ceph-mon[37158]: osdmap e613: 8 total, 8 up, 5 in 2024-02-05T07:38:49.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:49.212 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:49.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:49.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:49.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:49.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:49.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:49.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:49.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:49.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:49.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:49.548 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 45 16 59123 59107 82.0731 68.25 0.00770356 0.0121784 2024-02-05T07:38:49.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:49.642 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:49.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:49.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:49.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:49.770 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:49.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:49.872 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:49.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:49.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:38:49.964 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[29098]: 3.1c scrub starts 2024-02-05T07:38:49.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[29098]: 3.1c scrub ok 2024-02-05T07:38:49.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[29098]: 4.17 deep-scrub starts 2024-02-05T07:38:49.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[29098]: 4.17 deep-scrub ok 2024-02-05T07:38:49.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[29098]: 1.16 scrub starts 2024-02-05T07:38:49.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[29098]: 1.16 scrub ok 2024-02-05T07:38:49.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:38:49.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:38:49.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[29098]: osdmap e614: 8 total, 8 up, 5 in 2024-02-05T07:38:49.965 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[37158]: 3.1c scrub starts 2024-02-05T07:38:49.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[37158]: 3.1c scrub ok 2024-02-05T07:38:49.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[37158]: 4.17 deep-scrub starts 2024-02-05T07:38:49.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[37158]: 4.17 deep-scrub ok 2024-02-05T07:38:49.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[37158]: 1.16 scrub starts 2024-02-05T07:38:49.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[37158]: 1.16 scrub ok 2024-02-05T07:38:49.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:38:49.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:38:49.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:49 smithi007 ceph-mon[37158]: osdmap e614: 8 total, 8 up, 5 in 2024-02-05T07:38:49.997 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:50.023 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:50.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:50.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:49 smithi090 ceph-mon[33213]: 3.1c scrub starts 2024-02-05T07:38:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:49 smithi090 ceph-mon[33213]: 3.1c scrub ok 2024-02-05T07:38:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:49 smithi090 ceph-mon[33213]: 4.17 deep-scrub starts 2024-02-05T07:38:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:49 smithi090 ceph-mon[33213]: 4.17 deep-scrub ok 2024-02-05T07:38:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:49 smithi090 ceph-mon[33213]: 1.16 scrub starts 2024-02-05T07:38:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:49 smithi090 ceph-mon[33213]: 1.16 scrub ok 2024-02-05T07:38:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:49 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:38:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:49 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:38:50.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:49 smithi090 ceph-mon[33213]: osdmap e614: 8 total, 8 up, 5 in 2024-02-05T07:38:50.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:50.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:50.164 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:50.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:50.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:50.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:50.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:50.463 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:50.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:50.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:50.548 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 46 16 60205 60189 81.7589 67.625 0.00790767 0.0122251 2024-02-05T07:38:50.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:50.596 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:50.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:50.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:50.678 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 6, 3, 2] out_osds: [5, 7, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:38:50.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 0 1 2024-02-05T07:38:50.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:50.725 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:50.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:50.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:50.849 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:50.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:50.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:50.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:50.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:50.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:51.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:50 smithi090 ceph-mon[33213]: 6.7 scrub starts 2024-02-05T07:38:51.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:50 smithi090 ceph-mon[33213]: pgmap v1112: 178 pgs: 2 peering, 4 remapped+peering, 4 active+remapped+backfill_toofull, 168 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-05T07:38:51.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:50 smithi090 ceph-mon[33213]: 6.7 scrub ok 2024-02-05T07:38:51.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:51.125 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:51.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:51.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:50 smithi007 ceph-mon[29098]: 6.7 scrub starts 2024-02-05T07:38:51.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:50 smithi007 ceph-mon[29098]: pgmap v1112: 178 pgs: 2 peering, 4 remapped+peering, 4 active+remapped+backfill_toofull, 168 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-05T07:38:51.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:50 smithi007 ceph-mon[29098]: 6.7 scrub ok 2024-02-05T07:38:51.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:50 smithi007 ceph-mon[37158]: 6.7 scrub starts 2024-02-05T07:38:51.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:50 smithi007 ceph-mon[37158]: pgmap v1112: 178 pgs: 2 peering, 4 remapped+peering, 4 active+remapped+backfill_toofull, 168 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-05T07:38:51.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:50 smithi007 ceph-mon[37158]: 6.7 scrub ok 2024-02-05T07:38:51.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:51.252 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:51.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:51.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:51.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:51.380 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:51.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:51.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:51.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:51.508 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:51.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:51.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:51.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:38:51 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:38:51.548 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 47 16 61333 61317 81.5192 70.5 0.0160382 0.0122611 2024-02-05T07:38:51.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:51.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:51.634 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:51.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:51.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:51.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:51.761 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:51.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:51.823 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:51.828 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:51.866 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:51.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:51.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:51.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:51 smithi007 ceph-mon[29098]: 6.5 scrub starts 2024-02-05T07:38:51.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:51 smithi007 ceph-mon[29098]: 6.5 scrub ok 2024-02-05T07:38:51.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:51 smithi007 ceph-mon[29098]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-05T07:38:51.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:51 smithi007 ceph-mon[37158]: 6.5 scrub starts 2024-02-05T07:38:51.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:51 smithi007 ceph-mon[37158]: 6.5 scrub ok 2024-02-05T07:38:51.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:51 smithi007 ceph-mon[37158]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-05T07:38:52.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:51 smithi090 ceph-mon[33213]: 6.5 scrub starts 2024-02-05T07:38:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:51 smithi090 ceph-mon[33213]: 6.5 scrub ok 2024-02-05T07:38:52.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:51 smithi090 ceph-mon[33213]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-05T07:38:52.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:52.054 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:52.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:52.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:52.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:52.181 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:52.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:52.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:52.307 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:52.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:52.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:52.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:52.435 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:52.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:52.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:52.537 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:52.548 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 48 16 62494 62478 81.3325 72.5625 0.0147907 0.0122899 2024-02-05T07:38:52.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:52.580 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:52.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:52.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:52.721 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:52.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:52.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:38:52.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:52.848 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:52.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:52.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[29098]: 4.2 scrub starts 2024-02-05T07:38:52.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[29098]: 4.2 scrub ok 2024-02-05T07:38:52.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:52.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[29098]: 5.14 scrub starts 2024-02-05T07:38:52.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[29098]: 5.14 scrub ok 2024-02-05T07:38:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[29098]: pgmap v1113: 178 pgs: 2 peering, 7 active+remapped+backfill_toofull, 169 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 891 B/s rd, 63 MiB/s wr, 1.00k op/s; 2/175734 objects misplaced (0.001%) 2024-02-05T07:38:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[29098]: 6.18 scrub starts 2024-02-05T07:38:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[29098]: 6.18 scrub ok 2024-02-05T07:38:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[29098]: 4.a scrub starts 2024-02-05T07:38:52.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[29098]: 4.a scrub ok 2024-02-05T07:38:52.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[37158]: 4.2 scrub starts 2024-02-05T07:38:52.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[37158]: 4.2 scrub ok 2024-02-05T07:38:52.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:52.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[37158]: 5.14 scrub starts 2024-02-05T07:38:52.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[37158]: 5.14 scrub ok 2024-02-05T07:38:52.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[37158]: pgmap v1113: 178 pgs: 2 peering, 7 active+remapped+backfill_toofull, 169 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 891 B/s rd, 63 MiB/s wr, 1.00k op/s; 2/175734 objects misplaced (0.001%) 2024-02-05T07:38:52.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[37158]: 6.18 scrub starts 2024-02-05T07:38:52.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[37158]: 6.18 scrub ok 2024-02-05T07:38:52.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[37158]: 4.a scrub starts 2024-02-05T07:38:52.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:52 smithi007 ceph-mon[37158]: 4.a scrub ok 2024-02-05T07:38:52.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:53.004 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:53.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:53.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:53.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:53.164 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:53.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:53.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:53.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:52 smithi090 ceph-mon[33213]: 4.2 scrub starts 2024-02-05T07:38:53.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:52 smithi090 ceph-mon[33213]: 4.2 scrub ok 2024-02-05T07:38:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:52 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:38:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:52 smithi090 ceph-mon[33213]: 5.14 scrub starts 2024-02-05T07:38:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:52 smithi090 ceph-mon[33213]: 5.14 scrub ok 2024-02-05T07:38:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:52 smithi090 ceph-mon[33213]: pgmap v1113: 178 pgs: 2 peering, 7 active+remapped+backfill_toofull, 169 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 891 B/s rd, 63 MiB/s wr, 1.00k op/s; 2/175734 objects misplaced (0.001%) 2024-02-05T07:38:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:52 smithi090 ceph-mon[33213]: 6.18 scrub starts 2024-02-05T07:38:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:52 smithi090 ceph-mon[33213]: 6.18 scrub ok 2024-02-05T07:38:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:52 smithi090 ceph-mon[33213]: 4.a scrub starts 2024-02-05T07:38:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:52 smithi090 ceph-mon[33213]: 4.a scrub ok 2024-02-05T07:38:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:53.290 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:53.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:53.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:53.416 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:53.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:53.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:53.544 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:53.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:53.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:53.548 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 49 16 63650 63634 81.147 72.25 0.0133476 0.0123177 2024-02-05T07:38:53.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:53.671 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:53.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:53.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:53.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:53.798 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:53.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:53.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:53.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:38:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:38:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:38:53.926 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:53.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:54.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:54.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:54.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:54.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:54.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:54.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:54.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:54.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:54.317 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:54.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:54.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:54.445 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:54.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:54.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:54.550 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 50 16 64877 64861 81.0544 76.6875 0.0175885 0.0123314 2024-02-05T07:38:54.572 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:54.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:54.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:54.699 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:54.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:54.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:54.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:54.826 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:54.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:54.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:54.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:54 smithi007 ceph-mon[29098]: 4.f scrub starts 2024-02-05T07:38:54.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:54 smithi007 ceph-mon[29098]: 4.f scrub ok 2024-02-05T07:38:54.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:54 smithi007 ceph-mon[29098]: 5.17 scrub starts 2024-02-05T07:38:54.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:54 smithi007 ceph-mon[29098]: 5.17 scrub ok 2024-02-05T07:38:54.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:54 smithi007 ceph-mon[29098]: pgmap v1114: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 762 B/s rd, 70 MiB/s wr, 1.12k op/s; 2/181119 objects misplaced (0.001%) 2024-02-05T07:38:54.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:54 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:38:54.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:54 smithi007 ceph-mon[37158]: 4.f scrub starts 2024-02-05T07:38:54.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:54 smithi007 ceph-mon[37158]: 4.f scrub ok 2024-02-05T07:38:54.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:54 smithi007 ceph-mon[37158]: 5.17 scrub starts 2024-02-05T07:38:54.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:54 smithi007 ceph-mon[37158]: 5.17 scrub ok 2024-02-05T07:38:54.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:54 smithi007 ceph-mon[37158]: pgmap v1114: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 762 B/s rd, 70 MiB/s wr, 1.12k op/s; 2/181119 objects misplaced (0.001%) 2024-02-05T07:38:54.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:54 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:38:54.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:54.952 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:54.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:54.974 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:38:54.974 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:38:55.054 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:55.103 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:55.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:55.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:55.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:55.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:55.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:54 smithi090 ceph-mon[33213]: 4.f scrub starts 2024-02-05T07:38:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:54 smithi090 ceph-mon[33213]: 4.f scrub ok 2024-02-05T07:38:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:54 smithi090 ceph-mon[33213]: 5.17 scrub starts 2024-02-05T07:38:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:54 smithi090 ceph-mon[33213]: 5.17 scrub ok 2024-02-05T07:38:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:54 smithi090 ceph-mon[33213]: pgmap v1114: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 762 B/s rd, 70 MiB/s wr, 1.12k op/s; 2/181119 objects misplaced (0.001%) 2024-02-05T07:38:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:54 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:38:55.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:55.368 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:55.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:55.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:55.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:55.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:55.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:55.551 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 51 16 66041 66025 80.8915 72.75 0.0114055 0.0123563 2024-02-05T07:38:55.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:55.665 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:55.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:55.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:55.791 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:55.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:55.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:55.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:55.936 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:55.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:55.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:56.034 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[29098]: 5.1d scrub starts 2024-02-05T07:38:56.034 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[29098]: 5.1d scrub ok 2024-02-05T07:38:56.034 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[29098]: 5.13 scrub starts 2024-02-05T07:38:56.034 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[29098]: 5.13 scrub ok 2024-02-05T07:38:56.034 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[29098]: 4.6 scrub starts 2024-02-05T07:38:56.035 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[29098]: 4.6 scrub ok 2024-02-05T07:38:56.035 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[29098]: 4.1f scrub starts 2024-02-05T07:38:56.035 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[29098]: 4.1f scrub ok 2024-02-05T07:38:56.035 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[37158]: 5.1d scrub starts 2024-02-05T07:38:56.035 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[37158]: 5.1d scrub ok 2024-02-05T07:38:56.035 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[37158]: 5.13 scrub starts 2024-02-05T07:38:56.035 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[37158]: 5.13 scrub ok 2024-02-05T07:38:56.035 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[37158]: 4.6 scrub starts 2024-02-05T07:38:56.035 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[37158]: 4.6 scrub ok 2024-02-05T07:38:56.035 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[37158]: 4.1f scrub starts 2024-02-05T07:38:56.035 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:55 smithi007 ceph-mon[37158]: 4.1f scrub ok 2024-02-05T07:38:56.039 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:56.088 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:56.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:56.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:56.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:56.232 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:56.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:55 smithi090 ceph-mon[33213]: 5.1d scrub starts 2024-02-05T07:38:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:55 smithi090 ceph-mon[33213]: 5.1d scrub ok 2024-02-05T07:38:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:55 smithi090 ceph-mon[33213]: 5.13 scrub starts 2024-02-05T07:38:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:55 smithi090 ceph-mon[33213]: 5.13 scrub ok 2024-02-05T07:38:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:55 smithi090 ceph-mon[33213]: 4.6 scrub starts 2024-02-05T07:38:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:55 smithi090 ceph-mon[33213]: 4.6 scrub ok 2024-02-05T07:38:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:55 smithi090 ceph-mon[33213]: 4.1f scrub starts 2024-02-05T07:38:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:55 smithi090 ceph-mon[33213]: 4.1f scrub ok 2024-02-05T07:38:56.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:56.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:56.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:56.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:56.501 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:56.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:56.551 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 52 16 67097 67081 80.6051 66 0.00554686 0.0124005 2024-02-05T07:38:56.602 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:56.628 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:56.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:56.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:56.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:56.757 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:56.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:56.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T07:38:56.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:56.881 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:56.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:56.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:57.007 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:57.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:57.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:57.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:57.116 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[37158]: 3.16 scrub starts 2024-02-05T07:38:57.116 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[37158]: 3.16 scrub ok 2024-02-05T07:38:57.116 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[37158]: 4.16 scrub starts 2024-02-05T07:38:57.116 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[37158]: 4.16 scrub ok 2024-02-05T07:38:57.116 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[37158]: pgmap v1115: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 3.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 2/188733 objects misplaced (0.001%) 2024-02-05T07:38:57.116 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[37158]: 6.f scrub starts 2024-02-05T07:38:57.116 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[37158]: 3.1d scrub starts 2024-02-05T07:38:57.116 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[37158]: 3.1d scrub ok 2024-02-05T07:38:57.117 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[29098]: 3.16 scrub starts 2024-02-05T07:38:57.118 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[29098]: 3.16 scrub ok 2024-02-05T07:38:57.118 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[29098]: 4.16 scrub starts 2024-02-05T07:38:57.118 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[29098]: 4.16 scrub ok 2024-02-05T07:38:57.118 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[29098]: pgmap v1115: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 3.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 2/188733 objects misplaced (0.001%) 2024-02-05T07:38:57.118 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[29098]: 6.f scrub starts 2024-02-05T07:38:57.118 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[29098]: 3.1d scrub starts 2024-02-05T07:38:57.118 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:56 smithi007 ceph-mon[29098]: 3.1d scrub ok 2024-02-05T07:38:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:57.170 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:57.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:57.254 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:57.272 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:57.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:56 smithi090 ceph-mon[33213]: 3.16 scrub starts 2024-02-05T07:38:57.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:56 smithi090 ceph-mon[33213]: 3.16 scrub ok 2024-02-05T07:38:57.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:56 smithi090 ceph-mon[33213]: 4.16 scrub starts 2024-02-05T07:38:57.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:56 smithi090 ceph-mon[33213]: 4.16 scrub ok 2024-02-05T07:38:57.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:56 smithi090 ceph-mon[33213]: pgmap v1115: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 3.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 2/188733 objects misplaced (0.001%) 2024-02-05T07:38:57.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:56 smithi090 ceph-mon[33213]: 6.f scrub starts 2024-02-05T07:38:57.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:56 smithi090 ceph-mon[33213]: 3.1d scrub starts 2024-02-05T07:38:57.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:56 smithi090 ceph-mon[33213]: 3.1d scrub ok 2024-02-05T07:38:57.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:57.297 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:57.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:57.385 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:38:57.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:57.449 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:38:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:57.450 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:57.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:57.501 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:38:57.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:57.552 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 53 16 68059 68043 80.2184 60.125 0.0260409 0.0124588 2024-02-05T07:38:57.559 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:38:57.603 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:38:57.606 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:57.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:57.640 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:38:57.678 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:38:57.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:38:57.720 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:38:57.726 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:38:57.726 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:38:57.726 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:38:57.726 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:38:57.726 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:38:57.727 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:38:57.727 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:38:57.727 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:38:57.733 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:57.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:57.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:38:57.815 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.0 to 1 (10000) 2024-02-05T07:38:57.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:38:57.859 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:57.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:57.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:38:57.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:57 smithi007 ceph-mon[29098]: 3.f deep-scrub starts 2024-02-05T07:38:57.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:57 smithi007 ceph-mon[29098]: 3.f deep-scrub ok 2024-02-05T07:38:57.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:57 smithi007 ceph-mon[29098]: 6.f scrub ok 2024-02-05T07:38:57.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:57 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1387023242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:57.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:57 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2921797255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:38:57.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:57 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:38:57.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:57 smithi007 ceph-mon[37158]: 3.f deep-scrub starts 2024-02-05T07:38:57.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:57 smithi007 ceph-mon[37158]: 3.f deep-scrub ok 2024-02-05T07:38:57.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:57 smithi007 ceph-mon[37158]: 6.f scrub ok 2024-02-05T07:38:57.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:57 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1387023242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:57.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:57 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2921797255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:38:57.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:57 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:38:57.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:58.003 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:58.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:58.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:58.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:58.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:58.174 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:57 smithi090 ceph-mon[33213]: 3.f deep-scrub starts 2024-02-05T07:38:58.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:57 smithi090 ceph-mon[33213]: 3.f deep-scrub ok 2024-02-05T07:38:58.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:57 smithi090 ceph-mon[33213]: 6.f scrub ok 2024-02-05T07:38:58.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:57 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1387023242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:38:58.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:57 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2921797255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:38:58.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:57 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:38:58.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:58.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:58.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:58.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:58.425 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:58.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:58.484 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:38:58.527 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:58.551 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 54 16 69147 69131 79.9919 68 0.0161297 0.0124951 2024-02-05T07:38:58.579 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:58.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:58.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:58.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:58.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:58.820 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:58.865 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:58.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:58.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[29098]: pgmap v1116: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 2/198366 objects misplaced (0.001%) 2024-02-05T07:38:58.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[29098]: 3.7 scrub starts 2024-02-05T07:38:58.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[29098]: 3.7 scrub ok 2024-02-05T07:38:58.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[29098]: 3.0 scrub starts 2024-02-05T07:38:58.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[29098]: 3.0 scrub ok 2024-02-05T07:38:58.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T07:38:58.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[29098]: osdmap e615: 8 total, 8 up, 5 in 2024-02-05T07:38:58.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[29098]: 1.12 scrub starts 2024-02-05T07:38:58.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[29098]: 1.12 scrub ok 2024-02-05T07:38:58.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[37158]: pgmap v1116: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 2/198366 objects misplaced (0.001%) 2024-02-05T07:38:58.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[37158]: 3.7 scrub starts 2024-02-05T07:38:58.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[37158]: 3.7 scrub ok 2024-02-05T07:38:58.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[37158]: 3.0 scrub starts 2024-02-05T07:38:58.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[37158]: 3.0 scrub ok 2024-02-05T07:38:58.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T07:38:58.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[37158]: osdmap e615: 8 total, 8 up, 5 in 2024-02-05T07:38:58.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[37158]: 1.12 scrub starts 2024-02-05T07:38:58.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:58 smithi007 ceph-mon[37158]: 1.12 scrub ok 2024-02-05T07:38:58.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:38:58.996 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:58.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:38:59.057 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:58 smithi090 ceph-mon[33213]: pgmap v1116: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 2/198366 objects misplaced (0.001%) 2024-02-05T07:38:59.057 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:58 smithi090 ceph-mon[33213]: 3.7 scrub starts 2024-02-05T07:38:59.057 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:58 smithi090 ceph-mon[33213]: 3.7 scrub ok 2024-02-05T07:38:59.057 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:58 smithi090 ceph-mon[33213]: 3.0 scrub starts 2024-02-05T07:38:59.057 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:58 smithi090 ceph-mon[33213]: 3.0 scrub ok 2024-02-05T07:38:59.057 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:58 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T07:38:59.057 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:58 smithi090 ceph-mon[33213]: osdmap e615: 8 total, 8 up, 5 in 2024-02-05T07:38:59.057 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:58 smithi090 ceph-mon[33213]: 1.12 scrub starts 2024-02-05T07:38:59.057 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:58 smithi090 ceph-mon[33213]: 1.12 scrub ok 2024-02-05T07:38:59.098 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:38:59.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:59.148 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:59.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:38:59.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:59.292 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:59.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:59.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:59.394 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:38:59.447 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 1 1 2024-02-05T07:38:59.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:59.456 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:59.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:38:59.551 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 55 16 70270 70254 79.8136 70.1875 0.0174217 0.0125232 2024-02-05T07:38:59.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:38:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:59.611 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:59.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:38:59.713 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:38:59.739 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:38:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:59.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:38:59.841 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:38:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:38:59.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:38:59.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:38:59.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:00.037 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:00.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:00.039 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:59 smithi007 ceph-mon[37158]: 6.1b deep-scrub starts 2024-02-05T07:39:00.039 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:59 smithi007 ceph-mon[37158]: 6.1b deep-scrub ok 2024-02-05T07:39:00.040 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:59 smithi007 ceph-mon[37158]: 5.10 scrub starts 2024-02-05T07:39:00.040 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:59 smithi007 ceph-mon[37158]: 5.10 scrub ok 2024-02-05T07:39:00.040 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:38:59 smithi007 ceph-mon[37158]: osdmap e616: 8 total, 8 up, 5 in 2024-02-05T07:39:00.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:59 smithi007 ceph-mon[29098]: 6.1b deep-scrub starts 2024-02-05T07:39:00.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:59 smithi007 ceph-mon[29098]: 6.1b deep-scrub ok 2024-02-05T07:39:00.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:59 smithi007 ceph-mon[29098]: 5.10 scrub starts 2024-02-05T07:39:00.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:59 smithi007 ceph-mon[29098]: 5.10 scrub ok 2024-02-05T07:39:00.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:38:59 smithi007 ceph-mon[29098]: osdmap e616: 8 total, 8 up, 5 in 2024-02-05T07:39:00.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:00.182 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:00.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:00.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:00.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:59 smithi090 ceph-mon[33213]: 6.1b deep-scrub starts 2024-02-05T07:39:00.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:59 smithi090 ceph-mon[33213]: 6.1b deep-scrub ok 2024-02-05T07:39:00.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:59 smithi090 ceph-mon[33213]: 5.10 scrub starts 2024-02-05T07:39:00.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:59 smithi090 ceph-mon[33213]: 5.10 scrub ok 2024-02-05T07:39:00.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:38:59 smithi090 ceph-mon[33213]: osdmap e616: 8 total, 8 up, 5 in 2024-02-05T07:39:00.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:00.322 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:00.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:00.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:00.464 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:00.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:00.551 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 56 16 71438 71422 79.6918 73 0.0105269 0.0125436 2024-02-05T07:39:00.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:00.606 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:00.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:00.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:00.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:00.736 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:00.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:00.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:00.881 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:00.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:00.982 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:01.026 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:01.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:01.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:01.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:01.135 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:00 smithi090 ceph-mon[33213]: pgmap v1118: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 9.4 KiB/s rd, 71 MiB/s wr, 1.16k op/s; 2/202503 objects misplaced (0.001%) 2024-02-05T07:39:01.135 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:00 smithi090 ceph-mon[33213]: 3.11 scrub starts 2024-02-05T07:39:01.135 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:00 smithi090 ceph-mon[33213]: 3.11 scrub ok 2024-02-05T07:39:01.153 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:01.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:01.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:01.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:00 smithi007 ceph-mon[29098]: pgmap v1118: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 9.4 KiB/s rd, 71 MiB/s wr, 1.16k op/s; 2/202503 objects misplaced (0.001%) 2024-02-05T07:39:01.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:00 smithi007 ceph-mon[29098]: 3.11 scrub starts 2024-02-05T07:39:01.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:00 smithi007 ceph-mon[29098]: 3.11 scrub ok 2024-02-05T07:39:01.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:00 smithi007 ceph-mon[37158]: pgmap v1118: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 9.4 KiB/s rd, 71 MiB/s wr, 1.16k op/s; 2/202503 objects misplaced (0.001%) 2024-02-05T07:39:01.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:00 smithi007 ceph-mon[37158]: 3.11 scrub starts 2024-02-05T07:39:01.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:00 smithi007 ceph-mon[37158]: 3.11 scrub ok 2024-02-05T07:39:01.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:01.280 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:01.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:01.381 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:01.423 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:01.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:01.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:01.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:39:01 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:39:01.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:01.552 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 57 16 72657 72641 79.6302 76.1875 0.0166666 0.0125523 2024-02-05T07:39:01.568 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:01.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:01.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:01.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:01.696 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:01.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:01.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:01.835 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:01.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:01.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:01.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:01.983 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:01.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:01.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:02.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:02.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:02.133 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:02.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:02.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:02.279 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:02.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:02.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:02.381 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:02.407 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:02.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:02.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:39:02.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:39:02.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:02.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[29098]: pgmap v1120: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 31 KiB/s rd, 74 MiB/s wr, 1.24k op/s; 2/209685 objects misplaced (0.001%) 2024-02-05T07:39:02.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[29098]: 6.16 scrub starts 2024-02-05T07:39:02.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[29098]: 6.16 scrub ok 2024-02-05T07:39:02.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:39:02.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:39:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:39:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:39:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[37158]: pgmap v1120: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 31 KiB/s rd, 74 MiB/s wr, 1.24k op/s; 2/209685 objects misplaced (0.001%) 2024-02-05T07:39:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[37158]: 6.16 scrub starts 2024-02-05T07:39:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[37158]: 6.16 scrub ok 2024-02-05T07:39:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:39:02.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:02 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:39:02.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:02.523 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:02 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:39:02.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:02 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:39:02.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:02 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:02.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:02 smithi090 ceph-mon[33213]: pgmap v1120: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 31 KiB/s rd, 74 MiB/s wr, 1.24k op/s; 2/209685 objects misplaced (0.001%) 2024-02-05T07:39:02.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:02 smithi090 ceph-mon[33213]: 6.16 scrub starts 2024-02-05T07:39:02.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:02 smithi090 ceph-mon[33213]: 6.16 scrub ok 2024-02-05T07:39:02.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:02 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:39:02.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:02 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:39:02.538 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:02.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:02.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:02.551 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 58 16 73852 73836 79.5449 74.6875 0.0102712 0.0125655 2024-02-05T07:39:02.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:02.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:02.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:02.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:02.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:02.809 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:02.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:02.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:02.952 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:02.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:02.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:03.054 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:03.098 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:03.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:03.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:03.163 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:03 smithi007 ceph-mon[29098]: osdmap e617: 8 total, 8 up, 5 in 2024-02-05T07:39:03.163 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:03 smithi007 ceph-mon[37158]: osdmap e617: 8 total, 8 up, 5 in 2024-02-05T07:39:03.200 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:03.248 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:03.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:03.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:03.350 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:03.376 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:03.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:03.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:03.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:03.503 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:03.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:03.523 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:03 smithi090 ceph-mon[33213]: osdmap e617: 8 total, 8 up, 5 in 2024-02-05T07:39:03.552 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 59 16 74822 74806 79.2242 60.625 0.00515681 0.0126164 2024-02-05T07:39:03.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:03.653 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:03.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:03.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:03.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:03.800 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:03.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:03.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:03.922 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:39:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:39:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:39:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:03.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:04.045 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:04.071 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:04.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:04.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:04.170 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:39:04.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:04.197 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:04.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:04.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:04.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:04.326 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:04.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:04.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:04.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[29098]: 11.c scrub starts 2024-02-05T07:39:04.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[29098]: pgmap v1122: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 63 KiB/s rd, 57 MiB/s wr, 1.02k op/s; 2/214893 objects misplaced (0.001%) 2024-02-05T07:39:04.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[29098]: 6.6 scrub starts 2024-02-05T07:39:04.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[29098]: 6.6 scrub ok 2024-02-05T07:39:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[29098]: 5.d scrub starts 2024-02-05T07:39:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[29098]: 5.d scrub ok 2024-02-05T07:39:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/847656818' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:39:04.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:39:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[37158]: 11.c scrub starts 2024-02-05T07:39:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[37158]: pgmap v1122: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 63 KiB/s rd, 57 MiB/s wr, 1.02k op/s; 2/214893 objects misplaced (0.001%) 2024-02-05T07:39:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[37158]: 6.6 scrub starts 2024-02-05T07:39:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[37158]: 6.6 scrub ok 2024-02-05T07:39:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[37158]: 5.d scrub starts 2024-02-05T07:39:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[37158]: 5.d scrub ok 2024-02-05T07:39:04.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/847656818' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:39:04.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:04 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:39:04.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:04.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:04.455 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:04.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:04.523 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:04 smithi090 ceph-mon[33213]: 11.c scrub starts 2024-02-05T07:39:04.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:04 smithi090 ceph-mon[33213]: pgmap v1122: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 63 KiB/s rd, 57 MiB/s wr, 1.02k op/s; 2/214893 objects misplaced (0.001%) 2024-02-05T07:39:04.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:04 smithi090 ceph-mon[33213]: 6.6 scrub starts 2024-02-05T07:39:04.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:04 smithi090 ceph-mon[33213]: 6.6 scrub ok 2024-02-05T07:39:04.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:04 smithi090 ceph-mon[33213]: 5.d scrub starts 2024-02-05T07:39:04.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:04 smithi090 ceph-mon[33213]: 5.d scrub ok 2024-02-05T07:39:04.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:04 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/847656818' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:39:04.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:04 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:39:04.551 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:39:04.551996+0000 min lat: 0.00118746 max lat: 1.24847 avg lat: 0.0126364 2024-02-05T07:39:04.551 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:39:04.552 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 60 16 75974 75958 79.1037 72 0.0141502 0.0126364 2024-02-05T07:39:04.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:04.581 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:04.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:04.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:04.709 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:04.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:04.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:04.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:04.854 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:04.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:04.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:04.982 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:04.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:05.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:05.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:05.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:05.265 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:05.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:05.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:05.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:05.372 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:39:05.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:05 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:39:05.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:05 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:39:05.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:05 smithi007 ceph-mon[37158]: osdmap e618: 8 total, 8 up, 5 in 2024-02-05T07:39:05.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:05 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:39:05.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:05 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:39:05.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:05 smithi007 ceph-mon[29098]: osdmap e618: 8 total, 8 up, 5 in 2024-02-05T07:39:05.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:05.433 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:05.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:05.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:05 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:39:05.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:05 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:39:05.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:05 smithi090 ceph-mon[33213]: osdmap e618: 8 total, 8 up, 5 in 2024-02-05T07:39:05.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:05.552 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 61 16 77195 77179 79.0568 76.3125 0.0163276 0.0126435 2024-02-05T07:39:05.584 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:05.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:05.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:05.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:05.732 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:05.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:05.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:05.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:05.892 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:05.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:05.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:05.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:06.034 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:06.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:06.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:06.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:06.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:06.180 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:06.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:06.195 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:39:06.282 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:06.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:06.309 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:06.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:06.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:06.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:06 smithi007 ceph-mon[29098]: pgmap v1124: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 59 KiB/s rd, 69 MiB/s wr, 1.20k op/s; 2/222300 objects misplaced (0.001%) 2024-02-05T07:39:06.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:06 smithi007 ceph-mon[37158]: pgmap v1124: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 59 KiB/s rd, 69 MiB/s wr, 1.20k op/s; 2/222300 objects misplaced (0.001%) 2024-02-05T07:39:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:06.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:06.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:06.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:06 smithi090 ceph-mon[33213]: pgmap v1124: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 59 KiB/s rd, 69 MiB/s wr, 1.20k op/s; 2/222300 objects misplaced (0.001%) 2024-02-05T07:39:06.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:06.555 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 62 16 78356 78340 78.9488 72.5625 0.0124806 0.0126614 2024-02-05T07:39:06.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:06.588 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:06.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:06.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:06.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:06.720 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:06.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:06.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:06.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:06.848 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:06.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:06.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:39:06.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:07.024 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:07.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:07.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:07.126 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:07.151 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:07.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:07.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:07.278 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:07.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:07.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:07.405 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:07.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:07.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:07.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:07.532 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:07.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:07.555 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 63 16 79370 79354 78.7015 63.375 0.00712457 0.0126999 2024-02-05T07:39:07.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:07.673 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:07.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:07.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:07.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:07.801 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:07.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:07.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:07.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:07.929 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:07.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:08.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:08.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:08.057 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:08.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:08.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:08.184 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:08.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:08.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:08.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:08.317 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:08.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:08.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:08.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:08.444 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:08.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:08.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:08.555 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 64 16 80560 80544 78.6339 74.375 0.013589 0.0127096 2024-02-05T07:39:08.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:08.570 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:08.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:08.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:08.711 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:08.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:08.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:08.773 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:08 smithi090 ceph-mon[33213]: pgmap v1125: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.7 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 46 KiB/s rd, 79 MiB/s wr, 1.34k op/s; 2/231999 objects misplaced (0.001%) 2024-02-05T07:39:08.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:08.841 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:08.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:08.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:08.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:08 smithi007 ceph-mon[29098]: pgmap v1125: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.7 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 46 KiB/s rd, 79 MiB/s wr, 1.34k op/s; 2/231999 objects misplaced (0.001%) 2024-02-05T07:39:08.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:08 smithi007 ceph-mon[37158]: pgmap v1125: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.7 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 46 KiB/s rd, 79 MiB/s wr, 1.34k op/s; 2/231999 objects misplaced (0.001%) 2024-02-05T07:39:08.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:08.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:08.986 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:08.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:09.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:09.113 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:09.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:09.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:09.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:09.256 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:09.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:09.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:09.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:09.384 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:09.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:09.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:09.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:09.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:09.510 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:09.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:09.555 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 65 16 81723 81707 78.5423 72.6875 0.00667385 0.0127266 2024-02-05T07:39:09.612 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:09.636 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:09.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:09.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:09.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:09.762 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:09.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:09.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:09.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:09.890 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:09.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:09.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:09.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:10.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:10.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:10.160 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:10.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:10.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:10.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:10.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:10.306 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:10.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:10.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:10.431 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:10.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:10.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:10.555 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 66 15 82837 82822 78.408 69.6875 0.0108603 0.0127493 2024-02-05T07:39:10.559 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:10.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:10.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:10.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:10.687 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:10.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:10.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:10.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:10.831 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:10.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:10.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:10 smithi007 ceph-mon[29098]: pgmap v1126: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 24 KiB/s rd, 69 MiB/s wr, 1.15k op/s; 2/236331 objects misplaced (0.001%) 2024-02-05T07:39:10.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:10 smithi007 ceph-mon[37158]: pgmap v1126: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 24 KiB/s rd, 69 MiB/s wr, 1.15k op/s; 2/236331 objects misplaced (0.001%) 2024-02-05T07:39:10.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:10.978 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:10.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:11.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:10 smithi090 ceph-mon[33213]: pgmap v1126: 178 pgs: 7 active+remapped+backfill_toofull, 171 active+clean; 4.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 24 KiB/s rd, 69 MiB/s wr, 1.15k op/s; 2/236331 objects misplaced (0.001%) 2024-02-05T07:39:11.078 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:39:11.078 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-02-05T07:39:11.078 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-02-05T07:39:11.079 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:39:11.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.079 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:142776 10.12 10:4ee87194:::benchmark_data_smithi007_157098_object13846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-02-05T07:39:11.079 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:31.907061+0000", 2024-02-05T07:39:11.079 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 219.16782544099999, 2024-02-05T07:39:11.079 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.15728530499999999, 2024-02-05T07:39:11.079 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.079 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.079 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:11.079 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:39:11.079 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:39:11.079 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 142776 2024-02-05T07:39:11.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.080 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907061+0000", 2024-02-05T07:39:11.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907061+0000", 2024-02-05T07:39:11.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907063+0000", 2024-02-05T07:39:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6020000000000001e-06 2024-02-05T07:39:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907068+0000", 2024-02-05T07:39:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.434e-06 2024-02-05T07:39:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907080+0000", 2024-02-05T07:39:11.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1977000000000001e-05 2024-02-05T07:39:11.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.082 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.082 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907089+0000", 2024-02-05T07:39:11.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1389999999999995e-06 2024-02-05T07:39:11.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.082 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.082 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907798+0000", 2024-02-05T07:39:11.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00070967800000000004 2024-02-05T07:39:11.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.907937+0000", 2024-02-05T07:39:11.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013838899999999999 2024-02-05T07:39:11.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,3,4,6", 2024-02-05T07:39:11.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.908227+0000", 2024-02-05T07:39:11.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028989700000000001 2024-02-05T07:39:11.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:11.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.909450+0000", 2024-02-05T07:39:11.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012234520000000001 2024-02-05T07:39:11.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.084 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.084 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.913246+0000", 2024-02-05T07:39:11.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037962099999999999 2024-02-05T07:39:11.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.084 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.084 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.913297+0000", 2024-02-05T07:39:11.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0652999999999998e-05 2024-02-05T07:39:11.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.913341+0000", 2024-02-05T07:39:11.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4206999999999999e-05 2024-02-05T07:39:11.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.913805+0000", 2024-02-05T07:39:11.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00046370599999999998 2024-02-05T07:39:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.063525+0000", 2024-02-05T07:39:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14971975700000001 2024-02-05T07:39:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.064187+0000", 2024-02-05T07:39:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00066257000000000002 2024-02-05T07:39:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.087 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.064347+0000", 2024-02-05T07:39:11.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000159634 2024-02-05T07:39:11.087 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.087 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.087 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.087 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:142777 10.12 10:4f2b3996:::benchmark_data_smithi007_157098_object13847:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-02-05T07:39:11.087 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:31.910713+0000", 2024-02-05T07:39:11.087 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 219.16417423499999, 2024-02-05T07:39:11.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16056696200000001, 2024-02-05T07:39:11.087 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.087 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.088 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:11.088 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:39:11.088 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:39:11.088 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 142777 2024-02-05T07:39:11.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.088 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.088 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.088 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.910713+0000", 2024-02-05T07:39:11.088 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.088 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.089 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.910713+0000", 2024-02-05T07:39:11.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.089 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.910714+0000", 2024-02-05T07:39:11.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.378e-06 2024-02-05T07:39:11.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.089 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.910721+0000", 2024-02-05T07:39:11.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8279999999999997e-06 2024-02-05T07:39:11.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.910733+0000", 2024-02-05T07:39:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1854e-05 2024-02-05T07:39:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.910755+0000", 2024-02-05T07:39:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1997999999999998e-05 2024-02-05T07:39:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.910838+0000", 2024-02-05T07:39:11.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3765999999999997e-05 2024-02-05T07:39:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.911102+0000", 2024-02-05T07:39:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000263396 2024-02-05T07:39:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,3,4,6", 2024-02-05T07:39:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.911410+0000", 2024-02-05T07:39:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030852000000000001 2024-02-05T07:39:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.913445+0000", 2024-02-05T07:39:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020343739999999998 2024-02-05T07:39:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.913845+0000", 2024-02-05T07:39:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000400216 2024-02-05T07:39:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.913886+0000", 2024-02-05T07:39:11.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1032999999999998e-05 2024-02-05T07:39:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.917094+0000", 2024-02-05T07:39:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032076700000000001 2024-02-05T07:39:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.917112+0000", 2024-02-05T07:39:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8170000000000001e-05 2024-02-05T07:39:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:11.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.071032+0000", 2024-02-05T07:39:11.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.15391991399999999 2024-02-05T07:39:11.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.071138+0000", 2024-02-05T07:39:11.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010628600000000001 2024-02-05T07:39:11.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.071280+0000", 2024-02-05T07:39:11.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000141559 2024-02-05T07:39:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:142781 10.12 10:4dd68cc0:::benchmark_data_smithi007_157098_object13851:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-02-05T07:39:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:31.912235+0000", 2024-02-05T07:39:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 219.16265161600001, 2024-02-05T07:39:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.15995722300000001, 2024-02-05T07:39:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:39:11.096 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 142781 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.912235+0000", 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.912235+0000", 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.912236+0000", 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.15e-06 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.912241+0000", 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7199999999999997e-06 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.912252+0000", 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1028e-05 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.912260+0000", 2024-02-05T07:39:11.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6839999999999996e-06 2024-02-05T07:39:11.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.914094+0000", 2024-02-05T07:39:11.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0018345620000000001 2024-02-05T07:39:11.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.914218+0000", 2024-02-05T07:39:11.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012376600000000001 2024-02-05T07:39:11.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,3,4,6", 2024-02-05T07:39:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.914480+0000", 2024-02-05T07:39:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026203000000000001 2024-02-05T07:39:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.917221+0000", 2024-02-05T07:39:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002740503 2024-02-05T07:39:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.917384+0000", 2024-02-05T07:39:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016310399999999999 2024-02-05T07:39:11.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.917991+0000", 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00060672500000000002 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.918319+0000", 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032889600000000002 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.918360+0000", 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0249e-05 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.071995+0000", 2024-02-05T07:39:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.153635825 2024-02-05T07:39:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.072058+0000", 2024-02-05T07:39:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2402000000000003e-05 2024-02-05T07:39:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.072192+0000", 2024-02-05T07:39:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013457899999999999 2024-02-05T07:39:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.103 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.103 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.103 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:142793 10.d e506/423 10:b95366dc:::benchmark_data_smithi007_157098_object13863:head v 506'8875, mlcod=506'8873)", 2024-02-05T07:39:11.103 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:31.920032+0000", 2024-02-05T07:39:11.103 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 219.15485441000001, 2024-02-05T07:39:11.103 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14482795200000001, 2024-02-05T07:39:11.103 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.103 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.103 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.103 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920032+0000", 2024-02-05T07:39:11.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920032+0000", 2024-02-05T07:39:11.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920033+0000", 2024-02-05T07:39:11.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1800000000000002e-07 2024-02-05T07:39:11.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920037+0000", 2024-02-05T07:39:11.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1579999999999998e-06 2024-02-05T07:39:11.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920045+0000", 2024-02-05T07:39:11.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5299999999999996e-06 2024-02-05T07:39:11.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920052+0000", 2024-02-05T07:39:11.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9979999999999997e-06 2024-02-05T07:39:11.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920106+0000", 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3356000000000002e-05 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920127+0000", 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0766000000000001e-05 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.064820+0000", 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14469389699999999 2024-02-05T07:39:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.064860+0000", 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9928999999999997e-05 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "MOSDPGPushReply(10.16 506/496 [PushReplyOp(10:6b6bf244:::benchmark_data_smithi007_157098_object58135:head)])", 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:31.920064+0000", 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 219.154822846, 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14048513200000001, 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.107 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:39:11.108 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920064+0000", 2024-02-05T07:39:11.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920064+0000", 2024-02-05T07:39:11.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920064+0000", 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1100000000000002e-07 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920067+0000", 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1750000000000001e-06 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920075+0000", 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.238e-06 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920081+0000", 2024-02-05T07:39:11.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2369999999999997e-06 2024-02-05T07:39:11.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920120+0000", 2024-02-05T07:39:11.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8860999999999999e-05 2024-02-05T07:39:11.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.060549+0000", 2024-02-05T07:39:11.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14042931 2024-02-05T07:39:11.111 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.111 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.111 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.111 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:142796 10.10 e506/496 10:08cf4633:::benchmark_data_smithi007_157098_object13866:head v 506'8437, mlcod=506'8435)", 2024-02-05T07:39:11.111 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:31.920963+0000", 2024-02-05T07:39:11.111 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 219.15392409200001, 2024-02-05T07:39:11.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14140070699999999, 2024-02-05T07:39:11.111 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.111 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.111 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.111 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920963+0000", 2024-02-05T07:39:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920963+0000", 2024-02-05T07:39:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920963+0000", 2024-02-05T07:39:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4799999999999998e-07 2024-02-05T07:39:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:31.920970+0000", 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.81e-06 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.060394+0000", 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.13942379699999999 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.060407+0000", 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3054000000000001e-05 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.060525+0000", 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000118078 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.060550+0000", 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5344e-05 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.062318+0000", 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017671570000000001 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:32.062363+0000", 2024-02-05T07:39:11.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5819000000000003e-05 2024-02-05T07:39:11.115 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.115 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.115 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.115 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:160264 10.12 10:4e54d21a:::benchmark_data_smithi007_157098_object31334:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-02-05T07:39:11.115 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:49.708086+0000", 2024-02-05T07:39:11.115 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 201.366800977, 2024-02-05T07:39:11.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.142616413, 2024-02-05T07:39:11.115 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.115 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.115 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:11.116 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:39:11.116 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:39:11.116 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 160264 2024-02-05T07:39:11.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.116 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.116 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.116 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.708086+0000", 2024-02-05T07:39:11.116 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.116 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.116 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.708086+0000", 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.708087+0000", 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.531e-06 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.708093+0000", 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7579999999999998e-06 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.708104+0000", 2024-02-05T07:39:11.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1260000000000001e-05 2024-02-05T07:39:11.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.708113+0000", 2024-02-05T07:39:11.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3040000000000005e-06 2024-02-05T07:39:11.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.710237+0000", 2024-02-05T07:39:11.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021241329999999998 2024-02-05T07:39:11.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.710365+0000", 2024-02-05T07:39:11.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000127928 2024-02-05T07:39:11.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,3,4,6", 2024-02-05T07:39:11.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.710637+0000", 2024-02-05T07:39:11.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00027227499999999999 2024-02-05T07:39:11.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:11.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.711977+0000", 2024-02-05T07:39:11.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0013402010000000001 2024-02-05T07:39:11.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.712348+0000", 2024-02-05T07:39:11.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00037092700000000001 2024-02-05T07:39:11.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.712387+0000", 2024-02-05T07:39:11.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8436000000000003e-05 2024-02-05T07:39:11.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.713949+0000", 2024-02-05T07:39:11.121 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001562169 2024-02-05T07:39:11.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.121 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.715969+0000", 2024-02-05T07:39:11.121 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020206159999999998 2024-02-05T07:39:11.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.121 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.718719+0000", 2024-02-05T07:39:11.121 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0027495919999999999 2024-02-05T07:39:11.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.122 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.122 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.122 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.718874+0000", 2024-02-05T07:39:11.122 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000155014 2024-02-05T07:39:11.123 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.123 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.123 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.850702+0000", 2024-02-05T07:39:11.123 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.131828269 2024-02-05T07:39:11.123 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.123 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.123 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.124 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:160268 10.14 10:2a3c80e9:::benchmark_data_smithi007_157098_object31338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-02-05T07:39:11.124 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:49.710885+0000", 2024-02-05T07:39:11.124 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 201.36400177300001, 2024-02-05T07:39:11.124 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14318925900000001, 2024-02-05T07:39:11.124 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.124 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.124 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:11.124 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:39:11.124 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:39:11.125 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 160268 2024-02-05T07:39:11.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.125 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.125 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.125 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.710885+0000", 2024-02-05T07:39:11.125 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.125 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.125 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.710885+0000", 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.710886+0000", 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.147e-06 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.710891+0000", 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6460000000000001e-06 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.710909+0000", 2024-02-05T07:39:11.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8627999999999999e-05 2024-02-05T07:39:11.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.127 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.710918+0000", 2024-02-05T07:39:11.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1210000000000007e-06 2024-02-05T07:39:11.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.127 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.711032+0000", 2024-02-05T07:39:11.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000114059 2024-02-05T07:39:11.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.128 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.711154+0000", 2024-02-05T07:39:11.128 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012228799999999999 2024-02-05T07:39:11.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.128 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,3,4", 2024-02-05T07:39:11.128 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.711465+0000", 2024-02-05T07:39:11.128 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00031057200000000001 2024-02-05T07:39:11.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.128 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.128 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.717005+0000", 2024-02-05T07:39:11.128 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0055402760000000002 2024-02-05T07:39:11.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.717053+0000", 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7719999999999997e-05 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.717111+0000", 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8267000000000003e-05 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.717126+0000", 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4868000000000001e-05 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.130 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.130 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.853870+0000", 2024-02-05T07:39:11.130 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.136744427 2024-02-05T07:39:11.130 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.130 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.130 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.130 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.853944+0000", 2024-02-05T07:39:11.130 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4268000000000001e-05 2024-02-05T07:39:11.130 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.130 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.130 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.130 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.854074+0000", 2024-02-05T07:39:11.130 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012997200000000001 2024-02-05T07:39:11.130 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:160267 10.18 e514/496 10:18879764:::benchmark_data_smithi007_157098_object31337:head v 514'9004, mlcod=514'9001)", 2024-02-05T07:39:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:49.717455+0000", 2024-02-05T07:39:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 201.35743228199999, 2024-02-05T07:39:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14011641499999999, 2024-02-05T07:39:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.131 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.717455+0000", 2024-02-05T07:39:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.717455+0000", 2024-02-05T07:39:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.717455+0000", 2024-02-05T07:39:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8999999999999998e-07 2024-02-05T07:39:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.717458+0000", 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3330000000000001e-06 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.717463+0000", 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2390000000000002e-06 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.717468+0000", 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6319999999999999e-06 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.853968+0000", 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.13649950499999999 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.853992+0000", 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4598000000000001e-05 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.857541+0000", 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0035490819999999998 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.857571+0000", 2024-02-05T07:39:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9736000000000001e-05 2024-02-05T07:39:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:160280 10.10 e514/496 10:0f788ca1:::benchmark_data_smithi007_157098_object31350:head v 514'8978, mlcod=514'8976)", 2024-02-05T07:39:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:49.719222+0000", 2024-02-05T07:39:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 201.35566457499999, 2024-02-05T07:39:11.135 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.13831418000000001, 2024-02-05T07:39:11.136 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.136 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.136 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.136 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.136 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.136 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.719222+0000", 2024-02-05T07:39:11.136 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.136 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.136 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.136 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.719222+0000", 2024-02-05T07:39:11.136 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.719223+0000", 2024-02-05T07:39:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9800000000000003e-07 2024-02-05T07:39:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.719229+0000", 2024-02-05T07:39:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2140000000000001e-06 2024-02-05T07:39:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.850556+0000", 2024-02-05T07:39:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.131326425 2024-02-05T07:39:11.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.138 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.138 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.850565+0000", 2024-02-05T07:39:11.138 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2089999999999994e-06 2024-02-05T07:39:11.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.138 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.138 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.852151+0000", 2024-02-05T07:39:11.138 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001585817 2024-02-05T07:39:11.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.138 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.139 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.852173+0000", 2024-02-05T07:39:11.139 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2203000000000001e-05 2024-02-05T07:39:11.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.139 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.139 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.139 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.857498+0000", 2024-02-05T07:39:11.139 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0053249930000000001 2024-02-05T07:39:11.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.139 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.139 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.139 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:49.857536+0000", 2024-02-05T07:39:11.139 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8621000000000002e-05 2024-02-05T07:39:11.139 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.139 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.140 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.140 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.140 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:163077 10.11 10:8bd269b9:::benchmark_data_smithi007_157098_object34147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-02-05T07:39:11.140 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:52.301085+0000", 2024-02-05T07:39:11.140 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 198.77380140599999, 2024-02-05T07:39:11.140 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16956718700000001, 2024-02-05T07:39:11.140 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.140 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.140 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:11.140 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:39:11.141 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:39:11.141 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 163077 2024-02-05T07:39:11.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.141 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.141 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.141 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.301085+0000", 2024-02-05T07:39:11.141 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.141 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.141 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.301085+0000", 2024-02-05T07:39:11.142 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.142 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.142 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.142 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.301087+0000", 2024-02-05T07:39:11.142 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.502e-06 2024-02-05T07:39:11.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.142 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.142 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.142 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.301093+0000", 2024-02-05T07:39:11.142 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6869999999999999e-06 2024-02-05T07:39:11.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.142 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.143 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.143 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.301115+0000", 2024-02-05T07:39:11.143 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2123000000000001e-05 2024-02-05T07:39:11.143 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.143 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.143 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.143 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.301123+0000", 2024-02-05T07:39:11.143 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7139999999999999e-06 2024-02-05T07:39:11.143 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.143 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.143 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.143 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.301289+0000", 2024-02-05T07:39:11.143 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016535000000000001 2024-02-05T07:39:11.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.144 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.144 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.301414+0000", 2024-02-05T07:39:11.144 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012505 2024-02-05T07:39:11.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.144 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4,6", 2024-02-05T07:39:11.144 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.301777+0000", 2024-02-05T07:39:11.144 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00036362699999999999 2024-02-05T07:39:11.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.145 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:11.145 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303582+0000", 2024-02-05T07:39:11.145 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0018040440000000001 2024-02-05T07:39:11.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.145 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.145 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.307544+0000", 2024-02-05T07:39:11.145 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003962836 2024-02-05T07:39:11.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.145 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.145 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.307736+0000", 2024-02-05T07:39:11.145 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019127799999999999 2024-02-05T07:39:11.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.146 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.146 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.307854+0000", 2024-02-05T07:39:11.146 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000118118 2024-02-05T07:39:11.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.146 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.146 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.470461+0000", 2024-02-05T07:39:11.146 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16260687500000001 2024-02-05T07:39:11.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.146 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.470526+0000", 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5817999999999999e-05 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.470653+0000", 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000126165 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:163081 10.11 10:8d69986a:::benchmark_data_smithi007_157098_object34151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:52.303749+0000", 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 198.77113776600001, 2024-02-05T07:39:11.147 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.1671301, 2024-02-05T07:39:11.148 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.148 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.148 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:11.148 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:39:11.148 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:39:11.148 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 163081 2024-02-05T07:39:11.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.148 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.148 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.148 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303749+0000", 2024-02-05T07:39:11.149 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.149 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.149 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.149 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.149 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303749+0000", 2024-02-05T07:39:11.149 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.149 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.149 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.149 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.149 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303750+0000", 2024-02-05T07:39:11.149 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2750000000000001e-06 2024-02-05T07:39:11.149 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.149 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.149 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303755+0000", 2024-02-05T07:39:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3460000000000004e-06 2024-02-05T07:39:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303762+0000", 2024-02-05T07:39:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1180000000000003e-06 2024-02-05T07:39:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303767+0000", 2024-02-05T07:39:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.604e-06 2024-02-05T07:39:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.151 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.151 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303821+0000", 2024-02-05T07:39:11.151 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3949999999999997e-05 2024-02-05T07:39:11.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.151 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.151 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303923+0000", 2024-02-05T07:39:11.151 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010158999999999999 2024-02-05T07:39:11.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.151 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4,6", 2024-02-05T07:39:11.151 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.304235+0000", 2024-02-05T07:39:11.151 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00031194399999999998 2024-02-05T07:39:11.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.152 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.152 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:11.152 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.307572+0000", 2024-02-05T07:39:11.152 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033372039999999999 2024-02-05T07:39:11.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.152 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.152 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.152 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.313825+0000", 2024-02-05T07:39:11.152 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0062527379999999999 2024-02-05T07:39:11.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.152 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.152 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.152 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.313865+0000", 2024-02-05T07:39:11.153 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0238999999999999e-05 2024-02-05T07:39:11.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.153 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.153 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.153 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.317367+0000", 2024-02-05T07:39:11.153 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0035016919999999998 2024-02-05T07:39:11.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.153 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.153 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.153 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.470698+0000", 2024-02-05T07:39:11.153 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.15333086300000001 2024-02-05T07:39:11.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.153 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.154 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.154 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.470760+0000", 2024-02-05T07:39:11.154 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2341999999999994e-05 2024-02-05T07:39:11.154 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.154 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.154 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.154 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.470879+0000", 2024-02-05T07:39:11.154 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000119195 2024-02-05T07:39:11.154 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.154 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.154 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.154 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:163082 10.11 10:8a17134e:::benchmark_data_smithi007_157098_object34152:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-02-05T07:39:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:52.303782+0000", 2024-02-05T07:39:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 198.77110482000001, 2024-02-05T07:39:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.167339829, 2024-02-05T07:39:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:39:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:39:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 163082 2024-02-05T07:39:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.155 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303782+0000", 2024-02-05T07:39:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303782+0000", 2024-02-05T07:39:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303782+0000", 2024-02-05T07:39:11.156 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4400000000000001e-07 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303785+0000", 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4700000000000001e-06 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303789+0000", 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.76e-06 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.303792+0000", 2024-02-05T07:39:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.354e-06 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.305727+0000", 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001934729 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.305864+0000", 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013714399999999999 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4,6", 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.306151+0000", 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028766099999999998 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.307580+0000", 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0014287410000000001 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.313913+0000", 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0063331200000000002 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.313952+0000", 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8553999999999997e-05 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.317408+0000", 2024-02-05T07:39:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0034556880000000002 2024-02-05T07:39:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.470950+0000", 2024-02-05T07:39:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.15354273600000001 2024-02-05T07:39:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.471005+0000", 2024-02-05T07:39:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4846000000000003e-05 2024-02-05T07:39:11.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:52.471122+0000", 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011668200000000001 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:227407 10.12 10:4f6f3afe:::benchmark_data_smithi007_157098_object98477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:54.218424+0000", 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 136.856462897, 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.21464757800000001, 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.161 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:39:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:39:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 227407 2024-02-05T07:39:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.218424+0000", 2024-02-05T07:39:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.162 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.218424+0000", 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.218426+0000", 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.801e-06 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.218431+0000", 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5679999999999999e-06 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.163 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.218442+0000", 2024-02-05T07:39:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0661e-05 2024-02-05T07:39:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.218450+0000", 2024-02-05T07:39:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6199999999999999e-06 2024-02-05T07:39:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.219975+0000", 2024-02-05T07:39:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0015250699999999999 2024-02-05T07:39:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.220123+0000", 2024-02-05T07:39:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014797700000000001 2024-02-05T07:39:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.165 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,3,4,6", 2024-02-05T07:39:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.220490+0000", 2024-02-05T07:39:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00036688299999999998 2024-02-05T07:39:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.229543+0000", 2024-02-05T07:39:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0090536920000000003 2024-02-05T07:39:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.229593+0000", 2024-02-05T07:39:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9836000000000002e-05 2024-02-05T07:39:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.229644+0000", 2024-02-05T07:39:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1121000000000001e-05 2024-02-05T07:39:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.229754+0000", 2024-02-05T07:39:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000110226 2024-02-05T07:39:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.168 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.229794+0000", 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9733000000000001e-05 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.432801+0000", 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.20300699 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.432935+0000", 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000133805 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.433072+0000", 2024-02-05T07:39:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013659499999999999 2024-02-05T07:39:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:227414 10.12 10:4d3a9289:::benchmark_data_smithi007_157098_object98484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-02-05T07:39:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:54.224462+0000", 2024-02-05T07:39:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 136.85042484799999, 2024-02-05T07:39:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.21199794999999999, 2024-02-05T07:39:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:39:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:39:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 227414 2024-02-05T07:39:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.224462+0000", 2024-02-05T07:39:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.224462+0000", 2024-02-05T07:39:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.224464+0000", 2024-02-05T07:39:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9130000000000001e-06 2024-02-05T07:39:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.224470+0000", 2024-02-05T07:39:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2910000000000004e-06 2024-02-05T07:39:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.224483+0000", 2024-02-05T07:39:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2927e-05 2024-02-05T07:39:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.224492+0000", 2024-02-05T07:39:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.346e-06 2024-02-05T07:39:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.224541+0000", 2024-02-05T07:39:11.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8621000000000001e-05 2024-02-05T07:39:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.225460+0000", 2024-02-05T07:39:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00091891800000000001 2024-02-05T07:39:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,3,4,6", 2024-02-05T07:39:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.225930+0000", 2024-02-05T07:39:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00047017199999999998 2024-02-05T07:39:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:11.174 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.229659+0000", 2024-02-05T07:39:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003728664 2024-02-05T07:39:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.229709+0000", 2024-02-05T07:39:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0158000000000003e-05 2024-02-05T07:39:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.229827+0000", 2024-02-05T07:39:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011822100000000001 2024-02-05T07:39:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.229866+0000", 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8695999999999997e-05 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.351233+0000", 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.12136696700000001 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.436242+0000", 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.085009046000000005 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.176 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.436314+0000", 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1660000000000002e-05 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:54.436460+0000", 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014635000000000001 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15846.0:38181 11.5 e602/579 11:adb74ced:::benchmark_data_smithi007_183880_object38180:head v 602'2325, mlcod=602'2323)", 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:31.606524+0000", 2024-02-05T07:39:11.177 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 39.468362632000002, 2024-02-05T07:39:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.103137431, 2024-02-05T07:39:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.606524+0000", 2024-02-05T07:39:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.606524+0000", 2024-02-05T07:39:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.606525+0000", 2024-02-05T07:39:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4e-07 2024-02-05T07:39:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.606555+0000", 2024-02-05T07:39:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.0434999999999999e-05 2024-02-05T07:39:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.606567+0000", 2024-02-05T07:39:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1027e-05 2024-02-05T07:39:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.606574+0000", 2024-02-05T07:39:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6450000000000005e-06 2024-02-05T07:39:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.606630+0000", 2024-02-05T07:39:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5708000000000003e-05 2024-02-05T07:39:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.180 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.606651+0000", 2024-02-05T07:39:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1444999999999999e-05 2024-02-05T07:39:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.610162+0000", 2024-02-05T07:39:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003510511 2024-02-05T07:39:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.709662+0000", 2024-02-05T07:39:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.099499820000000003 2024-02-05T07:39:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.181 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15846.0:38174 11.9 e602/579 11:94dc4b06:::benchmark_data_smithi007_183880_object38173:head v 602'2303, mlcod=602'2300)", 2024-02-05T07:39:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:31.607416+0000", 2024-02-05T07:39:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 39.467471168000003, 2024-02-05T07:39:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.107725379, 2024-02-05T07:39:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607416+0000", 2024-02-05T07:39:11.182 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607416+0000", 2024-02-05T07:39:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607416+0000", 2024-02-05T07:39:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0500000000000002e-07 2024-02-05T07:39:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.183 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607438+0000", 2024-02-05T07:39:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1321000000000001e-05 2024-02-05T07:39:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607449+0000", 2024-02-05T07:39:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0747999999999999e-05 2024-02-05T07:39:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607457+0000", 2024-02-05T07:39:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1429999999999998e-06 2024-02-05T07:39:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.184 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607514+0000", 2024-02-05T07:39:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7247999999999998e-05 2024-02-05T07:39:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607535+0000", 2024-02-05T07:39:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1395000000000001e-05 2024-02-05T07:39:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.714921+0000", 2024-02-05T07:39:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10738597800000001 2024-02-05T07:39:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.715141+0000", 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000219741 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15846.0:38179 11.8 e602/579 11:17649cc1:::benchmark_data_smithi007_183880_object38178:head v 602'2335, mlcod=602'2333)", 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:31.607478+0000", 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 39.467408376999998, 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.102190169, 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.186 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607478+0000", 2024-02-05T07:39:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607478+0000", 2024-02-05T07:39:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607479+0000", 2024-02-05T07:39:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7e-07 2024-02-05T07:39:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607502+0000", 2024-02-05T07:39:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2731999999999999e-05 2024-02-05T07:39:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607508+0000", 2024-02-05T07:39:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6050000000000003e-06 2024-02-05T07:39:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607513+0000", 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2689999999999999e-06 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607632+0000", 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000118688 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.607652+0000", 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9966000000000002e-05 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.709616+0000", 2024-02-05T07:39:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10196356099999999 2024-02-05T07:39:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.709669+0000", 2024-02-05T07:39:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2978e-05 2024-02-05T07:39:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15846.0:38183 11.9 e602/579 11:905dbc0a:::benchmark_data_smithi007_183880_object38182:head v 602'2304, mlcod=602'2301)", 2024-02-05T07:39:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:31.608999+0000", 2024-02-05T07:39:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 39.465887993999999, 2024-02-05T07:39:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.108393401, 2024-02-05T07:39:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.608999+0000", 2024-02-05T07:39:11.191 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.608999+0000", 2024-02-05T07:39:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.609000+0000", 2024-02-05T07:39:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3099999999999997e-07 2024-02-05T07:39:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.609025+0000", 2024-02-05T07:39:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.529e-05 2024-02-05T07:39:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.609035+0000", 2024-02-05T07:39:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0412e-05 2024-02-05T07:39:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.609044+0000", 2024-02-05T07:39:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2770000000000005e-06 2024-02-05T07:39:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.709647+0000", 2024-02-05T07:39:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.100603673 2024-02-05T07:39:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.709675+0000", 2024-02-05T07:39:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7413e-05 2024-02-05T07:39:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.717301+0000", 2024-02-05T07:39:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0076264560000000002 2024-02-05T07:39:11.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.717392+0000", 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1149000000000004e-05 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15846.0:38188 11.1 e602/579 11:809a558b:::benchmark_data_smithi007_183880_object38187:head v 602'2366, mlcod=602'2364)", 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:31.609074+0000", 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 39.465812771000003, 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10837954700000001, 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:11.195 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.609074+0000", 2024-02-05T07:39:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.609074+0000", 2024-02-05T07:39:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:11.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.609074+0000", 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0699999999999998e-07 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.609104+0000", 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9555000000000001e-05 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.709595+0000", 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.100490994 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:11.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.709609+0000", 2024-02-05T07:39:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.438e-05 2024-02-05T07:39:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.709659+0000", 2024-02-05T07:39:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9495e-05 2024-02-05T07:39:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.709704+0000", 2024-02-05T07:39:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5498999999999999e-05 2024-02-05T07:39:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.717395+0000", 2024-02-05T07:39:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0076908360000000004 2024-02-05T07:39:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:31.717454+0000", 2024-02-05T07:39:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8381000000000002e-05 2024-02-05T07:39:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:11.199 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:11.200 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:39:11.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:11.200 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:11.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:11.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:11.328 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:11.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:11.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:11.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:11.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:11.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:11.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:39:11 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:39:11.556 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 67 16 83953 83937 78.2778 69.6875 0.010858 0.0127693 2024-02-05T07:39:11.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:11.582 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:11.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:11.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:11.684 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:11.709 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:11.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:11.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:11.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:11.837 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:11.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:11.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:11.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:11 smithi007 ceph-mon[29098]: osdmap e619: 8 total, 8 up, 5 in 2024-02-05T07:39:11.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:11 smithi007 ceph-mon[37158]: osdmap e619: 8 total, 8 up, 5 in 2024-02-05T07:39:11.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:11.982 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:11.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:12.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:11 smithi090 ceph-mon[33213]: osdmap e619: 8 total, 8 up, 5 in 2024-02-05T07:39:12.084 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:12.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:12.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:12.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:12.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:12.274 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:12.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:12.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:12.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:12.424 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:12.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:12.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:12.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:12.555 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 68 16 85170 85154 78.2451 76.0625 0.0205152 0.012775 2024-02-05T07:39:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:12.570 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:12.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:12.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:12.696 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:12.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:12.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:12.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:12.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:12.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:12 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:12 smithi007 ceph-mon[29098]: pgmap v1128: 178 pgs: 1 active+clean+remapped, 6 active+remapped+backfill_toofull, 171 active+clean; 4.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 7.2 KiB/s rd, 75 MiB/s wr, 1.21k op/s; 2/243681 objects misplaced (0.001%) 2024-02-05T07:39:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:12 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:39:12.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:12 smithi007 ceph-mon[29098]: osdmap e620: 8 total, 8 up, 5 in 2024-02-05T07:39:12.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:12 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:12.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:12 smithi007 ceph-mon[37158]: pgmap v1128: 178 pgs: 1 active+clean+remapped, 6 active+remapped+backfill_toofull, 171 active+clean; 4.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 7.2 KiB/s rd, 75 MiB/s wr, 1.21k op/s; 2/243681 objects misplaced (0.001%) 2024-02-05T07:39:12.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:12 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:39:12.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:12 smithi007 ceph-mon[37158]: osdmap e620: 8 total, 8 up, 5 in 2024-02-05T07:39:12.925 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:12.963 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:12.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:12.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:12 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:12 smithi090 ceph-mon[33213]: pgmap v1128: 178 pgs: 1 active+clean+remapped, 6 active+remapped+backfill_toofull, 171 active+clean; 4.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 7.2 KiB/s rd, 75 MiB/s wr, 1.21k op/s; 2/243681 objects misplaced (0.001%) 2024-02-05T07:39:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:12 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:39:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:12 smithi090 ceph-mon[33213]: osdmap e620: 8 total, 8 up, 5 in 2024-02-05T07:39:13.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:13.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:13.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:13.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:13.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:13.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:13.346 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:13.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:13.449 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:13.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:13.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:13.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:13.556 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 69 15 86394 86379 78.2206 76.5625 0.0200642 0.0127789 2024-02-05T07:39:13.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:13.604 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:13.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:13.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:13.706 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:13.731 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:13.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:13.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:13.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:13.875 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:13.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:13.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:13.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:39:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:39:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:39:13.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:13 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:39:13.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:13 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:39:13.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:13 smithi007 ceph-mon[29098]: osdmap e621: 8 total, 8 up, 5 in 2024-02-05T07:39:13.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:13 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:39:13.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:13 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:39:13.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:13 smithi007 ceph-mon[37158]: osdmap e621: 8 total, 8 up, 5 in 2024-02-05T07:39:13.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:14.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:13 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:39:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:13 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:39:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:13 smithi090 ceph-mon[33213]: osdmap e621: 8 total, 8 up, 5 in 2024-02-05T07:39:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:14.038 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:14.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:14.139 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:14.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:14.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:14.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:14.291 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:14.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:14.318 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:14.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:14.419 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:14.459 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:14.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:14.556 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 70 16 87644 87628 78.2183 78.0625 0.00363596 0.0127781 2024-02-05T07:39:14.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:14.584 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:14.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:14.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:14.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:14.732 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:14.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:14.744 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:39:14.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T07:39:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:39:14.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:14.859 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:14.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:14.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:14.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:15.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:15.020 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:15.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:15.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:14 smithi090 ceph-mon[33213]: pgmap v1131: 178 pgs: 1 active+clean+remapped, 5 active+remapped+backfill_toofull, 172 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 852 B/s rd, 59 MiB/s wr, 951 op/s; 2/249111 objects misplaced (0.001%) 2024-02-05T07:39:15.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:14 smithi090 ceph-mon[33213]: osdmap e622: 8 total, 8 up, 5 in 2024-02-05T07:39:15.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:14 smithi007 ceph-mon[29098]: pgmap v1131: 178 pgs: 1 active+clean+remapped, 5 active+remapped+backfill_toofull, 172 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 852 B/s rd, 59 MiB/s wr, 951 op/s; 2/249111 objects misplaced (0.001%) 2024-02-05T07:39:15.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:14 smithi007 ceph-mon[29098]: osdmap e622: 8 total, 8 up, 5 in 2024-02-05T07:39:15.055 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:14 smithi007 ceph-mon[37158]: pgmap v1131: 178 pgs: 1 active+clean+remapped, 5 active+remapped+backfill_toofull, 172 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 852 B/s rd, 59 MiB/s wr, 951 op/s; 2/249111 objects misplaced (0.001%) 2024-02-05T07:39:15.055 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:14 smithi007 ceph-mon[37158]: osdmap e622: 8 total, 8 up, 5 in 2024-02-05T07:39:15.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:15.147 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:15.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:15.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:15.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:15.274 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:15.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:15.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:15.426 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:15.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:15.527 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:15.556 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 71 16 88696 88680 78.0425 65.75 0.0177288 0.0128078 2024-02-05T07:39:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:15.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:15.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:15.703 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:15.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:15.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:15.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:15.854 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:15.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:15.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:15.999 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:16.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:16.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:16.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:16.142 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:16.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:16.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:16.268 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:16.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:16.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:16.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:16.395 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:16.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:16.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:16.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:16.522 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:16.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:16.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:16.556 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 72 16 89859 89843 77.9681 72.6875 0.0118739 0.0128212 2024-02-05T07:39:16.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:16.648 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:16.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:16.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:16.776 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:16.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:16.878 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:16.905 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:16.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:16.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:17.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:17.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:16 smithi090 ceph-mon[33213]: pgmap v1133: 178 pgs: 1 active+clean+remapped, 5 active+remapped+backfill_toofull, 172 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 924 op/s; 2/256821 objects misplaced (0.001%) 2024-02-05T07:39:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:16 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:39:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:16 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:39:17.034 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:17.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:17.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:17.161 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:17.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:17.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:17.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:16 smithi007 ceph-mon[29098]: pgmap v1133: 178 pgs: 1 active+clean+remapped, 5 active+remapped+backfill_toofull, 172 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 924 op/s; 2/256821 objects misplaced (0.001%) 2024-02-05T07:39:17.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:16 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:39:17.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:16 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:39:17.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:16 smithi007 ceph-mon[37158]: pgmap v1133: 178 pgs: 1 active+clean+remapped, 5 active+remapped+backfill_toofull, 172 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 924 op/s; 2/256821 objects misplaced (0.001%) 2024-02-05T07:39:17.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:16 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:39:17.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:16 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:39:17.263 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:17.288 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:17.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:17.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:17.397 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:39:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:17.448 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:17.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:17.549 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:17.556 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 73 15 91081 91066 77.947 76.4375 0.00990298 0.0128245 2024-02-05T07:39:17.596 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:17.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:17.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:17.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:17.734 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.1 to 1 (10000) 2024-02-05T07:39:17.750 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:17.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:17.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:17.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:17.901 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:17.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:18.003 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:18.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:17 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1380929100' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:39:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:17 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:39:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:17 smithi090 ceph-mon[33213]: osdmap e623: 8 total, 8 up, 5 in 2024-02-05T07:39:18.028 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:18.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:18.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:18.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:17 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1380929100' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:39:18.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:17 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:39:18.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:17 smithi007 ceph-mon[29098]: osdmap e623: 8 total, 8 up, 5 in 2024-02-05T07:39:18.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:17 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1380929100' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:39:18.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:17 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:39:18.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:17 smithi007 ceph-mon[37158]: osdmap e623: 8 total, 8 up, 5 in 2024-02-05T07:39:18.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:18.170 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:18.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:18.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:18.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:18.296 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:18.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:18.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:18.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:18.426 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:18.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:18.527 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:18.553 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:18.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:18.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:18.556 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 74 16 92297 92281 77.9198 75.9375 0.0201244 0.0128289 2024-02-05T07:39:18.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:18.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:18.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:18.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:18.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 6 1 2024-02-05T07:39:18.808 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:18.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:18.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:18.961 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:18.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:18.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:18 smithi090 ceph-mon[33213]: pgmap v1135: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 173 active+clean; 5.4 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 893 B/s rd, 84 MiB/s wr, 1.35k op/s; 2/266898 objects misplaced (0.001%) 2024-02-05T07:39:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:18 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:39:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:18 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T07:39:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:18 smithi090 ceph-mon[33213]: osdmap e624: 8 total, 8 up, 5 in 2024-02-05T07:39:19.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:18 smithi007 ceph-mon[29098]: pgmap v1135: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 173 active+clean; 5.4 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 893 B/s rd, 84 MiB/s wr, 1.35k op/s; 2/266898 objects misplaced (0.001%) 2024-02-05T07:39:19.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:18 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:39:19.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:18 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T07:39:19.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:18 smithi007 ceph-mon[29098]: osdmap e624: 8 total, 8 up, 5 in 2024-02-05T07:39:19.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:18 smithi007 ceph-mon[37158]: pgmap v1135: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 173 active+clean; 5.4 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 893 B/s rd, 84 MiB/s wr, 1.35k op/s; 2/266898 objects misplaced (0.001%) 2024-02-05T07:39:19.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:18 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:39:19.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:18 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T07:39:19.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:18 smithi007 ceph-mon[37158]: osdmap e624: 8 total, 8 up, 5 in 2024-02-05T07:39:19.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:19.090 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:19.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:19.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:19.218 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:19.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:19.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:19.344 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:19.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:19.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:19.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:19.470 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:19.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:19.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:19.557 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 75 16 93483 93467 77.8684 74.125 0.02014 0.012837 2024-02-05T07:39:19.572 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:19.598 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:19.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:19.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:19.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:19.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:19.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:19.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:19.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:19.873 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:19.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:19.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:19.998 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:19.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:20.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:20.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:20.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:20.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:20.251 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:20.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:20.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:20.390 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:20.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:20.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:20.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:20.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:20.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:20.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:20.557 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 76 16 94565 94549 77.7335 67.625 0.00977789 0.0128591 2024-02-05T07:39:20.635 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:20.676 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:20.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:20.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:20.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:20.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:20.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:20.908 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:20 smithi007 ceph-mon[29098]: pgmap v1137: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 173 active+clean; 5.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.23k op/s; 2/271311 objects misplaced (0.001%) 2024-02-05T07:39:20.908 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:20 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/892892655' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:39:20.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:20 smithi007 ceph-mon[37158]: pgmap v1137: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 173 active+clean; 5.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.23k op/s; 2/271311 objects misplaced (0.001%) 2024-02-05T07:39:20.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:20 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/892892655' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:39:20.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:20.964 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:20.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:21.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:20 smithi090 ceph-mon[33213]: pgmap v1137: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 173 active+clean; 5.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.23k op/s; 2/271311 objects misplaced (0.001%) 2024-02-05T07:39:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:20 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/892892655' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:39:21.065 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:21.089 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:21.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:21.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:21.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:21.217 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:21.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:21.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:21.345 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:21.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:21.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:21.472 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:21.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:21.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:21.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:39:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:39:21.554 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:39:21.555 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:39:21.555 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:39:21.555 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:39:21.555 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:39:21.555 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:39:21.555 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:39:21.555 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:39:21.555 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:39:21.555 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:39:21.556 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:39:21.556 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:39:21.556 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:39:21.556 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:39:21.556 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:39:21.556 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:39:21.558 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 77 16 95646 95630 77.6004 67.5625 0.0110812 0.0128814 2024-02-05T07:39:21.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:21.602 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:21.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:21.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:21.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:21.729 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:21.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:21.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:21.759 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:39:21.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:21.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:21.858 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:21.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:21.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:22.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:22.006 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:22.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:22.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:21 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:39:22.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:21 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:39:22.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:21 smithi090 ceph-mon[33213]: osdmap e625: 8 total, 8 up, 5 in 2024-02-05T07:39:22.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:21 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/874801348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:39:22.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:21 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:39:22.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:22.148 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:22.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:22.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:22.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:21 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:39:22.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:21 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:39:22.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:21 smithi007 ceph-mon[29098]: osdmap e625: 8 total, 8 up, 5 in 2024-02-05T07:39:22.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:21 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/874801348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:39:22.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:21 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:39:22.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:21 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:39:22.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:21 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:39:22.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:21 smithi007 ceph-mon[37158]: osdmap e625: 8 total, 8 up, 5 in 2024-02-05T07:39:22.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:21 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/874801348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:39:22.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:21 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:39:22.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:22.275 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:22.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:22.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:22.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:22.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:22.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:22.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:22.547 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:22.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:22.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:22.558 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 78 16 96892 96876 77.6038 77.875 0.0143744 0.0128811 2024-02-05T07:39:22.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:22.694 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:22.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:22.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:22.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:22.822 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:22.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:22.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:22.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:22 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:22.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:22 smithi007 ceph-mon[29098]: pgmap v1139: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 5.7 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.23k op/s; 43 KiB/s, 0 objects/s recovering 2024-02-05T07:39:22.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:22 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:39:22.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:22 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:39:22.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:22 smithi007 ceph-mon[29098]: osdmap e626: 8 total, 8 up, 5 in 2024-02-05T07:39:22.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:22 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:22.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:22 smithi007 ceph-mon[37158]: pgmap v1139: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 5.7 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.23k op/s; 43 KiB/s, 0 objects/s recovering 2024-02-05T07:39:22.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:22 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:39:22.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:22 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:39:22.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:22 smithi007 ceph-mon[37158]: osdmap e626: 8 total, 8 up, 5 in 2024-02-05T07:39:22.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:22.951 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:22.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:22.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:22.971 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:39:23.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:22 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:23.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:22 smithi090 ceph-mon[33213]: pgmap v1139: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 5.7 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.23k op/s; 43 KiB/s, 0 objects/s recovering 2024-02-05T07:39:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:22 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:39:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:22 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:39:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:22 smithi090 ceph-mon[33213]: osdmap e626: 8 total, 8 up, 5 in 2024-02-05T07:39:23.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:23.100 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:23.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:23.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:23.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:23.226 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:23.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:23.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:23.353 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:23.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:23.454 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:23.497 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:23.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:23.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:23.558 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 79 16 98118 98102 77.5914 76.625 0.0153848 0.0128824 2024-02-05T07:39:23.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:23.625 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:23.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:23.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:23.752 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:23.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:23.811 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:39:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:39:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:39:23.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:23.879 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:23.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:23.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:23.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:24.025 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:24.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:24.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:24.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:24.152 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:24.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:24.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:24.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:39:24.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:24.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:24.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:24.443 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:24.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:24.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:24.562 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-02-05T07:39:24.562444+0000 min lat: 0.00118746 max lat: 1.24847 avg lat: 0.0128974 2024-02-05T07:39:24.562 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T07:39:24.562 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 80 16 99251 99235 77.5031 70.8125 0.015922 0.0128974 2024-02-05T07:39:24.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:24.635 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:24.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:24.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:24.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:24.787 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:24.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:24.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:24.866 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:39:24.866 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:39:24.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:24.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:24.941 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:24.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:25.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:39:25.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:24 smithi090 ceph-mon[33213]: pgmap v1141: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 5.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 853 B/s rd, 61 MiB/s wr, 974 op/s; 43 KiB/s, 0 objects/s recovering 2024-02-05T07:39:25.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:24 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:39:25.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:25.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:24 smithi007 ceph-mon[29098]: pgmap v1141: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 5.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 853 B/s rd, 61 MiB/s wr, 974 op/s; 43 KiB/s, 0 objects/s recovering 2024-02-05T07:39:25.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:24 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:39:25.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:24 smithi007 ceph-mon[37158]: pgmap v1141: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 5.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 853 B/s rd, 61 MiB/s wr, 974 op/s; 43 KiB/s, 0 objects/s recovering 2024-02-05T07:39:25.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:24 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:39:25.069 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:25.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:25.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:25.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:25.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:25.199 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:25.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:25.300 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:25.325 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:25.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:25.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:25.453 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:25.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:25.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:25.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:25.563 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 81 15 100417 100402 77.4459 72.9375 0.013329 0.012907 2024-02-05T07:39:25.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:25.598 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:25.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:25.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:25.725 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:25.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:25.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:25.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:25.854 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:25.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:25.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:25.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:25.981 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:25.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:26.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:26.107 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:26.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:26.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:26.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:26.234 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:26.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:26.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:26.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:26.361 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:26.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:26.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:26.488 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:26.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:26.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:26.563 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 82 16 101423 101407 77.2675 62.8125 0.00515771 0.0129379 2024-02-05T07:39:26.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:26.632 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:26.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:26.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:26.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T07:39:26.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:26.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:26.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:26.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:26.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:26 smithi007 ceph-mon[29098]: pgmap v1142: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 5.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.3 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 33 KiB/s, 0 objects/s recovering 2024-02-05T07:39:26.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:26 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:39:26.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:26 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:39:26.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:26 smithi007 ceph-mon[37158]: pgmap v1142: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 5.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.3 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 33 KiB/s, 0 objects/s recovering 2024-02-05T07:39:26.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:26 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:39:26.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:26 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:39:26.946 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:26.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:26.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:27.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:26 smithi090 ceph-mon[33213]: pgmap v1142: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 5.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.3 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 33 KiB/s, 0 objects/s recovering 2024-02-05T07:39:27.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:26 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:39:27.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:26 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:39:27.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:27.072 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:27.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:27.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:27.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:27.225 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:27.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:27.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:27.379 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:27.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:27.418 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:39:27.447 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:39:27.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:27.543 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:27.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:27.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:27.565 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 83 16 102414 102398 77.0814 61.9375 0.0105119 0.012968 2024-02-05T07:39:27.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:27.671 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:27.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:27.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:27.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:27.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:27.823 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:27.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:27.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:27.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:27.970 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:27.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:28.060 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:27 smithi007 ceph-mon[29098]: osdmap e627: 8 total, 8 up, 5 in 2024-02-05T07:39:28.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:27 smithi007 ceph-mon[37158]: osdmap e627: 8 total, 8 up, 5 in 2024-02-05T07:39:28.067 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:39:28.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:28.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:28.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:28.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:28.228 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:28.272 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:28.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:28.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:27 smithi090 ceph-mon[33213]: osdmap e627: 8 total, 8 up, 5 in 2024-02-05T07:39:28.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:28.455 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:28.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:28.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:28.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:28.564 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 84 15 103507 103492 76.9777 68.375 0.0237986 0.0129855 2024-02-05T07:39:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:28.604 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:28.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:28.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:28.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:28.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:28.802 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.6 to 1 (10000) 2024-02-05T07:39:28.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:28 smithi007 ceph-mon[29098]: pgmap v1144: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 767 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-02-05T07:39:28.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:28 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1207388894' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:39:28.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:28 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:39:28.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:28 smithi007 ceph-mon[37158]: pgmap v1144: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 767 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-02-05T07:39:28.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:28 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1207388894' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:39:28.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:28 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:39:28.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:28.859 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:28.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:28.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:28.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:28.987 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:28.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:29.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:29.126 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:29.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:29.174 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:28 smithi090 ceph-mon[33213]: pgmap v1144: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 767 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-02-05T07:39:29.174 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:28 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1207388894' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:39:29.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:28 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:39:29.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:29.266 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:29.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:29.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:29.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:29.394 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:29.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:29.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:29.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:29.521 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:29.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:29.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:29.565 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 85 16 104711 104695 76.9566 75.1875 0.00681967 0.0129894 2024-02-05T07:39:29.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:29.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:29.654 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:29.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:29.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:29.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:29.799 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:29.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:29.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:29 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T07:39:29.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:29 smithi007 ceph-mon[29098]: osdmap e628: 8 total, 8 up, 5 in 2024-02-05T07:39:29.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:29 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T07:39:29.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:29 smithi007 ceph-mon[37158]: osdmap e628: 8 total, 8 up, 5 in 2024-02-05T07:39:29.925 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:29.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:29.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:30.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:30.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:30.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:30.213 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:30.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:30.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:30.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:29 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T07:39:30.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:29 smithi090 ceph-mon[33213]: osdmap e628: 8 total, 8 up, 5 in 2024-02-05T07:39:30.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:30.341 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:30.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:30.443 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:30.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 3 1 2024-02-05T07:39:30.491 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:30.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:30.565 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 86 16 105912 105896 76.9338 75.0625 0.0186519 0.012993 2024-02-05T07:39:30.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:30.646 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:30.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:30.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:30.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:30.774 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:30.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:30.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:30.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:30.902 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:30.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:30.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:31.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:31.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:31.029 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:31.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:31.060 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:30 smithi007 ceph-mon[29098]: pgmap v1146: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 668 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-05T07:39:31.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:30 smithi007 ceph-mon[37158]: pgmap v1146: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 668 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-05T07:39:31.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:31.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:31.164 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:30 smithi090 ceph-mon[33213]: pgmap v1146: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 668 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-05T07:39:31.258 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:31.300 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:31.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:31.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:31.427 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:31.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:31.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:31.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:39:31 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:39:31.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:31.566 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 87 16 107118 107102 76.9158 75.375 0.009479 0.0129963 2024-02-05T07:39:31.573 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:31.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:31.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:31.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:31.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:31.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:31.825 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:31.866 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:31.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:31.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:31.968 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:32.009 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:32.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:32.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:32.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:32.135 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:32.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:32.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:32.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:31 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:39:32.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:31 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:39:32.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:32.263 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:32.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:32.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:31 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:39:32.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:32.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:32.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:32.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:32.540 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:32.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:32.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:32.566 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 88 16 108205 108189 76.8137 67.9375 0.0138706 0.0130133 2024-02-05T07:39:32.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:32.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:32.667 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:32.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:32.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:32.810 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:32.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:32.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:32.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:32.963 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:32.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:33.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:33.107 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:33.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:33.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:33.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:32 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:33.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:32 smithi007 ceph-mon[29098]: pgmap v1147: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.4 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-05T07:39:33.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:32 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:33.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:32 smithi007 ceph-mon[37158]: pgmap v1147: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.4 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-05T07:39:33.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:33.236 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:33.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:33.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:32 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:32 smithi090 ceph-mon[33213]: pgmap v1147: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.4 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-05T07:39:33.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:33.363 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:33.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:33.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:33.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:33.505 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:33.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:33.566 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 89 16 109273 109257 76.7006 66.75 0.0782285 0.0130299 2024-02-05T07:39:33.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:33.632 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:33.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:33.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:33.803 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:33.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:33.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:33.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:33.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:39:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:39:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:39:33.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:33.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:34.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:34.057 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:34.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:34.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:34.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:34.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:34.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:34.313 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:34.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:34.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:34.415 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:34.440 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:34.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:34.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:34.564 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:34.569 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total time run: 90.0058 2024-02-05T07:39:34.569 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total writes made: 110278 2024-02-05T07:39:34.569 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Write size: 65536 2024-02-05T07:39:34.569 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object size: 65536 2024-02-05T07:39:34.569 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Bandwidth (MB/sec): 76.577 2024-02-05T07:39:34.569 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Bandwidth: 22.1332 2024-02-05T07:39:34.569 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max bandwidth (MB/sec): 156.375 2024-02-05T07:39:34.569 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min bandwidth (MB/sec): 0 2024-02-05T07:39:34.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average IOPS: 1225 2024-02-05T07:39:34.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev IOPS: 354.139 2024-02-05T07:39:34.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max IOPS: 2502 2024-02-05T07:39:34.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min IOPS: 0 2024-02-05T07:39:34.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average Latency(s): 0.0130546 2024-02-05T07:39:34.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Latency(s): 0.0239509 2024-02-05T07:39:34.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max latency(s): 1.24847 2024-02-05T07:39:34.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min latency(s): 0.00118746 2024-02-05T07:39:34.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T07:39:34.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:34.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:34.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:34.697 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:34.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:34.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:34.837 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:34.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:34.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:34.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:34.963 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:34.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:35.064 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:35.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:35.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:35.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:34 smithi007 ceph-mon[29098]: pgmap v1148: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.5 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T07:39:35.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:34 smithi007 ceph-mon[37158]: pgmap v1148: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.5 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T07:39:35.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:35.227 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:35.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:35.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:34 smithi090 ceph-mon[33213]: pgmap v1148: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.5 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T07:39:35.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:35.355 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:35.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:35.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:35.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:35.497 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:35.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:35.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:35.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:35.624 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:35.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:35.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:35.749 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:35.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:35.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:35.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:35.876 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:35.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:35.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:35.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:36.015 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:36.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:36.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:36.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:36.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:36.147 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:36.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:36.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:36.273 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:36.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:36.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:36.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:36.400 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:36.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:36.505 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:36.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:36.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:36.657 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:36.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:36.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:36.784 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:36.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:36.886 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:36.911 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:36.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:37.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:37.038 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:37.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:37.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:37.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:36 smithi007 ceph-mon[29098]: pgmap v1149: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-02-05T07:39:37.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:36 smithi007 ceph-mon[37158]: pgmap v1149: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-02-05T07:39:37.177 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:37.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:37.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:36 smithi090 ceph-mon[33213]: pgmap v1149: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-02-05T07:39:37.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:37.303 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:37.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:37.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:37.431 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:37.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:37.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:37.575 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:37.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:37.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:37.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:37.735 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:37.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:37.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:37.867 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:37.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:37.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:37.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:37.995 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:37.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:38.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:38.123 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:38.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:38.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:38.250 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:38.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:38.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:38.377 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:38.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:38.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:38.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:38.517 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:38.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:38.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:38.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:38.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:38.790 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:38.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:38.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:38.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:38.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:38.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:39.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:39.047 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:39.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:39.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:39.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:39.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:38 smithi007 ceph-mon[29098]: pgmap v1150: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.1 KiB/s rd, 61 MiB/s wr, 1.12k op/s 2024-02-05T07:39:39.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:38 smithi007 ceph-mon[37158]: pgmap v1150: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.1 KiB/s rd, 61 MiB/s wr, 1.12k op/s 2024-02-05T07:39:39.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:39.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:39.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:38 smithi090 ceph-mon[33213]: pgmap v1150: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.1 KiB/s rd, 61 MiB/s wr, 1.12k op/s 2024-02-05T07:39:39.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:39.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:39.314 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:39.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:39.415 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:39.440 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:39.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:39.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:39.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:39.569 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:39.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:39.670 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:39.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:39.710 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:39.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:39.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:39.836 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:39.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:39.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:39.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:39.964 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:39.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:39.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:40.066 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:40.110 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:40.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:40.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:40.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:40.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:40.251 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:40.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:40.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:40.377 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:40.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:40.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:40.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:40.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:40.510 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:39:40.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:40.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:40.664 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:40.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:40.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:40.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:40.810 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:40.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:40.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:40.922 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:40 smithi007 ceph-mon[29098]: pgmap v1151: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 1.19k op/s 2024-02-05T07:39:40.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:40 smithi007 ceph-mon[37158]: pgmap v1151: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 1.19k op/s 2024-02-05T07:39:40.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:40.950 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:40.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:41.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:41.100 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:41.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:41.177 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:40 smithi090 ceph-mon[33213]: pgmap v1151: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 1.19k op/s 2024-02-05T07:39:41.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:41.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:41.245 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:41.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:41.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:41.371 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:41.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:41.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:41.497 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:41.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:41.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:41.516 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:39:41.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:39:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:39:41.532 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:39:41.575 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:39:41.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:41.622 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:39:41.643 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:41.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:41.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:41.665 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:39:41.715 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:39:41.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:41.759 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:39:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:41.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:41.811 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:39:41.822 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:39:41.822 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:39:41.823 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:39:41.823 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:39:41.823 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:39:41.823 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:39:41.823 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:39:41.823 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:39:41.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:41.920 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:41.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:41.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:42.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:42.065 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:42.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:42.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:42.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:41 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:39:42.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:41 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:39:42.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:41 smithi007 ceph-mon[29098]: osdmap e629: 8 total, 8 up, 5 in 2024-02-05T07:39:42.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:41 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3905016518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:39:42.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:41 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:39:42.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:41 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:39:42.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:41 smithi007 ceph-mon[37158]: osdmap e629: 8 total, 8 up, 5 in 2024-02-05T07:39:42.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:41 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3905016518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:39:42.191 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:42.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:42.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:42.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:41 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:39:42.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:41 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:39:42.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:41 smithi090 ceph-mon[33213]: osdmap e629: 8 total, 8 up, 5 in 2024-02-05T07:39:42.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:41 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3905016518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:39:42.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:42.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:42.353 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:42.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:42.454 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:42.496 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:42.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:42.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:42.642 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:39:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:39:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-02-05T07:39:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-02-05T07:39:42.642 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:39:42.645 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:42.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:42.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:42.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:42.789 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:42.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:42.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:42.878 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:39:42.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:42.918 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:42.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:43.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:43.045 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:43.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:43.147 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:43.156 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:42 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:43.157 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:42 smithi007 ceph-mon[29098]: pgmap v1153: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.4 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 1.16k op/s 2024-02-05T07:39:43.157 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:42 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1692612781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:39:43.157 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:42 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:39:43.157 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:42 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:43.157 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:42 smithi007 ceph-mon[37158]: pgmap v1153: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.4 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 1.16k op/s 2024-02-05T07:39:43.157 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:42 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1692612781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:39:43.157 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:42 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:39:43.191 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:43.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:43.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:42 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:43.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:42 smithi090 ceph-mon[33213]: pgmap v1153: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.4 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 1.16k op/s 2024-02-05T07:39:43.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:42 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1692612781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:39:43.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:42 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:39:43.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:43.333 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:43.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:43.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:43.483 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:43.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:43.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:43.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:43.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:43.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:43.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:43.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:43.740 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:43.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:43.842 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:43.883 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:39:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:39:43.897 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:43.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:43.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:43.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:44.051 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:44.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:44.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:39:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:39:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[29098]: osdmap e630: 8 total, 8 up, 5 in 2024-02-05T07:39:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[29098]: pgmap v1155: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.3 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 767 B/s rd, 16 MiB/s wr, 1.16k op/s 2024-02-05T07:39:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:39:44.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[29098]: 4.18 scrub starts 2024-02-05T07:39:44.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[29098]: 4.18 scrub ok 2024-02-05T07:39:44.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[29098]: 11.c scrub starts 2024-02-05T07:39:44.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:39:44.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:39:44.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[37158]: osdmap e630: 8 total, 8 up, 5 in 2024-02-05T07:39:44.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[37158]: pgmap v1155: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.3 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 767 B/s rd, 16 MiB/s wr, 1.16k op/s 2024-02-05T07:39:44.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:39:44.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[37158]: 4.18 scrub starts 2024-02-05T07:39:44.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[37158]: 4.18 scrub ok 2024-02-05T07:39:44.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:43 smithi007 ceph-mon[37158]: 11.c scrub starts 2024-02-05T07:39:44.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:44.209 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:44.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:44.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:43 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:39:44.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:43 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:39:44.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:43 smithi090 ceph-mon[33213]: osdmap e630: 8 total, 8 up, 5 in 2024-02-05T07:39:44.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:43 smithi090 ceph-mon[33213]: pgmap v1155: 178 pgs: 3 active+remapped+backfill_toofull, 175 active+clean; 6.3 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 767 B/s rd, 16 MiB/s wr, 1.16k op/s 2024-02-05T07:39:44.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:43 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:39:44.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:43 smithi090 ceph-mon[33213]: 4.18 scrub starts 2024-02-05T07:39:44.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:43 smithi090 ceph-mon[33213]: 4.18 scrub ok 2024-02-05T07:39:44.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:43 smithi090 ceph-mon[33213]: 11.c scrub starts 2024-02-05T07:39:44.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:44.357 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:44.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:44.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:44.518 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:44.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:44.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T07:39:44.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:44.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:44.675 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:44.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:44.776 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:44.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:44.820 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:44.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:44.921 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:44.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:44.948 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:44.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:45.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:45.075 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:45.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:45.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:45.201 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:45.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:45.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:45.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:45.328 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:45.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:45.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:45.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:45.457 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:45.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:45.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:45.600 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:45.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:45.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:45.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:45.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:45.853 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:45.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:45.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:45.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:45.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:45 smithi007 ceph-mon[29098]: 5.3 scrub starts 2024-02-05T07:39:45.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:45 smithi007 ceph-mon[29098]: 5.3 scrub ok 2024-02-05T07:39:45.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:45 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:39:45.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:45 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:39:45.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:45 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:39:45.998 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:45 smithi007 ceph-mon[37158]: 5.3 scrub starts 2024-02-05T07:39:45.999 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:45 smithi007 ceph-mon[37158]: 5.3 scrub ok 2024-02-05T07:39:45.999 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:45 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:39:45.999 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:45 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:39:45.999 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:45 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:39:46.003 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:46.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:46.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:46.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:45 smithi090 ceph-mon[33213]: 5.3 scrub starts 2024-02-05T07:39:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:45 smithi090 ceph-mon[33213]: 5.3 scrub ok 2024-02-05T07:39:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:45 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:39:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:45 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:39:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:45 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:39:46.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:39:46.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:46.220 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:46.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:46.321 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:46.383 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:46.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:46.458 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:39:46.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:46.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:46.564 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:46.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:46.665 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:46.720 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:46.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:46.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:46.847 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:46.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:46.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:46.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:46 smithi007 ceph-mon[29098]: pgmap v1156: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 6.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 1.10k op/s 2024-02-05T07:39:46.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:46 smithi007 ceph-mon[29098]: 1.e scrub starts 2024-02-05T07:39:46.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:46 smithi007 ceph-mon[29098]: 1.e scrub ok 2024-02-05T07:39:46.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:46 smithi007 ceph-mon[29098]: 5.1a scrub starts 2024-02-05T07:39:46.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:46 smithi007 ceph-mon[29098]: 5.1a scrub ok 2024-02-05T07:39:46.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:39:46.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:46 smithi007 ceph-mon[29098]: osdmap e631: 8 total, 8 up, 5 in 2024-02-05T07:39:46.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:46 smithi007 ceph-mon[37158]: pgmap v1156: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 6.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 1.10k op/s 2024-02-05T07:39:46.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:46 smithi007 ceph-mon[37158]: 1.e scrub starts 2024-02-05T07:39:46.907 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:46 smithi007 ceph-mon[37158]: 1.e scrub ok 2024-02-05T07:39:46.908 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:46 smithi007 ceph-mon[37158]: 5.1a scrub starts 2024-02-05T07:39:46.908 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:46 smithi007 ceph-mon[37158]: 5.1a scrub ok 2024-02-05T07:39:46.908 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:39:46.908 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:46 smithi007 ceph-mon[37158]: osdmap e631: 8 total, 8 up, 5 in 2024-02-05T07:39:46.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:46.993 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:46.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:47.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:47.140 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:47.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:47.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:39:47.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:47.267 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:47.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:47.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:46 smithi090 ceph-mon[33213]: pgmap v1156: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 6.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 1.10k op/s 2024-02-05T07:39:47.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:46 smithi090 ceph-mon[33213]: 1.e scrub starts 2024-02-05T07:39:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:46 smithi090 ceph-mon[33213]: 1.e scrub ok 2024-02-05T07:39:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:46 smithi090 ceph-mon[33213]: 5.1a scrub starts 2024-02-05T07:39:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:46 smithi090 ceph-mon[33213]: 5.1a scrub ok 2024-02-05T07:39:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:39:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:46 smithi090 ceph-mon[33213]: osdmap e631: 8 total, 8 up, 5 in 2024-02-05T07:39:47.279 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:39:47.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:47.419 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:47.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:47.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:47.544 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:47.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:47.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:47.707 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:47.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:47.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:47.725 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.3 to 1 (10000) 2024-02-05T07:39:47.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:47.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:47.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:47.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:47.960 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:47.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:47.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:47.964 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:39:47.964 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:39:47.964 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[29098]: 11.c scrub ok 2024-02-05T07:39:47.964 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[29098]: 1.15 scrub starts 2024-02-05T07:39:47.964 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[29098]: 1.15 scrub ok 2024-02-05T07:39:47.964 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/564644441' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:39:47.964 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:39:47.964 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T07:39:47.964 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[29098]: osdmap e632: 8 total, 8 up, 5 in 2024-02-05T07:39:47.965 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:39:47.965 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:39:47.965 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[37158]: 11.c scrub ok 2024-02-05T07:39:47.965 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[37158]: 1.15 scrub starts 2024-02-05T07:39:47.965 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[37158]: 1.15 scrub ok 2024-02-05T07:39:47.965 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/564644441' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:39:47.965 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:39:47.965 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T07:39:47.965 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:47 smithi007 ceph-mon[37158]: osdmap e632: 8 total, 8 up, 5 in 2024-02-05T07:39:48.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:48.086 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:48.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:48.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:48.188 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:48.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:48.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:47 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:39:48.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:47 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:39:48.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:47 smithi090 ceph-mon[33213]: 11.c scrub ok 2024-02-05T07:39:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:47 smithi090 ceph-mon[33213]: 1.15 scrub starts 2024-02-05T07:39:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:47 smithi090 ceph-mon[33213]: 1.15 scrub ok 2024-02-05T07:39:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:47 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/564644441' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:39:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:47 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:39:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:47 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T07:39:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:47 smithi090 ceph-mon[33213]: osdmap e632: 8 total, 8 up, 5 in 2024-02-05T07:39:48.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:48.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:48.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:48.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:48.463 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:48.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:48.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:48.608 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:48.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:48.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:48.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:48.735 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:48.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:48.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:48.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:48.879 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:48.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:48.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:48.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:49.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:49.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:49.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:49.132 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:49.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:48 smithi007 ceph-mon[29098]: pgmap v1158: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 5.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T07:39:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:48 smithi007 ceph-mon[29098]: 6.b scrub starts 2024-02-05T07:39:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:48 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:39:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:48 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:39:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:48 smithi007 ceph-mon[29098]: 6.b scrub ok 2024-02-05T07:39:49.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:48 smithi007 ceph-mon[37158]: pgmap v1158: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 5.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T07:39:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:48 smithi007 ceph-mon[37158]: 6.b scrub starts 2024-02-05T07:39:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:48 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:39:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:48 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:39:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:48 smithi007 ceph-mon[37158]: 6.b scrub ok 2024-02-05T07:39:49.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:49.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:49.270 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:49.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:49.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:48 smithi090 ceph-mon[33213]: pgmap v1158: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 5.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T07:39:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:48 smithi090 ceph-mon[33213]: 6.b scrub starts 2024-02-05T07:39:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:48 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:39:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:48 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:39:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:48 smithi090 ceph-mon[33213]: 6.b scrub ok 2024-02-05T07:39:49.372 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:49.410 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:49.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:49.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:49.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:49.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:49.536 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:49.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:49.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:49.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:49.670 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:49.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:49.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:49.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:49.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:49.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:49.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:49.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:49.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:50.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:50.084 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:50.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:50.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[29098]: 11.e scrub starts 2024-02-05T07:39:50.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[29098]: 1.1a scrub starts 2024-02-05T07:39:50.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[29098]: 1.1a scrub ok 2024-02-05T07:39:50.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[29098]: 4.0 scrub starts 2024-02-05T07:39:50.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[29098]: 4.0 scrub ok 2024-02-05T07:39:50.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[29098]: osdmap e633: 8 total, 8 up, 5 in 2024-02-05T07:39:50.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[29098]: pgmap v1161: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 5.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T07:39:50.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[29098]: osdmap e634: 8 total, 8 up, 5 in 2024-02-05T07:39:50.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[37158]: 11.e scrub starts 2024-02-05T07:39:50.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[37158]: 1.1a scrub starts 2024-02-05T07:39:50.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[37158]: 1.1a scrub ok 2024-02-05T07:39:50.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[37158]: 4.0 scrub starts 2024-02-05T07:39:50.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[37158]: 4.0 scrub ok 2024-02-05T07:39:50.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[37158]: osdmap e633: 8 total, 8 up, 5 in 2024-02-05T07:39:50.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[37158]: pgmap v1161: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 5.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T07:39:50.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:49 smithi007 ceph-mon[37158]: osdmap e634: 8 total, 8 up, 5 in 2024-02-05T07:39:50.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:50.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:50.238 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:50.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:49 smithi090 ceph-mon[33213]: 11.e scrub starts 2024-02-05T07:39:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:49 smithi090 ceph-mon[33213]: 1.1a scrub starts 2024-02-05T07:39:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:49 smithi090 ceph-mon[33213]: 1.1a scrub ok 2024-02-05T07:39:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:49 smithi090 ceph-mon[33213]: 4.0 scrub starts 2024-02-05T07:39:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:49 smithi090 ceph-mon[33213]: 4.0 scrub ok 2024-02-05T07:39:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:49 smithi090 ceph-mon[33213]: osdmap e633: 8 total, 8 up, 5 in 2024-02-05T07:39:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:49 smithi090 ceph-mon[33213]: pgmap v1161: 178 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 174 active+clean; 5.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T07:39:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:49 smithi090 ceph-mon[33213]: osdmap e634: 8 total, 8 up, 5 in 2024-02-05T07:39:50.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:50.364 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:50.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:50.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:50.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:50.512 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:50.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:50.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:50.614 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:50.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:50.639 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:50.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:50.740 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:50.766 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:50.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:50.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:50.868 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:50.912 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:50.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:50.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:51.014 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:51.054 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:51.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:51.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:51.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:50 smithi007 ceph-mon[29098]: 6.4 scrub starts 2024-02-05T07:39:51.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:50 smithi007 ceph-mon[29098]: 6.4 scrub ok 2024-02-05T07:39:51.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:50 smithi007 ceph-mon[29098]: 6.d scrub starts 2024-02-05T07:39:51.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:50 smithi007 ceph-mon[29098]: 11.e scrub ok 2024-02-05T07:39:51.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:50 smithi007 ceph-mon[29098]: 6.d scrub ok 2024-02-05T07:39:51.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:50 smithi007 ceph-mon[37158]: 6.4 scrub starts 2024-02-05T07:39:51.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:50 smithi007 ceph-mon[37158]: 6.4 scrub ok 2024-02-05T07:39:51.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:50 smithi007 ceph-mon[37158]: 6.d scrub starts 2024-02-05T07:39:51.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:50 smithi007 ceph-mon[37158]: 11.e scrub ok 2024-02-05T07:39:51.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:50 smithi007 ceph-mon[37158]: 6.d scrub ok 2024-02-05T07:39:51.187 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:50 smithi090 ceph-mon[33213]: 6.4 scrub starts 2024-02-05T07:39:51.187 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:50 smithi090 ceph-mon[33213]: 6.4 scrub ok 2024-02-05T07:39:51.188 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:50 smithi090 ceph-mon[33213]: 6.d scrub starts 2024-02-05T07:39:51.188 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:50 smithi090 ceph-mon[33213]: 11.e scrub ok 2024-02-05T07:39:51.188 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:50 smithi090 ceph-mon[33213]: 6.d scrub ok 2024-02-05T07:39:51.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:51.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:51.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:51.323 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:51.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:51.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:51.425 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:51.451 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:51.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:51.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:51.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:39:51 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:39:51.553 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:51.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:51.579 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:51.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:51.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:51.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:51.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:51.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:51.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:51.892 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:51.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:51.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:51.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:52.019 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:52.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:52.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 2 1 2024-02-05T07:39:52.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:52.130 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:51 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:52.130 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:51 smithi007 ceph-mon[37158]: pgmap v1163: 178 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 176 active+clean; 5.7 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s rd, 0 B/s wr, 1.20k op/s 2024-02-05T07:39:52.134 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:51 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:52.135 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:51 smithi007 ceph-mon[29098]: pgmap v1163: 178 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 176 active+clean; 5.7 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s rd, 0 B/s wr, 1.20k op/s 2024-02-05T07:39:52.193 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:52.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:52.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:51 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:39:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:51 smithi090 ceph-mon[33213]: pgmap v1163: 178 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 176 active+clean; 5.7 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s rd, 0 B/s wr, 1.20k op/s 2024-02-05T07:39:52.293 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:39:52.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:52.321 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:52.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:52.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:52.349 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:39:52.349 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-02-05T07:39:52.349 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-02-05T07:39:52.349 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:39:52.349 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.349 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:103989 9.d 9:b04f585f:::benchmark_data_smithi007_136200_object24144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:17.620192+0000", 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 454.72379729599999, 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0380600419999999, 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 103989 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.620192+0000", 2024-02-05T07:39:52.350 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.351 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.351 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.351 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.351 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.620192+0000", 2024-02-05T07:39:52.351 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.351 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.351 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.351 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.351 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.620193+0000", 2024-02-05T07:39:52.351 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4600000000000004e-07 2024-02-05T07:39:52.351 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.351 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.351 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.351 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.620196+0000", 2024-02-05T07:39:52.352 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3859999999999999e-06 2024-02-05T07:39:52.352 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.352 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.352 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.352 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.620204+0000", 2024-02-05T07:39:52.352 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1100000000000003e-06 2024-02-05T07:39:52.352 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.352 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.352 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.352 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.620210+0000", 2024-02-05T07:39:52.352 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0419999999999999e-06 2024-02-05T07:39:52.352 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.352 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.352 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.353 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.620265+0000", 2024-02-05T07:39:52.353 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4716000000000003e-05 2024-02-05T07:39:52.353 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.353 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.353 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.353 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.620402+0000", 2024-02-05T07:39:52.353 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000136861 2024-02-05T07:39:52.353 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.353 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.353 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6", 2024-02-05T07:39:52.353 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.620537+0000", 2024-02-05T07:39:52.353 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000134995 2024-02-05T07:39:52.353 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.353 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.622610+0000", 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020734460000000001 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.633350+0000", 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010739589000000001 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.649642+0000", 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016292239 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.354 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.355 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.355 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.649651+0000", 2024-02-05T07:39:52.355 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2869999999999993e-06 2024-02-05T07:39:52.355 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.355 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.355 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.355 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.658160+0000", 2024-02-05T07:39:52.355 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.008508773 2024-02-05T07:39:52.355 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.355 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.355 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.355 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.658252+0000", 2024-02-05T07:39:52.355 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1852000000000001e-05 2024-02-05T07:39:52.355 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:104029 9.d 9:b7c63e21:::benchmark_data_smithi007_136200_object24184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:17.674609+0000", 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 454.669379929, 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99013345399999997, 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 104029 2024-02-05T07:39:52.356 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.674609+0000", 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.674609+0000", 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.674611+0000", 2024-02-05T07:39:52.357 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.528e-06 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.674617+0000", 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2129999999999996e-06 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.674626+0000", 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4460000000000007e-06 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.674634+0000", 2024-02-05T07:39:52.358 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0910000000000004e-06 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.674697+0000", 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3511999999999998e-05 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.674707+0000", 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8519999999999999e-06 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.658768+0000", 2024-02-05T07:39:52.359 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98406127899999996 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.658879+0000", 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011112600000000001 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6", 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.659012+0000", 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000132924 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.664473+0000", 2024-02-05T07:39:52.360 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054604570000000002 2024-02-05T07:39:52.361 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.361 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.361 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.361 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.664592+0000", 2024-02-05T07:39:52.361 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000119725 2024-02-05T07:39:52.361 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.361 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.361 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.361 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.664638+0000", 2024-02-05T07:39:52.361 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5621999999999998e-05 2024-02-05T07:39:52.361 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.361 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.361 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:52.361 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.664716+0000", 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7998999999999996e-05 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.664743+0000", 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6679999999999999e-05 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:104038 9.d 9:b8b0cc0b:::benchmark_data_smithi007_136200_object24193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:17.684035+0000", 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 454.65995380200002, 2024-02-05T07:39:52.362 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98400056999999996, 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 104038 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.684035+0000", 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.363 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.364 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.684035+0000", 2024-02-05T07:39:52.364 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.364 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.364 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.364 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.364 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.684036+0000", 2024-02-05T07:39:52.364 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9699999999999996e-07 2024-02-05T07:39:52.364 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.364 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.364 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.364 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.684039+0000", 2024-02-05T07:39:52.364 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.717e-06 2024-02-05T07:39:52.364 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.364 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.684045+0000", 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3230000000000003e-06 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.684051+0000", 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0310000000000004e-06 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.684602+0000", 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00055138799999999999 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.365 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.684612+0000", 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5920000000000007e-06 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.662613+0000", 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97800077200000002 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.662735+0000", 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012216199999999999 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.366 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6", 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.662866+0000", 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000131505 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.667874+0000", 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.005007849 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.667892+0000", 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7448e-05 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.367 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.667939+0000", 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7348000000000001e-05 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.667997+0000", 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8436000000000001e-05 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.668036+0000", 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8501999999999998e-05 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.368 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25268.0:104044 9.d 9:bc28a87a:::benchmark_data_smithi007_136200_object24199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:32:17.691098+0000", 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 454.652891469, 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98771838099999998, 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25268", 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 104044 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.369 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.691098+0000", 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.691098+0000", 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.691099+0000", 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3480000000000001e-06 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.370 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.691105+0000", 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8839999999999999e-06 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.691121+0000", 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6129000000000001e-05 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.691130+0000", 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7469999999999994e-06 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.371 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.691189+0000", 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9647e-05 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:17.691200+0000", 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.031e-05 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.668245+0000", 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97704539599999995 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.372 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.373 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.668414+0000", 2024-02-05T07:39:52.373 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016861900000000001 2024-02-05T07:39:52.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.373 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.373 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6", 2024-02-05T07:39:52.373 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.668531+0000", 2024-02-05T07:39:52.373 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000117513 2024-02-05T07:39:52.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.373 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.373 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.373 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.674373+0000", 2024-02-05T07:39:52.373 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0058412430000000003 2024-02-05T07:39:52.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.373 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.674412+0000", 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9944000000000002e-05 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.678734+0000", 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043217760000000003 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.678780+0000", 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5521e-05 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.374 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:32:18.678816+0000", 2024-02-05T07:39:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6303999999999997e-05 2024-02-05T07:39:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.375 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:170176 10.19 10:9d6fb8fd:::benchmark_data_smithi007_157098_object41246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e517)", 2024-02-05T07:39:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:58.819268+0000", 2024-02-05T07:39:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 233.524721105, 2024-02-05T07:39:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.79300742300000004, 2024-02-05T07:39:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:39:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:39:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 170176 2024-02-05T07:39:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.376 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.819268+0000", 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.819268+0000", 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.819270+0000", 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6950000000000001e-06 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.377 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.819276+0000", 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0290000000000002e-06 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.819288+0000", 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2565000000000001e-05 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.819298+0000", 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4040000000000001e-06 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.378 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.379 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.819364+0000", 2024-02-05T07:39:52.379 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6354999999999997e-05 2024-02-05T07:39:52.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.379 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.379 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.819374+0000", 2024-02-05T07:39:52.379 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4439999999999997e-06 2024-02-05T07:39:52.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.380 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.380 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.602345+0000", 2024-02-05T07:39:52.380 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.78297119599999998 2024-02-05T07:39:52.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.380 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.380 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.602450+0000", 2024-02-05T07:39:52.380 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000105707 2024-02-05T07:39:52.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.381 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-02-05T07:39:52.381 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.602850+0000", 2024-02-05T07:39:52.381 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00039953100000000001 2024-02-05T07:39:52.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.381 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.381 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.605477+0000", 2024-02-05T07:39:52.381 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0026272460000000002 2024-02-05T07:39:52.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.381 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.381 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.609519+0000", 2024-02-05T07:39:52.381 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0040418939999999999 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.612150+0000", 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0026304200000000001 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.612237+0000", 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7563000000000002e-05 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.612276+0000", 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8374000000000002e-05 2024-02-05T07:39:52.382 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:170215 10.19 10:9c2141b4:::benchmark_data_smithi007_157098_object41285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e517)", 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:35:58.851572+0000", 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 233.49241678499999, 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.76114158899999995, 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 170215 2024-02-05T07:39:52.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.851572+0000", 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.851572+0000", 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.851574+0000", 2024-02-05T07:39:52.384 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1829999999999999e-06 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.851579+0000", 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9989999999999999e-06 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.851589+0000", 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0671e-05 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.851597+0000", 2024-02-05T07:39:52.385 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4410000000000004e-06 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.851664+0000", 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7405999999999998e-05 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:58.851672+0000", 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1219999999999995e-06 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.605509+0000", 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.75383697400000005 2024-02-05T07:39:52.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.605620+0000", 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000111184 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.605968+0000", 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00034755300000000001 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.609479+0000", 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003511194 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.387 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.609585+0000", 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010566200000000001 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.612617+0000", 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030323709999999999 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.612674+0000", 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6876000000000002e-05 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.388 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.389 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.389 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:35:59.612714+0000", 2024-02-05T07:39:52.389 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9953000000000002e-05 2024-02-05T07:39:52.389 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.389 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.389 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.389 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.389 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9053 11.c 11:34de575c:::benchmark_data_smithi007_183880_object9052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-02-05T07:39:52.389 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.428410+0000", 2024-02-05T07:39:52.389 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 103.91557952700001, 2024-02-05T07:39:52.389 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2415406390000001, 2024-02-05T07:39:52.389 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.389 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9053 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428410+0000", 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428410+0000", 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428411+0000", 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.781e-06 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428437+0000", 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5075999999999998e-05 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428455+0000", 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8660000000000001e-05 2024-02-05T07:39:52.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.392 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.392 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428464+0000", 2024-02-05T07:39:52.392 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2299999999999997e-06 2024-02-05T07:39:52.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.392 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.392 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428677+0000", 2024-02-05T07:39:52.392 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00021306999999999999 2024-02-05T07:39:52.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.392 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.392 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428801+0000", 2024-02-05T07:39:52.392 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012398800000000001 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,4", 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428945+0000", 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014369900000000001 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.433109+0000", 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041642040000000003 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.462929+0000", 2024-02-05T07:39:52.393 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.029819735999999999 2024-02-05T07:39:52.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.394 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.394 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.462941+0000", 2024-02-05T07:39:52.394 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.208e-05 2024-02-05T07:39:52.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.394 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.394 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.452757+0000", 2024-02-05T07:39:52.394 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98981587299999996 2024-02-05T07:39:52.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.452774+0000", 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6829000000000001e-05 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.660647+0000", 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.20787291099999999 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.661191+0000", 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00054389100000000001 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.396 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.396 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.669828+0000", 2024-02-05T07:39:52.396 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0086373700000000001 2024-02-05T07:39:52.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.396 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.396 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.669950+0000", 2024-02-05T07:39:52.396 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012224100000000001 2024-02-05T07:39:52.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.396 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.396 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9054 11.c 11:3aa3c728:::benchmark_data_smithi007_183880_object9053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.428484+0000", 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 103.915504787, 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2467161410000001, 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9054 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428484+0000", 2024-02-05T07:39:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428484+0000", 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428485+0000", 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5000000000000002e-07 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428509+0000", 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3476e-05 2024-02-05T07:39:52.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428515+0000", 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8619999999999997e-06 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428521+0000", 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0499999999999997e-06 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.430743+0000", 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0022218989999999998 2024-02-05T07:39:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.430856+0000", 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011233199999999999 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,4", 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.430996+0000", 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000140813 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.433202+0000", 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002205631 2024-02-05T07:39:52.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.463242+0000", 2024-02-05T07:39:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.030040291 2024-02-05T07:39:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.463254+0000", 2024-02-05T07:39:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1518999999999999e-05 2024-02-05T07:39:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.401 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453065+0000", 2024-02-05T07:39:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98981125199999997 2024-02-05T07:39:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453077+0000", 2024-02-05T07:39:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1953000000000001e-05 2024-02-05T07:39:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.402 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.670057+0000", 2024-02-05T07:39:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.21697988800000001 2024-02-05T07:39:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-02-05T07:39:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.670611+0000", 2024-02-05T07:39:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00055360800000000003 2024-02-05T07:39:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.675097+0000", 2024-02-05T07:39:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044864789999999998 2024-02-05T07:39:52.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.404 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.404 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.404 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.675201+0000", 2024-02-05T07:39:52.404 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010343799999999999 2024-02-05T07:39:52.404 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.404 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.404 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.404 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.404 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.404 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9056 11.7 11:e61436ac:::benchmark_data_smithi007_183880_object9055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-02-05T07:39:52.404 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.428530+0000", 2024-02-05T07:39:52.404 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 103.915458798, 2024-02-05T07:39:52.404 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.228157197, 2024-02-05T07:39:52.404 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:39:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:39:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:39:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9056 2024-02-05T07:39:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428530+0000", 2024-02-05T07:39:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428530+0000", 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428531+0000", 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0299999999999999e-07 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428551+0000", 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9644999999999999e-05 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.406 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428557+0000", 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.331e-06 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428563+0000", 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7610000000000004e-06 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428627+0000", 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4387000000000005e-05 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.407 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.408 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428774+0000", 2024-02-05T07:39:52.408 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000147319 2024-02-05T07:39:52.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.408 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,4", 2024-02-05T07:39:52.408 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.428934+0000", 2024-02-05T07:39:52.408 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015985500000000001 2024-02-05T07:39:52.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.408 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.408 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.432342+0000", 2024-02-05T07:39:52.408 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003407818 2024-02-05T07:39:52.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.409 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.409 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.437218+0000", 2024-02-05T07:39:52.409 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.004876282 2024-02-05T07:39:52.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.409 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.409 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.410 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.462644+0000", 2024-02-05T07:39:52.410 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.025425738 2024-02-05T07:39:52.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.410 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.410 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.410 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.462658+0000", 2024-02-05T07:39:52.410 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3799e-05 2024-02-05T07:39:52.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.410 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.410 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.410 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.456214+0000", 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99355635600000003 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.456226+0000", 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1989e-05 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.656473+0000", 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.200247022 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.411 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.656688+0000", 2024-02-05T07:39:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00021439199999999999 2024-02-05T07:39:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9059 11.7 11:e1ac2e66:::benchmark_data_smithi007_183880_object9058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-02-05T07:39:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.429417+0000", 2024-02-05T07:39:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 103.914571858, 2024-02-05T07:39:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.229019305, 2024-02-05T07:39:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:39:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9059 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.429417+0000", 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.429417+0000", 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.413 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.429419+0000", 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3769999999999999e-06 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.429441+0000", 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2240999999999999e-05 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.429452+0000", 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1216000000000001e-05 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.414 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.429461+0000", 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9039999999999998e-06 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.429913+0000", 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00045225499999999999 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.430029+0000", 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011539700000000001 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.415 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.416 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,4", 2024-02-05T07:39:52.416 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.430195+0000", 2024-02-05T07:39:52.416 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000166038 2024-02-05T07:39:52.416 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.416 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.416 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.416 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.432681+0000", 2024-02-05T07:39:52.416 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024860160000000002 2024-02-05T07:39:52.416 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.416 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.416 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.416 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.437270+0000", 2024-02-05T07:39:52.416 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0045892880000000004 2024-02-05T07:39:52.416 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.417 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.417 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.417 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.462696+0000", 2024-02-05T07:39:52.417 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.025425458000000001 2024-02-05T07:39:52.417 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.417 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.417 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.417 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.462706+0000", 2024-02-05T07:39:52.417 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0837e-05 2024-02-05T07:39:52.417 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.417 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.417 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.417 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.456262+0000", 2024-02-05T07:39:52.417 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99355588299999997 2024-02-05T07:39:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.456273+0000", 2024-02-05T07:39:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.046e-05 2024-02-05T07:39:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.658337+0000", 2024-02-05T07:39:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.20206384899999999 2024-02-05T07:39:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.658437+0000", 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000100086 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9060 11.c 11:379708f4:::benchmark_data_smithi007_183880_object9059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.429474+0000", 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 103.914515235, 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2479900079999999, 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:39:52.419 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:39:52.420 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9060 2024-02-05T07:39:52.420 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.420 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.420 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.420 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.420 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.429474+0000", 2024-02-05T07:39:52.420 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.420 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.420 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.420 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.420 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.429474+0000", 2024-02-05T07:39:52.420 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.420 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.420 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.429474+0000", 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2300000000000001e-07 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.429495+0000", 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0497999999999999e-05 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.429502+0000", 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9589999999999998e-06 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.422 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.422 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.429508+0000", 2024-02-05T07:39:52.422 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2199999999999997e-06 2024-02-05T07:39:52.422 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.422 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.422 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.422 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.431381+0000", 2024-02-05T07:39:52.422 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001873261 2024-02-05T07:39:52.422 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.422 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.422 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.422 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.431499+0000", 2024-02-05T07:39:52.422 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000117853 2024-02-05T07:39:52.422 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.423 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.423 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,4", 2024-02-05T07:39:52.423 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.431633+0000", 2024-02-05T07:39:52.423 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013350999999999999 2024-02-05T07:39:52.423 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.423 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.423 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.423 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.434965+0000", 2024-02-05T07:39:52.423 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033318459999999999 2024-02-05T07:39:52.423 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.423 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.423 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.423 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.465360+0000", 2024-02-05T07:39:52.424 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.030395062 2024-02-05T07:39:52.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.424 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.424 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.424 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.465373+0000", 2024-02-05T07:39:52.424 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3689000000000001e-05 2024-02-05T07:39:52.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.424 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.424 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.424 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453115+0000", 2024-02-05T07:39:52.424 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98774174000000003 2024-02-05T07:39:52.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.424 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.424 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453125+0000", 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7969999999999995e-06 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.670664+0000", 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.21753890100000001 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.671359+0000", 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00069508999999999999 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.425 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.677370+0000", 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0060108970000000003 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.677464+0000", 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4161999999999997e-05 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9052 11.e 11:7344e182:::benchmark_data_smithi007_183880_object9051:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e588)", 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.443553+0000", 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 103.90043621300001, 2024-02-05T07:39:52.426 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2262886049999999, 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9052 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.443553+0000", 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.427 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.443553+0000", 2024-02-05T07:39:52.428 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.428 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.428 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.443555+0000", 2024-02-05T07:39:52.428 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7600000000000001e-06 2024-02-05T07:39:52.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.428 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.428 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.443583+0000", 2024-02-05T07:39:52.428 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8441000000000001e-05 2024-02-05T07:39:52.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.428 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.429 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.443594+0000", 2024-02-05T07:39:52.429 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0781999999999999e-05 2024-02-05T07:39:52.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.429 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.429 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:52.430 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.443606+0000", 2024-02-05T07:39:52.430 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2278e-05 2024-02-05T07:39:52.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.431 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.431 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.431 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.453784+0000", 2024-02-05T07:39:52.431 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010177581999999999 2024-02-05T07:39:52.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.431 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.431 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.431 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.453796+0000", 2024-02-05T07:39:52.431 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2581e-05 2024-02-05T07:39:52.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.431 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.431 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.431 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453533+0000", 2024-02-05T07:39:52.432 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99973690400000004 2024-02-05T07:39:52.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.432 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.432 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453546+0000", 2024-02-05T07:39:52.432 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3145999999999999e-05 2024-02-05T07:39:52.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.432 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.432 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.669534+0000", 2024-02-05T07:39:52.432 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.21598804799999999 2024-02-05T07:39:52.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.433 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.433 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.669842+0000", 2024-02-05T07:39:52.433 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030708299999999999 2024-02-05T07:39:52.433 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.433 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.433 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.433 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9072 11.e 11:7a64285d:::benchmark_data_smithi007_183880_object9071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-02-05T07:39:52.433 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.470288+0000", 2024-02-05T07:39:52.433 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 103.873701541, 2024-02-05T07:39:52.433 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.217309499, 2024-02-05T07:39:52.434 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.434 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:52.434 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.434 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:39:52.434 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:39:52.434 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9072 2024-02-05T07:39:52.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.434 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.434 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.434 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.470288+0000", 2024-02-05T07:39:52.434 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.470288+0000", 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.470289+0000", 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.649e-06 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.470315+0000", 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5531e-05 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.436 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.436 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.470326+0000", 2024-02-05T07:39:52.436 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1399e-05 2024-02-05T07:39:52.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.436 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.436 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.436 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.470335+0000", 2024-02-05T07:39:52.436 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0129999999999999e-06 2024-02-05T07:39:52.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.436 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.436 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.436 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.470399+0000", 2024-02-05T07:39:52.436 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3396000000000001e-05 2024-02-05T07:39:52.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.470411+0000", 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2486e-05 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453575+0000", 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98316424800000002 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453585+0000", 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9969999999999993e-06 2024-02-05T07:39:52.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.438 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.438 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.438 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.670304+0000", 2024-02-05T07:39:52.438 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.21671885099999999 2024-02-05T07:39:52.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.438 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.438 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.438 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.670490+0000", 2024-02-05T07:39:52.438 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018613399999999999 2024-02-05T07:39:52.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.438 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.438 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,3", 2024-02-05T07:39:52.438 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.670643+0000", 2024-02-05T07:39:52.438 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015253700000000001 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.673891+0000", 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032483859999999998 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.681627+0000", 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.007735942 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.686781+0000", 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0051536009999999998 2024-02-05T07:39:52.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.440 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:52.440 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.686839+0000", 2024-02-05T07:39:52.440 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8653999999999998e-05 2024-02-05T07:39:52.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.440 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.440 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.687597+0000", 2024-02-05T07:39:52.440 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00075767499999999997 2024-02-05T07:39:52.440 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.440 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.440 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9074 11.c 11:30e00818:::benchmark_data_smithi007_183880_object9073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.470366+0000", 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 103.87362274199999, 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2082120759999999, 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9074 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.470366+0000", 2024-02-05T07:39:52.441 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.470366+0000", 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.470367+0000", 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8899999999999999e-07 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.470389+0000", 2024-02-05T07:39:52.442 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1889e-05 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.470396+0000", 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8900000000000001e-06 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.470403+0000", 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9199999999999998e-06 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.471084+0000", 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00068104199999999998 2024-02-05T07:39:52.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.471092+0000", 2024-02-05T07:39:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3690000000000007e-06 2024-02-05T07:39:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453163+0000", 2024-02-05T07:39:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98207028600000001 2024-02-05T07:39:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.444 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453170+0000", 2024-02-05T07:39:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5009999999999998e-06 2024-02-05T07:39:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.672010+0000", 2024-02-05T07:39:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.21883965 2024-02-05T07:39:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.672170+0000", 2024-02-05T07:39:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016077499999999999 2024-02-05T07:39:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.445 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:39:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.672467+0000", 2024-02-05T07:39:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029617199999999998 2024-02-05T07:39:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.675994+0000", 2024-02-05T07:39:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0035273769999999999 2024-02-05T07:39:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.677506+0000", 2024-02-05T07:39:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0015119459999999999 2024-02-05T07:39:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.678015+0000", 2024-02-05T07:39:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00050897900000000001 2024-02-05T07:39:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.678115+0000", 2024-02-05T07:39:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000100108 2024-02-05T07:39:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.678579+0000", 2024-02-05T07:39:52.447 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00046348299999999999 2024-02-05T07:39:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9077 11.e 11:7533e084:::benchmark_data_smithi007_183880_object9076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-02-05T07:39:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.473605+0000", 2024-02-05T07:39:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 103.870384403, 2024-02-05T07:39:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2143414720000001, 2024-02-05T07:39:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.448 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:39:52.449 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:39:52.449 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9077 2024-02-05T07:39:52.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.449 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.449 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.449 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.473605+0000", 2024-02-05T07:39:52.449 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.449 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.449 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.473605+0000", 2024-02-05T07:39:52.449 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.450 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.450 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.450 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.473606+0000", 2024-02-05T07:39:52.450 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.579e-06 2024-02-05T07:39:52.450 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.450 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.450 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.450 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.473659+0000", 2024-02-05T07:39:52.450 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2249000000000001e-05 2024-02-05T07:39:52.450 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.450 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.451 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.451 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.473674+0000", 2024-02-05T07:39:52.451 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5007999999999999e-05 2024-02-05T07:39:52.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.451 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.451 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.451 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.473683+0000", 2024-02-05T07:39:52.451 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9309999999999997e-06 2024-02-05T07:39:52.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.451 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.451 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.451 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.473746+0000", 2024-02-05T07:39:52.451 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3668000000000001e-05 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.473765+0000", 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8997000000000001e-05 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453617+0000", 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97985213699999996 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453629+0000", 2024-02-05T07:39:52.452 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1573000000000001e-05 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.672320+0000", 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.21869149099999999 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.672473+0000", 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000152965 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,3", 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.672646+0000", 2024-02-05T07:39:52.453 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000172239 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.681683+0000", 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0090377419999999997 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.681734+0000", 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0154999999999997e-05 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.687607+0000", 2024-02-05T07:39:52.454 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0058737930000000004 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.687654+0000", 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6897999999999997e-05 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.687946+0000", 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029204699999999998 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.455 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9082 11.e 11:7a3bcaed:::benchmark_data_smithi007_183880_object9081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.481410+0000", 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 103.862579228, 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.206831475, 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9082 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.456 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481410+0000", 2024-02-05T07:39:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481410+0000", 2024-02-05T07:39:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481412+0000", 2024-02-05T07:39:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.278e-06 2024-02-05T07:39:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.457 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481467+0000", 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4333999999999999e-05 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481479+0000", 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2343e-05 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481488+0000", 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6009999999999995e-06 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.458 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481777+0000", 2024-02-05T07:39:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028944300000000001 2024-02-05T07:39:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481785+0000", 2024-02-05T07:39:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0229999999999992e-06 2024-02-05T07:39:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453666+0000", 2024-02-05T07:39:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.971881404 2024-02-05T07:39:52.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453676+0000", 2024-02-05T07:39:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1209999999999996e-06 2024-02-05T07:39:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.673933+0000", 2024-02-05T07:39:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.22025797699999999 2024-02-05T07:39:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.460 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.674783+0000", 2024-02-05T07:39:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00084982000000000005 2024-02-05T07:39:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,3", 2024-02-05T07:39:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.675296+0000", 2024-02-05T07:39:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000512566 2024-02-05T07:39:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.681782+0000", 2024-02-05T07:39:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0064862239999999996 2024-02-05T07:39:52.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.681832+0000", 2024-02-05T07:39:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0142999999999997e-05 2024-02-05T07:39:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.687955+0000", 2024-02-05T07:39:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0061230570000000003 2024-02-05T07:39:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.688003+0000", 2024-02-05T07:39:52.462 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8196999999999999e-05 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.688241+0000", 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000237944 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9084 11.e 11:7880e7d4:::benchmark_data_smithi007_183880_object9083:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.481723+0000", 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 103.862265697, 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2068470419999999, 2024-02-05T07:39:52.463 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.464 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:52.464 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.464 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:39:52.464 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:39:52.464 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9084 2024-02-05T07:39:52.464 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.464 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.464 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.464 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.464 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481723+0000", 2024-02-05T07:39:52.464 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.464 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.464 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.464 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.465 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481723+0000", 2024-02-05T07:39:52.465 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.465 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.465 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.465 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.465 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481724+0000", 2024-02-05T07:39:52.465 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5099999999999999e-07 2024-02-05T07:39:52.465 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.465 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.465 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.465 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481746+0000", 2024-02-05T07:39:52.465 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2076999999999999e-05 2024-02-05T07:39:52.465 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.465 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.466 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.466 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481757+0000", 2024-02-05T07:39:52.466 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0572e-05 2024-02-05T07:39:52.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.466 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.466 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.466 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481765+0000", 2024-02-05T07:39:52.466 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8809999999999996e-06 2024-02-05T07:39:52.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.466 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.466 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.466 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481823+0000", 2024-02-05T07:39:52.466 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7741000000000002e-05 2024-02-05T07:39:52.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.467 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.467 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.467 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481832+0000", 2024-02-05T07:39:52.467 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9699999999999994e-06 2024-02-05T07:39:52.467 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.467 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.467 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.467 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.454210+0000", 2024-02-05T07:39:52.467 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97237788800000002 2024-02-05T07:39:52.467 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.467 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.467 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.467 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.454222+0000", 2024-02-05T07:39:52.467 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1986e-05 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.678671+0000", 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.224448919 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.679046+0000", 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00037505900000000002 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,3", 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.679218+0000", 2024-02-05T07:39:52.468 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017129800000000001 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.681885+0000", 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0026676040000000001 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.684735+0000", 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002849702 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.688250+0000", 2024-02-05T07:39:52.469 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003515345 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.688335+0000", 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4820000000000004e-05 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.688571+0000", 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000235429 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.470 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.471 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9098 11.c 11:3bdc3e2e:::benchmark_data_smithi007_183880_object9097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-02-05T07:39:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.489205+0000", 2024-02-05T07:39:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 103.854784649, 2024-02-05T07:39:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2012975960000001, 2024-02-05T07:39:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:39:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:39:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9098 2024-02-05T07:39:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.489205+0000", 2024-02-05T07:39:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.489205+0000", 2024-02-05T07:39:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.489206+0000", 2024-02-05T07:39:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8759999999999999e-06 2024-02-05T07:39:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.489230+0000", 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3113e-05 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.489242+0000", 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2819999999999999e-05 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.489250+0000", 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0800000000000006e-06 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.474 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.489317+0000", 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6991999999999998e-05 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.489332+0000", 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4123e-05 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453199+0000", 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96386785699999999 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.475 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.476 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.476 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.453208+0000", 2024-02-05T07:39:52.476 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1599999999999998e-06 2024-02-05T07:39:52.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.476 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.476 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.476 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.678701+0000", 2024-02-05T07:39:52.476 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.22549367300000001 2024-02-05T07:39:52.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.476 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.476 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.476 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.678833+0000", 2024-02-05T07:39:52.476 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000131595 2024-02-05T07:39:52.477 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.477 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.477 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:39:52.477 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.680820+0000", 2024-02-05T07:39:52.477 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0019875719999999999 2024-02-05T07:39:52.477 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.477 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.477 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.477 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.687284+0000", 2024-02-05T07:39:52.477 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0064634970000000003 2024-02-05T07:39:52.477 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.477 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.477 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.477 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.687319+0000", 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5553000000000003e-05 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.687378+0000", 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9002999999999997e-05 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.687436+0000", 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7513999999999999e-05 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.478 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.690502+0000", 2024-02-05T07:39:52.479 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030661680000000002 2024-02-05T07:39:52.479 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.479 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.479 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.479 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.479 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.479 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9099 11.7 11:e5b88fbc:::benchmark_data_smithi007_183880_object9098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-02-05T07:39:52.479 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.490428+0000", 2024-02-05T07:39:52.479 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 103.85356158899999, 2024-02-05T07:39:52.479 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.171342332, 2024-02-05T07:39:52.479 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.479 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:52.479 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.479 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9099 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.490428+0000", 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.490428+0000", 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.480 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.481 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.481 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.490429+0000", 2024-02-05T07:39:52.481 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0599999999999999e-07 2024-02-05T07:39:52.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.481 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.481 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.481 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.490446+0000", 2024-02-05T07:39:52.481 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7897000000000002e-05 2024-02-05T07:39:52.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.481 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.481 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.481 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.490459+0000", 2024-02-05T07:39:52.481 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2143999999999999e-05 2024-02-05T07:39:52.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.490467+0000", 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7290000000000006e-06 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.490535+0000", 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7497999999999996e-05 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.490548+0000", 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3499000000000001e-05 2024-02-05T07:39:52.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.456301+0000", 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96575290700000005 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.456310+0000", 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4309999999999994e-06 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.658478+0000", 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.202168662 2024-02-05T07:39:52.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.484 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.484 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.484 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.658654+0000", 2024-02-05T07:39:52.484 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017597199999999999 2024-02-05T07:39:52.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.484 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.484 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-02-05T07:39:52.484 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.659263+0000", 2024-02-05T07:39:52.484 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00060851000000000004 2024-02-05T07:39:52.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.484 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.484 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.484 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.660986+0000", 2024-02-05T07:39:52.484 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001723238 2024-02-05T07:39:52.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.485 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.485 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.485 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.661004+0000", 2024-02-05T07:39:52.485 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8340000000000001e-05 2024-02-05T07:39:52.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.485 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.485 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.485 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.661052+0000", 2024-02-05T07:39:52.485 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.782e-05 2024-02-05T07:39:52.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.485 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.485 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:52.485 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.661126+0000", 2024-02-05T07:39:52.486 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3919999999999997e-05 2024-02-05T07:39:52.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.486 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.486 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.486 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.661770+0000", 2024-02-05T07:39:52.486 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00064385899999999999 2024-02-05T07:39:52.486 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.486 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.486 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.486 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.486 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:43511 11.7 11:e3bfb7ae:::benchmark_data_smithi007_183880_object43510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e604)", 2024-02-05T07:39:52.486 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:35.778366+0000", 2024-02-05T07:39:52.486 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 76.565623049999999, 2024-02-05T07:39:52.487 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16211499200000001, 2024-02-05T07:39:52.487 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:39:52.487 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:39:52.487 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:39:52.487 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:39:52.487 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:39:52.487 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 43511 2024-02-05T07:39:52.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.487 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:39:52.487 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.487 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:39:52.487 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:35.778366+0000", 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:35.778366+0000", 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:35.778368+0000", 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6869999999999999e-06 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:39:52.488 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:35.778389+0000", 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1673000000000001e-05 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:35.778401+0000", 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1804000000000001e-05 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:35.778409+0000", 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1249999999999993e-06 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:39:52.489 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:35.780697+0000", 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0022870799999999999 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:35.780813+0000", 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000116464 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:35.780963+0000", 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000150099 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.490 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:35.787856+0000", 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0068925849999999997 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:35.794565+0000", 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0067093090000000001 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:35.940259+0000", 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.14569435999999999 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:39:52.491 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:35.940420+0000", 2024-02-05T07:39:52.492 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000160216 2024-02-05T07:39:52.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:39:52.492 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:39:52.492 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:39:52.492 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:35.940481+0000", 2024-02-05T07:39:52.492 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1589999999999998e-05 2024-02-05T07:39:52.492 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.492 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.492 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.492 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:39:52.492 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:39:52.492 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:39:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:52.519 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:52.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:52.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:52.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:52.663 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:52.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:52.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:52.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:52.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:52.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:52.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:52.953 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:52.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:53.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:53.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:52 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:39:53.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:52 smithi007 ceph-mon[29098]: 5.7 scrub starts 2024-02-05T07:39:53.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:52 smithi007 ceph-mon[29098]: 5.7 scrub ok 2024-02-05T07:39:53.097 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:53.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:52 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:39:53.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:52 smithi007 ceph-mon[37158]: 5.7 scrub starts 2024-02-05T07:39:53.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:52 smithi007 ceph-mon[37158]: 5.7 scrub ok 2024-02-05T07:39:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:53.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:53.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:53.236 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:53.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:53.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:53.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:52 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:39:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:52 smithi090 ceph-mon[33213]: 5.7 scrub starts 2024-02-05T07:39:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:52 smithi090 ceph-mon[33213]: 5.7 scrub ok 2024-02-05T07:39:53.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:53.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:53.380 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:53.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:53.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:53.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:53.507 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:53.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:53.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:53.633 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:53.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:53.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:53.735 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:53.778 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:53.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:53.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:53.797 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:39:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:39:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:39:53.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T07:39:53.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:53.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:54.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:54.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:54.062 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:54.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:54.104 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:39:54.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:53 smithi007 ceph-mon[29098]: 3.13 scrub starts 2024-02-05T07:39:54.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:53 smithi007 ceph-mon[29098]: 3.13 scrub ok 2024-02-05T07:39:54.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:53 smithi007 ceph-mon[29098]: pgmap v1164: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 176 active+clean; 5.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s 2024-02-05T07:39:54.105 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:53 smithi007 ceph-mon[37158]: 3.13 scrub starts 2024-02-05T07:39:54.105 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:53 smithi007 ceph-mon[37158]: 3.13 scrub ok 2024-02-05T07:39:54.105 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:53 smithi007 ceph-mon[37158]: pgmap v1164: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 176 active+clean; 5.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s 2024-02-05T07:39:54.156 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:39:54.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:54.188 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:54.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:54.195 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:39:54.244 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:39:54.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:53 smithi090 ceph-mon[33213]: 3.13 scrub starts 2024-02-05T07:39:54.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:53 smithi090 ceph-mon[33213]: 3.13 scrub ok 2024-02-05T07:39:54.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:53 smithi090 ceph-mon[33213]: pgmap v1164: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 176 active+clean; 5.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s 2024-02-05T07:39:54.287 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:39:54.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:54.314 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:54.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:54.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:54.326 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:39:54.367 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:39:54.408 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:39:54.413 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:39:54.413 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:39:54.413 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:39:54.414 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:39:54.414 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:39:54.414 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:39:54.414 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:39:54.414 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:39:54.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:54.464 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:54.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:54.565 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:54.590 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:54.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:54.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:54.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:54.731 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:54.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:54.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:54.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:54.898 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:54.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:55.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:55.025 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:55.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:55.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:55.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:55.151 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:55.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:55.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:54 smithi007 ceph-mon[29098]: 6.e scrub starts 2024-02-05T07:39:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:54 smithi007 ceph-mon[29098]: 6.e scrub ok 2024-02-05T07:39:55.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:54 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4001849802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:39:55.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:54 smithi007 ceph-mon[37158]: 6.e scrub starts 2024-02-05T07:39:55.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:54 smithi007 ceph-mon[37158]: 6.e scrub ok 2024-02-05T07:39:55.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:54 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4001849802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:39:55.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:55.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:54 smithi090 ceph-mon[33213]: 6.e scrub starts 2024-02-05T07:39:55.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:54 smithi090 ceph-mon[33213]: 6.e scrub ok 2024-02-05T07:39:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:54 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4001849802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:39:55.277 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:55.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:55.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:55.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:55.423 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:55.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:55.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:55.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:55.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:55.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:55.693 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:55.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:55.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:55.820 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:55.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:55.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:55.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:55.947 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:55.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:55.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:56.049 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:56.093 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:56.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:56.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:56.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:55 smithi007 ceph-mon[29098]: 6.0 scrub starts 2024-02-05T07:39:56.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:55 smithi007 ceph-mon[29098]: 6.0 scrub ok 2024-02-05T07:39:56.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:55 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:39:56.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:55 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:39:56.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:55 smithi007 ceph-mon[29098]: pgmap v1165: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 5.4 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T07:39:56.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:55 smithi007 ceph-mon[29098]: 4.1 scrub starts 2024-02-05T07:39:56.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:55 smithi007 ceph-mon[29098]: 4.1 scrub ok 2024-02-05T07:39:56.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:55 smithi007 ceph-mon[37158]: 6.0 scrub starts 2024-02-05T07:39:56.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:55 smithi007 ceph-mon[37158]: 6.0 scrub ok 2024-02-05T07:39:56.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:55 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:39:56.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:55 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:39:56.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:55 smithi007 ceph-mon[37158]: pgmap v1165: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 5.4 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T07:39:56.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:55 smithi007 ceph-mon[37158]: 4.1 scrub starts 2024-02-05T07:39:56.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:55 smithi007 ceph-mon[37158]: 4.1 scrub ok 2024-02-05T07:39:56.195 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:56.220 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:56.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:56.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:56.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:55 smithi090 ceph-mon[33213]: 6.0 scrub starts 2024-02-05T07:39:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:55 smithi090 ceph-mon[33213]: 6.0 scrub ok 2024-02-05T07:39:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:55 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:39:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:55 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:39:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:55 smithi090 ceph-mon[33213]: pgmap v1165: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 5.4 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T07:39:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:55 smithi090 ceph-mon[33213]: 4.1 scrub starts 2024-02-05T07:39:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:55 smithi090 ceph-mon[33213]: 4.1 scrub ok 2024-02-05T07:39:56.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:56.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:56.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:56.509 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:56.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:56.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:56.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:56.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:56.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:56.776 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:56.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:56.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:56.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:56.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:56.918 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:56.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:57.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:57.045 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:57.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:57.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:57.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:56 smithi007 ceph-mon[29098]: 5.12 scrub starts 2024-02-05T07:39:57.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:56 smithi007 ceph-mon[29098]: 5.12 scrub ok 2024-02-05T07:39:57.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:56 smithi007 ceph-mon[29098]: osdmap e635: 8 total, 8 up, 5 in 2024-02-05T07:39:57.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:56 smithi007 ceph-mon[37158]: 5.12 scrub starts 2024-02-05T07:39:57.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:56 smithi007 ceph-mon[37158]: 5.12 scrub ok 2024-02-05T07:39:57.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:56 smithi007 ceph-mon[37158]: osdmap e635: 8 total, 8 up, 5 in 2024-02-05T07:39:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:57.174 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:57.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:57.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:56 smithi090 ceph-mon[33213]: 5.12 scrub starts 2024-02-05T07:39:57.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:56 smithi090 ceph-mon[33213]: 5.12 scrub ok 2024-02-05T07:39:57.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:56 smithi090 ceph-mon[33213]: osdmap e635: 8 total, 8 up, 5 in 2024-02-05T07:39:57.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:57.301 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:57.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:57.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:57.428 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:57.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:57.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:57.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:57.567 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:57.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:57.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:57.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:57.694 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:57.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:57.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:57.820 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:57.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:57.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:57.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:57.962 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:57.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:57.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:58.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:39:58.090 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:58.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:39:58.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:58.233 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:58.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:58.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:57 smithi090 ceph-mon[33213]: 3.4 scrub starts 2024-02-05T07:39:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:57 smithi090 ceph-mon[33213]: 3.4 scrub ok 2024-02-05T07:39:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:39:57 smithi090 ceph-mon[33213]: pgmap v1167: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-02-05T07:39:58.306 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:57 smithi007 ceph-mon[29098]: 3.4 scrub starts 2024-02-05T07:39:58.306 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:57 smithi007 ceph-mon[29098]: 3.4 scrub ok 2024-02-05T07:39:58.307 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:39:57 smithi007 ceph-mon[29098]: pgmap v1167: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-02-05T07:39:58.307 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:57 smithi007 ceph-mon[37158]: 3.4 scrub starts 2024-02-05T07:39:58.307 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:57 smithi007 ceph-mon[37158]: 3.4 scrub ok 2024-02-05T07:39:58.307 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:39:57 smithi007 ceph-mon[37158]: pgmap v1167: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-02-05T07:39:58.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:58.360 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:58.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:58.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:58.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:39:58.502 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:58.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:39:58.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:39:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:58.639 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:58.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:39:58.740 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:58.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:58.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:58.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:58.892 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:58.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:58.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:58.993 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:59.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:59.030 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:59.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:59.131 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:39:59.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:59.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:39:59.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:59.300 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:59.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:39:59.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:39:59.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:59.426 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:59.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:39:59.527 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:59.568 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:59.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:59.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:39:59.696 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:59.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:59.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:39:59.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:39:59.835 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:39:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:59.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:39:59.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:39:59.962 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:39:59.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:39:59.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:00.064 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:00.087 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:00.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:00.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:00.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:00.230 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:00.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:00.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:00.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:00.356 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:00.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:00.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:00.481 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:00.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:00.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:00.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:00.624 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:00.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:00.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:00.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:00.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:00.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:00.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:00.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:00.880 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:00.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:00.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:00.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[29098]: pgmap v1168: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 5.1 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T07:40:00.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[29098]: 5.c scrub starts 2024-02-05T07:40:00.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[29098]: 5.c scrub ok 2024-02-05T07:40:00.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[29098]: osdmap e636: 8 total, 8 up, 5 in 2024-02-05T07:40:00.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[29098]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-02-05T07:40:00.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[29098]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-02-05T07:40:00.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[29098]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-02-05T07:40:00.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[29098]: pg 1.1e is active+remapped+backfill_toofull, acting [3,0,6] 2024-02-05T07:40:00.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[37158]: pgmap v1168: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 5.1 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T07:40:00.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[37158]: 5.c scrub starts 2024-02-05T07:40:00.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[37158]: 5.c scrub ok 2024-02-05T07:40:00.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[37158]: osdmap e636: 8 total, 8 up, 5 in 2024-02-05T07:40:00.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[37158]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-02-05T07:40:00.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[37158]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-02-05T07:40:00.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[37158]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-02-05T07:40:00.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:00 smithi007 ceph-mon[37158]: pg 1.1e is active+remapped+backfill_toofull, acting [3,0,6] 2024-02-05T07:40:00.975 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:00 smithi090 ceph-mon[33213]: pgmap v1168: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 5.1 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T07:40:00.976 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:00 smithi090 ceph-mon[33213]: 5.c scrub starts 2024-02-05T07:40:00.976 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:00 smithi090 ceph-mon[33213]: 5.c scrub ok 2024-02-05T07:40:00.976 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:00 smithi090 ceph-mon[33213]: osdmap e636: 8 total, 8 up, 5 in 2024-02-05T07:40:00.976 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:00 smithi090 ceph-mon[33213]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-02-05T07:40:00.976 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:00 smithi090 ceph-mon[33213]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-02-05T07:40:00.976 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:00 smithi090 ceph-mon[33213]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-02-05T07:40:00.976 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:00 smithi090 ceph-mon[33213]: pg 1.1e is active+remapped+backfill_toofull, acting [3,0,6] 2024-02-05T07:40:00.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:01.006 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:01.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:01.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:01.132 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:01.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:01.233 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:01.258 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:01.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:01.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:01.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:40:01 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:40:01.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:01.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:01.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:01.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:01.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:01.512 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:01.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:01.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:01.638 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:01.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:01.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:01.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:01.779 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:01.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:01.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:01.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:01.906 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:01.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:02.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:02.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:01 smithi090 ceph-mon[33213]: 11.7 scrub starts 2024-02-05T07:40:02.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:01 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:40:02.030 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:02.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:02.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:02.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:02.155 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:02.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:02.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:02.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:01 smithi007 ceph-mon[29098]: 11.7 scrub starts 2024-02-05T07:40:02.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:01 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:40:02.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:01 smithi007 ceph-mon[37158]: 11.7 scrub starts 2024-02-05T07:40:02.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:01 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:40:02.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:02.281 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:02.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:02.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:02.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:02.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:02.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:02.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:02.554 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:02.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:02.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:02.655 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:02.696 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:02.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:02.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:02.783 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:02.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:02.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:02.873 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:02.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:02.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:03.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:02 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:03.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:02 smithi090 ceph-mon[33213]: pgmap v1170: 178 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 176 active+clean; 4.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-02-05T07:40:03.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:02 smithi090 ceph-mon[33213]: 6.10 scrub starts 2024-02-05T07:40:03.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:02 smithi090 ceph-mon[33213]: 6.10 scrub ok 2024-02-05T07:40:03.035 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:03.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:03.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:03.045 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:02 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:03.045 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:02 smithi007 ceph-mon[29098]: pgmap v1170: 178 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 176 active+clean; 4.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-02-05T07:40:03.045 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:02 smithi007 ceph-mon[29098]: 6.10 scrub starts 2024-02-05T07:40:03.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:02 smithi007 ceph-mon[29098]: 6.10 scrub ok 2024-02-05T07:40:03.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:02 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:03.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:02 smithi007 ceph-mon[37158]: pgmap v1170: 178 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 176 active+clean; 4.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-02-05T07:40:03.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:02 smithi007 ceph-mon[37158]: 6.10 scrub starts 2024-02-05T07:40:03.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:02 smithi007 ceph-mon[37158]: 6.10 scrub ok 2024-02-05T07:40:03.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:03.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:03.163 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:03.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:03.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:03.291 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:03.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:03.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:03.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:03.433 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:03.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:03.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:03.560 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:03.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:03.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:03.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:03.699 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:03.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:03.801 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:03.847 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:03.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:03.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:03 smithi007 ceph-mon[29098]: 5.11 scrub starts 2024-02-05T07:40:03.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:03 smithi007 ceph-mon[29098]: 5.11 scrub ok 2024-02-05T07:40:03.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:03 smithi007 ceph-mon[29098]: 11.7 scrub ok 2024-02-05T07:40:03.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:40:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:40:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:40:03.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:03 smithi007 ceph-mon[37158]: 5.11 scrub starts 2024-02-05T07:40:03.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:03 smithi007 ceph-mon[37158]: 5.11 scrub ok 2024-02-05T07:40:03.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:03 smithi007 ceph-mon[37158]: 11.7 scrub ok 2024-02-05T07:40:03.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:03.974 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:03.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:04.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:03 smithi090 ceph-mon[33213]: 5.11 scrub starts 2024-02-05T07:40:04.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:03 smithi090 ceph-mon[33213]: 5.11 scrub ok 2024-02-05T07:40:04.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:03 smithi090 ceph-mon[33213]: 11.7 scrub ok 2024-02-05T07:40:04.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:04.102 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:04.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:04.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:04.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:04.229 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:04.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:04.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:04.331 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:04.356 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:04.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:04.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:04.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:04.483 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:04.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:04.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:04.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:04.629 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:04.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:04.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:04.755 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:04.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:04.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:04.786 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:40:04.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:04.909 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:04.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:04.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:05.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:05.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:04 smithi090 ceph-mon[33213]: 3.d scrub starts 2024-02-05T07:40:05.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:04 smithi090 ceph-mon[33213]: 3.d scrub ok 2024-02-05T07:40:05.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:04 smithi090 ceph-mon[33213]: pgmap v1171: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 176 active+clean; 4.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.15k op/s 2024-02-05T07:40:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:05.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:05.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:05.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:04 smithi007 ceph-mon[29098]: 3.d scrub starts 2024-02-05T07:40:05.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:04 smithi007 ceph-mon[29098]: 3.d scrub ok 2024-02-05T07:40:05.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:04 smithi007 ceph-mon[29098]: pgmap v1171: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 176 active+clean; 4.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.15k op/s 2024-02-05T07:40:05.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:04 smithi007 ceph-mon[37158]: 3.d scrub starts 2024-02-05T07:40:05.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:04 smithi007 ceph-mon[37158]: 3.d scrub ok 2024-02-05T07:40:05.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:04 smithi007 ceph-mon[37158]: pgmap v1171: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 176 active+clean; 4.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.15k op/s 2024-02-05T07:40:05.184 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:05.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:05.286 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:05.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:05.312 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:05.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:05.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:05.462 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:05.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:05.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:05.589 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:05.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:05.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:05.726 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:05.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:05.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:05.870 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:05.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:05.972 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:06.027 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:06.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:06.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:06.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:06.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:06.324 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:06.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:06.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:06.468 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:06.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:06.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:06.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:06.612 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:06.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:06.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:06.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:06.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:06.761 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:06.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:06.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:06.906 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:06.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:06.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:06.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:06 smithi007 ceph-mon[29098]: pgmap v1172: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 176 active+clean; 4.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 538 B/s rd, 0 B/s wr, 909 op/s 2024-02-05T07:40:06.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:06 smithi007 ceph-mon[29098]: 5.15 scrub starts 2024-02-05T07:40:06.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:06 smithi007 ceph-mon[29098]: 5.15 scrub ok 2024-02-05T07:40:06.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:06 smithi007 ceph-mon[37158]: pgmap v1172: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 176 active+clean; 4.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 538 B/s rd, 0 B/s wr, 909 op/s 2024-02-05T07:40:06.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:06 smithi007 ceph-mon[37158]: 5.15 scrub starts 2024-02-05T07:40:06.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:06 smithi007 ceph-mon[37158]: 5.15 scrub ok 2024-02-05T07:40:07.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:07.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:06 smithi090 ceph-mon[33213]: pgmap v1172: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 176 active+clean; 4.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 538 B/s rd, 0 B/s wr, 909 op/s 2024-02-05T07:40:07.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:06 smithi090 ceph-mon[33213]: 5.15 scrub starts 2024-02-05T07:40:07.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:06 smithi090 ceph-mon[33213]: 5.15 scrub ok 2024-02-05T07:40:07.051 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:07.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:07.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:07.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:07.194 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:07.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:07.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:07.296 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:07.352 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:07.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:07.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:07.477 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:07.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:07.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:07.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:07.606 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:07.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:07.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:07.734 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:07.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:07.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:07.736 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:40:07.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:07.876 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:07.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:07.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:07.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:08.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:08.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:07 smithi090 ceph-mon[33213]: 4.1d scrub starts 2024-02-05T07:40:08.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:07 smithi090 ceph-mon[33213]: 4.1d scrub ok 2024-02-05T07:40:08.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:07 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2006469710' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:40:08.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:08.161 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:08.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:08.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:07 smithi007 ceph-mon[29098]: 4.1d scrub starts 2024-02-05T07:40:08.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:07 smithi007 ceph-mon[29098]: 4.1d scrub ok 2024-02-05T07:40:08.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:07 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2006469710' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:40:08.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:07 smithi007 ceph-mon[37158]: 4.1d scrub starts 2024-02-05T07:40:08.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:07 smithi007 ceph-mon[37158]: 4.1d scrub ok 2024-02-05T07:40:08.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:07 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2006469710' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:40:08.263 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:08.288 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:08.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:08.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:08.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:08.414 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:08.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:08.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:08.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:08.557 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:08.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:08.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:08.698 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:08.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:08.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:08.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:08.843 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:08.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:08.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:08.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:08.970 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:08.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:08 smithi090 ceph-mon[33213]: pgmap v1173: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T07:40:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:08 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:40:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:08 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2006469710' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:40:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:08 smithi090 ceph-mon[33213]: osdmap e637: 8 total, 8 up, 5 in 2024-02-05T07:40:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:08 smithi090 ceph-mon[33213]: 3.17 scrub starts 2024-02-05T07:40:09.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:08 smithi090 ceph-mon[33213]: 3.17 scrub ok 2024-02-05T07:40:09.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:09.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:09.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:09.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:09.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:08 smithi007 ceph-mon[29098]: pgmap v1173: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T07:40:09.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:08 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:40:09.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:08 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2006469710' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:40:09.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:08 smithi007 ceph-mon[29098]: osdmap e637: 8 total, 8 up, 5 in 2024-02-05T07:40:09.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:08 smithi007 ceph-mon[29098]: 3.17 scrub starts 2024-02-05T07:40:09.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:08 smithi007 ceph-mon[29098]: 3.17 scrub ok 2024-02-05T07:40:09.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:08 smithi007 ceph-mon[37158]: pgmap v1173: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T07:40:09.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:08 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:40:09.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:08 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2006469710' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:40:09.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:08 smithi007 ceph-mon[37158]: osdmap e637: 8 total, 8 up, 5 in 2024-02-05T07:40:09.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:08 smithi007 ceph-mon[37158]: 3.17 scrub starts 2024-02-05T07:40:09.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:08 smithi007 ceph-mon[37158]: 3.17 scrub ok 2024-02-05T07:40:09.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:09.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:09.246 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:09.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:09.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:09.371 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:09.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:09.473 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:09.498 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:09.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:09.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:09.626 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:09.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:09.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:09.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:09.752 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:09.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:09.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:09.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:09.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:09 smithi007 ceph-mon[29098]: 3.2 scrub starts 2024-02-05T07:40:09.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:09 smithi007 ceph-mon[29098]: 3.2 scrub ok 2024-02-05T07:40:09.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:09 smithi007 ceph-mon[29098]: osdmap e638: 8 total, 8 up, 5 in 2024-02-05T07:40:09.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:09 smithi007 ceph-mon[37158]: 3.2 scrub starts 2024-02-05T07:40:09.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:09 smithi007 ceph-mon[37158]: 3.2 scrub ok 2024-02-05T07:40:09.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:09 smithi007 ceph-mon[37158]: osdmap e638: 8 total, 8 up, 5 in 2024-02-05T07:40:09.879 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:09.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:09.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:10.006 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:10.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:10.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:10.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:09 smithi090 ceph-mon[33213]: 3.2 scrub starts 2024-02-05T07:40:10.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:09 smithi090 ceph-mon[33213]: 3.2 scrub ok 2024-02-05T07:40:10.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:09 smithi090 ceph-mon[33213]: osdmap e638: 8 total, 8 up, 5 in 2024-02-05T07:40:10.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:10.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:10.151 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:10.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:10.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:10.294 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:10.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:10.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:10.422 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:10.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:10.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:10.549 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:10.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:10.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:10.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:10.694 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:10.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:10.796 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:10.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:10.847 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:10.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:10.879 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:10.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:10.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:10 smithi007 ceph-mon[29098]: 6.1 scrub starts 2024-02-05T07:40:10.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:10 smithi007 ceph-mon[29098]: 6.1 scrub ok 2024-02-05T07:40:10.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:10 smithi007 ceph-mon[29098]: 6.1c scrub starts 2024-02-05T07:40:10.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:10 smithi007 ceph-mon[29098]: 6.1c scrub ok 2024-02-05T07:40:10.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:10 smithi007 ceph-mon[29098]: pgmap v1175: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T07:40:10.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:10 smithi007 ceph-mon[37158]: 6.1 scrub starts 2024-02-05T07:40:10.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:10 smithi007 ceph-mon[37158]: 6.1 scrub ok 2024-02-05T07:40:10.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:10 smithi007 ceph-mon[37158]: 6.1c scrub starts 2024-02-05T07:40:10.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:10 smithi007 ceph-mon[37158]: 6.1c scrub ok 2024-02-05T07:40:10.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:10 smithi007 ceph-mon[37158]: pgmap v1175: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T07:40:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:11.001 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:11.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:11.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:10 smithi090 ceph-mon[33213]: 6.1 scrub starts 2024-02-05T07:40:11.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:10 smithi090 ceph-mon[33213]: 6.1 scrub ok 2024-02-05T07:40:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:10 smithi090 ceph-mon[33213]: 6.1c scrub starts 2024-02-05T07:40:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:10 smithi090 ceph-mon[33213]: 6.1c scrub ok 2024-02-05T07:40:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:10 smithi090 ceph-mon[33213]: pgmap v1175: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T07:40:11.102 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:11.129 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:11.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:11.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:11.254 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:11.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:11.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:11.359 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:11.404 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:11.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:11.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:11.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:11.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:40:11 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:40:11.531 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:11.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:11.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:11.673 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:11.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:11.775 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:11.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:11.803 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:11.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:11.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:11.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:11.929 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:11.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:12.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:12.071 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:12.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:12.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:12.213 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:12.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:12.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:12.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:40:12.354 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:12.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:12.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:12.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:12.481 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:12.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:12.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:12.607 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:12.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:12.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:12.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:12.735 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:12.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:12.750 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.2 to 1 (10000) 2024-02-05T07:40:12.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:12.886 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:12.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:12.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:12.988 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:12 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:12 smithi090 ceph-mon[33213]: pgmap v1177: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T07:40:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:12 smithi090 ceph-mon[33213]: 3.18 deep-scrub starts 2024-02-05T07:40:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:12 smithi090 ceph-mon[33213]: 3.18 deep-scrub ok 2024-02-05T07:40:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:12 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/461712864' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:40:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:12 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:40:13.034 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:13.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:13.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:13.097 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:40:13.097 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:40:13.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:12 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:13.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:12 smithi007 ceph-mon[29098]: pgmap v1177: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T07:40:13.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:12 smithi007 ceph-mon[29098]: 3.18 deep-scrub starts 2024-02-05T07:40:13.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:12 smithi007 ceph-mon[29098]: 3.18 deep-scrub ok 2024-02-05T07:40:13.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:12 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/461712864' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:40:13.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:12 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:40:13.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:12 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:13.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:12 smithi007 ceph-mon[37158]: pgmap v1177: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T07:40:13.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:12 smithi007 ceph-mon[37158]: 3.18 deep-scrub starts 2024-02-05T07:40:13.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:12 smithi007 ceph-mon[37158]: 3.18 deep-scrub ok 2024-02-05T07:40:13.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:12 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/461712864' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:40:13.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:12 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:40:13.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:13.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:13.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:13.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:13.302 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:13.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:13.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:13.443 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:13.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:13.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:13.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:13.569 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:13.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:13.670 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:13.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:40:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:40:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:40:13.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:13.712 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:13.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:13.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:13.866 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:13.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:13.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T07:40:13.968 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:14.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:13 smithi090 ceph-mon[33213]: 5.2 scrub starts 2024-02-05T07:40:14.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:13 smithi090 ceph-mon[33213]: 5.2 scrub ok 2024-02-05T07:40:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:13 smithi090 ceph-mon[33213]: 2.2 scrub starts 2024-02-05T07:40:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:13 smithi090 ceph-mon[33213]: 2.2 scrub ok 2024-02-05T07:40:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:13 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T07:40:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:13 smithi090 ceph-mon[33213]: osdmap e639: 8 total, 8 up, 5 in 2024-02-05T07:40:14.024 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:14.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:14.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:14.152 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:14.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:14.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:13 smithi007 ceph-mon[29098]: 5.2 scrub starts 2024-02-05T07:40:14.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:13 smithi007 ceph-mon[29098]: 5.2 scrub ok 2024-02-05T07:40:14.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:13 smithi007 ceph-mon[29098]: 2.2 scrub starts 2024-02-05T07:40:14.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:13 smithi007 ceph-mon[29098]: 2.2 scrub ok 2024-02-05T07:40:14.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:13 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T07:40:14.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:13 smithi007 ceph-mon[29098]: osdmap e639: 8 total, 8 up, 5 in 2024-02-05T07:40:14.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:13 smithi007 ceph-mon[37158]: 5.2 scrub starts 2024-02-05T07:40:14.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:13 smithi007 ceph-mon[37158]: 5.2 scrub ok 2024-02-05T07:40:14.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:13 smithi007 ceph-mon[37158]: 2.2 scrub starts 2024-02-05T07:40:14.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:13 smithi007 ceph-mon[37158]: 2.2 scrub ok 2024-02-05T07:40:14.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:13 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T07:40:14.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:13 smithi007 ceph-mon[37158]: osdmap e639: 8 total, 8 up, 5 in 2024-02-05T07:40:14.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:14.280 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:14.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:14.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:14.334 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-05T07:40:14.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:40:14.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:14.407 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:14.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:14.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:14.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:14.561 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:14.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:14.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:14.663 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:14.689 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:14.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:14.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:14.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:14.794 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:14.865 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:14.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:14.881 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:14.925 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:14.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:14 smithi007 ceph-mon[29098]: 3.6 deep-scrub starts 2024-02-05T07:40:14.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:14 smithi007 ceph-mon[29098]: 3.6 deep-scrub ok 2024-02-05T07:40:14.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:14 smithi007 ceph-mon[29098]: pgmap v1179: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.2 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T07:40:14.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:14 smithi007 ceph-mon[29098]: osdmap e640: 8 total, 8 up, 5 in 2024-02-05T07:40:14.932 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:14 smithi007 ceph-mon[37158]: 3.6 deep-scrub starts 2024-02-05T07:40:14.932 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:14 smithi007 ceph-mon[37158]: 3.6 deep-scrub ok 2024-02-05T07:40:14.932 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:14 smithi007 ceph-mon[37158]: pgmap v1179: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.2 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T07:40:14.932 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:14 smithi007 ceph-mon[37158]: osdmap e640: 8 total, 8 up, 5 in 2024-02-05T07:40:14.933 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:40:14.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:14.981 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:40:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:14.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:15.021 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:40:15.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:14 smithi090 ceph-mon[33213]: 3.6 deep-scrub starts 2024-02-05T07:40:15.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:14 smithi090 ceph-mon[33213]: 3.6 deep-scrub ok 2024-02-05T07:40:15.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:14 smithi090 ceph-mon[33213]: pgmap v1179: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.2 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T07:40:15.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:14 smithi090 ceph-mon[33213]: osdmap e640: 8 total, 8 up, 5 in 2024-02-05T07:40:15.065 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:40:15.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:15.115 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:40:15.117 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:15.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:15.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:15.155 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:40:15.202 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:40:15.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:15.244 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:40:15.244 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:15.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:15.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:40:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:40:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:40:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:40:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:40:15.251 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:40:15.251 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:40:15.251 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:40:15.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:15.372 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:15.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:15.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:15.514 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:15.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:15.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:15.641 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:15.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:15.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:15.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:15.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:15.784 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:15.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:15.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:15.927 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:15.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:15.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:16.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:15 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3094510460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:40:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:15 smithi090 ceph-mon[33213]: 3.3 scrub starts 2024-02-05T07:40:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:15 smithi090 ceph-mon[33213]: 3.3 scrub ok 2024-02-05T07:40:16.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:16.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:15 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3094510460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:40:16.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:15 smithi007 ceph-mon[29098]: 3.3 scrub starts 2024-02-05T07:40:16.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:15 smithi007 ceph-mon[29098]: 3.3 scrub ok 2024-02-05T07:40:16.030 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:15 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3094510460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:40:16.030 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:15 smithi007 ceph-mon[37158]: 3.3 scrub starts 2024-02-05T07:40:16.030 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:15 smithi007 ceph-mon[37158]: 3.3 scrub ok 2024-02-05T07:40:16.070 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:16.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:16.173 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:16.212 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:16.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:16.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:16.360 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:16.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:16.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:16.486 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:16.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:16.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:16.613 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:16.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:16.715 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:16.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:16.761 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:16.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:16.862 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:16.887 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:16.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:16.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[29098]: 5.1e scrub starts 2024-02-05T07:40:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[29098]: 5.1e scrub ok 2024-02-05T07:40:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[29098]: pgmap v1181: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-02-05T07:40:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[29098]: osdmap e641: 8 total, 8 up, 5 in 2024-02-05T07:40:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:40:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:40:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:40:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:40:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[37158]: 5.1e scrub starts 2024-02-05T07:40:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[37158]: 5.1e scrub ok 2024-02-05T07:40:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[37158]: pgmap v1181: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-02-05T07:40:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[37158]: osdmap e641: 8 total, 8 up, 5 in 2024-02-05T07:40:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:40:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:40:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:40:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:16 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:40:16.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:17.029 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:17.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:17.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:17.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:17.156 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:17.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:17.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:17.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:17.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:16 smithi090 ceph-mon[33213]: 5.1e scrub starts 2024-02-05T07:40:17.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:16 smithi090 ceph-mon[33213]: 5.1e scrub ok 2024-02-05T07:40:17.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:16 smithi090 ceph-mon[33213]: pgmap v1181: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 4.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-02-05T07:40:17.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:16 smithi090 ceph-mon[33213]: osdmap e641: 8 total, 8 up, 5 in 2024-02-05T07:40:17.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:16 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:40:17.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:16 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:40:17.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:16 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:40:17.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:16 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:40:17.283 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:17.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:17.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:17.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:17.430 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:17.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:17.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:17.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:17.557 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:17.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:17.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:17.686 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:17.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:17.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:17.831 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:17.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:17.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:17.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:17 smithi007 ceph-mon[29098]: osdmap e642: 8 total, 8 up, 5 in 2024-02-05T07:40:17.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:17 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-05T07:40:17.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:17 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-05T07:40:17.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:17 smithi007 ceph-mon[37158]: osdmap e642: 8 total, 8 up, 5 in 2024-02-05T07:40:17.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:17 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-05T07:40:17.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:17 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-05T07:40:17.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:17.958 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:17.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:18.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:18.098 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:18.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:18.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:18.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:40:18.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:18.225 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:18.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:18.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:17 smithi090 ceph-mon[33213]: osdmap e642: 8 total, 8 up, 5 in 2024-02-05T07:40:18.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:17 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-05T07:40:18.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:17 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-05T07:40:18.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:18.378 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:18.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:18.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:18.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:18.505 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:18.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:18.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:18.680 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:18.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:18.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:18.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:18.827 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:18.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:18.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:18 smithi007 ceph-mon[29098]: pgmap v1184: 178 pgs: 178 active+clean; 3.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T07:40:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:18 smithi007 ceph-mon[29098]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T07:40:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:18 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:40:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:18 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-05T07:40:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:18 smithi007 ceph-mon[29098]: osdmap e643: 8 total, 8 up, 5 in 2024-02-05T07:40:18.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:18 smithi007 ceph-mon[37158]: pgmap v1184: 178 pgs: 178 active+clean; 3.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T07:40:18.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:18 smithi007 ceph-mon[37158]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T07:40:18.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:18 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:40:18.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:18 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-05T07:40:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:18 smithi007 ceph-mon[37158]: osdmap e643: 8 total, 8 up, 5 in 2024-02-05T07:40:18.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:18.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:18.955 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:18.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:19.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:19.103 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:19.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:19.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:19.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:19.229 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:19.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:19.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:19.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:18 smithi090 ceph-mon[33213]: pgmap v1184: 178 pgs: 178 active+clean; 3.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T07:40:19.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:18 smithi090 ceph-mon[33213]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T07:40:19.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:18 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:40:19.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:18 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-05T07:40:19.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:18 smithi090 ceph-mon[33213]: osdmap e643: 8 total, 8 up, 5 in 2024-02-05T07:40:19.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:19.349 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:40:19.349 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:40:19.349 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-02-05T07:40:19.350 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-02-05T07:40:19.350 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:40:19.381 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:19.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:19.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:19.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:19.519 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:19.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:19.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:19.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:19.647 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:19.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:19.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:19.787 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:19.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:19.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:19.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:19.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:19.933 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:19.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:20.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:20.079 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:20.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:20.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:20.133 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:19 smithi007 ceph-mon[29098]: 3.e scrub starts 2024-02-05T07:40:20.133 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:19 smithi007 ceph-mon[29098]: 3.e scrub ok 2024-02-05T07:40:20.133 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:19 smithi007 ceph-mon[29098]: 6.1e deep-scrub starts 2024-02-05T07:40:20.133 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:19 smithi007 ceph-mon[29098]: 6.1e deep-scrub ok 2024-02-05T07:40:20.133 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:19 smithi007 ceph-mon[29098]: osdmap e644: 8 total, 8 up, 5 in 2024-02-05T07:40:20.134 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:19 smithi007 ceph-mon[37158]: 3.e scrub starts 2024-02-05T07:40:20.134 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:19 smithi007 ceph-mon[37158]: 3.e scrub ok 2024-02-05T07:40:20.134 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:19 smithi007 ceph-mon[37158]: 6.1e deep-scrub starts 2024-02-05T07:40:20.134 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:19 smithi007 ceph-mon[37158]: 6.1e deep-scrub ok 2024-02-05T07:40:20.134 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:19 smithi007 ceph-mon[37158]: osdmap e644: 8 total, 8 up, 5 in 2024-02-05T07:40:20.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:20.223 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:20.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:20.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:19 smithi090 ceph-mon[33213]: 3.e scrub starts 2024-02-05T07:40:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:19 smithi090 ceph-mon[33213]: 3.e scrub ok 2024-02-05T07:40:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:19 smithi090 ceph-mon[33213]: 6.1e deep-scrub starts 2024-02-05T07:40:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:19 smithi090 ceph-mon[33213]: 6.1e deep-scrub ok 2024-02-05T07:40:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:19 smithi090 ceph-mon[33213]: osdmap e644: 8 total, 8 up, 5 in 2024-02-05T07:40:20.325 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:20.366 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:20.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:20.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:20.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:20.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:20.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:20.594 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:20.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T07:40:20.647 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:20.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:20.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:20.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:20.773 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:20.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:20.861 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:40:20.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:20.898 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:20.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:20.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:21.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:21.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:21.027 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:21.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:21.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:20 smithi007 ceph-mon[29098]: 5.5 scrub starts 2024-02-05T07:40:21.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:20 smithi007 ceph-mon[29098]: 5.5 scrub ok 2024-02-05T07:40:21.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:20 smithi007 ceph-mon[29098]: pgmap v1187: 178 pgs: 178 active+clean; 3.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-02-05T07:40:21.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:20 smithi007 ceph-mon[29098]: osdmap e645: 8 total, 8 up, 5 in 2024-02-05T07:40:21.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:20 smithi007 ceph-mon[29098]: 6.13 scrub starts 2024-02-05T07:40:21.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:20 smithi007 ceph-mon[29098]: 6.13 scrub ok 2024-02-05T07:40:21.113 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:20 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1560749220' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:40:21.113 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:20 smithi007 ceph-mon[37158]: 5.5 scrub starts 2024-02-05T07:40:21.113 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:20 smithi007 ceph-mon[37158]: 5.5 scrub ok 2024-02-05T07:40:21.113 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:20 smithi007 ceph-mon[37158]: pgmap v1187: 178 pgs: 178 active+clean; 3.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-02-05T07:40:21.113 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:20 smithi007 ceph-mon[37158]: osdmap e645: 8 total, 8 up, 5 in 2024-02-05T07:40:21.113 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:20 smithi007 ceph-mon[37158]: 6.13 scrub starts 2024-02-05T07:40:21.113 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:20 smithi007 ceph-mon[37158]: 6.13 scrub ok 2024-02-05T07:40:21.113 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:20 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1560749220' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:40:21.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:21.153 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:21.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:21.217 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:20 smithi090 ceph-mon[33213]: 5.5 scrub starts 2024-02-05T07:40:21.217 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:20 smithi090 ceph-mon[33213]: 5.5 scrub ok 2024-02-05T07:40:21.217 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:20 smithi090 ceph-mon[33213]: pgmap v1187: 178 pgs: 178 active+clean; 3.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-02-05T07:40:21.217 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:20 smithi090 ceph-mon[33213]: osdmap e645: 8 total, 8 up, 5 in 2024-02-05T07:40:21.217 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:20 smithi090 ceph-mon[33213]: 6.13 scrub starts 2024-02-05T07:40:21.217 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:20 smithi090 ceph-mon[33213]: 6.13 scrub ok 2024-02-05T07:40:21.217 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:20 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1560749220' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:40:21.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:21.278 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:21.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:21.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:21.405 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:21.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:21.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:21.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:40:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:40:21.532 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:21.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:21.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:21.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:21.658 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:21.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:21.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:21.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:21.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:21.804 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:21.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:21.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:21.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:21 smithi007 ceph-mon[29098]: 6.1d scrub starts 2024-02-05T07:40:21.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:21 smithi007 ceph-mon[29098]: 6.1d scrub ok 2024-02-05T07:40:21.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:21 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:40:21.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:21 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1560749220' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:40:21.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:21 smithi007 ceph-mon[29098]: osdmap e646: 8 total, 8 up, 5 in 2024-02-05T07:40:21.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:21 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-05T07:40:21.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:21 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-05T07:40:21.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:21 smithi007 ceph-mon[37158]: 6.1d scrub starts 2024-02-05T07:40:21.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:21 smithi007 ceph-mon[37158]: 6.1d scrub ok 2024-02-05T07:40:21.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:21 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:40:21.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:21 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1560749220' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:40:21.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:21 smithi007 ceph-mon[37158]: osdmap e646: 8 total, 8 up, 5 in 2024-02-05T07:40:21.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:21 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-05T07:40:21.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:21 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-05T07:40:21.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:21.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:21.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:22.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:22.073 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:22.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:22.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:22.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:22.199 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:22.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:22.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:21 smithi090 ceph-mon[33213]: 6.1d scrub starts 2024-02-05T07:40:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:21 smithi090 ceph-mon[33213]: 6.1d scrub ok 2024-02-05T07:40:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:21 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:40:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:21 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1560749220' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:40:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:21 smithi090 ceph-mon[33213]: osdmap e646: 8 total, 8 up, 5 in 2024-02-05T07:40:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:21 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-05T07:40:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:21 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-05T07:40:22.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:22.337 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:22.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:22.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:22.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:22.487 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:22.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:22.513 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:22.515 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:22.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:22.647 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:22.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:22.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:22.799 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:22.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:22.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:22.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:22.926 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:22.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:23.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:23.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:22 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:23.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:22 smithi007 ceph-mon[29098]: pgmap v1190: 177 pgs: 177 active+clean; 3.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T07:40:23.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:22 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-05T07:40:23.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:22 smithi007 ceph-mon[29098]: osdmap e647: 8 total, 8 up, 5 in 2024-02-05T07:40:23.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:22 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:23.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:22 smithi007 ceph-mon[37158]: pgmap v1190: 177 pgs: 177 active+clean; 3.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T07:40:23.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:22 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-05T07:40:23.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:22 smithi007 ceph-mon[37158]: osdmap e647: 8 total, 8 up, 5 in 2024-02-05T07:40:23.039 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:40:23.046 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:40:23.072 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:23.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:23.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:23.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:23.202 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:23.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:23.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:22 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:22 smithi090 ceph-mon[33213]: pgmap v1190: 177 pgs: 177 active+clean; 3.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T07:40:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:22 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-05T07:40:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:22 smithi090 ceph-mon[33213]: osdmap e647: 8 total, 8 up, 5 in 2024-02-05T07:40:23.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:23.341 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:23.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:23.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:23.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:23.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:23.469 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:23.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:23.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:23.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:23.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:23.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:23.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:23.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:23.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:23.773 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:40:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a[63442]: logger=cleanup t=2024-02-05T07:40:23.437139927Z level=info msg="Completed cleanup jobs" duration=24.646362ms 2024-02-05T07:40:23.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:23.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:23.879 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:23.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:23.884 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:40:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:40:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:40:23.980 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:24.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T07:40:24.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:24.006 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:24.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:24.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:24.166 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:24.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:24.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:23 smithi007 ceph-mon[29098]: osdmap e648: 8 total, 8 up, 5 in 2024-02-05T07:40:24.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:23 smithi007 ceph-mon[29098]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:40:24.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:23 smithi007 ceph-mon[29098]: pgmap v1193: 177 pgs: 177 active+clean; 3.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-02-05T07:40:24.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:23 smithi007 ceph-mon[37158]: osdmap e648: 8 total, 8 up, 5 in 2024-02-05T07:40:24.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:23 smithi007 ceph-mon[37158]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:40:24.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:23 smithi007 ceph-mon[37158]: pgmap v1193: 177 pgs: 177 active+clean; 3.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-02-05T07:40:24.268 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:24.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:23 smithi090 ceph-mon[33213]: osdmap e648: 8 total, 8 up, 5 in 2024-02-05T07:40:24.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:23 smithi090 ceph-mon[33213]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:40:24.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:23 smithi090 ceph-mon[33213]: pgmap v1193: 177 pgs: 177 active+clean; 3.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-02-05T07:40:24.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:24.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:24.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:24.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:24.449 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:24.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:24.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:24.576 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:24.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:24.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:24.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:24.704 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:24.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:24.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:24.844 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:24.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:24.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:24 smithi007 ceph-mon[29098]: osdmap e649: 8 total, 8 up, 5 in 2024-02-05T07:40:24.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:24 smithi007 ceph-mon[29098]: osdmap e650: 8 total, 8 up, 5 in 2024-02-05T07:40:24.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:24 smithi007 ceph-mon[37158]: osdmap e649: 8 total, 8 up, 5 in 2024-02-05T07:40:24.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:24 smithi007 ceph-mon[37158]: osdmap e650: 8 total, 8 up, 5 in 2024-02-05T07:40:24.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:24.970 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:24.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:24.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:25.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:25.128 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:25.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:25.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:25.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:24 smithi090 ceph-mon[33213]: osdmap e649: 8 total, 8 up, 5 in 2024-02-05T07:40:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:24 smithi090 ceph-mon[33213]: osdmap e650: 8 total, 8 up, 5 in 2024-02-05T07:40:25.274 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:25.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:25.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:25.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:25.415 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:25.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:25.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:25.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:25.542 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:25.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:25.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:25.685 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:25.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:25.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:25.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:25.813 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:25.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:25.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:25.940 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:25.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:26.041 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:26.067 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:26.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:26.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:26.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:25 smithi007 ceph-mon[29098]: pgmap v1196: 176 pgs: 176 active+clean; 3.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 864 op/s 2024-02-05T07:40:26.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:25 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-05T07:40:26.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:25 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-05T07:40:26.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:25 smithi007 ceph-mon[37158]: pgmap v1196: 176 pgs: 176 active+clean; 3.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 864 op/s 2024-02-05T07:40:26.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:25 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-05T07:40:26.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:25 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-05T07:40:26.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:26.204 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:26.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:26.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:25 smithi090 ceph-mon[33213]: pgmap v1196: 176 pgs: 176 active+clean; 3.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 864 op/s 2024-02-05T07:40:26.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:25 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-05T07:40:26.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:25 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-05T07:40:26.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:26.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:26.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:26.472 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:26.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:26.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:26.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:26.615 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:26.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:26.717 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:26.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:26.761 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:26.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:26.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:26.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:26.904 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:26.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:27.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:27.044 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:27.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:27.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:27.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:27.172 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:27.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:27.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:27.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:26 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-05T07:40:27.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:26 smithi007 ceph-mon[29098]: osdmap e651: 8 total, 8 up, 5 in 2024-02-05T07:40:27.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:26 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-05T07:40:27.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:26 smithi007 ceph-mon[37158]: osdmap e651: 8 total, 8 up, 5 in 2024-02-05T07:40:27.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:26 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-05T07:40:27.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:26 smithi090 ceph-mon[33213]: osdmap e651: 8 total, 8 up, 5 in 2024-02-05T07:40:27.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:27.298 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:27.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:27.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:27.399 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:27.423 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:27.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:27.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:27.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:27.565 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:27.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:27.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:27.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:27.711 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:27.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:27.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:27.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:27.859 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:27.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:27.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:28.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:28.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:28.141 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:28.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:28.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:27 smithi007 ceph-mon[29098]: osdmap e652: 8 total, 8 up, 5 in 2024-02-05T07:40:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:27 smithi007 ceph-mon[29098]: pgmap v1199: 176 pgs: 2 clean+premerge+peered, 174 active+clean; 3.1 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-02-05T07:40:28.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:27 smithi007 ceph-mon[37158]: osdmap e652: 8 total, 8 up, 5 in 2024-02-05T07:40:28.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:27 smithi007 ceph-mon[37158]: pgmap v1199: 176 pgs: 2 clean+premerge+peered, 174 active+clean; 3.1 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-02-05T07:40:28.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:28.268 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:28.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:28.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:27 smithi090 ceph-mon[33213]: osdmap e652: 8 total, 8 up, 5 in 2024-02-05T07:40:28.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:27 smithi090 ceph-mon[33213]: pgmap v1199: 176 pgs: 2 clean+premerge+peered, 174 active+clean; 3.1 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-02-05T07:40:28.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:28.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:28.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:28.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:28.526 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:28.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:28.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:28.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:28.654 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:28.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:28.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:28.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:28.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:28.897 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:28.922 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:28.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:29.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:29.049 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:29.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:29.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:29.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:29.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:28 smithi007 ceph-mon[29098]: osdmap e653: 8 total, 8 up, 5 in 2024-02-05T07:40:29.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:28 smithi007 ceph-mon[37158]: osdmap e653: 8 total, 8 up, 5 in 2024-02-05T07:40:29.176 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:29.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:29.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:29.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:28 smithi090 ceph-mon[33213]: osdmap e653: 8 total, 8 up, 5 in 2024-02-05T07:40:29.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:29.302 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:29.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:29.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:29.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:29.445 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:29.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:29.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:29.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:29.572 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:29.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:29.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:29.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:29.784 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:29.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:29.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:29.909 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:29.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:30.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:30.054 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:30.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:30.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:30.180 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:30.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:30.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:30.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:29 smithi090 ceph-mon[33213]: osdmap e654: 8 total, 8 up, 5 in 2024-02-05T07:40:30.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:29 smithi090 ceph-mon[33213]: pgmap v1202: 175 pgs: 1 clean+premerge+peered, 174 active+clean; 3.0 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-02-05T07:40:30.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:29 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-05T07:40:30.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:29 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-05T07:40:30.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:30.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:30.323 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:30.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:30.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:29 smithi007 ceph-mon[29098]: osdmap e654: 8 total, 8 up, 5 in 2024-02-05T07:40:30.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:29 smithi007 ceph-mon[29098]: pgmap v1202: 175 pgs: 1 clean+premerge+peered, 174 active+clean; 3.0 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-02-05T07:40:30.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:29 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-05T07:40:30.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:29 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-05T07:40:30.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:29 smithi007 ceph-mon[37158]: osdmap e654: 8 total, 8 up, 5 in 2024-02-05T07:40:30.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:29 smithi007 ceph-mon[37158]: pgmap v1202: 175 pgs: 1 clean+premerge+peered, 174 active+clean; 3.0 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-02-05T07:40:30.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:29 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-05T07:40:30.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:29 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-05T07:40:30.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:30.466 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:30.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:30.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:30.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:30.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:30.595 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:30.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:30.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:30.739 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:30.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:30.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:30.867 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:30.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:30.969 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:31.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:31.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:31.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:31.116 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:31.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:31.141 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:31.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:31.227 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:30 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-05T07:40:31.227 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:30 smithi090 ceph-mon[33213]: osdmap e655: 8 total, 8 up, 5 in 2024-02-05T07:40:31.241 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:31.269 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:31.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:31.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:31.396 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:31.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:31.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:31.422 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:30 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-05T07:40:31.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:30 smithi007 ceph-mon[29098]: osdmap e655: 8 total, 8 up, 5 in 2024-02-05T07:40:31.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:30 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-05T07:40:31.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:30 smithi007 ceph-mon[37158]: osdmap e655: 8 total, 8 up, 5 in 2024-02-05T07:40:31.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:31.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:40:31 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:40:31.534 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:31.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:31.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:31.676 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:31.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:31.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:31.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:31.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:31.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:31.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:31.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:31.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:32.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:32.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:32.071 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:32.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:32.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:32.214 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:32.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:32.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:31 smithi090 ceph-mon[33213]: osdmap e656: 8 total, 8 up, 5 in 2024-02-05T07:40:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:31 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:31 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:40:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:31 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:40:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:31 smithi090 ceph-mon[33213]: pgmap v1205: 175 pgs: 2 peering, 1 clean+premerge+peered, 172 active+clean; 2.8 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T07:40:32.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:32.323 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:31 smithi007 ceph-mon[29098]: osdmap e656: 8 total, 8 up, 5 in 2024-02-05T07:40:32.323 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:31 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:32.323 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:31 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:40:32.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:31 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:40:32.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:31 smithi007 ceph-mon[29098]: pgmap v1205: 175 pgs: 2 peering, 1 clean+premerge+peered, 172 active+clean; 2.8 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T07:40:32.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:31 smithi007 ceph-mon[37158]: osdmap e656: 8 total, 8 up, 5 in 2024-02-05T07:40:32.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:31 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:32.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:31 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:40:32.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:31 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:40:32.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:31 smithi007 ceph-mon[37158]: pgmap v1205: 175 pgs: 2 peering, 1 clean+premerge+peered, 172 active+clean; 2.8 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T07:40:32.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:32.341 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:32.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:32.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:32.483 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:32.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:32.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:32.628 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:32.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:32.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:32.766 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:32.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:32.868 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:32.909 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:32.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:33.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:33.033 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:33.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:33.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:33.160 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:33.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:33.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:33.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:32 smithi007 ceph-mon[29098]: osdmap e657: 8 total, 8 up, 5 in 2024-02-05T07:40:33.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:32 smithi007 ceph-mon[37158]: osdmap e657: 8 total, 8 up, 5 in 2024-02-05T07:40:33.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:33.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:32 smithi090 ceph-mon[33213]: osdmap e657: 8 total, 8 up, 5 in 2024-02-05T07:40:33.304 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:33.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:33.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:33.446 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:33.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:33.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:33.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:33.589 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:33.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:33.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:33.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:33.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:33.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:33.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:33.817 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:33.857 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:33.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:33.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:33.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:40:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:40:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:40:33.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:33.999 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:34.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:34.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:34.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:34.125 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:34.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:34.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:34.253 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:34.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:34.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:34.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:33 smithi090 ceph-mon[33213]: osdmap e658: 8 total, 8 up, 5 in 2024-02-05T07:40:34.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:33 smithi090 ceph-mon[33213]: pgmap v1208: 174 pgs: 1 peering, 173 active+clean; 2.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-02-05T07:40:34.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:33 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:40:34.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:33 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:40:34.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:34.394 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:34.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:34.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:34.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:33 smithi007 ceph-mon[29098]: osdmap e658: 8 total, 8 up, 5 in 2024-02-05T07:40:34.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:33 smithi007 ceph-mon[29098]: pgmap v1208: 174 pgs: 1 peering, 173 active+clean; 2.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-02-05T07:40:34.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:33 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:40:34.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:33 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:40:34.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:33 smithi007 ceph-mon[37158]: osdmap e658: 8 total, 8 up, 5 in 2024-02-05T07:40:34.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:34 smithi007 ceph-mon[37158]: pgmap v1208: 174 pgs: 1 peering, 173 active+clean; 2.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-02-05T07:40:34.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:34 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:40:34.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:34 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:40:34.496 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:34.538 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:34.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:34.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:34.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:34.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:34.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:34.822 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:34.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:34.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:34.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:34.962 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:34.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:35.065 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:35.091 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:35.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:35.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:35.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:34 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:40:35.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:34 smithi007 ceph-mon[29098]: osdmap e659: 8 total, 8 up, 5 in 2024-02-05T07:40:35.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:34 smithi007 ceph-mon[29098]: osdmap e660: 8 total, 8 up, 5 in 2024-02-05T07:40:35.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:34 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:40:35.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:34 smithi007 ceph-mon[37158]: osdmap e659: 8 total, 8 up, 5 in 2024-02-05T07:40:35.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:34 smithi007 ceph-mon[37158]: osdmap e660: 8 total, 8 up, 5 in 2024-02-05T07:40:35.193 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:35.217 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:35.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:35.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:34 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:40:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:34 smithi090 ceph-mon[33213]: osdmap e659: 8 total, 8 up, 5 in 2024-02-05T07:40:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:34 smithi090 ceph-mon[33213]: osdmap e660: 8 total, 8 up, 5 in 2024-02-05T07:40:35.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:35.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:35.346 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:35.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:35.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:35.471 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:35.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:35.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:35.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:35.599 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:35.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:35.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:35.726 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:35.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:35.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:35.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:35.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:35.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:36.012 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:36.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:36.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:36.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:36.138 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:36.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:36.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:36.265 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:36.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:36.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:36.419 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1192,"stamp":"2024-02-05T07:40:22.874285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3681098100,"num_objects":56359,"num_object_clones":0,"num_object_copies":169077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56359,"num_whiteouts":0,"num_read":2111,"num_read_kb":1863,"num_write":165405,"num_write_kb":7057838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2799,"num_bytes_recovered":150681091,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47323,"ondisk_log_size":47323,"up":531,"acting":531,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":15784452,"kb_used_data":10867448,"kb_used_omap":11,"kb_used_meta":4916916,"kb_avail":452941308,"statfs":{"total":479975178240,"available":463811899392,"internally_reserved":0,"allocated":11128266752,"data_stored":11121713665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12005,"internal_metadata":5034922267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,10,6,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-412024832,"num_objects":-6287,"num_object_clones":0,"num_object_copies":-18861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6287,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6287,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.062773"},"pg_stats":[{"pgid":"11.3","version":"643'10104","reported_seq":10245,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818400+0000","last_change":"2024-02-05T07:37:52.811299+0000","last_active":"2024-02-05T07:40:18.818400+0000","last_peered":"2024-02-05T07:40:18.818400+0000","last_clean":"2024-02-05T07:40:18.818400+0000","last_became_active":"2024-02-05T07:37:52.811051+0000","last_became_peered":"2024-02-05T07:37:52.811051+0000","last_unstale":"2024-02-05T07:40:18.818400+0000","last_undegraded":"2024-02-05T07:40:18.818400+0000","last_fullsized":"2024-02-05T07:40:18.818400+0000","mapping_epoch":579,"log_start":"628'7320","ondisk_log_start":"628'7320","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2784,"log_dups_size":3000,"ondisk_log_size":2784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236453888,"num_objects":3608,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10104,"num_write_kb":438784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":1417,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874914+0000","last_change":"2024-02-05T07:37:41.337312+0000","last_active":"2024-02-05T07:40:21.874914+0000","last_peered":"2024-02-05T07:40:21.874914+0000","last_clean":"2024-02-05T07:40:21.874914+0000","last_became_active":"2024-02-05T07:37:36.688127+0000","last_became_peered":"2024-02-05T07:37:36.688127+0000","last_unstale":"2024-02-05T07:40:21.874914+0000","last_undegraded":"2024-02-05T07:40:21.874914+0000","last_fullsized":"2024-02-05T07:40:21.874914+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.337196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.337196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"615'35","reported_seq":781,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.875101+0000","last_change":"2024-02-05T07:38:09.523613+0000","last_active":"2024-02-05T07:40:21.875101+0000","last_peered":"2024-02-05T07:40:21.875101+0000","last_clean":"2024-02-05T07:40:21.875101+0000","last_became_active":"2024-02-05T07:38:09.493734+0000","last_became_peered":"2024-02-05T07:38:09.493734+0000","last_unstale":"2024-02-05T07:40:21.875101+0000","last_undegraded":"2024-02-05T07:40:21.875101+0000","last_fullsized":"2024-02-05T07:40:21.875101+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:37:02.567716+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:37:02.567716+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":804,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.878694+0000","last_change":"2024-02-05T07:38:37.381786+0000","last_active":"2024-02-05T07:40:21.878694+0000","last_peered":"2024-02-05T07:40:21.878694+0000","last_clean":"2024-02-05T07:40:21.878694+0000","last_became_active":"2024-02-05T07:38:09.496706+0000","last_became_peered":"2024-02-05T07:38:09.496706+0000","last_unstale":"2024-02-05T07:40:21.878694+0000","last_undegraded":"2024-02-05T07:40:21.878694+0000","last_fullsized":"2024-02-05T07:40:21.878694+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:37.381317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:37.381317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"538'13","reported_seq":288,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.851673+0000","last_change":"2024-02-05T07:39:03.630111+0000","last_active":"2024-02-05T07:40:19.851673+0000","last_peered":"2024-02-05T07:40:19.851673+0000","last_clean":"2024-02-05T07:40:19.851673+0000","last_became_active":"2024-02-05T07:36:32.743613+0000","last_became_peered":"2024-02-05T07:36:32.743613+0000","last_unstale":"2024-02-05T07:40:19.851673+0000","last_undegraded":"2024-02-05T07:40:19.851673+0000","last_fullsized":"2024-02-05T07:40:19.851673+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'13","last_scrub_stamp":"2024-02-05T07:39:03.629988+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:39:03.629988+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1107,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.835872+0000","last_change":"2024-02-05T07:39:53.910643+0000","last_active":"2024-02-05T07:40:19.835872+0000","last_peered":"2024-02-05T07:40:19.835872+0000","last_clean":"2024-02-05T07:40:19.835872+0000","last_became_active":"2024-02-05T07:38:09.495645+0000","last_became_peered":"2024-02-05T07:38:09.495645+0000","last_unstale":"2024-02-05T07:40:19.835872+0000","last_undegraded":"2024-02-05T07:40:19.835872+0000","last_fullsized":"2024-02-05T07:40:19.835872+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:53.910541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:39:53.910541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:00.438016+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2","version":"645'10203","reported_seq":10346,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:20.685866+0000","last_change":"2024-02-05T07:37:53.089553+0000","last_active":"2024-02-05T07:40:20.685866+0000","last_peered":"2024-02-05T07:40:20.685866+0000","last_clean":"2024-02-05T07:40:20.685866+0000","last_became_active":"2024-02-05T07:37:53.089215+0000","last_became_peered":"2024-02-05T07:37:53.089215+0000","last_unstale":"2024-02-05T07:40:20.685866+0000","last_undegraded":"2024-02-05T07:40:20.685866+0000","last_fullsized":"2024-02-05T07:40:20.685866+0000","mapping_epoch":579,"log_start":"628'7036","ondisk_log_start":"628'7036","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":3167,"log_dups_size":0,"ondisk_log_size":3167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231800832,"num_objects":3537,"num_object_clones":0,"num_object_copies":10611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10203,"num_write_kb":439680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":714,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.846282+0000","last_change":"2024-02-05T07:37:38.390073+0000","last_active":"2024-02-05T07:40:19.846282+0000","last_peered":"2024-02-05T07:40:19.846282+0000","last_clean":"2024-02-05T07:40:19.846282+0000","last_became_active":"2024-02-05T07:37:36.688448+0000","last_became_peered":"2024-02-05T07:37:36.688448+0000","last_unstale":"2024-02-05T07:40:19.846282+0000","last_undegraded":"2024-02-05T07:40:19.846282+0000","last_fullsized":"2024-02-05T07:40:19.846282+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.389984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.389984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":69,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"616'45","reported_seq":1222,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.851771+0000","last_change":"2024-02-05T07:38:09.530860+0000","last_active":"2024-02-05T07:40:19.851771+0000","last_peered":"2024-02-05T07:40:19.851771+0000","last_clean":"2024-02-05T07:40:19.851771+0000","last_became_active":"2024-02-05T07:38:09.503330+0000","last_became_peered":"2024-02-05T07:38:09.503330+0000","last_unstale":"2024-02-05T07:40:19.851771+0000","last_undegraded":"2024-02-05T07:40:19.851771+0000","last_fullsized":"2024-02-05T07:40:19.851771+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'31","last_scrub_stamp":"2024-02-05T07:37:54.045831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.045831+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1070,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818621+0000","last_change":"2024-02-05T07:38:09.471668+0000","last_active":"2024-02-05T07:40:18.818621+0000","last_peered":"2024-02-05T07:40:18.818621+0000","last_clean":"2024-02-05T07:40:18.818621+0000","last_became_active":"2024-02-05T07:38:09.471049+0000","last_became_peered":"2024-02-05T07:38:09.471049+0000","last_unstale":"2024-02-05T07:40:18.818621+0000","last_undegraded":"2024-02-05T07:40:18.818621+0000","last_fullsized":"2024-02-05T07:40:18.818621+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:46.657168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:37:46.657168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":129,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819063+0000","last_change":"2024-02-05T07:39:59.662598+0000","last_active":"2024-02-05T07:40:18.819063+0000","last_peered":"2024-02-05T07:40:18.819063+0000","last_clean":"2024-02-05T07:40:18.819063+0000","last_became_active":"2024-02-05T07:38:09.464116+0000","last_became_peered":"2024-02-05T07:38:09.464116+0000","last_unstale":"2024-02-05T07:40:18.819063+0000","last_undegraded":"2024-02-05T07:40:18.819063+0000","last_fullsized":"2024-02-05T07:40:18.819063+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:59.662530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:59.662530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:25.420020+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":244,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.849149+0000","last_change":"2024-02-05T07:38:55.897373+0000","last_active":"2024-02-05T07:40:19.849149+0000","last_peered":"2024-02-05T07:40:19.849149+0000","last_clean":"2024-02-05T07:40:19.849149+0000","last_became_active":"2024-02-05T07:38:09.482289+0000","last_became_peered":"2024-02-05T07:38:09.482289+0000","last_unstale":"2024-02-05T07:40:19.849149+0000","last_undegraded":"2024-02-05T07:40:19.849149+0000","last_fullsized":"2024-02-05T07:40:19.849149+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:55.897133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.897133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"643'10063","reported_seq":10204,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818513+0000","last_change":"2024-02-05T07:37:53.089618+0000","last_active":"2024-02-05T07:40:18.818513+0000","last_peered":"2024-02-05T07:40:18.818513+0000","last_clean":"2024-02-05T07:40:18.818513+0000","last_became_active":"2024-02-05T07:37:53.089261+0000","last_became_peered":"2024-02-05T07:37:53.089261+0000","last_unstale":"2024-02-05T07:40:18.818513+0000","last_undegraded":"2024-02-05T07:40:18.818513+0000","last_fullsized":"2024-02-05T07:40:18.818513+0000","mapping_epoch":579,"log_start":"628'7231","ondisk_log_start":"628'7231","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2832,"log_dups_size":3000,"ondisk_log_size":2832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234160128,"num_objects":3573,"num_object_clones":0,"num_object_copies":10719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10063,"num_write_kb":436352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":1001,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.850048+0000","last_change":"2024-02-05T07:37:39.417183+0000","last_active":"2024-02-05T07:40:19.850048+0000","last_peered":"2024-02-05T07:40:19.850048+0000","last_clean":"2024-02-05T07:40:19.850048+0000","last_became_active":"2024-02-05T07:37:36.693385+0000","last_became_peered":"2024-02-05T07:37:36.693385+0000","last_unstale":"2024-02-05T07:40:19.850048+0000","last_undegraded":"2024-02-05T07:40:19.850048+0000","last_fullsized":"2024-02-05T07:40:19.850048+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.417059+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.417059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"616'32","reported_seq":747,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819302+0000","last_change":"2024-02-05T07:38:36.705553+0000","last_active":"2024-02-05T07:40:18.819302+0000","last_peered":"2024-02-05T07:40:18.819302+0000","last_clean":"2024-02-05T07:40:18.819302+0000","last_became_active":"2024-02-05T07:38:09.471120+0000","last_became_peered":"2024-02-05T07:38:09.471120+0000","last_unstale":"2024-02-05T07:40:18.819302+0000","last_undegraded":"2024-02-05T07:40:18.819302+0000","last_fullsized":"2024-02-05T07:40:18.819302+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:38:36.705434+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:38:36.705434+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1120,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.847199+0000","last_change":"2024-02-05T07:38:09.490148+0000","last_active":"2024-02-05T07:40:19.847199+0000","last_peered":"2024-02-05T07:40:19.847199+0000","last_clean":"2024-02-05T07:40:19.847199+0000","last_became_active":"2024-02-05T07:38:09.489807+0000","last_became_peered":"2024-02-05T07:38:09.489807+0000","last_unstale":"2024-02-05T07:40:19.847199+0000","last_undegraded":"2024-02-05T07:40:19.847199+0000","last_fullsized":"2024-02-05T07:40:19.847199+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:03.618665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:37:03.618665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":953,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819543+0000","last_change":"2024-02-05T07:38:37.703938+0000","last_active":"2024-02-05T07:40:18.819543+0000","last_peered":"2024-02-05T07:40:18.819543+0000","last_clean":"2024-02-05T07:40:18.819543+0000","last_became_active":"2024-02-05T07:38:09.472546+0000","last_became_peered":"2024-02-05T07:38:09.472546+0000","last_unstale":"2024-02-05T07:40:18.819543+0000","last_undegraded":"2024-02-05T07:40:18.819543+0000","last_fullsized":"2024-02-05T07:40:18.819543+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:37.703824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:38:37.703824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":807,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874463+0000","last_change":"2024-02-05T07:37:52.481462+0000","last_active":"2024-02-05T07:40:21.874463+0000","last_peered":"2024-02-05T07:40:21.874463+0000","last_clean":"2024-02-05T07:40:21.874463+0000","last_became_active":"2024-02-05T07:30:40.498529+0000","last_became_peered":"2024-02-05T07:30:40.498529+0000","last_unstale":"2024-02-05T07:40:21.874463+0000","last_undegraded":"2024-02-05T07:40:21.874463+0000","last_fullsized":"2024-02-05T07:40:21.874463+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:52.481255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:52.481255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"647'10168","reported_seq":10313,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:22.217242+0000","last_change":"2024-02-05T07:37:52.817253+0000","last_active":"2024-02-05T07:40:22.217242+0000","last_peered":"2024-02-05T07:40:22.217242+0000","last_clean":"2024-02-05T07:40:22.217242+0000","last_became_active":"2024-02-05T07:37:52.816932+0000","last_became_peered":"2024-02-05T07:37:52.816932+0000","last_unstale":"2024-02-05T07:40:22.217242+0000","last_undegraded":"2024-02-05T07:40:22.217242+0000","last_fullsized":"2024-02-05T07:40:22.217242+0000","mapping_epoch":579,"log_start":"629'7300","ondisk_log_start":"629'7300","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2868,"log_dups_size":3000,"ondisk_log_size":2868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222822400,"num_objects":3400,"num_object_clones":0,"num_object_copies":10200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10168,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":597,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.846997+0000","last_change":"2024-02-05T07:38:11.671455+0000","last_active":"2024-02-05T07:40:19.846997+0000","last_peered":"2024-02-05T07:40:19.846997+0000","last_clean":"2024-02-05T07:40:19.846997+0000","last_became_active":"2024-02-05T07:38:11.671087+0000","last_became_peered":"2024-02-05T07:38:11.671087+0000","last_unstale":"2024-02-05T07:40:19.846997+0000","last_undegraded":"2024-02-05T07:40:19.846997+0000","last_fullsized":"2024-02-05T07:40:19.846997+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:40.411931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:40.411931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"616'46","reported_seq":1352,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.837503+0000","last_change":"2024-02-05T07:38:52.880432+0000","last_active":"2024-02-05T07:40:19.837503+0000","last_peered":"2024-02-05T07:40:19.837503+0000","last_clean":"2024-02-05T07:40:19.837503+0000","last_became_active":"2024-02-05T07:38:09.500221+0000","last_became_peered":"2024-02-05T07:38:09.500221+0000","last_unstale":"2024-02-05T07:40:19.837503+0000","last_undegraded":"2024-02-05T07:40:19.837503+0000","last_fullsized":"2024-02-05T07:40:19.837503+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'34","last_scrub_stamp":"2024-02-05T07:38:52.880291+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:38:52.880291+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":811,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.879863+0000","last_change":"2024-02-05T07:38:09.485474+0000","last_active":"2024-02-05T07:40:21.879863+0000","last_peered":"2024-02-05T07:40:21.879863+0000","last_clean":"2024-02-05T07:40:21.879863+0000","last_became_active":"2024-02-05T07:38:09.485164+0000","last_became_peered":"2024-02-05T07:38:09.485164+0000","last_unstale":"2024-02-05T07:40:21.879863+0000","last_undegraded":"2024-02-05T07:40:21.879863+0000","last_fullsized":"2024-02-05T07:40:21.879863+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:47.116727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:47.116727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"589'12","reported_seq":1370,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.849794+0000","last_change":"2024-02-05T07:38:09.514237+0000","last_active":"2024-02-05T07:40:19.849794+0000","last_peered":"2024-02-05T07:40:19.849794+0000","last_clean":"2024-02-05T07:40:19.849794+0000","last_became_active":"2024-02-05T07:38:09.488856+0000","last_became_peered":"2024-02-05T07:38:09.488856+0000","last_unstale":"2024-02-05T07:40:19.849794+0000","last_undegraded":"2024-02-05T07:40:19.849794+0000","last_fullsized":"2024-02-05T07:40:19.849794+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'11","last_scrub_stamp":"2024-02-05T07:36:59.523421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.523421+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":238,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.850777+0000","last_change":"2024-02-05T07:39:50.551862+0000","last_active":"2024-02-05T07:40:19.850777+0000","last_peered":"2024-02-05T07:40:19.850777+0000","last_clean":"2024-02-05T07:40:19.850777+0000","last_became_active":"2024-02-05T07:36:32.744070+0000","last_became_peered":"2024-02-05T07:36:32.744070+0000","last_unstale":"2024-02-05T07:40:19.850777+0000","last_undegraded":"2024-02-05T07:40:19.850777+0000","last_fullsized":"2024-02-05T07:40:19.850777+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:50.551740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:50.551740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:06.027327+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"645'10285","reported_seq":10428,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:20.636152+0000","last_change":"2024-02-05T07:37:53.089610+0000","last_active":"2024-02-05T07:40:20.636152+0000","last_peered":"2024-02-05T07:40:20.636152+0000","last_clean":"2024-02-05T07:40:20.636152+0000","last_became_active":"2024-02-05T07:37:53.089296+0000","last_became_peered":"2024-02-05T07:37:53.089296+0000","last_unstale":"2024-02-05T07:40:20.636152+0000","last_undegraded":"2024-02-05T07:40:20.636152+0000","last_fullsized":"2024-02-05T07:40:20.636152+0000","mapping_epoch":579,"log_start":"628'7172","ondisk_log_start":"628'7172","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228917248,"num_objects":3493,"num_object_clones":0,"num_object_copies":10479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10285,"num_write_kb":440896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":1191,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.850297+0000","last_change":"2024-02-05T07:38:34.512785+0000","last_active":"2024-02-05T07:40:19.850297+0000","last_peered":"2024-02-05T07:40:19.850297+0000","last_clean":"2024-02-05T07:40:19.850297+0000","last_became_active":"2024-02-05T07:38:13.714462+0000","last_became_peered":"2024-02-05T07:38:13.714462+0000","last_unstale":"2024-02-05T07:40:19.850297+0000","last_undegraded":"2024-02-05T07:40:19.850297+0000","last_fullsized":"2024-02-05T07:40:19.850297+0000","mapping_epoch":593,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:34.509015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:34.509015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":763,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874226+0000","last_change":"2024-02-05T07:38:09.479255+0000","last_active":"2024-02-05T07:40:21.874226+0000","last_peered":"2024-02-05T07:40:21.874226+0000","last_clean":"2024-02-05T07:40:21.874226+0000","last_became_active":"2024-02-05T07:38:09.478485+0000","last_became_peered":"2024-02-05T07:38:09.478485+0000","last_unstale":"2024-02-05T07:40:21.874226+0000","last_undegraded":"2024-02-05T07:40:21.874226+0000","last_fullsized":"2024-02-05T07:40:21.874226+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.155552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.155552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"615'50","reported_seq":893,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.879572+0000","last_change":"2024-02-05T07:39:48.737503+0000","last_active":"2024-02-05T07:40:21.879572+0000","last_peered":"2024-02-05T07:40:21.879572+0000","last_clean":"2024-02-05T07:40:21.879572+0000","last_became_active":"2024-02-05T07:30:40.496940+0000","last_became_peered":"2024-02-05T07:30:40.496940+0000","last_unstale":"2024-02-05T07:40:21.879572+0000","last_undegraded":"2024-02-05T07:40:21.879572+0000","last_fullsized":"2024-02-05T07:40:21.879572+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'50","last_scrub_stamp":"2024-02-05T07:39:48.737259+0000","last_deep_scrub":"454'38","last_deep_scrub_stamp":"2024-02-05T07:34:38.942830+0000","last_clean_scrub_stamp":"2024-02-05T07:39:48.737259+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:07.832109+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":811,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.875752+0000","last_change":"2024-02-05T07:38:57.455422+0000","last_active":"2024-02-05T07:40:21.875752+0000","last_peered":"2024-02-05T07:40:21.875752+0000","last_clean":"2024-02-05T07:40:21.875752+0000","last_became_active":"2024-02-05T07:38:09.488092+0000","last_became_peered":"2024-02-05T07:38:09.488092+0000","last_unstale":"2024-02-05T07:40:21.875752+0000","last_undegraded":"2024-02-05T07:40:21.875752+0000","last_fullsized":"2024-02-05T07:40:21.875752+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:57.455281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:57.455281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1121,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.883242+0000","last_change":"2024-02-05T07:38:09.480292+0000","last_active":"2024-02-05T07:40:21.883242+0000","last_peered":"2024-02-05T07:40:21.883242+0000","last_clean":"2024-02-05T07:40:21.883242+0000","last_became_active":"2024-02-05T07:38:09.479403+0000","last_became_peered":"2024-02-05T07:38:09.479403+0000","last_unstale":"2024-02-05T07:40:21.883242+0000","last_undegraded":"2024-02-05T07:40:21.883242+0000","last_fullsized":"2024-02-05T07:40:21.883242+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:01.252507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:38:01.252507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"645'10359","reported_seq":11758,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:20.543938+0000","last_change":"2024-02-05T07:39:50.484334+0000","last_active":"2024-02-05T07:40:20.543938+0000","last_peered":"2024-02-05T07:40:20.543938+0000","last_clean":"2024-02-05T07:40:20.543938+0000","last_became_active":"2024-02-05T07:38:16.656778+0000","last_became_peered":"2024-02-05T07:38:16.656778+0000","last_unstale":"2024-02-05T07:40:20.543938+0000","last_undegraded":"2024-02-05T07:40:20.543938+0000","last_fullsized":"2024-02-05T07:40:20.543938+0000","mapping_epoch":596,"log_start":"630'7700","ondisk_log_start":"630'7700","created":579,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"634'8172","last_scrub_stamp":"2024-02-05T07:39:50.484069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:39:50.484069+0000","objects_scrubbed":5766,"log_size":2659,"log_dups_size":3000,"ondisk_log_size":2659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:09.764020+0000","scrub_duration":2626,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227475456,"num_objects":3471,"num_object_clones":0,"num_object_copies":10413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10359,"num_write_kb":442560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":589,"num_bytes_recovered":38600704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":1561,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874687+0000","last_change":"2024-02-05T07:38:11.676831+0000","last_active":"2024-02-05T07:40:21.874687+0000","last_peered":"2024-02-05T07:40:21.874687+0000","last_clean":"2024-02-05T07:40:21.874687+0000","last_became_active":"2024-02-05T07:38:11.676482+0000","last_became_peered":"2024-02-05T07:38:11.676482+0000","last_unstale":"2024-02-05T07:40:21.874687+0000","last_undegraded":"2024-02-05T07:40:21.874687+0000","last_fullsized":"2024-02-05T07:40:21.874687+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:07.353770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:07.353770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"589'11","reported_seq":971,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.879916+0000","last_change":"2024-02-05T07:38:09.522356+0000","last_active":"2024-02-05T07:40:21.879916+0000","last_peered":"2024-02-05T07:40:21.879916+0000","last_clean":"2024-02-05T07:40:21.879916+0000","last_became_active":"2024-02-05T07:38:09.495697+0000","last_became_peered":"2024-02-05T07:38:09.495697+0000","last_unstale":"2024-02-05T07:40:21.879916+0000","last_undegraded":"2024-02-05T07:40:21.879916+0000","last_fullsized":"2024-02-05T07:40:21.879916+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'10","last_scrub_stamp":"2024-02-05T07:37:59.362892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:59.362892+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"616'49","reported_seq":786,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818844+0000","last_change":"2024-02-05T07:39:55.713122+0000","last_active":"2024-02-05T07:40:18.818844+0000","last_peered":"2024-02-05T07:40:18.818844+0000","last_clean":"2024-02-05T07:40:18.818844+0000","last_became_active":"2024-02-05T07:38:09.480688+0000","last_became_peered":"2024-02-05T07:38:09.480688+0000","last_unstale":"2024-02-05T07:40:18.818844+0000","last_undegraded":"2024-02-05T07:40:18.818844+0000","last_fullsized":"2024-02-05T07:40:18.818844+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'49","last_scrub_stamp":"2024-02-05T07:39:55.713036+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:39:55.713036+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:05.586770+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"639'2","reported_seq":994,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.835598+0000","last_change":"2024-02-05T07:40:13.213597+0000","last_active":"2024-02-05T07:40:19.835598+0000","last_peered":"2024-02-05T07:40:19.835598+0000","last_clean":"2024-02-05T07:40:19.835598+0000","last_became_active":"2024-02-05T07:38:09.467535+0000","last_became_peered":"2024-02-05T07:38:09.467535+0000","last_unstale":"2024-02-05T07:40:19.835598+0000","last_undegraded":"2024-02-05T07:40:19.835598+0000","last_fullsized":"2024-02-05T07:40:19.835598+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'2","last_scrub_stamp":"2024-02-05T07:40:13.213477+0000","last_deep_scrub":"639'2","last_deep_scrub_stamp":"2024-02-05T07:40:13.213477+0000","last_clean_scrub_stamp":"2024-02-05T07:40:13.213477+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:26.617373+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1096,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818811+0000","last_change":"2024-02-05T07:37:58.737127+0000","last_active":"2024-02-05T07:40:18.818811+0000","last_peered":"2024-02-05T07:40:18.818811+0000","last_clean":"2024-02-05T07:40:18.818811+0000","last_became_active":"2024-02-05T07:36:32.701203+0000","last_became_peered":"2024-02-05T07:36:32.701203+0000","last_unstale":"2024-02-05T07:40:18.818811+0000","last_undegraded":"2024-02-05T07:40:18.818811+0000","last_fullsized":"2024-02-05T07:40:18.818811+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:58.737033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:58.737033+0000","last_clean_scrub_stamp":"2024-02-05T07:37:58.737033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"645'10364","reported_seq":11705,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:20.647194+0000","last_change":"2024-02-05T07:38:16.745449+0000","last_active":"2024-02-05T07:40:20.647194+0000","last_peered":"2024-02-05T07:40:20.647194+0000","last_clean":"2024-02-05T07:40:20.647194+0000","last_became_active":"2024-02-05T07:38:16.742262+0000","last_became_peered":"2024-02-05T07:38:16.742262+0000","last_unstale":"2024-02-05T07:40:20.647194+0000","last_undegraded":"2024-02-05T07:40:20.647194+0000","last_fullsized":"2024-02-05T07:40:20.647194+0000","mapping_epoch":596,"log_start":"628'7229","ondisk_log_start":"628'7229","created":579,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230752280,"num_objects":3522,"num_object_clones":0,"num_object_copies":10566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10364,"num_write_kb":444289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":593,"num_bytes_recovered":38862848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":1177,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819794+0000","last_change":"2024-02-05T07:37:42.825173+0000","last_active":"2024-02-05T07:40:18.819794+0000","last_peered":"2024-02-05T07:40:18.819794+0000","last_clean":"2024-02-05T07:40:18.819794+0000","last_became_active":"2024-02-05T07:37:36.688626+0000","last_became_peered":"2024-02-05T07:37:36.688626+0000","last_unstale":"2024-02-05T07:40:18.819794+0000","last_undegraded":"2024-02-05T07:40:18.819794+0000","last_fullsized":"2024-02-05T07:40:18.819794+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:42.822474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:42.822474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":961,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819077+0000","last_change":"2024-02-05T07:39:44.702368+0000","last_active":"2024-02-05T07:40:18.819077+0000","last_peered":"2024-02-05T07:40:18.819077+0000","last_clean":"2024-02-05T07:40:18.819077+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:40:18.819077+0000","last_undegraded":"2024-02-05T07:40:18.819077+0000","last_fullsized":"2024-02-05T07:40:18.819077+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:39:44.702291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:44.702291+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:52.040389+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"616'35","reported_seq":1313,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.835695+0000","last_change":"2024-02-05T07:38:50.868322+0000","last_active":"2024-02-05T07:40:19.835695+0000","last_peered":"2024-02-05T07:40:19.835695+0000","last_clean":"2024-02-05T07:40:19.835695+0000","last_became_active":"2024-02-05T07:38:09.496732+0000","last_became_peered":"2024-02-05T07:38:09.496732+0000","last_unstale":"2024-02-05T07:40:19.835695+0000","last_undegraded":"2024-02-05T07:40:19.835695+0000","last_fullsized":"2024-02-05T07:40:19.835695+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:38:50.868179+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:38:50.868179+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":814,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874719+0000","last_change":"2024-02-05T07:36:32.741872+0000","last_active":"2024-02-05T07:40:21.874719+0000","last_peered":"2024-02-05T07:40:21.874719+0000","last_clean":"2024-02-05T07:40:21.874719+0000","last_became_active":"2024-02-05T07:36:32.741377+0000","last_became_peered":"2024-02-05T07:36:32.741377+0000","last_unstale":"2024-02-05T07:40:21.874719+0000","last_undegraded":"2024-02-05T07:40:21.874719+0000","last_fullsized":"2024-02-05T07:40:21.874719+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:17.067204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:35:17.067204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":502,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.835104+0000","last_change":"2024-02-05T07:39:54.880406+0000","last_active":"2024-02-05T07:40:19.835104+0000","last_peered":"2024-02-05T07:40:19.835104+0000","last_clean":"2024-02-05T07:40:19.835104+0000","last_became_active":"2024-02-05T07:32:58.598709+0000","last_became_peered":"2024-02-05T07:32:58.598709+0000","last_unstale":"2024-02-05T07:40:19.835104+0000","last_undegraded":"2024-02-05T07:40:19.835104+0000","last_fullsized":"2024-02-05T07:40:19.835104+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:54.879841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:54.879841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:07.394562+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"645'10264","reported_seq":11682,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:20.577817+0000","last_change":"2024-02-05T07:39:46.125582+0000","last_active":"2024-02-05T07:40:20.577817+0000","last_peered":"2024-02-05T07:40:20.577817+0000","last_clean":"2024-02-05T07:40:20.577817+0000","last_became_active":"2024-02-05T07:38:15.646279+0000","last_became_peered":"2024-02-05T07:38:15.646279+0000","last_unstale":"2024-02-05T07:40:20.577817+0000","last_undegraded":"2024-02-05T07:40:20.577817+0000","last_fullsized":"2024-02-05T07:40:20.577817+0000","mapping_epoch":595,"log_start":"630'7611","ondisk_log_start":"630'7611","created":579,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'7750","last_scrub_stamp":"2024-02-05T07:39:46.125414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:39:46.125414+0000","objects_scrubbed":6078,"log_size":2653,"log_dups_size":3000,"ondisk_log_size":2653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:08.891141+0000","scrub_duration":2339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227803136,"num_objects":3476,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10264,"num_write_kb":439680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":577,"num_bytes_recovered":37814272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":1149,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819073+0000","last_change":"2024-02-05T07:37:39.746542+0000","last_active":"2024-02-05T07:40:18.819073+0000","last_peered":"2024-02-05T07:40:18.819073+0000","last_clean":"2024-02-05T07:40:18.819073+0000","last_became_active":"2024-02-05T07:37:36.693264+0000","last_became_peered":"2024-02-05T07:37:36.693264+0000","last_unstale":"2024-02-05T07:40:18.819073+0000","last_undegraded":"2024-02-05T07:40:18.819073+0000","last_fullsized":"2024-02-05T07:40:18.819073+0000","mapping_epoch":302,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.746435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.746435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1071,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818405+0000","last_change":"2024-02-05T07:40:11.774832+0000","last_active":"2024-02-05T07:40:18.818405+0000","last_peered":"2024-02-05T07:40:18.818405+0000","last_clean":"2024-02-05T07:40:18.818405+0000","last_became_active":"2024-02-05T07:36:32.709025+0000","last_became_peered":"2024-02-05T07:36:32.709025+0000","last_unstale":"2024-02-05T07:40:18.818405+0000","last_undegraded":"2024-02-05T07:40:18.818405+0000","last_fullsized":"2024-02-05T07:40:18.818405+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:11.774744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:40:11.774744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:36.409236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"615'39","reported_seq":911,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.837713+0000","last_change":"2024-02-05T07:38:41.807307+0000","last_active":"2024-02-05T07:40:19.837713+0000","last_peered":"2024-02-05T07:40:19.837713+0000","last_clean":"2024-02-05T07:40:19.837713+0000","last_became_active":"2024-02-05T07:36:32.712346+0000","last_became_peered":"2024-02-05T07:36:32.712346+0000","last_unstale":"2024-02-05T07:40:19.837713+0000","last_undegraded":"2024-02-05T07:40:19.837713+0000","last_fullsized":"2024-02-05T07:40:19.837713+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'29","last_scrub_stamp":"2024-02-05T07:38:41.806977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:41.806977+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":238,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.852478+0000","last_change":"2024-02-05T07:39:57.311987+0000","last_active":"2024-02-05T07:40:19.852478+0000","last_peered":"2024-02-05T07:40:19.852478+0000","last_clean":"2024-02-05T07:40:19.852478+0000","last_became_active":"2024-02-05T07:36:32.732781+0000","last_became_peered":"2024-02-05T07:36:32.732781+0000","last_unstale":"2024-02-05T07:40:19.852478+0000","last_undegraded":"2024-02-05T07:40:19.852478+0000","last_fullsized":"2024-02-05T07:40:19.852478+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:57.311857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:39:57.311857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:04.385252+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1232,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.833938+0000","last_change":"2024-02-05T07:40:09.083562+0000","last_active":"2024-02-05T07:40:19.833938+0000","last_peered":"2024-02-05T07:40:19.833938+0000","last_clean":"2024-02-05T07:40:19.833938+0000","last_became_active":"2024-02-05T07:36:32.709872+0000","last_became_peered":"2024-02-05T07:36:32.709872+0000","last_unstale":"2024-02-05T07:40:19.833938+0000","last_undegraded":"2024-02-05T07:40:19.833938+0000","last_fullsized":"2024-02-05T07:40:19.833938+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:09.083452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:40:09.083452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:10.818855+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"647'10379","reported_seq":10526,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:22.194884+0000","last_change":"2024-02-05T07:37:53.093098+0000","last_active":"2024-02-05T07:40:22.194884+0000","last_peered":"2024-02-05T07:40:22.194884+0000","last_clean":"2024-02-05T07:40:22.194884+0000","last_became_active":"2024-02-05T07:37:53.091347+0000","last_became_peered":"2024-02-05T07:37:53.091347+0000","last_unstale":"2024-02-05T07:40:22.194884+0000","last_undegraded":"2024-02-05T07:40:22.194884+0000","last_fullsized":"2024-02-05T07:40:22.194884+0000","mapping_epoch":579,"log_start":"630'7500","ondisk_log_start":"630'7500","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2879,"log_dups_size":3000,"ondisk_log_size":2879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220397568,"num_objects":3363,"num_object_clones":0,"num_object_copies":10089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10379,"num_write_kb":439744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":809,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819410+0000","last_change":"2024-02-05T07:38:11.676690+0000","last_active":"2024-02-05T07:40:18.819410+0000","last_peered":"2024-02-05T07:40:18.819410+0000","last_clean":"2024-02-05T07:40:18.819410+0000","last_became_active":"2024-02-05T07:38:11.676446+0000","last_became_peered":"2024-02-05T07:38:11.676446+0000","last_unstale":"2024-02-05T07:40:18.819410+0000","last_undegraded":"2024-02-05T07:40:18.819410+0000","last_fullsized":"2024-02-05T07:40:18.819410+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.778695+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.778695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":826,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.838040+0000","last_change":"2024-02-05T07:39:51.915848+0000","last_active":"2024-02-05T07:40:19.838040+0000","last_peered":"2024-02-05T07:40:19.838040+0000","last_clean":"2024-02-05T07:40:19.838040+0000","last_became_active":"2024-02-05T07:38:09.496845+0000","last_became_peered":"2024-02-05T07:38:09.496845+0000","last_unstale":"2024-02-05T07:40:19.838040+0000","last_undegraded":"2024-02-05T07:40:19.838040+0000","last_fullsized":"2024-02-05T07:40:19.838040+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:51.915741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:51.915741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:18.553450+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"615'36","reported_seq":270,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.850130+0000","last_change":"2024-02-05T07:38:54.585085+0000","last_active":"2024-02-05T07:40:19.850130+0000","last_peered":"2024-02-05T07:40:19.850130+0000","last_clean":"2024-02-05T07:40:19.850130+0000","last_became_active":"2024-02-05T07:36:32.753752+0000","last_became_peered":"2024-02-05T07:36:32.753752+0000","last_unstale":"2024-02-05T07:40:19.850130+0000","last_undegraded":"2024-02-05T07:40:19.850130+0000","last_fullsized":"2024-02-05T07:40:19.850130+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:38:54.584948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:54.584948+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:21.834141+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1215,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.835589+0000","last_change":"2024-02-05T07:38:31.740800+0000","last_active":"2024-02-05T07:40:19.835589+0000","last_peered":"2024-02-05T07:40:19.835589+0000","last_clean":"2024-02-05T07:40:19.835589+0000","last_became_active":"2024-02-05T07:30:40.493770+0000","last_became_peered":"2024-02-05T07:30:40.493770+0000","last_unstale":"2024-02-05T07:40:19.835589+0000","last_undegraded":"2024-02-05T07:40:19.835589+0000","last_fullsized":"2024-02-05T07:40:19.835589+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:38:31.740660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:38:31.740660+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":584,"num_read_kb":579,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":530,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874481+0000","last_change":"2024-02-05T07:38:41.375038+0000","last_active":"2024-02-05T07:40:21.874481+0000","last_peered":"2024-02-05T07:40:21.874481+0000","last_clean":"2024-02-05T07:40:21.874481+0000","last_became_active":"2024-02-05T07:38:09.493827+0000","last_became_peered":"2024-02-05T07:38:09.493827+0000","last_unstale":"2024-02-05T07:40:21.874481+0000","last_undegraded":"2024-02-05T07:40:21.874481+0000","last_fullsized":"2024-02-05T07:40:21.874481+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:41.374804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:38:41.374804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":1248,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.836625+0000","last_change":"2024-02-05T07:39:49.884158+0000","last_active":"2024-02-05T07:40:19.836625+0000","last_peered":"2024-02-05T07:40:19.836625+0000","last_clean":"2024-02-05T07:40:19.836625+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:40:19.836625+0000","last_undegraded":"2024-02-05T07:40:19.836625+0000","last_fullsized":"2024-02-05T07:40:19.836625+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:39:49.883906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:49.883906+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:05.915566+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"647'10340","reported_seq":10487,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:22.187159+0000","last_change":"2024-02-05T07:37:52.817223+0000","last_active":"2024-02-05T07:40:22.187159+0000","last_peered":"2024-02-05T07:40:22.187159+0000","last_clean":"2024-02-05T07:40:22.187159+0000","last_became_active":"2024-02-05T07:37:52.816946+0000","last_became_peered":"2024-02-05T07:37:52.816946+0000","last_unstale":"2024-02-05T07:40:22.187159+0000","last_undegraded":"2024-02-05T07:40:22.187159+0000","last_fullsized":"2024-02-05T07:40:22.187159+0000","mapping_epoch":579,"log_start":"629'7500","ondisk_log_start":"629'7500","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2840,"log_dups_size":3000,"ondisk_log_size":2840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223346688,"num_objects":3408,"num_object_clones":0,"num_object_copies":10224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10340,"num_write_kb":439936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":238,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.852611+0000","last_change":"2024-02-05T07:38:43.594222+0000","last_active":"2024-02-05T07:40:19.852611+0000","last_peered":"2024-02-05T07:40:19.852611+0000","last_clean":"2024-02-05T07:40:19.852611+0000","last_became_active":"2024-02-05T07:36:32.732334+0000","last_became_peered":"2024-02-05T07:36:32.732334+0000","last_unstale":"2024-02-05T07:40:19.852611+0000","last_undegraded":"2024-02-05T07:40:19.852611+0000","last_fullsized":"2024-02-05T07:40:19.852611+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:43.593810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:43.593810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"616'44","reported_seq":1400,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819935+0000","last_change":"2024-02-05T07:38:09.529174+0000","last_active":"2024-02-05T07:40:18.819935+0000","last_peered":"2024-02-05T07:40:18.819935+0000","last_clean":"2024-02-05T07:40:18.819935+0000","last_became_active":"2024-02-05T07:38:09.479528+0000","last_became_peered":"2024-02-05T07:38:09.479528+0000","last_unstale":"2024-02-05T07:40:18.819935+0000","last_undegraded":"2024-02-05T07:40:18.819935+0000","last_fullsized":"2024-02-05T07:40:18.819935+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'31","last_scrub_stamp":"2024-02-05T07:37:45.644963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.644963+0000","objects_scrubbed":8,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":103,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.846070+0000","last_change":"2024-02-05T07:40:08.312293+0000","last_active":"2024-02-05T07:40:19.846070+0000","last_peered":"2024-02-05T07:40:19.846070+0000","last_clean":"2024-02-05T07:40:19.846070+0000","last_became_active":"2024-02-05T07:38:31.807354+0000","last_became_peered":"2024-02-05T07:38:31.807354+0000","last_unstale":"2024-02-05T07:40:19.846070+0000","last_undegraded":"2024-02-05T07:40:19.846070+0000","last_fullsized":"2024-02-05T07:40:19.846070+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:08.312041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:40:08.312041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:35.861633+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":979,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818331+0000","last_change":"2024-02-05T07:38:29.812121+0000","last_active":"2024-02-05T07:40:18.818331+0000","last_peered":"2024-02-05T07:40:18.818331+0000","last_clean":"2024-02-05T07:40:18.818331+0000","last_became_active":"2024-02-05T07:37:06.977158+0000","last_became_peered":"2024-02-05T07:37:06.977158+0000","last_unstale":"2024-02-05T07:40:18.818331+0000","last_undegraded":"2024-02-05T07:40:18.818331+0000","last_fullsized":"2024-02-05T07:40:18.818331+0000","mapping_epoch":367,"log_start":"86'128","ondisk_log_start":"86'128","created":20,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:29.812050+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:38:29.812050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":805,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.875672+0000","last_change":"2024-02-05T07:38:49.589976+0000","last_active":"2024-02-05T07:40:21.875672+0000","last_peered":"2024-02-05T07:40:21.875672+0000","last_clean":"2024-02-05T07:40:21.875672+0000","last_became_active":"2024-02-05T07:38:09.488951+0000","last_became_peered":"2024-02-05T07:38:09.488951+0000","last_unstale":"2024-02-05T07:40:21.875672+0000","last_undegraded":"2024-02-05T07:40:21.875672+0000","last_fullsized":"2024-02-05T07:40:21.875672+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:49.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:49.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"647'10298","reported_seq":10445,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:22.182546+0000","last_change":"2024-02-05T07:37:53.090431+0000","last_active":"2024-02-05T07:40:22.182546+0000","last_peered":"2024-02-05T07:40:22.182546+0000","last_clean":"2024-02-05T07:40:22.182546+0000","last_became_active":"2024-02-05T07:37:53.090153+0000","last_became_peered":"2024-02-05T07:37:53.090153+0000","last_unstale":"2024-02-05T07:40:22.182546+0000","last_undegraded":"2024-02-05T07:40:22.182546+0000","last_fullsized":"2024-02-05T07:40:22.182546+0000","mapping_epoch":579,"log_start":"629'7400","ondisk_log_start":"629'7400","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2898,"log_dups_size":3000,"ondisk_log_size":2898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222167040,"num_objects":3390,"num_object_clones":0,"num_object_copies":10170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10298,"num_write_kb":438016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":1204,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.847164+0000","last_change":"2024-02-05T07:38:32.532424+0000","last_active":"2024-02-05T07:40:19.847164+0000","last_peered":"2024-02-05T07:40:19.847164+0000","last_clean":"2024-02-05T07:40:19.847164+0000","last_became_active":"2024-02-05T07:38:11.674897+0000","last_became_peered":"2024-02-05T07:38:11.674897+0000","last_unstale":"2024-02-05T07:40:19.847164+0000","last_undegraded":"2024-02-05T07:40:19.847164+0000","last_fullsized":"2024-02-05T07:40:19.847164+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:32.532304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:32.532304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":244,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.849994+0000","last_change":"2024-02-05T07:38:35.498140+0000","last_active":"2024-02-05T07:40:19.849994+0000","last_peered":"2024-02-05T07:40:19.849994+0000","last_clean":"2024-02-05T07:40:19.849994+0000","last_became_active":"2024-02-05T07:38:09.500885+0000","last_became_peered":"2024-02-05T07:38:09.500885+0000","last_unstale":"2024-02-05T07:40:19.849994+0000","last_undegraded":"2024-02-05T07:40:19.849994+0000","last_fullsized":"2024-02-05T07:40:19.849994+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:35.497879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:35.497879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"615'26","reported_seq":192,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.848638+0000","last_change":"2024-02-05T07:38:39.505608+0000","last_active":"2024-02-05T07:40:19.848638+0000","last_peered":"2024-02-05T07:40:19.848638+0000","last_clean":"2024-02-05T07:40:19.848638+0000","last_became_active":"2024-02-05T07:37:31.671370+0000","last_became_peered":"2024-02-05T07:37:31.671370+0000","last_unstale":"2024-02-05T07:40:19.848638+0000","last_undegraded":"2024-02-05T07:40:19.848638+0000","last_fullsized":"2024-02-05T07:40:19.848638+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'22","last_scrub_stamp":"2024-02-05T07:38:39.505230+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:38:39.505230+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":238,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.847591+0000","last_change":"2024-02-05T07:38:45.513185+0000","last_active":"2024-02-05T07:40:19.847591+0000","last_peered":"2024-02-05T07:40:19.847591+0000","last_clean":"2024-02-05T07:40:19.847591+0000","last_became_active":"2024-02-05T07:36:32.733822+0000","last_became_peered":"2024-02-05T07:36:32.733822+0000","last_unstale":"2024-02-05T07:40:19.847591+0000","last_undegraded":"2024-02-05T07:40:19.847591+0000","last_fullsized":"2024-02-05T07:40:19.847591+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:45.512912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:38:45.512912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":242,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.846105+0000","last_change":"2024-02-05T07:38:57.649523+0000","last_active":"2024-02-05T07:40:19.846105+0000","last_peered":"2024-02-05T07:40:19.846105+0000","last_clean":"2024-02-05T07:40:19.846105+0000","last_became_active":"2024-02-05T07:36:32.781195+0000","last_became_peered":"2024-02-05T07:36:32.781195+0000","last_unstale":"2024-02-05T07:40:19.846105+0000","last_undegraded":"2024-02-05T07:40:19.846105+0000","last_fullsized":"2024-02-05T07:40:19.846105+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:38:57.649351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:57.649351+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1108,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819772+0000","last_change":"2024-02-05T07:38:51.640312+0000","last_active":"2024-02-05T07:40:18.819772+0000","last_peered":"2024-02-05T07:40:18.819772+0000","last_clean":"2024-02-05T07:40:18.819772+0000","last_became_active":"2024-02-05T07:36:32.718684+0000","last_became_peered":"2024-02-05T07:36:32.718684+0000","last_unstale":"2024-02-05T07:40:18.819772+0000","last_undegraded":"2024-02-05T07:40:18.819772+0000","last_fullsized":"2024-02-05T07:40:18.819772+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:51.640224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:38:51.640224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"647'10508","reported_seq":10655,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:22.199882+0000","last_change":"2024-02-05T07:37:53.089611+0000","last_active":"2024-02-05T07:40:22.199882+0000","last_peered":"2024-02-05T07:40:22.199882+0000","last_clean":"2024-02-05T07:40:22.199882+0000","last_became_active":"2024-02-05T07:37:53.089298+0000","last_became_peered":"2024-02-05T07:37:53.089298+0000","last_unstale":"2024-02-05T07:40:22.199882+0000","last_undegraded":"2024-02-05T07:40:22.199882+0000","last_fullsized":"2024-02-05T07:40:22.199882+0000","mapping_epoch":579,"log_start":"630'7700","ondisk_log_start":"630'7700","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2808,"log_dups_size":3000,"ondisk_log_size":2808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223346688,"num_objects":3408,"num_object_clones":0,"num_object_copies":10224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10508,"num_write_kb":445312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":591,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.847408+0000","last_change":"2024-02-05T07:38:15.643159+0000","last_active":"2024-02-05T07:40:19.847408+0000","last_peered":"2024-02-05T07:40:19.847408+0000","last_clean":"2024-02-05T07:40:19.847408+0000","last_became_active":"2024-02-05T07:38:15.642779+0000","last_became_peered":"2024-02-05T07:38:15.642779+0000","last_unstale":"2024-02-05T07:40:19.847408+0000","last_undegraded":"2024-02-05T07:40:19.847408+0000","last_fullsized":"2024-02-05T07:40:19.847408+0000","mapping_epoch":595,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:08.283492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:08.283492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":981,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.834535+0000","last_change":"2024-02-05T07:40:19.104414+0000","last_active":"2024-02-05T07:40:19.834535+0000","last_peered":"2024-02-05T07:40:19.834535+0000","last_clean":"2024-02-05T07:40:19.834535+0000","last_became_active":"2024-02-05T07:38:09.463763+0000","last_became_peered":"2024-02-05T07:38:09.463763+0000","last_unstale":"2024-02-05T07:40:19.834535+0000","last_undegraded":"2024-02-05T07:40:19.834535+0000","last_fullsized":"2024-02-05T07:40:19.834535+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:19.104295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:40:19.104295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:47.076546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"616'68","reported_seq":334,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.852408+0000","last_change":"2024-02-05T07:36:56.264458+0000","last_active":"2024-02-05T07:40:19.852408+0000","last_peered":"2024-02-05T07:40:19.852408+0000","last_clean":"2024-02-05T07:40:19.852408+0000","last_became_active":"2024-02-05T07:36:32.738270+0000","last_became_peered":"2024-02-05T07:36:32.738270+0000","last_unstale":"2024-02-05T07:40:19.852408+0000","last_undegraded":"2024-02-05T07:40:19.852408+0000","last_fullsized":"2024-02-05T07:40:19.852408+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'56","last_scrub_stamp":"2024-02-05T07:36:56.264300+0000","last_deep_scrub":"550'56","last_deep_scrub_stamp":"2024-02-05T07:36:56.264300+0000","last_clean_scrub_stamp":"2024-02-05T07:36:56.264300+0000","objects_scrubbed":10,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":82,"num_read_kb":67,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":637,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819513+0000","last_change":"2024-02-05T07:40:15.743056+0000","last_active":"2024-02-05T07:40:18.819513+0000","last_peered":"2024-02-05T07:40:18.819513+0000","last_clean":"2024-02-05T07:40:18.819513+0000","last_became_active":"2024-02-05T07:38:09.484410+0000","last_became_peered":"2024-02-05T07:38:09.484410+0000","last_unstale":"2024-02-05T07:40:18.819513+0000","last_undegraded":"2024-02-05T07:40:18.819513+0000","last_fullsized":"2024-02-05T07:40:18.819513+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:15.742953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:40:15.742953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:22.968200+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":1145,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.835257+0000","last_change":"2024-02-05T07:40:12.141885+0000","last_active":"2024-02-05T07:40:19.835257+0000","last_peered":"2024-02-05T07:40:19.835257+0000","last_clean":"2024-02-05T07:40:19.835257+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:40:19.835257+0000","last_undegraded":"2024-02-05T07:40:19.835257+0000","last_fullsized":"2024-02-05T07:40:19.835257+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:40:12.141744+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:40:12.141744+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:22.389002+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":819,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.879834+0000","last_change":"2024-02-05T07:39:03.439286+0000","last_active":"2024-02-05T07:40:21.879834+0000","last_peered":"2024-02-05T07:40:21.879834+0000","last_clean":"2024-02-05T07:40:21.879834+0000","last_became_active":"2024-02-05T07:38:09.486846+0000","last_became_peered":"2024-02-05T07:38:09.486846+0000","last_unstale":"2024-02-05T07:40:21.879834+0000","last_undegraded":"2024-02-05T07:40:21.879834+0000","last_fullsized":"2024-02-05T07:40:21.879834+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:03.439161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:03.439161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"644'10413","reported_seq":10553,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.835925+0000","last_change":"2024-02-05T07:37:52.813698+0000","last_active":"2024-02-05T07:40:18.835925+0000","last_peered":"2024-02-05T07:40:18.835925+0000","last_clean":"2024-02-05T07:40:18.835925+0000","last_became_active":"2024-02-05T07:37:52.813405+0000","last_became_peered":"2024-02-05T07:37:52.813405+0000","last_unstale":"2024-02-05T07:40:18.835925+0000","last_undegraded":"2024-02-05T07:40:18.835925+0000","last_fullsized":"2024-02-05T07:40:18.835925+0000","mapping_epoch":579,"log_start":"629'7617","ondisk_log_start":"629'7617","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2796,"log_dups_size":3000,"ondisk_log_size":2796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245825536,"num_objects":3751,"num_object_clones":0,"num_object_copies":11253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":453248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":558,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.880466+0000","last_change":"2024-02-05T07:40:20.876587+0000","last_active":"2024-02-05T07:40:21.880466+0000","last_peered":"2024-02-05T07:40:21.880466+0000","last_clean":"2024-02-05T07:40:21.880466+0000","last_became_active":"2024-02-05T07:40:20.876256+0000","last_became_peered":"2024-02-05T07:40:20.876256+0000","last_unstale":"2024-02-05T07:40:21.880466+0000","last_undegraded":"2024-02-05T07:40:21.880466+0000","last_fullsized":"2024-02-05T07:40:21.880466+0000","mapping_epoch":619,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:39:45.710637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:39:45.710637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:07.509837+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"615'21","reported_seq":1035,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.837580+0000","last_change":"2024-02-05T07:38:35.790500+0000","last_active":"2024-02-05T07:40:19.837580+0000","last_peered":"2024-02-05T07:40:19.837580+0000","last_clean":"2024-02-05T07:40:19.837580+0000","last_became_active":"2024-02-05T07:38:09.479890+0000","last_became_peered":"2024-02-05T07:38:09.479890+0000","last_unstale":"2024-02-05T07:40:19.837580+0000","last_undegraded":"2024-02-05T07:40:19.837580+0000","last_fullsized":"2024-02-05T07:40:19.837580+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:38:35.790379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:35.790379+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":801,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.875290+0000","last_change":"2024-02-05T07:36:54.146303+0000","last_active":"2024-02-05T07:40:21.875290+0000","last_peered":"2024-02-05T07:40:21.875290+0000","last_clean":"2024-02-05T07:40:21.875290+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:40:21.875290+0000","last_undegraded":"2024-02-05T07:40:21.875290+0000","last_fullsized":"2024-02-05T07:40:21.875290+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:54.146194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:54.146194+0000","last_clean_scrub_stamp":"2024-02-05T07:36:54.146194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":244,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.846771+0000","last_change":"2024-02-05T07:38:09.492094+0000","last_active":"2024-02-05T07:40:19.846771+0000","last_peered":"2024-02-05T07:40:19.846771+0000","last_clean":"2024-02-05T07:40:19.846771+0000","last_became_active":"2024-02-05T07:38:09.491746+0000","last_became_peered":"2024-02-05T07:38:09.491746+0000","last_unstale":"2024-02-05T07:40:19.846771+0000","last_undegraded":"2024-02-05T07:40:19.846771+0000","last_fullsized":"2024-02-05T07:40:19.846771+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:45.362685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.362685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":238,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.847247+0000","last_change":"2024-02-05T07:37:00.310446+0000","last_active":"2024-02-05T07:40:19.847247+0000","last_peered":"2024-02-05T07:40:19.847247+0000","last_clean":"2024-02-05T07:40:19.847247+0000","last_became_active":"2024-02-05T07:36:32.734230+0000","last_became_peered":"2024-02-05T07:36:32.734230+0000","last_unstale":"2024-02-05T07:40:19.847247+0000","last_undegraded":"2024-02-05T07:40:19.847247+0000","last_fullsized":"2024-02-05T07:40:19.847247+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:00.309850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:00.309850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"643'10057","reported_seq":10198,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819864+0000","last_change":"2024-02-05T07:37:53.089683+0000","last_active":"2024-02-05T07:40:18.819864+0000","last_peered":"2024-02-05T07:40:18.819864+0000","last_clean":"2024-02-05T07:40:18.819864+0000","last_became_active":"2024-02-05T07:37:53.089386+0000","last_became_peered":"2024-02-05T07:37:53.089386+0000","last_unstale":"2024-02-05T07:40:18.819864+0000","last_undegraded":"2024-02-05T07:40:18.819864+0000","last_fullsized":"2024-02-05T07:40:18.819864+0000","mapping_epoch":579,"log_start":"628'7229","ondisk_log_start":"628'7229","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2828,"log_dups_size":3000,"ondisk_log_size":2828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242286592,"num_objects":3697,"num_object_clones":0,"num_object_copies":11091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":440128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":1201,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819285+0000","last_change":"2024-02-05T07:38:11.660956+0000","last_active":"2024-02-05T07:40:18.819285+0000","last_peered":"2024-02-05T07:40:18.819285+0000","last_clean":"2024-02-05T07:40:18.819285+0000","last_became_active":"2024-02-05T07:38:11.660622+0000","last_became_peered":"2024-02-05T07:38:11.660622+0000","last_unstale":"2024-02-05T07:40:18.819285+0000","last_undegraded":"2024-02-05T07:40:18.819285+0000","last_fullsized":"2024-02-05T07:40:18.819285+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.730473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.730473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"616'51","reported_seq":1184,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818520+0000","last_change":"2024-02-05T07:38:52.517441+0000","last_active":"2024-02-05T07:40:18.818520+0000","last_peered":"2024-02-05T07:40:18.818520+0000","last_clean":"2024-02-05T07:40:18.818520+0000","last_became_active":"2024-02-05T07:36:16.736226+0000","last_became_peered":"2024-02-05T07:36:16.736226+0000","last_unstale":"2024-02-05T07:40:18.818520+0000","last_undegraded":"2024-02-05T07:40:18.818520+0000","last_fullsized":"2024-02-05T07:40:18.818520+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:38:52.517337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:52.517337+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":103,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.849811+0000","last_change":"2024-02-05T07:40:03.305634+0000","last_active":"2024-02-05T07:40:19.849811+0000","last_peered":"2024-02-05T07:40:19.849811+0000","last_clean":"2024-02-05T07:40:19.849811+0000","last_became_active":"2024-02-05T07:38:31.795149+0000","last_became_peered":"2024-02-05T07:38:31.795149+0000","last_unstale":"2024-02-05T07:40:19.849811+0000","last_undegraded":"2024-02-05T07:40:19.849811+0000","last_fullsized":"2024-02-05T07:40:19.849811+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:03.305531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_clean_scrub_stamp":"2024-02-05T07:40:03.305531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:24.210343+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":238,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.850520+0000","last_change":"2024-02-05T07:37:54.312733+0000","last_active":"2024-02-05T07:40:19.850520+0000","last_peered":"2024-02-05T07:40:19.850520+0000","last_clean":"2024-02-05T07:40:19.850520+0000","last_became_active":"2024-02-05T07:36:32.733494+0000","last_became_peered":"2024-02-05T07:36:32.733494+0000","last_unstale":"2024-02-05T07:40:19.850520+0000","last_undegraded":"2024-02-05T07:40:19.850520+0000","last_fullsized":"2024-02-05T07:40:19.850520+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:54.312642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.312642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":548,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.873675+0000","last_change":"2024-02-05T07:38:31.803859+0000","last_active":"2024-02-05T07:40:21.873675+0000","last_peered":"2024-02-05T07:40:21.873675+0000","last_clean":"2024-02-05T07:40:21.873675+0000","last_became_active":"2024-02-05T07:38:31.803474+0000","last_became_peered":"2024-02-05T07:38:31.803474+0000","last_unstale":"2024-02-05T07:40:21.873675+0000","last_undegraded":"2024-02-05T07:40:21.873675+0000","last_fullsized":"2024-02-05T07:40:21.873675+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:01.478555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:38:01.478555+0000","last_clean_scrub_stamp":"2024-02-05T07:38:01.478555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"645'10382","reported_seq":10525,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:20.688496+0000","last_change":"2024-02-05T07:37:53.090459+0000","last_active":"2024-02-05T07:40:20.688496+0000","last_peered":"2024-02-05T07:40:20.688496+0000","last_clean":"2024-02-05T07:40:20.688496+0000","last_became_active":"2024-02-05T07:37:53.090173+0000","last_became_peered":"2024-02-05T07:37:53.090173+0000","last_unstale":"2024-02-05T07:40:20.688496+0000","last_undegraded":"2024-02-05T07:40:20.688496+0000","last_fullsized":"2024-02-05T07:40:20.688496+0000","mapping_epoch":579,"log_start":"628'7206","ondisk_log_start":"628'7206","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":3176,"log_dups_size":0,"ondisk_log_size":3176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229900288,"num_objects":3508,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10382,"num_write_kb":444480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":1566,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874288+0000","last_change":"2024-02-05T07:37:39.295414+0000","last_active":"2024-02-05T07:40:21.874288+0000","last_peered":"2024-02-05T07:40:21.874288+0000","last_clean":"2024-02-05T07:40:21.874288+0000","last_became_active":"2024-02-05T07:37:36.695548+0000","last_became_peered":"2024-02-05T07:37:36.695548+0000","last_unstale":"2024-02-05T07:40:21.874288+0000","last_undegraded":"2024-02-05T07:40:21.874288+0000","last_fullsized":"2024-02-05T07:40:21.874288+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.295329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.295329+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":238,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.846378+0000","last_change":"2024-02-05T07:37:51.288853+0000","last_active":"2024-02-05T07:40:19.846378+0000","last_peered":"2024-02-05T07:40:19.846378+0000","last_clean":"2024-02-05T07:40:19.846378+0000","last_became_active":"2024-02-05T07:36:32.733003+0000","last_became_peered":"2024-02-05T07:36:32.733003+0000","last_unstale":"2024-02-05T07:40:19.846378+0000","last_undegraded":"2024-02-05T07:40:19.846378+0000","last_fullsized":"2024-02-05T07:40:19.846378+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.288772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.288772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"615'24","reported_seq":1209,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.833689+0000","last_change":"2024-02-05T07:38:30.797759+0000","last_active":"2024-02-05T07:40:19.833689+0000","last_peered":"2024-02-05T07:40:19.833689+0000","last_clean":"2024-02-05T07:40:19.833689+0000","last_became_active":"2024-02-05T07:38:09.480893+0000","last_became_peered":"2024-02-05T07:38:09.480893+0000","last_unstale":"2024-02-05T07:40:19.833689+0000","last_undegraded":"2024-02-05T07:40:19.833689+0000","last_fullsized":"2024-02-05T07:40:19.833689+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'18","last_scrub_stamp":"2024-02-05T07:38:30.797628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:30.797628+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":135,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.835345+0000","last_change":"2024-02-05T07:40:18.107509+0000","last_active":"2024-02-05T07:40:19.835345+0000","last_peered":"2024-02-05T07:40:19.835345+0000","last_clean":"2024-02-05T07:40:19.835345+0000","last_became_active":"2024-02-05T07:38:09.485926+0000","last_became_peered":"2024-02-05T07:38:09.485926+0000","last_unstale":"2024-02-05T07:40:19.835345+0000","last_undegraded":"2024-02-05T07:40:19.835345+0000","last_fullsized":"2024-02-05T07:40:19.835345+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:18.107395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:40:18.107395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:42.598865+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":812,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.878526+0000","last_change":"2024-02-05T07:39:47.983852+0000","last_active":"2024-02-05T07:40:21.878526+0000","last_peered":"2024-02-05T07:40:21.878526+0000","last_clean":"2024-02-05T07:40:21.878526+0000","last_became_active":"2024-02-05T07:30:40.485991+0000","last_became_peered":"2024-02-05T07:30:40.485991+0000","last_unstale":"2024-02-05T07:40:21.878526+0000","last_undegraded":"2024-02-05T07:40:21.878526+0000","last_fullsized":"2024-02-05T07:40:21.878526+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:47.982166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:47.982166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.045882+0000","scrub_duration":274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"645'10210","reported_seq":11493,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:20.542933+0000","last_change":"2024-02-05T07:40:02.890033+0000","last_active":"2024-02-05T07:40:20.542933+0000","last_peered":"2024-02-05T07:40:20.542933+0000","last_clean":"2024-02-05T07:40:20.542933+0000","last_became_active":"2024-02-05T07:38:15.650851+0000","last_became_peered":"2024-02-05T07:38:15.650851+0000","last_unstale":"2024-02-05T07:40:20.542933+0000","last_undegraded":"2024-02-05T07:40:20.542933+0000","last_fullsized":"2024-02-05T07:40:20.542933+0000","mapping_epoch":595,"log_start":"630'7500","ondisk_log_start":"630'7500","created":579,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'8919","last_scrub_stamp":"2024-02-05T07:40:02.889877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:40:02.889877+0000","objects_scrubbed":4908,"log_size":2710,"log_dups_size":3000,"ondisk_log_size":2710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:14.278208+0000","scrub_duration":2873,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233046016,"num_objects":3556,"num_object_clones":0,"num_object_copies":10668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10210,"num_write_kb":440512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":536,"num_bytes_recovered":35127296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":1190,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.849971+0000","last_change":"2024-02-05T07:38:33.547522+0000","last_active":"2024-02-05T07:40:19.849971+0000","last_peered":"2024-02-05T07:40:19.849971+0000","last_clean":"2024-02-05T07:40:19.849971+0000","last_became_active":"2024-02-05T07:38:15.650809+0000","last_became_peered":"2024-02-05T07:38:15.650809+0000","last_unstale":"2024-02-05T07:40:19.849971+0000","last_undegraded":"2024-02-05T07:40:19.849971+0000","last_fullsized":"2024-02-05T07:40:19.849971+0000","mapping_epoch":595,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:33.547397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:33.547397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"589'13","reported_seq":1345,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819369+0000","last_change":"2024-02-05T07:38:09.523382+0000","last_active":"2024-02-05T07:40:18.819369+0000","last_peered":"2024-02-05T07:40:18.819369+0000","last_clean":"2024-02-05T07:40:18.819369+0000","last_became_active":"2024-02-05T07:38:09.471365+0000","last_became_peered":"2024-02-05T07:38:09.471365+0000","last_unstale":"2024-02-05T07:40:18.819369+0000","last_undegraded":"2024-02-05T07:40:18.819369+0000","last_fullsized":"2024-02-05T07:40:18.819369+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'12","last_scrub_stamp":"2024-02-05T07:37:53.638258+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:37:53.638258+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"615'39","reported_seq":894,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818557+0000","last_change":"2024-02-05T07:38:09.513629+0000","last_active":"2024-02-05T07:40:18.818557+0000","last_peered":"2024-02-05T07:40:18.818557+0000","last_clean":"2024-02-05T07:40:18.818557+0000","last_became_active":"2024-02-05T07:38:09.471133+0000","last_became_peered":"2024-02-05T07:38:09.471133+0000","last_unstale":"2024-02-05T07:40:18.818557+0000","last_undegraded":"2024-02-05T07:40:18.818557+0000","last_fullsized":"2024-02-05T07:40:18.818557+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:38:00.687570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:00.687570+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"614'4","reported_seq":152,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.837031+0000","last_change":"2024-02-05T07:38:55.830017+0000","last_active":"2024-02-05T07:40:19.837031+0000","last_peered":"2024-02-05T07:40:19.837031+0000","last_clean":"2024-02-05T07:40:19.837031+0000","last_became_active":"2024-02-05T07:38:09.482757+0000","last_became_peered":"2024-02-05T07:38:09.482757+0000","last_unstale":"2024-02-05T07:40:19.837031+0000","last_undegraded":"2024-02-05T07:40:19.837031+0000","last_fullsized":"2024-02-05T07:40:19.837031+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"614'4","last_scrub_stamp":"2024-02-05T07:38:55.829386+0000","last_deep_scrub":"614'4","last_deep_scrub_stamp":"2024-02-05T07:38:55.829386+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.829386+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:21.790215+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":893,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818387+0000","last_change":"2024-02-05T07:37:02.526520+0000","last_active":"2024-02-05T07:40:18.818387+0000","last_peered":"2024-02-05T07:40:18.818387+0000","last_clean":"2024-02-05T07:40:18.818387+0000","last_became_active":"2024-02-05T07:36:32.701535+0000","last_became_peered":"2024-02-05T07:36:32.701535+0000","last_unstale":"2024-02-05T07:40:18.818387+0000","last_undegraded":"2024-02-05T07:40:18.818387+0000","last_fullsized":"2024-02-05T07:40:18.818387+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:02.526362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:37:02.526362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":522,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.834758+0000","last_change":"2024-02-05T07:38:57.845026+0000","last_active":"2024-02-05T07:40:19.834758+0000","last_peered":"2024-02-05T07:40:19.834758+0000","last_clean":"2024-02-05T07:40:19.834758+0000","last_became_active":"2024-02-05T07:37:41.643328+0000","last_became_peered":"2024-02-05T07:37:41.643328+0000","last_unstale":"2024-02-05T07:40:19.834758+0000","last_undegraded":"2024-02-05T07:40:19.834758+0000","last_fullsized":"2024-02-05T07:40:19.834758+0000","mapping_epoch":576,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:57.844891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:57.844891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:22.230419+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"615'21","reported_seq":869,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.875487+0000","last_change":"2024-02-05T07:38:48.343585+0000","last_active":"2024-02-05T07:40:21.875487+0000","last_peered":"2024-02-05T07:40:21.875487+0000","last_clean":"2024-02-05T07:40:21.875487+0000","last_became_active":"2024-02-05T07:36:32.742483+0000","last_became_peered":"2024-02-05T07:36:32.742483+0000","last_unstale":"2024-02-05T07:40:21.875487+0000","last_undegraded":"2024-02-05T07:40:21.875487+0000","last_fullsized":"2024-02-05T07:40:21.875487+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'15","last_scrub_stamp":"2024-02-05T07:38:48.343448+0000","last_deep_scrub":"613'15","last_deep_scrub_stamp":"2024-02-05T07:38:48.343448+0000","last_clean_scrub_stamp":"2024-02-05T07:38:48.343448+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":1025,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818152+0000","last_change":"2024-02-05T07:37:47.639730+0000","last_active":"2024-02-05T07:40:18.818152+0000","last_peered":"2024-02-05T07:40:18.818152+0000","last_clean":"2024-02-05T07:40:18.818152+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:40:18.818152+0000","last_undegraded":"2024-02-05T07:40:18.818152+0000","last_fullsized":"2024-02-05T07:40:18.818152+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:37:47.639652+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:37:47.639652+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":801,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.880130+0000","last_change":"2024-02-05T07:37:01.178948+0000","last_active":"2024-02-05T07:40:21.880130+0000","last_peered":"2024-02-05T07:40:21.880130+0000","last_clean":"2024-02-05T07:40:21.880130+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:40:21.880130+0000","last_undegraded":"2024-02-05T07:40:21.880130+0000","last_fullsized":"2024-02-05T07:40:21.880130+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:01.178816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:01.178816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1088,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819553+0000","last_change":"2024-02-05T07:38:41.820804+0000","last_active":"2024-02-05T07:40:18.819553+0000","last_peered":"2024-02-05T07:40:18.819553+0000","last_clean":"2024-02-05T07:40:18.819553+0000","last_became_active":"2024-02-05T07:38:09.480228+0000","last_became_peered":"2024-02-05T07:38:09.480228+0000","last_unstale":"2024-02-05T07:40:18.819553+0000","last_undegraded":"2024-02-05T07:40:18.819553+0000","last_fullsized":"2024-02-05T07:40:18.819553+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:41.820713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:38:41.820713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":1018,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.878204+0000","last_change":"2024-02-05T07:38:41.750699+0000","last_active":"2024-02-05T07:40:21.878204+0000","last_peered":"2024-02-05T07:40:21.878204+0000","last_clean":"2024-02-05T07:40:21.878204+0000","last_became_active":"2024-02-05T07:38:41.750174+0000","last_became_peered":"2024-02-05T07:38:41.750174+0000","last_unstale":"2024-02-05T07:40:21.878204+0000","last_undegraded":"2024-02-05T07:40:21.878204+0000","last_fullsized":"2024-02-05T07:40:21.878204+0000","mapping_epoch":607,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.260984+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.260984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"615'21","reported_seq":874,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.879886+0000","last_change":"2024-02-05T07:38:55.387276+0000","last_active":"2024-02-05T07:40:21.879886+0000","last_peered":"2024-02-05T07:40:21.879886+0000","last_clean":"2024-02-05T07:40:21.879886+0000","last_became_active":"2024-02-05T07:36:32.747221+0000","last_became_peered":"2024-02-05T07:36:32.747221+0000","last_unstale":"2024-02-05T07:40:21.879886+0000","last_undegraded":"2024-02-05T07:40:21.879886+0000","last_fullsized":"2024-02-05T07:40:21.879886+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:38:55.387145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.387145+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":639,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819302+0000","last_change":"2024-02-05T07:39:00.488430+0000","last_active":"2024-02-05T07:40:18.819302+0000","last_peered":"2024-02-05T07:40:18.819302+0000","last_clean":"2024-02-05T07:40:18.819302+0000","last_became_active":"2024-02-05T07:38:09.479116+0000","last_became_peered":"2024-02-05T07:38:09.479116+0000","last_unstale":"2024-02-05T07:40:18.819302+0000","last_undegraded":"2024-02-05T07:40:18.819302+0000","last_fullsized":"2024-02-05T07:40:18.819302+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:00.488357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:39:00.488357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":804,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.875586+0000","last_change":"2024-02-05T07:38:53.372481+0000","last_active":"2024-02-05T07:40:21.875586+0000","last_peered":"2024-02-05T07:40:21.875586+0000","last_clean":"2024-02-05T07:40:21.875586+0000","last_became_active":"2024-02-05T07:30:40.496076+0000","last_became_peered":"2024-02-05T07:30:40.496076+0000","last_unstale":"2024-02-05T07:40:21.875586+0000","last_undegraded":"2024-02-05T07:40:21.875586+0000","last_fullsized":"2024-02-05T07:40:21.875586+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:53.372355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_clean_scrub_stamp":"2024-02-05T07:38:53.372355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":244,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.852620+0000","last_change":"2024-02-05T07:38:09.489174+0000","last_active":"2024-02-05T07:40:19.852620+0000","last_peered":"2024-02-05T07:40:19.852620+0000","last_clean":"2024-02-05T07:40:19.852620+0000","last_became_active":"2024-02-05T07:38:09.488755+0000","last_became_peered":"2024-02-05T07:38:09.488755+0000","last_unstale":"2024-02-05T07:40:19.852620+0000","last_undegraded":"2024-02-05T07:40:19.852620+0000","last_fullsized":"2024-02-05T07:40:19.852620+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:47.330693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:37:47.330693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":1003,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819285+0000","last_change":"2024-02-05T07:38:41.725370+0000","last_active":"2024-02-05T07:40:18.819285+0000","last_peered":"2024-02-05T07:40:18.819285+0000","last_clean":"2024-02-05T07:40:18.819285+0000","last_became_active":"2024-02-05T07:38:41.725034+0000","last_became_peered":"2024-02-05T07:38:41.725034+0000","last_unstale":"2024-02-05T07:40:18.819285+0000","last_undegraded":"2024-02-05T07:40:18.819285+0000","last_fullsized":"2024-02-05T07:40:18.819285+0000","mapping_epoch":607,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:37.743034+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:37:37.743034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"615'25","reported_seq":1162,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.875029+0000","last_change":"2024-02-05T07:37:06.319292+0000","last_active":"2024-02-05T07:40:21.875029+0000","last_peered":"2024-02-05T07:40:21.875029+0000","last_clean":"2024-02-05T07:40:21.875029+0000","last_became_active":"2024-02-05T07:30:40.483700+0000","last_became_peered":"2024-02-05T07:30:40.483700+0000","last_unstale":"2024-02-05T07:40:21.875029+0000","last_undegraded":"2024-02-05T07:40:21.875029+0000","last_fullsized":"2024-02-05T07:40:21.875029+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:37:06.318619+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:37:06.318619+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":967,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.882995+0000","last_change":"2024-02-05T07:36:46.059245+0000","last_active":"2024-02-05T07:40:21.882995+0000","last_peered":"2024-02-05T07:40:21.882995+0000","last_clean":"2024-02-05T07:40:21.882995+0000","last_became_active":"2024-02-05T07:30:40.476058+0000","last_became_peered":"2024-02-05T07:30:40.476058+0000","last_unstale":"2024-02-05T07:40:21.882995+0000","last_undegraded":"2024-02-05T07:40:21.882995+0000","last_fullsized":"2024-02-05T07:40:21.882995+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:46.059129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.059129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"538'13","reported_seq":989,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874915+0000","last_change":"2024-02-05T07:38:51.305914+0000","last_active":"2024-02-05T07:40:21.874915+0000","last_peered":"2024-02-05T07:40:21.874915+0000","last_clean":"2024-02-05T07:40:21.874915+0000","last_became_active":"2024-02-05T07:36:32.746270+0000","last_became_peered":"2024-02-05T07:36:32.746270+0000","last_unstale":"2024-02-05T07:40:21.874915+0000","last_undegraded":"2024-02-05T07:40:21.874915+0000","last_fullsized":"2024-02-05T07:40:21.874915+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'13","last_scrub_stamp":"2024-02-05T07:38:51.305782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:51.305782+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1106,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.848920+0000","last_change":"2024-02-05T07:38:09.504184+0000","last_active":"2024-02-05T07:40:19.848920+0000","last_peered":"2024-02-05T07:40:19.848920+0000","last_clean":"2024-02-05T07:40:19.848920+0000","last_became_active":"2024-02-05T07:38:09.503255+0000","last_became_peered":"2024-02-05T07:38:09.503255+0000","last_unstale":"2024-02-05T07:40:19.848920+0000","last_undegraded":"2024-02-05T07:40:19.848920+0000","last_fullsized":"2024-02-05T07:40:19.848920+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:58.277028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:58.277028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":575,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.875249+0000","last_change":"2024-02-05T07:37:40.289255+0000","last_active":"2024-02-05T07:40:21.875249+0000","last_peered":"2024-02-05T07:40:21.875249+0000","last_clean":"2024-02-05T07:40:21.875249+0000","last_became_active":"2024-02-05T07:37:12.040170+0000","last_became_peered":"2024-02-05T07:37:12.040170+0000","last_unstale":"2024-02-05T07:40:21.875249+0000","last_undegraded":"2024-02-05T07:40:21.875249+0000","last_fullsized":"2024-02-05T07:40:21.875249+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:40.289174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:40.289174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"615'35","reported_seq":894,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.875196+0000","last_change":"2024-02-05T07:36:59.117388+0000","last_active":"2024-02-05T07:40:21.875196+0000","last_peered":"2024-02-05T07:40:21.875196+0000","last_clean":"2024-02-05T07:40:21.875196+0000","last_became_active":"2024-02-05T07:36:32.746757+0000","last_became_peered":"2024-02-05T07:36:32.746757+0000","last_unstale":"2024-02-05T07:40:21.875196+0000","last_undegraded":"2024-02-05T07:40:21.875196+0000","last_fullsized":"2024-02-05T07:40:21.875196+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:36:59.117218+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.117218+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":499,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.850301+0000","last_change":"2024-02-05T07:39:53.396917+0000","last_active":"2024-02-05T07:40:19.850301+0000","last_peered":"2024-02-05T07:40:19.850301+0000","last_clean":"2024-02-05T07:40:19.850301+0000","last_became_active":"2024-02-05T07:38:09.495040+0000","last_became_peered":"2024-02-05T07:38:09.495040+0000","last_unstale":"2024-02-05T07:40:19.850301+0000","last_undegraded":"2024-02-05T07:40:19.850301+0000","last_fullsized":"2024-02-05T07:40:19.850301+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:53.396795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:39:53.396795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:06.394109+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"584'12","reported_seq":178,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874438+0000","last_change":"2024-02-05T07:40:05.765708+0000","last_active":"2024-02-05T07:40:21.874438+0000","last_peered":"2024-02-05T07:40:21.874438+0000","last_clean":"2024-02-05T07:40:21.874438+0000","last_became_active":"2024-02-05T07:37:58.781263+0000","last_became_peered":"2024-02-05T07:37:58.781263+0000","last_unstale":"2024-02-05T07:40:21.874438+0000","last_undegraded":"2024-02-05T07:40:21.874438+0000","last_fullsized":"2024-02-05T07:40:21.874438+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'12","last_scrub_stamp":"2024-02-05T07:40:05.765574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:40:05.765574+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:13.477481+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":969,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.880026+0000","last_change":"2024-02-05T07:39:01.508990+0000","last_active":"2024-02-05T07:40:21.880026+0000","last_peered":"2024-02-05T07:40:21.880026+0000","last_clean":"2024-02-05T07:40:21.880026+0000","last_became_active":"2024-02-05T07:38:09.487782+0000","last_became_peered":"2024-02-05T07:38:09.487782+0000","last_unstale":"2024-02-05T07:40:21.880026+0000","last_undegraded":"2024-02-05T07:40:21.880026+0000","last_fullsized":"2024-02-05T07:40:21.880026+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:01.508866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:01.508866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.16","version":"86'128","reported_seq":417,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818756+0000","last_change":"2024-02-05T07:38:48.574887+0000","last_active":"2024-02-05T07:40:18.818756+0000","last_peered":"2024-02-05T07:40:18.818756+0000","last_clean":"2024-02-05T07:40:18.818756+0000","last_became_active":"2024-02-05T07:38:47.704946+0000","last_became_peered":"2024-02-05T07:38:47.704946+0000","last_unstale":"2024-02-05T07:40:18.818756+0000","last_undegraded":"2024-02-05T07:40:18.818756+0000","last_fullsized":"2024-02-05T07:40:18.818756+0000","mapping_epoch":612,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:48.574781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:48.574781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"615'27","reported_seq":752,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818966+0000","last_change":"2024-02-05T07:38:09.532591+0000","last_active":"2024-02-05T07:40:18.818966+0000","last_peered":"2024-02-05T07:40:18.818966+0000","last_clean":"2024-02-05T07:40:18.818966+0000","last_became_active":"2024-02-05T07:38:09.484252+0000","last_became_peered":"2024-02-05T07:38:09.484252+0000","last_unstale":"2024-02-05T07:40:18.818966+0000","last_undegraded":"2024-02-05T07:40:18.818966+0000","last_fullsized":"2024-02-05T07:40:18.818966+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:37:48.219951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:48.219951+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1133,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.820071+0000","last_change":"2024-02-05T07:38:32.811706+0000","last_active":"2024-02-05T07:40:18.820071+0000","last_peered":"2024-02-05T07:40:18.820071+0000","last_clean":"2024-02-05T07:40:18.820071+0000","last_became_active":"2024-02-05T07:38:09.472022+0000","last_became_peered":"2024-02-05T07:38:09.472022+0000","last_unstale":"2024-02-05T07:40:18.820071+0000","last_undegraded":"2024-02-05T07:40:18.820071+0000","last_fullsized":"2024-02-05T07:40:18.820071+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:32.811595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:38:32.811595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1236,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.835599+0000","last_change":"2024-02-05T07:39:55.907631+0000","last_active":"2024-02-05T07:40:19.835599+0000","last_peered":"2024-02-05T07:40:19.835599+0000","last_clean":"2024-02-05T07:40:19.835599+0000","last_became_active":"2024-02-05T07:30:40.487141+0000","last_became_peered":"2024-02-05T07:30:40.487141+0000","last_unstale":"2024-02-05T07:40:19.835599+0000","last_undegraded":"2024-02-05T07:40:19.835599+0000","last_fullsized":"2024-02-05T07:40:19.835599+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:55.907516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_clean_scrub_stamp":"2024-02-05T07:39:55.907516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:23.259237+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":808,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874693+0000","last_change":"2024-02-05T07:37:54.469820+0000","last_active":"2024-02-05T07:40:21.874693+0000","last_peered":"2024-02-05T07:40:21.874693+0000","last_clean":"2024-02-05T07:40:21.874693+0000","last_became_active":"2024-02-05T07:36:32.742150+0000","last_became_peered":"2024-02-05T07:36:32.742150+0000","last_unstale":"2024-02-05T07:40:21.874693+0000","last_undegraded":"2024-02-05T07:40:21.874693+0000","last_fullsized":"2024-02-05T07:40:21.874693+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:54.469701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.469701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"86'128","reported_seq":1209,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819329+0000","last_change":"2024-02-05T07:38:47.700603+0000","last_active":"2024-02-05T07:40:18.819329+0000","last_peered":"2024-02-05T07:40:18.819329+0000","last_clean":"2024-02-05T07:40:18.819329+0000","last_became_active":"2024-02-05T07:38:47.700371+0000","last_became_peered":"2024-02-05T07:38:47.700371+0000","last_unstale":"2024-02-05T07:40:18.819329+0000","last_undegraded":"2024-02-05T07:40:18.819329+0000","last_fullsized":"2024-02-05T07:40:18.819329+0000","mapping_epoch":612,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:34.751313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:34.751313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"615'21","reported_seq":882,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.835034+0000","last_change":"2024-02-05T07:38:37.758235+0000","last_active":"2024-02-05T07:40:19.835034+0000","last_peered":"2024-02-05T07:40:19.835034+0000","last_clean":"2024-02-05T07:40:19.835034+0000","last_became_active":"2024-02-05T07:36:32.710090+0000","last_became_peered":"2024-02-05T07:36:32.710090+0000","last_unstale":"2024-02-05T07:40:19.835034+0000","last_undegraded":"2024-02-05T07:40:19.835034+0000","last_fullsized":"2024-02-05T07:40:19.835034+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:38:37.758114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:37.758114+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1018,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874941+0000","last_change":"2024-02-05T07:38:09.473515+0000","last_active":"2024-02-05T07:40:21.874941+0000","last_peered":"2024-02-05T07:40:21.874941+0000","last_clean":"2024-02-05T07:40:21.874941+0000","last_became_active":"2024-02-05T07:38:09.472668+0000","last_became_peered":"2024-02-05T07:38:09.472668+0000","last_unstale":"2024-02-05T07:40:21.874941+0000","last_undegraded":"2024-02-05T07:40:21.874941+0000","last_fullsized":"2024-02-05T07:40:21.874941+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:32.120410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:32.120410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":798,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.878848+0000","last_change":"2024-02-05T07:38:54.397360+0000","last_active":"2024-02-05T07:40:21.878848+0000","last_peered":"2024-02-05T07:40:21.878848+0000","last_clean":"2024-02-05T07:40:21.878848+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:40:21.878848+0000","last_undegraded":"2024-02-05T07:40:21.878848+0000","last_fullsized":"2024-02-05T07:40:21.878848+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:54.397113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:38:54.397113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":678,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818913+0000","last_change":"2024-02-05T07:40:01.732873+0000","last_active":"2024-02-05T07:40:18.818913+0000","last_peered":"2024-02-05T07:40:18.818913+0000","last_clean":"2024-02-05T07:40:18.818913+0000","last_became_active":"2024-02-05T07:34:30.670179+0000","last_became_peered":"2024-02-05T07:34:30.670179+0000","last_unstale":"2024-02-05T07:40:18.818913+0000","last_undegraded":"2024-02-05T07:40:18.818913+0000","last_fullsized":"2024-02-05T07:40:18.818913+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:01.732803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:40:01.732803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:29.909417+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":1575,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.852094+0000","last_change":"2024-02-05T07:39:11.702922+0000","last_active":"2024-02-05T07:40:19.852094+0000","last_peered":"2024-02-05T07:40:19.852094+0000","last_clean":"2024-02-05T07:40:19.852094+0000","last_became_active":"2024-02-05T07:39:11.702286+0000","last_became_peered":"2024-02-05T07:39:11.702286+0000","last_unstale":"2024-02-05T07:40:19.852094+0000","last_undegraded":"2024-02-05T07:40:19.852094+0000","last_fullsized":"2024-02-05T07:40:19.852094+0000","mapping_epoch":619,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.395638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.395638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"616'27","reported_seq":855,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874317+0000","last_change":"2024-02-05T07:36:48.088544+0000","last_active":"2024-02-05T07:40:21.874317+0000","last_peered":"2024-02-05T07:40:21.874317+0000","last_clean":"2024-02-05T07:40:21.874317+0000","last_became_active":"2024-02-05T07:30:40.501893+0000","last_became_peered":"2024-02-05T07:30:40.501893+0000","last_unstale":"2024-02-05T07:40:21.874317+0000","last_undegraded":"2024-02-05T07:40:21.874317+0000","last_fullsized":"2024-02-05T07:40:21.874317+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:36:48.088406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:36:48.088406+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":629,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.835544+0000","last_change":"2024-02-05T07:38:54.868354+0000","last_active":"2024-02-05T07:40:19.835544+0000","last_peered":"2024-02-05T07:40:19.835544+0000","last_clean":"2024-02-05T07:40:19.835544+0000","last_became_active":"2024-02-05T07:36:32.712612+0000","last_became_peered":"2024-02-05T07:36:32.712612+0000","last_unstale":"2024-02-05T07:40:19.835544+0000","last_undegraded":"2024-02-05T07:40:19.835544+0000","last_fullsized":"2024-02-05T07:40:19.835544+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:54.867940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:54.867940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":631,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818247+0000","last_change":"2024-02-05T07:38:59.527516+0000","last_active":"2024-02-05T07:40:18.818247+0000","last_peered":"2024-02-05T07:40:18.818247+0000","last_clean":"2024-02-05T07:40:18.818247+0000","last_became_active":"2024-02-05T07:38:09.480566+0000","last_became_peered":"2024-02-05T07:38:09.480566+0000","last_unstale":"2024-02-05T07:40:18.818247+0000","last_undegraded":"2024-02-05T07:40:18.818247+0000","last_fullsized":"2024-02-05T07:40:18.818247+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:59.527436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:38:59.527436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:22.924335+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":810,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.878359+0000","last_change":"2024-02-05T07:40:20.094896+0000","last_active":"2024-02-05T07:40:21.878359+0000","last_peered":"2024-02-05T07:40:21.878359+0000","last_clean":"2024-02-05T07:40:21.878359+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:40:21.878359+0000","last_undegraded":"2024-02-05T07:40:21.878359+0000","last_fullsized":"2024-02-05T07:40:21.878359+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:20.094078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:40:20.094078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:33.467712+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"86'128","reported_seq":1196,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.850534+0000","last_change":"2024-02-05T07:39:46.502942+0000","last_active":"2024-02-05T07:40:19.850534+0000","last_peered":"2024-02-05T07:40:19.850534+0000","last_clean":"2024-02-05T07:40:19.850534+0000","last_became_active":"2024-02-05T07:39:17.741563+0000","last_became_peered":"2024-02-05T07:39:17.741563+0000","last_unstale":"2024-02-05T07:40:19.850534+0000","last_undegraded":"2024-02-05T07:40:19.850534+0000","last_fullsized":"2024-02-05T07:40:19.850534+0000","mapping_epoch":623,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:39:46.502804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:39:46.502804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:58.140615+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"615'12","reported_seq":820,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.880100+0000","last_change":"2024-02-05T07:35:47.070886+0000","last_active":"2024-02-05T07:40:21.880100+0000","last_peered":"2024-02-05T07:40:21.880100+0000","last_clean":"2024-02-05T07:40:21.880100+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:40:21.880100+0000","last_undegraded":"2024-02-05T07:40:21.880100+0000","last_fullsized":"2024-02-05T07:40:21.880100+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:35:47.070805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:47.070805+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":809,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.875333+0000","last_change":"2024-02-05T07:40:07.758232+0000","last_active":"2024-02-05T07:40:21.875333+0000","last_peered":"2024-02-05T07:40:21.875333+0000","last_clean":"2024-02-05T07:40:21.875333+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:40:21.875333+0000","last_undegraded":"2024-02-05T07:40:21.875333+0000","last_fullsized":"2024-02-05T07:40:21.875333+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:07.758093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:34:39.980067+0000","last_clean_scrub_stamp":"2024-02-05T07:40:07.758093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:33.830081+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1073,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819033+0000","last_change":"2024-02-05T07:40:02.708852+0000","last_active":"2024-02-05T07:40:18.819033+0000","last_peered":"2024-02-05T07:40:18.819033+0000","last_clean":"2024-02-05T07:40:18.819033+0000","last_became_active":"2024-02-05T07:38:09.464446+0000","last_became_peered":"2024-02-05T07:38:09.464446+0000","last_unstale":"2024-02-05T07:40:18.819033+0000","last_undegraded":"2024-02-05T07:40:18.819033+0000","last_fullsized":"2024-02-05T07:40:18.819033+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:02.708785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:40:02.708785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:06.491220+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":244,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.847267+0000","last_change":"2024-02-05T07:38:09.504998+0000","last_active":"2024-02-05T07:40:19.847267+0000","last_peered":"2024-02-05T07:40:19.847267+0000","last_clean":"2024-02-05T07:40:19.847267+0000","last_became_active":"2024-02-05T07:38:09.503369+0000","last_became_peered":"2024-02-05T07:38:09.503369+0000","last_unstale":"2024-02-05T07:40:19.847267+0000","last_undegraded":"2024-02-05T07:40:19.847267+0000","last_fullsized":"2024-02-05T07:40:19.847267+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:58.296469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:58.296469+0000","last_clean_scrub_stamp":"2024-02-05T07:36:58.296469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1a","version":"86'128","reported_seq":529,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.846466+0000","last_change":"2024-02-05T07:39:48.483978+0000","last_active":"2024-02-05T07:40:19.846466+0000","last_peered":"2024-02-05T07:40:19.846466+0000","last_clean":"2024-02-05T07:40:19.846466+0000","last_became_active":"2024-02-05T07:39:47.747990+0000","last_became_peered":"2024-02-05T07:39:47.747990+0000","last_unstale":"2024-02-05T07:40:19.846466+0000","last_undegraded":"2024-02-05T07:40:19.846466+0000","last_fullsized":"2024-02-05T07:40:19.846466+0000","mapping_epoch":631,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:39:48.483675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:39:48.483675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:54.816352+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"616'27","reported_seq":1065,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819473+0000","last_change":"2024-02-05T07:38:55.551835+0000","last_active":"2024-02-05T07:40:18.819473+0000","last_peered":"2024-02-05T07:40:18.819473+0000","last_clean":"2024-02-05T07:40:18.819473+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:40:18.819473+0000","last_undegraded":"2024-02-05T07:40:18.819473+0000","last_fullsized":"2024-02-05T07:40:18.819473+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:38:55.551751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.551751+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:19.953801+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":806,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.879483+0000","last_change":"2024-02-05T07:40:11.850388+0000","last_active":"2024-02-05T07:40:21.879483+0000","last_peered":"2024-02-05T07:40:21.879483+0000","last_clean":"2024-02-05T07:40:21.879483+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:40:21.879483+0000","last_undegraded":"2024-02-05T07:40:21.879483+0000","last_fullsized":"2024-02-05T07:40:21.879483+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:11.849558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:40:11.849558+0000","last_clean_scrub_stamp":"2024-02-05T07:40:11.849558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:35.605390+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":246,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.849896+0000","last_change":"2024-02-05T07:40:15.407981+0000","last_active":"2024-02-05T07:40:19.849896+0000","last_peered":"2024-02-05T07:40:19.849896+0000","last_clean":"2024-02-05T07:40:19.849896+0000","last_became_active":"2024-02-05T07:36:32.752879+0000","last_became_peered":"2024-02-05T07:36:32.752879+0000","last_unstale":"2024-02-05T07:40:19.849896+0000","last_undegraded":"2024-02-05T07:40:19.849896+0000","last_fullsized":"2024-02-05T07:40:19.849896+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:15.407749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:40:15.407749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:41.593247+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":732,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:20.624422+0000","last_change":"2024-02-05T07:40:20.624459+0000","last_active":"2024-02-05T07:40:20.624422+0000","last_peered":"2024-02-05T07:40:20.624422+0000","last_clean":"2024-02-05T07:40:20.624422+0000","last_became_active":"2024-02-05T07:38:09.503835+0000","last_became_peered":"2024-02-05T07:38:09.503835+0000","last_unstale":"2024-02-05T07:40:20.624422+0000","last_undegraded":"2024-02-05T07:40:20.624422+0000","last_fullsized":"2024-02-05T07:40:20.624422+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:20.624132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:40:20.624132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:28.219914+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1b","version":"86'128","reported_seq":1006,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.850664+0000","last_change":"2024-02-05T07:39:47.744437+0000","last_active":"2024-02-05T07:40:19.850664+0000","last_peered":"2024-02-05T07:40:19.850664+0000","last_clean":"2024-02-05T07:40:19.850664+0000","last_became_active":"2024-02-05T07:39:47.744063+0000","last_became_peered":"2024-02-05T07:39:47.744063+0000","last_unstale":"2024-02-05T07:40:19.850664+0000","last_undegraded":"2024-02-05T07:40:19.850664+0000","last_fullsized":"2024-02-05T07:40:19.850664+0000","mapping_epoch":631,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:31.516142+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:38:31.516142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"616'30","reported_seq":980,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.880212+0000","last_change":"2024-02-05T07:35:13.094823+0000","last_active":"2024-02-05T07:40:21.880212+0000","last_peered":"2024-02-05T07:40:21.880212+0000","last_clean":"2024-02-05T07:40:21.880212+0000","last_became_active":"2024-02-05T07:30:40.495954+0000","last_became_peered":"2024-02-05T07:30:40.495954+0000","last_unstale":"2024-02-05T07:40:21.880212+0000","last_undegraded":"2024-02-05T07:40:21.880212+0000","last_fullsized":"2024-02-05T07:40:21.880212+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:35:13.094694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:13.094694+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"585'2","reported_seq":99,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.846578+0000","last_change":"2024-02-05T07:38:31.834917+0000","last_active":"2024-02-05T07:40:19.846578+0000","last_peered":"2024-02-05T07:40:19.846578+0000","last_clean":"2024-02-05T07:40:19.846578+0000","last_became_active":"2024-02-05T07:38:31.811200+0000","last_became_peered":"2024-02-05T07:38:31.811200+0000","last_unstale":"2024-02-05T07:40:19.846578+0000","last_undegraded":"2024-02-05T07:40:19.846578+0000","last_fullsized":"2024-02-05T07:40:19.846578+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'2","last_scrub_stamp":"2024-02-05T07:38:00.631235+0000","last_deep_scrub":"585'2","last_deep_scrub_stamp":"2024-02-05T07:38:00.631235+0000","last_clean_scrub_stamp":"2024-02-05T07:38:00.631235+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"589'14","reported_seq":1330,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819644+0000","last_change":"2024-02-05T07:38:09.524876+0000","last_active":"2024-02-05T07:40:18.819644+0000","last_peered":"2024-02-05T07:40:18.819644+0000","last_clean":"2024-02-05T07:40:18.819644+0000","last_became_active":"2024-02-05T07:38:09.472910+0000","last_became_peered":"2024-02-05T07:38:09.472910+0000","last_unstale":"2024-02-05T07:40:18.819644+0000","last_undegraded":"2024-02-05T07:40:18.819644+0000","last_fullsized":"2024-02-05T07:40:18.819644+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'13","last_scrub_stamp":"2024-02-05T07:36:57.487913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:57.487913+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":252,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.852513+0000","last_change":"2024-02-05T07:40:09.332550+0000","last_active":"2024-02-05T07:40:19.852513+0000","last_peered":"2024-02-05T07:40:19.852513+0000","last_clean":"2024-02-05T07:40:19.852513+0000","last_became_active":"2024-02-05T07:38:09.482346+0000","last_became_peered":"2024-02-05T07:38:09.482346+0000","last_unstale":"2024-02-05T07:40:19.852513+0000","last_undegraded":"2024-02-05T07:40:19.852513+0000","last_fullsized":"2024-02-05T07:40:19.852513+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:40:09.332235+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:40:09.332235+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:35.166123+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"86'128","reported_seq":754,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.845996+0000","last_change":"2024-02-05T07:39:17.754088+0000","last_active":"2024-02-05T07:40:19.845996+0000","last_peered":"2024-02-05T07:40:19.845996+0000","last_clean":"2024-02-05T07:40:19.845996+0000","last_became_active":"2024-02-05T07:39:17.753640+0000","last_became_peered":"2024-02-05T07:39:17.753640+0000","last_unstale":"2024-02-05T07:40:19.845996+0000","last_undegraded":"2024-02-05T07:40:19.845996+0000","last_fullsized":"2024-02-05T07:40:19.845996+0000","mapping_epoch":623,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:42.405286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:42.405286+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":69,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"616'29","reported_seq":859,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874620+0000","last_change":"2024-02-05T07:40:06.764929+0000","last_active":"2024-02-05T07:40:21.874620+0000","last_peered":"2024-02-05T07:40:21.874620+0000","last_clean":"2024-02-05T07:40:21.874620+0000","last_became_active":"2024-02-05T07:38:09.486225+0000","last_became_peered":"2024-02-05T07:38:09.486225+0000","last_unstale":"2024-02-05T07:40:21.874620+0000","last_undegraded":"2024-02-05T07:40:21.874620+0000","last_fullsized":"2024-02-05T07:40:21.874620+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'29","last_scrub_stamp":"2024-02-05T07:40:06.764791+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:40:06.764791+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:23.569118+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":244,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.847357+0000","last_change":"2024-02-05T07:38:09.489856+0000","last_active":"2024-02-05T07:40:19.847357+0000","last_peered":"2024-02-05T07:40:19.847357+0000","last_clean":"2024-02-05T07:40:19.847357+0000","last_became_active":"2024-02-05T07:38:09.489555+0000","last_became_peered":"2024-02-05T07:38:09.489555+0000","last_unstale":"2024-02-05T07:40:19.847357+0000","last_undegraded":"2024-02-05T07:40:19.847357+0000","last_fullsized":"2024-02-05T07:40:19.847357+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:59.269484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.269484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1123,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.875012+0000","last_change":"2024-02-05T07:38:09.484586+0000","last_active":"2024-02-05T07:40:21.875012+0000","last_peered":"2024-02-05T07:40:21.875012+0000","last_clean":"2024-02-05T07:40:21.875012+0000","last_became_active":"2024-02-05T07:38:09.484261+0000","last_became_peered":"2024-02-05T07:38:09.484261+0000","last_unstale":"2024-02-05T07:40:21.875012+0000","last_undegraded":"2024-02-05T07:40:21.875012+0000","last_fullsized":"2024-02-05T07:40:21.875012+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:47.437414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:47.437414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":195,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819440+0000","last_change":"2024-02-05T07:38:35.940570+0000","last_active":"2024-02-05T07:40:18.819440+0000","last_peered":"2024-02-05T07:40:18.819440+0000","last_clean":"2024-02-05T07:40:18.819440+0000","last_became_active":"2024-02-05T07:38:31.812780+0000","last_became_peered":"2024-02-05T07:38:31.812780+0000","last_unstale":"2024-02-05T07:40:18.819440+0000","last_undegraded":"2024-02-05T07:40:18.819440+0000","last_fullsized":"2024-02-05T07:40:18.819440+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:35.940472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:35.940472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.19","version":"86'128","reported_seq":1445,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.875405+0000","last_change":"2024-02-05T07:38:47.716030+0000","last_active":"2024-02-05T07:40:21.875405+0000","last_peered":"2024-02-05T07:40:21.875405+0000","last_clean":"2024-02-05T07:40:21.875405+0000","last_became_active":"2024-02-05T07:38:47.710478+0000","last_became_peered":"2024-02-05T07:38:47.710478+0000","last_unstale":"2024-02-05T07:40:21.875405+0000","last_undegraded":"2024-02-05T07:40:21.875405+0000","last_fullsized":"2024-02-05T07:40:21.875405+0000","mapping_epoch":612,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:42.340630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:42.340630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"615'39","reported_seq":272,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.852235+0000","last_change":"2024-02-05T07:37:57.440904+0000","last_active":"2024-02-05T07:40:19.852235+0000","last_peered":"2024-02-05T07:40:19.852235+0000","last_clean":"2024-02-05T07:40:19.852235+0000","last_became_active":"2024-02-05T07:36:32.744301+0000","last_became_peered":"2024-02-05T07:36:32.744301+0000","last_unstale":"2024-02-05T07:40:19.852235+0000","last_undegraded":"2024-02-05T07:40:19.852235+0000","last_fullsized":"2024-02-05T07:40:19.852235+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:37:57.440806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:57.440806+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":172,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.850145+0000","last_change":"2024-02-05T07:38:09.501225+0000","last_active":"2024-02-05T07:40:19.850145+0000","last_peered":"2024-02-05T07:40:19.850145+0000","last_clean":"2024-02-05T07:40:19.850145+0000","last_became_active":"2024-02-05T07:38:09.500871+0000","last_became_peered":"2024-02-05T07:38:09.500871+0000","last_unstale":"2024-02-05T07:40:19.850145+0000","last_undegraded":"2024-02-05T07:40:19.850145+0000","last_fullsized":"2024-02-05T07:40:19.850145+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:48.410161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:48.410161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1219,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.834450+0000","last_change":"2024-02-05T07:38:53.859910+0000","last_active":"2024-02-05T07:40:19.834450+0000","last_peered":"2024-02-05T07:40:19.834450+0000","last_clean":"2024-02-05T07:40:19.834450+0000","last_became_active":"2024-02-05T07:38:09.477634+0000","last_became_peered":"2024-02-05T07:38:09.477634+0000","last_unstale":"2024-02-05T07:40:19.834450+0000","last_undegraded":"2024-02-05T07:40:19.834450+0000","last_fullsized":"2024-02-05T07:40:19.834450+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:53.859782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:53.859782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":103,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.849380+0000","last_change":"2024-02-05T07:40:18.398849+0000","last_active":"2024-02-05T07:40:19.849380+0000","last_peered":"2024-02-05T07:40:19.849380+0000","last_clean":"2024-02-05T07:40:19.849380+0000","last_became_active":"2024-02-05T07:38:31.795971+0000","last_became_peered":"2024-02-05T07:38:31.795971+0000","last_unstale":"2024-02-05T07:40:19.849380+0000","last_undegraded":"2024-02-05T07:40:19.849380+0000","last_fullsized":"2024-02-05T07:40:19.849380+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:18.398592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:40:18.398592+0000","last_clean_scrub_stamp":"2024-02-05T07:40:18.398592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:45.308821+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"615'9","reported_seq":720,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.878611+0000","last_change":"2024-02-05T07:38:09.527356+0000","last_active":"2024-02-05T07:40:21.878611+0000","last_peered":"2024-02-05T07:40:21.878611+0000","last_clean":"2024-02-05T07:40:21.878611+0000","last_became_active":"2024-02-05T07:38:09.496642+0000","last_became_peered":"2024-02-05T07:38:09.496642+0000","last_unstale":"2024-02-05T07:40:21.878611+0000","last_undegraded":"2024-02-05T07:40:21.878611+0000","last_fullsized":"2024-02-05T07:40:21.878611+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'7","last_scrub_stamp":"2024-02-05T07:37:06.495165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:06.495165+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":454,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.836529+0000","last_change":"2024-02-05T07:38:47.813224+0000","last_active":"2024-02-05T07:40:19.836529+0000","last_peered":"2024-02-05T07:40:19.836529+0000","last_clean":"2024-02-05T07:40:19.836529+0000","last_became_active":"2024-02-05T07:38:09.495573+0000","last_became_peered":"2024-02-05T07:38:09.495573+0000","last_unstale":"2024-02-05T07:40:19.836529+0000","last_undegraded":"2024-02-05T07:40:19.836529+0000","last_fullsized":"2024-02-05T07:40:19.836529+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:47.812860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:47.812860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":135,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.835417+0000","last_change":"2024-02-05T07:39:45.831179+0000","last_active":"2024-02-05T07:40:19.835417+0000","last_peered":"2024-02-05T07:40:19.835417+0000","last_clean":"2024-02-05T07:40:19.835417+0000","last_became_active":"2024-02-05T07:38:09.462036+0000","last_became_peered":"2024-02-05T07:38:09.462036+0000","last_unstale":"2024-02-05T07:40:19.835417+0000","last_undegraded":"2024-02-05T07:40:19.835417+0000","last_fullsized":"2024-02-05T07:40:19.835417+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:45.831067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:45.831067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:57.942891+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1032,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.834036+0000","last_change":"2024-02-05T07:38:33.873655+0000","last_active":"2024-02-05T07:40:19.834036+0000","last_peered":"2024-02-05T07:40:19.834036+0000","last_clean":"2024-02-05T07:40:19.834036+0000","last_became_active":"2024-02-05T07:36:32.710575+0000","last_became_peered":"2024-02-05T07:36:32.710575+0000","last_unstale":"2024-02-05T07:40:19.834036+0000","last_undegraded":"2024-02-05T07:40:19.834036+0000","last_fullsized":"2024-02-05T07:40:19.834036+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:33.873524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:33.873524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"615'25","reported_seq":754,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.883088+0000","last_change":"2024-02-05T07:38:09.506020+0000","last_active":"2024-02-05T07:40:21.883088+0000","last_peered":"2024-02-05T07:40:21.883088+0000","last_clean":"2024-02-05T07:40:21.883088+0000","last_became_active":"2024-02-05T07:38:09.479937+0000","last_became_peered":"2024-02-05T07:38:09.479937+0000","last_unstale":"2024-02-05T07:40:21.883088+0000","last_undegraded":"2024-02-05T07:40:21.883088+0000","last_fullsized":"2024-02-05T07:40:21.883088+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'19","last_scrub_stamp":"2024-02-05T07:35:18.020665+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:35:18.020665+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":455,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.819710+0000","last_change":"2024-02-05T07:38:56.526617+0000","last_active":"2024-02-05T07:40:18.819710+0000","last_peered":"2024-02-05T07:40:18.819710+0000","last_clean":"2024-02-05T07:40:18.819710+0000","last_became_active":"2024-02-05T07:38:09.484875+0000","last_became_peered":"2024-02-05T07:38:09.484875+0000","last_unstale":"2024-02-05T07:40:18.819710+0000","last_undegraded":"2024-02-05T07:40:18.819710+0000","last_fullsized":"2024-02-05T07:40:18.819710+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:56.526522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:56.526522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":135,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818570+0000","last_change":"2024-02-05T07:38:47.564845+0000","last_active":"2024-02-05T07:40:18.818570+0000","last_peered":"2024-02-05T07:40:18.818570+0000","last_clean":"2024-02-05T07:40:18.818570+0000","last_became_active":"2024-02-05T07:38:09.471013+0000","last_became_peered":"2024-02-05T07:38:09.471013+0000","last_unstale":"2024-02-05T07:40:18.818570+0000","last_undegraded":"2024-02-05T07:40:18.818570+0000","last_fullsized":"2024-02-05T07:40:18.818570+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:47.564750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:47.564750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":238,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.846472+0000","last_change":"2024-02-05T07:38:51.521757+0000","last_active":"2024-02-05T07:40:19.846472+0000","last_peered":"2024-02-05T07:40:19.846472+0000","last_clean":"2024-02-05T07:40:19.846472+0000","last_became_active":"2024-02-05T07:36:32.733405+0000","last_became_peered":"2024-02-05T07:36:32.733405+0000","last_unstale":"2024-02-05T07:40:19.846472+0000","last_undegraded":"2024-02-05T07:40:19.846472+0000","last_fullsized":"2024-02-05T07:40:19.846472+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:51.521494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:51.521494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"86'128","reported_seq":1506,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874120+0000","last_change":"2024-02-05T07:38:47.704237+0000","last_active":"2024-02-05T07:40:21.874120+0000","last_peered":"2024-02-05T07:40:21.874120+0000","last_clean":"2024-02-05T07:40:21.874120+0000","last_became_active":"2024-02-05T07:38:47.703858+0000","last_became_peered":"2024-02-05T07:38:47.703858+0000","last_unstale":"2024-02-05T07:40:21.874120+0000","last_undegraded":"2024-02-05T07:40:21.874120+0000","last_fullsized":"2024-02-05T07:40:21.874120+0000","mapping_epoch":612,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:46.355408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:46.355408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"616'39","reported_seq":894,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874124+0000","last_change":"2024-02-05T07:37:50.422750+0000","last_active":"2024-02-05T07:40:21.874124+0000","last_peered":"2024-02-05T07:40:21.874124+0000","last_clean":"2024-02-05T07:40:21.874124+0000","last_became_active":"2024-02-05T07:36:32.741691+0000","last_became_peered":"2024-02-05T07:36:32.741691+0000","last_unstale":"2024-02-05T07:40:21.874124+0000","last_undegraded":"2024-02-05T07:40:21.874124+0000","last_fullsized":"2024-02-05T07:40:21.874124+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:37:50.422661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:50.422661+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":815,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.874554+0000","last_change":"2024-02-05T07:37:44.381379+0000","last_active":"2024-02-05T07:40:21.874554+0000","last_peered":"2024-02-05T07:40:21.874554+0000","last_clean":"2024-02-05T07:40:21.874554+0000","last_became_active":"2024-02-05T07:36:32.746731+0000","last_became_peered":"2024-02-05T07:36:32.746731+0000","last_unstale":"2024-02-05T07:40:21.874554+0000","last_undegraded":"2024-02-05T07:40:21.874554+0000","last_fullsized":"2024-02-05T07:40:21.874554+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:44.381300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:37:44.381300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":619,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818451+0000","last_change":"2024-02-05T07:38:09.473135+0000","last_active":"2024-02-05T07:40:18.818451+0000","last_peered":"2024-02-05T07:40:18.818451+0000","last_clean":"2024-02-05T07:40:18.818451+0000","last_became_active":"2024-02-05T07:38:09.472733+0000","last_became_peered":"2024-02-05T07:38:09.472733+0000","last_unstale":"2024-02-05T07:40:18.818451+0000","last_undegraded":"2024-02-05T07:40:18.818451+0000","last_fullsized":"2024-02-05T07:40:18.818451+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:45.170539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.170539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":816,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-05T07:40:21.878771+0000","last_change":"2024-02-05T07:38:58.711149+0000","last_active":"2024-02-05T07:40:21.878771+0000","last_peered":"2024-02-05T07:40:21.878771+0000","last_clean":"2024-02-05T07:40:21.878771+0000","last_became_active":"2024-02-05T07:30:40.493086+0000","last_became_peered":"2024-02-05T07:30:40.493086+0000","last_unstale":"2024-02-05T07:40:21.878771+0000","last_undegraded":"2024-02-05T07:40:21.878771+0000","last_fullsized":"2024-02-05T07:40:21.878771+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:58.711041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:38:58.711041+0000","last_clean_scrub_stamp":"2024-02-05T07:38:58.711041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"86'128","reported_seq":1313,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.850743+0000","last_change":"2024-02-05T07:38:47.709033+0000","last_active":"2024-02-05T07:40:19.850743+0000","last_peered":"2024-02-05T07:40:19.850743+0000","last_clean":"2024-02-05T07:40:19.850743+0000","last_became_active":"2024-02-05T07:38:47.706789+0000","last_became_peered":"2024-02-05T07:38:47.706789+0000","last_unstale":"2024-02-05T07:40:19.850743+0000","last_undegraded":"2024-02-05T07:40:19.850743+0000","last_fullsized":"2024-02-05T07:40:19.850743+0000","mapping_epoch":612,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.590393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.590393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"615'25","reported_seq":994,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818662+0000","last_change":"2024-02-05T07:39:43.728715+0000","last_active":"2024-02-05T07:40:18.818662+0000","last_peered":"2024-02-05T07:40:18.818662+0000","last_clean":"2024-02-05T07:40:18.818662+0000","last_became_active":"2024-02-05T07:38:09.470878+0000","last_became_peered":"2024-02-05T07:38:09.470878+0000","last_unstale":"2024-02-05T07:40:18.818662+0000","last_undegraded":"2024-02-05T07:40:18.818662+0000","last_fullsized":"2024-02-05T07:40:18.818662+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'25","last_scrub_stamp":"2024-02-05T07:39:43.728551+0000","last_deep_scrub":"450'19","last_deep_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_clean_scrub_stamp":"2024-02-05T07:39:43.728551+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:01.829731+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":238,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.849292+0000","last_change":"2024-02-05T07:38:47.508209+0000","last_active":"2024-02-05T07:40:19.849292+0000","last_peered":"2024-02-05T07:40:19.849292+0000","last_clean":"2024-02-05T07:40:19.849292+0000","last_became_active":"2024-02-05T07:36:32.734502+0000","last_became_peered":"2024-02-05T07:36:32.734502+0000","last_unstale":"2024-02-05T07:40:19.849292+0000","last_undegraded":"2024-02-05T07:40:19.849292+0000","last_fullsized":"2024-02-05T07:40:19.849292+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:47.508088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:38:47.508088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":238,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-05T07:40:19.847508+0000","last_change":"2024-02-05T07:37:50.267485+0000","last_active":"2024-02-05T07:40:19.847508+0000","last_peered":"2024-02-05T07:40:19.847508+0000","last_clean":"2024-02-05T07:40:19.847508+0000","last_became_active":"2024-02-05T07:36:32.708917+0000","last_became_peered":"2024-02-05T07:36:32.708917+0000","last_unstale":"2024-02-05T07:40:19.847508+0000","last_undegraded":"2024-02-05T07:40:19.847508+0000","last_fullsized":"2024-02-05T07:40:19.847508+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:50.267364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:50.267364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":745,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-05T07:40:18.818236+0000","last_change":"2024-02-05T07:38:09.480452+0000","last_active":"2024-02-05T07:40:18.818236+0000","last_peered":"2024-02-05T07:40:18.818236+0000","last_clean":"2024-02-05T07:40:18.818236+0000","last_became_active":"2024-02-05T07:38:09.480027+0000","last_became_peered":"2024-02-05T07:38:09.480027+0000","last_unstale":"2024-02-05T07:40:18.818236+0000","last_undegraded":"2024-02-05T07:40:18.818236+0000","last_fullsized":"2024-02-05T07:40:18.818236+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:49.186617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:49.186617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3680501784,"num_objects":56161,"num_object_clones":0,"num_object_copies":168483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56161,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164397,"num_write_kb":7057793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2295,"num_bytes_recovered":150405120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11189170176,"data_stored":11189157960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46146,"ondisk_log_size":46146,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":584,"num_read_kb":579,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":30,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":90,"acting":90,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":4552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1526,"num_read_kb":1283,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":10072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":503808,"data_stored":13460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1052,"ondisk_log_size":1052,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":257698038023,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":301,"seq":1292785156240,"num_pgs":110,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3318924,"kb_used_data":2582588,"kb_used_omap":5,"kb_used_meta":736314,"kb_avail":90426228,"statfs":{"total":95995035648,"available":92596457472,"internally_reserved":0,"allocated":2644570112,"data_stored":2643285394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":753986053},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.1319999999999999,"15min":1.339},"min":{"1min":0.28599999999999998,"5min":0.19600000000000001,"15min":0.192},"max":{"1min":4.4210000000000003,"5min":13.976000000000001,"15min":14.218},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.107,"15min":1.3779999999999999},"min":{"1min":0.23200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.7799999999999998,"5min":13.962999999999999,"15min":14.175000000000001},"last":0.24199999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:39:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1779999999999999,"5min":2.2709999999999999,"15min":2.0350000000000001},"min":{"1min":0.26600000000000001,"5min":0.24299999999999999,"15min":0.217},"max":{"1min":56.325000000000003,"5min":56.325000000000003,"15min":56.325000000000003},"last":3.9569999999999999},{"interface":"front","average":{"1min":4.6120000000000001,"5min":2.387,"15min":2.4980000000000002},"min":{"1min":0.252,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":60.115000000000002,"5min":60.115000000000002,"15min":61.103000000000002},"last":0.65300000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:39:47 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.9370000000000001,"15min":2.0190000000000001},"min":{"1min":0.38400000000000001,"5min":0.311,"15min":0.249},"max":{"1min":6.2930000000000001,"5min":46.927,"15min":46.927},"last":2.7869999999999999},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.7310000000000001,"15min":2.2080000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.3239999999999998,"5min":46.506,"15min":46.506},"last":2.0019999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:39:47 2024","interfaces":[{"interface":"back","average":{"1min":4.4119999999999999,"5min":1.8839999999999999,"15min":2.3820000000000001},"min":{"1min":0.30399999999999999,"5min":0.27300000000000002,"15min":0.22500000000000001},"max":{"1min":56.271000000000001,"5min":56.271000000000001,"15min":56.271000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":4.452,"5min":2.0169999999999999,"15min":2.5379999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":56.235999999999997,"5min":56.235999999999997,"15min":56.235999999999997},"last":0.47799999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.64600000000000002,"15min":1.0820000000000001},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.224},"max":{"1min":0.47399999999999998,"5min":23.155000000000001,"15min":151.02099999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.79000000000000004,"15min":1.103},"min":{"1min":0.26800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.67300000000000004,"5min":24.140000000000001,"15min":149.21799999999999},"last":0.38600000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.60599999999999998,"15min":0.57299999999999995},"min":{"1min":0.32700000000000001,"5min":0.29999999999999999,"15min":0.26800000000000002},"max":{"1min":0.71299999999999997,"5min":2.431,"15min":2.8919999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.47899999999999998,"15min":0.51700000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28100000000000003},"max":{"1min":0.53300000000000003,"5min":1.0669999999999999,"15min":1.526},"last":0.44900000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.505,"15min":2.3109999999999999},"min":{"1min":0.32200000000000001,"5min":0.315,"15min":0.28000000000000003},"max":{"1min":0.58599999999999997,"5min":2.0550000000000002,"15min":84.158000000000001},"last":0.315},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.71799999999999997,"15min":2.452},"min":{"1min":0.32400000000000001,"5min":0.317,"15min":0.312},"max":{"1min":0.621,"5min":23.055,"15min":85.018000000000001},"last":0.35599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575141,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3591796,"kb_used_data":2504808,"kb_used_omap":1,"kb_used_meta":1086974,"kb_avail":90153356,"statfs":{"total":95995035648,"available":92317036544,"internally_reserved":0,"allocated":2564923392,"data_stored":2563607936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1113061829},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:40:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1560000000000001,"5min":2.7999999999999998,"15min":1.909},"min":{"1min":0.192,"5min":0.192,"15min":0.16800000000000001},"max":{"1min":15.327,"5min":29.449999999999999,"15min":29.449999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":2.919,"5min":3.028,"15min":2.1909999999999998},"min":{"1min":0.49299999999999999,"5min":0.28299999999999997,"15min":0.20899999999999999},"max":{"1min":7.9260000000000002,"5min":29.268999999999998,"15min":29.268999999999998},"last":1.78}]},{"osd":2,"last update":"Mon Feb 5 07:39:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9489999999999998,"5min":3.3119999999999998,"15min":2.23},"min":{"1min":0.23999999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":12.848000000000001,"5min":42.823999999999998,"15min":42.823999999999998},"last":1.744},{"interface":"front","average":{"1min":2.3740000000000001,"5min":2.9500000000000002,"15min":2.069},"min":{"1min":0.32500000000000001,"5min":0.23999999999999999,"15min":0.20699999999999999},"max":{"1min":9.8030000000000008,"5min":42.426000000000002,"15min":42.426000000000002},"last":1.109}]},{"osd":3,"last update":"Mon Feb 5 07:39:36 2024","interfaces":[{"interface":"back","average":{"1min":2.4710000000000001,"5min":2.1299999999999999,"15min":2.0960000000000001},"min":{"1min":0.495,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":9.5519999999999996,"5min":9.5519999999999996,"15min":46.777000000000001},"last":2.1179999999999999},{"interface":"front","average":{"1min":2.7269999999999999,"5min":1.958,"15min":1.986},"min":{"1min":0.25,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.8799999999999999,"5min":9.5419999999999998,"15min":50.164999999999999},"last":1.1439999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.054,"15min":1.8959999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.0069999999999997,"5min":6.7329999999999997,"15min":89.409000000000006},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.4099999999999999,"15min":1.9019999999999999},"min":{"1min":0.42699999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":5.8979999999999997,"5min":14.629,"15min":89.915000000000006},"last":0.57899999999999996}]},{"osd":5,"last update":"Mon Feb 5 07:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":1.1679999999999999,"15min":2.0059999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.309},"max":{"1min":1.6359999999999999,"5min":6.7610000000000001,"15min":106.672},"last":0.51600000000000001},{"interface":"front","average":{"1min":2.2330000000000001,"5min":1.6599999999999999,"15min":2.165},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.27600000000000002},"max":{"1min":8.2739999999999991,"5min":14.613,"15min":107.348},"last":1.6200000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:39:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.288,"15min":1.3740000000000001},"min":{"1min":0.39800000000000002,"5min":0.314,"15min":0.29199999999999998},"max":{"1min":3.8220000000000001,"5min":12.685,"15min":12.685},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.2569999999999999,"15min":1.556},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.51,"5min":6.8179999999999996,"15min":8.923},"last":0.42899999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:39:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2919999999999998,"5min":1.587,"15min":2.0259999999999998},"min":{"1min":0.49399999999999999,"5min":0.36599999999999999,"15min":0.318},"max":{"1min":8.2439999999999998,"5min":8.2439999999999998,"15min":66.703999999999994},"last":1.679},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.3280000000000001,"15min":1.603},"min":{"1min":0.40999999999999998,"5min":0.40500000000000003,"15min":0.34000000000000002},"max":{"1min":12.781000000000001,"5min":12.781000000000001,"15min":61.594999999999999},"last":0.60499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738666,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3283708,"kb_used_data":2227948,"kb_used_omap":1,"kb_used_meta":1055742,"kb_avail":90461444,"statfs":{"total":95995035648,"available":92632518656,"internally_reserved":0,"allocated":2281418752,"data_stored":2280105753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1081080261},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:39:50 2024","interfaces":[{"interface":"back","average":{"1min":6.7060000000000004,"5min":3.1669999999999998,"15min":2.6280000000000001},"min":{"1min":0.39000000000000001,"5min":0.189,"15min":0.16500000000000001},"max":{"1min":65.799000000000007,"5min":65.799000000000007,"15min":65.799000000000007},"last":0.80200000000000005},{"interface":"front","average":{"1min":6.2720000000000002,"5min":2.4969999999999999,"15min":2.399},"min":{"1min":0.33900000000000002,"5min":0.192,"15min":0.192},"max":{"1min":65.093999999999994,"5min":65.093999999999994,"15min":65.093999999999994},"last":5.1340000000000003}]},{"osd":2,"last update":"Mon Feb 5 07:39:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":2.7589999999999999,"15min":2.2650000000000001},"min":{"1min":0.34899999999999998,"5min":0.22900000000000001,"15min":0.191},"max":{"1min":6.6719999999999997,"5min":47.137999999999998,"15min":65.903999999999996},"last":2.6789999999999998},{"interface":"front","average":{"1min":1.5569999999999999,"5min":2.528,"15min":2.1720000000000002},"min":{"1min":0.32000000000000001,"5min":0.20300000000000001,"15min":0.186},"max":{"1min":3.7469999999999999,"5min":47.170999999999999,"15min":65.694000000000003},"last":1.77}]},{"osd":3,"last update":"Mon Feb 5 07:39:29 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":1.77,"15min":1.7609999999999999},"min":{"1min":0.25900000000000001,"5min":0.25600000000000001,"15min":0.21299999999999999},"max":{"1min":5.3360000000000003,"5min":26.43,"15min":29.629000000000001},"last":1.7430000000000001},{"interface":"front","average":{"1min":2.8650000000000002,"5min":2.802,"15min":2.2509999999999999},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":7.8799999999999999,"5min":25.337,"15min":29.463999999999999},"last":0.82399999999999995}]},{"osd":4,"last update":"Mon Feb 5 07:39:50 2024","interfaces":[{"interface":"back","average":{"1min":2.1720000000000002,"5min":2.113,"15min":1.9890000000000001},"min":{"1min":0.35499999999999998,"5min":0.308,"15min":0.26700000000000002},"max":{"1min":7.8460000000000001,"5min":12.718999999999999,"15min":50.545000000000002},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.573,"5min":1.349,"15min":1.802},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.27100000000000002},"max":{"1min":5.4770000000000003,"5min":5.4770000000000003,"15min":50.310000000000002},"last":1.1479999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:39:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.167,"15min":1.462},"min":{"1min":0.35599999999999998,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":3.448,"5min":4.9690000000000003,"15min":95.772999999999996},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.4430000000000001,"15min":1.577},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.27600000000000002},"max":{"1min":5.4770000000000003,"5min":8.7599999999999998,"15min":96.028000000000006},"last":1.1879999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:39:48 2024","interfaces":[{"interface":"back","average":{"1min":2.4649999999999999,"5min":2.0800000000000001,"15min":1.4790000000000001},"min":{"1min":0.41999999999999998,"5min":0.27800000000000002,"15min":0.26300000000000001},"max":{"1min":7.7889999999999997,"5min":12.670999999999999,"15min":12.670999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.4470000000000001,"15min":1.4219999999999999},"min":{"1min":0.377,"5min":0.376,"15min":0.35099999999999998},"max":{"1min":5.5140000000000002,"5min":5.7229999999999999,"15min":8.718},"last":1.232}]},{"osd":7,"last update":"Mon Feb 5 07:40:10 2024","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":2.0939999999999999,"15min":1.798},"min":{"1min":0.38500000000000001,"5min":0.33500000000000002,"15min":0.28599999999999998},"max":{"1min":7.8200000000000003,"5min":12.766999999999999,"15min":43.012},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.742,"5min":1.4199999999999999,"15min":1.6459999999999999},"min":{"1min":0.48799999999999999,"5min":0.39100000000000001,"15min":0.309},"max":{"1min":5.641,"5min":5.641,"15min":43.512},"last":1.266}]}]},{"osd":2,"up_from":18,"seq":77309411616,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2550116,"kb_used_data":1584596,"kb_used_omap":1,"kb_used_meta":965502,"kb_avail":91195036,"statfs":{"total":95995035648,"available":93383716864,"internally_reserved":0,"allocated":1622626304,"data_stored":1621331426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":988674501},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:39:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":2.5649999999999999,"15min":2.1560000000000001},"min":{"1min":0.28000000000000003,"5min":0.21099999999999999,"15min":0.182},"max":{"1min":3.911,"5min":80.427000000000007,"15min":92.638000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":2.0699999999999998,"5min":3.0609999999999999,"15min":2.4169999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.20699999999999999},"max":{"1min":11.583,"5min":80.939999999999998,"15min":93.481999999999999},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Feb 5 07:39:26 2024","interfaces":[{"interface":"back","average":{"1min":2.8239999999999998,"5min":3.8199999999999998,"15min":3.2050000000000001},"min":{"1min":0.498,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.0830000000000002,"5min":110.917,"15min":110.917},"last":1.5049999999999999},{"interface":"front","average":{"1min":2.306,"5min":1.5860000000000001,"15min":2.605},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":7.9509999999999996,"5min":7.9509999999999996,"15min":93.004000000000005},"last":2.3290000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:39:39 2024","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":3.8479999999999999,"15min":2.8519999999999999},"min":{"1min":0.247,"5min":0.214,"15min":0.214},"max":{"1min":6.6950000000000003,"5min":110.773,"15min":110.773},"last":2.3109999999999999},{"interface":"front","average":{"1min":2.0590000000000002,"5min":1.984,"15min":2.1480000000000001},"min":{"1min":0.35999999999999999,"5min":0.28299999999999997,"15min":0.22600000000000001},"max":{"1min":6.7309999999999999,"5min":47.368000000000002,"15min":63.521000000000001},"last":4.1920000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":2.0979999999999999,"15min":2.3690000000000002},"min":{"1min":0.28699999999999998,"5min":0.27700000000000002,"15min":0.24299999999999999},"max":{"1min":2.629,"5min":46.521999999999998,"15min":168.11699999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.5640000000000001,"15min":2.153},"min":{"1min":0.39800000000000002,"5min":0.36599999999999999,"15min":0.27200000000000002},"max":{"1min":2.8359999999999999,"5min":40.329000000000001,"15min":168.589},"last":0.33800000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.1080000000000001,"15min":1.2150000000000001},"min":{"1min":0.35899999999999999,"5min":0.35699999999999998,"15min":0.315},"max":{"1min":4.4649999999999999,"5min":6.0129999999999999,"15min":39.923000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":2.0710000000000002,"5min":2.476,"15min":1.663},"min":{"1min":0.52100000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.5679999999999996,"5min":47.761000000000003,"15min":47.761000000000003},"last":1.403}]},{"osd":6,"last update":"Mon Feb 5 07:39:51 2024","interfaces":[{"interface":"back","average":{"1min":1.8560000000000001,"5min":3.0619999999999998,"15min":1.782},"min":{"1min":0.40100000000000002,"5min":0.309,"15min":0.28799999999999998},"max":{"1min":5.2750000000000004,"5min":47.726999999999997,"15min":47.726999999999997},"last":1.446},{"interface":"front","average":{"1min":1.244,"5min":1.643,"15min":1.3620000000000001},"min":{"1min":0.39600000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":4.5149999999999997,"5min":40.872999999999998,"15min":40.872999999999998},"last":0.48299999999999998}]},{"osd":7,"last update":"Mon Feb 5 07:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":1.206,"15min":1.4139999999999999},"min":{"1min":0.36399999999999999,"5min":0.35699999999999998,"15min":0.30499999999999999},"max":{"1min":2.802,"5min":6.5709999999999997,"15min":37.573999999999998},"last":0.69999999999999996},{"interface":"front","average":{"1min":2.1190000000000002,"5min":3.5150000000000001,"15min":2.1339999999999999},"min":{"1min":0.53500000000000003,"5min":0.40100000000000002,"15min":0.33700000000000002},"max":{"1min":4.641,"5min":110.86799999999999,"15min":110.86799999999999},"last":1.474}]}]},{"osd":3,"up_from":26,"seq":111669149979,"num_pgs":107,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3039908,"kb_used_data":1967508,"kb_used_omap":1,"kb_used_meta":1072382,"kb_avail":90705244,"statfs":{"total":95995035648,"available":92882169856,"internally_reserved":0,"allocated":2014728192,"data_stored":2013383156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1098119623},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:39:53 2024","interfaces":[{"interface":"back","average":{"1min":1.919,"5min":2.3929999999999998,"15min":1.7170000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.782,"5min":48.688000000000002,"15min":48.688000000000002},"last":12.092000000000001},{"interface":"front","average":{"1min":2.5619999999999998,"5min":2.968,"15min":2.3199999999999998},"min":{"1min":0.36599999999999999,"5min":0.216,"15min":0.185},"max":{"1min":8.4930000000000003,"5min":48.573,"15min":99.989999999999995},"last":6.3499999999999996}]},{"osd":1,"last update":"Mon Feb 5 07:39:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8250000000000002,"5min":3.069,"15min":2.702},"min":{"1min":0.59099999999999997,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":7.274,"5min":61.941000000000003,"15min":102.20399999999999},"last":3.0089999999999999},{"interface":"front","average":{"1min":2.1190000000000002,"5min":2.73,"15min":2.6030000000000002},"min":{"1min":0.29999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.5469999999999997,"5min":61.902000000000001,"15min":102.286},"last":11.539}]},{"osd":2,"last update":"Mon Feb 5 07:39:53 2024","interfaces":[{"interface":"back","average":{"1min":2.206,"5min":3.2519999999999998,"15min":2.4729999999999999},"min":{"1min":0.45700000000000002,"5min":0.23999999999999999,"15min":0.22700000000000001},"max":{"1min":6.8200000000000003,"5min":145.94999999999999,"15min":145.94999999999999},"last":12.6},{"interface":"front","average":{"1min":1.8799999999999999,"5min":3.4340000000000002,"15min":2.4969999999999999},"min":{"1min":0.32800000000000001,"5min":0.218,"15min":0.20200000000000001},"max":{"1min":7.0110000000000001,"5min":146.006,"15min":146.006},"last":1.1060000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:40:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.3049999999999999,"15min":1.79},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27000000000000002},"max":{"1min":3.2389999999999999,"5min":9.2639999999999993,"15min":100.024},"last":1.1719999999999999},{"interface":"front","average":{"1min":2.0139999999999998,"5min":1.7470000000000001,"15min":2.0830000000000002},"min":{"1min":0.36399999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.3200000000000003,"5min":16.253,"15min":99.859999999999999},"last":1.875}]},{"osd":5,"last update":"Mon Feb 5 07:39:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.177,"15min":1.573},"min":{"1min":0.41099999999999998,"5min":0.308,"15min":0.308},"max":{"1min":4.1200000000000001,"5min":6.7599999999999998,"15min":99.911000000000001},"last":11.507999999999999},{"interface":"front","average":{"1min":1.139,"5min":1.304,"15min":1.663},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":4.1950000000000003,"5min":8.4350000000000005,"15min":100.053},"last":1.198}]},{"osd":6,"last update":"Mon Feb 5 07:39:53 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":1.7789999999999999,"15min":1.6519999999999999},"min":{"1min":0.33200000000000002,"5min":0.32700000000000001,"15min":0.29299999999999998},"max":{"1min":4.5830000000000002,"5min":32.505000000000003,"15min":99.807000000000002},"last":11.563000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.7969999999999999,"15min":1.7370000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.1020000000000003,"5min":31.084,"15min":101.489},"last":1.1439999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:40:11 2024","interfaces":[{"interface":"back","average":{"1min":1.99,"5min":1.593,"15min":2.387},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.29499999999999998},"max":{"1min":6.3609999999999998,"5min":8.4329999999999998,"15min":99.893000000000001},"last":1.923},{"interface":"front","average":{"1min":1.794,"5min":1.294,"15min":2.2679999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.6509999999999998,"5min":6.726,"15min":99.879000000000005},"last":11.457000000000001}]}]},{"osd":4,"up_from":0,"seq":137438953750,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":176093659409,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2258501632,"data_stored":2258501632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2536837120,"data_stored":2536833048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1613434880,"data_stored":1613430808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1997078528,"data_stored":1997078528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":150011904,"data_stored":150011904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2633306112,"data_stored":2633302040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:40:36.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:40:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:36.429 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:36.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:36.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:36.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:36.575 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:36.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:36.676 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:36.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:36.720 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:36.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:36.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:36.846 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:36.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:36.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:36.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:36.971 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:36.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:36.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:37.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:36 smithi007 ceph-mon[29098]: pgmap v1211: 174 pgs: 1 peering, 173 active+clean; 2.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 823 op/s 2024-02-05T07:40:37.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:36 smithi007 ceph-mon[29098]: osdmap e661: 8 total, 8 up, 5 in 2024-02-05T07:40:37.000 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:36 smithi007 ceph-mon[37158]: pgmap v1211: 174 pgs: 1 peering, 173 active+clean; 2.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 823 op/s 2024-02-05T07:40:37.000 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:36 smithi007 ceph-mon[37158]: osdmap e661: 8 total, 8 up, 5 in 2024-02-05T07:40:37.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:36 smithi090 ceph-mon[33213]: pgmap v1211: 174 pgs: 1 peering, 173 active+clean; 2.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 823 op/s 2024-02-05T07:40:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:36 smithi090 ceph-mon[33213]: osdmap e661: 8 total, 8 up, 5 in 2024-02-05T07:40:37.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:37.098 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:37.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:37.101 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:37.203 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:37.268 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:37.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:37.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:37.418 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:37.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:37.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:37.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:37.545 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:37.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:37.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:37.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:37.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:37.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:37.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:37.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:37.830 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:37.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:37 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:40:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:37 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:40:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:37 smithi007 ceph-mon[29098]: osdmap e662: 8 total, 8 up, 5 in 2024-02-05T07:40:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:37 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-05T07:40:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:37 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-05T07:40:37.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:37 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:40:37.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:37 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:40:37.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:37 smithi007 ceph-mon[37158]: osdmap e662: 8 total, 8 up, 5 in 2024-02-05T07:40:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:37 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-05T07:40:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:37 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-05T07:40:37.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:37.957 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:37.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:37.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:38.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:37 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:40:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:37 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:40:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:37 smithi090 ceph-mon[33213]: osdmap e662: 8 total, 8 up, 5 in 2024-02-05T07:40:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:37 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-05T07:40:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:37 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-05T07:40:38.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:38.099 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:38.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:38.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:38.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:38.227 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:38.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:38.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:38.351 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:38.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:38.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:38.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:38.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:38.505 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:38.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:38.542 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:40:38.542 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-02-05T07:40:38.542 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-02-05T07:40:38.542 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-02-05T07:40:38.542 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.542 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.25454.0:190462 10.15 e527/524 10:abd78c7a:::benchmark_data_smithi007_157098_object61532:head v 527'9875, mlcod=526'9873)", 2024-02-05T07:40:38.542 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:19.777482+0000", 2024-02-05T07:40:38.542 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 258.76014076400003, 2024-02-05T07:40:38.542 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.43455628000000002, 2024-02-05T07:40:38.542 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777482+0000", 2024-02-05T07:40:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777482+0000", 2024-02-05T07:40:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.543 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777483+0000", 2024-02-05T07:40:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2249999999999999e-06 2024-02-05T07:40:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777498+0000", 2024-02-05T07:40:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5310000000000001e-05 2024-02-05T07:40:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777510+0000", 2024-02-05T07:40:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1960999999999999e-05 2024-02-05T07:40:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.544 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777519+0000", 2024-02-05T07:40:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7809999999999994e-06 2024-02-05T07:40:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777849+0000", 2024-02-05T07:40:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032970900000000002 2024-02-05T07:40:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:19.777871+0000", 2024-02-05T07:40:38.545 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2288000000000001e-05 2024-02-05T07:40:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:20.211883+0000", 2024-02-05T07:40:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.43401152500000001 2024-02-05T07:40:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:20.212038+0000", 2024-02-05T07:40:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015548100000000001 2024-02-05T07:40:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.546 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217805 10.18 10:1ea1ad8a:::benchmark_data_smithi007_157098_object88875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:45.879289+0000", 2024-02-05T07:40:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.65833375099999, 2024-02-05T07:40:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96299109100000002, 2024-02-05T07:40:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217805 2024-02-05T07:40:38.547 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879289+0000", 2024-02-05T07:40:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879289+0000", 2024-02-05T07:40:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.548 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879290+0000", 2024-02-05T07:40:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3510000000000001e-06 2024-02-05T07:40:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879295+0000", 2024-02-05T07:40:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6920000000000001e-06 2024-02-05T07:40:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879318+0000", 2024-02-05T07:40:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3315999999999998e-05 2024-02-05T07:40:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879327+0000", 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8149999999999995e-06 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879380+0000", 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3075999999999998e-05 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.879399+0000", 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8719000000000002e-05 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.550 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.835865+0000", 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.956466339 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.835977+0000", 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011222000000000001 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.836346+0000", 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00036861100000000003 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.551 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.840534+0000", 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041881720000000004 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.842031+0000", 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001497246 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.842205+0000", 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000173382 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.552 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.842254+0000", 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9375000000000001e-05 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.842280+0000", 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5777000000000001e-05 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217840 10.18 10:19c295f5:::benchmark_data_smithi007_157098_object88910:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:45.903257+0000", 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.63436556900001, 2024-02-05T07:40:38.553 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.94821095200000005, 2024-02-05T07:40:38.554 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.554 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.554 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.554 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.554 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.554 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217840 2024-02-05T07:40:38.554 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.554 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.554 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.554 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.554 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903257+0000", 2024-02-05T07:40:38.554 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.554 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.554 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.555 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.555 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903257+0000", 2024-02-05T07:40:38.555 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.555 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.555 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.555 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.555 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903258+0000", 2024-02-05T07:40:38.555 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4600000000000003e-07 2024-02-05T07:40:38.555 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.555 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.555 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.556 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903263+0000", 2024-02-05T07:40:38.556 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9490000000000004e-06 2024-02-05T07:40:38.556 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.556 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.556 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.556 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903274+0000", 2024-02-05T07:40:38.556 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1116e-05 2024-02-05T07:40:38.556 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.556 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.557 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.557 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903282+0000", 2024-02-05T07:40:38.557 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1480000000000006e-06 2024-02-05T07:40:38.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.557 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.557 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.557 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903364+0000", 2024-02-05T07:40:38.557 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2136000000000001e-05 2024-02-05T07:40:38.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.557 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.557 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.557 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.903373+0000", 2024-02-05T07:40:38.557 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3950000000000001e-06 2024-02-05T07:40:38.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.558 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.558 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.558 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.838826+0000", 2024-02-05T07:40:38.558 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.93545278899999995 2024-02-05T07:40:38.558 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.558 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.558 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.558 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.838978+0000", 2024-02-05T07:40:38.558 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000152325 2024-02-05T07:40:38.558 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.558 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.558 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.558 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.839306+0000", 2024-02-05T07:40:38.558 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032797799999999998 2024-02-05T07:40:38.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.559 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.559 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.559 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.844005+0000", 2024-02-05T07:40:38.559 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0046995270000000002 2024-02-05T07:40:38.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.559 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.559 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.559 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.847770+0000", 2024-02-05T07:40:38.559 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003764117 2024-02-05T07:40:38.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.559 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.559 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.559 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851395+0000", 2024-02-05T07:40:38.560 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0036254 2024-02-05T07:40:38.560 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.560 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.560 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.560 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851439+0000", 2024-02-05T07:40:38.560 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4477000000000001e-05 2024-02-05T07:40:38.560 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.560 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.560 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.560 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851468+0000", 2024-02-05T07:40:38.560 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8649e-05 2024-02-05T07:40:38.560 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.560 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217841 10.18 10:1fd606f1:::benchmark_data_smithi007_157098_object88911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:45.905042+0000", 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.63258094899999, 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.946656622, 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217841 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.561 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905042+0000", 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905042+0000", 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905043+0000", 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2720000000000001e-06 2024-02-05T07:40:38.562 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.563 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.563 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.563 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905048+0000", 2024-02-05T07:40:38.563 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7419999999999997e-06 2024-02-05T07:40:38.563 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.563 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.563 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.563 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905059+0000", 2024-02-05T07:40:38.563 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1039000000000001e-05 2024-02-05T07:40:38.563 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.563 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.563 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.563 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905066+0000", 2024-02-05T07:40:38.563 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7309999999999993e-06 2024-02-05T07:40:38.564 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.564 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.564 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.564 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905159+0000", 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2286000000000004e-05 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.905167+0000", 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7290000000000006e-06 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.840574+0000", 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.93540616200000004 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.565 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.840650+0000", 2024-02-05T07:40:38.566 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6518000000000002e-05 2024-02-05T07:40:38.566 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.566 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.566 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.566 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.840854+0000", 2024-02-05T07:40:38.566 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020390100000000001 2024-02-05T07:40:38.566 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.566 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.566 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.566 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.844013+0000", 2024-02-05T07:40:38.566 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0031590780000000001 2024-02-05T07:40:38.566 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.566 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.566 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.567 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.847836+0000", 2024-02-05T07:40:38.567 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038230949999999999 2024-02-05T07:40:38.567 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.567 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.567 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.567 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851594+0000", 2024-02-05T07:40:38.567 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037579369999999998 2024-02-05T07:40:38.567 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.567 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.567 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.567 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851656+0000", 2024-02-05T07:40:38.567 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2031000000000003e-05 2024-02-05T07:40:38.567 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.567 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.568 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.568 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851698+0000", 2024-02-05T07:40:38.568 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2101000000000002e-05 2024-02-05T07:40:38.568 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.568 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.568 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.568 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.568 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.568 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217870 10.18 10:1ed08b88:::benchmark_data_smithi007_157098_object88940:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.568 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:45.960539+0000", 2024-02-05T07:40:38.568 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.57708348200001, 2024-02-05T07:40:38.568 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.89133640199999997, 2024-02-05T07:40:38.568 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217870 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960539+0000", 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.569 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960539+0000", 2024-02-05T07:40:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960541+0000", 2024-02-05T07:40:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4869999999999999e-06 2024-02-05T07:40:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960545+0000", 2024-02-05T07:40:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8359999999999999e-06 2024-02-05T07:40:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960557+0000", 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1575e-05 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960565+0000", 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0819999999999999e-06 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960624+0000", 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9039000000000002e-05 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.572 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.572 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960634+0000", 2024-02-05T07:40:38.572 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8770000000000005e-06 2024-02-05T07:40:38.572 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.842355+0000", 2024-02-05T07:40:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.88172097500000002 2024-02-05T07:40:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.842453+0000", 2024-02-05T07:40:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8101000000000001e-05 2024-02-05T07:40:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.842665+0000", 2024-02-05T07:40:38.573 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00021142200000000001 2024-02-05T07:40:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.847852+0000", 2024-02-05T07:40:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0051875929999999999 2024-02-05T07:40:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.847885+0000", 2024-02-05T07:40:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2909e-05 2024-02-05T07:40:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.574 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851777+0000", 2024-02-05T07:40:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038922800000000001 2024-02-05T07:40:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851835+0000", 2024-02-05T07:40:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7788999999999998e-05 2024-02-05T07:40:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851876+0000", 2024-02-05T07:40:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0436999999999999e-05 2024-02-05T07:40:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.575 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217871 10.18 10:1967d2fc:::benchmark_data_smithi007_157098_object88941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:45.960911+0000", 2024-02-05T07:40:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.57671201400001, 2024-02-05T07:40:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.89109762400000003, 2024-02-05T07:40:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217871 2024-02-05T07:40:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.576 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960911+0000", 2024-02-05T07:40:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960911+0000", 2024-02-05T07:40:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960912+0000", 2024-02-05T07:40:38.578 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7499999999999999e-07 2024-02-05T07:40:38.578 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.578 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.578 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.578 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960916+0000", 2024-02-05T07:40:38.578 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5630000000000002e-06 2024-02-05T07:40:38.578 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.578 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.578 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.578 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960925+0000", 2024-02-05T07:40:38.578 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2869999999999993e-06 2024-02-05T07:40:38.578 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.578 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.578 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.579 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.960933+0000", 2024-02-05T07:40:38.579 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.695e-06 2024-02-05T07:40:38.579 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.579 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.579 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.579 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.961021+0000", 2024-02-05T07:40:38.579 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7545000000000003e-05 2024-02-05T07:40:38.579 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.579 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.579 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.579 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.961029+0000", 2024-02-05T07:40:38.579 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9079999999999995e-06 2024-02-05T07:40:38.579 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.579 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.580 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.580 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.844064+0000", 2024-02-05T07:40:38.580 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.883035668 2024-02-05T07:40:38.580 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.580 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.580 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.580 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.844182+0000", 2024-02-05T07:40:38.580 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011744900000000001 2024-02-05T07:40:38.580 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.580 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.580 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.580 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.844465+0000", 2024-02-05T07:40:38.580 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028307199999999999 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.847860+0000", 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033948450000000001 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851335+0000", 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003475634 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851935+0000", 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000599831 2024-02-05T07:40:38.581 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851983+0000", 2024-02-05T07:40:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8065000000000001e-05 2024-02-05T07:40:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852008+0000", 2024-02-05T07:40:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5187000000000001e-05 2024-02-05T07:40:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.582 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217905 10.18 10:1b875e8f:::benchmark_data_smithi007_157098_object88975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:45.983757+0000", 2024-02-05T07:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.553865204, 2024-02-05T07:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.86843629099999997, 2024-02-05T07:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217905 2024-02-05T07:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.584 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983757+0000", 2024-02-05T07:40:38.584 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.584 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.584 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.584 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.584 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983757+0000", 2024-02-05T07:40:38.584 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.584 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.584 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.584 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.585 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983759+0000", 2024-02-05T07:40:38.585 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1489999999999999e-06 2024-02-05T07:40:38.585 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.585 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.585 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.585 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983762+0000", 2024-02-05T07:40:38.585 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.428e-06 2024-02-05T07:40:38.585 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983770+0000", 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4930000000000001e-06 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983777+0000", 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9979999999999997e-06 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983837+0000", 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0634000000000003e-05 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.587 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.587 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.983846+0000", 2024-02-05T07:40:38.587 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0499999999999997e-06 2024-02-05T07:40:38.587 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.587 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.587 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.587 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.846009+0000", 2024-02-05T07:40:38.587 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.86216245199999997 2024-02-05T07:40:38.587 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.587 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.587 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.587 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.846089+0000", 2024-02-05T07:40:38.587 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0739000000000001e-05 2024-02-05T07:40:38.588 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.588 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.588 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.588 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.846345+0000", 2024-02-05T07:40:38.588 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00025539399999999999 2024-02-05T07:40:38.588 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.588 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.588 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.588 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.849745+0000", 2024-02-05T07:40:38.588 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0034002979999999999 2024-02-05T07:40:38.588 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.588 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.588 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.588 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851533+0000", 2024-02-05T07:40:38.589 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001787935 2024-02-05T07:40:38.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.589 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.589 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.589 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852120+0000", 2024-02-05T07:40:38.589 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00058712100000000002 2024-02-05T07:40:38.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.589 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.589 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.589 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852157+0000", 2024-02-05T07:40:38.589 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6579000000000003e-05 2024-02-05T07:40:38.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.589 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.589 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.590 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852194+0000", 2024-02-05T07:40:38.590 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7020999999999997e-05 2024-02-05T07:40:38.590 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.590 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.590 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.590 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.590 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.590 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217931 10.18 10:1ecf49c3:::benchmark_data_smithi007_157098_object89001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.590 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:45.998689+0000", 2024-02-05T07:40:38.590 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.53893324399999, 2024-02-05T07:40:38.590 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.85374700999999997, 2024-02-05T07:40:38.590 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.590 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.590 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.591 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.591 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.591 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217931 2024-02-05T07:40:38.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.591 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.591 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.591 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.998689+0000", 2024-02-05T07:40:38.591 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.591 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.591 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.998689+0000", 2024-02-05T07:40:38.592 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.592 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.592 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.592 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.998690+0000", 2024-02-05T07:40:38.592 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8700000000000004e-07 2024-02-05T07:40:38.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.592 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.592 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.592 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.998694+0000", 2024-02-05T07:40:38.592 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6009999999999999e-06 2024-02-05T07:40:38.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.592 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.592 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.593 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.998704+0000", 2024-02-05T07:40:38.593 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0013000000000001e-05 2024-02-05T07:40:38.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.593 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.593 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.593 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.998711+0000", 2024-02-05T07:40:38.593 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3130000000000001e-06 2024-02-05T07:40:38.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.593 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.593 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.593 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.999722+0000", 2024-02-05T07:40:38.593 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0010110830000000001 2024-02-05T07:40:38.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.594 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.594 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.594 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:45.999731+0000", 2024-02-05T07:40:38.594 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8759999999999994e-06 2024-02-05T07:40:38.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.594 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.594 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.594 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.847942+0000", 2024-02-05T07:40:38.594 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.84821022800000001 2024-02-05T07:40:38.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.594 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.594 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.595 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.848067+0000", 2024-02-05T07:40:38.595 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000125079 2024-02-05T07:40:38.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.595 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.595 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.595 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.848370+0000", 2024-02-05T07:40:38.595 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030292200000000001 2024-02-05T07:40:38.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.595 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.595 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.595 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851552+0000", 2024-02-05T07:40:38.595 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0031822859999999999 2024-02-05T07:40:38.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.595 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.596 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.596 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852061+0000", 2024-02-05T07:40:38.596 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00050924800000000001 2024-02-05T07:40:38.596 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.596 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.596 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.596 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852347+0000", 2024-02-05T07:40:38.596 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028602899999999998 2024-02-05T07:40:38.596 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.596 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.596 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.596 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852400+0000", 2024-02-05T07:40:38.597 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2794999999999999e-05 2024-02-05T07:40:38.597 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.597 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.597 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.597 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852436+0000", 2024-02-05T07:40:38.597 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6550000000000001e-05 2024-02-05T07:40:38.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.597 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.597 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.597 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.597 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217938 10.18 10:1d8d2565:::benchmark_data_smithi007_157098_object89008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.597 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.003702+0000", 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.53392072400001, 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.85442914000000003, 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217938 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003702+0000", 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.599 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.599 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003702+0000", 2024-02-05T07:40:38.599 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.599 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.599 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003718+0000", 2024-02-05T07:40:38.599 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5654e-05 2024-02-05T07:40:38.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.599 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.599 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003724+0000", 2024-02-05T07:40:38.600 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2360000000000001e-06 2024-02-05T07:40:38.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.600 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.600 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003736+0000", 2024-02-05T07:40:38.600 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.221e-05 2024-02-05T07:40:38.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.600 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.600 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003745+0000", 2024-02-05T07:40:38.600 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9139999999999997e-06 2024-02-05T07:40:38.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.601 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.601 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003812+0000", 2024-02-05T07:40:38.601 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6923999999999998e-05 2024-02-05T07:40:38.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.601 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.601 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.003821+0000", 2024-02-05T07:40:38.601 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6229999999999992e-06 2024-02-05T07:40:38.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.849791+0000", 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.84596975200000002 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.849887+0000", 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5978999999999997e-05 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.850108+0000", 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00022115700000000001 2024-02-05T07:40:38.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.603 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.603 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.851713+0000", 2024-02-05T07:40:38.603 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001604926 2024-02-05T07:40:38.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.603 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.603 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852282+0000", 2024-02-05T07:40:38.603 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00056861800000000001 2024-02-05T07:40:38.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.603 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.603 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.858030+0000", 2024-02-05T07:40:38.604 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0057477659999999996 2024-02-05T07:40:38.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.604 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.604 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.858088+0000", 2024-02-05T07:40:38.604 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8162000000000003e-05 2024-02-05T07:40:38.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.604 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.604 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.858131+0000", 2024-02-05T07:40:38.604 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3219000000000001e-05 2024-02-05T07:40:38.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.604 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.605 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217953 10.18 10:1831d604:::benchmark_data_smithi007_157098_object89023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.605 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.012541+0000", 2024-02-05T07:40:38.605 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.52508116199999, 2024-02-05T07:40:38.605 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.85552399199999996, 2024-02-05T07:40:38.605 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.605 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.605 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.605 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.605 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.605 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217953 2024-02-05T07:40:38.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.605 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.606 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.606 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012541+0000", 2024-02-05T07:40:38.606 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.606 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.606 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012541+0000", 2024-02-05T07:40:38.606 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.606 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.606 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012543+0000", 2024-02-05T07:40:38.606 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0189999999999999e-06 2024-02-05T07:40:38.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.607 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.607 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.607 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012546+0000", 2024-02-05T07:40:38.607 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8099999999999999e-06 2024-02-05T07:40:38.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.607 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.607 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.607 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012555+0000", 2024-02-05T07:40:38.607 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0480000000000004e-06 2024-02-05T07:40:38.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.607 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.607 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.607 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012562+0000", 2024-02-05T07:40:38.608 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4749999999999998e-06 2024-02-05T07:40:38.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.608 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.608 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.608 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012615+0000", 2024-02-05T07:40:38.608 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3467999999999997e-05 2024-02-05T07:40:38.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.608 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.608 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.608 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012625+0000", 2024-02-05T07:40:38.608 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0131e-05 2024-02-05T07:40:38.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.611 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.611 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.611 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852479+0000", 2024-02-05T07:40:38.611 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.83985321499999999 2024-02-05T07:40:38.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.612 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.612 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.612 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852593+0000", 2024-02-05T07:40:38.612 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000114287 2024-02-05T07:40:38.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.612 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.612 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.612 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.852937+0000", 2024-02-05T07:40:38.612 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00034402199999999998 2024-02-05T07:40:38.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.612 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.612 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.613 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.857954+0000", 2024-02-05T07:40:38.613 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0050172649999999999 2024-02-05T07:40:38.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.613 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.613 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.613 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.867855+0000", 2024-02-05T07:40:38.613 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0099010669999999995 2024-02-05T07:40:38.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.613 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.613 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.613 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.867964+0000", 2024-02-05T07:40:38.613 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010865 2024-02-05T07:40:38.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.613 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.614 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.614 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868022+0000", 2024-02-05T07:40:38.614 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8421000000000002e-05 2024-02-05T07:40:38.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.614 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.614 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.614 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868065+0000", 2024-02-05T07:40:38.614 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3114000000000001e-05 2024-02-05T07:40:38.614 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.614 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.614 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.614 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.614 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217954 10.18 10:190cb164:::benchmark_data_smithi007_157098_object89024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.615 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.012763+0000", 2024-02-05T07:40:38.615 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.524859306, 2024-02-05T07:40:38.615 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.85552692900000005, 2024-02-05T07:40:38.615 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.615 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.615 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.615 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.615 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.615 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217954 2024-02-05T07:40:38.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.615 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.615 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.615 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.615 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012763+0000", 2024-02-05T07:40:38.616 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.616 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.616 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012763+0000", 2024-02-05T07:40:38.616 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.616 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.616 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012764+0000", 2024-02-05T07:40:38.616 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1299999999999999e-07 2024-02-05T07:40:38.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.617 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.617 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012768+0000", 2024-02-05T07:40:38.617 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1479999999999999e-06 2024-02-05T07:40:38.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.617 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.617 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.617 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012778+0000", 2024-02-05T07:40:38.617 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7399999999999999e-06 2024-02-05T07:40:38.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.617 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.617 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.617 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012786+0000", 2024-02-05T07:40:38.617 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8399999999999995e-06 2024-02-05T07:40:38.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.618 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.618 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.618 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012836+0000", 2024-02-05T07:40:38.618 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0311e-05 2024-02-05T07:40:38.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.618 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.618 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.618 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.012844+0000", 2024-02-05T07:40:38.618 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.678e-06 2024-02-05T07:40:38.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.618 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.618 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.619 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.855466+0000", 2024-02-05T07:40:38.619 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.84262185000000001 2024-02-05T07:40:38.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.619 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.619 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.855597+0000", 2024-02-05T07:40:38.619 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013140600000000001 2024-02-05T07:40:38.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.619 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.619 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.855985+0000", 2024-02-05T07:40:38.619 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00038769599999999998 2024-02-05T07:40:38.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.620 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.620 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.867870+0000", 2024-02-05T07:40:38.620 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011885578000000001 2024-02-05T07:40:38.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.620 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.620 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868141+0000", 2024-02-05T07:40:38.620 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00027049099999999998 2024-02-05T07:40:38.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.620 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.620 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868205+0000", 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3759999999999999e-05 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868254+0000", 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9206999999999998e-05 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868290+0000", 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6411000000000001e-05 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217964 10.18 10:1bd88409:::benchmark_data_smithi007_157098_object89034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.019472+0000", 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.51815064600001, 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.84900089400000001, 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217964 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019472+0000", 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019472+0000", 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019473+0000", 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.248e-06 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019478+0000", 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5360000000000003e-06 2024-02-05T07:40:38.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.624 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.624 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019487+0000", 2024-02-05T07:40:38.624 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4180000000000003e-06 2024-02-05T07:40:38.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.624 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.624 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019493+0000", 2024-02-05T07:40:38.624 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0599999999999996e-06 2024-02-05T07:40:38.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.624 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.624 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019544+0000", 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0637000000000003e-05 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.019554+0000", 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0282000000000001e-05 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.858164+0000", 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.83860974300000002 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.625 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.858280+0000", 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011600899999999999 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.858648+0000", 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000368296 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.867876+0000", 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0092275440000000007 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.626 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868334+0000", 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00045833199999999999 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868378+0000", 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3856000000000002e-05 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868437+0000", 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9039999999999997e-05 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.627 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868473+0000", 2024-02-05T07:40:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5892999999999997e-05 2024-02-05T07:40:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:217998 10.18 10:1a72f410:::benchmark_data_smithi007_157098_object89068:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.052088+0000", 2024-02-05T07:40:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.48553415200001, 2024-02-05T07:40:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.82240571100000004, 2024-02-05T07:40:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.628 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 217998 2024-02-05T07:40:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052088+0000", 2024-02-05T07:40:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.630 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052088+0000", 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052090+0000", 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.198e-06 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052095+0000", 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.079e-06 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.631 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052109+0000", 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4113e-05 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052118+0000", 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0990000000000005e-06 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052249+0000", 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013130600000000001 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.632 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.052259+0000", 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2189999999999993e-06 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.861087+0000", 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.80882802600000003 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.861213+0000", 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012615400000000001 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.861671+0000", 2024-02-05T07:40:38.633 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00045862600000000001 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868525+0000", 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0068532360000000004 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868566+0000", 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1046e-05 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.874398+0000", 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.005832556 2024-02-05T07:40:38.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.635 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.635 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.874449+0000", 2024-02-05T07:40:38.635 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1103000000000002e-05 2024-02-05T07:40:38.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.635 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.635 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.874494+0000", 2024-02-05T07:40:38.635 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4950000000000002e-05 2024-02-05T07:40:38.635 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.635 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.635 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.636 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:218069 10.18 10:1d5b1d6a:::benchmark_data_smithi007_157098_object89139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.636 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.127842+0000", 2024-02-05T07:40:38.636 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.409780254, 2024-02-05T07:40:38.636 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.75143445600000003, 2024-02-05T07:40:38.636 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.636 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.636 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.636 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.636 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.636 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 218069 2024-02-05T07:40:38.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.636 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.636 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.636 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.637 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127842+0000", 2024-02-05T07:40:38.637 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.637 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.637 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.637 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127842+0000", 2024-02-05T07:40:38.637 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.637 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.637 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.637 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127844+0000", 2024-02-05T07:40:38.637 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2410000000000001e-06 2024-02-05T07:40:38.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127849+0000", 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4380000000000003e-06 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127859+0000", 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0084e-05 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127867+0000", 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2849999999999995e-06 2024-02-05T07:40:38.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127948+0000", 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0420999999999995e-05 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.127958+0000", 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.049e-05 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.865082+0000", 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.73712396099999999 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.640 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.640 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.865207+0000", 2024-02-05T07:40:38.640 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000124919 2024-02-05T07:40:38.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.640 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.640 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.865760+0000", 2024-02-05T07:40:38.640 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00055247599999999999 2024-02-05T07:40:38.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.640 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.640 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879087+0000", 2024-02-05T07:40:38.640 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013327105000000001 2024-02-05T07:40:38.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879137+0000", 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9985999999999999e-05 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879185+0000", 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8028000000000002e-05 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879236+0000", 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1190000000000003e-05 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.641 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.642 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.642 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879277+0000", 2024-02-05T07:40:38.642 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0831999999999998e-05 2024-02-05T07:40:38.642 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.642 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.642 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.642 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:218139 10.18 10:1cee83cf:::benchmark_data_smithi007_157098_object89209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.642 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.208744+0000", 2024-02-05T07:40:38.642 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.32887830600001, 2024-02-05T07:40:38.642 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.67071878399999996, 2024-02-05T07:40:38.642 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.642 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.643 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.643 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.643 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.643 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 218139 2024-02-05T07:40:38.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.643 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.643 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.643 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.643 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208744+0000", 2024-02-05T07:40:38.643 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.643 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.643 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.643 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208744+0000", 2024-02-05T07:40:38.644 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.644 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.644 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208746+0000", 2024-02-05T07:40:38.644 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1710000000000001e-06 2024-02-05T07:40:38.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.644 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.644 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208750+0000", 2024-02-05T07:40:38.644 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4619999999999998e-06 2024-02-05T07:40:38.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.644 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.645 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208762+0000", 2024-02-05T07:40:38.645 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1678999999999999e-05 2024-02-05T07:40:38.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.645 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.645 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.645 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208768+0000", 2024-02-05T07:40:38.645 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.494e-06 2024-02-05T07:40:38.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.645 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.645 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.645 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208814+0000", 2024-02-05T07:40:38.645 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5813000000000003e-05 2024-02-05T07:40:38.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.646 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.646 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.208823+0000", 2024-02-05T07:40:38.646 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8659999999999995e-06 2024-02-05T07:40:38.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.646 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.646 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868596+0000", 2024-02-05T07:40:38.646 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.65977350499999998 2024-02-05T07:40:38.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.646 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.868709+0000", 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011272700000000001 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.869138+0000", 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00042847799999999999 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879094+0000", 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0099561029999999991 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.647 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879335+0000", 2024-02-05T07:40:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00024086100000000001 2024-02-05T07:40:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879380+0000", 2024-02-05T07:40:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5540000000000001e-05 2024-02-05T07:40:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879426+0000", 2024-02-05T07:40:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6338999999999998e-05 2024-02-05T07:40:38.648 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879463+0000", 2024-02-05T07:40:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6745999999999997e-05 2024-02-05T07:40:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:218157 10.18 10:198d6fce:::benchmark_data_smithi007_157098_object89227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.234341+0000", 2024-02-05T07:40:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.303281731, 2024-02-05T07:40:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.64906166700000001, 2024-02-05T07:40:38.649 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 218157 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234341+0000", 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.650 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234341+0000", 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234342+0000", 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3200000000000004e-07 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234345+0000", 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.63e-06 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.651 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234356+0000", 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0413e-05 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234362+0000", 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8189999999999997e-06 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234412+0000", 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0738000000000001e-05 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.652 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.234422+0000", 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3230000000000004e-06 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.871037+0000", 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.63661556100000005 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.871162+0000", 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012524300000000001 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.653 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.872236+0000", 2024-02-05T07:40:38.654 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0010734119999999999 2024-02-05T07:40:38.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.654 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.654 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879149+0000", 2024-02-05T07:40:38.654 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0069134670000000004 2024-02-05T07:40:38.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.654 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.654 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879505+0000", 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00035600399999999999 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.883309+0000", 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038041389999999998 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.883361+0000", 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1496000000000003e-05 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.883403+0000", 2024-02-05T07:40:38.655 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1690000000000002e-05 2024-02-05T07:40:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.25454.0:218164 10.18 10:1a060940:::benchmark_data_smithi007_157098_object89234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e546)", 2024-02-05T07:40:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:36:46.251100+0000", 2024-02-05T07:40:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 232.28652291399999, 2024-02-05T07:40:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.63564669699999998, 2024-02-05T07:40:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.656 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.657 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.25454", 2024-02-05T07:40:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:40:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 218164 2024-02-05T07:40:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251100+0000", 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251100+0000", 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251101+0000", 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3179999999999999e-06 2024-02-05T07:40:38.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251105+0000", 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3699999999999997e-06 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251116+0000", 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0961999999999999e-05 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251123+0000", 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5200000000000003e-06 2024-02-05T07:40:38.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.660 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.660 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.660 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251169+0000", 2024-02-05T07:40:38.660 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5741999999999997e-05 2024-02-05T07:40:38.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.660 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.660 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.660 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.251176+0000", 2024-02-05T07:40:38.660 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8900000000000001e-06 2024-02-05T07:40:38.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.660 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.660 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.660 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.874530+0000", 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.62335404500000002 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.874646+0000", 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000116889 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.875018+0000", 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00037153499999999999 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.661 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.879566+0000", 2024-02-05T07:40:38.662 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0045484200000000001 2024-02-05T07:40:38.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.662 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.662 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.662 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.883271+0000", 2024-02-05T07:40:38.662 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037046729999999999 2024-02-05T07:40:38.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.662 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.662 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.662 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.884149+0000", 2024-02-05T07:40:38.662 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00087832700000000004 2024-02-05T07:40:38.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.662 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.662 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.884261+0000", 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000111209 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:36:46.886746+0000", 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024857970000000001 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9083 11.d 11:b2ec523a:::benchmark_data_smithi007_183880_object9082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.481230+0000", 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 150.05639237899999, 2024-02-05T07:40:38.663 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.202288376, 2024-02-05T07:40:38.664 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.664 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.664 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.664 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:40:38.664 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:40:38.664 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9083 2024-02-05T07:40:38.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.664 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.664 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.664 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.664 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481230+0000", 2024-02-05T07:40:38.664 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.664 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481230+0000", 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481232+0000", 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0839999999999999e-06 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481288+0000", 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5543e-05 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481312+0000", 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4360000000000001e-05 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481325+0000", 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2510000000000001e-05 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481403+0000", 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8112e-05 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.666 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.481415+0000", 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2181000000000001e-05 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.464981+0000", 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98356552500000005 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.464993+0000", 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2462e-05 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.677368+0000", 2024-02-05T07:40:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.21237463200000001 2024-02-05T07:40:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.677505+0000", 2024-02-05T07:40:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013759199999999999 2024-02-05T07:40:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-02-05T07:40:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.677667+0000", 2024-02-05T07:40:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000161559 2024-02-05T07:40:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.669 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.669 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.680176+0000", 2024-02-05T07:40:38.669 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025089969999999998 2024-02-05T07:40:38.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.669 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.669 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.669 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.680193+0000", 2024-02-05T07:40:38.669 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7385999999999999e-05 2024-02-05T07:40:38.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.669 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.669 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.669 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.682932+0000", 2024-02-05T07:40:38.669 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0027384639999999999 2024-02-05T07:40:38.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.670 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.670 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.670 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.682997+0000", 2024-02-05T07:40:38.670 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4846999999999998e-05 2024-02-05T07:40:38.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.670 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.670 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.670 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.683519+0000", 2024-02-05T07:40:38.670 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00052212199999999997 2024-02-05T07:40:38.670 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.670 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.671 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.671 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.671 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9095 11.d 11:b9aaf6db:::benchmark_data_smithi007_183880_object9094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-02-05T07:40:38.671 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.487919+0000", 2024-02-05T07:40:38.671 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 150.04970366500001, 2024-02-05T07:40:38.671 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.196086666, 2024-02-05T07:40:38.671 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.671 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.671 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.671 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:40:38.671 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:40:38.671 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9095 2024-02-05T07:40:38.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.672 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.672 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.672 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.487919+0000", 2024-02-05T07:40:38.672 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.672 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.672 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.487919+0000", 2024-02-05T07:40:38.672 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.673 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.673 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.487921+0000", 2024-02-05T07:40:38.673 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5859999999999999e-06 2024-02-05T07:40:38.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.673 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.673 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.673 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.487946+0000", 2024-02-05T07:40:38.673 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5261000000000001e-05 2024-02-05T07:40:38.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.673 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.673 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.673 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.487964+0000", 2024-02-05T07:40:38.673 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7949999999999999e-05 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.487973+0000", 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4220000000000006e-06 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.488050+0000", 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6329999999999996e-05 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.488061+0000", 2024-02-05T07:40:38.674 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1374e-05 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.465021+0000", 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97695975000000002 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.465030+0000", 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0799999999999995e-06 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.679031+0000", 2024-02-05T07:40:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.21400076600000001 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.679188+0000", 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015706499999999999 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.679381+0000", 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000193585 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.681368+0000", 2024-02-05T07:40:38.676 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0019868519999999999 2024-02-05T07:40:38.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.677 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.677 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.682353+0000", 2024-02-05T07:40:38.677 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00098529900000000003 2024-02-05T07:40:38.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.677 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.677 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.683574+0000", 2024-02-05T07:40:38.677 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012203159999999999 2024-02-05T07:40:38.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.677 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.677 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.683653+0000", 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9381000000000005e-05 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.684006+0000", 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00035264900000000002 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15846.0:9111 11.d 11:bc62cb26:::benchmark_data_smithi007_183880_object9110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-02-05T07:38:08.506846+0000", 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 150.03077690699999, 2024-02-05T07:40:38.678 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1812617329999999, 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15846", 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9111 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.506846+0000", 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.679 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-02-05T07:40:38.680 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.506846+0000", 2024-02-05T07:40:38.680 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-02-05T07:40:38.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.680 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-02-05T07:40:38.680 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.506847+0000", 2024-02-05T07:40:38.680 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.412e-06 2024-02-05T07:40:38.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.680 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-02-05T07:40:38.680 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.506865+0000", 2024-02-05T07:40:38.680 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7858000000000001e-05 2024-02-05T07:40:38.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.506876+0000", 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0944999999999999e-05 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.506882+0000", 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3090000000000001e-06 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.506941+0000", 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8900999999999997e-05 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.681 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:08.506948+0000", 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9609999999999999e-06 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.465056+0000", 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.95810786299999995 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.465064+0000", 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1210000000000007e-06 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.682 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.680219+0000", 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.21515505100000001 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.680368+0000", 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014931699999999999 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.680546+0000", 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000177178 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.683 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-02-05T07:40:38.684 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.682360+0000", 2024-02-05T07:40:38.684 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001814635 2024-02-05T07:40:38.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.684 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.684 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.684 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.682392+0000", 2024-02-05T07:40:38.684 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1612999999999997e-05 2024-02-05T07:40:38.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.684 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.684 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:40:38.684 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.687495+0000", 2024-02-05T07:40:38.684 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0051035730000000001 2024-02-05T07:40:38.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.684 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.685 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-02-05T07:40:38.685 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.687555+0000", 2024-02-05T07:40:38.685 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9364000000000003e-05 2024-02-05T07:40:38.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:40:38.685 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:40:38.685 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-02-05T07:40:38.685 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-02-05T07:38:09.688107+0000", 2024-02-05T07:40:38.685 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00055263199999999995 2024-02-05T07:40:38.685 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.685 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.685 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.685 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:40:38.685 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:40:38.685 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:40:38.710 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:38.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:38.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:38.812 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:38.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:38 smithi007 ceph-mon[37158]: pgmap v1214: 173 pgs: 1 peering, 172 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-02-05T07:40:38.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:38 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-05T07:40:38.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:38 smithi007 ceph-mon[37158]: osdmap e663: 8 total, 8 up, 5 in 2024-02-05T07:40:38.840 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:38 smithi007 ceph-mon[29098]: pgmap v1214: 173 pgs: 1 peering, 172 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-02-05T07:40:38.840 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:38 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-05T07:40:38.840 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:38 smithi007 ceph-mon[29098]: osdmap e663: 8 total, 8 up, 5 in 2024-02-05T07:40:38.873 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:38.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:38.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:38.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:39.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:39.001 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:39.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:39.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:38 smithi090 ceph-mon[33213]: pgmap v1214: 173 pgs: 1 peering, 172 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-02-05T07:40:39.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:38 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-05T07:40:39.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:38 smithi090 ceph-mon[33213]: osdmap e663: 8 total, 8 up, 5 in 2024-02-05T07:40:39.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:39.144 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:39.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:39.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:39.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:39.270 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:39.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:39.371 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:39.412 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:39.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:39.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:39.552 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:39.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:39.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:39.678 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:39.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:39.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:39.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:39.818 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:39.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:39.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:39.946 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:39.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:40.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:39 smithi090 ceph-mon[33213]: osdmap e664: 8 total, 8 up, 5 in 2024-02-05T07:40:40.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:40.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:40.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:40.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:39 smithi007 ceph-mon[29098]: osdmap e664: 8 total, 8 up, 5 in 2024-02-05T07:40:40.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:39 smithi007 ceph-mon[37158]: osdmap e664: 8 total, 8 up, 5 in 2024-02-05T07:40:40.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:40.199 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:40.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:40.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:40.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:40.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:40.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:40.451 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:40.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:40.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:40.553 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:40.578 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:40.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:40.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:40.704 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:40.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:40.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:40.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:40.845 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:40.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:40.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:40.971 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:40.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:41.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:40 smithi090 ceph-mon[33213]: pgmap v1217: 173 pgs: 1 peering, 172 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-02-05T07:40:41.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:40 smithi090 ceph-mon[33213]: osdmap e665: 8 total, 8 up, 5 in 2024-02-05T07:40:41.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:40 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:40:41.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:40 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:40:41.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:41.097 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:41.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:41.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:40 smithi007 ceph-mon[29098]: pgmap v1217: 173 pgs: 1 peering, 172 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-02-05T07:40:41.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:40 smithi007 ceph-mon[29098]: osdmap e665: 8 total, 8 up, 5 in 2024-02-05T07:40:41.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:40 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:40:41.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:40 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:40:41.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:40 smithi007 ceph-mon[37158]: pgmap v1217: 173 pgs: 1 peering, 172 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-02-05T07:40:41.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:40 smithi007 ceph-mon[37158]: osdmap e665: 8 total, 8 up, 5 in 2024-02-05T07:40:41.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:40 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:40:41.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:40 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:40:41.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:41.224 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:41.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:41.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:41.350 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:41.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:41.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:41.433 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T07:40:41.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:41.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:40:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:40:41.525 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:41.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:41.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:41.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:41.652 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:41.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:41.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:41.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:41.811 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:41.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:41.913 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:41.959 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:41.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:41.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:42.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:42.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:42.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:42.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:41 smithi007 ceph-mon[29098]: osdmap e666: 8 total, 8 up, 5 in 2024-02-05T07:40:42.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:41 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-05T07:40:42.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:41 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-05T07:40:42.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:41 smithi007 ceph-mon[37158]: osdmap e666: 8 total, 8 up, 5 in 2024-02-05T07:40:42.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:41 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-05T07:40:42.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:41 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-05T07:40:42.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:42.210 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:42.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:42.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:42.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:41 smithi090 ceph-mon[33213]: osdmap e666: 8 total, 8 up, 5 in 2024-02-05T07:40:42.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:41 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-05T07:40:42.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:41 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-05T07:40:42.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:42.337 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:42.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:42.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:42.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:42.465 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:42.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:42.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:42.607 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:42.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:42.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:42.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:42.734 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:42.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:42.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:42.861 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:42.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:42.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:42.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:42.988 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:42.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:43.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:42 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:43.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:42 smithi007 ceph-mon[29098]: pgmap v1220: 172 pgs: 172 active+clean; 2.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 969 op/s 2024-02-05T07:40:43.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:42 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-05T07:40:43.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:42 smithi007 ceph-mon[29098]: osdmap e667: 8 total, 8 up, 5 in 2024-02-05T07:40:43.029 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:42 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:43.030 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:42 smithi007 ceph-mon[37158]: pgmap v1220: 172 pgs: 172 active+clean; 2.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 969 op/s 2024-02-05T07:40:43.030 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:42 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-05T07:40:43.030 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:42 smithi007 ceph-mon[37158]: osdmap e667: 8 total, 8 up, 5 in 2024-02-05T07:40:43.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:43.114 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:43.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:43.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:43.173 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:40:43.173 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:40:43.173 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:40:43.173 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:40:43.173 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:40:43.212 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:40:43.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:43.249 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:40:43.270 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:43.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:43.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:42 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:43.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:42 smithi090 ceph-mon[33213]: pgmap v1220: 172 pgs: 172 active+clean; 2.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 969 op/s 2024-02-05T07:40:43.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:42 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-05T07:40:43.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:42 smithi090 ceph-mon[33213]: osdmap e667: 8 total, 8 up, 5 in 2024-02-05T07:40:43.296 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:40:43.305 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:40:43.305 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:40:43.305 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:40:43.305 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:40:43.305 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:40:43.305 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:40:43.305 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:40:43.305 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:40:43.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:43.396 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:43.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:43.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:43.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:43.536 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:43.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:43.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:43.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:43.663 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:43.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:43.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:40:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:40:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:40:43.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:43.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:43.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:43.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:43.950 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:43.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:43.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:44.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:44.077 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:44.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:44.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:43 smithi007 ceph-mon[29098]: osdmap e668: 8 total, 8 up, 5 in 2024-02-05T07:40:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:43 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1381116053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:40:44.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:43 smithi007 ceph-mon[37158]: osdmap e668: 8 total, 8 up, 5 in 2024-02-05T07:40:44.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:43 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1381116053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:40:44.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:44.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:44.206 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:44.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:44.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:43 smithi090 ceph-mon[33213]: osdmap e668: 8 total, 8 up, 5 in 2024-02-05T07:40:44.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:43 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1381116053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:40:44.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:44.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:44.360 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:44.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:44.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:44.486 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:44.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:44.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:44.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:44.611 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:44.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:44.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:44.713 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:44.737 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:44.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:44.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:44.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:44.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:44.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:44.989 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:44.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:45.050 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:44 smithi007 ceph-mon[29098]: pgmap v1223: 172 pgs: 172 active+clean; 2.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T07:40:45.050 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:44 smithi007 ceph-mon[29098]: osdmap e669: 8 total, 8 up, 5 in 2024-02-05T07:40:45.050 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:44 smithi007 ceph-mon[29098]: osdmap e670: 8 total, 8 up, 5 in 2024-02-05T07:40:45.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:44 smithi007 ceph-mon[37158]: pgmap v1223: 172 pgs: 172 active+clean; 2.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T07:40:45.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:44 smithi007 ceph-mon[37158]: osdmap e669: 8 total, 8 up, 5 in 2024-02-05T07:40:45.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:44 smithi007 ceph-mon[37158]: osdmap e670: 8 total, 8 up, 5 in 2024-02-05T07:40:45.091 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:45.119 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:45.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:45.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:45.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:44 smithi090 ceph-mon[33213]: pgmap v1223: 172 pgs: 172 active+clean; 2.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T07:40:45.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:44 smithi090 ceph-mon[33213]: osdmap e669: 8 total, 8 up, 5 in 2024-02-05T07:40:45.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:44 smithi090 ceph-mon[33213]: osdmap e670: 8 total, 8 up, 5 in 2024-02-05T07:40:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:45.276 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:45.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:45.377 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:45.407 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:45.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:45.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:45.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:45.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:45.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:45.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:45.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:45.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:45.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:45.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:46.023 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:46.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:46.123 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:45 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:40:46.123 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:45 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-05T07:40:46.123 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:45 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-05T07:40:46.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:46.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:45 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:40:46.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:45 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-05T07:40:46.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:45 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-05T07:40:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:46.171 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:46.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:46.234 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:46.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:46.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:45 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:40:46.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:45 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-05T07:40:46.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:45 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-05T07:40:46.317 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:46.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:46.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:46.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:46.445 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:46.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:46.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:46.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:46.590 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:46.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:46.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:46.715 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:46.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:46.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:46.817 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:46.840 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-02-05T07:40:46.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:46.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:46.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:46.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:46.986 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:46.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:46.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:47.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[29098]: pgmap v1226: 171 pgs: 171 active+clean; 1.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 862 op/s 2024-02-05T07:40:47.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-05T07:40:47.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[29098]: osdmap e671: 8 total, 8 up, 5 in 2024-02-05T07:40:47.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:40:47.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:40:47.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:40:47.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:40:47.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/787425405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:40:47.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:40:47.040 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[37158]: pgmap v1226: 171 pgs: 171 active+clean; 1.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 862 op/s 2024-02-05T07:40:47.040 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-05T07:40:47.041 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[37158]: osdmap e671: 8 total, 8 up, 5 in 2024-02-05T07:40:47.041 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:40:47.041 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:40:47.041 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:40:47.041 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:40:47.041 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/787425405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:40:47.041 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:46 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:40:47.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:47.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:40:47.130 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:47.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:47.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:47.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:46 smithi090 ceph-mon[33213]: pgmap v1226: 171 pgs: 171 active+clean; 1.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 862 op/s 2024-02-05T07:40:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:46 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-05T07:40:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:46 smithi090 ceph-mon[33213]: osdmap e671: 8 total, 8 up, 5 in 2024-02-05T07:40:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:40:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:40:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:46 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:40:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:40:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:46 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/787425405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:40:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:46 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:40:47.278 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:47.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:47.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:47.381 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:47.405 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:47.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:47.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:47.533 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:47.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:47.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:47.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:47.660 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:47.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:47.761 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:47.807 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:47.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:47.820 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:40:47.836 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:40:47.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:47.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:47 smithi007 ceph-mon[29098]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:40:47.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:47 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:40:47.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:47 smithi007 ceph-mon[29098]: osdmap e672: 8 total, 8 up, 5 in 2024-02-05T07:40:47.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:47 smithi007 ceph-mon[37158]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:40:47.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:47 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:40:47.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:47 smithi007 ceph-mon[37158]: osdmap e672: 8 total, 8 up, 5 in 2024-02-05T07:40:47.934 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:47.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:47.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:48.036 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:48.077 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:48.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:48.178 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:48.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:48.205 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:48.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:48.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:47 smithi090 ceph-mon[33213]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:40:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:47 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T07:40:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:47 smithi090 ceph-mon[33213]: osdmap e672: 8 total, 8 up, 5 in 2024-02-05T07:40:48.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:48.347 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:48.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:48.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:48.491 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:48.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:48.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:48.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:48.618 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:48.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:48.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:48.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:48.764 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:48.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:48.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:48.866 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:48.871 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:48 smithi007 ceph-mon[37158]: pgmap v1229: 171 pgs: 171 active+clean; 1.6 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-02-05T07:40:48.871 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:48 smithi007 ceph-mon[37158]: from='client.15948 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:40:48.871 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:48 smithi007 ceph-mon[37158]: osdmap e673: 8 total, 8 up, 5 in 2024-02-05T07:40:48.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:48 smithi007 ceph-mon[29098]: pgmap v1229: 171 pgs: 171 active+clean; 1.6 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-02-05T07:40:48.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:48 smithi007 ceph-mon[29098]: from='client.15948 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:40:48.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:48 smithi007 ceph-mon[29098]: osdmap e673: 8 total, 8 up, 5 in 2024-02-05T07:40:48.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:48.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:49.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:49.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:49.064 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:49.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:49.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:49.207 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:49.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:49.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:48 smithi090 ceph-mon[33213]: pgmap v1229: 171 pgs: 171 active+clean; 1.6 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-02-05T07:40:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:48 smithi090 ceph-mon[33213]: from='client.15948 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:40:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:48 smithi090 ceph-mon[33213]: osdmap e673: 8 total, 8 up, 5 in 2024-02-05T07:40:49.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:49.351 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:49.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:49.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:49.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:49.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:49.501 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:49.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:49.526 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1229,"stamp":"2024-02-05T07:40:47.441479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1741756788,"num_objects":26767,"num_object_clones":0,"num_object_copies":80301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26767,"num_whiteouts":0,"num_read":2136,"num_read_kb":1888,"num_write":194997,"num_write_kb":7057838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2799,"num_bytes_recovered":150681091,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48915,"ondisk_log_size":48915,"up":513,"acting":513,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":516,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":10217452,"kb_used_data":5210068,"kb_used_omap":11,"kb_used_meta":5006644,"kb_avail":458508308,"statfs":{"total":479975178240,"available":469512507392,"internally_reserved":0,"allocated":5335109632,"data_stored":5327931367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12005,"internal_metadata":5126803739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-387645440,"num_objects":-5915,"num_object_clones":0,"num_object_copies":-17745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5915,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5915,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001253"},"pg_stats":[{"pgid":"11.3","version":"668'11899","reported_seq":12090,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.827910+0000","last_change":"2024-02-05T07:37:52.811299+0000","last_active":"2024-02-05T07:40:43.827910+0000","last_peered":"2024-02-05T07:40:43.827910+0000","last_clean":"2024-02-05T07:40:43.827910+0000","last_became_active":"2024-02-05T07:37:52.811051+0000","last_became_peered":"2024-02-05T07:37:52.811051+0000","last_unstale":"2024-02-05T07:40:43.827910+0000","last_undegraded":"2024-02-05T07:40:43.827910+0000","last_fullsized":"2024-02-05T07:40:43.827910+0000","mapping_epoch":579,"log_start":"636'9020","ondisk_log_start":"636'9020","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2879,"log_dups_size":3000,"ondisk_log_size":2879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118816768,"num_objects":1813,"num_object_clones":0,"num_object_copies":5439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11899,"num_write_kb":438784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":1489,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.832628+0000","last_change":"2024-02-05T07:40:40.778976+0000","last_active":"2024-02-05T07:40:46.832628+0000","last_peered":"2024-02-05T07:40:46.832628+0000","last_clean":"2024-02-05T07:40:46.832628+0000","last_became_active":"2024-02-05T07:40:40.778437+0000","last_became_peered":"2024-02-05T07:40:40.778437+0000","last_unstale":"2024-02-05T07:40:46.832628+0000","last_undegraded":"2024-02-05T07:40:46.832628+0000","last_fullsized":"2024-02-05T07:40:46.832628+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.337196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.337196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"615'35","reported_seq":832,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.835101+0000","last_change":"2024-02-05T07:38:09.523613+0000","last_active":"2024-02-05T07:40:46.835101+0000","last_peered":"2024-02-05T07:40:46.835101+0000","last_clean":"2024-02-05T07:40:46.835101+0000","last_became_active":"2024-02-05T07:38:09.493734+0000","last_became_peered":"2024-02-05T07:38:09.493734+0000","last_unstale":"2024-02-05T07:40:46.835101+0000","last_undegraded":"2024-02-05T07:40:46.835101+0000","last_fullsized":"2024-02-05T07:40:46.835101+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:37:02.567716+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:37:02.567716+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":854,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.845940+0000","last_change":"2024-02-05T07:38:37.381786+0000","last_active":"2024-02-05T07:40:46.845940+0000","last_peered":"2024-02-05T07:40:46.845940+0000","last_clean":"2024-02-05T07:40:46.845940+0000","last_became_active":"2024-02-05T07:38:09.496706+0000","last_became_peered":"2024-02-05T07:38:09.496706+0000","last_unstale":"2024-02-05T07:40:46.845940+0000","last_undegraded":"2024-02-05T07:40:46.845940+0000","last_fullsized":"2024-02-05T07:40:46.845940+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:37.381317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:37.381317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"538'13","reported_seq":343,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710977+0000","last_change":"2024-02-05T07:39:03.630111+0000","last_active":"2024-02-05T07:40:44.710977+0000","last_peered":"2024-02-05T07:40:44.710977+0000","last_clean":"2024-02-05T07:40:44.710977+0000","last_became_active":"2024-02-05T07:36:32.743613+0000","last_became_peered":"2024-02-05T07:36:32.743613+0000","last_unstale":"2024-02-05T07:40:44.710977+0000","last_undegraded":"2024-02-05T07:40:44.710977+0000","last_fullsized":"2024-02-05T07:40:44.710977+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'13","last_scrub_stamp":"2024-02-05T07:39:03.629988+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:39:03.629988+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.624333+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1158,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.713559+0000","last_change":"2024-02-05T07:39:53.910643+0000","last_active":"2024-02-05T07:40:44.713559+0000","last_peered":"2024-02-05T07:40:44.713559+0000","last_clean":"2024-02-05T07:40:44.713559+0000","last_became_active":"2024-02-05T07:38:09.495645+0000","last_became_peered":"2024-02-05T07:38:09.495645+0000","last_unstale":"2024-02-05T07:40:44.713559+0000","last_undegraded":"2024-02-05T07:40:44.713559+0000","last_fullsized":"2024-02-05T07:40:44.713559+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:53.910541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:39:53.910541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:00.438016+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2","version":"670'12092","reported_seq":12285,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:45.834884+0000","last_change":"2024-02-05T07:37:53.089553+0000","last_active":"2024-02-05T07:40:45.834884+0000","last_peered":"2024-02-05T07:40:45.834884+0000","last_clean":"2024-02-05T07:40:45.834884+0000","last_became_active":"2024-02-05T07:37:53.089215+0000","last_became_peered":"2024-02-05T07:37:53.089215+0000","last_unstale":"2024-02-05T07:40:45.834884+0000","last_undegraded":"2024-02-05T07:40:45.834884+0000","last_fullsized":"2024-02-05T07:40:45.834884+0000","mapping_epoch":579,"log_start":"636'8736","ondisk_log_start":"636'8736","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":3356,"log_dups_size":0,"ondisk_log_size":3356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.618793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108003328,"num_objects":1648,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12092,"num_write_kb":439680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":786,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.724483+0000","last_change":"2024-02-05T07:40:44.724483+0000","last_active":"2024-02-05T07:40:44.724483+0000","last_peered":"2024-02-05T07:40:44.724483+0000","last_clean":"2024-02-05T07:40:44.724483+0000","last_became_active":"2024-02-05T07:40:44.724105+0000","last_became_peered":"2024-02-05T07:40:44.724105+0000","last_unstale":"2024-02-05T07:40:44.724483+0000","last_undegraded":"2024-02-05T07:40:44.724483+0000","last_fullsized":"2024-02-05T07:40:44.724483+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":670,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.389984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.389984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":138,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"616'45","reported_seq":1273,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.711030+0000","last_change":"2024-02-05T07:38:09.530860+0000","last_active":"2024-02-05T07:40:44.711030+0000","last_peered":"2024-02-05T07:40:44.711030+0000","last_clean":"2024-02-05T07:40:44.711030+0000","last_became_active":"2024-02-05T07:38:09.503330+0000","last_became_peered":"2024-02-05T07:38:09.503330+0000","last_unstale":"2024-02-05T07:40:44.711030+0000","last_undegraded":"2024-02-05T07:40:44.711030+0000","last_fullsized":"2024-02-05T07:40:44.711030+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'31","last_scrub_stamp":"2024-02-05T07:37:54.045831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.045831+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.621664+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1120,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821122+0000","last_change":"2024-02-05T07:38:09.471668+0000","last_active":"2024-02-05T07:40:43.821122+0000","last_peered":"2024-02-05T07:40:43.821122+0000","last_clean":"2024-02-05T07:40:43.821122+0000","last_became_active":"2024-02-05T07:38:09.471049+0000","last_became_peered":"2024-02-05T07:38:09.471049+0000","last_unstale":"2024-02-05T07:40:43.821122+0000","last_undegraded":"2024-02-05T07:40:43.821122+0000","last_fullsized":"2024-02-05T07:40:43.821122+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:46.657168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:37:46.657168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":180,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821326+0000","last_change":"2024-02-05T07:39:59.662598+0000","last_active":"2024-02-05T07:40:43.821326+0000","last_peered":"2024-02-05T07:40:43.821326+0000","last_clean":"2024-02-05T07:40:43.821326+0000","last_became_active":"2024-02-05T07:38:09.464116+0000","last_became_peered":"2024-02-05T07:38:09.464116+0000","last_unstale":"2024-02-05T07:40:43.821326+0000","last_undegraded":"2024-02-05T07:40:43.821326+0000","last_fullsized":"2024-02-05T07:40:43.821326+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:59.662530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:59.662530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:25.420020+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":294,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.714422+0000","last_change":"2024-02-05T07:38:55.897373+0000","last_active":"2024-02-05T07:40:44.714422+0000","last_peered":"2024-02-05T07:40:44.714422+0000","last_clean":"2024-02-05T07:40:44.714422+0000","last_became_active":"2024-02-05T07:38:09.482289+0000","last_became_peered":"2024-02-05T07:38:09.482289+0000","last_unstale":"2024-02-05T07:40:44.714422+0000","last_undegraded":"2024-02-05T07:40:44.714422+0000","last_fullsized":"2024-02-05T07:40:44.714422+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:55.897133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.897133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.624386+0000","scrub_duration":294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"669'11913","reported_seq":12104,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.820773+0000","last_change":"2024-02-05T07:37:53.089618+0000","last_active":"2024-02-05T07:40:43.820773+0000","last_peered":"2024-02-05T07:40:43.820773+0000","last_clean":"2024-02-05T07:40:43.820773+0000","last_became_active":"2024-02-05T07:37:53.089261+0000","last_became_peered":"2024-02-05T07:37:53.089261+0000","last_unstale":"2024-02-05T07:40:43.820773+0000","last_undegraded":"2024-02-05T07:40:43.820773+0000","last_fullsized":"2024-02-05T07:40:43.820773+0000","mapping_epoch":579,"log_start":"636'9031","ondisk_log_start":"636'9031","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2882,"log_dups_size":3000,"ondisk_log_size":2882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112918528,"num_objects":1723,"num_object_clones":0,"num_object_copies":5169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11913,"num_write_kb":436352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":1073,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.712792+0000","last_change":"2024-02-05T07:40:33.082486+0000","last_active":"2024-02-05T07:40:44.712792+0000","last_peered":"2024-02-05T07:40:44.712792+0000","last_clean":"2024-02-05T07:40:44.712792+0000","last_became_active":"2024-02-05T07:40:33.082113+0000","last_became_peered":"2024-02-05T07:40:33.082113+0000","last_unstale":"2024-02-05T07:40:44.712792+0000","last_undegraded":"2024-02-05T07:40:44.712792+0000","last_fullsized":"2024-02-05T07:40:44.712792+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.417059+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.417059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"616'32","reported_seq":798,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821691+0000","last_change":"2024-02-05T07:38:36.705553+0000","last_active":"2024-02-05T07:40:43.821691+0000","last_peered":"2024-02-05T07:40:43.821691+0000","last_clean":"2024-02-05T07:40:43.821691+0000","last_became_active":"2024-02-05T07:38:09.471120+0000","last_became_peered":"2024-02-05T07:38:09.471120+0000","last_unstale":"2024-02-05T07:40:43.821691+0000","last_undegraded":"2024-02-05T07:40:43.821691+0000","last_fullsized":"2024-02-05T07:40:43.821691+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:38:36.705434+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:38:36.705434+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770859+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1170,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.705801+0000","last_change":"2024-02-05T07:38:09.490148+0000","last_active":"2024-02-05T07:40:44.705801+0000","last_peered":"2024-02-05T07:40:44.705801+0000","last_clean":"2024-02-05T07:40:44.705801+0000","last_became_active":"2024-02-05T07:38:09.489807+0000","last_became_peered":"2024-02-05T07:38:09.489807+0000","last_unstale":"2024-02-05T07:40:44.705801+0000","last_undegraded":"2024-02-05T07:40:44.705801+0000","last_fullsized":"2024-02-05T07:40:44.705801+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:03.618665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:37:03.618665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.618545+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1004,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.822680+0000","last_change":"2024-02-05T07:38:37.703938+0000","last_active":"2024-02-05T07:40:43.822680+0000","last_peered":"2024-02-05T07:40:43.822680+0000","last_clean":"2024-02-05T07:40:43.822680+0000","last_became_active":"2024-02-05T07:38:09.472546+0000","last_became_peered":"2024-02-05T07:38:09.472546+0000","last_unstale":"2024-02-05T07:40:43.822680+0000","last_undegraded":"2024-02-05T07:40:43.822680+0000","last_fullsized":"2024-02-05T07:40:43.822680+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:37.703824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:38:37.703824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770712+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":857,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.832600+0000","last_change":"2024-02-05T07:37:52.481462+0000","last_active":"2024-02-05T07:40:46.832600+0000","last_peered":"2024-02-05T07:40:46.832600+0000","last_clean":"2024-02-05T07:40:46.832600+0000","last_became_active":"2024-02-05T07:30:40.498529+0000","last_became_peered":"2024-02-05T07:30:40.498529+0000","last_unstale":"2024-02-05T07:40:46.832600+0000","last_undegraded":"2024-02-05T07:40:46.832600+0000","last_fullsized":"2024-02-05T07:40:46.832600+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:52.481255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:52.481255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"672'12023","reported_seq":12220,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:47.224088+0000","last_change":"2024-02-05T07:37:52.817253+0000","last_active":"2024-02-05T07:40:47.224088+0000","last_peered":"2024-02-05T07:40:47.224088+0000","last_clean":"2024-02-05T07:40:47.224088+0000","last_became_active":"2024-02-05T07:37:52.816932+0000","last_became_peered":"2024-02-05T07:37:52.816932+0000","last_unstale":"2024-02-05T07:40:47.224088+0000","last_undegraded":"2024-02-05T07:40:47.224088+0000","last_fullsized":"2024-02-05T07:40:47.224088+0000","mapping_epoch":579,"log_start":"636'9100","ondisk_log_start":"636'9100","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2923,"log_dups_size":3000,"ondisk_log_size":2923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101253120,"num_objects":1545,"num_object_clones":0,"num_object_copies":4635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12023,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":669,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710865+0000","last_change":"2024-02-05T07:40:36.746220+0000","last_active":"2024-02-05T07:40:44.710865+0000","last_peered":"2024-02-05T07:40:44.710865+0000","last_clean":"2024-02-05T07:40:44.710865+0000","last_became_active":"2024-02-05T07:40:36.745864+0000","last_became_peered":"2024-02-05T07:40:36.745864+0000","last_unstale":"2024-02-05T07:40:44.710865+0000","last_undegraded":"2024-02-05T07:40:44.710865+0000","last_fullsized":"2024-02-05T07:40:44.710865+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:40.411931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:40.411931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"616'46","reported_seq":1402,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.720955+0000","last_change":"2024-02-05T07:38:52.880432+0000","last_active":"2024-02-05T07:40:44.720955+0000","last_peered":"2024-02-05T07:40:44.720955+0000","last_clean":"2024-02-05T07:40:44.720955+0000","last_became_active":"2024-02-05T07:38:09.500221+0000","last_became_peered":"2024-02-05T07:38:09.500221+0000","last_unstale":"2024-02-05T07:40:44.720955+0000","last_undegraded":"2024-02-05T07:40:44.720955+0000","last_fullsized":"2024-02-05T07:40:44.720955+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'34","last_scrub_stamp":"2024-02-05T07:38:52.880291+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:38:52.880291+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.376668+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":861,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.848145+0000","last_change":"2024-02-05T07:38:09.485474+0000","last_active":"2024-02-05T07:40:46.848145+0000","last_peered":"2024-02-05T07:40:46.848145+0000","last_clean":"2024-02-05T07:40:46.848145+0000","last_became_active":"2024-02-05T07:38:09.485164+0000","last_became_peered":"2024-02-05T07:38:09.485164+0000","last_unstale":"2024-02-05T07:40:46.848145+0000","last_undegraded":"2024-02-05T07:40:46.848145+0000","last_fullsized":"2024-02-05T07:40:46.848145+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:47.116727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:47.116727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"589'12","reported_seq":1426,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.711492+0000","last_change":"2024-02-05T07:38:09.514237+0000","last_active":"2024-02-05T07:40:44.711492+0000","last_peered":"2024-02-05T07:40:44.711492+0000","last_clean":"2024-02-05T07:40:44.711492+0000","last_became_active":"2024-02-05T07:38:09.488856+0000","last_became_peered":"2024-02-05T07:38:09.488856+0000","last_unstale":"2024-02-05T07:40:44.711492+0000","last_undegraded":"2024-02-05T07:40:44.711492+0000","last_fullsized":"2024-02-05T07:40:44.711492+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'11","last_scrub_stamp":"2024-02-05T07:36:59.523421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.523421+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.618577+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":288,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710829+0000","last_change":"2024-02-05T07:39:50.551862+0000","last_active":"2024-02-05T07:40:44.710829+0000","last_peered":"2024-02-05T07:40:44.710829+0000","last_clean":"2024-02-05T07:40:44.710829+0000","last_became_active":"2024-02-05T07:36:32.744070+0000","last_became_peered":"2024-02-05T07:36:32.744070+0000","last_unstale":"2024-02-05T07:40:44.710829+0000","last_undegraded":"2024-02-05T07:40:44.710829+0000","last_fullsized":"2024-02-05T07:40:44.710829+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:50.551740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:50.551740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:06.027327+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"670'12065","reported_seq":12258,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:45.791789+0000","last_change":"2024-02-05T07:37:53.089610+0000","last_active":"2024-02-05T07:40:45.791789+0000","last_peered":"2024-02-05T07:40:45.791789+0000","last_clean":"2024-02-05T07:40:45.791789+0000","last_became_active":"2024-02-05T07:37:53.089296+0000","last_became_peered":"2024-02-05T07:37:53.089296+0000","last_unstale":"2024-02-05T07:40:45.791789+0000","last_undegraded":"2024-02-05T07:40:45.791789+0000","last_fullsized":"2024-02-05T07:40:45.791789+0000","mapping_epoch":579,"log_start":"635'8772","ondisk_log_start":"635'8772","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":3293,"log_dups_size":0,"ondisk_log_size":3293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.621556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112263168,"num_objects":1713,"num_object_clones":0,"num_object_copies":5139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12065,"num_write_kb":440896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":1253,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710349+0000","last_change":"2024-02-05T07:38:34.512785+0000","last_active":"2024-02-05T07:40:44.710349+0000","last_peered":"2024-02-05T07:40:44.710349+0000","last_clean":"2024-02-05T07:40:44.710349+0000","last_became_active":"2024-02-05T07:38:13.714462+0000","last_became_peered":"2024-02-05T07:38:13.714462+0000","last_unstale":"2024-02-05T07:40:44.710349+0000","last_undegraded":"2024-02-05T07:40:44.710349+0000","last_fullsized":"2024-02-05T07:40:44.710349+0000","mapping_epoch":593,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:34.509015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:34.509015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":813,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.832067+0000","last_change":"2024-02-05T07:38:09.479255+0000","last_active":"2024-02-05T07:40:46.832067+0000","last_peered":"2024-02-05T07:40:46.832067+0000","last_clean":"2024-02-05T07:40:46.832067+0000","last_became_active":"2024-02-05T07:38:09.478485+0000","last_became_peered":"2024-02-05T07:38:09.478485+0000","last_unstale":"2024-02-05T07:40:46.832067+0000","last_undegraded":"2024-02-05T07:40:46.832067+0000","last_fullsized":"2024-02-05T07:40:46.832067+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.155552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.155552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"615'50","reported_seq":943,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.848036+0000","last_change":"2024-02-05T07:39:48.737503+0000","last_active":"2024-02-05T07:40:46.848036+0000","last_peered":"2024-02-05T07:40:46.848036+0000","last_clean":"2024-02-05T07:40:46.848036+0000","last_became_active":"2024-02-05T07:30:40.496940+0000","last_became_peered":"2024-02-05T07:30:40.496940+0000","last_unstale":"2024-02-05T07:40:46.848036+0000","last_undegraded":"2024-02-05T07:40:46.848036+0000","last_fullsized":"2024-02-05T07:40:46.848036+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'50","last_scrub_stamp":"2024-02-05T07:39:48.737259+0000","last_deep_scrub":"454'38","last_deep_scrub_stamp":"2024-02-05T07:34:38.942830+0000","last_clean_scrub_stamp":"2024-02-05T07:39:48.737259+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:07.832109+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":861,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.834693+0000","last_change":"2024-02-05T07:38:57.455422+0000","last_active":"2024-02-05T07:40:46.834693+0000","last_peered":"2024-02-05T07:40:46.834693+0000","last_clean":"2024-02-05T07:40:46.834693+0000","last_became_active":"2024-02-05T07:38:09.488092+0000","last_became_peered":"2024-02-05T07:38:09.488092+0000","last_unstale":"2024-02-05T07:40:46.834693+0000","last_undegraded":"2024-02-05T07:40:46.834693+0000","last_fullsized":"2024-02-05T07:40:46.834693+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:57.455281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:57.455281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1171,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.845495+0000","last_change":"2024-02-05T07:38:09.480292+0000","last_active":"2024-02-05T07:40:46.845495+0000","last_peered":"2024-02-05T07:40:46.845495+0000","last_clean":"2024-02-05T07:40:46.845495+0000","last_became_active":"2024-02-05T07:38:09.479403+0000","last_became_peered":"2024-02-05T07:38:09.479403+0000","last_unstale":"2024-02-05T07:40:46.845495+0000","last_undegraded":"2024-02-05T07:40:46.845495+0000","last_fullsized":"2024-02-05T07:40:46.845495+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:01.252507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:38:01.252507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"670'12169","reported_seq":13618,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:45.633414+0000","last_change":"2024-02-05T07:39:50.484334+0000","last_active":"2024-02-05T07:40:45.633414+0000","last_peered":"2024-02-05T07:40:45.633414+0000","last_clean":"2024-02-05T07:40:45.633414+0000","last_became_active":"2024-02-05T07:38:16.656778+0000","last_became_peered":"2024-02-05T07:38:16.656778+0000","last_unstale":"2024-02-05T07:40:45.633414+0000","last_undegraded":"2024-02-05T07:40:45.633414+0000","last_fullsized":"2024-02-05T07:40:45.633414+0000","mapping_epoch":596,"log_start":"637'9400","ondisk_log_start":"637'9400","created":579,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"634'8172","last_scrub_stamp":"2024-02-05T07:39:50.484069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:39:50.484069+0000","objects_scrubbed":5766,"log_size":2769,"log_dups_size":3000,"ondisk_log_size":2769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:09.764020+0000","scrub_duration":2626,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108855296,"num_objects":1661,"num_object_clones":0,"num_object_copies":4983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12169,"num_write_kb":442560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":589,"num_bytes_recovered":38600704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":1623,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.832770+0000","last_change":"2024-02-05T07:38:11.676831+0000","last_active":"2024-02-05T07:40:46.832770+0000","last_peered":"2024-02-05T07:40:46.832770+0000","last_clean":"2024-02-05T07:40:46.832770+0000","last_became_active":"2024-02-05T07:38:11.676482+0000","last_became_peered":"2024-02-05T07:38:11.676482+0000","last_unstale":"2024-02-05T07:40:46.832770+0000","last_undegraded":"2024-02-05T07:40:46.832770+0000","last_fullsized":"2024-02-05T07:40:46.832770+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:07.353770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:07.353770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"589'11","reported_seq":1026,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.848198+0000","last_change":"2024-02-05T07:38:09.522356+0000","last_active":"2024-02-05T07:40:46.848198+0000","last_peered":"2024-02-05T07:40:46.848198+0000","last_clean":"2024-02-05T07:40:46.848198+0000","last_became_active":"2024-02-05T07:38:09.495697+0000","last_became_peered":"2024-02-05T07:38:09.495697+0000","last_unstale":"2024-02-05T07:40:46.848198+0000","last_undegraded":"2024-02-05T07:40:46.848198+0000","last_fullsized":"2024-02-05T07:40:46.848198+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'10","last_scrub_stamp":"2024-02-05T07:37:59.362892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:59.362892+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"616'49","reported_seq":836,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821284+0000","last_change":"2024-02-05T07:39:55.713122+0000","last_active":"2024-02-05T07:40:43.821284+0000","last_peered":"2024-02-05T07:40:43.821284+0000","last_clean":"2024-02-05T07:40:43.821284+0000","last_became_active":"2024-02-05T07:38:09.480688+0000","last_became_peered":"2024-02-05T07:38:09.480688+0000","last_unstale":"2024-02-05T07:40:43.821284+0000","last_undegraded":"2024-02-05T07:40:43.821284+0000","last_fullsized":"2024-02-05T07:40:43.821284+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'49","last_scrub_stamp":"2024-02-05T07:39:55.713036+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:39:55.713036+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:05.586770+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"639'2","reported_seq":1044,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.713325+0000","last_change":"2024-02-05T07:40:13.213597+0000","last_active":"2024-02-05T07:40:44.713325+0000","last_peered":"2024-02-05T07:40:44.713325+0000","last_clean":"2024-02-05T07:40:44.713325+0000","last_became_active":"2024-02-05T07:38:09.467535+0000","last_became_peered":"2024-02-05T07:38:09.467535+0000","last_unstale":"2024-02-05T07:40:44.713325+0000","last_undegraded":"2024-02-05T07:40:44.713325+0000","last_fullsized":"2024-02-05T07:40:44.713325+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'2","last_scrub_stamp":"2024-02-05T07:40:13.213477+0000","last_deep_scrub":"639'2","last_deep_scrub_stamp":"2024-02-05T07:40:13.213477+0000","last_clean_scrub_stamp":"2024-02-05T07:40:13.213477+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:26.617373+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1146,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821162+0000","last_change":"2024-02-05T07:37:58.737127+0000","last_active":"2024-02-05T07:40:43.821162+0000","last_peered":"2024-02-05T07:40:43.821162+0000","last_clean":"2024-02-05T07:40:43.821162+0000","last_became_active":"2024-02-05T07:36:32.701203+0000","last_became_peered":"2024-02-05T07:36:32.701203+0000","last_unstale":"2024-02-05T07:40:43.821162+0000","last_undegraded":"2024-02-05T07:40:43.821162+0000","last_fullsized":"2024-02-05T07:40:43.821162+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:58.737033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:58.737033+0000","last_clean_scrub_stamp":"2024-02-05T07:37:58.737033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770585+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"670'12251","reported_seq":13642,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:45.809825+0000","last_change":"2024-02-05T07:38:16.745449+0000","last_active":"2024-02-05T07:40:45.809825+0000","last_peered":"2024-02-05T07:40:45.809825+0000","last_clean":"2024-02-05T07:40:45.809825+0000","last_became_active":"2024-02-05T07:38:16.742262+0000","last_became_peered":"2024-02-05T07:38:16.742262+0000","last_unstale":"2024-02-05T07:40:45.809825+0000","last_undegraded":"2024-02-05T07:40:45.809825+0000","last_fullsized":"2024-02-05T07:40:45.809825+0000","mapping_epoch":596,"log_start":"636'8929","ondisk_log_start":"636'8929","created":579,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":3322,"log_dups_size":0,"ondisk_log_size":3322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.621814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107085848,"num_objects":1635,"num_object_clones":0,"num_object_copies":4905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12251,"num_write_kb":444289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":593,"num_bytes_recovered":38862848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":1240,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.822795+0000","last_change":"2024-02-05T07:37:42.825173+0000","last_active":"2024-02-05T07:40:43.822795+0000","last_peered":"2024-02-05T07:40:43.822795+0000","last_clean":"2024-02-05T07:40:43.822795+0000","last_became_active":"2024-02-05T07:37:36.688626+0000","last_became_peered":"2024-02-05T07:37:36.688626+0000","last_unstale":"2024-02-05T07:40:43.822795+0000","last_undegraded":"2024-02-05T07:40:43.822795+0000","last_fullsized":"2024-02-05T07:40:43.822795+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:42.822474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:42.822474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":1016,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821421+0000","last_change":"2024-02-05T07:39:44.702368+0000","last_active":"2024-02-05T07:40:43.821421+0000","last_peered":"2024-02-05T07:40:43.821421+0000","last_clean":"2024-02-05T07:40:43.821421+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:40:43.821421+0000","last_undegraded":"2024-02-05T07:40:43.821421+0000","last_fullsized":"2024-02-05T07:40:43.821421+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:39:44.702291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:44.702291+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:52.040389+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"616'35","reported_seq":1363,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710257+0000","last_change":"2024-02-05T07:38:50.868322+0000","last_active":"2024-02-05T07:40:44.710257+0000","last_peered":"2024-02-05T07:40:44.710257+0000","last_clean":"2024-02-05T07:40:44.710257+0000","last_became_active":"2024-02-05T07:38:09.496732+0000","last_became_peered":"2024-02-05T07:38:09.496732+0000","last_unstale":"2024-02-05T07:40:44.710257+0000","last_undegraded":"2024-02-05T07:40:44.710257+0000","last_fullsized":"2024-02-05T07:40:44.710257+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:38:50.868179+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:38:50.868179+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.376609+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":864,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.835858+0000","last_change":"2024-02-05T07:36:32.741872+0000","last_active":"2024-02-05T07:40:46.835858+0000","last_peered":"2024-02-05T07:40:46.835858+0000","last_clean":"2024-02-05T07:40:46.835858+0000","last_became_active":"2024-02-05T07:36:32.741377+0000","last_became_peered":"2024-02-05T07:36:32.741377+0000","last_unstale":"2024-02-05T07:40:46.835858+0000","last_undegraded":"2024-02-05T07:40:46.835858+0000","last_fullsized":"2024-02-05T07:40:46.835858+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:17.067204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:35:17.067204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":552,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.707846+0000","last_change":"2024-02-05T07:39:54.880406+0000","last_active":"2024-02-05T07:40:44.707846+0000","last_peered":"2024-02-05T07:40:44.707846+0000","last_clean":"2024-02-05T07:40:44.707846+0000","last_became_active":"2024-02-05T07:32:58.598709+0000","last_became_peered":"2024-02-05T07:32:58.598709+0000","last_unstale":"2024-02-05T07:40:44.707846+0000","last_undegraded":"2024-02-05T07:40:44.707846+0000","last_fullsized":"2024-02-05T07:40:44.707846+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:54.879841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:54.879841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:07.394562+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"670'12135","reported_seq":13602,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:45.657769+0000","last_change":"2024-02-05T07:39:46.125582+0000","last_active":"2024-02-05T07:40:45.657769+0000","last_peered":"2024-02-05T07:40:45.657769+0000","last_clean":"2024-02-05T07:40:45.657769+0000","last_became_active":"2024-02-05T07:38:15.646279+0000","last_became_peered":"2024-02-05T07:38:15.646279+0000","last_unstale":"2024-02-05T07:40:45.657769+0000","last_undegraded":"2024-02-05T07:40:45.657769+0000","last_fullsized":"2024-02-05T07:40:45.657769+0000","mapping_epoch":595,"log_start":"637'9411","ondisk_log_start":"637'9411","created":579,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'7750","last_scrub_stamp":"2024-02-05T07:39:46.125414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:39:46.125414+0000","objects_scrubbed":6078,"log_size":2724,"log_dups_size":3000,"ondisk_log_size":2724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:08.891141+0000","scrub_duration":2339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105185280,"num_objects":1605,"num_object_clones":0,"num_object_copies":4815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12135,"num_write_kb":439680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":577,"num_bytes_recovered":37814272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":1212,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821576+0000","last_change":"2024-02-05T07:37:39.746542+0000","last_active":"2024-02-05T07:40:43.821576+0000","last_peered":"2024-02-05T07:40:43.821576+0000","last_clean":"2024-02-05T07:40:43.821576+0000","last_became_active":"2024-02-05T07:37:36.693264+0000","last_became_peered":"2024-02-05T07:37:36.693264+0000","last_unstale":"2024-02-05T07:40:43.821576+0000","last_undegraded":"2024-02-05T07:40:43.821576+0000","last_fullsized":"2024-02-05T07:40:43.821576+0000","mapping_epoch":302,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.746435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.746435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1121,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.820906+0000","last_change":"2024-02-05T07:40:11.774832+0000","last_active":"2024-02-05T07:40:43.820906+0000","last_peered":"2024-02-05T07:40:43.820906+0000","last_clean":"2024-02-05T07:40:43.820906+0000","last_became_active":"2024-02-05T07:36:32.709025+0000","last_became_peered":"2024-02-05T07:36:32.709025+0000","last_unstale":"2024-02-05T07:40:43.820906+0000","last_undegraded":"2024-02-05T07:40:43.820906+0000","last_fullsized":"2024-02-05T07:40:43.820906+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:11.774744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:40:11.774744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:36.409236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"615'39","reported_seq":961,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.708707+0000","last_change":"2024-02-05T07:38:41.807307+0000","last_active":"2024-02-05T07:40:44.708707+0000","last_peered":"2024-02-05T07:40:44.708707+0000","last_clean":"2024-02-05T07:40:44.708707+0000","last_became_active":"2024-02-05T07:36:32.712346+0000","last_became_peered":"2024-02-05T07:36:32.712346+0000","last_unstale":"2024-02-05T07:40:44.708707+0000","last_undegraded":"2024-02-05T07:40:44.708707+0000","last_fullsized":"2024-02-05T07:40:44.708707+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'29","last_scrub_stamp":"2024-02-05T07:38:41.806977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:41.806977+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.376374+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":289,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.712410+0000","last_change":"2024-02-05T07:39:57.311987+0000","last_active":"2024-02-05T07:40:44.712410+0000","last_peered":"2024-02-05T07:40:44.712410+0000","last_clean":"2024-02-05T07:40:44.712410+0000","last_became_active":"2024-02-05T07:36:32.732781+0000","last_became_peered":"2024-02-05T07:36:32.732781+0000","last_unstale":"2024-02-05T07:40:44.712410+0000","last_undegraded":"2024-02-05T07:40:44.712410+0000","last_fullsized":"2024-02-05T07:40:44.712410+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:57.311857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:39:57.311857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:04.385252+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1282,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.708332+0000","last_change":"2024-02-05T07:40:09.083562+0000","last_active":"2024-02-05T07:40:44.708332+0000","last_peered":"2024-02-05T07:40:44.708332+0000","last_clean":"2024-02-05T07:40:44.708332+0000","last_became_active":"2024-02-05T07:36:32.709872+0000","last_became_peered":"2024-02-05T07:36:32.709872+0000","last_unstale":"2024-02-05T07:40:44.708332+0000","last_undegraded":"2024-02-05T07:40:44.708332+0000","last_fullsized":"2024-02-05T07:40:44.708332+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:09.083452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:40:09.083452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:10.818855+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"672'12195","reported_seq":12392,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:47.271801+0000","last_change":"2024-02-05T07:37:53.093098+0000","last_active":"2024-02-05T07:40:47.271801+0000","last_peered":"2024-02-05T07:40:47.271801+0000","last_clean":"2024-02-05T07:40:47.271801+0000","last_became_active":"2024-02-05T07:37:53.091347+0000","last_became_peered":"2024-02-05T07:37:53.091347+0000","last_unstale":"2024-02-05T07:40:47.271801+0000","last_undegraded":"2024-02-05T07:40:47.271801+0000","last_fullsized":"2024-02-05T07:40:47.271801+0000","mapping_epoch":579,"log_start":"637'9300","ondisk_log_start":"637'9300","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2895,"log_dups_size":3000,"ondisk_log_size":2895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101384192,"num_objects":1547,"num_object_clones":0,"num_object_copies":4641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12195,"num_write_kb":439744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":872,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821601+0000","last_change":"2024-02-05T07:38:11.676690+0000","last_active":"2024-02-05T07:40:43.821601+0000","last_peered":"2024-02-05T07:40:43.821601+0000","last_clean":"2024-02-05T07:40:43.821601+0000","last_became_active":"2024-02-05T07:38:11.676446+0000","last_became_peered":"2024-02-05T07:38:11.676446+0000","last_unstale":"2024-02-05T07:40:43.821601+0000","last_undegraded":"2024-02-05T07:40:43.821601+0000","last_fullsized":"2024-02-05T07:40:43.821601+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.778695+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.778695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":876,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.721307+0000","last_change":"2024-02-05T07:39:51.915848+0000","last_active":"2024-02-05T07:40:44.721307+0000","last_peered":"2024-02-05T07:40:44.721307+0000","last_clean":"2024-02-05T07:40:44.721307+0000","last_became_active":"2024-02-05T07:38:09.496845+0000","last_became_peered":"2024-02-05T07:38:09.496845+0000","last_unstale":"2024-02-05T07:40:44.721307+0000","last_undegraded":"2024-02-05T07:40:44.721307+0000","last_fullsized":"2024-02-05T07:40:44.721307+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:51.915741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:51.915741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:18.553450+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"615'36","reported_seq":320,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.711807+0000","last_change":"2024-02-05T07:38:54.585085+0000","last_active":"2024-02-05T07:40:44.711807+0000","last_peered":"2024-02-05T07:40:44.711807+0000","last_clean":"2024-02-05T07:40:44.711807+0000","last_became_active":"2024-02-05T07:36:32.753752+0000","last_became_peered":"2024-02-05T07:36:32.753752+0000","last_unstale":"2024-02-05T07:40:44.711807+0000","last_undegraded":"2024-02-05T07:40:44.711807+0000","last_fullsized":"2024-02-05T07:40:44.711807+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:38:54.584948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:54.584948+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.624472+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1291,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:45.245852+0000","last_change":"2024-02-05T07:38:31.740800+0000","last_active":"2024-02-05T07:40:45.245852+0000","last_peered":"2024-02-05T07:40:45.245852+0000","last_clean":"2024-02-05T07:40:45.245852+0000","last_became_active":"2024-02-05T07:30:40.493770+0000","last_became_peered":"2024-02-05T07:30:40.493770+0000","last_unstale":"2024-02-05T07:40:45.245852+0000","last_undegraded":"2024-02-05T07:40:45.245852+0000","last_fullsized":"2024-02-05T07:40:45.245852+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:38:31.740660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:38:31.740660+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.376467+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":609,"num_read_kb":604,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":580,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.832308+0000","last_change":"2024-02-05T07:38:41.375038+0000","last_active":"2024-02-05T07:40:46.832308+0000","last_peered":"2024-02-05T07:40:46.832308+0000","last_clean":"2024-02-05T07:40:46.832308+0000","last_became_active":"2024-02-05T07:38:09.493827+0000","last_became_peered":"2024-02-05T07:38:09.493827+0000","last_unstale":"2024-02-05T07:40:46.832308+0000","last_undegraded":"2024-02-05T07:40:46.832308+0000","last_fullsized":"2024-02-05T07:40:46.832308+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:41.374804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:38:41.374804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":1298,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.711028+0000","last_change":"2024-02-05T07:39:49.884158+0000","last_active":"2024-02-05T07:40:44.711028+0000","last_peered":"2024-02-05T07:40:44.711028+0000","last_clean":"2024-02-05T07:40:44.711028+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:40:44.711028+0000","last_undegraded":"2024-02-05T07:40:44.711028+0000","last_fullsized":"2024-02-05T07:40:44.711028+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:39:49.883906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:49.883906+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:05.915566+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"672'12183","reported_seq":12380,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:47.218097+0000","last_change":"2024-02-05T07:37:52.817223+0000","last_active":"2024-02-05T07:40:47.218097+0000","last_peered":"2024-02-05T07:40:47.218097+0000","last_clean":"2024-02-05T07:40:47.218097+0000","last_became_active":"2024-02-05T07:37:52.816946+0000","last_became_peered":"2024-02-05T07:37:52.816946+0000","last_unstale":"2024-02-05T07:40:47.218097+0000","last_undegraded":"2024-02-05T07:40:47.218097+0000","last_fullsized":"2024-02-05T07:40:47.218097+0000","mapping_epoch":579,"log_start":"636'9200","ondisk_log_start":"636'9200","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2983,"log_dups_size":3000,"ondisk_log_size":2983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102563840,"num_objects":1565,"num_object_clones":0,"num_object_copies":4695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12183,"num_write_kb":439936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":289,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.712817+0000","last_change":"2024-02-05T07:38:43.594222+0000","last_active":"2024-02-05T07:40:44.712817+0000","last_peered":"2024-02-05T07:40:44.712817+0000","last_clean":"2024-02-05T07:40:44.712817+0000","last_became_active":"2024-02-05T07:36:32.732334+0000","last_became_peered":"2024-02-05T07:36:32.732334+0000","last_unstale":"2024-02-05T07:40:44.712817+0000","last_undegraded":"2024-02-05T07:40:44.712817+0000","last_fullsized":"2024-02-05T07:40:44.712817+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:43.593810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:43.593810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.624272+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"616'44","reported_seq":1455,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.822711+0000","last_change":"2024-02-05T07:38:09.529174+0000","last_active":"2024-02-05T07:40:43.822711+0000","last_peered":"2024-02-05T07:40:43.822711+0000","last_clean":"2024-02-05T07:40:43.822711+0000","last_became_active":"2024-02-05T07:38:09.479528+0000","last_became_peered":"2024-02-05T07:38:09.479528+0000","last_unstale":"2024-02-05T07:40:43.822711+0000","last_undegraded":"2024-02-05T07:40:43.822711+0000","last_fullsized":"2024-02-05T07:40:43.822711+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'31","last_scrub_stamp":"2024-02-05T07:37:45.644963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.644963+0000","objects_scrubbed":8,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":154,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.707683+0000","last_change":"2024-02-05T07:40:08.312293+0000","last_active":"2024-02-05T07:40:44.707683+0000","last_peered":"2024-02-05T07:40:44.707683+0000","last_clean":"2024-02-05T07:40:44.707683+0000","last_became_active":"2024-02-05T07:38:31.807354+0000","last_became_peered":"2024-02-05T07:38:31.807354+0000","last_unstale":"2024-02-05T07:40:44.707683+0000","last_undegraded":"2024-02-05T07:40:44.707683+0000","last_fullsized":"2024-02-05T07:40:44.707683+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:08.312041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:40:08.312041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:35.861633+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1043,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.820883+0000","last_change":"2024-02-05T07:38:29.812121+0000","last_active":"2024-02-05T07:40:43.820883+0000","last_peered":"2024-02-05T07:40:43.820883+0000","last_clean":"2024-02-05T07:40:43.820883+0000","last_became_active":"2024-02-05T07:37:06.977158+0000","last_became_peered":"2024-02-05T07:37:06.977158+0000","last_unstale":"2024-02-05T07:40:43.820883+0000","last_undegraded":"2024-02-05T07:40:43.820883+0000","last_fullsized":"2024-02-05T07:40:43.820883+0000","mapping_epoch":367,"log_start":"86'128","ondisk_log_start":"86'128","created":20,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:29.812050+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:38:29.812050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":855,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.834612+0000","last_change":"2024-02-05T07:38:49.589976+0000","last_active":"2024-02-05T07:40:46.834612+0000","last_peered":"2024-02-05T07:40:46.834612+0000","last_clean":"2024-02-05T07:40:46.834612+0000","last_became_active":"2024-02-05T07:38:09.488951+0000","last_became_peered":"2024-02-05T07:38:09.488951+0000","last_unstale":"2024-02-05T07:40:46.834612+0000","last_undegraded":"2024-02-05T07:40:46.834612+0000","last_fullsized":"2024-02-05T07:40:46.834612+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:49.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:49.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"672'12109","reported_seq":12306,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:47.230297+0000","last_change":"2024-02-05T07:37:53.090431+0000","last_active":"2024-02-05T07:40:47.230297+0000","last_peered":"2024-02-05T07:40:47.230297+0000","last_clean":"2024-02-05T07:40:47.230297+0000","last_became_active":"2024-02-05T07:37:53.090153+0000","last_became_peered":"2024-02-05T07:37:53.090153+0000","last_unstale":"2024-02-05T07:40:47.230297+0000","last_undegraded":"2024-02-05T07:40:47.230297+0000","last_fullsized":"2024-02-05T07:40:47.230297+0000","mapping_epoch":579,"log_start":"637'9200","ondisk_log_start":"637'9200","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2909,"log_dups_size":3000,"ondisk_log_size":2909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103481344,"num_objects":1579,"num_object_clones":0,"num_object_copies":4737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12109,"num_write_kb":438016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":1266,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710980+0000","last_change":"2024-02-05T07:38:32.532424+0000","last_active":"2024-02-05T07:40:44.710980+0000","last_peered":"2024-02-05T07:40:44.710980+0000","last_clean":"2024-02-05T07:40:44.710980+0000","last_became_active":"2024-02-05T07:38:11.674897+0000","last_became_peered":"2024-02-05T07:38:11.674897+0000","last_unstale":"2024-02-05T07:40:44.710980+0000","last_undegraded":"2024-02-05T07:40:44.710980+0000","last_fullsized":"2024-02-05T07:40:44.710980+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:32.532304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:32.532304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":294,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.711725+0000","last_change":"2024-02-05T07:38:35.498140+0000","last_active":"2024-02-05T07:40:44.711725+0000","last_peered":"2024-02-05T07:40:44.711725+0000","last_clean":"2024-02-05T07:40:44.711725+0000","last_became_active":"2024-02-05T07:38:09.500885+0000","last_became_peered":"2024-02-05T07:38:09.500885+0000","last_unstale":"2024-02-05T07:40:44.711725+0000","last_undegraded":"2024-02-05T07:40:44.711725+0000","last_fullsized":"2024-02-05T07:40:44.711725+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:35.497879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:35.497879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.624144+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"615'26","reported_seq":242,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.713923+0000","last_change":"2024-02-05T07:38:39.505608+0000","last_active":"2024-02-05T07:40:44.713923+0000","last_peered":"2024-02-05T07:40:44.713923+0000","last_clean":"2024-02-05T07:40:44.713923+0000","last_became_active":"2024-02-05T07:37:31.671370+0000","last_became_peered":"2024-02-05T07:37:31.671370+0000","last_unstale":"2024-02-05T07:40:44.713923+0000","last_undegraded":"2024-02-05T07:40:44.713923+0000","last_fullsized":"2024-02-05T07:40:44.713923+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'22","last_scrub_stamp":"2024-02-05T07:38:39.505230+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:38:39.505230+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.624219+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":289,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.706155+0000","last_change":"2024-02-05T07:38:45.513185+0000","last_active":"2024-02-05T07:40:44.706155+0000","last_peered":"2024-02-05T07:40:44.706155+0000","last_clean":"2024-02-05T07:40:44.706155+0000","last_became_active":"2024-02-05T07:36:32.733822+0000","last_became_peered":"2024-02-05T07:36:32.733822+0000","last_unstale":"2024-02-05T07:40:44.706155+0000","last_undegraded":"2024-02-05T07:40:44.706155+0000","last_fullsized":"2024-02-05T07:40:44.706155+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:45.512912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:38:45.512912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.624360+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":293,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710332+0000","last_change":"2024-02-05T07:38:57.649523+0000","last_active":"2024-02-05T07:40:44.710332+0000","last_peered":"2024-02-05T07:40:44.710332+0000","last_clean":"2024-02-05T07:40:44.710332+0000","last_became_active":"2024-02-05T07:36:32.781195+0000","last_became_peered":"2024-02-05T07:36:32.781195+0000","last_unstale":"2024-02-05T07:40:44.710332+0000","last_undegraded":"2024-02-05T07:40:44.710332+0000","last_fullsized":"2024-02-05T07:40:44.710332+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:38:57.649351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:57.649351+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.624416+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1158,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.822567+0000","last_change":"2024-02-05T07:38:51.640312+0000","last_active":"2024-02-05T07:40:43.822567+0000","last_peered":"2024-02-05T07:40:43.822567+0000","last_clean":"2024-02-05T07:40:43.822567+0000","last_became_active":"2024-02-05T07:36:32.718684+0000","last_became_peered":"2024-02-05T07:36:32.718684+0000","last_unstale":"2024-02-05T07:40:43.822567+0000","last_undegraded":"2024-02-05T07:40:43.822567+0000","last_fullsized":"2024-02-05T07:40:43.822567+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:51.640224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:38:51.640224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770796+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"672'12373","reported_seq":12568,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:47.233213+0000","last_change":"2024-02-05T07:37:53.089611+0000","last_active":"2024-02-05T07:40:47.233213+0000","last_peered":"2024-02-05T07:40:47.233213+0000","last_clean":"2024-02-05T07:40:47.233213+0000","last_became_active":"2024-02-05T07:37:53.089298+0000","last_became_peered":"2024-02-05T07:37:53.089298+0000","last_unstale":"2024-02-05T07:40:47.233213+0000","last_undegraded":"2024-02-05T07:40:47.233213+0000","last_fullsized":"2024-02-05T07:40:47.233213+0000","mapping_epoch":579,"log_start":"636'9400","ondisk_log_start":"636'9400","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":3000,"ondisk_log_size":2973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101122048,"num_objects":1543,"num_object_clones":0,"num_object_copies":4629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12373,"num_write_kb":445312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":653,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.705968+0000","last_change":"2024-02-05T07:38:15.643159+0000","last_active":"2024-02-05T07:40:44.705968+0000","last_peered":"2024-02-05T07:40:44.705968+0000","last_clean":"2024-02-05T07:40:44.705968+0000","last_became_active":"2024-02-05T07:38:15.642779+0000","last_became_peered":"2024-02-05T07:38:15.642779+0000","last_unstale":"2024-02-05T07:40:44.705968+0000","last_undegraded":"2024-02-05T07:40:44.705968+0000","last_fullsized":"2024-02-05T07:40:44.705968+0000","mapping_epoch":595,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:08.283492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:08.283492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1031,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.708820+0000","last_change":"2024-02-05T07:40:19.104414+0000","last_active":"2024-02-05T07:40:44.708820+0000","last_peered":"2024-02-05T07:40:44.708820+0000","last_clean":"2024-02-05T07:40:44.708820+0000","last_became_active":"2024-02-05T07:38:09.463763+0000","last_became_peered":"2024-02-05T07:38:09.463763+0000","last_unstale":"2024-02-05T07:40:44.708820+0000","last_undegraded":"2024-02-05T07:40:44.708820+0000","last_fullsized":"2024-02-05T07:40:44.708820+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:19.104295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:40:19.104295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:47.076546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"616'68","reported_seq":390,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.739589+0000","last_change":"2024-02-05T07:36:56.264458+0000","last_active":"2024-02-05T07:40:44.739589+0000","last_peered":"2024-02-05T07:40:44.739589+0000","last_clean":"2024-02-05T07:40:44.739589+0000","last_became_active":"2024-02-05T07:36:32.738270+0000","last_became_peered":"2024-02-05T07:36:32.738270+0000","last_unstale":"2024-02-05T07:40:44.739589+0000","last_undegraded":"2024-02-05T07:40:44.739589+0000","last_fullsized":"2024-02-05T07:40:44.739589+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'56","last_scrub_stamp":"2024-02-05T07:36:56.264300+0000","last_deep_scrub":"550'56","last_deep_scrub_stamp":"2024-02-05T07:36:56.264300+0000","last_clean_scrub_stamp":"2024-02-05T07:36:56.264300+0000","objects_scrubbed":10,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.618608+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":82,"num_read_kb":67,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":687,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821659+0000","last_change":"2024-02-05T07:40:15.743056+0000","last_active":"2024-02-05T07:40:43.821659+0000","last_peered":"2024-02-05T07:40:43.821659+0000","last_clean":"2024-02-05T07:40:43.821659+0000","last_became_active":"2024-02-05T07:38:09.484410+0000","last_became_peered":"2024-02-05T07:38:09.484410+0000","last_unstale":"2024-02-05T07:40:43.821659+0000","last_undegraded":"2024-02-05T07:40:43.821659+0000","last_fullsized":"2024-02-05T07:40:43.821659+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:15.742953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:40:15.742953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:22.968200+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":1195,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.709785+0000","last_change":"2024-02-05T07:40:12.141885+0000","last_active":"2024-02-05T07:40:44.709785+0000","last_peered":"2024-02-05T07:40:44.709785+0000","last_clean":"2024-02-05T07:40:44.709785+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:40:44.709785+0000","last_undegraded":"2024-02-05T07:40:44.709785+0000","last_fullsized":"2024-02-05T07:40:44.709785+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:40:12.141744+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:40:12.141744+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:22.389002+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":869,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.835838+0000","last_change":"2024-02-05T07:39:03.439286+0000","last_active":"2024-02-05T07:40:46.835838+0000","last_peered":"2024-02-05T07:40:46.835838+0000","last_clean":"2024-02-05T07:40:46.835838+0000","last_became_active":"2024-02-05T07:38:09.486846+0000","last_became_peered":"2024-02-05T07:38:09.486846+0000","last_unstale":"2024-02-05T07:40:46.835838+0000","last_undegraded":"2024-02-05T07:40:46.835838+0000","last_fullsized":"2024-02-05T07:40:46.835838+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:03.439161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:03.439161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"668'12338","reported_seq":12529,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821560+0000","last_change":"2024-02-05T07:37:52.813698+0000","last_active":"2024-02-05T07:40:43.821560+0000","last_peered":"2024-02-05T07:40:43.821560+0000","last_clean":"2024-02-05T07:40:43.821560+0000","last_became_active":"2024-02-05T07:37:52.813405+0000","last_became_peered":"2024-02-05T07:37:52.813405+0000","last_unstale":"2024-02-05T07:40:43.821560+0000","last_undegraded":"2024-02-05T07:40:43.821560+0000","last_fullsized":"2024-02-05T07:40:43.821560+0000","mapping_epoch":579,"log_start":"636'9417","ondisk_log_start":"636'9417","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2921,"log_dups_size":3000,"ondisk_log_size":2921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119668736,"num_objects":1826,"num_object_clones":0,"num_object_copies":5478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12338,"num_write_kb":453248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":621,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.836381+0000","last_change":"2024-02-05T07:40:20.876587+0000","last_active":"2024-02-05T07:40:46.836381+0000","last_peered":"2024-02-05T07:40:46.836381+0000","last_clean":"2024-02-05T07:40:46.836381+0000","last_became_active":"2024-02-05T07:40:20.876256+0000","last_became_peered":"2024-02-05T07:40:20.876256+0000","last_unstale":"2024-02-05T07:40:46.836381+0000","last_undegraded":"2024-02-05T07:40:46.836381+0000","last_fullsized":"2024-02-05T07:40:46.836381+0000","mapping_epoch":619,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:39:45.710637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:39:45.710637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.071227+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"615'21","reported_seq":1085,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.708561+0000","last_change":"2024-02-05T07:38:35.790500+0000","last_active":"2024-02-05T07:40:44.708561+0000","last_peered":"2024-02-05T07:40:44.708561+0000","last_clean":"2024-02-05T07:40:44.708561+0000","last_became_active":"2024-02-05T07:38:09.479890+0000","last_became_peered":"2024-02-05T07:38:09.479890+0000","last_unstale":"2024-02-05T07:40:44.708561+0000","last_undegraded":"2024-02-05T07:40:44.708561+0000","last_fullsized":"2024-02-05T07:40:44.708561+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:38:35.790379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:35.790379+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.376549+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":851,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.835278+0000","last_change":"2024-02-05T07:36:54.146303+0000","last_active":"2024-02-05T07:40:46.835278+0000","last_peered":"2024-02-05T07:40:46.835278+0000","last_clean":"2024-02-05T07:40:46.835278+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:40:46.835278+0000","last_undegraded":"2024-02-05T07:40:46.835278+0000","last_fullsized":"2024-02-05T07:40:46.835278+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:54.146194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:54.146194+0000","last_clean_scrub_stamp":"2024-02-05T07:36:54.146194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":294,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710709+0000","last_change":"2024-02-05T07:38:09.492094+0000","last_active":"2024-02-05T07:40:44.710709+0000","last_peered":"2024-02-05T07:40:44.710709+0000","last_clean":"2024-02-05T07:40:44.710709+0000","last_became_active":"2024-02-05T07:38:09.491746+0000","last_became_peered":"2024-02-05T07:38:09.491746+0000","last_unstale":"2024-02-05T07:40:44.710709+0000","last_undegraded":"2024-02-05T07:40:44.710709+0000","last_fullsized":"2024-02-05T07:40:44.710709+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:45.362685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.362685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.618822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":288,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.705875+0000","last_change":"2024-02-05T07:37:00.310446+0000","last_active":"2024-02-05T07:40:44.705875+0000","last_peered":"2024-02-05T07:40:44.705875+0000","last_clean":"2024-02-05T07:40:44.705875+0000","last_became_active":"2024-02-05T07:36:32.734230+0000","last_became_peered":"2024-02-05T07:36:32.734230+0000","last_unstale":"2024-02-05T07:40:44.705875+0000","last_undegraded":"2024-02-05T07:40:44.705875+0000","last_fullsized":"2024-02-05T07:40:44.705875+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:00.309850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:00.309850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.618634+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"668'11942","reported_seq":12133,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.822663+0000","last_change":"2024-02-05T07:37:53.089683+0000","last_active":"2024-02-05T07:40:43.822663+0000","last_peered":"2024-02-05T07:40:43.822663+0000","last_clean":"2024-02-05T07:40:43.822663+0000","last_became_active":"2024-02-05T07:37:53.089386+0000","last_became_peered":"2024-02-05T07:37:53.089386+0000","last_unstale":"2024-02-05T07:40:43.822663+0000","last_undegraded":"2024-02-05T07:40:43.822663+0000","last_fullsized":"2024-02-05T07:40:43.822663+0000","mapping_epoch":579,"log_start":"636'9029","ondisk_log_start":"636'9029","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2913,"log_dups_size":3000,"ondisk_log_size":2913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118751232,"num_objects":1812,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11942,"num_write_kb":440128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":1264,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821829+0000","last_change":"2024-02-05T07:38:11.660956+0000","last_active":"2024-02-05T07:40:43.821829+0000","last_peered":"2024-02-05T07:40:43.821829+0000","last_clean":"2024-02-05T07:40:43.821829+0000","last_became_active":"2024-02-05T07:38:11.660622+0000","last_became_peered":"2024-02-05T07:38:11.660622+0000","last_unstale":"2024-02-05T07:40:43.821829+0000","last_undegraded":"2024-02-05T07:40:43.821829+0000","last_fullsized":"2024-02-05T07:40:43.821829+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.730473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.730473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"616'51","reported_seq":1234,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821023+0000","last_change":"2024-02-05T07:38:52.517441+0000","last_active":"2024-02-05T07:40:43.821023+0000","last_peered":"2024-02-05T07:40:43.821023+0000","last_clean":"2024-02-05T07:40:43.821023+0000","last_became_active":"2024-02-05T07:36:16.736226+0000","last_became_peered":"2024-02-05T07:36:16.736226+0000","last_unstale":"2024-02-05T07:40:43.821023+0000","last_undegraded":"2024-02-05T07:40:43.821023+0000","last_fullsized":"2024-02-05T07:40:43.821023+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:38:52.517337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:52.517337+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770781+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":153,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.709987+0000","last_change":"2024-02-05T07:40:03.305634+0000","last_active":"2024-02-05T07:40:44.709987+0000","last_peered":"2024-02-05T07:40:44.709987+0000","last_clean":"2024-02-05T07:40:44.709987+0000","last_became_active":"2024-02-05T07:38:31.795149+0000","last_became_peered":"2024-02-05T07:38:31.795149+0000","last_unstale":"2024-02-05T07:40:44.709987+0000","last_undegraded":"2024-02-05T07:40:44.709987+0000","last_fullsized":"2024-02-05T07:40:44.709987+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:03.305531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_clean_scrub_stamp":"2024-02-05T07:40:03.305531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:24.210343+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":288,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.713475+0000","last_change":"2024-02-05T07:37:54.312733+0000","last_active":"2024-02-05T07:40:44.713475+0000","last_peered":"2024-02-05T07:40:44.713475+0000","last_clean":"2024-02-05T07:40:44.713475+0000","last_became_active":"2024-02-05T07:36:32.733494+0000","last_became_peered":"2024-02-05T07:36:32.733494+0000","last_unstale":"2024-02-05T07:40:44.713475+0000","last_undegraded":"2024-02-05T07:40:44.713475+0000","last_fullsized":"2024-02-05T07:40:44.713475+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:54.312642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.312642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.621693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":599,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.847672+0000","last_change":"2024-02-05T07:38:31.803859+0000","last_active":"2024-02-05T07:40:46.847672+0000","last_peered":"2024-02-05T07:40:46.847672+0000","last_clean":"2024-02-05T07:40:46.847672+0000","last_became_active":"2024-02-05T07:38:31.803474+0000","last_became_peered":"2024-02-05T07:38:31.803474+0000","last_unstale":"2024-02-05T07:40:46.847672+0000","last_undegraded":"2024-02-05T07:40:46.847672+0000","last_fullsized":"2024-02-05T07:40:46.847672+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:01.478555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:38:01.478555+0000","last_clean_scrub_stamp":"2024-02-05T07:38:01.478555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"670'12223","reported_seq":12416,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:45.766289+0000","last_change":"2024-02-05T07:37:53.090459+0000","last_active":"2024-02-05T07:40:45.766289+0000","last_peered":"2024-02-05T07:40:45.766289+0000","last_clean":"2024-02-05T07:40:45.766289+0000","last_became_active":"2024-02-05T07:37:53.090173+0000","last_became_peered":"2024-02-05T07:37:53.090173+0000","last_unstale":"2024-02-05T07:40:45.766289+0000","last_undegraded":"2024-02-05T07:40:45.766289+0000","last_fullsized":"2024-02-05T07:40:45.766289+0000","mapping_epoch":579,"log_start":"636'8906","ondisk_log_start":"636'8906","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":3317,"log_dups_size":0,"ondisk_log_size":3317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.621845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109248512,"num_objects":1667,"num_object_clones":0,"num_object_copies":5001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12223,"num_write_kb":444480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":1638,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.832438+0000","last_change":"2024-02-05T07:40:28.961455+0000","last_active":"2024-02-05T07:40:46.832438+0000","last_peered":"2024-02-05T07:40:46.832438+0000","last_clean":"2024-02-05T07:40:46.832438+0000","last_became_active":"2024-02-05T07:40:28.961096+0000","last_became_peered":"2024-02-05T07:40:28.961096+0000","last_unstale":"2024-02-05T07:40:46.832438+0000","last_undegraded":"2024-02-05T07:40:46.832438+0000","last_fullsized":"2024-02-05T07:40:46.832438+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.295329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.295329+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":118,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":288,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.711359+0000","last_change":"2024-02-05T07:37:51.288853+0000","last_active":"2024-02-05T07:40:44.711359+0000","last_peered":"2024-02-05T07:40:44.711359+0000","last_clean":"2024-02-05T07:40:44.711359+0000","last_became_active":"2024-02-05T07:36:32.733003+0000","last_became_peered":"2024-02-05T07:36:32.733003+0000","last_unstale":"2024-02-05T07:40:44.711359+0000","last_undegraded":"2024-02-05T07:40:44.711359+0000","last_fullsized":"2024-02-05T07:40:44.711359+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.288772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.288772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.621633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"615'24","reported_seq":1259,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.708027+0000","last_change":"2024-02-05T07:38:30.797759+0000","last_active":"2024-02-05T07:40:44.708027+0000","last_peered":"2024-02-05T07:40:44.708027+0000","last_clean":"2024-02-05T07:40:44.708027+0000","last_became_active":"2024-02-05T07:38:09.480893+0000","last_became_peered":"2024-02-05T07:38:09.480893+0000","last_unstale":"2024-02-05T07:40:44.708027+0000","last_undegraded":"2024-02-05T07:40:44.708027+0000","last_fullsized":"2024-02-05T07:40:44.708027+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'18","last_scrub_stamp":"2024-02-05T07:38:30.797628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:30.797628+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.376437+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":186,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710643+0000","last_change":"2024-02-05T07:40:18.107509+0000","last_active":"2024-02-05T07:40:44.710643+0000","last_peered":"2024-02-05T07:40:44.710643+0000","last_clean":"2024-02-05T07:40:44.710643+0000","last_became_active":"2024-02-05T07:38:09.485926+0000","last_became_peered":"2024-02-05T07:38:09.485926+0000","last_unstale":"2024-02-05T07:40:44.710643+0000","last_undegraded":"2024-02-05T07:40:44.710643+0000","last_fullsized":"2024-02-05T07:40:44.710643+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:18.107395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:40:18.107395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:42.598865+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":862,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.845728+0000","last_change":"2024-02-05T07:39:47.983852+0000","last_active":"2024-02-05T07:40:46.845728+0000","last_peered":"2024-02-05T07:40:46.845728+0000","last_clean":"2024-02-05T07:40:46.845728+0000","last_became_active":"2024-02-05T07:30:40.485991+0000","last_became_peered":"2024-02-05T07:30:40.485991+0000","last_unstale":"2024-02-05T07:40:46.845728+0000","last_undegraded":"2024-02-05T07:40:46.845728+0000","last_fullsized":"2024-02-05T07:40:46.845728+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:47.982166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:47.982166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.045882+0000","scrub_duration":274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"670'12079","reported_seq":13412,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:45.611359+0000","last_change":"2024-02-05T07:40:02.890033+0000","last_active":"2024-02-05T07:40:45.611359+0000","last_peered":"2024-02-05T07:40:45.611359+0000","last_clean":"2024-02-05T07:40:45.611359+0000","last_became_active":"2024-02-05T07:38:15.650851+0000","last_became_peered":"2024-02-05T07:38:15.650851+0000","last_unstale":"2024-02-05T07:40:45.611359+0000","last_undegraded":"2024-02-05T07:40:45.611359+0000","last_fullsized":"2024-02-05T07:40:45.611359+0000","mapping_epoch":595,"log_start":"638'9400","ondisk_log_start":"638'9400","created":579,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'8919","last_scrub_stamp":"2024-02-05T07:40:02.889877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:40:02.889877+0000","objects_scrubbed":4908,"log_size":2679,"log_dups_size":3000,"ondisk_log_size":2679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:14.278208+0000","scrub_duration":2873,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110559232,"num_objects":1687,"num_object_clones":0,"num_object_copies":5061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12079,"num_write_kb":440512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":536,"num_bytes_recovered":35127296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":1263,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710090+0000","last_change":"2024-02-05T07:40:24.720852+0000","last_active":"2024-02-05T07:40:44.710090+0000","last_peered":"2024-02-05T07:40:44.710090+0000","last_clean":"2024-02-05T07:40:44.710090+0000","last_became_active":"2024-02-05T07:40:24.720423+0000","last_became_peered":"2024-02-05T07:40:24.720423+0000","last_unstale":"2024-02-05T07:40:44.710090+0000","last_undegraded":"2024-02-05T07:40:44.710090+0000","last_fullsized":"2024-02-05T07:40:44.710090+0000","mapping_epoch":595,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:33.547397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:33.547397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"589'13","reported_seq":1400,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821407+0000","last_change":"2024-02-05T07:38:09.523382+0000","last_active":"2024-02-05T07:40:43.821407+0000","last_peered":"2024-02-05T07:40:43.821407+0000","last_clean":"2024-02-05T07:40:43.821407+0000","last_became_active":"2024-02-05T07:38:09.471365+0000","last_became_peered":"2024-02-05T07:38:09.471365+0000","last_unstale":"2024-02-05T07:40:43.821407+0000","last_undegraded":"2024-02-05T07:40:43.821407+0000","last_fullsized":"2024-02-05T07:40:43.821407+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'12","last_scrub_stamp":"2024-02-05T07:37:53.638258+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:37:53.638258+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"615'39","reported_seq":944,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.820834+0000","last_change":"2024-02-05T07:38:09.513629+0000","last_active":"2024-02-05T07:40:43.820834+0000","last_peered":"2024-02-05T07:40:43.820834+0000","last_clean":"2024-02-05T07:40:43.820834+0000","last_became_active":"2024-02-05T07:38:09.471133+0000","last_became_peered":"2024-02-05T07:38:09.471133+0000","last_unstale":"2024-02-05T07:40:43.820834+0000","last_undegraded":"2024-02-05T07:40:43.820834+0000","last_fullsized":"2024-02-05T07:40:43.820834+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:38:00.687570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:00.687570+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770660+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"614'4","reported_seq":203,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.720778+0000","last_change":"2024-02-05T07:38:55.830017+0000","last_active":"2024-02-05T07:40:44.720778+0000","last_peered":"2024-02-05T07:40:44.720778+0000","last_clean":"2024-02-05T07:40:44.720778+0000","last_became_active":"2024-02-05T07:38:09.482757+0000","last_became_peered":"2024-02-05T07:38:09.482757+0000","last_unstale":"2024-02-05T07:40:44.720778+0000","last_undegraded":"2024-02-05T07:40:44.720778+0000","last_fullsized":"2024-02-05T07:40:44.720778+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"614'4","last_scrub_stamp":"2024-02-05T07:38:55.829386+0000","last_deep_scrub":"614'4","last_deep_scrub_stamp":"2024-02-05T07:38:55.829386+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.829386+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.376727+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":943,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.820897+0000","last_change":"2024-02-05T07:37:02.526520+0000","last_active":"2024-02-05T07:40:43.820897+0000","last_peered":"2024-02-05T07:40:43.820897+0000","last_clean":"2024-02-05T07:40:43.820897+0000","last_became_active":"2024-02-05T07:36:32.701535+0000","last_became_peered":"2024-02-05T07:36:32.701535+0000","last_unstale":"2024-02-05T07:40:43.820897+0000","last_undegraded":"2024-02-05T07:40:43.820897+0000","last_fullsized":"2024-02-05T07:40:43.820897+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:02.526362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:37:02.526362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770268+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":584,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.709449+0000","last_change":"2024-02-05T07:38:57.845026+0000","last_active":"2024-02-05T07:40:44.709449+0000","last_peered":"2024-02-05T07:40:44.709449+0000","last_clean":"2024-02-05T07:40:44.709449+0000","last_became_active":"2024-02-05T07:37:41.643328+0000","last_became_peered":"2024-02-05T07:37:41.643328+0000","last_unstale":"2024-02-05T07:40:44.709449+0000","last_undegraded":"2024-02-05T07:40:44.709449+0000","last_fullsized":"2024-02-05T07:40:44.709449+0000","mapping_epoch":576,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:57.844891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:57.844891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.376578+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"615'21","reported_seq":919,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.834433+0000","last_change":"2024-02-05T07:38:48.343585+0000","last_active":"2024-02-05T07:40:46.834433+0000","last_peered":"2024-02-05T07:40:46.834433+0000","last_clean":"2024-02-05T07:40:46.834433+0000","last_became_active":"2024-02-05T07:36:32.742483+0000","last_became_peered":"2024-02-05T07:36:32.742483+0000","last_unstale":"2024-02-05T07:40:46.834433+0000","last_undegraded":"2024-02-05T07:40:46.834433+0000","last_fullsized":"2024-02-05T07:40:46.834433+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'15","last_scrub_stamp":"2024-02-05T07:38:48.343448+0000","last_deep_scrub":"613'15","last_deep_scrub_stamp":"2024-02-05T07:38:48.343448+0000","last_clean_scrub_stamp":"2024-02-05T07:38:48.343448+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":1075,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.820595+0000","last_change":"2024-02-05T07:37:47.639730+0000","last_active":"2024-02-05T07:40:43.820595+0000","last_peered":"2024-02-05T07:40:43.820595+0000","last_clean":"2024-02-05T07:40:43.820595+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:40:43.820595+0000","last_undegraded":"2024-02-05T07:40:43.820595+0000","last_fullsized":"2024-02-05T07:40:43.820595+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:37:47.639652+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:37:47.639652+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770387+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":851,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.836184+0000","last_change":"2024-02-05T07:37:01.178948+0000","last_active":"2024-02-05T07:40:46.836184+0000","last_peered":"2024-02-05T07:40:46.836184+0000","last_clean":"2024-02-05T07:40:46.836184+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:40:46.836184+0000","last_undegraded":"2024-02-05T07:40:46.836184+0000","last_fullsized":"2024-02-05T07:40:46.836184+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:01.178816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:01.178816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1139,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821816+0000","last_change":"2024-02-05T07:38:41.820804+0000","last_active":"2024-02-05T07:40:43.821816+0000","last_peered":"2024-02-05T07:40:43.821816+0000","last_clean":"2024-02-05T07:40:43.821816+0000","last_became_active":"2024-02-05T07:38:09.480228+0000","last_became_peered":"2024-02-05T07:38:09.480228+0000","last_unstale":"2024-02-05T07:40:43.821816+0000","last_undegraded":"2024-02-05T07:40:43.821816+0000","last_fullsized":"2024-02-05T07:40:43.821816+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:41.820713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:38:41.820713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770842+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":1080,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.832838+0000","last_change":"2024-02-05T07:38:41.750699+0000","last_active":"2024-02-05T07:40:46.832838+0000","last_peered":"2024-02-05T07:40:46.832838+0000","last_clean":"2024-02-05T07:40:46.832838+0000","last_became_active":"2024-02-05T07:38:41.750174+0000","last_became_peered":"2024-02-05T07:38:41.750174+0000","last_unstale":"2024-02-05T07:40:46.832838+0000","last_undegraded":"2024-02-05T07:40:46.832838+0000","last_fullsized":"2024-02-05T07:40:46.832838+0000","mapping_epoch":607,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.260984+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.260984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"615'21","reported_seq":924,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.835944+0000","last_change":"2024-02-05T07:38:55.387276+0000","last_active":"2024-02-05T07:40:46.835944+0000","last_peered":"2024-02-05T07:40:46.835944+0000","last_clean":"2024-02-05T07:40:46.835944+0000","last_became_active":"2024-02-05T07:36:32.747221+0000","last_became_peered":"2024-02-05T07:36:32.747221+0000","last_unstale":"2024-02-05T07:40:46.835944+0000","last_undegraded":"2024-02-05T07:40:46.835944+0000","last_fullsized":"2024-02-05T07:40:46.835944+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:38:55.387145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.387145+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":690,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821511+0000","last_change":"2024-02-05T07:39:00.488430+0000","last_active":"2024-02-05T07:40:43.821511+0000","last_peered":"2024-02-05T07:40:43.821511+0000","last_clean":"2024-02-05T07:40:43.821511+0000","last_became_active":"2024-02-05T07:38:09.479116+0000","last_became_peered":"2024-02-05T07:38:09.479116+0000","last_unstale":"2024-02-05T07:40:43.821511+0000","last_undegraded":"2024-02-05T07:40:43.821511+0000","last_fullsized":"2024-02-05T07:40:43.821511+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:00.488357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:39:00.488357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770811+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":854,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.834533+0000","last_change":"2024-02-05T07:38:53.372481+0000","last_active":"2024-02-05T07:40:46.834533+0000","last_peered":"2024-02-05T07:40:46.834533+0000","last_clean":"2024-02-05T07:40:46.834533+0000","last_became_active":"2024-02-05T07:30:40.496076+0000","last_became_peered":"2024-02-05T07:30:40.496076+0000","last_unstale":"2024-02-05T07:40:46.834533+0000","last_undegraded":"2024-02-05T07:40:46.834533+0000","last_fullsized":"2024-02-05T07:40:46.834533+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:53.372355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_clean_scrub_stamp":"2024-02-05T07:38:53.372355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":294,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.712868+0000","last_change":"2024-02-05T07:38:09.489174+0000","last_active":"2024-02-05T07:40:44.712868+0000","last_peered":"2024-02-05T07:40:44.712868+0000","last_clean":"2024-02-05T07:40:44.712868+0000","last_became_active":"2024-02-05T07:38:09.488755+0000","last_became_peered":"2024-02-05T07:38:09.488755+0000","last_unstale":"2024-02-05T07:40:44.712868+0000","last_undegraded":"2024-02-05T07:40:44.712868+0000","last_fullsized":"2024-02-05T07:40:44.712868+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:47.330693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:37:47.330693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.621749+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":1066,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.822625+0000","last_change":"2024-02-05T07:38:41.725370+0000","last_active":"2024-02-05T07:40:43.822625+0000","last_peered":"2024-02-05T07:40:43.822625+0000","last_clean":"2024-02-05T07:40:43.822625+0000","last_became_active":"2024-02-05T07:38:41.725034+0000","last_became_peered":"2024-02-05T07:38:41.725034+0000","last_unstale":"2024-02-05T07:40:43.822625+0000","last_undegraded":"2024-02-05T07:40:43.822625+0000","last_fullsized":"2024-02-05T07:40:43.822625+0000","mapping_epoch":607,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:37.743034+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:37:37.743034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"615'25","reported_seq":1212,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.836387+0000","last_change":"2024-02-05T07:37:06.319292+0000","last_active":"2024-02-05T07:40:46.836387+0000","last_peered":"2024-02-05T07:40:46.836387+0000","last_clean":"2024-02-05T07:40:46.836387+0000","last_became_active":"2024-02-05T07:30:40.483700+0000","last_became_peered":"2024-02-05T07:30:40.483700+0000","last_unstale":"2024-02-05T07:40:46.836387+0000","last_undegraded":"2024-02-05T07:40:46.836387+0000","last_fullsized":"2024-02-05T07:40:46.836387+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:37:06.318619+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:37:06.318619+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1017,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.837020+0000","last_change":"2024-02-05T07:36:46.059245+0000","last_active":"2024-02-05T07:40:46.837020+0000","last_peered":"2024-02-05T07:40:46.837020+0000","last_clean":"2024-02-05T07:40:46.837020+0000","last_became_active":"2024-02-05T07:30:40.476058+0000","last_became_peered":"2024-02-05T07:30:40.476058+0000","last_unstale":"2024-02-05T07:40:46.837020+0000","last_undegraded":"2024-02-05T07:40:46.837020+0000","last_fullsized":"2024-02-05T07:40:46.837020+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:46.059129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.059129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"538'13","reported_seq":1044,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.833018+0000","last_change":"2024-02-05T07:38:51.305914+0000","last_active":"2024-02-05T07:40:46.833018+0000","last_peered":"2024-02-05T07:40:46.833018+0000","last_clean":"2024-02-05T07:40:46.833018+0000","last_became_active":"2024-02-05T07:36:32.746270+0000","last_became_peered":"2024-02-05T07:36:32.746270+0000","last_unstale":"2024-02-05T07:40:46.833018+0000","last_undegraded":"2024-02-05T07:40:46.833018+0000","last_fullsized":"2024-02-05T07:40:46.833018+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'13","last_scrub_stamp":"2024-02-05T07:38:51.305782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:51.305782+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1157,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.714216+0000","last_change":"2024-02-05T07:38:09.504184+0000","last_active":"2024-02-05T07:40:44.714216+0000","last_peered":"2024-02-05T07:40:44.714216+0000","last_clean":"2024-02-05T07:40:44.714216+0000","last_became_active":"2024-02-05T07:38:09.503255+0000","last_became_peered":"2024-02-05T07:38:09.503255+0000","last_unstale":"2024-02-05T07:40:44.714216+0000","last_undegraded":"2024-02-05T07:40:44.714216+0000","last_fullsized":"2024-02-05T07:40:44.714216+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:58.277028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:58.277028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.621912+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":637,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.832788+0000","last_change":"2024-02-05T07:37:40.289255+0000","last_active":"2024-02-05T07:40:46.832788+0000","last_peered":"2024-02-05T07:40:46.832788+0000","last_clean":"2024-02-05T07:40:46.832788+0000","last_became_active":"2024-02-05T07:37:12.040170+0000","last_became_peered":"2024-02-05T07:37:12.040170+0000","last_unstale":"2024-02-05T07:40:46.832788+0000","last_undegraded":"2024-02-05T07:40:46.832788+0000","last_fullsized":"2024-02-05T07:40:46.832788+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:40.289174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:40.289174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"615'35","reported_seq":944,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.835188+0000","last_change":"2024-02-05T07:36:59.117388+0000","last_active":"2024-02-05T07:40:46.835188+0000","last_peered":"2024-02-05T07:40:46.835188+0000","last_clean":"2024-02-05T07:40:46.835188+0000","last_became_active":"2024-02-05T07:36:32.746757+0000","last_became_peered":"2024-02-05T07:36:32.746757+0000","last_unstale":"2024-02-05T07:40:46.835188+0000","last_undegraded":"2024-02-05T07:40:46.835188+0000","last_fullsized":"2024-02-05T07:40:46.835188+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:36:59.117218+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.117218+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":550,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.713072+0000","last_change":"2024-02-05T07:39:53.396917+0000","last_active":"2024-02-05T07:40:44.713072+0000","last_peered":"2024-02-05T07:40:44.713072+0000","last_clean":"2024-02-05T07:40:44.713072+0000","last_became_active":"2024-02-05T07:38:09.495040+0000","last_became_peered":"2024-02-05T07:38:09.495040+0000","last_unstale":"2024-02-05T07:40:44.713072+0000","last_undegraded":"2024-02-05T07:40:44.713072+0000","last_fullsized":"2024-02-05T07:40:44.713072+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:53.396795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:39:53.396795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:06.394109+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"584'12","reported_seq":234,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.835604+0000","last_change":"2024-02-05T07:40:05.765708+0000","last_active":"2024-02-05T07:40:46.835604+0000","last_peered":"2024-02-05T07:40:46.835604+0000","last_clean":"2024-02-05T07:40:46.835604+0000","last_became_active":"2024-02-05T07:37:58.781263+0000","last_became_peered":"2024-02-05T07:37:58.781263+0000","last_unstale":"2024-02-05T07:40:46.835604+0000","last_undegraded":"2024-02-05T07:40:46.835604+0000","last_fullsized":"2024-02-05T07:40:46.835604+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'12","last_scrub_stamp":"2024-02-05T07:40:05.765574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:40:05.765574+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:13.477481+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1019,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.836038+0000","last_change":"2024-02-05T07:39:01.508990+0000","last_active":"2024-02-05T07:40:46.836038+0000","last_peered":"2024-02-05T07:40:46.836038+0000","last_clean":"2024-02-05T07:40:46.836038+0000","last_became_active":"2024-02-05T07:38:09.487782+0000","last_became_peered":"2024-02-05T07:38:09.487782+0000","last_unstale":"2024-02-05T07:40:46.836038+0000","last_undegraded":"2024-02-05T07:40:46.836038+0000","last_fullsized":"2024-02-05T07:40:46.836038+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:01.508866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:01.508866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.16","version":"86'128","reported_seq":480,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821201+0000","last_change":"2024-02-05T07:38:48.574887+0000","last_active":"2024-02-05T07:40:43.821201+0000","last_peered":"2024-02-05T07:40:43.821201+0000","last_clean":"2024-02-05T07:40:43.821201+0000","last_became_active":"2024-02-05T07:38:47.704946+0000","last_became_peered":"2024-02-05T07:38:47.704946+0000","last_unstale":"2024-02-05T07:40:43.821201+0000","last_undegraded":"2024-02-05T07:40:43.821201+0000","last_fullsized":"2024-02-05T07:40:43.821201+0000","mapping_epoch":612,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:48.574781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:48.574781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"615'27","reported_seq":803,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821216+0000","last_change":"2024-02-05T07:38:09.532591+0000","last_active":"2024-02-05T07:40:43.821216+0000","last_peered":"2024-02-05T07:40:43.821216+0000","last_clean":"2024-02-05T07:40:43.821216+0000","last_became_active":"2024-02-05T07:38:09.484252+0000","last_became_peered":"2024-02-05T07:38:09.484252+0000","last_unstale":"2024-02-05T07:40:43.821216+0000","last_undegraded":"2024-02-05T07:40:43.821216+0000","last_fullsized":"2024-02-05T07:40:43.821216+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:37:48.219951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:48.219951+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770370+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1184,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821668+0000","last_change":"2024-02-05T07:38:32.811706+0000","last_active":"2024-02-05T07:40:43.821668+0000","last_peered":"2024-02-05T07:40:43.821668+0000","last_clean":"2024-02-05T07:40:43.821668+0000","last_became_active":"2024-02-05T07:38:09.472022+0000","last_became_peered":"2024-02-05T07:38:09.472022+0000","last_unstale":"2024-02-05T07:40:43.821668+0000","last_undegraded":"2024-02-05T07:40:43.821668+0000","last_fullsized":"2024-02-05T07:40:43.821668+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:32.811595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:38:32.811595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770688+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1286,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710167+0000","last_change":"2024-02-05T07:39:55.907631+0000","last_active":"2024-02-05T07:40:44.710167+0000","last_peered":"2024-02-05T07:40:44.710167+0000","last_clean":"2024-02-05T07:40:44.710167+0000","last_became_active":"2024-02-05T07:30:40.487141+0000","last_became_peered":"2024-02-05T07:30:40.487141+0000","last_unstale":"2024-02-05T07:40:44.710167+0000","last_undegraded":"2024-02-05T07:40:44.710167+0000","last_fullsized":"2024-02-05T07:40:44.710167+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:55.907516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_clean_scrub_stamp":"2024-02-05T07:39:55.907516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:23.259237+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":858,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.832533+0000","last_change":"2024-02-05T07:37:54.469820+0000","last_active":"2024-02-05T07:40:46.832533+0000","last_peered":"2024-02-05T07:40:46.832533+0000","last_clean":"2024-02-05T07:40:46.832533+0000","last_became_active":"2024-02-05T07:36:32.742150+0000","last_became_peered":"2024-02-05T07:36:32.742150+0000","last_unstale":"2024-02-05T07:40:46.832533+0000","last_undegraded":"2024-02-05T07:40:46.832533+0000","last_fullsized":"2024-02-05T07:40:46.832533+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:54.469701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.469701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"86'128","reported_seq":1272,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.822431+0000","last_change":"2024-02-05T07:38:47.700603+0000","last_active":"2024-02-05T07:40:43.822431+0000","last_peered":"2024-02-05T07:40:43.822431+0000","last_clean":"2024-02-05T07:40:43.822431+0000","last_became_active":"2024-02-05T07:38:47.700371+0000","last_became_peered":"2024-02-05T07:38:47.700371+0000","last_unstale":"2024-02-05T07:40:43.822431+0000","last_undegraded":"2024-02-05T07:40:43.822431+0000","last_fullsized":"2024-02-05T07:40:43.822431+0000","mapping_epoch":612,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:34.751313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:34.751313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"615'21","reported_seq":932,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.709561+0000","last_change":"2024-02-05T07:38:37.758235+0000","last_active":"2024-02-05T07:40:44.709561+0000","last_peered":"2024-02-05T07:40:44.709561+0000","last_clean":"2024-02-05T07:40:44.709561+0000","last_became_active":"2024-02-05T07:36:32.710090+0000","last_became_peered":"2024-02-05T07:36:32.710090+0000","last_unstale":"2024-02-05T07:40:44.709561+0000","last_undegraded":"2024-02-05T07:40:44.709561+0000","last_fullsized":"2024-02-05T07:40:44.709561+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:38:37.758114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:37.758114+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.376408+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1068,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.836298+0000","last_change":"2024-02-05T07:38:09.473515+0000","last_active":"2024-02-05T07:40:46.836298+0000","last_peered":"2024-02-05T07:40:46.836298+0000","last_clean":"2024-02-05T07:40:46.836298+0000","last_became_active":"2024-02-05T07:38:09.472668+0000","last_became_peered":"2024-02-05T07:38:09.472668+0000","last_unstale":"2024-02-05T07:40:46.836298+0000","last_undegraded":"2024-02-05T07:40:46.836298+0000","last_fullsized":"2024-02-05T07:40:46.836298+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:32.120410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:32.120410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":848,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.846134+0000","last_change":"2024-02-05T07:38:54.397360+0000","last_active":"2024-02-05T07:40:46.846134+0000","last_peered":"2024-02-05T07:40:46.846134+0000","last_clean":"2024-02-05T07:40:46.846134+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:40:46.846134+0000","last_undegraded":"2024-02-05T07:40:46.846134+0000","last_fullsized":"2024-02-05T07:40:46.846134+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:54.397113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:38:54.397113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":728,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821819+0000","last_change":"2024-02-05T07:40:01.732873+0000","last_active":"2024-02-05T07:40:43.821819+0000","last_peered":"2024-02-05T07:40:43.821819+0000","last_clean":"2024-02-05T07:40:43.821819+0000","last_became_active":"2024-02-05T07:34:30.670179+0000","last_became_peered":"2024-02-05T07:34:30.670179+0000","last_unstale":"2024-02-05T07:40:43.821819+0000","last_undegraded":"2024-02-05T07:40:43.821819+0000","last_fullsized":"2024-02-05T07:40:43.821819+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:01.732803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:40:01.732803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:29.909417+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":1637,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.712028+0000","last_change":"2024-02-05T07:39:11.702922+0000","last_active":"2024-02-05T07:40:44.712028+0000","last_peered":"2024-02-05T07:40:44.712028+0000","last_clean":"2024-02-05T07:40:44.712028+0000","last_became_active":"2024-02-05T07:39:11.702286+0000","last_became_peered":"2024-02-05T07:39:11.702286+0000","last_unstale":"2024-02-05T07:40:44.712028+0000","last_undegraded":"2024-02-05T07:40:44.712028+0000","last_fullsized":"2024-02-05T07:40:44.712028+0000","mapping_epoch":619,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.395638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.395638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"616'27","reported_seq":905,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.832153+0000","last_change":"2024-02-05T07:36:48.088544+0000","last_active":"2024-02-05T07:40:46.832153+0000","last_peered":"2024-02-05T07:40:46.832153+0000","last_clean":"2024-02-05T07:40:46.832153+0000","last_became_active":"2024-02-05T07:30:40.501893+0000","last_became_peered":"2024-02-05T07:30:40.501893+0000","last_unstale":"2024-02-05T07:40:46.832153+0000","last_undegraded":"2024-02-05T07:40:46.832153+0000","last_fullsized":"2024-02-05T07:40:46.832153+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:36:48.088406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:36:48.088406+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":679,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.713230+0000","last_change":"2024-02-05T07:38:54.868354+0000","last_active":"2024-02-05T07:40:44.713230+0000","last_peered":"2024-02-05T07:40:44.713230+0000","last_clean":"2024-02-05T07:40:44.713230+0000","last_became_active":"2024-02-05T07:36:32.712612+0000","last_became_peered":"2024-02-05T07:36:32.712612+0000","last_unstale":"2024-02-05T07:40:44.713230+0000","last_undegraded":"2024-02-05T07:40:44.713230+0000","last_fullsized":"2024-02-05T07:40:44.713230+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:54.867940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:54.867940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.376637+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":682,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.820639+0000","last_change":"2024-02-05T07:38:59.527516+0000","last_active":"2024-02-05T07:40:43.820639+0000","last_peered":"2024-02-05T07:40:43.820639+0000","last_clean":"2024-02-05T07:40:43.820639+0000","last_became_active":"2024-02-05T07:38:09.480566+0000","last_became_peered":"2024-02-05T07:38:09.480566+0000","last_unstale":"2024-02-05T07:40:43.820639+0000","last_undegraded":"2024-02-05T07:40:43.820639+0000","last_fullsized":"2024-02-05T07:40:43.820639+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:59.527436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:38:59.527436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770921+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":860,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.845275+0000","last_change":"2024-02-05T07:40:20.094896+0000","last_active":"2024-02-05T07:40:46.845275+0000","last_peered":"2024-02-05T07:40:46.845275+0000","last_clean":"2024-02-05T07:40:46.845275+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:40:46.845275+0000","last_undegraded":"2024-02-05T07:40:46.845275+0000","last_fullsized":"2024-02-05T07:40:46.845275+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:20.094078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:40:20.094078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:33.467712+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"86'128","reported_seq":1258,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710670+0000","last_change":"2024-02-05T07:39:46.502942+0000","last_active":"2024-02-05T07:40:44.710670+0000","last_peered":"2024-02-05T07:40:44.710670+0000","last_clean":"2024-02-05T07:40:44.710670+0000","last_became_active":"2024-02-05T07:39:17.741563+0000","last_became_peered":"2024-02-05T07:39:17.741563+0000","last_unstale":"2024-02-05T07:40:44.710670+0000","last_undegraded":"2024-02-05T07:40:44.710670+0000","last_fullsized":"2024-02-05T07:40:44.710670+0000","mapping_epoch":623,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:39:46.502804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:39:46.502804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:50.763967+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"615'12","reported_seq":870,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.848339+0000","last_change":"2024-02-05T07:35:47.070886+0000","last_active":"2024-02-05T07:40:46.848339+0000","last_peered":"2024-02-05T07:40:46.848339+0000","last_clean":"2024-02-05T07:40:46.848339+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:40:46.848339+0000","last_undegraded":"2024-02-05T07:40:46.848339+0000","last_fullsized":"2024-02-05T07:40:46.848339+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:35:47.070805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:47.070805+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":859,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.834288+0000","last_change":"2024-02-05T07:40:07.758232+0000","last_active":"2024-02-05T07:40:46.834288+0000","last_peered":"2024-02-05T07:40:46.834288+0000","last_clean":"2024-02-05T07:40:46.834288+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:40:46.834288+0000","last_undegraded":"2024-02-05T07:40:46.834288+0000","last_fullsized":"2024-02-05T07:40:46.834288+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:07.758093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:34:39.980067+0000","last_clean_scrub_stamp":"2024-02-05T07:40:07.758093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:33.830081+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1123,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821441+0000","last_change":"2024-02-05T07:40:02.708852+0000","last_active":"2024-02-05T07:40:43.821441+0000","last_peered":"2024-02-05T07:40:43.821441+0000","last_clean":"2024-02-05T07:40:43.821441+0000","last_became_active":"2024-02-05T07:38:09.464446+0000","last_became_peered":"2024-02-05T07:38:09.464446+0000","last_unstale":"2024-02-05T07:40:43.821441+0000","last_undegraded":"2024-02-05T07:40:43.821441+0000","last_fullsized":"2024-02-05T07:40:43.821441+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:02.708785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:40:02.708785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:06.491220+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":294,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.711078+0000","last_change":"2024-02-05T07:38:09.504998+0000","last_active":"2024-02-05T07:40:44.711078+0000","last_peered":"2024-02-05T07:40:44.711078+0000","last_clean":"2024-02-05T07:40:44.711078+0000","last_became_active":"2024-02-05T07:38:09.503369+0000","last_became_peered":"2024-02-05T07:38:09.503369+0000","last_unstale":"2024-02-05T07:40:44.711078+0000","last_undegraded":"2024-02-05T07:40:44.711078+0000","last_fullsized":"2024-02-05T07:40:44.711078+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:58.296469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:58.296469+0000","last_clean_scrub_stamp":"2024-02-05T07:36:58.296469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.618501+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"616'27","reported_seq":1115,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.822617+0000","last_change":"2024-02-05T07:38:55.551835+0000","last_active":"2024-02-05T07:40:43.822617+0000","last_peered":"2024-02-05T07:40:43.822617+0000","last_clean":"2024-02-05T07:40:43.822617+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:40:43.822617+0000","last_undegraded":"2024-02-05T07:40:43.822617+0000","last_fullsized":"2024-02-05T07:40:43.822617+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:38:55.551751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.551751+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770907+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":856,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.847958+0000","last_change":"2024-02-05T07:40:11.850388+0000","last_active":"2024-02-05T07:40:46.847958+0000","last_peered":"2024-02-05T07:40:46.847958+0000","last_clean":"2024-02-05T07:40:46.847958+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:40:46.847958+0000","last_undegraded":"2024-02-05T07:40:46.847958+0000","last_fullsized":"2024-02-05T07:40:46.847958+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:11.849558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:40:11.849558+0000","last_clean_scrub_stamp":"2024-02-05T07:40:11.849558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:35.605390+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":297,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.711547+0000","last_change":"2024-02-05T07:40:15.407981+0000","last_active":"2024-02-05T07:40:44.711547+0000","last_peered":"2024-02-05T07:40:44.711547+0000","last_clean":"2024-02-05T07:40:44.711547+0000","last_became_active":"2024-02-05T07:36:32.752879+0000","last_became_peered":"2024-02-05T07:36:32.752879+0000","last_unstale":"2024-02-05T07:40:44.711547+0000","last_undegraded":"2024-02-05T07:40:44.711547+0000","last_fullsized":"2024-02-05T07:40:44.711547+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:15.407749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:40:15.407749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:41.593247+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":783,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710573+0000","last_change":"2024-02-05T07:40:20.624459+0000","last_active":"2024-02-05T07:40:44.710573+0000","last_peered":"2024-02-05T07:40:44.710573+0000","last_clean":"2024-02-05T07:40:44.710573+0000","last_became_active":"2024-02-05T07:38:09.503835+0000","last_became_peered":"2024-02-05T07:38:09.503835+0000","last_unstale":"2024-02-05T07:40:44.710573+0000","last_undegraded":"2024-02-05T07:40:44.710573+0000","last_fullsized":"2024-02-05T07:40:44.710573+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:20.624132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:40:20.624132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:28.219914+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"616'30","reported_seq":1031,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.836277+0000","last_change":"2024-02-05T07:35:13.094823+0000","last_active":"2024-02-05T07:40:46.836277+0000","last_peered":"2024-02-05T07:40:46.836277+0000","last_clean":"2024-02-05T07:40:46.836277+0000","last_became_active":"2024-02-05T07:30:40.495954+0000","last_became_peered":"2024-02-05T07:30:40.495954+0000","last_unstale":"2024-02-05T07:40:46.836277+0000","last_undegraded":"2024-02-05T07:40:46.836277+0000","last_fullsized":"2024-02-05T07:40:46.836277+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:35:13.094694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:13.094694+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"585'2","reported_seq":149,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.704981+0000","last_change":"2024-02-05T07:38:31.834917+0000","last_active":"2024-02-05T07:40:44.704981+0000","last_peered":"2024-02-05T07:40:44.704981+0000","last_clean":"2024-02-05T07:40:44.704981+0000","last_became_active":"2024-02-05T07:38:31.811200+0000","last_became_peered":"2024-02-05T07:38:31.811200+0000","last_unstale":"2024-02-05T07:40:44.704981+0000","last_undegraded":"2024-02-05T07:40:44.704981+0000","last_fullsized":"2024-02-05T07:40:44.704981+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'2","last_scrub_stamp":"2024-02-05T07:38:00.631235+0000","last_deep_scrub":"585'2","last_deep_scrub_stamp":"2024-02-05T07:38:00.631235+0000","last_clean_scrub_stamp":"2024-02-05T07:38:00.631235+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.621873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"589'14","reported_seq":1385,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.822730+0000","last_change":"2024-02-05T07:38:09.524876+0000","last_active":"2024-02-05T07:40:43.822730+0000","last_peered":"2024-02-05T07:40:43.822730+0000","last_clean":"2024-02-05T07:40:43.822730+0000","last_became_active":"2024-02-05T07:38:09.472910+0000","last_became_peered":"2024-02-05T07:38:09.472910+0000","last_unstale":"2024-02-05T07:40:43.822730+0000","last_undegraded":"2024-02-05T07:40:43.822730+0000","last_fullsized":"2024-02-05T07:40:43.822730+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'13","last_scrub_stamp":"2024-02-05T07:36:57.487913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:57.487913+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770288+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":302,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.712625+0000","last_change":"2024-02-05T07:40:09.332550+0000","last_active":"2024-02-05T07:40:44.712625+0000","last_peered":"2024-02-05T07:40:44.712625+0000","last_clean":"2024-02-05T07:40:44.712625+0000","last_became_active":"2024-02-05T07:38:09.482346+0000","last_became_peered":"2024-02-05T07:38:09.482346+0000","last_unstale":"2024-02-05T07:40:44.712625+0000","last_undegraded":"2024-02-05T07:40:44.712625+0000","last_fullsized":"2024-02-05T07:40:44.712625+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:40:09.332235+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:40:09.332235+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:35.166123+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"616'29","reported_seq":909,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.835762+0000","last_change":"2024-02-05T07:40:06.764929+0000","last_active":"2024-02-05T07:40:46.835762+0000","last_peered":"2024-02-05T07:40:46.835762+0000","last_clean":"2024-02-05T07:40:46.835762+0000","last_became_active":"2024-02-05T07:38:09.486225+0000","last_became_peered":"2024-02-05T07:38:09.486225+0000","last_unstale":"2024-02-05T07:40:46.835762+0000","last_undegraded":"2024-02-05T07:40:46.835762+0000","last_fullsized":"2024-02-05T07:40:46.835762+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'29","last_scrub_stamp":"2024-02-05T07:40:06.764791+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:40:06.764791+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:23.569118+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":294,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.711169+0000","last_change":"2024-02-05T07:38:09.489856+0000","last_active":"2024-02-05T07:40:44.711169+0000","last_peered":"2024-02-05T07:40:44.711169+0000","last_clean":"2024-02-05T07:40:44.711169+0000","last_became_active":"2024-02-05T07:38:09.489555+0000","last_became_peered":"2024-02-05T07:38:09.489555+0000","last_unstale":"2024-02-05T07:40:44.711169+0000","last_undegraded":"2024-02-05T07:40:44.711169+0000","last_fullsized":"2024-02-05T07:40:44.711169+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:59.269484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.269484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.618660+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1174,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.835011+0000","last_change":"2024-02-05T07:38:09.484586+0000","last_active":"2024-02-05T07:40:46.835011+0000","last_peered":"2024-02-05T07:40:46.835011+0000","last_clean":"2024-02-05T07:40:46.835011+0000","last_became_active":"2024-02-05T07:38:09.484261+0000","last_became_peered":"2024-02-05T07:38:09.484261+0000","last_unstale":"2024-02-05T07:40:46.835011+0000","last_undegraded":"2024-02-05T07:40:46.835011+0000","last_fullsized":"2024-02-05T07:40:46.835011+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:47.437414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:47.437414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":246,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.822563+0000","last_change":"2024-02-05T07:38:35.940570+0000","last_active":"2024-02-05T07:40:43.822563+0000","last_peered":"2024-02-05T07:40:43.822563+0000","last_clean":"2024-02-05T07:40:43.822563+0000","last_became_active":"2024-02-05T07:38:31.812780+0000","last_became_peered":"2024-02-05T07:38:31.812780+0000","last_unstale":"2024-02-05T07:40:43.822563+0000","last_undegraded":"2024-02-05T07:40:43.822563+0000","last_fullsized":"2024-02-05T07:40:43.822563+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:35.940472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:35.940472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770763+0000","scrub_duration":230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"615'39","reported_seq":322,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.712271+0000","last_change":"2024-02-05T07:37:57.440904+0000","last_active":"2024-02-05T07:40:44.712271+0000","last_peered":"2024-02-05T07:40:44.712271+0000","last_clean":"2024-02-05T07:40:44.712271+0000","last_became_active":"2024-02-05T07:36:32.744301+0000","last_became_peered":"2024-02-05T07:36:32.744301+0000","last_unstale":"2024-02-05T07:40:44.712271+0000","last_undegraded":"2024-02-05T07:40:44.712271+0000","last_fullsized":"2024-02-05T07:40:44.712271+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:37:57.440806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:57.440806+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.621782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":222,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.712908+0000","last_change":"2024-02-05T07:38:09.501225+0000","last_active":"2024-02-05T07:40:44.712908+0000","last_peered":"2024-02-05T07:40:44.712908+0000","last_clean":"2024-02-05T07:40:44.712908+0000","last_became_active":"2024-02-05T07:38:09.500871+0000","last_became_peered":"2024-02-05T07:38:09.500871+0000","last_unstale":"2024-02-05T07:40:44.712908+0000","last_undegraded":"2024-02-05T07:40:44.712908+0000","last_fullsized":"2024-02-05T07:40:44.712908+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:48.410161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:48.410161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.618465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1269,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.708726+0000","last_change":"2024-02-05T07:38:53.859910+0000","last_active":"2024-02-05T07:40:44.708726+0000","last_peered":"2024-02-05T07:40:44.708726+0000","last_clean":"2024-02-05T07:40:44.708726+0000","last_became_active":"2024-02-05T07:38:09.477634+0000","last_became_peered":"2024-02-05T07:38:09.477634+0000","last_unstale":"2024-02-05T07:40:44.708726+0000","last_undegraded":"2024-02-05T07:40:44.708726+0000","last_fullsized":"2024-02-05T07:40:44.708726+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:53.859782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:53.859782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.376695+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":154,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710089+0000","last_change":"2024-02-05T07:40:18.398849+0000","last_active":"2024-02-05T07:40:44.710089+0000","last_peered":"2024-02-05T07:40:44.710089+0000","last_clean":"2024-02-05T07:40:44.710089+0000","last_became_active":"2024-02-05T07:38:31.795971+0000","last_became_peered":"2024-02-05T07:38:31.795971+0000","last_unstale":"2024-02-05T07:40:44.710089+0000","last_undegraded":"2024-02-05T07:40:44.710089+0000","last_fullsized":"2024-02-05T07:40:44.710089+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:18.398592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:40:18.398592+0000","last_clean_scrub_stamp":"2024-02-05T07:40:18.398592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:45.308821+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"615'9","reported_seq":771,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.845835+0000","last_change":"2024-02-05T07:38:09.527356+0000","last_active":"2024-02-05T07:40:46.845835+0000","last_peered":"2024-02-05T07:40:46.845835+0000","last_clean":"2024-02-05T07:40:46.845835+0000","last_became_active":"2024-02-05T07:38:09.496642+0000","last_became_peered":"2024-02-05T07:38:09.496642+0000","last_unstale":"2024-02-05T07:40:46.845835+0000","last_undegraded":"2024-02-05T07:40:46.845835+0000","last_fullsized":"2024-02-05T07:40:46.845835+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'7","last_scrub_stamp":"2024-02-05T07:37:06.495165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:06.495165+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":505,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.710933+0000","last_change":"2024-02-05T07:38:47.813224+0000","last_active":"2024-02-05T07:40:44.710933+0000","last_peered":"2024-02-05T07:40:44.710933+0000","last_clean":"2024-02-05T07:40:44.710933+0000","last_became_active":"2024-02-05T07:38:09.495573+0000","last_became_peered":"2024-02-05T07:38:09.495573+0000","last_unstale":"2024-02-05T07:40:44.710933+0000","last_undegraded":"2024-02-05T07:40:44.710933+0000","last_fullsized":"2024-02-05T07:40:44.710933+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:47.812860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:47.812860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.376493+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":186,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.709969+0000","last_change":"2024-02-05T07:39:45.831179+0000","last_active":"2024-02-05T07:40:44.709969+0000","last_peered":"2024-02-05T07:40:44.709969+0000","last_clean":"2024-02-05T07:40:44.709969+0000","last_became_active":"2024-02-05T07:38:09.462036+0000","last_became_peered":"2024-02-05T07:38:09.462036+0000","last_unstale":"2024-02-05T07:40:44.709969+0000","last_undegraded":"2024-02-05T07:40:44.709969+0000","last_fullsized":"2024-02-05T07:40:44.709969+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:45.831067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:45.831067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:57.942891+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1082,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.708429+0000","last_change":"2024-02-05T07:38:33.873655+0000","last_active":"2024-02-05T07:40:44.708429+0000","last_peered":"2024-02-05T07:40:44.708429+0000","last_clean":"2024-02-05T07:40:44.708429+0000","last_became_active":"2024-02-05T07:36:32.710575+0000","last_became_peered":"2024-02-05T07:36:32.710575+0000","last_unstale":"2024-02-05T07:40:44.708429+0000","last_undegraded":"2024-02-05T07:40:44.708429+0000","last_fullsized":"2024-02-05T07:40:44.708429+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:33.873524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:33.873524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.376519+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"615'25","reported_seq":805,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.845523+0000","last_change":"2024-02-05T07:38:09.506020+0000","last_active":"2024-02-05T07:40:46.845523+0000","last_peered":"2024-02-05T07:40:46.845523+0000","last_clean":"2024-02-05T07:40:46.845523+0000","last_became_active":"2024-02-05T07:38:09.479937+0000","last_became_peered":"2024-02-05T07:38:09.479937+0000","last_unstale":"2024-02-05T07:40:46.845523+0000","last_undegraded":"2024-02-05T07:40:46.845523+0000","last_fullsized":"2024-02-05T07:40:46.845523+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'19","last_scrub_stamp":"2024-02-05T07:35:18.020665+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:35:18.020665+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":506,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.822514+0000","last_change":"2024-02-05T07:38:56.526617+0000","last_active":"2024-02-05T07:40:43.822514+0000","last_peered":"2024-02-05T07:40:43.822514+0000","last_clean":"2024-02-05T07:40:43.822514+0000","last_became_active":"2024-02-05T07:38:09.484875+0000","last_became_peered":"2024-02-05T07:38:09.484875+0000","last_unstale":"2024-02-05T07:40:43.822514+0000","last_undegraded":"2024-02-05T07:40:43.822514+0000","last_fullsized":"2024-02-05T07:40:43.822514+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:56.526522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:56.526522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770891+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":186,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821013+0000","last_change":"2024-02-05T07:38:47.564845+0000","last_active":"2024-02-05T07:40:43.821013+0000","last_peered":"2024-02-05T07:40:43.821013+0000","last_clean":"2024-02-05T07:40:43.821013+0000","last_became_active":"2024-02-05T07:38:09.471013+0000","last_became_peered":"2024-02-05T07:38:09.471013+0000","last_unstale":"2024-02-05T07:40:43.821013+0000","last_undegraded":"2024-02-05T07:40:43.821013+0000","last_fullsized":"2024-02-05T07:40:43.821013+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:47.564750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:47.564750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770874+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":288,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.711460+0000","last_change":"2024-02-05T07:38:51.521757+0000","last_active":"2024-02-05T07:40:44.711460+0000","last_peered":"2024-02-05T07:40:44.711460+0000","last_clean":"2024-02-05T07:40:44.711460+0000","last_became_active":"2024-02-05T07:36:32.733405+0000","last_became_peered":"2024-02-05T07:36:32.733405+0000","last_unstale":"2024-02-05T07:40:44.711460+0000","last_undegraded":"2024-02-05T07:40:44.711460+0000","last_fullsized":"2024-02-05T07:40:44.711460+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:51.521494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:51.521494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.624442+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"616'39","reported_seq":944,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.831946+0000","last_change":"2024-02-05T07:37:50.422750+0000","last_active":"2024-02-05T07:40:46.831946+0000","last_peered":"2024-02-05T07:40:46.831946+0000","last_clean":"2024-02-05T07:40:46.831946+0000","last_became_active":"2024-02-05T07:36:32.741691+0000","last_became_peered":"2024-02-05T07:36:32.741691+0000","last_unstale":"2024-02-05T07:40:46.831946+0000","last_undegraded":"2024-02-05T07:40:46.831946+0000","last_fullsized":"2024-02-05T07:40:46.831946+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:37:50.422661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:50.422661+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":865,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.835739+0000","last_change":"2024-02-05T07:37:44.381379+0000","last_active":"2024-02-05T07:40:46.835739+0000","last_peered":"2024-02-05T07:40:46.835739+0000","last_clean":"2024-02-05T07:40:46.835739+0000","last_became_active":"2024-02-05T07:36:32.746731+0000","last_became_peered":"2024-02-05T07:36:32.746731+0000","last_unstale":"2024-02-05T07:40:46.835739+0000","last_undegraded":"2024-02-05T07:40:46.835739+0000","last_fullsized":"2024-02-05T07:40:46.835739+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:44.381300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:37:44.381300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":670,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.820735+0000","last_change":"2024-02-05T07:38:09.473135+0000","last_active":"2024-02-05T07:40:43.820735+0000","last_peered":"2024-02-05T07:40:43.820735+0000","last_clean":"2024-02-05T07:40:43.820735+0000","last_became_active":"2024-02-05T07:38:09.472733+0000","last_became_peered":"2024-02-05T07:38:09.472733+0000","last_unstale":"2024-02-05T07:40:43.820735+0000","last_undegraded":"2024-02-05T07:40:43.820735+0000","last_fullsized":"2024-02-05T07:40:43.820735+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:45.170539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.170539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":866,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-02-05T07:40:46.846048+0000","last_change":"2024-02-05T07:38:58.711149+0000","last_active":"2024-02-05T07:40:46.846048+0000","last_peered":"2024-02-05T07:40:46.846048+0000","last_clean":"2024-02-05T07:40:46.846048+0000","last_became_active":"2024-02-05T07:30:40.493086+0000","last_became_peered":"2024-02-05T07:30:40.493086+0000","last_unstale":"2024-02-05T07:40:46.846048+0000","last_undegraded":"2024-02-05T07:40:46.846048+0000","last_fullsized":"2024-02-05T07:40:46.846048+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:58.711041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:38:58.711041+0000","last_clean_scrub_stamp":"2024-02-05T07:38:58.711041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"615'25","reported_seq":1045,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.821214+0000","last_change":"2024-02-05T07:39:43.728715+0000","last_active":"2024-02-05T07:40:43.821214+0000","last_peered":"2024-02-05T07:40:43.821214+0000","last_clean":"2024-02-05T07:40:43.821214+0000","last_became_active":"2024-02-05T07:38:09.470878+0000","last_became_peered":"2024-02-05T07:38:09.470878+0000","last_unstale":"2024-02-05T07:40:43.821214+0000","last_undegraded":"2024-02-05T07:40:43.821214+0000","last_fullsized":"2024-02-05T07:40:43.821214+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'25","last_scrub_stamp":"2024-02-05T07:39:43.728551+0000","last_deep_scrub":"450'19","last_deep_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_clean_scrub_stamp":"2024-02-05T07:39:43.728551+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:01.829731+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":289,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.714632+0000","last_change":"2024-02-05T07:38:47.508209+0000","last_active":"2024-02-05T07:40:44.714632+0000","last_peered":"2024-02-05T07:40:44.714632+0000","last_clean":"2024-02-05T07:40:44.714632+0000","last_became_active":"2024-02-05T07:36:32.734502+0000","last_became_peered":"2024-02-05T07:36:32.734502+0000","last_unstale":"2024-02-05T07:40:44.714632+0000","last_undegraded":"2024-02-05T07:40:44.714632+0000","last_fullsized":"2024-02-05T07:40:44.714632+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:47.508088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:38:47.508088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.624246+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":288,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-05T07:40:44.706055+0000","last_change":"2024-02-05T07:37:50.267485+0000","last_active":"2024-02-05T07:40:44.706055+0000","last_peered":"2024-02-05T07:40:44.706055+0000","last_clean":"2024-02-05T07:40:44.706055+0000","last_became_active":"2024-02-05T07:36:32.708917+0000","last_became_peered":"2024-02-05T07:36:32.708917+0000","last_unstale":"2024-02-05T07:40:44.706055+0000","last_undegraded":"2024-02-05T07:40:44.706055+0000","last_fullsized":"2024-02-05T07:40:44.706055+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:50.267364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:50.267364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:47.621721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":796,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-05T07:40:43.820739+0000","last_change":"2024-02-05T07:38:09.480452+0000","last_active":"2024-02-05T07:40:43.820739+0000","last_peered":"2024-02-05T07:40:43.820739+0000","last_clean":"2024-02-05T07:40:43.820739+0000","last_became_active":"2024-02-05T07:38:09.480027+0000","last_became_peered":"2024-02-05T07:38:09.480027+0000","last_unstale":"2024-02-05T07:40:43.820739+0000","last_undegraded":"2024-02-05T07:40:43.820739+0000","last_fullsized":"2024-02-05T07:40:43.820739+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:49.186617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:49.186617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:44.770337+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1741160472,"num_objects":26569,"num_object_clones":0,"num_object_copies":79707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193989,"num_write_kb":7057793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2295,"num_bytes_recovered":150405120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5372588032,"data_stored":5372575816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47738,"ondisk_log_size":47738,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":609,"num_read_kb":604,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":24,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":72,"acting":72,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":4552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1526,"num_read_kb":1283,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":10072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":503808,"data_stored":13460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1052,"ondisk_log_size":1052,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":257698038028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":301,"seq":1292785156245,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2070780,"kb_used_data":1316392,"kb_used_omap":5,"kb_used_meta":754362,"kb_avail":91674372,"statfs":{"total":95995035648,"available":93874556928,"internally_reserved":0,"allocated":1347985408,"data_stored":1346575582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":772467205},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.1319999999999999,"15min":1.339},"min":{"1min":0.28599999999999998,"5min":0.19600000000000001,"15min":0.192},"max":{"1min":4.4210000000000003,"5min":13.976000000000001,"15min":14.218},"last":5.1959999999999997},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.107,"15min":1.3779999999999999},"min":{"1min":0.23200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.7799999999999998,"5min":13.962999999999999,"15min":14.175000000000001},"last":0.58299999999999996}]},{"osd":1,"last update":"Mon Feb 5 07:39:47 2024","interfaces":[{"interface":"back","average":{"1min":4.1779999999999999,"5min":2.2709999999999999,"15min":2.0350000000000001},"min":{"1min":0.26600000000000001,"5min":0.24299999999999999,"15min":0.217},"max":{"1min":56.325000000000003,"5min":56.325000000000003,"15min":56.325000000000003},"last":2.8029999999999999},{"interface":"front","average":{"1min":4.6120000000000001,"5min":2.387,"15min":2.4980000000000002},"min":{"1min":0.252,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":60.115000000000002,"5min":60.115000000000002,"15min":61.103000000000002},"last":0.32600000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:39:47 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.9370000000000001,"15min":2.0190000000000001},"min":{"1min":0.38400000000000001,"5min":0.311,"15min":0.249},"max":{"1min":6.2930000000000001,"5min":46.927,"15min":46.927},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.7310000000000001,"15min":2.2080000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.3239999999999998,"5min":46.506,"15min":46.506},"last":2.294}]},{"osd":3,"last update":"Mon Feb 5 07:39:47 2024","interfaces":[{"interface":"back","average":{"1min":4.4119999999999999,"5min":1.8839999999999999,"15min":2.3820000000000001},"min":{"1min":0.30399999999999999,"5min":0.27300000000000002,"15min":0.22500000000000001},"max":{"1min":56.271000000000001,"5min":56.271000000000001,"15min":56.271000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":4.452,"5min":2.0169999999999999,"15min":2.5379999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":56.235999999999997,"5min":56.235999999999997,"15min":56.235999999999997},"last":1.806}]},{"osd":4,"last update":"Mon Feb 5 07:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.64600000000000002,"15min":1.0820000000000001},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.224},"max":{"1min":0.47399999999999998,"5min":23.155000000000001,"15min":151.02099999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.79000000000000004,"15min":1.103},"min":{"1min":0.26800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.67300000000000004,"5min":24.140000000000001,"15min":149.21799999999999},"last":0.55000000000000004}]},{"osd":5,"last update":"Mon Feb 5 07:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.60599999999999998,"15min":0.57299999999999995},"min":{"1min":0.32700000000000001,"5min":0.29999999999999999,"15min":0.26800000000000002},"max":{"1min":0.71299999999999997,"5min":2.431,"15min":2.8919999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.47899999999999998,"15min":0.51700000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28100000000000003},"max":{"1min":0.53300000000000003,"5min":1.0669999999999999,"15min":1.526},"last":0.56499999999999995}]},{"osd":7,"last update":"Mon Feb 5 07:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.505,"15min":2.3109999999999999},"min":{"1min":0.32200000000000001,"5min":0.315,"15min":0.28000000000000003},"max":{"1min":0.58599999999999997,"5min":2.0550000000000002,"15min":84.158000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.71799999999999997,"15min":2.452},"min":{"1min":0.32400000000000001,"5min":0.317,"15min":0.312},"max":{"1min":0.621,"5min":23.055,"15min":85.018000000000001},"last":0.51500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575146,"num_pgs":112,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2290532,"kb_used_data":1189268,"kb_used_omap":1,"kb_used_meta":1100862,"kb_avail":91454620,"statfs":{"total":95995035648,"available":93649530880,"internally_reserved":0,"allocated":1217810432,"data_stored":1216369868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1127283141},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:40:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1560000000000001,"5min":2.7999999999999998,"15min":1.909},"min":{"1min":0.192,"5min":0.192,"15min":0.16800000000000001},"max":{"1min":15.327,"5min":29.449999999999999,"15min":29.449999999999999},"last":1.286},{"interface":"front","average":{"1min":2.919,"5min":3.028,"15min":2.1909999999999998},"min":{"1min":0.49299999999999999,"5min":0.28299999999999997,"15min":0.20899999999999999},"max":{"1min":7.9260000000000002,"5min":29.268999999999998,"15min":29.268999999999998},"last":1.5900000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:40:35 2024","interfaces":[{"interface":"back","average":{"1min":2.6920000000000002,"5min":3.4300000000000002,"15min":2.3130000000000002},"min":{"1min":0.373,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":6.7949999999999999,"5min":42.823999999999998,"15min":42.823999999999998},"last":1.407},{"interface":"front","average":{"1min":1.6819999999999999,"5min":2.843,"15min":2.0939999999999999},"min":{"1min":0.36199999999999999,"5min":0.26500000000000001,"15min":0.20699999999999999},"max":{"1min":3.984,"5min":42.426000000000002,"15min":42.426000000000002},"last":0.97899999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:40:39 2024","interfaces":[{"interface":"back","average":{"1min":2.3420000000000001,"5min":2.177,"15min":2.2109999999999999},"min":{"1min":0.39600000000000002,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":7.9009999999999998,"5min":9.5519999999999996,"15min":46.777000000000001},"last":5.7080000000000002},{"interface":"front","average":{"1min":1.756,"5min":1.9570000000000001,"15min":2.0550000000000002},"min":{"1min":0.38700000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":7.3899999999999997,"5min":9.5419999999999998,"15min":50.164999999999999},"last":1.0009999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.054,"15min":1.8959999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.0069999999999997,"5min":6.7329999999999997,"15min":89.409000000000006},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.4099999999999999,"15min":1.9019999999999999},"min":{"1min":0.42699999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":5.8979999999999997,"5min":14.629,"15min":89.915000000000006},"last":1.244}]},{"osd":5,"last update":"Mon Feb 5 07:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0509999999999999,"15min":1.7769999999999999},"min":{"1min":0.42099999999999999,"5min":0.32200000000000001,"15min":0.309},"max":{"1min":3.9580000000000002,"5min":6.7610000000000001,"15min":106.672},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.835,"15min":1.998},"min":{"1min":0.53300000000000003,"5min":0.34799999999999998,"15min":0.27600000000000002},"max":{"1min":7.8250000000000002,"5min":14.613,"15min":107.348},"last":1.6339999999999999}]},{"osd":6,"last update":"Mon Feb 5 07:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.3520000000000001,"15min":1.361},"min":{"1min":0.36699999999999999,"5min":0.314,"15min":0.29199999999999998},"max":{"1min":5.7990000000000004,"5min":12.685,"15min":12.685},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1319999999999999,"15min":1.49},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":3.9300000000000002,"5min":6.8179999999999996,"15min":8.923},"last":0.89600000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.9530000000000001,"5min":1.71,"15min":1.835},"min":{"1min":0.436,"5min":0.36599999999999999,"15min":0.318},"max":{"1min":7.8719999999999999,"5min":8.2439999999999998,"15min":46.351999999999997},"last":1.667},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.405,"15min":1.387},"min":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.34000000000000002},"max":{"1min":5.8710000000000004,"5min":12.781000000000001,"15min":12.781000000000001},"last":1.5029999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738671,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2145452,"kb_used_data":1049944,"kb_used_omap":1,"kb_used_meta":1095486,"kb_avail":91599700,"statfs":{"total":95995035648,"available":93798092800,"internally_reserved":0,"allocated":1075142656,"data_stored":1073704911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1121778117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:39:50 2024","interfaces":[{"interface":"back","average":{"1min":6.7060000000000004,"5min":3.1669999999999998,"15min":2.6280000000000001},"min":{"1min":0.39000000000000001,"5min":0.189,"15min":0.16500000000000001},"max":{"1min":65.799000000000007,"5min":65.799000000000007,"15min":65.799000000000007},"last":1.9850000000000001},{"interface":"front","average":{"1min":6.2720000000000002,"5min":2.4969999999999999,"15min":2.399},"min":{"1min":0.33900000000000002,"5min":0.192,"15min":0.192},"max":{"1min":65.093999999999994,"5min":65.093999999999994,"15min":65.093999999999994},"last":0.47499999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:40:24 2024","interfaces":[{"interface":"back","average":{"1min":2.032,"5min":2.5619999999999998,"15min":2.044},"min":{"1min":0.24299999999999999,"5min":0.22900000000000001,"15min":0.191},"max":{"1min":5.5949999999999998,"5min":47.137999999999998,"15min":47.137999999999998},"last":1.462},{"interface":"front","average":{"1min":1.379,"5min":2.2930000000000001,"15min":1.897},"min":{"1min":0.25900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.3440000000000003,"5min":47.170999999999999,"15min":47.170999999999999},"last":5.0499999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:40:31 2024","interfaces":[{"interface":"back","average":{"1min":2.883,"5min":2.0579999999999998,"15min":1.823},"min":{"1min":0.433,"5min":0.25600000000000001,"15min":0.21299999999999999},"max":{"1min":22.129000000000001,"5min":26.43,"15min":29.629000000000001},"last":5.9500000000000002},{"interface":"front","average":{"1min":2.6949999999999998,"5min":2.871,"15min":2.2930000000000001},"min":{"1min":0.435,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":22.155999999999999,"5min":25.337,"15min":29.463999999999999},"last":2.9510000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:39:50 2024","interfaces":[{"interface":"back","average":{"1min":2.1720000000000002,"5min":2.113,"15min":1.9890000000000001},"min":{"1min":0.35499999999999998,"5min":0.308,"15min":0.26700000000000002},"max":{"1min":7.8460000000000001,"5min":12.718999999999999,"15min":50.545000000000002},"last":1.9279999999999999},{"interface":"front","average":{"1min":1.573,"5min":1.349,"15min":1.802},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.27100000000000002},"max":{"1min":5.4770000000000003,"5min":5.4770000000000003,"15min":50.310000000000002},"last":0.51400000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.157,"15min":1.419},"min":{"1min":0.374,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":5.3840000000000003,"5min":5.3840000000000003,"15min":95.772999999999996},"last":5.1280000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.268,"15min":1.546},"min":{"1min":0.372,"5min":0.32900000000000001,"15min":0.28199999999999997},"max":{"1min":5.5810000000000004,"5min":5.5810000000000004,"15min":96.028000000000006},"last":0.55700000000000005}]},{"osd":6,"last update":"Mon Feb 5 07:39:48 2024","interfaces":[{"interface":"back","average":{"1min":2.4649999999999999,"5min":2.0800000000000001,"15min":1.4790000000000001},"min":{"1min":0.41999999999999998,"5min":0.27800000000000002,"15min":0.26300000000000001},"max":{"1min":7.7889999999999997,"5min":12.670999999999999,"15min":12.670999999999999},"last":1.883},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.4470000000000001,"15min":1.4219999999999999},"min":{"1min":0.377,"5min":0.376,"15min":0.35099999999999998},"max":{"1min":5.5140000000000002,"5min":5.7229999999999999,"15min":8.718},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:40:10 2024","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":2.0939999999999999,"15min":1.798},"min":{"1min":0.38500000000000001,"5min":0.33500000000000002,"15min":0.28599999999999998},"max":{"1min":7.8200000000000003,"5min":12.766999999999999,"15min":43.012},"last":1.96},{"interface":"front","average":{"1min":1.742,"5min":1.4199999999999999,"15min":1.6459999999999999},"min":{"1min":0.48799999999999999,"5min":0.39100000000000001,"15min":0.309},"max":{"1min":5.641,"5min":5.641,"15min":43.512},"last":0.58499999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411621,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1731664,"kb_used_data":753600,"kb_used_omap":1,"kb_used_meta":977982,"kb_avail":92013488,"statfs":{"total":95995035648,"available":94221811712,"internally_reserved":0,"allocated":771686400,"data_stored":770266414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1001454021},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:40:29 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":2.5169999999999999,"15min":2.1499999999999999},"min":{"1min":0.25,"5min":0.21099999999999999,"15min":0.19600000000000001},"max":{"1min":5.1340000000000003,"5min":80.427000000000007,"15min":92.638000000000005},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.8979999999999999,"5min":2.9199999999999999,"15min":2.427},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.0129999999999999,"5min":80.939999999999998,"15min":93.481999999999999},"last":2.722}]},{"osd":1,"last update":"Mon Feb 5 07:40:29 2024","interfaces":[{"interface":"back","average":{"1min":2.302,"5min":3.9660000000000002,"15min":3.2890000000000001},"min":{"1min":0.63400000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":6.1399999999999997,"5min":110.917,"15min":110.917},"last":1.2350000000000001},{"interface":"front","average":{"1min":2.6240000000000001,"5min":1.7849999999999999,"15min":2.6949999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.20300000000000001},"max":{"1min":5.7309999999999999,"5min":7.9509999999999996,"15min":93.004000000000005},"last":1.1619999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.722,"5min":3.609,"15min":2.9300000000000002},"min":{"1min":0.29199999999999998,"5min":0.214,"15min":0.214},"max":{"1min":5.3339999999999996,"5min":110.773,"15min":110.773},"last":0.80700000000000005},{"interface":"front","average":{"1min":2.532,"5min":2.1509999999999998,"15min":2.2850000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":5.2670000000000003,"5min":47.368000000000002,"15min":63.521000000000001},"last":0.82999999999999996}]},{"osd":4,"last update":"Mon Feb 5 07:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":2.0979999999999999,"15min":2.3690000000000002},"min":{"1min":0.28699999999999998,"5min":0.27700000000000002,"15min":0.24299999999999999},"max":{"1min":2.629,"5min":46.521999999999998,"15min":168.11699999999999},"last":2.6560000000000001},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.5640000000000001,"15min":2.153},"min":{"1min":0.39800000000000002,"5min":0.36599999999999999,"15min":0.27200000000000002},"max":{"1min":2.8359999999999999,"5min":40.329000000000001,"15min":168.589},"last":0.47399999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:40:29 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.0309999999999999,"15min":1.212},"min":{"1min":0.41899999999999998,"5min":0.35699999999999998,"15min":0.315},"max":{"1min":2.8679999999999999,"5min":6.0129999999999999,"15min":39.923000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.456,"5min":2.464,"15min":1.694},"min":{"1min":0.441,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.2329999999999997,"5min":47.761000000000003,"15min":47.761000000000003},"last":1.2010000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:39:51 2024","interfaces":[{"interface":"back","average":{"1min":1.8560000000000001,"5min":3.0619999999999998,"15min":1.782},"min":{"1min":0.40100000000000002,"5min":0.309,"15min":0.28799999999999998},"max":{"1min":5.2750000000000004,"5min":47.726999999999997,"15min":47.726999999999997},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.244,"5min":1.643,"15min":1.3620000000000001},"min":{"1min":0.39600000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":4.5149999999999997,"5min":40.872999999999998,"15min":40.872999999999998},"last":0.51600000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:40:29 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":1.0700000000000001,"15min":1.403},"min":{"1min":0.374,"5min":0.35699999999999998,"15min":0.30499999999999999},"max":{"1min":2.673,"5min":6.4669999999999996,"15min":37.573999999999998},"last":2.694},{"interface":"front","average":{"1min":1.536,"5min":3.5310000000000001,"15min":2.137},"min":{"1min":0.501,"5min":0.40100000000000002,"15min":0.33700000000000002},"max":{"1min":5.3049999999999997,"5min":110.86799999999999,"15min":110.86799999999999},"last":1.0489999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149984,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1979024,"kb_used_data":900864,"kb_used_omap":1,"kb_used_meta":1077950,"kb_avail":91766128,"statfs":{"total":95995035648,"available":93968515072,"internally_reserved":0,"allocated":922484736,"data_stored":921014592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1103821255},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:39:53 2024","interfaces":[{"interface":"back","average":{"1min":1.919,"5min":2.3929999999999998,"15min":1.7170000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.782,"5min":48.688000000000002,"15min":48.688000000000002},"last":4.4349999999999996},{"interface":"front","average":{"1min":2.5619999999999998,"5min":2.968,"15min":2.3199999999999998},"min":{"1min":0.36599999999999999,"5min":0.216,"15min":0.185},"max":{"1min":8.4930000000000003,"5min":48.573,"15min":99.989999999999995},"last":3.8959999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:39:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8250000000000002,"5min":3.069,"15min":2.702},"min":{"1min":0.59099999999999997,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":7.274,"5min":61.941000000000003,"15min":102.20399999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":2.1190000000000002,"5min":2.73,"15min":2.6030000000000002},"min":{"1min":0.29999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.5469999999999997,"5min":61.902000000000001,"15min":102.286},"last":4.5570000000000004}]},{"osd":2,"last update":"Mon Feb 5 07:39:53 2024","interfaces":[{"interface":"back","average":{"1min":2.206,"5min":3.2519999999999998,"15min":2.4729999999999999},"min":{"1min":0.45700000000000002,"5min":0.23999999999999999,"15min":0.22700000000000001},"max":{"1min":6.8200000000000003,"5min":145.94999999999999,"15min":145.94999999999999},"last":4.5140000000000002},{"interface":"front","average":{"1min":1.8799999999999999,"5min":3.4340000000000002,"15min":2.4969999999999999},"min":{"1min":0.32800000000000001,"5min":0.218,"15min":0.20200000000000001},"max":{"1min":7.0110000000000001,"5min":146.006,"15min":146.006},"last":2.2240000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:40:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.3049999999999999,"15min":1.79},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27000000000000002},"max":{"1min":3.2389999999999999,"5min":9.2639999999999993,"15min":100.024},"last":2.2869999999999999},{"interface":"front","average":{"1min":2.0139999999999998,"5min":1.7470000000000001,"15min":2.0830000000000002},"min":{"1min":0.36399999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.3200000000000003,"5min":16.253,"15min":99.859999999999999},"last":0.375}]},{"osd":5,"last update":"Mon Feb 5 07:40:27 2024","interfaces":[{"interface":"back","average":{"1min":2.456,"5min":1.49,"15min":1.5469999999999999},"min":{"1min":0.313,"5min":0.308,"15min":0.308},"max":{"1min":11.507999999999999,"5min":11.507999999999999,"15min":99.911000000000001},"last":4.3979999999999997},{"interface":"front","average":{"1min":1.536,"5min":1.363,"15min":1.591},"min":{"1min":0.34100000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.2789999999999999,"5min":8.4350000000000005,"15min":100.053},"last":2.319}]},{"osd":6,"last update":"Mon Feb 5 07:39:53 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":1.7789999999999999,"15min":1.6519999999999999},"min":{"1min":0.33200000000000002,"5min":0.32700000000000001,"15min":0.29299999999999998},"max":{"1min":4.5830000000000002,"5min":32.505000000000003,"15min":99.807000000000002},"last":4.3490000000000002},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.7969999999999999,"15min":1.7370000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.1020000000000003,"5min":31.084,"15min":101.489},"last":2.2610000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:40:11 2024","interfaces":[{"interface":"back","average":{"1min":1.99,"5min":1.593,"15min":2.387},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.29499999999999998},"max":{"1min":6.3609999999999998,"5min":8.4329999999999998,"15min":99.893000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.794,"5min":1.294,"15min":2.2679999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.6509999999999998,"5min":6.726,"15min":99.879000000000005},"last":4.4829999999999997}]}]},{"osd":4,"up_from":0,"seq":137438953755,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":176093659414,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1056374784,"data_stored":1056374784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1200820224,"data_stored":1200816152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":760352768,"data_stored":760348696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":913571840,"data_stored":913571840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":150011904,"data_stored":150011904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1291456512,"data_stored":1291452440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:40:49.533 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-05T07:40:49.533 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:40:49.534 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-02-05T07:40:49.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:40:49.602 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:49.626 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:49.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:49.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:49.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:49.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:49.813 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:40:49.813 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:40:49.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:49.879 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:49.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:49.980 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:50.004 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:50.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:50.106 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:50.152 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:50.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:50.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:49 smithi007 ceph-mon[29098]: osdmap e674: 8 total, 8 up, 5 in 2024-02-05T07:40:50.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:49 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-05T07:40:50.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:49 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-05T07:40:50.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:49 smithi007 ceph-mon[37158]: osdmap e674: 8 total, 8 up, 5 in 2024-02-05T07:40:50.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:49 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-05T07:40:50.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:49 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-05T07:40:50.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:50.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:49 smithi090 ceph-mon[33213]: osdmap e674: 8 total, 8 up, 5 in 2024-02-05T07:40:50.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:49 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-05T07:40:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:49 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-05T07:40:50.278 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:50.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:50.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:50.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:50.405 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:50.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:50.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:50.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:50.547 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:50.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:50.648 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:50.674 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:50.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:50.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:50.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T07:40:50.776 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:50.800 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:50.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:50.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:50.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:50.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:40:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:50.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:51.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:51.051 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:51.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:51.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:51.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:51.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:50 smithi007 ceph-mon[29098]: pgmap v1232: 170 pgs: 1 peering, 169 active+clean; 1.5 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 64 KiB/s, 0 objects/s recovering 2024-02-05T07:40:51.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:50 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:40:51.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:50 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:40:51.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:50 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-05T07:40:51.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:50 smithi007 ceph-mon[29098]: osdmap e675: 8 total, 8 up, 5 in 2024-02-05T07:40:51.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:50 smithi007 ceph-mon[37158]: pgmap v1232: 170 pgs: 1 peering, 169 active+clean; 1.5 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 64 KiB/s, 0 objects/s recovering 2024-02-05T07:40:51.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:50 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:40:51.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:50 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:40:51.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:50 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-05T07:40:51.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:50 smithi007 ceph-mon[37158]: osdmap e675: 8 total, 8 up, 5 in 2024-02-05T07:40:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:51.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:51.248 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:50 smithi090 ceph-mon[33213]: pgmap v1232: 170 pgs: 1 peering, 169 active+clean; 1.5 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 64 KiB/s, 0 objects/s recovering 2024-02-05T07:40:51.248 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:50 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:40:51.248 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:50 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:40:51.248 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:50 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-05T07:40:51.248 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:50 smithi090 ceph-mon[33213]: osdmap e675: 8 total, 8 up, 5 in 2024-02-05T07:40:51.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:51.304 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:51.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:51.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:51.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:51.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:51.470 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:51.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:51.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:40:51 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:40:51.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:51.641 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:51.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:51.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:51.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:51.782 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:51.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:51.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:51.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:51.926 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:51.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:51.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:52.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:52.052 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:52.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:52.115 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:52.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:51 smithi007 ceph-mon[29098]: osdmap e676: 8 total, 8 up, 5 in 2024-02-05T07:40:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:51 smithi007 ceph-mon[37158]: osdmap e676: 8 total, 8 up, 5 in 2024-02-05T07:40:52.209 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:52.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:52.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:51 smithi090 ceph-mon[33213]: osdmap e676: 8 total, 8 up, 5 in 2024-02-05T07:40:52.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:52.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:52.356 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:52.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:52.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:52.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:52.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:52.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:52.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:52.625 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:52.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:52.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:52.770 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:52.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:52.871 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:52.913 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:52.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:52.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:53.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:53.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:53.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:53.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:53.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:53.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:52 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:53.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:52 smithi007 ceph-mon[29098]: pgmap v1235: 170 pgs: 1 peering, 169 active+clean; 1.4 GiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1k op/s 2024-02-05T07:40:53.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:52 smithi007 ceph-mon[29098]: osdmap e677: 8 total, 8 up, 5 in 2024-02-05T07:40:53.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:52 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:53.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:52 smithi007 ceph-mon[37158]: pgmap v1235: 170 pgs: 1 peering, 169 active+clean; 1.4 GiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1k op/s 2024-02-05T07:40:53.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:52 smithi007 ceph-mon[37158]: osdmap e677: 8 total, 8 up, 5 in 2024-02-05T07:40:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:53.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:53.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:52 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:40:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:52 smithi090 ceph-mon[33213]: pgmap v1235: 170 pgs: 1 peering, 169 active+clean; 1.4 GiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1k op/s 2024-02-05T07:40:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:52 smithi090 ceph-mon[33213]: osdmap e677: 8 total, 8 up, 5 in 2024-02-05T07:40:53.296 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:53.345 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:53.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:53.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:53.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:53.464 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:40:53.471 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:53.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:53.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:53.517 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:40:53.559 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:40:53.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:53.598 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:53.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:53.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:53.605 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:40:53.645 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:40:53.685 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:40:53.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:53.722 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:40:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:40:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:40:53.723 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:40:53.724 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:53.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:53.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:53.766 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:40:53.777 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:40:53.777 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:40:53.777 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:40:53.777 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:40:53.777 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:40:53.778 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:40:53.778 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:40:53.778 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:40:53.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:53.898 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:53.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:53.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:54.040 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:54.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:54.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:54.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:53 smithi007 ceph-mon[29098]: osdmap e678: 8 total, 8 up, 5 in 2024-02-05T07:40:54.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:53 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/834731064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:40:54.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:53 smithi007 ceph-mon[29098]: pgmap v1238: 169 pgs: 1 peering, 168 active+clean; 1.3 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T07:40:54.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:53 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:40:54.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:53 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:40:54.153 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:53 smithi007 ceph-mon[37158]: osdmap e678: 8 total, 8 up, 5 in 2024-02-05T07:40:54.153 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:53 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/834731064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:40:54.153 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:53 smithi007 ceph-mon[37158]: pgmap v1238: 169 pgs: 1 peering, 168 active+clean; 1.3 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T07:40:54.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:53 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:40:54.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:53 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:40:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:54.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:54.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:54.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:53 smithi090 ceph-mon[33213]: osdmap e678: 8 total, 8 up, 5 in 2024-02-05T07:40:54.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:53 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/834731064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:40:54.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:53 smithi090 ceph-mon[33213]: pgmap v1238: 169 pgs: 1 peering, 168 active+clean; 1.3 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T07:40:54.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:53 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:40:54.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:53 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T07:40:54.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:54.314 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:54.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:54.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:54.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:54.455 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:54.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:54.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:54.496 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:40:54.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:54.581 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:54.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:54.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:54.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:54.735 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:54.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:54.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:54.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:54.893 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:54.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:54.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:55.019 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:55.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:55.121 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:55.168 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:55.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:55.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:54 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T07:40:55.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:54 smithi007 ceph-mon[29098]: osdmap e679: 8 total, 8 up, 5 in 2024-02-05T07:40:55.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:54 smithi007 ceph-mon[29098]: osdmap e680: 8 total, 8 up, 5 in 2024-02-05T07:40:55.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:54 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T07:40:55.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:54 smithi007 ceph-mon[37158]: osdmap e679: 8 total, 8 up, 5 in 2024-02-05T07:40:55.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:54 smithi007 ceph-mon[37158]: osdmap e680: 8 total, 8 up, 5 in 2024-02-05T07:40:55.269 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:55.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:54 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T07:40:55.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:54 smithi090 ceph-mon[33213]: osdmap e679: 8 total, 8 up, 5 in 2024-02-05T07:40:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:54 smithi090 ceph-mon[33213]: osdmap e680: 8 total, 8 up, 5 in 2024-02-05T07:40:55.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:55.324 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:55.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:55.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:55.468 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:55.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:55.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:55.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:55.596 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:55.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:55.691 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:40:55.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:55.702 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:40:55.726 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:55.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:55.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:55.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:55.868 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:55.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:55.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:55.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:55.994 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:55.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:55.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:56.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:56.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:56.154 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:56.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:56.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:56.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:56.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:56.424 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:56.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:56.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:56.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:56.564 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:56.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:56.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:56.691 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:56.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:56.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:56.818 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:56.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:56.920 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:56.962 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:56.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:57.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:56 smithi090 ceph-mon[33213]: pgmap v1241: 169 pgs: 169 active+clean; 1.2 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 838 op/s 2024-02-05T07:40:57.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:56 smithi090 ceph-mon[33213]: from='client.15957 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:40:57.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:56 smithi090 ceph-mon[33213]: osdmap e681: 8 total, 8 up, 5 in 2024-02-05T07:40:57.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:57.088 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:57.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:57.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:57.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:56 smithi007 ceph-mon[29098]: pgmap v1241: 169 pgs: 169 active+clean; 1.2 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 838 op/s 2024-02-05T07:40:57.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:56 smithi007 ceph-mon[29098]: from='client.15957 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:40:57.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:56 smithi007 ceph-mon[29098]: osdmap e681: 8 total, 8 up, 5 in 2024-02-05T07:40:57.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:56 smithi007 ceph-mon[37158]: pgmap v1241: 169 pgs: 169 active+clean; 1.2 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 838 op/s 2024-02-05T07:40:57.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:56 smithi007 ceph-mon[37158]: from='client.15957 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:40:57.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:56 smithi007 ceph-mon[37158]: osdmap e681: 8 total, 8 up, 5 in 2024-02-05T07:40:57.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:40:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:57.215 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:57.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:40:57.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:57.357 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:57.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:57.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:57.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:57.484 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:57.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:57.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:57.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:57.611 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:57.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:57.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:57.713 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:40:57.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:57.739 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:57.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:40:57.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:40:57.877 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:57.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:57.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:40:57.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:58.002 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:58.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:58.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:58.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:57 smithi090 ceph-mon[33213]: osdmap e682: 8 total, 8 up, 5 in 2024-02-05T07:40:58.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:58.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:58.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:57 smithi007 ceph-mon[29098]: osdmap e682: 8 total, 8 up, 5 in 2024-02-05T07:40:58.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:57 smithi007 ceph-mon[37158]: osdmap e682: 8 total, 8 up, 5 in 2024-02-05T07:40:58.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:58.272 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:58.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:58.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:58.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:58.417 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:58.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:58.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:58.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T07:40:58.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:58.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:58.569 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:58.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:58.670 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:58.724 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:58.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:58.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:58.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:58.851 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:58.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:58.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:40:58 smithi007 ceph-mon[29098]: pgmap v1244: 168 pgs: 1 peering, 167 active+clean; 926 MiB data, 7.5 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-02-05T07:40:58.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:40:58 smithi007 ceph-mon[37158]: pgmap v1244: 168 pgs: 1 peering, 167 active+clean; 926 MiB data, 7.5 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-02-05T07:40:58.953 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:58.981 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:58.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:59.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:40:58 smithi090 ceph-mon[33213]: pgmap v1244: 168 pgs: 1 peering, 167 active+clean; 926 MiB data, 7.5 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-02-05T07:40:59.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:40:59.122 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:59.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:40:59.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:59.248 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:59.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:59.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:59.349 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:40:59.373 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:59.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:59.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:40:59.475 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:40:59.513 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:59.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:59.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:40:59.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:59.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:59.640 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:59.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:40:59.741 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:40:59.787 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:40:59.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:59.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:40:59.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:40:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:40:59.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:40:59.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:00.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:00.051 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:00.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:00.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:00.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:00.198 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:00.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:00.299 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:00.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:00.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:00.502 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:00.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:00.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:00.509 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1241,"stamp":"2024-02-05T07:40:55.444992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1259673972,"num_objects":19411,"num_object_clones":0,"num_object_copies":58233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19411,"num_whiteouts":0,"num_read":2141,"num_read_kb":1893,"num_write":202353,"num_write_kb":7057838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2799,"num_bytes_recovered":150681091,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49771,"ondisk_log_size":49771,"up":507,"acting":507,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":511,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":8813564,"kb_used_data":3790012,"kb_used_omap":11,"kb_used_meta":5023284,"kb_avail":459912196,"statfs":{"total":479975178240,"available":470950088704,"internally_reserved":0,"allocated":3880972288,"data_stored":3873644609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12005,"internal_metadata":5143843099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-220069888,"num_objects":-3358,"num_object_clones":0,"num_object_copies":-10074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3358,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.002406"},"pg_stats":[{"pgid":"11.3","version":"678'12581","reported_seq":12790,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:53.776867+0000","last_change":"2024-02-05T07:37:52.811299+0000","last_active":"2024-02-05T07:40:53.776867+0000","last_peered":"2024-02-05T07:40:53.776867+0000","last_clean":"2024-02-05T07:40:53.776867+0000","last_became_active":"2024-02-05T07:37:52.811051+0000","last_became_peered":"2024-02-05T07:37:52.811051+0000","last_unstale":"2024-02-05T07:40:53.776867+0000","last_undegraded":"2024-02-05T07:40:53.776867+0000","last_fullsized":"2024-02-05T07:40:53.776867+0000","mapping_epoch":579,"log_start":"639'9620","ondisk_log_start":"639'9620","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2961,"log_dups_size":3000,"ondisk_log_size":2961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74121216,"num_objects":1131,"num_object_clones":0,"num_object_copies":3393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12581,"num_write_kb":438784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":1502,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.897853+0000","last_change":"2024-02-05T07:40:40.778976+0000","last_active":"2024-02-05T07:40:51.897853+0000","last_peered":"2024-02-05T07:40:51.897853+0000","last_clean":"2024-02-05T07:40:51.897853+0000","last_became_active":"2024-02-05T07:40:40.778437+0000","last_became_peered":"2024-02-05T07:40:40.778437+0000","last_unstale":"2024-02-05T07:40:51.897853+0000","last_undegraded":"2024-02-05T07:40:51.897853+0000","last_fullsized":"2024-02-05T07:40:51.897853+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.337196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.337196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"615'35","reported_seq":842,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.890893+0000","last_change":"2024-02-05T07:38:09.523613+0000","last_active":"2024-02-05T07:40:51.890893+0000","last_peered":"2024-02-05T07:40:51.890893+0000","last_clean":"2024-02-05T07:40:51.890893+0000","last_became_active":"2024-02-05T07:38:09.493734+0000","last_became_peered":"2024-02-05T07:38:09.493734+0000","last_unstale":"2024-02-05T07:40:51.890893+0000","last_undegraded":"2024-02-05T07:40:51.890893+0000","last_fullsized":"2024-02-05T07:40:51.890893+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:37:02.567716+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:37:02.567716+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":865,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.901554+0000","last_change":"2024-02-05T07:38:37.381786+0000","last_active":"2024-02-05T07:40:51.901554+0000","last_peered":"2024-02-05T07:40:51.901554+0000","last_clean":"2024-02-05T07:40:51.901554+0000","last_became_active":"2024-02-05T07:38:09.496706+0000","last_became_peered":"2024-02-05T07:38:09.496706+0000","last_unstale":"2024-02-05T07:40:51.901554+0000","last_undegraded":"2024-02-05T07:40:51.901554+0000","last_fullsized":"2024-02-05T07:40:51.901554+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:37.381317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:37.381317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:57.073659+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"538'13","reported_seq":354,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.876638+0000","last_change":"2024-02-05T07:39:03.630111+0000","last_active":"2024-02-05T07:40:49.876638+0000","last_peered":"2024-02-05T07:40:49.876638+0000","last_clean":"2024-02-05T07:40:49.876638+0000","last_became_active":"2024-02-05T07:36:32.743613+0000","last_became_peered":"2024-02-05T07:36:32.743613+0000","last_unstale":"2024-02-05T07:40:49.876638+0000","last_undegraded":"2024-02-05T07:40:49.876638+0000","last_fullsized":"2024-02-05T07:40:49.876638+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'13","last_scrub_stamp":"2024-02-05T07:39:03.629988+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:39:03.629988+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1168,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.888123+0000","last_change":"2024-02-05T07:39:53.910643+0000","last_active":"2024-02-05T07:40:49.888123+0000","last_peered":"2024-02-05T07:40:49.888123+0000","last_clean":"2024-02-05T07:40:49.888123+0000","last_became_active":"2024-02-05T07:38:09.495645+0000","last_became_peered":"2024-02-05T07:38:09.495645+0000","last_unstale":"2024-02-05T07:40:49.888123+0000","last_undegraded":"2024-02-05T07:40:49.888123+0000","last_fullsized":"2024-02-05T07:40:49.888123+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:53.910541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:39:53.910541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:00.438016+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2","version":"675'12471","reported_seq":12674,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:50.838251+0000","last_change":"2024-02-05T07:37:53.089553+0000","last_active":"2024-02-05T07:40:50.838251+0000","last_peered":"2024-02-05T07:40:50.838251+0000","last_clean":"2024-02-05T07:40:50.838251+0000","last_became_active":"2024-02-05T07:37:53.089215+0000","last_became_peered":"2024-02-05T07:37:53.089215+0000","last_unstale":"2024-02-05T07:40:50.838251+0000","last_undegraded":"2024-02-05T07:40:50.838251+0000","last_fullsized":"2024-02-05T07:40:50.838251+0000","mapping_epoch":579,"log_start":"636'9136","ondisk_log_start":"636'9136","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":3335,"log_dups_size":0,"ondisk_log_size":3335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:55.631547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83165184,"num_objects":1269,"num_object_clones":0,"num_object_copies":3807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12471,"num_write_kb":439680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":799,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.874197+0000","last_change":"2024-02-05T07:40:44.724483+0000","last_active":"2024-02-05T07:40:49.874197+0000","last_peered":"2024-02-05T07:40:49.874197+0000","last_clean":"2024-02-05T07:40:49.874197+0000","last_became_active":"2024-02-05T07:40:44.724105+0000","last_became_peered":"2024-02-05T07:40:44.724105+0000","last_unstale":"2024-02-05T07:40:49.874197+0000","last_undegraded":"2024-02-05T07:40:49.874197+0000","last_fullsized":"2024-02-05T07:40:49.874197+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":670,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.389984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.389984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":138,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"616'45","reported_seq":1283,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.876742+0000","last_change":"2024-02-05T07:38:09.530860+0000","last_active":"2024-02-05T07:40:49.876742+0000","last_peered":"2024-02-05T07:40:49.876742+0000","last_clean":"2024-02-05T07:40:49.876742+0000","last_became_active":"2024-02-05T07:38:09.503330+0000","last_became_peered":"2024-02-05T07:38:09.503330+0000","last_unstale":"2024-02-05T07:40:49.876742+0000","last_undegraded":"2024-02-05T07:40:49.876742+0000","last_fullsized":"2024-02-05T07:40:49.876742+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'31","last_scrub_stamp":"2024-02-05T07:37:54.045831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.045831+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1138,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892154+0000","last_change":"2024-02-05T07:38:09.471668+0000","last_active":"2024-02-05T07:40:52.892154+0000","last_peered":"2024-02-05T07:40:52.892154+0000","last_clean":"2024-02-05T07:40:52.892154+0000","last_became_active":"2024-02-05T07:38:09.471049+0000","last_became_peered":"2024-02-05T07:38:09.471049+0000","last_unstale":"2024-02-05T07:40:52.892154+0000","last_undegraded":"2024-02-05T07:40:52.892154+0000","last_fullsized":"2024-02-05T07:40:52.892154+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:46.657168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:37:46.657168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":198,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892603+0000","last_change":"2024-02-05T07:39:59.662598+0000","last_active":"2024-02-05T07:40:52.892603+0000","last_peered":"2024-02-05T07:40:52.892603+0000","last_clean":"2024-02-05T07:40:52.892603+0000","last_became_active":"2024-02-05T07:38:09.464116+0000","last_became_peered":"2024-02-05T07:38:09.464116+0000","last_unstale":"2024-02-05T07:40:52.892603+0000","last_undegraded":"2024-02-05T07:40:52.892603+0000","last_fullsized":"2024-02-05T07:40:52.892603+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:59.662530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:59.662530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:25.420020+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":304,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.880062+0000","last_change":"2024-02-05T07:38:55.897373+0000","last_active":"2024-02-05T07:40:49.880062+0000","last_peered":"2024-02-05T07:40:49.880062+0000","last_clean":"2024-02-05T07:40:49.880062+0000","last_became_active":"2024-02-05T07:38:09.482289+0000","last_became_peered":"2024-02-05T07:38:09.482289+0000","last_unstale":"2024-02-05T07:40:49.880062+0000","last_undegraded":"2024-02-05T07:40:49.880062+0000","last_fullsized":"2024-02-05T07:40:49.880062+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:55.897133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.897133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"678'12616","reported_seq":12825,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:53.836291+0000","last_change":"2024-02-05T07:37:53.089618+0000","last_active":"2024-02-05T07:40:53.836291+0000","last_peered":"2024-02-05T07:40:53.836291+0000","last_clean":"2024-02-05T07:40:53.836291+0000","last_became_active":"2024-02-05T07:37:53.089261+0000","last_became_peered":"2024-02-05T07:37:53.089261+0000","last_unstale":"2024-02-05T07:40:53.836291+0000","last_undegraded":"2024-02-05T07:40:53.836291+0000","last_fullsized":"2024-02-05T07:40:53.836291+0000","mapping_epoch":579,"log_start":"639'9631","ondisk_log_start":"639'9631","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2985,"log_dups_size":3000,"ondisk_log_size":2985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66846720,"num_objects":1020,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12616,"num_write_kb":436352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":1086,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.876161+0000","last_change":"2024-02-05T07:40:33.082486+0000","last_active":"2024-02-05T07:40:49.876161+0000","last_peered":"2024-02-05T07:40:49.876161+0000","last_clean":"2024-02-05T07:40:49.876161+0000","last_became_active":"2024-02-05T07:40:33.082113+0000","last_became_peered":"2024-02-05T07:40:33.082113+0000","last_unstale":"2024-02-05T07:40:49.876161+0000","last_undegraded":"2024-02-05T07:40:49.876161+0000","last_fullsized":"2024-02-05T07:40:49.876161+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.417059+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.417059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"616'32","reported_seq":816,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892393+0000","last_change":"2024-02-05T07:38:36.705553+0000","last_active":"2024-02-05T07:40:52.892393+0000","last_peered":"2024-02-05T07:40:52.892393+0000","last_clean":"2024-02-05T07:40:52.892393+0000","last_became_active":"2024-02-05T07:38:09.471120+0000","last_became_peered":"2024-02-05T07:38:09.471120+0000","last_unstale":"2024-02-05T07:40:52.892393+0000","last_undegraded":"2024-02-05T07:40:52.892393+0000","last_fullsized":"2024-02-05T07:40:52.892393+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:38:36.705434+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:38:36.705434+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1180,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.874572+0000","last_change":"2024-02-05T07:38:09.490148+0000","last_active":"2024-02-05T07:40:49.874572+0000","last_peered":"2024-02-05T07:40:49.874572+0000","last_clean":"2024-02-05T07:40:49.874572+0000","last_became_active":"2024-02-05T07:38:09.489807+0000","last_became_peered":"2024-02-05T07:38:09.489807+0000","last_unstale":"2024-02-05T07:40:49.874572+0000","last_undegraded":"2024-02-05T07:40:49.874572+0000","last_fullsized":"2024-02-05T07:40:49.874572+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:03.618665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:37:03.618665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1022,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892769+0000","last_change":"2024-02-05T07:38:37.703938+0000","last_active":"2024-02-05T07:40:52.892769+0000","last_peered":"2024-02-05T07:40:52.892769+0000","last_clean":"2024-02-05T07:40:52.892769+0000","last_became_active":"2024-02-05T07:38:09.472546+0000","last_became_peered":"2024-02-05T07:38:09.472546+0000","last_unstale":"2024-02-05T07:40:52.892769+0000","last_undegraded":"2024-02-05T07:40:52.892769+0000","last_fullsized":"2024-02-05T07:40:52.892769+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:37.703824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:38:37.703824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":868,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.890301+0000","last_change":"2024-02-05T07:37:52.481462+0000","last_active":"2024-02-05T07:40:51.890301+0000","last_peered":"2024-02-05T07:40:51.890301+0000","last_clean":"2024-02-05T07:40:51.890301+0000","last_became_active":"2024-02-05T07:30:40.498529+0000","last_became_peered":"2024-02-05T07:30:40.498529+0000","last_unstale":"2024-02-05T07:40:51.890301+0000","last_undegraded":"2024-02-05T07:40:51.890301+0000","last_fullsized":"2024-02-05T07:40:51.890301+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:52.481255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:52.481255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:57.073632+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"677'12398","reported_seq":12605,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.269015+0000","last_change":"2024-02-05T07:37:52.817253+0000","last_active":"2024-02-05T07:40:52.269015+0000","last_peered":"2024-02-05T07:40:52.269015+0000","last_clean":"2024-02-05T07:40:52.269015+0000","last_became_active":"2024-02-05T07:37:52.816932+0000","last_became_peered":"2024-02-05T07:37:52.816932+0000","last_unstale":"2024-02-05T07:40:52.269015+0000","last_undegraded":"2024-02-05T07:40:52.269015+0000","last_fullsized":"2024-02-05T07:40:52.269015+0000","mapping_epoch":579,"log_start":"638'9400","ondisk_log_start":"638'9400","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2998,"log_dups_size":3000,"ondisk_log_size":2998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:57.073578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76677120,"num_objects":1170,"num_object_clones":0,"num_object_copies":3510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12398,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":682,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.883792+0000","last_change":"2024-02-05T07:40:36.746220+0000","last_active":"2024-02-05T07:40:49.883792+0000","last_peered":"2024-02-05T07:40:49.883792+0000","last_clean":"2024-02-05T07:40:49.883792+0000","last_became_active":"2024-02-05T07:40:36.745864+0000","last_became_peered":"2024-02-05T07:40:36.745864+0000","last_unstale":"2024-02-05T07:40:49.883792+0000","last_undegraded":"2024-02-05T07:40:49.883792+0000","last_fullsized":"2024-02-05T07:40:49.883792+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:40.411931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:40.411931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"616'46","reported_seq":1412,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.889925+0000","last_change":"2024-02-05T07:38:52.880432+0000","last_active":"2024-02-05T07:40:49.889925+0000","last_peered":"2024-02-05T07:40:49.889925+0000","last_clean":"2024-02-05T07:40:49.889925+0000","last_became_active":"2024-02-05T07:38:09.500221+0000","last_became_peered":"2024-02-05T07:38:09.500221+0000","last_unstale":"2024-02-05T07:40:49.889925+0000","last_undegraded":"2024-02-05T07:40:49.889925+0000","last_fullsized":"2024-02-05T07:40:49.889925+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'34","last_scrub_stamp":"2024-02-05T07:38:52.880291+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:38:52.880291+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":871,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.891015+0000","last_change":"2024-02-05T07:38:09.485474+0000","last_active":"2024-02-05T07:40:51.891015+0000","last_peered":"2024-02-05T07:40:51.891015+0000","last_clean":"2024-02-05T07:40:51.891015+0000","last_became_active":"2024-02-05T07:38:09.485164+0000","last_became_peered":"2024-02-05T07:38:09.485164+0000","last_unstale":"2024-02-05T07:40:51.891015+0000","last_undegraded":"2024-02-05T07:40:51.891015+0000","last_fullsized":"2024-02-05T07:40:51.891015+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:47.116727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:47.116727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"589'12","reported_seq":1437,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.883218+0000","last_change":"2024-02-05T07:38:09.514237+0000","last_active":"2024-02-05T07:40:49.883218+0000","last_peered":"2024-02-05T07:40:49.883218+0000","last_clean":"2024-02-05T07:40:49.883218+0000","last_became_active":"2024-02-05T07:38:09.488856+0000","last_became_peered":"2024-02-05T07:38:09.488856+0000","last_unstale":"2024-02-05T07:40:49.883218+0000","last_undegraded":"2024-02-05T07:40:49.883218+0000","last_fullsized":"2024-02-05T07:40:49.883218+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'11","last_scrub_stamp":"2024-02-05T07:36:59.523421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.523421+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":298,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.876506+0000","last_change":"2024-02-05T07:39:50.551862+0000","last_active":"2024-02-05T07:40:49.876506+0000","last_peered":"2024-02-05T07:40:49.876506+0000","last_clean":"2024-02-05T07:40:49.876506+0000","last_became_active":"2024-02-05T07:36:32.744070+0000","last_became_peered":"2024-02-05T07:36:32.744070+0000","last_unstale":"2024-02-05T07:40:49.876506+0000","last_undegraded":"2024-02-05T07:40:49.876506+0000","last_fullsized":"2024-02-05T07:40:49.876506+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:50.551740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:50.551740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:06.027327+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"675'12444","reported_seq":12647,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:50.783177+0000","last_change":"2024-02-05T07:37:53.089610+0000","last_active":"2024-02-05T07:40:50.783177+0000","last_peered":"2024-02-05T07:40:50.783177+0000","last_clean":"2024-02-05T07:40:50.783177+0000","last_became_active":"2024-02-05T07:37:53.089296+0000","last_became_peered":"2024-02-05T07:37:53.089296+0000","last_unstale":"2024-02-05T07:40:50.783177+0000","last_undegraded":"2024-02-05T07:40:50.783177+0000","last_fullsized":"2024-02-05T07:40:50.783177+0000","mapping_epoch":579,"log_start":"636'9072","ondisk_log_start":"636'9072","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":3372,"log_dups_size":0,"ondisk_log_size":3372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:55.631611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87425024,"num_objects":1334,"num_object_clones":0,"num_object_copies":4002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12444,"num_write_kb":440896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":1266,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.876087+0000","last_change":"2024-02-05T07:38:34.512785+0000","last_active":"2024-02-05T07:40:49.876087+0000","last_peered":"2024-02-05T07:40:49.876087+0000","last_clean":"2024-02-05T07:40:49.876087+0000","last_became_active":"2024-02-05T07:38:13.714462+0000","last_became_peered":"2024-02-05T07:38:13.714462+0000","last_unstale":"2024-02-05T07:40:49.876087+0000","last_undegraded":"2024-02-05T07:40:49.876087+0000","last_fullsized":"2024-02-05T07:40:49.876087+0000","mapping_epoch":593,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:34.509015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:34.509015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":823,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.896687+0000","last_change":"2024-02-05T07:38:09.479255+0000","last_active":"2024-02-05T07:40:51.896687+0000","last_peered":"2024-02-05T07:40:51.896687+0000","last_clean":"2024-02-05T07:40:51.896687+0000","last_became_active":"2024-02-05T07:38:09.478485+0000","last_became_peered":"2024-02-05T07:38:09.478485+0000","last_unstale":"2024-02-05T07:40:51.896687+0000","last_undegraded":"2024-02-05T07:40:51.896687+0000","last_fullsized":"2024-02-05T07:40:51.896687+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.155552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.155552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"615'50","reported_seq":953,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.890754+0000","last_change":"2024-02-05T07:39:48.737503+0000","last_active":"2024-02-05T07:40:51.890754+0000","last_peered":"2024-02-05T07:40:51.890754+0000","last_clean":"2024-02-05T07:40:51.890754+0000","last_became_active":"2024-02-05T07:30:40.496940+0000","last_became_peered":"2024-02-05T07:30:40.496940+0000","last_unstale":"2024-02-05T07:40:51.890754+0000","last_undegraded":"2024-02-05T07:40:51.890754+0000","last_fullsized":"2024-02-05T07:40:51.890754+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'50","last_scrub_stamp":"2024-02-05T07:39:48.737259+0000","last_deep_scrub":"454'38","last_deep_scrub_stamp":"2024-02-05T07:34:38.942830+0000","last_clean_scrub_stamp":"2024-02-05T07:39:48.737259+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:07.832109+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":871,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.897166+0000","last_change":"2024-02-05T07:38:57.455422+0000","last_active":"2024-02-05T07:40:51.897166+0000","last_peered":"2024-02-05T07:40:51.897166+0000","last_clean":"2024-02-05T07:40:51.897166+0000","last_became_active":"2024-02-05T07:38:09.488092+0000","last_became_peered":"2024-02-05T07:38:09.488092+0000","last_unstale":"2024-02-05T07:40:51.897166+0000","last_undegraded":"2024-02-05T07:40:51.897166+0000","last_fullsized":"2024-02-05T07:40:51.897166+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:57.455281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:57.455281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1181,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.897947+0000","last_change":"2024-02-05T07:38:09.480292+0000","last_active":"2024-02-05T07:40:51.897947+0000","last_peered":"2024-02-05T07:40:51.897947+0000","last_clean":"2024-02-05T07:40:51.897947+0000","last_became_active":"2024-02-05T07:38:09.479403+0000","last_became_peered":"2024-02-05T07:38:09.479403+0000","last_unstale":"2024-02-05T07:40:51.897947+0000","last_undegraded":"2024-02-05T07:40:51.897947+0000","last_fullsized":"2024-02-05T07:40:51.897947+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:01.252507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:38:01.252507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"675'12559","reported_seq":14018,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:50.614363+0000","last_change":"2024-02-05T07:39:50.484334+0000","last_active":"2024-02-05T07:40:50.614363+0000","last_peered":"2024-02-05T07:40:50.614363+0000","last_clean":"2024-02-05T07:40:50.614363+0000","last_became_active":"2024-02-05T07:38:16.656778+0000","last_became_peered":"2024-02-05T07:38:16.656778+0000","last_unstale":"2024-02-05T07:40:50.614363+0000","last_undegraded":"2024-02-05T07:40:50.614363+0000","last_fullsized":"2024-02-05T07:40:50.614363+0000","mapping_epoch":596,"log_start":"639'9800","ondisk_log_start":"639'9800","created":579,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"634'8172","last_scrub_stamp":"2024-02-05T07:39:50.484069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:39:50.484069+0000","objects_scrubbed":5766,"log_size":2759,"log_dups_size":3000,"ondisk_log_size":2759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:09.764020+0000","scrub_duration":2626,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83296256,"num_objects":1271,"num_object_clones":0,"num_object_copies":3813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12559,"num_write_kb":442560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":589,"num_bytes_recovered":38600704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":1636,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.890495+0000","last_change":"2024-02-05T07:38:11.676831+0000","last_active":"2024-02-05T07:40:51.890495+0000","last_peered":"2024-02-05T07:40:51.890495+0000","last_clean":"2024-02-05T07:40:51.890495+0000","last_became_active":"2024-02-05T07:38:11.676482+0000","last_became_peered":"2024-02-05T07:38:11.676482+0000","last_unstale":"2024-02-05T07:40:51.890495+0000","last_undegraded":"2024-02-05T07:40:51.890495+0000","last_fullsized":"2024-02-05T07:40:51.890495+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:07.353770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:07.353770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"589'11","reported_seq":1037,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.891093+0000","last_change":"2024-02-05T07:38:09.522356+0000","last_active":"2024-02-05T07:40:51.891093+0000","last_peered":"2024-02-05T07:40:51.891093+0000","last_clean":"2024-02-05T07:40:51.891093+0000","last_became_active":"2024-02-05T07:38:09.495697+0000","last_became_peered":"2024-02-05T07:38:09.495697+0000","last_unstale":"2024-02-05T07:40:51.891093+0000","last_undegraded":"2024-02-05T07:40:51.891093+0000","last_fullsized":"2024-02-05T07:40:51.891093+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'10","last_scrub_stamp":"2024-02-05T07:37:59.362892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:59.362892+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"616'49","reported_seq":854,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891587+0000","last_change":"2024-02-05T07:39:55.713122+0000","last_active":"2024-02-05T07:40:52.891587+0000","last_peered":"2024-02-05T07:40:52.891587+0000","last_clean":"2024-02-05T07:40:52.891587+0000","last_became_active":"2024-02-05T07:38:09.480688+0000","last_became_peered":"2024-02-05T07:38:09.480688+0000","last_unstale":"2024-02-05T07:40:52.891587+0000","last_undegraded":"2024-02-05T07:40:52.891587+0000","last_fullsized":"2024-02-05T07:40:52.891587+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'49","last_scrub_stamp":"2024-02-05T07:39:55.713036+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:39:55.713036+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:05.586770+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"639'2","reported_seq":1054,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.887942+0000","last_change":"2024-02-05T07:40:13.213597+0000","last_active":"2024-02-05T07:40:49.887942+0000","last_peered":"2024-02-05T07:40:49.887942+0000","last_clean":"2024-02-05T07:40:49.887942+0000","last_became_active":"2024-02-05T07:38:09.467535+0000","last_became_peered":"2024-02-05T07:38:09.467535+0000","last_unstale":"2024-02-05T07:40:49.887942+0000","last_undegraded":"2024-02-05T07:40:49.887942+0000","last_fullsized":"2024-02-05T07:40:49.887942+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'2","last_scrub_stamp":"2024-02-05T07:40:13.213477+0000","last_deep_scrub":"639'2","last_deep_scrub_stamp":"2024-02-05T07:40:13.213477+0000","last_clean_scrub_stamp":"2024-02-05T07:40:13.213477+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:26.617373+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1164,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891961+0000","last_change":"2024-02-05T07:37:58.737127+0000","last_active":"2024-02-05T07:40:52.891961+0000","last_peered":"2024-02-05T07:40:52.891961+0000","last_clean":"2024-02-05T07:40:52.891961+0000","last_became_active":"2024-02-05T07:36:32.701203+0000","last_became_peered":"2024-02-05T07:36:32.701203+0000","last_unstale":"2024-02-05T07:40:52.891961+0000","last_undegraded":"2024-02-05T07:40:52.891961+0000","last_fullsized":"2024-02-05T07:40:52.891961+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:58.737033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:58.737033+0000","last_clean_scrub_stamp":"2024-02-05T07:37:58.737033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"675'12605","reported_seq":14006,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:50.750614+0000","last_change":"2024-02-05T07:38:16.745449+0000","last_active":"2024-02-05T07:40:50.750614+0000","last_peered":"2024-02-05T07:40:50.750614+0000","last_clean":"2024-02-05T07:40:50.750614+0000","last_became_active":"2024-02-05T07:38:16.742262+0000","last_became_peered":"2024-02-05T07:38:16.742262+0000","last_unstale":"2024-02-05T07:40:50.750614+0000","last_undegraded":"2024-02-05T07:40:50.750614+0000","last_fullsized":"2024-02-05T07:40:50.750614+0000","mapping_epoch":596,"log_start":"636'9229","ondisk_log_start":"636'9229","created":579,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":3376,"log_dups_size":0,"ondisk_log_size":3376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:55.633861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886104,"num_objects":1281,"num_object_clones":0,"num_object_copies":3843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1281,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12605,"num_write_kb":444289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":593,"num_bytes_recovered":38862848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":1272,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892971+0000","last_change":"2024-02-05T07:40:48.866053+0000","last_active":"2024-02-05T07:40:52.892971+0000","last_peered":"2024-02-05T07:40:52.892971+0000","last_clean":"2024-02-05T07:40:52.892971+0000","last_became_active":"2024-02-05T07:40:48.865775+0000","last_became_peered":"2024-02-05T07:40:48.865775+0000","last_unstale":"2024-02-05T07:40:52.892971+0000","last_undegraded":"2024-02-05T07:40:52.892971+0000","last_fullsized":"2024-02-05T07:40:52.892971+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:42.822474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:42.822474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":1037,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892174+0000","last_change":"2024-02-05T07:39:44.702368+0000","last_active":"2024-02-05T07:40:52.892174+0000","last_peered":"2024-02-05T07:40:52.892174+0000","last_clean":"2024-02-05T07:40:52.892174+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:40:52.892174+0000","last_undegraded":"2024-02-05T07:40:52.892174+0000","last_fullsized":"2024-02-05T07:40:52.892174+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:39:44.702291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:44.702291+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"616'35","reported_seq":1373,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.872569+0000","last_change":"2024-02-05T07:38:50.868322+0000","last_active":"2024-02-05T07:40:49.872569+0000","last_peered":"2024-02-05T07:40:49.872569+0000","last_clean":"2024-02-05T07:40:49.872569+0000","last_became_active":"2024-02-05T07:38:09.496732+0000","last_became_peered":"2024-02-05T07:38:09.496732+0000","last_unstale":"2024-02-05T07:40:49.872569+0000","last_undegraded":"2024-02-05T07:40:49.872569+0000","last_fullsized":"2024-02-05T07:40:49.872569+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:38:50.868179+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:38:50.868179+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":874,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.898274+0000","last_change":"2024-02-05T07:36:32.741872+0000","last_active":"2024-02-05T07:40:51.898274+0000","last_peered":"2024-02-05T07:40:51.898274+0000","last_clean":"2024-02-05T07:40:51.898274+0000","last_became_active":"2024-02-05T07:36:32.741377+0000","last_became_peered":"2024-02-05T07:36:32.741377+0000","last_unstale":"2024-02-05T07:40:51.898274+0000","last_undegraded":"2024-02-05T07:40:51.898274+0000","last_fullsized":"2024-02-05T07:40:51.898274+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:17.067204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:35:17.067204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":562,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.866475+0000","last_change":"2024-02-05T07:39:54.880406+0000","last_active":"2024-02-05T07:40:49.866475+0000","last_peered":"2024-02-05T07:40:49.866475+0000","last_clean":"2024-02-05T07:40:49.866475+0000","last_became_active":"2024-02-05T07:32:58.598709+0000","last_became_peered":"2024-02-05T07:32:58.598709+0000","last_unstale":"2024-02-05T07:40:49.866475+0000","last_undegraded":"2024-02-05T07:40:49.866475+0000","last_fullsized":"2024-02-05T07:40:49.866475+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:54.879841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:54.879841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:07.394562+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"675'12486","reported_seq":13964,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:50.610551+0000","last_change":"2024-02-05T07:39:46.125582+0000","last_active":"2024-02-05T07:40:50.610551+0000","last_peered":"2024-02-05T07:40:50.610551+0000","last_clean":"2024-02-05T07:40:50.610551+0000","last_became_active":"2024-02-05T07:38:15.646279+0000","last_became_peered":"2024-02-05T07:38:15.646279+0000","last_unstale":"2024-02-05T07:40:50.610551+0000","last_undegraded":"2024-02-05T07:40:50.610551+0000","last_fullsized":"2024-02-05T07:40:50.610551+0000","mapping_epoch":595,"log_start":"639'9711","ondisk_log_start":"639'9711","created":579,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'7750","last_scrub_stamp":"2024-02-05T07:39:46.125414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:39:46.125414+0000","objects_scrubbed":6078,"log_size":2775,"log_dups_size":3000,"ondisk_log_size":2775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:08.891141+0000","scrub_duration":2339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82182144,"num_objects":1254,"num_object_clones":0,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12486,"num_write_kb":439680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":577,"num_bytes_recovered":37814272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":1244,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.908246+0000","last_change":"2024-02-05T07:40:52.908246+0000","last_active":"2024-02-05T07:40:52.908246+0000","last_peered":"2024-02-05T07:40:52.908246+0000","last_clean":"2024-02-05T07:40:52.908246+0000","last_became_active":"2024-02-05T07:40:52.907985+0000","last_became_peered":"2024-02-05T07:40:52.907985+0000","last_unstale":"2024-02-05T07:40:52.908246+0000","last_undegraded":"2024-02-05T07:40:52.908246+0000","last_fullsized":"2024-02-05T07:40:52.908246+0000","mapping_epoch":302,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.746435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.746435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1140,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892024+0000","last_change":"2024-02-05T07:40:11.774832+0000","last_active":"2024-02-05T07:40:52.892024+0000","last_peered":"2024-02-05T07:40:52.892024+0000","last_clean":"2024-02-05T07:40:52.892024+0000","last_became_active":"2024-02-05T07:36:32.709025+0000","last_became_peered":"2024-02-05T07:36:32.709025+0000","last_unstale":"2024-02-05T07:40:52.892024+0000","last_undegraded":"2024-02-05T07:40:52.892024+0000","last_fullsized":"2024-02-05T07:40:52.892024+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:11.774744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:40:11.774744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:36.409236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"615'39","reported_seq":971,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.871958+0000","last_change":"2024-02-05T07:38:41.807307+0000","last_active":"2024-02-05T07:40:49.871958+0000","last_peered":"2024-02-05T07:40:49.871958+0000","last_clean":"2024-02-05T07:40:49.871958+0000","last_became_active":"2024-02-05T07:36:32.712346+0000","last_became_peered":"2024-02-05T07:36:32.712346+0000","last_unstale":"2024-02-05T07:40:49.871958+0000","last_undegraded":"2024-02-05T07:40:49.871958+0000","last_fullsized":"2024-02-05T07:40:49.871958+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'29","last_scrub_stamp":"2024-02-05T07:38:41.806977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:41.806977+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":299,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.880026+0000","last_change":"2024-02-05T07:39:57.311987+0000","last_active":"2024-02-05T07:40:49.880026+0000","last_peered":"2024-02-05T07:40:49.880026+0000","last_clean":"2024-02-05T07:40:49.880026+0000","last_became_active":"2024-02-05T07:36:32.732781+0000","last_became_peered":"2024-02-05T07:36:32.732781+0000","last_unstale":"2024-02-05T07:40:49.880026+0000","last_undegraded":"2024-02-05T07:40:49.880026+0000","last_fullsized":"2024-02-05T07:40:49.880026+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:57.311857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:39:57.311857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:04.385252+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1292,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.878887+0000","last_change":"2024-02-05T07:40:09.083562+0000","last_active":"2024-02-05T07:40:49.878887+0000","last_peered":"2024-02-05T07:40:49.878887+0000","last_clean":"2024-02-05T07:40:49.878887+0000","last_became_active":"2024-02-05T07:36:32.709872+0000","last_became_peered":"2024-02-05T07:36:32.709872+0000","last_unstale":"2024-02-05T07:40:49.878887+0000","last_undegraded":"2024-02-05T07:40:49.878887+0000","last_fullsized":"2024-02-05T07:40:49.878887+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:09.083452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:40:09.083452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:10.818855+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"677'12565","reported_seq":12770,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.249694+0000","last_change":"2024-02-05T07:37:53.093098+0000","last_active":"2024-02-05T07:40:52.249694+0000","last_peered":"2024-02-05T07:40:52.249694+0000","last_clean":"2024-02-05T07:40:52.249694+0000","last_became_active":"2024-02-05T07:37:53.091347+0000","last_became_peered":"2024-02-05T07:37:53.091347+0000","last_unstale":"2024-02-05T07:40:52.249694+0000","last_undegraded":"2024-02-05T07:40:52.249694+0000","last_fullsized":"2024-02-05T07:40:52.249694+0000","mapping_epoch":579,"log_start":"639'9600","ondisk_log_start":"639'9600","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2965,"log_dups_size":3000,"ondisk_log_size":2965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:57.073546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77135872,"num_objects":1177,"num_object_clones":0,"num_object_copies":3531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12565,"num_write_kb":439744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":895,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892227+0000","last_change":"2024-02-05T07:38:11.676690+0000","last_active":"2024-02-05T07:40:52.892227+0000","last_peered":"2024-02-05T07:40:52.892227+0000","last_clean":"2024-02-05T07:40:52.892227+0000","last_became_active":"2024-02-05T07:38:11.676446+0000","last_became_peered":"2024-02-05T07:38:11.676446+0000","last_unstale":"2024-02-05T07:40:52.892227+0000","last_undegraded":"2024-02-05T07:40:52.892227+0000","last_fullsized":"2024-02-05T07:40:52.892227+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.778695+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.778695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":886,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.890281+0000","last_change":"2024-02-05T07:39:51.915848+0000","last_active":"2024-02-05T07:40:49.890281+0000","last_peered":"2024-02-05T07:40:49.890281+0000","last_clean":"2024-02-05T07:40:49.890281+0000","last_became_active":"2024-02-05T07:38:09.496845+0000","last_became_peered":"2024-02-05T07:38:09.496845+0000","last_unstale":"2024-02-05T07:40:49.890281+0000","last_undegraded":"2024-02-05T07:40:49.890281+0000","last_fullsized":"2024-02-05T07:40:49.890281+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:51.915741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:51.915741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:18.553450+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"615'36","reported_seq":330,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.883495+0000","last_change":"2024-02-05T07:38:54.585085+0000","last_active":"2024-02-05T07:40:49.883495+0000","last_peered":"2024-02-05T07:40:49.883495+0000","last_clean":"2024-02-05T07:40:49.883495+0000","last_became_active":"2024-02-05T07:36:32.753752+0000","last_became_peered":"2024-02-05T07:36:32.753752+0000","last_unstale":"2024-02-05T07:40:49.883495+0000","last_undegraded":"2024-02-05T07:40:49.883495+0000","last_fullsized":"2024-02-05T07:40:49.883495+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:38:54.584948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:54.584948+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1306,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.867030+0000","last_change":"2024-02-05T07:38:31.740800+0000","last_active":"2024-02-05T07:40:49.867030+0000","last_peered":"2024-02-05T07:40:49.867030+0000","last_clean":"2024-02-05T07:40:49.867030+0000","last_became_active":"2024-02-05T07:30:40.493770+0000","last_became_peered":"2024-02-05T07:30:40.493770+0000","last_unstale":"2024-02-05T07:40:49.867030+0000","last_undegraded":"2024-02-05T07:40:49.867030+0000","last_fullsized":"2024-02-05T07:40:49.867030+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T07:38:31.740660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:38:31.740660+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":614,"num_read_kb":609,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":590,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.896927+0000","last_change":"2024-02-05T07:38:41.375038+0000","last_active":"2024-02-05T07:40:51.896927+0000","last_peered":"2024-02-05T07:40:51.896927+0000","last_clean":"2024-02-05T07:40:51.896927+0000","last_became_active":"2024-02-05T07:38:09.493827+0000","last_became_peered":"2024-02-05T07:38:09.493827+0000","last_unstale":"2024-02-05T07:40:51.896927+0000","last_undegraded":"2024-02-05T07:40:51.896927+0000","last_fullsized":"2024-02-05T07:40:51.896927+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:41.374804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:38:41.374804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":1308,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.883096+0000","last_change":"2024-02-05T07:39:49.884158+0000","last_active":"2024-02-05T07:40:49.883096+0000","last_peered":"2024-02-05T07:40:49.883096+0000","last_clean":"2024-02-05T07:40:49.883096+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:40:49.883096+0000","last_undegraded":"2024-02-05T07:40:49.883096+0000","last_fullsized":"2024-02-05T07:40:49.883096+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:39:49.883906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:49.883906+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:05.915566+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"677'12548","reported_seq":12755,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.283774+0000","last_change":"2024-02-05T07:37:52.817223+0000","last_active":"2024-02-05T07:40:52.283774+0000","last_peered":"2024-02-05T07:40:52.283774+0000","last_clean":"2024-02-05T07:40:52.283774+0000","last_became_active":"2024-02-05T07:37:52.816946+0000","last_became_peered":"2024-02-05T07:37:52.816946+0000","last_unstale":"2024-02-05T07:40:52.283774+0000","last_undegraded":"2024-02-05T07:40:52.283774+0000","last_fullsized":"2024-02-05T07:40:52.283774+0000","mapping_epoch":579,"log_start":"638'9600","ondisk_log_start":"638'9600","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2948,"log_dups_size":3000,"ondisk_log_size":2948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:57.073315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78643200,"num_objects":1200,"num_object_clones":0,"num_object_copies":3600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12548,"num_write_kb":439936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":299,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.884455+0000","last_change":"2024-02-05T07:38:43.594222+0000","last_active":"2024-02-05T07:40:49.884455+0000","last_peered":"2024-02-05T07:40:49.884455+0000","last_clean":"2024-02-05T07:40:49.884455+0000","last_became_active":"2024-02-05T07:36:32.732334+0000","last_became_peered":"2024-02-05T07:36:32.732334+0000","last_unstale":"2024-02-05T07:40:49.884455+0000","last_undegraded":"2024-02-05T07:40:49.884455+0000","last_fullsized":"2024-02-05T07:40:49.884455+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:43.593810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:43.593810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"616'44","reported_seq":1476,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892839+0000","last_change":"2024-02-05T07:38:09.529174+0000","last_active":"2024-02-05T07:40:52.892839+0000","last_peered":"2024-02-05T07:40:52.892839+0000","last_clean":"2024-02-05T07:40:52.892839+0000","last_became_active":"2024-02-05T07:38:09.479528+0000","last_became_peered":"2024-02-05T07:38:09.479528+0000","last_unstale":"2024-02-05T07:40:52.892839+0000","last_undegraded":"2024-02-05T07:40:52.892839+0000","last_fullsized":"2024-02-05T07:40:52.892839+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'31","last_scrub_stamp":"2024-02-05T07:37:45.644963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.644963+0000","objects_scrubbed":8,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":164,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.874090+0000","last_change":"2024-02-05T07:40:08.312293+0000","last_active":"2024-02-05T07:40:49.874090+0000","last_peered":"2024-02-05T07:40:49.874090+0000","last_clean":"2024-02-05T07:40:49.874090+0000","last_became_active":"2024-02-05T07:38:31.807354+0000","last_became_peered":"2024-02-05T07:38:31.807354+0000","last_unstale":"2024-02-05T07:40:49.874090+0000","last_undegraded":"2024-02-05T07:40:49.874090+0000","last_fullsized":"2024-02-05T07:40:49.874090+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:08.312041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:40:08.312041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:35.861633+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1065,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891266+0000","last_change":"2024-02-05T07:38:29.812121+0000","last_active":"2024-02-05T07:40:52.891266+0000","last_peered":"2024-02-05T07:40:52.891266+0000","last_clean":"2024-02-05T07:40:52.891266+0000","last_became_active":"2024-02-05T07:37:06.977158+0000","last_became_peered":"2024-02-05T07:37:06.977158+0000","last_unstale":"2024-02-05T07:40:52.891266+0000","last_undegraded":"2024-02-05T07:40:52.891266+0000","last_fullsized":"2024-02-05T07:40:52.891266+0000","mapping_epoch":367,"log_start":"86'128","ondisk_log_start":"86'128","created":20,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:29.812050+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:38:29.812050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":865,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.897072+0000","last_change":"2024-02-05T07:38:49.589976+0000","last_active":"2024-02-05T07:40:51.897072+0000","last_peered":"2024-02-05T07:40:51.897072+0000","last_clean":"2024-02-05T07:40:51.897072+0000","last_became_active":"2024-02-05T07:38:09.488951+0000","last_became_peered":"2024-02-05T07:38:09.488951+0000","last_unstale":"2024-02-05T07:40:51.897072+0000","last_undegraded":"2024-02-05T07:40:51.897072+0000","last_fullsized":"2024-02-05T07:40:51.897072+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:49.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:49.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"677'12491","reported_seq":12698,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.245530+0000","last_change":"2024-02-05T07:37:53.090431+0000","last_active":"2024-02-05T07:40:52.245530+0000","last_peered":"2024-02-05T07:40:52.245530+0000","last_clean":"2024-02-05T07:40:52.245530+0000","last_became_active":"2024-02-05T07:37:53.090153+0000","last_became_peered":"2024-02-05T07:37:53.090153+0000","last_unstale":"2024-02-05T07:40:52.245530+0000","last_undegraded":"2024-02-05T07:40:52.245530+0000","last_fullsized":"2024-02-05T07:40:52.245530+0000","mapping_epoch":579,"log_start":"638'9500","ondisk_log_start":"638'9500","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2991,"log_dups_size":3000,"ondisk_log_size":2991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:57.073256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78446592,"num_objects":1197,"num_object_clones":0,"num_object_copies":3591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12491,"num_write_kb":438016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":1279,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.883972+0000","last_change":"2024-02-05T07:38:32.532424+0000","last_active":"2024-02-05T07:40:49.883972+0000","last_peered":"2024-02-05T07:40:49.883972+0000","last_clean":"2024-02-05T07:40:49.883972+0000","last_became_active":"2024-02-05T07:38:11.674897+0000","last_became_peered":"2024-02-05T07:38:11.674897+0000","last_unstale":"2024-02-05T07:40:49.883972+0000","last_undegraded":"2024-02-05T07:40:49.883972+0000","last_fullsized":"2024-02-05T07:40:49.883972+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:32.532304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:32.532304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":304,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.883405+0000","last_change":"2024-02-05T07:38:35.498140+0000","last_active":"2024-02-05T07:40:49.883405+0000","last_peered":"2024-02-05T07:40:49.883405+0000","last_clean":"2024-02-05T07:40:49.883405+0000","last_became_active":"2024-02-05T07:38:09.500885+0000","last_became_peered":"2024-02-05T07:38:09.500885+0000","last_unstale":"2024-02-05T07:40:49.883405+0000","last_undegraded":"2024-02-05T07:40:49.883405+0000","last_fullsized":"2024-02-05T07:40:49.883405+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:35.497879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:35.497879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"615'26","reported_seq":252,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.879230+0000","last_change":"2024-02-05T07:38:39.505608+0000","last_active":"2024-02-05T07:40:49.879230+0000","last_peered":"2024-02-05T07:40:49.879230+0000","last_clean":"2024-02-05T07:40:49.879230+0000","last_became_active":"2024-02-05T07:37:31.671370+0000","last_became_peered":"2024-02-05T07:37:31.671370+0000","last_unstale":"2024-02-05T07:40:49.879230+0000","last_undegraded":"2024-02-05T07:40:49.879230+0000","last_fullsized":"2024-02-05T07:40:49.879230+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'22","last_scrub_stamp":"2024-02-05T07:38:39.505230+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:38:39.505230+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":299,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.875045+0000","last_change":"2024-02-05T07:38:45.513185+0000","last_active":"2024-02-05T07:40:49.875045+0000","last_peered":"2024-02-05T07:40:49.875045+0000","last_clean":"2024-02-05T07:40:49.875045+0000","last_became_active":"2024-02-05T07:36:32.733822+0000","last_became_peered":"2024-02-05T07:36:32.733822+0000","last_unstale":"2024-02-05T07:40:49.875045+0000","last_undegraded":"2024-02-05T07:40:49.875045+0000","last_fullsized":"2024-02-05T07:40:49.875045+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:45.512912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:38:45.512912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":303,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.874087+0000","last_change":"2024-02-05T07:38:57.649523+0000","last_active":"2024-02-05T07:40:49.874087+0000","last_peered":"2024-02-05T07:40:49.874087+0000","last_clean":"2024-02-05T07:40:49.874087+0000","last_became_active":"2024-02-05T07:36:32.781195+0000","last_became_peered":"2024-02-05T07:36:32.781195+0000","last_unstale":"2024-02-05T07:40:49.874087+0000","last_undegraded":"2024-02-05T07:40:49.874087+0000","last_fullsized":"2024-02-05T07:40:49.874087+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:38:57.649351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:57.649351+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1176,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892449+0000","last_change":"2024-02-05T07:38:51.640312+0000","last_active":"2024-02-05T07:40:52.892449+0000","last_peered":"2024-02-05T07:40:52.892449+0000","last_clean":"2024-02-05T07:40:52.892449+0000","last_became_active":"2024-02-05T07:36:32.718684+0000","last_became_peered":"2024-02-05T07:36:32.718684+0000","last_unstale":"2024-02-05T07:40:52.892449+0000","last_undegraded":"2024-02-05T07:40:52.892449+0000","last_fullsized":"2024-02-05T07:40:52.892449+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:51.640224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:38:51.640224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"677'12753","reported_seq":12960,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.274050+0000","last_change":"2024-02-05T07:37:53.089611+0000","last_active":"2024-02-05T07:40:52.274050+0000","last_peered":"2024-02-05T07:40:52.274050+0000","last_clean":"2024-02-05T07:40:52.274050+0000","last_became_active":"2024-02-05T07:37:53.089298+0000","last_became_peered":"2024-02-05T07:37:53.089298+0000","last_unstale":"2024-02-05T07:40:52.274050+0000","last_undegraded":"2024-02-05T07:40:52.274050+0000","last_fullsized":"2024-02-05T07:40:52.274050+0000","mapping_epoch":579,"log_start":"638'9800","ondisk_log_start":"638'9800","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2953,"log_dups_size":3000,"ondisk_log_size":2953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:57.073487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76218368,"num_objects":1163,"num_object_clones":0,"num_object_copies":3489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12753,"num_write_kb":445312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":666,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.874846+0000","last_change":"2024-02-05T07:38:15.643159+0000","last_active":"2024-02-05T07:40:49.874846+0000","last_peered":"2024-02-05T07:40:49.874846+0000","last_clean":"2024-02-05T07:40:49.874846+0000","last_became_active":"2024-02-05T07:38:15.642779+0000","last_became_peered":"2024-02-05T07:38:15.642779+0000","last_unstale":"2024-02-05T07:40:49.874846+0000","last_undegraded":"2024-02-05T07:40:49.874846+0000","last_fullsized":"2024-02-05T07:40:49.874846+0000","mapping_epoch":595,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:08.283492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:08.283492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1041,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.879819+0000","last_change":"2024-02-05T07:40:19.104414+0000","last_active":"2024-02-05T07:40:49.879819+0000","last_peered":"2024-02-05T07:40:49.879819+0000","last_clean":"2024-02-05T07:40:49.879819+0000","last_became_active":"2024-02-05T07:38:09.463763+0000","last_became_peered":"2024-02-05T07:38:09.463763+0000","last_unstale":"2024-02-05T07:40:49.879819+0000","last_undegraded":"2024-02-05T07:40:49.879819+0000","last_fullsized":"2024-02-05T07:40:49.879819+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:19.104295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:40:19.104295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:47.076546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"616'68","reported_seq":401,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.884135+0000","last_change":"2024-02-05T07:36:56.264458+0000","last_active":"2024-02-05T07:40:49.884135+0000","last_peered":"2024-02-05T07:40:49.884135+0000","last_clean":"2024-02-05T07:40:49.884135+0000","last_became_active":"2024-02-05T07:36:32.738270+0000","last_became_peered":"2024-02-05T07:36:32.738270+0000","last_unstale":"2024-02-05T07:40:49.884135+0000","last_undegraded":"2024-02-05T07:40:49.884135+0000","last_fullsized":"2024-02-05T07:40:49.884135+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'56","last_scrub_stamp":"2024-02-05T07:36:56.264300+0000","last_deep_scrub":"550'56","last_deep_scrub_stamp":"2024-02-05T07:36:56.264300+0000","last_clean_scrub_stamp":"2024-02-05T07:36:56.264300+0000","objects_scrubbed":10,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":82,"num_read_kb":67,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":705,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892280+0000","last_change":"2024-02-05T07:40:15.743056+0000","last_active":"2024-02-05T07:40:52.892280+0000","last_peered":"2024-02-05T07:40:52.892280+0000","last_clean":"2024-02-05T07:40:52.892280+0000","last_became_active":"2024-02-05T07:38:09.484410+0000","last_became_peered":"2024-02-05T07:38:09.484410+0000","last_unstale":"2024-02-05T07:40:52.892280+0000","last_undegraded":"2024-02-05T07:40:52.892280+0000","last_fullsized":"2024-02-05T07:40:52.892280+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:15.742953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:40:15.742953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:22.968200+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":1205,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.872023+0000","last_change":"2024-02-05T07:40:12.141885+0000","last_active":"2024-02-05T07:40:49.872023+0000","last_peered":"2024-02-05T07:40:49.872023+0000","last_clean":"2024-02-05T07:40:49.872023+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:40:49.872023+0000","last_undegraded":"2024-02-05T07:40:49.872023+0000","last_fullsized":"2024-02-05T07:40:49.872023+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:40:12.141744+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:40:12.141744+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:22.389002+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":879,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.892500+0000","last_change":"2024-02-05T07:39:03.439286+0000","last_active":"2024-02-05T07:40:51.892500+0000","last_peered":"2024-02-05T07:40:51.892500+0000","last_clean":"2024-02-05T07:40:51.892500+0000","last_became_active":"2024-02-05T07:38:09.486846+0000","last_became_peered":"2024-02-05T07:38:09.486846+0000","last_unstale":"2024-02-05T07:40:51.892500+0000","last_undegraded":"2024-02-05T07:40:51.892500+0000","last_fullsized":"2024-02-05T07:40:51.892500+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:03.439161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:03.439161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"678'13113","reported_seq":13322,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:53.809709+0000","last_change":"2024-02-05T07:37:52.813698+0000","last_active":"2024-02-05T07:40:53.809709+0000","last_peered":"2024-02-05T07:40:53.809709+0000","last_clean":"2024-02-05T07:40:53.809709+0000","last_became_active":"2024-02-05T07:37:52.813405+0000","last_became_peered":"2024-02-05T07:37:52.813405+0000","last_unstale":"2024-02-05T07:40:53.809709+0000","last_undegraded":"2024-02-05T07:40:53.809709+0000","last_fullsized":"2024-02-05T07:40:53.809709+0000","mapping_epoch":579,"log_start":"640'10117","ondisk_log_start":"640'10117","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":2996,"log_dups_size":3000,"ondisk_log_size":2996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68878336,"num_objects":1051,"num_object_clones":0,"num_object_copies":3153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13113,"num_write_kb":453248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":634,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.893024+0000","last_change":"2024-02-05T07:40:20.876587+0000","last_active":"2024-02-05T07:40:51.893024+0000","last_peered":"2024-02-05T07:40:51.893024+0000","last_clean":"2024-02-05T07:40:51.893024+0000","last_became_active":"2024-02-05T07:40:20.876256+0000","last_became_peered":"2024-02-05T07:40:20.876256+0000","last_unstale":"2024-02-05T07:40:51.893024+0000","last_undegraded":"2024-02-05T07:40:51.893024+0000","last_fullsized":"2024-02-05T07:40:51.893024+0000","mapping_epoch":619,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:39:45.710637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:39:45.710637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:15.453859+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"615'21","reported_seq":1095,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.871754+0000","last_change":"2024-02-05T07:38:35.790500+0000","last_active":"2024-02-05T07:40:49.871754+0000","last_peered":"2024-02-05T07:40:49.871754+0000","last_clean":"2024-02-05T07:40:49.871754+0000","last_became_active":"2024-02-05T07:38:09.479890+0000","last_became_peered":"2024-02-05T07:38:09.479890+0000","last_unstale":"2024-02-05T07:40:49.871754+0000","last_undegraded":"2024-02-05T07:40:49.871754+0000","last_fullsized":"2024-02-05T07:40:49.871754+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:38:35.790379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:35.790379+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":862,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.891116+0000","last_change":"2024-02-05T07:36:54.146303+0000","last_active":"2024-02-05T07:40:51.891116+0000","last_peered":"2024-02-05T07:40:51.891116+0000","last_clean":"2024-02-05T07:40:51.891116+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:40:51.891116+0000","last_undegraded":"2024-02-05T07:40:51.891116+0000","last_fullsized":"2024-02-05T07:40:51.891116+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:54.146194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:54.146194+0000","last_clean_scrub_stamp":"2024-02-05T07:36:54.146194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:57.073090+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":304,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.874531+0000","last_change":"2024-02-05T07:38:09.492094+0000","last_active":"2024-02-05T07:40:49.874531+0000","last_peered":"2024-02-05T07:40:49.874531+0000","last_clean":"2024-02-05T07:40:49.874531+0000","last_became_active":"2024-02-05T07:38:09.491746+0000","last_became_peered":"2024-02-05T07:38:09.491746+0000","last_unstale":"2024-02-05T07:40:49.874531+0000","last_undegraded":"2024-02-05T07:40:49.874531+0000","last_fullsized":"2024-02-05T07:40:49.874531+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:45.362685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.362685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":298,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.874660+0000","last_change":"2024-02-05T07:37:00.310446+0000","last_active":"2024-02-05T07:40:49.874660+0000","last_peered":"2024-02-05T07:40:49.874660+0000","last_clean":"2024-02-05T07:40:49.874660+0000","last_became_active":"2024-02-05T07:36:32.734230+0000","last_became_peered":"2024-02-05T07:36:32.734230+0000","last_unstale":"2024-02-05T07:40:49.874660+0000","last_undegraded":"2024-02-05T07:40:49.874660+0000","last_fullsized":"2024-02-05T07:40:49.874660+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:00.309850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:00.309850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"678'12659","reported_seq":12868,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:53.836280+0000","last_change":"2024-02-05T07:37:53.089683+0000","last_active":"2024-02-05T07:40:53.836280+0000","last_peered":"2024-02-05T07:40:53.836280+0000","last_clean":"2024-02-05T07:40:53.836280+0000","last_became_active":"2024-02-05T07:37:53.089386+0000","last_became_peered":"2024-02-05T07:37:53.089386+0000","last_unstale":"2024-02-05T07:40:53.836280+0000","last_undegraded":"2024-02-05T07:40:53.836280+0000","last_fullsized":"2024-02-05T07:40:53.836280+0000","mapping_epoch":579,"log_start":"639'9629","ondisk_log_start":"639'9629","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":3030,"log_dups_size":3000,"ondisk_log_size":3030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71761920,"num_objects":1095,"num_object_clones":0,"num_object_copies":3285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12659,"num_write_kb":440128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":1286,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892585+0000","last_change":"2024-02-05T07:38:11.660956+0000","last_active":"2024-02-05T07:40:52.892585+0000","last_peered":"2024-02-05T07:40:52.892585+0000","last_clean":"2024-02-05T07:40:52.892585+0000","last_became_active":"2024-02-05T07:38:11.660622+0000","last_became_peered":"2024-02-05T07:38:11.660622+0000","last_unstale":"2024-02-05T07:40:52.892585+0000","last_undegraded":"2024-02-05T07:40:52.892585+0000","last_fullsized":"2024-02-05T07:40:52.892585+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.730473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.730473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"616'51","reported_seq":1252,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892089+0000","last_change":"2024-02-05T07:38:52.517441+0000","last_active":"2024-02-05T07:40:52.892089+0000","last_peered":"2024-02-05T07:40:52.892089+0000","last_clean":"2024-02-05T07:40:52.892089+0000","last_became_active":"2024-02-05T07:36:16.736226+0000","last_became_peered":"2024-02-05T07:36:16.736226+0000","last_unstale":"2024-02-05T07:40:52.892089+0000","last_undegraded":"2024-02-05T07:40:52.892089+0000","last_fullsized":"2024-02-05T07:40:52.892089+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:38:52.517337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:52.517337+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":164,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.875658+0000","last_change":"2024-02-05T07:40:03.305634+0000","last_active":"2024-02-05T07:40:49.875658+0000","last_peered":"2024-02-05T07:40:49.875658+0000","last_clean":"2024-02-05T07:40:49.875658+0000","last_became_active":"2024-02-05T07:38:31.795149+0000","last_became_peered":"2024-02-05T07:38:31.795149+0000","last_unstale":"2024-02-05T07:40:49.875658+0000","last_undegraded":"2024-02-05T07:40:49.875658+0000","last_fullsized":"2024-02-05T07:40:49.875658+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:03.305531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_clean_scrub_stamp":"2024-02-05T07:40:03.305531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:24.210343+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":298,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.876688+0000","last_change":"2024-02-05T07:37:54.312733+0000","last_active":"2024-02-05T07:40:49.876688+0000","last_peered":"2024-02-05T07:40:49.876688+0000","last_clean":"2024-02-05T07:40:49.876688+0000","last_became_active":"2024-02-05T07:36:32.733494+0000","last_became_peered":"2024-02-05T07:36:32.733494+0000","last_unstale":"2024-02-05T07:40:49.876688+0000","last_undegraded":"2024-02-05T07:40:49.876688+0000","last_fullsized":"2024-02-05T07:40:49.876688+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:54.312642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.312642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":609,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.890380+0000","last_change":"2024-02-05T07:38:31.803859+0000","last_active":"2024-02-05T07:40:51.890380+0000","last_peered":"2024-02-05T07:40:51.890380+0000","last_clean":"2024-02-05T07:40:51.890380+0000","last_became_active":"2024-02-05T07:38:31.803474+0000","last_became_peered":"2024-02-05T07:38:31.803474+0000","last_unstale":"2024-02-05T07:40:51.890380+0000","last_undegraded":"2024-02-05T07:40:51.890380+0000","last_fullsized":"2024-02-05T07:40:51.890380+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:01.478555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:38:01.478555+0000","last_clean_scrub_stamp":"2024-02-05T07:38:01.478555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"675'12590","reported_seq":12791,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:50.808281+0000","last_change":"2024-02-05T07:37:53.090459+0000","last_active":"2024-02-05T07:40:50.808281+0000","last_peered":"2024-02-05T07:40:50.808281+0000","last_clean":"2024-02-05T07:40:50.808281+0000","last_became_active":"2024-02-05T07:37:53.090173+0000","last_became_peered":"2024-02-05T07:37:53.090173+0000","last_unstale":"2024-02-05T07:40:50.808281+0000","last_undegraded":"2024-02-05T07:40:50.808281+0000","last_fullsized":"2024-02-05T07:40:50.808281+0000","mapping_epoch":579,"log_start":"636'9206","ondisk_log_start":"636'9206","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.799787+0000","objects_scrubbed":0,"log_size":3384,"log_dups_size":0,"ondisk_log_size":3384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:55.633919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85196800,"num_objects":1300,"num_object_clones":0,"num_object_copies":3900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":444480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":1651,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.890068+0000","last_change":"2024-02-05T07:40:28.961455+0000","last_active":"2024-02-05T07:40:51.890068+0000","last_peered":"2024-02-05T07:40:51.890068+0000","last_clean":"2024-02-05T07:40:51.890068+0000","last_became_active":"2024-02-05T07:40:28.961096+0000","last_became_peered":"2024-02-05T07:40:28.961096+0000","last_unstale":"2024-02-05T07:40:51.890068+0000","last_undegraded":"2024-02-05T07:40:51.890068+0000","last_fullsized":"2024-02-05T07:40:51.890068+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.295329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.295329+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":118,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":298,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.874183+0000","last_change":"2024-02-05T07:37:51.288853+0000","last_active":"2024-02-05T07:40:49.874183+0000","last_peered":"2024-02-05T07:40:49.874183+0000","last_clean":"2024-02-05T07:40:49.874183+0000","last_became_active":"2024-02-05T07:36:32.733003+0000","last_became_peered":"2024-02-05T07:36:32.733003+0000","last_unstale":"2024-02-05T07:40:49.874183+0000","last_undegraded":"2024-02-05T07:40:49.874183+0000","last_fullsized":"2024-02-05T07:40:49.874183+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.288772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.288772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"615'24","reported_seq":1269,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.878618+0000","last_change":"2024-02-05T07:38:30.797759+0000","last_active":"2024-02-05T07:40:49.878618+0000","last_peered":"2024-02-05T07:40:49.878618+0000","last_clean":"2024-02-05T07:40:49.878618+0000","last_became_active":"2024-02-05T07:38:09.480893+0000","last_became_peered":"2024-02-05T07:38:09.480893+0000","last_unstale":"2024-02-05T07:40:49.878618+0000","last_undegraded":"2024-02-05T07:40:49.878618+0000","last_fullsized":"2024-02-05T07:40:49.878618+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'18","last_scrub_stamp":"2024-02-05T07:38:30.797628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:30.797628+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":196,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.887709+0000","last_change":"2024-02-05T07:40:18.107509+0000","last_active":"2024-02-05T07:40:49.887709+0000","last_peered":"2024-02-05T07:40:49.887709+0000","last_clean":"2024-02-05T07:40:49.887709+0000","last_became_active":"2024-02-05T07:38:09.485926+0000","last_became_peered":"2024-02-05T07:38:09.485926+0000","last_unstale":"2024-02-05T07:40:49.887709+0000","last_undegraded":"2024-02-05T07:40:49.887709+0000","last_fullsized":"2024-02-05T07:40:49.887709+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:18.107395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:40:18.107395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:42.598865+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":873,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.901386+0000","last_change":"2024-02-05T07:39:47.983852+0000","last_active":"2024-02-05T07:40:51.901386+0000","last_peered":"2024-02-05T07:40:51.901386+0000","last_clean":"2024-02-05T07:40:51.901386+0000","last_became_active":"2024-02-05T07:30:40.485991+0000","last_became_peered":"2024-02-05T07:30:40.485991+0000","last_unstale":"2024-02-05T07:40:51.901386+0000","last_undegraded":"2024-02-05T07:40:51.901386+0000","last_fullsized":"2024-02-05T07:40:51.901386+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:47.982166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:47.982166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:57.074007+0000","scrub_duration":274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"675'12466","reported_seq":13809,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:50.655104+0000","last_change":"2024-02-05T07:40:02.890033+0000","last_active":"2024-02-05T07:40:50.655104+0000","last_peered":"2024-02-05T07:40:50.655104+0000","last_clean":"2024-02-05T07:40:50.655104+0000","last_became_active":"2024-02-05T07:38:15.650851+0000","last_became_peered":"2024-02-05T07:38:15.650851+0000","last_unstale":"2024-02-05T07:40:50.655104+0000","last_undegraded":"2024-02-05T07:40:50.655104+0000","last_fullsized":"2024-02-05T07:40:50.655104+0000","mapping_epoch":595,"log_start":"639'9700","ondisk_log_start":"639'9700","created":579,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'8919","last_scrub_stamp":"2024-02-05T07:40:02.889877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:51.799787+0000","last_clean_scrub_stamp":"2024-02-05T07:40:02.889877+0000","objects_scrubbed":4908,"log_size":2766,"log_dups_size":3000,"ondisk_log_size":2766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:14.278208+0000","scrub_duration":2873,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85196800,"num_objects":1300,"num_object_clones":0,"num_object_copies":3900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12466,"num_write_kb":440512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":536,"num_bytes_recovered":35127296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":1276,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.875773+0000","last_change":"2024-02-05T07:40:24.720852+0000","last_active":"2024-02-05T07:40:49.875773+0000","last_peered":"2024-02-05T07:40:49.875773+0000","last_clean":"2024-02-05T07:40:49.875773+0000","last_became_active":"2024-02-05T07:40:24.720423+0000","last_became_peered":"2024-02-05T07:40:24.720423+0000","last_unstale":"2024-02-05T07:40:49.875773+0000","last_undegraded":"2024-02-05T07:40:49.875773+0000","last_fullsized":"2024-02-05T07:40:49.875773+0000","mapping_epoch":595,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:33.547397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:33.547397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"589'13","reported_seq":1420,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891714+0000","last_change":"2024-02-05T07:38:09.523382+0000","last_active":"2024-02-05T07:40:52.891714+0000","last_peered":"2024-02-05T07:40:52.891714+0000","last_clean":"2024-02-05T07:40:52.891714+0000","last_became_active":"2024-02-05T07:38:09.471365+0000","last_became_peered":"2024-02-05T07:38:09.471365+0000","last_unstale":"2024-02-05T07:40:52.891714+0000","last_undegraded":"2024-02-05T07:40:52.891714+0000","last_fullsized":"2024-02-05T07:40:52.891714+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'12","last_scrub_stamp":"2024-02-05T07:37:53.638258+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:37:53.638258+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"615'39","reported_seq":962,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891366+0000","last_change":"2024-02-05T07:38:09.513629+0000","last_active":"2024-02-05T07:40:52.891366+0000","last_peered":"2024-02-05T07:40:52.891366+0000","last_clean":"2024-02-05T07:40:52.891366+0000","last_became_active":"2024-02-05T07:38:09.471133+0000","last_became_peered":"2024-02-05T07:38:09.471133+0000","last_unstale":"2024-02-05T07:40:52.891366+0000","last_undegraded":"2024-02-05T07:40:52.891366+0000","last_fullsized":"2024-02-05T07:40:52.891366+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:38:00.687570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:00.687570+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"614'4","reported_seq":213,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.889758+0000","last_change":"2024-02-05T07:38:55.830017+0000","last_active":"2024-02-05T07:40:49.889758+0000","last_peered":"2024-02-05T07:40:49.889758+0000","last_clean":"2024-02-05T07:40:49.889758+0000","last_became_active":"2024-02-05T07:38:09.482757+0000","last_became_peered":"2024-02-05T07:38:09.482757+0000","last_unstale":"2024-02-05T07:40:49.889758+0000","last_undegraded":"2024-02-05T07:40:49.889758+0000","last_fullsized":"2024-02-05T07:40:49.889758+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"614'4","last_scrub_stamp":"2024-02-05T07:38:55.829386+0000","last_deep_scrub":"614'4","last_deep_scrub_stamp":"2024-02-05T07:38:55.829386+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.829386+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":961,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891883+0000","last_change":"2024-02-05T07:37:02.526520+0000","last_active":"2024-02-05T07:40:52.891883+0000","last_peered":"2024-02-05T07:40:52.891883+0000","last_clean":"2024-02-05T07:40:52.891883+0000","last_became_active":"2024-02-05T07:36:32.701535+0000","last_became_peered":"2024-02-05T07:36:32.701535+0000","last_unstale":"2024-02-05T07:40:52.891883+0000","last_undegraded":"2024-02-05T07:40:52.891883+0000","last_fullsized":"2024-02-05T07:40:52.891883+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:02.526362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:37:02.526362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":597,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.871667+0000","last_change":"2024-02-05T07:38:57.845026+0000","last_active":"2024-02-05T07:40:49.871667+0000","last_peered":"2024-02-05T07:40:49.871667+0000","last_clean":"2024-02-05T07:40:49.871667+0000","last_became_active":"2024-02-05T07:37:41.643328+0000","last_became_peered":"2024-02-05T07:37:41.643328+0000","last_unstale":"2024-02-05T07:40:49.871667+0000","last_undegraded":"2024-02-05T07:40:49.871667+0000","last_fullsized":"2024-02-05T07:40:49.871667+0000","mapping_epoch":576,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:57.844891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:57.844891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"615'21","reported_seq":929,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.896865+0000","last_change":"2024-02-05T07:38:48.343585+0000","last_active":"2024-02-05T07:40:51.896865+0000","last_peered":"2024-02-05T07:40:51.896865+0000","last_clean":"2024-02-05T07:40:51.896865+0000","last_became_active":"2024-02-05T07:36:32.742483+0000","last_became_peered":"2024-02-05T07:36:32.742483+0000","last_unstale":"2024-02-05T07:40:51.896865+0000","last_undegraded":"2024-02-05T07:40:51.896865+0000","last_fullsized":"2024-02-05T07:40:51.896865+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'15","last_scrub_stamp":"2024-02-05T07:38:48.343448+0000","last_deep_scrub":"613'15","last_deep_scrub_stamp":"2024-02-05T07:38:48.343448+0000","last_clean_scrub_stamp":"2024-02-05T07:38:48.343448+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":1093,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891171+0000","last_change":"2024-02-05T07:37:47.639730+0000","last_active":"2024-02-05T07:40:52.891171+0000","last_peered":"2024-02-05T07:40:52.891171+0000","last_clean":"2024-02-05T07:40:52.891171+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:40:52.891171+0000","last_undegraded":"2024-02-05T07:40:52.891171+0000","last_fullsized":"2024-02-05T07:40:52.891171+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:37:47.639652+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:37:47.639652+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":861,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.892797+0000","last_change":"2024-02-05T07:37:01.178948+0000","last_active":"2024-02-05T07:40:51.892797+0000","last_peered":"2024-02-05T07:40:51.892797+0000","last_clean":"2024-02-05T07:40:51.892797+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:40:51.892797+0000","last_undegraded":"2024-02-05T07:40:51.892797+0000","last_fullsized":"2024-02-05T07:40:51.892797+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:01.178816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:01.178816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1157,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892126+0000","last_change":"2024-02-05T07:38:41.820804+0000","last_active":"2024-02-05T07:40:52.892126+0000","last_peered":"2024-02-05T07:40:52.892126+0000","last_clean":"2024-02-05T07:40:52.892126+0000","last_became_active":"2024-02-05T07:38:09.480228+0000","last_became_peered":"2024-02-05T07:38:09.480228+0000","last_unstale":"2024-02-05T07:40:52.892126+0000","last_undegraded":"2024-02-05T07:40:52.892126+0000","last_fullsized":"2024-02-05T07:40:52.892126+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:41.820713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:38:41.820713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":1093,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.901050+0000","last_change":"2024-02-05T07:38:41.750699+0000","last_active":"2024-02-05T07:40:51.901050+0000","last_peered":"2024-02-05T07:40:51.901050+0000","last_clean":"2024-02-05T07:40:51.901050+0000","last_became_active":"2024-02-05T07:38:41.750174+0000","last_became_peered":"2024-02-05T07:38:41.750174+0000","last_unstale":"2024-02-05T07:40:51.901050+0000","last_undegraded":"2024-02-05T07:40:51.901050+0000","last_fullsized":"2024-02-05T07:40:51.901050+0000","mapping_epoch":607,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.260984+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.260984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"615'21","reported_seq":934,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.892583+0000","last_change":"2024-02-05T07:38:55.387276+0000","last_active":"2024-02-05T07:40:51.892583+0000","last_peered":"2024-02-05T07:40:51.892583+0000","last_clean":"2024-02-05T07:40:51.892583+0000","last_became_active":"2024-02-05T07:36:32.747221+0000","last_became_peered":"2024-02-05T07:36:32.747221+0000","last_unstale":"2024-02-05T07:40:51.892583+0000","last_undegraded":"2024-02-05T07:40:51.892583+0000","last_fullsized":"2024-02-05T07:40:51.892583+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:38:55.387145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.387145+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":708,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892535+0000","last_change":"2024-02-05T07:39:00.488430+0000","last_active":"2024-02-05T07:40:52.892535+0000","last_peered":"2024-02-05T07:40:52.892535+0000","last_clean":"2024-02-05T07:40:52.892535+0000","last_became_active":"2024-02-05T07:38:09.479116+0000","last_became_peered":"2024-02-05T07:38:09.479116+0000","last_unstale":"2024-02-05T07:40:52.892535+0000","last_undegraded":"2024-02-05T07:40:52.892535+0000","last_fullsized":"2024-02-05T07:40:52.892535+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:00.488357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:39:00.488357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":864,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.896980+0000","last_change":"2024-02-05T07:38:53.372481+0000","last_active":"2024-02-05T07:40:51.896980+0000","last_peered":"2024-02-05T07:40:51.896980+0000","last_clean":"2024-02-05T07:40:51.896980+0000","last_became_active":"2024-02-05T07:30:40.496076+0000","last_became_peered":"2024-02-05T07:30:40.496076+0000","last_unstale":"2024-02-05T07:40:51.896980+0000","last_undegraded":"2024-02-05T07:40:51.896980+0000","last_fullsized":"2024-02-05T07:40:51.896980+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:53.372355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_clean_scrub_stamp":"2024-02-05T07:38:53.372355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":304,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.884544+0000","last_change":"2024-02-05T07:38:09.489174+0000","last_active":"2024-02-05T07:40:49.884544+0000","last_peered":"2024-02-05T07:40:49.884544+0000","last_clean":"2024-02-05T07:40:49.884544+0000","last_became_active":"2024-02-05T07:38:09.488755+0000","last_became_peered":"2024-02-05T07:38:09.488755+0000","last_unstale":"2024-02-05T07:40:49.884544+0000","last_undegraded":"2024-02-05T07:40:49.884544+0000","last_fullsized":"2024-02-05T07:40:49.884544+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:47.330693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:37:47.330693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":1088,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891791+0000","last_change":"2024-02-05T07:38:41.725370+0000","last_active":"2024-02-05T07:40:52.891791+0000","last_peered":"2024-02-05T07:40:52.891791+0000","last_clean":"2024-02-05T07:40:52.891791+0000","last_became_active":"2024-02-05T07:38:41.725034+0000","last_became_peered":"2024-02-05T07:38:41.725034+0000","last_unstale":"2024-02-05T07:40:52.891791+0000","last_undegraded":"2024-02-05T07:40:52.891791+0000","last_fullsized":"2024-02-05T07:40:52.891791+0000","mapping_epoch":607,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:37.743034+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:37:37.743034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"615'25","reported_seq":1222,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.897352+0000","last_change":"2024-02-05T07:37:06.319292+0000","last_active":"2024-02-05T07:40:51.897352+0000","last_peered":"2024-02-05T07:40:51.897352+0000","last_clean":"2024-02-05T07:40:51.897352+0000","last_became_active":"2024-02-05T07:30:40.483700+0000","last_became_peered":"2024-02-05T07:30:40.483700+0000","last_unstale":"2024-02-05T07:40:51.897352+0000","last_undegraded":"2024-02-05T07:40:51.897352+0000","last_fullsized":"2024-02-05T07:40:51.897352+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:37:06.318619+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:37:06.318619+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1027,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.897723+0000","last_change":"2024-02-05T07:36:46.059245+0000","last_active":"2024-02-05T07:40:51.897723+0000","last_peered":"2024-02-05T07:40:51.897723+0000","last_clean":"2024-02-05T07:40:51.897723+0000","last_became_active":"2024-02-05T07:30:40.476058+0000","last_became_peered":"2024-02-05T07:30:40.476058+0000","last_unstale":"2024-02-05T07:40:51.897723+0000","last_undegraded":"2024-02-05T07:40:51.897723+0000","last_fullsized":"2024-02-05T07:40:51.897723+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:46.059129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.059129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"538'13","reported_seq":1055,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.890715+0000","last_change":"2024-02-05T07:38:51.305914+0000","last_active":"2024-02-05T07:40:51.890715+0000","last_peered":"2024-02-05T07:40:51.890715+0000","last_clean":"2024-02-05T07:40:51.890715+0000","last_became_active":"2024-02-05T07:36:32.746270+0000","last_became_peered":"2024-02-05T07:36:32.746270+0000","last_unstale":"2024-02-05T07:40:51.890715+0000","last_undegraded":"2024-02-05T07:40:51.890715+0000","last_fullsized":"2024-02-05T07:40:51.890715+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'13","last_scrub_stamp":"2024-02-05T07:38:51.305782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:51.305782+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1167,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.879823+0000","last_change":"2024-02-05T07:38:09.504184+0000","last_active":"2024-02-05T07:40:49.879823+0000","last_peered":"2024-02-05T07:40:49.879823+0000","last_clean":"2024-02-05T07:40:49.879823+0000","last_became_active":"2024-02-05T07:38:09.503255+0000","last_became_peered":"2024-02-05T07:38:09.503255+0000","last_unstale":"2024-02-05T07:40:49.879823+0000","last_undegraded":"2024-02-05T07:40:49.879823+0000","last_fullsized":"2024-02-05T07:40:49.879823+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:58.277028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:58.277028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":650,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.898211+0000","last_change":"2024-02-05T07:37:40.289255+0000","last_active":"2024-02-05T07:40:51.898211+0000","last_peered":"2024-02-05T07:40:51.898211+0000","last_clean":"2024-02-05T07:40:51.898211+0000","last_became_active":"2024-02-05T07:37:12.040170+0000","last_became_peered":"2024-02-05T07:37:12.040170+0000","last_unstale":"2024-02-05T07:40:51.898211+0000","last_undegraded":"2024-02-05T07:40:51.898211+0000","last_fullsized":"2024-02-05T07:40:51.898211+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:40.289174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:40.289174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"615'35","reported_seq":954,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.890993+0000","last_change":"2024-02-05T07:36:59.117388+0000","last_active":"2024-02-05T07:40:51.890993+0000","last_peered":"2024-02-05T07:40:51.890993+0000","last_clean":"2024-02-05T07:40:51.890993+0000","last_became_active":"2024-02-05T07:36:32.746757+0000","last_became_peered":"2024-02-05T07:36:32.746757+0000","last_unstale":"2024-02-05T07:40:51.890993+0000","last_undegraded":"2024-02-05T07:40:51.890993+0000","last_fullsized":"2024-02-05T07:40:51.890993+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:36:59.117218+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.117218+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":560,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.876428+0000","last_change":"2024-02-05T07:39:53.396917+0000","last_active":"2024-02-05T07:40:49.876428+0000","last_peered":"2024-02-05T07:40:49.876428+0000","last_clean":"2024-02-05T07:40:49.876428+0000","last_became_active":"2024-02-05T07:38:09.495040+0000","last_became_peered":"2024-02-05T07:38:09.495040+0000","last_unstale":"2024-02-05T07:40:49.876428+0000","last_undegraded":"2024-02-05T07:40:49.876428+0000","last_fullsized":"2024-02-05T07:40:49.876428+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:53.396795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:39:53.396795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:06.394109+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"584'12","reported_seq":245,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.891989+0000","last_change":"2024-02-05T07:40:05.765708+0000","last_active":"2024-02-05T07:40:51.891989+0000","last_peered":"2024-02-05T07:40:51.891989+0000","last_clean":"2024-02-05T07:40:51.891989+0000","last_became_active":"2024-02-05T07:37:58.781263+0000","last_became_peered":"2024-02-05T07:37:58.781263+0000","last_unstale":"2024-02-05T07:40:51.891989+0000","last_undegraded":"2024-02-05T07:40:51.891989+0000","last_fullsized":"2024-02-05T07:40:51.891989+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'12","last_scrub_stamp":"2024-02-05T07:40:05.765574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:40:05.765574+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:13.477481+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1029,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.892669+0000","last_change":"2024-02-05T07:39:01.508990+0000","last_active":"2024-02-05T07:40:51.892669+0000","last_peered":"2024-02-05T07:40:51.892669+0000","last_clean":"2024-02-05T07:40:51.892669+0000","last_became_active":"2024-02-05T07:38:09.487782+0000","last_became_peered":"2024-02-05T07:38:09.487782+0000","last_unstale":"2024-02-05T07:40:51.892669+0000","last_undegraded":"2024-02-05T07:40:51.892669+0000","last_fullsized":"2024-02-05T07:40:51.892669+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:01.508866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:01.508866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"615'27","reported_seq":821,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892019+0000","last_change":"2024-02-05T07:38:09.532591+0000","last_active":"2024-02-05T07:40:52.892019+0000","last_peered":"2024-02-05T07:40:52.892019+0000","last_clean":"2024-02-05T07:40:52.892019+0000","last_became_active":"2024-02-05T07:38:09.484252+0000","last_became_peered":"2024-02-05T07:38:09.484252+0000","last_unstale":"2024-02-05T07:40:52.892019+0000","last_undegraded":"2024-02-05T07:40:52.892019+0000","last_fullsized":"2024-02-05T07:40:52.892019+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:37:48.219951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:48.219951+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1202,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.893901+0000","last_change":"2024-02-05T07:38:32.811706+0000","last_active":"2024-02-05T07:40:52.893901+0000","last_peered":"2024-02-05T07:40:52.893901+0000","last_clean":"2024-02-05T07:40:52.893901+0000","last_became_active":"2024-02-05T07:38:09.472022+0000","last_became_peered":"2024-02-05T07:38:09.472022+0000","last_unstale":"2024-02-05T07:40:52.893901+0000","last_undegraded":"2024-02-05T07:40:52.893901+0000","last_fullsized":"2024-02-05T07:40:52.893901+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:32.811595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:38:32.811595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1296,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.872371+0000","last_change":"2024-02-05T07:39:55.907631+0000","last_active":"2024-02-05T07:40:49.872371+0000","last_peered":"2024-02-05T07:40:49.872371+0000","last_clean":"2024-02-05T07:40:49.872371+0000","last_became_active":"2024-02-05T07:30:40.487141+0000","last_became_peered":"2024-02-05T07:30:40.487141+0000","last_unstale":"2024-02-05T07:40:49.872371+0000","last_undegraded":"2024-02-05T07:40:49.872371+0000","last_fullsized":"2024-02-05T07:40:49.872371+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:55.907516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_clean_scrub_stamp":"2024-02-05T07:39:55.907516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:23.259237+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":868,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.897101+0000","last_change":"2024-02-05T07:37:54.469820+0000","last_active":"2024-02-05T07:40:51.897101+0000","last_peered":"2024-02-05T07:40:51.897101+0000","last_clean":"2024-02-05T07:40:51.897101+0000","last_became_active":"2024-02-05T07:36:32.742150+0000","last_became_peered":"2024-02-05T07:36:32.742150+0000","last_unstale":"2024-02-05T07:40:51.897101+0000","last_undegraded":"2024-02-05T07:40:51.897101+0000","last_fullsized":"2024-02-05T07:40:51.897101+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:54.469701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.469701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"615'21","reported_seq":942,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.871783+0000","last_change":"2024-02-05T07:38:37.758235+0000","last_active":"2024-02-05T07:40:49.871783+0000","last_peered":"2024-02-05T07:40:49.871783+0000","last_clean":"2024-02-05T07:40:49.871783+0000","last_became_active":"2024-02-05T07:36:32.710090+0000","last_became_peered":"2024-02-05T07:36:32.710090+0000","last_unstale":"2024-02-05T07:40:49.871783+0000","last_undegraded":"2024-02-05T07:40:49.871783+0000","last_fullsized":"2024-02-05T07:40:49.871783+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:38:37.758114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:37.758114+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1078,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.897219+0000","last_change":"2024-02-05T07:38:09.473515+0000","last_active":"2024-02-05T07:40:51.897219+0000","last_peered":"2024-02-05T07:40:51.897219+0000","last_clean":"2024-02-05T07:40:51.897219+0000","last_became_active":"2024-02-05T07:38:09.472668+0000","last_became_peered":"2024-02-05T07:38:09.472668+0000","last_unstale":"2024-02-05T07:40:51.897219+0000","last_undegraded":"2024-02-05T07:40:51.897219+0000","last_fullsized":"2024-02-05T07:40:51.897219+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:32.120410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:32.120410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":858,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.901676+0000","last_change":"2024-02-05T07:38:54.397360+0000","last_active":"2024-02-05T07:40:51.901676+0000","last_peered":"2024-02-05T07:40:51.901676+0000","last_clean":"2024-02-05T07:40:51.901676+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:40:51.901676+0000","last_undegraded":"2024-02-05T07:40:51.901676+0000","last_fullsized":"2024-02-05T07:40:51.901676+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:54.397113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:38:54.397113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":746,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891577+0000","last_change":"2024-02-05T07:40:01.732873+0000","last_active":"2024-02-05T07:40:52.891577+0000","last_peered":"2024-02-05T07:40:52.891577+0000","last_clean":"2024-02-05T07:40:52.891577+0000","last_became_active":"2024-02-05T07:34:30.670179+0000","last_became_peered":"2024-02-05T07:34:30.670179+0000","last_unstale":"2024-02-05T07:40:52.891577+0000","last_undegraded":"2024-02-05T07:40:52.891577+0000","last_fullsized":"2024-02-05T07:40:52.891577+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:01.732803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:40:01.732803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:29.909417+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":1650,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.879616+0000","last_change":"2024-02-05T07:39:11.702922+0000","last_active":"2024-02-05T07:40:49.879616+0000","last_peered":"2024-02-05T07:40:49.879616+0000","last_clean":"2024-02-05T07:40:49.879616+0000","last_became_active":"2024-02-05T07:39:11.702286+0000","last_became_peered":"2024-02-05T07:39:11.702286+0000","last_unstale":"2024-02-05T07:40:49.879616+0000","last_undegraded":"2024-02-05T07:40:49.879616+0000","last_fullsized":"2024-02-05T07:40:49.879616+0000","mapping_epoch":619,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.395638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.395638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"616'27","reported_seq":915,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.896771+0000","last_change":"2024-02-05T07:36:48.088544+0000","last_active":"2024-02-05T07:40:51.896771+0000","last_peered":"2024-02-05T07:40:51.896771+0000","last_clean":"2024-02-05T07:40:51.896771+0000","last_became_active":"2024-02-05T07:30:40.501893+0000","last_became_peered":"2024-02-05T07:30:40.501893+0000","last_unstale":"2024-02-05T07:40:51.896771+0000","last_undegraded":"2024-02-05T07:40:51.896771+0000","last_fullsized":"2024-02-05T07:40:51.896771+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:36:48.088406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:36:48.088406+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":689,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.887853+0000","last_change":"2024-02-05T07:38:54.868354+0000","last_active":"2024-02-05T07:40:49.887853+0000","last_peered":"2024-02-05T07:40:49.887853+0000","last_clean":"2024-02-05T07:40:49.887853+0000","last_became_active":"2024-02-05T07:36:32.712612+0000","last_became_peered":"2024-02-05T07:36:32.712612+0000","last_unstale":"2024-02-05T07:40:49.887853+0000","last_undegraded":"2024-02-05T07:40:49.887853+0000","last_fullsized":"2024-02-05T07:40:49.887853+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:54.867940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:54.867940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":700,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891230+0000","last_change":"2024-02-05T07:38:59.527516+0000","last_active":"2024-02-05T07:40:52.891230+0000","last_peered":"2024-02-05T07:40:52.891230+0000","last_clean":"2024-02-05T07:40:52.891230+0000","last_became_active":"2024-02-05T07:38:09.480566+0000","last_became_peered":"2024-02-05T07:38:09.480566+0000","last_unstale":"2024-02-05T07:40:52.891230+0000","last_undegraded":"2024-02-05T07:40:52.891230+0000","last_fullsized":"2024-02-05T07:40:52.891230+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:59.527436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:38:59.527436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":870,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.901212+0000","last_change":"2024-02-05T07:40:20.094896+0000","last_active":"2024-02-05T07:40:51.901212+0000","last_peered":"2024-02-05T07:40:51.901212+0000","last_clean":"2024-02-05T07:40:51.901212+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:40:51.901212+0000","last_undegraded":"2024-02-05T07:40:51.901212+0000","last_fullsized":"2024-02-05T07:40:51.901212+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:20.094078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:40:20.094078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:33.467712+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"86'128","reported_seq":1271,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.876342+0000","last_change":"2024-02-05T07:39:46.502942+0000","last_active":"2024-02-05T07:40:49.876342+0000","last_peered":"2024-02-05T07:40:49.876342+0000","last_clean":"2024-02-05T07:40:49.876342+0000","last_became_active":"2024-02-05T07:39:17.741563+0000","last_became_peered":"2024-02-05T07:39:17.741563+0000","last_unstale":"2024-02-05T07:40:49.876342+0000","last_undegraded":"2024-02-05T07:40:49.876342+0000","last_fullsized":"2024-02-05T07:40:49.876342+0000","mapping_epoch":623,"log_start":"86'128","ondisk_log_start":"86'128","created":571,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:39:46.502804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:39:46.502804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:49.898734+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"615'12","reported_seq":880,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.891286+0000","last_change":"2024-02-05T07:35:47.070886+0000","last_active":"2024-02-05T07:40:51.891286+0000","last_peered":"2024-02-05T07:40:51.891286+0000","last_clean":"2024-02-05T07:40:51.891286+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:40:51.891286+0000","last_undegraded":"2024-02-05T07:40:51.891286+0000","last_fullsized":"2024-02-05T07:40:51.891286+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:35:47.070805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:47.070805+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":869,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.896706+0000","last_change":"2024-02-05T07:40:07.758232+0000","last_active":"2024-02-05T07:40:51.896706+0000","last_peered":"2024-02-05T07:40:51.896706+0000","last_clean":"2024-02-05T07:40:51.896706+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:40:51.896706+0000","last_undegraded":"2024-02-05T07:40:51.896706+0000","last_fullsized":"2024-02-05T07:40:51.896706+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:07.758093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:34:39.980067+0000","last_clean_scrub_stamp":"2024-02-05T07:40:07.758093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:33.830081+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1141,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891717+0000","last_change":"2024-02-05T07:40:02.708852+0000","last_active":"2024-02-05T07:40:52.891717+0000","last_peered":"2024-02-05T07:40:52.891717+0000","last_clean":"2024-02-05T07:40:52.891717+0000","last_became_active":"2024-02-05T07:38:09.464446+0000","last_became_peered":"2024-02-05T07:38:09.464446+0000","last_unstale":"2024-02-05T07:40:52.891717+0000","last_undegraded":"2024-02-05T07:40:52.891717+0000","last_fullsized":"2024-02-05T07:40:52.891717+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:02.708785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:40:02.708785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:06.491220+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":304,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.883945+0000","last_change":"2024-02-05T07:38:09.504998+0000","last_active":"2024-02-05T07:40:49.883945+0000","last_peered":"2024-02-05T07:40:49.883945+0000","last_clean":"2024-02-05T07:40:49.883945+0000","last_became_active":"2024-02-05T07:38:09.503369+0000","last_became_peered":"2024-02-05T07:38:09.503369+0000","last_unstale":"2024-02-05T07:40:49.883945+0000","last_undegraded":"2024-02-05T07:40:49.883945+0000","last_fullsized":"2024-02-05T07:40:49.883945+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:58.296469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:58.296469+0000","last_clean_scrub_stamp":"2024-02-05T07:36:58.296469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"616'27","reported_seq":1134,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892774+0000","last_change":"2024-02-05T07:38:55.551835+0000","last_active":"2024-02-05T07:40:52.892774+0000","last_peered":"2024-02-05T07:40:52.892774+0000","last_clean":"2024-02-05T07:40:52.892774+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:40:52.892774+0000","last_undegraded":"2024-02-05T07:40:52.892774+0000","last_fullsized":"2024-02-05T07:40:52.892774+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:38:55.551751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.551751+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":866,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.890665+0000","last_change":"2024-02-05T07:40:11.850388+0000","last_active":"2024-02-05T07:40:51.890665+0000","last_peered":"2024-02-05T07:40:51.890665+0000","last_clean":"2024-02-05T07:40:51.890665+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:40:51.890665+0000","last_undegraded":"2024-02-05T07:40:51.890665+0000","last_fullsized":"2024-02-05T07:40:51.890665+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:11.849558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:40:11.849558+0000","last_clean_scrub_stamp":"2024-02-05T07:40:11.849558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:35.605390+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":307,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.883319+0000","last_change":"2024-02-05T07:40:15.407981+0000","last_active":"2024-02-05T07:40:49.883319+0000","last_peered":"2024-02-05T07:40:49.883319+0000","last_clean":"2024-02-05T07:40:49.883319+0000","last_became_active":"2024-02-05T07:36:32.752879+0000","last_became_peered":"2024-02-05T07:36:32.752879+0000","last_unstale":"2024-02-05T07:40:49.883319+0000","last_undegraded":"2024-02-05T07:40:49.883319+0000","last_fullsized":"2024-02-05T07:40:49.883319+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:15.407749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:40:15.407749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:41.593247+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":793,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.876181+0000","last_change":"2024-02-05T07:40:20.624459+0000","last_active":"2024-02-05T07:40:49.876181+0000","last_peered":"2024-02-05T07:40:49.876181+0000","last_clean":"2024-02-05T07:40:49.876181+0000","last_became_active":"2024-02-05T07:38:09.503835+0000","last_became_peered":"2024-02-05T07:38:09.503835+0000","last_unstale":"2024-02-05T07:40:49.876181+0000","last_undegraded":"2024-02-05T07:40:49.876181+0000","last_fullsized":"2024-02-05T07:40:49.876181+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:20.624132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:40:20.624132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:28.219914+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"616'30","reported_seq":1041,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.892874+0000","last_change":"2024-02-05T07:35:13.094823+0000","last_active":"2024-02-05T07:40:51.892874+0000","last_peered":"2024-02-05T07:40:51.892874+0000","last_clean":"2024-02-05T07:40:51.892874+0000","last_became_active":"2024-02-05T07:30:40.495954+0000","last_became_peered":"2024-02-05T07:30:40.495954+0000","last_unstale":"2024-02-05T07:40:51.892874+0000","last_undegraded":"2024-02-05T07:40:51.892874+0000","last_fullsized":"2024-02-05T07:40:51.892874+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:35:13.094694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:13.094694+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"585'2","reported_seq":159,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.873833+0000","last_change":"2024-02-05T07:38:31.834917+0000","last_active":"2024-02-05T07:40:49.873833+0000","last_peered":"2024-02-05T07:40:49.873833+0000","last_clean":"2024-02-05T07:40:49.873833+0000","last_became_active":"2024-02-05T07:38:31.811200+0000","last_became_peered":"2024-02-05T07:38:31.811200+0000","last_unstale":"2024-02-05T07:40:49.873833+0000","last_undegraded":"2024-02-05T07:40:49.873833+0000","last_fullsized":"2024-02-05T07:40:49.873833+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'2","last_scrub_stamp":"2024-02-05T07:38:00.631235+0000","last_deep_scrub":"585'2","last_deep_scrub_stamp":"2024-02-05T07:38:00.631235+0000","last_clean_scrub_stamp":"2024-02-05T07:38:00.631235+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"589'14","reported_seq":1406,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892911+0000","last_change":"2024-02-05T07:38:09.524876+0000","last_active":"2024-02-05T07:40:52.892911+0000","last_peered":"2024-02-05T07:40:52.892911+0000","last_clean":"2024-02-05T07:40:52.892911+0000","last_became_active":"2024-02-05T07:38:09.472910+0000","last_became_peered":"2024-02-05T07:38:09.472910+0000","last_unstale":"2024-02-05T07:40:52.892911+0000","last_undegraded":"2024-02-05T07:40:52.892911+0000","last_fullsized":"2024-02-05T07:40:52.892911+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'13","last_scrub_stamp":"2024-02-05T07:36:57.487913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:57.487913+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":312,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.884293+0000","last_change":"2024-02-05T07:40:09.332550+0000","last_active":"2024-02-05T07:40:49.884293+0000","last_peered":"2024-02-05T07:40:49.884293+0000","last_clean":"2024-02-05T07:40:49.884293+0000","last_became_active":"2024-02-05T07:38:09.482346+0000","last_became_peered":"2024-02-05T07:38:09.482346+0000","last_unstale":"2024-02-05T07:40:49.884293+0000","last_undegraded":"2024-02-05T07:40:49.884293+0000","last_fullsized":"2024-02-05T07:40:49.884293+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:40:09.332235+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:40:09.332235+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:35.166123+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"616'29","reported_seq":919,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.892197+0000","last_change":"2024-02-05T07:40:06.764929+0000","last_active":"2024-02-05T07:40:51.892197+0000","last_peered":"2024-02-05T07:40:51.892197+0000","last_clean":"2024-02-05T07:40:51.892197+0000","last_became_active":"2024-02-05T07:38:09.486225+0000","last_became_peered":"2024-02-05T07:38:09.486225+0000","last_unstale":"2024-02-05T07:40:51.892197+0000","last_undegraded":"2024-02-05T07:40:51.892197+0000","last_fullsized":"2024-02-05T07:40:51.892197+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'29","last_scrub_stamp":"2024-02-05T07:40:06.764791+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:40:06.764791+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:23.569118+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":304,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.884040+0000","last_change":"2024-02-05T07:38:09.489856+0000","last_active":"2024-02-05T07:40:49.884040+0000","last_peered":"2024-02-05T07:40:49.884040+0000","last_clean":"2024-02-05T07:40:49.884040+0000","last_became_active":"2024-02-05T07:38:09.489555+0000","last_became_peered":"2024-02-05T07:38:09.489555+0000","last_unstale":"2024-02-05T07:40:49.884040+0000","last_undegraded":"2024-02-05T07:40:49.884040+0000","last_fullsized":"2024-02-05T07:40:49.884040+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:59.269484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.269484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1184,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.890809+0000","last_change":"2024-02-05T07:38:09.484586+0000","last_active":"2024-02-05T07:40:51.890809+0000","last_peered":"2024-02-05T07:40:51.890809+0000","last_clean":"2024-02-05T07:40:51.890809+0000","last_became_active":"2024-02-05T07:38:09.484261+0000","last_became_peered":"2024-02-05T07:38:09.484261+0000","last_unstale":"2024-02-05T07:40:51.890809+0000","last_undegraded":"2024-02-05T07:40:51.890809+0000","last_fullsized":"2024-02-05T07:40:51.890809+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:47.437414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:47.437414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":264,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892684+0000","last_change":"2024-02-05T07:38:35.940570+0000","last_active":"2024-02-05T07:40:52.892684+0000","last_peered":"2024-02-05T07:40:52.892684+0000","last_clean":"2024-02-05T07:40:52.892684+0000","last_became_active":"2024-02-05T07:38:31.812780+0000","last_became_peered":"2024-02-05T07:38:31.812780+0000","last_unstale":"2024-02-05T07:40:52.892684+0000","last_undegraded":"2024-02-05T07:40:52.892684+0000","last_fullsized":"2024-02-05T07:40:52.892684+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:35.940472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:35.940472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"615'39","reported_seq":332,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.879925+0000","last_change":"2024-02-05T07:37:57.440904+0000","last_active":"2024-02-05T07:40:49.879925+0000","last_peered":"2024-02-05T07:40:49.879925+0000","last_clean":"2024-02-05T07:40:49.879925+0000","last_became_active":"2024-02-05T07:36:32.744301+0000","last_became_peered":"2024-02-05T07:36:32.744301+0000","last_unstale":"2024-02-05T07:40:49.879925+0000","last_undegraded":"2024-02-05T07:40:49.879925+0000","last_fullsized":"2024-02-05T07:40:49.879925+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:37:57.440806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:57.440806+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":232,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.876265+0000","last_change":"2024-02-05T07:38:09.501225+0000","last_active":"2024-02-05T07:40:49.876265+0000","last_peered":"2024-02-05T07:40:49.876265+0000","last_clean":"2024-02-05T07:40:49.876265+0000","last_became_active":"2024-02-05T07:38:09.500871+0000","last_became_peered":"2024-02-05T07:38:09.500871+0000","last_unstale":"2024-02-05T07:40:49.876265+0000","last_undegraded":"2024-02-05T07:40:49.876265+0000","last_fullsized":"2024-02-05T07:40:49.876265+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:48.410161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:48.410161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1279,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.879656+0000","last_change":"2024-02-05T07:38:53.859910+0000","last_active":"2024-02-05T07:40:49.879656+0000","last_peered":"2024-02-05T07:40:49.879656+0000","last_clean":"2024-02-05T07:40:49.879656+0000","last_became_active":"2024-02-05T07:38:09.477634+0000","last_became_peered":"2024-02-05T07:38:09.477634+0000","last_unstale":"2024-02-05T07:40:49.879656+0000","last_undegraded":"2024-02-05T07:40:49.879656+0000","last_fullsized":"2024-02-05T07:40:49.879656+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:53.859782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:53.859782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":164,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.882567+0000","last_change":"2024-02-05T07:40:18.398849+0000","last_active":"2024-02-05T07:40:49.882567+0000","last_peered":"2024-02-05T07:40:49.882567+0000","last_clean":"2024-02-05T07:40:49.882567+0000","last_became_active":"2024-02-05T07:38:31.795971+0000","last_became_peered":"2024-02-05T07:38:31.795971+0000","last_unstale":"2024-02-05T07:40:49.882567+0000","last_undegraded":"2024-02-05T07:40:49.882567+0000","last_fullsized":"2024-02-05T07:40:49.882567+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:18.398592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:40:18.398592+0000","last_clean_scrub_stamp":"2024-02-05T07:40:18.398592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:45.308821+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"615'9","reported_seq":781,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.901437+0000","last_change":"2024-02-05T07:38:09.527356+0000","last_active":"2024-02-05T07:40:51.901437+0000","last_peered":"2024-02-05T07:40:51.901437+0000","last_clean":"2024-02-05T07:40:51.901437+0000","last_became_active":"2024-02-05T07:38:09.496642+0000","last_became_peered":"2024-02-05T07:38:09.496642+0000","last_unstale":"2024-02-05T07:40:51.901437+0000","last_undegraded":"2024-02-05T07:40:51.901437+0000","last_fullsized":"2024-02-05T07:40:51.901437+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'7","last_scrub_stamp":"2024-02-05T07:37:06.495165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:06.495165+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":515,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.883005+0000","last_change":"2024-02-05T07:38:47.813224+0000","last_active":"2024-02-05T07:40:49.883005+0000","last_peered":"2024-02-05T07:40:49.883005+0000","last_clean":"2024-02-05T07:40:49.883005+0000","last_became_active":"2024-02-05T07:38:09.495573+0000","last_became_peered":"2024-02-05T07:38:09.495573+0000","last_unstale":"2024-02-05T07:40:49.883005+0000","last_undegraded":"2024-02-05T07:40:49.883005+0000","last_fullsized":"2024-02-05T07:40:49.883005+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:47.812860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:47.812860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":196,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.872183+0000","last_change":"2024-02-05T07:39:45.831179+0000","last_active":"2024-02-05T07:40:49.872183+0000","last_peered":"2024-02-05T07:40:49.872183+0000","last_clean":"2024-02-05T07:40:49.872183+0000","last_became_active":"2024-02-05T07:38:09.462036+0000","last_became_peered":"2024-02-05T07:38:09.462036+0000","last_unstale":"2024-02-05T07:40:49.872183+0000","last_undegraded":"2024-02-05T07:40:49.872183+0000","last_fullsized":"2024-02-05T07:40:49.872183+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:45.831067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:45.831067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:40:57.942891+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1092,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.878993+0000","last_change":"2024-02-05T07:38:33.873655+0000","last_active":"2024-02-05T07:40:49.878993+0000","last_peered":"2024-02-05T07:40:49.878993+0000","last_clean":"2024-02-05T07:40:49.878993+0000","last_became_active":"2024-02-05T07:36:32.710575+0000","last_became_peered":"2024-02-05T07:36:32.710575+0000","last_unstale":"2024-02-05T07:40:49.878993+0000","last_undegraded":"2024-02-05T07:40:49.878993+0000","last_fullsized":"2024-02-05T07:40:49.878993+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:33.873524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:33.873524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"615'25","reported_seq":815,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.897805+0000","last_change":"2024-02-05T07:38:09.506020+0000","last_active":"2024-02-05T07:40:51.897805+0000","last_peered":"2024-02-05T07:40:51.897805+0000","last_clean":"2024-02-05T07:40:51.897805+0000","last_became_active":"2024-02-05T07:38:09.479937+0000","last_became_peered":"2024-02-05T07:38:09.479937+0000","last_unstale":"2024-02-05T07:40:51.897805+0000","last_undegraded":"2024-02-05T07:40:51.897805+0000","last_fullsized":"2024-02-05T07:40:51.897805+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'19","last_scrub_stamp":"2024-02-05T07:35:18.020665+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:35:18.020665+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":524,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.892384+0000","last_change":"2024-02-05T07:38:56.526617+0000","last_active":"2024-02-05T07:40:52.892384+0000","last_peered":"2024-02-05T07:40:52.892384+0000","last_clean":"2024-02-05T07:40:52.892384+0000","last_became_active":"2024-02-05T07:38:09.484875+0000","last_became_peered":"2024-02-05T07:38:09.484875+0000","last_unstale":"2024-02-05T07:40:52.892384+0000","last_undegraded":"2024-02-05T07:40:52.892384+0000","last_fullsized":"2024-02-05T07:40:52.892384+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:56.526522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:56.526522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":204,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891846+0000","last_change":"2024-02-05T07:38:47.564845+0000","last_active":"2024-02-05T07:40:52.891846+0000","last_peered":"2024-02-05T07:40:52.891846+0000","last_clean":"2024-02-05T07:40:52.891846+0000","last_became_active":"2024-02-05T07:38:09.471013+0000","last_became_peered":"2024-02-05T07:38:09.471013+0000","last_unstale":"2024-02-05T07:40:52.891846+0000","last_undegraded":"2024-02-05T07:40:52.891846+0000","last_fullsized":"2024-02-05T07:40:52.891846+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:47.564750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:47.564750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":298,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.874268+0000","last_change":"2024-02-05T07:38:51.521757+0000","last_active":"2024-02-05T07:40:49.874268+0000","last_peered":"2024-02-05T07:40:49.874268+0000","last_clean":"2024-02-05T07:40:49.874268+0000","last_became_active":"2024-02-05T07:36:32.733405+0000","last_became_peered":"2024-02-05T07:36:32.733405+0000","last_unstale":"2024-02-05T07:40:49.874268+0000","last_undegraded":"2024-02-05T07:40:49.874268+0000","last_fullsized":"2024-02-05T07:40:49.874268+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:51.521494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:51.521494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"616'39","reported_seq":954,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.896575+0000","last_change":"2024-02-05T07:37:50.422750+0000","last_active":"2024-02-05T07:40:51.896575+0000","last_peered":"2024-02-05T07:40:51.896575+0000","last_clean":"2024-02-05T07:40:51.896575+0000","last_became_active":"2024-02-05T07:36:32.741691+0000","last_became_peered":"2024-02-05T07:36:32.741691+0000","last_unstale":"2024-02-05T07:40:51.896575+0000","last_undegraded":"2024-02-05T07:40:51.896575+0000","last_fullsized":"2024-02-05T07:40:51.896575+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:37:50.422661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:50.422661+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":875,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.892338+0000","last_change":"2024-02-05T07:37:44.381379+0000","last_active":"2024-02-05T07:40:51.892338+0000","last_peered":"2024-02-05T07:40:51.892338+0000","last_clean":"2024-02-05T07:40:51.892338+0000","last_became_active":"2024-02-05T07:36:32.746731+0000","last_became_peered":"2024-02-05T07:36:32.746731+0000","last_unstale":"2024-02-05T07:40:51.892338+0000","last_undegraded":"2024-02-05T07:40:51.892338+0000","last_fullsized":"2024-02-05T07:40:51.892338+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:44.381300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:37:44.381300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":688,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891323+0000","last_change":"2024-02-05T07:38:09.473135+0000","last_active":"2024-02-05T07:40:52.891323+0000","last_peered":"2024-02-05T07:40:52.891323+0000","last_clean":"2024-02-05T07:40:52.891323+0000","last_became_active":"2024-02-05T07:38:09.472733+0000","last_became_peered":"2024-02-05T07:38:09.472733+0000","last_unstale":"2024-02-05T07:40:52.891323+0000","last_undegraded":"2024-02-05T07:40:52.891323+0000","last_fullsized":"2024-02-05T07:40:52.891323+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:45.170539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.170539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":876,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-02-05T07:40:51.901599+0000","last_change":"2024-02-05T07:38:58.711149+0000","last_active":"2024-02-05T07:40:51.901599+0000","last_peered":"2024-02-05T07:40:51.901599+0000","last_clean":"2024-02-05T07:40:51.901599+0000","last_became_active":"2024-02-05T07:30:40.493086+0000","last_became_peered":"2024-02-05T07:30:40.493086+0000","last_unstale":"2024-02-05T07:40:51.901599+0000","last_undegraded":"2024-02-05T07:40:51.901599+0000","last_fullsized":"2024-02-05T07:40:51.901599+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:58.711041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:38:58.711041+0000","last_clean_scrub_stamp":"2024-02-05T07:38:58.711041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"615'25","reported_seq":1063,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891483+0000","last_change":"2024-02-05T07:39:43.728715+0000","last_active":"2024-02-05T07:40:52.891483+0000","last_peered":"2024-02-05T07:40:52.891483+0000","last_clean":"2024-02-05T07:40:52.891483+0000","last_became_active":"2024-02-05T07:38:09.470878+0000","last_became_peered":"2024-02-05T07:38:09.470878+0000","last_unstale":"2024-02-05T07:40:52.891483+0000","last_undegraded":"2024-02-05T07:40:52.891483+0000","last_fullsized":"2024-02-05T07:40:52.891483+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'25","last_scrub_stamp":"2024-02-05T07:39:43.728551+0000","last_deep_scrub":"450'19","last_deep_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_clean_scrub_stamp":"2024-02-05T07:39:43.728551+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:01.829731+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":299,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.882468+0000","last_change":"2024-02-05T07:38:47.508209+0000","last_active":"2024-02-05T07:40:49.882468+0000","last_peered":"2024-02-05T07:40:49.882468+0000","last_clean":"2024-02-05T07:40:49.882468+0000","last_became_active":"2024-02-05T07:36:32.734502+0000","last_became_peered":"2024-02-05T07:36:32.734502+0000","last_unstale":"2024-02-05T07:40:49.882468+0000","last_undegraded":"2024-02-05T07:40:49.882468+0000","last_fullsized":"2024-02-05T07:40:49.882468+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:47.508088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:38:47.508088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":298,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-02-05T07:40:49.874958+0000","last_change":"2024-02-05T07:37:50.267485+0000","last_active":"2024-02-05T07:40:49.874958+0000","last_peered":"2024-02-05T07:40:49.874958+0000","last_clean":"2024-02-05T07:40:49.874958+0000","last_became_active":"2024-02-05T07:36:32.708917+0000","last_became_peered":"2024-02-05T07:36:32.708917+0000","last_unstale":"2024-02-05T07:40:49.874958+0000","last_undegraded":"2024-02-05T07:40:49.874958+0000","last_fullsized":"2024-02-05T07:40:49.874958+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:50.267364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:50.267364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":814,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-02-05T07:40:52.891550+0000","last_change":"2024-02-05T07:38:09.480452+0000","last_active":"2024-02-05T07:40:52.891550+0000","last_peered":"2024-02-05T07:40:52.891550+0000","last_clean":"2024-02-05T07:40:52.891550+0000","last_became_active":"2024-02-05T07:38:09.480027+0000","last_became_peered":"2024-02-05T07:38:09.480027+0000","last_unstale":"2024-02-05T07:40:52.891550+0000","last_undegraded":"2024-02-05T07:40:52.891550+0000","last_fullsized":"2024-02-05T07:40:52.891550+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:49.186617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:49.186617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1259077656,"num_objects":19213,"num_object_clones":0,"num_object_copies":57639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201345,"num_write_kb":7057793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2295,"num_bytes_recovered":150405120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3956355072,"data_stored":3956342856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48594,"ondisk_log_size":48594,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":614,"num_read_kb":609,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":22,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":66,"acting":66,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":4552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1526,"num_read_kb":1283,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":10072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":503808,"data_stored":13460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1052,"ondisk_log_size":1052,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":257698038029,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":301,"seq":1292785156247,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509996,"kb_used_data":755616,"kb_used_omap":5,"kb_used_meta":754362,"kb_avail":92235156,"statfs":{"total":95995035648,"available":94448799744,"internally_reserved":0,"allocated":773750784,"data_stored":772291002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":772467205},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.94899999999999995,"15min":1.2410000000000001},"min":{"1min":0.254,"5min":0.19600000000000001,"15min":0.192},"max":{"1min":5.1959999999999997,"5min":7.0990000000000002,"15min":14.218},"last":2.6070000000000002},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.98499999999999999,"15min":1.2989999999999999},"min":{"1min":0.23899999999999999,"5min":0.23200000000000001,"15min":0.22700000000000001},"max":{"1min":5.7460000000000004,"5min":5.7460000000000004,"15min":14.175000000000001},"last":2.6190000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:40:48 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":2.3300000000000001,"15min":2.0230000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.217},"max":{"1min":5.4290000000000003,"5min":56.325000000000003,"15min":56.325000000000003},"last":0.247},{"interface":"front","average":{"1min":1.2769999999999999,"5min":2.399,"15min":2.4089999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.7149999999999999,"5min":60.115000000000002,"15min":61.103000000000002},"last":0.311}]},{"osd":2,"last update":"Mon Feb 5 07:40:48 2024","interfaces":[{"interface":"back","average":{"1min":1.927,"5min":1.9379999999999999,"15min":1.9490000000000001},"min":{"1min":0.317,"5min":0.311,"15min":0.249},"max":{"1min":7.4009999999999998,"5min":46.927,"15min":46.927},"last":2.4609999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.653,"15min":2.0529999999999999},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.3140000000000001,"5min":46.506,"15min":46.506},"last":0.28899999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:40:48 2024","interfaces":[{"interface":"back","average":{"1min":3.625,"5min":2.3029999999999999,"15min":2.4780000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":48.600999999999999,"5min":56.271000000000001,"15min":56.271000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":4.1130000000000004,"5min":2.5419999999999998,"15min":2.645},"min":{"1min":0.31900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":49.828000000000003,"5min":56.235999999999997,"15min":56.235999999999997},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.42199999999999999,"15min":1.0569999999999999},"min":{"1min":0.29399999999999998,"5min":0.252,"15min":0.224},"max":{"1min":0.625,"5min":1.3149999999999999,"15min":151.02099999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.51800000000000002,"15min":1.0860000000000001},"min":{"1min":0.308,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55000000000000004,"5min":1.907,"15min":149.21799999999999},"last":0.33400000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.34999999999999998,"5min":0.30599999999999999,"15min":0.26800000000000002},"max":{"1min":0.58199999999999996,"5min":2.431,"15min":2.8919999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46700000000000003,"15min":0.50700000000000001},"min":{"1min":0.36699999999999999,"5min":0.28499999999999998,"15min":0.28100000000000003},"max":{"1min":0.56499999999999995,"5min":0.97099999999999997,"15min":1.526},"last":0.40600000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.48899999999999999,"15min":1.966},"min":{"1min":0.315,"5min":0.315,"15min":0.28000000000000003},"max":{"1min":0.59999999999999998,"5min":2.0550000000000002,"15min":84.158000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.48099999999999998,"15min":2.0960000000000001},"min":{"1min":0.34399999999999997,"5min":0.317,"15min":0.312},"max":{"1min":0.66100000000000003,"5min":1.097,"15min":85.018000000000001},"last":0.63}]}]},{"osd":1,"up_from":13,"seq":55834575147,"num_pgs":112,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2053468,"kb_used_data":940112,"kb_used_omap":1,"kb_used_meta":1113342,"kb_avail":91691684,"statfs":{"total":95995035648,"available":93892284416,"internally_reserved":0,"allocated":962674688,"data_stored":961209348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1140062661},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:40:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1560000000000001,"5min":2.7999999999999998,"15min":1.909},"min":{"1min":0.192,"5min":0.192,"15min":0.16800000000000001},"max":{"1min":15.327,"5min":29.449999999999999,"15min":29.449999999999999},"last":1.258},{"interface":"front","average":{"1min":2.919,"5min":3.028,"15min":2.1909999999999998},"min":{"1min":0.49299999999999999,"5min":0.28299999999999997,"15min":0.20899999999999999},"max":{"1min":7.9260000000000002,"5min":29.268999999999998,"15min":29.268999999999998},"last":0.49299999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:40:35 2024","interfaces":[{"interface":"back","average":{"1min":2.6920000000000002,"5min":3.4300000000000002,"15min":2.3130000000000002},"min":{"1min":0.373,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":6.7949999999999999,"5min":42.823999999999998,"15min":42.823999999999998},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.6819999999999999,"5min":2.843,"15min":2.0939999999999999},"min":{"1min":0.36199999999999999,"5min":0.26500000000000001,"15min":0.20699999999999999},"max":{"1min":3.984,"5min":42.426000000000002,"15min":42.426000000000002},"last":5.3109999999999999}]},{"osd":3,"last update":"Mon Feb 5 07:40:39 2024","interfaces":[{"interface":"back","average":{"1min":2.3420000000000001,"5min":2.177,"15min":2.2109999999999999},"min":{"1min":0.39600000000000002,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":7.9009999999999998,"5min":9.5519999999999996,"15min":46.777000000000001},"last":1.165},{"interface":"front","average":{"1min":1.756,"5min":1.9570000000000001,"15min":2.0550000000000002},"min":{"1min":0.38700000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":7.3899999999999997,"5min":9.5419999999999998,"15min":50.164999999999999},"last":3.7730000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.054,"15min":1.8959999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.0069999999999997,"5min":6.7329999999999997,"15min":89.409000000000006},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.4099999999999999,"15min":1.9019999999999999},"min":{"1min":0.42699999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":5.8979999999999997,"5min":14.629,"15min":89.915000000000006},"last":1.1890000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0509999999999999,"15min":1.7769999999999999},"min":{"1min":0.42099999999999999,"5min":0.32200000000000001,"15min":0.309},"max":{"1min":3.9580000000000002,"5min":6.7610000000000001,"15min":106.672},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.835,"15min":1.998},"min":{"1min":0.53300000000000003,"5min":0.34799999999999998,"15min":0.27600000000000002},"max":{"1min":7.8250000000000002,"5min":14.613,"15min":107.348},"last":0.53700000000000003}]},{"osd":6,"last update":"Mon Feb 5 07:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.3520000000000001,"15min":1.361},"min":{"1min":0.36699999999999999,"5min":0.314,"15min":0.29199999999999998},"max":{"1min":5.7990000000000004,"5min":12.685,"15min":12.685},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1319999999999999,"15min":1.49},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":3.9300000000000002,"5min":6.8179999999999996,"15min":8.923},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon Feb 5 07:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.9530000000000001,"5min":1.71,"15min":1.835},"min":{"1min":0.436,"5min":0.36599999999999999,"15min":0.318},"max":{"1min":7.8719999999999999,"5min":8.2439999999999998,"15min":46.351999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.405,"15min":1.387},"min":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.34000000000000002},"max":{"1min":5.8710000000000004,"5min":12.781000000000001,"15min":12.781000000000001},"last":1.224}]}]},{"osd":0,"up_from":8,"seq":34359738672,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1905124,"kb_used_data":813588,"kb_used_omap":1,"kb_used_meta":1091326,"kb_avail":91840028,"statfs":{"total":95995035648,"available":94044188672,"internally_reserved":0,"allocated":833114112,"data_stored":831651229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1117518277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:39:50 2024","interfaces":[{"interface":"back","average":{"1min":6.7060000000000004,"5min":3.1669999999999998,"15min":2.6280000000000001},"min":{"1min":0.39000000000000001,"5min":0.189,"15min":0.16500000000000001},"max":{"1min":65.799000000000007,"5min":65.799000000000007,"15min":65.799000000000007},"last":5.2469999999999999},{"interface":"front","average":{"1min":6.2720000000000002,"5min":2.4969999999999999,"15min":2.399},"min":{"1min":0.33900000000000002,"5min":0.192,"15min":0.192},"max":{"1min":65.093999999999994,"5min":65.093999999999994,"15min":65.093999999999994},"last":6.2380000000000004}]},{"osd":2,"last update":"Mon Feb 5 07:40:24 2024","interfaces":[{"interface":"back","average":{"1min":2.032,"5min":2.5619999999999998,"15min":2.044},"min":{"1min":0.24299999999999999,"5min":0.22900000000000001,"15min":0.191},"max":{"1min":5.5949999999999998,"5min":47.137999999999998,"15min":47.137999999999998},"last":6.2709999999999999},{"interface":"front","average":{"1min":1.379,"5min":2.2930000000000001,"15min":1.897},"min":{"1min":0.25900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.3440000000000003,"5min":47.170999999999999,"15min":47.170999999999999},"last":0.67700000000000005}]},{"osd":3,"last update":"Mon Feb 5 07:40:31 2024","interfaces":[{"interface":"back","average":{"1min":2.883,"5min":2.0579999999999998,"15min":1.823},"min":{"1min":0.433,"5min":0.25600000000000001,"15min":0.21299999999999999},"max":{"1min":22.129000000000001,"5min":26.43,"15min":29.629000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":2.6949999999999998,"5min":2.871,"15min":2.2930000000000001},"min":{"1min":0.435,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":22.155999999999999,"5min":25.337,"15min":29.463999999999999},"last":3.6299999999999999}]},{"osd":4,"last update":"Mon Feb 5 07:39:50 2024","interfaces":[{"interface":"back","average":{"1min":2.1720000000000002,"5min":2.113,"15min":1.9890000000000001},"min":{"1min":0.35499999999999998,"5min":0.308,"15min":0.26700000000000002},"max":{"1min":7.8460000000000001,"5min":12.718999999999999,"15min":50.545000000000002},"last":2.8439999999999999},{"interface":"front","average":{"1min":1.573,"5min":1.349,"15min":1.802},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.27100000000000002},"max":{"1min":5.4770000000000003,"5min":5.4770000000000003,"15min":50.310000000000002},"last":6.1230000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.157,"15min":1.419},"min":{"1min":0.374,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":5.3840000000000003,"5min":5.3840000000000003,"15min":95.772999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.268,"15min":1.546},"min":{"1min":0.372,"5min":0.32900000000000001,"15min":0.28199999999999997},"max":{"1min":5.5810000000000004,"5min":5.5810000000000004,"15min":96.028000000000006},"last":6.1580000000000004}]},{"osd":6,"last update":"Mon Feb 5 07:40:49 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":3.0619999999999998,"15min":1.8979999999999999},"min":{"1min":0.312,"5min":0.27800000000000002,"15min":0.26300000000000001},"max":{"1min":141.59899999999999,"5min":141.59899999999999,"15min":141.59899999999999},"last":2.806},{"interface":"front","average":{"1min":7.5549999999999997,"5min":2.5289999999999999,"15min":1.8120000000000001},"min":{"1min":0.39900000000000002,"5min":0.376,"15min":0.35099999999999998},"max":{"1min":141.39599999999999,"5min":141.39599999999999,"15min":141.39599999999999},"last":6.032}]},{"osd":7,"last update":"Mon Feb 5 07:40:10 2024","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":2.0939999999999999,"15min":1.798},"min":{"1min":0.38500000000000001,"5min":0.33500000000000002,"15min":0.28599999999999998},"max":{"1min":7.8200000000000003,"5min":12.766999999999999,"15min":43.012},"last":2.8730000000000002},{"interface":"front","average":{"1min":1.742,"5min":1.4199999999999999,"15min":1.6459999999999999},"min":{"1min":0.48799999999999999,"5min":0.39100000000000001,"15min":0.309},"max":{"1min":5.641,"5min":5.641,"15min":43.512},"last":6.0869999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411622,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1564168,"kb_used_data":590332,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":92180984,"statfs":{"total":95995035648,"available":94393327616,"internally_reserved":0,"allocated":604499968,"data_stored":603055206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:40:29 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":2.5169999999999999,"15min":2.1499999999999999},"min":{"1min":0.25,"5min":0.21099999999999999,"15min":0.19600000000000001},"max":{"1min":5.1340000000000003,"5min":80.427000000000007,"15min":92.638000000000005},"last":4.1689999999999996},{"interface":"front","average":{"1min":1.8979999999999999,"5min":2.9199999999999999,"15min":2.427},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.0129999999999999,"5min":80.939999999999998,"15min":93.481999999999999},"last":4.2140000000000004}]},{"osd":1,"last update":"Mon Feb 5 07:40:29 2024","interfaces":[{"interface":"back","average":{"1min":2.302,"5min":3.9660000000000002,"15min":3.2890000000000001},"min":{"1min":0.63400000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":6.1399999999999997,"5min":110.917,"15min":110.917},"last":2.9470000000000001},{"interface":"front","average":{"1min":2.6240000000000001,"5min":1.7849999999999999,"15min":2.6949999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.20300000000000001},"max":{"1min":5.7309999999999999,"5min":7.9509999999999996,"15min":93.004000000000005},"last":1.841}]},{"osd":3,"last update":"Mon Feb 5 07:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.722,"5min":3.609,"15min":2.9300000000000002},"min":{"1min":0.29199999999999998,"5min":0.214,"15min":0.214},"max":{"1min":5.3339999999999996,"5min":110.773,"15min":110.773},"last":2.1720000000000002},{"interface":"front","average":{"1min":2.532,"5min":2.1509999999999998,"15min":2.2850000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":5.2670000000000003,"5min":47.368000000000002,"15min":63.521000000000001},"last":1.002}]},{"osd":4,"last update":"Mon Feb 5 07:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":2.0979999999999999,"15min":2.3690000000000002},"min":{"1min":0.28699999999999998,"5min":0.27700000000000002,"15min":0.24299999999999999},"max":{"1min":2.629,"5min":46.521999999999998,"15min":168.11699999999999},"last":1.9179999999999999},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.5640000000000001,"15min":2.153},"min":{"1min":0.39800000000000002,"5min":0.36599999999999999,"15min":0.27200000000000002},"max":{"1min":2.8359999999999999,"5min":40.329000000000001,"15min":168.589},"last":0.371}]},{"osd":5,"last update":"Mon Feb 5 07:40:29 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.0309999999999999,"15min":1.212},"min":{"1min":0.41899999999999998,"5min":0.35699999999999998,"15min":0.315},"max":{"1min":2.8679999999999999,"5min":6.0129999999999999,"15min":39.923000000000002},"last":0.437},{"interface":"front","average":{"1min":1.456,"5min":2.464,"15min":1.694},"min":{"1min":0.441,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.2329999999999997,"5min":47.761000000000003,"15min":47.761000000000003},"last":2.7210000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:39:51 2024","interfaces":[{"interface":"back","average":{"1min":1.8560000000000001,"5min":3.0619999999999998,"15min":1.782},"min":{"1min":0.40100000000000002,"5min":0.309,"15min":0.28799999999999998},"max":{"1min":5.2750000000000004,"5min":47.726999999999997,"15min":47.726999999999997},"last":2.6840000000000002},{"interface":"front","average":{"1min":1.244,"5min":1.643,"15min":1.3620000000000001},"min":{"1min":0.39600000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":4.5149999999999997,"5min":40.872999999999998,"15min":40.872999999999998},"last":0.40699999999999997}]},{"osd":7,"last update":"Mon Feb 5 07:40:29 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":1.0700000000000001,"15min":1.403},"min":{"1min":0.374,"5min":0.35699999999999998,"15min":0.30499999999999999},"max":{"1min":2.673,"5min":6.4669999999999996,"15min":37.573999999999998},"last":1.974},{"interface":"front","average":{"1min":1.536,"5min":3.5310000000000001,"15min":2.137},"min":{"1min":0.501,"5min":0.40100000000000002,"15min":0.33700000000000002},"max":{"1min":5.3049999999999997,"5min":110.86799999999999,"15min":110.86799999999999},"last":2.7480000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149985,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1780808,"kb_used_data":690364,"kb_used_omap":1,"kb_used_meta":1090430,"kb_avail":91964344,"statfs":{"total":95995035648,"available":94171488256,"internally_reserved":0,"allocated":706932736,"data_stored":705437824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1116600775},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:39:53 2024","interfaces":[{"interface":"back","average":{"1min":1.919,"5min":2.3929999999999998,"15min":1.7170000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.782,"5min":48.688000000000002,"15min":48.688000000000002},"last":2.956},{"interface":"front","average":{"1min":2.5619999999999998,"5min":2.968,"15min":2.3199999999999998},"min":{"1min":0.36599999999999999,"5min":0.216,"15min":0.185},"max":{"1min":8.4930000000000003,"5min":48.573,"15min":99.989999999999995},"last":0.82299999999999995}]},{"osd":1,"last update":"Mon Feb 5 07:39:53 2024","interfaces":[{"interface":"back","average":{"1min":2.8250000000000002,"5min":3.069,"15min":2.702},"min":{"1min":0.59099999999999997,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":7.274,"5min":61.941000000000003,"15min":102.20399999999999},"last":2.927},{"interface":"front","average":{"1min":2.1190000000000002,"5min":2.73,"15min":2.6030000000000002},"min":{"1min":0.29999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.5469999999999997,"5min":61.902000000000001,"15min":102.286},"last":3.157}]},{"osd":2,"last update":"Mon Feb 5 07:39:53 2024","interfaces":[{"interface":"back","average":{"1min":2.206,"5min":3.2519999999999998,"15min":2.4729999999999999},"min":{"1min":0.45700000000000002,"5min":0.23999999999999999,"15min":0.22700000000000001},"max":{"1min":6.8200000000000003,"5min":145.94999999999999,"15min":145.94999999999999},"last":3.1259999999999999},{"interface":"front","average":{"1min":1.8799999999999999,"5min":3.4340000000000002,"15min":2.4969999999999999},"min":{"1min":0.32800000000000001,"5min":0.218,"15min":0.20200000000000001},"max":{"1min":7.0110000000000001,"5min":146.006,"15min":146.006},"last":0.79200000000000004}]},{"osd":4,"last update":"Mon Feb 5 07:40:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.3049999999999999,"15min":1.79},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27000000000000002},"max":{"1min":3.2389999999999999,"5min":9.2639999999999993,"15min":100.024},"last":0.84699999999999998},{"interface":"front","average":{"1min":2.0139999999999998,"5min":1.7470000000000001,"15min":2.0830000000000002},"min":{"1min":0.36399999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.3200000000000003,"5min":16.253,"15min":99.859999999999999},"last":2.0209999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:40:27 2024","interfaces":[{"interface":"back","average":{"1min":2.456,"5min":1.49,"15min":1.5469999999999999},"min":{"1min":0.313,"5min":0.308,"15min":0.308},"max":{"1min":11.507999999999999,"5min":11.507999999999999,"15min":99.911000000000001},"last":3.0939999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.363,"15min":1.591},"min":{"1min":0.34100000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.2789999999999999,"5min":8.4350000000000005,"15min":100.053},"last":0.91100000000000003}]},{"osd":6,"last update":"Mon Feb 5 07:39:53 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":1.7789999999999999,"15min":1.6519999999999999},"min":{"1min":0.33200000000000002,"5min":0.32700000000000001,"15min":0.29299999999999998},"max":{"1min":4.5830000000000002,"5min":32.505000000000003,"15min":99.807000000000002},"last":3.0169999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.7969999999999999,"15min":1.7370000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.1020000000000003,"5min":31.084,"15min":101.489},"last":0.877}]},{"osd":7,"last update":"Mon Feb 5 07:40:11 2024","interfaces":[{"interface":"back","average":{"1min":1.99,"5min":1.593,"15min":2.387},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.29499999999999998},"max":{"1min":6.3609999999999998,"5min":8.4329999999999998,"15min":99.893000000000001},"last":1.9750000000000001},{"interface":"front","average":{"1min":1.794,"5min":1.294,"15min":2.2679999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.6509999999999998,"5min":6.726,"15min":99.879000000000005},"last":3.0590000000000002}]}]},{"osd":4,"up_from":0,"seq":137438953756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":176093659416,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":811991040,"data_stored":811991040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":937758720,"data_stored":937754648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":594022400,"data_stored":594018328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":693436416,"data_stored":693436416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":150011904,"data_stored":150011904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":769134592,"data_stored":769130520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:41:00.516 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-05T07:41:00.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set unique_pool_4 pg_num 26 2024-02-05T07:41:00.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:00.628 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:00.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:00.730 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:00.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:00.786 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:00.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:00.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:00.936 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:00.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:00.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:01.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:00 smithi090 ceph-mon[33213]: pgmap v1245: 168 pgs: 1 peering, 167 active+clean; 834 MiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 922 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T07:41:01.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:01.062 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:01.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:01.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:01.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:00 smithi007 ceph-mon[29098]: pgmap v1245: 168 pgs: 1 peering, 167 active+clean; 834 MiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 922 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T07:41:01.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:00 smithi007 ceph-mon[37158]: pgmap v1245: 168 pgs: 1 peering, 167 active+clean; 834 MiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 922 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T07:41:01.209 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:01.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:01.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:01.335 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:01.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:01.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:01.462 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:01.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:01.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:41:01 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:41:01.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:01.589 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:01.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:01.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:01.690 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:01.727 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:41:01.727 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-02-05T07:41:01.727 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-02-05T07:41:01.728 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-02-05T07:41:01.728 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:41:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:01.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:01.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:01.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:41:01.861 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:01.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:01.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:41:01.920 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:01 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:41:01.920 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:01 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:41:01.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:01.988 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:01.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:02.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:01 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:41:02.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:02.132 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:02.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:02.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:02.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:02.286 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:02.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:02.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:02.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:02.446 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:02.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:02.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:02.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:02.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:02.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:41:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:02.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:02.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:41:02.815 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:02.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:02.860 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:02.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:02.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:41:02.985 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:02.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:02.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:41:02.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T07:41:03.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:02 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:03.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:02 smithi090 ceph-mon[33213]: pgmap v1246: 168 pgs: 168 active+clean; 675 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T07:41:03.025 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:03.047 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:02 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:03.047 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:02 smithi007 ceph-mon[29098]: pgmap v1246: 168 pgs: 168 active+clean; 675 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T07:41:03.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:02 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:03.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:02 smithi007 ceph-mon[37158]: pgmap v1246: 168 pgs: 168 active+clean; 675 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T07:41:03.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:03.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:03.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:03.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:03.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:03.294 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:03.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:03.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:03.419 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:03.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:03.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:03.545 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:03.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:03.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:03.634 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:41:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:41:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:41:03.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:03.689 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:03.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:03.775 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:41:03.791 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:03.815 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:03.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:03.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:03.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:03 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4133939944' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:41:03.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:03 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:41:03.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:03 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:41:03.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:03 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4133939944' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:41:03.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:03 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:41:03.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:03 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:41:03.956 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:03.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:03.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:04.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:03 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4133939944' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:41:04.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:03 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:41:04.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:03 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:41:04.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:41:04.082 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:04.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:41:04.184 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:04.209 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:04.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:04.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:04.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:04.351 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:04.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:04.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:04.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:04.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:04.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:04.618 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:04.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:04.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:04.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:41:04.743 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:04.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:41:04.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:04.869 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:04.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:04.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:04.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:04 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:41:04.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:04 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:41:04.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:04 smithi007 ceph-mon[29098]: pgmap v1247: 168 pgs: 168 active+clean; 568 MiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T07:41:04.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:04 smithi007 ceph-mon[29098]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:41:04.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:04 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4133939944' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:41:04.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:04 smithi007 ceph-mon[29098]: osdmap e683: 8 total, 8 up, 5 in 2024-02-05T07:41:04.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:04 smithi007 ceph-mon[29098]: osdmap e684: 8 total, 8 up, 5 in 2024-02-05T07:41:04.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:04 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:41:04.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:04 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:41:04.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:04 smithi007 ceph-mon[37158]: pgmap v1247: 168 pgs: 168 active+clean; 568 MiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T07:41:04.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:04 smithi007 ceph-mon[37158]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:41:04.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:04 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4133939944' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:41:04.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:04 smithi007 ceph-mon[37158]: osdmap e683: 8 total, 8 up, 5 in 2024-02-05T07:41:04.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:04 smithi007 ceph-mon[37158]: osdmap e684: 8 total, 8 up, 5 in 2024-02-05T07:41:04.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:05.021 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:05.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:05.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:05.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:04 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:41:05.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:04 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:41:05.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:04 smithi090 ceph-mon[33213]: pgmap v1247: 168 pgs: 168 active+clean; 568 MiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T07:41:05.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:04 smithi090 ceph-mon[33213]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:41:05.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:04 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4133939944' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:41:05.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:04 smithi090 ceph-mon[33213]: osdmap e683: 8 total, 8 up, 5 in 2024-02-05T07:41:05.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:04 smithi090 ceph-mon[33213]: osdmap e684: 8 total, 8 up, 5 in 2024-02-05T07:41:05.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:05.163 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:05.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:05.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:05.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:05.304 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:05.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:05.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:05.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:05.444 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:05.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:05.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:05.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:05.585 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:05.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:05.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:05.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:05.728 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:05.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:05.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:05.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:05.868 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:05.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:05.969 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:05.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:05.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:06.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:05 smithi090 ceph-mon[33213]: 3.4 scrub starts 2024-02-05T07:41:06.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:05 smithi090 ceph-mon[33213]: 3.4 scrub ok 2024-02-05T07:41:06.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:06.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:06.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:05 smithi007 ceph-mon[29098]: 3.4 scrub starts 2024-02-05T07:41:06.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:05 smithi007 ceph-mon[29098]: 3.4 scrub ok 2024-02-05T07:41:06.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:05 smithi007 ceph-mon[37158]: 3.4 scrub starts 2024-02-05T07:41:06.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:05 smithi007 ceph-mon[37158]: 3.4 scrub ok 2024-02-05T07:41:06.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:06.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:06.274 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:06.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:06.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:06.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:06.416 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:06.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:06.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:06.542 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:06.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:06.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:06.685 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:06.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:06.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:06.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:06.811 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:06.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:06.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:06.936 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:06.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:06.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:07.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:06 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:41:07.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:06 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:41:07.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:06 smithi090 ceph-mon[33213]: pgmap v1250: 168 pgs: 168 active+clean; 475 MiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 901 op/s 2024-02-05T07:41:07.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:06 smithi090 ceph-mon[33213]: osdmap e685: 8 total, 8 up, 5 in 2024-02-05T07:41:07.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:06 smithi090 ceph-mon[33213]: 6.a scrub starts 2024-02-05T07:41:07.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:06 smithi090 ceph-mon[33213]: 6.a scrub ok 2024-02-05T07:41:07.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:07.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:07.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:07.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:06 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:41:07.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:06 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:41:07.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:06 smithi007 ceph-mon[29098]: pgmap v1250: 168 pgs: 168 active+clean; 475 MiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 901 op/s 2024-02-05T07:41:07.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:06 smithi007 ceph-mon[29098]: osdmap e685: 8 total, 8 up, 5 in 2024-02-05T07:41:07.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:06 smithi007 ceph-mon[29098]: 6.a scrub starts 2024-02-05T07:41:07.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:06 smithi007 ceph-mon[29098]: 6.a scrub ok 2024-02-05T07:41:07.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:06 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:41:07.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:06 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:41:07.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:06 smithi007 ceph-mon[37158]: pgmap v1250: 168 pgs: 168 active+clean; 475 MiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 901 op/s 2024-02-05T07:41:07.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:06 smithi007 ceph-mon[37158]: osdmap e685: 8 total, 8 up, 5 in 2024-02-05T07:41:07.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:06 smithi007 ceph-mon[37158]: 6.a scrub starts 2024-02-05T07:41:07.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:06 smithi007 ceph-mon[37158]: 6.a scrub ok 2024-02-05T07:41:07.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:07.207 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:07.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:07.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:07.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:07.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:07.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:07.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:07.460 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:07.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:07.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:07.585 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:07.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:07.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:07.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:07.728 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:07.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:07.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:07.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:07.865 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:07.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:07.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:07.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:08.007 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:08.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:08.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:08.149 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:08.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:08.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:08.294 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:08.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:08.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:08.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:41:08.420 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:08.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:41:08.521 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:08.553 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:08.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:08.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:08.555 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Removed 110278 objects 2024-02-05T07:41:08.555 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Clean up completed and total clean up time :93.9863 2024-02-05T07:41:08.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:08.677 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:08.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:08.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:08.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:08.804 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:08.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:08.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:08.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:08.930 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:08.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:09.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:08 smithi090 ceph-mon[33213]: pgmap v1252: 168 pgs: 168 active+clean; 198 MiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T07:41:09.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:41:09.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:09.055 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:09.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:41:09.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:08 smithi007 ceph-mon[29098]: pgmap v1252: 168 pgs: 168 active+clean; 198 MiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T07:41:09.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:08 smithi007 ceph-mon[37158]: pgmap v1252: 168 pgs: 168 active+clean; 198 MiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T07:41:09.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:09.180 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:09.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:09.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:09.305 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:09.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:09.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:41:09.431 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:09.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:09.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:41:09.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:09.556 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:09.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:09.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:09.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:09.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:09.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:09.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:09.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:09.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:09.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:09.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:09.931 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:09.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:09.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:10.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:10.058 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:10.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:10.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:10.186 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:10.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:10.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:10.288 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:10.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:10.315 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:10.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:10.416 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:41:10.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:10.443 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:10.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:41:10.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:10.572 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:10.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:10.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:10.674 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:10.702 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:10.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:10.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:10.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:10.830 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:10.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:10.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:41:10.961 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:10.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:10.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:41:10.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:10 smithi007 ceph-mon[29098]: 5.1f scrub starts 2024-02-05T07:41:10.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:10 smithi007 ceph-mon[29098]: 5.1f scrub ok 2024-02-05T07:41:10.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:10 smithi007 ceph-mon[29098]: pgmap v1253: 168 pgs: 168 active+clean; 113 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s 2024-02-05T07:41:10.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:10 smithi007 ceph-mon[29098]: 3.1b scrub starts 2024-02-05T07:41:10.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:10 smithi007 ceph-mon[29098]: 3.1b scrub ok 2024-02-05T07:41:11.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:10 smithi007 ceph-mon[29098]: osdmap e686: 8 total, 8 up, 5 in 2024-02-05T07:41:11.000 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:10 smithi007 ceph-mon[37158]: 5.1f scrub starts 2024-02-05T07:41:11.000 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:10 smithi007 ceph-mon[37158]: 5.1f scrub ok 2024-02-05T07:41:11.000 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:10 smithi007 ceph-mon[37158]: pgmap v1253: 168 pgs: 168 active+clean; 113 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s 2024-02-05T07:41:11.000 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:10 smithi007 ceph-mon[37158]: 3.1b scrub starts 2024-02-05T07:41:11.000 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:10 smithi007 ceph-mon[37158]: 3.1b scrub ok 2024-02-05T07:41:11.000 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:10 smithi007 ceph-mon[37158]: osdmap e686: 8 total, 8 up, 5 in 2024-02-05T07:41:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:10 smithi090 ceph-mon[33213]: 5.1f scrub starts 2024-02-05T07:41:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:10 smithi090 ceph-mon[33213]: 5.1f scrub ok 2024-02-05T07:41:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:10 smithi090 ceph-mon[33213]: pgmap v1253: 168 pgs: 168 active+clean; 113 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s 2024-02-05T07:41:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:10 smithi090 ceph-mon[33213]: 3.1b scrub starts 2024-02-05T07:41:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:10 smithi090 ceph-mon[33213]: 3.1b scrub ok 2024-02-05T07:41:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:10 smithi090 ceph-mon[33213]: osdmap e686: 8 total, 8 up, 5 in 2024-02-05T07:41:11.024 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:11.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:11.090 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:11.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:11.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:11.218 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:11.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:11.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:11.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:11.346 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:11.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:11.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:11.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:11.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:11.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:11.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:41:11 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:41:11.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:11.603 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:11.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:11.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:11.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:11.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:11.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:11.858 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:11.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:11.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:11.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:11.985 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:11.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:12.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:11 smithi090 ceph-mon[33213]: 6.12 scrub starts 2024-02-05T07:41:12.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:11 smithi090 ceph-mon[33213]: 6.12 scrub ok 2024-02-05T07:41:12.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:12.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:11 smithi007 ceph-mon[29098]: 6.12 scrub starts 2024-02-05T07:41:12.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:11 smithi007 ceph-mon[29098]: 6.12 scrub ok 2024-02-05T07:41:12.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:11 smithi007 ceph-mon[37158]: 6.12 scrub starts 2024-02-05T07:41:12.098 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:11 smithi007 ceph-mon[37158]: 6.12 scrub ok 2024-02-05T07:41:12.112 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:12.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:12.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:12.239 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:12.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:12.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:12.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:12.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:12.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:12.494 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:12.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:12.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:12.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:12.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:12.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:12.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:12.751 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:12.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:12.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:41:12.877 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:12.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:12.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:41:12.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[29098]: 1.7 scrub starts 2024-02-05T07:41:12.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[29098]: 1.7 scrub ok 2024-02-05T07:41:12.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:12.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[29098]: pgmap v1255: 168 pgs: 168 active+clean; 25 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-02-05T07:41:12.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[29098]: 3.9 scrub starts 2024-02-05T07:41:12.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[29098]: 3.9 scrub ok 2024-02-05T07:41:12.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[29098]: 4.3 scrub starts 2024-02-05T07:41:12.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[29098]: 4.3 scrub ok 2024-02-05T07:41:12.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[29098]: 4.1a scrub starts 2024-02-05T07:41:12.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[29098]: 4.1a scrub ok 2024-02-05T07:41:12.894 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[37158]: 1.7 scrub starts 2024-02-05T07:41:12.894 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[37158]: 1.7 scrub ok 2024-02-05T07:41:12.894 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:12.894 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[37158]: pgmap v1255: 168 pgs: 168 active+clean; 25 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-02-05T07:41:12.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[37158]: 3.9 scrub starts 2024-02-05T07:41:12.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[37158]: 3.9 scrub ok 2024-02-05T07:41:12.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[37158]: 4.3 scrub starts 2024-02-05T07:41:12.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[37158]: 4.3 scrub ok 2024-02-05T07:41:12.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[37158]: 4.1a scrub starts 2024-02-05T07:41:12.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:12 smithi007 ceph-mon[37158]: 4.1a scrub ok 2024-02-05T07:41:12.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:41:12.913 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:41:12.933 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:41:12.955 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:41:12.976 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:41:12.979 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:12.997 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:41:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:13.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:13.018 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:41:13.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:12 smithi090 ceph-mon[33213]: 1.7 scrub starts 2024-02-05T07:41:13.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:12 smithi090 ceph-mon[33213]: 1.7 scrub ok 2024-02-05T07:41:13.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:12 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:12 smithi090 ceph-mon[33213]: pgmap v1255: 168 pgs: 168 active+clean; 25 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-02-05T07:41:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:12 smithi090 ceph-mon[33213]: 3.9 scrub starts 2024-02-05T07:41:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:12 smithi090 ceph-mon[33213]: 3.9 scrub ok 2024-02-05T07:41:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:12 smithi090 ceph-mon[33213]: 4.3 scrub starts 2024-02-05T07:41:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:12 smithi090 ceph-mon[33213]: 4.3 scrub ok 2024-02-05T07:41:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:12 smithi090 ceph-mon[33213]: 4.1a scrub starts 2024-02-05T07:41:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:12 smithi090 ceph-mon[33213]: 4.1a scrub ok 2024-02-05T07:41:13.039 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:41:13.060 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:41:13.062 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:41:13.062 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:41:13.062 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:41:13.062 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:41:13.062 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:41:13.062 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:41:13.062 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:41:13.062 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:41:13.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:13.132 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:13.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:13.233 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:13.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:13.261 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:13.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:13.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:13.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:13.387 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:13.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:13.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:13.512 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:13.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:13.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:13.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:13.637 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:13.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:13.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:13.672 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:41:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:41:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:41:13.721 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 11 pg_num to 26 2024-02-05T07:41:13.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:13.767 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:13.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:13.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:13.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:13.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:13.970 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:13.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:14.022 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:14.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:13 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2652629960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:41:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:13 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1932239670' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:41:14.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:13 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:41:14.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:13 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2652629960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:41:14.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:13 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1932239670' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:41:14.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:13 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:41:14.094 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:13 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2652629960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:41:14.094 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:13 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1932239670' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:41:14.094 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:13 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T07:41:14.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:14.153 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:14.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:14.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:14.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:14.283 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:14.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:14.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:14.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:14.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:14.426 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-02-05T07:41:14.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-02-05T07:41:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:14.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:14.543 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:14.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:14.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:14.671 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:14.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:14.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:14.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:14.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:14.804 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:14.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:14.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:14.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:14 smithi007 ceph-mon[37158]: 3.a scrub starts 2024-02-05T07:41:14.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:14 smithi007 ceph-mon[37158]: 3.a scrub ok 2024-02-05T07:41:14.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:14 smithi007 ceph-mon[37158]: pgmap v1256: 168 pgs: 168 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 949 op/s 2024-02-05T07:41:14.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:14 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-02-05T07:41:14.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:14 smithi007 ceph-mon[37158]: osdmap e687: 8 total, 8 up, 5 in 2024-02-05T07:41:14.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:14 smithi007 ceph-mon[37158]: osdmap e688: 8 total, 8 up, 5 in 2024-02-05T07:41:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:14 smithi007 ceph-mon[29098]: 3.a scrub starts 2024-02-05T07:41:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:14 smithi007 ceph-mon[29098]: 3.a scrub ok 2024-02-05T07:41:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:14 smithi007 ceph-mon[29098]: pgmap v1256: 168 pgs: 168 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 949 op/s 2024-02-05T07:41:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:14 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-02-05T07:41:14.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:14 smithi007 ceph-mon[29098]: osdmap e687: 8 total, 8 up, 5 in 2024-02-05T07:41:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:14 smithi007 ceph-mon[29098]: osdmap e688: 8 total, 8 up, 5 in 2024-02-05T07:41:14.932 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:14.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:14.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:15.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:14 smithi090 ceph-mon[33213]: 3.a scrub starts 2024-02-05T07:41:15.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:14 smithi090 ceph-mon[33213]: 3.a scrub ok 2024-02-05T07:41:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:14 smithi090 ceph-mon[33213]: pgmap v1256: 168 pgs: 168 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 949 op/s 2024-02-05T07:41:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:14 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-02-05T07:41:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:14 smithi090 ceph-mon[33213]: osdmap e687: 8 total, 8 up, 5 in 2024-02-05T07:41:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:14 smithi090 ceph-mon[33213]: osdmap e688: 8 total, 8 up, 5 in 2024-02-05T07:41:15.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:15.061 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:15.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:15.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:15.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:15.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:15.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:15.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:15.312 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:15.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:15.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:15.442 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:15.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:15.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:15.544 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:15.572 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:15.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:15.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:15.674 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:15.697 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:15.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:15.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:15.705 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T07:41:15.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:15.867 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:15.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:15.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:15.969 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:15.976 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:15 smithi007 ceph-mon[37158]: 1.6 scrub starts 2024-02-05T07:41:15.976 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:15 smithi007 ceph-mon[37158]: 1.6 scrub ok 2024-02-05T07:41:15.976 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:15 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:41:15.976 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:15 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:41:15.976 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:15 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:41:15.977 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:15 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:41:15.977 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:15 smithi007 ceph-mon[29098]: 1.6 scrub starts 2024-02-05T07:41:15.977 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:15 smithi007 ceph-mon[29098]: 1.6 scrub ok 2024-02-05T07:41:15.977 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:15 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:41:15.977 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:15 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:41:15.977 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:15 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:41:15.977 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:15 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:41:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:15 smithi090 ceph-mon[33213]: 1.6 scrub starts 2024-02-05T07:41:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:15 smithi090 ceph-mon[33213]: 1.6 scrub ok 2024-02-05T07:41:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:15 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:41:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:15 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:41:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:15 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:41:16.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:15 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T07:41:16.074 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:16.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:16.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:16.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:16.199 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:16.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:16.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:16.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:16.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:16.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:16.475 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:16.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:16.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:16.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:16.601 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:16.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:16.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:16.729 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:16.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:16.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:16.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:41:16.856 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:16.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:41:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[29098]: 5.e scrub starts 2024-02-05T07:41:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[29098]: 5.e scrub ok 2024-02-05T07:41:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[29098]: pgmap v1259: 168 pgs: 168 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 301 op/s 2024-02-05T07:41:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[29098]: 4.12 scrub starts 2024-02-05T07:41:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[29098]: 4.12 scrub ok 2024-02-05T07:41:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:41:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[29098]: osdmap e689: 8 total, 8 up, 5 in 2024-02-05T07:41:16.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:41:16.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[37158]: 5.e scrub starts 2024-02-05T07:41:16.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[37158]: 5.e scrub ok 2024-02-05T07:41:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[37158]: pgmap v1259: 168 pgs: 168 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 301 op/s 2024-02-05T07:41:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[37158]: 4.12 scrub starts 2024-02-05T07:41:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[37158]: 4.12 scrub ok 2024-02-05T07:41:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:41:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[37158]: osdmap e689: 8 total, 8 up, 5 in 2024-02-05T07:41:16.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:16 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:41:16.958 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:16.982 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:16.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:17.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:16 smithi090 ceph-mon[33213]: 5.e scrub starts 2024-02-05T07:41:17.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:16 smithi090 ceph-mon[33213]: 5.e scrub ok 2024-02-05T07:41:17.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:16 smithi090 ceph-mon[33213]: pgmap v1259: 168 pgs: 168 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 301 op/s 2024-02-05T07:41:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:16 smithi090 ceph-mon[33213]: 4.12 scrub starts 2024-02-05T07:41:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:16 smithi090 ceph-mon[33213]: 4.12 scrub ok 2024-02-05T07:41:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:16 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T07:41:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:16 smithi090 ceph-mon[33213]: osdmap e689: 8 total, 8 up, 5 in 2024-02-05T07:41:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:16 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:41:17.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:17.107 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:17.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:17.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:17.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:17.232 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:17.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:17.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:17.358 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:17.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:17.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:17.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:17.484 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:17.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:17.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:17.609 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:17.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:17.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:17.611 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:17.648 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:17.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:41:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:17.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:17.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:17.842 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:17.866 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:17.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:17.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:17.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[29098]: 1.10 scrub starts 2024-02-05T07:41:17.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[29098]: 1.10 scrub ok 2024-02-05T07:41:17.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[29098]: osdmap e690: 8 total, 8 up, 5 in 2024-02-05T07:41:17.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[29098]: 11.8 scrub starts 2024-02-05T07:41:17.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[29098]: 11.8 scrub ok 2024-02-05T07:41:17.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2863905464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:41:17.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:41:17.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:41:17.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[37158]: 1.10 scrub starts 2024-02-05T07:41:17.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[37158]: 1.10 scrub ok 2024-02-05T07:41:17.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[37158]: osdmap e690: 8 total, 8 up, 5 in 2024-02-05T07:41:17.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[37158]: 11.8 scrub starts 2024-02-05T07:41:17.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[37158]: 11.8 scrub ok 2024-02-05T07:41:17.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2863905464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:41:17.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:41:17.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:17 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:41:17.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:17.992 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:17.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:17.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:18.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:17 smithi090 ceph-mon[33213]: 1.10 scrub starts 2024-02-05T07:41:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:17 smithi090 ceph-mon[33213]: 1.10 scrub ok 2024-02-05T07:41:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:17 smithi090 ceph-mon[33213]: osdmap e690: 8 total, 8 up, 5 in 2024-02-05T07:41:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:17 smithi090 ceph-mon[33213]: 11.8 scrub starts 2024-02-05T07:41:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:17 smithi090 ceph-mon[33213]: 11.8 scrub ok 2024-02-05T07:41:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:17 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2863905464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:41:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:17 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:41:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:17 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T07:41:18.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:18.118 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:18.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:18.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:18.244 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:18.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:18.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:18.370 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:18.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:18.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:18.495 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:18.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:18.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:18.621 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:18.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:18.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:18.754 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:18.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:18.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:18.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:18.880 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:18.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:18.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[29098]: 11.3 scrub starts 2024-02-05T07:41:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[29098]: 11.3 scrub ok 2024-02-05T07:41:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[29098]: 11.6 scrub starts 2024-02-05T07:41:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[29098]: 11.6 scrub ok 2024-02-05T07:41:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[29098]: pgmap v1262: 178 pgs: 2 peering, 7 unknown, 169 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:41:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[29098]: 11.7 scrub starts 2024-02-05T07:41:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[29098]: 11.7 scrub ok 2024-02-05T07:41:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:41:18.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:41:18.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2863905464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:41:18.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T07:41:18.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[29098]: osdmap e691: 8 total, 8 up, 5 in 2024-02-05T07:41:18.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[29098]: 11.0 scrub starts 2024-02-05T07:41:18.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[29098]: 11.0 scrub ok 2024-02-05T07:41:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[37158]: 11.3 scrub starts 2024-02-05T07:41:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[37158]: 11.3 scrub ok 2024-02-05T07:41:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[37158]: 11.6 scrub starts 2024-02-05T07:41:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[37158]: 11.6 scrub ok 2024-02-05T07:41:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[37158]: pgmap v1262: 178 pgs: 2 peering, 7 unknown, 169 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:41:18.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[37158]: 11.7 scrub starts 2024-02-05T07:41:18.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[37158]: 11.7 scrub ok 2024-02-05T07:41:18.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:41:18.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:41:18.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2863905464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:41:18.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T07:41:18.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[37158]: osdmap e691: 8 total, 8 up, 5 in 2024-02-05T07:41:18.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[37158]: 11.0 scrub starts 2024-02-05T07:41:18.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:18 smithi007 ceph-mon[37158]: 11.0 scrub ok 2024-02-05T07:41:18.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:19.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:19.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:18 smithi090 ceph-mon[33213]: 11.3 scrub starts 2024-02-05T07:41:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:18 smithi090 ceph-mon[33213]: 11.3 scrub ok 2024-02-05T07:41:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:18 smithi090 ceph-mon[33213]: 11.6 scrub starts 2024-02-05T07:41:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:18 smithi090 ceph-mon[33213]: 11.6 scrub ok 2024-02-05T07:41:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:18 smithi090 ceph-mon[33213]: pgmap v1262: 178 pgs: 2 peering, 7 unknown, 169 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:41:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:18 smithi090 ceph-mon[33213]: 11.7 scrub starts 2024-02-05T07:41:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:18 smithi090 ceph-mon[33213]: 11.7 scrub ok 2024-02-05T07:41:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:18 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T07:41:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:18 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:41:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:18 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2863905464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:41:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:18 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T07:41:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:18 smithi090 ceph-mon[33213]: osdmap e691: 8 total, 8 up, 5 in 2024-02-05T07:41:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:18 smithi090 ceph-mon[33213]: 11.0 scrub starts 2024-02-05T07:41:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:18 smithi090 ceph-mon[33213]: 11.0 scrub ok 2024-02-05T07:41:19.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:19.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:19.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:19.258 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:19.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:19.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:19.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:19.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:19.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:19.426 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 6, 3, 2] out_osds: [5, 7, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T07:41:19.427 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T07:41:19.427 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-05T07:41:19.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T07:41:19.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:19.509 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:19.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:19.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:19.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:19.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:19.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:19.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:19.766 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:19.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:19.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[29098]: 11.1 scrub starts 2024-02-05T07:41:19.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[29098]: 11.1 scrub ok 2024-02-05T07:41:19.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[29098]: 11.2 scrub starts 2024-02-05T07:41:19.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[29098]: 11.2 scrub ok 2024-02-05T07:41:19.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[29098]: osdmap e692: 8 total, 8 up, 5 in 2024-02-05T07:41:19.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[29098]: 11.9 scrub starts 2024-02-05T07:41:19.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[29098]: 11.9 scrub ok 2024-02-05T07:41:19.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[29098]: osdmap e693: 8 total, 8 up, 5 in 2024-02-05T07:41:19.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[37158]: 11.1 scrub starts 2024-02-05T07:41:19.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[37158]: 11.1 scrub ok 2024-02-05T07:41:19.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[37158]: 11.2 scrub starts 2024-02-05T07:41:19.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[37158]: 11.2 scrub ok 2024-02-05T07:41:19.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[37158]: osdmap e692: 8 total, 8 up, 5 in 2024-02-05T07:41:19.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[37158]: 11.9 scrub starts 2024-02-05T07:41:19.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[37158]: 11.9 scrub ok 2024-02-05T07:41:19.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:19 smithi007 ceph-mon[37158]: osdmap e693: 8 total, 8 up, 5 in 2024-02-05T07:41:19.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:19.891 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:19.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:19.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:19.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:20.016 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:20.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:20.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:20.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:19 smithi090 ceph-mon[33213]: 11.1 scrub starts 2024-02-05T07:41:20.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:19 smithi090 ceph-mon[33213]: 11.1 scrub ok 2024-02-05T07:41:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:19 smithi090 ceph-mon[33213]: 11.2 scrub starts 2024-02-05T07:41:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:19 smithi090 ceph-mon[33213]: 11.2 scrub ok 2024-02-05T07:41:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:19 smithi090 ceph-mon[33213]: osdmap e692: 8 total, 8 up, 5 in 2024-02-05T07:41:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:19 smithi090 ceph-mon[33213]: 11.9 scrub starts 2024-02-05T07:41:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:19 smithi090 ceph-mon[33213]: 11.9 scrub ok 2024-02-05T07:41:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:19 smithi090 ceph-mon[33213]: osdmap e693: 8 total, 8 up, 5 in 2024-02-05T07:41:20.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:20.141 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:20.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:20.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:20.267 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:20.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:20.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:20.368 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:20.392 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:20.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:20.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:20.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:20.520 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:20.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:20.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4 2024-02-05T07:41:20.647 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:20.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.4' 2024-02-05T07:41:20.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:20.776 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:20.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:20.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:20.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2 2024-02-05T07:41:20.906 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:20.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:20.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.2' 2024-02-05T07:41:21.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:21.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.4 scrub starts 2024-02-05T07:41:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.4 scrub ok 2024-02-05T07:41:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.12 scrub starts 2024-02-05T07:41:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.12 scrub ok 2024-02-05T07:41:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: pgmap v1265: 178 pgs: 3 remapped+peering, 2 peering, 4 unknown, 169 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 4.2 MiB/s, 66 objects/s recovering 2024-02-05T07:41:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 4.9 scrub starts 2024-02-05T07:41:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 4.9 scrub ok 2024-02-05T07:41:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 685'13568 2024-02-05T07:41:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 685'13712 2024-02-05T07:41:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13636 2024-02-05T07:41:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.19 starting backfill to osd.1 from (0'0,0'0] MAX to 685'13742 2024-02-05T07:41:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13688 2024-02-05T07:41:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 685'13890 2024-02-05T07:41:21.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.17 starting backfill to osd.3 from (0'0,0'0] MAX to 685'13766 2024-02-05T07:41:21.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 685'13754 2024-02-05T07:41:21.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13568 2024-02-05T07:41:21.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13742 2024-02-05T07:41:21.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13754 2024-02-05T07:41:21.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.18 scrub starts 2024-02-05T07:41:21.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:20 smithi090 ceph-mon[33213]: 11.18 scrub ok 2024-02-05T07:41:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:21.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:21.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:21.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:21.160 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:21.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:21.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:21.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.4 scrub starts 2024-02-05T07:41:21.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.4 scrub ok 2024-02-05T07:41:21.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.12 scrub starts 2024-02-05T07:41:21.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.12 scrub ok 2024-02-05T07:41:21.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: pgmap v1265: 178 pgs: 3 remapped+peering, 2 peering, 4 unknown, 169 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 4.2 MiB/s, 66 objects/s recovering 2024-02-05T07:41:21.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 4.9 scrub starts 2024-02-05T07:41:21.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 4.9 scrub ok 2024-02-05T07:41:21.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 685'13568 2024-02-05T07:41:21.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 685'13712 2024-02-05T07:41:21.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13636 2024-02-05T07:41:21.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.19 starting backfill to osd.1 from (0'0,0'0] MAX to 685'13742 2024-02-05T07:41:21.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13688 2024-02-05T07:41:21.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 685'13890 2024-02-05T07:41:21.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.17 starting backfill to osd.3 from (0'0,0'0] MAX to 685'13766 2024-02-05T07:41:21.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 685'13754 2024-02-05T07:41:21.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13568 2024-02-05T07:41:21.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13742 2024-02-05T07:41:21.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13754 2024-02-05T07:41:21.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.18 scrub starts 2024-02-05T07:41:21.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[37158]: 11.18 scrub ok 2024-02-05T07:41:21.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.4 scrub starts 2024-02-05T07:41:21.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.4 scrub ok 2024-02-05T07:41:21.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.12 scrub starts 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.12 scrub ok 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: pgmap v1265: 178 pgs: 3 remapped+peering, 2 peering, 4 unknown, 169 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 4.2 MiB/s, 66 objects/s recovering 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 4.9 scrub starts 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 4.9 scrub ok 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 685'13568 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 685'13712 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13636 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.19 starting backfill to osd.1 from (0'0,0'0] MAX to 685'13742 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13688 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 685'13890 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.17 starting backfill to osd.3 from (0'0,0'0] MAX to 685'13766 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 685'13754 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13568 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13742 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 685'13754 2024-02-05T07:41:21.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.18 scrub starts 2024-02-05T07:41:21.176 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:20 smithi007 ceph-mon[29098]: 11.18 scrub ok 2024-02-05T07:41:21.262 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:21.289 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:21.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:21.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:21.313 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-02-05T07:41:21.313 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T07:41:21.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:21.415 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:21.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:21.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:21.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0 2024-02-05T07:41:21.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:21.542 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:21.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.0' 2024-02-05T07:41:21.643 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:41:21.671 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:21.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:21.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:41:21.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:41:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:41:21.767 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_4' removed 2024-02-05T07:41:21.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3 2024-02-05T07:41:21.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T07:41:21.830 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:21.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.3' 2024-02-05T07:41:21.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6 2024-02-05T07:41:21.958 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:21.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.6' 2024-02-05T07:41:22.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:21 smithi090 ceph-mon[33213]: 11.5 scrub starts 2024-02-05T07:41:22.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:21 smithi090 ceph-mon[33213]: 11.5 scrub ok 2024-02-05T07:41:22.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:21 smithi090 ceph-mon[33213]: osdmap e694: 8 total, 8 up, 5 in 2024-02-05T07:41:22.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:21 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4096983350' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:41:22.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:21 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:41:22.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1 2024-02-05T07:41:22.087 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-02-05T07:41:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:22.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.1' 2024-02-05T07:41:22.136 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:21 smithi007 ceph-mon[29098]: 11.5 scrub starts 2024-02-05T07:41:22.136 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:21 smithi007 ceph-mon[29098]: 11.5 scrub ok 2024-02-05T07:41:22.136 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:21 smithi007 ceph-mon[29098]: osdmap e694: 8 total, 8 up, 5 in 2024-02-05T07:41:22.136 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:21 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4096983350' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:41:22.136 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:21 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:41:22.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:21 smithi007 ceph-mon[37158]: 11.5 scrub starts 2024-02-05T07:41:22.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:21 smithi007 ceph-mon[37158]: 11.5 scrub ok 2024-02-05T07:41:22.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:21 smithi007 ceph-mon[37158]: osdmap e694: 8 total, 8 up, 5 in 2024-02-05T07:41:22.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:21 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4096983350' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:41:22.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:21 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T07:41:22.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:22.213 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:22.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:22.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:22.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5 2024-02-05T07:41:22.339 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:22.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.5' 2024-02-05T07:41:22.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7 2024-02-05T07:41:22.464 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: docker: command not found 2024-02-05T07:41:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:41:22.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi090 with status 1: 'sudo docker kill -s 1 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd.7' 2024-02-05T07:41:22.495 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-02-05T07:41:22.505 INFO:tasks.thrashosds:joining thrashosds 2024-02-05T07:41:22.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T07:41:23.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:22 smithi090 ceph-mon[33213]: 11.14 scrub starts 2024-02-05T07:41:23.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:22 smithi090 ceph-mon[33213]: 11.14 scrub ok 2024-02-05T07:41:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:22 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:22 smithi090 ceph-mon[33213]: pgmap v1268: 178 pgs: 3 peering, 3 remapped+peering, 2 unknown, 170 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:41:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:22 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:41:23.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:22 smithi090 ceph-mon[33213]: osdmap e695: 8 total, 8 up, 5 in 2024-02-05T07:41:23.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:22 smithi007 ceph-mon[29098]: 11.14 scrub starts 2024-02-05T07:41:23.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:22 smithi007 ceph-mon[29098]: 11.14 scrub ok 2024-02-05T07:41:23.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:22 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:23.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:22 smithi007 ceph-mon[29098]: pgmap v1268: 178 pgs: 3 peering, 3 remapped+peering, 2 unknown, 170 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:41:23.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:22 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:41:23.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:22 smithi007 ceph-mon[29098]: osdmap e695: 8 total, 8 up, 5 in 2024-02-05T07:41:23.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:22 smithi007 ceph-mon[37158]: 11.14 scrub starts 2024-02-05T07:41:23.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:22 smithi007 ceph-mon[37158]: 11.14 scrub ok 2024-02-05T07:41:23.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:22 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:23.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:22 smithi007 ceph-mon[37158]: pgmap v1268: 178 pgs: 3 peering, 3 remapped+peering, 2 unknown, 170 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:41:23.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:22 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T07:41:23.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:22 smithi007 ceph-mon[37158]: osdmap e695: 8 total, 8 up, 5 in 2024-02-05T07:41:23.292 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:41:23.406 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:23.621 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:41:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:41:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:41:23.855 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:23.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:23 smithi007 ceph-mon[29098]: 2.0 deep-scrub starts 2024-02-05T07:41:23.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:23 smithi007 ceph-mon[29098]: 2.0 deep-scrub ok 2024-02-05T07:41:23.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:23 smithi007 ceph-mon[37158]: 2.0 deep-scrub starts 2024-02-05T07:41:23.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:23 smithi007 ceph-mon[37158]: 2.0 deep-scrub ok 2024-02-05T07:41:24.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:23 smithi090 ceph-mon[33213]: 2.0 deep-scrub starts 2024-02-05T07:41:24.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:23 smithi090 ceph-mon[33213]: 2.0 deep-scrub ok 2024-02-05T07:41:24.503 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-02-05T07:41:24.523 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-02-05T07:41:24.543 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-02-05T07:41:24.562 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-02-05T07:41:24.583 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-02-05T07:41:24.603 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-02-05T07:41:24.623 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-02-05T07:41:24.644 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-02-05T07:41:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T07:41:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T07:41:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T07:41:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T07:41:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T07:41:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T07:41:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T07:41:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T07:41:24.922 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:24 smithi007 ceph-mon[29098]: 6.1a scrub starts 2024-02-05T07:41:24.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:24 smithi007 ceph-mon[29098]: 6.1a scrub ok 2024-02-05T07:41:24.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:24 smithi007 ceph-mon[29098]: pgmap v1270: 152 pgs: 152 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 867 B/s rd, 0 op/s 2024-02-05T07:41:24.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:24 smithi007 ceph-mon[29098]: 4.13 scrub starts 2024-02-05T07:41:24.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:24 smithi007 ceph-mon[29098]: 4.13 scrub ok 2024-02-05T07:41:24.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:24 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2782054301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:41:24.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:24 smithi007 ceph-mon[29098]: osdmap e696: 8 total, 8 up, 5 in 2024-02-05T07:41:24.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:24 smithi007 ceph-mon[37158]: 6.1a scrub starts 2024-02-05T07:41:24.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:24 smithi007 ceph-mon[37158]: 6.1a scrub ok 2024-02-05T07:41:24.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:24 smithi007 ceph-mon[37158]: pgmap v1270: 152 pgs: 152 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 867 B/s rd, 0 op/s 2024-02-05T07:41:24.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:24 smithi007 ceph-mon[37158]: 4.13 scrub starts 2024-02-05T07:41:24.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:24 smithi007 ceph-mon[37158]: 4.13 scrub ok 2024-02-05T07:41:24.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:24 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2782054301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:41:24.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:24 smithi007 ceph-mon[37158]: osdmap e696: 8 total, 8 up, 5 in 2024-02-05T07:41:25.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:24 smithi090 ceph-mon[33213]: 6.1a scrub starts 2024-02-05T07:41:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:24 smithi090 ceph-mon[33213]: 6.1a scrub ok 2024-02-05T07:41:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:24 smithi090 ceph-mon[33213]: pgmap v1270: 152 pgs: 152 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 867 B/s rd, 0 op/s 2024-02-05T07:41:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:24 smithi090 ceph-mon[33213]: 4.13 scrub starts 2024-02-05T07:41:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:24 smithi090 ceph-mon[33213]: 4.13 scrub ok 2024-02-05T07:41:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:24 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2782054301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T07:41:25.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:24 smithi090 ceph-mon[33213]: osdmap e696: 8 total, 8 up, 5 in 2024-02-05T07:41:25.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:25 smithi007 ceph-mon[29098]: 4.4 scrub starts 2024-02-05T07:41:25.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:25 smithi007 ceph-mon[29098]: 4.4 scrub ok 2024-02-05T07:41:25.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:25 smithi007 ceph-mon[29098]: 3.10 scrub starts 2024-02-05T07:41:25.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:25 smithi007 ceph-mon[29098]: 3.10 scrub ok 2024-02-05T07:41:25.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:25 smithi007 ceph-mon[37158]: 4.4 scrub starts 2024-02-05T07:41:25.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:25 smithi007 ceph-mon[37158]: 4.4 scrub ok 2024-02-05T07:41:25.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:25 smithi007 ceph-mon[37158]: 3.10 scrub starts 2024-02-05T07:41:25.879 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:25 smithi007 ceph-mon[37158]: 3.10 scrub ok 2024-02-05T07:41:25.919 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:41:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:41:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-02-05T07:41:25.919 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:41:26.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:25 smithi090 ceph-mon[33213]: 4.4 scrub starts 2024-02-05T07:41:26.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:25 smithi090 ceph-mon[33213]: 4.4 scrub ok 2024-02-05T07:41:26.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:25 smithi090 ceph-mon[33213]: 3.10 scrub starts 2024-02-05T07:41:26.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:25 smithi090 ceph-mon[33213]: 3.10 scrub ok 2024-02-05T07:41:26.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T07:41:26.851 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:41:26.851 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-05T07:41:26.851 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 696, 2024-02-05T07:41:26.851 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "e3e7059e-c3f5-11ee-95b6-87774f69a715", 2024-02-05T07:41:26.851 INFO:teuthology.orchestra.run.smithi007.stdout: "created": "2024-02-05T07:13:14.603885+0000", 2024-02-05T07:41:26.851 INFO:teuthology.orchestra.run.smithi007.stdout: "modified": "2024-02-05T07:41:24.703037+0000", 2024-02-05T07:41:26.851 INFO:teuthology.orchestra.run.smithi007.stdout: "last_up_change": "2024-02-05T07:28:24.555976+0000", 2024-02-05T07:41:26.851 INFO:teuthology.orchestra.run.smithi007.stdout: "last_in_change": "2024-02-05T07:38:08.421173+0000", 2024-02-05T07:41:26.851 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T07:41:26.852 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_num": 5799936, 2024-02-05T07:41:26.852 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_set": [ 2024-02-05T07:41:26.852 INFO:teuthology.orchestra.run.smithi007.stdout: "pglog_hardlimit", 2024-02-05T07:41:26.852 INFO:teuthology.orchestra.run.smithi007.stdout: "purged_snapdirs", 2024-02-05T07:41:26.852 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_deletes", 2024-02-05T07:41:26.852 INFO:teuthology.orchestra.run.smithi007.stdout: "sortbitwise" 2024-02-05T07:41:26.852 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:41:26.852 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_version": 19, 2024-02-05T07:41:26.852 INFO:teuthology.orchestra.run.smithi007.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T07:41:26.852 INFO:teuthology.orchestra.run.smithi007.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T07:41:26.852 INFO:teuthology.orchestra.run.smithi007.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T07:41:26.852 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_snapshot": "", 2024-02-05T07:41:26.853 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_max": 11, 2024-02-05T07:41:26.853 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd": 8, 2024-02-05T07:41:26.853 INFO:teuthology.orchestra.run.smithi007.stdout: "require_min_compat_client": "luminous", 2024-02-05T07:41:26.853 INFO:teuthology.orchestra.run.smithi007.stdout: "min_compat_client": "luminous", 2024-02-05T07:41:26.853 INFO:teuthology.orchestra.run.smithi007.stdout: "require_osd_release": "squid", 2024-02-05T07:41:26.853 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_crimson": false, 2024-02-05T07:41:26.853 INFO:teuthology.orchestra.run.smithi007.stdout: "pools": [ 2024-02-05T07:41:26.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.853 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 1, 2024-02-05T07:41:26.853 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".mgr", 2024-02-05T07:41:26.853 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:16:32.505567+0000", 2024-02-05T07:41:26.853 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:41:26.853 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:41:26.853 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:41:26.854 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:41:26.854 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:41:26.854 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:41:26.854 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:41:26.854 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:41:26.854 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:41:26.854 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:41:26.854 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:41:26.854 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:41:26.854 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 21, 2024-02-05T07:41:26.854 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 21, 2024-02-05T07:41:26.854 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 21, 2024-02-05T07:41:26.854 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 21, 2024-02-05T07:41:26.854 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 21, 2024-02-05T07:41:26.855 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:41:26.855 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "1.15", 2024-02-05T07:41:26.855 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 681, 2024-02-05T07:41:26.855 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 680, 2024-02-05T07:41:26.855 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 680, 2024-02-05T07:41:26.855 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "86'128", 2024-02-05T07:41:26.855 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "86'128" 2024-02-05T07:41:26.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.855 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "681", 2024-02-05T07:41:26.855 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:41:26.855 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "681", 2024-02-05T07:41:26.855 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "679", 2024-02-05T07:41:26.855 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:41:26.855 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:41:26.856 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:41:26.856 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:41:26.856 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:41:26.856 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:41:26.856 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:41:26.856 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:41:26.856 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:41:26.856 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:41:26.856 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:41:26.856 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:41:26.856 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:41:26.856 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:41:26.856 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:41:26.856 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:41:26.857 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:41:26.857 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:41:26.857 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:41:26.857 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:41:26.857 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:41:26.857 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:41:26.857 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:41:26.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.857 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:41:26.857 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:41:26.857 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:41:26.857 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:41:26.857 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:41:26.857 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:41:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:41:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:41:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:41:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:41:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:41:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-02-05T07:41:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_max": 32, 2024-02-05T07:41:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_min": 1 2024-02-05T07:41:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:41:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": {} 2024-02-05T07:41:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:41:26.858 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:41:26.859 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.6800000667572021, 2024-02-05T07:41:26.859 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.6800000667572021, 2024-02-05T07:41:26.859 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.70999997854232788, 2024-02-05T07:41:26.859 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.8999999761581421, 2024-02-05T07:41:26.859 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.8999999761581421, 2024-02-05T07:41:26.859 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.7850494384765625, 2024-02-05T07:41:26.859 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.89999997615814209, 2024-02-05T07:41:26.859 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-02-05T07:41:26.859 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.859 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 2, 2024-02-05T07:41:26.859 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "datapool", 2024-02-05T07:41:26.859 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:47.259462+0000", 2024-02-05T07:41:26.860 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 8193, 2024-02-05T07:41:26.860 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T07:41:26.860 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:41:26.860 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:41:26.860 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:41:26.860 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:41:26.860 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:41:26.860 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:41:26.860 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:41:26.860 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:41:26.860 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:41:26.860 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:41:26.861 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 3, 2024-02-05T07:41:26.861 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 3, 2024-02-05T07:41:26.861 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 3, 2024-02-05T07:41:26.861 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 3, 2024-02-05T07:41:26.861 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 3, 2024-02-05T07:41:26.861 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:41:26.861 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:41:26.861 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:41:26.861 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:41:26.861 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:41:26.861 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:41:26.861 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:41:26.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.862 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "71", 2024-02-05T07:41:26.862 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:41:26.862 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:41:26.862 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:41:26.862 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:41:26.862 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:41:26.862 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 2, 2024-02-05T07:41:26.862 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 71, 2024-02-05T07:41:26.862 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:41:26.862 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:41:26.862 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:41:26.862 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:41:26.862 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:41:26.862 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.863 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:41:26.864 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:41:26.864 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:41:26.864 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:41:26.864 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:41:26.864 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:41:26.864 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:41:26.864 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:41:26.864 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:41:26.864 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:41:26.864 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:41:26.864 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:41:26.864 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:41:26.864 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd": {} 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 4.690000057220459, 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 4.690000057220459, 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.70999997854232788, 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 3.3299999237060547, 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 3.3299999237060547, 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.7850494384765625, 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.89999997615814209, 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.865 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 3, 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".rgw.root", 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:47.762336+0000", 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:41:26.866 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "69", 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:41:26.867 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:41:26.868 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:41:26.868 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:41:26.868 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:41:26.868 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:41:26.868 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:41:26.868 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:41:26.868 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:41:26.868 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:41:26.868 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:41:26.868 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:41:26.868 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:41:26.868 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:41:26.868 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:41:26.869 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:41:26.869 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:41:26.869 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:41:26.869 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:41:26.869 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:41:26.869 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:41:26.869 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:41:26.869 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:41:26.869 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:41:26.869 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:41:26.869 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:41:26.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.870 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:41:26.870 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:41:26.870 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:41:26.870 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:41:26.870 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:41:26.870 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:41:26.870 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:41:26.870 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:41:26.870 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:41:26.870 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:41:26.870 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:41:26.870 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:41:26.870 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.4200000762939453, 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.4200000762939453, 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.70999997854232788, 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.7200000286102295, 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.7200000286102295, 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.7850494384765625, 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.89999997615814209, 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.872 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 4, 2024-02-05T07:41:26.872 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.log", 2024-02-05T07:41:26.872 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:49.736599+0000", 2024-02-05T07:41:26.872 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:41:26.872 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:41:26.872 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:41:26.872 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:41:26.872 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:41:26.872 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:41:26.872 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:41:26.872 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:41:26.872 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:41:26.872 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:41:26.872 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "71", 2024-02-05T07:41:26.873 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:41:26.874 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:41:26.875 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.8599998950958252, 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.8599998950958252, 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.70999997854232788, 2024-02-05T07:41:26.876 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.0299999713897705, 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.0299999713897705, 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.7850494384765625, 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.89999997615814209, 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 5, 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.control", 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:51.696948+0000", 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:41:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:41:26.878 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:41:26.879 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:41:26.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.879 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "73", 2024-02-05T07:41:26.879 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:41:26.879 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:41:26.879 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:41:26.879 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:41:26.879 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:41:26.879 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:41:26.879 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:41:26.879 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:41:26.879 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:41:26.879 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:41:26.879 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:41:26.880 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:41:26.880 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:41:26.880 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:41:26.880 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:41:26.880 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:41:26.880 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:41:26.880 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:41:26.880 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:41:26.880 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:41:26.880 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:41:26.880 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:41:26.880 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:41:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.2000000476837158, 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.2000000476837158, 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.70999997854232788, 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.559999942779541, 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.559999942779541, 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.7850494384765625, 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.89999997615814209, 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 6, 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.meta", 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-02-05T07:18:53.768555+0000", 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-02-05T07:41:26.883 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-02-05T07:41:26.884 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-02-05T07:41:26.884 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-02-05T07:41:26.884 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-02-05T07:41:26.884 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-02-05T07:41:26.884 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-02-05T07:41:26.884 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-02-05T07:41:26.884 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-02-05T07:41:26.884 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-02-05T07:41:26.884 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-02-05T07:41:26.884 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-02-05T07:41:26.884 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-02-05T07:41:26.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.884 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "75", 2024-02-05T07:41:26.885 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-02-05T07:41:26.885 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T07:41:26.885 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T07:41:26.885 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-02-05T07:41:26.885 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-02-05T07:41:26.885 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-02-05T07:41:26.885 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-02-05T07:41:26.885 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-02-05T07:41:26.885 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-02-05T07:41:26.885 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-02-05T07:41:26.885 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-02-05T07:41:26.885 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-02-05T07:41:26.885 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-02-05T07:41:26.886 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-02-05T07:41:26.886 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-02-05T07:41:26.886 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-02-05T07:41:26.886 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-02-05T07:41:26.886 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-02-05T07:41:26.886 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T07:41:26.886 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T07:41:26.886 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T07:41:26.886 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-02-05T07:41:26.886 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-02-05T07:41:26.886 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-02-05T07:41:26.886 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-02-05T07:41:26.886 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-02-05T07:41:26.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.887 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-02-05T07:41:26.887 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-02-05T07:41:26.887 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-02-05T07:41:26.887 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-02-05T07:41:26.887 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-02-05T07:41:26.887 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T07:41:26.887 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-02-05T07:41:26.887 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-02-05T07:41:26.887 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-02-05T07:41:26.887 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-02-05T07:41:26.887 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-02-05T07:41:26.887 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-02-05T07:41:26.888 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_bias": 4 2024-02-05T07:41:26.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.888 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-02-05T07:41:26.888 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-02-05T07:41:26.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.888 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-02-05T07:41:26.888 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-02-05T07:41:26.888 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.2000000476837158, 2024-02-05T07:41:26.888 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.2000000476837158, 2024-02-05T07:41:26.888 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.70999997854232788, 2024-02-05T07:41:26.888 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.559999942779541, 2024-02-05T07:41:26.888 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.559999942779541, 2024-02-05T07:41:26.889 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.7850494384765625, 2024-02-05T07:41:26.889 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.89999997615814209, 2024-02-05T07:41:26.889 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-02-05T07:41:26.889 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.889 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.889 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:41:26.889 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:41:26.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.889 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-02-05T07:41:26.889 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "703c3948-8003-4015-9dd8-21019cf0eeb8", 2024-02-05T07:41:26.889 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:41:26.889 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:41:26.889 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:41:26.889 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0, 2024-02-05T07:41:26.890 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:41:26.890 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:41:26.890 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 8, 2024-02-05T07:41:26.890 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 693, 2024-02-05T07:41:26.890 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:41:26.890 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:41:26.890 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:41:26.890 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.890 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.890 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-02-05T07:41:26.890 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:41:26.890 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.890 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6802", 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.891 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6804", 2024-02-05T07:41:26.892 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:41:26.892 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.892 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.892 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:41:26.892 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.892 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.892 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6803", 2024-02-05T07:41:26.892 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2069016689 2024-02-05T07:41:26.892 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.892 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.893 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6801/2069016689", 2024-02-05T07:41:26.893 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6802/2069016689", 2024-02-05T07:41:26.893 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6804/2069016689", 2024-02-05T07:41:26.893 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6803/2069016689", 2024-02-05T07:41:26.893 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:41:26.893 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:41:26.893 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:41:26.893 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.893 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-02-05T07:41:26.893 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "5c3e7d1e-a61d-44b9-b21e-f91c99a096bc", 2024-02-05T07:41:26.893 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:41:26.894 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:41:26.894 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:41:26.894 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0.5441131591796875, 2024-02-05T07:41:26.894 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:41:26.894 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:41:26.894 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 13, 2024-02-05T07:41:26.894 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 689, 2024-02-05T07:41:26.894 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:41:26.894 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:41:26.894 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:41:26.894 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.895 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.895 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6805", 2024-02-05T07:41:26.895 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:41:26.895 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.895 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.895 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:41:26.895 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.895 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.895 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6806", 2024-02-05T07:41:26.895 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:41:26.895 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.896 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.896 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:41:26.896 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.896 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.896 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6808", 2024-02-05T07:41:26.896 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:41:26.896 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.896 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.896 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:41:26.896 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.897 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.897 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6807", 2024-02-05T07:41:26.897 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 865486532 2024-02-05T07:41:26.897 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.897 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.897 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6805/865486532", 2024-02-05T07:41:26.897 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6806/865486532", 2024-02-05T07:41:26.897 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6808/865486532", 2024-02-05T07:41:26.897 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6807/865486532", 2024-02-05T07:41:26.897 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:41:26.897 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:41:26.897 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:41:26.898 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.898 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-02-05T07:41:26.898 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "f4fd930f-7bb2-46ca-9964-98235bd47b3a", 2024-02-05T07:41:26.898 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:41:26.898 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:41:26.898 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:41:26.898 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:41:26.898 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:41:26.898 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:41:26.898 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 18, 2024-02-05T07:41:26.898 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 694, 2024-02-05T07:41:26.899 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:41:26.899 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:41:26.899 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:41:26.899 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.899 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.899 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6809", 2024-02-05T07:41:26.899 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:41:26.899 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.899 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.899 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:41:26.899 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.900 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.900 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6810", 2024-02-05T07:41:26.900 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:41:26.900 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.900 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.900 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:41:26.900 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.900 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.900 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6812", 2024-02-05T07:41:26.900 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:41:26.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.901 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.901 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:41:26.901 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.901 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.901 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6811", 2024-02-05T07:41:26.901 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 904871370 2024-02-05T07:41:26.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.901 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.901 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6809/904871370", 2024-02-05T07:41:26.901 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6810/904871370", 2024-02-05T07:41:26.902 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6812/904871370", 2024-02-05T07:41:26.902 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6811/904871370", 2024-02-05T07:41:26.902 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:41:26.902 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:41:26.902 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:41:26.902 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.902 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-02-05T07:41:26.902 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "149ed0d6-b049-4478-9819-cafc446e94d3", 2024-02-05T07:41:26.902 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:41:26.902 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:41:26.903 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:41:26.903 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:41:26.903 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:41:26.903 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:41:26.903 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 26, 2024-02-05T07:41:26.903 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 694, 2024-02-05T07:41:26.903 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:41:26.903 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:41:26.903 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:41:26.903 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.903 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.903 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6813", 2024-02-05T07:41:26.904 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:41:26.904 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.904 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.904 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:41:26.904 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.904 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.904 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6814", 2024-02-05T07:41:26.904 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:41:26.904 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.904 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.905 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:41:26.905 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.905 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.905 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6816", 2024-02-05T07:41:26.905 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:41:26.905 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.905 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.905 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:41:26.905 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.905 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.906 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6815", 2024-02-05T07:41:26.906 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2758586242 2024-02-05T07:41:26.906 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.906 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.906 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6813/2758586242", 2024-02-05T07:41:26.906 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6814/2758586242", 2024-02-05T07:41:26.906 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6816/2758586242", 2024-02-05T07:41:26.906 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6815/2758586242", 2024-02-05T07:41:26.906 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:41:26.907 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:41:26.907 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:41:26.907 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.907 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-02-05T07:41:26.907 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d", 2024-02-05T07:41:26.907 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:41:26.907 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-02-05T07:41:26.907 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-02-05T07:41:26.907 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:41:26.907 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:41:26.907 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:41:26.907 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 32, 2024-02-05T07:41:26.908 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 605, 2024-02-05T07:41:26.908 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:41:26.908 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:41:26.908 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:41:26.908 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.908 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.908 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6800", 2024-02-05T07:41:26.908 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:41:26.908 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.908 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.908 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:41:26.909 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.909 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.909 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6801", 2024-02-05T07:41:26.909 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:41:26.909 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.909 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.909 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:41:26.909 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.909 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.910 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6803", 2024-02-05T07:41:26.910 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:41:26.910 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.910 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.910 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:41:26.910 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.910 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.910 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.910 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6802", 2024-02-05T07:41:26.910 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1154929470 2024-02-05T07:41:26.910 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.910 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.911 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6800/1154929470", 2024-02-05T07:41:26.911 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6801/1154929470", 2024-02-05T07:41:26.911 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6803/1154929470", 2024-02-05T07:41:26.911 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6802/1154929470", 2024-02-05T07:41:26.911 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:41:26.911 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:41:26.911 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:41:26.911 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.911 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.911 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-02-05T07:41:26.911 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "244a7930-c22c-413e-86e1-11e0749f7367", 2024-02-05T07:41:26.912 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:41:26.912 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-02-05T07:41:26.912 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-02-05T07:41:26.912 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0.7362823486328125, 2024-02-05T07:41:26.912 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:41:26.912 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:41:26.912 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 41, 2024-02-05T07:41:26.912 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 336, 2024-02-05T07:41:26.912 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:41:26.912 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:41:26.912 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:41:26.912 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.912 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.913 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6804", 2024-02-05T07:41:26.913 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:41:26.913 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.913 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.913 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:41:26.913 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.913 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.913 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6805", 2024-02-05T07:41:26.913 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:41:26.913 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.913 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.914 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:41:26.914 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.914 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.914 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6807", 2024-02-05T07:41:26.914 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:41:26.914 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.914 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.914 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:41:26.914 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.914 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.914 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6806", 2024-02-05T07:41:26.915 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 398113577 2024-02-05T07:41:26.915 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.915 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.915 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6804/398113577", 2024-02-05T07:41:26.915 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6805/398113577", 2024-02-05T07:41:26.915 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6807/398113577", 2024-02-05T07:41:26.915 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6806/398113577", 2024-02-05T07:41:26.915 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:41:26.915 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:41:26.915 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:41:26.915 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.915 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "0adbb562-3012-4f60-b5ec-02550a10b2c7", 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 52, 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 230, 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 301, 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 694, 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 231, 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.916 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.917 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.917 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6808", 2024-02-05T07:41:26.917 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:41:26.917 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.917 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.917 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:41:26.917 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.917 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.917 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6809", 2024-02-05T07:41:26.917 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:41:26.918 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.918 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.918 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:41:26.918 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.918 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.918 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6811", 2024-02-05T07:41:26.918 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:41:26.918 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.918 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.919 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:41:26.919 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.919 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.919 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.919 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6810", 2024-02-05T07:41:26.919 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 313196533 2024-02-05T07:41:26.919 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.919 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.919 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6808/313196533", 2024-02-05T07:41:26.919 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6809/313196533", 2024-02-05T07:41:26.919 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6811/313196533", 2024-02-05T07:41:26.920 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6810/313196533", 2024-02-05T07:41:26.920 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:41:26.920 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:41:26.920 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:41:26.920 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.920 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.920 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-02-05T07:41:26.920 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "dbc26c36-a0e9-4f21-9e98-739270cea6d9", 2024-02-05T07:41:26.920 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-02-05T07:41:26.920 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-02-05T07:41:26.920 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-02-05T07:41:26.920 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 60, 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 349, 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6812", 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.921 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-02-05T07:41:26.922 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.922 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.922 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6813", 2024-02-05T07:41:26.922 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:41:26.922 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.922 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.922 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-02-05T07:41:26.922 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.922 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.922 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6815", 2024-02-05T07:41:26.922 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.90:6814", 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2983605980 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.90:6812/2983605980", 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.90:6813/2983605980", 2024-02-05T07:41:26.923 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2983605980", 2024-02-05T07:41:26.924 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.90:6814/2983605980", 2024-02-05T07:41:26.924 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-02-05T07:41:26.924 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-02-05T07:41:26.924 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-02-05T07:41:26.924 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.924 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.924 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:41:26.924 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_xinfo": [ 2024-02-05T07:41:26.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.924 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-02-05T07:41:26.924 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:41:26.924 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:41:26.924 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:40:40.189325+0000", 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:41:03.111657+0000", 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:41:26.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:41:05.056598+0000", 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:41:26.926 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:40:49.595209+0000", 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:41:03.118101+0000", 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-02-05T07:41:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:41:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:41:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:41:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:41:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-02-05T07:41:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:41:15.037578+0000", 2024-02-05T07:41:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:41:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-02-05T07:41:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:41:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:41:26.928 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:41:26.929 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:41:26.929 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-02-05T07:41:26.929 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:40:16.467562+0000", 2024-02-05T07:41:26.929 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 230 2024-02-05T07:41:26.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.929 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.929 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-02-05T07:41:26.929 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-02-05T07:41:26.929 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-02-05T07:41:26.929 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-02-05T07:41:26.929 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540701272860131327, 2024-02-05T07:41:26.929 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-02-05T07:41:26.930 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-02-05T07:40:35.938344+0000", 2024-02-05T07:41:26.930 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-02-05T07:41:26.930 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.930 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:41:26.930 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap": [ 2024-02-05T07:41:26.930 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.930 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.d", 2024-02-05T07:41:26.930 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:41:26.930 INFO:teuthology.orchestra.run.smithi007.stdout: 4, 2024-02-05T07:41:26.930 INFO:teuthology.orchestra.run.smithi007.stdout: 7, 2024-02-05T07:41:26.930 INFO:teuthology.orchestra.run.smithi007.stdout: 1 2024-02-05T07:41:26.930 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "6.0", 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: 1, 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: 0, 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: 6 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_items": [ 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.2", 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.931 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 3, 2024-02-05T07:41:26.932 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:41:26.932 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.932 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.932 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.d", 2024-02-05T07:41:26.932 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:41:26.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.932 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 6, 2024-02-05T07:41:26.932 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:41:26.932 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.932 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.19", 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.1b", 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 0, 2024-02-05T07:41:26.933 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.4", 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 0 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.7", 2024-02-05T07:41:26.934 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "6.8", 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 2, 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 0 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.936 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "6.1e", 2024-02-05T07:41:26.936 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:41:26.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.936 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 6, 2024-02-05T07:41:26.936 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-02-05T07:41:26.936 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.936 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.936 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "6.1f", 2024-02-05T07:41:26.936 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-02-05T07:41:26.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-02-05T07:41:26.936 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 3, 2024-02-05T07:41:26.937 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 0 2024-02-05T07:41:26.937 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.937 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-02-05T07:41:26.937 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.937 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-02-05T07:41:26.937 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_primaries": [], 2024-02-05T07:41:26.937 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_temp": [], 2024-02-05T07:41:26.937 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_temp": [], 2024-02-05T07:41:26.937 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklist": { 2024-02-05T07:41:26.937 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2180087772": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:41:26.937 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2066685086": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:41:26.937 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/4100020626": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:41:26.937 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1953351182": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:41:26.937 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3203189162": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3418781614": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1518592084": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/3380493069": "2024-02-06T07:19:28.895331+0000", 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2208590106": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/223579766": "2024-02-06T07:13:47.004765+0000", 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/2634895574": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1462812719": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2823976078": "2024-02-06T07:14:14.164720+0000", 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1905520582": "2024-02-06T07:13:47.004765+0000" 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: "range_blocklist": {}, 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profiles": { 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: "default": { 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-failure-domain": "osd", 2024-02-05T07:41:26.938 INFO:teuthology.orchestra.run.smithi007.stdout: "k": "2", 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "m": "1", 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin": "jerasure", 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "technique": "reed_sol_van" 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps_queue": [], 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "new_removed_snaps": [], 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "new_purged_snaps": [], 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_node_flags": {}, 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "device_class_flags": {}, 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode": { 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_enabled": false, 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_bucket_count": 0, 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "degraded_stretch_mode": 0, 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "recovering_stretch_mode": 0, 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_bucket": 0 2024-02-05T07:41:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-02-05T07:41:26.940 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-05T07:41:26.940 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:41:26.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:26 smithi007 ceph-mon[29098]: 6.9 scrub starts 2024-02-05T07:41:26.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:26 smithi007 ceph-mon[29098]: 6.9 scrub ok 2024-02-05T07:41:26.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:26 smithi007 ceph-mon[29098]: pgmap v1272: 152 pgs: 152 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 716 B/s rd, 0 op/s 2024-02-05T07:41:26.941 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:26 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/728307593' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:41:26.941 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:26 smithi007 ceph-mon[37158]: 6.9 scrub starts 2024-02-05T07:41:26.941 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:26 smithi007 ceph-mon[37158]: 6.9 scrub ok 2024-02-05T07:41:26.941 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:26 smithi007 ceph-mon[37158]: pgmap v1272: 152 pgs: 152 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 716 B/s rd, 0 op/s 2024-02-05T07:41:26.941 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:26 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/728307593' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:41:27.016 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:26 smithi090 ceph-mon[33213]: 6.9 scrub starts 2024-02-05T07:41:27.017 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:26 smithi090 ceph-mon[33213]: 6.9 scrub ok 2024-02-05T07:41:27.017 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:26 smithi090 ceph-mon[33213]: pgmap v1272: 152 pgs: 152 active+clean; 582 KiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 716 B/s rd, 0 op/s 2024-02-05T07:41:27.017 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:26 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/728307593' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T07:41:27.862 INFO:tasks.thrashosds.thrasher:j is {'epoch': 696, 'fsid': 'e3e7059e-c3f5-11ee-95b6-87774f69a715', 'created': '2024-02-05T07:13:14.603885+0000', 'modified': '2024-02-05T07:41:24.703037+0000', 'last_up_change': '2024-02-05T07:28:24.555976+0000', 'last_in_change': '2024-02-05T07:38:08.421173+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T07:16:32.505567+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '1.15', 'ready_epoch': 681, 'last_epoch_started': 680, 'last_epoch_clean': 680, 'source_version': "86'128", 'target_version': "86'128"}, 'last_change': '681', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '681', 'last_force_op_resend_preluminous': '679', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.680000066757202, 'score_stable': 2.680000066757202, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 1.899999976158142, 'raw_score_stable': 1.899999976158142, 'primary_affinity_weighted': 0.7850494384765625, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T07:18:47.259462+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.690000057220459, 'score_stable': 4.690000057220459, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 3.3299999237060547, 'raw_score_stable': 3.3299999237060547, 'primary_affinity_weighted': 0.7850494384765625, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T07:18:47.762336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4200000762939453, 'score_stable': 2.4200000762939453, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 1.7200000286102295, 'raw_score_stable': 1.7200000286102295, 'primary_affinity_weighted': 0.7850494384765625, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T07:18:49.736599+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.859999895095825, 'score_stable': 2.859999895095825, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 2.0299999713897705, 'raw_score_stable': 2.0299999713897705, 'primary_affinity_weighted': 0.7850494384765625, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T07:18:51.696948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.200000047683716, 'score_stable': 2.200000047683716, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 0.7850494384765625, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T07:18:53.768555+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.200000047683716, 'score_stable': 2.200000047683716, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 0.7850494384765625, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}], 'osds': [{'osd': 0, 'uuid': '703c3948-8003-4015-9dd8-21019cf0eeb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 693, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6801', 'nonce': 2069016689}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6802', 'nonce': 2069016689}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6804', 'nonce': 2069016689}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6803', 'nonce': 2069016689}]}, 'public_addr': '172.21.15.7:6801/2069016689', 'cluster_addr': '172.21.15.7:6802/2069016689', 'heartbeat_back_addr': '172.21.15.7:6804/2069016689', 'heartbeat_front_addr': '172.21.15.7:6803/2069016689', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '5c3e7d1e-a61d-44b9-b21e-f91c99a096bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5441131591796875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 689, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6805', 'nonce': 865486532}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6806', 'nonce': 865486532}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6808', 'nonce': 865486532}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6807', 'nonce': 865486532}]}, 'public_addr': '172.21.15.7:6805/865486532', 'cluster_addr': '172.21.15.7:6806/865486532', 'heartbeat_back_addr': '172.21.15.7:6808/865486532', 'heartbeat_front_addr': '172.21.15.7:6807/865486532', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4fd930f-7bb2-46ca-9964-98235bd47b3a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 694, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6809', 'nonce': 904871370}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6810', 'nonce': 904871370}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6812', 'nonce': 904871370}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6811', 'nonce': 904871370}]}, 'public_addr': '172.21.15.7:6809/904871370', 'cluster_addr': '172.21.15.7:6810/904871370', 'heartbeat_back_addr': '172.21.15.7:6812/904871370', 'heartbeat_front_addr': '172.21.15.7:6811/904871370', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '149ed0d6-b049-4478-9819-cafc446e94d3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 694, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6813', 'nonce': 2758586242}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6814', 'nonce': 2758586242}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6816', 'nonce': 2758586242}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6815', 'nonce': 2758586242}]}, 'public_addr': '172.21.15.7:6813/2758586242', 'cluster_addr': '172.21.15.7:6814/2758586242', 'heartbeat_back_addr': '172.21.15.7:6816/2758586242', 'heartbeat_front_addr': '172.21.15.7:6815/2758586242', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 605, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 1154929470}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6801', 'nonce': 1154929470}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6803', 'nonce': 1154929470}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 1154929470}]}, 'public_addr': '172.21.15.90:6800/1154929470', 'cluster_addr': '172.21.15.90:6801/1154929470', 'heartbeat_back_addr': '172.21.15.90:6803/1154929470', 'heartbeat_front_addr': '172.21.15.90:6802/1154929470', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '244a7930-c22c-413e-86e1-11e0749f7367', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.7362823486328125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 336, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 398113577}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6805', 'nonce': 398113577}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6807', 'nonce': 398113577}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 398113577}]}, 'public_addr': '172.21.15.90:6804/398113577', 'cluster_addr': '172.21.15.90:6805/398113577', 'heartbeat_back_addr': '172.21.15.90:6807/398113577', 'heartbeat_front_addr': '172.21.15.90:6806/398113577', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0adbb562-3012-4f60-b5ec-02550a10b2c7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 52, 'last_clean_end': 230, 'up_from': 301, 'up_thru': 694, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 313196533}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 313196533}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6811', 'nonce': 313196533}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6810', 'nonce': 313196533}]}, 'public_addr': '172.21.15.90:6808/313196533', 'cluster_addr': '172.21.15.90:6809/313196533', 'heartbeat_back_addr': '172.21.15.90:6811/313196533', 'heartbeat_front_addr': '172.21.15.90:6810/313196533', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dbc26c36-a0e9-4f21-9e98-739270cea6d9', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 349, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 2983605980}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 2983605980}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6815', 'nonce': 2983605980}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6814', 'nonce': 2983605980}]}, 'public_addr': '172.21.15.90:6812/2983605980', 'cluster_addr': '172.21.15.90:6813/2983605980', 'heartbeat_back_addr': '172.21.15.90:6815/2983605980', 'heartbeat_front_addr': '172.21.15.90:6814/2983605980', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:40:40.189325+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:41:03.111657+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:41:05.056598+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:40:49.595209+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T07:41:03.118101+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T07:41:15.037578+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T07:40:16.467562+0000', 'dead_epoch': 230}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T07:40:35.938344+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.d', 'osds': [4, 7, 1]}, {'pgid': '6.0', 'osds': [1, 0, 6]}], 'pg_upmap_items': [{'pgid': '3.2', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.d', 'mappings': [{'from': 6, 'to': 2}]}, {'pgid': '3.19', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '3.1b', 'mappings': [{'from': 0, 'to': 2}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '6.8', 'mappings': [{'from': 2, 'to': 0}]}, {'pgid': '6.1e', 'mappings': [{'from': 6, 'to': 2}]}, {'pgid': '6.1f', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.7:0/2180087772': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/2066685086': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/4100020626': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/1953351182': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/3203189162': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/3418781614': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:6800/1518592084': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:6800/3380493069': '2024-02-06T07:19:28.895331+0000', '172.21.15.7:0/2208590106': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:0/223579766': '2024-02-06T07:13:47.004765+0000', '172.21.15.7:6800/2634895574': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/1462812719': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/2823976078': '2024-02-06T07:14:14.164720+0000', '172.21.15.7:0/1905520582': '2024-02-06T07:13:47.004765+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T07:41:27.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:41:28.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:27 smithi007 ceph-mon[29098]: 3.1a scrub starts 2024-02-05T07:41:28.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:27 smithi007 ceph-mon[29098]: 3.1a scrub ok 2024-02-05T07:41:28.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:27 smithi007 ceph-mon[29098]: 3.1c scrub starts 2024-02-05T07:41:28.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:27 smithi007 ceph-mon[29098]: 3.1c scrub ok 2024-02-05T07:41:28.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:27 smithi007 ceph-mon[37158]: 3.1a scrub starts 2024-02-05T07:41:28.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:27 smithi007 ceph-mon[37158]: 3.1a scrub ok 2024-02-05T07:41:28.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:27 smithi007 ceph-mon[37158]: 3.1c scrub starts 2024-02-05T07:41:28.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:27 smithi007 ceph-mon[37158]: 3.1c scrub ok 2024-02-05T07:41:28.236 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:27 smithi090 ceph-mon[33213]: 3.1a scrub starts 2024-02-05T07:41:28.236 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:27 smithi090 ceph-mon[33213]: 3.1a scrub ok 2024-02-05T07:41:28.236 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:27 smithi090 ceph-mon[33213]: 3.1c scrub starts 2024-02-05T07:41:28.236 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:27 smithi090 ceph-mon[33213]: 3.1c scrub ok 2024-02-05T07:41:28.825 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:28.866 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:28 smithi090 ceph-mon[33213]: 1.1 scrub starts 2024-02-05T07:41:28.867 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:28 smithi090 ceph-mon[33213]: 1.1 scrub ok 2024-02-05T07:41:28.867 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:28 smithi090 ceph-mon[33213]: pgmap v1273: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 610 B/s rd, 0 op/s 2024-02-05T07:41:28.867 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:28 smithi090 ceph-mon[33213]: 6.19 scrub starts 2024-02-05T07:41:28.867 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:28 smithi090 ceph-mon[33213]: 6.19 scrub ok 2024-02-05T07:41:28.867 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:28 smithi090 ceph-mon[33213]: 4.1e scrub starts 2024-02-05T07:41:28.867 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:28 smithi090 ceph-mon[33213]: 4.1e scrub ok 2024-02-05T07:41:28.867 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:28 smithi090 ceph-mon[33213]: 1.f scrub starts 2024-02-05T07:41:28.867 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:28 smithi090 ceph-mon[33213]: 1.f scrub ok 2024-02-05T07:41:28.886 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:41:28.886 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:41:28.886 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:41:28.886 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:41:28.886 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:41:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[29098]: 1.1 scrub starts 2024-02-05T07:41:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[29098]: 1.1 scrub ok 2024-02-05T07:41:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[29098]: pgmap v1273: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 610 B/s rd, 0 op/s 2024-02-05T07:41:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[29098]: 6.19 scrub starts 2024-02-05T07:41:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[29098]: 6.19 scrub ok 2024-02-05T07:41:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[29098]: 4.1e scrub starts 2024-02-05T07:41:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[29098]: 4.1e scrub ok 2024-02-05T07:41:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[29098]: 1.f scrub starts 2024-02-05T07:41:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[29098]: 1.f scrub ok 2024-02-05T07:41:28.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[37158]: 1.1 scrub starts 2024-02-05T07:41:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[37158]: 1.1 scrub ok 2024-02-05T07:41:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[37158]: pgmap v1273: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 610 B/s rd, 0 op/s 2024-02-05T07:41:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[37158]: 6.19 scrub starts 2024-02-05T07:41:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[37158]: 6.19 scrub ok 2024-02-05T07:41:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[37158]: 4.1e scrub starts 2024-02-05T07:41:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[37158]: 4.1e scrub ok 2024-02-05T07:41:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[37158]: 1.f scrub starts 2024-02-05T07:41:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:28 smithi007 ceph-mon[37158]: 1.f scrub ok 2024-02-05T07:41:29.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T07:41:29.785 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-02-05T07:41:29.924 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:41:29.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:29 smithi007 ceph-mon[29098]: 5.9 scrub starts 2024-02-05T07:41:29.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:29 smithi007 ceph-mon[29098]: 5.9 scrub ok 2024-02-05T07:41:29.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:29 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1797891358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:41:29.943 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:29 smithi007 ceph-mon[37158]: 5.9 scrub starts 2024-02-05T07:41:29.943 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:29 smithi007 ceph-mon[37158]: 5.9 scrub ok 2024-02-05T07:41:29.943 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:29 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1797891358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:41:30.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:29 smithi090 ceph-mon[33213]: 5.9 scrub starts 2024-02-05T07:41:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:29 smithi090 ceph-mon[33213]: 5.9 scrub ok 2024-02-05T07:41:30.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:29 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1797891358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T07:41:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:30 smithi090 ceph-mon[33213]: pgmap v1274: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T07:41:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:30 smithi090 ceph-mon[33213]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:41:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:30 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1797891358' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:41:31.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:30 smithi090 ceph-mon[33213]: osdmap e697: 8 total, 8 up, 5 in 2024-02-05T07:41:31.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:30 smithi007 ceph-mon[29098]: pgmap v1274: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T07:41:31.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:30 smithi007 ceph-mon[29098]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:41:31.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:30 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1797891358' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:41:31.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:30 smithi007 ceph-mon[29098]: osdmap e697: 8 total, 8 up, 5 in 2024-02-05T07:41:31.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:30 smithi007 ceph-mon[37158]: pgmap v1274: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T07:41:31.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:30 smithi007 ceph-mon[37158]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T07:41:31.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:30 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1797891358' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T07:41:31.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:30 smithi007 ceph-mon[37158]: osdmap e697: 8 total, 8 up, 5 in 2024-02-05T07:41:31.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:41:31 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:41:31.650 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:41:31.652 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:41:31.704 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:41:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-02-05T07:41:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-02-05T07:41:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:41:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:90079 9.e 9:71a255a5:::benchmark_data_smithi007_136200_object10234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-02-05T07:41:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:32:00.549204+0000", 2024-02-05T07:41:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 571.15256532900003, 2024-02-05T07:41:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0150999940000001, 2024-02-05T07:41:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:41:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:41:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:41:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90079 2024-02-05T07:41:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549204+0000", 2024-02-05T07:41:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549204+0000", 2024-02-05T07:41:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549206+0000", 2024-02-05T07:41:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3859999999999999e-06 2024-02-05T07:41:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549227+0000", 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1117000000000001e-05 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549238+0000", 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1507e-05 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549245+0000", 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0469999999999996e-06 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549303+0000", 2024-02-05T07:41:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7494000000000002e-05 2024-02-05T07:41:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549437+0000", 2024-02-05T07:41:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134525 2024-02-05T07:41:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:41:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.549532+0000", 2024-02-05T07:41:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4881000000000002e-05 2024-02-05T07:41:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.551286+0000", 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001753665 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.554566+0000", 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032797939999999999 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.554578+0000", 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2604e-05 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.564255+0000", 2024-02-05T07:41:31.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0096767259999999 2024-02-05T07:41:31.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.564304+0000", 2024-02-05T07:41:31.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9248e-05 2024-02-05T07:41:31.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.710 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.710 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:90088 9.e 9:7a4763be:::benchmark_data_smithi007_136200_object10243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:32:00.560900+0000", 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 571.14086998200003, 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0181687180000001, 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90088 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560900+0000", 2024-02-05T07:41:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560900+0000", 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560901+0000", 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.234e-06 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560905+0000", 2024-02-05T07:41:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0840000000000002e-06 2024-02-05T07:41:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560916+0000", 2024-02-05T07:41:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1270000000000001e-05 2024-02-05T07:41:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560922+0000", 2024-02-05T07:41:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9619999999999998e-06 2024-02-05T07:41:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560982+0000", 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0009000000000001e-05 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.560994+0000", 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2305e-05 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.564648+0000", 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.003653269 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.564736+0000", 2024-02-05T07:41:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8554000000000006e-05 2024-02-05T07:41:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:41:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.564834+0000", 2024-02-05T07:41:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7688000000000004e-05 2024-02-05T07:41:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.569842+0000", 2024-02-05T07:41:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050076779999999998 2024-02-05T07:41:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.574424+0000", 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045827799999999998 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.579011+0000", 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045867130000000001 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.579043+0000", 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1834e-05 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.579068+0000", 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5338e-05 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:90091 9.e 9:73fec898:::benchmark_data_smithi007_136200_object10246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:32:00.571555+0000", 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 571.13021468600004, 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0075982779999999, 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:41:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:41:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90091 2024-02-05T07:41:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571555+0000", 2024-02-05T07:41:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571555+0000", 2024-02-05T07:41:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571556+0000", 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.243e-06 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571561+0000", 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4630000000000003e-06 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571571+0000", 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0285999999999999e-05 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571576+0000", 2024-02-05T07:41:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1780000000000002e-06 2024-02-05T07:41:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571617+0000", 2024-02-05T07:41:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1325000000000002e-05 2024-02-05T07:41:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.571626+0000", 2024-02-05T07:41:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1859999999999992e-06 2024-02-05T07:41:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.566392+0000", 2024-02-05T07:41:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99476686599999997 2024-02-05T07:41:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.566514+0000", 2024-02-05T07:41:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121883 2024-02-05T07:41:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:41:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.566620+0000", 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000105686 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.571944+0000", 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053239339999999998 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.574471+0000", 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002527446 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.579107+0000", 2024-02-05T07:41:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046361029999999999 2024-02-05T07:41:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.579134+0000", 2024-02-05T07:41:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6461999999999998e-05 2024-02-05T07:41:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.579153+0000", 2024-02-05T07:41:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9216999999999999e-05 2024-02-05T07:41:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:90094 9.e 9:7011d553:::benchmark_data_smithi007_136200_object10249:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-02-05T07:41:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:32:00.577316+0000", 2024-02-05T07:41:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 571.12445364799999, 2024-02-05T07:41:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0049051760000001, 2024-02-05T07:41:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:41:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:41:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90094 2024-02-05T07:41:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577316+0000", 2024-02-05T07:41:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577316+0000", 2024-02-05T07:41:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577317+0000", 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.068e-06 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577321+0000", 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8630000000000001e-06 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577332+0000", 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0740000000000001e-05 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577338+0000", 2024-02-05T07:41:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0680000000000002e-06 2024-02-05T07:41:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577382+0000", 2024-02-05T07:41:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4008000000000003e-05 2024-02-05T07:41:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.577389+0000", 2024-02-05T07:41:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8340000000000001e-06 2024-02-05T07:41:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.568360+0000", 2024-02-05T07:41:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99097137400000002 2024-02-05T07:41:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.568512+0000", 2024-02-05T07:41:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015188499999999999 2024-02-05T07:41:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:41:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.568626+0000", 2024-02-05T07:41:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011432700000000001 2024-02-05T07:41:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.574381+0000", 2024-02-05T07:41:31.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0057548419999999996 2024-02-05T07:41:31.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.574517+0000", 2024-02-05T07:41:31.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136496 2024-02-05T07:41:31.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.582160+0000", 2024-02-05T07:41:31.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0076425549999999997 2024-02-05T07:41:31.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.582193+0000", 2024-02-05T07:41:31.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2756999999999998e-05 2024-02-05T07:41:31.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.582221+0000", 2024-02-05T07:41:31.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8359000000000001e-05 2024-02-05T07:41:31.730 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.730 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.730 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.731 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:90110 9.e 9:731cf300:::benchmark_data_smithi007_136200_object10265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-02-05T07:41:31.732 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:32:00.600929+0000", 2024-02-05T07:41:31.732 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 571.10084074400004, 2024-02-05T07:41:31.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99471859399999996, 2024-02-05T07:41:31.732 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.732 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.732 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.732 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:41:31.732 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:41:31.732 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90110 2024-02-05T07:41:31.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.732 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.600929+0000", 2024-02-05T07:41:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.600929+0000", 2024-02-05T07:41:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.600930+0000", 2024-02-05T07:41:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.102e-06 2024-02-05T07:41:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.600934+0000", 2024-02-05T07:41:31.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2289999999999997e-06 2024-02-05T07:41:31.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.600945+0000", 2024-02-05T07:41:31.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0657e-05 2024-02-05T07:41:31.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.600951+0000", 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7649999999999999e-06 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.601002+0000", 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1172999999999999e-05 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.601018+0000", 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6612999999999998e-05 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.569888+0000", 2024-02-05T07:41:31.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.96886916199999995 2024-02-05T07:41:31.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.570002+0000", 2024-02-05T07:41:31.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011484699999999999 2024-02-05T07:41:31.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:41:31.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.570079+0000", 2024-02-05T07:41:31.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6959999999999995e-05 2024-02-05T07:41:31.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.574387+0000", 2024-02-05T07:41:31.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043078680000000003 2024-02-05T07:41:31.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.582267+0000", 2024-02-05T07:41:31.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0078799030000000006 2024-02-05T07:41:31.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.595585+0000", 2024-02-05T07:41:31.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013318194 2024-02-05T07:41:31.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.595621+0000", 2024-02-05T07:41:31.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5772000000000002e-05 2024-02-05T07:41:31.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.595647+0000", 2024-02-05T07:41:31.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6349000000000001e-05 2024-02-05T07:41:31.738 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.738 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.738 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.739 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25268.0:90126 9.e 9:7051948f:::benchmark_data_smithi007_136200_object10281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-02-05T07:41:31.739 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:32:00.630567+0000", 2024-02-05T07:41:31.739 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 571.07120212699999, 2024-02-05T07:41:31.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.967434503, 2024-02-05T07:41:31.739 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.739 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.739 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.739 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25268", 2024-02-05T07:41:31.739 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1730877968", 2024-02-05T07:41:31.739 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90126 2024-02-05T07:41:31.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.739 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630567+0000", 2024-02-05T07:41:31.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630567+0000", 2024-02-05T07:41:31.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630570+0000", 2024-02-05T07:41:31.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3360000000000002e-06 2024-02-05T07:41:31.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630579+0000", 2024-02-05T07:41:31.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6069999999999997e-06 2024-02-05T07:41:31.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630607+0000", 2024-02-05T07:41:31.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8087000000000001e-05 2024-02-05T07:41:31.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630623+0000", 2024-02-05T07:41:31.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5716e-05 2024-02-05T07:41:31.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630756+0000", 2024-02-05T07:41:31.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013253200000000001 2024-02-05T07:41:31.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:31.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:00.630773+0000", 2024-02-05T07:41:31.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7391999999999999e-05 2024-02-05T07:41:31.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.571034+0000", 2024-02-05T07:41:31.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94026063100000001 2024-02-05T07:41:31.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.571111+0000", 2024-02-05T07:41:31.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6854000000000007e-05 2024-02-05T07:41:31.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,1", 2024-02-05T07:41:31.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.571177+0000", 2024-02-05T07:41:31.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5925000000000003e-05 2024-02-05T07:41:31.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.574481+0000", 2024-02-05T07:41:31.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033049580000000002 2024-02-05T07:41:31.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.583698+0000", 2024-02-05T07:41:31.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092162170000000005 2024-02-05T07:41:31.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.597946+0000", 2024-02-05T07:41:31.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014248475 2024-02-05T07:41:31.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.597979+0000", 2024-02-05T07:41:31.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3009999999999997e-05 2024-02-05T07:41:31.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:32:01.598002+0000", 2024-02-05T07:41:31.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2762999999999999e-05 2024-02-05T07:41:31.748 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.748 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.748 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.748 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:158023 10.17 10:e94d5d52:::benchmark_data_smithi007_157098_object29093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-02-05T07:41:31.749 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:46.698078+0000", 2024-02-05T07:41:31.749 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 345.003691554, 2024-02-05T07:41:31.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0361030979999999, 2024-02-05T07:41:31.749 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.749 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.749 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.749 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:31.749 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:31.749 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158023 2024-02-05T07:41:31.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.749 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.698078+0000", 2024-02-05T07:41:31.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.698078+0000", 2024-02-05T07:41:31.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.698079+0000", 2024-02-05T07:41:31.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.066e-06 2024-02-05T07:41:31.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.698083+0000", 2024-02-05T07:41:31.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5949999999999999e-06 2024-02-05T07:41:31.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.698090+0000", 2024-02-05T07:41:31.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7929999999999998e-06 2024-02-05T07:41:31.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.698100+0000", 2024-02-05T07:41:31.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1279999999999997e-06 2024-02-05T07:41:31.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.698157+0000", 2024-02-05T07:41:31.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7850999999999999e-05 2024-02-05T07:41:31.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:31.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.698165+0000", 2024-02-05T07:41:31.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7980000000000006e-06 2024-02-05T07:41:31.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.729369+0000", 2024-02-05T07:41:31.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031203697 2024-02-05T07:41:31.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.729434+0000", 2024-02-05T07:41:31.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5111999999999996e-05 2024-02-05T07:41:31.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2024-02-05T07:41:31.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.729647+0000", 2024-02-05T07:41:31.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000212668 2024-02-05T07:41:31.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.733907+0000", 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042605669999999998 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.734050+0000", 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000142977 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.734101+0000", 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1082000000000003e-05 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.734147+0000", 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5459e-05 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.734181+0000", 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4304999999999998e-05 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:158031 10.17 10:ec231209:::benchmark_data_smithi007_157098_object29101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:46.706792+0000", 2024-02-05T07:41:31.755 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 344.994977425, 2024-02-05T07:41:31.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0299326710000001, 2024-02-05T07:41:31.756 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.756 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:31.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:31.756 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158031 2024-02-05T07:41:31.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.756 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706792+0000", 2024-02-05T07:41:31.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706792+0000", 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706793+0000", 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.046e-06 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706799+0000", 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2989999999999996e-06 2024-02-05T07:41:31.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706809+0000", 2024-02-05T07:41:31.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0885e-05 2024-02-05T07:41:31.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706818+0000", 2024-02-05T07:41:31.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3280000000000006e-06 2024-02-05T07:41:31.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706880+0000", 2024-02-05T07:41:31.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1885999999999997e-05 2024-02-05T07:41:31.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:31.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706889+0000", 2024-02-05T07:41:31.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0019999999999995e-06 2024-02-05T07:41:31.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.731321+0000", 2024-02-05T07:41:31.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0244321789999999 2024-02-05T07:41:31.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.731434+0000", 2024-02-05T07:41:31.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011311699999999999 2024-02-05T07:41:31.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2024-02-05T07:41:31.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.731652+0000", 2024-02-05T07:41:31.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000217975 2024-02-05T07:41:31.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.734063+0000", 2024-02-05T07:41:31.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024108250000000001 2024-02-05T07:41:31.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.736563+0000", 2024-02-05T07:41:31.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002500572 2024-02-05T07:41:31.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.736604+0000", 2024-02-05T07:41:31.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0249e-05 2024-02-05T07:41:31.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.736686+0000", 2024-02-05T07:41:31.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2361999999999995e-05 2024-02-05T07:41:31.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.736725+0000", 2024-02-05T07:41:31.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8946000000000003e-05 2024-02-05T07:41:31.762 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.762 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.762 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.762 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:158097 10.17 10:ee64b248:::benchmark_data_smithi007_157098_object29167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-02-05T07:41:31.762 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:46.753221+0000", 2024-02-05T07:41:31.762 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 344.948548894, 2024-02-05T07:41:31.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98368597000000002, 2024-02-05T07:41:31.763 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.763 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:31.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:31.763 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158097 2024-02-05T07:41:31.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.763 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.753221+0000", 2024-02-05T07:41:31.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.753221+0000", 2024-02-05T07:41:31.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.753221+0000", 2024-02-05T07:41:31.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4199999999999995e-07 2024-02-05T07:41:31.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.753224+0000", 2024-02-05T07:41:31.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9679999999999998e-06 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.753233+0000", 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8210000000000008e-06 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.753239+0000", 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.502e-06 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.753281+0000", 2024-02-05T07:41:31.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1950000000000003e-05 2024-02-05T07:41:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.753287+0000", 2024-02-05T07:41:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6499999999999999e-06 2024-02-05T07:41:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.732712+0000", 2024-02-05T07:41:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.97942435999999999 2024-02-05T07:41:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.732824+0000", 2024-02-05T07:41:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111889 2024-02-05T07:41:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2024-02-05T07:41:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.732989+0000", 2024-02-05T07:41:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165033 2024-02-05T07:41:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.736528+0000", 2024-02-05T07:41:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003539665 2024-02-05T07:41:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.736760+0000", 2024-02-05T07:41:31.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023222 2024-02-05T07:41:31.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.736796+0000", 2024-02-05T07:41:31.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5293999999999998e-05 2024-02-05T07:41:31.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.736868+0000", 2024-02-05T07:41:31.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2615999999999997e-05 2024-02-05T07:41:31.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.736907+0000", 2024-02-05T07:41:31.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260000000000003e-05 2024-02-05T07:41:31.769 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.769 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.769 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.769 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:158105 10.17 10:ec6441d2:::benchmark_data_smithi007_157098_object29175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-02-05T07:41:31.769 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:46.760490+0000", 2024-02-05T07:41:31.769 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 344.94127999599999, 2024-02-05T07:41:31.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98262503999999995, 2024-02-05T07:41:31.770 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.770 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:31.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:31.770 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158105 2024-02-05T07:41:31.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.770 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.760490+0000", 2024-02-05T07:41:31.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.760490+0000", 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.760491+0000", 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.176e-06 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.760498+0000", 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8299999999999998e-06 2024-02-05T07:41:31.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.760508+0000", 2024-02-05T07:41:31.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0712e-05 2024-02-05T07:41:31.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.760514+0000", 2024-02-05T07:41:31.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7799999999999997e-06 2024-02-05T07:41:31.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.760565+0000", 2024-02-05T07:41:31.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0498999999999999e-05 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.760574+0000", 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3729999999999999e-06 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.734215+0000", 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.97364081499999999 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.734309+0000", 2024-02-05T07:41:31.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4115000000000003e-05 2024-02-05T07:41:31.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2024-02-05T07:41:31.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.734473+0000", 2024-02-05T07:41:31.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000164141 2024-02-05T07:41:31.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.736574+0000", 2024-02-05T07:41:31.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021014670000000001 2024-02-05T07:41:31.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.740514+0000", 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039390129999999999 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.743033+0000", 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025199659999999998 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.743081+0000", 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7756999999999997e-05 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.743115+0000", 2024-02-05T07:41:31.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3395999999999997e-05 2024-02-05T07:41:31.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.776 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.776 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:158112 10.17 10:ed668f68:::benchmark_data_smithi007_157098_object29182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-02-05T07:41:31.776 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:46.765941+0000", 2024-02-05T07:41:31.776 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 344.93582824399999, 2024-02-05T07:41:31.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.97731131100000002, 2024-02-05T07:41:31.776 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.776 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.777 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:31.777 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:31.777 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158112 2024-02-05T07:41:31.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.777 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.765941+0000", 2024-02-05T07:41:31.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.765941+0000", 2024-02-05T07:41:31.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.765942+0000", 2024-02-05T07:41:31.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1900000000000002e-07 2024-02-05T07:41:31.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.765946+0000", 2024-02-05T07:41:31.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.523e-06 2024-02-05T07:41:31.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.765954+0000", 2024-02-05T07:41:31.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8470000000000002e-06 2024-02-05T07:41:31.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.765960+0000", 2024-02-05T07:41:31.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.412e-06 2024-02-05T07:41:31.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.766016+0000", 2024-02-05T07:41:31.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6217000000000001e-05 2024-02-05T07:41:31.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:31.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.766027+0000", 2024-02-05T07:41:31.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0566e-05 2024-02-05T07:41:31.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.735406+0000", 2024-02-05T07:41:31.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.96937901900000001 2024-02-05T07:41:31.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.735514+0000", 2024-02-05T07:41:31.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108421 2024-02-05T07:41:31.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2024-02-05T07:41:31.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.735677+0000", 2024-02-05T07:41:31.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163065 2024-02-05T07:41:31.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.740466+0000", 2024-02-05T07:41:31.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047886509999999997 2024-02-05T07:41:31.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.743149+0000", 2024-02-05T07:41:31.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002683469 2024-02-05T07:41:31.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.743182+0000", 2024-02-05T07:41:31.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2317999999999998e-05 2024-02-05T07:41:31.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.743223+0000", 2024-02-05T07:41:31.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1e-05 2024-02-05T07:41:31.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:47.743253+0000", 2024-02-05T07:41:31.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0083999999999999e-05 2024-02-05T07:41:31.783 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.783 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.783 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.783 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:235364 10.e 10:79d7b3fe:::benchmark_data_smithi007_157098_object106434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-02-05T07:41:31.783 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:37:00.535802+0000", 2024-02-05T07:41:31.783 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 271.16596756799999, 2024-02-05T07:41:31.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097021198000000003, 2024-02-05T07:41:31.783 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.784 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.784 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.784 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:31.784 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:31.784 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 235364 2024-02-05T07:41:31.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.784 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.535802+0000", 2024-02-05T07:41:31.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.535802+0000", 2024-02-05T07:41:31.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.535803+0000", 2024-02-05T07:41:31.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1999999999999999e-06 2024-02-05T07:41:31.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.535808+0000", 2024-02-05T07:41:31.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.87e-06 2024-02-05T07:41:31.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.535820+0000", 2024-02-05T07:41:31.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1442e-05 2024-02-05T07:41:31.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.535829+0000", 2024-02-05T07:41:31.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.91e-06 2024-02-05T07:41:31.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.535891+0000", 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.143e-05 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.536009+0000", 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117831 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.536121+0000", 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112402 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.540401+0000", 2024-02-05T07:41:31.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042794790000000001 2024-02-05T07:41:31.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.541001+0000", 2024-02-05T07:41:31.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060018699999999996 2024-02-05T07:41:31.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.632728+0000", 2024-02-05T07:41:31.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.091727068999999994 2024-02-05T07:41:31.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.632791+0000", 2024-02-05T07:41:31.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3131000000000003e-05 2024-02-05T07:41:31.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.632823+0000", 2024-02-05T07:41:31.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2246999999999998e-05 2024-02-05T07:41:31.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.789 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.789 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:235370 10.e 10:7315b34f:::benchmark_data_smithi007_157098_object106440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-02-05T07:41:31.790 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:37:00.538184+0000", 2024-02-05T07:41:31.790 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 271.16358555300002, 2024-02-05T07:41:31.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094775743999999995, 2024-02-05T07:41:31.790 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.790 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:31.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:31.790 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 235370 2024-02-05T07:41:31.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.790 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.538184+0000", 2024-02-05T07:41:31.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.538184+0000", 2024-02-05T07:41:31.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.538185+0000", 2024-02-05T07:41:31.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1399999999999996e-07 2024-02-05T07:41:31.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.538188+0000", 2024-02-05T07:41:31.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3790000000000002e-06 2024-02-05T07:41:31.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.538196+0000", 2024-02-05T07:41:31.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2960000000000001e-06 2024-02-05T07:41:31.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.538202+0000", 2024-02-05T07:41:31.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6259999999999997e-06 2024-02-05T07:41:31.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.538279+0000", 2024-02-05T07:41:31.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-02-05T07:41:31.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.538385+0000", 2024-02-05T07:41:31.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010637900000000001 2024-02-05T07:41:31.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:41:31.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.538515+0000", 2024-02-05T07:41:31.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012926300000000001 2024-02-05T07:41:31.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.541041+0000", 2024-02-05T07:41:31.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002526754 2024-02-05T07:41:31.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.541140+0000", 2024-02-05T07:41:31.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8404000000000002e-05 2024-02-05T07:41:31.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.632874+0000", 2024-02-05T07:41:31.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.091734289999999996 2024-02-05T07:41:31.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.632924+0000", 2024-02-05T07:41:31.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0275999999999998e-05 2024-02-05T07:41:31.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.632960+0000", 2024-02-05T07:41:31.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5499e-05 2024-02-05T07:41:31.796 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.796 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.796 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.796 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:235382 10.e 10:7e6965eb:::benchmark_data_smithi007_157098_object106452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-02-05T07:41:31.796 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:37:00.544895+0000", 2024-02-05T07:41:31.796 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 271.15687410300001, 2024-02-05T07:41:31.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.088215947000000003, 2024-02-05T07:41:31.796 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.796 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:31.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:31.797 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 235382 2024-02-05T07:41:31.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.797 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.544895+0000", 2024-02-05T07:41:31.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.544895+0000", 2024-02-05T07:41:31.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.544896+0000", 2024-02-05T07:41:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5499999999999997e-07 2024-02-05T07:41:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.544900+0000", 2024-02-05T07:41:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7589999999999999e-06 2024-02-05T07:41:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.544908+0000", 2024-02-05T07:41:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4359999999999996e-06 2024-02-05T07:41:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.544913+0000", 2024-02-05T07:41:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7440000000000004e-06 2024-02-05T07:41:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.544964+0000", 2024-02-05T07:41:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0671000000000003e-05 2024-02-05T07:41:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.545068+0000", 2024-02-05T07:41:31.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010377799999999999 2024-02-05T07:41:31.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2024-02-05T07:41:31.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.545135+0000", 2024-02-05T07:41:31.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7601000000000005e-05 2024-02-05T07:41:31.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.546836+0000", 2024-02-05T07:41:31.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017007929999999999 2024-02-05T07:41:31.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.547527+0000", 2024-02-05T07:41:31.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069117899999999999 2024-02-05T07:41:31.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.633015+0000", 2024-02-05T07:41:31.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.085487456000000003 2024-02-05T07:41:31.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.633072+0000", 2024-02-05T07:41:31.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7244000000000003e-05 2024-02-05T07:41:31.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:00.633111+0000", 2024-02-05T07:41:31.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9431000000000002e-05 2024-02-05T07:41:31.802 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.802 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.802 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.802 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:236434 10.17 10:e95a4c33:::benchmark_data_smithi007_157098_object107504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-02-05T07:41:31.802 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:37:01.411966+0000", 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 270.28980309299999, 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.082909132999999996, 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 236434 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:01.411966+0000", 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:01.411966+0000", 2024-02-05T07:41:31.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:01.411967+0000", 2024-02-05T07:41:31.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1999999999999999e-07 2024-02-05T07:41:31.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:01.411973+0000", 2024-02-05T07:41:31.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5949999999999998e-06 2024-02-05T07:41:31.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:01.411980+0000", 2024-02-05T07:41:31.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3649999999999998e-06 2024-02-05T07:41:31.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:01.411985+0000", 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8579999999999999e-06 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:01.412027+0000", 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1928000000000002e-05 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:01.412103+0000", 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6077999999999999e-05 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2024-02-05T07:41:31.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:01.412168+0000", 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5223999999999998e-05 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:01.485901+0000", 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.073732945999999994 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:01.486086+0000", 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185107 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:01.494787+0000", 2024-02-05T07:41:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087005940000000007 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:01.494839+0000", 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1641000000000002e-05 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:01.494876+0000", 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7076999999999998e-05 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:255381 10.c 10:34b6573f:::benchmark_data_smithi007_157098_object126451:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e562)", 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:37:16.419515+0000", 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 255.28225450299999, 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068108400999999999, 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 255381 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:16.419515+0000", 2024-02-05T07:41:31.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:31.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:16.419515+0000", 2024-02-05T07:41:31.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:31.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:31.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:16.419516+0000", 2024-02-05T07:41:31.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8000000000000005e-07 2024-02-05T07:41:31.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:31.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:16.419519+0000", 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1609999999999999e-06 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:16.419532+0000", 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2853999999999999e-05 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:16.419539+0000", 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.216e-06 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:31.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:16.419599+0000", 2024-02-05T07:41:31.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0053999999999998e-05 2024-02-05T07:41:31.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:31.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:16.419703+0000", 2024-02-05T07:41:31.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103358 2024-02-05T07:41:31.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3", 2024-02-05T07:41:31.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:16.419804+0000", 2024-02-05T07:41:31.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010131 2024-02-05T07:41:31.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:31.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:16.421769+0000", 2024-02-05T07:41:31.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001964721 2024-02-05T07:41:31.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:16.422429+0000", 2024-02-05T07:41:31.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066033599999999997 2024-02-05T07:41:31.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:31.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:16.487561+0000", 2024-02-05T07:41:31.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.065131720000000004 2024-02-05T07:41:31.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:16.487593+0000", 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2243999999999998e-05 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:37:16.487623+0000", 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0647e-05 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:31.814 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:41:31.922 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:31 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:41:31.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:31 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:41:32.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:31 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:41:32.368 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1276,"stamp":"2024-02-05T07:41:31.455043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2180,"num_read_kb":1932,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":504,"num_bytes_recovered":275971,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1178,"ondisk_log_size":1178,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":456,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":5193464,"kb_used_data":19712,"kb_used_omap":11,"kb_used_meta":5173684,"kb_avail":463532296,"statfs":{"total":479975178240,"available":474657071104,"internally_reserved":0,"allocated":20185088,"data_stored":12338905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12005,"internal_metadata":5297852699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002015"},"pg_stats":[{"pgid":"1.9","version":"86'128","reported_seq":1542,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824311+0000","last_change":"2024-02-05T07:40:40.778976+0000","last_active":"2024-02-05T07:41:24.824311+0000","last_peered":"2024-02-05T07:41:24.824311+0000","last_clean":"2024-02-05T07:41:24.824311+0000","last_became_active":"2024-02-05T07:40:40.778437+0000","last_became_peered":"2024-02-05T07:40:40.778437+0000","last_unstale":"2024-02-05T07:41:24.824311+0000","last_undegraded":"2024-02-05T07:41:24.824311+0000","last_fullsized":"2024-02-05T07:41:24.824311+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.337196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.337196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"615'35","reported_seq":880,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825158+0000","last_change":"2024-02-05T07:38:09.523613+0000","last_active":"2024-02-05T07:41:24.825158+0000","last_peered":"2024-02-05T07:41:24.825158+0000","last_clean":"2024-02-05T07:41:24.825158+0000","last_became_active":"2024-02-05T07:38:09.493734+0000","last_became_peered":"2024-02-05T07:38:09.493734+0000","last_unstale":"2024-02-05T07:41:24.825158+0000","last_undegraded":"2024-02-05T07:41:24.825158+0000","last_fullsized":"2024-02-05T07:41:24.825158+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:37:02.567716+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:37:02.567716+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":903,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824406+0000","last_change":"2024-02-05T07:38:37.381786+0000","last_active":"2024-02-05T07:41:24.824406+0000","last_peered":"2024-02-05T07:41:24.824406+0000","last_clean":"2024-02-05T07:41:24.824406+0000","last_became_active":"2024-02-05T07:38:09.496706+0000","last_became_peered":"2024-02-05T07:38:09.496706+0000","last_unstale":"2024-02-05T07:41:24.824406+0000","last_undegraded":"2024-02-05T07:41:24.824406+0000","last_fullsized":"2024-02-05T07:41:24.824406+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:37.381317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:37.381317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"538'13","reported_seq":407,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784487+0000","last_change":"2024-02-05T07:39:03.630111+0000","last_active":"2024-02-05T07:41:29.784487+0000","last_peered":"2024-02-05T07:41:29.784487+0000","last_clean":"2024-02-05T07:41:29.784487+0000","last_became_active":"2024-02-05T07:36:32.743613+0000","last_became_peered":"2024-02-05T07:36:32.743613+0000","last_unstale":"2024-02-05T07:41:29.784487+0000","last_undegraded":"2024-02-05T07:41:29.784487+0000","last_fullsized":"2024-02-05T07:41:29.784487+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'13","last_scrub_stamp":"2024-02-05T07:39:03.629988+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:39:03.629988+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1212,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.782863+0000","last_change":"2024-02-05T07:39:53.910643+0000","last_active":"2024-02-05T07:41:29.782863+0000","last_peered":"2024-02-05T07:41:29.782863+0000","last_clean":"2024-02-05T07:41:29.782863+0000","last_became_active":"2024-02-05T07:38:09.495645+0000","last_became_peered":"2024-02-05T07:38:09.495645+0000","last_unstale":"2024-02-05T07:41:29.782863+0000","last_undegraded":"2024-02-05T07:41:29.782863+0000","last_fullsized":"2024-02-05T07:41:29.782863+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:53.910541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:39:53.910541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":847,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.783026+0000","last_change":"2024-02-05T07:40:44.724483+0000","last_active":"2024-02-05T07:41:29.783026+0000","last_peered":"2024-02-05T07:41:29.783026+0000","last_clean":"2024-02-05T07:41:29.783026+0000","last_became_active":"2024-02-05T07:40:44.724105+0000","last_became_peered":"2024-02-05T07:40:44.724105+0000","last_unstale":"2024-02-05T07:41:29.783026+0000","last_undegraded":"2024-02-05T07:41:29.783026+0000","last_fullsized":"2024-02-05T07:41:29.783026+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":670,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.389984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.389984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":138,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"616'45","reported_seq":1327,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784560+0000","last_change":"2024-02-05T07:38:09.530860+0000","last_active":"2024-02-05T07:41:29.784560+0000","last_peered":"2024-02-05T07:41:29.784560+0000","last_clean":"2024-02-05T07:41:29.784560+0000","last_became_active":"2024-02-05T07:38:09.503330+0000","last_became_peered":"2024-02-05T07:38:09.503330+0000","last_unstale":"2024-02-05T07:41:29.784560+0000","last_undegraded":"2024-02-05T07:41:29.784560+0000","last_fullsized":"2024-02-05T07:41:29.784560+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'31","last_scrub_stamp":"2024-02-05T07:37:54.045831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.045831+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1183,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822755+0000","last_change":"2024-02-05T07:41:12.908537+0000","last_active":"2024-02-05T07:41:24.822755+0000","last_peered":"2024-02-05T07:41:24.822755+0000","last_clean":"2024-02-05T07:41:24.822755+0000","last_became_active":"2024-02-05T07:38:09.471049+0000","last_became_peered":"2024-02-05T07:38:09.471049+0000","last_unstale":"2024-02-05T07:41:24.822755+0000","last_undegraded":"2024-02-05T07:41:24.822755+0000","last_fullsized":"2024-02-05T07:41:24.822755+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:12.908449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:41:12.908449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:30.154544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":234,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822421+0000","last_change":"2024-02-05T07:39:59.662598+0000","last_active":"2024-02-05T07:41:24.822421+0000","last_peered":"2024-02-05T07:41:24.822421+0000","last_clean":"2024-02-05T07:41:24.822421+0000","last_became_active":"2024-02-05T07:38:09.464116+0000","last_became_peered":"2024-02-05T07:38:09.464116+0000","last_unstale":"2024-02-05T07:41:24.822421+0000","last_undegraded":"2024-02-05T07:41:24.822421+0000","last_fullsized":"2024-02-05T07:41:24.822421+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:59.662530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:59.662530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:25.420020+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":349,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784972+0000","last_change":"2024-02-05T07:38:55.897373+0000","last_active":"2024-02-05T07:41:29.784972+0000","last_peered":"2024-02-05T07:41:29.784972+0000","last_clean":"2024-02-05T07:41:29.784972+0000","last_became_active":"2024-02-05T07:38:09.482289+0000","last_became_peered":"2024-02-05T07:38:09.482289+0000","last_unstale":"2024-02-05T07:41:29.784972+0000","last_undegraded":"2024-02-05T07:41:29.784972+0000","last_fullsized":"2024-02-05T07:41:29.784972+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:55.897133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.897133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":1133,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784110+0000","last_change":"2024-02-05T07:40:33.082486+0000","last_active":"2024-02-05T07:41:29.784110+0000","last_peered":"2024-02-05T07:41:29.784110+0000","last_clean":"2024-02-05T07:41:29.784110+0000","last_became_active":"2024-02-05T07:40:33.082113+0000","last_became_peered":"2024-02-05T07:40:33.082113+0000","last_unstale":"2024-02-05T07:41:29.784110+0000","last_undegraded":"2024-02-05T07:41:29.784110+0000","last_fullsized":"2024-02-05T07:41:29.784110+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.417059+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.417059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"616'32","reported_seq":852,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823160+0000","last_change":"2024-02-05T07:38:36.705553+0000","last_active":"2024-02-05T07:41:24.823160+0000","last_peered":"2024-02-05T07:41:24.823160+0000","last_clean":"2024-02-05T07:41:24.823160+0000","last_became_active":"2024-02-05T07:38:09.471120+0000","last_became_peered":"2024-02-05T07:38:09.471120+0000","last_unstale":"2024-02-05T07:41:24.823160+0000","last_undegraded":"2024-02-05T07:41:24.823160+0000","last_fullsized":"2024-02-05T07:41:24.823160+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:38:36.705434+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:38:36.705434+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1232,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.783785+0000","last_change":"2024-02-05T07:41:11.464646+0000","last_active":"2024-02-05T07:41:29.783785+0000","last_peered":"2024-02-05T07:41:29.783785+0000","last_clean":"2024-02-05T07:41:29.783785+0000","last_became_active":"2024-02-05T07:38:09.489807+0000","last_became_peered":"2024-02-05T07:38:09.489807+0000","last_unstale":"2024-02-05T07:41:29.783785+0000","last_undegraded":"2024-02-05T07:41:29.783785+0000","last_fullsized":"2024-02-05T07:41:29.783785+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:11.464549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:41:11.464549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:33.050668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1058,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823438+0000","last_change":"2024-02-05T07:38:37.703938+0000","last_active":"2024-02-05T07:41:24.823438+0000","last_peered":"2024-02-05T07:41:24.823438+0000","last_clean":"2024-02-05T07:41:24.823438+0000","last_became_active":"2024-02-05T07:38:09.472546+0000","last_became_peered":"2024-02-05T07:38:09.472546+0000","last_unstale":"2024-02-05T07:41:24.823438+0000","last_undegraded":"2024-02-05T07:41:24.823438+0000","last_fullsized":"2024-02-05T07:41:24.823438+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:37.703824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:45.012821+0000","last_clean_scrub_stamp":"2024-02-05T07:38:37.703824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":906,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824310+0000","last_change":"2024-02-05T07:37:52.481462+0000","last_active":"2024-02-05T07:41:24.824310+0000","last_peered":"2024-02-05T07:41:24.824310+0000","last_clean":"2024-02-05T07:41:24.824310+0000","last_became_active":"2024-02-05T07:30:40.498529+0000","last_became_peered":"2024-02-05T07:30:40.498529+0000","last_unstale":"2024-02-05T07:41:24.824310+0000","last_undegraded":"2024-02-05T07:41:24.824310+0000","last_fullsized":"2024-02-05T07:41:24.824310+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:52.481255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:52.481255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":730,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784839+0000","last_change":"2024-02-05T07:40:36.746220+0000","last_active":"2024-02-05T07:41:29.784839+0000","last_peered":"2024-02-05T07:41:29.784839+0000","last_clean":"2024-02-05T07:41:29.784839+0000","last_became_active":"2024-02-05T07:40:36.745864+0000","last_became_peered":"2024-02-05T07:40:36.745864+0000","last_unstale":"2024-02-05T07:41:29.784839+0000","last_undegraded":"2024-02-05T07:41:29.784839+0000","last_fullsized":"2024-02-05T07:41:29.784839+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:40.411931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:40.411931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"616'46","reported_seq":1457,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.783385+0000","last_change":"2024-02-05T07:38:52.880432+0000","last_active":"2024-02-05T07:41:29.783385+0000","last_peered":"2024-02-05T07:41:29.783385+0000","last_clean":"2024-02-05T07:41:29.783385+0000","last_became_active":"2024-02-05T07:38:09.500221+0000","last_became_peered":"2024-02-05T07:38:09.500221+0000","last_unstale":"2024-02-05T07:41:29.783385+0000","last_undegraded":"2024-02-05T07:41:29.783385+0000","last_fullsized":"2024-02-05T07:41:29.783385+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'34","last_scrub_stamp":"2024-02-05T07:38:52.880291+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:38:52.880291+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:35.527553+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":909,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823647+0000","last_change":"2024-02-05T07:38:09.485474+0000","last_active":"2024-02-05T07:41:24.823647+0000","last_peered":"2024-02-05T07:41:24.823647+0000","last_clean":"2024-02-05T07:41:24.823647+0000","last_became_active":"2024-02-05T07:38:09.485164+0000","last_became_peered":"2024-02-05T07:38:09.485164+0000","last_unstale":"2024-02-05T07:41:24.823647+0000","last_undegraded":"2024-02-05T07:41:24.823647+0000","last_fullsized":"2024-02-05T07:41:24.823647+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:47.116727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:47.116727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"589'12","reported_seq":1497,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784807+0000","last_change":"2024-02-05T07:41:15.416693+0000","last_active":"2024-02-05T07:41:29.784807+0000","last_peered":"2024-02-05T07:41:29.784807+0000","last_clean":"2024-02-05T07:41:29.784807+0000","last_became_active":"2024-02-05T07:38:09.488856+0000","last_became_peered":"2024-02-05T07:38:09.488856+0000","last_unstale":"2024-02-05T07:41:29.784807+0000","last_undegraded":"2024-02-05T07:41:29.784807+0000","last_fullsized":"2024-02-05T07:41:29.784807+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"589'12","last_scrub_stamp":"2024-02-05T07:41:15.416604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:41:15.416604+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:22.921263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":343,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784437+0000","last_change":"2024-02-05T07:39:50.551862+0000","last_active":"2024-02-05T07:41:29.784437+0000","last_peered":"2024-02-05T07:41:29.784437+0000","last_clean":"2024-02-05T07:41:29.784437+0000","last_became_active":"2024-02-05T07:36:32.744070+0000","last_became_peered":"2024-02-05T07:36:32.744070+0000","last_unstale":"2024-02-05T07:41:29.784437+0000","last_undegraded":"2024-02-05T07:41:29.784437+0000","last_fullsized":"2024-02-05T07:41:29.784437+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:50.551740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:50.551740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":1323,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784308+0000","last_change":"2024-02-05T07:40:56.744734+0000","last_active":"2024-02-05T07:41:29.784308+0000","last_peered":"2024-02-05T07:41:29.784308+0000","last_clean":"2024-02-05T07:41:29.784308+0000","last_became_active":"2024-02-05T07:40:56.744007+0000","last_became_peered":"2024-02-05T07:40:56.744007+0000","last_unstale":"2024-02-05T07:41:29.784308+0000","last_undegraded":"2024-02-05T07:41:29.784308+0000","last_fullsized":"2024-02-05T07:41:29.784308+0000","mapping_epoch":593,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:34.509015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:34.509015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":861,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824009+0000","last_change":"2024-02-05T07:38:09.479255+0000","last_active":"2024-02-05T07:41:24.824009+0000","last_peered":"2024-02-05T07:41:24.824009+0000","last_clean":"2024-02-05T07:41:24.824009+0000","last_became_active":"2024-02-05T07:38:09.478485+0000","last_became_peered":"2024-02-05T07:38:09.478485+0000","last_unstale":"2024-02-05T07:41:24.824009+0000","last_undegraded":"2024-02-05T07:41:24.824009+0000","last_fullsized":"2024-02-05T07:41:24.824009+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.155552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.155552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"615'50","reported_seq":992,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823558+0000","last_change":"2024-02-05T07:39:48.737503+0000","last_active":"2024-02-05T07:41:24.823558+0000","last_peered":"2024-02-05T07:41:24.823558+0000","last_clean":"2024-02-05T07:41:24.823558+0000","last_became_active":"2024-02-05T07:30:40.496940+0000","last_became_peered":"2024-02-05T07:30:40.496940+0000","last_unstale":"2024-02-05T07:41:24.823558+0000","last_undegraded":"2024-02-05T07:41:24.823558+0000","last_fullsized":"2024-02-05T07:41:24.823558+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'50","last_scrub_stamp":"2024-02-05T07:39:48.737259+0000","last_deep_scrub":"454'38","last_deep_scrub_stamp":"2024-02-05T07:34:38.942830+0000","last_clean_scrub_stamp":"2024-02-05T07:39:48.737259+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":909,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825880+0000","last_change":"2024-02-05T07:38:57.455422+0000","last_active":"2024-02-05T07:41:24.825880+0000","last_peered":"2024-02-05T07:41:24.825880+0000","last_clean":"2024-02-05T07:41:24.825880+0000","last_became_active":"2024-02-05T07:38:09.488092+0000","last_became_peered":"2024-02-05T07:38:09.488092+0000","last_unstale":"2024-02-05T07:41:24.825880+0000","last_undegraded":"2024-02-05T07:41:24.825880+0000","last_fullsized":"2024-02-05T07:41:24.825880+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:57.455281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:57.455281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1219,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824128+0000","last_change":"2024-02-05T07:38:09.480292+0000","last_active":"2024-02-05T07:41:24.824128+0000","last_peered":"2024-02-05T07:41:24.824128+0000","last_clean":"2024-02-05T07:41:24.824128+0000","last_became_active":"2024-02-05T07:38:09.479403+0000","last_became_peered":"2024-02-05T07:38:09.479403+0000","last_unstale":"2024-02-05T07:41:24.824128+0000","last_undegraded":"2024-02-05T07:41:24.824128+0000","last_fullsized":"2024-02-05T07:41:24.824128+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:01.252507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:38:01.252507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":1676,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824312+0000","last_change":"2024-02-05T07:38:11.676831+0000","last_active":"2024-02-05T07:41:24.824312+0000","last_peered":"2024-02-05T07:41:24.824312+0000","last_clean":"2024-02-05T07:41:24.824312+0000","last_became_active":"2024-02-05T07:38:11.676482+0000","last_became_peered":"2024-02-05T07:38:11.676482+0000","last_unstale":"2024-02-05T07:41:24.824312+0000","last_undegraded":"2024-02-05T07:41:24.824312+0000","last_fullsized":"2024-02-05T07:41:24.824312+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:07.353770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:07.353770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"589'11","reported_seq":1083,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823652+0000","last_change":"2024-02-05T07:38:09.522356+0000","last_active":"2024-02-05T07:41:24.823652+0000","last_peered":"2024-02-05T07:41:24.823652+0000","last_clean":"2024-02-05T07:41:24.823652+0000","last_became_active":"2024-02-05T07:38:09.495697+0000","last_became_peered":"2024-02-05T07:38:09.495697+0000","last_unstale":"2024-02-05T07:41:24.823652+0000","last_undegraded":"2024-02-05T07:41:24.823652+0000","last_fullsized":"2024-02-05T07:41:24.823652+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'10","last_scrub_stamp":"2024-02-05T07:37:59.362892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:59.362892+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"616'49","reported_seq":890,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.821990+0000","last_change":"2024-02-05T07:39:55.713122+0000","last_active":"2024-02-05T07:41:24.821990+0000","last_peered":"2024-02-05T07:41:24.821990+0000","last_clean":"2024-02-05T07:41:24.821990+0000","last_became_active":"2024-02-05T07:38:09.480688+0000","last_became_peered":"2024-02-05T07:38:09.480688+0000","last_unstale":"2024-02-05T07:41:24.821990+0000","last_undegraded":"2024-02-05T07:41:24.821990+0000","last_fullsized":"2024-02-05T07:41:24.821990+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'49","last_scrub_stamp":"2024-02-05T07:39:55.713036+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:39:55.713036+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"639'2","reported_seq":1098,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.782769+0000","last_change":"2024-02-05T07:40:13.213597+0000","last_active":"2024-02-05T07:41:29.782769+0000","last_peered":"2024-02-05T07:41:29.782769+0000","last_clean":"2024-02-05T07:41:29.782769+0000","last_became_active":"2024-02-05T07:38:09.467535+0000","last_became_peered":"2024-02-05T07:38:09.467535+0000","last_unstale":"2024-02-05T07:41:29.782769+0000","last_undegraded":"2024-02-05T07:41:29.782769+0000","last_fullsized":"2024-02-05T07:41:29.782769+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'2","last_scrub_stamp":"2024-02-05T07:40:13.213477+0000","last_deep_scrub":"639'2","last_deep_scrub_stamp":"2024-02-05T07:40:13.213477+0000","last_clean_scrub_stamp":"2024-02-05T07:40:13.213477+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1201,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822230+0000","last_change":"2024-02-05T07:37:58.737127+0000","last_active":"2024-02-05T07:41:24.822230+0000","last_peered":"2024-02-05T07:41:24.822230+0000","last_clean":"2024-02-05T07:41:24.822230+0000","last_became_active":"2024-02-05T07:36:32.701203+0000","last_became_peered":"2024-02-05T07:36:32.701203+0000","last_unstale":"2024-02-05T07:41:24.822230+0000","last_undegraded":"2024-02-05T07:41:24.822230+0000","last_fullsized":"2024-02-05T07:41:24.822230+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:58.737033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:58.737033+0000","last_clean_scrub_stamp":"2024-02-05T07:37:58.737033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":1319,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823567+0000","last_change":"2024-02-05T07:41:10.927278+0000","last_active":"2024-02-05T07:41:24.823567+0000","last_peered":"2024-02-05T07:41:24.823567+0000","last_clean":"2024-02-05T07:41:24.823567+0000","last_became_active":"2024-02-05T07:40:48.865775+0000","last_became_peered":"2024-02-05T07:40:48.865775+0000","last_unstale":"2024-02-05T07:41:24.823567+0000","last_undegraded":"2024-02-05T07:41:24.823567+0000","last_fullsized":"2024-02-05T07:41:24.823567+0000","mapping_epoch":538,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:41:10.927166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:41:10.927166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:20.488593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"303'15","reported_seq":1080,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822350+0000","last_change":"2024-02-05T07:39:44.702368+0000","last_active":"2024-02-05T07:41:24.822350+0000","last_peered":"2024-02-05T07:41:24.822350+0000","last_clean":"2024-02-05T07:41:24.822350+0000","last_became_active":"2024-02-05T07:28:26.596623+0000","last_became_peered":"2024-02-05T07:28:26.596623+0000","last_unstale":"2024-02-05T07:41:24.822350+0000","last_undegraded":"2024-02-05T07:41:24.822350+0000","last_fullsized":"2024-02-05T07:41:24.822350+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-02-05T07:39:44.702291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:44.702291+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"616'35","reported_seq":1417,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.782124+0000","last_change":"2024-02-05T07:38:50.868322+0000","last_active":"2024-02-05T07:41:29.782124+0000","last_peered":"2024-02-05T07:41:29.782124+0000","last_clean":"2024-02-05T07:41:29.782124+0000","last_became_active":"2024-02-05T07:38:09.496732+0000","last_became_peered":"2024-02-05T07:38:09.496732+0000","last_unstale":"2024-02-05T07:41:29.782124+0000","last_undegraded":"2024-02-05T07:41:29.782124+0000","last_fullsized":"2024-02-05T07:41:29.782124+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:38:50.868179+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:38:50.868179+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":912,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825478+0000","last_change":"2024-02-05T07:36:32.741872+0000","last_active":"2024-02-05T07:41:24.825478+0000","last_peered":"2024-02-05T07:41:24.825478+0000","last_clean":"2024-02-05T07:41:24.825478+0000","last_became_active":"2024-02-05T07:36:32.741377+0000","last_became_peered":"2024-02-05T07:36:32.741377+0000","last_unstale":"2024-02-05T07:41:24.825478+0000","last_undegraded":"2024-02-05T07:41:24.825478+0000","last_fullsized":"2024-02-05T07:41:24.825478+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:35:17.067204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:35:17.067204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":607,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.780596+0000","last_change":"2024-02-05T07:39:54.880406+0000","last_active":"2024-02-05T07:41:29.780596+0000","last_peered":"2024-02-05T07:41:29.780596+0000","last_clean":"2024-02-05T07:41:29.780596+0000","last_became_active":"2024-02-05T07:32:58.598709+0000","last_became_peered":"2024-02-05T07:32:58.598709+0000","last_unstale":"2024-02-05T07:41:29.780596+0000","last_undegraded":"2024-02-05T07:41:29.780596+0000","last_fullsized":"2024-02-05T07:41:29.780596+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:54.879841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:54.879841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":1291,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823054+0000","last_change":"2024-02-05T07:41:13.923027+0000","last_active":"2024-02-05T07:41:24.823054+0000","last_peered":"2024-02-05T07:41:24.823054+0000","last_clean":"2024-02-05T07:41:24.823054+0000","last_became_active":"2024-02-05T07:40:52.907985+0000","last_became_peered":"2024-02-05T07:40:52.907985+0000","last_unstale":"2024-02-05T07:41:24.823054+0000","last_undegraded":"2024-02-05T07:41:24.823054+0000","last_fullsized":"2024-02-05T07:41:24.823054+0000","mapping_epoch":302,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:41:13.922831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:41:13.922831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:30.989266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1176,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822331+0000","last_change":"2024-02-05T07:40:11.774832+0000","last_active":"2024-02-05T07:41:24.822331+0000","last_peered":"2024-02-05T07:41:24.822331+0000","last_clean":"2024-02-05T07:41:24.822331+0000","last_became_active":"2024-02-05T07:36:32.709025+0000","last_became_peered":"2024-02-05T07:36:32.709025+0000","last_unstale":"2024-02-05T07:41:24.822331+0000","last_undegraded":"2024-02-05T07:41:24.822331+0000","last_fullsized":"2024-02-05T07:41:24.822331+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:11.774744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:40:11.774744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:36.409236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"615'39","reported_seq":1024,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.781754+0000","last_change":"2024-02-05T07:41:11.504402+0000","last_active":"2024-02-05T07:41:29.781754+0000","last_peered":"2024-02-05T07:41:29.781754+0000","last_clean":"2024-02-05T07:41:29.781754+0000","last_became_active":"2024-02-05T07:36:32.712346+0000","last_became_peered":"2024-02-05T07:36:32.712346+0000","last_unstale":"2024-02-05T07:41:29.781754+0000","last_undegraded":"2024-02-05T07:41:29.781754+0000","last_fullsized":"2024-02-05T07:41:29.781754+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'39","last_scrub_stamp":"2024-02-05T07:41:11.504301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:41:11.504301+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:26.385778+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":351,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784584+0000","last_change":"2024-02-05T07:41:04.555918+0000","last_active":"2024-02-05T07:41:29.784584+0000","last_peered":"2024-02-05T07:41:29.784584+0000","last_clean":"2024-02-05T07:41:29.784584+0000","last_became_active":"2024-02-05T07:36:32.732781+0000","last_became_peered":"2024-02-05T07:36:32.732781+0000","last_unstale":"2024-02-05T07:41:29.784584+0000","last_undegraded":"2024-02-05T07:41:29.784584+0000","last_fullsized":"2024-02-05T07:41:29.784584+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:04.555788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:41:04.555788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:15.526481+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1336,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.781018+0000","last_change":"2024-02-05T07:40:09.083562+0000","last_active":"2024-02-05T07:41:29.781018+0000","last_peered":"2024-02-05T07:41:29.781018+0000","last_clean":"2024-02-05T07:41:29.781018+0000","last_became_active":"2024-02-05T07:36:32.709872+0000","last_became_peered":"2024-02-05T07:36:32.709872+0000","last_unstale":"2024-02-05T07:41:29.781018+0000","last_undegraded":"2024-02-05T07:41:29.781018+0000","last_fullsized":"2024-02-05T07:41:29.781018+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:09.083452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:40:09.083452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":933,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822423+0000","last_change":"2024-02-05T07:38:11.676690+0000","last_active":"2024-02-05T07:41:24.822423+0000","last_peered":"2024-02-05T07:41:24.822423+0000","last_clean":"2024-02-05T07:41:24.822423+0000","last_became_active":"2024-02-05T07:38:11.676446+0000","last_became_peered":"2024-02-05T07:38:11.676446+0000","last_unstale":"2024-02-05T07:41:24.822423+0000","last_undegraded":"2024-02-05T07:41:24.822423+0000","last_fullsized":"2024-02-05T07:41:24.822423+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.778695+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.778695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":930,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.783935+0000","last_change":"2024-02-05T07:39:51.915848+0000","last_active":"2024-02-05T07:41:29.783935+0000","last_peered":"2024-02-05T07:41:29.783935+0000","last_clean":"2024-02-05T07:41:29.783935+0000","last_became_active":"2024-02-05T07:38:09.496845+0000","last_became_peered":"2024-02-05T07:38:09.496845+0000","last_unstale":"2024-02-05T07:41:29.783935+0000","last_undegraded":"2024-02-05T07:41:29.783935+0000","last_fullsized":"2024-02-05T07:41:29.783935+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:51.915741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:51.915741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"615'36","reported_seq":375,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.785784+0000","last_change":"2024-02-05T07:38:54.585085+0000","last_active":"2024-02-05T07:41:29.785784+0000","last_peered":"2024-02-05T07:41:29.785784+0000","last_clean":"2024-02-05T07:41:29.785784+0000","last_became_active":"2024-02-05T07:36:32.753752+0000","last_became_peered":"2024-02-05T07:36:32.753752+0000","last_unstale":"2024-02-05T07:41:29.785784+0000","last_undegraded":"2024-02-05T07:41:29.785784+0000","last_fullsized":"2024-02-05T07:41:29.785784+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'24","last_scrub_stamp":"2024-02-05T07:38:54.584948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:54.584948+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"695'6","reported_seq":1399,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.780920+0000","last_change":"2024-02-05T07:41:22.536451+0000","last_active":"2024-02-05T07:41:29.780920+0000","last_peered":"2024-02-05T07:41:29.780920+0000","last_clean":"2024-02-05T07:41:29.780920+0000","last_became_active":"2024-02-05T07:30:40.493770+0000","last_became_peered":"2024-02-05T07:30:40.493770+0000","last_unstale":"2024-02-05T07:41:29.780920+0000","last_undegraded":"2024-02-05T07:41:29.780920+0000","last_fullsized":"2024-02-05T07:41:29.780920+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'6","last_scrub_stamp":"2024-02-05T07:41:22.536369+0000","last_deep_scrub":"695'6","last_deep_scrub_stamp":"2024-02-05T07:41:22.536369+0000","last_clean_scrub_stamp":"2024-02-05T07:41:22.536369+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:41.402932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":654,"num_read_kb":649,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":628,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824187+0000","last_change":"2024-02-05T07:38:41.375038+0000","last_active":"2024-02-05T07:41:24.824187+0000","last_peered":"2024-02-05T07:41:24.824187+0000","last_clean":"2024-02-05T07:41:24.824187+0000","last_became_active":"2024-02-05T07:38:09.493827+0000","last_became_peered":"2024-02-05T07:38:09.493827+0000","last_unstale":"2024-02-05T07:41:24.824187+0000","last_undegraded":"2024-02-05T07:41:24.824187+0000","last_fullsized":"2024-02-05T07:41:24.824187+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:41.374804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:48.775665+0000","last_clean_scrub_stamp":"2024-02-05T07:38:41.374804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":1352,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.783551+0000","last_change":"2024-02-05T07:39:49.884158+0000","last_active":"2024-02-05T07:41:29.783551+0000","last_peered":"2024-02-05T07:41:29.783551+0000","last_clean":"2024-02-05T07:41:29.783551+0000","last_became_active":"2024-02-05T07:28:26.598049+0000","last_became_peered":"2024-02-05T07:28:26.598049+0000","last_unstale":"2024-02-05T07:41:29.783551+0000","last_undegraded":"2024-02-05T07:41:29.783551+0000","last_fullsized":"2024-02-05T07:41:29.783551+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:39:49.883906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:49.883906+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":343,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784908+0000","last_change":"2024-02-05T07:38:43.594222+0000","last_active":"2024-02-05T07:41:29.784908+0000","last_peered":"2024-02-05T07:41:29.784908+0000","last_clean":"2024-02-05T07:41:29.784908+0000","last_became_active":"2024-02-05T07:36:32.732334+0000","last_became_peered":"2024-02-05T07:36:32.732334+0000","last_unstale":"2024-02-05T07:41:29.784908+0000","last_undegraded":"2024-02-05T07:41:29.784908+0000","last_fullsized":"2024-02-05T07:41:29.784908+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:43.593810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:43.593810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"616'44","reported_seq":1519,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823251+0000","last_change":"2024-02-05T07:38:09.529174+0000","last_active":"2024-02-05T07:41:24.823251+0000","last_peered":"2024-02-05T07:41:24.823251+0000","last_clean":"2024-02-05T07:41:24.823251+0000","last_became_active":"2024-02-05T07:38:09.479528+0000","last_became_peered":"2024-02-05T07:38:09.479528+0000","last_unstale":"2024-02-05T07:41:24.823251+0000","last_undegraded":"2024-02-05T07:41:24.823251+0000","last_fullsized":"2024-02-05T07:41:24.823251+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'31","last_scrub_stamp":"2024-02-05T07:37:45.644963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.644963+0000","objects_scrubbed":8,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":208,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.783893+0000","last_change":"2024-02-05T07:40:08.312293+0000","last_active":"2024-02-05T07:41:29.783893+0000","last_peered":"2024-02-05T07:41:29.783893+0000","last_clean":"2024-02-05T07:41:29.783893+0000","last_became_active":"2024-02-05T07:38:31.807354+0000","last_became_peered":"2024-02-05T07:38:31.807354+0000","last_unstale":"2024-02-05T07:41:29.783893+0000","last_undegraded":"2024-02-05T07:41:29.783893+0000","last_fullsized":"2024-02-05T07:41:29.783893+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:08.312041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:40:08.312041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:35.861633+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1103,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822104+0000","last_change":"2024-02-05T07:38:29.812121+0000","last_active":"2024-02-05T07:41:24.822104+0000","last_peered":"2024-02-05T07:41:24.822104+0000","last_clean":"2024-02-05T07:41:24.822104+0000","last_became_active":"2024-02-05T07:37:06.977158+0000","last_became_peered":"2024-02-05T07:37:06.977158+0000","last_unstale":"2024-02-05T07:41:24.822104+0000","last_undegraded":"2024-02-05T07:41:24.822104+0000","last_fullsized":"2024-02-05T07:41:24.822104+0000","mapping_epoch":367,"log_start":"86'128","ondisk_log_start":"86'128","created":20,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:29.812050+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:38:29.812050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":903,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825755+0000","last_change":"2024-02-05T07:38:49.589976+0000","last_active":"2024-02-05T07:41:24.825755+0000","last_peered":"2024-02-05T07:41:24.825755+0000","last_clean":"2024-02-05T07:41:24.825755+0000","last_became_active":"2024-02-05T07:38:09.488951+0000","last_became_peered":"2024-02-05T07:38:09.488951+0000","last_unstale":"2024-02-05T07:41:24.825755+0000","last_undegraded":"2024-02-05T07:41:24.825755+0000","last_fullsized":"2024-02-05T07:41:24.825755+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:49.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:49.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":1326,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784766+0000","last_change":"2024-02-05T07:38:32.532424+0000","last_active":"2024-02-05T07:41:29.784766+0000","last_peered":"2024-02-05T07:41:29.784766+0000","last_clean":"2024-02-05T07:41:29.784766+0000","last_became_active":"2024-02-05T07:38:11.674897+0000","last_became_peered":"2024-02-05T07:38:11.674897+0000","last_unstale":"2024-02-05T07:41:29.784766+0000","last_undegraded":"2024-02-05T07:41:29.784766+0000","last_fullsized":"2024-02-05T07:41:29.784766+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:32.532304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:32.532304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":349,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.785771+0000","last_change":"2024-02-05T07:38:35.498140+0000","last_active":"2024-02-05T07:41:29.785771+0000","last_peered":"2024-02-05T07:41:29.785771+0000","last_clean":"2024-02-05T07:41:29.785771+0000","last_became_active":"2024-02-05T07:38:09.500885+0000","last_became_peered":"2024-02-05T07:38:09.500885+0000","last_unstale":"2024-02-05T07:41:29.785771+0000","last_undegraded":"2024-02-05T07:41:29.785771+0000","last_fullsized":"2024-02-05T07:41:29.785771+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:35.497879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:35.497879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"615'26","reported_seq":296,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784615+0000","last_change":"2024-02-05T07:38:39.505608+0000","last_active":"2024-02-05T07:41:29.784615+0000","last_peered":"2024-02-05T07:41:29.784615+0000","last_clean":"2024-02-05T07:41:29.784615+0000","last_became_active":"2024-02-05T07:37:31.671370+0000","last_became_peered":"2024-02-05T07:37:31.671370+0000","last_unstale":"2024-02-05T07:41:29.784615+0000","last_undegraded":"2024-02-05T07:41:29.784615+0000","last_fullsized":"2024-02-05T07:41:29.784615+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'22","last_scrub_stamp":"2024-02-05T07:38:39.505230+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:38:39.505230+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":343,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784015+0000","last_change":"2024-02-05T07:38:45.513185+0000","last_active":"2024-02-05T07:41:29.784015+0000","last_peered":"2024-02-05T07:41:29.784015+0000","last_clean":"2024-02-05T07:41:29.784015+0000","last_became_active":"2024-02-05T07:36:32.733822+0000","last_became_peered":"2024-02-05T07:36:32.733822+0000","last_unstale":"2024-02-05T07:41:29.784015+0000","last_undegraded":"2024-02-05T07:41:29.784015+0000","last_fullsized":"2024-02-05T07:41:29.784015+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:45.512912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:38:45.512912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":347,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.782929+0000","last_change":"2024-02-05T07:38:57.649523+0000","last_active":"2024-02-05T07:41:29.782929+0000","last_peered":"2024-02-05T07:41:29.782929+0000","last_clean":"2024-02-05T07:41:29.782929+0000","last_became_active":"2024-02-05T07:36:32.781195+0000","last_became_peered":"2024-02-05T07:36:32.781195+0000","last_unstale":"2024-02-05T07:41:29.782929+0000","last_undegraded":"2024-02-05T07:41:29.782929+0000","last_fullsized":"2024-02-05T07:41:29.782929+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:38:57.649351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:57.649351+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1213,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823060+0000","last_change":"2024-02-05T07:38:51.640312+0000","last_active":"2024-02-05T07:41:24.823060+0000","last_peered":"2024-02-05T07:41:24.823060+0000","last_clean":"2024-02-05T07:41:24.823060+0000","last_became_active":"2024-02-05T07:36:32.718684+0000","last_became_peered":"2024-02-05T07:36:32.718684+0000","last_unstale":"2024-02-05T07:41:24.823060+0000","last_undegraded":"2024-02-05T07:41:24.823060+0000","last_fullsized":"2024-02-05T07:41:24.823060+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:51.640224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:06.673081+0000","last_clean_scrub_stamp":"2024-02-05T07:38:51.640224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":722,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.783907+0000","last_change":"2024-02-05T07:41:27.411867+0000","last_active":"2024-02-05T07:41:29.783907+0000","last_peered":"2024-02-05T07:41:29.783907+0000","last_clean":"2024-02-05T07:41:29.783907+0000","last_became_active":"2024-02-05T07:38:15.642779+0000","last_became_peered":"2024-02-05T07:38:15.642779+0000","last_unstale":"2024-02-05T07:41:29.783907+0000","last_undegraded":"2024-02-05T07:41:29.783907+0000","last_fullsized":"2024-02-05T07:41:29.783907+0000","mapping_epoch":595,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:41:27.411780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:41:27.411780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:42.019739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1086,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.781698+0000","last_change":"2024-02-05T07:40:19.104414+0000","last_active":"2024-02-05T07:41:29.781698+0000","last_peered":"2024-02-05T07:41:29.781698+0000","last_clean":"2024-02-05T07:41:29.781698+0000","last_became_active":"2024-02-05T07:38:09.463763+0000","last_became_peered":"2024-02-05T07:38:09.463763+0000","last_unstale":"2024-02-05T07:41:29.781698+0000","last_undegraded":"2024-02-05T07:41:29.781698+0000","last_fullsized":"2024-02-05T07:41:29.781698+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:19.104295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:40:19.104295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:47.076546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"616'68","reported_seq":461,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784725+0000","last_change":"2024-02-05T07:41:24.429246+0000","last_active":"2024-02-05T07:41:29.784725+0000","last_peered":"2024-02-05T07:41:29.784725+0000","last_clean":"2024-02-05T07:41:29.784725+0000","last_became_active":"2024-02-05T07:36:32.738270+0000","last_became_peered":"2024-02-05T07:36:32.738270+0000","last_unstale":"2024-02-05T07:41:29.784725+0000","last_undegraded":"2024-02-05T07:41:29.784725+0000","last_fullsized":"2024-02-05T07:41:29.784725+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'68","last_scrub_stamp":"2024-02-05T07:41:24.429102+0000","last_deep_scrub":"550'56","last_deep_scrub_stamp":"2024-02-05T07:36:56.264300+0000","last_clean_scrub_stamp":"2024-02-05T07:41:24.429102+0000","objects_scrubbed":10,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:36.932172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":82,"num_read_kb":67,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":741,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822816+0000","last_change":"2024-02-05T07:40:15.743056+0000","last_active":"2024-02-05T07:41:24.822816+0000","last_peered":"2024-02-05T07:41:24.822816+0000","last_clean":"2024-02-05T07:41:24.822816+0000","last_became_active":"2024-02-05T07:38:09.484410+0000","last_became_peered":"2024-02-05T07:38:09.484410+0000","last_unstale":"2024-02-05T07:41:24.822816+0000","last_undegraded":"2024-02-05T07:41:24.822816+0000","last_fullsized":"2024-02-05T07:41:24.822816+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:15.742953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:40:15.742953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":1249,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.781685+0000","last_change":"2024-02-05T07:40:12.141885+0000","last_active":"2024-02-05T07:41:29.781685+0000","last_peered":"2024-02-05T07:41:29.781685+0000","last_clean":"2024-02-05T07:41:29.781685+0000","last_became_active":"2024-02-05T07:28:26.590442+0000","last_became_peered":"2024-02-05T07:28:26.590442+0000","last_unstale":"2024-02-05T07:41:29.781685+0000","last_undegraded":"2024-02-05T07:41:29.781685+0000","last_fullsized":"2024-02-05T07:41:29.781685+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:40:12.141744+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:40:12.141744+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":917,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825294+0000","last_change":"2024-02-05T07:39:03.439286+0000","last_active":"2024-02-05T07:41:24.825294+0000","last_peered":"2024-02-05T07:41:24.825294+0000","last_clean":"2024-02-05T07:41:24.825294+0000","last_became_active":"2024-02-05T07:38:09.486846+0000","last_became_peered":"2024-02-05T07:38:09.486846+0000","last_unstale":"2024-02-05T07:41:24.825294+0000","last_undegraded":"2024-02-05T07:41:24.825294+0000","last_fullsized":"2024-02-05T07:41:24.825294+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:03.439161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:03.439161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":674,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825586+0000","last_change":"2024-02-05T07:40:20.876587+0000","last_active":"2024-02-05T07:41:24.825586+0000","last_peered":"2024-02-05T07:41:24.825586+0000","last_clean":"2024-02-05T07:41:24.825586+0000","last_became_active":"2024-02-05T07:40:20.876256+0000","last_became_peered":"2024-02-05T07:40:20.876256+0000","last_unstale":"2024-02-05T07:41:24.825586+0000","last_undegraded":"2024-02-05T07:41:24.825586+0000","last_fullsized":"2024-02-05T07:41:24.825586+0000","mapping_epoch":619,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:39:45.710637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:39:45.710637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"615'21","reported_seq":1139,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.781562+0000","last_change":"2024-02-05T07:38:35.790500+0000","last_active":"2024-02-05T07:41:29.781562+0000","last_peered":"2024-02-05T07:41:29.781562+0000","last_clean":"2024-02-05T07:41:29.781562+0000","last_became_active":"2024-02-05T07:38:09.479890+0000","last_became_peered":"2024-02-05T07:38:09.479890+0000","last_unstale":"2024-02-05T07:41:29.781562+0000","last_undegraded":"2024-02-05T07:41:29.781562+0000","last_fullsized":"2024-02-05T07:41:29.781562+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'15","last_scrub_stamp":"2024-02-05T07:38:35.790379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:35.790379+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":900,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825352+0000","last_change":"2024-02-05T07:36:54.146303+0000","last_active":"2024-02-05T07:41:24.825352+0000","last_peered":"2024-02-05T07:41:24.825352+0000","last_clean":"2024-02-05T07:41:24.825352+0000","last_became_active":"2024-02-05T07:28:26.602532+0000","last_became_peered":"2024-02-05T07:28:26.602532+0000","last_unstale":"2024-02-05T07:41:24.825352+0000","last_undegraded":"2024-02-05T07:41:24.825352+0000","last_fullsized":"2024-02-05T07:41:24.825352+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:54.146194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:54.146194+0000","last_clean_scrub_stamp":"2024-02-05T07:36:54.146194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":349,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784429+0000","last_change":"2024-02-05T07:38:09.492094+0000","last_active":"2024-02-05T07:41:29.784429+0000","last_peered":"2024-02-05T07:41:29.784429+0000","last_clean":"2024-02-05T07:41:29.784429+0000","last_became_active":"2024-02-05T07:38:09.491746+0000","last_became_peered":"2024-02-05T07:38:09.491746+0000","last_unstale":"2024-02-05T07:41:29.784429+0000","last_undegraded":"2024-02-05T07:41:29.784429+0000","last_fullsized":"2024-02-05T07:41:29.784429+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:45.362685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.362685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":351,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.783824+0000","last_change":"2024-02-05T07:41:25.451481+0000","last_active":"2024-02-05T07:41:29.783824+0000","last_peered":"2024-02-05T07:41:29.783824+0000","last_clean":"2024-02-05T07:41:29.783824+0000","last_became_active":"2024-02-05T07:36:32.734230+0000","last_became_peered":"2024-02-05T07:36:32.734230+0000","last_unstale":"2024-02-05T07:41:29.783824+0000","last_undegraded":"2024-02-05T07:41:29.783824+0000","last_fullsized":"2024-02-05T07:41:29.783824+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:25.451396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:41:25.451396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:44.745780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":1332,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:27.806677+0000","last_change":"2024-02-05T07:41:27.806704+0000","last_active":"2024-02-05T07:41:27.806677+0000","last_peered":"2024-02-05T07:41:27.806677+0000","last_clean":"2024-02-05T07:41:27.806677+0000","last_became_active":"2024-02-05T07:38:11.660622+0000","last_became_peered":"2024-02-05T07:38:11.660622+0000","last_unstale":"2024-02-05T07:41:27.806677+0000","last_undegraded":"2024-02-05T07:41:27.806677+0000","last_fullsized":"2024-02-05T07:41:27.806677+0000","mapping_epoch":591,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:41:27.806590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:41:27.806590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:34.873184+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"616'51","reported_seq":1289,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822572+0000","last_change":"2024-02-05T07:38:52.517441+0000","last_active":"2024-02-05T07:41:24.822572+0000","last_peered":"2024-02-05T07:41:24.822572+0000","last_clean":"2024-02-05T07:41:24.822572+0000","last_became_active":"2024-02-05T07:36:16.736226+0000","last_became_peered":"2024-02-05T07:36:16.736226+0000","last_unstale":"2024-02-05T07:41:24.822572+0000","last_undegraded":"2024-02-05T07:41:24.822572+0000","last_fullsized":"2024-02-05T07:41:24.822572+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'35","last_scrub_stamp":"2024-02-05T07:38:52.517337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:52.517337+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":208,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784165+0000","last_change":"2024-02-05T07:40:03.305634+0000","last_active":"2024-02-05T07:41:29.784165+0000","last_peered":"2024-02-05T07:41:29.784165+0000","last_clean":"2024-02-05T07:41:29.784165+0000","last_became_active":"2024-02-05T07:38:31.795149+0000","last_became_peered":"2024-02-05T07:38:31.795149+0000","last_unstale":"2024-02-05T07:41:29.784165+0000","last_undegraded":"2024-02-05T07:41:29.784165+0000","last_fullsized":"2024-02-05T07:41:29.784165+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:03.305531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_clean_scrub_stamp":"2024-02-05T07:40:03.305531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":343,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784501+0000","last_change":"2024-02-05T07:37:54.312733+0000","last_active":"2024-02-05T07:41:29.784501+0000","last_peered":"2024-02-05T07:41:29.784501+0000","last_clean":"2024-02-05T07:41:29.784501+0000","last_became_active":"2024-02-05T07:36:32.733494+0000","last_became_peered":"2024-02-05T07:36:32.733494+0000","last_unstale":"2024-02-05T07:41:29.784501+0000","last_undegraded":"2024-02-05T07:41:29.784501+0000","last_fullsized":"2024-02-05T07:41:29.784501+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:54.312642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.312642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":647,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823339+0000","last_change":"2024-02-05T07:38:31.803859+0000","last_active":"2024-02-05T07:41:24.823339+0000","last_peered":"2024-02-05T07:41:24.823339+0000","last_clean":"2024-02-05T07:41:24.823339+0000","last_became_active":"2024-02-05T07:38:31.803474+0000","last_became_peered":"2024-02-05T07:38:31.803474+0000","last_unstale":"2024-02-05T07:41:24.823339+0000","last_undegraded":"2024-02-05T07:41:24.823339+0000","last_fullsized":"2024-02-05T07:41:24.823339+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:01.478555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:38:01.478555+0000","last_clean_scrub_stamp":"2024-02-05T07:38:01.478555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":1691,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824019+0000","last_change":"2024-02-05T07:40:28.961455+0000","last_active":"2024-02-05T07:41:24.824019+0000","last_peered":"2024-02-05T07:41:24.824019+0000","last_clean":"2024-02-05T07:41:24.824019+0000","last_became_active":"2024-02-05T07:40:28.961096+0000","last_became_peered":"2024-02-05T07:40:28.961096+0000","last_unstale":"2024-02-05T07:41:24.824019+0000","last_undegraded":"2024-02-05T07:41:24.824019+0000","last_fullsized":"2024-02-05T07:41:24.824019+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:39.295329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:39.295329+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":118,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":343,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.783121+0000","last_change":"2024-02-05T07:37:51.288853+0000","last_active":"2024-02-05T07:41:29.783121+0000","last_peered":"2024-02-05T07:41:29.783121+0000","last_clean":"2024-02-05T07:41:29.783121+0000","last_became_active":"2024-02-05T07:36:32.733003+0000","last_became_peered":"2024-02-05T07:36:32.733003+0000","last_unstale":"2024-02-05T07:41:29.783121+0000","last_undegraded":"2024-02-05T07:41:29.783121+0000","last_fullsized":"2024-02-05T07:41:29.783121+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:51.288772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:51.288772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"615'24","reported_seq":1321,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.780767+0000","last_change":"2024-02-05T07:41:19.498668+0000","last_active":"2024-02-05T07:41:29.780767+0000","last_peered":"2024-02-05T07:41:29.780767+0000","last_clean":"2024-02-05T07:41:29.780767+0000","last_became_active":"2024-02-05T07:38:09.480893+0000","last_became_peered":"2024-02-05T07:38:09.480893+0000","last_unstale":"2024-02-05T07:41:29.780767+0000","last_undegraded":"2024-02-05T07:41:29.780767+0000","last_fullsized":"2024-02-05T07:41:29.780767+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'24","last_scrub_stamp":"2024-02-05T07:41:19.498574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:41:19.498574+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:39.954419+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":240,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.782594+0000","last_change":"2024-02-05T07:40:18.107509+0000","last_active":"2024-02-05T07:41:29.782594+0000","last_peered":"2024-02-05T07:41:29.782594+0000","last_clean":"2024-02-05T07:41:29.782594+0000","last_became_active":"2024-02-05T07:38:09.485926+0000","last_became_peered":"2024-02-05T07:38:09.485926+0000","last_unstale":"2024-02-05T07:41:29.782594+0000","last_undegraded":"2024-02-05T07:41:29.782594+0000","last_fullsized":"2024-02-05T07:41:29.782594+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:18.107395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:40:18.107395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:42.598865+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":911,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824287+0000","last_change":"2024-02-05T07:39:47.983852+0000","last_active":"2024-02-05T07:41:24.824287+0000","last_peered":"2024-02-05T07:41:24.824287+0000","last_clean":"2024-02-05T07:41:24.824287+0000","last_became_active":"2024-02-05T07:30:40.485991+0000","last_became_peered":"2024-02-05T07:30:40.485991+0000","last_unstale":"2024-02-05T07:41:24.824287+0000","last_undegraded":"2024-02-05T07:41:24.824287+0000","last_fullsized":"2024-02-05T07:41:24.824287+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:47.982166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:47.982166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":1323,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784186+0000","last_change":"2024-02-05T07:40:24.720852+0000","last_active":"2024-02-05T07:41:29.784186+0000","last_peered":"2024-02-05T07:41:29.784186+0000","last_clean":"2024-02-05T07:41:29.784186+0000","last_became_active":"2024-02-05T07:40:24.720423+0000","last_became_peered":"2024-02-05T07:40:24.720423+0000","last_unstale":"2024-02-05T07:41:29.784186+0000","last_undegraded":"2024-02-05T07:41:29.784186+0000","last_fullsized":"2024-02-05T07:41:29.784186+0000","mapping_epoch":595,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:33.547397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:33.547397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"589'13","reported_seq":1471,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:28.808382+0000","last_change":"2024-02-05T07:41:28.808405+0000","last_active":"2024-02-05T07:41:28.808382+0000","last_peered":"2024-02-05T07:41:28.808382+0000","last_clean":"2024-02-05T07:41:28.808382+0000","last_became_active":"2024-02-05T07:38:09.471365+0000","last_became_peered":"2024-02-05T07:38:09.471365+0000","last_unstale":"2024-02-05T07:41:28.808382+0000","last_undegraded":"2024-02-05T07:41:28.808382+0000","last_fullsized":"2024-02-05T07:41:28.808382+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"589'13","last_scrub_stamp":"2024-02-05T07:41:28.808327+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:41:28.808327+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:49.072623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"615'39","reported_seq":998,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822549+0000","last_change":"2024-02-05T07:38:09.513629+0000","last_active":"2024-02-05T07:41:24.822549+0000","last_peered":"2024-02-05T07:41:24.822549+0000","last_clean":"2024-02-05T07:41:24.822549+0000","last_became_active":"2024-02-05T07:38:09.471133+0000","last_became_peered":"2024-02-05T07:38:09.471133+0000","last_unstale":"2024-02-05T07:41:24.822549+0000","last_undegraded":"2024-02-05T07:41:24.822549+0000","last_fullsized":"2024-02-05T07:41:24.822549+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:38:00.687570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:00.687570+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"614'4","reported_seq":257,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.782575+0000","last_change":"2024-02-05T07:38:55.830017+0000","last_active":"2024-02-05T07:41:29.782575+0000","last_peered":"2024-02-05T07:41:29.782575+0000","last_clean":"2024-02-05T07:41:29.782575+0000","last_became_active":"2024-02-05T07:38:09.482757+0000","last_became_peered":"2024-02-05T07:38:09.482757+0000","last_unstale":"2024-02-05T07:41:29.782575+0000","last_undegraded":"2024-02-05T07:41:29.782575+0000","last_fullsized":"2024-02-05T07:41:29.782575+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"614'4","last_scrub_stamp":"2024-02-05T07:38:55.829386+0000","last_deep_scrub":"614'4","last_deep_scrub_stamp":"2024-02-05T07:38:55.829386+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.829386+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1006,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822225+0000","last_change":"2024-02-05T07:41:05.992783+0000","last_active":"2024-02-05T07:41:24.822225+0000","last_peered":"2024-02-05T07:41:24.822225+0000","last_clean":"2024-02-05T07:41:24.822225+0000","last_became_active":"2024-02-05T07:36:32.701535+0000","last_became_peered":"2024-02-05T07:36:32.701535+0000","last_unstale":"2024-02-05T07:41:24.822225+0000","last_undegraded":"2024-02-05T07:41:24.822225+0000","last_fullsized":"2024-02-05T07:41:24.822225+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:05.992703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:41:05.992703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:34.595129+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":644,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.781326+0000","last_change":"2024-02-05T07:38:57.845026+0000","last_active":"2024-02-05T07:41:29.781326+0000","last_peered":"2024-02-05T07:41:29.781326+0000","last_clean":"2024-02-05T07:41:29.781326+0000","last_became_active":"2024-02-05T07:37:41.643328+0000","last_became_peered":"2024-02-05T07:37:41.643328+0000","last_unstale":"2024-02-05T07:41:29.781326+0000","last_undegraded":"2024-02-05T07:41:29.781326+0000","last_fullsized":"2024-02-05T07:41:29.781326+0000","mapping_epoch":576,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:38:57.844891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:38:57.844891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"615'21","reported_seq":967,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825574+0000","last_change":"2024-02-05T07:38:48.343585+0000","last_active":"2024-02-05T07:41:24.825574+0000","last_peered":"2024-02-05T07:41:24.825574+0000","last_clean":"2024-02-05T07:41:24.825574+0000","last_became_active":"2024-02-05T07:36:32.742483+0000","last_became_peered":"2024-02-05T07:36:32.742483+0000","last_unstale":"2024-02-05T07:41:24.825574+0000","last_undegraded":"2024-02-05T07:41:24.825574+0000","last_fullsized":"2024-02-05T07:41:24.825574+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'15","last_scrub_stamp":"2024-02-05T07:38:48.343448+0000","last_deep_scrub":"613'15","last_deep_scrub_stamp":"2024-02-05T07:38:48.343448+0000","last_clean_scrub_stamp":"2024-02-05T07:38:48.343448+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":1137,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824501+0000","last_change":"2024-02-05T07:41:24.824524+0000","last_active":"2024-02-05T07:41:24.824501+0000","last_peered":"2024-02-05T07:41:24.824501+0000","last_clean":"2024-02-05T07:41:24.824501+0000","last_became_active":"2024-02-05T07:28:26.597829+0000","last_became_peered":"2024-02-05T07:28:26.597829+0000","last_unstale":"2024-02-05T07:41:24.824501+0000","last_undegraded":"2024-02-05T07:41:24.824501+0000","last_fullsized":"2024-02-05T07:41:24.824501+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:41:24.824446+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:41:24.824446+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:51.694317+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":899,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825495+0000","last_change":"2024-02-05T07:37:01.178948+0000","last_active":"2024-02-05T07:41:24.825495+0000","last_peered":"2024-02-05T07:41:24.825495+0000","last_clean":"2024-02-05T07:41:24.825495+0000","last_became_active":"2024-02-05T07:28:26.599413+0000","last_became_peered":"2024-02-05T07:28:26.599413+0000","last_unstale":"2024-02-05T07:41:24.825495+0000","last_undegraded":"2024-02-05T07:41:24.825495+0000","last_fullsized":"2024-02-05T07:41:24.825495+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:01.178816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:01.178816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1193,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822731+0000","last_change":"2024-02-05T07:38:41.820804+0000","last_active":"2024-02-05T07:41:24.822731+0000","last_peered":"2024-02-05T07:41:24.822731+0000","last_clean":"2024-02-05T07:41:24.822731+0000","last_became_active":"2024-02-05T07:38:09.480228+0000","last_became_peered":"2024-02-05T07:38:09.480228+0000","last_unstale":"2024-02-05T07:41:24.822731+0000","last_undegraded":"2024-02-05T07:41:24.822731+0000","last_fullsized":"2024-02-05T07:41:24.822731+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:41.820713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:38:41.820713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":1133,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824085+0000","last_change":"2024-02-05T07:38:41.750699+0000","last_active":"2024-02-05T07:41:24.824085+0000","last_peered":"2024-02-05T07:41:24.824085+0000","last_clean":"2024-02-05T07:41:24.824085+0000","last_became_active":"2024-02-05T07:38:41.750174+0000","last_became_peered":"2024-02-05T07:38:41.750174+0000","last_unstale":"2024-02-05T07:41:24.824085+0000","last_undegraded":"2024-02-05T07:41:24.824085+0000","last_fullsized":"2024-02-05T07:41:24.824085+0000","mapping_epoch":607,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:38.260984+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:37:38.260984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"615'21","reported_seq":972,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825369+0000","last_change":"2024-02-05T07:38:55.387276+0000","last_active":"2024-02-05T07:41:24.825369+0000","last_peered":"2024-02-05T07:41:24.825369+0000","last_clean":"2024-02-05T07:41:24.825369+0000","last_became_active":"2024-02-05T07:36:32.747221+0000","last_became_peered":"2024-02-05T07:36:32.747221+0000","last_unstale":"2024-02-05T07:41:24.825369+0000","last_undegraded":"2024-02-05T07:41:24.825369+0000","last_fullsized":"2024-02-05T07:41:24.825369+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-02-05T07:38:55.387145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.387145+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":744,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822238+0000","last_change":"2024-02-05T07:39:00.488430+0000","last_active":"2024-02-05T07:41:24.822238+0000","last_peered":"2024-02-05T07:41:24.822238+0000","last_clean":"2024-02-05T07:41:24.822238+0000","last_became_active":"2024-02-05T07:38:09.479116+0000","last_became_peered":"2024-02-05T07:38:09.479116+0000","last_unstale":"2024-02-05T07:41:24.822238+0000","last_undegraded":"2024-02-05T07:41:24.822238+0000","last_fullsized":"2024-02-05T07:41:24.822238+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:00.488357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:39:00.488357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":902,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825654+0000","last_change":"2024-02-05T07:38:53.372481+0000","last_active":"2024-02-05T07:41:24.825654+0000","last_peered":"2024-02-05T07:41:24.825654+0000","last_clean":"2024-02-05T07:41:24.825654+0000","last_became_active":"2024-02-05T07:30:40.496076+0000","last_became_peered":"2024-02-05T07:30:40.496076+0000","last_unstale":"2024-02-05T07:41:24.825654+0000","last_undegraded":"2024-02-05T07:41:24.825654+0000","last_fullsized":"2024-02-05T07:41:24.825654+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:53.372355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_clean_scrub_stamp":"2024-02-05T07:38:53.372355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":349,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784955+0000","last_change":"2024-02-05T07:38:09.489174+0000","last_active":"2024-02-05T07:41:29.784955+0000","last_peered":"2024-02-05T07:41:29.784955+0000","last_clean":"2024-02-05T07:41:29.784955+0000","last_became_active":"2024-02-05T07:38:09.488755+0000","last_became_peered":"2024-02-05T07:38:09.488755+0000","last_unstale":"2024-02-05T07:41:29.784955+0000","last_undegraded":"2024-02-05T07:41:29.784955+0000","last_fullsized":"2024-02-05T07:41:29.784955+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:47.330693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:37:47.330693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":1134,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822684+0000","last_change":"2024-02-05T07:41:15.995646+0000","last_active":"2024-02-05T07:41:24.822684+0000","last_peered":"2024-02-05T07:41:24.822684+0000","last_clean":"2024-02-05T07:41:24.822684+0000","last_became_active":"2024-02-05T07:38:41.725034+0000","last_became_peered":"2024-02-05T07:38:41.725034+0000","last_unstale":"2024-02-05T07:41:24.822684+0000","last_undegraded":"2024-02-05T07:41:24.822684+0000","last_fullsized":"2024-02-05T07:41:24.822684+0000","mapping_epoch":607,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:41:15.995540+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:41:15.995540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:29.116642+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"615'25","reported_seq":1260,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825722+0000","last_change":"2024-02-05T07:37:06.319292+0000","last_active":"2024-02-05T07:41:24.825722+0000","last_peered":"2024-02-05T07:41:24.825722+0000","last_clean":"2024-02-05T07:41:24.825722+0000","last_became_active":"2024-02-05T07:30:40.483700+0000","last_became_peered":"2024-02-05T07:30:40.483700+0000","last_unstale":"2024-02-05T07:41:24.825722+0000","last_undegraded":"2024-02-05T07:41:24.825722+0000","last_fullsized":"2024-02-05T07:41:24.825722+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:37:06.318619+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:37:06.318619+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1065,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823978+0000","last_change":"2024-02-05T07:36:46.059245+0000","last_active":"2024-02-05T07:41:24.823978+0000","last_peered":"2024-02-05T07:41:24.823978+0000","last_clean":"2024-02-05T07:41:24.823978+0000","last_became_active":"2024-02-05T07:30:40.476058+0000","last_became_peered":"2024-02-05T07:30:40.476058+0000","last_unstale":"2024-02-05T07:41:24.823978+0000","last_undegraded":"2024-02-05T07:41:24.823978+0000","last_fullsized":"2024-02-05T07:41:24.823978+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:46.059129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:46.059129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"538'13","reported_seq":1101,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824568+0000","last_change":"2024-02-05T07:38:51.305914+0000","last_active":"2024-02-05T07:41:24.824568+0000","last_peered":"2024-02-05T07:41:24.824568+0000","last_clean":"2024-02-05T07:41:24.824568+0000","last_became_active":"2024-02-05T07:36:32.746270+0000","last_became_peered":"2024-02-05T07:36:32.746270+0000","last_unstale":"2024-02-05T07:41:24.824568+0000","last_undegraded":"2024-02-05T07:41:24.824568+0000","last_fullsized":"2024-02-05T07:41:24.824568+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'13","last_scrub_stamp":"2024-02-05T07:38:51.305782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:51.305782+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1211,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784852+0000","last_change":"2024-02-05T07:38:09.504184+0000","last_active":"2024-02-05T07:41:29.784852+0000","last_peered":"2024-02-05T07:41:29.784852+0000","last_clean":"2024-02-05T07:41:29.784852+0000","last_became_active":"2024-02-05T07:38:09.503255+0000","last_became_peered":"2024-02-05T07:38:09.503255+0000","last_unstale":"2024-02-05T07:41:29.784852+0000","last_undegraded":"2024-02-05T07:41:29.784852+0000","last_fullsized":"2024-02-05T07:41:29.784852+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:58.277028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:37:58.277028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":690,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825184+0000","last_change":"2024-02-05T07:37:40.289255+0000","last_active":"2024-02-05T07:41:24.825184+0000","last_peered":"2024-02-05T07:41:24.825184+0000","last_clean":"2024-02-05T07:41:24.825184+0000","last_became_active":"2024-02-05T07:37:12.040170+0000","last_became_peered":"2024-02-05T07:37:12.040170+0000","last_unstale":"2024-02-05T07:41:24.825184+0000","last_undegraded":"2024-02-05T07:41:24.825184+0000","last_fullsized":"2024-02-05T07:41:24.825184+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:40.289174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:40.289174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"615'35","reported_seq":993,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825253+0000","last_change":"2024-02-05T07:36:59.117388+0000","last_active":"2024-02-05T07:41:24.825253+0000","last_peered":"2024-02-05T07:41:24.825253+0000","last_clean":"2024-02-05T07:41:24.825253+0000","last_became_active":"2024-02-05T07:36:32.746757+0000","last_became_peered":"2024-02-05T07:36:32.746757+0000","last_unstale":"2024-02-05T07:41:24.825253+0000","last_undegraded":"2024-02-05T07:41:24.825253+0000","last_fullsized":"2024-02-05T07:41:24.825253+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'25","last_scrub_stamp":"2024-02-05T07:36:59.117218+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:36:59.117218+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":604,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784587+0000","last_change":"2024-02-05T07:39:53.396917+0000","last_active":"2024-02-05T07:41:29.784587+0000","last_peered":"2024-02-05T07:41:29.784587+0000","last_clean":"2024-02-05T07:41:29.784587+0000","last_became_active":"2024-02-05T07:38:09.495040+0000","last_became_peered":"2024-02-05T07:38:09.495040+0000","last_unstale":"2024-02-05T07:41:29.784587+0000","last_undegraded":"2024-02-05T07:41:29.784587+0000","last_fullsized":"2024-02-05T07:41:29.784587+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:53.396795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:39:53.396795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"584'12","reported_seq":290,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825221+0000","last_change":"2024-02-05T07:40:05.765708+0000","last_active":"2024-02-05T07:41:24.825221+0000","last_peered":"2024-02-05T07:41:24.825221+0000","last_clean":"2024-02-05T07:41:24.825221+0000","last_became_active":"2024-02-05T07:37:58.781263+0000","last_became_peered":"2024-02-05T07:37:58.781263+0000","last_unstale":"2024-02-05T07:41:24.825221+0000","last_undegraded":"2024-02-05T07:41:24.825221+0000","last_fullsized":"2024-02-05T07:41:24.825221+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'12","last_scrub_stamp":"2024-02-05T07:40:05.765574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:40:05.765574+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1067,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825388+0000","last_change":"2024-02-05T07:39:01.508990+0000","last_active":"2024-02-05T07:41:24.825388+0000","last_peered":"2024-02-05T07:41:24.825388+0000","last_clean":"2024-02-05T07:41:24.825388+0000","last_became_active":"2024-02-05T07:38:09.487782+0000","last_became_peered":"2024-02-05T07:38:09.487782+0000","last_unstale":"2024-02-05T07:41:24.825388+0000","last_undegraded":"2024-02-05T07:41:24.825388+0000","last_fullsized":"2024-02-05T07:41:24.825388+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:01.508866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:39:01.508866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"615'27","reported_seq":865,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822304+0000","last_change":"2024-02-05T07:41:23.842983+0000","last_active":"2024-02-05T07:41:24.822304+0000","last_peered":"2024-02-05T07:41:24.822304+0000","last_clean":"2024-02-05T07:41:24.822304+0000","last_became_active":"2024-02-05T07:38:09.484252+0000","last_became_peered":"2024-02-05T07:38:09.484252+0000","last_unstale":"2024-02-05T07:41:24.822304+0000","last_undegraded":"2024-02-05T07:41:24.822304+0000","last_fullsized":"2024-02-05T07:41:24.822304+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'27","last_scrub_stamp":"2024-02-05T07:41:23.842876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:41:23.842876+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:52.110321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1238,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823060+0000","last_change":"2024-02-05T07:38:32.811706+0000","last_active":"2024-02-05T07:41:24.823060+0000","last_peered":"2024-02-05T07:41:24.823060+0000","last_clean":"2024-02-05T07:41:24.823060+0000","last_became_active":"2024-02-05T07:38:09.472022+0000","last_became_peered":"2024-02-05T07:38:09.472022+0000","last_unstale":"2024-02-05T07:41:24.823060+0000","last_undegraded":"2024-02-05T07:41:24.823060+0000","last_fullsized":"2024-02-05T07:41:24.823060+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:32.811595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:38:32.811595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1340,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.782040+0000","last_change":"2024-02-05T07:39:55.907631+0000","last_active":"2024-02-05T07:41:29.782040+0000","last_peered":"2024-02-05T07:41:29.782040+0000","last_clean":"2024-02-05T07:41:29.782040+0000","last_became_active":"2024-02-05T07:30:40.487141+0000","last_became_peered":"2024-02-05T07:30:40.487141+0000","last_unstale":"2024-02-05T07:41:29.782040+0000","last_undegraded":"2024-02-05T07:41:29.782040+0000","last_fullsized":"2024-02-05T07:41:29.782040+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:55.907516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_clean_scrub_stamp":"2024-02-05T07:39:55.907516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":907,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824196+0000","last_change":"2024-02-05T07:37:54.469820+0000","last_active":"2024-02-05T07:41:24.824196+0000","last_peered":"2024-02-05T07:41:24.824196+0000","last_clean":"2024-02-05T07:41:24.824196+0000","last_became_active":"2024-02-05T07:36:32.742150+0000","last_became_peered":"2024-02-05T07:36:32.742150+0000","last_unstale":"2024-02-05T07:41:24.824196+0000","last_undegraded":"2024-02-05T07:41:24.824196+0000","last_fullsized":"2024-02-05T07:41:24.824196+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:54.469701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:37:54.469701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"615'21","reported_seq":995,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.781463+0000","last_change":"2024-02-05T07:41:15.590638+0000","last_active":"2024-02-05T07:41:29.781463+0000","last_peered":"2024-02-05T07:41:29.781463+0000","last_clean":"2024-02-05T07:41:29.781463+0000","last_became_active":"2024-02-05T07:36:32.710090+0000","last_became_peered":"2024-02-05T07:36:32.710090+0000","last_unstale":"2024-02-05T07:41:29.781463+0000","last_undegraded":"2024-02-05T07:41:29.781463+0000","last_fullsized":"2024-02-05T07:41:29.781463+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'21","last_scrub_stamp":"2024-02-05T07:41:15.590544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:41:15.590544+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:27.533601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1116,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825626+0000","last_change":"2024-02-05T07:38:09.473515+0000","last_active":"2024-02-05T07:41:24.825626+0000","last_peered":"2024-02-05T07:41:24.825626+0000","last_clean":"2024-02-05T07:41:24.825626+0000","last_became_active":"2024-02-05T07:38:09.472668+0000","last_became_peered":"2024-02-05T07:38:09.472668+0000","last_unstale":"2024-02-05T07:41:24.825626+0000","last_undegraded":"2024-02-05T07:41:24.825626+0000","last_fullsized":"2024-02-05T07:41:24.825626+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:32.120410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:36:32.120410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":897,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824497+0000","last_change":"2024-02-05T07:38:54.397360+0000","last_active":"2024-02-05T07:41:24.824497+0000","last_peered":"2024-02-05T07:41:24.824497+0000","last_clean":"2024-02-05T07:41:24.824497+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:41:24.824497+0000","last_undegraded":"2024-02-05T07:41:24.824497+0000","last_fullsized":"2024-02-05T07:41:24.824497+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:54.397113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:38:54.397113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":782,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822503+0000","last_change":"2024-02-05T07:40:01.732873+0000","last_active":"2024-02-05T07:41:24.822503+0000","last_peered":"2024-02-05T07:41:24.822503+0000","last_clean":"2024-02-05T07:41:24.822503+0000","last_became_active":"2024-02-05T07:34:30.670179+0000","last_became_peered":"2024-02-05T07:34:30.670179+0000","last_unstale":"2024-02-05T07:41:24.822503+0000","last_undegraded":"2024-02-05T07:41:24.822503+0000","last_fullsized":"2024-02-05T07:41:24.822503+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:01.732803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:24:20.215062+0000","last_clean_scrub_stamp":"2024-02-05T07:40:01.732803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:29.909417+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":1697,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784147+0000","last_change":"2024-02-05T07:39:11.702922+0000","last_active":"2024-02-05T07:41:29.784147+0000","last_peered":"2024-02-05T07:41:29.784147+0000","last_clean":"2024-02-05T07:41:29.784147+0000","last_became_active":"2024-02-05T07:39:11.702286+0000","last_became_peered":"2024-02-05T07:39:11.702286+0000","last_unstale":"2024-02-05T07:41:29.784147+0000","last_undegraded":"2024-02-05T07:41:29.784147+0000","last_fullsized":"2024-02-05T07:41:29.784147+0000","mapping_epoch":619,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:37:41.395638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:37:41.395638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"616'27","reported_seq":954,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824010+0000","last_change":"2024-02-05T07:36:48.088544+0000","last_active":"2024-02-05T07:41:24.824010+0000","last_peered":"2024-02-05T07:41:24.824010+0000","last_clean":"2024-02-05T07:41:24.824010+0000","last_became_active":"2024-02-05T07:30:40.501893+0000","last_became_peered":"2024-02-05T07:30:40.501893+0000","last_unstale":"2024-02-05T07:41:24.824010+0000","last_undegraded":"2024-02-05T07:41:24.824010+0000","last_fullsized":"2024-02-05T07:41:24.824010+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:36:48.088406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:36:48.088406+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":733,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.782684+0000","last_change":"2024-02-05T07:38:54.868354+0000","last_active":"2024-02-05T07:41:29.782684+0000","last_peered":"2024-02-05T07:41:29.782684+0000","last_clean":"2024-02-05T07:41:29.782684+0000","last_became_active":"2024-02-05T07:36:32.712612+0000","last_became_peered":"2024-02-05T07:36:32.712612+0000","last_unstale":"2024-02-05T07:41:29.782684+0000","last_undegraded":"2024-02-05T07:41:29.782684+0000","last_fullsized":"2024-02-05T07:41:29.782684+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:54.867940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:54.867940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":736,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.821839+0000","last_change":"2024-02-05T07:38:59.527516+0000","last_active":"2024-02-05T07:41:24.821839+0000","last_peered":"2024-02-05T07:41:24.821839+0000","last_clean":"2024-02-05T07:41:24.821839+0000","last_became_active":"2024-02-05T07:38:09.480566+0000","last_became_peered":"2024-02-05T07:38:09.480566+0000","last_unstale":"2024-02-05T07:41:24.821839+0000","last_undegraded":"2024-02-05T07:41:24.821839+0000","last_fullsized":"2024-02-05T07:41:24.821839+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:59.527436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:38:59.527436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":909,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824224+0000","last_change":"2024-02-05T07:40:20.094896+0000","last_active":"2024-02-05T07:41:24.824224+0000","last_peered":"2024-02-05T07:41:24.824224+0000","last_clean":"2024-02-05T07:41:24.824224+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:41:24.824224+0000","last_undegraded":"2024-02-05T07:41:24.824224+0000","last_fullsized":"2024-02-05T07:41:24.824224+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:20.094078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:40:20.094078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:33.467712+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"615'12","reported_seq":919,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823697+0000","last_change":"2024-02-05T07:35:47.070886+0000","last_active":"2024-02-05T07:41:24.823697+0000","last_peered":"2024-02-05T07:41:24.823697+0000","last_clean":"2024-02-05T07:41:24.823697+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:41:24.823697+0000","last_undegraded":"2024-02-05T07:41:24.823697+0000","last_fullsized":"2024-02-05T07:41:24.823697+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'8","last_scrub_stamp":"2024-02-05T07:35:47.070805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:47.070805+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":907,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825378+0000","last_change":"2024-02-05T07:40:07.758232+0000","last_active":"2024-02-05T07:41:24.825378+0000","last_peered":"2024-02-05T07:41:24.825378+0000","last_clean":"2024-02-05T07:41:24.825378+0000","last_became_active":"2024-02-05T07:28:26.601932+0000","last_became_peered":"2024-02-05T07:28:26.601932+0000","last_unstale":"2024-02-05T07:41:24.825378+0000","last_undegraded":"2024-02-05T07:41:24.825378+0000","last_fullsized":"2024-02-05T07:41:24.825378+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:07.758093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:34:39.980067+0000","last_clean_scrub_stamp":"2024-02-05T07:40:07.758093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:33.830081+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1177,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822291+0000","last_change":"2024-02-05T07:40:02.708852+0000","last_active":"2024-02-05T07:41:24.822291+0000","last_peered":"2024-02-05T07:41:24.822291+0000","last_clean":"2024-02-05T07:41:24.822291+0000","last_became_active":"2024-02-05T07:38:09.464446+0000","last_became_peered":"2024-02-05T07:38:09.464446+0000","last_unstale":"2024-02-05T07:41:24.822291+0000","last_undegraded":"2024-02-05T07:41:24.822291+0000","last_fullsized":"2024-02-05T07:41:24.822291+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:02.708785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:40:02.708785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":357,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.785719+0000","last_change":"2024-02-05T07:41:10.476563+0000","last_active":"2024-02-05T07:41:29.785719+0000","last_peered":"2024-02-05T07:41:29.785719+0000","last_clean":"2024-02-05T07:41:29.785719+0000","last_became_active":"2024-02-05T07:38:09.503369+0000","last_became_peered":"2024-02-05T07:38:09.503369+0000","last_unstale":"2024-02-05T07:41:29.785719+0000","last_undegraded":"2024-02-05T07:41:29.785719+0000","last_fullsized":"2024-02-05T07:41:29.785719+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:10.476432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:58.296469+0000","last_clean_scrub_stamp":"2024-02-05T07:41:10.476432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:27.212178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"616'27","reported_seq":1170,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823374+0000","last_change":"2024-02-05T07:38:55.551835+0000","last_active":"2024-02-05T07:41:24.823374+0000","last_peered":"2024-02-05T07:41:24.823374+0000","last_clean":"2024-02-05T07:41:24.823374+0000","last_became_active":"2024-02-05T07:28:26.589215+0000","last_became_peered":"2024-02-05T07:28:26.589215+0000","last_unstale":"2024-02-05T07:41:24.823374+0000","last_undegraded":"2024-02-05T07:41:24.823374+0000","last_fullsized":"2024-02-05T07:41:24.823374+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'19","last_scrub_stamp":"2024-02-05T07:38:55.551751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:38:55.551751+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":904,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823503+0000","last_change":"2024-02-05T07:40:11.850388+0000","last_active":"2024-02-05T07:41:24.823503+0000","last_peered":"2024-02-05T07:41:24.823503+0000","last_clean":"2024-02-05T07:41:24.823503+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:41:24.823503+0000","last_undegraded":"2024-02-05T07:41:24.823503+0000","last_fullsized":"2024-02-05T07:41:24.823503+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:11.849558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:40:11.849558+0000","last_clean_scrub_stamp":"2024-02-05T07:40:11.849558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:35.605390+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":351,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784917+0000","last_change":"2024-02-05T07:40:15.407981+0000","last_active":"2024-02-05T07:41:29.784917+0000","last_peered":"2024-02-05T07:41:29.784917+0000","last_clean":"2024-02-05T07:41:29.784917+0000","last_became_active":"2024-02-05T07:36:32.752879+0000","last_became_peered":"2024-02-05T07:36:32.752879+0000","last_unstale":"2024-02-05T07:41:29.784917+0000","last_undegraded":"2024-02-05T07:41:29.784917+0000","last_fullsized":"2024-02-05T07:41:29.784917+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:15.407749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:40:15.407749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:41.593247+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":837,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784337+0000","last_change":"2024-02-05T07:40:20.624459+0000","last_active":"2024-02-05T07:41:29.784337+0000","last_peered":"2024-02-05T07:41:29.784337+0000","last_clean":"2024-02-05T07:41:29.784337+0000","last_became_active":"2024-02-05T07:38:09.503835+0000","last_became_peered":"2024-02-05T07:38:09.503835+0000","last_unstale":"2024-02-05T07:41:29.784337+0000","last_undegraded":"2024-02-05T07:41:29.784337+0000","last_fullsized":"2024-02-05T07:41:29.784337+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:20.624132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:40:20.624132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"616'30","reported_seq":1079,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825560+0000","last_change":"2024-02-05T07:35:13.094823+0000","last_active":"2024-02-05T07:41:24.825560+0000","last_peered":"2024-02-05T07:41:24.825560+0000","last_clean":"2024-02-05T07:41:24.825560+0000","last_became_active":"2024-02-05T07:30:40.495954+0000","last_became_peered":"2024-02-05T07:30:40.495954+0000","last_unstale":"2024-02-05T07:41:24.825560+0000","last_undegraded":"2024-02-05T07:41:24.825560+0000","last_fullsized":"2024-02-05T07:41:24.825560+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'20","last_scrub_stamp":"2024-02-05T07:35:13.094694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:35:13.094694+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"585'2","reported_seq":204,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.783325+0000","last_change":"2024-02-05T07:38:31.834917+0000","last_active":"2024-02-05T07:41:29.783325+0000","last_peered":"2024-02-05T07:41:29.783325+0000","last_clean":"2024-02-05T07:41:29.783325+0000","last_became_active":"2024-02-05T07:38:31.811200+0000","last_became_peered":"2024-02-05T07:38:31.811200+0000","last_unstale":"2024-02-05T07:41:29.783325+0000","last_undegraded":"2024-02-05T07:41:29.783325+0000","last_fullsized":"2024-02-05T07:41:29.783325+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'2","last_scrub_stamp":"2024-02-05T07:38:00.631235+0000","last_deep_scrub":"585'2","last_deep_scrub_stamp":"2024-02-05T07:38:00.631235+0000","last_clean_scrub_stamp":"2024-02-05T07:38:00.631235+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"589'14","reported_seq":1457,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823504+0000","last_change":"2024-02-05T07:41:08.957920+0000","last_active":"2024-02-05T07:41:24.823504+0000","last_peered":"2024-02-05T07:41:24.823504+0000","last_clean":"2024-02-05T07:41:24.823504+0000","last_became_active":"2024-02-05T07:38:09.472910+0000","last_became_peered":"2024-02-05T07:38:09.472910+0000","last_unstale":"2024-02-05T07:41:24.823504+0000","last_undegraded":"2024-02-05T07:41:24.823504+0000","last_fullsized":"2024-02-05T07:41:24.823504+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"589'14","last_scrub_stamp":"2024-02-05T07:41:08.957841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:41:08.957841+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:26.527068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":356,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784826+0000","last_change":"2024-02-05T07:40:09.332550+0000","last_active":"2024-02-05T07:41:29.784826+0000","last_peered":"2024-02-05T07:41:29.784826+0000","last_clean":"2024-02-05T07:41:29.784826+0000","last_became_active":"2024-02-05T07:38:09.482346+0000","last_became_peered":"2024-02-05T07:38:09.482346+0000","last_unstale":"2024-02-05T07:41:29.784826+0000","last_undegraded":"2024-02-05T07:41:29.784826+0000","last_fullsized":"2024-02-05T07:41:29.784826+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:40:09.332235+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:40:09.332235+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:35.166123+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"616'29","reported_seq":957,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825387+0000","last_change":"2024-02-05T07:40:06.764929+0000","last_active":"2024-02-05T07:41:24.825387+0000","last_peered":"2024-02-05T07:41:24.825387+0000","last_clean":"2024-02-05T07:41:24.825387+0000","last_became_active":"2024-02-05T07:38:09.486225+0000","last_became_peered":"2024-02-05T07:38:09.486225+0000","last_unstale":"2024-02-05T07:41:24.825387+0000","last_undegraded":"2024-02-05T07:41:24.825387+0000","last_fullsized":"2024-02-05T07:41:24.825387+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'29","last_scrub_stamp":"2024-02-05T07:40:06.764791+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:40:06.764791+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":356,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.785831+0000","last_change":"2024-02-05T07:41:26.429025+0000","last_active":"2024-02-05T07:41:29.785831+0000","last_peered":"2024-02-05T07:41:29.785831+0000","last_clean":"2024-02-05T07:41:29.785831+0000","last_became_active":"2024-02-05T07:38:09.489555+0000","last_became_peered":"2024-02-05T07:38:09.489555+0000","last_unstale":"2024-02-05T07:41:29.785831+0000","last_undegraded":"2024-02-05T07:41:29.785831+0000","last_fullsized":"2024-02-05T07:41:29.785831+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:26.428915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:41:26.428915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:51.567963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1222,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825067+0000","last_change":"2024-02-05T07:38:09.484586+0000","last_active":"2024-02-05T07:41:24.825067+0000","last_peered":"2024-02-05T07:41:24.825067+0000","last_clean":"2024-02-05T07:41:24.825067+0000","last_became_active":"2024-02-05T07:38:09.484261+0000","last_became_peered":"2024-02-05T07:38:09.484261+0000","last_unstale":"2024-02-05T07:41:24.825067+0000","last_undegraded":"2024-02-05T07:41:24.825067+0000","last_fullsized":"2024-02-05T07:41:24.825067+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:36:47.437414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:36:47.437414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":300,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823384+0000","last_change":"2024-02-05T07:38:35.940570+0000","last_active":"2024-02-05T07:41:24.823384+0000","last_peered":"2024-02-05T07:41:24.823384+0000","last_clean":"2024-02-05T07:41:24.823384+0000","last_became_active":"2024-02-05T07:38:31.812780+0000","last_became_peered":"2024-02-05T07:38:31.812780+0000","last_unstale":"2024-02-05T07:41:24.823384+0000","last_undegraded":"2024-02-05T07:41:24.823384+0000","last_fullsized":"2024-02-05T07:41:24.823384+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:35.940472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:35.940472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"615'39","reported_seq":376,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784437+0000","last_change":"2024-02-05T07:37:57.440904+0000","last_active":"2024-02-05T07:41:29.784437+0000","last_peered":"2024-02-05T07:41:29.784437+0000","last_clean":"2024-02-05T07:41:29.784437+0000","last_became_active":"2024-02-05T07:36:32.744301+0000","last_became_peered":"2024-02-05T07:36:32.744301+0000","last_unstale":"2024-02-05T07:41:29.784437+0000","last_undegraded":"2024-02-05T07:41:29.784437+0000","last_fullsized":"2024-02-05T07:41:29.784437+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:37:57.440806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:57.440806+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":284,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784218+0000","last_change":"2024-02-05T07:41:09.517142+0000","last_active":"2024-02-05T07:41:29.784218+0000","last_peered":"2024-02-05T07:41:29.784218+0000","last_clean":"2024-02-05T07:41:29.784218+0000","last_became_active":"2024-02-05T07:38:09.500871+0000","last_became_peered":"2024-02-05T07:38:09.500871+0000","last_unstale":"2024-02-05T07:41:29.784218+0000","last_undegraded":"2024-02-05T07:41:29.784218+0000","last_fullsized":"2024-02-05T07:41:29.784218+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:09.517063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:41:09.517063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:18.468997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1323,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.781615+0000","last_change":"2024-02-05T07:38:53.859910+0000","last_active":"2024-02-05T07:41:29.781615+0000","last_peered":"2024-02-05T07:41:29.781615+0000","last_clean":"2024-02-05T07:41:29.781615+0000","last_became_active":"2024-02-05T07:38:09.477634+0000","last_became_peered":"2024-02-05T07:38:09.477634+0000","last_unstale":"2024-02-05T07:41:29.781615+0000","last_undegraded":"2024-02-05T07:41:29.781615+0000","last_fullsized":"2024-02-05T07:41:29.781615+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:53.859782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:53.859782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":208,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.784384+0000","last_change":"2024-02-05T07:40:18.398849+0000","last_active":"2024-02-05T07:41:29.784384+0000","last_peered":"2024-02-05T07:41:29.784384+0000","last_clean":"2024-02-05T07:41:29.784384+0000","last_became_active":"2024-02-05T07:38:31.795971+0000","last_became_peered":"2024-02-05T07:38:31.795971+0000","last_unstale":"2024-02-05T07:41:29.784384+0000","last_undegraded":"2024-02-05T07:41:29.784384+0000","last_fullsized":"2024-02-05T07:41:29.784384+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:40:18.398592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:40:18.398592+0000","last_clean_scrub_stamp":"2024-02-05T07:40:18.398592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:41:45.308821+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"615'9","reported_seq":819,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824327+0000","last_change":"2024-02-05T07:38:09.527356+0000","last_active":"2024-02-05T07:41:24.824327+0000","last_peered":"2024-02-05T07:41:24.824327+0000","last_clean":"2024-02-05T07:41:24.824327+0000","last_became_active":"2024-02-05T07:38:09.496642+0000","last_became_peered":"2024-02-05T07:38:09.496642+0000","last_unstale":"2024-02-05T07:41:24.824327+0000","last_undegraded":"2024-02-05T07:41:24.824327+0000","last_fullsized":"2024-02-05T07:41:24.824327+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'7","last_scrub_stamp":"2024-02-05T07:37:06.495165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:06.495165+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":567,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.783542+0000","last_change":"2024-02-05T07:41:26.613289+0000","last_active":"2024-02-05T07:41:29.783542+0000","last_peered":"2024-02-05T07:41:29.783542+0000","last_clean":"2024-02-05T07:41:29.783542+0000","last_became_active":"2024-02-05T07:38:09.495573+0000","last_became_peered":"2024-02-05T07:38:09.495573+0000","last_unstale":"2024-02-05T07:41:29.783542+0000","last_undegraded":"2024-02-05T07:41:29.783542+0000","last_fullsized":"2024-02-05T07:41:29.783542+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:26.613182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:41:26.613182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:42.657348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":240,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.781845+0000","last_change":"2024-02-05T07:39:45.831179+0000","last_active":"2024-02-05T07:41:29.781845+0000","last_peered":"2024-02-05T07:41:29.781845+0000","last_clean":"2024-02-05T07:41:29.781845+0000","last_became_active":"2024-02-05T07:38:09.462036+0000","last_became_peered":"2024-02-05T07:38:09.462036+0000","last_unstale":"2024-02-05T07:41:29.781845+0000","last_undegraded":"2024-02-05T07:41:29.781845+0000","last_fullsized":"2024-02-05T07:41:29.781845+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:39:45.831067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:39:45.831067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1144,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.781115+0000","last_change":"2024-02-05T07:41:27.716082+0000","last_active":"2024-02-05T07:41:29.781115+0000","last_peered":"2024-02-05T07:41:29.781115+0000","last_clean":"2024-02-05T07:41:29.781115+0000","last_became_active":"2024-02-05T07:36:32.710575+0000","last_became_peered":"2024-02-05T07:36:32.710575+0000","last_unstale":"2024-02-05T07:41:29.781115+0000","last_undegraded":"2024-02-05T07:41:29.781115+0000","last_fullsized":"2024-02-05T07:41:29.781115+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:27.715979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:41:27.715979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:49.491256+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"615'25","reported_seq":861,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824053+0000","last_change":"2024-02-05T07:41:11.835882+0000","last_active":"2024-02-05T07:41:24.824053+0000","last_peered":"2024-02-05T07:41:24.824053+0000","last_clean":"2024-02-05T07:41:24.824053+0000","last_became_active":"2024-02-05T07:38:09.479937+0000","last_became_peered":"2024-02-05T07:38:09.479937+0000","last_unstale":"2024-02-05T07:41:24.824053+0000","last_undegraded":"2024-02-05T07:41:24.824053+0000","last_fullsized":"2024-02-05T07:41:24.824053+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'25","last_scrub_stamp":"2024-02-05T07:41:11.835783+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:41:11.835783+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:16.512248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":560,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823040+0000","last_change":"2024-02-05T07:38:56.526617+0000","last_active":"2024-02-05T07:41:24.823040+0000","last_peered":"2024-02-05T07:41:24.823040+0000","last_clean":"2024-02-05T07:41:24.823040+0000","last_became_active":"2024-02-05T07:38:09.484875+0000","last_became_peered":"2024-02-05T07:38:09.484875+0000","last_unstale":"2024-02-05T07:41:24.823040+0000","last_undegraded":"2024-02-05T07:41:24.823040+0000","last_fullsized":"2024-02-05T07:41:24.823040+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:56.526522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:38:56.526522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":240,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822130+0000","last_change":"2024-02-05T07:38:47.564845+0000","last_active":"2024-02-05T07:41:24.822130+0000","last_peered":"2024-02-05T07:41:24.822130+0000","last_clean":"2024-02-05T07:41:24.822130+0000","last_became_active":"2024-02-05T07:38:09.471013+0000","last_became_peered":"2024-02-05T07:38:09.471013+0000","last_unstale":"2024-02-05T07:41:24.822130+0000","last_undegraded":"2024-02-05T07:41:24.822130+0000","last_fullsized":"2024-02-05T07:41:24.822130+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:47.564750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:38:47.564750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":343,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.783301+0000","last_change":"2024-02-05T07:38:51.521757+0000","last_active":"2024-02-05T07:41:29.783301+0000","last_peered":"2024-02-05T07:41:29.783301+0000","last_clean":"2024-02-05T07:41:29.783301+0000","last_became_active":"2024-02-05T07:36:32.733405+0000","last_became_peered":"2024-02-05T07:36:32.733405+0000","last_unstale":"2024-02-05T07:41:29.783301+0000","last_undegraded":"2024-02-05T07:41:29.783301+0000","last_fullsized":"2024-02-05T07:41:29.783301+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:51.521494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:38:51.521494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"616'39","reported_seq":992,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.823879+0000","last_change":"2024-02-05T07:37:50.422750+0000","last_active":"2024-02-05T07:41:24.823879+0000","last_peered":"2024-02-05T07:41:24.823879+0000","last_clean":"2024-02-05T07:41:24.823879+0000","last_became_active":"2024-02-05T07:36:32.741691+0000","last_became_peered":"2024-02-05T07:36:32.741691+0000","last_unstale":"2024-02-05T07:41:24.823879+0000","last_undegraded":"2024-02-05T07:41:24.823879+0000","last_fullsized":"2024-02-05T07:41:24.823879+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'27","last_scrub_stamp":"2024-02-05T07:37:50.422661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:37:50.422661+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":914,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.825082+0000","last_change":"2024-02-05T07:37:44.381379+0000","last_active":"2024-02-05T07:41:24.825082+0000","last_peered":"2024-02-05T07:41:24.825082+0000","last_clean":"2024-02-05T07:41:24.825082+0000","last_became_active":"2024-02-05T07:36:32.746731+0000","last_became_peered":"2024-02-05T07:36:32.746731+0000","last_unstale":"2024-02-05T07:41:24.825082+0000","last_undegraded":"2024-02-05T07:41:24.825082+0000","last_fullsized":"2024-02-05T07:41:24.825082+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:44.381300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:37:44.381300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":724,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.821961+0000","last_change":"2024-02-05T07:38:09.473135+0000","last_active":"2024-02-05T07:41:24.821961+0000","last_peered":"2024-02-05T07:41:24.821961+0000","last_clean":"2024-02-05T07:41:24.821961+0000","last_became_active":"2024-02-05T07:38:09.472733+0000","last_became_peered":"2024-02-05T07:38:09.472733+0000","last_unstale":"2024-02-05T07:41:24.821961+0000","last_undegraded":"2024-02-05T07:41:24.821961+0000","last_fullsized":"2024-02-05T07:41:24.821961+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:45.170539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:45.170539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":914,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.824435+0000","last_change":"2024-02-05T07:38:58.711149+0000","last_active":"2024-02-05T07:41:24.824435+0000","last_peered":"2024-02-05T07:41:24.824435+0000","last_clean":"2024-02-05T07:41:24.824435+0000","last_became_active":"2024-02-05T07:30:40.493086+0000","last_became_peered":"2024-02-05T07:30:40.493086+0000","last_unstale":"2024-02-05T07:41:24.824435+0000","last_undegraded":"2024-02-05T07:41:24.824435+0000","last_fullsized":"2024-02-05T07:41:24.824435+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:58.711041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:38:58.711041+0000","last_clean_scrub_stamp":"2024-02-05T07:38:58.711041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"615'25","reported_seq":1099,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822324+0000","last_change":"2024-02-05T07:39:43.728715+0000","last_active":"2024-02-05T07:41:24.822324+0000","last_peered":"2024-02-05T07:41:24.822324+0000","last_clean":"2024-02-05T07:41:24.822324+0000","last_became_active":"2024-02-05T07:38:09.470878+0000","last_became_peered":"2024-02-05T07:38:09.470878+0000","last_unstale":"2024-02-05T07:41:24.822324+0000","last_undegraded":"2024-02-05T07:41:24.822324+0000","last_fullsized":"2024-02-05T07:41:24.822324+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'25","last_scrub_stamp":"2024-02-05T07:39:43.728551+0000","last_deep_scrub":"450'19","last_deep_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_clean_scrub_stamp":"2024-02-05T07:39:43.728551+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":343,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.785813+0000","last_change":"2024-02-05T07:38:47.508209+0000","last_active":"2024-02-05T07:41:29.785813+0000","last_peered":"2024-02-05T07:41:29.785813+0000","last_clean":"2024-02-05T07:41:29.785813+0000","last_became_active":"2024-02-05T07:36:32.734502+0000","last_became_peered":"2024-02-05T07:36:32.734502+0000","last_unstale":"2024-02-05T07:41:29.785813+0000","last_undegraded":"2024-02-05T07:41:29.785813+0000","last_fullsized":"2024-02-05T07:41:29.785813+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:38:47.508088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:38:47.508088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":343,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-02-05T07:41:29.783940+0000","last_change":"2024-02-05T07:37:50.267485+0000","last_active":"2024-02-05T07:41:29.783940+0000","last_peered":"2024-02-05T07:41:29.783940+0000","last_clean":"2024-02-05T07:41:29.783940+0000","last_became_active":"2024-02-05T07:36:32.708917+0000","last_became_peered":"2024-02-05T07:36:32.708917+0000","last_unstale":"2024-02-05T07:41:29.783940+0000","last_undegraded":"2024-02-05T07:41:29.783940+0000","last_fullsized":"2024-02-05T07:41:29.783940+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:37:50.267364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:37:50.267364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":858,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-02-05T07:41:24.822371+0000","last_change":"2024-02-05T07:41:22.800223+0000","last_active":"2024-02-05T07:41:24.822371+0000","last_peered":"2024-02-05T07:41:24.822371+0000","last_clean":"2024-02-05T07:41:24.822371+0000","last_became_active":"2024-02-05T07:38:09.480027+0000","last_became_peered":"2024-02-05T07:38:09.480027+0000","last_unstale":"2024-02-05T07:41:24.822371+0000","last_undegraded":"2024-02-05T07:41:24.822371+0000","last_fullsized":"2024-02-05T07:41:24.822371+0000","mapping_epoch":588,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:22.800142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:41:22.800142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:42:32.708944+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":654,"num_read_kb":649,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":4552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1526,"num_read_kb":1283,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":10072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":503808,"data_stored":13460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1052,"ondisk_log_size":1052,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":257698038037,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":301,"seq":1292785156254,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":794624,"kb_used_data":4152,"kb_used_omap":5,"kb_used_meta":790458,"kb_avail":92950528,"statfs":{"total":95995035648,"available":95181340672,"internally_reserved":0,"allocated":4251648,"data_stored":2701126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":809429509},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.94899999999999995,"15min":1.2410000000000001},"min":{"1min":0.254,"5min":0.19600000000000001,"15min":0.192},"max":{"1min":5.1959999999999997,"5min":7.0990000000000002,"15min":14.218},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.98499999999999999,"15min":1.2989999999999999},"min":{"1min":0.23899999999999999,"5min":0.23200000000000001,"15min":0.22700000000000001},"max":{"1min":5.7460000000000004,"5min":5.7460000000000004,"15min":14.175000000000001},"last":0.65400000000000003}]},{"osd":1,"last update":"Mon Feb 5 07:40:48 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":2.3300000000000001,"15min":2.0230000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.217},"max":{"1min":5.4290000000000003,"5min":56.325000000000003,"15min":56.325000000000003},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.2769999999999999,"5min":2.399,"15min":2.4089999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.7149999999999999,"5min":60.115000000000002,"15min":61.103000000000002},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Feb 5 07:40:48 2024","interfaces":[{"interface":"back","average":{"1min":1.927,"5min":1.9379999999999999,"15min":1.9490000000000001},"min":{"1min":0.317,"5min":0.311,"15min":0.249},"max":{"1min":7.4009999999999998,"5min":46.927,"15min":46.927},"last":0.628},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.653,"15min":2.0529999999999999},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.3140000000000001,"5min":46.506,"15min":46.506},"last":0.68000000000000005}]},{"osd":3,"last update":"Mon Feb 5 07:40:48 2024","interfaces":[{"interface":"back","average":{"1min":3.625,"5min":2.3029999999999999,"15min":2.4780000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":48.600999999999999,"5min":56.271000000000001,"15min":56.271000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":4.1130000000000004,"5min":2.5419999999999998,"15min":2.645},"min":{"1min":0.31900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":49.828000000000003,"5min":56.235999999999997,"15min":56.235999999999997},"last":0.69399999999999995}]},{"osd":4,"last update":"Mon Feb 5 07:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.42199999999999999,"15min":1.0569999999999999},"min":{"1min":0.29399999999999998,"5min":0.252,"15min":0.224},"max":{"1min":0.625,"5min":1.3149999999999999,"15min":151.02099999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.51800000000000002,"15min":1.0860000000000001},"min":{"1min":0.308,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55000000000000004,"5min":1.907,"15min":149.21799999999999},"last":0.503}]},{"osd":5,"last update":"Mon Feb 5 07:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.34999999999999998,"5min":0.30599999999999999,"15min":0.26800000000000002},"max":{"1min":0.58199999999999996,"5min":2.431,"15min":2.8919999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46700000000000003,"15min":0.50700000000000001},"min":{"1min":0.36699999999999999,"5min":0.28499999999999998,"15min":0.28100000000000003},"max":{"1min":0.56499999999999995,"5min":0.97099999999999997,"15min":1.526},"last":0.48699999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.48899999999999999,"15min":1.966},"min":{"1min":0.315,"5min":0.315,"15min":0.28000000000000003},"max":{"1min":0.59999999999999998,"5min":2.0550000000000002,"15min":84.158000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.48099999999999998,"15min":2.0960000000000001},"min":{"1min":0.34399999999999997,"5min":0.317,"15min":0.312},"max":{"1min":0.66100000000000003,"5min":1.097,"15min":85.018000000000001},"last":0.54800000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575155,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1153060,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":1149438,"kb_avail":92592092,"statfs":{"total":95995035648,"available":94814302208,"internally_reserved":0,"allocated":3694592,"data_stored":2117773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1177024965},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:41:05 2024","interfaces":[{"interface":"back","average":{"1min":1.9910000000000001,"5min":2.0840000000000001,"15min":2.0049999999999999},"min":{"1min":0.21299999999999999,"5min":0.192,"15min":0.16800000000000001},"max":{"1min":8.4269999999999996,"5min":15.327,"15min":29.449999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.95,"5min":2.218,"15min":2.2839999999999998},"min":{"1min":0.48099999999999998,"5min":0.28299999999999997,"15min":0.20899999999999999},"max":{"1min":4.944,"5min":8.3930000000000007,"15min":29.268999999999998},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:40:35 2024","interfaces":[{"interface":"back","average":{"1min":2.6920000000000002,"5min":3.4300000000000002,"15min":2.3130000000000002},"min":{"1min":0.373,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":6.7949999999999999,"5min":42.823999999999998,"15min":42.823999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.6819999999999999,"5min":2.843,"15min":2.0939999999999999},"min":{"1min":0.36199999999999999,"5min":0.26500000000000001,"15min":0.20699999999999999},"max":{"1min":3.984,"5min":42.426000000000002,"15min":42.426000000000002},"last":0.27100000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:40:39 2024","interfaces":[{"interface":"back","average":{"1min":2.3420000000000001,"5min":2.177,"15min":2.2109999999999999},"min":{"1min":0.39600000000000002,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":7.9009999999999998,"5min":9.5519999999999996,"15min":46.777000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.756,"5min":1.9570000000000001,"15min":2.0550000000000002},"min":{"1min":0.38700000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":7.3899999999999997,"5min":9.5419999999999998,"15min":50.164999999999999},"last":0.315}]},{"osd":4,"last update":"Mon Feb 5 07:41:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.0680000000000001,"15min":1.9139999999999999},"min":{"1min":0.40600000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.5739999999999998,"5min":6.7329999999999997,"15min":89.409000000000006},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.4950000000000001,"15min":1.9350000000000001},"min":{"1min":0.437,"5min":0.29399999999999998,"15min":0.28199999999999997},"max":{"1min":5.226,"5min":14.629,"15min":89.915000000000006},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0509999999999999,"15min":1.7769999999999999},"min":{"1min":0.42099999999999999,"5min":0.32200000000000001,"15min":0.309},"max":{"1min":3.9580000000000002,"5min":6.7610000000000001,"15min":106.672},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.835,"15min":1.998},"min":{"1min":0.53300000000000003,"5min":0.34799999999999998,"15min":0.27600000000000002},"max":{"1min":7.8250000000000002,"5min":14.613,"15min":107.348},"last":0.47299999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.3520000000000001,"15min":1.361},"min":{"1min":0.36699999999999999,"5min":0.314,"15min":0.29199999999999998},"max":{"1min":5.7990000000000004,"5min":12.685,"15min":12.685},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1319999999999999,"15min":1.49},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":3.9300000000000002,"5min":6.8179999999999996,"15min":8.923},"last":0.53800000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.9530000000000001,"5min":1.71,"15min":1.835},"min":{"1min":0.436,"5min":0.36599999999999999,"15min":0.318},"max":{"1min":7.8719999999999999,"5min":8.2439999999999998,"15min":46.351999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.405,"15min":1.387},"min":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.34000000000000002},"max":{"1min":5.8710000000000004,"5min":12.781000000000001,"15min":12.781000000000001},"last":0.45800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738679,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1127188,"kb_used_data":3912,"kb_used_omap":1,"kb_used_meta":1123262,"kb_avail":92617964,"statfs":{"total":95995035648,"available":94840795136,"internally_reserved":0,"allocated":4005888,"data_stored":2438195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1150220741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:40:53 2024","interfaces":[{"interface":"back","average":{"1min":2.2799999999999998,"5min":2.9940000000000002,"15min":2.621},"min":{"1min":0.25,"5min":0.214,"15min":0.16500000000000001},"max":{"1min":7.5090000000000003,"5min":65.799000000000007,"15min":65.799000000000007},"last":0.39800000000000002},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.5219999999999998,"15min":2.403},"min":{"1min":0.39900000000000002,"5min":0.218,"15min":0.192},"max":{"1min":7.5640000000000001,"5min":65.093999999999994,"15min":65.093999999999994},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:40:24 2024","interfaces":[{"interface":"back","average":{"1min":2.032,"5min":2.5619999999999998,"15min":2.044},"min":{"1min":0.24299999999999999,"5min":0.22900000000000001,"15min":0.191},"max":{"1min":5.5949999999999998,"5min":47.137999999999998,"15min":47.137999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.379,"5min":2.2930000000000001,"15min":1.897},"min":{"1min":0.25900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.3440000000000003,"5min":47.170999999999999,"15min":47.170999999999999},"last":0.317}]},{"osd":3,"last update":"Mon Feb 5 07:40:31 2024","interfaces":[{"interface":"back","average":{"1min":2.883,"5min":2.0579999999999998,"15min":1.823},"min":{"1min":0.433,"5min":0.25600000000000001,"15min":0.21299999999999999},"max":{"1min":22.129000000000001,"5min":26.43,"15min":29.629000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":2.6949999999999998,"5min":2.871,"15min":2.2930000000000001},"min":{"1min":0.435,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":22.155999999999999,"5min":25.337,"15min":29.463999999999999},"last":0.45500000000000002}]},{"osd":4,"last update":"Mon Feb 5 07:40:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.835,"15min":1.9510000000000001},"min":{"1min":0.34300000000000003,"5min":0.308,"15min":0.26700000000000002},"max":{"1min":7.476,"5min":12.718999999999999,"15min":50.545000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.292,"15min":1.78},"min":{"1min":0.45700000000000002,"5min":0.28000000000000003,"15min":0.27100000000000002},"max":{"1min":7.3959999999999999,"5min":7.3959999999999999,"15min":50.310000000000002},"last":0.38500000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.157,"15min":1.419},"min":{"1min":0.374,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":5.3840000000000003,"5min":5.3840000000000003,"15min":95.772999999999996},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.327,"5min":1.268,"15min":1.546},"min":{"1min":0.372,"5min":0.32900000000000001,"15min":0.28199999999999997},"max":{"1min":5.5810000000000004,"5min":5.5810000000000004,"15min":96.028000000000006},"last":0.434}]},{"osd":6,"last update":"Mon Feb 5 07:40:49 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":3.0619999999999998,"15min":1.8979999999999999},"min":{"1min":0.312,"5min":0.27800000000000002,"15min":0.26300000000000001},"max":{"1min":141.59899999999999,"5min":141.59899999999999,"15min":141.59899999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":7.5549999999999997,"5min":2.5289999999999999,"15min":1.8120000000000001},"min":{"1min":0.39900000000000002,"5min":0.376,"15min":0.35099999999999998},"max":{"1min":141.39599999999999,"5min":141.39599999999999,"15min":141.39599999999999},"last":0.53000000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:41:13 2024","interfaces":[{"interface":"back","average":{"1min":1.7490000000000001,"5min":1.8480000000000001,"15min":1.869},"min":{"1min":0.441,"5min":0.33500000000000002,"15min":0.28599999999999998},"max":{"1min":7.7190000000000003,"5min":12.766999999999999,"15min":43.012},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.774,"5min":1.4630000000000001,"15min":1.714},"min":{"1min":0.58499999999999996,"5min":0.41699999999999998,"15min":0.309},"max":{"1min":7.883,"5min":7.883,"15min":43.512},"last":0.56100000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411630,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":995728,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92749424,"statfs":{"total":95995035648,"available":94975410176,"internally_reserved":0,"allocated":3936256,"data_stored":2380015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1015675333},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:41:30 2024","interfaces":[{"interface":"back","average":{"1min":2.202,"5min":1.4350000000000001,"15min":2.2639999999999998},"min":{"1min":0.23899999999999999,"5min":0.21099999999999999,"15min":0.19600000000000001},"max":{"1min":17.398,"5min":17.398,"15min":92.638000000000005},"last":0.36399999999999999},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.5720000000000001,"15min":2.496},"min":{"1min":0.24399999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.8869999999999996,"5min":11.583,"15min":93.481999999999999},"last":0.28899999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:41:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0009999999999999,"5min":2.028,"15min":3.3759999999999999},"min":{"1min":0.27300000000000002,"5min":0.21099999999999999,"15min":0.17899999999999999},"max":{"1min":18.135999999999999,"5min":18.135999999999999,"15min":110.917},"last":0.39000000000000001},{"interface":"front","average":{"1min":1.627,"5min":1.833,"15min":2.762},"min":{"1min":0.27500000000000002,"5min":0.214,"15min":0.20300000000000001},"max":{"1min":6.1740000000000004,"5min":7.9509999999999996,"15min":93.004000000000005},"last":0.45600000000000002}]},{"osd":3,"last update":"Mon Feb 5 07:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.722,"5min":3.609,"15min":2.9300000000000002},"min":{"1min":0.29199999999999998,"5min":0.214,"15min":0.214},"max":{"1min":5.3339999999999996,"5min":110.773,"15min":110.773},"last":0.621},{"interface":"front","average":{"1min":2.532,"5min":2.1509999999999998,"15min":2.2850000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":5.2670000000000003,"5min":47.368000000000002,"15min":63.521000000000001},"last":0.40600000000000003}]},{"osd":4,"last update":"Mon Feb 5 07:41:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.869,"15min":2.4020000000000001},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.24299999999999999},"max":{"1min":3.3719999999999999,"5min":46.521999999999998,"15min":168.11699999999999},"last":0.441},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.4419999999999999,"15min":2.1659999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.27200000000000002},"max":{"1min":2.629,"5min":40.329000000000001,"15min":168.589},"last":0.55900000000000005}]},{"osd":5,"last update":"Mon Feb 5 07:41:30 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.1919999999999999,"15min":1.242},"min":{"1min":0.437,"5min":0.35899999999999999,"15min":0.315},"max":{"1min":17.347999999999999,"5min":17.347999999999999,"15min":39.923000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.5620000000000001,"15min":1.71},"min":{"1min":0.36399999999999999,"5min":0.36199999999999999,"15min":0.30499999999999999},"max":{"1min":5.601,"5min":5.601,"15min":47.761000000000003},"last":0.46999999999999997}]},{"osd":6,"last update":"Mon Feb 5 07:40:52 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":2.762,"15min":1.796},"min":{"1min":0.40100000000000002,"5min":0.309,"15min":0.28799999999999998},"max":{"1min":3.1360000000000001,"5min":47.726999999999997,"15min":47.726999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.84199999999999997,"5min":1.496,"15min":1.3169999999999999},"min":{"1min":0.34599999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.6589999999999998,"5min":40.872999999999998,"15min":40.872999999999998},"last":0.72099999999999997}]},{"osd":7,"last update":"Mon Feb 5 07:41:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":0.99299999999999999,"15min":1.4199999999999999},"min":{"1min":0.39200000000000002,"5min":0.35699999999999998,"15min":0.30499999999999999},"max":{"1min":4.0510000000000002,"5min":4.0510000000000002,"15min":37.573999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.337,"5min":1.607,"15min":2.1720000000000002},"min":{"1min":0.42299999999999999,"5min":0.40100000000000002,"15min":0.33700000000000002},"max":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":110.86799999999999},"last":0.70099999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149992,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1122864,"kb_used_data":4196,"kb_used_omap":1,"kb_used_meta":1118654,"kb_avail":92622288,"statfs":{"total":95995035648,"available":94845222912,"internally_reserved":0,"allocated":4296704,"data_stored":2701796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1145502151},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:40:53 2024","interfaces":[{"interface":"back","average":{"1min":2.5070000000000001,"5min":2.278,"15min":1.851},"min":{"1min":0.49099999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":12.092000000000001,"5min":48.688000000000002,"15min":48.688000000000002},"last":0.33900000000000002},{"interface":"front","average":{"1min":2.274,"5min":2.5409999999999999,"15min":2.4300000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.185},"max":{"1min":6.3499999999999996,"5min":48.573,"15min":99.989999999999995},"last":0.246}]},{"osd":1,"last update":"Mon Feb 5 07:40:53 2024","interfaces":[{"interface":"back","average":{"1min":2.762,"5min":3.0459999999999998,"15min":2.8570000000000002},"min":{"1min":0.28199999999999997,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":10.106999999999999,"5min":61.941000000000003,"15min":102.20399999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":3.681,"5min":2.8940000000000001,"15min":2.8159999999999998},"min":{"1min":0.60499999999999998,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.539,"5min":61.902000000000001,"15min":102.286},"last":0.28699999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:40:53 2024","interfaces":[{"interface":"back","average":{"1min":2.6040000000000001,"5min":2.0009999999999999,"15min":2.6179999999999999},"min":{"1min":0.495,"5min":0.23999999999999999,"15min":0.22700000000000001},"max":{"1min":12.6,"5min":12.6,"15min":145.94999999999999},"last":0.22600000000000001},{"interface":"front","average":{"1min":2.262,"5min":1.925,"15min":2.6150000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20200000000000001},"max":{"1min":9.7370000000000001,"5min":10.695,"15min":146.006},"last":0.32400000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:41:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.381,"15min":1.8540000000000001},"min":{"1min":0.28899999999999998,"5min":0.28799999999999998,"15min":0.27000000000000002},"max":{"1min":5.992,"5min":9.2639999999999993,"15min":100.024},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.113,"5min":1.7450000000000001,"15min":2.1160000000000001},"min":{"1min":0.28899999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.016,"5min":16.253,"15min":99.859999999999999},"last":0.59699999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:40:27 2024","interfaces":[{"interface":"back","average":{"1min":2.456,"5min":1.49,"15min":1.5469999999999999},"min":{"1min":0.313,"5min":0.308,"15min":0.308},"max":{"1min":11.507999999999999,"5min":11.507999999999999,"15min":99.911000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.363,"15min":1.591},"min":{"1min":0.34100000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.2789999999999999,"5min":8.4350000000000005,"15min":100.053},"last":0.498}]},{"osd":6,"last update":"Mon Feb 5 07:40:53 2024","interfaces":[{"interface":"back","average":{"1min":2.488,"5min":1.5600000000000001,"15min":1.7549999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":11.563000000000001,"5min":11.563000000000001,"15min":99.807000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.4610000000000001,"15min":1.7809999999999999},"min":{"1min":0.374,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.2919999999999998,"5min":10.209,"15min":101.489},"last":0.68100000000000005}]},{"osd":7,"last update":"Mon Feb 5 07:41:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.573,"15min":2.4199999999999999},"min":{"1min":0.41899999999999998,"5min":0.38200000000000001,"15min":0.29499999999999998},"max":{"1min":4.0609999999999999,"5min":8.4329999999999998,"15min":99.893000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":2.2799999999999998,"5min":1.488,"15min":2.375},"min":{"1min":0.47599999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":11.457000000000001,"5min":11.457000000000001,"15min":99.879000000000005},"last":0.54700000000000004}]}]},{"osd":4,"up_from":0,"seq":137438953764,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":176093659423,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:41:32.374 INFO:tasks.thrashosds.thrasher:Setting 5.6 to [3, 0, 1, 2, 5, 4] 2024-02-05T07:41:32.374 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '5.6', '3', '0', '1', '2', '5', '4'] 2024-02-05T07:41:32.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pg-upmap-items 5.6 3 0 1 2 5 4 2024-02-05T07:41:32.621 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:32.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T07:41:32.768 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T07:41:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:41:33.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:32 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:33.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:32 smithi090 ceph-mon[33213]: pgmap v1276: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:41:33.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:32 smithi090 ceph-mon[33213]: from='client.16008 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:41:33.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:32 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:33.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:32 smithi007 ceph-mon[29098]: pgmap v1276: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:41:33.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:32 smithi007 ceph-mon[29098]: from='client.16008 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:41:33.033 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:32 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:33.033 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:32 smithi007 ceph-mon[37158]: pgmap v1276: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:41:33.033 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:32 smithi007 ceph-mon[37158]: from='client.16008 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:41:33.344 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:33.770 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:41:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:41:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:41:34.639 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:41:34.639 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:41:34.639 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-02-05T07:41:34.639 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:41:34.961 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:34 smithi090 ceph-mon[33213]: pgmap v1277: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:41:34.961 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:34 smithi090 ceph-mon[33213]: osdmap e698: 8 total, 8 up, 5 in 2024-02-05T07:41:35.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:34 smithi007 ceph-mon[29098]: pgmap v1277: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:41:35.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:34 smithi007 ceph-mon[29098]: osdmap e698: 8 total, 8 up, 5 in 2024-02-05T07:41:35.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:34 smithi007 ceph-mon[37158]: pgmap v1277: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:41:35.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:34 smithi007 ceph-mon[37158]: osdmap e698: 8 total, 8 up, 5 in 2024-02-05T07:41:35.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T07:41:35.486 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:41:35.801 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-02-05T07:41:35.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:35 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2899601788' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:41:35.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:35 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:41:35.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:35 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2899601788' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:41:35.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:35 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:41:35.932 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:35 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2899601788' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:41:35.932 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:35 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T07:41:36.490 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T07:41:36.738 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:36.814 INFO:teuthology.orchestra.run.smithi007.stderr:set 5.6 pg_upmap_items mapping to [3->0,1->2,5->4] 2024-02-05T07:41:37.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[37158]: pgmap v1279: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:41:37.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[37158]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-05T07:41:37.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:41:37.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:41:37.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[37158]: osdmap e699: 8 total, 8 up, 5 in 2024-02-05T07:41:37.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:41:37.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:41:37.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1386477210' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.6", "id": [3, 0, 1, 2, 5, 4]}]: dispatch 2024-02-05T07:41:37.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[29098]: pgmap v1279: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:41:37.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[29098]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-05T07:41:37.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:41:37.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:41:37.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[29098]: osdmap e699: 8 total, 8 up, 5 in 2024-02-05T07:41:37.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:41:37.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:41:37.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:36 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1386477210' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.6", "id": [3, 0, 1, 2, 5, 4]}]: dispatch 2024-02-05T07:41:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:36 smithi090 ceph-mon[33213]: pgmap v1279: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:41:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:36 smithi090 ceph-mon[33213]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-05T07:41:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:36 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:41:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:36 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T07:41:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:36 smithi090 ceph-mon[33213]: osdmap e699: 8 total, 8 up, 5 in 2024-02-05T07:41:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:36 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:41:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:36 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:41:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:36 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1386477210' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.6", "id": [3, 0, 1, 2, 5, 4]}]: dispatch 2024-02-05T07:41:37.604 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:41:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:41:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:41:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:41:37.604 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:41:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:37 smithi090 ceph-mon[33213]: 3.5 scrub starts 2024-02-05T07:41:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:37 smithi090 ceph-mon[33213]: 3.5 scrub ok 2024-02-05T07:41:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:37 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1386477210' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.6", "id": [3, 0, 1, 2, 5, 4]}]': finished 2024-02-05T07:41:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:37 smithi090 ceph-mon[33213]: osdmap e700: 8 total, 8 up, 5 in 2024-02-05T07:41:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:37 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:41:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:37 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:41:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:37 smithi090 ceph-mon[33213]: 5.2 scrub starts 2024-02-05T07:41:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:37 smithi090 ceph-mon[33213]: 5.2 scrub ok 2024-02-05T07:41:38.108 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[29098]: 3.5 scrub starts 2024-02-05T07:41:38.108 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[29098]: 3.5 scrub ok 2024-02-05T07:41:38.108 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1386477210' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.6", "id": [3, 0, 1, 2, 5, 4]}]': finished 2024-02-05T07:41:38.108 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[29098]: osdmap e700: 8 total, 8 up, 5 in 2024-02-05T07:41:38.108 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:41:38.108 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:41:38.109 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[29098]: 5.2 scrub starts 2024-02-05T07:41:38.109 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[29098]: 5.2 scrub ok 2024-02-05T07:41:38.109 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[37158]: 3.5 scrub starts 2024-02-05T07:41:38.109 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[37158]: 3.5 scrub ok 2024-02-05T07:41:38.109 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1386477210' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.6", "id": [3, 0, 1, 2, 5, 4]}]': finished 2024-02-05T07:41:38.109 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[37158]: osdmap e700: 8 total, 8 up, 5 in 2024-02-05T07:41:38.109 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:41:38.109 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:41:38.109 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[37158]: 5.2 scrub starts 2024-02-05T07:41:38.109 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:37 smithi007 ceph-mon[37158]: 5.2 scrub ok 2024-02-05T07:41:38.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T07:41:38.403 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:41:39.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:38 smithi007 ceph-mon[37158]: pgmap v1282: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:41:39.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:38 smithi007 ceph-mon[37158]: 4.b scrub starts 2024-02-05T07:41:39.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:38 smithi007 ceph-mon[37158]: 4.b scrub ok 2024-02-05T07:41:39.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:38 smithi007 ceph-mon[37158]: osdmap e701: 8 total, 8 up, 5 in 2024-02-05T07:41:39.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:38 smithi007 ceph-mon[29098]: pgmap v1282: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:41:39.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:38 smithi007 ceph-mon[29098]: 4.b scrub starts 2024-02-05T07:41:39.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:38 smithi007 ceph-mon[29098]: 4.b scrub ok 2024-02-05T07:41:39.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:38 smithi007 ceph-mon[29098]: osdmap e701: 8 total, 8 up, 5 in 2024-02-05T07:41:39.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:38 smithi090 ceph-mon[33213]: pgmap v1282: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:41:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:38 smithi090 ceph-mon[33213]: 4.b scrub starts 2024-02-05T07:41:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:38 smithi090 ceph-mon[33213]: 4.b scrub ok 2024-02-05T07:41:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:38 smithi090 ceph-mon[33213]: osdmap e701: 8 total, 8 up, 5 in 2024-02-05T07:41:39.719 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-02-05T07:41:39.954 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:39 smithi090 ceph-mon[33213]: 1.8 scrub starts 2024-02-05T07:41:39.954 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:39 smithi090 ceph-mon[33213]: 1.8 scrub ok 2024-02-05T07:41:39.954 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:39 smithi090 ceph-mon[33213]: 4.2 scrub starts 2024-02-05T07:41:39.954 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:39 smithi090 ceph-mon[33213]: 4.2 scrub ok 2024-02-05T07:41:39.955 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:39 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2662815740' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:41:39.955 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:39 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:41:39.955 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:39 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:41:39.955 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:39 smithi090 ceph-mon[33213]: osdmap e702: 8 total, 8 up, 5 in 2024-02-05T07:41:39.977 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:41:39.977 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-02-05T07:41:39.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-02-05T07:41:39.977 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-02-05T07:41:39.977 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:41:40.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[29098]: 1.8 scrub starts 2024-02-05T07:41:40.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[29098]: 1.8 scrub ok 2024-02-05T07:41:40.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[29098]: 4.2 scrub starts 2024-02-05T07:41:40.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[29098]: 4.2 scrub ok 2024-02-05T07:41:40.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2662815740' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:41:40.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:41:40.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:41:40.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[29098]: osdmap e702: 8 total, 8 up, 5 in 2024-02-05T07:41:40.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[37158]: 1.8 scrub starts 2024-02-05T07:41:40.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[37158]: 1.8 scrub ok 2024-02-05T07:41:40.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[37158]: 4.2 scrub starts 2024-02-05T07:41:40.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[37158]: 4.2 scrub ok 2024-02-05T07:41:40.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2662815740' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:41:40.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T07:41:40.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T07:41:40.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:39 smithi007 ceph-mon[37158]: osdmap e702: 8 total, 8 up, 5 in 2024-02-05T07:41:40.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T07:41:40.807 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:41:40.929 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:40 smithi090 ceph-mon[33213]: pgmap v1284: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:41:40.929 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:40 smithi090 ceph-mon[33213]: 3.16 scrub starts 2024-02-05T07:41:40.929 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:40 smithi090 ceph-mon[33213]: 3.16 scrub ok 2024-02-05T07:41:40.929 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:40 smithi090 ceph-mon[33213]: 4.5 scrub starts 2024-02-05T07:41:40.929 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:40 smithi090 ceph-mon[33213]: 4.5 scrub ok 2024-02-05T07:41:41.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:40 smithi007 ceph-mon[29098]: pgmap v1284: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:41:41.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:40 smithi007 ceph-mon[29098]: 3.16 scrub starts 2024-02-05T07:41:41.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:40 smithi007 ceph-mon[29098]: 3.16 scrub ok 2024-02-05T07:41:41.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:40 smithi007 ceph-mon[29098]: 4.5 scrub starts 2024-02-05T07:41:41.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:40 smithi007 ceph-mon[29098]: 4.5 scrub ok 2024-02-05T07:41:41.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:40 smithi007 ceph-mon[37158]: pgmap v1284: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:41:41.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:40 smithi007 ceph-mon[37158]: 3.16 scrub starts 2024-02-05T07:41:41.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:40 smithi007 ceph-mon[37158]: 3.16 scrub ok 2024-02-05T07:41:41.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:40 smithi007 ceph-mon[37158]: 4.5 scrub starts 2024-02-05T07:41:41.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:40 smithi007 ceph-mon[37158]: 4.5 scrub ok 2024-02-05T07:41:41.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:41:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:41:42.136 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:41 smithi090 ceph-mon[33213]: 5.a deep-scrub starts 2024-02-05T07:41:42.136 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:41 smithi090 ceph-mon[33213]: 5.a deep-scrub ok 2024-02-05T07:41:42.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:41 smithi007 ceph-mon[29098]: 5.a deep-scrub starts 2024-02-05T07:41:42.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:41 smithi007 ceph-mon[29098]: 5.a deep-scrub ok 2024-02-05T07:41:42.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:41 smithi007 ceph-mon[37158]: 5.a deep-scrub starts 2024-02-05T07:41:42.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:41 smithi007 ceph-mon[37158]: 5.a deep-scrub ok 2024-02-05T07:41:42.371 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:41:42.371 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:41:42.371 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-02-05T07:41:42.371 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:41:42.581 INFO:tasks.thrashosds.thrasher:inning osd 2024-02-05T07:41:42.582 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-02-05T07:41:42.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd in 7 2024-02-05T07:41:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:42.919 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:42 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:42.919 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:42 smithi090 ceph-mon[33213]: pgmap v1286: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 903 B/s rd, 0 op/s 2024-02-05T07:41:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:42 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:42.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:42 smithi007 ceph-mon[29098]: pgmap v1286: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 903 B/s rd, 0 op/s 2024-02-05T07:41:42.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:42 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:42.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:42 smithi007 ceph-mon[37158]: pgmap v1286: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 903 B/s rd, 0 op/s 2024-02-05T07:41:42.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T07:41:43.184 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:41:43.828 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:41:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:41:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:41:43.977 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:43 smithi090 ceph-mon[33213]: 4.f scrub starts 2024-02-05T07:41:43.977 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:43 smithi090 ceph-mon[33213]: 4.f scrub ok 2024-02-05T07:41:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:43 smithi007 ceph-mon[29098]: 4.f scrub starts 2024-02-05T07:41:44.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:43 smithi007 ceph-mon[29098]: 4.f scrub ok 2024-02-05T07:41:44.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:43 smithi007 ceph-mon[37158]: 4.f scrub starts 2024-02-05T07:41:44.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:43 smithi007 ceph-mon[37158]: 4.f scrub ok 2024-02-05T07:41:44.801 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:41:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:41:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:41:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:41:44.801 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:41:44.849 INFO:teuthology.orchestra.run.smithi007.stderr:marked in osd.7. 2024-02-05T07:41:45.120 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:44 smithi090 ceph-mon[33213]: pgmap v1287: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 769 B/s rd, 0 op/s 2024-02-05T07:41:45.120 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:44 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1917894300' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-05T07:41:45.120 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:44 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-05T07:41:45.164 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:44 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a[29094]: 2024-02-05T07:41:44.680+0000 7fe3564d3700 -1 mon.a@0(leader).osd e702 definitely_dead 0 2024-02-05T07:41:45.165 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:44 smithi007 ceph-mon[29098]: pgmap v1287: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 769 B/s rd, 0 op/s 2024-02-05T07:41:45.165 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:44 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1917894300' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-05T07:41:45.165 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:44 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-05T07:41:45.165 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:44 smithi007 ceph-mon[37158]: pgmap v1287: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 769 B/s rd, 0 op/s 2024-02-05T07:41:45.165 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:44 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1917894300' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-05T07:41:45.165 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:44 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-05T07:41:45.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T07:41:45.581 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-02-05T07:41:45.581 INFO:tasks.thrashosds.thrasher:inning osd 2024-02-05T07:41:45.581 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-02-05T07:41:45.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd in 5 2024-02-05T07:41:45.637 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:41:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:45.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:45 smithi007 ceph-mon[37158]: 1.14 scrub starts 2024-02-05T07:41:45.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:45 smithi007 ceph-mon[37158]: 1.14 scrub ok 2024-02-05T07:41:45.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:45 smithi007 ceph-mon[37158]: 4.10 scrub starts 2024-02-05T07:41:45.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:45 smithi007 ceph-mon[37158]: 4.10 scrub ok 2024-02-05T07:41:45.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:45 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-05T07:41:45.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:45 smithi007 ceph-mon[37158]: osdmap e703: 8 total, 8 up, 6 in 2024-02-05T07:41:45.858 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:45 smithi007 ceph-mon[37158]: osdmap e704: 8 total, 8 up, 6 in 2024-02-05T07:41:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:45 smithi090 ceph-mon[33213]: 1.14 scrub starts 2024-02-05T07:41:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:45 smithi090 ceph-mon[33213]: 1.14 scrub ok 2024-02-05T07:41:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:45 smithi090 ceph-mon[33213]: 4.10 scrub starts 2024-02-05T07:41:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:45 smithi090 ceph-mon[33213]: 4.10 scrub ok 2024-02-05T07:41:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:45 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-05T07:41:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:45 smithi090 ceph-mon[33213]: osdmap e703: 8 total, 8 up, 6 in 2024-02-05T07:41:46.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:45 smithi090 ceph-mon[33213]: osdmap e704: 8 total, 8 up, 6 in 2024-02-05T07:41:46.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:45 smithi007 ceph-mon[29098]: 1.14 scrub starts 2024-02-05T07:41:46.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:45 smithi007 ceph-mon[29098]: 1.14 scrub ok 2024-02-05T07:41:46.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:45 smithi007 ceph-mon[29098]: 4.10 scrub starts 2024-02-05T07:41:46.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:45 smithi007 ceph-mon[29098]: 4.10 scrub ok 2024-02-05T07:41:46.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:45 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-05T07:41:46.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:45 smithi007 ceph-mon[29098]: osdmap e703: 8 total, 8 up, 6 in 2024-02-05T07:41:46.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:45 smithi007 ceph-mon[29098]: osdmap e704: 8 total, 8 up, 6 in 2024-02-05T07:41:47.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:46 smithi090 ceph-mon[33213]: pgmap v1289: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 669 B/s rd, 0 op/s 2024-02-05T07:41:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:46 smithi090 ceph-mon[33213]: 5.16 scrub starts 2024-02-05T07:41:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:46 smithi090 ceph-mon[33213]: 5.16 scrub ok 2024-02-05T07:41:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:46 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:41:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:41:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:41:47.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:46 smithi090 ceph-mon[33213]: osdmap e705: 8 total, 8 up, 6 in 2024-02-05T07:41:47.123 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:46 smithi007 ceph-mon[29098]: pgmap v1289: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 669 B/s rd, 0 op/s 2024-02-05T07:41:47.123 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:46 smithi007 ceph-mon[29098]: 5.16 scrub starts 2024-02-05T07:41:47.123 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:46 smithi007 ceph-mon[29098]: 5.16 scrub ok 2024-02-05T07:41:47.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:46 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:41:47.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:41:47.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:41:47.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:46 smithi007 ceph-mon[29098]: osdmap e705: 8 total, 8 up, 6 in 2024-02-05T07:41:47.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:46 smithi007 ceph-mon[37158]: pgmap v1289: 152 pgs: 152 active+clean; 582 KiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 669 B/s rd, 0 op/s 2024-02-05T07:41:47.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:46 smithi007 ceph-mon[37158]: 5.16 scrub starts 2024-02-05T07:41:47.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:46 smithi007 ceph-mon[37158]: 5.16 scrub ok 2024-02-05T07:41:47.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:46 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:41:47.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:41:47.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:41:47.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:46 smithi007 ceph-mon[37158]: osdmap e705: 8 total, 8 up, 6 in 2024-02-05T07:41:47.311 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:41:47.311 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-02-05T07:41:47.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-02-05T07:41:47.311 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:41:47.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.311 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24356.0:4594 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 93862608744832 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-02-05T07:41:47.311 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:44.830699+0000", 2024-02-05T07:41:47.312 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.4781731950000001, 2024-02-05T07:41:47.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0784568219999999, 2024-02-05T07:41:47.312 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.312 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.312 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:47.312 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24356", 2024-02-05T07:41:47.312 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2716729371", 2024-02-05T07:41:47.312 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4594 2024-02-05T07:41:47.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.312 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830699+0000", 2024-02-05T07:41:47.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830699+0000", 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830701+0000", 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0320000000000002e-06 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830712+0000", 2024-02-05T07:41:47.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1542999999999999e-05 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830733+0000", 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0579999999999999e-05 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830744+0000", 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0946e-05 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.839626+0000", 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0088819209999999992 2024-02-05T07:41:47.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:47.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.839635+0000", 2024-02-05T07:41:47.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4409999999999999e-06 2024-02-05T07:41:47.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.881765+0000", 2024-02-05T07:41:47.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0421301940000001 2024-02-05T07:41:47.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-02-05T07:41:47.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.881890+0000", 2024-02-05T07:41:47.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012460100000000001 2024-02-05T07:41:47.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.907577+0000", 2024-02-05T07:41:47.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025687498 2024-02-05T07:41:47.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-02-05T07:41:47.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.907643+0000", 2024-02-05T07:41:47.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5143999999999998e-05 2024-02-05T07:41:47.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.907866+0000", 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022382700000000001 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.907914+0000", 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7969000000000001e-05 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.909155+0000", 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012411259999999999 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.14532.0:1358 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e703)", 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.372034+0000", 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.93683798, 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51612570499999999, 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.14532", 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3929198940", 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1358 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.372034+0000", 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.372034+0000", 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.372036+0000", 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5569999999999998e-06 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.372048+0000", 2024-02-05T07:41:47.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.204e-05 2024-02-05T07:41:47.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.372072+0000", 2024-02-05T07:41:47.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3754999999999999e-05 2024-02-05T07:41:47.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.372087+0000", 2024-02-05T07:41:47.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4341999999999999e-05 2024-02-05T07:41:47.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.372163+0000", 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6110000000000001e-05 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.372173+0000", 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0104e-05 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.884106+0000", 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51193328500000002 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.884233+0000", 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127205 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.888007+0000", 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003773392 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.888069+0000", 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1929999999999998e-05 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.888160+0000", 2024-02-05T07:41:47.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0984999999999997e-05 2024-02-05T07:41:47.323 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.323 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.323 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.323 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(4.5 704/703 [PushOp(4:a60f9269:lc::lua_package_allowlist:head, version: 75'4, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a60f9269:lc::lua_package_allowlist:head@75'4, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.323 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.879163+0000", 2024-02-05T07:41:47.323 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.4297090729999999, 2024-02-05T07:41:47.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035271899999999998, 2024-02-05T07:41:47.323 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.323 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.324 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.879163+0000", 2024-02-05T07:41:47.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.879163+0000", 2024-02-05T07:41:47.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.879163+0000", 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3399999999999998e-07 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.879187+0000", 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3808999999999999e-05 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.879211+0000", 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3907999999999999e-05 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.879234+0000", 2024-02-05T07:41:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3184e-05 2024-02-05T07:41:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.879328+0000", 2024-02-05T07:41:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4077000000000001e-05 2024-02-05T07:41:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.879344+0000", 2024-02-05T07:41:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5045000000000001e-05 2024-02-05T07:41:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.879515+0000", 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171962 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(5.d 704/703 [PushOp(5:b4812045:::notify.4:head, version: 363'10, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:b4812045:::notify.4:head@363'10, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.881585+0000", 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.427286638, 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036850099999999997, 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.327 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.881585+0000", 2024-02-05T07:41:47.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.881585+0000", 2024-02-05T07:41:47.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.881586+0000", 2024-02-05T07:41:47.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9699999999999996e-07 2024-02-05T07:41:47.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.881600+0000", 2024-02-05T07:41:47.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4379000000000001e-05 2024-02-05T07:41:47.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.881621+0000", 2024-02-05T07:41:47.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0948999999999999e-05 2024-02-05T07:41:47.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.881665+0000", 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4397999999999997e-05 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.881831+0000", 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016575800000000001 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.881839+0000", 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3810000000000001e-06 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.881954+0000", 2024-02-05T07:41:47.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115139 2024-02-05T07:41:47.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.331 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.331 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "pg_update_log_missing(4.5 epoch 704/703 rep_tid 161840 entries 704'45 (0'0) error 4:a60f9269:lc::lua_package_allowlist:head by client.24356.0:4595 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-02-05T07:41:47.331 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.882035+0000", 2024-02-05T07:41:47.331 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.4268369750000001, 2024-02-05T07:41:47.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000695394, 2024-02-05T07:41:47.331 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.331 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:41:47.331 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.882035+0000", 2024-02-05T07:41:47.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.882035+0000", 2024-02-05T07:41:47.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.882035+0000", 2024-02-05T07:41:47.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7599999999999997e-07 2024-02-05T07:41:47.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.882043+0000", 2024-02-05T07:41:47.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6590000000000007e-06 2024-02-05T07:41:47.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.882057+0000", 2024-02-05T07:41:47.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3645000000000001e-05 2024-02-05T07:41:47.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.882069+0000", 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2146e-05 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.882129+0000", 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0115999999999998e-05 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.882730+0000", 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060125200000000004 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.334 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.335 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "pg_update_log_missing(5.d epoch 704/703 rep_tid 178212 entries 704'14 (0'0) error 5:b4812045:::notify.4:head by client.24356.0:4591 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-02-05T07:41:47.335 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.884512+0000", 2024-02-05T07:41:47.335 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.4243601029999999, 2024-02-05T07:41:47.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042646900000000001, 2024-02-05T07:41:47.335 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.335 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:41:47.335 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.884512+0000", 2024-02-05T07:41:47.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.884512+0000", 2024-02-05T07:41:47.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.884512+0000", 2024-02-05T07:41:47.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3900000000000001e-07 2024-02-05T07:41:47.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.884518+0000", 2024-02-05T07:41:47.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6609999999999997e-06 2024-02-05T07:41:47.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.884529+0000", 2024-02-05T07:41:47.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1199000000000001e-05 2024-02-05T07:41:47.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.884541+0000", 2024-02-05T07:41:47.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2125e-05 2024-02-05T07:41:47.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.884608+0000", 2024-02-05T07:41:47.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6673000000000004e-05 2024-02-05T07:41:47.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.884938+0000", 2024-02-05T07:41:47.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033057200000000001 2024-02-05T07:41:47.339 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.339 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.339 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.340 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(4.5 704/703 [PushOp(4:a2ace8b3:::data_log.0:head, version: 72'1, data_included: [0~154], data_size: 154, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a2ace8b3:::data_log.0:head@72'1, size: 154, copy_subset: [(0, 154)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 154, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.340 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.896040+0000", 2024-02-05T07:41:47.340 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.4128313640000001, 2024-02-05T07:41:47.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000364705, 2024-02-05T07:41:47.340 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.340 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.340 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.896040+0000", 2024-02-05T07:41:47.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.896040+0000", 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.896041+0000", 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3300000000000002e-07 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.896049+0000", 2024-02-05T07:41:47.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7759999999999997e-06 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.896064+0000", 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5074e-05 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.896081+0000", 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6665999999999999e-05 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.896175+0000", 2024-02-05T07:41:47.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4560000000000003e-05 2024-02-05T07:41:47.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.896187+0000", 2024-02-05T07:41:47.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2167999999999999e-05 2024-02-05T07:41:47.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.896405+0000", 2024-02-05T07:41:47.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021792799999999999 2024-02-05T07:41:47.343 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.343 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.343 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.344 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(5.9 704/703 [PushOp(5:93e5b521:::notify.7:head, version: 89'9, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:93e5b521:::notify.7:head@89'9, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.344 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.907135+0000", 2024-02-05T07:41:47.344 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.4017366199999999, 2024-02-05T07:41:47.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039812400000000003, 2024-02-05T07:41:47.344 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.344 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.344 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.907135+0000", 2024-02-05T07:41:47.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.907135+0000", 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.907136+0000", 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6300000000000004e-07 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.907149+0000", 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2884999999999999e-05 2024-02-05T07:41:47.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.907163+0000", 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3932e-05 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.907173+0000", 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0558e-05 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.907227+0000", 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3183000000000002e-05 2024-02-05T07:41:47.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.907234+0000", 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9759999999999998e-06 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.907533+0000", 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029982700000000001 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(4.3 704/703 [PushOp(4:c623b173:gc::gc.16:head, version: 75'5, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:c623b173:gc::gc.16:head@75'5, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.347 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.909728+0000", 2024-02-05T07:41:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.3991439370000001, 2024-02-05T07:41:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000419961, 2024-02-05T07:41:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.909728+0000", 2024-02-05T07:41:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.909728+0000", 2024-02-05T07:41:47.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.909728+0000", 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8000000000000002e-07 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.909739+0000", 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0380000000000001e-05 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.909749+0000", 2024-02-05T07:41:47.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0402e-05 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.909763+0000", 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4207e-05 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.909818+0000", 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4506000000000003e-05 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.909827+0000", 2024-02-05T07:41:47.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5699999999999999e-06 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.910148+0000", 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032061600000000002 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(4.3 704/703 [PushOp(4:c31f41d7:gc::gc.10:head, version: 75'13, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:c31f41d7:gc::gc.10:head@75'13, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.913309+0000", 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.395563205, 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071654900000000003, 2024-02-05T07:41:47.351 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.352 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.352 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.913309+0000", 2024-02-05T07:41:47.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.913309+0000", 2024-02-05T07:41:47.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.913310+0000", 2024-02-05T07:41:47.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1829999999999999e-06 2024-02-05T07:41:47.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.913331+0000", 2024-02-05T07:41:47.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1483000000000001e-05 2024-02-05T07:41:47.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.913362+0000", 2024-02-05T07:41:47.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0470000000000001e-05 2024-02-05T07:41:47.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.913389+0000", 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7239000000000001e-05 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.913548+0000", 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015871799999999999 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.913570+0000", 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2633000000000002e-05 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.914025+0000", 2024-02-05T07:41:47.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045482299999999998 2024-02-05T07:41:47.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.355 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.355 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.f 704/703 [PushOp(3:f4c53578:::zonegroups_names.default:head, version: 614'4, data_included: [0~46], data_size: 46, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:f4c53578:::zonegroups_names.default:head@614'4, size: 46, copy_subset: [(0, 46)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 46, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.355 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.918478+0000", 2024-02-05T07:41:47.355 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.3903940100000001, 2024-02-05T07:41:47.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035804000000000001, 2024-02-05T07:41:47.355 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.355 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.356 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.918478+0000", 2024-02-05T07:41:47.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.918478+0000", 2024-02-05T07:41:47.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.918478+0000", 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0999999999999999e-07 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.918489+0000", 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0915e-05 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.918504+0000", 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5109999999999999e-05 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.918516+0000", 2024-02-05T07:41:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1481000000000001e-05 2024-02-05T07:41:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.918607+0000", 2024-02-05T07:41:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1456e-05 2024-02-05T07:41:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.918615+0000", 2024-02-05T07:41:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4359999999999996e-06 2024-02-05T07:41:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.918836+0000", 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221132 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(4.3 704/703 [PushOp(4:c3a682d4:reshard::reshard.0000000004:head, version: 614'31, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:c3a682d4:reshard::reshard.0000000004:head@614'31, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.920250+0000", 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.3886222209999999, 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038579700000000001, 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.359 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.920250+0000", 2024-02-05T07:41:47.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.920250+0000", 2024-02-05T07:41:47.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.920250+0000", 2024-02-05T07:41:47.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4099999999999999e-07 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.920260+0000", 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4690000000000003e-06 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.920274+0000", 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4423999999999999e-05 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.920285+0000", 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0959e-05 2024-02-05T07:41:47.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.920356+0000", 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1301000000000001e-05 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.920364+0000", 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9300000000000003e-06 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.920635+0000", 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027117299999999999 2024-02-05T07:41:47.362 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(4.3 704/703 [PushOp(4:c35966f0:::obj_delete_at_hint.0000000109:head, version: 615'39, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:c35966f0:::obj_delete_at_hint.0000000109:head@615'39, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.925564+0000", 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.3833073819999999, 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000387573, 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.925564+0000", 2024-02-05T07:41:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.925564+0000", 2024-02-05T07:41:47.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.925565+0000", 2024-02-05T07:41:47.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5499999999999999e-07 2024-02-05T07:41:47.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.925573+0000", 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2199999999999992e-06 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.925583+0000", 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0411000000000001e-05 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.925591+0000", 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9710000000000004e-06 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.925658+0000", 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6112000000000007e-05 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.925671+0000", 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3067e-05 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.925952+0000", 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000281437 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(4.a 704/703 [PushOp(4:5127e642:gc::gc.21:head, version: 75'7, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:5127e642:gc::gc.21:head@75'7, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.928684+0000", 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.380187421, 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000429818, 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.928684+0000", 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.928684+0000", 2024-02-05T07:41:47.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.928685+0000", 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7299999999999995e-07 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.928696+0000", 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1266e-05 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.928752+0000", 2024-02-05T07:41:47.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5574e-05 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.928768+0000", 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6271000000000001e-05 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.928916+0000", 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014830799999999999 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.928929+0000", 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2965e-05 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.929114+0000", 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000184761 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(4.a 704/703 [PushOp(4:517698f6:::obj_delete_at_hint.0000000121:head, version: 616'49, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:517698f6:::obj_delete_at_hint.0000000121:head@616'49, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.938787+0000", 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.3700851000000001, 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039727399999999998, 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.370 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.938787+0000", 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.938787+0000", 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.938787+0000", 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.37e-07 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.938803+0000", 2024-02-05T07:41:47.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6002e-05 2024-02-05T07:41:47.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.938820+0000", 2024-02-05T07:41:47.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6738000000000001e-05 2024-02-05T07:41:47.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.938836+0000", 2024-02-05T07:41:47.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6186999999999999e-05 2024-02-05T07:41:47.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.939034+0000", 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000197481 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.939048+0000", 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3736e-05 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.939184+0000", 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013649300000000001 2024-02-05T07:41:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.374 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.374 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.374 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(4.1e 704/703 [PushOp(4:7af5a05b:::obj_delete_at_hint.0000000044:head, version: 615'22, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:7af5a05b:::obj_delete_at_hint.0000000044:head@615'22, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.374 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.951713+0000", 2024-02-05T07:41:47.374 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.3571586019999999, 2024-02-05T07:41:47.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035732199999999998, 2024-02-05T07:41:47.374 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.374 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.374 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.951713+0000", 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.951713+0000", 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.951714+0000", 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1100000000000001e-07 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.951723+0000", 2024-02-05T07:41:47.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9339999999999995e-06 2024-02-05T07:41:47.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.951737+0000", 2024-02-05T07:41:47.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4318000000000001e-05 2024-02-05T07:41:47.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.951751+0000", 2024-02-05T07:41:47.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.42e-05 2024-02-05T07:41:47.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.951824+0000", 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3271999999999999e-05 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.951837+0000", 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2836999999999999e-05 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.952071+0000", 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023335000000000001 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(4.1e 704/703 [PushOp(4:7a5c7c5f:::obj_delete_at_hint.0000000049:head, version: 615'24, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:7a5c7c5f:::obj_delete_at_hint.0000000049:head@615'24, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.953095+0000", 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.355776849, 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044425500000000001, 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.953095+0000", 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.953095+0000", 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.953095+0000", 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4799999999999999e-07 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.953103+0000", 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2830000000000004e-06 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.953113+0000", 2024-02-05T07:41:47.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0382e-05 2024-02-05T07:41:47.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.953122+0000", 2024-02-05T07:41:47.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2024-02-05T07:41:47.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.953190+0000", 2024-02-05T07:41:47.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7997000000000006e-05 2024-02-05T07:41:47.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.953203+0000", 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2763e-05 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.953539+0000", 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033658499999999998 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(4.1e 704/703 [PushOp(4:7a480f67:::obj_delete_at_hint.0000000079:head, version: 615'28, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:7a480f67:::obj_delete_at_hint.0000000079:head@615'28, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.955556+0000", 2024-02-05T07:41:47.381 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.3533161339999999, 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043320700000000001, 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.955556+0000", 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.955556+0000", 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.955556+0000", 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0200000000000002e-07 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.955565+0000", 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1779999999999992e-06 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.955576+0000", 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0677e-05 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.955586+0000", 2024-02-05T07:41:47.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0360000000000001e-05 2024-02-05T07:41:47.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.955657+0000", 2024-02-05T07:41:47.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0748999999999996e-05 2024-02-05T07:41:47.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.955671+0000", 2024-02-05T07:41:47.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3448999999999999e-05 2024-02-05T07:41:47.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.955989+0000", 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031829200000000002 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(4.1e 704/703 [PushOp(4:7adbb485:::obj_delete_at_hint.0000000117:head, version: 616'30, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:7adbb485:::obj_delete_at_hint.0000000117:head@616'30, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.958558+0000", 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.350313436, 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036244800000000002, 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.385 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.958558+0000", 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.958558+0000", 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.958559+0000", 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.92e-07 2024-02-05T07:41:47.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.958564+0000", 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0359999999999997e-06 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.958570+0000", 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4409999999999998e-06 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.958577+0000", 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1740000000000003e-06 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.958634+0000", 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6848000000000002e-05 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.958646+0000", 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.958921+0000", 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027530099999999999 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(4.15 704/703 [PushOp(4:ad3f17aa:::obj_delete_at_hint.0000000111:head, version: 615'25, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ad3f17aa:::obj_delete_at_hint.0000000111:head@615'25, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:45.962847+0000", 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.3460247000000001, 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040817300000000001, 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.962847+0000", 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:47.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.962847+0000", 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.962848+0000", 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2499999999999995e-07 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.962857+0000", 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3670000000000003e-06 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.962868+0000", 2024-02-05T07:41:47.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0961999999999999e-05 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.962879+0000", 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0626e-05 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.962947+0000", 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7952000000000003e-05 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.962959+0000", 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2024-02-05T07:41:47.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:47.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:47.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:47.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.963255+0000", 2024-02-05T07:41:47.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029639599999999999 2024-02-05T07:41:47.392 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.392 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.392 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.392 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:47.392 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:47.392 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:41:47.856 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 3.15 scrub starts 2024-02-05T07:41:47.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 3.15 scrub ok 2024-02-05T07:41:47.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 1.3 scrub starts 2024-02-05T07:41:47.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 5.1d deep-scrub starts 2024-02-05T07:41:47.857 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 1.3 scrub ok 2024-02-05T07:41:48.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 5.1d deep-scrub ok 2024-02-05T07:41:48.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 1.9 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 1.14 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 1.1 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 1.10 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 1.2 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 1.0 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 1.12 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 1.d starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 1.e starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[37158]: 1.4 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 3.15 scrub starts 2024-02-05T07:41:48.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 3.15 scrub ok 2024-02-05T07:41:48.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 1.3 scrub starts 2024-02-05T07:41:48.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 5.1d deep-scrub starts 2024-02-05T07:41:48.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 1.3 scrub ok 2024-02-05T07:41:48.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 5.1d deep-scrub ok 2024-02-05T07:41:48.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 1.9 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 1.14 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 1.1 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 1.10 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 1.2 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 1.0 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 1.12 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 1.d starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 1.e starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-mon[29098]: 1.4 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:47 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a[29094]: 2024-02-05T07:41:47.925+0000 7fe3564d3700 -1 mon.a@0(leader).osd e706 definitely_dead 0 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 3.15 scrub starts 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 3.15 scrub ok 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 1.3 scrub starts 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 5.1d deep-scrub starts 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 1.3 scrub ok 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 5.1d deep-scrub ok 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 1.9 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 1.14 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 1.1 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 1.10 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 1.2 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 1.0 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 1.12 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 1.d starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 1.e starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:47 smithi090 ceph-mon[33213]: 1.4 starting backfill to osd.7 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:48.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T07:41:48.568 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:41:48.874 INFO:teuthology.orchestra.run.smithi007.stderr:marked in osd.5. 2024-02-05T07:41:48.937 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:48 smithi090 ceph-mon[33213]: pgmap v1292: 152 pgs: 1 activating+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 activating, 6 remapped+peering, 38 peering, 2 unknown, 100 active+clean; 648 KiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 46 KiB/s wr, 3 op/s; 5/594 objects degraded (0.842%); 33 KiB/s, 4 objects/s recovering 2024-02-05T07:41:48.937 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:48 smithi090 ceph-mon[33213]: 3.12 deep-scrub starts 2024-02-05T07:41:48.937 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:48 smithi090 ceph-mon[33213]: 3.12 deep-scrub ok 2024-02-05T07:41:48.937 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:48 smithi090 ceph-mon[33213]: Health check failed: Reduced data availability: 23 pgs inactive, 32 pgs peering (PG_AVAILABILITY) 2024-02-05T07:41:48.937 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:48 smithi090 ceph-mon[33213]: Health check failed: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:41:48.937 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:48 smithi090 ceph-mon[33213]: osdmap e706: 8 total, 8 up, 6 in 2024-02-05T07:41:48.937 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:48 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:41:48.937 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:48 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:41:48.938 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:48 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:41:48.938 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:48 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3410433154' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-05T07:41:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[29098]: pgmap v1292: 152 pgs: 1 activating+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 activating, 6 remapped+peering, 38 peering, 2 unknown, 100 active+clean; 648 KiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 46 KiB/s wr, 3 op/s; 5/594 objects degraded (0.842%); 33 KiB/s, 4 objects/s recovering 2024-02-05T07:41:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[29098]: 3.12 deep-scrub starts 2024-02-05T07:41:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[29098]: 3.12 deep-scrub ok 2024-02-05T07:41:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[29098]: Health check failed: Reduced data availability: 23 pgs inactive, 32 pgs peering (PG_AVAILABILITY) 2024-02-05T07:41:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[29098]: Health check failed: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:41:49.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[29098]: osdmap e706: 8 total, 8 up, 6 in 2024-02-05T07:41:49.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:41:49.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:41:49.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:41:49.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3410433154' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-05T07:41:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[37158]: pgmap v1292: 152 pgs: 1 activating+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 activating, 6 remapped+peering, 38 peering, 2 unknown, 100 active+clean; 648 KiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 46 KiB/s wr, 3 op/s; 5/594 objects degraded (0.842%); 33 KiB/s, 4 objects/s recovering 2024-02-05T07:41:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[37158]: 3.12 deep-scrub starts 2024-02-05T07:41:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[37158]: 3.12 deep-scrub ok 2024-02-05T07:41:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[37158]: Health check failed: Reduced data availability: 23 pgs inactive, 32 pgs peering (PG_AVAILABILITY) 2024-02-05T07:41:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[37158]: Health check failed: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded (PG_DEGRADED) 2024-02-05T07:41:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[37158]: osdmap e706: 8 total, 8 up, 6 in 2024-02-05T07:41:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:41:49.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:41:49.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:41:49.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:48 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3410433154' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-05T07:41:49.621 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-02-05T07:41:49.622 INFO:tasks.thrashosds.thrasher:inning osd 2024-02-05T07:41:49.622 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-02-05T07:41:49.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd in 4 2024-02-05T07:41:49.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:49 smithi007 ceph-mon[29098]: 6.e scrub starts 2024-02-05T07:41:49.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:49 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3410433154' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-02-05T07:41:49.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:49 smithi007 ceph-mon[29098]: osdmap e707: 8 total, 8 up, 7 in 2024-02-05T07:41:49.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:49 smithi007 ceph-mon[29098]: osdmap e708: 8 total, 8 up, 7 in 2024-02-05T07:41:49.867 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:49 smithi007 ceph-mon[37158]: 6.e scrub starts 2024-02-05T07:41:49.867 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:49 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3410433154' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-02-05T07:41:49.867 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:49 smithi007 ceph-mon[37158]: osdmap e707: 8 total, 8 up, 7 in 2024-02-05T07:41:49.867 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:49 smithi007 ceph-mon[37158]: osdmap e708: 8 total, 8 up, 7 in 2024-02-05T07:41:49.928 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:50.083 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:49 smithi090 ceph-mon[33213]: 6.e scrub starts 2024-02-05T07:41:50.083 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:49 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3410433154' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-02-05T07:41:50.083 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:49 smithi090 ceph-mon[33213]: osdmap e707: 8 total, 8 up, 7 in 2024-02-05T07:41:50.084 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:49 smithi090 ceph-mon[33213]: osdmap e708: 8 total, 8 up, 7 in 2024-02-05T07:41:50.206 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:41:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:41:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-02-05T07:41:50.206 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:41:50.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T07:41:51.061 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:41:51.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[29098]: pgmap v1295: 152 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 activating, 6 remapped+peering, 38 peering, 2 unknown, 99 active+clean; 648 KiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 68 KiB/s wr, 4 op/s; 5/594 objects degraded (0.842%); 49 KiB/s, 12 objects/s recovering 2024-02-05T07:41:51.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[29098]: 1.9 scrub starts 2024-02-05T07:41:51.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[29098]: 1.9 scrub ok 2024-02-05T07:41:51.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[29098]: 1.12 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:51.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[29098]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:41:51.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[29098]: osdmap e709: 8 total, 8 up, 7 in 2024-02-05T07:41:51.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[29098]: 1.f starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:51.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[29098]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:51.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[37158]: pgmap v1295: 152 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 activating, 6 remapped+peering, 38 peering, 2 unknown, 99 active+clean; 648 KiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 68 KiB/s wr, 4 op/s; 5/594 objects degraded (0.842%); 49 KiB/s, 12 objects/s recovering 2024-02-05T07:41:51.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[37158]: 1.9 scrub starts 2024-02-05T07:41:51.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[37158]: 1.9 scrub ok 2024-02-05T07:41:51.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[37158]: 1.12 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:51.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[37158]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:41:51.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[37158]: osdmap e709: 8 total, 8 up, 7 in 2024-02-05T07:41:51.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[37158]: 1.f starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:51.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:50 smithi007 ceph-mon[37158]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:51.182 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:50 smithi090 ceph-mon[33213]: pgmap v1295: 152 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 activating, 6 remapped+peering, 38 peering, 2 unknown, 99 active+clean; 648 KiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 68 KiB/s wr, 4 op/s; 5/594 objects degraded (0.842%); 49 KiB/s, 12 objects/s recovering 2024-02-05T07:41:51.182 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:50 smithi090 ceph-mon[33213]: 1.9 scrub starts 2024-02-05T07:41:51.182 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:50 smithi090 ceph-mon[33213]: 1.9 scrub ok 2024-02-05T07:41:51.182 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:50 smithi090 ceph-mon[33213]: 1.12 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:51.182 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:50 smithi090 ceph-mon[33213]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:41:51.182 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:50 smithi090 ceph-mon[33213]: osdmap e709: 8 total, 8 up, 7 in 2024-02-05T07:41:51.182 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:50 smithi090 ceph-mon[33213]: 1.f starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:51.182 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:50 smithi090 ceph-mon[33213]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:51.501 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:41:52.139 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 1.2 scrub starts 2024-02-05T07:41:52.139 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 1.2 scrub ok 2024-02-05T07:41:52.139 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 1.4 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.139 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 1.10 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.139 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 1.11 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.139 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 1.b starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.139 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 1.8 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.139 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 1.1 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.139 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 1.a starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.139 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 1.5 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.139 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 5.18 deep-scrub starts 2024-02-05T07:41:52.140 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 5.18 deep-scrub ok 2024-02-05T07:41:52.140 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 4.15 scrub starts 2024-02-05T07:41:52.140 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 4.15 scrub ok 2024-02-05T07:41:52.140 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 1.b scrub starts 2024-02-05T07:41:52.140 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: 1.b scrub ok 2024-02-05T07:41:52.140 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:51 smithi090 ceph-mon[33213]: osdmap e710: 8 total, 8 up, 7 in 2024-02-05T07:41:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 1.2 scrub starts 2024-02-05T07:41:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 1.2 scrub ok 2024-02-05T07:41:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 1.4 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 1.10 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 1.11 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 1.b starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 1.8 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 1.1 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 1.a starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 1.5 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 5.18 deep-scrub starts 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 5.18 deep-scrub ok 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 4.15 scrub starts 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 4.15 scrub ok 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 1.b scrub starts 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: 1.b scrub ok 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[29098]: osdmap e710: 8 total, 8 up, 7 in 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a[29094]: 2024-02-05T07:41:51.990+0000 7fe3564d3700 -1 mon.a@0(leader).osd e710 definitely_dead 0 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 1.2 scrub starts 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 1.2 scrub ok 2024-02-05T07:41:52.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 1.4 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 1.10 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 1.11 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 1.b starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 1.8 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 1.1 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 1.a starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 1.5 starting backfill to osd.5 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:52.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 5.18 deep-scrub starts 2024-02-05T07:41:52.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 5.18 deep-scrub ok 2024-02-05T07:41:52.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 4.15 scrub starts 2024-02-05T07:41:52.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 4.15 scrub ok 2024-02-05T07:41:52.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 1.b scrub starts 2024-02-05T07:41:52.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: 1.b scrub ok 2024-02-05T07:41:52.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:51 smithi007 ceph-mon[37158]: osdmap e710: 8 total, 8 up, 7 in 2024-02-05T07:41:52.658 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:41:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-02-05T07:41:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-02-05T07:41:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-02-05T07:41:52.659 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:41:52.738 INFO:teuthology.orchestra.run.smithi007.stderr:marked in osd.4. 2024-02-05T07:41:53.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:53.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[29098]: 5.b scrub starts 2024-02-05T07:41:53.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[29098]: 5.b scrub ok 2024-02-05T07:41:53.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[29098]: pgmap v1298: 152 pgs: 4 active+clean+remapped, 4 active+remapped+backfill_toofull, 1 activating+remapped, 2 remapped+peering, 141 active+clean; 645 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 62 KiB/s, 20 objects/s recovering 2024-02-05T07:41:53.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[29098]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded) 2024-02-05T07:41:53.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1633308247' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-05T07:41:53.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-05T07:41:53.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[29098]: 6.10 deep-scrub starts 2024-02-05T07:41:53.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[29098]: 6.10 deep-scrub ok 2024-02-05T07:41:53.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-05T07:41:53.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[29098]: osdmap e711: 8 total, 8 up, 8 in 2024-02-05T07:41:53.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[29098]: 6.1f deep-scrub starts 2024-02-05T07:41:53.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:53.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[37158]: 5.b scrub starts 2024-02-05T07:41:53.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[37158]: 5.b scrub ok 2024-02-05T07:41:53.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[37158]: pgmap v1298: 152 pgs: 4 active+clean+remapped, 4 active+remapped+backfill_toofull, 1 activating+remapped, 2 remapped+peering, 141 active+clean; 645 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 62 KiB/s, 20 objects/s recovering 2024-02-05T07:41:53.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[37158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded) 2024-02-05T07:41:53.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1633308247' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-05T07:41:53.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-05T07:41:53.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[37158]: 6.10 deep-scrub starts 2024-02-05T07:41:53.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[37158]: 6.10 deep-scrub ok 2024-02-05T07:41:53.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-05T07:41:53.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[37158]: osdmap e711: 8 total, 8 up, 8 in 2024-02-05T07:41:53.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:52 smithi007 ceph-mon[37158]: 6.1f deep-scrub starts 2024-02-05T07:41:53.248 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:52 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:41:53.249 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:52 smithi090 ceph-mon[33213]: 5.b scrub starts 2024-02-05T07:41:53.249 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:52 smithi090 ceph-mon[33213]: 5.b scrub ok 2024-02-05T07:41:53.249 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:52 smithi090 ceph-mon[33213]: pgmap v1298: 152 pgs: 4 active+clean+remapped, 4 active+remapped+backfill_toofull, 1 activating+remapped, 2 remapped+peering, 141 active+clean; 645 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 62 KiB/s, 20 objects/s recovering 2024-02-05T07:41:53.249 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:52 smithi090 ceph-mon[33213]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded) 2024-02-05T07:41:53.249 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:52 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1633308247' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-05T07:41:53.249 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:52 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-05T07:41:53.249 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:52 smithi090 ceph-mon[33213]: 6.10 deep-scrub starts 2024-02-05T07:41:53.249 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:52 smithi090 ceph-mon[33213]: 6.10 deep-scrub ok 2024-02-05T07:41:53.249 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:52 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-05T07:41:53.249 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:52 smithi090 ceph-mon[33213]: osdmap e711: 8 total, 8 up, 8 in 2024-02-05T07:41:53.249 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:52 smithi090 ceph-mon[33213]: 6.1f deep-scrub starts 2024-02-05T07:41:53.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T07:41:53.444 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-02-05T07:41:53.444 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-02-05T07:41:53.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pgp_num 21 2024-02-05T07:41:53.512 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T07:41:53.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:41:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:41:53.695 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:53.885 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:53 smithi090 ceph-mon[33213]: 5.8 scrub starts 2024-02-05T07:41:53.885 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:53 smithi090 ceph-mon[33213]: 5.8 scrub ok 2024-02-05T07:41:53.885 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:53 smithi090 ceph-mon[33213]: 3.8 scrub starts 2024-02-05T07:41:53.885 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:53 smithi090 ceph-mon[33213]: 3.8 scrub ok 2024-02-05T07:41:53.885 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:53 smithi090 ceph-mon[33213]: 6.1f deep-scrub ok 2024-02-05T07:41:53.885 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:53 smithi090 ceph-mon[33213]: 1.b scrub starts 2024-02-05T07:41:53.885 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:53 smithi090 ceph-mon[33213]: 1.b scrub ok 2024-02-05T07:41:53.885 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:53 smithi090 ceph-mon[33213]: osdmap e712: 8 total, 8 up, 8 in 2024-02-05T07:41:53.885 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:53 smithi090 ceph-mon[33213]: 1.f starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:53.885 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:53 smithi090 ceph-mon[33213]: 1.f continuing backfill to osd.5 from (86'128,86'128] MIN to 86'128 2024-02-05T07:41:54.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[29098]: 5.8 scrub starts 2024-02-05T07:41:54.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[29098]: 5.8 scrub ok 2024-02-05T07:41:54.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[29098]: 3.8 scrub starts 2024-02-05T07:41:54.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[29098]: 3.8 scrub ok 2024-02-05T07:41:54.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[29098]: 6.1f deep-scrub ok 2024-02-05T07:41:54.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[29098]: 1.b scrub starts 2024-02-05T07:41:54.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[29098]: 1.b scrub ok 2024-02-05T07:41:54.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[29098]: osdmap e712: 8 total, 8 up, 8 in 2024-02-05T07:41:54.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[29098]: 1.f starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:54.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[29098]: 1.f continuing backfill to osd.5 from (86'128,86'128] MIN to 86'128 2024-02-05T07:41:54.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[37158]: 5.8 scrub starts 2024-02-05T07:41:54.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[37158]: 5.8 scrub ok 2024-02-05T07:41:54.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[37158]: 3.8 scrub starts 2024-02-05T07:41:54.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[37158]: 3.8 scrub ok 2024-02-05T07:41:54.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[37158]: 6.1f deep-scrub ok 2024-02-05T07:41:54.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[37158]: 1.b scrub starts 2024-02-05T07:41:54.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[37158]: 1.b scrub ok 2024-02-05T07:41:54.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[37158]: osdmap e712: 8 total, 8 up, 8 in 2024-02-05T07:41:54.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[37158]: 1.f starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:54.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:53 smithi007 ceph-mon[37158]: 1.f continuing backfill to osd.5 from (86'128,86'128] MIN to 86'128 2024-02-05T07:41:55.060 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: pgmap v1301: 152 pgs: 1 peering, 6 active+clean+remapped, 6 active+remapped+backfill_toofull, 139 active+clean; 671 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 4.5 KiB/s rd, 24 KiB/s wr, 5 op/s; 82 KiB/s, 14 objects/s recovering 2024-02-05T07:41:55.060 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: 5.19 scrub starts 2024-02-05T07:41:55.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: 5.19 scrub ok 2024-02-05T07:41:55.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: 1.d starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:55.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: 1.d continuing backfill to osd.7 from (86'128,86'128] MIN to 86'128 2024-02-05T07:41:55.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: 1.e starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:55.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: 1.e continuing backfill to osd.7 from (86'128,86'128] MIN to 86'128 2024-02-05T07:41:55.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: 1.d deep-scrub starts 2024-02-05T07:41:55.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: 1.d deep-scrub ok 2024-02-05T07:41:55.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-02-05T07:41:55.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: 1.11 scrub starts 2024-02-05T07:41:55.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: 1.11 scrub ok 2024-02-05T07:41:55.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:41:55.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:41:55.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: osdmap e713: 8 total, 8 up, 8 in 2024-02-05T07:41:55.062 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[29098]: 1.3 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:55.062 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: pgmap v1301: 152 pgs: 1 peering, 6 active+clean+remapped, 6 active+remapped+backfill_toofull, 139 active+clean; 671 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 4.5 KiB/s rd, 24 KiB/s wr, 5 op/s; 82 KiB/s, 14 objects/s recovering 2024-02-05T07:41:55.062 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: 5.19 scrub starts 2024-02-05T07:41:55.062 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: 5.19 scrub ok 2024-02-05T07:41:55.062 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: 1.d starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:55.062 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: 1.d continuing backfill to osd.7 from (86'128,86'128] MIN to 86'128 2024-02-05T07:41:55.062 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: 1.e starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:55.062 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: 1.e continuing backfill to osd.7 from (86'128,86'128] MIN to 86'128 2024-02-05T07:41:55.062 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: 1.d deep-scrub starts 2024-02-05T07:41:55.062 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: 1.d deep-scrub ok 2024-02-05T07:41:55.062 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-02-05T07:41:55.063 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: 1.11 scrub starts 2024-02-05T07:41:55.063 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: 1.11 scrub ok 2024-02-05T07:41:55.063 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:41:55.063 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:41:55.063 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: osdmap e713: 8 total, 8 up, 8 in 2024-02-05T07:41:55.063 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:54 smithi007 ceph-mon[37158]: 1.3 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:55.130 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: pgmap v1301: 152 pgs: 1 peering, 6 active+clean+remapped, 6 active+remapped+backfill_toofull, 139 active+clean; 671 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 4.5 KiB/s rd, 24 KiB/s wr, 5 op/s; 82 KiB/s, 14 objects/s recovering 2024-02-05T07:41:55.130 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: 5.19 scrub starts 2024-02-05T07:41:55.130 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: 5.19 scrub ok 2024-02-05T07:41:55.130 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: 1.d starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:55.131 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: 1.d continuing backfill to osd.7 from (86'128,86'128] MIN to 86'128 2024-02-05T07:41:55.131 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: 1.e starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:55.131 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: 1.e continuing backfill to osd.7 from (86'128,86'128] MIN to 86'128 2024-02-05T07:41:55.131 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: 1.d deep-scrub starts 2024-02-05T07:41:55.131 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: 1.d deep-scrub ok 2024-02-05T07:41:55.131 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-02-05T07:41:55.131 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: 1.11 scrub starts 2024-02-05T07:41:55.131 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: 1.11 scrub ok 2024-02-05T07:41:55.131 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:41:55.131 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:41:55.131 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: osdmap e713: 8 total, 8 up, 8 in 2024-02-05T07:41:55.131 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:54 smithi090 ceph-mon[33213]: 1.3 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:55.172 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-05T07:41:55.172 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-02-05T07:41:55.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-02-05T07:41:55.172 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-02-05T07:41:55.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.172 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:157380 10.16 10:6c7dc5bb:::benchmark_data_smithi007_157098_object28450:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e511)", 2024-02-05T07:41:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:45.622639+0000", 2024-02-05T07:41:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 369.54718911399999, 2024-02-05T07:41:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0701192349999999, 2024-02-05T07:41:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-02-05T07:41:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157380 2024-02-05T07:41:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.622639+0000", 2024-02-05T07:41:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.622639+0000", 2024-02-05T07:41:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.622640+0000", 2024-02-05T07:41:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.141e-06 2024-02-05T07:41:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.622643+0000", 2024-02-05T07:41:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.343e-06 2024-02-05T07:41:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.622655+0000", 2024-02-05T07:41:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1888999999999999e-05 2024-02-05T07:41:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.622664+0000", 2024-02-05T07:41:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3790000000000006e-06 2024-02-05T07:41:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.622714+0000", 2024-02-05T07:41:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0640999999999998e-05 2024-02-05T07:41:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.622720+0000", 2024-02-05T07:41:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2089999999999996e-06 2024-02-05T07:41:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.692708+0000", 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0699883130000001 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.692758+0000", 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0319999999999999e-05 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:157442 10.16 10:6d0a09e2:::benchmark_data_smithi007_157098_object28512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e511)", 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:45.667603+0000", 2024-02-05T07:41:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 369.50222473100001, 2024-02-05T07:41:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029763532, 2024-02-05T07:41:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157442 2024-02-05T07:41:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.667603+0000", 2024-02-05T07:41:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.667603+0000", 2024-02-05T07:41:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.667604+0000", 2024-02-05T07:41:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6000000000000003e-07 2024-02-05T07:41:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.667609+0000", 2024-02-05T07:41:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7509999999999999e-06 2024-02-05T07:41:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.667618+0000", 2024-02-05T07:41:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7790000000000001e-06 2024-02-05T07:41:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.667623+0000", 2024-02-05T07:41:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8949999999999997e-06 2024-02-05T07:41:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.667668+0000", 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5234000000000001e-05 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.667673+0000", 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4450000000000004e-06 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.693101+0000", 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254277599999999 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.693220+0000", 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118857 2024-02-05T07:41:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.693464+0000", 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000244501 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.697206+0000", 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003741893 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.697214+0000", 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2269999999999993e-06 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.697293+0000", 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8992999999999994e-05 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.697337+0000", 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3495999999999998e-05 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.697367+0000", 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9941e-05 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:157452 10.16 10:6d933ea3:::benchmark_data_smithi007_157098_object28522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e511)", 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:45.669986+0000", 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 369.49984157199998, 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.027492353, 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157452 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.669986+0000", 2024-02-05T07:41:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.669986+0000", 2024-02-05T07:41:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.669987+0000", 2024-02-05T07:41:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7999999999999995e-07 2024-02-05T07:41:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.669990+0000", 2024-02-05T07:41:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2389999999999998e-06 2024-02-05T07:41:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.670001+0000", 2024-02-05T07:41:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0302999999999999e-05 2024-02-05T07:41:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.670007+0000", 2024-02-05T07:41:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0109999999999997e-06 2024-02-05T07:41:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.670054+0000", 2024-02-05T07:41:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7645999999999997e-05 2024-02-05T07:41:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.670063+0000", 2024-02-05T07:41:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2789999999999998e-06 2024-02-05T07:41:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.694729+0000", 2024-02-05T07:41:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0246662799999999 2024-02-05T07:41:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.694835+0000", 2024-02-05T07:41:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106344 2024-02-05T07:41:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:41:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.695006+0000", 2024-02-05T07:41:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017037700000000001 2024-02-05T07:41:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.697245+0000", 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022396619999999999 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.697378+0000", 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013293199999999999 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.697417+0000", 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8609000000000003e-05 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.697445+0000", 2024-02-05T07:41:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8479999999999998e-05 2024-02-05T07:41:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.697479+0000", 2024-02-05T07:41:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3611000000000001e-05 2024-02-05T07:41:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:157468 10.16 10:684821cb:::benchmark_data_smithi007_157098_object28538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e511)", 2024-02-05T07:41:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:45.682757+0000", 2024-02-05T07:41:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 369.487071335, 2024-02-05T07:41:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0174302129999999, 2024-02-05T07:41:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157468 2024-02-05T07:41:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.682757+0000", 2024-02-05T07:41:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.682757+0000", 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.682758+0000", 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8700000000000005e-07 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.682761+0000", 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8460000000000001e-06 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.682770+0000", 2024-02-05T07:41:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7090000000000008e-06 2024-02-05T07:41:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.682777+0000", 2024-02-05T07:41:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8179999999999998e-06 2024-02-05T07:41:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.682836+0000", 2024-02-05T07:41:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780999999999998e-05 2024-02-05T07:41:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.682844+0000", 2024-02-05T07:41:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.568e-06 2024-02-05T07:41:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.696028+0000", 2024-02-05T07:41:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0131838360000001 2024-02-05T07:41:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.696100+0000", 2024-02-05T07:41:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2211000000000004e-05 2024-02-05T07:41:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:41:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.696237+0000", 2024-02-05T07:41:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013678599999999999 2024-02-05T07:41:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.698746+0000", 2024-02-05T07:41:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025086029999999999 2024-02-05T07:41:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.700028+0000", 2024-02-05T07:41:55.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012825930000000001 2024-02-05T07:41:55.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.700095+0000", 2024-02-05T07:41:55.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6581000000000005e-05 2024-02-05T07:41:55.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.700149+0000", 2024-02-05T07:41:55.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3825000000000001e-05 2024-02-05T07:41:55.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.700187+0000", 2024-02-05T07:41:55.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8269000000000002e-05 2024-02-05T07:41:55.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.198 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.198 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:157477 10.16 10:6b325865:::benchmark_data_smithi007_157098_object28547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e511)", 2024-02-05T07:41:55.198 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:45.689665+0000", 2024-02-05T07:41:55.198 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 369.48016333800001, 2024-02-05T07:41:55.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.01260576, 2024-02-05T07:41:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157477 2024-02-05T07:41:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.689665+0000", 2024-02-05T07:41:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.689665+0000", 2024-02-05T07:41:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.689666+0000", 2024-02-05T07:41:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8100000000000001e-07 2024-02-05T07:41:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.689669+0000", 2024-02-05T07:41:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8600000000000003e-06 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.689679+0000", 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0005e-05 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.689687+0000", 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5390000000000002e-06 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.689745+0000", 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8038999999999998e-05 2024-02-05T07:41:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.689762+0000", 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6541999999999998e-05 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.697519+0000", 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0077571729999999 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.697607+0000", 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8049999999999999e-05 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.697768+0000", 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016120300000000001 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.700248+0000", 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024800450000000002 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.700255+0000", 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3679999999999996e-06 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.702178+0000", 2024-02-05T07:41:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001922258 2024-02-05T07:41:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.702235+0000", 2024-02-05T07:41:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6975000000000003e-05 2024-02-05T07:41:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.702270+0000", 2024-02-05T07:41:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5821999999999997e-05 2024-02-05T07:41:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:157499 10.16 10:6af597ed:::benchmark_data_smithi007_157098_object28569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e511)", 2024-02-05T07:41:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:45.705500+0000", 2024-02-05T07:41:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 369.464328271, 2024-02-05T07:41:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.00092044, 2024-02-05T07:41:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157499 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.705500+0000", 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.705500+0000", 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.705501+0000", 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0999999999999997e-07 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.705505+0000", 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9389999999999999e-06 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.705515+0000", 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0210000000000001e-05 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.705522+0000", 2024-02-05T07:41:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7320000000000001e-06 2024-02-05T07:41:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.705582+0000", 2024-02-05T07:41:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0733999999999998e-05 2024-02-05T07:41:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.705592+0000", 2024-02-05T07:41:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.713e-06 2024-02-05T07:41:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.698798+0000", 2024-02-05T07:41:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99320615300000004 2024-02-05T07:41:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.698916+0000", 2024-02-05T07:41:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118288 2024-02-05T07:41:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:41:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.699059+0000", 2024-02-05T07:41:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014260999999999999 2024-02-05T07:41:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.702097+0000", 2024-02-05T07:41:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030382069999999998 2024-02-05T07:41:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.702132+0000", 2024-02-05T07:41:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4941000000000003e-05 2024-02-05T07:41:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706351+0000", 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042191449999999997 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706392+0000", 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0822000000000003e-05 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706420+0000", 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8036000000000001e-05 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:157507 10.16 10:6acefcdb:::benchmark_data_smithi007_157098_object28577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e511)", 2024-02-05T07:41:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:45.713062+0000", 2024-02-05T07:41:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 369.45676647699997, 2024-02-05T07:41:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99584896499999997, 2024-02-05T07:41:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157507 2024-02-05T07:41:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.713062+0000", 2024-02-05T07:41:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.713062+0000", 2024-02-05T07:41:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.713062+0000", 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4399999999999999e-07 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.713066+0000", 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.715e-06 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.713074+0000", 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7340000000000008e-06 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.713080+0000", 2024-02-05T07:41:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0340000000000002e-06 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.713122+0000", 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2169000000000002e-05 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.713128+0000", 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9190000000000004e-06 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.700285+0000", 2024-02-05T07:41:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.98715752000000001 2024-02-05T07:41:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.700354+0000", 2024-02-05T07:41:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8362999999999996e-05 2024-02-05T07:41:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:41:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.700480+0000", 2024-02-05T07:41:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012607699999999999 2024-02-05T07:41:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.702146+0000", 2024-02-05T07:41:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016663170000000001 2024-02-05T07:41:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.704401+0000", 2024-02-05T07:41:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022551619999999998 2024-02-05T07:41:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.708823+0000", 2024-02-05T07:41:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004421421 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.708872+0000", 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9616000000000001e-05 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.708911+0000", 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8173999999999997e-05 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:157576 10.16 10:6c7d9738:::benchmark_data_smithi007_157098_object28646:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e511)", 2024-02-05T07:41:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:45.829082+0000", 2024-02-05T07:41:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 369.34074631599998, 2024-02-05T07:41:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87994899299999996, 2024-02-05T07:41:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157576 2024-02-05T07:41:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.829082+0000", 2024-02-05T07:41:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.829082+0000", 2024-02-05T07:41:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.829083+0000", 2024-02-05T07:41:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6000000000000002e-07 2024-02-05T07:41:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.829087+0000", 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9099999999999998e-06 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.829096+0000", 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6320000000000003e-06 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.829104+0000", 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0059999999999992e-06 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.829153+0000", 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8789000000000003e-05 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.829159+0000", 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4489999999999996e-06 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.701070+0000", 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87191027399999999 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.701174+0000", 2024-02-05T07:41:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010385 2024-02-05T07:41:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:41:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.701334+0000", 2024-02-05T07:41:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016040999999999999 2024-02-05T07:41:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.704465+0000", 2024-02-05T07:41:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031312829999999999 2024-02-05T07:41:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.705294+0000", 2024-02-05T07:41:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082868099999999995 2024-02-05T07:41:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.708950+0000", 2024-02-05T07:41:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036558990000000002 2024-02-05T07:41:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.709000+0000", 2024-02-05T07:41:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9960000000000003e-05 2024-02-05T07:41:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.709031+0000", 2024-02-05T07:41:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.099e-05 2024-02-05T07:41:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:157579 10.16 10:6fe66432:::benchmark_data_smithi007_157098_object28649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e511)", 2024-02-05T07:41:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:45.830223+0000", 2024-02-05T07:41:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 369.33960543000001, 2024-02-05T07:41:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87893903200000001, 2024-02-05T07:41:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157579 2024-02-05T07:41:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.830223+0000", 2024-02-05T07:41:55.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.830223+0000", 2024-02-05T07:41:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.830223+0000", 2024-02-05T07:41:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1699999999999997e-07 2024-02-05T07:41:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.830227+0000", 2024-02-05T07:41:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9020000000000001e-06 2024-02-05T07:41:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.830237+0000", 2024-02-05T07:41:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6190000000000006e-06 2024-02-05T07:41:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.830245+0000", 2024-02-05T07:41:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9829999999999996e-06 2024-02-05T07:41:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.830297+0000", 2024-02-05T07:41:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2602999999999998e-05 2024-02-05T07:41:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.830305+0000", 2024-02-05T07:41:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1160000000000001e-06 2024-02-05T07:41:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.702313+0000", 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87200889199999998 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.702413+0000", 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9957000000000005e-05 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.702601+0000", 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187679 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.705301+0000", 2024-02-05T07:41:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026996820000000001 2024-02-05T07:41:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706192+0000", 2024-02-05T07:41:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089077299999999998 2024-02-05T07:41:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.709084+0000", 2024-02-05T07:41:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028926220000000001 2024-02-05T07:41:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.709116+0000", 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1877e-05 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.709162+0000", 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5609999999999999e-05 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:157615 10.16 10:6edeef46:::benchmark_data_smithi007_157098_object28685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e511)", 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:45.864354+0000", 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 369.30547403000003, 2024-02-05T07:41:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.84611338599999997, 2024-02-05T07:41:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157615 2024-02-05T07:41:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.864354+0000", 2024-02-05T07:41:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.864354+0000", 2024-02-05T07:41:55.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.864355+0000", 2024-02-05T07:41:55.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0109999999999999e-06 2024-02-05T07:41:55.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.864359+0000", 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9149999999999998e-06 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.864366+0000", 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8730000000000001e-06 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.864373+0000", 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9959999999999996e-06 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.864432+0000", 2024-02-05T07:41:55.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9540999999999998e-05 2024-02-05T07:41:55.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:55.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.864442+0000", 2024-02-05T07:41:55.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5389999999999996e-06 2024-02-05T07:41:55.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.703399+0000", 2024-02-05T07:41:55.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.83895688199999996 2024-02-05T07:41:55.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.703523+0000", 2024-02-05T07:41:55.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000123862 2024-02-05T07:41:55.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:41:55.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.703639+0000", 2024-02-05T07:41:55.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116452 2024-02-05T07:41:55.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706207+0000", 2024-02-05T07:41:55.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025680410000000001 2024-02-05T07:41:55.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706253+0000", 2024-02-05T07:41:55.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6264999999999998e-05 2024-02-05T07:41:55.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.710383+0000", 2024-02-05T07:41:55.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041291000000000001 2024-02-05T07:41:55.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.710431+0000", 2024-02-05T07:41:55.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8809e-05 2024-02-05T07:41:55.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.710467+0000", 2024-02-05T07:41:55.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6100000000000003e-05 2024-02-05T07:41:55.240 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.240 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.240 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.240 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:157708 10.16 10:6cdd049c:::benchmark_data_smithi007_157098_object28778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e511)", 2024-02-05T07:41:55.240 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:45.958033+0000", 2024-02-05T07:41:55.240 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 369.21179500300002, 2024-02-05T07:41:55.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.753513082, 2024-02-05T07:41:55.241 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.241 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.241 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.241 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:55.241 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:55.241 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157708 2024-02-05T07:41:55.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.241 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.958033+0000", 2024-02-05T07:41:55.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.958033+0000", 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.958034+0000", 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6599999999999995e-07 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.958038+0000", 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6739999999999999e-06 2024-02-05T07:41:55.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.958047+0000", 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4169999999999998e-06 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.958053+0000", 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6030000000000004e-06 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.958102+0000", 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8974999999999998e-05 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:45.958110+0000", 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.704499+0000", 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.74638953900000005 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.704566+0000", 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6863e-05 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.704686+0000", 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011961 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706305+0000", 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016192870000000001 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706317+0000", 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2463000000000001e-05 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.711448+0000", 2024-02-05T07:41:55.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051310219999999998 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.711509+0000", 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0806e-05 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.711546+0000", 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6927000000000001e-05 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.246 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25454.0:157747 10.16 10:6bbaff7a:::benchmark_data_smithi007_157098_object28817:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e511)", 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:35:46.003852+0000", 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 369.16597583200002, 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70783331800000004, 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25454", 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/1073973298", 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157747 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.003852+0000", 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.003852+0000", 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.003853+0000", 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1900000000000003e-07 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.003856+0000", 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7530000000000002e-06 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.003862+0000", 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4969999999999998e-06 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.003868+0000", 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.178e-06 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.003919+0000", 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0562000000000001e-05 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.003926+0000", 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8220000000000001e-06 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.705334+0000", 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70140844899999999 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.705432+0000", 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7706000000000003e-05 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.705603+0000", 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017103100000000001 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.706324+0000", 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072118699999999998 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.708753+0000", 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024294030000000001 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.711600+0000", 2024-02-05T07:41:55.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028463609999999999 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.711651+0000", 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1452999999999997e-05 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:35:46.711686+0000", 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4397000000000003e-05 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.15846.0:50219 11.1 11:8a6403b5:::benchmark_data_smithi007_183880_object50218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e607)", 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:38:41.587129+0000", 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 193.58269898200001, 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.131261244, 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.15846", 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 50219 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:38:41.587129+0000", 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:38:41.587129+0000", 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:38:41.587130+0000", 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.06e-07 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:38:41.587239+0000", 2024-02-05T07:41:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108884 2024-02-05T07:41:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:38:41.587246+0000", 2024-02-05T07:41:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1400000000000002e-06 2024-02-05T07:41:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:38:41.587251+0000", 2024-02-05T07:41:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5790000000000003e-06 2024-02-05T07:41:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:38:41.587299+0000", 2024-02-05T07:41:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7676999999999998e-05 2024-02-05T07:41:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:38:41.587408+0000", 2024-02-05T07:41:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109447 2024-02-05T07:41:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3", 2024-02-05T07:41:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:38:41.587519+0000", 2024-02-05T07:41:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011084199999999999 2024-02-05T07:41:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:38:41.588828+0000", 2024-02-05T07:41:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013085760000000001 2024-02-05T07:41:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:38:41.590131+0000", 2024-02-05T07:41:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013028860000000001 2024-02-05T07:41:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:38:41.718331+0000", 2024-02-05T07:41:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12820026100000001 2024-02-05T07:41:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:38:41.718367+0000", 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5806999999999998e-05 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:38:41.718390+0000", 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3538999999999999e-05 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.15846.0:74024 11.3 11:ceefb60a:::benchmark_data_smithi007_183880_object74023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e617)", 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:39:02.680610+0000", 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.48921837099999, 2024-02-05T07:41:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.129066026, 2024-02-05T07:41:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.15846", 2024-02-05T07:41:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:41:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 74024 2024-02-05T07:41:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.680610+0000", 2024-02-05T07:41:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.680610+0000", 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.680611+0000", 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1909999999999999e-06 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.680655+0000", 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3639e-05 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.680665+0000", 2024-02-05T07:41:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0582e-05 2024-02-05T07:41:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.680671+0000", 2024-02-05T07:41:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8080000000000001e-06 2024-02-05T07:41:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.680719+0000", 2024-02-05T07:41:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8028999999999997e-05 2024-02-05T07:41:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.680824+0000", 2024-02-05T07:41:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104994 2024-02-05T07:41:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:41:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.680918+0000", 2024-02-05T07:41:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3938999999999996e-05 2024-02-05T07:41:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.682098+0000", 2024-02-05T07:41:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001179954 2024-02-05T07:41:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.687682+0000", 2024-02-05T07:41:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055843009999999998 2024-02-05T07:41:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.809612+0000", 2024-02-05T07:41:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12192995500000001 2024-02-05T07:41:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.809648+0000", 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6220000000000002e-05 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.809676+0000", 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7413999999999999e-05 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.15846.0:74029 11.3 11:c59f0baa:::benchmark_data_smithi007_183880_object74028:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e617)", 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:39:02.684179+0000", 2024-02-05T07:41:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.485648841, 2024-02-05T07:41:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12785095199999999, 2024-02-05T07:41:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.15846", 2024-02-05T07:41:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:41:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 74029 2024-02-05T07:41:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.684179+0000", 2024-02-05T07:41:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.684179+0000", 2024-02-05T07:41:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.684180+0000", 2024-02-05T07:41:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3700000000000001e-07 2024-02-05T07:41:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.684267+0000", 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7379000000000005e-05 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.684273+0000", 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7980000000000002e-06 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.684277+0000", 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6959999999999998e-06 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.684317+0000", 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9978999999999998e-05 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.684434+0000", 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117198 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.684528+0000", 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4296999999999995e-05 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.685770+0000", 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012418290000000001 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.691066+0000", 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052966200000000001 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.811937+0000", 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120870423 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.811987+0000", 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0466000000000001e-05 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.812030+0000", 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2830000000000002e-05 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.15846.0:74033 11.3 11:c013095c:::benchmark_data_smithi007_183880_object74032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e617)", 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:39:02.687833+0000", 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.48199536300001, 2024-02-05T07:41:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.127095492, 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.15846", 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 74033 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.687833+0000", 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.687833+0000", 2024-02-05T07:41:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.687833+0000", 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3900000000000004e-07 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.687884+0000", 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0251999999999999e-05 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.687892+0000", 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.028e-06 2024-02-05T07:41:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.687898+0000", 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7669999999999998e-06 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.687941+0000", 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2453999999999997e-05 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.688027+0000", 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6533000000000007e-05 2024-02-05T07:41:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.688119+0000", 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1108999999999998e-05 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.689109+0000", 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00099078800000000004 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.691106+0000", 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019965899999999999 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.814871+0000", 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12376498900000001 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.814905+0000", 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4022000000000001e-05 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.814928+0000", 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3221e-05 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.15846.0:74034 11.3 11:ce40954f:::benchmark_data_smithi007_183880_object74033:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e617)", 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:39:02.690072+0000", 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.479756291, 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.124967541, 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.15846", 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 74034 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.690072+0000", 2024-02-05T07:41:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.690072+0000", 2024-02-05T07:41:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.690072+0000", 2024-02-05T07:41:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4099999999999999e-07 2024-02-05T07:41:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.690095+0000", 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2331999999999999e-05 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.690102+0000", 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8189999999999992e-06 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.690109+0000", 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9500000000000004e-06 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.690185+0000", 2024-02-05T07:41:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5111999999999995e-05 2024-02-05T07:41:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.690271+0000", 2024-02-05T07:41:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6489000000000005e-05 2024-02-05T07:41:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2024-02-05T07:41:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.690360+0000", 2024-02-05T07:41:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9248999999999999e-05 2024-02-05T07:41:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.691942+0000", 2024-02-05T07:41:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015820770000000001 2024-02-05T07:41:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.694540+0000", 2024-02-05T07:41:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002597748 2024-02-05T07:41:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.814970+0000", 2024-02-05T07:41:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120430285 2024-02-05T07:41:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.815016+0000", 2024-02-05T07:41:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5600999999999999e-05 2024-02-05T07:41:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:39:02.815039+0000", 2024-02-05T07:41:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3337999999999999e-05 2024-02-05T07:41:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.285 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.15846.0:220537 11.5 11:a543fd94:::benchmark_data_smithi007_183880_object110256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e685)", 2024-02-05T07:41:55.285 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:08.410748+0000", 2024-02-05T07:41:55.285 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.759079796000002, 2024-02-05T07:41:55.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13255274, 2024-02-05T07:41:55.285 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.285 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T07:41:55.285 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.285 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.15846", 2024-02-05T07:41:55.285 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2747371084", 2024-02-05T07:41:55.285 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 220537 2024-02-05T07:41:55.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.285 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:08.410748+0000", 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:08.410748+0000", 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:08.410749+0000", 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3500000000000005e-07 2024-02-05T07:41:55.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:08.410754+0000", 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4159999999999997e-06 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:08.410764+0000", 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0134e-05 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:08.410770+0000", 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.801e-06 2024-02-05T07:41:55.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:08.410830+0000", 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0250000000000001e-05 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:08.410969+0000", 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013931499999999999 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,3", 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:08.411109+0000", 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013988299999999999 2024-02-05T07:41:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:08.412872+0000", 2024-02-05T07:41:55.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001763255 2024-02-05T07:41:55.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-02-05T07:41:55.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:08.412983+0000", 2024-02-05T07:41:55.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110289 2024-02-05T07:41:55.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-02-05T07:41:55.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:08.543173+0000", 2024-02-05T07:41:55.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.130190795 2024-02-05T07:41:55.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-02-05T07:41:55.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:08.543263+0000", 2024-02-05T07:41:55.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0152e-05 2024-02-05T07:41:55.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:08.543301+0000", 2024-02-05T07:41:55.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7515000000000003e-05 2024-02-05T07:41:55.290 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.290 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.290 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.291 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24356.0:4593 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 93862608739072 gen 8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-02-05T07:41:55.291 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:44.830698+0000", 2024-02-05T07:41:55.291 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.339130235000001, 2024-02-05T07:41:55.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.036927401, 2024-02-05T07:41:55.291 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.291 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:55.291 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.291 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24356", 2024-02-05T07:41:55.291 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2716729371", 2024-02-05T07:41:55.291 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4593 2024-02-05T07:41:55.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.291 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830698+0000", 2024-02-05T07:41:55.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830698+0000", 2024-02-05T07:41:55.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830700+0000", 2024-02-05T07:41:55.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.922e-06 2024-02-05T07:41:55.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830706+0000", 2024-02-05T07:41:55.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3110000000000002e-06 2024-02-05T07:41:55.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830722+0000", 2024-02-05T07:41:55.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6044000000000001e-05 2024-02-05T07:41:55.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830735+0000", 2024-02-05T07:41:55.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2456000000000001e-05 2024-02-05T07:41:55.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830811+0000", 2024-02-05T07:41:55.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2024-02-05T07:41:55.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-02-05T07:41:55.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830819+0000", 2024-02-05T07:41:55.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5279999999999998e-06 2024-02-05T07:41:55.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.840626+0000", 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098072130000000004 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.840637+0000", 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0553999999999999e-05 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.864291+0000", 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0236545889999999 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-02-05T07:41:55.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.864535+0000", 2024-02-05T07:41:55.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024379600000000001 2024-02-05T07:41:55.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.865250+0000", 2024-02-05T07:41:55.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071546200000000004 2024-02-05T07:41:55.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.865302+0000", 2024-02-05T07:41:55.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1578e-05 2024-02-05T07:41:55.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.867625+0000", 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023233009999999998 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24356.0:4595 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 93862613159296 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-02-05T07:41:44.830761+0000", 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.339066605999999, 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.05846247, 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-02-05T07:41:55.297 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24356", 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.7:0/2716729371", 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4595 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830761+0000", 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830761+0000", 2024-02-05T07:41:55.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-02-05T07:41:55.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-02-05T07:41:55.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830762+0000", 2024-02-05T07:41:55.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7999999999999996e-07 2024-02-05T07:41:55.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-02-05T07:41:55.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830765+0000", 2024-02-05T07:41:55.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6730000000000001e-06 2024-02-05T07:41:55.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-02-05T07:41:55.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830775+0000", 2024-02-05T07:41:55.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.075e-05 2024-02-05T07:41:55.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-02-05T07:41:55.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830782+0000", 2024-02-05T07:41:55.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.156e-06 2024-02-05T07:41:55.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830835+0000", 2024-02-05T07:41:55.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3341999999999999e-05 2024-02-05T07:41:55.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.830841+0000", 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7470000000000002e-06 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.843297+0000", 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012456752999999999 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:44.843304+0000", 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5289999999999997e-06 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.878735+0000", 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0354311519999999 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.879100+0000", 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036537499999999998 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.880191+0000", 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001090089 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-02-05T07:41:55.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.880288+0000", 2024-02-05T07:41:55.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7623999999999999e-05 2024-02-05T07:41:55.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-02-05T07:41:55.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-02-05T07:41:55.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-02-05T07:41:55.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-02-05T07:41:45.889224+0000", 2024-02-05T07:41:55.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0089358000000000007 2024-02-05T07:41:55.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.303 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-02-05T07:41:55.303 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-05T07:41:55.303 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-05T07:41:55.730 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pgp_num to 21 2024-02-05T07:41:56.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[29098]: 1.14 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:56.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[29098]: 1.5 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:56.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[29098]: 1.2 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:56.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[29098]: 1.13 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:56.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[29098]: 4.8 scrub starts 2024-02-05T07:41:56.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[29098]: 4.8 scrub ok 2024-02-05T07:41:56.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[29098]: 1.5 scrub starts 2024-02-05T07:41:56.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[29098]: 1.5 scrub ok 2024-02-05T07:41:56.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2144752202' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-05T07:41:56.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-05T07:41:56.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-02-05T07:41:56.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[29098]: osdmap e714: 8 total, 8 up, 8 in 2024-02-05T07:41:56.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[37158]: 1.14 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:56.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[37158]: 1.5 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:56.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[37158]: 1.2 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:56.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[37158]: 1.13 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:56.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[37158]: 4.8 scrub starts 2024-02-05T07:41:56.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[37158]: 4.8 scrub ok 2024-02-05T07:41:56.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[37158]: 1.5 scrub starts 2024-02-05T07:41:56.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[37158]: 1.5 scrub ok 2024-02-05T07:41:56.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2144752202' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-05T07:41:56.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-05T07:41:56.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-02-05T07:41:56.031 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:55 smithi007 ceph-mon[37158]: osdmap e714: 8 total, 8 up, 8 in 2024-02-05T07:41:56.186 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:55 smithi090 ceph-mon[33213]: 1.14 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:56.186 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:55 smithi090 ceph-mon[33213]: 1.5 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:56.186 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:55 smithi090 ceph-mon[33213]: 1.2 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:56.186 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:55 smithi090 ceph-mon[33213]: 1.13 starting backfill to osd.4 from (0'0,0'0] MAX to 86'128 2024-02-05T07:41:56.186 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:55 smithi090 ceph-mon[33213]: 4.8 scrub starts 2024-02-05T07:41:56.186 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:55 smithi090 ceph-mon[33213]: 4.8 scrub ok 2024-02-05T07:41:56.186 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:55 smithi090 ceph-mon[33213]: 1.5 scrub starts 2024-02-05T07:41:56.186 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:55 smithi090 ceph-mon[33213]: 1.5 scrub ok 2024-02-05T07:41:56.186 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:55 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2144752202' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-05T07:41:56.186 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:55 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-05T07:41:56.186 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:55 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-02-05T07:41:56.186 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:55 smithi090 ceph-mon[33213]: osdmap e714: 8 total, 8 up, 8 in 2024-02-05T07:41:56.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-02-05T07:41:56.670 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:57.109 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[29098]: pgmap v1304: 152 pgs: 1 remapped+peering, 1 peering, 3 active+clean+remapped, 7 active+remapped+backfill_toofull, 140 active+clean; 645 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 165 B/s, 3 objects/s recovering 2024-02-05T07:41:57.109 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[29098]: 6.3 scrub starts 2024-02-05T07:41:57.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[29098]: 6.3 scrub ok 2024-02-05T07:41:57.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:41:57.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[29098]: 1.e scrub starts 2024-02-05T07:41:57.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[29098]: 1.e scrub ok 2024-02-05T07:41:57.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[29098]: 5.1b scrub starts 2024-02-05T07:41:57.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[29098]: 5.1b scrub ok 2024-02-05T07:41:57.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[29098]: osdmap e715: 8 total, 8 up, 8 in 2024-02-05T07:41:57.110 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[37158]: pgmap v1304: 152 pgs: 1 remapped+peering, 1 peering, 3 active+clean+remapped, 7 active+remapped+backfill_toofull, 140 active+clean; 645 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 165 B/s, 3 objects/s recovering 2024-02-05T07:41:57.110 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[37158]: 6.3 scrub starts 2024-02-05T07:41:57.110 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[37158]: 6.3 scrub ok 2024-02-05T07:41:57.111 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:41:57.111 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[37158]: 1.e scrub starts 2024-02-05T07:41:57.111 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[37158]: 1.e scrub ok 2024-02-05T07:41:57.111 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[37158]: 5.1b scrub starts 2024-02-05T07:41:57.111 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[37158]: 5.1b scrub ok 2024-02-05T07:41:57.111 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:56 smithi007 ceph-mon[37158]: osdmap e715: 8 total, 8 up, 8 in 2024-02-05T07:41:57.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:56 smithi090 ceph-mon[33213]: pgmap v1304: 152 pgs: 1 remapped+peering, 1 peering, 3 active+clean+remapped, 7 active+remapped+backfill_toofull, 140 active+clean; 645 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 165 B/s, 3 objects/s recovering 2024-02-05T07:41:57.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:56 smithi090 ceph-mon[33213]: 6.3 scrub starts 2024-02-05T07:41:57.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:56 smithi090 ceph-mon[33213]: 6.3 scrub ok 2024-02-05T07:41:57.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:56 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:41:57.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:56 smithi090 ceph-mon[33213]: 1.e scrub starts 2024-02-05T07:41:57.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:56 smithi090 ceph-mon[33213]: 1.e scrub ok 2024-02-05T07:41:57.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:56 smithi090 ceph-mon[33213]: 5.1b scrub starts 2024-02-05T07:41:57.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:56 smithi090 ceph-mon[33213]: 5.1b scrub ok 2024-02-05T07:41:57.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:56 smithi090 ceph-mon[33213]: osdmap e715: 8 total, 8 up, 8 in 2024-02-05T07:41:58.008 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:57 smithi007 ceph-mon[29098]: 6.14 scrub starts 2024-02-05T07:41:58.008 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:57 smithi007 ceph-mon[29098]: 6.14 scrub ok 2024-02-05T07:41:58.008 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:57 smithi007 ceph-mon[29098]: 6.4 scrub starts 2024-02-05T07:41:58.008 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:57 smithi007 ceph-mon[29098]: 6.4 scrub ok 2024-02-05T07:41:58.008 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:57 smithi007 ceph-mon[29098]: 1.5 scrub starts 2024-02-05T07:41:58.008 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:57 smithi007 ceph-mon[29098]: 1.5 scrub ok 2024-02-05T07:41:58.009 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:57 smithi007 ceph-mon[37158]: 6.14 scrub starts 2024-02-05T07:41:58.009 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:57 smithi007 ceph-mon[37158]: 6.14 scrub ok 2024-02-05T07:41:58.009 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:57 smithi007 ceph-mon[37158]: 6.4 scrub starts 2024-02-05T07:41:58.009 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:57 smithi007 ceph-mon[37158]: 6.4 scrub ok 2024-02-05T07:41:58.009 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:57 smithi007 ceph-mon[37158]: 1.5 scrub starts 2024-02-05T07:41:58.009 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:57 smithi007 ceph-mon[37158]: 1.5 scrub ok 2024-02-05T07:41:58.229 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a: {} 2024-02-05T07:41:58.240 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b: {} 2024-02-05T07:41:58.250 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c: {} 2024-02-05T07:41:58.251 INFO:teuthology.orchestra.run.smithi007.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T07:41:58.251 INFO:teuthology.orchestra.run.smithi007.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T07:41:58.251 INFO:teuthology.orchestra.run.smithi007.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T07:41:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:57 smithi090 ceph-mon[33213]: 6.14 scrub starts 2024-02-05T07:41:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:57 smithi090 ceph-mon[33213]: 6.14 scrub ok 2024-02-05T07:41:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:57 smithi090 ceph-mon[33213]: 6.4 scrub starts 2024-02-05T07:41:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:57 smithi090 ceph-mon[33213]: 6.4 scrub ok 2024-02-05T07:41:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:57 smithi090 ceph-mon[33213]: 1.5 scrub starts 2024-02-05T07:41:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:57 smithi090 ceph-mon[33213]: 1.5 scrub ok 2024-02-05T07:41:58.822 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 0 1 2024-02-05T07:41:58.890 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:58 smithi090 ceph-mon[33213]: pgmap v1307: 152 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 3 peering, 1 active+clean+remapped, 6 active+remapped+backfill_toofull, 140 active+clean; 645 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 19 objects/s recovering 2024-02-05T07:41:59.071 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:41:59.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[29098]: pgmap v1307: 152 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 3 peering, 1 active+clean+remapped, 6 active+remapped+backfill_toofull, 140 active+clean; 645 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 19 objects/s recovering 2024-02-05T07:41:59.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[29098]: 4.1c scrub starts 2024-02-05T07:41:59.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[29098]: 4.1c scrub ok 2024-02-05T07:41:59.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[29098]: 6.8 scrub starts 2024-02-05T07:41:59.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[29098]: 6.8 scrub ok 2024-02-05T07:41:59.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[29098]: 4.1f scrub starts 2024-02-05T07:41:59.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[29098]: 4.1f scrub ok 2024-02-05T07:41:59.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[29098]: 5.12 scrub starts 2024-02-05T07:41:59.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[29098]: 5.12 scrub ok 2024-02-05T07:41:59.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2640338668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:41:59.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[37158]: pgmap v1307: 152 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 3 peering, 1 active+clean+remapped, 6 active+remapped+backfill_toofull, 140 active+clean; 645 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 19 objects/s recovering 2024-02-05T07:41:59.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[37158]: 4.1c scrub starts 2024-02-05T07:41:59.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[37158]: 4.1c scrub ok 2024-02-05T07:41:59.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[37158]: 6.8 scrub starts 2024-02-05T07:41:59.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[37158]: 6.8 scrub ok 2024-02-05T07:41:59.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[37158]: 4.1f scrub starts 2024-02-05T07:41:59.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[37158]: 4.1f scrub ok 2024-02-05T07:41:59.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[37158]: 5.12 scrub starts 2024-02-05T07:41:59.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[37158]: 5.12 scrub ok 2024-02-05T07:41:59.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:58 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2640338668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:41:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:58 smithi090 ceph-mon[33213]: 4.1c scrub starts 2024-02-05T07:41:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:58 smithi090 ceph-mon[33213]: 4.1c scrub ok 2024-02-05T07:41:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:58 smithi090 ceph-mon[33213]: 6.8 scrub starts 2024-02-05T07:41:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:58 smithi090 ceph-mon[33213]: 6.8 scrub ok 2024-02-05T07:41:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:58 smithi090 ceph-mon[33213]: 4.1f scrub starts 2024-02-05T07:41:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:58 smithi090 ceph-mon[33213]: 4.1f scrub ok 2024-02-05T07:41:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:58 smithi090 ceph-mon[33213]: 5.12 scrub starts 2024-02-05T07:41:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:58 smithi090 ceph-mon[33213]: 5.12 scrub ok 2024-02-05T07:41:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:58 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2640338668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T07:42:00.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[29098]: 4.14 scrub starts 2024-02-05T07:42:00.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[29098]: 4.14 scrub ok 2024-02-05T07:42:00.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[29098]: 4.a scrub starts 2024-02-05T07:42:00.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[29098]: 4.a scrub ok 2024-02-05T07:42:00.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[29098]: 6.7 scrub starts 2024-02-05T07:42:00.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[29098]: 6.7 scrub ok 2024-02-05T07:42:00.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[29098]: pgmap v1308: 152 pgs: 1 active+clean+scrubbing, 3 peering, 1 active+clean+remapped, 6 active+remapped+backfill_toofull, 141 active+clean; 645 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 13 objects/s recovering 2024-02-05T07:42:00.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[29098]: osdmap e716: 8 total, 8 up, 8 in 2024-02-05T07:42:00.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[29098]: 5.3 scrub starts 2024-02-05T07:42:00.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[29098]: 5.3 scrub ok 2024-02-05T07:42:00.121 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[37158]: 4.14 scrub starts 2024-02-05T07:42:00.121 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[37158]: 4.14 scrub ok 2024-02-05T07:42:00.121 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[37158]: 4.a scrub starts 2024-02-05T07:42:00.121 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[37158]: 4.a scrub ok 2024-02-05T07:42:00.121 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[37158]: 6.7 scrub starts 2024-02-05T07:42:00.121 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[37158]: 6.7 scrub ok 2024-02-05T07:42:00.121 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[37158]: pgmap v1308: 152 pgs: 1 active+clean+scrubbing, 3 peering, 1 active+clean+remapped, 6 active+remapped+backfill_toofull, 141 active+clean; 645 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 13 objects/s recovering 2024-02-05T07:42:00.121 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[37158]: osdmap e716: 8 total, 8 up, 8 in 2024-02-05T07:42:00.121 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[37158]: 5.3 scrub starts 2024-02-05T07:42:00.121 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:41:59 smithi007 ceph-mon[37158]: 5.3 scrub ok 2024-02-05T07:42:00.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:59 smithi090 ceph-mon[33213]: 4.14 scrub starts 2024-02-05T07:42:00.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:59 smithi090 ceph-mon[33213]: 4.14 scrub ok 2024-02-05T07:42:00.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:59 smithi090 ceph-mon[33213]: 4.a scrub starts 2024-02-05T07:42:00.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:59 smithi090 ceph-mon[33213]: 4.a scrub ok 2024-02-05T07:42:00.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:59 smithi090 ceph-mon[33213]: 6.7 scrub starts 2024-02-05T07:42:00.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:59 smithi090 ceph-mon[33213]: 6.7 scrub ok 2024-02-05T07:42:00.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:59 smithi090 ceph-mon[33213]: pgmap v1308: 152 pgs: 1 active+clean+scrubbing, 3 peering, 1 active+clean+remapped, 6 active+remapped+backfill_toofull, 141 active+clean; 645 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 13 objects/s recovering 2024-02-05T07:42:00.176 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:59 smithi090 ceph-mon[33213]: osdmap e716: 8 total, 8 up, 8 in 2024-02-05T07:42:00.176 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:59 smithi090 ceph-mon[33213]: 5.3 scrub starts 2024-02-05T07:42:00.176 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:41:59 smithi090 ceph-mon[33213]: 5.3 scrub ok 2024-02-05T07:42:00.902 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.0 to 1 (10000) 2024-02-05T07:42:01.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:00 smithi007 ceph-mon[29098]: 6.15 scrub starts 2024-02-05T07:42:01.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:00 smithi007 ceph-mon[29098]: 6.15 scrub ok 2024-02-05T07:42:01.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:00 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3505643092' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:42:01.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:00 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:42:01.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:00 smithi007 ceph-mon[37158]: 6.15 scrub starts 2024-02-05T07:42:01.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:00 smithi007 ceph-mon[37158]: 6.15 scrub ok 2024-02-05T07:42:01.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:00 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3505643092' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:42:01.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:00 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:42:01.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:00 smithi090 ceph-mon[33213]: 6.15 scrub starts 2024-02-05T07:42:01.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:00 smithi090 ceph-mon[33213]: 6.15 scrub ok 2024-02-05T07:42:01.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:00 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3505643092' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:42:01.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:00 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:42:01.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:42:01.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 0 1 2024-02-05T07:42:02.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[37158]: 4.11 scrub starts 2024-02-05T07:42:02.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[37158]: 4.11 scrub ok 2024-02-05T07:42:02.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[37158]: 6.5 deep-scrub starts 2024-02-05T07:42:02.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[37158]: 6.5 deep-scrub ok 2024-02-05T07:42:02.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T07:42:02.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[37158]: osdmap e717: 8 total, 8 up, 8 in 2024-02-05T07:42:02.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[37158]: 1.12 scrub starts 2024-02-05T07:42:02.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[37158]: 1.12 scrub ok 2024-02-05T07:42:02.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[37158]: 4.c scrub starts 2024-02-05T07:42:02.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[37158]: 4.c scrub ok 2024-02-05T07:42:02.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:02.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:42:02.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:42:02.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[37158]: pgmap v1311: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 891 B/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-02-05T07:42:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[29098]: 4.11 scrub starts 2024-02-05T07:42:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[29098]: 4.11 scrub ok 2024-02-05T07:42:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[29098]: 6.5 deep-scrub starts 2024-02-05T07:42:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[29098]: 6.5 deep-scrub ok 2024-02-05T07:42:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T07:42:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[29098]: osdmap e717: 8 total, 8 up, 8 in 2024-02-05T07:42:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[29098]: 1.12 scrub starts 2024-02-05T07:42:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[29098]: 1.12 scrub ok 2024-02-05T07:42:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[29098]: 4.c scrub starts 2024-02-05T07:42:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[29098]: 4.c scrub ok 2024-02-05T07:42:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:02.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:42:02.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:42:02.175 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:01 smithi007 ceph-mon[29098]: pgmap v1311: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 891 B/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-02-05T07:42:02.176 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-mon[33213]: 4.11 scrub starts 2024-02-05T07:42:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-mon[33213]: 4.11 scrub ok 2024-02-05T07:42:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-mon[33213]: 6.5 deep-scrub starts 2024-02-05T07:42:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-mon[33213]: 6.5 deep-scrub ok 2024-02-05T07:42:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T07:42:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-mon[33213]: osdmap e717: 8 total, 8 up, 8 in 2024-02-05T07:42:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-mon[33213]: 1.12 scrub starts 2024-02-05T07:42:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-mon[33213]: 1.12 scrub ok 2024-02-05T07:42:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-mon[33213]: 4.c scrub starts 2024-02-05T07:42:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-mon[33213]: 4.c scrub ok 2024-02-05T07:42:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:42:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:42:02.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:01 smithi090 ceph-mon[33213]: pgmap v1311: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 891 B/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-02-05T07:42:03.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:02 smithi007 ceph-mon[29098]: 1.4 scrub starts 2024-02-05T07:42:03.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:02 smithi007 ceph-mon[29098]: 1.4 scrub ok 2024-02-05T07:42:03.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:02 smithi007 ceph-mon[29098]: 6.0 scrub starts 2024-02-05T07:42:03.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:02 smithi007 ceph-mon[29098]: 6.0 scrub ok 2024-02-05T07:42:03.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:02 smithi007 ceph-mon[37158]: 1.4 scrub starts 2024-02-05T07:42:03.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:02 smithi007 ceph-mon[37158]: 1.4 scrub ok 2024-02-05T07:42:03.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:02 smithi007 ceph-mon[37158]: 6.0 scrub starts 2024-02-05T07:42:03.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:02 smithi007 ceph-mon[37158]: 6.0 scrub ok 2024-02-05T07:42:03.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:02 smithi090 ceph-mon[33213]: 1.4 scrub starts 2024-02-05T07:42:03.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:02 smithi090 ceph-mon[33213]: 1.4 scrub ok 2024-02-05T07:42:03.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:02 smithi090 ceph-mon[33213]: 6.0 scrub starts 2024-02-05T07:42:03.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:02 smithi090 ceph-mon[33213]: 6.0 scrub ok 2024-02-05T07:42:03.612 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:42:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:42:03.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-mon[37158]: 3.c scrub starts 2024-02-05T07:42:03.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-mon[37158]: 3.c scrub ok 2024-02-05T07:42:03.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-mon[37158]: 5.4 scrub starts 2024-02-05T07:42:03.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-mon[37158]: 5.4 scrub ok 2024-02-05T07:42:03.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-mon[37158]: 5.1c scrub starts 2024-02-05T07:42:03.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-mon[37158]: 5.1c scrub ok 2024-02-05T07:42:03.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-mon[37158]: pgmap v1312: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:42:04.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-mon[29098]: 3.c scrub starts 2024-02-05T07:42:04.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-mon[29098]: 3.c scrub ok 2024-02-05T07:42:04.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-mon[29098]: 5.4 scrub starts 2024-02-05T07:42:04.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-mon[29098]: 5.4 scrub ok 2024-02-05T07:42:04.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-mon[29098]: 5.1c scrub starts 2024-02-05T07:42:04.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-mon[29098]: 5.1c scrub ok 2024-02-05T07:42:04.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:03 smithi007 ceph-mon[29098]: pgmap v1312: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:42:04.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:03 smithi090 ceph-mon[33213]: 3.c scrub starts 2024-02-05T07:42:04.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:03 smithi090 ceph-mon[33213]: 3.c scrub ok 2024-02-05T07:42:04.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:03 smithi090 ceph-mon[33213]: 5.4 scrub starts 2024-02-05T07:42:04.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:03 smithi090 ceph-mon[33213]: 5.4 scrub ok 2024-02-05T07:42:04.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:03 smithi090 ceph-mon[33213]: 5.1c scrub starts 2024-02-05T07:42:04.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:03 smithi090 ceph-mon[33213]: 5.1c scrub ok 2024-02-05T07:42:04.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:03 smithi090 ceph-mon[33213]: pgmap v1312: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:42:04.917 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-02-05T07:42:05.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:42:05.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:42:05.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[29098]: 6.16 scrub starts 2024-02-05T07:42:05.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[29098]: 6.16 scrub ok 2024-02-05T07:42:05.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[29098]: osdmap e718: 8 total, 8 up, 8 in 2024-02-05T07:42:05.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[29098]: 6.1 scrub starts 2024-02-05T07:42:05.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[29098]: 6.1 scrub ok 2024-02-05T07:42:05.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/162550421' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:42:05.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:05.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:42:05.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:42:05.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[37158]: 6.16 scrub starts 2024-02-05T07:42:05.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[37158]: 6.16 scrub ok 2024-02-05T07:42:05.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[37158]: osdmap e718: 8 total, 8 up, 8 in 2024-02-05T07:42:05.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[37158]: 6.1 scrub starts 2024-02-05T07:42:05.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[37158]: 6.1 scrub ok 2024-02-05T07:42:05.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/162550421' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:42:05.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:04 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:05.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:04 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:42:05.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:04 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:42:05.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:04 smithi090 ceph-mon[33213]: 6.16 scrub starts 2024-02-05T07:42:05.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:04 smithi090 ceph-mon[33213]: 6.16 scrub ok 2024-02-05T07:42:05.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:04 smithi090 ceph-mon[33213]: osdmap e718: 8 total, 8 up, 8 in 2024-02-05T07:42:05.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:04 smithi090 ceph-mon[33213]: 6.1 scrub starts 2024-02-05T07:42:05.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:04 smithi090 ceph-mon[33213]: 6.1 scrub ok 2024-02-05T07:42:05.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:04 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/162550421' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T07:42:05.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:04 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:05.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 1 1 2024-02-05T07:42:05.904 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:06.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[29098]: 1.c scrub starts 2024-02-05T07:42:06.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[29098]: 1.c scrub ok 2024-02-05T07:42:06.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/162550421' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-05T07:42:06.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[29098]: osdmap e719: 8 total, 8 up, 8 in 2024-02-05T07:42:06.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[29098]: 4.1b deep-scrub starts 2024-02-05T07:42:06.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[29098]: 4.1b deep-scrub ok 2024-02-05T07:42:06.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[29098]: pgmap v1315: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:42:06.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[29098]: osdmap e720: 8 total, 8 up, 8 in 2024-02-05T07:42:06.259 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[37158]: 1.c scrub starts 2024-02-05T07:42:06.259 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[37158]: 1.c scrub ok 2024-02-05T07:42:06.259 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/162550421' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-05T07:42:06.259 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[37158]: osdmap e719: 8 total, 8 up, 8 in 2024-02-05T07:42:06.259 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[37158]: 4.1b deep-scrub starts 2024-02-05T07:42:06.259 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[37158]: 4.1b deep-scrub ok 2024-02-05T07:42:06.259 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[37158]: pgmap v1315: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:42:06.259 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:05 smithi007 ceph-mon[37158]: osdmap e720: 8 total, 8 up, 8 in 2024-02-05T07:42:06.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:05 smithi090 ceph-mon[33213]: 1.c scrub starts 2024-02-05T07:42:06.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:05 smithi090 ceph-mon[33213]: 1.c scrub ok 2024-02-05T07:42:06.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:05 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/162550421' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-05T07:42:06.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:05 smithi090 ceph-mon[33213]: osdmap e719: 8 total, 8 up, 8 in 2024-02-05T07:42:06.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:05 smithi090 ceph-mon[33213]: 4.1b deep-scrub starts 2024-02-05T07:42:06.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:05 smithi090 ceph-mon[33213]: 4.1b deep-scrub ok 2024-02-05T07:42:06.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:05 smithi090 ceph-mon[33213]: pgmap v1315: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:42:06.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:05 smithi090 ceph-mon[33213]: osdmap e720: 8 total, 8 up, 8 in 2024-02-05T07:42:07.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:06 smithi007 ceph-mon[29098]: 3.19 scrub starts 2024-02-05T07:42:07.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:06 smithi007 ceph-mon[29098]: 3.19 scrub ok 2024-02-05T07:42:07.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:06 smithi007 ceph-mon[29098]: 5.7 deep-scrub starts 2024-02-05T07:42:07.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:06 smithi007 ceph-mon[29098]: 5.7 deep-scrub ok 2024-02-05T07:42:07.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:06 smithi007 ceph-mon[29098]: 1.1 scrub starts 2024-02-05T07:42:07.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:06 smithi007 ceph-mon[29098]: 1.1 scrub ok 2024-02-05T07:42:07.005 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:06 smithi007 ceph-mon[37158]: 3.19 scrub starts 2024-02-05T07:42:07.005 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:06 smithi007 ceph-mon[37158]: 3.19 scrub ok 2024-02-05T07:42:07.005 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:06 smithi007 ceph-mon[37158]: 5.7 deep-scrub starts 2024-02-05T07:42:07.005 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:06 smithi007 ceph-mon[37158]: 5.7 deep-scrub ok 2024-02-05T07:42:07.005 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:06 smithi007 ceph-mon[37158]: 1.1 scrub starts 2024-02-05T07:42:07.005 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:06 smithi007 ceph-mon[37158]: 1.1 scrub ok 2024-02-05T07:42:07.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:06 smithi090 ceph-mon[33213]: 3.19 scrub starts 2024-02-05T07:42:07.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:06 smithi090 ceph-mon[33213]: 3.19 scrub ok 2024-02-05T07:42:07.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:06 smithi090 ceph-mon[33213]: 5.7 deep-scrub starts 2024-02-05T07:42:07.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:06 smithi090 ceph-mon[33213]: 5.7 deep-scrub ok 2024-02-05T07:42:07.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:06 smithi090 ceph-mon[33213]: 1.1 scrub starts 2024-02-05T07:42:07.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:06 smithi090 ceph-mon[33213]: 1.1 scrub ok 2024-02-05T07:42:07.937 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.1 to 1 (10000) 2024-02-05T07:42:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:07 smithi090 ceph-mon[33213]: 3.f scrub starts 2024-02-05T07:42:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:07 smithi090 ceph-mon[33213]: 3.f scrub ok 2024-02-05T07:42:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:07 smithi090 ceph-mon[33213]: 2.2 scrub starts 2024-02-05T07:42:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:07 smithi090 ceph-mon[33213]: 2.2 scrub ok 2024-02-05T07:42:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:07 smithi090 ceph-mon[33213]: 1.11 scrub starts 2024-02-05T07:42:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:07 smithi090 ceph-mon[33213]: 1.11 scrub ok 2024-02-05T07:42:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:07 smithi090 ceph-mon[33213]: pgmap v1317: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 661 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.5 KiB/s rd, 18 KiB/s wr, 3 op/s; 13 KiB/s, 0 objects/s recovering 2024-02-05T07:42:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:07 smithi090 ceph-mon[33213]: 5.11 scrub starts 2024-02-05T07:42:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:07 smithi090 ceph-mon[33213]: 5.11 scrub ok 2024-02-05T07:42:08.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:07 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/27885829' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:42:08.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[29098]: 3.f scrub starts 2024-02-05T07:42:08.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[29098]: 3.f scrub ok 2024-02-05T07:42:08.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[29098]: 2.2 scrub starts 2024-02-05T07:42:08.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[29098]: 2.2 scrub ok 2024-02-05T07:42:08.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[29098]: 1.11 scrub starts 2024-02-05T07:42:08.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[29098]: 1.11 scrub ok 2024-02-05T07:42:08.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[29098]: pgmap v1317: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 661 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.5 KiB/s rd, 18 KiB/s wr, 3 op/s; 13 KiB/s, 0 objects/s recovering 2024-02-05T07:42:08.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[29098]: 5.11 scrub starts 2024-02-05T07:42:08.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[29098]: 5.11 scrub ok 2024-02-05T07:42:08.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/27885829' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:42:08.309 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[37158]: 3.f scrub starts 2024-02-05T07:42:08.309 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[37158]: 3.f scrub ok 2024-02-05T07:42:08.309 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[37158]: 2.2 scrub starts 2024-02-05T07:42:08.309 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[37158]: 2.2 scrub ok 2024-02-05T07:42:08.309 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[37158]: 1.11 scrub starts 2024-02-05T07:42:08.310 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[37158]: 1.11 scrub ok 2024-02-05T07:42:08.310 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[37158]: pgmap v1317: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 661 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.5 KiB/s rd, 18 KiB/s wr, 3 op/s; 13 KiB/s, 0 objects/s recovering 2024-02-05T07:42:08.310 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[37158]: 5.11 scrub starts 2024-02-05T07:42:08.310 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[37158]: 5.11 scrub ok 2024-02-05T07:42:08.310 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:07 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/27885829' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:42:08.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 1 1 2024-02-05T07:42:08.864 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:09.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:08 smithi007 ceph-mon[29098]: 3.1e scrub starts 2024-02-05T07:42:09.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:08 smithi007 ceph-mon[29098]: 3.1e scrub ok 2024-02-05T07:42:09.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:08 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/27885829' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T07:42:09.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:08 smithi007 ceph-mon[29098]: osdmap e721: 8 total, 8 up, 8 in 2024-02-05T07:42:09.242 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:08 smithi007 ceph-mon[37158]: 3.1e scrub starts 2024-02-05T07:42:09.242 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:08 smithi007 ceph-mon[37158]: 3.1e scrub ok 2024-02-05T07:42:09.242 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:08 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/27885829' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T07:42:09.242 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:08 smithi007 ceph-mon[37158]: osdmap e721: 8 total, 8 up, 8 in 2024-02-05T07:42:09.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:08 smithi090 ceph-mon[33213]: 3.1e scrub starts 2024-02-05T07:42:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:08 smithi090 ceph-mon[33213]: 3.1e scrub ok 2024-02-05T07:42:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:08 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/27885829' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T07:42:09.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:08 smithi090 ceph-mon[33213]: osdmap e721: 8 total, 8 up, 8 in 2024-02-05T07:42:10.224 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:09 smithi090 ceph-mon[33213]: 4.19 scrub starts 2024-02-05T07:42:10.225 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:09 smithi090 ceph-mon[33213]: 4.19 scrub ok 2024-02-05T07:42:10.225 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:09 smithi090 ceph-mon[33213]: pgmap v1319: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 661 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.7 KiB/s rd, 20 KiB/s wr, 3 op/s; 14 KiB/s, 0 objects/s recovering 2024-02-05T07:42:10.225 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:09 smithi090 ceph-mon[33213]: osdmap e722: 8 total, 8 up, 8 in 2024-02-05T07:42:10.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:09 smithi007 ceph-mon[29098]: 4.19 scrub starts 2024-02-05T07:42:10.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:09 smithi007 ceph-mon[29098]: 4.19 scrub ok 2024-02-05T07:42:10.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:09 smithi007 ceph-mon[29098]: pgmap v1319: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 661 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.7 KiB/s rd, 20 KiB/s wr, 3 op/s; 14 KiB/s, 0 objects/s recovering 2024-02-05T07:42:10.389 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:09 smithi007 ceph-mon[29098]: osdmap e722: 8 total, 8 up, 8 in 2024-02-05T07:42:10.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:09 smithi007 ceph-mon[37158]: 4.19 scrub starts 2024-02-05T07:42:10.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:09 smithi007 ceph-mon[37158]: 4.19 scrub ok 2024-02-05T07:42:10.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:09 smithi007 ceph-mon[37158]: pgmap v1319: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 661 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.7 KiB/s rd, 20 KiB/s wr, 3 op/s; 14 KiB/s, 0 objects/s recovering 2024-02-05T07:42:10.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:09 smithi007 ceph-mon[37158]: osdmap e722: 8 total, 8 up, 8 in 2024-02-05T07:42:10.956 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-02-05T07:42:11.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:10 smithi090 ceph-mon[33213]: 6.11 scrub starts 2024-02-05T07:42:11.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:10 smithi090 ceph-mon[33213]: 6.11 scrub ok 2024-02-05T07:42:11.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:10 smithi090 ceph-mon[33213]: 3.11 scrub starts 2024-02-05T07:42:11.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:10 smithi090 ceph-mon[33213]: 3.11 scrub ok 2024-02-05T07:42:11.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:10 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/372707819' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:42:11.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:10 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:42:11.367 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:10 smithi007 ceph-mon[29098]: 6.11 scrub starts 2024-02-05T07:42:11.367 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:10 smithi007 ceph-mon[29098]: 6.11 scrub ok 2024-02-05T07:42:11.367 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:10 smithi007 ceph-mon[29098]: 3.11 scrub starts 2024-02-05T07:42:11.368 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:10 smithi007 ceph-mon[29098]: 3.11 scrub ok 2024-02-05T07:42:11.368 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:10 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/372707819' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:42:11.368 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:10 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:42:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:10 smithi007 ceph-mon[37158]: 6.11 scrub starts 2024-02-05T07:42:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:10 smithi007 ceph-mon[37158]: 6.11 scrub ok 2024-02-05T07:42:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:10 smithi007 ceph-mon[37158]: 3.11 scrub starts 2024-02-05T07:42:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:10 smithi007 ceph-mon[37158]: 3.11 scrub ok 2024-02-05T07:42:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:10 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/372707819' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:42:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:10 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T07:42:11.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 2 1 2024-02-05T07:42:11.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:42:11 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:42:11.982 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:12.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[29098]: 6.c scrub starts 2024-02-05T07:42:12.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[29098]: 6.c scrub ok 2024-02-05T07:42:12.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[29098]: 5.10 scrub starts 2024-02-05T07:42:12.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[29098]: 5.10 scrub ok 2024-02-05T07:42:12.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:42:12.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:42:12.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-05T07:42:12.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[29098]: osdmap e723: 8 total, 8 up, 8 in 2024-02-05T07:42:12.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[29098]: 6.18 scrub starts 2024-02-05T07:42:12.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[29098]: 6.18 scrub ok 2024-02-05T07:42:12.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:12.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[29098]: pgmap v1322: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 661 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.7 KiB/s rd, 20 KiB/s wr, 3 op/s; 14 KiB/s, 0 objects/s recovering 2024-02-05T07:42:12.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[29098]: osdmap e724: 8 total, 8 up, 8 in 2024-02-05T07:42:12.250 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[37158]: 6.c scrub starts 2024-02-05T07:42:12.250 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[37158]: 6.c scrub ok 2024-02-05T07:42:12.250 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[37158]: 5.10 scrub starts 2024-02-05T07:42:12.250 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[37158]: 5.10 scrub ok 2024-02-05T07:42:12.250 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:42:12.250 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:42:12.250 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-05T07:42:12.250 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[37158]: osdmap e723: 8 total, 8 up, 8 in 2024-02-05T07:42:12.250 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[37158]: 6.18 scrub starts 2024-02-05T07:42:12.250 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[37158]: 6.18 scrub ok 2024-02-05T07:42:12.250 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:12.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[37158]: pgmap v1322: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 661 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.7 KiB/s rd, 20 KiB/s wr, 3 op/s; 14 KiB/s, 0 objects/s recovering 2024-02-05T07:42:12.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:11 smithi007 ceph-mon[37158]: osdmap e724: 8 total, 8 up, 8 in 2024-02-05T07:42:12.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:11 smithi090 ceph-mon[33213]: 6.c scrub starts 2024-02-05T07:42:12.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:11 smithi090 ceph-mon[33213]: 6.c scrub ok 2024-02-05T07:42:12.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:11 smithi090 ceph-mon[33213]: 5.10 scrub starts 2024-02-05T07:42:12.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:11 smithi090 ceph-mon[33213]: 5.10 scrub ok 2024-02-05T07:42:12.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:11 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:42:12.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:11 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:42:12.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:11 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-05T07:42:12.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:11 smithi090 ceph-mon[33213]: osdmap e723: 8 total, 8 up, 8 in 2024-02-05T07:42:12.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:11 smithi090 ceph-mon[33213]: 6.18 scrub starts 2024-02-05T07:42:12.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:11 smithi090 ceph-mon[33213]: 6.18 scrub ok 2024-02-05T07:42:12.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:11 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:12.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:11 smithi090 ceph-mon[33213]: pgmap v1322: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 661 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.7 KiB/s rd, 20 KiB/s wr, 3 op/s; 14 KiB/s, 0 objects/s recovering 2024-02-05T07:42:12.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:11 smithi090 ceph-mon[33213]: osdmap e724: 8 total, 8 up, 8 in 2024-02-05T07:42:13.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:12 smithi007 ceph-mon[29098]: 1.3 scrub starts 2024-02-05T07:42:13.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:12 smithi007 ceph-mon[29098]: 1.3 scrub ok 2024-02-05T07:42:13.013 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:12 smithi007 ceph-mon[37158]: 1.3 scrub starts 2024-02-05T07:42:13.014 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:12 smithi007 ceph-mon[37158]: 1.3 scrub ok 2024-02-05T07:42:13.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:12 smithi090 ceph-mon[33213]: 1.3 scrub starts 2024-02-05T07:42:13.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:12 smithi090 ceph-mon[33213]: 1.3 scrub ok 2024-02-05T07:42:13.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:42:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:42:13.980 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.2 to 1 (10000) 2024-02-05T07:42:14.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:13 smithi090 ceph-mon[33213]: osdmap e725: 8 total, 8 up, 8 in 2024-02-05T07:42:14.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:13 smithi090 ceph-mon[33213]: 1.13 continuing backfill to osd.4 from (86'128,86'128] MIN to 86'128 2024-02-05T07:42:14.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:13 smithi090 ceph-mon[33213]: 6.d deep-scrub starts 2024-02-05T07:42:14.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:13 smithi090 ceph-mon[33213]: 1.b scrub starts 2024-02-05T07:42:14.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:13 smithi090 ceph-mon[33213]: 1.b scrub ok 2024-02-05T07:42:14.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:13 smithi090 ceph-mon[33213]: 6.d deep-scrub ok 2024-02-05T07:42:14.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:13 smithi090 ceph-mon[33213]: pgmap v1325: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:14.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:13 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2966342841' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:42:14.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:13 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:42:14.288 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[29098]: osdmap e725: 8 total, 8 up, 8 in 2024-02-05T07:42:14.288 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[29098]: 1.13 continuing backfill to osd.4 from (86'128,86'128] MIN to 86'128 2024-02-05T07:42:14.288 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[29098]: 6.d deep-scrub starts 2024-02-05T07:42:14.288 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[29098]: 1.b scrub starts 2024-02-05T07:42:14.288 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[29098]: 1.b scrub ok 2024-02-05T07:42:14.288 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[29098]: 6.d deep-scrub ok 2024-02-05T07:42:14.288 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[29098]: pgmap v1325: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:14.288 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2966342841' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:42:14.288 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:42:14.289 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[37158]: osdmap e725: 8 total, 8 up, 8 in 2024-02-05T07:42:14.289 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[37158]: 1.13 continuing backfill to osd.4 from (86'128,86'128] MIN to 86'128 2024-02-05T07:42:14.289 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[37158]: 6.d deep-scrub starts 2024-02-05T07:42:14.289 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[37158]: 1.b scrub starts 2024-02-05T07:42:14.289 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[37158]: 1.b scrub ok 2024-02-05T07:42:14.289 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[37158]: 6.d deep-scrub ok 2024-02-05T07:42:14.289 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[37158]: pgmap v1325: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:14.289 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2966342841' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:42:14.289 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:13 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:42:14.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 2 1 2024-02-05T07:42:14.980 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:15.244 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T07:42:15.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[29098]: osdmap e726: 8 total, 8 up, 8 in 2024-02-05T07:42:15.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[29098]: 3.0 scrub starts 2024-02-05T07:42:15.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[29098]: 3.0 scrub ok 2024-02-05T07:42:15.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[29098]: 5.1 deep-scrub starts 2024-02-05T07:42:15.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[29098]: 5.1 deep-scrub ok 2024-02-05T07:42:15.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[29098]: 3.1 deep-scrub starts 2024-02-05T07:42:15.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[29098]: 3.1 deep-scrub ok 2024-02-05T07:42:15.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[29098]: 5.1a scrub starts 2024-02-05T07:42:15.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[29098]: 5.1a scrub ok 2024-02-05T07:42:15.245 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T07:42:15.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[37158]: osdmap e726: 8 total, 8 up, 8 in 2024-02-05T07:42:15.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[37158]: 3.0 scrub starts 2024-02-05T07:42:15.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[37158]: 3.0 scrub ok 2024-02-05T07:42:15.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[37158]: 5.1 deep-scrub starts 2024-02-05T07:42:15.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[37158]: 5.1 deep-scrub ok 2024-02-05T07:42:15.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[37158]: 3.1 deep-scrub starts 2024-02-05T07:42:15.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[37158]: 3.1 deep-scrub ok 2024-02-05T07:42:15.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[37158]: 5.1a scrub starts 2024-02-05T07:42:15.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:14 smithi007 ceph-mon[37158]: 5.1a scrub ok 2024-02-05T07:42:15.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:14 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T07:42:15.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:14 smithi090 ceph-mon[33213]: osdmap e726: 8 total, 8 up, 8 in 2024-02-05T07:42:15.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:14 smithi090 ceph-mon[33213]: 3.0 scrub starts 2024-02-05T07:42:15.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:14 smithi090 ceph-mon[33213]: 3.0 scrub ok 2024-02-05T07:42:15.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:14 smithi090 ceph-mon[33213]: 5.1 deep-scrub starts 2024-02-05T07:42:15.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:14 smithi090 ceph-mon[33213]: 5.1 deep-scrub ok 2024-02-05T07:42:15.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:14 smithi090 ceph-mon[33213]: 3.1 deep-scrub starts 2024-02-05T07:42:15.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:14 smithi090 ceph-mon[33213]: 3.1 deep-scrub ok 2024-02-05T07:42:15.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:14 smithi090 ceph-mon[33213]: 5.1a scrub starts 2024-02-05T07:42:15.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:14 smithi090 ceph-mon[33213]: 5.1a scrub ok 2024-02-05T07:42:16.063 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:15 smithi007 ceph-mon[29098]: osdmap e727: 8 total, 8 up, 8 in 2024-02-05T07:42:16.063 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:15 smithi007 ceph-mon[29098]: pgmap v1328: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:16.063 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:15 smithi007 ceph-mon[37158]: osdmap e727: 8 total, 8 up, 8 in 2024-02-05T07:42:16.063 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:15 smithi007 ceph-mon[37158]: pgmap v1328: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:16.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:15 smithi090 ceph-mon[33213]: osdmap e727: 8 total, 8 up, 8 in 2024-02-05T07:42:16.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:15 smithi090 ceph-mon[33213]: pgmap v1328: 152 pgs: 6 active+remapped+backfill_toofull, 146 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:16.995 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-02-05T07:42:17.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:16 smithi090 ceph-mon[33213]: 1.13 scrub starts 2024-02-05T07:42:17.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:16 smithi090 ceph-mon[33213]: 1.13 scrub ok 2024-02-05T07:42:17.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:16 smithi090 ceph-mon[33213]: 5.15 deep-scrub starts 2024-02-05T07:42:17.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:16 smithi090 ceph-mon[33213]: 5.15 deep-scrub ok 2024-02-05T07:42:17.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:16 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:42:17.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:16 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:42:17.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:16 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:42:17.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:16 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4186897883' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:42:17.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[29098]: 1.13 scrub starts 2024-02-05T07:42:17.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[29098]: 1.13 scrub ok 2024-02-05T07:42:17.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[29098]: 5.15 deep-scrub starts 2024-02-05T07:42:17.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[29098]: 5.15 deep-scrub ok 2024-02-05T07:42:17.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:42:17.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:42:17.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:42:17.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4186897883' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:42:17.287 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[37158]: 1.13 scrub starts 2024-02-05T07:42:17.287 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[37158]: 1.13 scrub ok 2024-02-05T07:42:17.287 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[37158]: 5.15 deep-scrub starts 2024-02-05T07:42:17.287 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[37158]: 5.15 deep-scrub ok 2024-02-05T07:42:17.287 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:42:17.287 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:42:17.287 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:42:17.287 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:16 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4186897883' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T07:42:17.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 3 1 2024-02-05T07:42:17.963 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:18.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:17 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4186897883' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-05T07:42:18.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:17 smithi090 ceph-mon[33213]: osdmap e728: 8 total, 8 up, 8 in 2024-02-05T07:42:18.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:17 smithi090 ceph-mon[33213]: pgmap v1330: 152 pgs: 5 active+remapped+backfill_toofull, 147 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-05T07:42:18.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:17 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4186897883' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-05T07:42:18.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:17 smithi007 ceph-mon[29098]: osdmap e728: 8 total, 8 up, 8 in 2024-02-05T07:42:18.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:17 smithi007 ceph-mon[29098]: pgmap v1330: 152 pgs: 5 active+remapped+backfill_toofull, 147 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-05T07:42:18.291 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:17 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4186897883' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-05T07:42:18.291 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:17 smithi007 ceph-mon[37158]: osdmap e728: 8 total, 8 up, 8 in 2024-02-05T07:42:18.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:17 smithi007 ceph-mon[37158]: pgmap v1330: 152 pgs: 5 active+remapped+backfill_toofull, 147 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-05T07:42:19.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:19 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:19.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:19 smithi007 ceph-mon[29098]: osdmap e729: 8 total, 8 up, 8 in 2024-02-05T07:42:19.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:19 smithi007 ceph-mon[29098]: 5.6 scrub starts 2024-02-05T07:42:19.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:19 smithi007 ceph-mon[29098]: 5.6 scrub ok 2024-02-05T07:42:19.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:19 smithi007 ceph-mon[29098]: 3.13 scrub starts 2024-02-05T07:42:19.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:19 smithi007 ceph-mon[29098]: 3.13 scrub ok 2024-02-05T07:42:19.259 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:19 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:19.259 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:19 smithi007 ceph-mon[37158]: osdmap e729: 8 total, 8 up, 8 in 2024-02-05T07:42:19.259 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:19 smithi007 ceph-mon[37158]: 5.6 scrub starts 2024-02-05T07:42:19.259 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:19 smithi007 ceph-mon[37158]: 5.6 scrub ok 2024-02-05T07:42:19.260 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:19 smithi007 ceph-mon[37158]: 3.13 scrub starts 2024-02-05T07:42:19.260 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:19 smithi007 ceph-mon[37158]: 3.13 scrub ok 2024-02-05T07:42:19.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:19 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:19.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:19 smithi090 ceph-mon[33213]: osdmap e729: 8 total, 8 up, 8 in 2024-02-05T07:42:19.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:19 smithi090 ceph-mon[33213]: 5.6 scrub starts 2024-02-05T07:42:19.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:19 smithi090 ceph-mon[33213]: 5.6 scrub ok 2024-02-05T07:42:19.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:19 smithi090 ceph-mon[33213]: 3.13 scrub starts 2024-02-05T07:42:19.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:19 smithi090 ceph-mon[33213]: 3.13 scrub ok 2024-02-05T07:42:20.014 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.3 to 1 (10000) 2024-02-05T07:42:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:20 smithi090 ceph-mon[33213]: osdmap e730: 8 total, 8 up, 8 in 2024-02-05T07:42:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:20 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:42:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:20 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:42:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:20 smithi090 ceph-mon[33213]: pgmap v1333: 152 pgs: 1 peering, 4 active+remapped+backfill_toofull, 147 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-05T07:42:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:20 smithi090 ceph-mon[33213]: 3.d scrub starts 2024-02-05T07:42:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:20 smithi090 ceph-mon[33213]: 3.d scrub ok 2024-02-05T07:42:20.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:20 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4230349014' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:42:20.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:20 smithi007 ceph-mon[29098]: osdmap e730: 8 total, 8 up, 8 in 2024-02-05T07:42:20.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:20 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:42:20.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:20 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:42:20.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:20 smithi007 ceph-mon[29098]: pgmap v1333: 152 pgs: 1 peering, 4 active+remapped+backfill_toofull, 147 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-05T07:42:20.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:20 smithi007 ceph-mon[29098]: 3.d scrub starts 2024-02-05T07:42:20.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:20 smithi007 ceph-mon[29098]: 3.d scrub ok 2024-02-05T07:42:20.312 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:20 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4230349014' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:42:20.312 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:20 smithi007 ceph-mon[37158]: osdmap e730: 8 total, 8 up, 8 in 2024-02-05T07:42:20.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:20 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:42:20.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:20 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:42:20.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:20 smithi007 ceph-mon[37158]: pgmap v1333: 152 pgs: 1 peering, 4 active+remapped+backfill_toofull, 147 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-05T07:42:20.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:20 smithi007 ceph-mon[37158]: 3.d scrub starts 2024-02-05T07:42:20.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:20 smithi007 ceph-mon[37158]: 3.d scrub ok 2024-02-05T07:42:20.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:20 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4230349014' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:42:20.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 3 1 2024-02-05T07:42:20.948 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:21.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:21 smithi090 ceph-mon[33213]: 5.0 scrub starts 2024-02-05T07:42:21.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:21 smithi090 ceph-mon[33213]: 5.0 scrub ok 2024-02-05T07:42:21.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:21 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4230349014' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T07:42:21.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:21 smithi090 ceph-mon[33213]: osdmap e731: 8 total, 8 up, 8 in 2024-02-05T07:42:21.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:21 smithi090 ceph-mon[33213]: 4.7 scrub starts 2024-02-05T07:42:21.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:21 smithi090 ceph-mon[33213]: 4.7 scrub ok 2024-02-05T07:42:21.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:21 smithi090 ceph-mon[33213]: 3.1f scrub starts 2024-02-05T07:42:21.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:21 smithi090 ceph-mon[33213]: 3.1f scrub ok 2024-02-05T07:42:21.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[29098]: 5.0 scrub starts 2024-02-05T07:42:21.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[29098]: 5.0 scrub ok 2024-02-05T07:42:21.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4230349014' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T07:42:21.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[29098]: osdmap e731: 8 total, 8 up, 8 in 2024-02-05T07:42:21.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[29098]: 4.7 scrub starts 2024-02-05T07:42:21.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[29098]: 4.7 scrub ok 2024-02-05T07:42:21.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[29098]: 3.1f scrub starts 2024-02-05T07:42:21.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[29098]: 3.1f scrub ok 2024-02-05T07:42:21.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[37158]: 5.0 scrub starts 2024-02-05T07:42:21.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[37158]: 5.0 scrub ok 2024-02-05T07:42:21.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4230349014' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T07:42:21.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[37158]: osdmap e731: 8 total, 8 up, 8 in 2024-02-05T07:42:21.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[37158]: 4.7 scrub starts 2024-02-05T07:42:21.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[37158]: 4.7 scrub ok 2024-02-05T07:42:21.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[37158]: 3.1f scrub starts 2024-02-05T07:42:21.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:21 smithi007 ceph-mon[37158]: 3.1f scrub ok 2024-02-05T07:42:21.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:42:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:42:22.266 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:22 smithi007 ceph-mon[29098]: 4.17 scrub starts 2024-02-05T07:42:22.266 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:22 smithi007 ceph-mon[29098]: 4.17 scrub ok 2024-02-05T07:42:22.266 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:22 smithi007 ceph-mon[29098]: osdmap e732: 8 total, 8 up, 8 in 2024-02-05T07:42:22.267 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:22 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:22.267 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:22 smithi007 ceph-mon[29098]: 4.16 scrub starts 2024-02-05T07:42:22.267 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:22 smithi007 ceph-mon[29098]: 4.16 scrub ok 2024-02-05T07:42:22.267 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:22 smithi007 ceph-mon[29098]: pgmap v1336: 152 pgs: 1 peering, 4 active+remapped+backfill_toofull, 147 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:42:22.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:22 smithi007 ceph-mon[37158]: 4.17 scrub starts 2024-02-05T07:42:22.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:22 smithi007 ceph-mon[37158]: 4.17 scrub ok 2024-02-05T07:42:22.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:22 smithi007 ceph-mon[37158]: osdmap e732: 8 total, 8 up, 8 in 2024-02-05T07:42:22.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:22 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:22.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:22 smithi007 ceph-mon[37158]: 4.16 scrub starts 2024-02-05T07:42:22.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:22 smithi007 ceph-mon[37158]: 4.16 scrub ok 2024-02-05T07:42:22.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:22 smithi007 ceph-mon[37158]: pgmap v1336: 152 pgs: 1 peering, 4 active+remapped+backfill_toofull, 147 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:42:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:22 smithi090 ceph-mon[33213]: 4.17 scrub starts 2024-02-05T07:42:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:22 smithi090 ceph-mon[33213]: 4.17 scrub ok 2024-02-05T07:42:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:22 smithi090 ceph-mon[33213]: osdmap e732: 8 total, 8 up, 8 in 2024-02-05T07:42:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:22 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:22 smithi090 ceph-mon[33213]: 4.16 scrub starts 2024-02-05T07:42:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:22 smithi090 ceph-mon[33213]: 4.16 scrub ok 2024-02-05T07:42:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:22 smithi090 ceph-mon[33213]: pgmap v1336: 152 pgs: 1 peering, 4 active+remapped+backfill_toofull, 147 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:42:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-02-05T07:42:23.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:23 smithi007 ceph-mon[29098]: osdmap e733: 8 total, 8 up, 8 in 2024-02-05T07:42:23.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:23 smithi007 ceph-mon[29098]: 1.10 deep-scrub starts 2024-02-05T07:42:23.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:23 smithi007 ceph-mon[29098]: 1.10 deep-scrub ok 2024-02-05T07:42:23.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:23 smithi007 ceph-mon[29098]: 4.d scrub starts 2024-02-05T07:42:23.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:23 smithi007 ceph-mon[29098]: 4.d scrub ok 2024-02-05T07:42:23.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:23 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3440779974' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:42:23.405 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:23 smithi007 ceph-mon[37158]: osdmap e733: 8 total, 8 up, 8 in 2024-02-05T07:42:23.405 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:23 smithi007 ceph-mon[37158]: 1.10 deep-scrub starts 2024-02-05T07:42:23.405 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:23 smithi007 ceph-mon[37158]: 1.10 deep-scrub ok 2024-02-05T07:42:23.405 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:23 smithi007 ceph-mon[37158]: 4.d scrub starts 2024-02-05T07:42:23.405 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:23 smithi007 ceph-mon[37158]: 4.d scrub ok 2024-02-05T07:42:23.405 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:23 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3440779974' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:42:23.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:23 smithi090 ceph-mon[33213]: osdmap e733: 8 total, 8 up, 8 in 2024-02-05T07:42:23.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:23 smithi090 ceph-mon[33213]: 1.10 deep-scrub starts 2024-02-05T07:42:23.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:23 smithi090 ceph-mon[33213]: 1.10 deep-scrub ok 2024-02-05T07:42:23.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:23 smithi090 ceph-mon[33213]: 4.d scrub starts 2024-02-05T07:42:23.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:23 smithi090 ceph-mon[33213]: 4.d scrub ok 2024-02-05T07:42:23.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:23 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3440779974' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T07:42:23.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:42:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:42:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:42:23.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 4 1 2024-02-05T07:42:23.962 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:24.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:24 smithi007 ceph-mon[29098]: 5.c scrub starts 2024-02-05T07:42:24.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:24 smithi007 ceph-mon[29098]: 5.c scrub ok 2024-02-05T07:42:24.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:24 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3440779974' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-05T07:42:24.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:24 smithi007 ceph-mon[29098]: osdmap e734: 8 total, 8 up, 8 in 2024-02-05T07:42:24.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:24 smithi007 ceph-mon[29098]: pgmap v1339: 152 pgs: 1 peering, 4 active+remapped+backfill_toofull, 147 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:42:24.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:24 smithi007 ceph-mon[37158]: 5.c scrub starts 2024-02-05T07:42:24.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:24 smithi007 ceph-mon[37158]: 5.c scrub ok 2024-02-05T07:42:24.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:24 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3440779974' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-05T07:42:24.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:24 smithi007 ceph-mon[37158]: osdmap e734: 8 total, 8 up, 8 in 2024-02-05T07:42:24.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:24 smithi007 ceph-mon[37158]: pgmap v1339: 152 pgs: 1 peering, 4 active+remapped+backfill_toofull, 147 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:42:24.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:24 smithi090 ceph-mon[33213]: 5.c scrub starts 2024-02-05T07:42:24.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:24 smithi090 ceph-mon[33213]: 5.c scrub ok 2024-02-05T07:42:24.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:24 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3440779974' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-05T07:42:24.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:24 smithi090 ceph-mon[33213]: osdmap e734: 8 total, 8 up, 8 in 2024-02-05T07:42:24.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:24 smithi090 ceph-mon[33213]: pgmap v1339: 152 pgs: 1 peering, 4 active+remapped+backfill_toofull, 147 active+clean; 635 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:42:25.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:25 smithi007 ceph-mon[29098]: osdmap e735: 8 total, 8 up, 8 in 2024-02-05T07:42:25.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:25 smithi007 ceph-mon[29098]: 2.1 scrub starts 2024-02-05T07:42:25.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:25 smithi007 ceph-mon[29098]: 2.1 scrub ok 2024-02-05T07:42:25.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:25 smithi007 ceph-mon[29098]: 6.1c scrub starts 2024-02-05T07:42:25.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:25 smithi007 ceph-mon[29098]: 6.1c scrub ok 2024-02-05T07:42:25.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:25 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:25.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:25 smithi007 ceph-mon[29098]: osdmap e736: 8 total, 8 up, 8 in 2024-02-05T07:42:25.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:25 smithi007 ceph-mon[37158]: osdmap e735: 8 total, 8 up, 8 in 2024-02-05T07:42:25.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:25 smithi007 ceph-mon[37158]: 2.1 scrub starts 2024-02-05T07:42:25.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:25 smithi007 ceph-mon[37158]: 2.1 scrub ok 2024-02-05T07:42:25.146 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:25 smithi007 ceph-mon[37158]: 6.1c scrub starts 2024-02-05T07:42:25.147 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:25 smithi007 ceph-mon[37158]: 6.1c scrub ok 2024-02-05T07:42:25.147 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:25 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:25.147 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:25 smithi007 ceph-mon[37158]: osdmap e736: 8 total, 8 up, 8 in 2024-02-05T07:42:25.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:25 smithi090 ceph-mon[33213]: osdmap e735: 8 total, 8 up, 8 in 2024-02-05T07:42:25.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:25 smithi090 ceph-mon[33213]: 2.1 scrub starts 2024-02-05T07:42:25.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:25 smithi090 ceph-mon[33213]: 2.1 scrub ok 2024-02-05T07:42:25.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:25 smithi090 ceph-mon[33213]: 6.1c scrub starts 2024-02-05T07:42:25.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:25 smithi090 ceph-mon[33213]: 6.1c scrub ok 2024-02-05T07:42:25.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:25 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:25.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:25 smithi090 ceph-mon[33213]: osdmap e736: 8 total, 8 up, 8 in 2024-02-05T07:42:26.038 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.4 to 1 (10000) 2024-02-05T07:42:26.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:26 smithi007 ceph-mon[29098]: 6.f scrub starts 2024-02-05T07:42:26.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:26 smithi007 ceph-mon[29098]: 6.f scrub ok 2024-02-05T07:42:26.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:26 smithi007 ceph-mon[29098]: pgmap v1342: 152 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 149 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:26.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:26 smithi007 ceph-mon[29098]: 6.a scrub starts 2024-02-05T07:42:26.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:26 smithi007 ceph-mon[29098]: 6.a scrub ok 2024-02-05T07:42:26.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:26 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4198626155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:42:26.413 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:26 smithi007 ceph-mon[37158]: 6.f scrub starts 2024-02-05T07:42:26.413 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:26 smithi007 ceph-mon[37158]: 6.f scrub ok 2024-02-05T07:42:26.413 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:26 smithi007 ceph-mon[37158]: pgmap v1342: 152 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 149 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:26.413 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:26 smithi007 ceph-mon[37158]: 6.a scrub starts 2024-02-05T07:42:26.413 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:26 smithi007 ceph-mon[37158]: 6.a scrub ok 2024-02-05T07:42:26.413 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:26 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4198626155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:42:26.523 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:26 smithi090 ceph-mon[33213]: 6.f scrub starts 2024-02-05T07:42:26.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:26 smithi090 ceph-mon[33213]: 6.f scrub ok 2024-02-05T07:42:26.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:26 smithi090 ceph-mon[33213]: pgmap v1342: 152 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 149 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:26.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:26 smithi090 ceph-mon[33213]: 6.a scrub starts 2024-02-05T07:42:26.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:26 smithi090 ceph-mon[33213]: 6.a scrub ok 2024-02-05T07:42:26.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:26 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4198626155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:42:26.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 4 1 2024-02-05T07:42:27.063 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:27.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:27 smithi007 ceph-mon[29098]: 6.1d scrub starts 2024-02-05T07:42:27.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:27 smithi007 ceph-mon[29098]: 6.1d scrub ok 2024-02-05T07:42:27.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:27 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4198626155' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T07:42:27.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:27 smithi007 ceph-mon[29098]: osdmap e737: 8 total, 8 up, 8 in 2024-02-05T07:42:27.306 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:27 smithi007 ceph-mon[29098]: 6.17 scrub starts 2024-02-05T07:42:27.306 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:27 smithi007 ceph-mon[29098]: 6.17 scrub ok 2024-02-05T07:42:27.306 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:27 smithi007 ceph-mon[37158]: 6.1d scrub starts 2024-02-05T07:42:27.306 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:27 smithi007 ceph-mon[37158]: 6.1d scrub ok 2024-02-05T07:42:27.306 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:27 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4198626155' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T07:42:27.306 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:27 smithi007 ceph-mon[37158]: osdmap e737: 8 total, 8 up, 8 in 2024-02-05T07:42:27.306 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:27 smithi007 ceph-mon[37158]: 6.17 scrub starts 2024-02-05T07:42:27.306 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:27 smithi007 ceph-mon[37158]: 6.17 scrub ok 2024-02-05T07:42:27.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:27 smithi090 ceph-mon[33213]: 6.1d scrub starts 2024-02-05T07:42:27.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:27 smithi090 ceph-mon[33213]: 6.1d scrub ok 2024-02-05T07:42:27.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:27 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4198626155' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T07:42:27.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:27 smithi090 ceph-mon[33213]: osdmap e737: 8 total, 8 up, 8 in 2024-02-05T07:42:27.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:27 smithi090 ceph-mon[33213]: 6.17 scrub starts 2024-02-05T07:42:27.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:27 smithi090 ceph-mon[33213]: 6.17 scrub ok 2024-02-05T07:42:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[29098]: 3.4 scrub starts 2024-02-05T07:42:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[29098]: 3.4 scrub ok 2024-02-05T07:42:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[29098]: 3.b scrub starts 2024-02-05T07:42:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[29098]: 3.b scrub ok 2024-02-05T07:42:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[29098]: 5.d scrub starts 2024-02-05T07:42:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[29098]: 5.d scrub ok 2024-02-05T07:42:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[29098]: 4.6 scrub starts 2024-02-05T07:42:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[29098]: 4.6 scrub ok 2024-02-05T07:42:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[29098]: pgmap v1344: 152 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 149 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[37158]: 3.4 scrub starts 2024-02-05T07:42:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[37158]: 3.4 scrub ok 2024-02-05T07:42:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[37158]: 3.b scrub starts 2024-02-05T07:42:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[37158]: 3.b scrub ok 2024-02-05T07:42:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[37158]: 5.d scrub starts 2024-02-05T07:42:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[37158]: 5.d scrub ok 2024-02-05T07:42:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[37158]: 4.6 scrub starts 2024-02-05T07:42:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[37158]: 4.6 scrub ok 2024-02-05T07:42:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:28 smithi007 ceph-mon[37158]: pgmap v1344: 152 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 149 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:28 smithi090 ceph-mon[33213]: 3.4 scrub starts 2024-02-05T07:42:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:28 smithi090 ceph-mon[33213]: 3.4 scrub ok 2024-02-05T07:42:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:28 smithi090 ceph-mon[33213]: 3.b scrub starts 2024-02-05T07:42:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:28 smithi090 ceph-mon[33213]: 3.b scrub ok 2024-02-05T07:42:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:28 smithi090 ceph-mon[33213]: 5.d scrub starts 2024-02-05T07:42:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:28 smithi090 ceph-mon[33213]: 5.d scrub ok 2024-02-05T07:42:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:28 smithi090 ceph-mon[33213]: 4.6 scrub starts 2024-02-05T07:42:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:28 smithi090 ceph-mon[33213]: 4.6 scrub ok 2024-02-05T07:42:28.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:28 smithi090 ceph-mon[33213]: pgmap v1344: 152 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 149 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:29.178 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-02-05T07:42:29.523 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:29 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4242417832' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:42:29.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:29 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:42:29.546 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:29 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4242417832' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:42:29.546 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:29 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:42:29.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:29 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4242417832' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:42:29.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:29 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T07:42:29.935 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 5 1 2024-02-05T07:42:30.183 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:30.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:30 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-05T07:42:30.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:30 smithi007 ceph-mon[29098]: osdmap e738: 8 total, 8 up, 8 in 2024-02-05T07:42:30.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:30 smithi007 ceph-mon[29098]: 3.2 scrub starts 2024-02-05T07:42:30.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:30 smithi007 ceph-mon[29098]: 3.2 scrub ok 2024-02-05T07:42:30.501 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:30 smithi007 ceph-mon[29098]: pgmap v1346: 152 pgs: 2 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:30.501 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:30 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:30.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:30 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-05T07:42:30.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:30 smithi007 ceph-mon[37158]: osdmap e738: 8 total, 8 up, 8 in 2024-02-05T07:42:30.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:30 smithi007 ceph-mon[37158]: 3.2 scrub starts 2024-02-05T07:42:30.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:30 smithi007 ceph-mon[37158]: 3.2 scrub ok 2024-02-05T07:42:30.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:30 smithi007 ceph-mon[37158]: pgmap v1346: 152 pgs: 2 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:30.502 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:30 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:30.523 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:30 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-05T07:42:30.523 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:30 smithi090 ceph-mon[33213]: osdmap e738: 8 total, 8 up, 8 in 2024-02-05T07:42:30.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:30 smithi090 ceph-mon[33213]: 3.2 scrub starts 2024-02-05T07:42:30.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:30 smithi090 ceph-mon[33213]: 3.2 scrub ok 2024-02-05T07:42:30.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:30 smithi090 ceph-mon[33213]: pgmap v1346: 152 pgs: 2 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:30.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:30 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:31.330 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:31 smithi007 ceph-mon[29098]: 1.6 scrub starts 2024-02-05T07:42:31.330 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:31 smithi007 ceph-mon[29098]: 1.6 scrub ok 2024-02-05T07:42:31.330 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:31 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-05T07:42:31.330 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:31 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-05T07:42:31.331 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:31 smithi007 ceph-mon[37158]: 1.6 scrub starts 2024-02-05T07:42:31.331 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:31 smithi007 ceph-mon[37158]: 1.6 scrub ok 2024-02-05T07:42:31.331 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:31 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-05T07:42:31.331 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:31 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-05T07:42:31.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:42:31 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:42:31.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:31 smithi090 ceph-mon[33213]: 1.6 scrub starts 2024-02-05T07:42:31.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:31 smithi090 ceph-mon[33213]: 1.6 scrub ok 2024-02-05T07:42:31.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:31 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-05T07:42:31.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:31 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-05T07:42:32.204 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.5 to 1 (10000) 2024-02-05T07:42:32.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:32 smithi090 ceph-mon[33213]: 5.13 scrub starts 2024-02-05T07:42:32.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:32 smithi090 ceph-mon[33213]: 5.13 scrub ok 2024-02-05T07:42:32.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:32 smithi090 ceph-mon[33213]: 4.12 scrub starts 2024-02-05T07:42:32.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:32 smithi090 ceph-mon[33213]: 4.12 scrub ok 2024-02-05T07:42:32.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:32 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-05T07:42:32.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:32 smithi090 ceph-mon[33213]: osdmap e739: 8 total, 8 up, 8 in 2024-02-05T07:42:32.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:32 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:32.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:32 smithi090 ceph-mon[33213]: 3.6 scrub starts 2024-02-05T07:42:32.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:32 smithi090 ceph-mon[33213]: 3.6 scrub ok 2024-02-05T07:42:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:32 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:42:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:32 smithi090 ceph-mon[33213]: pgmap v1348: 152 pgs: 2 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T07:42:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:32 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3938079299' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T07:42:32.525 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:32 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T07:42:32.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[37158]: 5.13 scrub starts 2024-02-05T07:42:32.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[37158]: 5.13 scrub ok 2024-02-05T07:42:32.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[37158]: 4.12 scrub starts 2024-02-05T07:42:32.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[37158]: 4.12 scrub ok 2024-02-05T07:42:32.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-05T07:42:32.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[37158]: osdmap e739: 8 total, 8 up, 8 in 2024-02-05T07:42:32.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:32.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[37158]: 3.6 scrub starts 2024-02-05T07:42:32.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[37158]: 3.6 scrub ok 2024-02-05T07:42:32.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:42:32.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[37158]: pgmap v1348: 152 pgs: 2 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T07:42:32.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3938079299' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T07:42:32.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T07:42:32.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[29098]: 5.13 scrub starts 2024-02-05T07:42:32.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[29098]: 5.13 scrub ok 2024-02-05T07:42:32.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[29098]: 4.12 scrub starts 2024-02-05T07:42:32.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[29098]: 4.12 scrub ok 2024-02-05T07:42:32.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-05T07:42:32.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[29098]: osdmap e739: 8 total, 8 up, 8 in 2024-02-05T07:42:32.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:32.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[29098]: 3.6 scrub starts 2024-02-05T07:42:32.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[29098]: 3.6 scrub ok 2024-02-05T07:42:32.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:42:32.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[29098]: pgmap v1348: 152 pgs: 2 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T07:42:32.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3938079299' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T07:42:32.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:32 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T07:42:33.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 5 1 2024-02-05T07:42:33.521 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:33.523 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:33 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T07:42:33.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:33 smithi090 ceph-mon[33213]: osdmap e740: 8 total, 8 up, 8 in 2024-02-05T07:42:33.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:33 smithi090 ceph-mon[33213]: 5.1e scrub starts 2024-02-05T07:42:33.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:33 smithi090 ceph-mon[33213]: 5.1e scrub ok 2024-02-05T07:42:33.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:33 smithi090 ceph-mon[33213]: 3.a deep-scrub starts 2024-02-05T07:42:33.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:33 smithi090 ceph-mon[33213]: 3.a deep-scrub ok 2024-02-05T07:42:33.534 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:33 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T07:42:33.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:33 smithi007 ceph-mon[29098]: osdmap e740: 8 total, 8 up, 8 in 2024-02-05T07:42:33.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:33 smithi007 ceph-mon[29098]: 5.1e scrub starts 2024-02-05T07:42:33.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:33 smithi007 ceph-mon[29098]: 5.1e scrub ok 2024-02-05T07:42:33.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:33 smithi007 ceph-mon[29098]: 3.a deep-scrub starts 2024-02-05T07:42:33.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:33 smithi007 ceph-mon[29098]: 3.a deep-scrub ok 2024-02-05T07:42:33.535 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:33 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T07:42:33.535 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:33 smithi007 ceph-mon[37158]: osdmap e740: 8 total, 8 up, 8 in 2024-02-05T07:42:33.535 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:33 smithi007 ceph-mon[37158]: 5.1e scrub starts 2024-02-05T07:42:33.535 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:33 smithi007 ceph-mon[37158]: 5.1e scrub ok 2024-02-05T07:42:33.535 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:33 smithi007 ceph-mon[37158]: 3.a deep-scrub starts 2024-02-05T07:42:33.535 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:33 smithi007 ceph-mon[37158]: 3.a deep-scrub ok 2024-02-05T07:42:33.884 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:42:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:42:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:42:34.524 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:34 smithi090 ceph-mon[33213]: pgmap v1350: 152 pgs: 2 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:42:34.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:34 smithi007 ceph-mon[29098]: pgmap v1350: 152 pgs: 2 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:42:34.664 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:34 smithi007 ceph-mon[37158]: pgmap v1350: 152 pgs: 2 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:42:35.738 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-02-05T07:42:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:35 smithi090 ceph-mon[33213]: 6.2 scrub starts 2024-02-05T07:42:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:35 smithi090 ceph-mon[33213]: 6.2 scrub ok 2024-02-05T07:42:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:35 smithi090 ceph-mon[33213]: 5.14 scrub starts 2024-02-05T07:42:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:35 smithi090 ceph-mon[33213]: 5.14 scrub ok 2024-02-05T07:42:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:35 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2755104739' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T07:42:36.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:35 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T07:42:36.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:35 smithi007 ceph-mon[29098]: 6.2 scrub starts 2024-02-05T07:42:36.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:35 smithi007 ceph-mon[29098]: 6.2 scrub ok 2024-02-05T07:42:36.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:35 smithi007 ceph-mon[29098]: 5.14 scrub starts 2024-02-05T07:42:36.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:35 smithi007 ceph-mon[29098]: 5.14 scrub ok 2024-02-05T07:42:36.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:35 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2755104739' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T07:42:36.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:35 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T07:42:36.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:35 smithi007 ceph-mon[37158]: 6.2 scrub starts 2024-02-05T07:42:36.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:35 smithi007 ceph-mon[37158]: 6.2 scrub ok 2024-02-05T07:42:36.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:35 smithi007 ceph-mon[37158]: 5.14 scrub starts 2024-02-05T07:42:36.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:35 smithi007 ceph-mon[37158]: 5.14 scrub ok 2024-02-05T07:42:36.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:35 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2755104739' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T07:42:36.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:35 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T07:42:36.481 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 7 1 2024-02-05T07:42:36.744 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:37.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:36 smithi007 ceph-mon[29098]: pgmap v1351: 152 pgs: 2 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 812 B/s rd, 0 op/s 2024-02-05T07:42:37.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:36 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-05T07:42:37.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:36 smithi007 ceph-mon[29098]: osdmap e741: 8 total, 8 up, 8 in 2024-02-05T07:42:37.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:36 smithi007 ceph-mon[29098]: osdmap e742: 8 total, 8 up, 8 in 2024-02-05T07:42:37.016 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:36 smithi007 ceph-mon[37158]: pgmap v1351: 152 pgs: 2 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 812 B/s rd, 0 op/s 2024-02-05T07:42:37.016 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:36 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-05T07:42:37.016 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:36 smithi007 ceph-mon[37158]: osdmap e741: 8 total, 8 up, 8 in 2024-02-05T07:42:37.016 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:36 smithi007 ceph-mon[37158]: osdmap e742: 8 total, 8 up, 8 in 2024-02-05T07:42:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:36 smithi090 ceph-mon[33213]: pgmap v1351: 152 pgs: 2 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 812 B/s rd, 0 op/s 2024-02-05T07:42:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:36 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-05T07:42:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:36 smithi090 ceph-mon[33213]: osdmap e741: 8 total, 8 up, 8 in 2024-02-05T07:42:37.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:36 smithi090 ceph-mon[33213]: osdmap e742: 8 total, 8 up, 8 in 2024-02-05T07:42:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[29098]: 4.e scrub starts 2024-02-05T07:42:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[29098]: 4.e scrub ok 2024-02-05T07:42:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[29098]: 6.1b scrub starts 2024-02-05T07:42:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[29098]: 5.1f scrub starts 2024-02-05T07:42:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[29098]: 5.1f scrub ok 2024-02-05T07:42:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[29098]: 1.a continuing backfill to osd.5 from (86'128,86'128] MIN to 86'128 2024-02-05T07:42:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[29098]: 6.1b scrub ok 2024-02-05T07:42:37.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[29098]: 3.17 scrub starts 2024-02-05T07:42:37.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[29098]: 3.17 scrub ok 2024-02-05T07:42:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[37158]: 4.e scrub starts 2024-02-05T07:42:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[37158]: 4.e scrub ok 2024-02-05T07:42:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[37158]: 6.1b scrub starts 2024-02-05T07:42:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[37158]: 5.1f scrub starts 2024-02-05T07:42:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[37158]: 5.1f scrub ok 2024-02-05T07:42:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[37158]: 1.a continuing backfill to osd.5 from (86'128,86'128] MIN to 86'128 2024-02-05T07:42:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[37158]: 6.1b scrub ok 2024-02-05T07:42:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[37158]: 3.17 scrub starts 2024-02-05T07:42:37.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:37 smithi007 ceph-mon[37158]: 3.17 scrub ok 2024-02-05T07:42:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:37 smithi090 ceph-mon[33213]: 4.e scrub starts 2024-02-05T07:42:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:37 smithi090 ceph-mon[33213]: 4.e scrub ok 2024-02-05T07:42:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:37 smithi090 ceph-mon[33213]: 6.1b scrub starts 2024-02-05T07:42:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:37 smithi090 ceph-mon[33213]: 5.1f scrub starts 2024-02-05T07:42:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:37 smithi090 ceph-mon[33213]: 5.1f scrub ok 2024-02-05T07:42:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:37 smithi090 ceph-mon[33213]: 1.a continuing backfill to osd.5 from (86'128,86'128] MIN to 86'128 2024-02-05T07:42:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:37 smithi090 ceph-mon[33213]: 6.1b scrub ok 2024-02-05T07:42:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:37 smithi090 ceph-mon[33213]: 3.17 scrub starts 2024-02-05T07:42:38.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:37 smithi090 ceph-mon[33213]: 3.17 scrub ok 2024-02-05T07:42:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:38 smithi007 ceph-mon[29098]: pgmap v1354: 152 pgs: 1 remapped+peering, 1 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:42:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:38 smithi007 ceph-mon[29098]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:38 smithi007 ceph-mon[29098]: osdmap e743: 8 total, 8 up, 8 in 2024-02-05T07:42:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:38 smithi007 ceph-mon[29098]: 5.f deep-scrub starts 2024-02-05T07:42:38.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:38 smithi007 ceph-mon[29098]: 5.f deep-scrub ok 2024-02-05T07:42:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:38 smithi007 ceph-mon[37158]: pgmap v1354: 152 pgs: 1 remapped+peering, 1 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:42:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:38 smithi007 ceph-mon[37158]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:38 smithi007 ceph-mon[37158]: osdmap e743: 8 total, 8 up, 8 in 2024-02-05T07:42:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:38 smithi007 ceph-mon[37158]: 5.f deep-scrub starts 2024-02-05T07:42:38.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:38 smithi007 ceph-mon[37158]: 5.f deep-scrub ok 2024-02-05T07:42:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:38 smithi090 ceph-mon[33213]: pgmap v1354: 152 pgs: 1 remapped+peering, 1 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T07:42:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:38 smithi090 ceph-mon[33213]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T07:42:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:38 smithi090 ceph-mon[33213]: osdmap e743: 8 total, 8 up, 8 in 2024-02-05T07:42:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:38 smithi090 ceph-mon[33213]: 5.f deep-scrub starts 2024-02-05T07:42:39.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:38 smithi090 ceph-mon[33213]: 5.f deep-scrub ok 2024-02-05T07:42:39.763 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.7 to 1 (10000) 2024-02-05T07:42:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:39 smithi090 ceph-mon[33213]: 5.17 scrub starts 2024-02-05T07:42:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:39 smithi090 ceph-mon[33213]: 5.17 scrub ok 2024-02-05T07:42:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:39 smithi090 ceph-mon[33213]: osdmap e744: 8 total, 8 up, 8 in 2024-02-05T07:42:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:39 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2681179047' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:42:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:39 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:42:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:39 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:42:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:39 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:42:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:39 smithi090 ceph-mon[33213]: 1.a scrub starts 2024-02-05T07:42:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:39 smithi090 ceph-mon[33213]: 1.a scrub ok 2024-02-05T07:42:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:39 smithi090 ceph-mon[33213]: 5.5 scrub starts 2024-02-05T07:42:40.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:39 smithi090 ceph-mon[33213]: 5.5 scrub ok 2024-02-05T07:42:40.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[29098]: 5.17 scrub starts 2024-02-05T07:42:40.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[29098]: 5.17 scrub ok 2024-02-05T07:42:40.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[29098]: osdmap e744: 8 total, 8 up, 8 in 2024-02-05T07:42:40.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2681179047' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:42:40.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:42:40.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:42:40.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:42:40.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[29098]: 1.a scrub starts 2024-02-05T07:42:40.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[29098]: 1.a scrub ok 2024-02-05T07:42:40.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[29098]: 5.5 scrub starts 2024-02-05T07:42:40.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[29098]: 5.5 scrub ok 2024-02-05T07:42:40.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[37158]: 5.17 scrub starts 2024-02-05T07:42:40.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[37158]: 5.17 scrub ok 2024-02-05T07:42:40.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[37158]: osdmap e744: 8 total, 8 up, 8 in 2024-02-05T07:42:40.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2681179047' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:42:40.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:42:40.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:42:40.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:42:40.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[37158]: 1.a scrub starts 2024-02-05T07:42:40.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[37158]: 1.a scrub ok 2024-02-05T07:42:40.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[37158]: 5.5 scrub starts 2024-02-05T07:42:40.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:39 smithi007 ceph-mon[37158]: 5.5 scrub ok 2024-02-05T07:42:40.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 7 1 2024-02-05T07:42:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:40 smithi090 ceph-mon[33213]: pgmap v1357: 152 pgs: 1 remapped+peering, 1 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-02-05T07:42:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:40 smithi090 ceph-mon[33213]: 1.7 scrub starts 2024-02-05T07:42:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:40 smithi090 ceph-mon[33213]: 1.7 scrub ok 2024-02-05T07:42:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:40 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T07:42:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:40 smithi090 ceph-mon[33213]: osdmap e745: 8 total, 8 up, 8 in 2024-02-05T07:42:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:40 smithi090 ceph-mon[33213]: 6.6 scrub starts 2024-02-05T07:42:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:40 smithi090 ceph-mon[33213]: 6.6 scrub ok 2024-02-05T07:42:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:40 smithi090 ceph-mon[33213]: 3.1d scrub starts 2024-02-05T07:42:41.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:40 smithi090 ceph-mon[33213]: 3.1d scrub ok 2024-02-05T07:42:41.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[29098]: pgmap v1357: 152 pgs: 1 remapped+peering, 1 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-02-05T07:42:41.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[29098]: 1.7 scrub starts 2024-02-05T07:42:41.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[29098]: 1.7 scrub ok 2024-02-05T07:42:41.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T07:42:41.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[29098]: osdmap e745: 8 total, 8 up, 8 in 2024-02-05T07:42:41.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[29098]: 6.6 scrub starts 2024-02-05T07:42:41.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[29098]: 6.6 scrub ok 2024-02-05T07:42:41.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[29098]: 3.1d scrub starts 2024-02-05T07:42:41.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[29098]: 3.1d scrub ok 2024-02-05T07:42:41.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[37158]: pgmap v1357: 152 pgs: 1 remapped+peering, 1 active+remapped+backfill_toofull, 150 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-02-05T07:42:41.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[37158]: 1.7 scrub starts 2024-02-05T07:42:41.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[37158]: 1.7 scrub ok 2024-02-05T07:42:41.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T07:42:41.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[37158]: osdmap e745: 8 total, 8 up, 8 in 2024-02-05T07:42:41.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[37158]: 6.6 scrub starts 2024-02-05T07:42:41.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[37158]: 6.6 scrub ok 2024-02-05T07:42:41.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[37158]: 3.1d scrub starts 2024-02-05T07:42:41.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:40 smithi007 ceph-mon[37158]: 3.1d scrub ok 2024-02-05T07:42:41.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:42:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:42:41.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:41 smithi007 ceph-mon[29098]: 3.e scrub starts 2024-02-05T07:42:41.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:41 smithi007 ceph-mon[29098]: 3.e scrub ok 2024-02-05T07:42:41.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:41 smithi007 ceph-mon[29098]: 3.1c scrub starts 2024-02-05T07:42:41.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:41 smithi007 ceph-mon[29098]: 3.1c scrub ok 2024-02-05T07:42:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:41 smithi007 ceph-mon[37158]: 3.e scrub starts 2024-02-05T07:42:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:41 smithi007 ceph-mon[37158]: 3.e scrub ok 2024-02-05T07:42:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:41 smithi007 ceph-mon[37158]: 3.1c scrub starts 2024-02-05T07:42:41.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:41 smithi007 ceph-mon[37158]: 3.1c scrub ok 2024-02-05T07:42:42.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:41 smithi090 ceph-mon[33213]: 3.e scrub starts 2024-02-05T07:42:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:41 smithi090 ceph-mon[33213]: 3.e scrub ok 2024-02-05T07:42:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:41 smithi090 ceph-mon[33213]: 3.1c scrub starts 2024-02-05T07:42:42.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:41 smithi090 ceph-mon[33213]: 3.1c scrub ok 2024-02-05T07:42:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:42 smithi090 ceph-mon[33213]: 3.14 scrub starts 2024-02-05T07:42:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:42 smithi090 ceph-mon[33213]: 3.14 scrub ok 2024-02-05T07:42:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:42 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:43.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:42 smithi090 ceph-mon[33213]: pgmap v1359: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:42:43.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:42 smithi007 ceph-mon[29098]: 3.14 scrub starts 2024-02-05T07:42:43.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:42 smithi007 ceph-mon[29098]: 3.14 scrub ok 2024-02-05T07:42:43.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:42 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:43.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:42 smithi007 ceph-mon[29098]: pgmap v1359: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:42:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:42 smithi007 ceph-mon[37158]: 3.14 scrub starts 2024-02-05T07:42:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:42 smithi007 ceph-mon[37158]: 3.14 scrub ok 2024-02-05T07:42:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:42 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:43.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:42 smithi007 ceph-mon[37158]: pgmap v1359: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T07:42:43.780 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-02-05T07:42:43.793 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:42:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:42:43.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[29098]: 6.e scrub starts 2024-02-05T07:42:43.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[29098]: 3.1b scrub starts 2024-02-05T07:42:43.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[29098]: 3.1b scrub ok 2024-02-05T07:42:43.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[29098]: 6.e scrub ok 2024-02-05T07:42:43.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[29098]: 6.12 scrub starts 2024-02-05T07:42:43.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[29098]: 6.12 scrub ok 2024-02-05T07:42:43.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/622404220' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:42:43.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:42:43.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[37158]: 6.e scrub starts 2024-02-05T07:42:43.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[37158]: 3.1b scrub starts 2024-02-05T07:42:43.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[37158]: 3.1b scrub ok 2024-02-05T07:42:43.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[37158]: 6.e scrub ok 2024-02-05T07:42:43.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[37158]: 6.12 scrub starts 2024-02-05T07:42:43.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[37158]: 6.12 scrub ok 2024-02-05T07:42:43.795 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/622404220' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:42:43.795 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:43 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:42:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:43 smithi090 ceph-mon[33213]: 6.e scrub starts 2024-02-05T07:42:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:43 smithi090 ceph-mon[33213]: 3.1b scrub starts 2024-02-05T07:42:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:43 smithi090 ceph-mon[33213]: 3.1b scrub ok 2024-02-05T07:42:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:43 smithi090 ceph-mon[33213]: 6.e scrub ok 2024-02-05T07:42:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:43 smithi090 ceph-mon[33213]: 6.12 scrub starts 2024-02-05T07:42:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:43 smithi090 ceph-mon[33213]: 6.12 scrub ok 2024-02-05T07:42:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:43 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/622404220' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:42:44.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:43 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T07:42:44.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd reweight 6 1 2024-02-05T07:42:44.766 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:45.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:44 smithi007 ceph-mon[29098]: pgmap v1360: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:42:45.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:44 smithi007 ceph-mon[29098]: 5.9 scrub starts 2024-02-05T07:42:45.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:44 smithi007 ceph-mon[29098]: 5.9 scrub ok 2024-02-05T07:42:45.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:44 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-05T07:42:45.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:44 smithi007 ceph-mon[29098]: osdmap e746: 8 total, 8 up, 8 in 2024-02-05T07:42:45.034 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:44 smithi007 ceph-mon[37158]: pgmap v1360: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:42:45.034 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:44 smithi007 ceph-mon[37158]: 5.9 scrub starts 2024-02-05T07:42:45.034 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:44 smithi007 ceph-mon[37158]: 5.9 scrub ok 2024-02-05T07:42:45.034 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:44 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-05T07:42:45.034 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:44 smithi007 ceph-mon[37158]: osdmap e746: 8 total, 8 up, 8 in 2024-02-05T07:42:45.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:44 smithi090 ceph-mon[33213]: pgmap v1360: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:42:45.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:44 smithi090 ceph-mon[33213]: 5.9 scrub starts 2024-02-05T07:42:45.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:44 smithi090 ceph-mon[33213]: 5.9 scrub ok 2024-02-05T07:42:45.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:44 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-05T07:42:45.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:44 smithi090 ceph-mon[33213]: osdmap e746: 8 total, 8 up, 8 in 2024-02-05T07:42:46.035 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:45 smithi007 ceph-mon[29098]: 3.7 deep-scrub starts 2024-02-05T07:42:46.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:45 smithi007 ceph-mon[29098]: 3.7 deep-scrub ok 2024-02-05T07:42:46.036 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:45 smithi007 ceph-mon[37158]: 3.7 deep-scrub starts 2024-02-05T07:42:46.036 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:45 smithi007 ceph-mon[37158]: 3.7 deep-scrub ok 2024-02-05T07:42:46.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:45 smithi090 ceph-mon[33213]: 3.7 deep-scrub starts 2024-02-05T07:42:46.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:45 smithi090 ceph-mon[33213]: 3.7 deep-scrub ok 2024-02-05T07:42:46.793 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.6 to 1 (10000) 2024-02-05T07:42:47.117 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:46 smithi007 ceph-mon[29098]: pgmap v1362: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-02-05T07:42:47.117 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:46 smithi007 ceph-mon[29098]: 4.3 scrub starts 2024-02-05T07:42:47.117 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:46 smithi007 ceph-mon[29098]: 4.3 scrub ok 2024-02-05T07:42:47.117 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:46 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:42:47.117 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:46 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2035767166' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:42:47.117 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:46 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:42:47.117 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:46 smithi007 ceph-mon[37158]: pgmap v1362: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-02-05T07:42:47.117 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:46 smithi007 ceph-mon[37158]: 4.3 scrub starts 2024-02-05T07:42:47.118 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:46 smithi007 ceph-mon[37158]: 4.3 scrub ok 2024-02-05T07:42:47.118 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:46 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:42:47.118 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:46 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2035767166' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:42:47.118 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:46 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:42:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:46 smithi090 ceph-mon[33213]: pgmap v1362: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-02-05T07:42:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:46 smithi090 ceph-mon[33213]: 4.3 scrub starts 2024-02-05T07:42:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:46 smithi090 ceph-mon[33213]: 4.3 scrub ok 2024-02-05T07:42:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:46 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:42:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:46 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2035767166' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:42:47.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:46 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:42:47.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 6 1 2024-02-05T07:42:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:48.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[29098]: 1.14 scrub starts 2024-02-05T07:42:48.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[29098]: 1.14 scrub ok 2024-02-05T07:42:48.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[29098]: 2.0 scrub starts 2024-02-05T07:42:48.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[29098]: 2.0 scrub ok 2024-02-05T07:42:48.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[29098]: 6.b deep-scrub starts 2024-02-05T07:42:48.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T07:42:48.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[29098]: osdmap e747: 8 total, 8 up, 8 in 2024-02-05T07:42:48.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[29098]: 6.b deep-scrub ok 2024-02-05T07:42:48.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[29098]: 6.9 scrub starts 2024-02-05T07:42:48.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[29098]: 1.f scrub starts 2024-02-05T07:42:48.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[29098]: 1.f scrub ok 2024-02-05T07:42:48.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[37158]: 1.14 scrub starts 2024-02-05T07:42:48.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[37158]: 1.14 scrub ok 2024-02-05T07:42:48.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[37158]: 2.0 scrub starts 2024-02-05T07:42:48.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[37158]: 2.0 scrub ok 2024-02-05T07:42:48.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[37158]: 6.b deep-scrub starts 2024-02-05T07:42:48.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T07:42:48.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[37158]: osdmap e747: 8 total, 8 up, 8 in 2024-02-05T07:42:48.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[37158]: 6.b deep-scrub ok 2024-02-05T07:42:48.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[37158]: 6.9 scrub starts 2024-02-05T07:42:48.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[37158]: 1.f scrub starts 2024-02-05T07:42:48.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:47 smithi007 ceph-mon[37158]: 1.f scrub ok 2024-02-05T07:42:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:47 smithi090 ceph-mon[33213]: 1.14 scrub starts 2024-02-05T07:42:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:47 smithi090 ceph-mon[33213]: 1.14 scrub ok 2024-02-05T07:42:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:47 smithi090 ceph-mon[33213]: 2.0 scrub starts 2024-02-05T07:42:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:47 smithi090 ceph-mon[33213]: 2.0 scrub ok 2024-02-05T07:42:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:47 smithi090 ceph-mon[33213]: 6.b deep-scrub starts 2024-02-05T07:42:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:47 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T07:42:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:47 smithi090 ceph-mon[33213]: osdmap e747: 8 total, 8 up, 8 in 2024-02-05T07:42:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:47 smithi090 ceph-mon[33213]: 6.b deep-scrub ok 2024-02-05T07:42:48.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:47 smithi090 ceph-mon[33213]: 6.9 scrub starts 2024-02-05T07:42:48.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:47 smithi090 ceph-mon[33213]: 1.f scrub starts 2024-02-05T07:42:48.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:47 smithi090 ceph-mon[33213]: 1.f scrub ok 2024-02-05T07:42:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[29098]: pgmap v1364: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:42:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[29098]: 6.9 scrub ok 2024-02-05T07:42:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[29098]: 4.0 scrub starts 2024-02-05T07:42:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[29098]: 4.0 scrub ok 2024-02-05T07:42:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[29098]: osdmap e748: 8 total, 8 up, 8 in 2024-02-05T07:42:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:42:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:42:48.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:42:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:42:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[29098]: 3.10 scrub starts 2024-02-05T07:42:48.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[29098]: 3.10 scrub ok 2024-02-05T07:42:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[37158]: pgmap v1364: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:42:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[37158]: 6.9 scrub ok 2024-02-05T07:42:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[37158]: 4.0 scrub starts 2024-02-05T07:42:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[37158]: 4.0 scrub ok 2024-02-05T07:42:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[37158]: osdmap e748: 8 total, 8 up, 8 in 2024-02-05T07:42:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:42:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:42:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:42:48.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:42:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[37158]: 3.10 scrub starts 2024-02-05T07:42:48.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:48 smithi007 ceph-mon[37158]: 3.10 scrub ok 2024-02-05T07:42:49.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:48 smithi090 ceph-mon[33213]: pgmap v1364: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:42:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:48 smithi090 ceph-mon[33213]: 6.9 scrub ok 2024-02-05T07:42:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:48 smithi090 ceph-mon[33213]: 4.0 scrub starts 2024-02-05T07:42:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:48 smithi090 ceph-mon[33213]: 4.0 scrub ok 2024-02-05T07:42:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:48 smithi090 ceph-mon[33213]: osdmap e748: 8 total, 8 up, 8 in 2024-02-05T07:42:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:48 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:42:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:48 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:42:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:48 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:42:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:48 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:42:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:48 smithi090 ceph-mon[33213]: 3.10 scrub starts 2024-02-05T07:42:49.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:48 smithi090 ceph-mon[33213]: 3.10 scrub ok 2024-02-05T07:42:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-02-05T07:42:50.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[29098]: 4.1d scrub starts 2024-02-05T07:42:50.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[29098]: 4.1d scrub ok 2024-02-05T07:42:50.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[29098]: osdmap e749: 8 total, 8 up, 8 in 2024-02-05T07:42:50.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[29098]: 4.18 scrub starts 2024-02-05T07:42:50.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[29098]: 4.18 scrub ok 2024-02-05T07:42:50.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3745919613' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:42:50.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3745919613' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-05T07:42:50.102 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[29098]: osdmap e750: 8 total, 8 up, 8 in 2024-02-05T07:42:50.102 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[37158]: 4.1d scrub starts 2024-02-05T07:42:50.102 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[37158]: 4.1d scrub ok 2024-02-05T07:42:50.102 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[37158]: osdmap e749: 8 total, 8 up, 8 in 2024-02-05T07:42:50.103 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[37158]: 4.18 scrub starts 2024-02-05T07:42:50.103 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[37158]: 4.18 scrub ok 2024-02-05T07:42:50.103 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3745919613' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:42:50.103 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3745919613' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-05T07:42:50.103 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:49 smithi007 ceph-mon[37158]: osdmap e750: 8 total, 8 up, 8 in 2024-02-05T07:42:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:49 smithi090 ceph-mon[33213]: 4.1d scrub starts 2024-02-05T07:42:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:49 smithi090 ceph-mon[33213]: 4.1d scrub ok 2024-02-05T07:42:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:49 smithi090 ceph-mon[33213]: osdmap e749: 8 total, 8 up, 8 in 2024-02-05T07:42:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:49 smithi090 ceph-mon[33213]: 4.18 scrub starts 2024-02-05T07:42:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:49 smithi090 ceph-mon[33213]: 4.18 scrub ok 2024-02-05T07:42:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:49 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3745919613' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T07:42:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:49 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3745919613' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-05T07:42:50.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:49 smithi090 ceph-mon[33213]: osdmap e750: 8 total, 8 up, 8 in 2024-02-05T07:42:50.517 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-02-05T07:42:50.517 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-02-05T07:42:50.518 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-02-05T07:42:50.518 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-02-05T07:42:50.518 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-05T07:42:50.518 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T07:42:50.782 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:51.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:50 smithi007 ceph-mon[29098]: pgmap v1367: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 897 B/s rd, 0 op/s 2024-02-05T07:42:51.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:50 smithi007 ceph-mon[29098]: 1.0 scrub starts 2024-02-05T07:42:51.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:50 smithi007 ceph-mon[29098]: 1.0 scrub ok 2024-02-05T07:42:51.149 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:50 smithi007 ceph-mon[29098]: 4.b deep-scrub starts 2024-02-05T07:42:51.149 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:50 smithi007 ceph-mon[29098]: 4.b deep-scrub ok 2024-02-05T07:42:51.149 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:50 smithi007 ceph-mon[29098]: 5.2 scrub starts 2024-02-05T07:42:51.149 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:50 smithi007 ceph-mon[29098]: 5.2 scrub ok 2024-02-05T07:42:51.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:50 smithi007 ceph-mon[37158]: pgmap v1367: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 897 B/s rd, 0 op/s 2024-02-05T07:42:51.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:50 smithi007 ceph-mon[37158]: 1.0 scrub starts 2024-02-05T07:42:51.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:50 smithi007 ceph-mon[37158]: 1.0 scrub ok 2024-02-05T07:42:51.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:50 smithi007 ceph-mon[37158]: 4.b deep-scrub starts 2024-02-05T07:42:51.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:50 smithi007 ceph-mon[37158]: 4.b deep-scrub ok 2024-02-05T07:42:51.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:50 smithi007 ceph-mon[37158]: 5.2 scrub starts 2024-02-05T07:42:51.150 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:50 smithi007 ceph-mon[37158]: 5.2 scrub ok 2024-02-05T07:42:51.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:50 smithi090 ceph-mon[33213]: pgmap v1367: 152 pgs: 1 active+remapped+backfill_toofull, 151 active+clean; 609 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 897 B/s rd, 0 op/s 2024-02-05T07:42:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:50 smithi090 ceph-mon[33213]: 1.0 scrub starts 2024-02-05T07:42:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:50 smithi090 ceph-mon[33213]: 1.0 scrub ok 2024-02-05T07:42:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:50 smithi090 ceph-mon[33213]: 4.b deep-scrub starts 2024-02-05T07:42:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:50 smithi090 ceph-mon[33213]: 4.b deep-scrub ok 2024-02-05T07:42:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:50 smithi090 ceph-mon[33213]: 5.2 scrub starts 2024-02-05T07:42:51.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:50 smithi090 ceph-mon[33213]: 5.2 scrub ok 2024-02-05T07:42:51.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:42:51 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:42:52.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:51 smithi007 ceph-mon[29098]: 6.13 deep-scrub starts 2024-02-05T07:42:52.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:51 smithi007 ceph-mon[29098]: 6.13 deep-scrub ok 2024-02-05T07:42:52.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:51 smithi007 ceph-mon[29098]: 6.1e scrub starts 2024-02-05T07:42:52.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:51 smithi007 ceph-mon[29098]: 6.1e scrub ok 2024-02-05T07:42:52.101 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:51 smithi007 ceph-mon[37158]: 6.13 deep-scrub starts 2024-02-05T07:42:52.101 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:51 smithi007 ceph-mon[37158]: 6.13 deep-scrub ok 2024-02-05T07:42:52.101 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:51 smithi007 ceph-mon[37158]: 6.1e scrub starts 2024-02-05T07:42:52.101 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:51 smithi007 ceph-mon[37158]: 6.1e scrub ok 2024-02-05T07:42:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:51 smithi090 ceph-mon[33213]: 6.13 deep-scrub starts 2024-02-05T07:42:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:51 smithi090 ceph-mon[33213]: 6.13 deep-scrub ok 2024-02-05T07:42:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:51 smithi090 ceph-mon[33213]: 6.1e scrub starts 2024-02-05T07:42:52.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:51 smithi090 ceph-mon[33213]: 6.1e scrub ok 2024-02-05T07:42:52.546 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:42:52.546 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":750,"fsid":"e3e7059e-c3f5-11ee-95b6-87774f69a715","created":"2024-02-05T07:13:14.603885+0000","modified":"2024-02-05T07:42:49.726865+0000","last_up_change":"2024-02-05T07:28:24.555976+0000","last_in_change":"2024-02-05T07:41:52.714677+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T07:16:32.505567+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"1.15","ready_epoch":681,"last_epoch_started":680,"last_epoch_clean":680,"source_version":"86'128","target_version":"86'128"},"last_change":"714","last_force_op_resend":"0","last_force_op_resend_prenautilus":"681","last_force_op_resend_preluminous":"679","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-05T07:18:47.259462+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-05T07:18:47.762336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-05T07:18:49.736599+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-05T07:18:51.696948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-05T07:18:53.768555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"703c3948-8003-4015-9dd8-21019cf0eeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":747,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6801","nonce":2069016689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2069016689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2069016689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":2069016689}]},"public_addr":"172.21.15.7:6801/2069016689","cluster_addr":"172.21.15.7:6802/2069016689","heartbeat_back_addr":"172.21.15.7:6804/2069016689","heartbeat_front_addr":"172.21.15.7:6803/2069016689","state":["exists","up"]},{"osd":1,"uuid":"5c3e7d1e-a61d-44b9-b21e-f91c99a096bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":726,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":865486532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":865486532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":865486532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6807","nonce":865486532}]},"public_addr":"172.21.15.7:6805/865486532","cluster_addr":"172.21.15.7:6806/865486532","heartbeat_back_addr":"172.21.15.7:6808/865486532","heartbeat_front_addr":"172.21.15.7:6807/865486532","state":["exists","up"]},{"osd":2,"uuid":"f4fd930f-7bb2-46ca-9964-98235bd47b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":742,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":904871370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":904871370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":904871370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":904871370}]},"public_addr":"172.21.15.7:6809/904871370","cluster_addr":"172.21.15.7:6810/904871370","heartbeat_back_addr":"172.21.15.7:6812/904871370","heartbeat_front_addr":"172.21.15.7:6811/904871370","state":["exists","up"]},{"osd":3,"uuid":"149ed0d6-b049-4478-9819-cafc446e94d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":723,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":2758586242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2758586242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2758586242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2758586242}]},"public_addr":"172.21.15.7:6813/2758586242","cluster_addr":"172.21.15.7:6814/2758586242","heartbeat_back_addr":"172.21.15.7:6816/2758586242","heartbeat_front_addr":"172.21.15.7:6815/2758586242","state":["exists","up"]},{"osd":4,"uuid":"3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":714,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":1154929470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6801","nonce":1154929470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6803","nonce":1154929470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1154929470}]},"public_addr":"172.21.15.90:6800/1154929470","cluster_addr":"172.21.15.90:6801/1154929470","heartbeat_back_addr":"172.21.15.90:6803/1154929470","heartbeat_front_addr":"172.21.15.90:6802/1154929470","state":["exists","up"]},{"osd":5,"uuid":"244a7930-c22c-413e-86e1-11e0749f7367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":743,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":398113577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6805","nonce":398113577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6807","nonce":398113577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":398113577}]},"public_addr":"172.21.15.90:6804/398113577","cluster_addr":"172.21.15.90:6805/398113577","heartbeat_back_addr":"172.21.15.90:6807/398113577","heartbeat_front_addr":"172.21.15.90:6806/398113577","state":["exists","up"]},{"osd":6,"uuid":"0adbb562-3012-4f60-b5ec-02550a10b2c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":52,"last_clean_end":230,"up_from":301,"up_thru":735,"down_at":231,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":313196533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":313196533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6811","nonce":313196533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":313196533}]},"public_addr":"172.21.15.90:6808/313196533","cluster_addr":"172.21.15.90:6809/313196533","heartbeat_back_addr":"172.21.15.90:6811/313196533","heartbeat_front_addr":"172.21.15.90:6810/313196533","state":["exists","up"]},{"osd":7,"uuid":"dbc26c36-a0e9-4f21-9e98-739270cea6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":748,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2983605980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":2983605980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":2983605980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2983605980}]},"public_addr":"172.21.15.90:6812/2983605980","cluster_addr":"172.21.15.90:6813/2983605980","heartbeat_back_addr":"172.21.15.90:6815/2983605980","heartbeat_front_addr":"172.21.15.90:6814/2983605980","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:41:54.408221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:42:10.829525+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:42:19.161399+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:42:03.480845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:42:16.436670+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:42:38.979003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:41:35.879130+0000","dead_epoch":230},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:41:36.893546+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.d","osds":[4,7,1]},{"pgid":"6.0","osds":[1,0,6]}],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":6,"to":2}]},{"pgid":"3.19","mappings":[{"from":1,"to":2}]},{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":0}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"6.8","mappings":[{"from":2,"to":0}]},{"pgid":"6.1e","mappings":[{"from":6,"to":2}]},{"pgid":"6.1f","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/2180087772":"2024-02-06T07:19:28.895331+0000","172.21.15.7:0/2066685086":"2024-02-06T07:19:28.895331+0000","172.21.15.7:0/4100020626":"2024-02-06T07:19:28.895331+0000","172.21.15.7:0/1953351182":"2024-02-06T07:19:28.895331+0000","172.21.15.7:0/3203189162":"2024-02-06T07:19:28.895331+0000","172.21.15.7:0/3418781614":"2024-02-06T07:14:14.164720+0000","172.21.15.7:6800/1518592084":"2024-02-06T07:13:47.004765+0000","172.21.15.7:6800/3380493069":"2024-02-06T07:19:28.895331+0000","172.21.15.7:0/2208590106":"2024-02-06T07:13:47.004765+0000","172.21.15.7:0/223579766":"2024-02-06T07:13:47.004765+0000","172.21.15.7:6800/2634895574":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/1462812719":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/2823976078":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/1905520582":"2024-02-06T07:13:47.004765+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T07:42:53.119 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:52 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:53.119 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:52 smithi007 ceph-mon[29098]: pgmap v1369: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:53.119 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:52 smithi007 ceph-mon[29098]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T07:42:53.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:52 smithi007 ceph-mon[29098]: Cluster is now healthy 2024-02-05T07:42:53.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:52 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2219659944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:42:53.120 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:52 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:53.120 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:52 smithi007 ceph-mon[37158]: pgmap v1369: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:53.120 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:52 smithi007 ceph-mon[37158]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T07:42:53.120 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:52 smithi007 ceph-mon[37158]: Cluster is now healthy 2024-02-05T07:42:53.120 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:52 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2219659944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:42:53.146 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-05T07:42:53.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T07:42:53.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:52 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:42:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:52 smithi090 ceph-mon[33213]: pgmap v1369: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:52 smithi090 ceph-mon[33213]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T07:42:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:52 smithi090 ceph-mon[33213]: Cluster is now healthy 2024-02-05T07:42:53.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:52 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2219659944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:42:53.389 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:53.815 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:42:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:42:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:42:54.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:53 smithi007 ceph-mon[29098]: 4.1 scrub starts 2024-02-05T07:42:54.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:53 smithi007 ceph-mon[29098]: 4.1 scrub ok 2024-02-05T07:42:54.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:53 smithi007 ceph-mon[37158]: 4.1 scrub starts 2024-02-05T07:42:54.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:53 smithi007 ceph-mon[37158]: 4.1 scrub ok 2024-02-05T07:42:54.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:53 smithi090 ceph-mon[33213]: 4.1 scrub starts 2024-02-05T07:42:54.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:53 smithi090 ceph-mon[33213]: 4.1 scrub ok 2024-02-05T07:42:54.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:54 smithi007 ceph-mon[37158]: pgmap v1370: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:54.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:54 smithi007 ceph-mon[37158]: 3.3 deep-scrub starts 2024-02-05T07:42:54.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:54 smithi007 ceph-mon[37158]: 3.3 deep-scrub ok 2024-02-05T07:42:54.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:54 smithi007 ceph-mon[37158]: 1.8 scrub starts 2024-02-05T07:42:54.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:54 smithi007 ceph-mon[37158]: 1.8 scrub ok 2024-02-05T07:42:54.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:54 smithi007 ceph-mon[37158]: 4.4 deep-scrub starts 2024-02-05T07:42:54.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:54 smithi007 ceph-mon[37158]: 4.4 deep-scrub ok 2024-02-05T07:42:54.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:54 smithi007 ceph-mon[29098]: pgmap v1370: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:54.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:54 smithi007 ceph-mon[29098]: 3.3 deep-scrub starts 2024-02-05T07:42:54.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:54 smithi007 ceph-mon[29098]: 3.3 deep-scrub ok 2024-02-05T07:42:54.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:54 smithi007 ceph-mon[29098]: 1.8 scrub starts 2024-02-05T07:42:54.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:54 smithi007 ceph-mon[29098]: 1.8 scrub ok 2024-02-05T07:42:54.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:54 smithi007 ceph-mon[29098]: 4.4 deep-scrub starts 2024-02-05T07:42:54.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:54 smithi007 ceph-mon[29098]: 4.4 deep-scrub ok 2024-02-05T07:42:55.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:54 smithi090 ceph-mon[33213]: pgmap v1370: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:54 smithi090 ceph-mon[33213]: 3.3 deep-scrub starts 2024-02-05T07:42:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:54 smithi090 ceph-mon[33213]: 3.3 deep-scrub ok 2024-02-05T07:42:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:54 smithi090 ceph-mon[33213]: 1.8 scrub starts 2024-02-05T07:42:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:54 smithi090 ceph-mon[33213]: 1.8 scrub ok 2024-02-05T07:42:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:54 smithi090 ceph-mon[33213]: 4.4 deep-scrub starts 2024-02-05T07:42:55.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:54 smithi090 ceph-mon[33213]: 4.4 deep-scrub ok 2024-02-05T07:42:55.338 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:42:55.338 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":750,"fsid":"e3e7059e-c3f5-11ee-95b6-87774f69a715","created":"2024-02-05T07:13:14.603885+0000","modified":"2024-02-05T07:42:49.726865+0000","last_up_change":"2024-02-05T07:28:24.555976+0000","last_in_change":"2024-02-05T07:41:52.714677+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T07:16:32.505567+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"1.15","ready_epoch":681,"last_epoch_started":680,"last_epoch_clean":680,"source_version":"86'128","target_version":"86'128"},"last_change":"714","last_force_op_resend":"0","last_force_op_resend_prenautilus":"681","last_force_op_resend_preluminous":"679","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-05T07:18:47.259462+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-05T07:18:47.762336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-05T07:18:49.736599+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-05T07:18:51.696948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-05T07:18:53.768555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"703c3948-8003-4015-9dd8-21019cf0eeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":747,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6801","nonce":2069016689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2069016689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2069016689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":2069016689}]},"public_addr":"172.21.15.7:6801/2069016689","cluster_addr":"172.21.15.7:6802/2069016689","heartbeat_back_addr":"172.21.15.7:6804/2069016689","heartbeat_front_addr":"172.21.15.7:6803/2069016689","state":["exists","up"]},{"osd":1,"uuid":"5c3e7d1e-a61d-44b9-b21e-f91c99a096bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":726,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":865486532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":865486532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":865486532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6807","nonce":865486532}]},"public_addr":"172.21.15.7:6805/865486532","cluster_addr":"172.21.15.7:6806/865486532","heartbeat_back_addr":"172.21.15.7:6808/865486532","heartbeat_front_addr":"172.21.15.7:6807/865486532","state":["exists","up"]},{"osd":2,"uuid":"f4fd930f-7bb2-46ca-9964-98235bd47b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":742,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":904871370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":904871370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":904871370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":904871370}]},"public_addr":"172.21.15.7:6809/904871370","cluster_addr":"172.21.15.7:6810/904871370","heartbeat_back_addr":"172.21.15.7:6812/904871370","heartbeat_front_addr":"172.21.15.7:6811/904871370","state":["exists","up"]},{"osd":3,"uuid":"149ed0d6-b049-4478-9819-cafc446e94d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":723,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":2758586242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2758586242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2758586242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2758586242}]},"public_addr":"172.21.15.7:6813/2758586242","cluster_addr":"172.21.15.7:6814/2758586242","heartbeat_back_addr":"172.21.15.7:6816/2758586242","heartbeat_front_addr":"172.21.15.7:6815/2758586242","state":["exists","up"]},{"osd":4,"uuid":"3fbc7933-c77f-4a9e-8d8a-a3629f76bf6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":714,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":1154929470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6801","nonce":1154929470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6803","nonce":1154929470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1154929470}]},"public_addr":"172.21.15.90:6800/1154929470","cluster_addr":"172.21.15.90:6801/1154929470","heartbeat_back_addr":"172.21.15.90:6803/1154929470","heartbeat_front_addr":"172.21.15.90:6802/1154929470","state":["exists","up"]},{"osd":5,"uuid":"244a7930-c22c-413e-86e1-11e0749f7367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":743,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":398113577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6805","nonce":398113577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6807","nonce":398113577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":398113577}]},"public_addr":"172.21.15.90:6804/398113577","cluster_addr":"172.21.15.90:6805/398113577","heartbeat_back_addr":"172.21.15.90:6807/398113577","heartbeat_front_addr":"172.21.15.90:6806/398113577","state":["exists","up"]},{"osd":6,"uuid":"0adbb562-3012-4f60-b5ec-02550a10b2c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":52,"last_clean_end":230,"up_from":301,"up_thru":735,"down_at":231,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":313196533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":313196533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6811","nonce":313196533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":313196533}]},"public_addr":"172.21.15.90:6808/313196533","cluster_addr":"172.21.15.90:6809/313196533","heartbeat_back_addr":"172.21.15.90:6811/313196533","heartbeat_front_addr":"172.21.15.90:6810/313196533","state":["exists","up"]},{"osd":7,"uuid":"dbc26c36-a0e9-4f21-9e98-739270cea6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":748,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2983605980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":2983605980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":2983605980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2983605980}]},"public_addr":"172.21.15.90:6812/2983605980","cluster_addr":"172.21.15.90:6813/2983605980","heartbeat_back_addr":"172.21.15.90:6815/2983605980","heartbeat_front_addr":"172.21.15.90:6814/2983605980","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:41:54.408221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:42:10.829525+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:42:19.161399+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:42:03.480845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:42:16.436670+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:42:38.979003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:41:35.879130+0000","dead_epoch":230},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T07:41:36.893546+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.d","osds":[4,7,1]},{"pgid":"6.0","osds":[1,0,6]}],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":6,"to":2}]},{"pgid":"3.19","mappings":[{"from":1,"to":2}]},{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":0}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"6.8","mappings":[{"from":2,"to":0}]},{"pgid":"6.1e","mappings":[{"from":6,"to":2}]},{"pgid":"6.1f","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/2180087772":"2024-02-06T07:19:28.895331+0000","172.21.15.7:0/2066685086":"2024-02-06T07:19:28.895331+0000","172.21.15.7:0/4100020626":"2024-02-06T07:19:28.895331+0000","172.21.15.7:0/1953351182":"2024-02-06T07:19:28.895331+0000","172.21.15.7:0/3203189162":"2024-02-06T07:19:28.895331+0000","172.21.15.7:0/3418781614":"2024-02-06T07:14:14.164720+0000","172.21.15.7:6800/1518592084":"2024-02-06T07:13:47.004765+0000","172.21.15.7:6800/3380493069":"2024-02-06T07:19:28.895331+0000","172.21.15.7:0/2208590106":"2024-02-06T07:13:47.004765+0000","172.21.15.7:0/223579766":"2024-02-06T07:13:47.004765+0000","172.21.15.7:6800/2634895574":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/1462812719":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/2823976078":"2024-02-06T07:14:14.164720+0000","172.21.15.7:0/1905520582":"2024-02-06T07:13:47.004765+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T07:42:56.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:55 smithi007 ceph-mon[29098]: 3.18 scrub starts 2024-02-05T07:42:56.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:55 smithi007 ceph-mon[29098]: 3.18 scrub ok 2024-02-05T07:42:56.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:55 smithi007 ceph-mon[29098]: 3.9 scrub starts 2024-02-05T07:42:56.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:55 smithi007 ceph-mon[29098]: 3.9 scrub ok 2024-02-05T07:42:56.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:55 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3373333225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:42:56.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:55 smithi007 ceph-mon[37158]: 3.18 scrub starts 2024-02-05T07:42:56.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:55 smithi007 ceph-mon[37158]: 3.18 scrub ok 2024-02-05T07:42:56.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:55 smithi007 ceph-mon[37158]: 3.9 scrub starts 2024-02-05T07:42:56.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:55 smithi007 ceph-mon[37158]: 3.9 scrub ok 2024-02-05T07:42:56.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:55 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3373333225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:42:56.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-05T07:42:56.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-05T07:42:56.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-05T07:42:56.110 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-05T07:42:56.110 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-05T07:42:56.110 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-05T07:42:56.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-05T07:42:56.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-05T07:42:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:55 smithi090 ceph-mon[33213]: 3.18 scrub starts 2024-02-05T07:42:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:55 smithi090 ceph-mon[33213]: 3.18 scrub ok 2024-02-05T07:42:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:55 smithi090 ceph-mon[33213]: 3.9 scrub starts 2024-02-05T07:42:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:55 smithi090 ceph-mon[33213]: 3.9 scrub ok 2024-02-05T07:42:56.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:55 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3373333225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T07:42:56.602 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:56.623 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:56.625 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:56.626 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:56.627 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:56.627 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:42:56.991 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:56 smithi007 ceph-mon[37158]: pgmap v1371: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:56.992 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:56 smithi007 ceph-mon[37158]: 3.5 scrub starts 2024-02-05T07:42:56.992 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:56 smithi007 ceph-mon[37158]: 3.5 scrub ok 2024-02-05T07:42:56.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:56 smithi007 ceph-mon[29098]: pgmap v1371: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:56.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:56 smithi007 ceph-mon[29098]: 3.5 scrub starts 2024-02-05T07:42:56.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:56 smithi007 ceph-mon[29098]: 3.5 scrub ok 2024-02-05T07:42:57.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:56 smithi090 ceph-mon[33213]: pgmap v1371: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-02-05T07:42:57.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:56 smithi090 ceph-mon[33213]: 3.5 scrub starts 2024-02-05T07:42:57.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:56 smithi090 ceph-mon[33213]: 3.5 scrub ok 2024-02-05T07:42:57.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:57 smithi007 ceph-mon[29098]: 5.a scrub starts 2024-02-05T07:42:57.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:57 smithi007 ceph-mon[29098]: 5.a scrub ok 2024-02-05T07:42:57.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:57 smithi007 ceph-mon[37158]: 5.a scrub starts 2024-02-05T07:42:57.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:57 smithi007 ceph-mon[37158]: 5.a scrub ok 2024-02-05T07:42:57.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:57 smithi007 ceph-mon[37158]: 5.1d scrub starts 2024-02-05T07:42:58.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:57 smithi007 ceph-mon[29098]: 5.1d scrub starts 2024-02-05T07:42:58.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:57 smithi007 ceph-mon[29098]: 5.1d scrub ok 2024-02-05T07:42:58.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:57 smithi007 ceph-mon[29098]: 6.19 scrub starts 2024-02-05T07:42:58.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:57 smithi007 ceph-mon[29098]: 6.19 scrub ok 2024-02-05T07:42:58.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:57 smithi007 ceph-mon[37158]: 5.1d scrub ok 2024-02-05T07:42:58.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:57 smithi007 ceph-mon[37158]: 6.19 scrub starts 2024-02-05T07:42:58.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:57 smithi007 ceph-mon[37158]: 6.19 scrub ok 2024-02-05T07:42:58.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:57 smithi090 ceph-mon[33213]: 5.a scrub starts 2024-02-05T07:42:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:57 smithi090 ceph-mon[33213]: 5.a scrub ok 2024-02-05T07:42:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:57 smithi090 ceph-mon[33213]: 5.1d scrub starts 2024-02-05T07:42:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:57 smithi090 ceph-mon[33213]: 5.1d scrub ok 2024-02-05T07:42:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:57 smithi090 ceph-mon[33213]: 6.19 scrub starts 2024-02-05T07:42:58.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:57 smithi090 ceph-mon[33213]: 6.19 scrub ok 2024-02-05T07:42:58.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:58 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:42:58.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:58 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:42:58.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:58 smithi007 ceph-mon[29098]: pgmap v1372: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:42:58.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:58 smithi007 ceph-mon[29098]: 4.1e scrub starts 2024-02-05T07:42:58.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:58 smithi007 ceph-mon[29098]: 4.1e scrub ok 2024-02-05T07:42:58.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:58 smithi007 ceph-mon[29098]: osdmap e751: 8 total, 8 up, 8 in 2024-02-05T07:42:58.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:58 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:42:58.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:58 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:42:58.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:58 smithi007 ceph-mon[37158]: pgmap v1372: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:42:58.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:58 smithi007 ceph-mon[37158]: 4.1e scrub starts 2024-02-05T07:42:58.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:58 smithi007 ceph-mon[37158]: 4.1e scrub ok 2024-02-05T07:42:58.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:58 smithi007 ceph-mon[37158]: osdmap e751: 8 total, 8 up, 8 in 2024-02-05T07:42:59.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:58 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:42:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:58 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:42:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:58 smithi090 ceph-mon[33213]: pgmap v1372: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:42:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:58 smithi090 ceph-mon[33213]: 4.1e scrub starts 2024-02-05T07:42:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:58 smithi090 ceph-mon[33213]: 4.1e scrub ok 2024-02-05T07:42:59.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:58 smithi090 ceph-mon[33213]: osdmap e751: 8 total, 8 up, 8 in 2024-02-05T07:43:00.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:59 smithi007 ceph-mon[29098]: 4.2 scrub starts 2024-02-05T07:43:00.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:59 smithi007 ceph-mon[29098]: 4.2 scrub ok 2024-02-05T07:43:00.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:59 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:43:00.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:59 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:43:00.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:42:59 smithi007 ceph-mon[29098]: osdmap e752: 8 total, 8 up, 8 in 2024-02-05T07:43:00.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:59 smithi007 ceph-mon[37158]: 4.2 scrub starts 2024-02-05T07:43:00.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:59 smithi007 ceph-mon[37158]: 4.2 scrub ok 2024-02-05T07:43:00.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:59 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:43:00.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:59 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:43:00.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:42:59 smithi007 ceph-mon[37158]: osdmap e752: 8 total, 8 up, 8 in 2024-02-05T07:43:00.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:59 smithi090 ceph-mon[33213]: 4.2 scrub starts 2024-02-05T07:43:00.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:59 smithi090 ceph-mon[33213]: 4.2 scrub ok 2024-02-05T07:43:00.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:59 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:43:00.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:59 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:43:00.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:42:59 smithi090 ceph-mon[33213]: osdmap e752: 8 total, 8 up, 8 in 2024-02-05T07:43:00.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:00 smithi007 ceph-mon[29098]: pgmap v1374: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:43:00.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:00 smithi007 ceph-mon[29098]: 5.16 scrub starts 2024-02-05T07:43:00.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:00 smithi007 ceph-mon[29098]: 5.16 scrub ok 2024-02-05T07:43:00.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:00 smithi007 ceph-mon[37158]: pgmap v1374: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:43:00.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:00 smithi007 ceph-mon[37158]: 5.16 scrub starts 2024-02-05T07:43:00.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:00 smithi007 ceph-mon[37158]: 5.16 scrub ok 2024-02-05T07:43:01.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:00 smithi090 ceph-mon[33213]: pgmap v1374: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-05T07:43:01.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:00 smithi090 ceph-mon[33213]: 5.16 scrub starts 2024-02-05T07:43:01.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:00 smithi090 ceph-mon[33213]: 5.16 scrub ok 2024-02-05T07:43:01.773 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:43:01 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:43:02.579 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:02 smithi007 ceph-mon[29098]: 3.12 scrub starts 2024-02-05T07:43:02.579 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:02 smithi007 ceph-mon[29098]: 3.12 scrub ok 2024-02-05T07:43:02.579 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:02 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:43:02.579 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:02 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:43:02.579 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:02 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:43:02.579 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:02 smithi007 ceph-mon[29098]: pgmap v1376: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:02.580 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:02 smithi007 ceph-mon[37158]: 3.12 scrub starts 2024-02-05T07:43:02.580 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:02 smithi007 ceph-mon[37158]: 3.12 scrub ok 2024-02-05T07:43:02.580 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:02 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:43:02.580 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:02 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:43:02.580 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:02 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:43:02.580 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:02 smithi007 ceph-mon[37158]: pgmap v1376: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:02 smithi090 ceph-mon[33213]: 3.12 scrub starts 2024-02-05T07:43:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:02 smithi090 ceph-mon[33213]: 3.12 scrub ok 2024-02-05T07:43:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:02 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:43:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:02 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:43:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:02 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:43:02.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:02 smithi090 ceph-mon[33213]: pgmap v1376: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:03.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:03 smithi007 ceph-mon[29098]: 4.f scrub starts 2024-02-05T07:43:03.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:03 smithi007 ceph-mon[29098]: 4.f scrub ok 2024-02-05T07:43:03.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:03 smithi007 ceph-mon[29098]: 4.5 scrub starts 2024-02-05T07:43:03.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:03 smithi007 ceph-mon[29098]: 4.5 scrub ok 2024-02-05T07:43:03.536 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:03 smithi007 ceph-mon[37158]: 4.f scrub starts 2024-02-05T07:43:03.536 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:03 smithi007 ceph-mon[37158]: 4.f scrub ok 2024-02-05T07:43:03.536 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:03 smithi007 ceph-mon[37158]: 4.5 scrub starts 2024-02-05T07:43:03.536 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:03 smithi007 ceph-mon[37158]: 4.5 scrub ok 2024-02-05T07:43:03.773 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:03 smithi090 ceph-mon[33213]: 4.f scrub starts 2024-02-05T07:43:03.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:03 smithi090 ceph-mon[33213]: 4.f scrub ok 2024-02-05T07:43:03.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:03 smithi090 ceph-mon[33213]: 4.5 scrub starts 2024-02-05T07:43:03.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:03 smithi090 ceph-mon[33213]: 4.5 scrub ok 2024-02-05T07:43:03.825 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:03 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:43:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:43:04.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:04 smithi007 ceph-mon[29098]: pgmap v1377: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:04.599 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:04 smithi007 ceph-mon[37158]: pgmap v1377: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:04.773 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:04 smithi090 ceph-mon[33213]: pgmap v1377: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:05.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:05 smithi007 ceph-mon[29098]: 5.3 scrub starts 2024-02-05T07:43:05.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:05 smithi007 ceph-mon[29098]: 5.3 scrub ok 2024-02-05T07:43:05.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:05 smithi007 ceph-mon[29098]: 5.19 scrub starts 2024-02-05T07:43:05.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:05 smithi007 ceph-mon[29098]: 5.19 scrub ok 2024-02-05T07:43:05.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:05 smithi007 ceph-mon[29098]: 4.15 scrub starts 2024-02-05T07:43:05.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:05 smithi007 ceph-mon[29098]: 4.15 scrub ok 2024-02-05T07:43:05.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:05 smithi007 ceph-mon[37158]: 5.3 scrub starts 2024-02-05T07:43:05.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:05 smithi007 ceph-mon[37158]: 5.3 scrub ok 2024-02-05T07:43:05.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:05 smithi007 ceph-mon[37158]: 5.19 scrub starts 2024-02-05T07:43:05.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:05 smithi007 ceph-mon[37158]: 5.19 scrub ok 2024-02-05T07:43:05.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:05 smithi007 ceph-mon[37158]: 4.15 scrub starts 2024-02-05T07:43:05.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:05 smithi007 ceph-mon[37158]: 4.15 scrub ok 2024-02-05T07:43:05.773 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:05 smithi090 ceph-mon[33213]: 5.3 scrub starts 2024-02-05T07:43:05.773 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:05 smithi090 ceph-mon[33213]: 5.3 scrub ok 2024-02-05T07:43:05.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:05 smithi090 ceph-mon[33213]: 5.19 scrub starts 2024-02-05T07:43:05.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:05 smithi090 ceph-mon[33213]: 5.19 scrub ok 2024-02-05T07:43:05.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:05 smithi090 ceph-mon[33213]: 4.15 scrub starts 2024-02-05T07:43:05.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:05 smithi090 ceph-mon[33213]: 4.15 scrub ok 2024-02-05T07:43:06.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:06 smithi007 ceph-mon[29098]: 4.13 scrub starts 2024-02-05T07:43:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:06 smithi007 ceph-mon[29098]: 4.13 scrub ok 2024-02-05T07:43:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:06 smithi007 ceph-mon[29098]: pgmap v1378: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:06 smithi007 ceph-mon[29098]: 6.4 scrub starts 2024-02-05T07:43:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:06 smithi007 ceph-mon[29098]: 6.4 scrub ok 2024-02-05T07:43:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:06 smithi007 ceph-mon[29098]: 3.c scrub starts 2024-02-05T07:43:06.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:06 smithi007 ceph-mon[29098]: 3.c scrub ok 2024-02-05T07:43:06.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:06 smithi007 ceph-mon[37158]: 4.13 scrub starts 2024-02-05T07:43:06.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:06 smithi007 ceph-mon[37158]: 4.13 scrub ok 2024-02-05T07:43:06.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:06 smithi007 ceph-mon[37158]: pgmap v1378: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:06.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:06 smithi007 ceph-mon[37158]: 6.4 scrub starts 2024-02-05T07:43:06.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:06 smithi007 ceph-mon[37158]: 6.4 scrub ok 2024-02-05T07:43:06.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:06 smithi007 ceph-mon[37158]: 3.c scrub starts 2024-02-05T07:43:06.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:06 smithi007 ceph-mon[37158]: 3.c scrub ok 2024-02-05T07:43:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:06 smithi090 ceph-mon[33213]: 4.13 scrub starts 2024-02-05T07:43:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:06 smithi090 ceph-mon[33213]: 4.13 scrub ok 2024-02-05T07:43:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:06 smithi090 ceph-mon[33213]: pgmap v1378: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:06 smithi090 ceph-mon[33213]: 6.4 scrub starts 2024-02-05T07:43:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:06 smithi090 ceph-mon[33213]: 6.4 scrub ok 2024-02-05T07:43:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:06 smithi090 ceph-mon[33213]: 3.c scrub starts 2024-02-05T07:43:06.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:06 smithi090 ceph-mon[33213]: 3.c scrub ok 2024-02-05T07:43:07.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:07 smithi090 ceph-mon[33213]: 4.9 scrub starts 2024-02-05T07:43:07.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:07 smithi090 ceph-mon[33213]: 4.9 scrub ok 2024-02-05T07:43:07.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:07 smithi090 ceph-mon[33213]: 6.0 scrub starts 2024-02-05T07:43:07.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:07 smithi090 ceph-mon[33213]: 6.0 scrub ok 2024-02-05T07:43:07.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:07 smithi007 ceph-mon[29098]: 4.9 scrub starts 2024-02-05T07:43:07.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:07 smithi007 ceph-mon[29098]: 4.9 scrub ok 2024-02-05T07:43:07.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:07 smithi007 ceph-mon[29098]: 6.0 scrub starts 2024-02-05T07:43:07.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:07 smithi007 ceph-mon[29098]: 6.0 scrub ok 2024-02-05T07:43:07.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:07 smithi007 ceph-mon[37158]: 4.9 scrub starts 2024-02-05T07:43:07.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:07 smithi007 ceph-mon[37158]: 4.9 scrub ok 2024-02-05T07:43:07.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:07 smithi007 ceph-mon[37158]: 6.0 scrub starts 2024-02-05T07:43:07.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:07 smithi007 ceph-mon[37158]: 6.0 scrub ok 2024-02-05T07:43:07.844 INFO:teuthology.orchestra.run.smithi007.stdout:1292785156274 2024-02-05T07:43:07.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-05T07:43:08.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:08 smithi007 ceph-mon[29098]: 6.1f scrub starts 2024-02-05T07:43:08.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:08 smithi007 ceph-mon[29098]: 5.e scrub starts 2024-02-05T07:43:08.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:08 smithi007 ceph-mon[29098]: 5.e scrub ok 2024-02-05T07:43:08.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:08 smithi007 ceph-mon[29098]: 6.1f scrub ok 2024-02-05T07:43:08.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:08 smithi007 ceph-mon[29098]: pgmap v1379: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:43:08.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:08 smithi007 ceph-mon[29098]: 4.14 scrub starts 2024-02-05T07:43:08.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:08 smithi007 ceph-mon[29098]: 4.14 scrub ok 2024-02-05T07:43:08.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:08 smithi007 ceph-mon[37158]: 6.1f scrub starts 2024-02-05T07:43:08.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:08 smithi007 ceph-mon[37158]: 5.e scrub starts 2024-02-05T07:43:08.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:08 smithi007 ceph-mon[37158]: 5.e scrub ok 2024-02-05T07:43:08.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:08 smithi007 ceph-mon[37158]: 6.1f scrub ok 2024-02-05T07:43:08.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:08 smithi007 ceph-mon[37158]: pgmap v1379: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:43:08.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:08 smithi007 ceph-mon[37158]: 4.14 scrub starts 2024-02-05T07:43:08.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:08 smithi007 ceph-mon[37158]: 4.14 scrub ok 2024-02-05T07:43:08.773 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:08 smithi090 ceph-mon[33213]: 6.1f scrub starts 2024-02-05T07:43:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:08 smithi090 ceph-mon[33213]: 5.e scrub starts 2024-02-05T07:43:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:08 smithi090 ceph-mon[33213]: 5.e scrub ok 2024-02-05T07:43:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:08 smithi090 ceph-mon[33213]: 6.1f scrub ok 2024-02-05T07:43:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:08 smithi090 ceph-mon[33213]: pgmap v1379: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T07:43:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:08 smithi090 ceph-mon[33213]: 4.14 scrub starts 2024-02-05T07:43:08.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:08 smithi090 ceph-mon[33213]: 4.14 scrub ok 2024-02-05T07:43:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:09 smithi090 ceph-mon[33213]: 4.1a scrub starts 2024-02-05T07:43:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:09 smithi090 ceph-mon[33213]: 4.1a scrub ok 2024-02-05T07:43:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:09 smithi090 ceph-mon[33213]: 6.14 scrub starts 2024-02-05T07:43:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:09 smithi090 ceph-mon[33213]: 6.14 scrub ok 2024-02-05T07:43:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:09 smithi090 ceph-mon[33213]: 4.1f scrub starts 2024-02-05T07:43:09.774 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:09 smithi090 ceph-mon[33213]: 4.1f scrub ok 2024-02-05T07:43:09.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:09 smithi007 ceph-mon[29098]: 4.1a scrub starts 2024-02-05T07:43:09.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:09 smithi007 ceph-mon[29098]: 4.1a scrub ok 2024-02-05T07:43:09.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:09 smithi007 ceph-mon[29098]: 6.14 scrub starts 2024-02-05T07:43:09.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:09 smithi007 ceph-mon[29098]: 6.14 scrub ok 2024-02-05T07:43:09.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:09 smithi007 ceph-mon[29098]: 4.1f scrub starts 2024-02-05T07:43:09.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:09 smithi007 ceph-mon[29098]: 4.1f scrub ok 2024-02-05T07:43:09.883 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:09 smithi007 ceph-mon[37158]: 4.1a scrub starts 2024-02-05T07:43:09.883 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:09 smithi007 ceph-mon[37158]: 4.1a scrub ok 2024-02-05T07:43:09.883 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:09 smithi007 ceph-mon[37158]: 6.14 scrub starts 2024-02-05T07:43:09.883 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:09 smithi007 ceph-mon[37158]: 6.14 scrub ok 2024-02-05T07:43:09.883 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:09 smithi007 ceph-mon[37158]: 4.1f scrub starts 2024-02-05T07:43:09.883 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:09 smithi007 ceph-mon[37158]: 4.1f scrub ok 2024-02-05T07:43:10.100 INFO:teuthology.orchestra.run.smithi007.stdout:55834575174 2024-02-05T07:43:10.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-05T07:43:10.280 INFO:teuthology.orchestra.run.smithi007.stdout:176093659443 2024-02-05T07:43:10.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-05T07:43:11.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:10 smithi090 ceph-mon[33213]: pgmap v1380: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T07:43:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:10 smithi090 ceph-mon[33213]: 3.8 deep-scrub starts 2024-02-05T07:43:11.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:10 smithi090 ceph-mon[33213]: 3.8 deep-scrub ok 2024-02-05T07:43:11.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:10 smithi007 ceph-mon[29098]: pgmap v1380: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T07:43:11.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:10 smithi007 ceph-mon[29098]: 3.8 deep-scrub starts 2024-02-05T07:43:11.148 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:10 smithi007 ceph-mon[29098]: 3.8 deep-scrub ok 2024-02-05T07:43:11.148 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:10 smithi007 ceph-mon[37158]: pgmap v1380: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T07:43:11.148 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:10 smithi007 ceph-mon[37158]: 3.8 deep-scrub starts 2024-02-05T07:43:11.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:10 smithi007 ceph-mon[37158]: 3.8 deep-scrub ok 2024-02-05T07:43:11.176 INFO:teuthology.orchestra.run.smithi007.stdout:77309411650 2024-02-05T07:43:11.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-05T07:43:11.596 INFO:teuthology.orchestra.run.smithi007.stdout:257698038057 2024-02-05T07:43:11.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-05T07:43:11.742 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:43:11 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:43:11.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:11 smithi007 ceph-mon[29098]: 4.1c scrub starts 2024-02-05T07:43:11.827 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:11 smithi007 ceph-mon[29098]: 4.1c scrub ok 2024-02-05T07:43:11.827 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:11 smithi007 ceph-mon[29098]: 6.10 deep-scrub starts 2024-02-05T07:43:11.827 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:11 smithi007 ceph-mon[29098]: 6.10 deep-scrub ok 2024-02-05T07:43:11.827 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:11 smithi007 ceph-mon[29098]: 3.11 scrub starts 2024-02-05T07:43:11.827 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:11 smithi007 ceph-mon[29098]: 3.11 scrub ok 2024-02-05T07:43:11.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:11 smithi007 ceph-mon[37158]: 4.1c scrub starts 2024-02-05T07:43:11.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:11 smithi007 ceph-mon[37158]: 4.1c scrub ok 2024-02-05T07:43:11.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:11 smithi007 ceph-mon[37158]: 6.10 deep-scrub starts 2024-02-05T07:43:11.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:11 smithi007 ceph-mon[37158]: 6.10 deep-scrub ok 2024-02-05T07:43:11.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:11 smithi007 ceph-mon[37158]: 3.11 scrub starts 2024-02-05T07:43:11.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:11 smithi007 ceph-mon[37158]: 3.11 scrub ok 2024-02-05T07:43:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:11 smithi090 ceph-mon[33213]: 4.1c scrub starts 2024-02-05T07:43:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:11 smithi090 ceph-mon[33213]: 4.1c scrub ok 2024-02-05T07:43:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:11 smithi090 ceph-mon[33213]: 6.10 deep-scrub starts 2024-02-05T07:43:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:11 smithi090 ceph-mon[33213]: 6.10 deep-scrub ok 2024-02-05T07:43:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:11 smithi090 ceph-mon[33213]: 3.11 scrub starts 2024-02-05T07:43:12.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:11 smithi090 ceph-mon[33213]: 3.11 scrub ok 2024-02-05T07:43:12.468 INFO:teuthology.orchestra.run.smithi007.stdout:34359738700 2024-02-05T07:43:12.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-05T07:43:12.943 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:43:12.944 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:43:12.944 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:43:12.944 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:43:13.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:12 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:43:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:12 smithi090 ceph-mon[33213]: pgmap v1381: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:43:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:12 smithi090 ceph-mon[33213]: 4.10 scrub starts 2024-02-05T07:43:13.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:12 smithi090 ceph-mon[33213]: 4.10 scrub ok 2024-02-05T07:43:13.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:12 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:43:13.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:12 smithi007 ceph-mon[29098]: pgmap v1381: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:43:13.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:12 smithi007 ceph-mon[29098]: 4.10 scrub starts 2024-02-05T07:43:13.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:12 smithi007 ceph-mon[29098]: 4.10 scrub ok 2024-02-05T07:43:13.036 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:12 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:43:13.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:12 smithi007 ceph-mon[37158]: pgmap v1381: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T07:43:13.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:12 smithi007 ceph-mon[37158]: 4.10 scrub starts 2024-02-05T07:43:13.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:12 smithi007 ceph-mon[37158]: 4.10 scrub ok 2024-02-05T07:43:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:43:13.573 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:43:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:43:14.362 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:43:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:14 smithi090 ceph-mon[33213]: 6.1a scrub starts 2024-02-05T07:43:15.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:14 smithi090 ceph-mon[33213]: 6.1a scrub ok 2024-02-05T07:43:15.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:14 smithi090 ceph-mon[33213]: pgmap v1382: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:43:15.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:14 smithi090 ceph-mon[33213]: 4.11 scrub starts 2024-02-05T07:43:15.025 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:14 smithi090 ceph-mon[33213]: 4.11 scrub ok 2024-02-05T07:43:15.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:14 smithi007 ceph-mon[29098]: 6.1a scrub starts 2024-02-05T07:43:15.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:14 smithi007 ceph-mon[29098]: 6.1a scrub ok 2024-02-05T07:43:15.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:14 smithi007 ceph-mon[29098]: pgmap v1382: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:43:15.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:14 smithi007 ceph-mon[29098]: 4.11 scrub starts 2024-02-05T07:43:15.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:14 smithi007 ceph-mon[29098]: 4.11 scrub ok 2024-02-05T07:43:15.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:14 smithi007 ceph-mon[37158]: 6.1a scrub starts 2024-02-05T07:43:15.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:14 smithi007 ceph-mon[37158]: 6.1a scrub ok 2024-02-05T07:43:15.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:14 smithi007 ceph-mon[37158]: pgmap v1382: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:43:15.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:14 smithi007 ceph-mon[37158]: 4.11 scrub starts 2024-02-05T07:43:15.092 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:14 smithi007 ceph-mon[37158]: 4.11 scrub ok 2024-02-05T07:43:16.780 INFO:teuthology.orchestra.run.smithi007.stdout:111669150014 2024-02-05T07:43:16.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-05T07:43:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:16 smithi090 ceph-mon[33213]: 3.1a scrub starts 2024-02-05T07:43:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:16 smithi090 ceph-mon[33213]: 3.1a scrub ok 2024-02-05T07:43:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:16 smithi090 ceph-mon[33213]: 5.b scrub starts 2024-02-05T07:43:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:16 smithi090 ceph-mon[33213]: 5.b scrub ok 2024-02-05T07:43:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:16 smithi090 ceph-mon[33213]: 5.12 scrub starts 2024-02-05T07:43:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:16 smithi090 ceph-mon[33213]: 5.12 scrub ok 2024-02-05T07:43:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:16 smithi090 ceph-mon[33213]: pgmap v1383: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:43:17.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:16 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:43:17.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[29098]: 3.1a scrub starts 2024-02-05T07:43:17.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[29098]: 3.1a scrub ok 2024-02-05T07:43:17.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[29098]: 5.b scrub starts 2024-02-05T07:43:17.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[29098]: 5.b scrub ok 2024-02-05T07:43:17.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[29098]: 5.12 scrub starts 2024-02-05T07:43:17.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[29098]: 5.12 scrub ok 2024-02-05T07:43:17.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[29098]: pgmap v1383: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:43:17.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:43:17.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[37158]: 3.1a scrub starts 2024-02-05T07:43:17.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[37158]: 3.1a scrub ok 2024-02-05T07:43:17.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[37158]: 5.b scrub starts 2024-02-05T07:43:17.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[37158]: 5.b scrub ok 2024-02-05T07:43:17.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[37158]: 5.12 scrub starts 2024-02-05T07:43:17.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[37158]: 5.12 scrub ok 2024-02-05T07:43:17.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[37158]: pgmap v1383: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:43:17.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:16 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:43:17.917 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:17 smithi007 ceph-mon[29098]: 1.5 scrub starts 2024-02-05T07:43:17.917 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:17 smithi007 ceph-mon[29098]: 1.5 scrub ok 2024-02-05T07:43:17.917 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:17 smithi007 ceph-mon[29098]: 6.5 scrub starts 2024-02-05T07:43:17.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:17 smithi007 ceph-mon[29098]: 6.5 scrub ok 2024-02-05T07:43:17.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:17 smithi007 ceph-mon[29098]: 3.16 scrub starts 2024-02-05T07:43:17.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:17 smithi007 ceph-mon[29098]: 3.16 scrub ok 2024-02-05T07:43:17.918 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:17 smithi007 ceph-mon[37158]: 1.5 scrub starts 2024-02-05T07:43:17.918 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:17 smithi007 ceph-mon[37158]: 1.5 scrub ok 2024-02-05T07:43:17.918 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:17 smithi007 ceph-mon[37158]: 6.5 scrub starts 2024-02-05T07:43:17.918 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:17 smithi007 ceph-mon[37158]: 6.5 scrub ok 2024-02-05T07:43:17.918 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:17 smithi007 ceph-mon[37158]: 3.16 scrub starts 2024-02-05T07:43:17.918 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:17 smithi007 ceph-mon[37158]: 3.16 scrub ok 2024-02-05T07:43:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:17 smithi090 ceph-mon[33213]: 1.5 scrub starts 2024-02-05T07:43:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:17 smithi090 ceph-mon[33213]: 1.5 scrub ok 2024-02-05T07:43:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:17 smithi090 ceph-mon[33213]: 6.5 scrub starts 2024-02-05T07:43:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:17 smithi090 ceph-mon[33213]: 6.5 scrub ok 2024-02-05T07:43:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:17 smithi090 ceph-mon[33213]: 3.16 scrub starts 2024-02-05T07:43:18.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:17 smithi090 ceph-mon[33213]: 3.16 scrub ok 2024-02-05T07:43:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:18 smithi007 ceph-mon[29098]: 5.18 scrub starts 2024-02-05T07:43:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:18 smithi007 ceph-mon[29098]: 5.18 scrub ok 2024-02-05T07:43:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:18 smithi007 ceph-mon[29098]: pgmap v1384: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:18 smithi007 ceph-mon[29098]: 5.4 scrub starts 2024-02-05T07:43:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:18 smithi007 ceph-mon[29098]: 5.4 scrub ok 2024-02-05T07:43:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:18 smithi007 ceph-mon[29098]: 5.1b scrub starts 2024-02-05T07:43:18.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:18 smithi007 ceph-mon[29098]: 5.1b scrub ok 2024-02-05T07:43:18.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:18 smithi007 ceph-mon[37158]: 5.18 scrub starts 2024-02-05T07:43:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:18 smithi007 ceph-mon[37158]: 5.18 scrub ok 2024-02-05T07:43:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:18 smithi007 ceph-mon[37158]: pgmap v1384: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:18 smithi007 ceph-mon[37158]: 5.4 scrub starts 2024-02-05T07:43:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:18 smithi007 ceph-mon[37158]: 5.4 scrub ok 2024-02-05T07:43:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:18 smithi007 ceph-mon[37158]: 5.1b scrub starts 2024-02-05T07:43:18.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:18 smithi007 ceph-mon[37158]: 5.1b scrub ok 2024-02-05T07:43:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:18 smithi090 ceph-mon[33213]: 5.18 scrub starts 2024-02-05T07:43:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:18 smithi090 ceph-mon[33213]: 5.18 scrub ok 2024-02-05T07:43:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:18 smithi090 ceph-mon[33213]: pgmap v1384: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:18 smithi090 ceph-mon[33213]: 5.4 scrub starts 2024-02-05T07:43:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:18 smithi090 ceph-mon[33213]: 5.4 scrub ok 2024-02-05T07:43:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:18 smithi090 ceph-mon[33213]: 5.1b scrub starts 2024-02-05T07:43:19.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:18 smithi090 ceph-mon[33213]: 5.1b scrub ok 2024-02-05T07:43:19.271 INFO:teuthology.orchestra.run.smithi007.stdout:137438953785 2024-02-05T07:43:19.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-05T07:43:19.678 INFO:teuthology.orchestra.run.smithi007.stdout:1292785156276 2024-02-05T07:43:19.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:19 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:43:19.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:19 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:43:19.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:19 smithi007 ceph-mon[29098]: 6.1 scrub starts 2024-02-05T07:43:19.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:19 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2067277808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T07:43:19.897 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:19 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:43:19.897 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:19 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:43:19.897 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:19 smithi007 ceph-mon[37158]: 6.1 scrub starts 2024-02-05T07:43:19.897 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:19 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2067277808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T07:43:20.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:19 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:43:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:19 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:43:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:19 smithi090 ceph-mon[33213]: 6.1 scrub starts 2024-02-05T07:43:20.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:19 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2067277808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T07:43:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[29098]: 5.6 deep-scrub starts 2024-02-05T07:43:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[29098]: 5.6 deep-scrub ok 2024-02-05T07:43:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[29098]: 6.1 scrub ok 2024-02-05T07:43:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[29098]: 4.19 deep-scrub starts 2024-02-05T07:43:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[29098]: 4.19 deep-scrub ok 2024-02-05T07:43:20.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[29098]: pgmap v1385: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:43:20.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[29098]: osdmap e753: 8 total, 8 up, 8 in 2024-02-05T07:43:20.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[29098]: 3.15 scrub starts 2024-02-05T07:43:20.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[29098]: 3.15 scrub ok 2024-02-05T07:43:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[37158]: 5.6 deep-scrub starts 2024-02-05T07:43:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[37158]: 5.6 deep-scrub ok 2024-02-05T07:43:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[37158]: 6.1 scrub ok 2024-02-05T07:43:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[37158]: 4.19 deep-scrub starts 2024-02-05T07:43:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[37158]: 4.19 deep-scrub ok 2024-02-05T07:43:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[37158]: pgmap v1385: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:43:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[37158]: osdmap e753: 8 total, 8 up, 8 in 2024-02-05T07:43:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[37158]: 3.15 scrub starts 2024-02-05T07:43:20.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:20 smithi007 ceph-mon[37158]: 3.15 scrub ok 2024-02-05T07:43:21.023 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:20 smithi090 ceph-mon[33213]: 5.6 deep-scrub starts 2024-02-05T07:43:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:20 smithi090 ceph-mon[33213]: 5.6 deep-scrub ok 2024-02-05T07:43:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:20 smithi090 ceph-mon[33213]: 6.1 scrub ok 2024-02-05T07:43:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:20 smithi090 ceph-mon[33213]: 4.19 deep-scrub starts 2024-02-05T07:43:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:20 smithi090 ceph-mon[33213]: 4.19 deep-scrub ok 2024-02-05T07:43:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:20 smithi090 ceph-mon[33213]: pgmap v1385: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T07:43:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:20 smithi090 ceph-mon[33213]: osdmap e753: 8 total, 8 up, 8 in 2024-02-05T07:43:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:20 smithi090 ceph-mon[33213]: 3.15 scrub starts 2024-02-05T07:43:21.024 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:20 smithi090 ceph-mon[33213]: 3.15 scrub ok 2024-02-05T07:43:21.384 INFO:teuthology.orchestra.run.smithi007.stdout:77309411652 2024-02-05T07:43:21.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:43:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:43:21.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:21 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:43:21.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:21 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:43:21.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:21 smithi007 ceph-mon[29098]: 6.16 deep-scrub starts 2024-02-05T07:43:21.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:21 smithi007 ceph-mon[29098]: 6.16 deep-scrub ok 2024-02-05T07:43:21.993 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:21 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2820097352' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T07:43:21.993 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:21 smithi007 ceph-mon[29098]: 1.3 scrub starts 2024-02-05T07:43:21.993 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:21 smithi007 ceph-mon[29098]: 1.3 scrub ok 2024-02-05T07:43:21.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:21 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:43:21.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:21 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:43:21.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:21 smithi007 ceph-mon[37158]: 6.16 deep-scrub starts 2024-02-05T07:43:21.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:21 smithi007 ceph-mon[37158]: 6.16 deep-scrub ok 2024-02-05T07:43:21.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:21 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2820097352' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T07:43:21.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:21 smithi007 ceph-mon[37158]: 1.3 scrub starts 2024-02-05T07:43:21.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:21 smithi007 ceph-mon[37158]: 1.3 scrub ok 2024-02-05T07:43:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:21 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:43:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:21 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:43:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:21 smithi090 ceph-mon[33213]: 6.16 deep-scrub starts 2024-02-05T07:43:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:21 smithi090 ceph-mon[33213]: 6.16 deep-scrub ok 2024-02-05T07:43:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:21 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2820097352' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T07:43:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:21 smithi090 ceph-mon[33213]: 1.3 scrub starts 2024-02-05T07:43:22.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:21 smithi090 ceph-mon[33213]: 1.3 scrub ok 2024-02-05T07:43:22.757 INFO:teuthology.orchestra.run.smithi007.stdout:34359738702 2024-02-05T07:43:23.014 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:43:23.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[29098]: 3.1e scrub starts 2024-02-05T07:43:23.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[29098]: 3.1e scrub ok 2024-02-05T07:43:23.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[29098]: pgmap v1387: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:43:23.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[29098]: 4.a scrub starts 2024-02-05T07:43:23.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[29098]: 4.a scrub ok 2024-02-05T07:43:23.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[29098]: osdmap e754: 8 total, 8 up, 8 in 2024-02-05T07:43:23.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/386559270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T07:43:23.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:43:23.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[37158]: 3.1e scrub starts 2024-02-05T07:43:23.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[37158]: 3.1e scrub ok 2024-02-05T07:43:23.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[37158]: pgmap v1387: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:43:23.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[37158]: 4.a scrub starts 2024-02-05T07:43:23.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[37158]: 4.a scrub ok 2024-02-05T07:43:23.016 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[37158]: osdmap e754: 8 total, 8 up, 8 in 2024-02-05T07:43:23.016 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:22 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/386559270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T07:43:23.043 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1292785156274 got 1292785156276 for osd.6 2024-02-05T07:43:23.043 DEBUG:teuthology.parallel:result is None 2024-02-05T07:43:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:22 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:43:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:22 smithi090 ceph-mon[33213]: 3.1e scrub starts 2024-02-05T07:43:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:22 smithi090 ceph-mon[33213]: 3.1e scrub ok 2024-02-05T07:43:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:22 smithi090 ceph-mon[33213]: pgmap v1387: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T07:43:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:22 smithi090 ceph-mon[33213]: 4.a scrub starts 2024-02-05T07:43:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:22 smithi090 ceph-mon[33213]: 4.a scrub ok 2024-02-05T07:43:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:22 smithi090 ceph-mon[33213]: osdmap e754: 8 total, 8 up, 8 in 2024-02-05T07:43:23.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:22 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/386559270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T07:43:23.673 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:23 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:43:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:43:24.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:23 smithi007 ceph-mon[29098]: 4.1b scrub starts 2024-02-05T07:43:24.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:23 smithi007 ceph-mon[29098]: 4.1b scrub ok 2024-02-05T07:43:24.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:23 smithi007 ceph-mon[29098]: 1.9 deep-scrub starts 2024-02-05T07:43:24.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:23 smithi007 ceph-mon[29098]: 1.9 deep-scrub ok 2024-02-05T07:43:24.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:23 smithi007 ceph-mon[29098]: 6.7 scrub starts 2024-02-05T07:43:24.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:23 smithi007 ceph-mon[29098]: 6.7 scrub ok 2024-02-05T07:43:24.101 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:23 smithi007 ceph-mon[37158]: 4.1b scrub starts 2024-02-05T07:43:24.101 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:23 smithi007 ceph-mon[37158]: 4.1b scrub ok 2024-02-05T07:43:24.101 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:23 smithi007 ceph-mon[37158]: 1.9 deep-scrub starts 2024-02-05T07:43:24.101 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:23 smithi007 ceph-mon[37158]: 1.9 deep-scrub ok 2024-02-05T07:43:24.101 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:23 smithi007 ceph-mon[37158]: 6.7 scrub starts 2024-02-05T07:43:24.101 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:23 smithi007 ceph-mon[37158]: 6.7 scrub ok 2024-02-05T07:43:24.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:23 smithi090 ceph-mon[33213]: 4.1b scrub starts 2024-02-05T07:43:24.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:23 smithi090 ceph-mon[33213]: 4.1b scrub ok 2024-02-05T07:43:24.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:23 smithi090 ceph-mon[33213]: 1.9 deep-scrub starts 2024-02-05T07:43:24.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:23 smithi090 ceph-mon[33213]: 1.9 deep-scrub ok 2024-02-05T07:43:24.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:23 smithi090 ceph-mon[33213]: 6.7 scrub starts 2024-02-05T07:43:24.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:23 smithi090 ceph-mon[33213]: 6.7 scrub ok 2024-02-05T07:43:24.721 INFO:teuthology.orchestra.run.smithi007.stdout:55834575178 2024-02-05T07:43:24.974 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[29098]: 4.c scrub starts 2024-02-05T07:43:24.974 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[29098]: 4.c scrub ok 2024-02-05T07:43:24.974 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[29098]: 5.8 scrub starts 2024-02-05T07:43:24.974 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[29098]: 5.8 scrub ok 2024-02-05T07:43:24.974 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[29098]: pgmap v1389: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:24.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[29098]: 1.2 scrub starts 2024-02-05T07:43:24.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[29098]: 1.2 scrub ok 2024-02-05T07:43:24.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[29098]: 1.12 scrub starts 2024-02-05T07:43:24.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[29098]: 1.12 scrub ok 2024-02-05T07:43:24.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1356290560' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T07:43:24.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[37158]: 4.c scrub starts 2024-02-05T07:43:24.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[37158]: 4.c scrub ok 2024-02-05T07:43:24.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[37158]: 5.8 scrub starts 2024-02-05T07:43:24.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[37158]: 5.8 scrub ok 2024-02-05T07:43:24.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[37158]: pgmap v1389: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:24.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[37158]: 1.2 scrub starts 2024-02-05T07:43:24.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[37158]: 1.2 scrub ok 2024-02-05T07:43:24.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[37158]: 1.12 scrub starts 2024-02-05T07:43:24.976 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[37158]: 1.12 scrub ok 2024-02-05T07:43:24.976 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:24 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1356290560' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T07:43:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:24 smithi090 ceph-mon[33213]: 4.c scrub starts 2024-02-05T07:43:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:24 smithi090 ceph-mon[33213]: 4.c scrub ok 2024-02-05T07:43:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:24 smithi090 ceph-mon[33213]: 5.8 scrub starts 2024-02-05T07:43:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:24 smithi090 ceph-mon[33213]: 5.8 scrub ok 2024-02-05T07:43:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:24 smithi090 ceph-mon[33213]: pgmap v1389: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:24 smithi090 ceph-mon[33213]: 1.2 scrub starts 2024-02-05T07:43:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:24 smithi090 ceph-mon[33213]: 1.2 scrub ok 2024-02-05T07:43:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:24 smithi090 ceph-mon[33213]: 1.12 scrub starts 2024-02-05T07:43:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:24 smithi090 ceph-mon[33213]: 1.12 scrub ok 2024-02-05T07:43:25.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:24 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1356290560' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T07:43:25.302 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738700 got 34359738702 for osd.0 2024-02-05T07:43:25.302 DEBUG:teuthology.parallel:result is None 2024-02-05T07:43:25.496 INFO:teuthology.orchestra.run.smithi007.stdout:176093659446 2024-02-05T07:43:25.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:25 smithi007 ceph-mon[29098]: 5.1 scrub starts 2024-02-05T07:43:25.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:25 smithi007 ceph-mon[29098]: 5.1 scrub ok 2024-02-05T07:43:25.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:25 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/1686176189' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T07:43:25.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:25 smithi007 ceph-mon[37158]: 5.1 scrub starts 2024-02-05T07:43:25.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:25 smithi007 ceph-mon[37158]: 5.1 scrub ok 2024-02-05T07:43:25.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:25 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/1686176189' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T07:43:26.269 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575174 got 55834575178 for osd.1 2024-02-05T07:43:26.269 DEBUG:teuthology.parallel:result is None 2024-02-05T07:43:26.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:25 smithi090 ceph-mon[33213]: 5.1 scrub starts 2024-02-05T07:43:26.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:25 smithi090 ceph-mon[33213]: 5.1 scrub ok 2024-02-05T07:43:26.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:25 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/1686176189' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T07:43:27.030 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411650 got 77309411652 for osd.2 2024-02-05T07:43:27.031 DEBUG:teuthology.parallel:result is None 2024-02-05T07:43:27.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:26 smithi007 ceph-mon[29098]: pgmap v1390: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:27.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:26 smithi007 ceph-mon[29098]: 1.e scrub starts 2024-02-05T07:43:27.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:26 smithi007 ceph-mon[29098]: 1.e scrub ok 2024-02-05T07:43:27.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:26 smithi007 ceph-mon[29098]: 3.0 scrub starts 2024-02-05T07:43:27.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:26 smithi007 ceph-mon[29098]: 3.0 scrub ok 2024-02-05T07:43:27.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:26 smithi007 ceph-mon[37158]: pgmap v1390: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:27.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:26 smithi007 ceph-mon[37158]: 1.e scrub starts 2024-02-05T07:43:27.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:26 smithi007 ceph-mon[37158]: 1.e scrub ok 2024-02-05T07:43:27.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:26 smithi007 ceph-mon[37158]: 3.0 scrub starts 2024-02-05T07:43:27.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:26 smithi007 ceph-mon[37158]: 3.0 scrub ok 2024-02-05T07:43:27.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:26 smithi090 ceph-mon[33213]: pgmap v1390: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:27.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:26 smithi090 ceph-mon[33213]: 1.e scrub starts 2024-02-05T07:43:27.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:26 smithi090 ceph-mon[33213]: 1.e scrub ok 2024-02-05T07:43:27.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:26 smithi090 ceph-mon[33213]: 3.0 scrub starts 2024-02-05T07:43:27.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:26 smithi090 ceph-mon[33213]: 3.0 scrub ok 2024-02-05T07:43:27.282 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:43:27.283 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:43:27.316 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659443 got 176093659446 for osd.5 2024-02-05T07:43:27.316 DEBUG:teuthology.parallel:result is None 2024-02-05T07:43:27.687 INFO:teuthology.orchestra.run.smithi007.stdout:257698038060 2024-02-05T07:43:27.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:27 smithi007 ceph-mon[29098]: 4.8 deep-scrub starts 2024-02-05T07:43:27.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:27 smithi007 ceph-mon[29098]: 4.8 deep-scrub ok 2024-02-05T07:43:27.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:27 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2383793496' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T07:43:27.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:27 smithi007 ceph-mon[37158]: 4.8 deep-scrub starts 2024-02-05T07:43:27.883 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:27 smithi007 ceph-mon[37158]: 4.8 deep-scrub ok 2024-02-05T07:43:27.883 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:27 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2383793496' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T07:43:28.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:27 smithi090 ceph-mon[33213]: 4.8 deep-scrub starts 2024-02-05T07:43:28.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:27 smithi090 ceph-mon[33213]: 4.8 deep-scrub ok 2024-02-05T07:43:28.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:27 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2383793496' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T07:43:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:28 smithi007 ceph-mon[29098]: pgmap v1391: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:28 smithi007 ceph-mon[29098]: 5.1a scrub starts 2024-02-05T07:43:28.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:28 smithi007 ceph-mon[29098]: 5.1a scrub ok 2024-02-05T07:43:28.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:28 smithi007 ceph-mon[37158]: pgmap v1391: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:28 smithi007 ceph-mon[37158]: 5.1a scrub starts 2024-02-05T07:43:28.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:28 smithi007 ceph-mon[37158]: 5.1a scrub ok 2024-02-05T07:43:29.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:28 smithi090 ceph-mon[33213]: pgmap v1391: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:29.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:28 smithi090 ceph-mon[33213]: 5.1a scrub starts 2024-02-05T07:43:29.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:28 smithi090 ceph-mon[33213]: 5.1a scrub ok 2024-02-05T07:43:29.514 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698038057 got 257698038060 for osd.7 2024-02-05T07:43:29.514 DEBUG:teuthology.parallel:result is None 2024-02-05T07:43:30.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:29 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:43:30.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:29 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:43:30.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:29 smithi007 ceph-mon[29098]: osdmap e755: 8 total, 8 up, 8 in 2024-02-05T07:43:30.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:29 smithi007 ceph-mon[29098]: osdmap e756: 8 total, 8 up, 8 in 2024-02-05T07:43:30.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:29 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:43:30.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:29 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:43:30.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:29 smithi007 ceph-mon[37158]: osdmap e755: 8 total, 8 up, 8 in 2024-02-05T07:43:30.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:29 smithi007 ceph-mon[37158]: osdmap e756: 8 total, 8 up, 8 in 2024-02-05T07:43:30.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:29 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:43:30.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:29 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:43:30.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:29 smithi090 ceph-mon[33213]: osdmap e755: 8 total, 8 up, 8 in 2024-02-05T07:43:30.175 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:29 smithi090 ceph-mon[33213]: osdmap e756: 8 total, 8 up, 8 in 2024-02-05T07:43:30.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:30 smithi007 ceph-mon[29098]: pgmap v1393: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:30.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:30 smithi007 ceph-mon[37158]: pgmap v1393: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:31.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:30 smithi090 ceph-mon[33213]: pgmap v1393: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:31.350 INFO:teuthology.orchestra.run.smithi007.stdout:137438953788 2024-02-05T07:43:31.653 INFO:teuthology.orchestra.run.smithi007.stdout:111669150017 2024-02-05T07:43:31.773 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:43:31 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:43:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:31 smithi007 ceph-mon[29098]: 4.7 scrub starts 2024-02-05T07:43:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:31 smithi007 ceph-mon[29098]: 4.7 scrub ok 2024-02-05T07:43:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:31 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/3155931927' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T07:43:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:31 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:43:31.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:31 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/2302912555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T07:43:31.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:31 smithi007 ceph-mon[37158]: 4.7 scrub starts 2024-02-05T07:43:31.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:31 smithi007 ceph-mon[37158]: 4.7 scrub ok 2024-02-05T07:43:31.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:31 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/3155931927' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T07:43:31.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:31 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:43:31.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:31 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/2302912555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T07:43:32.174 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953785 got 137438953788 for osd.4 2024-02-05T07:43:32.174 DEBUG:teuthology.parallel:result is None 2024-02-05T07:43:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:31 smithi090 ceph-mon[33213]: 4.7 scrub starts 2024-02-05T07:43:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:31 smithi090 ceph-mon[33213]: 4.7 scrub ok 2024-02-05T07:43:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:31 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/3155931927' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T07:43:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:31 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T07:43:32.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:31 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/2302912555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T07:43:32.585 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669150014 got 111669150017 for osd.3 2024-02-05T07:43:32.585 DEBUG:teuthology.parallel:result is None 2024-02-05T07:43:32.585 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-05T07:43:32.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:43:32.834 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:43:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[29098]: 3.19 scrub starts 2024-02-05T07:43:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[29098]: 3.19 scrub ok 2024-02-05T07:43:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[29098]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:43:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[29098]: pgmap v1395: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[29098]: 6.8 deep-scrub starts 2024-02-05T07:43:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[29098]: 6.8 deep-scrub ok 2024-02-05T07:43:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[29098]: 6.1d scrub starts 2024-02-05T07:43:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[29098]: 5.7 scrub starts 2024-02-05T07:43:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[29098]: 5.7 scrub ok 2024-02-05T07:43:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[29098]: 5.11 scrub starts 2024-02-05T07:43:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[29098]: 5.11 scrub ok 2024-02-05T07:43:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[37158]: 3.19 scrub starts 2024-02-05T07:43:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[37158]: 3.19 scrub ok 2024-02-05T07:43:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[37158]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:43:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[37158]: pgmap v1395: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[37158]: 6.8 deep-scrub starts 2024-02-05T07:43:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[37158]: 6.8 deep-scrub ok 2024-02-05T07:43:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[37158]: 6.1d scrub starts 2024-02-05T07:43:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[37158]: 5.7 scrub starts 2024-02-05T07:43:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[37158]: 5.7 scrub ok 2024-02-05T07:43:32.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[37158]: 5.11 scrub starts 2024-02-05T07:43:32.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:32 smithi007 ceph-mon[37158]: 5.11 scrub ok 2024-02-05T07:43:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:32 smithi090 ceph-mon[33213]: 3.19 scrub starts 2024-02-05T07:43:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:32 smithi090 ceph-mon[33213]: 3.19 scrub ok 2024-02-05T07:43:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:32 smithi090 ceph-mon[33213]: from='client.24431 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T07:43:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:32 smithi090 ceph-mon[33213]: pgmap v1395: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:32 smithi090 ceph-mon[33213]: 6.8 deep-scrub starts 2024-02-05T07:43:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:32 smithi090 ceph-mon[33213]: 6.8 deep-scrub ok 2024-02-05T07:43:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:32 smithi090 ceph-mon[33213]: 6.1d scrub starts 2024-02-05T07:43:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:32 smithi090 ceph-mon[33213]: 5.7 scrub starts 2024-02-05T07:43:33.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:32 smithi090 ceph-mon[33213]: 5.7 scrub ok 2024-02-05T07:43:33.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:32 smithi090 ceph-mon[33213]: 5.11 scrub starts 2024-02-05T07:43:33.275 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:32 smithi090 ceph-mon[33213]: 5.11 scrub ok 2024-02-05T07:43:33.819 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:33 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:43:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:43:34.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:33 smithi007 ceph-mon[29098]: 6.1d scrub ok 2024-02-05T07:43:34.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:33 smithi007 ceph-mon[37158]: 6.1d scrub ok 2024-02-05T07:43:34.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:33 smithi090 ceph-mon[33213]: 6.1d scrub ok 2024-02-05T07:43:34.790 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:43:34.793 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:43:35.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:34 smithi007 ceph-mon[29098]: pgmap v1396: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:35.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:34 smithi007 ceph-mon[29098]: 1.4 scrub starts 2024-02-05T07:43:35.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:34 smithi007 ceph-mon[29098]: 1.4 scrub ok 2024-02-05T07:43:35.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:34 smithi007 ceph-mon[29098]: 1.d scrub starts 2024-02-05T07:43:35.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:34 smithi007 ceph-mon[29098]: 1.d scrub ok 2024-02-05T07:43:35.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:34 smithi007 ceph-mon[29098]: 1.1 scrub starts 2024-02-05T07:43:35.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:34 smithi007 ceph-mon[29098]: 1.1 scrub ok 2024-02-05T07:43:35.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:34 smithi007 ceph-mon[37158]: pgmap v1396: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:35.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:34 smithi007 ceph-mon[37158]: 1.4 scrub starts 2024-02-05T07:43:35.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:34 smithi007 ceph-mon[37158]: 1.4 scrub ok 2024-02-05T07:43:35.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:34 smithi007 ceph-mon[37158]: 1.d scrub starts 2024-02-05T07:43:35.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:34 smithi007 ceph-mon[37158]: 1.d scrub ok 2024-02-05T07:43:35.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:34 smithi007 ceph-mon[37158]: 1.1 scrub starts 2024-02-05T07:43:35.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:34 smithi007 ceph-mon[37158]: 1.1 scrub ok 2024-02-05T07:43:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:34 smithi090 ceph-mon[33213]: pgmap v1396: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:34 smithi090 ceph-mon[33213]: 1.4 scrub starts 2024-02-05T07:43:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:34 smithi090 ceph-mon[33213]: 1.4 scrub ok 2024-02-05T07:43:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:34 smithi090 ceph-mon[33213]: 1.d scrub starts 2024-02-05T07:43:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:34 smithi090 ceph-mon[33213]: 1.d scrub ok 2024-02-05T07:43:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:34 smithi090 ceph-mon[33213]: 1.1 scrub starts 2024-02-05T07:43:35.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:34 smithi090 ceph-mon[33213]: 1.1 scrub ok 2024-02-05T07:43:35.610 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1396,"stamp":"2024-02-05T07:43:33.488536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2268,"num_read_kb":2020,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":669,"num_bytes_recovered":542611,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1215,"ondisk_log_size":1215,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":456,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6981604,"kb_used_data":35840,"kb_used_omap":16,"kb_used_meta":6945647,"kb_avail":742979612,"statfs":{"total":767960285184,"available":760811122688,"internally_reserved":0,"allocated":36700160,"data_stored":21972154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16785,"internal_metadata":7112343151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002405"},"pg_stats":[{"pgid":"1.9","version":"86'128","reported_seq":1297,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897843+0000","last_change":"2024-02-05T07:43:22.748182+0000","last_active":"2024-02-05T07:43:29.897843+0000","last_peered":"2024-02-05T07:43:29.897843+0000","last_clean":"2024-02-05T07:43:29.897843+0000","last_became_active":"2024-02-05T07:41:48.892873+0000","last_became_peered":"2024-02-05T07:41:48.892873+0000","last_unstale":"2024-02-05T07:43:29.897843+0000","last_undegraded":"2024-02-05T07:43:29.897843+0000","last_fullsized":"2024-02-05T07:43:29.897843+0000","mapping_epoch":706,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":707,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:22.748084+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:43:22.748084+0000","last_clean_scrub_stamp":"2024-02-05T07:43:22.748084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:51.823285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"615'35","reported_seq":127,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913793+0000","last_change":"2024-02-05T07:43:22.879048+0000","last_active":"2024-02-05T07:43:29.913793+0000","last_peered":"2024-02-05T07:43:29.913793+0000","last_clean":"2024-02-05T07:43:29.913793+0000","last_became_active":"2024-02-05T07:41:53.755648+0000","last_became_peered":"2024-02-05T07:41:53.755648+0000","last_unstale":"2024-02-05T07:43:29.913793+0000","last_undegraded":"2024-02-05T07:43:29.913793+0000","last_fullsized":"2024-02-05T07:43:29.913793+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'35","last_scrub_stamp":"2024-02-05T07:43:22.878910+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:43:22.878910+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:26.132943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1038,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.686001+0000","last_change":"2024-02-05T07:42:26.591966+0000","last_active":"2024-02-05T07:43:30.686001+0000","last_peered":"2024-02-05T07:43:30.686001+0000","last_clean":"2024-02-05T07:43:30.686001+0000","last_became_active":"2024-02-05T07:41:53.752202+0000","last_became_peered":"2024-02-05T07:41:53.752202+0000","last_unstale":"2024-02-05T07:43:30.686001+0000","last_undegraded":"2024-02-05T07:43:30.686001+0000","last_fullsized":"2024-02-05T07:43:30.686001+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:26.591880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:42:26.591880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:53.456483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"708'15","reported_seq":579,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.984454+0000","last_change":"2024-02-05T07:42:27.151867+0000","last_active":"2024-02-05T07:43:29.984454+0000","last_peered":"2024-02-05T07:43:29.984454+0000","last_clean":"2024-02-05T07:43:29.984454+0000","last_became_active":"2024-02-05T07:41:49.759014+0000","last_became_peered":"2024-02-05T07:41:49.759014+0000","last_unstale":"2024-02-05T07:43:29.984454+0000","last_undegraded":"2024-02-05T07:43:29.984454+0000","last_fullsized":"2024-02-05T07:43:29.984454+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"708'15","last_scrub_stamp":"2024-02-05T07:42:27.151770+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:42:27.151770+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:33.374275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913157+0000","last_change":"2024-02-05T07:42:42.170041+0000","last_active":"2024-02-05T07:43:29.913157+0000","last_peered":"2024-02-05T07:43:29.913157+0000","last_clean":"2024-02-05T07:43:29.913157+0000","last_became_active":"2024-02-05T07:41:53.747848+0000","last_became_peered":"2024-02-05T07:41:53.747848+0000","last_unstale":"2024-02-05T07:43:29.913157+0000","last_undegraded":"2024-02-05T07:43:29.913157+0000","last_fullsized":"2024-02-05T07:43:29.913157+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:42.169959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:42:42.169959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:47.324283+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":934,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743996+0000","last_change":"2024-02-05T07:42:54.112250+0000","last_active":"2024-02-05T07:43:29.743996+0000","last_peered":"2024-02-05T07:43:29.743996+0000","last_clean":"2024-02-05T07:43:29.743996+0000","last_became_active":"2024-02-05T07:42:36.737646+0000","last_became_peered":"2024-02-05T07:42:36.737646+0000","last_unstale":"2024-02-05T07:43:29.743996+0000","last_undegraded":"2024-02-05T07:43:29.743996+0000","last_fullsized":"2024-02-05T07:43:29.743996+0000","mapping_epoch":741,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:54.112157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:54.112157+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":138,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:11.718443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"616'45","reported_seq":133,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912856+0000","last_change":"2024-02-05T07:42:22.221425+0000","last_active":"2024-02-05T07:43:29.912856+0000","last_peered":"2024-02-05T07:43:29.912856+0000","last_clean":"2024-02-05T07:43:29.912856+0000","last_became_active":"2024-02-05T07:41:53.748344+0000","last_became_peered":"2024-02-05T07:41:53.748344+0000","last_unstale":"2024-02-05T07:43:29.912856+0000","last_undegraded":"2024-02-05T07:43:29.912856+0000","last_fullsized":"2024-02-05T07:43:29.912856+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'45","last_scrub_stamp":"2024-02-05T07:42:22.221306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:42:22.221306+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:44.464889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1313,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-02-05T07:43:22.223653+0000","last_change":"2024-02-05T07:42:32.638969+0000","last_active":"2024-02-05T07:43:22.223653+0000","last_peered":"2024-02-05T07:43:22.223653+0000","last_clean":"2024-02-05T07:43:22.223653+0000","last_became_active":"2024-02-05T07:41:53.744933+0000","last_became_peered":"2024-02-05T07:41:53.744933+0000","last_unstale":"2024-02-05T07:43:22.223653+0000","last_undegraded":"2024-02-05T07:43:22.223653+0000","last_fullsized":"2024-02-05T07:43:22.223653+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:32.638874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:32.638874+0000","last_clean_scrub_stamp":"2024-02-05T07:42:32.638874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:45.225778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":878,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742214+0000","last_change":"2024-02-05T07:42:22.148309+0000","last_active":"2024-02-05T07:43:29.742214+0000","last_peered":"2024-02-05T07:43:29.742214+0000","last_clean":"2024-02-05T07:43:29.742214+0000","last_became_active":"2024-02-05T07:42:11.957630+0000","last_became_peered":"2024-02-05T07:42:11.957630+0000","last_unstale":"2024-02-05T07:43:29.742214+0000","last_undegraded":"2024-02-05T07:43:29.742214+0000","last_fullsized":"2024-02-05T07:43:29.742214+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:22.148187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:42:22.148187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:45.099164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":482,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.984745+0000","last_change":"2024-02-05T07:42:25.428561+0000","last_active":"2024-02-05T07:43:29.984745+0000","last_peered":"2024-02-05T07:43:29.984745+0000","last_clean":"2024-02-05T07:43:29.984745+0000","last_became_active":"2024-02-05T07:41:53.747418+0000","last_became_peered":"2024-02-05T07:41:53.747418+0000","last_unstale":"2024-02-05T07:43:29.984745+0000","last_undegraded":"2024-02-05T07:43:29.984745+0000","last_fullsized":"2024-02-05T07:43:29.984745+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:25.428462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:25.428462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:35.307176+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":888,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743260+0000","last_change":"2024-02-05T07:42:13.115144+0000","last_active":"2024-02-05T07:43:29.743260+0000","last_peered":"2024-02-05T07:43:29.743260+0000","last_clean":"2024-02-05T07:43:29.743260+0000","last_became_active":"2024-02-05T07:42:11.959147+0000","last_became_peered":"2024-02-05T07:42:11.959147+0000","last_unstale":"2024-02-05T07:43:29.743260+0000","last_undegraded":"2024-02-05T07:43:29.743260+0000","last_fullsized":"2024-02-05T07:43:29.743260+0000","mapping_epoch":723,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:13.114996+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:42:13.114996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:34.990635+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"616'32","reported_seq":119,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912837+0000","last_change":"2024-02-05T07:42:36.129167+0000","last_active":"2024-02-05T07:43:29.912837+0000","last_peered":"2024-02-05T07:43:29.912837+0000","last_clean":"2024-02-05T07:43:29.912837+0000","last_became_active":"2024-02-05T07:41:53.755063+0000","last_became_peered":"2024-02-05T07:41:53.755063+0000","last_unstale":"2024-02-05T07:43:29.912837+0000","last_undegraded":"2024-02-05T07:43:29.912837+0000","last_fullsized":"2024-02-05T07:43:29.912837+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'32","last_scrub_stamp":"2024-02-05T07:42:36.129021+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:42:36.129021+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:03.237577+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913186+0000","last_change":"2024-02-05T07:42:54.971153+0000","last_active":"2024-02-05T07:43:29.913186+0000","last_peered":"2024-02-05T07:43:29.913186+0000","last_clean":"2024-02-05T07:43:29.913186+0000","last_became_active":"2024-02-05T07:41:53.747999+0000","last_became_peered":"2024-02-05T07:41:53.747999+0000","last_unstale":"2024-02-05T07:43:29.913186+0000","last_undegraded":"2024-02-05T07:43:29.913186+0000","last_fullsized":"2024-02-05T07:43:29.913186+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:54.971057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:42:54.971057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:13.720219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1127,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.745076+0000","last_change":"2024-02-05T07:42:38.151143+0000","last_active":"2024-02-05T07:43:29.745076+0000","last_peered":"2024-02-05T07:43:29.745076+0000","last_clean":"2024-02-05T07:43:29.745076+0000","last_became_active":"2024-02-05T07:42:36.743340+0000","last_became_peered":"2024-02-05T07:42:36.743340+0000","last_unstale":"2024-02-05T07:43:29.745076+0000","last_undegraded":"2024-02-05T07:43:29.745076+0000","last_fullsized":"2024-02-05T07:43:29.745076+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:38.151005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:38.151005+0000","last_clean_scrub_stamp":"2024-02-05T07:42:38.151005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:02.208451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1047,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.686835+0000","last_change":"2024-02-05T07:42:10.547377+0000","last_active":"2024-02-05T07:43:30.686835+0000","last_peered":"2024-02-05T07:43:30.686835+0000","last_clean":"2024-02-05T07:43:30.686835+0000","last_became_active":"2024-02-05T07:41:49.749351+0000","last_became_peered":"2024-02-05T07:41:49.749351+0000","last_unstale":"2024-02-05T07:43:30.686835+0000","last_undegraded":"2024-02-05T07:43:30.686835+0000","last_fullsized":"2024-02-05T07:43:30.686835+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:10.547297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:10.547297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:37.447504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":883,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744490+0000","last_change":"2024-02-05T07:42:39.115312+0000","last_active":"2024-02-05T07:43:29.744490+0000","last_peered":"2024-02-05T07:43:29.744490+0000","last_clean":"2024-02-05T07:43:29.744490+0000","last_became_active":"2024-02-05T07:42:38.753738+0000","last_became_peered":"2024-02-05T07:42:38.753738+0000","last_unstale":"2024-02-05T07:43:29.744490+0000","last_undegraded":"2024-02-05T07:43:29.744490+0000","last_fullsized":"2024-02-05T07:43:29.744490+0000","mapping_epoch":743,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":744,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:39.115200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:39.115200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:03.322489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"616'46","reported_seq":1635,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743221+0000","last_change":"2024-02-05T07:43:02.155276+0000","last_active":"2024-02-05T07:43:29.743221+0000","last_peered":"2024-02-05T07:43:29.743221+0000","last_clean":"2024-02-05T07:43:29.743221+0000","last_became_active":"2024-02-05T07:41:53.749695+0000","last_became_peered":"2024-02-05T07:41:53.749695+0000","last_unstale":"2024-02-05T07:43:29.743221+0000","last_undegraded":"2024-02-05T07:43:29.743221+0000","last_fullsized":"2024-02-05T07:43:29.743221+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'46","last_scrub_stamp":"2024-02-05T07:43:02.155154+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:43:02.155154+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:13.933013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1052,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.685806+0000","last_change":"2024-02-05T07:43:09.511558+0000","last_active":"2024-02-05T07:43:30.685806+0000","last_peered":"2024-02-05T07:43:30.685806+0000","last_clean":"2024-02-05T07:43:30.685806+0000","last_became_active":"2024-02-05T07:41:45.938795+0000","last_became_peered":"2024-02-05T07:41:45.938795+0000","last_unstale":"2024-02-05T07:43:30.685806+0000","last_undegraded":"2024-02-05T07:43:30.685806+0000","last_fullsized":"2024-02-05T07:43:30.685806+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:09.511469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:43:09.511469+0000","last_clean_scrub_stamp":"2024-02-05T07:43:09.511469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:10.937664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"713'14","reported_seq":131,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912967+0000","last_change":"2024-02-05T07:43:06.868299+0000","last_active":"2024-02-05T07:43:29.912967+0000","last_peered":"2024-02-05T07:43:29.912967+0000","last_clean":"2024-02-05T07:43:29.912967+0000","last_became_active":"2024-02-05T07:41:53.759990+0000","last_became_peered":"2024-02-05T07:41:53.759990+0000","last_unstale":"2024-02-05T07:43:29.912967+0000","last_undegraded":"2024-02-05T07:43:29.912967+0000","last_fullsized":"2024-02-05T07:43:29.912967+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'14","last_scrub_stamp":"2024-02-05T07:43:06.868200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:06.868200+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:29.648510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":115,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.745071+0000","last_change":"2024-02-05T07:42:13.176258+0000","last_active":"2024-02-05T07:43:29.745071+0000","last_peered":"2024-02-05T07:43:29.745071+0000","last_clean":"2024-02-05T07:43:29.745071+0000","last_became_active":"2024-02-05T07:41:49.742477+0000","last_became_peered":"2024-02-05T07:41:49.742477+0000","last_unstale":"2024-02-05T07:43:29.745071+0000","last_undegraded":"2024-02-05T07:43:29.745071+0000","last_fullsized":"2024-02-05T07:43:29.745071+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:13.176166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:13.176166+0000","last_clean_scrub_stamp":"2024-02-05T07:42:13.176166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:34.406372+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":1183,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.914126+0000","last_change":"2024-02-05T07:43:15.807217+0000","last_active":"2024-02-05T07:43:29.914126+0000","last_peered":"2024-02-05T07:43:29.914126+0000","last_clean":"2024-02-05T07:43:29.914126+0000","last_became_active":"2024-02-05T07:41:56.731867+0000","last_became_peered":"2024-02-05T07:41:56.731867+0000","last_unstale":"2024-02-05T07:43:29.914126+0000","last_undegraded":"2024-02-05T07:43:29.914126+0000","last_fullsized":"2024-02-05T07:43:29.914126+0000","mapping_epoch":714,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:15.807100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:43:15.807100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:34.256250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":115,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913352+0000","last_change":"2024-02-05T07:43:24.795918+0000","last_active":"2024-02-05T07:43:29.913352+0000","last_peered":"2024-02-05T07:43:29.913352+0000","last_clean":"2024-02-05T07:43:29.913352+0000","last_became_active":"2024-02-05T07:41:53.753278+0000","last_became_peered":"2024-02-05T07:41:53.753278+0000","last_unstale":"2024-02-05T07:43:29.913352+0000","last_undegraded":"2024-02-05T07:43:29.913352+0000","last_fullsized":"2024-02-05T07:43:29.913352+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:24.795825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:14.198146+0000","last_clean_scrub_stamp":"2024-02-05T07:43:24.795825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:48.247803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"615'50","reported_seq":1145,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.685990+0000","last_change":"2024-02-05T07:42:47.698407+0000","last_active":"2024-02-05T07:43:30.685990+0000","last_peered":"2024-02-05T07:43:30.685990+0000","last_clean":"2024-02-05T07:43:30.685990+0000","last_became_active":"2024-02-05T07:41:45.939280+0000","last_became_peered":"2024-02-05T07:41:45.939280+0000","last_unstale":"2024-02-05T07:43:30.685990+0000","last_undegraded":"2024-02-05T07:43:30.685990+0000","last_fullsized":"2024-02-05T07:43:30.685990+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'50","last_scrub_stamp":"2024-02-05T07:42:47.698301+0000","last_deep_scrub":"454'38","last_deep_scrub_stamp":"2024-02-05T07:34:38.942830+0000","last_clean_scrub_stamp":"2024-02-05T07:42:47.698301+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:07.731167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1044,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.686210+0000","last_change":"2024-02-05T07:42:44.718432+0000","last_active":"2024-02-05T07:43:30.686210+0000","last_peered":"2024-02-05T07:43:30.686210+0000","last_clean":"2024-02-05T07:43:30.686210+0000","last_became_active":"2024-02-05T07:41:45.943736+0000","last_became_peered":"2024-02-05T07:41:45.943736+0000","last_unstale":"2024-02-05T07:43:30.686210+0000","last_undegraded":"2024-02-05T07:43:30.686210+0000","last_fullsized":"2024-02-05T07:43:30.686210+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:44.718337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:44.718337+0000","last_clean_scrub_stamp":"2024-02-05T07:42:44.718337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:46.006994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913558+0000","last_change":"2024-02-05T07:42:34.098540+0000","last_active":"2024-02-05T07:43:29.913558+0000","last_peered":"2024-02-05T07:43:29.913558+0000","last_clean":"2024-02-05T07:43:29.913558+0000","last_became_active":"2024-02-05T07:41:53.749709+0000","last_became_peered":"2024-02-05T07:41:53.749709+0000","last_unstale":"2024-02-05T07:43:29.913558+0000","last_undegraded":"2024-02-05T07:43:29.913558+0000","last_fullsized":"2024-02-05T07:43:29.913558+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:34.098427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:42:34.098427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:57.922534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":1829,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.686966+0000","last_change":"2024-02-05T07:42:01.624851+0000","last_active":"2024-02-05T07:43:30.686966+0000","last_peered":"2024-02-05T07:43:30.686966+0000","last_clean":"2024-02-05T07:43:30.686966+0000","last_became_active":"2024-02-05T07:41:52.748451+0000","last_became_peered":"2024-02-05T07:41:52.748451+0000","last_unstale":"2024-02-05T07:43:30.686966+0000","last_undegraded":"2024-02-05T07:43:30.686966+0000","last_fullsized":"2024-02-05T07:43:30.686966+0000","mapping_epoch":710,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:01.624759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:01.624759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"713'12","reported_seq":1247,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.686554+0000","last_change":"2024-02-05T07:42:19.555579+0000","last_active":"2024-02-05T07:43:30.686554+0000","last_peered":"2024-02-05T07:43:30.686554+0000","last_clean":"2024-02-05T07:43:30.686554+0000","last_became_active":"2024-02-05T07:41:53.737961+0000","last_became_peered":"2024-02-05T07:41:53.737961+0000","last_unstale":"2024-02-05T07:43:30.686554+0000","last_undegraded":"2024-02-05T07:43:30.686554+0000","last_fullsized":"2024-02-05T07:43:30.686554+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'12","last_scrub_stamp":"2024-02-05T07:42:19.555495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:42:19.555495+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:40.583073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"616'49","reported_seq":123,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912685+0000","last_change":"2024-02-05T07:42:53.016225+0000","last_active":"2024-02-05T07:43:29.912685+0000","last_peered":"2024-02-05T07:43:29.912685+0000","last_clean":"2024-02-05T07:43:29.912685+0000","last_became_active":"2024-02-05T07:41:53.758571+0000","last_became_peered":"2024-02-05T07:41:53.758571+0000","last_unstale":"2024-02-05T07:43:29.912685+0000","last_undegraded":"2024-02-05T07:43:29.912685+0000","last_fullsized":"2024-02-05T07:43:29.912685+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'49","last_scrub_stamp":"2024-02-05T07:42:53.016112+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:42:53.016112+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:53.238724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"639'2","reported_seq":1068,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744238+0000","last_change":"2024-02-05T07:42:31.380189+0000","last_active":"2024-02-05T07:43:29.744238+0000","last_peered":"2024-02-05T07:43:29.744238+0000","last_clean":"2024-02-05T07:43:29.744238+0000","last_became_active":"2024-02-05T07:42:05.923781+0000","last_became_peered":"2024-02-05T07:42:05.923781+0000","last_unstale":"2024-02-05T07:43:29.744238+0000","last_undegraded":"2024-02-05T07:43:29.744238+0000","last_fullsized":"2024-02-05T07:43:29.744238+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'2","last_scrub_stamp":"2024-02-05T07:42:31.380092+0000","last_deep_scrub":"639'2","last_deep_scrub_stamp":"2024-02-05T07:40:13.213477+0000","last_clean_scrub_stamp":"2024-02-05T07:42:31.380092+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:38.380712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":123,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897470+0000","last_change":"2024-02-05T07:41:55.764320+0000","last_active":"2024-02-05T07:43:29.897470+0000","last_peered":"2024-02-05T07:43:29.897470+0000","last_clean":"2024-02-05T07:43:29.897470+0000","last_became_active":"2024-02-05T07:41:45.860439+0000","last_became_peered":"2024-02-05T07:41:45.860439+0000","last_unstale":"2024-02-05T07:43:29.897470+0000","last_undegraded":"2024-02-05T07:43:29.897470+0000","last_fullsized":"2024-02-05T07:43:29.897470+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:55.764170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:58.737033+0000","last_clean_scrub_stamp":"2024-02-05T07:41:55.764170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":1463,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-02-05T07:43:22.224669+0000","last_change":"2024-02-05T07:42:39.684922+0000","last_active":"2024-02-05T07:43:22.224669+0000","last_peered":"2024-02-05T07:43:22.224669+0000","last_clean":"2024-02-05T07:43:22.224669+0000","last_became_active":"2024-02-05T07:42:19.004211+0000","last_became_peered":"2024-02-05T07:42:19.004211+0000","last_unstale":"2024-02-05T07:43:22.224669+0000","last_undegraded":"2024-02-05T07:43:22.224669+0000","last_fullsized":"2024-02-05T07:43:22.224669+0000","mapping_epoch":729,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:39.684799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:39.684799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:41.196649+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"722'17","reported_seq":431,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743693+0000","last_change":"2024-02-05T07:43:03.674585+0000","last_active":"2024-02-05T07:43:29.743693+0000","last_peered":"2024-02-05T07:43:29.743693+0000","last_clean":"2024-02-05T07:43:29.743693+0000","last_became_active":"2024-02-05T07:42:05.931950+0000","last_became_peered":"2024-02-05T07:42:05.931950+0000","last_unstale":"2024-02-05T07:43:29.743693+0000","last_undegraded":"2024-02-05T07:43:29.743693+0000","last_fullsized":"2024-02-05T07:43:29.743693+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'17","last_scrub_stamp":"2024-02-05T07:43:03.674431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:03.674431+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:29.089732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"616'35","reported_seq":63,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.983965+0000","last_change":"2024-02-05T07:42:58.931836+0000","last_active":"2024-02-05T07:43:29.983965+0000","last_peered":"2024-02-05T07:43:29.983965+0000","last_clean":"2024-02-05T07:43:29.983965+0000","last_became_active":"2024-02-05T07:42:32.202250+0000","last_became_peered":"2024-02-05T07:42:32.202250+0000","last_unstale":"2024-02-05T07:43:29.983965+0000","last_undegraded":"2024-02-05T07:43:29.983965+0000","last_fullsized":"2024-02-05T07:43:29.983965+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'35","last_scrub_stamp":"2024-02-05T07:42:58.931730+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:42:58.931730+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:28.186531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":115,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744947+0000","last_change":"2024-02-05T07:42:56.041305+0000","last_active":"2024-02-05T07:43:29.744947+0000","last_peered":"2024-02-05T07:43:29.744947+0000","last_clean":"2024-02-05T07:43:29.744947+0000","last_became_active":"2024-02-05T07:41:49.764275+0000","last_became_peered":"2024-02-05T07:41:49.764275+0000","last_unstale":"2024-02-05T07:43:29.744947+0000","last_undegraded":"2024-02-05T07:43:29.744947+0000","last_fullsized":"2024-02-05T07:43:29.744947+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:56.041211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:42:56.041211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:23.083246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":742,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.741927+0000","last_change":"2024-02-05T07:43:07.136513+0000","last_active":"2024-02-05T07:43:29.741927+0000","last_peered":"2024-02-05T07:43:29.741927+0000","last_clean":"2024-02-05T07:43:29.741927+0000","last_became_active":"2024-02-05T07:32:58.598709+0000","last_became_peered":"2024-02-05T07:32:58.598709+0000","last_unstale":"2024-02-05T07:43:29.741927+0000","last_undegraded":"2024-02-05T07:43:29.741927+0000","last_fullsized":"2024-02-05T07:43:29.741927+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:07.136433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:43:07.136433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:32.604100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":1416,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-02-05T07:43:22.224070+0000","last_change":"2024-02-05T07:42:30.657385+0000","last_active":"2024-02-05T07:43:22.224070+0000","last_peered":"2024-02-05T07:43:22.224070+0000","last_clean":"2024-02-05T07:43:22.224070+0000","last_became_active":"2024-02-05T07:40:52.907985+0000","last_became_peered":"2024-02-05T07:40:52.907985+0000","last_unstale":"2024-02-05T07:43:22.224070+0000","last_undegraded":"2024-02-05T07:43:22.224070+0000","last_fullsized":"2024-02-05T07:43:22.224070+0000","mapping_epoch":302,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:30.657284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:30.657284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:51.264922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1315,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-02-05T07:43:22.223486+0000","last_change":"2024-02-05T07:42:50.752034+0000","last_active":"2024-02-05T07:43:22.223486+0000","last_peered":"2024-02-05T07:43:22.223486+0000","last_clean":"2024-02-05T07:43:22.223486+0000","last_became_active":"2024-02-05T07:41:49.735919+0000","last_became_peered":"2024-02-05T07:41:49.735919+0000","last_unstale":"2024-02-05T07:43:22.223486+0000","last_undegraded":"2024-02-05T07:43:22.223486+0000","last_fullsized":"2024-02-05T07:43:22.223486+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:50.751946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:42:50.751946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:03.358026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"615'39","reported_seq":940,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743729+0000","last_change":"2024-02-05T07:42:46.404152+0000","last_active":"2024-02-05T07:43:29.743729+0000","last_peered":"2024-02-05T07:43:29.743729+0000","last_clean":"2024-02-05T07:43:29.743729+0000","last_became_active":"2024-02-05T07:42:05.921622+0000","last_became_peered":"2024-02-05T07:42:05.921622+0000","last_unstale":"2024-02-05T07:43:29.743729+0000","last_undegraded":"2024-02-05T07:43:29.743729+0000","last_fullsized":"2024-02-05T07:43:29.743729+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'39","last_scrub_stamp":"2024-02-05T07:42:46.404018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:42:46.404018+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:46.499030+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":66,"num_read_kb":52,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":893,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742912+0000","last_change":"2024-02-05T07:42:26.164849+0000","last_active":"2024-02-05T07:43:29.742912+0000","last_peered":"2024-02-05T07:43:29.742912+0000","last_clean":"2024-02-05T07:43:29.742912+0000","last_became_active":"2024-02-05T07:42:11.957822+0000","last_became_peered":"2024-02-05T07:42:11.957822+0000","last_unstale":"2024-02-05T07:43:29.742912+0000","last_undegraded":"2024-02-05T07:43:29.742912+0000","last_fullsized":"2024-02-05T07:43:29.742912+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:26.164774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:42:26.164774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:55.863395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1471,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743029+0000","last_change":"2024-02-05T07:43:19.199842+0000","last_active":"2024-02-05T07:43:29.743029+0000","last_peered":"2024-02-05T07:43:29.743029+0000","last_clean":"2024-02-05T07:43:29.743029+0000","last_became_active":"2024-02-05T07:36:32.709872+0000","last_became_peered":"2024-02-05T07:36:32.709872+0000","last_unstale":"2024-02-05T07:43:29.743029+0000","last_undegraded":"2024-02-05T07:43:29.743029+0000","last_fullsized":"2024-02-05T07:43:29.743029+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:19.199710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:43:19.199710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:32.763255+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":880,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743208+0000","last_change":"2024-02-05T07:43:21.092918+0000","last_active":"2024-02-05T07:43:29.743208+0000","last_peered":"2024-02-05T07:43:29.743208+0000","last_clean":"2024-02-05T07:43:29.743208+0000","last_became_active":"2024-02-05T07:42:11.957957+0000","last_became_peered":"2024-02-05T07:42:11.957957+0000","last_unstale":"2024-02-05T07:43:29.743208+0000","last_undegraded":"2024-02-05T07:43:29.743208+0000","last_fullsized":"2024-02-05T07:43:29.743208+0000","mapping_epoch":723,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:21.092817+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:43:21.092817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:29.874000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":115,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.745169+0000","last_change":"2024-02-05T07:42:06.098522+0000","last_active":"2024-02-05T07:43:29.745169+0000","last_peered":"2024-02-05T07:43:29.745169+0000","last_clean":"2024-02-05T07:43:29.745169+0000","last_became_active":"2024-02-05T07:41:49.741714+0000","last_became_peered":"2024-02-05T07:41:49.741714+0000","last_unstale":"2024-02-05T07:43:29.745169+0000","last_undegraded":"2024-02-05T07:43:29.745169+0000","last_fullsized":"2024-02-05T07:43:29.745169+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:06.098389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:06.098389+0000","last_clean_scrub_stamp":"2024-02-05T07:42:06.098389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"615'36","reported_seq":300,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744099+0000","last_change":"2024-02-05T07:42:27.433721+0000","last_active":"2024-02-05T07:43:29.744099+0000","last_peered":"2024-02-05T07:43:29.744099+0000","last_clean":"2024-02-05T07:43:29.744099+0000","last_became_active":"2024-02-05T07:42:05.922131+0000","last_became_peered":"2024-02-05T07:42:05.922131+0000","last_unstale":"2024-02-05T07:43:29.744099+0000","last_undegraded":"2024-02-05T07:43:29.744099+0000","last_fullsized":"2024-02-05T07:43:29.744099+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'36","last_scrub_stamp":"2024-02-05T07:42:27.433619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:42:27.433619+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:57.072440+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":45,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"695'6","reported_seq":232,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897524+0000","last_change":"2024-02-05T07:42:46.622715+0000","last_active":"2024-02-05T07:43:29.897524+0000","last_peered":"2024-02-05T07:43:29.897524+0000","last_clean":"2024-02-05T07:43:29.897524+0000","last_became_active":"2024-02-05T07:41:45.883810+0000","last_became_peered":"2024-02-05T07:41:45.883810+0000","last_unstale":"2024-02-05T07:43:29.897524+0000","last_undegraded":"2024-02-05T07:43:29.897524+0000","last_fullsized":"2024-02-05T07:43:29.897524+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'6","last_scrub_stamp":"2024-02-05T07:42:46.622612+0000","last_deep_scrub":"695'6","last_deep_scrub_stamp":"2024-02-05T07:41:22.536369+0000","last_clean_scrub_stamp":"2024-02-05T07:42:46.622612+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:14.702968+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":750,"num_read_kb":745,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":747,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742476+0000","last_change":"2024-02-05T07:42:14.291981+0000","last_active":"2024-02-05T07:43:29.742476+0000","last_peered":"2024-02-05T07:43:29.742476+0000","last_clean":"2024-02-05T07:43:29.742476+0000","last_became_active":"2024-02-05T07:42:05.930992+0000","last_became_peered":"2024-02-05T07:42:05.930992+0000","last_unstale":"2024-02-05T07:43:29.742476+0000","last_undegraded":"2024-02-05T07:43:29.742476+0000","last_fullsized":"2024-02-05T07:43:29.742476+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:14.291888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:14.291888+0000","last_clean_scrub_stamp":"2024-02-05T07:42:14.291888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:34.231809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":1497,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742987+0000","last_change":"2024-02-05T07:43:06.107976+0000","last_active":"2024-02-05T07:43:29.742987+0000","last_peered":"2024-02-05T07:43:29.742987+0000","last_clean":"2024-02-05T07:43:29.742987+0000","last_became_active":"2024-02-05T07:41:49.749990+0000","last_became_peered":"2024-02-05T07:41:49.749990+0000","last_unstale":"2024-02-05T07:43:29.742987+0000","last_undegraded":"2024-02-05T07:43:29.742987+0000","last_fullsized":"2024-02-05T07:43:29.742987+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:43:06.107882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:43:06.107882+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:08.624572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":137,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898180+0000","last_change":"2024-02-05T07:43:17.778996+0000","last_active":"2024-02-05T07:43:29.898180+0000","last_peered":"2024-02-05T07:43:29.898180+0000","last_clean":"2024-02-05T07:43:29.898180+0000","last_became_active":"2024-02-05T07:41:49.729965+0000","last_became_peered":"2024-02-05T07:41:49.729965+0000","last_unstale":"2024-02-05T07:43:29.898180+0000","last_undegraded":"2024-02-05T07:43:29.898180+0000","last_fullsized":"2024-02-05T07:43:29.898180+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:17.778900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:17.778900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:45.044357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"713'46","reported_seq":1725,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-02-05T07:43:22.224319+0000","last_change":"2024-02-05T07:43:02.681388+0000","last_active":"2024-02-05T07:43:22.224319+0000","last_peered":"2024-02-05T07:43:22.224319+0000","last_clean":"2024-02-05T07:43:22.224319+0000","last_became_active":"2024-02-05T07:41:53.745616+0000","last_became_peered":"2024-02-05T07:41:53.745616+0000","last_unstale":"2024-02-05T07:43:22.224319+0000","last_undegraded":"2024-02-05T07:43:22.224319+0000","last_fullsized":"2024-02-05T07:43:22.224319+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'46","last_scrub_stamp":"2024-02-05T07:43:02.681256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:02.681256+0000","objects_scrubbed":8,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:13.499647+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":341,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.983849+0000","last_change":"2024-02-05T07:42:29.206268+0000","last_active":"2024-02-05T07:43:29.983849+0000","last_peered":"2024-02-05T07:43:29.983849+0000","last_clean":"2024-02-05T07:43:29.983849+0000","last_became_active":"2024-02-05T07:41:49.736707+0000","last_became_peered":"2024-02-05T07:41:49.736707+0000","last_unstale":"2024-02-05T07:43:29.983849+0000","last_undegraded":"2024-02-05T07:43:29.983849+0000","last_fullsized":"2024-02-05T07:43:29.983849+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:29.206178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:42:29.206178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:36.155973+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":439,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897874+0000","last_change":"2024-02-05T07:42:49.691175+0000","last_active":"2024-02-05T07:43:29.897874+0000","last_peered":"2024-02-05T07:43:29.897874+0000","last_clean":"2024-02-05T07:43:29.897874+0000","last_became_active":"2024-02-05T07:42:48.805828+0000","last_became_peered":"2024-02-05T07:42:48.805828+0000","last_unstale":"2024-02-05T07:43:29.897874+0000","last_undegraded":"2024-02-05T07:43:29.897874+0000","last_fullsized":"2024-02-05T07:43:29.897874+0000","mapping_epoch":748,"log_start":"86'128","ondisk_log_start":"86'128","created":20,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:49.691081+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:42:49.691081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:59.660676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":129,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.745274+0000","last_change":"2024-02-05T07:43:23.326134+0000","last_active":"2024-02-05T07:43:29.745274+0000","last_peered":"2024-02-05T07:43:29.745274+0000","last_clean":"2024-02-05T07:43:29.745274+0000","last_became_active":"2024-02-05T07:41:53.752144+0000","last_became_peered":"2024-02-05T07:41:53.752144+0000","last_unstale":"2024-02-05T07:43:29.745274+0000","last_undegraded":"2024-02-05T07:43:29.745274+0000","last_fullsized":"2024-02-05T07:43:29.745274+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:23.326009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:43:23.326009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:33.263326+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":453,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897852+0000","last_change":"2024-02-05T07:43:23.771265+0000","last_active":"2024-02-05T07:43:29.897852+0000","last_peered":"2024-02-05T07:43:29.897852+0000","last_clean":"2024-02-05T07:43:29.897852+0000","last_became_active":"2024-02-05T07:41:56.731825+0000","last_became_peered":"2024-02-05T07:41:56.731825+0000","last_unstale":"2024-02-05T07:43:29.897852+0000","last_undegraded":"2024-02-05T07:43:29.897852+0000","last_fullsized":"2024-02-05T07:43:29.897852+0000","mapping_epoch":714,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:23.771152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:43:23.771152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:46.984569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":502,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.984546+0000","last_change":"2024-02-05T07:43:19.014258+0000","last_active":"2024-02-05T07:43:29.984546+0000","last_peered":"2024-02-05T07:43:29.984546+0000","last_clean":"2024-02-05T07:43:29.984546+0000","last_became_active":"2024-02-05T07:41:49.756275+0000","last_became_peered":"2024-02-05T07:41:49.756275+0000","last_unstale":"2024-02-05T07:43:29.984546+0000","last_undegraded":"2024-02-05T07:43:29.984546+0000","last_fullsized":"2024-02-05T07:43:29.984546+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:19.014155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:43:19.014155+0000","last_clean_scrub_stamp":"2024-02-05T07:43:19.014155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:37.584468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"615'26","reported_seq":467,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.985542+0000","last_change":"2024-02-05T07:43:29.985572+0000","last_active":"2024-02-05T07:43:29.985542+0000","last_peered":"2024-02-05T07:43:29.985542+0000","last_clean":"2024-02-05T07:43:29.985542+0000","last_became_active":"2024-02-05T07:41:49.748077+0000","last_became_peered":"2024-02-05T07:41:49.748077+0000","last_unstale":"2024-02-05T07:43:29.985542+0000","last_undegraded":"2024-02-05T07:43:29.985542+0000","last_fullsized":"2024-02-05T07:43:29.985542+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'26","last_scrub_stamp":"2024-02-05T07:43:29.985485+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:43:29.985485+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:43.292062+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":470,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.984061+0000","last_change":"2024-02-05T07:42:24.167581+0000","last_active":"2024-02-05T07:43:29.984061+0000","last_peered":"2024-02-05T07:43:29.984061+0000","last_clean":"2024-02-05T07:43:29.984061+0000","last_became_active":"2024-02-05T07:36:32.733822+0000","last_became_peered":"2024-02-05T07:36:32.733822+0000","last_unstale":"2024-02-05T07:43:29.984061+0000","last_undegraded":"2024-02-05T07:43:29.984061+0000","last_fullsized":"2024-02-05T07:43:29.984061+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:24.167505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:42:24.167505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:32.116354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":918,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742248+0000","last_change":"2024-02-05T07:43:26.096612+0000","last_active":"2024-02-05T07:43:29.742248+0000","last_peered":"2024-02-05T07:43:29.742248+0000","last_clean":"2024-02-05T07:43:29.742248+0000","last_became_active":"2024-02-05T07:42:11.956148+0000","last_became_peered":"2024-02-05T07:42:11.956148+0000","last_unstale":"2024-02-05T07:43:29.742248+0000","last_undegraded":"2024-02-05T07:43:29.742248+0000","last_fullsized":"2024-02-05T07:43:29.742248+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:43:26.096520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:43:26.096520+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:42.328266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":131,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898424+0000","last_change":"2024-02-05T07:43:16.788580+0000","last_active":"2024-02-05T07:43:29.898424+0000","last_peered":"2024-02-05T07:43:29.898424+0000","last_clean":"2024-02-05T07:43:29.898424+0000","last_became_active":"2024-02-05T07:41:45.938599+0000","last_became_peered":"2024-02-05T07:41:45.938599+0000","last_unstale":"2024-02-05T07:43:29.898424+0000","last_undegraded":"2024-02-05T07:43:29.898424+0000","last_fullsized":"2024-02-05T07:43:29.898424+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:16.788465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:00.818054+0000","last_clean_scrub_stamp":"2024-02-05T07:43:16.788465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:41.462635+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":799,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744011+0000","last_change":"2024-02-05T07:42:36.739805+0000","last_active":"2024-02-05T07:43:29.744011+0000","last_peered":"2024-02-05T07:43:29.744011+0000","last_clean":"2024-02-05T07:43:29.744011+0000","last_became_active":"2024-02-05T07:42:36.739437+0000","last_became_peered":"2024-02-05T07:42:36.739437+0000","last_unstale":"2024-02-05T07:43:29.744011+0000","last_undegraded":"2024-02-05T07:43:29.744011+0000","last_fullsized":"2024-02-05T07:43:29.744011+0000","mapping_epoch":741,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:06.284725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:06.284725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1205,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743875+0000","last_change":"2024-02-05T07:42:39.333789+0000","last_active":"2024-02-05T07:43:29.743875+0000","last_peered":"2024-02-05T07:43:29.743875+0000","last_clean":"2024-02-05T07:43:29.743875+0000","last_became_active":"2024-02-05T07:42:05.922152+0000","last_became_peered":"2024-02-05T07:42:05.922152+0000","last_unstale":"2024-02-05T07:43:29.743875+0000","last_undegraded":"2024-02-05T07:43:29.743875+0000","last_fullsized":"2024-02-05T07:43:29.743875+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:39.333697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:42:39.333697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:47.330280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"722'69","reported_seq":366,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744159+0000","last_change":"2024-02-05T07:42:54.499621+0000","last_active":"2024-02-05T07:43:29.744159+0000","last_peered":"2024-02-05T07:43:29.744159+0000","last_clean":"2024-02-05T07:43:29.744159+0000","last_became_active":"2024-02-05T07:42:05.925978+0000","last_became_peered":"2024-02-05T07:42:05.925978+0000","last_unstale":"2024-02-05T07:43:29.744159+0000","last_undegraded":"2024-02-05T07:43:29.744159+0000","last_fullsized":"2024-02-05T07:43:29.744159+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'69","last_scrub_stamp":"2024-02-05T07:42:54.499501+0000","last_deep_scrub":"722'69","last_deep_scrub_stamp":"2024-02-05T07:42:54.499501+0000","last_clean_scrub_stamp":"2024-02-05T07:42:54.499501+0000","objects_scrubbed":10,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:59.389228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":81,"num_read_kb":66,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912785+0000","last_change":"2024-02-05T07:42:53.969116+0000","last_active":"2024-02-05T07:43:29.912785+0000","last_peered":"2024-02-05T07:43:29.912785+0000","last_clean":"2024-02-05T07:43:29.912785+0000","last_became_active":"2024-02-05T07:41:53.756727+0000","last_became_peered":"2024-02-05T07:41:53.756727+0000","last_unstale":"2024-02-05T07:43:29.912785+0000","last_undegraded":"2024-02-05T07:43:29.912785+0000","last_fullsized":"2024-02-05T07:43:29.912785+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:53.969006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:53.969006+0000","last_clean_scrub_stamp":"2024-02-05T07:42:53.969006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:55.164088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":1308,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744230+0000","last_change":"2024-02-05T07:42:36.742497+0000","last_active":"2024-02-05T07:43:29.744230+0000","last_peered":"2024-02-05T07:43:29.744230+0000","last_clean":"2024-02-05T07:43:29.744230+0000","last_became_active":"2024-02-05T07:42:36.742068+0000","last_became_peered":"2024-02-05T07:42:36.742068+0000","last_unstale":"2024-02-05T07:43:29.744230+0000","last_undegraded":"2024-02-05T07:43:29.744230+0000","last_fullsized":"2024-02-05T07:43:29.744230+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:42:07.050455+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:42:07.050455+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:36.981039+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1058,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.687380+0000","last_change":"2024-02-05T07:42:39.799450+0000","last_active":"2024-02-05T07:43:30.687380+0000","last_peered":"2024-02-05T07:43:30.687380+0000","last_clean":"2024-02-05T07:43:30.687380+0000","last_became_active":"2024-02-05T07:41:53.751831+0000","last_became_peered":"2024-02-05T07:41:53.751831+0000","last_unstale":"2024-02-05T07:43:30.687380+0000","last_undegraded":"2024-02-05T07:43:30.687380+0000","last_fullsized":"2024-02-05T07:43:30.687380+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:39.799335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:39.799335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:46.389673+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":616,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913904+0000","last_change":"2024-02-05T07:43:25.789659+0000","last_active":"2024-02-05T07:43:29.913904+0000","last_peered":"2024-02-05T07:43:29.913904+0000","last_clean":"2024-02-05T07:43:29.913904+0000","last_became_active":"2024-02-05T07:41:55.732199+0000","last_became_peered":"2024-02-05T07:41:55.732199+0000","last_unstale":"2024-02-05T07:43:29.913904+0000","last_undegraded":"2024-02-05T07:43:29.913904+0000","last_fullsized":"2024-02-05T07:43:29.913904+0000","mapping_epoch":713,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":714,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:25.789560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:43:25.789560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:55.227667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"750'25","reported_seq":1068,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743814+0000","last_change":"2024-02-05T07:42:50.453544+0000","last_active":"2024-02-05T07:43:29.743814+0000","last_peered":"2024-02-05T07:43:29.743814+0000","last_clean":"2024-02-05T07:43:29.743814+0000","last_became_active":"2024-02-05T07:42:05.924209+0000","last_became_peered":"2024-02-05T07:42:05.924209+0000","last_unstale":"2024-02-05T07:43:29.743814+0000","last_undegraded":"2024-02-05T07:43:29.743814+0000","last_fullsized":"2024-02-05T07:43:29.743814+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"750'25","last_scrub_stamp":"2024-02-05T07:42:50.453447+0000","last_deep_scrub":"750'25","last_deep_scrub_stamp":"2024-02-05T07:42:50.453447+0000","last_clean_scrub_stamp":"2024-02-05T07:42:50.453447+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:55.440697+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":969,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744672+0000","last_change":"2024-02-05T07:43:06.142521+0000","last_active":"2024-02-05T07:43:29.744672+0000","last_peered":"2024-02-05T07:43:29.744672+0000","last_clean":"2024-02-05T07:43:29.744672+0000","last_became_active":"2024-02-05T07:42:36.742368+0000","last_became_peered":"2024-02-05T07:42:36.742368+0000","last_unstale":"2024-02-05T07:43:29.744672+0000","last_undegraded":"2024-02-05T07:43:29.744672+0000","last_fullsized":"2024-02-05T07:43:29.744672+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:06.142417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:54.146194+0000","last_clean_scrub_stamp":"2024-02-05T07:43:06.142417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:33.560481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":490,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.984074+0000","last_change":"2024-02-05T07:42:55.945438+0000","last_active":"2024-02-05T07:43:29.984074+0000","last_peered":"2024-02-05T07:43:29.984074+0000","last_clean":"2024-02-05T07:43:29.984074+0000","last_became_active":"2024-02-05T07:41:53.737660+0000","last_became_peered":"2024-02-05T07:41:53.737660+0000","last_unstale":"2024-02-05T07:43:29.984074+0000","last_undegraded":"2024-02-05T07:43:29.984074+0000","last_fullsized":"2024-02-05T07:43:29.984074+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:55.945348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:41:40.550671+0000","last_clean_scrub_stamp":"2024-02-05T07:42:55.945348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:14.699029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":333,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742986+0000","last_change":"2024-02-05T07:42:47.509045+0000","last_active":"2024-02-05T07:43:29.742986+0000","last_peered":"2024-02-05T07:43:29.742986+0000","last_clean":"2024-02-05T07:43:29.742986+0000","last_became_active":"2024-02-05T07:42:05.925362+0000","last_became_peered":"2024-02-05T07:42:05.925362+0000","last_unstale":"2024-02-05T07:43:29.742986+0000","last_undegraded":"2024-02-05T07:43:29.742986+0000","last_fullsized":"2024-02-05T07:43:29.742986+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:47.508944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:47.508944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:49.041403+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":1486,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-02-05T07:43:22.224764+0000","last_change":"2024-02-05T07:42:47.700393+0000","last_active":"2024-02-05T07:43:22.224764+0000","last_peered":"2024-02-05T07:43:22.224764+0000","last_clean":"2024-02-05T07:43:22.224764+0000","last_became_active":"2024-02-05T07:42:24.729306+0000","last_became_peered":"2024-02-05T07:42:24.729306+0000","last_unstale":"2024-02-05T07:43:22.224764+0000","last_undegraded":"2024-02-05T07:43:22.224764+0000","last_fullsized":"2024-02-05T07:43:22.224764+0000","mapping_epoch":735,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":736,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:47.700266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:47.700266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:04.828316+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"616'51","reported_seq":1447,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-02-05T07:43:22.221293+0000","last_change":"2024-02-05T07:43:21.686492+0000","last_active":"2024-02-05T07:43:22.221293+0000","last_peered":"2024-02-05T07:43:22.221293+0000","last_clean":"2024-02-05T07:43:22.221293+0000","last_became_active":"2024-02-05T07:41:45.914903+0000","last_became_peered":"2024-02-05T07:41:45.914903+0000","last_unstale":"2024-02-05T07:43:22.221293+0000","last_undegraded":"2024-02-05T07:43:22.221293+0000","last_fullsized":"2024-02-05T07:43:22.221293+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'51","last_scrub_stamp":"2024-02-05T07:43:21.686362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:21.686362+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:50.670208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":129,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898340+0000","last_change":"2024-02-05T07:42:19.750470+0000","last_active":"2024-02-05T07:43:29.898340+0000","last_peered":"2024-02-05T07:43:29.898340+0000","last_clean":"2024-02-05T07:43:29.898340+0000","last_became_active":"2024-02-05T07:41:49.760281+0000","last_became_peered":"2024-02-05T07:41:49.760281+0000","last_unstale":"2024-02-05T07:43:29.898340+0000","last_undegraded":"2024-02-05T07:43:29.898340+0000","last_fullsized":"2024-02-05T07:43:29.898340+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:19.750373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_clean_scrub_stamp":"2024-02-05T07:42:19.750373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:41.082115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":490,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.984216+0000","last_change":"2024-02-05T07:43:15.021367+0000","last_active":"2024-02-05T07:43:29.984216+0000","last_peered":"2024-02-05T07:43:29.984216+0000","last_clean":"2024-02-05T07:43:29.984216+0000","last_became_active":"2024-02-05T07:41:49.747891+0000","last_became_peered":"2024-02-05T07:41:49.747891+0000","last_unstale":"2024-02-05T07:43:29.984216+0000","last_undegraded":"2024-02-05T07:43:29.984216+0000","last_fullsized":"2024-02-05T07:43:29.984216+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:15.021283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:15.021283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:19.170938+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":123,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897671+0000","last_change":"2024-02-05T07:41:57.666480+0000","last_active":"2024-02-05T07:43:29.897671+0000","last_peered":"2024-02-05T07:43:29.897671+0000","last_clean":"2024-02-05T07:43:29.897671+0000","last_became_active":"2024-02-05T07:41:45.937344+0000","last_became_peered":"2024-02-05T07:41:45.937344+0000","last_unstale":"2024-02-05T07:43:29.897671+0000","last_undegraded":"2024-02-05T07:43:29.897671+0000","last_fullsized":"2024-02-05T07:43:29.897671+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:57.666375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:38:01.478555+0000","last_clean_scrub_stamp":"2024-02-05T07:41:57.666375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":1821,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.687072+0000","last_change":"2024-02-05T07:42:04.595436+0000","last_active":"2024-02-05T07:43:30.687072+0000","last_peered":"2024-02-05T07:43:30.687072+0000","last_clean":"2024-02-05T07:43:30.687072+0000","last_became_active":"2024-02-05T07:40:28.961096+0000","last_became_peered":"2024-02-05T07:40:28.961096+0000","last_unstale":"2024-02-05T07:43:30.687072+0000","last_undegraded":"2024-02-05T07:43:30.687072+0000","last_fullsized":"2024-02-05T07:43:30.687072+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:04.595339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:04.595339+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":118,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:34.353568+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":478,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.983776+0000","last_change":"2024-02-05T07:43:23.079920+0000","last_active":"2024-02-05T07:43:29.983776+0000","last_peered":"2024-02-05T07:43:29.983776+0000","last_clean":"2024-02-05T07:43:29.983776+0000","last_became_active":"2024-02-05T07:36:32.733003+0000","last_became_peered":"2024-02-05T07:36:32.733003+0000","last_unstale":"2024-02-05T07:43:29.983776+0000","last_undegraded":"2024-02-05T07:43:29.983776+0000","last_fullsized":"2024-02-05T07:43:29.983776+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:23.079787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:23.079787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:34.512688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"615'24","reported_seq":119,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912668+0000","last_change":"2024-02-05T07:43:05.865185+0000","last_active":"2024-02-05T07:43:29.912668+0000","last_peered":"2024-02-05T07:43:29.912668+0000","last_clean":"2024-02-05T07:43:29.912668+0000","last_became_active":"2024-02-05T07:41:53.752799+0000","last_became_peered":"2024-02-05T07:41:53.752799+0000","last_unstale":"2024-02-05T07:43:29.912668+0000","last_undegraded":"2024-02-05T07:43:29.912668+0000","last_fullsized":"2024-02-05T07:43:29.912668+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'24","last_scrub_stamp":"2024-02-05T07:43:05.865075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:05.865075+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:12.836841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":129,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897491+0000","last_change":"2024-02-05T07:42:40.690417+0000","last_active":"2024-02-05T07:43:29.897491+0000","last_peered":"2024-02-05T07:43:29.897491+0000","last_clean":"2024-02-05T07:43:29.897491+0000","last_became_active":"2024-02-05T07:41:53.752835+0000","last_became_peered":"2024-02-05T07:41:53.752835+0000","last_unstale":"2024-02-05T07:43:29.897491+0000","last_undegraded":"2024-02-05T07:43:29.897491+0000","last_fullsized":"2024-02-05T07:43:29.897491+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:40.690284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:42:40.690284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:00.234498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1046,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.686093+0000","last_change":"2024-02-05T07:42:46.897648+0000","last_active":"2024-02-05T07:43:30.686093+0000","last_peered":"2024-02-05T07:43:30.686093+0000","last_clean":"2024-02-05T07:43:30.686093+0000","last_became_active":"2024-02-05T07:41:46.009533+0000","last_became_peered":"2024-02-05T07:41:46.009533+0000","last_unstale":"2024-02-05T07:43:30.686093+0000","last_undegraded":"2024-02-05T07:43:30.686093+0000","last_fullsized":"2024-02-05T07:43:30.686093+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:46.897527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:46.897527+0000","last_clean_scrub_stamp":"2024-02-05T07:42:46.897527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:14.490189+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":1287,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898608+0000","last_change":"2024-02-05T07:41:55.729741+0000","last_active":"2024-02-05T07:43:29.898608+0000","last_peered":"2024-02-05T07:43:29.898608+0000","last_clean":"2024-02-05T07:43:29.898608+0000","last_became_active":"2024-02-05T07:41:55.729341+0000","last_became_peered":"2024-02-05T07:41:55.729341+0000","last_unstale":"2024-02-05T07:43:29.898608+0000","last_undegraded":"2024-02-05T07:43:29.898608+0000","last_fullsized":"2024-02-05T07:43:29.898608+0000","mapping_epoch":713,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":714,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:41:53.850473+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:41:53.850473+0000","last_clean_scrub_stamp":"2024-02-05T07:41:53.850473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"713'15","reported_seq":159,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898000+0000","last_change":"2024-02-05T07:42:43.646159+0000","last_active":"2024-02-05T07:43:29.898000+0000","last_peered":"2024-02-05T07:43:29.898000+0000","last_clean":"2024-02-05T07:43:29.898000+0000","last_became_active":"2024-02-05T07:41:53.752883+0000","last_became_peered":"2024-02-05T07:41:53.752883+0000","last_unstale":"2024-02-05T07:43:29.898000+0000","last_undegraded":"2024-02-05T07:43:29.898000+0000","last_fullsized":"2024-02-05T07:43:29.898000+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'15","last_scrub_stamp":"2024-02-05T07:42:43.646056+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:42:43.646056+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:50.923210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"754'46","reported_seq":146,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744074+0000","last_change":"2024-02-05T07:43:27.201592+0000","last_active":"2024-02-05T07:43:29.744074+0000","last_peered":"2024-02-05T07:43:29.744074+0000","last_clean":"2024-02-05T07:43:29.744074+0000","last_became_active":"2024-02-05T07:41:49.734651+0000","last_became_peered":"2024-02-05T07:41:49.734651+0000","last_unstale":"2024-02-05T07:43:29.744074+0000","last_undegraded":"2024-02-05T07:43:29.744074+0000","last_fullsized":"2024-02-05T07:43:29.744074+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'46","last_scrub_stamp":"2024-02-05T07:43:27.201474+0000","last_deep_scrub":"754'46","last_deep_scrub_stamp":"2024-02-05T07:43:27.201474+0000","last_clean_scrub_stamp":"2024-02-05T07:43:27.201474+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:55.206882+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"614'4","reported_seq":141,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897514+0000","last_change":"2024-02-05T07:42:06.730315+0000","last_active":"2024-02-05T07:43:29.897514+0000","last_peered":"2024-02-05T07:43:29.897514+0000","last_clean":"2024-02-05T07:43:29.897514+0000","last_became_active":"2024-02-05T07:41:53.753442+0000","last_became_peered":"2024-02-05T07:41:53.753442+0000","last_unstale":"2024-02-05T07:43:29.897514+0000","last_undegraded":"2024-02-05T07:43:29.897514+0000","last_fullsized":"2024-02-05T07:43:29.897514+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"614'4","last_scrub_stamp":"2024-02-05T07:42:06.730218+0000","last_deep_scrub":"614'4","last_deep_scrub_stamp":"2024-02-05T07:38:55.829386+0000","last_clean_scrub_stamp":"2024-02-05T07:42:06.730218+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1067,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744466+0000","last_change":"2024-02-05T07:42:36.743895+0000","last_active":"2024-02-05T07:43:29.744466+0000","last_peered":"2024-02-05T07:43:29.744466+0000","last_clean":"2024-02-05T07:43:29.744466+0000","last_became_active":"2024-02-05T07:42:36.743154+0000","last_became_peered":"2024-02-05T07:42:36.743154+0000","last_unstale":"2024-02-05T07:43:29.744466+0000","last_undegraded":"2024-02-05T07:43:29.744466+0000","last_fullsized":"2024-02-05T07:43:29.744466+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:25.616991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:42:25.616991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:33.787790+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":803,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744341+0000","last_change":"2024-02-05T07:43:24.158620+0000","last_active":"2024-02-05T07:43:29.744341+0000","last_peered":"2024-02-05T07:43:29.744341+0000","last_clean":"2024-02-05T07:43:29.744341+0000","last_became_active":"2024-02-05T07:41:51.724911+0000","last_became_peered":"2024-02-05T07:41:51.724911+0000","last_unstale":"2024-02-05T07:43:29.744341+0000","last_undegraded":"2024-02-05T07:43:29.744341+0000","last_fullsized":"2024-02-05T07:43:29.744341+0000","mapping_epoch":709,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:24.158519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:43:24.158519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:30.053686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"615'21","reported_seq":1110,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.686599+0000","last_change":"2024-02-05T07:42:20.575553+0000","last_active":"2024-02-05T07:43:30.686599+0000","last_peered":"2024-02-05T07:43:30.686599+0000","last_clean":"2024-02-05T07:43:30.686599+0000","last_became_active":"2024-02-05T07:41:46.203574+0000","last_became_peered":"2024-02-05T07:41:46.203574+0000","last_unstale":"2024-02-05T07:43:30.686599+0000","last_undegraded":"2024-02-05T07:43:30.686599+0000","last_fullsized":"2024-02-05T07:43:30.686599+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'21","last_scrub_stamp":"2024-02-05T07:42:20.575458+0000","last_deep_scrub":"613'15","last_deep_scrub_stamp":"2024-02-05T07:38:48.343448+0000","last_clean_scrub_stamp":"2024-02-05T07:42:20.575458+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:48.441315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":1272,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-02-05T07:43:22.221769+0000","last_change":"2024-02-05T07:42:48.726688+0000","last_active":"2024-02-05T07:43:22.221769+0000","last_peered":"2024-02-05T07:43:22.221769+0000","last_clean":"2024-02-05T07:43:22.221769+0000","last_became_active":"2024-02-05T07:41:49.732960+0000","last_became_peered":"2024-02-05T07:41:49.732960+0000","last_unstale":"2024-02-05T07:43:22.221769+0000","last_undegraded":"2024-02-05T07:43:22.221769+0000","last_fullsized":"2024-02-05T07:43:22.221769+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:42:48.726605+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:42:48.726605+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:00.811869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":115,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.745041+0000","last_change":"2024-02-05T07:43:00.099868+0000","last_active":"2024-02-05T07:43:29.745041+0000","last_peered":"2024-02-05T07:43:29.745041+0000","last_clean":"2024-02-05T07:43:29.745041+0000","last_became_active":"2024-02-05T07:41:49.759544+0000","last_became_peered":"2024-02-05T07:41:49.759544+0000","last_unstale":"2024-02-05T07:43:29.745041+0000","last_undegraded":"2024-02-05T07:43:29.745041+0000","last_fullsized":"2024-02-05T07:43:29.745041+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:00.099772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:00.099772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:16.090460+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":135,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898478+0000","last_change":"2024-02-05T07:41:59.813914+0000","last_active":"2024-02-05T07:43:29.898478+0000","last_peered":"2024-02-05T07:43:29.898478+0000","last_clean":"2024-02-05T07:43:29.898478+0000","last_became_active":"2024-02-05T07:41:53.751823+0000","last_became_peered":"2024-02-05T07:41:53.751823+0000","last_unstale":"2024-02-05T07:43:29.898478+0000","last_undegraded":"2024-02-05T07:43:29.898478+0000","last_fullsized":"2024-02-05T07:43:29.898478+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:59.813778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:41:59.813778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":895,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743067+0000","last_change":"2024-02-05T07:42:15.136606+0000","last_active":"2024-02-05T07:43:29.743067+0000","last_peered":"2024-02-05T07:43:29.743067+0000","last_clean":"2024-02-05T07:43:29.743067+0000","last_became_active":"2024-02-05T07:42:14.979893+0000","last_became_peered":"2024-02-05T07:42:14.979893+0000","last_unstale":"2024-02-05T07:43:29.743067+0000","last_undegraded":"2024-02-05T07:43:29.743067+0000","last_fullsized":"2024-02-05T07:43:29.743067+0000","mapping_epoch":726,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":727,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:15.136518+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:42:15.136518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:35.687088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"615'21","reported_seq":899,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744539+0000","last_change":"2024-02-05T07:42:21.384042+0000","last_active":"2024-02-05T07:43:29.744539+0000","last_peered":"2024-02-05T07:43:29.744539+0000","last_clean":"2024-02-05T07:43:29.744539+0000","last_became_active":"2024-02-05T07:42:05.928414+0000","last_became_peered":"2024-02-05T07:42:05.928414+0000","last_unstale":"2024-02-05T07:43:29.744539+0000","last_undegraded":"2024-02-05T07:43:29.744539+0000","last_fullsized":"2024-02-05T07:43:29.744539+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'21","last_scrub_stamp":"2024-02-05T07:42:21.383957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:42:21.383957+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:43.387421+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":137,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897699+0000","last_change":"2024-02-05T07:43:10.746556+0000","last_active":"2024-02-05T07:43:29.897699+0000","last_peered":"2024-02-05T07:43:29.897699+0000","last_clean":"2024-02-05T07:43:29.897699+0000","last_became_active":"2024-02-05T07:41:53.751818+0000","last_became_peered":"2024-02-05T07:41:53.751818+0000","last_unstale":"2024-02-05T07:43:29.897699+0000","last_undegraded":"2024-02-05T07:43:29.897699+0000","last_fullsized":"2024-02-05T07:43:29.897699+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:10.746414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:43:10.746414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:33.189668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1037,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.686478+0000","last_change":"2024-02-05T07:42:38.723653+0000","last_active":"2024-02-05T07:43:30.686478+0000","last_peered":"2024-02-05T07:43:30.686478+0000","last_clean":"2024-02-05T07:43:30.686478+0000","last_became_active":"2024-02-05T07:41:46.203566+0000","last_became_peered":"2024-02-05T07:41:46.203566+0000","last_unstale":"2024-02-05T07:43:30.686478+0000","last_undegraded":"2024-02-05T07:43:30.686478+0000","last_fullsized":"2024-02-05T07:43:30.686478+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:38.723577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_clean_scrub_stamp":"2024-02-05T07:42:38.723577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:40.837376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":495,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.984636+0000","last_change":"2024-02-05T07:43:07.965205+0000","last_active":"2024-02-05T07:43:29.984636+0000","last_peered":"2024-02-05T07:43:29.984636+0000","last_clean":"2024-02-05T07:43:29.984636+0000","last_became_active":"2024-02-05T07:41:53.742721+0000","last_became_peered":"2024-02-05T07:41:53.742721+0000","last_unstale":"2024-02-05T07:43:29.984636+0000","last_undegraded":"2024-02-05T07:43:29.984636+0000","last_fullsized":"2024-02-05T07:43:29.984636+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:07.965126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:43:07.965126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:25.673625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":442,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744139+0000","last_change":"2024-02-05T07:42:22.195996+0000","last_active":"2024-02-05T07:43:29.744139+0000","last_peered":"2024-02-05T07:43:29.744139+0000","last_clean":"2024-02-05T07:43:29.744139+0000","last_became_active":"2024-02-05T07:42:22.025157+0000","last_became_peered":"2024-02-05T07:42:22.025157+0000","last_unstale":"2024-02-05T07:43:29.744139+0000","last_undegraded":"2024-02-05T07:43:29.744139+0000","last_fullsized":"2024-02-05T07:43:29.744139+0000","mapping_epoch":732,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":733,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:22.195891+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:42:22.195891+0000","last_clean_scrub_stamp":"2024-02-05T07:42:22.195891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"615'25","reported_seq":133,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744782+0000","last_change":"2024-02-05T07:43:05.168692+0000","last_active":"2024-02-05T07:43:29.744782+0000","last_peered":"2024-02-05T07:43:29.744782+0000","last_clean":"2024-02-05T07:43:29.744782+0000","last_became_active":"2024-02-05T07:41:49.764420+0000","last_became_peered":"2024-02-05T07:41:49.764420+0000","last_unstale":"2024-02-05T07:43:29.744782+0000","last_undegraded":"2024-02-05T07:43:29.744782+0000","last_fullsized":"2024-02-05T07:43:29.744782+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'25","last_scrub_stamp":"2024-02-05T07:43:05.168573+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:43:05.168573+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:08.800540+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":473,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742577+0000","last_change":"2024-02-05T07:43:00.548691+0000","last_active":"2024-02-05T07:43:29.742577+0000","last_peered":"2024-02-05T07:43:29.742577+0000","last_clean":"2024-02-05T07:43:29.742577+0000","last_became_active":"2024-02-05T07:42:05.926723+0000","last_became_peered":"2024-02-05T07:42:05.926723+0000","last_unstale":"2024-02-05T07:43:29.742577+0000","last_undegraded":"2024-02-05T07:43:29.742577+0000","last_fullsized":"2024-02-05T07:43:29.742577+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:00.548588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:41:47.678965+0000","last_clean_scrub_stamp":"2024-02-05T07:43:00.548588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:18.963269+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"704'14","reported_seq":1265,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.686051+0000","last_change":"2024-02-05T07:42:34.643665+0000","last_active":"2024-02-05T07:43:30.686051+0000","last_peered":"2024-02-05T07:43:30.686051+0000","last_clean":"2024-02-05T07:43:30.686051+0000","last_became_active":"2024-02-05T07:41:45.943759+0000","last_became_peered":"2024-02-05T07:41:45.943759+0000","last_unstale":"2024-02-05T07:43:30.686051+0000","last_undegraded":"2024-02-05T07:43:30.686051+0000","last_fullsized":"2024-02-05T07:43:30.686051+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'14","last_scrub_stamp":"2024-02-05T07:42:34.643555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:42:34.643555+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:54.840311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912887+0000","last_change":"2024-02-05T07:42:26.427884+0000","last_active":"2024-02-05T07:43:29.912887+0000","last_peered":"2024-02-05T07:43:29.912887+0000","last_clean":"2024-02-05T07:43:29.912887+0000","last_became_active":"2024-02-05T07:41:53.763253+0000","last_became_peered":"2024-02-05T07:41:53.763253+0000","last_unstale":"2024-02-05T07:43:29.912887+0000","last_undegraded":"2024-02-05T07:43:29.912887+0000","last_fullsized":"2024-02-05T07:43:29.912887+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:26.427765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:26.427765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:31.131882+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":483,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742019+0000","last_change":"2024-02-05T07:42:07.318649+0000","last_active":"2024-02-05T07:43:29.742019+0000","last_peered":"2024-02-05T07:43:29.742019+0000","last_clean":"2024-02-05T07:43:29.742019+0000","last_became_active":"2024-02-05T07:42:05.927588+0000","last_became_peered":"2024-02-05T07:42:05.927588+0000","last_unstale":"2024-02-05T07:43:29.742019+0000","last_undegraded":"2024-02-05T07:43:29.742019+0000","last_fullsized":"2024-02-05T07:43:29.742019+0000","mapping_epoch":719,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:07.318543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:07.318543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:37.101543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"615'35","reported_seq":1147,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.685871+0000","last_change":"2024-02-05T07:43:07.547909+0000","last_active":"2024-02-05T07:43:30.685871+0000","last_peered":"2024-02-05T07:43:30.685871+0000","last_clean":"2024-02-05T07:43:30.685871+0000","last_became_active":"2024-02-05T07:41:45.944922+0000","last_became_peered":"2024-02-05T07:41:45.944922+0000","last_unstale":"2024-02-05T07:43:30.685871+0000","last_undegraded":"2024-02-05T07:43:30.685871+0000","last_fullsized":"2024-02-05T07:43:30.685871+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'35","last_scrub_stamp":"2024-02-05T07:43:07.547817+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:43:07.547817+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:24.763038+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":129,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898252+0000","last_change":"2024-02-05T07:42:18.787273+0000","last_active":"2024-02-05T07:43:29.898252+0000","last_peered":"2024-02-05T07:43:29.898252+0000","last_clean":"2024-02-05T07:43:29.898252+0000","last_became_active":"2024-02-05T07:41:53.738482+0000","last_became_peered":"2024-02-05T07:41:53.738482+0000","last_unstale":"2024-02-05T07:43:29.898252+0000","last_undegraded":"2024-02-05T07:43:29.898252+0000","last_fullsized":"2024-02-05T07:43:29.898252+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:18.787178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:42:18.787178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:37.424205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"727'14","reported_seq":141,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.745072+0000","last_change":"2024-02-05T07:42:16.144597+0000","last_active":"2024-02-05T07:43:29.745072+0000","last_peered":"2024-02-05T07:43:29.745072+0000","last_clean":"2024-02-05T07:43:29.745072+0000","last_became_active":"2024-02-05T07:41:49.741112+0000","last_became_peered":"2024-02-05T07:41:49.741112+0000","last_unstale":"2024-02-05T07:43:29.745072+0000","last_undegraded":"2024-02-05T07:43:29.745072+0000","last_fullsized":"2024-02-05T07:43:29.745072+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'14","last_scrub_stamp":"2024-02-05T07:42:16.144496+0000","last_deep_scrub":"727'14","last_deep_scrub_stamp":"2024-02-05T07:42:16.144496+0000","last_clean_scrub_stamp":"2024-02-05T07:42:16.144496+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:36.503020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":927,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743646+0000","last_change":"2024-02-05T07:43:20.840005+0000","last_active":"2024-02-05T07:43:29.743646+0000","last_peered":"2024-02-05T07:43:29.743646+0000","last_clean":"2024-02-05T07:43:29.743646+0000","last_became_active":"2024-02-05T07:42:05.925936+0000","last_became_peered":"2024-02-05T07:42:05.925936+0000","last_unstale":"2024-02-05T07:43:29.743646+0000","last_undegraded":"2024-02-05T07:43:29.743646+0000","last_fullsized":"2024-02-05T07:43:29.743646+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:20.839896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:43:20.839896+0000","last_clean_scrub_stamp":"2024-02-05T07:43:20.839896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:37.416997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"615'27","reported_seq":119,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912742+0000","last_change":"2024-02-05T07:43:04.866672+0000","last_active":"2024-02-05T07:43:29.912742+0000","last_peered":"2024-02-05T07:43:29.912742+0000","last_clean":"2024-02-05T07:43:29.912742+0000","last_became_active":"2024-02-05T07:41:53.754844+0000","last_became_peered":"2024-02-05T07:41:53.754844+0000","last_unstale":"2024-02-05T07:43:29.912742+0000","last_undegraded":"2024-02-05T07:43:29.912742+0000","last_fullsized":"2024-02-05T07:43:29.912742+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'27","last_scrub_stamp":"2024-02-05T07:43:04.866562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:04.866562+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:17.895184+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913801+0000","last_change":"2024-02-05T07:42:41.124845+0000","last_active":"2024-02-05T07:43:29.913801+0000","last_peered":"2024-02-05T07:43:29.913801+0000","last_clean":"2024-02-05T07:43:29.913801+0000","last_became_active":"2024-02-05T07:41:53.751189+0000","last_became_peered":"2024-02-05T07:41:53.751189+0000","last_unstale":"2024-02-05T07:43:29.913801+0000","last_undegraded":"2024-02-05T07:43:29.913801+0000","last_fullsized":"2024-02-05T07:43:29.913801+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:41.124746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:42:41.124746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:09.119048+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1487,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742711+0000","last_change":"2024-02-05T07:43:15.180126+0000","last_active":"2024-02-05T07:43:29.742711+0000","last_peered":"2024-02-05T07:43:29.742711+0000","last_clean":"2024-02-05T07:43:29.742711+0000","last_became_active":"2024-02-05T07:41:49.750878+0000","last_became_peered":"2024-02-05T07:41:49.750878+0000","last_unstale":"2024-02-05T07:43:29.742711+0000","last_undegraded":"2024-02-05T07:43:29.742711+0000","last_fullsized":"2024-02-05T07:43:29.742711+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:15.180043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_clean_scrub_stamp":"2024-02-05T07:43:15.180043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:20.403842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1041,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.686652+0000","last_change":"2024-02-05T07:42:09.551986+0000","last_active":"2024-02-05T07:43:30.686652+0000","last_peered":"2024-02-05T07:43:30.686652+0000","last_clean":"2024-02-05T07:43:30.686652+0000","last_became_active":"2024-02-05T07:41:49.762328+0000","last_became_peered":"2024-02-05T07:41:49.762328+0000","last_unstale":"2024-02-05T07:43:30.686652+0000","last_undegraded":"2024-02-05T07:43:30.686652+0000","last_fullsized":"2024-02-05T07:43:30.686652+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:09.551894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:42:09.551894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:34.250213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"615'21","reported_seq":1122,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742270+0000","last_change":"2024-02-05T07:42:31.165246+0000","last_active":"2024-02-05T07:43:29.742270+0000","last_peered":"2024-02-05T07:43:29.742270+0000","last_clean":"2024-02-05T07:43:29.742270+0000","last_became_active":"2024-02-05T07:36:32.710090+0000","last_became_peered":"2024-02-05T07:36:32.710090+0000","last_unstale":"2024-02-05T07:43:29.742270+0000","last_undegraded":"2024-02-05T07:43:29.742270+0000","last_fullsized":"2024-02-05T07:43:29.742270+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'21","last_scrub_stamp":"2024-02-05T07:42:31.165083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:42:31.165083+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:54.549680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":137,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898545+0000","last_change":"2024-02-05T07:43:19.777472+0000","last_active":"2024-02-05T07:43:29.898545+0000","last_peered":"2024-02-05T07:43:29.898545+0000","last_clean":"2024-02-05T07:43:29.898545+0000","last_became_active":"2024-02-05T07:41:53.753389+0000","last_became_peered":"2024-02-05T07:41:53.753389+0000","last_unstale":"2024-02-05T07:43:29.898545+0000","last_undegraded":"2024-02-05T07:43:29.898545+0000","last_fullsized":"2024-02-05T07:43:29.898545+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:19.777339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:43:19.777339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:33.432123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1025,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.687118+0000","last_change":"2024-02-05T07:42:30.589866+0000","last_active":"2024-02-05T07:43:30.687118+0000","last_peered":"2024-02-05T07:43:30.687118+0000","last_clean":"2024-02-05T07:43:30.687118+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:43:30.687118+0000","last_undegraded":"2024-02-05T07:43:30.687118+0000","last_fullsized":"2024-02-05T07:43:30.687118+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:30.589751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:42:30.589751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:33.678636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":380,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742136+0000","last_change":"2024-02-05T07:43:10.734795+0000","last_active":"2024-02-05T07:43:29.742136+0000","last_peered":"2024-02-05T07:43:29.742136+0000","last_clean":"2024-02-05T07:43:29.742136+0000","last_became_active":"2024-02-05T07:41:49.724979+0000","last_became_peered":"2024-02-05T07:41:49.724979+0000","last_unstale":"2024-02-05T07:43:29.742136+0000","last_undegraded":"2024-02-05T07:43:29.742136+0000","last_fullsized":"2024-02-05T07:43:29.742136+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:10.734709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:43:10.734709+0000","last_clean_scrub_stamp":"2024-02-05T07:43:10.734709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:30.426933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":1862,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.984201+0000","last_change":"2024-02-05T07:42:46.142513+0000","last_active":"2024-02-05T07:43:29.984201+0000","last_peered":"2024-02-05T07:43:29.984201+0000","last_clean":"2024-02-05T07:43:29.984201+0000","last_became_active":"2024-02-05T07:41:56.731143+0000","last_became_peered":"2024-02-05T07:41:56.731143+0000","last_unstale":"2024-02-05T07:43:29.984201+0000","last_undegraded":"2024-02-05T07:43:29.984201+0000","last_fullsized":"2024-02-05T07:43:29.984201+0000","mapping_epoch":714,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:46.142430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:46.142430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:53.503737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"616'27","reported_seq":1108,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.685807+0000","last_change":"2024-02-05T07:43:13.488714+0000","last_active":"2024-02-05T07:43:30.685807+0000","last_peered":"2024-02-05T07:43:30.685807+0000","last_clean":"2024-02-05T07:43:30.685807+0000","last_became_active":"2024-02-05T07:41:45.948277+0000","last_became_peered":"2024-02-05T07:41:45.948277+0000","last_unstale":"2024-02-05T07:43:30.685807+0000","last_undegraded":"2024-02-05T07:43:30.685807+0000","last_fullsized":"2024-02-05T07:43:30.685807+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'27","last_scrub_stamp":"2024-02-05T07:43:13.488630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:13.488630+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:28.999480+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":115,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744939+0000","last_change":"2024-02-05T07:43:17.155583+0000","last_active":"2024-02-05T07:43:29.744939+0000","last_peered":"2024-02-05T07:43:29.744939+0000","last_clean":"2024-02-05T07:43:29.744939+0000","last_became_active":"2024-02-05T07:41:49.736869+0000","last_became_peered":"2024-02-05T07:41:49.736869+0000","last_unstale":"2024-02-05T07:43:29.744939+0000","last_undegraded":"2024-02-05T07:43:29.744939+0000","last_fullsized":"2024-02-05T07:43:29.744939+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:17.155488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:43:17.155488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:40.037279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":129,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897720+0000","last_change":"2024-02-05T07:42:10.745719+0000","last_active":"2024-02-05T07:43:29.897720+0000","last_peered":"2024-02-05T07:43:29.897720+0000","last_clean":"2024-02-05T07:43:29.897720+0000","last_became_active":"2024-02-05T07:41:53.737579+0000","last_became_peered":"2024-02-05T07:41:53.737579+0000","last_unstale":"2024-02-05T07:43:29.897720+0000","last_undegraded":"2024-02-05T07:43:29.897720+0000","last_fullsized":"2024-02-05T07:43:29.897720+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:10.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:42:10.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:38.258160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1037,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.686996+0000","last_change":"2024-02-05T07:42:50.783601+0000","last_active":"2024-02-05T07:43:30.686996+0000","last_peered":"2024-02-05T07:43:30.686996+0000","last_clean":"2024-02-05T07:43:30.686996+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:43:30.686996+0000","last_undegraded":"2024-02-05T07:43:30.686996+0000","last_fullsized":"2024-02-05T07:43:30.686996+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:50.783496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:50.783496+0000","last_clean_scrub_stamp":"2024-02-05T07:42:50.783496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:07.269592+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"615'12","reported_seq":1055,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.686657+0000","last_change":"2024-02-05T07:43:11.497632+0000","last_active":"2024-02-05T07:43:30.686657+0000","last_peered":"2024-02-05T07:43:30.686657+0000","last_clean":"2024-02-05T07:43:30.686657+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:43:30.686657+0000","last_undegraded":"2024-02-05T07:43:30.686657+0000","last_fullsized":"2024-02-05T07:43:30.686657+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'12","last_scrub_stamp":"2024-02-05T07:43:11.497509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:11.497509+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:15.037504+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":398,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744427+0000","last_change":"2024-02-05T07:42:37.359520+0000","last_active":"2024-02-05T07:43:29.744427+0000","last_peered":"2024-02-05T07:43:29.744427+0000","last_clean":"2024-02-05T07:43:29.744427+0000","last_became_active":"2024-02-05T07:42:05.928456+0000","last_became_peered":"2024-02-05T07:42:05.928456+0000","last_unstale":"2024-02-05T07:43:29.744427+0000","last_undegraded":"2024-02-05T07:43:29.744427+0000","last_fullsized":"2024-02-05T07:43:29.744427+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:37.359439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:34:39.980067+0000","last_clean_scrub_stamp":"2024-02-05T07:42:37.359439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:44.025447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1314,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-02-05T07:43:22.222751+0000","last_change":"2024-02-05T07:42:07.678508+0000","last_active":"2024-02-05T07:43:22.222751+0000","last_peered":"2024-02-05T07:43:22.222751+0000","last_clean":"2024-02-05T07:43:22.222751+0000","last_became_active":"2024-02-05T07:41:53.745802+0000","last_became_peered":"2024-02-05T07:41:53.745802+0000","last_unstale":"2024-02-05T07:43:22.222751+0000","last_undegraded":"2024-02-05T07:43:22.222751+0000","last_fullsized":"2024-02-05T07:43:22.222751+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:07.678407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:42:07.678407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:26.968688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":129,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897790+0000","last_change":"2024-02-05T07:42:42.719864+0000","last_active":"2024-02-05T07:43:29.897790+0000","last_peered":"2024-02-05T07:43:29.897790+0000","last_clean":"2024-02-05T07:43:29.897790+0000","last_became_active":"2024-02-05T07:41:53.737953+0000","last_became_peered":"2024-02-05T07:41:53.737953+0000","last_unstale":"2024-02-05T07:43:29.897790+0000","last_undegraded":"2024-02-05T07:43:29.897790+0000","last_fullsized":"2024-02-05T07:43:29.897790+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:42.719750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:58.296469+0000","last_clean_scrub_stamp":"2024-02-05T07:42:42.719750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:00.454523+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"616'27","reported_seq":1321,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-02-05T07:43:22.224390+0000","last_change":"2024-02-05T07:43:08.721766+0000","last_active":"2024-02-05T07:43:22.224390+0000","last_peered":"2024-02-05T07:43:22.224390+0000","last_clean":"2024-02-05T07:43:22.224390+0000","last_became_active":"2024-02-05T07:41:49.741249+0000","last_became_peered":"2024-02-05T07:41:49.741249+0000","last_unstale":"2024-02-05T07:43:22.224390+0000","last_undegraded":"2024-02-05T07:43:22.224390+0000","last_fullsized":"2024-02-05T07:43:22.224390+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'27","last_scrub_stamp":"2024-02-05T07:43:08.721624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:08.721624+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:35.548763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1033,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.686425+0000","last_change":"2024-02-05T07:42:54.586637+0000","last_active":"2024-02-05T07:43:30.686425+0000","last_peered":"2024-02-05T07:43:30.686425+0000","last_clean":"2024-02-05T07:43:30.686425+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:43:30.686425+0000","last_undegraded":"2024-02-05T07:43:30.686425+0000","last_fullsized":"2024-02-05T07:43:30.686425+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:54.586552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:40:11.849558+0000","last_clean_scrub_stamp":"2024-02-05T07:42:54.586552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:18.516132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":337,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744525+0000","last_change":"2024-02-05T07:42:32.406206+0000","last_active":"2024-02-05T07:43:29.744525+0000","last_peered":"2024-02-05T07:43:29.744525+0000","last_clean":"2024-02-05T07:43:29.744525+0000","last_became_active":"2024-02-05T07:42:05.921203+0000","last_became_peered":"2024-02-05T07:42:05.921203+0000","last_unstale":"2024-02-05T07:43:29.744525+0000","last_undegraded":"2024-02-05T07:43:29.744525+0000","last_fullsized":"2024-02-05T07:43:29.744525+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:32.406132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:42:32.406132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:58.099585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":888,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743620+0000","last_change":"2024-02-05T07:42:25.366238+0000","last_active":"2024-02-05T07:43:29.743620+0000","last_peered":"2024-02-05T07:43:29.743620+0000","last_clean":"2024-02-05T07:43:29.743620+0000","last_became_active":"2024-02-05T07:42:11.957914+0000","last_became_peered":"2024-02-05T07:42:11.957914+0000","last_unstale":"2024-02-05T07:43:29.743620+0000","last_undegraded":"2024-02-05T07:43:29.743620+0000","last_fullsized":"2024-02-05T07:43:29.743620+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:25.366113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:25.366113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:31.304667+0000","scrub_duration":233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"616'30","reported_seq":1008,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744354+0000","last_change":"2024-02-05T07:42:57.532987+0000","last_active":"2024-02-05T07:43:29.744354+0000","last_peered":"2024-02-05T07:43:29.744354+0000","last_clean":"2024-02-05T07:43:29.744354+0000","last_became_active":"2024-02-05T07:42:05.927928+0000","last_became_peered":"2024-02-05T07:42:05.927928+0000","last_unstale":"2024-02-05T07:43:29.744354+0000","last_undegraded":"2024-02-05T07:43:29.744354+0000","last_fullsized":"2024-02-05T07:43:29.744354+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'30","last_scrub_stamp":"2024-02-05T07:42:57.532884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:42:57.532884+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:11.267326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":37,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"585'2","reported_seq":341,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.983734+0000","last_change":"2024-02-05T07:42:05.295672+0000","last_active":"2024-02-05T07:43:29.983734+0000","last_peered":"2024-02-05T07:43:29.983734+0000","last_clean":"2024-02-05T07:43:29.983734+0000","last_became_active":"2024-02-05T07:41:53.747431+0000","last_became_peered":"2024-02-05T07:41:53.747431+0000","last_unstale":"2024-02-05T07:43:29.983734+0000","last_undegraded":"2024-02-05T07:43:29.983734+0000","last_fullsized":"2024-02-05T07:43:29.983734+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'2","last_scrub_stamp":"2024-02-05T07:42:05.295566+0000","last_deep_scrub":"585'2","last_deep_scrub_stamp":"2024-02-05T07:38:00.631235+0000","last_clean_scrub_stamp":"2024-02-05T07:42:05.295566+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"713'16","reported_seq":1627,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-02-05T07:43:22.224612+0000","last_change":"2024-02-05T07:42:36.667978+0000","last_active":"2024-02-05T07:43:22.224612+0000","last_peered":"2024-02-05T07:43:22.224612+0000","last_clean":"2024-02-05T07:43:22.224612+0000","last_became_active":"2024-02-05T07:41:53.756627+0000","last_became_peered":"2024-02-05T07:41:53.756627+0000","last_unstale":"2024-02-05T07:43:22.224612+0000","last_undegraded":"2024-02-05T07:43:22.224612+0000","last_fullsized":"2024-02-05T07:43:22.224612+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'16","last_scrub_stamp":"2024-02-05T07:42:36.667872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:42:36.667872+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:52.433126+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":137,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898299+0000","last_change":"2024-02-05T07:42:24.691285+0000","last_active":"2024-02-05T07:43:29.898299+0000","last_peered":"2024-02-05T07:43:29.898299+0000","last_clean":"2024-02-05T07:43:29.898299+0000","last_became_active":"2024-02-05T07:41:49.734786+0000","last_became_peered":"2024-02-05T07:41:49.734786+0000","last_unstale":"2024-02-05T07:43:29.898299+0000","last_undegraded":"2024-02-05T07:43:29.898299+0000","last_fullsized":"2024-02-05T07:43:29.898299+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:42:24.691117+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:42:24.691117+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:38.181577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"616'29","reported_seq":1104,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.685933+0000","last_change":"2024-02-05T07:42:48.671635+0000","last_active":"2024-02-05T07:43:30.685933+0000","last_peered":"2024-02-05T07:43:30.685933+0000","last_clean":"2024-02-05T07:43:30.685933+0000","last_became_active":"2024-02-05T07:41:53.751859+0000","last_became_peered":"2024-02-05T07:41:53.751859+0000","last_unstale":"2024-02-05T07:43:30.685933+0000","last_undegraded":"2024-02-05T07:43:30.685933+0000","last_fullsized":"2024-02-05T07:43:30.685933+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'29","last_scrub_stamp":"2024-02-05T07:42:48.671504+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:42:48.671504+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:02.519893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913487+0000","last_change":"2024-02-05T07:43:14.840381+0000","last_active":"2024-02-05T07:43:29.913487+0000","last_peered":"2024-02-05T07:43:29.913487+0000","last_clean":"2024-02-05T07:43:29.913487+0000","last_became_active":"2024-02-05T07:41:53.736908+0000","last_became_peered":"2024-02-05T07:41:53.736908+0000","last_unstale":"2024-02-05T07:43:29.913487+0000","last_undegraded":"2024-02-05T07:43:29.913487+0000","last_fullsized":"2024-02-05T07:43:29.913487+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:14.840237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:43:14.840237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:32.460118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913687+0000","last_change":"2024-02-05T07:42:03.128705+0000","last_active":"2024-02-05T07:43:29.913687+0000","last_peered":"2024-02-05T07:43:29.913687+0000","last_clean":"2024-02-05T07:43:29.913687+0000","last_became_active":"2024-02-05T07:41:53.766217+0000","last_became_peered":"2024-02-05T07:41:53.766217+0000","last_unstale":"2024-02-05T07:43:29.913687+0000","last_undegraded":"2024-02-05T07:43:29.913687+0000","last_fullsized":"2024-02-05T07:43:29.913687+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:03.128612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:42:03.128612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:31.984685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":413,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744097+0000","last_change":"2024-02-05T07:43:06.897431+0000","last_active":"2024-02-05T07:43:29.744097+0000","last_peered":"2024-02-05T07:43:29.744097+0000","last_clean":"2024-02-05T07:43:29.744097+0000","last_became_active":"2024-02-05T07:42:05.931798+0000","last_became_peered":"2024-02-05T07:42:05.931798+0000","last_unstale":"2024-02-05T07:43:29.744097+0000","last_undegraded":"2024-02-05T07:43:29.744097+0000","last_fullsized":"2024-02-05T07:43:29.744097+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:06.897325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:41:52.942441+0000","last_clean_scrub_stamp":"2024-02-05T07:43:06.897325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:08.569154+0000","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"615'39","reported_seq":511,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.984518+0000","last_change":"2024-02-05T07:43:09.996344+0000","last_active":"2024-02-05T07:43:29.984518+0000","last_peered":"2024-02-05T07:43:29.984518+0000","last_clean":"2024-02-05T07:43:29.984518+0000","last_became_active":"2024-02-05T07:36:32.744301+0000","last_became_peered":"2024-02-05T07:36:32.744301+0000","last_unstale":"2024-02-05T07:43:29.984518+0000","last_undegraded":"2024-02-05T07:43:29.984518+0000","last_fullsized":"2024-02-05T07:43:29.984518+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'39","last_scrub_stamp":"2024-02-05T07:43:09.996257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:09.996257+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:37.694273+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":423,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.984038+0000","last_change":"2024-02-05T07:42:42.160383+0000","last_active":"2024-02-05T07:43:29.984038+0000","last_peered":"2024-02-05T07:43:29.984038+0000","last_clean":"2024-02-05T07:43:29.984038+0000","last_became_active":"2024-02-05T07:41:53.739157+0000","last_became_peered":"2024-02-05T07:41:53.739157+0000","last_unstale":"2024-02-05T07:43:29.984038+0000","last_undegraded":"2024-02-05T07:43:29.984038+0000","last_fullsized":"2024-02-05T07:43:29.984038+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:42.160302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:42:42.160302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:59.700319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1470,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743749+0000","last_change":"2024-02-05T07:42:56.170275+0000","last_active":"2024-02-05T07:43:29.743749+0000","last_peered":"2024-02-05T07:43:29.743749+0000","last_clean":"2024-02-05T07:43:29.743749+0000","last_became_active":"2024-02-05T07:41:53.747506+0000","last_became_peered":"2024-02-05T07:41:53.747506+0000","last_unstale":"2024-02-05T07:43:29.743749+0000","last_undegraded":"2024-02-05T07:43:29.743749+0000","last_fullsized":"2024-02-05T07:43:29.743749+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:56.170120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:41:46.786494+0000","last_clean_scrub_stamp":"2024-02-05T07:42:56.170120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:15.811099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913186+0000","last_change":"2024-02-05T07:42:51.109730+0000","last_active":"2024-02-05T07:43:29.913186+0000","last_peered":"2024-02-05T07:43:29.913186+0000","last_clean":"2024-02-05T07:43:29.913186+0000","last_became_active":"2024-02-05T07:41:53.764501+0000","last_became_peered":"2024-02-05T07:41:53.764501+0000","last_unstale":"2024-02-05T07:43:29.913186+0000","last_undegraded":"2024-02-05T07:43:29.913186+0000","last_fullsized":"2024-02-05T07:43:29.913186+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:51.109628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:40:18.398592+0000","last_clean_scrub_stamp":"2024-02-05T07:42:51.109628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:02.490625+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"719'11","reported_seq":123,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913071+0000","last_change":"2024-02-05T07:43:21.923765+0000","last_active":"2024-02-05T07:43:29.913071+0000","last_peered":"2024-02-05T07:43:29.913071+0000","last_clean":"2024-02-05T07:43:29.913071+0000","last_became_active":"2024-02-05T07:41:53.754051+0000","last_became_peered":"2024-02-05T07:41:53.754051+0000","last_unstale":"2024-02-05T07:43:29.913071+0000","last_undegraded":"2024-02-05T07:43:29.913071+0000","last_fullsized":"2024-02-05T07:43:29.913071+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"719'11","last_scrub_stamp":"2024-02-05T07:43:21.923661+0000","last_deep_scrub":"719'11","last_deep_scrub_stamp":"2024-02-05T07:42:05.182833+0000","last_clean_scrub_stamp":"2024-02-05T07:43:21.923661+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:23.665787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":640,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744815+0000","last_change":"2024-02-05T07:42:41.079220+0000","last_active":"2024-02-05T07:43:29.744815+0000","last_peered":"2024-02-05T07:43:29.744815+0000","last_clean":"2024-02-05T07:43:29.744815+0000","last_became_active":"2024-02-05T07:42:36.735340+0000","last_became_peered":"2024-02-05T07:42:36.735340+0000","last_unstale":"2024-02-05T07:43:29.744815+0000","last_undegraded":"2024-02-05T07:43:29.744815+0000","last_fullsized":"2024-02-05T07:43:29.744815+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:41.079113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:42:41.079113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:05.250301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":137,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898007+0000","last_change":"2024-02-05T07:43:27.868178+0000","last_active":"2024-02-05T07:43:29.898007+0000","last_peered":"2024-02-05T07:43:29.898007+0000","last_clean":"2024-02-05T07:43:29.898007+0000","last_became_active":"2024-02-05T07:41:53.739127+0000","last_became_peered":"2024-02-05T07:41:53.739127+0000","last_unstale":"2024-02-05T07:43:29.898007+0000","last_undegraded":"2024-02-05T07:43:29.898007+0000","last_fullsized":"2024-02-05T07:43:29.898007+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:27.868035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:27.868035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:51.324777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":115,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744248+0000","last_change":"2024-02-05T07:42:57.168859+0000","last_active":"2024-02-05T07:43:29.744248+0000","last_peered":"2024-02-05T07:43:29.744248+0000","last_clean":"2024-02-05T07:43:29.744248+0000","last_became_active":"2024-02-05T07:41:49.765238+0000","last_became_peered":"2024-02-05T07:41:49.765238+0000","last_unstale":"2024-02-05T07:43:29.744248+0000","last_undegraded":"2024-02-05T07:43:29.744248+0000","last_fullsized":"2024-02-05T07:43:29.744248+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:57.168732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:57.168732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:09.273025+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"615'25","reported_seq":117,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912694+0000","last_change":"2024-02-05T07:43:07.869638+0000","last_active":"2024-02-05T07:43:29.912694+0000","last_peered":"2024-02-05T07:43:29.912694+0000","last_clean":"2024-02-05T07:43:29.912694+0000","last_became_active":"2024-02-05T07:41:53.751157+0000","last_became_peered":"2024-02-05T07:41:53.751157+0000","last_unstale":"2024-02-05T07:43:29.912694+0000","last_undegraded":"2024-02-05T07:43:29.912694+0000","last_fullsized":"2024-02-05T07:43:29.912694+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'25","last_scrub_stamp":"2024-02-05T07:43:07.869497+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:43:07.869497+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:16.515760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":629,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744798+0000","last_change":"2024-02-05T07:42:40.081185+0000","last_active":"2024-02-05T07:43:29.744798+0000","last_peered":"2024-02-05T07:43:29.744798+0000","last_clean":"2024-02-05T07:43:29.744798+0000","last_became_active":"2024-02-05T07:42:36.741191+0000","last_became_peered":"2024-02-05T07:42:36.741191+0000","last_unstale":"2024-02-05T07:43:29.744798+0000","last_undegraded":"2024-02-05T07:43:29.744798+0000","last_fullsized":"2024-02-05T07:43:29.744798+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:40.081091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:42:40.081091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:48.604227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":123,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744485+0000","last_change":"2024-02-05T07:43:18.178931+0000","last_active":"2024-02-05T07:43:29.744485+0000","last_peered":"2024-02-05T07:43:29.744485+0000","last_clean":"2024-02-05T07:43:29.744485+0000","last_became_active":"2024-02-05T07:41:49.732819+0000","last_became_peered":"2024-02-05T07:41:49.732819+0000","last_unstale":"2024-02-05T07:43:29.744485+0000","last_undegraded":"2024-02-05T07:43:29.744485+0000","last_fullsized":"2024-02-05T07:43:29.744485+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:18.178829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:18.178829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:19.583756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":478,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742369+0000","last_change":"2024-02-05T07:42:11.287040+0000","last_active":"2024-02-05T07:43:29.742369+0000","last_peered":"2024-02-05T07:43:29.742369+0000","last_clean":"2024-02-05T07:43:29.742369+0000","last_became_active":"2024-02-05T07:42:05.921956+0000","last_became_peered":"2024-02-05T07:42:05.921956+0000","last_unstale":"2024-02-05T07:43:29.742369+0000","last_undegraded":"2024-02-05T07:43:29.742369+0000","last_fullsized":"2024-02-05T07:43:29.742369+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:11.286964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:11.286964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:38.098619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"752'46","reported_seq":1136,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.686282+0000","last_change":"2024-02-05T07:43:19.464717+0000","last_active":"2024-02-05T07:43:30.686282+0000","last_peered":"2024-02-05T07:43:30.686282+0000","last_clean":"2024-02-05T07:43:30.686282+0000","last_became_active":"2024-02-05T07:36:32.741691+0000","last_became_peered":"2024-02-05T07:36:32.741691+0000","last_unstale":"2024-02-05T07:43:30.686282+0000","last_undegraded":"2024-02-05T07:43:30.686282+0000","last_fullsized":"2024-02-05T07:43:30.686282+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"752'46","last_scrub_stamp":"2024-02-05T07:43:19.464585+0000","last_deep_scrub":"752'46","last_deep_scrub_stamp":"2024-02-05T07:43:19.464585+0000","last_clean_scrub_stamp":"2024-02-05T07:43:19.464585+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:36.264092+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1051,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.687253+0000","last_change":"2024-02-05T07:43:21.483047+0000","last_active":"2024-02-05T07:43:30.687253+0000","last_peered":"2024-02-05T07:43:30.687253+0000","last_clean":"2024-02-05T07:43:30.687253+0000","last_became_active":"2024-02-05T07:36:32.746731+0000","last_became_peered":"2024-02-05T07:36:32.746731+0000","last_unstale":"2024-02-05T07:43:30.687253+0000","last_undegraded":"2024-02-05T07:43:30.687253+0000","last_fullsized":"2024-02-05T07:43:30.687253+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:21.482944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:43:21.482944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:51.309766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912660+0000","last_change":"2024-02-05T07:43:16.844544+0000","last_active":"2024-02-05T07:43:29.912660+0000","last_peered":"2024-02-05T07:43:29.912660+0000","last_clean":"2024-02-05T07:43:29.912660+0000","last_became_active":"2024-02-05T07:41:53.747813+0000","last_became_peered":"2024-02-05T07:41:53.747813+0000","last_unstale":"2024-02-05T07:43:29.912660+0000","last_undegraded":"2024-02-05T07:43:29.912660+0000","last_fullsized":"2024-02-05T07:43:29.912660+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:16.844418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:41:50.825097+0000","last_clean_scrub_stamp":"2024-02-05T07:43:16.844418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:31.959846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1049,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:30.685837+0000","last_change":"2024-02-05T07:42:36.827332+0000","last_active":"2024-02-05T07:43:30.685837+0000","last_peered":"2024-02-05T07:43:30.685837+0000","last_clean":"2024-02-05T07:43:30.685837+0000","last_became_active":"2024-02-05T07:41:45.939654+0000","last_became_peered":"2024-02-05T07:41:45.939654+0000","last_unstale":"2024-02-05T07:43:30.685837+0000","last_undegraded":"2024-02-05T07:43:30.685837+0000","last_fullsized":"2024-02-05T07:43:30.685837+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:36.827238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:38:58.711041+0000","last_clean_scrub_stamp":"2024-02-05T07:42:36.827238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:58.319667+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"615'25","reported_seq":117,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912672+0000","last_change":"2024-02-05T07:42:49.050344+0000","last_active":"2024-02-05T07:43:29.912672+0000","last_peered":"2024-02-05T07:43:29.912672+0000","last_clean":"2024-02-05T07:43:29.912672+0000","last_became_active":"2024-02-05T07:41:53.747734+0000","last_became_peered":"2024-02-05T07:41:53.747734+0000","last_unstale":"2024-02-05T07:43:29.912672+0000","last_undegraded":"2024-02-05T07:43:29.912672+0000","last_fullsized":"2024-02-05T07:43:29.912672+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'25","last_scrub_stamp":"2024-02-05T07:42:49.050261+0000","last_deep_scrub":"450'19","last_deep_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_clean_scrub_stamp":"2024-02-05T07:42:49.050261+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:55.058453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":454,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744242+0000","last_change":"2024-02-05T07:42:20.360388+0000","last_active":"2024-02-05T07:43:29.744242+0000","last_peered":"2024-02-05T07:43:29.744242+0000","last_clean":"2024-02-05T07:43:29.744242+0000","last_became_active":"2024-02-05T07:42:05.919133+0000","last_became_peered":"2024-02-05T07:42:05.919133+0000","last_unstale":"2024-02-05T07:43:29.744242+0000","last_undegraded":"2024-02-05T07:43:29.744242+0000","last_fullsized":"2024-02-05T07:43:29.744242+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:20.360311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:42:20.360311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:50.262787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":890,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742923+0000","last_change":"2024-02-05T07:43:05.105031+0000","last_active":"2024-02-05T07:43:29.742923+0000","last_peered":"2024-02-05T07:43:29.742923+0000","last_clean":"2024-02-05T07:43:29.742923+0000","last_became_active":"2024-02-05T07:42:12.072167+0000","last_became_peered":"2024-02-05T07:42:12.072167+0000","last_unstale":"2024-02-05T07:43:29.742923+0000","last_undegraded":"2024-02-05T07:43:29.742923+0000","last_fullsized":"2024-02-05T07:43:29.742923+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:05.104913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:05.104913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:28.535307+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912705+0000","last_change":"2024-02-05T07:43:12.929364+0000","last_active":"2024-02-05T07:43:29.912705+0000","last_peered":"2024-02-05T07:43:29.912705+0000","last_clean":"2024-02-05T07:43:29.912705+0000","last_became_active":"2024-02-05T07:41:53.758815+0000","last_became_peered":"2024-02-05T07:41:53.758815+0000","last_unstale":"2024-02-05T07:43:29.912705+0000","last_undegraded":"2024-02-05T07:43:29.912705+0000","last_fullsized":"2024-02-05T07:43:29.912705+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:12.929256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:43:12.929256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:13.065451+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":750,"num_read_kb":745,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":797,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1042,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1511,"num_read_kb":1268,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":617,"num_bytes_recovered":13130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1075,"ondisk_log_size":1075,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038062,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":518572,"kb_used_data":4256,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93226580,"statfs":{"total":95995035648,"available":95464017920,"internally_reserved":0,"allocated":4358144,"data_stored":2522586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":526645706},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.85199999999999998,"15min":0.89800000000000002},"min":{"1min":0.33300000000000002,"5min":0.23000000000000001,"15min":0.214},"max":{"1min":0.94599999999999995,"5min":4.7489999999999997,"15min":17.518999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":1.006,"15min":1.069},"min":{"1min":0.34000000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.85099999999999998,"5min":9.9169999999999998,"15min":17.151},"last":0.46500000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":1.5109999999999999,"15min":2.411},"min":{"1min":0.36799999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76500000000000001,"5min":36.286000000000001,"15min":65.792000000000002},"last":0.496},{"interface":"front","average":{"1min":0.53500000000000003,"5min":1.254,"15min":2.1019999999999999},"min":{"1min":0.36699999999999999,"5min":0.28199999999999997,"15min":0.27500000000000002},"max":{"1min":0.76300000000000001,"5min":36.125999999999998,"15min":65.430000000000007},"last":0.45000000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.84699999999999998,"15min":1.7909999999999999},"min":{"1min":0.38600000000000001,"5min":0.29299999999999998,"15min":0.26900000000000002},"max":{"1min":0.70899999999999996,"5min":4.5839999999999996,"15min":170.59100000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":1.2569999999999999,"15min":1.883},"min":{"1min":0.40000000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":11.401999999999999,"15min":170.25399999999999},"last":0.55100000000000005}]},{"osd":3,"last update":"Mon Feb 5 07:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":1.4550000000000001,"15min":1.948},"min":{"1min":0.39800000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.98799999999999999,"5min":36.673000000000002,"15min":61.079999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.595,"15min":2.0230000000000001},"min":{"1min":0.46999999999999997,"5min":0.28399999999999997,"15min":0.27300000000000002},"max":{"1min":0.84499999999999997,"5min":37.106000000000002,"15min":60.936999999999998},"last":0.53400000000000003}]},{"osd":4,"last update":"Mon Feb 5 07:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.433,"15min":0.71499999999999997},"min":{"1min":0.40899999999999997,"5min":0.27900000000000003,"15min":0.23000000000000001},"max":{"1min":0.69199999999999995,"5min":0.92100000000000004,"15min":72.484999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.432,"15min":0.72699999999999998},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.247},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":70.831999999999994},"last":0.42299999999999999}]},{"osd":5,"last update":"Mon Feb 5 07:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.42999999999999999,"15min":0.52600000000000002},"min":{"1min":0.34599999999999997,"5min":0.27200000000000002,"15min":0.248},"max":{"1min":0.75600000000000001,"5min":0.95299999999999996,"15min":4.0839999999999996},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.47999999999999998,"15min":0.48899999999999999},"min":{"1min":0.379,"5min":0.318,"15min":0.28999999999999998},"max":{"1min":0.74299999999999999,"5min":0.98199999999999998,"15min":1.8779999999999999},"last":0.379}]},{"osd":6,"last update":"Mon Feb 5 07:43:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.46800000000000003,"15min":0.85399999999999998},"min":{"1min":0.34499999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72399999999999998,"5min":1.405,"15min":71.091999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.504,"15min":0.83899999999999997},"min":{"1min":0.44500000000000001,"5min":0.314,"15min":0.30299999999999999},"max":{"1min":0.82099999999999995,"5min":1.0029999999999999,"15min":70.977000000000004},"last":0.40000000000000002}]}]},{"osd":6,"up_from":301,"seq":1292785156279,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":795020,"kb_used_data":4548,"kb_used_omap":5,"kb_used_meta":790458,"kb_avail":92950132,"statfs":{"total":95995035648,"available":95180935168,"internally_reserved":0,"allocated":4657152,"data_stored":2843567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":809429509},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.69199999999999995,"15min":1.022},"min":{"1min":0.32800000000000001,"5min":0.22700000000000001,"15min":0.192},"max":{"1min":0.76400000000000001,"5min":5.1959999999999997,"15min":14.218},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.71699999999999997,"15min":1.0880000000000001},"min":{"1min":0.44700000000000001,"5min":0.23200000000000001,"15min":0.22700000000000001},"max":{"1min":0.73799999999999999,"5min":5.7460000000000004,"15min":14.175000000000001},"last":0.61699999999999999}]},{"osd":1,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":1.548,"15min":1.877},"min":{"1min":0.29799999999999999,"5min":0.222,"15min":0.217},"max":{"1min":0.74399999999999999,"5min":56.325000000000003,"15min":56.325000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.6970000000000001,"15min":2.1619999999999999},"min":{"1min":0.36899999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.73199999999999998,"5min":60.115000000000002,"15min":61.103000000000002},"last":0.72099999999999997}]},{"osd":2,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":1.2010000000000001,"15min":1.669},"min":{"1min":0.40400000000000003,"5min":0.28299999999999997,"15min":0.249},"max":{"1min":0.67300000000000004,"5min":9.5109999999999992,"15min":46.927},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.93200000000000005,"15min":1.671},"min":{"1min":0.34899999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.71999999999999997,"5min":7.6479999999999997,"15min":46.506},"last":0.70599999999999996}]},{"osd":3,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":2.012,"15min":2.266},"min":{"1min":0.41099999999999998,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":0.70799999999999996,"5min":56.271000000000001,"15min":56.271000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":2.202,"15min":2.4009999999999998},"min":{"1min":0.46400000000000002,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":0.69599999999999995,"5min":56.235999999999997,"15min":56.235999999999997},"last":0.68700000000000006}]},{"osd":4,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.41699999999999998,"15min":1.0149999999999999},"min":{"1min":0.32200000000000001,"5min":0.26600000000000001,"15min":0.224},"max":{"1min":0.67600000000000005,"5min":0.68899999999999995,"15min":151.02099999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42099999999999999,"15min":1.056},"min":{"1min":0.35099999999999998,"5min":0.26300000000000001,"15min":0.22800000000000001},"max":{"1min":0.60799999999999998,"5min":0.77300000000000002,"15min":149.21799999999999},"last":0.52800000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.46899999999999997,"15min":0.54600000000000004},"min":{"1min":0.39100000000000001,"5min":0.313,"15min":0.26800000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":2.8919999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.47299999999999998,"15min":0.496},"min":{"1min":0.33000000000000002,"5min":0.28499999999999998,"15min":0.28100000000000003},"max":{"1min":0.72499999999999998,"5min":0.747,"15min":1.526},"last":0.55300000000000005}]},{"osd":7,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.46300000000000002,"15min":1.286},"min":{"1min":0.379,"5min":0.315,"15min":0.28000000000000003},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":84.158000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.48199999999999998,"15min":1.397},"min":{"1min":0.39600000000000002,"5min":0.318,"15min":0.312},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":85.018000000000001},"last":0.73999999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575180,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1153736,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":1149438,"kb_avail":92591416,"statfs":{"total":95995035648,"available":94813609984,"internally_reserved":0,"allocated":4386816,"data_stored":2530123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1177024965},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":1.51,"15min":1.8919999999999999},"min":{"1min":0.245,"5min":0.192,"15min":0.192},"max":{"1min":0.57699999999999996,"5min":15.327,"15min":29.449999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":1.601,"15min":2.1659999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.23799999999999999},"max":{"1min":0.624,"5min":8.3930000000000007,"15min":29.268999999999998},"last":0.40300000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":1.599,"15min":2.2599999999999998},"min":{"1min":0.33100000000000002,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":0.59099999999999997,"5min":12.848000000000001,"15min":42.823999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":1.246,"15min":2.0030000000000001},"min":{"1min":0.25,"5min":0.22800000000000001,"15min":0.20699999999999999},"max":{"1min":0.54900000000000004,"5min":9.8030000000000008,"15min":42.426000000000002},"last":0.53900000000000003}]},{"osd":3,"last update":"Mon Feb 5 07:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.5589999999999999,"15min":2.2149999999999999},"min":{"1min":0.28199999999999997,"5min":0.252,"15min":0.22900000000000001},"max":{"1min":0.63,"5min":9.5519999999999996,"15min":46.777000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.47199999999999998,"5min":1.327,"15min":1.982},"min":{"1min":0.36299999999999999,"5min":0.25,"15min":0.20000000000000001},"max":{"1min":0.59799999999999998,"5min":7.3899999999999997,"15min":50.164999999999999},"last":0.433}]},{"osd":4,"last update":"Mon Feb 5 07:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.79500000000000004,"15min":1.51},"min":{"1min":0.379,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.91800000000000004,"5min":4.0069999999999997,"15min":89.409000000000006},"last":0.75},{"interface":"front","average":{"1min":0.60499999999999998,"5min":1.127,"15min":1.629},"min":{"1min":0.45200000000000001,"5min":0.38900000000000001,"15min":0.28199999999999997},"max":{"1min":0.96899999999999997,"5min":12.743,"15min":89.915000000000006},"last":0.628}]},{"osd":5,"last update":"Mon Feb 5 07:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.78700000000000003,"15min":1.3340000000000001},"min":{"1min":0.436,"5min":0.32200000000000001,"15min":0.309},"max":{"1min":0.86899999999999999,"5min":3.9580000000000002,"15min":46.398000000000003},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":1.6499999999999999,"15min":1.6799999999999999},"min":{"1min":0.42499999999999999,"5min":0.34799999999999998,"15min":0.27600000000000002},"max":{"1min":0.83099999999999996,"5min":14.613,"15min":20.085999999999999},"last":0.66200000000000003}]},{"osd":6,"last update":"Mon Feb 5 07:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":1.131,"15min":1.2010000000000001},"min":{"1min":0.45900000000000002,"5min":0.36099999999999999,"15min":0.29199999999999998},"max":{"1min":0.88700000000000001,"5min":12.685,"15min":12.685},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.84199999999999997,"15min":1.3029999999999999},"min":{"1min":0.44400000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.81100000000000005,"5min":3.9300000000000002,"15min":8.923},"last":0.80900000000000005}]},{"osd":7,"last update":"Mon Feb 5 07:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":1.353,"15min":1.8029999999999999},"min":{"1min":0.41899999999999998,"5min":0.38200000000000001,"15min":0.318},"max":{"1min":0.91500000000000004,"5min":8.2439999999999998,"15min":46.351999999999997},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":1.3140000000000001,"15min":1.383},"min":{"1min":0.51200000000000001,"5min":0.40999999999999998,"15min":0.34000000000000002},"max":{"1min":0.98199999999999998,"5min":14.492000000000001,"15min":14.492000000000001},"last":0.76800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738705,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1127884,"kb_used_data":4608,"kb_used_omap":1,"kb_used_meta":1123262,"kb_avail":92617268,"statfs":{"total":95995035648,"available":94840082432,"internally_reserved":0,"allocated":4718592,"data_stored":2857415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1150220741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":2.431,"15min":2.371},"min":{"1min":0.21299999999999999,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":0.57399999999999995,"5min":65.799000000000007,"15min":65.799000000000007},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.373,"5min":2.286,"15min":2.1440000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.192},"max":{"1min":0.57599999999999996,"5min":65.093999999999994,"15min":65.093999999999994},"last":0.32700000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:42:32 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":1.4159999999999999,"15min":1.929},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.191},"max":{"1min":0.73399999999999999,"5min":7.4560000000000004,"15min":47.137999999999998},"last":0.377},{"interface":"front","average":{"1min":0.41199999999999998,"5min":1.196,"15min":1.796},"min":{"1min":0.222,"5min":0.222,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":7.9039999999999999,"15min":47.170999999999999},"last":0.29199999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":1.5469999999999999,"15min":1.8200000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.61299999999999999,"5min":22.129000000000001,"15min":29.629000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":1.7669999999999999,"15min":2.3260000000000001},"min":{"1min":0.253,"5min":0.23200000000000001,"15min":0.20899999999999999},"max":{"1min":0.71999999999999997,"5min":22.155999999999999,"15min":29.463999999999999},"last":0.312}]},{"osd":4,"last update":"Mon Feb 5 07:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":1.353,"15min":1.615},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.26700000000000002},"max":{"1min":0.753,"5min":7.8460000000000001,"15min":42.901000000000003},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.111,"15min":1.45},"min":{"1min":0.40799999999999997,"5min":0.28000000000000003,"15min":0.27100000000000002},"max":{"1min":0.78800000000000003,"5min":7.3959999999999999,"15min":42.960999999999999},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Feb 5 07:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.085,"15min":1.1000000000000001},"min":{"1min":0.38100000000000001,"5min":0.35599999999999998,"15min":0.30199999999999999},"max":{"1min":0.77800000000000002,"5min":7.492,"15min":7.492},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":1.149,"15min":1.224},"min":{"1min":0.41099999999999998,"5min":0.32900000000000001,"15min":0.28199999999999997},"max":{"1min":0.68500000000000005,"5min":7.8399999999999999,"15min":8.7599999999999998},"last":0.60199999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":2.5990000000000002,"15min":1.8120000000000001},"min":{"1min":0.40999999999999998,"5min":0.312,"15min":0.26300000000000001},"max":{"1min":0.76100000000000001,"5min":141.59899999999999,"15min":141.59899999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":2.391,"15min":1.7110000000000001},"min":{"1min":0.40600000000000003,"5min":0.376,"15min":0.35099999999999998},"max":{"1min":0.78100000000000003,"5min":141.39599999999999,"15min":141.39599999999999},"last":0.68500000000000005}]},{"osd":7,"last update":"Mon Feb 5 07:43:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.379,"15min":1.649},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.28599999999999998},"max":{"1min":0.73299999999999998,"5min":7.8200000000000003,"15min":43.012},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":1.169,"15min":1.524},"min":{"1min":0.441,"5min":0.41699999999999998,"15min":0.34000000000000002},"max":{"1min":0.82399999999999995,"5min":7.883,"15min":43.512},"last":0.626}]}]},{"osd":2,"up_from":18,"seq":77309411655,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":996388,"kb_used_data":4504,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92748764,"statfs":{"total":95995035648,"available":94974734336,"internally_reserved":0,"allocated":4612096,"data_stored":2785367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1015675333},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:42:33 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":1.1419999999999999,"15min":2.1960000000000002},"min":{"1min":0.218,"5min":0.21099999999999999,"15min":0.19600000000000001},"max":{"1min":0.71199999999999997,"5min":17.398,"15min":92.638000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.41899999999999998,"5min":1.319,"15min":2.4409999999999998},"min":{"1min":0.28199999999999997,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.626,"5min":11.583,"15min":93.481999999999999},"last":0.47299999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:42:33 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":1.611,"15min":3.2770000000000001},"min":{"1min":0.221,"5min":0.21099999999999999,"15min":0.17899999999999999},"max":{"1min":0.57499999999999996,"5min":18.135999999999999,"15min":110.917},"last":0.495},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.528,"15min":2.6819999999999999},"min":{"1min":0.29199999999999998,"5min":0.214,"15min":0.20300000000000001},"max":{"1min":0.70399999999999996,"5min":7.9509999999999996,"15min":93.004000000000005},"last":0.60099999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":1.47,"15min":2.9049999999999998},"min":{"1min":0.26100000000000001,"5min":0.247,"15min":0.214},"max":{"1min":0.625,"5min":18.338999999999999,"15min":110.773},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.46500000000000002,"5min":1.6200000000000001,"15min":2.3279999999999998},"min":{"1min":0.30599999999999999,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":0.67400000000000004,"5min":19.640000000000001,"15min":63.521000000000001},"last":0.51600000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.85299999999999998,"15min":2.2029999999999998},"min":{"1min":0.42099999999999999,"5min":0.28699999999999998,"15min":0.24299999999999999},"max":{"1min":0.81399999999999995,"5min":4.0759999999999996,"15min":168.11699999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.875,"15min":2.0009999999999999},"min":{"1min":0.433,"5min":0.33800000000000002,"15min":0.27200000000000002},"max":{"1min":0.83499999999999996,"5min":17.43,"15min":168.589},"last":0.624}]},{"osd":5,"last update":"Mon Feb 5 07:42:33 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.97499999999999998,"15min":1.0549999999999999},"min":{"1min":0.44500000000000001,"5min":0.35899999999999999,"15min":0.315},"max":{"1min":0.875,"5min":17.347999999999999,"15min":17.347999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.2410000000000001,"15min":1.5209999999999999},"min":{"1min":0.38900000000000001,"5min":0.36399999999999999,"15min":0.30499999999999999},"max":{"1min":0.90300000000000002,"5min":5.601,"15min":47.761000000000003},"last":0.72599999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":1.2490000000000001,"15min":1.7210000000000001},"min":{"1min":0.49299999999999999,"5min":0.35999999999999999,"15min":0.28799999999999998},"max":{"1min":0.95499999999999996,"5min":5.2750000000000004,"15min":47.726999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.94499999999999995,"15min":1.252},"min":{"1min":0.44600000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.99199999999999999,"5min":17.457999999999998,"15min":40.872999999999998},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:42:33 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.876,"15min":1.3660000000000001},"min":{"1min":0.442,"5min":0.36399999999999999,"15min":0.30499999999999999},"max":{"1min":1.02,"5min":4.0510000000000002,"15min":37.573999999999998},"last":0.751},{"interface":"front","average":{"1min":0.63600000000000001,"5min":1.2769999999999999,"15min":2.089},"min":{"1min":0.442,"5min":0.42299999999999999,"15min":0.33700000000000002},"max":{"1min":0.93100000000000005,"5min":5.6369999999999996,"15min":110.86799999999999},"last":0.76100000000000001}]}]},{"osd":3,"up_from":26,"seq":111669150018,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1127696,"kb_used_data":4864,"kb_used_omap":1,"kb_used_meta":1122814,"kb_avail":92617456,"statfs":{"total":95995035648,"available":94840274944,"internally_reserved":0,"allocated":4980736,"data_stored":3119457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1149761991},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":1.335,"15min":1.6379999999999999},"min":{"1min":0.28899999999999998,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.56399999999999995,"5min":12.092000000000001,"15min":48.688000000000002},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.372,"5min":1.5109999999999999,"15min":2.2010000000000001},"min":{"1min":0.23599999999999999,"5min":0.20799999999999999,"15min":0.185},"max":{"1min":0.495,"5min":8.4930000000000003,"15min":99.989999999999995},"last":0.314}]},{"osd":1,"last update":"Mon Feb 5 07:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":2.6200000000000001,"15min":2.6080000000000001},"min":{"1min":0.26400000000000001,"5min":0.23400000000000001,"15min":0.20399999999999999},"max":{"1min":0.61899999999999999,"5min":61.941000000000003,"15min":102.20399999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.433,"5min":2.556,"15min":2.5},"min":{"1min":0.309,"5min":0.28699999999999998,"15min":0.20999999999999999},"max":{"1min":0.624,"5min":61.902000000000001,"15min":102.286},"last":0.377}]},{"osd":2,"last update":"Mon Feb 5 07:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":1.4910000000000001,"15min":2.468},"min":{"1min":0.29899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.66800000000000004,"5min":12.6,"15min":145.94999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.40400000000000003,"5min":1.3839999999999999,"15min":2.5270000000000001},"min":{"1min":0.26300000000000001,"5min":0.21299999999999999,"15min":0.20200000000000001},"max":{"1min":0.55600000000000005,"5min":9.7370000000000001,"15min":146.006},"last":0.44}]},{"osd":4,"last update":"Mon Feb 5 07:43:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.997,"15min":1.7949999999999999},"min":{"1min":0.41799999999999998,"5min":0.28799999999999998,"15min":0.27000000000000002},"max":{"1min":0.82099999999999995,"5min":5.992,"15min":100.024},"last":0.622},{"interface":"front","average":{"1min":0.60799999999999998,"5min":1.2669999999999999,"15min":2.0590000000000002},"min":{"1min":0.42299999999999999,"5min":0.28899999999999998,"15min":0.27300000000000002},"max":{"1min":0.81299999999999994,"5min":6.3200000000000003,"15min":99.859999999999999},"last":0.51400000000000001}]},{"osd":5,"last update":"Mon Feb 5 07:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":1.306,"15min":1.5489999999999999},"min":{"1min":0.435,"5min":0.313,"15min":0.308},"max":{"1min":0.92500000000000004,"5min":11.507999999999999,"15min":99.911000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":1.04,"15min":1.514},"min":{"1min":0.46400000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.76100000000000001,"5min":5.9640000000000004,"15min":100.053},"last":0.52900000000000003}]},{"osd":6,"last update":"Mon Feb 5 07:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":1.3440000000000001,"15min":1.722},"min":{"1min":0.47199999999999998,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":1.034,"5min":11.563000000000001,"15min":99.807000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":1.137,"15min":1.752},"min":{"1min":0.44700000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.80600000000000005,"5min":5.9199999999999999,"15min":101.489},"last":0.58599999999999997}]},{"osd":7,"last update":"Mon Feb 5 07:43:19 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":1.2889999999999999,"15min":2.3679999999999999},"min":{"1min":0.41299999999999998,"5min":0.38200000000000001,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":6.3609999999999998,"15min":99.893000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.333,"15min":2.3149999999999999},"min":{"1min":0.52000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.97699999999999998,"5min":11.457000000000001,"15min":99.879000000000005},"last":0.55800000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953789,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":825392,"kb_used_data":4260,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92919760,"statfs":{"total":95995035648,"available":95149834240,"internally_reserved":0,"allocated":4362240,"data_stored":2522030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:43:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.0589999999999999,"15min":1.4319999999999999},"min":{"1min":0.40500000000000003,"5min":0.23499999999999999,"15min":0.20300000000000001},"max":{"1min":13.606999999999999,"5min":13.606999999999999,"15min":101.212},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":1.0529999999999999,"15min":1.4379999999999999},"min":{"1min":0.41299999999999998,"5min":0.26100000000000001,"15min":0.222},"max":{"1min":0.754,"5min":6.7350000000000003,"15min":102.07599999999999},"last":0.504}]},{"osd":1,"last update":"Mon Feb 5 07:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":1.115,"15min":1.909},"min":{"1min":0.38300000000000001,"5min":0.25800000000000001,"15min":0.253},"max":{"1min":0.72499999999999998,"5min":5.601,"15min":67.838999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":1.845,"15min":2.327},"min":{"1min":0.38100000000000001,"5min":0.26500000000000001,"15min":0.23300000000000001},"max":{"1min":0.877,"5min":32.598999999999997,"15min":68.108999999999995},"last":0.51600000000000001}]},{"osd":2,"last update":"Mon Feb 5 07:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":1.7689999999999999,"15min":2.532},"min":{"1min":0.47399999999999998,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":0.80600000000000005,"5min":22.738,"15min":180.40199999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":1.6259999999999999,"15min":2.528},"min":{"1min":0.37,"5min":0.26200000000000001,"15min":0.24299999999999999},"max":{"1min":0.76800000000000002,"5min":22.407,"15min":180.381},"last":0.47599999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":2.0819999999999999,"15min":2.3879999999999999},"min":{"1min":0.40799999999999997,"5min":0.27500000000000002,"15min":0.23799999999999999},"max":{"1min":0.72599999999999998,"5min":35.484000000000002,"15min":66.019999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":1.4870000000000001,"15min":2.335},"min":{"1min":0.41799999999999998,"5min":0.29799999999999999,"15min":0.26400000000000001},"max":{"1min":0.78700000000000003,"5min":33.344000000000001,"15min":101.001},"last":0.53000000000000003}]},{"osd":5,"last update":"Mon Feb 5 07:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.42099999999999999,"15min":0.51400000000000001},"min":{"1min":0.372,"5min":0.27800000000000002,"15min":0.24399999999999999},"max":{"1min":0.625,"5min":0.71399999999999997,"15min":10.917999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.434,"15min":0.55500000000000005},"min":{"1min":0.376,"5min":0.29699999999999999,"15min":0.24199999999999999},"max":{"1min":0.60499999999999998,"5min":0.82399999999999995,"15min":15.105},"last":0.40500000000000003}]},{"osd":6,"last update":"Mon Feb 5 07:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.44800000000000001,"15min":0.61599999999999999},"min":{"1min":0.35399999999999998,"5min":0.26100000000000001,"15min":0.245},"max":{"1min":0.70399999999999996,"5min":0.92200000000000004,"15min":14.606},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.48099999999999998,"15min":0.66900000000000004},"min":{"1min":0.41399999999999998,"5min":0.27400000000000002,"15min":0.254},"max":{"1min":0.71799999999999997,"5min":0.90200000000000002,"15min":13.114000000000001},"last":0.42799999999999999}]},{"osd":7,"last update":"Mon Feb 5 07:43:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.48399999999999999,"15min":0.625},"min":{"1min":0.41099999999999998,"5min":0.316,"15min":0.25700000000000001},"max":{"1min":0.69099999999999995,"5min":0.877,"15min":15.073},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.5,"15min":0.61599999999999999},"min":{"1min":0.39300000000000002,"5min":0.33900000000000002,"15min":0.28999999999999998},"max":{"1min":0.745,"5min":0.76500000000000001,"15min":14.581},"last":0.44600000000000001}]}]},{"osd":5,"up_from":41,"seq":176093659448,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":436916,"kb_used_data":4516,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93308236,"statfs":{"total":95995035648,"available":95547633664,"internally_reserved":0,"allocated":4624384,"data_stored":2791609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.95499999999999996,"15min":1.5409999999999999},"min":{"1min":0.28299999999999997,"5min":0.22900000000000001,"15min":0.20799999999999999},"max":{"1min":0.747,"5min":11.586,"15min":45.518000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.74399999999999999,"15min":1.4350000000000001},"min":{"1min":0.32700000000000001,"5min":0.248,"15min":0.24099999999999999},"max":{"1min":0.83899999999999997,"5min":11.234999999999999,"15min":49.914000000000001},"last":0.68999999999999995}]},{"osd":1,"last update":"Mon Feb 5 07:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":2.6899999999999999,"15min":2.7639999999999998},"min":{"1min":0.32900000000000001,"5min":0.254,"15min":0.23699999999999999},"max":{"1min":0.91700000000000004,"5min":77.477999999999994,"15min":77.477999999999994},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":2.714,"15min":2.7109999999999999},"min":{"1min":0.308,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.78400000000000003,"5min":79.176000000000002,"15min":79.176000000000002},"last":0.73099999999999998}]},{"osd":2,"last update":"Mon Feb 5 07:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":1.089,"15min":1.665},"min":{"1min":0.38,"5min":0.255,"15min":0.255},"max":{"1min":0.82099999999999995,"5min":35.546999999999997,"15min":74.126999999999995},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":1.214,"15min":1.6850000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.248},"max":{"1min":0.80200000000000005,"5min":35.896999999999998,"15min":73.688999999999993},"last":0.79000000000000004}]},{"osd":3,"last update":"Mon Feb 5 07:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":4.0999999999999996,"15min":3.3239999999999998},"min":{"1min":0.376,"5min":0.29499999999999998,"15min":0.252},"max":{"1min":0.76900000000000002,"5min":97.192999999999998,"15min":97.192999999999998},"last":0.749},{"interface":"front","average":{"1min":0.60499999999999998,"5min":3.8140000000000001,"15min":3.0670000000000002},"min":{"1min":0.33800000000000002,"5min":0.32400000000000001,"15min":0.28499999999999998},"max":{"1min":0.873,"5min":97.206999999999994,"15min":97.206999999999994},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Feb 5 07:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.42599999999999999,"15min":1.2589999999999999},"min":{"1min":0.34399999999999997,"5min":0.26500000000000001,"15min":0.22500000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":118.986},"last":0.372},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.45100000000000001,"15min":1.2829999999999999},"min":{"1min":0.31900000000000001,"5min":0.318,"15min":0.25900000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":120.893},"last":0.41599999999999998}]},{"osd":6,"last update":"Mon Feb 5 07:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.45600000000000002,"15min":1.2949999999999999},"min":{"1min":0.375,"5min":0.253,"15min":0.253},"max":{"1min":0.747,"5min":0.747,"15min":122.346},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.48599999999999999,"15min":1.3009999999999999},"min":{"1min":0.46700000000000003,"5min":0.28799999999999998,"15min":0.28599999999999998},"max":{"1min":0.749,"5min":0.89800000000000002,"15min":120.462},"last":0.66600000000000004}]},{"osd":7,"last update":"Mon Feb 5 07:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.496,"15min":1.0840000000000001},"min":{"1min":0.372,"5min":0.30399999999999999,"15min":0.222},"max":{"1min":0.76300000000000001,"5min":0.79700000000000004,"15min":125.23699999999999},"last":0.437},{"interface":"front","average":{"1min":0.628,"5min":0.50900000000000001,"15min":1.103},"min":{"1min":0.40100000000000002,"5min":0.312,"15min":0.27600000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":124.72},"last":0.51900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:43:35.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T07:43:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:43:36.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:35 smithi007 ceph-mon[29098]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:43:36.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:35 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:43:36.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:35 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:43:36.127 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:35 smithi007 ceph-mon[37158]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:43:36.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:35 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:43:36.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:35 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:43:36.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:35 smithi090 ceph-mon[33213]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:43:36.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:35 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:43:36.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:35 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:43:36.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:36 smithi007 ceph-mon[29098]: pgmap v1397: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:36.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:36 smithi007 ceph-mon[29098]: osdmap e757: 8 total, 8 up, 8 in 2024-02-05T07:43:36.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:36 smithi007 ceph-mon[29098]: 6.17 scrub starts 2024-02-05T07:43:36.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:36 smithi007 ceph-mon[29098]: 6.17 scrub ok 2024-02-05T07:43:36.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:36 smithi007 ceph-mon[37158]: pgmap v1397: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:36.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:36 smithi007 ceph-mon[37158]: osdmap e757: 8 total, 8 up, 8 in 2024-02-05T07:43:36.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:36 smithi007 ceph-mon[37158]: 6.17 scrub starts 2024-02-05T07:43:36.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:36 smithi007 ceph-mon[37158]: 6.17 scrub ok 2024-02-05T07:43:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:36 smithi090 ceph-mon[33213]: pgmap v1397: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:36 smithi090 ceph-mon[33213]: osdmap e757: 8 total, 8 up, 8 in 2024-02-05T07:43:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:36 smithi090 ceph-mon[33213]: 6.17 scrub starts 2024-02-05T07:43:37.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:36 smithi090 ceph-mon[33213]: 6.17 scrub ok 2024-02-05T07:43:37.799 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-05T07:43:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-05T07:43:38.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:37 smithi007 ceph-mon[29098]: 2.1 scrub starts 2024-02-05T07:43:38.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:37 smithi007 ceph-mon[29098]: 2.1 scrub ok 2024-02-05T07:43:38.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:37 smithi007 ceph-mon[29098]: 3.1 scrub starts 2024-02-05T07:43:38.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:37 smithi007 ceph-mon[29098]: 3.1 scrub ok 2024-02-05T07:43:38.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:37 smithi007 ceph-mon[29098]: 1.10 deep-scrub starts 2024-02-05T07:43:38.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:37 smithi007 ceph-mon[29098]: 1.10 deep-scrub ok 2024-02-05T07:43:38.100 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:37 smithi007 ceph-mon[37158]: 2.1 scrub starts 2024-02-05T07:43:38.100 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:37 smithi007 ceph-mon[37158]: 2.1 scrub ok 2024-02-05T07:43:38.100 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:37 smithi007 ceph-mon[37158]: 3.1 scrub starts 2024-02-05T07:43:38.100 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:37 smithi007 ceph-mon[37158]: 3.1 scrub ok 2024-02-05T07:43:38.100 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:37 smithi007 ceph-mon[37158]: 1.10 deep-scrub starts 2024-02-05T07:43:38.100 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:37 smithi007 ceph-mon[37158]: 1.10 deep-scrub ok 2024-02-05T07:43:38.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:37 smithi090 ceph-mon[33213]: 2.1 scrub starts 2024-02-05T07:43:38.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:37 smithi090 ceph-mon[33213]: 2.1 scrub ok 2024-02-05T07:43:38.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:37 smithi090 ceph-mon[33213]: 3.1 scrub starts 2024-02-05T07:43:38.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:37 smithi090 ceph-mon[33213]: 3.1 scrub ok 2024-02-05T07:43:38.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:37 smithi090 ceph-mon[33213]: 1.10 deep-scrub starts 2024-02-05T07:43:38.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:37 smithi090 ceph-mon[33213]: 1.10 deep-scrub ok 2024-02-05T07:43:38.427 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1399,"stamp":"2024-02-05T07:43:37.489613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2273,"num_read_kb":2025,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":669,"num_bytes_recovered":542611,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1215,"ondisk_log_size":1215,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":456,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6981656,"kb_used_data":35912,"kb_used_omap":16,"kb_used_meta":6945647,"kb_avail":742979560,"statfs":{"total":767960285184,"available":760811069440,"internally_reserved":0,"allocated":36773888,"data_stored":22012814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16785,"internal_metadata":7112343151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002260"},"pg_stats":[{"pgid":"1.9","version":"86'128","reported_seq":1297,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897843+0000","last_change":"2024-02-05T07:43:22.748182+0000","last_active":"2024-02-05T07:43:29.897843+0000","last_peered":"2024-02-05T07:43:29.897843+0000","last_clean":"2024-02-05T07:43:29.897843+0000","last_became_active":"2024-02-05T07:41:48.892873+0000","last_became_peered":"2024-02-05T07:41:48.892873+0000","last_unstale":"2024-02-05T07:43:29.897843+0000","last_undegraded":"2024-02-05T07:43:29.897843+0000","last_fullsized":"2024-02-05T07:43:29.897843+0000","mapping_epoch":706,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":707,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:22.748084+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:43:22.748084+0000","last_clean_scrub_stamp":"2024-02-05T07:43:22.748084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:51.823285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"615'35","reported_seq":127,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913793+0000","last_change":"2024-02-05T07:43:22.879048+0000","last_active":"2024-02-05T07:43:29.913793+0000","last_peered":"2024-02-05T07:43:29.913793+0000","last_clean":"2024-02-05T07:43:29.913793+0000","last_became_active":"2024-02-05T07:41:53.755648+0000","last_became_peered":"2024-02-05T07:41:53.755648+0000","last_unstale":"2024-02-05T07:43:29.913793+0000","last_undegraded":"2024-02-05T07:43:29.913793+0000","last_fullsized":"2024-02-05T07:43:29.913793+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'35","last_scrub_stamp":"2024-02-05T07:43:22.878910+0000","last_deep_scrub":"221'15","last_deep_scrub_stamp":"2024-02-05T07:25:44.205475+0000","last_clean_scrub_stamp":"2024-02-05T07:43:22.878910+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:26.132943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1040,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055987+0000","last_change":"2024-02-05T07:42:26.591966+0000","last_active":"2024-02-05T07:43:36.055987+0000","last_peered":"2024-02-05T07:43:36.055987+0000","last_clean":"2024-02-05T07:43:36.055987+0000","last_became_active":"2024-02-05T07:41:53.752202+0000","last_became_peered":"2024-02-05T07:41:53.752202+0000","last_unstale":"2024-02-05T07:43:36.055987+0000","last_undegraded":"2024-02-05T07:43:36.055987+0000","last_fullsized":"2024-02-05T07:43:36.055987+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:26.591880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:42:26.591880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:53.456483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"708'15","reported_seq":582,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.826980+0000","last_change":"2024-02-05T07:42:27.151867+0000","last_active":"2024-02-05T07:43:35.826980+0000","last_peered":"2024-02-05T07:43:35.826980+0000","last_clean":"2024-02-05T07:43:35.826980+0000","last_became_active":"2024-02-05T07:41:49.759014+0000","last_became_peered":"2024-02-05T07:41:49.759014+0000","last_unstale":"2024-02-05T07:43:35.826980+0000","last_undegraded":"2024-02-05T07:43:35.826980+0000","last_fullsized":"2024-02-05T07:43:35.826980+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"708'15","last_scrub_stamp":"2024-02-05T07:42:27.151770+0000","last_deep_scrub":"363'10","last_deep_scrub_stamp":"2024-02-05T07:30:38.946065+0000","last_clean_scrub_stamp":"2024-02-05T07:42:27.151770+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913157+0000","last_change":"2024-02-05T07:42:42.170041+0000","last_active":"2024-02-05T07:43:29.913157+0000","last_peered":"2024-02-05T07:43:29.913157+0000","last_clean":"2024-02-05T07:43:29.913157+0000","last_became_active":"2024-02-05T07:41:53.747848+0000","last_became_peered":"2024-02-05T07:41:53.747848+0000","last_unstale":"2024-02-05T07:43:29.913157+0000","last_undegraded":"2024-02-05T07:43:29.913157+0000","last_fullsized":"2024-02-05T07:43:29.913157+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:42.169959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:43.273019+0000","last_clean_scrub_stamp":"2024-02-05T07:42:42.169959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:47.324283+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":934,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743996+0000","last_change":"2024-02-05T07:42:54.112250+0000","last_active":"2024-02-05T07:43:29.743996+0000","last_peered":"2024-02-05T07:43:29.743996+0000","last_clean":"2024-02-05T07:43:29.743996+0000","last_became_active":"2024-02-05T07:42:36.737646+0000","last_became_peered":"2024-02-05T07:42:36.737646+0000","last_unstale":"2024-02-05T07:43:29.743996+0000","last_undegraded":"2024-02-05T07:43:29.743996+0000","last_fullsized":"2024-02-05T07:43:29.743996+0000","mapping_epoch":741,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:54.112157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:54.112157+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":138,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:11.718443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"616'45","reported_seq":133,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912856+0000","last_change":"2024-02-05T07:42:22.221425+0000","last_active":"2024-02-05T07:43:29.912856+0000","last_peered":"2024-02-05T07:43:29.912856+0000","last_clean":"2024-02-05T07:43:29.912856+0000","last_became_active":"2024-02-05T07:41:53.748344+0000","last_became_peered":"2024-02-05T07:41:53.748344+0000","last_unstale":"2024-02-05T07:43:29.912856+0000","last_undegraded":"2024-02-05T07:43:29.912856+0000","last_fullsized":"2024-02-05T07:43:29.912856+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'45","last_scrub_stamp":"2024-02-05T07:42:22.221306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:42:22.221306+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:44.464889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1318,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.764721+0000","last_change":"2024-02-05T07:42:32.638969+0000","last_active":"2024-02-05T07:43:29.764721+0000","last_peered":"2024-02-05T07:43:29.764721+0000","last_clean":"2024-02-05T07:43:29.764721+0000","last_became_active":"2024-02-05T07:41:53.744933+0000","last_became_peered":"2024-02-05T07:41:53.744933+0000","last_unstale":"2024-02-05T07:43:29.764721+0000","last_undegraded":"2024-02-05T07:43:29.764721+0000","last_fullsized":"2024-02-05T07:43:29.764721+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:32.638874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:32.638874+0000","last_clean_scrub_stamp":"2024-02-05T07:42:32.638874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:45.225778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":878,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742214+0000","last_change":"2024-02-05T07:42:22.148309+0000","last_active":"2024-02-05T07:43:29.742214+0000","last_peered":"2024-02-05T07:43:29.742214+0000","last_clean":"2024-02-05T07:43:29.742214+0000","last_became_active":"2024-02-05T07:42:11.957630+0000","last_became_peered":"2024-02-05T07:42:11.957630+0000","last_unstale":"2024-02-05T07:43:29.742214+0000","last_undegraded":"2024-02-05T07:43:29.742214+0000","last_fullsized":"2024-02-05T07:43:29.742214+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:22.148187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:42:22.148187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:45.099164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":484,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.827266+0000","last_change":"2024-02-05T07:42:25.428561+0000","last_active":"2024-02-05T07:43:35.827266+0000","last_peered":"2024-02-05T07:43:35.827266+0000","last_clean":"2024-02-05T07:43:35.827266+0000","last_became_active":"2024-02-05T07:41:53.747418+0000","last_became_peered":"2024-02-05T07:41:53.747418+0000","last_unstale":"2024-02-05T07:43:35.827266+0000","last_undegraded":"2024-02-05T07:43:35.827266+0000","last_fullsized":"2024-02-05T07:43:35.827266+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:25.428462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:25.428462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"86'128","reported_seq":888,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743260+0000","last_change":"2024-02-05T07:42:13.115144+0000","last_active":"2024-02-05T07:43:29.743260+0000","last_peered":"2024-02-05T07:43:29.743260+0000","last_clean":"2024-02-05T07:43:29.743260+0000","last_became_active":"2024-02-05T07:42:11.959147+0000","last_became_peered":"2024-02-05T07:42:11.959147+0000","last_unstale":"2024-02-05T07:43:29.743260+0000","last_undegraded":"2024-02-05T07:43:29.743260+0000","last_fullsized":"2024-02-05T07:43:29.743260+0000","mapping_epoch":723,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:13.114996+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:42:13.114996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:34.990635+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"616'32","reported_seq":119,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912837+0000","last_change":"2024-02-05T07:42:36.129167+0000","last_active":"2024-02-05T07:43:29.912837+0000","last_peered":"2024-02-05T07:43:29.912837+0000","last_clean":"2024-02-05T07:43:29.912837+0000","last_became_active":"2024-02-05T07:41:53.755063+0000","last_became_peered":"2024-02-05T07:41:53.755063+0000","last_unstale":"2024-02-05T07:43:29.912837+0000","last_undegraded":"2024-02-05T07:43:29.912837+0000","last_fullsized":"2024-02-05T07:43:29.912837+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'32","last_scrub_stamp":"2024-02-05T07:42:36.129021+0000","last_deep_scrub":"90'16","last_deep_scrub_stamp":"2024-02-05T07:20:09.873388+0000","last_clean_scrub_stamp":"2024-02-05T07:42:36.129021+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:03.237577+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913186+0000","last_change":"2024-02-05T07:42:54.971153+0000","last_active":"2024-02-05T07:43:29.913186+0000","last_peered":"2024-02-05T07:43:29.913186+0000","last_clean":"2024-02-05T07:43:29.913186+0000","last_became_active":"2024-02-05T07:41:53.747999+0000","last_became_peered":"2024-02-05T07:41:53.747999+0000","last_unstale":"2024-02-05T07:43:29.913186+0000","last_undegraded":"2024-02-05T07:43:29.913186+0000","last_fullsized":"2024-02-05T07:43:29.913186+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:54.971057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:46.149431+0000","last_clean_scrub_stamp":"2024-02-05T07:42:54.971057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:13.720219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1127,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.745076+0000","last_change":"2024-02-05T07:42:38.151143+0000","last_active":"2024-02-05T07:43:29.745076+0000","last_peered":"2024-02-05T07:43:29.745076+0000","last_clean":"2024-02-05T07:43:29.745076+0000","last_became_active":"2024-02-05T07:42:36.743340+0000","last_became_peered":"2024-02-05T07:42:36.743340+0000","last_unstale":"2024-02-05T07:43:29.745076+0000","last_undegraded":"2024-02-05T07:43:29.745076+0000","last_fullsized":"2024-02-05T07:43:29.745076+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:38.151005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:38.151005+0000","last_clean_scrub_stamp":"2024-02-05T07:42:38.151005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:02.208451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1049,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055739+0000","last_change":"2024-02-05T07:42:10.547377+0000","last_active":"2024-02-05T07:43:36.055739+0000","last_peered":"2024-02-05T07:43:36.055739+0000","last_clean":"2024-02-05T07:43:36.055739+0000","last_became_active":"2024-02-05T07:41:49.749351+0000","last_became_peered":"2024-02-05T07:41:49.749351+0000","last_unstale":"2024-02-05T07:43:36.055739+0000","last_undegraded":"2024-02-05T07:43:36.055739+0000","last_fullsized":"2024-02-05T07:43:36.055739+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:10.547297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:10.547297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:37.447504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":883,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744490+0000","last_change":"2024-02-05T07:42:39.115312+0000","last_active":"2024-02-05T07:43:29.744490+0000","last_peered":"2024-02-05T07:43:29.744490+0000","last_clean":"2024-02-05T07:43:29.744490+0000","last_became_active":"2024-02-05T07:42:38.753738+0000","last_became_peered":"2024-02-05T07:42:38.753738+0000","last_unstale":"2024-02-05T07:43:29.744490+0000","last_undegraded":"2024-02-05T07:43:29.744490+0000","last_fullsized":"2024-02-05T07:43:29.744490+0000","mapping_epoch":743,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":744,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:39.115200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:39.115200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:03.322489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"616'46","reported_seq":1635,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743221+0000","last_change":"2024-02-05T07:43:02.155276+0000","last_active":"2024-02-05T07:43:29.743221+0000","last_peered":"2024-02-05T07:43:29.743221+0000","last_clean":"2024-02-05T07:43:29.743221+0000","last_became_active":"2024-02-05T07:41:53.749695+0000","last_became_peered":"2024-02-05T07:41:53.749695+0000","last_unstale":"2024-02-05T07:43:29.743221+0000","last_undegraded":"2024-02-05T07:43:29.743221+0000","last_fullsized":"2024-02-05T07:43:29.743221+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'46","last_scrub_stamp":"2024-02-05T07:43:02.155154+0000","last_deep_scrub":"98'22","last_deep_scrub_stamp":"2024-02-05T07:20:36.466914+0000","last_clean_scrub_stamp":"2024-02-05T07:43:02.155154+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:13.933013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1054,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055370+0000","last_change":"2024-02-05T07:43:09.511558+0000","last_active":"2024-02-05T07:43:36.055370+0000","last_peered":"2024-02-05T07:43:36.055370+0000","last_clean":"2024-02-05T07:43:36.055370+0000","last_became_active":"2024-02-05T07:41:45.938795+0000","last_became_peered":"2024-02-05T07:41:45.938795+0000","last_unstale":"2024-02-05T07:43:36.055370+0000","last_undegraded":"2024-02-05T07:43:36.055370+0000","last_fullsized":"2024-02-05T07:43:36.055370+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:09.511469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:43:09.511469+0000","last_clean_scrub_stamp":"2024-02-05T07:43:09.511469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:10.937664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"713'14","reported_seq":132,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913044+0000","last_change":"2024-02-05T07:43:06.868299+0000","last_active":"2024-02-05T07:43:29.913044+0000","last_peered":"2024-02-05T07:43:29.913044+0000","last_clean":"2024-02-05T07:43:29.913044+0000","last_became_active":"2024-02-05T07:41:53.759990+0000","last_became_peered":"2024-02-05T07:41:53.759990+0000","last_unstale":"2024-02-05T07:43:29.913044+0000","last_undegraded":"2024-02-05T07:43:29.913044+0000","last_fullsized":"2024-02-05T07:43:29.913044+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'14","last_scrub_stamp":"2024-02-05T07:43:06.868200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:06.868200+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:29.648510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":115,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.745071+0000","last_change":"2024-02-05T07:42:13.176258+0000","last_active":"2024-02-05T07:43:29.745071+0000","last_peered":"2024-02-05T07:43:29.745071+0000","last_clean":"2024-02-05T07:43:29.745071+0000","last_became_active":"2024-02-05T07:41:49.742477+0000","last_became_peered":"2024-02-05T07:41:49.742477+0000","last_unstale":"2024-02-05T07:43:29.745071+0000","last_undegraded":"2024-02-05T07:43:29.745071+0000","last_fullsized":"2024-02-05T07:43:29.745071+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:13.176166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:13.176166+0000","last_clean_scrub_stamp":"2024-02-05T07:42:13.176166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:34.406372+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":1183,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.914126+0000","last_change":"2024-02-05T07:43:15.807217+0000","last_active":"2024-02-05T07:43:29.914126+0000","last_peered":"2024-02-05T07:43:29.914126+0000","last_clean":"2024-02-05T07:43:29.914126+0000","last_became_active":"2024-02-05T07:41:56.731867+0000","last_became_peered":"2024-02-05T07:41:56.731867+0000","last_unstale":"2024-02-05T07:43:29.914126+0000","last_undegraded":"2024-02-05T07:43:29.914126+0000","last_fullsized":"2024-02-05T07:43:29.914126+0000","mapping_epoch":714,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:15.807100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:43:15.807100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:34.256250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":115,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913352+0000","last_change":"2024-02-05T07:43:24.795918+0000","last_active":"2024-02-05T07:43:29.913352+0000","last_peered":"2024-02-05T07:43:29.913352+0000","last_clean":"2024-02-05T07:43:29.913352+0000","last_became_active":"2024-02-05T07:41:53.753278+0000","last_became_peered":"2024-02-05T07:41:53.753278+0000","last_unstale":"2024-02-05T07:43:29.913352+0000","last_undegraded":"2024-02-05T07:43:29.913352+0000","last_fullsized":"2024-02-05T07:43:29.913352+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:24.795825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:14.198146+0000","last_clean_scrub_stamp":"2024-02-05T07:43:24.795825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:48.247803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"615'50","reported_seq":1147,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055330+0000","last_change":"2024-02-05T07:42:47.698407+0000","last_active":"2024-02-05T07:43:36.055330+0000","last_peered":"2024-02-05T07:43:36.055330+0000","last_clean":"2024-02-05T07:43:36.055330+0000","last_became_active":"2024-02-05T07:41:45.939280+0000","last_became_peered":"2024-02-05T07:41:45.939280+0000","last_unstale":"2024-02-05T07:43:36.055330+0000","last_undegraded":"2024-02-05T07:43:36.055330+0000","last_fullsized":"2024-02-05T07:43:36.055330+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'50","last_scrub_stamp":"2024-02-05T07:42:47.698301+0000","last_deep_scrub":"454'38","last_deep_scrub_stamp":"2024-02-05T07:34:38.942830+0000","last_clean_scrub_stamp":"2024-02-05T07:42:47.698301+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:07.731167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1046,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.056507+0000","last_change":"2024-02-05T07:42:44.718432+0000","last_active":"2024-02-05T07:43:36.056507+0000","last_peered":"2024-02-05T07:43:36.056507+0000","last_clean":"2024-02-05T07:43:36.056507+0000","last_became_active":"2024-02-05T07:41:45.943736+0000","last_became_peered":"2024-02-05T07:41:45.943736+0000","last_unstale":"2024-02-05T07:43:36.056507+0000","last_undegraded":"2024-02-05T07:43:36.056507+0000","last_fullsized":"2024-02-05T07:43:36.056507+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:44.718337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:44.718337+0000","last_clean_scrub_stamp":"2024-02-05T07:42:44.718337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:46.006994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913558+0000","last_change":"2024-02-05T07:42:34.098540+0000","last_active":"2024-02-05T07:43:29.913558+0000","last_peered":"2024-02-05T07:43:29.913558+0000","last_clean":"2024-02-05T07:43:29.913558+0000","last_became_active":"2024-02-05T07:41:53.749709+0000","last_became_peered":"2024-02-05T07:41:53.749709+0000","last_unstale":"2024-02-05T07:43:29.913558+0000","last_undegraded":"2024-02-05T07:43:29.913558+0000","last_fullsized":"2024-02-05T07:43:29.913558+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:34.098427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:28:56.783168+0000","last_clean_scrub_stamp":"2024-02-05T07:42:34.098427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:57.922534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":1839,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055840+0000","last_change":"2024-02-05T07:43:33.719248+0000","last_active":"2024-02-05T07:43:36.055840+0000","last_peered":"2024-02-05T07:43:36.055840+0000","last_clean":"2024-02-05T07:43:36.055840+0000","last_became_active":"2024-02-05T07:41:52.748451+0000","last_became_peered":"2024-02-05T07:41:52.748451+0000","last_unstale":"2024-02-05T07:43:36.055840+0000","last_undegraded":"2024-02-05T07:43:36.055840+0000","last_fullsized":"2024-02-05T07:43:36.055840+0000","mapping_epoch":710,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:33.719153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:43:33.719153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:55.071579+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"713'12","reported_seq":1250,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055438+0000","last_change":"2024-02-05T07:42:19.555579+0000","last_active":"2024-02-05T07:43:36.055438+0000","last_peered":"2024-02-05T07:43:36.055438+0000","last_clean":"2024-02-05T07:43:36.055438+0000","last_became_active":"2024-02-05T07:41:53.737961+0000","last_became_peered":"2024-02-05T07:41:53.737961+0000","last_unstale":"2024-02-05T07:43:36.055438+0000","last_undegraded":"2024-02-05T07:43:36.055438+0000","last_fullsized":"2024-02-05T07:43:36.055438+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'12","last_scrub_stamp":"2024-02-05T07:42:19.555495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:42:19.555495+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:40.583073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"616'49","reported_seq":123,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912685+0000","last_change":"2024-02-05T07:42:53.016225+0000","last_active":"2024-02-05T07:43:29.912685+0000","last_peered":"2024-02-05T07:43:29.912685+0000","last_clean":"2024-02-05T07:43:29.912685+0000","last_became_active":"2024-02-05T07:41:53.758571+0000","last_became_peered":"2024-02-05T07:41:53.758571+0000","last_unstale":"2024-02-05T07:43:29.912685+0000","last_undegraded":"2024-02-05T07:43:29.912685+0000","last_fullsized":"2024-02-05T07:43:29.912685+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'49","last_scrub_stamp":"2024-02-05T07:42:53.016112+0000","last_deep_scrub":"202'21","last_deep_scrub_stamp":"2024-02-05T07:24:59.870542+0000","last_clean_scrub_stamp":"2024-02-05T07:42:53.016112+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:53.238724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"639'2","reported_seq":1070,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053930+0000","last_change":"2024-02-05T07:42:31.380189+0000","last_active":"2024-02-05T07:43:36.053930+0000","last_peered":"2024-02-05T07:43:36.053930+0000","last_clean":"2024-02-05T07:43:36.053930+0000","last_became_active":"2024-02-05T07:42:05.923781+0000","last_became_peered":"2024-02-05T07:42:05.923781+0000","last_unstale":"2024-02-05T07:43:36.053930+0000","last_undegraded":"2024-02-05T07:43:36.053930+0000","last_fullsized":"2024-02-05T07:43:36.053930+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'2","last_scrub_stamp":"2024-02-05T07:42:31.380092+0000","last_deep_scrub":"639'2","last_deep_scrub_stamp":"2024-02-05T07:40:13.213477+0000","last_clean_scrub_stamp":"2024-02-05T07:42:31.380092+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:38.380712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":123,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897470+0000","last_change":"2024-02-05T07:41:55.764320+0000","last_active":"2024-02-05T07:43:29.897470+0000","last_peered":"2024-02-05T07:43:29.897470+0000","last_clean":"2024-02-05T07:43:29.897470+0000","last_became_active":"2024-02-05T07:41:45.860439+0000","last_became_peered":"2024-02-05T07:41:45.860439+0000","last_unstale":"2024-02-05T07:43:29.897470+0000","last_undegraded":"2024-02-05T07:43:29.897470+0000","last_fullsized":"2024-02-05T07:43:29.897470+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:55.764170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:37:58.737033+0000","last_clean_scrub_stamp":"2024-02-05T07:41:55.764170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":1468,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.765895+0000","last_change":"2024-02-05T07:42:39.684922+0000","last_active":"2024-02-05T07:43:29.765895+0000","last_peered":"2024-02-05T07:43:29.765895+0000","last_clean":"2024-02-05T07:43:29.765895+0000","last_became_active":"2024-02-05T07:42:19.004211+0000","last_became_peered":"2024-02-05T07:42:19.004211+0000","last_unstale":"2024-02-05T07:43:29.765895+0000","last_undegraded":"2024-02-05T07:43:29.765895+0000","last_fullsized":"2024-02-05T07:43:29.765895+0000","mapping_epoch":729,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:39.684799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:39.684799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:41.196649+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"722'17","reported_seq":434,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053830+0000","last_change":"2024-02-05T07:43:03.674585+0000","last_active":"2024-02-05T07:43:36.053830+0000","last_peered":"2024-02-05T07:43:36.053830+0000","last_clean":"2024-02-05T07:43:36.053830+0000","last_became_active":"2024-02-05T07:42:05.931950+0000","last_became_peered":"2024-02-05T07:42:05.931950+0000","last_unstale":"2024-02-05T07:43:36.053830+0000","last_undegraded":"2024-02-05T07:43:36.053830+0000","last_fullsized":"2024-02-05T07:43:36.053830+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'17","last_scrub_stamp":"2024-02-05T07:43:03.674431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:03.674431+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:29.089732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"616'35","reported_seq":65,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.826605+0000","last_change":"2024-02-05T07:42:58.931836+0000","last_active":"2024-02-05T07:43:35.826605+0000","last_peered":"2024-02-05T07:43:35.826605+0000","last_clean":"2024-02-05T07:43:35.826605+0000","last_became_active":"2024-02-05T07:42:32.202250+0000","last_became_peered":"2024-02-05T07:42:32.202250+0000","last_unstale":"2024-02-05T07:43:35.826605+0000","last_undegraded":"2024-02-05T07:43:35.826605+0000","last_fullsized":"2024-02-05T07:43:35.826605+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'35","last_scrub_stamp":"2024-02-05T07:42:58.931730+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-02-05T07:23:26.247047+0000","last_clean_scrub_stamp":"2024-02-05T07:42:58.931730+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:28.186531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":115,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744947+0000","last_change":"2024-02-05T07:42:56.041305+0000","last_active":"2024-02-05T07:43:29.744947+0000","last_peered":"2024-02-05T07:43:29.744947+0000","last_clean":"2024-02-05T07:43:29.744947+0000","last_became_active":"2024-02-05T07:41:49.764275+0000","last_became_peered":"2024-02-05T07:41:49.764275+0000","last_unstale":"2024-02-05T07:43:29.744947+0000","last_undegraded":"2024-02-05T07:43:29.744947+0000","last_fullsized":"2024-02-05T07:43:29.744947+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:56.041211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:32.197687+0000","last_clean_scrub_stamp":"2024-02-05T07:42:56.041211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:23.083246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":742,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.741927+0000","last_change":"2024-02-05T07:43:07.136513+0000","last_active":"2024-02-05T07:43:29.741927+0000","last_peered":"2024-02-05T07:43:29.741927+0000","last_clean":"2024-02-05T07:43:29.741927+0000","last_became_active":"2024-02-05T07:32:58.598709+0000","last_became_peered":"2024-02-05T07:32:58.598709+0000","last_unstale":"2024-02-05T07:43:29.741927+0000","last_undegraded":"2024-02-05T07:43:29.741927+0000","last_fullsized":"2024-02-05T07:43:29.741927+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:07.136433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:43:07.136433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:32.604100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":1421,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.765666+0000","last_change":"2024-02-05T07:42:30.657385+0000","last_active":"2024-02-05T07:43:29.765666+0000","last_peered":"2024-02-05T07:43:29.765666+0000","last_clean":"2024-02-05T07:43:29.765666+0000","last_became_active":"2024-02-05T07:40:52.907985+0000","last_became_peered":"2024-02-05T07:40:52.907985+0000","last_unstale":"2024-02-05T07:43:29.765666+0000","last_undegraded":"2024-02-05T07:43:29.765666+0000","last_fullsized":"2024-02-05T07:43:29.765666+0000","mapping_epoch":302,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:30.657284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:30.657284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:51.264922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1319,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.765289+0000","last_change":"2024-02-05T07:42:50.752034+0000","last_active":"2024-02-05T07:43:29.765289+0000","last_peered":"2024-02-05T07:43:29.765289+0000","last_clean":"2024-02-05T07:43:29.765289+0000","last_became_active":"2024-02-05T07:41:49.735919+0000","last_became_peered":"2024-02-05T07:41:49.735919+0000","last_unstale":"2024-02-05T07:43:29.765289+0000","last_undegraded":"2024-02-05T07:43:29.765289+0000","last_fullsized":"2024-02-05T07:43:29.765289+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:50.751946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:37.374703+0000","last_clean_scrub_stamp":"2024-02-05T07:42:50.751946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:03.358026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"615'39","reported_seq":942,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.054325+0000","last_change":"2024-02-05T07:42:46.404152+0000","last_active":"2024-02-05T07:43:36.054325+0000","last_peered":"2024-02-05T07:43:36.054325+0000","last_clean":"2024-02-05T07:43:36.054325+0000","last_became_active":"2024-02-05T07:42:05.921622+0000","last_became_peered":"2024-02-05T07:42:05.921622+0000","last_unstale":"2024-02-05T07:43:36.054325+0000","last_undegraded":"2024-02-05T07:43:36.054325+0000","last_fullsized":"2024-02-05T07:43:36.054325+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'39","last_scrub_stamp":"2024-02-05T07:42:46.404018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:42:46.404018+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:46.499030+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":66,"num_read_kb":52,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":893,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742912+0000","last_change":"2024-02-05T07:42:26.164849+0000","last_active":"2024-02-05T07:43:29.742912+0000","last_peered":"2024-02-05T07:43:29.742912+0000","last_clean":"2024-02-05T07:43:29.742912+0000","last_became_active":"2024-02-05T07:42:11.957822+0000","last_became_peered":"2024-02-05T07:42:11.957822+0000","last_unstale":"2024-02-05T07:43:29.742912+0000","last_undegraded":"2024-02-05T07:43:29.742912+0000","last_fullsized":"2024-02-05T07:43:29.742912+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:26.164774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:10.436535+0000","last_clean_scrub_stamp":"2024-02-05T07:42:26.164774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:55.863395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1471,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743029+0000","last_change":"2024-02-05T07:43:19.199842+0000","last_active":"2024-02-05T07:43:29.743029+0000","last_peered":"2024-02-05T07:43:29.743029+0000","last_clean":"2024-02-05T07:43:29.743029+0000","last_became_active":"2024-02-05T07:36:32.709872+0000","last_became_peered":"2024-02-05T07:36:32.709872+0000","last_unstale":"2024-02-05T07:43:29.743029+0000","last_undegraded":"2024-02-05T07:43:29.743029+0000","last_fullsized":"2024-02-05T07:43:29.743029+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:19.199710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:43:19.199710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:32.763255+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":880,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743208+0000","last_change":"2024-02-05T07:43:21.092918+0000","last_active":"2024-02-05T07:43:29.743208+0000","last_peered":"2024-02-05T07:43:29.743208+0000","last_clean":"2024-02-05T07:43:29.743208+0000","last_became_active":"2024-02-05T07:42:11.957957+0000","last_became_peered":"2024-02-05T07:42:11.957957+0000","last_unstale":"2024-02-05T07:43:29.743208+0000","last_undegraded":"2024-02-05T07:43:29.743208+0000","last_fullsized":"2024-02-05T07:43:29.743208+0000","mapping_epoch":723,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:21.092817+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:43:21.092817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:29.874000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":123,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:32.208888+0000","last_change":"2024-02-05T07:43:32.208913+0000","last_active":"2024-02-05T07:43:32.208888+0000","last_peered":"2024-02-05T07:43:32.208888+0000","last_clean":"2024-02-05T07:43:32.208888+0000","last_became_active":"2024-02-05T07:41:49.741714+0000","last_became_peered":"2024-02-05T07:41:49.741714+0000","last_unstale":"2024-02-05T07:43:32.208888+0000","last_undegraded":"2024-02-05T07:43:32.208888+0000","last_fullsized":"2024-02-05T07:43:32.208888+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:32.208820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:06.098389+0000","last_clean_scrub_stamp":"2024-02-05T07:43:32.208820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:57.436409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"615'36","reported_seq":302,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053819+0000","last_change":"2024-02-05T07:42:27.433721+0000","last_active":"2024-02-05T07:43:36.053819+0000","last_peered":"2024-02-05T07:43:36.053819+0000","last_clean":"2024-02-05T07:43:36.053819+0000","last_became_active":"2024-02-05T07:42:05.922131+0000","last_became_peered":"2024-02-05T07:42:05.922131+0000","last_unstale":"2024-02-05T07:43:36.053819+0000","last_undegraded":"2024-02-05T07:43:36.053819+0000","last_fullsized":"2024-02-05T07:43:36.053819+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'36","last_scrub_stamp":"2024-02-05T07:42:27.433619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:42:27.433619+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:57.072440+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":45,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"695'6","reported_seq":237,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:34.048870+0000","last_change":"2024-02-05T07:42:46.622715+0000","last_active":"2024-02-05T07:43:34.048870+0000","last_peered":"2024-02-05T07:43:34.048870+0000","last_clean":"2024-02-05T07:43:34.048870+0000","last_became_active":"2024-02-05T07:41:45.883810+0000","last_became_peered":"2024-02-05T07:41:45.883810+0000","last_unstale":"2024-02-05T07:43:34.048870+0000","last_undegraded":"2024-02-05T07:43:34.048870+0000","last_fullsized":"2024-02-05T07:43:34.048870+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'6","last_scrub_stamp":"2024-02-05T07:42:46.622612+0000","last_deep_scrub":"695'6","last_deep_scrub_stamp":"2024-02-05T07:41:22.536369+0000","last_clean_scrub_stamp":"2024-02-05T07:42:46.622612+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:14.702968+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001202961,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":755,"num_read_kb":750,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":749,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053431+0000","last_change":"2024-02-05T07:42:14.291981+0000","last_active":"2024-02-05T07:43:36.053431+0000","last_peered":"2024-02-05T07:43:36.053431+0000","last_clean":"2024-02-05T07:43:36.053431+0000","last_became_active":"2024-02-05T07:42:05.930992+0000","last_became_peered":"2024-02-05T07:42:05.930992+0000","last_unstale":"2024-02-05T07:43:36.053431+0000","last_undegraded":"2024-02-05T07:43:36.053431+0000","last_fullsized":"2024-02-05T07:43:36.053431+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:14.291888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:14.291888+0000","last_clean_scrub_stamp":"2024-02-05T07:42:14.291888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"75'1","reported_seq":1497,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742987+0000","last_change":"2024-02-05T07:43:06.107976+0000","last_active":"2024-02-05T07:43:29.742987+0000","last_peered":"2024-02-05T07:43:29.742987+0000","last_clean":"2024-02-05T07:43:29.742987+0000","last_became_active":"2024-02-05T07:41:49.749990+0000","last_became_peered":"2024-02-05T07:41:49.749990+0000","last_unstale":"2024-02-05T07:43:29.742987+0000","last_undegraded":"2024-02-05T07:43:29.742987+0000","last_fullsized":"2024-02-05T07:43:29.742987+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-02-05T07:43:06.107882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:43:06.107882+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:08.624572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":137,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898180+0000","last_change":"2024-02-05T07:43:17.778996+0000","last_active":"2024-02-05T07:43:29.898180+0000","last_peered":"2024-02-05T07:43:29.898180+0000","last_clean":"2024-02-05T07:43:29.898180+0000","last_became_active":"2024-02-05T07:41:49.729965+0000","last_became_peered":"2024-02-05T07:41:49.729965+0000","last_unstale":"2024-02-05T07:43:29.898180+0000","last_undegraded":"2024-02-05T07:43:29.898180+0000","last_fullsized":"2024-02-05T07:43:29.898180+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:17.778900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:17.778900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:45.044357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"713'46","reported_seq":1730,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.765673+0000","last_change":"2024-02-05T07:43:02.681388+0000","last_active":"2024-02-05T07:43:29.765673+0000","last_peered":"2024-02-05T07:43:29.765673+0000","last_clean":"2024-02-05T07:43:29.765673+0000","last_became_active":"2024-02-05T07:41:53.745616+0000","last_became_peered":"2024-02-05T07:41:53.745616+0000","last_unstale":"2024-02-05T07:43:29.765673+0000","last_undegraded":"2024-02-05T07:43:29.765673+0000","last_fullsized":"2024-02-05T07:43:29.765673+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'46","last_scrub_stamp":"2024-02-05T07:43:02.681256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:02.681256+0000","objects_scrubbed":8,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:13.499647+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":343,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.826650+0000","last_change":"2024-02-05T07:42:29.206268+0000","last_active":"2024-02-05T07:43:35.826650+0000","last_peered":"2024-02-05T07:43:35.826650+0000","last_clean":"2024-02-05T07:43:35.826650+0000","last_became_active":"2024-02-05T07:41:49.736707+0000","last_became_peered":"2024-02-05T07:41:49.736707+0000","last_unstale":"2024-02-05T07:43:35.826650+0000","last_undegraded":"2024-02-05T07:43:35.826650+0000","last_fullsized":"2024-02-05T07:43:35.826650+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:29.206178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:42:29.206178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:36.155973+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":439,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897874+0000","last_change":"2024-02-05T07:42:49.691175+0000","last_active":"2024-02-05T07:43:29.897874+0000","last_peered":"2024-02-05T07:43:29.897874+0000","last_clean":"2024-02-05T07:43:29.897874+0000","last_became_active":"2024-02-05T07:42:48.805828+0000","last_became_peered":"2024-02-05T07:42:48.805828+0000","last_unstale":"2024-02-05T07:43:29.897874+0000","last_undegraded":"2024-02-05T07:43:29.897874+0000","last_fullsized":"2024-02-05T07:43:29.897874+0000","mapping_epoch":748,"log_start":"86'128","ondisk_log_start":"86'128","created":20,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:49.691081+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:48.221845+0000","last_clean_scrub_stamp":"2024-02-05T07:42:49.691081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:59.660676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":129,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.745274+0000","last_change":"2024-02-05T07:43:23.326134+0000","last_active":"2024-02-05T07:43:29.745274+0000","last_peered":"2024-02-05T07:43:29.745274+0000","last_clean":"2024-02-05T07:43:29.745274+0000","last_became_active":"2024-02-05T07:41:53.752144+0000","last_became_peered":"2024-02-05T07:41:53.752144+0000","last_unstale":"2024-02-05T07:43:29.745274+0000","last_undegraded":"2024-02-05T07:43:29.745274+0000","last_fullsized":"2024-02-05T07:43:29.745274+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:23.326009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:43:23.326009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:33.263326+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":453,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897852+0000","last_change":"2024-02-05T07:43:23.771265+0000","last_active":"2024-02-05T07:43:29.897852+0000","last_peered":"2024-02-05T07:43:29.897852+0000","last_clean":"2024-02-05T07:43:29.897852+0000","last_became_active":"2024-02-05T07:41:56.731825+0000","last_became_peered":"2024-02-05T07:41:56.731825+0000","last_unstale":"2024-02-05T07:43:29.897852+0000","last_undegraded":"2024-02-05T07:43:29.897852+0000","last_fullsized":"2024-02-05T07:43:29.897852+0000","mapping_epoch":714,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:23.771152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:43:23.771152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:46.984569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":504,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.827177+0000","last_change":"2024-02-05T07:43:19.014258+0000","last_active":"2024-02-05T07:43:35.827177+0000","last_peered":"2024-02-05T07:43:35.827177+0000","last_clean":"2024-02-05T07:43:35.827177+0000","last_became_active":"2024-02-05T07:41:49.756275+0000","last_became_peered":"2024-02-05T07:41:49.756275+0000","last_unstale":"2024-02-05T07:43:35.827177+0000","last_undegraded":"2024-02-05T07:43:35.827177+0000","last_fullsized":"2024-02-05T07:43:35.827177+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:19.014155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:43:19.014155+0000","last_clean_scrub_stamp":"2024-02-05T07:43:19.014155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:37.584468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"615'26","reported_seq":469,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.827137+0000","last_change":"2024-02-05T07:43:29.985572+0000","last_active":"2024-02-05T07:43:35.827137+0000","last_peered":"2024-02-05T07:43:35.827137+0000","last_clean":"2024-02-05T07:43:35.827137+0000","last_became_active":"2024-02-05T07:41:49.748077+0000","last_became_peered":"2024-02-05T07:41:49.748077+0000","last_unstale":"2024-02-05T07:43:35.827137+0000","last_undegraded":"2024-02-05T07:43:35.827137+0000","last_fullsized":"2024-02-05T07:43:35.827137+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'26","last_scrub_stamp":"2024-02-05T07:43:29.985485+0000","last_deep_scrub":"96'18","last_deep_scrub_stamp":"2024-02-05T07:20:28.409472+0000","last_clean_scrub_stamp":"2024-02-05T07:43:29.985485+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:43.292062+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":472,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.827177+0000","last_change":"2024-02-05T07:42:24.167581+0000","last_active":"2024-02-05T07:43:35.827177+0000","last_peered":"2024-02-05T07:43:35.827177+0000","last_clean":"2024-02-05T07:43:35.827177+0000","last_became_active":"2024-02-05T07:36:32.733822+0000","last_became_peered":"2024-02-05T07:36:32.733822+0000","last_unstale":"2024-02-05T07:43:35.827177+0000","last_undegraded":"2024-02-05T07:43:35.827177+0000","last_fullsized":"2024-02-05T07:43:35.827177+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:24.167505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:47.405257+0000","last_clean_scrub_stamp":"2024-02-05T07:42:24.167505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043553800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"68'1","reported_seq":918,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742248+0000","last_change":"2024-02-05T07:43:26.096612+0000","last_active":"2024-02-05T07:43:29.742248+0000","last_peered":"2024-02-05T07:43:29.742248+0000","last_clean":"2024-02-05T07:43:29.742248+0000","last_became_active":"2024-02-05T07:42:11.956148+0000","last_became_peered":"2024-02-05T07:42:11.956148+0000","last_unstale":"2024-02-05T07:43:29.742248+0000","last_undegraded":"2024-02-05T07:43:29.742248+0000","last_fullsized":"2024-02-05T07:43:29.742248+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-02-05T07:43:26.096520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:43:26.096520+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:42.328266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":131,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898424+0000","last_change":"2024-02-05T07:43:16.788580+0000","last_active":"2024-02-05T07:43:29.898424+0000","last_peered":"2024-02-05T07:43:29.898424+0000","last_clean":"2024-02-05T07:43:29.898424+0000","last_became_active":"2024-02-05T07:41:45.938599+0000","last_became_peered":"2024-02-05T07:41:45.938599+0000","last_unstale":"2024-02-05T07:43:29.898424+0000","last_undegraded":"2024-02-05T07:43:29.898424+0000","last_fullsized":"2024-02-05T07:43:29.898424+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:16.788465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:00.818054+0000","last_clean_scrub_stamp":"2024-02-05T07:43:16.788465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:41.462635+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":807,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:34.234452+0000","last_change":"2024-02-05T07:43:34.234485+0000","last_active":"2024-02-05T07:43:34.234452+0000","last_peered":"2024-02-05T07:43:34.234452+0000","last_clean":"2024-02-05T07:43:34.234452+0000","last_became_active":"2024-02-05T07:42:36.739437+0000","last_became_peered":"2024-02-05T07:42:36.739437+0000","last_unstale":"2024-02-05T07:43:34.234452+0000","last_undegraded":"2024-02-05T07:43:34.234452+0000","last_fullsized":"2024-02-05T07:43:34.234452+0000","mapping_epoch":741,"log_start":"86'128","ondisk_log_start":"86'128","created":224,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:34.234374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:43:34.234374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:34.753672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1207,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053679+0000","last_change":"2024-02-05T07:42:39.333789+0000","last_active":"2024-02-05T07:43:36.053679+0000","last_peered":"2024-02-05T07:43:36.053679+0000","last_clean":"2024-02-05T07:43:36.053679+0000","last_became_active":"2024-02-05T07:42:05.922152+0000","last_became_peered":"2024-02-05T07:42:05.922152+0000","last_unstale":"2024-02-05T07:43:36.053679+0000","last_undegraded":"2024-02-05T07:43:36.053679+0000","last_fullsized":"2024-02-05T07:43:36.053679+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:39.333697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:42:39.333697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:47.330280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"722'69","reported_seq":369,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053670+0000","last_change":"2024-02-05T07:42:54.499621+0000","last_active":"2024-02-05T07:43:36.053670+0000","last_peered":"2024-02-05T07:43:36.053670+0000","last_clean":"2024-02-05T07:43:36.053670+0000","last_became_active":"2024-02-05T07:42:05.925978+0000","last_became_peered":"2024-02-05T07:42:05.925978+0000","last_unstale":"2024-02-05T07:43:36.053670+0000","last_undegraded":"2024-02-05T07:43:36.053670+0000","last_fullsized":"2024-02-05T07:43:36.053670+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'69","last_scrub_stamp":"2024-02-05T07:42:54.499501+0000","last_deep_scrub":"722'69","last_deep_scrub_stamp":"2024-02-05T07:42:54.499501+0000","last_clean_scrub_stamp":"2024-02-05T07:42:54.499501+0000","objects_scrubbed":10,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:59.389228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":81,"num_read_kb":66,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912785+0000","last_change":"2024-02-05T07:42:53.969116+0000","last_active":"2024-02-05T07:43:29.912785+0000","last_peered":"2024-02-05T07:43:29.912785+0000","last_clean":"2024-02-05T07:43:29.912785+0000","last_became_active":"2024-02-05T07:41:53.756727+0000","last_became_peered":"2024-02-05T07:41:53.756727+0000","last_unstale":"2024-02-05T07:43:29.912785+0000","last_undegraded":"2024-02-05T07:43:29.912785+0000","last_fullsized":"2024-02-05T07:43:29.912785+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:53.969006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:53.969006+0000","last_clean_scrub_stamp":"2024-02-05T07:42:53.969006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:55.164088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"98'3","reported_seq":1308,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744230+0000","last_change":"2024-02-05T07:42:36.742497+0000","last_active":"2024-02-05T07:43:29.744230+0000","last_peered":"2024-02-05T07:43:29.744230+0000","last_clean":"2024-02-05T07:43:29.744230+0000","last_became_active":"2024-02-05T07:42:36.742068+0000","last_became_peered":"2024-02-05T07:42:36.742068+0000","last_unstale":"2024-02-05T07:43:29.744230+0000","last_undegraded":"2024-02-05T07:43:29.744230+0000","last_fullsized":"2024-02-05T07:43:29.744230+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-02-05T07:42:07.050455+0000","last_deep_scrub":"98'3","last_deep_scrub_stamp":"2024-02-05T07:28:55.477280+0000","last_clean_scrub_stamp":"2024-02-05T07:42:07.050455+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:36.981039+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0010070389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1060,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055962+0000","last_change":"2024-02-05T07:42:39.799450+0000","last_active":"2024-02-05T07:43:36.055962+0000","last_peered":"2024-02-05T07:43:36.055962+0000","last_clean":"2024-02-05T07:43:36.055962+0000","last_became_active":"2024-02-05T07:41:53.751831+0000","last_became_peered":"2024-02-05T07:41:53.751831+0000","last_unstale":"2024-02-05T07:43:36.055962+0000","last_undegraded":"2024-02-05T07:43:36.055962+0000","last_fullsized":"2024-02-05T07:43:36.055962+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:39.799335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:39.799335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:46.389673+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"86'128","reported_seq":616,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913904+0000","last_change":"2024-02-05T07:43:25.789659+0000","last_active":"2024-02-05T07:43:29.913904+0000","last_peered":"2024-02-05T07:43:29.913904+0000","last_clean":"2024-02-05T07:43:29.913904+0000","last_became_active":"2024-02-05T07:41:55.732199+0000","last_became_peered":"2024-02-05T07:41:55.732199+0000","last_unstale":"2024-02-05T07:43:29.913904+0000","last_undegraded":"2024-02-05T07:43:29.913904+0000","last_fullsized":"2024-02-05T07:43:29.913904+0000","mapping_epoch":713,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":714,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:25.789560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:43:25.789560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:55.227667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"750'25","reported_seq":1070,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.054233+0000","last_change":"2024-02-05T07:42:50.453544+0000","last_active":"2024-02-05T07:43:36.054233+0000","last_peered":"2024-02-05T07:43:36.054233+0000","last_clean":"2024-02-05T07:43:36.054233+0000","last_became_active":"2024-02-05T07:42:05.924209+0000","last_became_peered":"2024-02-05T07:42:05.924209+0000","last_unstale":"2024-02-05T07:43:36.054233+0000","last_undegraded":"2024-02-05T07:43:36.054233+0000","last_fullsized":"2024-02-05T07:43:36.054233+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"750'25","last_scrub_stamp":"2024-02-05T07:42:50.453447+0000","last_deep_scrub":"750'25","last_deep_scrub_stamp":"2024-02-05T07:42:50.453447+0000","last_clean_scrub_stamp":"2024-02-05T07:42:50.453447+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:55.440697+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":969,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744672+0000","last_change":"2024-02-05T07:43:06.142521+0000","last_active":"2024-02-05T07:43:29.744672+0000","last_peered":"2024-02-05T07:43:29.744672+0000","last_clean":"2024-02-05T07:43:29.744672+0000","last_became_active":"2024-02-05T07:42:36.742368+0000","last_became_peered":"2024-02-05T07:42:36.742368+0000","last_unstale":"2024-02-05T07:43:29.744672+0000","last_undegraded":"2024-02-05T07:43:29.744672+0000","last_fullsized":"2024-02-05T07:43:29.744672+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:06.142417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:54.146194+0000","last_clean_scrub_stamp":"2024-02-05T07:43:06.142417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:33.560481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":492,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.826784+0000","last_change":"2024-02-05T07:42:55.945438+0000","last_active":"2024-02-05T07:43:35.826784+0000","last_peered":"2024-02-05T07:43:35.826784+0000","last_clean":"2024-02-05T07:43:35.826784+0000","last_became_active":"2024-02-05T07:41:53.737660+0000","last_became_peered":"2024-02-05T07:41:53.737660+0000","last_unstale":"2024-02-05T07:43:35.826784+0000","last_undegraded":"2024-02-05T07:43:35.826784+0000","last_fullsized":"2024-02-05T07:43:35.826784+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:55.945348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:41:40.550671+0000","last_clean_scrub_stamp":"2024-02-05T07:42:55.945348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:14.699029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":335,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053635+0000","last_change":"2024-02-05T07:42:47.509045+0000","last_active":"2024-02-05T07:43:36.053635+0000","last_peered":"2024-02-05T07:43:36.053635+0000","last_clean":"2024-02-05T07:43:36.053635+0000","last_became_active":"2024-02-05T07:42:05.925362+0000","last_became_peered":"2024-02-05T07:42:05.925362+0000","last_unstale":"2024-02-05T07:43:36.053635+0000","last_undegraded":"2024-02-05T07:43:36.053635+0000","last_fullsized":"2024-02-05T07:43:36.053635+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:47.508944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:47.508944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:49.041403+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"86'128","reported_seq":1490,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.765695+0000","last_change":"2024-02-05T07:42:47.700393+0000","last_active":"2024-02-05T07:43:29.765695+0000","last_peered":"2024-02-05T07:43:29.765695+0000","last_clean":"2024-02-05T07:43:29.765695+0000","last_became_active":"2024-02-05T07:42:24.729306+0000","last_became_peered":"2024-02-05T07:42:24.729306+0000","last_unstale":"2024-02-05T07:43:29.765695+0000","last_undegraded":"2024-02-05T07:43:29.765695+0000","last_fullsized":"2024-02-05T07:43:29.765695+0000","mapping_epoch":735,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":736,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:47.700266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:47.700266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:04.828316+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"616'51","reported_seq":1452,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.765484+0000","last_change":"2024-02-05T07:43:21.686492+0000","last_active":"2024-02-05T07:43:29.765484+0000","last_peered":"2024-02-05T07:43:29.765484+0000","last_clean":"2024-02-05T07:43:29.765484+0000","last_became_active":"2024-02-05T07:41:45.914903+0000","last_became_peered":"2024-02-05T07:41:45.914903+0000","last_unstale":"2024-02-05T07:43:29.765484+0000","last_undegraded":"2024-02-05T07:43:29.765484+0000","last_fullsized":"2024-02-05T07:43:29.765484+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'51","last_scrub_stamp":"2024-02-05T07:43:21.686362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:21.686362+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:50.670208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":129,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898340+0000","last_change":"2024-02-05T07:42:19.750470+0000","last_active":"2024-02-05T07:43:29.898340+0000","last_peered":"2024-02-05T07:43:29.898340+0000","last_clean":"2024-02-05T07:43:29.898340+0000","last_became_active":"2024-02-05T07:41:49.760281+0000","last_became_peered":"2024-02-05T07:41:49.760281+0000","last_unstale":"2024-02-05T07:43:29.898340+0000","last_undegraded":"2024-02-05T07:43:29.898340+0000","last_fullsized":"2024-02-05T07:43:29.898340+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:19.750373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:39.793075+0000","last_clean_scrub_stamp":"2024-02-05T07:42:19.750373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:41.082115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":492,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.826905+0000","last_change":"2024-02-05T07:43:15.021367+0000","last_active":"2024-02-05T07:43:35.826905+0000","last_peered":"2024-02-05T07:43:35.826905+0000","last_clean":"2024-02-05T07:43:35.826905+0000","last_became_active":"2024-02-05T07:41:49.747891+0000","last_became_peered":"2024-02-05T07:41:49.747891+0000","last_unstale":"2024-02-05T07:43:35.826905+0000","last_undegraded":"2024-02-05T07:43:35.826905+0000","last_fullsized":"2024-02-05T07:43:35.826905+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:15.021283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:15.021283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:19.170938+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":131,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:31.939953+0000","last_change":"2024-02-05T07:43:31.939976+0000","last_active":"2024-02-05T07:43:31.939953+0000","last_peered":"2024-02-05T07:43:31.939953+0000","last_clean":"2024-02-05T07:43:31.939953+0000","last_became_active":"2024-02-05T07:41:45.937344+0000","last_became_peered":"2024-02-05T07:41:45.937344+0000","last_unstale":"2024-02-05T07:43:31.939953+0000","last_undegraded":"2024-02-05T07:43:31.939953+0000","last_fullsized":"2024-02-05T07:43:31.939953+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:31.939861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:43:31.939861+0000","last_clean_scrub_stamp":"2024-02-05T07:43:31.939861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:35.366197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"86'128","reported_seq":1823,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055574+0000","last_change":"2024-02-05T07:42:04.595436+0000","last_active":"2024-02-05T07:43:36.055574+0000","last_peered":"2024-02-05T07:43:36.055574+0000","last_clean":"2024-02-05T07:43:36.055574+0000","last_became_active":"2024-02-05T07:40:28.961096+0000","last_became_peered":"2024-02-05T07:40:28.961096+0000","last_unstale":"2024-02-05T07:43:36.055574+0000","last_undegraded":"2024-02-05T07:43:36.055574+0000","last_fullsized":"2024-02-05T07:43:36.055574+0000","mapping_epoch":560,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:04.595339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:04.595339+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":118,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":480,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.826567+0000","last_change":"2024-02-05T07:43:23.079920+0000","last_active":"2024-02-05T07:43:35.826567+0000","last_peered":"2024-02-05T07:43:35.826567+0000","last_clean":"2024-02-05T07:43:35.826567+0000","last_became_active":"2024-02-05T07:36:32.733003+0000","last_became_peered":"2024-02-05T07:36:32.733003+0000","last_unstale":"2024-02-05T07:43:35.826567+0000","last_undegraded":"2024-02-05T07:43:35.826567+0000","last_fullsized":"2024-02-05T07:43:35.826567+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:23.079787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:23.079787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:34.512688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"615'24","reported_seq":119,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912668+0000","last_change":"2024-02-05T07:43:05.865185+0000","last_active":"2024-02-05T07:43:29.912668+0000","last_peered":"2024-02-05T07:43:29.912668+0000","last_clean":"2024-02-05T07:43:29.912668+0000","last_became_active":"2024-02-05T07:41:53.752799+0000","last_became_peered":"2024-02-05T07:41:53.752799+0000","last_unstale":"2024-02-05T07:43:29.912668+0000","last_undegraded":"2024-02-05T07:43:29.912668+0000","last_fullsized":"2024-02-05T07:43:29.912668+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'24","last_scrub_stamp":"2024-02-05T07:43:05.865075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:05.865075+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:12.836841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":129,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897491+0000","last_change":"2024-02-05T07:42:40.690417+0000","last_active":"2024-02-05T07:43:29.897491+0000","last_peered":"2024-02-05T07:43:29.897491+0000","last_clean":"2024-02-05T07:43:29.897491+0000","last_became_active":"2024-02-05T07:41:53.752835+0000","last_became_peered":"2024-02-05T07:41:53.752835+0000","last_unstale":"2024-02-05T07:43:29.897491+0000","last_undegraded":"2024-02-05T07:43:29.897491+0000","last_fullsized":"2024-02-05T07:43:29.897491+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:40.690284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:42:40.690284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:00.234498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1048,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055820+0000","last_change":"2024-02-05T07:42:46.897648+0000","last_active":"2024-02-05T07:43:36.055820+0000","last_peered":"2024-02-05T07:43:36.055820+0000","last_clean":"2024-02-05T07:43:36.055820+0000","last_became_active":"2024-02-05T07:41:46.009533+0000","last_became_peered":"2024-02-05T07:41:46.009533+0000","last_unstale":"2024-02-05T07:43:36.055820+0000","last_undegraded":"2024-02-05T07:43:36.055820+0000","last_fullsized":"2024-02-05T07:43:36.055820+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:46.897527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:46.897527+0000","last_clean_scrub_stamp":"2024-02-05T07:42:46.897527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:14.490189+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"86'128","reported_seq":1295,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:33.917824+0000","last_change":"2024-02-05T07:43:33.917852+0000","last_active":"2024-02-05T07:43:33.917824+0000","last_peered":"2024-02-05T07:43:33.917824+0000","last_clean":"2024-02-05T07:43:33.917824+0000","last_became_active":"2024-02-05T07:41:55.729341+0000","last_became_peered":"2024-02-05T07:41:55.729341+0000","last_unstale":"2024-02-05T07:43:33.917824+0000","last_undegraded":"2024-02-05T07:43:33.917824+0000","last_fullsized":"2024-02-05T07:43:33.917824+0000","mapping_epoch":713,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":714,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:33.917737+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:41:53.850473+0000","last_clean_scrub_stamp":"2024-02-05T07:43:33.917737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:54.298535+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"713'15","reported_seq":160,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898068+0000","last_change":"2024-02-05T07:42:43.646159+0000","last_active":"2024-02-05T07:43:29.898068+0000","last_peered":"2024-02-05T07:43:29.898068+0000","last_clean":"2024-02-05T07:43:29.898068+0000","last_became_active":"2024-02-05T07:41:53.752883+0000","last_became_peered":"2024-02-05T07:41:53.752883+0000","last_unstale":"2024-02-05T07:43:29.898068+0000","last_undegraded":"2024-02-05T07:43:29.898068+0000","last_fullsized":"2024-02-05T07:43:29.898068+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'15","last_scrub_stamp":"2024-02-05T07:42:43.646056+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-02-05T07:20:00.685282+0000","last_clean_scrub_stamp":"2024-02-05T07:42:43.646056+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:50.923210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"754'46","reported_seq":146,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744074+0000","last_change":"2024-02-05T07:43:27.201592+0000","last_active":"2024-02-05T07:43:29.744074+0000","last_peered":"2024-02-05T07:43:29.744074+0000","last_clean":"2024-02-05T07:43:29.744074+0000","last_became_active":"2024-02-05T07:41:49.734651+0000","last_became_peered":"2024-02-05T07:41:49.734651+0000","last_unstale":"2024-02-05T07:43:29.744074+0000","last_undegraded":"2024-02-05T07:43:29.744074+0000","last_fullsized":"2024-02-05T07:43:29.744074+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'46","last_scrub_stamp":"2024-02-05T07:43:27.201474+0000","last_deep_scrub":"754'46","last_deep_scrub_stamp":"2024-02-05T07:43:27.201474+0000","last_clean_scrub_stamp":"2024-02-05T07:43:27.201474+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:55.206882+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"614'4","reported_seq":141,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897514+0000","last_change":"2024-02-05T07:42:06.730315+0000","last_active":"2024-02-05T07:43:29.897514+0000","last_peered":"2024-02-05T07:43:29.897514+0000","last_clean":"2024-02-05T07:43:29.897514+0000","last_became_active":"2024-02-05T07:41:53.753442+0000","last_became_peered":"2024-02-05T07:41:53.753442+0000","last_unstale":"2024-02-05T07:43:29.897514+0000","last_undegraded":"2024-02-05T07:43:29.897514+0000","last_fullsized":"2024-02-05T07:43:29.897514+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"614'4","last_scrub_stamp":"2024-02-05T07:42:06.730218+0000","last_deep_scrub":"614'4","last_deep_scrub_stamp":"2024-02-05T07:38:55.829386+0000","last_clean_scrub_stamp":"2024-02-05T07:42:06.730218+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1067,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744466+0000","last_change":"2024-02-05T07:42:36.743895+0000","last_active":"2024-02-05T07:43:29.744466+0000","last_peered":"2024-02-05T07:43:29.744466+0000","last_clean":"2024-02-05T07:43:29.744466+0000","last_became_active":"2024-02-05T07:42:36.743154+0000","last_became_peered":"2024-02-05T07:42:36.743154+0000","last_unstale":"2024-02-05T07:43:29.744466+0000","last_undegraded":"2024-02-05T07:43:29.744466+0000","last_fullsized":"2024-02-05T07:43:29.744466+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:25.616991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:40.130170+0000","last_clean_scrub_stamp":"2024-02-05T07:42:25.616991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:33.787790+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"86'128","reported_seq":803,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744341+0000","last_change":"2024-02-05T07:43:24.158620+0000","last_active":"2024-02-05T07:43:29.744341+0000","last_peered":"2024-02-05T07:43:29.744341+0000","last_clean":"2024-02-05T07:43:29.744341+0000","last_became_active":"2024-02-05T07:41:51.724911+0000","last_became_peered":"2024-02-05T07:41:51.724911+0000","last_unstale":"2024-02-05T07:43:29.744341+0000","last_undegraded":"2024-02-05T07:43:29.744341+0000","last_fullsized":"2024-02-05T07:43:29.744341+0000","mapping_epoch":709,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:43:24.158519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:43:24.158519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:30.053686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"615'21","reported_seq":1112,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.056324+0000","last_change":"2024-02-05T07:42:20.575553+0000","last_active":"2024-02-05T07:43:36.056324+0000","last_peered":"2024-02-05T07:43:36.056324+0000","last_clean":"2024-02-05T07:43:36.056324+0000","last_became_active":"2024-02-05T07:41:46.203574+0000","last_became_peered":"2024-02-05T07:41:46.203574+0000","last_unstale":"2024-02-05T07:43:36.056324+0000","last_undegraded":"2024-02-05T07:43:36.056324+0000","last_fullsized":"2024-02-05T07:43:36.056324+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'21","last_scrub_stamp":"2024-02-05T07:42:20.575458+0000","last_deep_scrub":"613'15","last_deep_scrub_stamp":"2024-02-05T07:38:48.343448+0000","last_clean_scrub_stamp":"2024-02-05T07:42:20.575458+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:48.441315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"319'2","reported_seq":1276,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.764839+0000","last_change":"2024-02-05T07:42:48.726688+0000","last_active":"2024-02-05T07:43:29.764839+0000","last_peered":"2024-02-05T07:43:29.764839+0000","last_clean":"2024-02-05T07:43:29.764839+0000","last_became_active":"2024-02-05T07:41:49.732960+0000","last_became_peered":"2024-02-05T07:41:49.732960+0000","last_unstale":"2024-02-05T07:43:29.764839+0000","last_undegraded":"2024-02-05T07:43:29.764839+0000","last_fullsized":"2024-02-05T07:43:29.764839+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'2","last_scrub_stamp":"2024-02-05T07:42:48.726605+0000","last_deep_scrub":"319'2","last_deep_scrub_stamp":"2024-02-05T07:28:45.571917+0000","last_clean_scrub_stamp":"2024-02-05T07:42:48.726605+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:00.811869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":115,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.745041+0000","last_change":"2024-02-05T07:43:00.099868+0000","last_active":"2024-02-05T07:43:29.745041+0000","last_peered":"2024-02-05T07:43:29.745041+0000","last_clean":"2024-02-05T07:43:29.745041+0000","last_became_active":"2024-02-05T07:41:49.759544+0000","last_became_peered":"2024-02-05T07:41:49.759544+0000","last_unstale":"2024-02-05T07:43:29.745041+0000","last_undegraded":"2024-02-05T07:43:29.745041+0000","last_fullsized":"2024-02-05T07:43:29.745041+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:00.099772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:00.099772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:16.090460+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":135,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898478+0000","last_change":"2024-02-05T07:41:59.813914+0000","last_active":"2024-02-05T07:43:29.898478+0000","last_peered":"2024-02-05T07:43:29.898478+0000","last_clean":"2024-02-05T07:43:29.898478+0000","last_became_active":"2024-02-05T07:41:53.751823+0000","last_became_peered":"2024-02-05T07:41:53.751823+0000","last_unstale":"2024-02-05T07:43:29.898478+0000","last_undegraded":"2024-02-05T07:43:29.898478+0000","last_fullsized":"2024-02-05T07:43:29.898478+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:41:59.813778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:21:44.435826+0000","last_clean_scrub_stamp":"2024-02-05T07:41:59.813778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"86'128","reported_seq":895,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743067+0000","last_change":"2024-02-05T07:42:15.136606+0000","last_active":"2024-02-05T07:43:29.743067+0000","last_peered":"2024-02-05T07:43:29.743067+0000","last_clean":"2024-02-05T07:43:29.743067+0000","last_became_active":"2024-02-05T07:42:14.979893+0000","last_became_peered":"2024-02-05T07:42:14.979893+0000","last_unstale":"2024-02-05T07:43:29.743067+0000","last_undegraded":"2024-02-05T07:43:29.743067+0000","last_fullsized":"2024-02-05T07:43:29.743067+0000","mapping_epoch":726,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":727,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:15.136518+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:33:42.423707+0000","last_clean_scrub_stamp":"2024-02-05T07:42:15.136518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:35.687088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"615'21","reported_seq":901,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.054173+0000","last_change":"2024-02-05T07:42:21.384042+0000","last_active":"2024-02-05T07:43:36.054173+0000","last_peered":"2024-02-05T07:43:36.054173+0000","last_clean":"2024-02-05T07:43:36.054173+0000","last_became_active":"2024-02-05T07:42:05.928414+0000","last_became_peered":"2024-02-05T07:42:05.928414+0000","last_unstale":"2024-02-05T07:43:36.054173+0000","last_undegraded":"2024-02-05T07:43:36.054173+0000","last_fullsized":"2024-02-05T07:43:36.054173+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'21","last_scrub_stamp":"2024-02-05T07:42:21.383957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:42:21.383957+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:43.387421+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":137,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897699+0000","last_change":"2024-02-05T07:43:10.746556+0000","last_active":"2024-02-05T07:43:29.897699+0000","last_peered":"2024-02-05T07:43:29.897699+0000","last_clean":"2024-02-05T07:43:29.897699+0000","last_became_active":"2024-02-05T07:41:53.751818+0000","last_became_peered":"2024-02-05T07:41:53.751818+0000","last_unstale":"2024-02-05T07:43:29.897699+0000","last_undegraded":"2024-02-05T07:43:29.897699+0000","last_fullsized":"2024-02-05T07:43:29.897699+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:10.746414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:02.673592+0000","last_clean_scrub_stamp":"2024-02-05T07:43:10.746414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:33.189668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1039,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.056537+0000","last_change":"2024-02-05T07:42:38.723653+0000","last_active":"2024-02-05T07:43:36.056537+0000","last_peered":"2024-02-05T07:43:36.056537+0000","last_clean":"2024-02-05T07:43:36.056537+0000","last_became_active":"2024-02-05T07:41:46.203566+0000","last_became_peered":"2024-02-05T07:41:46.203566+0000","last_unstale":"2024-02-05T07:43:36.056537+0000","last_undegraded":"2024-02-05T07:43:36.056537+0000","last_fullsized":"2024-02-05T07:43:36.056537+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:38.723577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:33:50.723212+0000","last_clean_scrub_stamp":"2024-02-05T07:42:38.723577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:40.837376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":498,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.827184+0000","last_change":"2024-02-05T07:43:07.965205+0000","last_active":"2024-02-05T07:43:35.827184+0000","last_peered":"2024-02-05T07:43:35.827184+0000","last_clean":"2024-02-05T07:43:35.827184+0000","last_became_active":"2024-02-05T07:41:53.742721+0000","last_became_peered":"2024-02-05T07:41:53.742721+0000","last_unstale":"2024-02-05T07:43:35.827184+0000","last_undegraded":"2024-02-05T07:43:35.827184+0000","last_fullsized":"2024-02-05T07:43:35.827184+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:07.965126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:25:47.862362+0000","last_clean_scrub_stamp":"2024-02-05T07:43:07.965126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:25.673625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"86'128","reported_seq":442,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744139+0000","last_change":"2024-02-05T07:42:22.195996+0000","last_active":"2024-02-05T07:43:29.744139+0000","last_peered":"2024-02-05T07:43:29.744139+0000","last_clean":"2024-02-05T07:43:29.744139+0000","last_became_active":"2024-02-05T07:42:22.025157+0000","last_became_peered":"2024-02-05T07:42:22.025157+0000","last_unstale":"2024-02-05T07:43:29.744139+0000","last_undegraded":"2024-02-05T07:43:29.744139+0000","last_fullsized":"2024-02-05T07:43:29.744139+0000","mapping_epoch":732,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":733,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:22.195891+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-02-05T07:42:22.195891+0000","last_clean_scrub_stamp":"2024-02-05T07:42:22.195891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"615'25","reported_seq":133,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744782+0000","last_change":"2024-02-05T07:43:05.168692+0000","last_active":"2024-02-05T07:43:29.744782+0000","last_peered":"2024-02-05T07:43:29.744782+0000","last_clean":"2024-02-05T07:43:29.744782+0000","last_became_active":"2024-02-05T07:41:49.764420+0000","last_became_peered":"2024-02-05T07:41:49.764420+0000","last_unstale":"2024-02-05T07:43:29.744782+0000","last_undegraded":"2024-02-05T07:43:29.744782+0000","last_fullsized":"2024-02-05T07:43:29.744782+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'25","last_scrub_stamp":"2024-02-05T07:43:05.168573+0000","last_deep_scrub":"222'13","last_deep_scrub_stamp":"2024-02-05T07:25:46.891183+0000","last_clean_scrub_stamp":"2024-02-05T07:43:05.168573+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:08.800540+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":475,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053408+0000","last_change":"2024-02-05T07:43:00.548691+0000","last_active":"2024-02-05T07:43:36.053408+0000","last_peered":"2024-02-05T07:43:36.053408+0000","last_clean":"2024-02-05T07:43:36.053408+0000","last_became_active":"2024-02-05T07:42:05.926723+0000","last_became_peered":"2024-02-05T07:42:05.926723+0000","last_unstale":"2024-02-05T07:43:36.053408+0000","last_undegraded":"2024-02-05T07:43:36.053408+0000","last_fullsized":"2024-02-05T07:43:36.053408+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:00.548588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:41:47.678965+0000","last_clean_scrub_stamp":"2024-02-05T07:43:00.548588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:18.963269+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"704'14","reported_seq":1268,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.056008+0000","last_change":"2024-02-05T07:42:34.643665+0000","last_active":"2024-02-05T07:43:36.056008+0000","last_peered":"2024-02-05T07:43:36.056008+0000","last_clean":"2024-02-05T07:43:36.056008+0000","last_became_active":"2024-02-05T07:41:45.943759+0000","last_became_peered":"2024-02-05T07:41:45.943759+0000","last_unstale":"2024-02-05T07:43:36.056008+0000","last_undegraded":"2024-02-05T07:43:36.056008+0000","last_fullsized":"2024-02-05T07:43:36.056008+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'14","last_scrub_stamp":"2024-02-05T07:42:34.643555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:42:34.643555+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:54.840311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912887+0000","last_change":"2024-02-05T07:42:26.427884+0000","last_active":"2024-02-05T07:43:29.912887+0000","last_peered":"2024-02-05T07:43:29.912887+0000","last_clean":"2024-02-05T07:43:29.912887+0000","last_became_active":"2024-02-05T07:41:53.763253+0000","last_became_peered":"2024-02-05T07:41:53.763253+0000","last_unstale":"2024-02-05T07:43:29.912887+0000","last_undegraded":"2024-02-05T07:43:29.912887+0000","last_fullsized":"2024-02-05T07:43:29.912887+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:26.427765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:26.427765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:31.131882+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"86'128","reported_seq":485,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053356+0000","last_change":"2024-02-05T07:42:07.318649+0000","last_active":"2024-02-05T07:43:36.053356+0000","last_peered":"2024-02-05T07:43:36.053356+0000","last_clean":"2024-02-05T07:43:36.053356+0000","last_became_active":"2024-02-05T07:42:05.927588+0000","last_became_peered":"2024-02-05T07:42:05.927588+0000","last_unstale":"2024-02-05T07:43:36.053356+0000","last_undegraded":"2024-02-05T07:43:36.053356+0000","last_fullsized":"2024-02-05T07:43:36.053356+0000","mapping_epoch":719,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:07.318543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:07.318543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:37.101543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"615'35","reported_seq":1149,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.056249+0000","last_change":"2024-02-05T07:43:07.547909+0000","last_active":"2024-02-05T07:43:36.056249+0000","last_peered":"2024-02-05T07:43:36.056249+0000","last_clean":"2024-02-05T07:43:36.056249+0000","last_became_active":"2024-02-05T07:41:45.944922+0000","last_became_peered":"2024-02-05T07:41:45.944922+0000","last_unstale":"2024-02-05T07:43:36.056249+0000","last_undegraded":"2024-02-05T07:43:36.056249+0000","last_fullsized":"2024-02-05T07:43:36.056249+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'35","last_scrub_stamp":"2024-02-05T07:43:07.547817+0000","last_deep_scrub":"113'15","last_deep_scrub_stamp":"2024-02-05T07:21:36.359546+0000","last_clean_scrub_stamp":"2024-02-05T07:43:07.547817+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:24.763038+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":129,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898252+0000","last_change":"2024-02-05T07:42:18.787273+0000","last_active":"2024-02-05T07:43:29.898252+0000","last_peered":"2024-02-05T07:43:29.898252+0000","last_clean":"2024-02-05T07:43:29.898252+0000","last_became_active":"2024-02-05T07:41:53.738482+0000","last_became_peered":"2024-02-05T07:41:53.738482+0000","last_unstale":"2024-02-05T07:43:29.898252+0000","last_undegraded":"2024-02-05T07:43:29.898252+0000","last_fullsized":"2024-02-05T07:43:29.898252+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:18.787178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:42:18.787178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:37.424205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"727'14","reported_seq":142,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.769773+0000","last_change":"2024-02-05T07:42:16.144597+0000","last_active":"2024-02-05T07:43:29.769773+0000","last_peered":"2024-02-05T07:43:29.769773+0000","last_clean":"2024-02-05T07:43:29.769773+0000","last_became_active":"2024-02-05T07:41:49.741112+0000","last_became_peered":"2024-02-05T07:41:49.741112+0000","last_unstale":"2024-02-05T07:43:29.769773+0000","last_undegraded":"2024-02-05T07:43:29.769773+0000","last_fullsized":"2024-02-05T07:43:29.769773+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'14","last_scrub_stamp":"2024-02-05T07:42:16.144496+0000","last_deep_scrub":"727'14","last_deep_scrub_stamp":"2024-02-05T07:42:16.144496+0000","last_clean_scrub_stamp":"2024-02-05T07:42:16.144496+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:36.503020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":929,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053779+0000","last_change":"2024-02-05T07:43:20.840005+0000","last_active":"2024-02-05T07:43:36.053779+0000","last_peered":"2024-02-05T07:43:36.053779+0000","last_clean":"2024-02-05T07:43:36.053779+0000","last_became_active":"2024-02-05T07:42:05.925936+0000","last_became_peered":"2024-02-05T07:42:05.925936+0000","last_unstale":"2024-02-05T07:43:36.053779+0000","last_undegraded":"2024-02-05T07:43:36.053779+0000","last_fullsized":"2024-02-05T07:43:36.053779+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:20.839896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:43:20.839896+0000","last_clean_scrub_stamp":"2024-02-05T07:43:20.839896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:37.416997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"615'27","reported_seq":119,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912742+0000","last_change":"2024-02-05T07:43:04.866672+0000","last_active":"2024-02-05T07:43:29.912742+0000","last_peered":"2024-02-05T07:43:29.912742+0000","last_clean":"2024-02-05T07:43:29.912742+0000","last_became_active":"2024-02-05T07:41:53.754844+0000","last_became_peered":"2024-02-05T07:41:53.754844+0000","last_unstale":"2024-02-05T07:43:29.912742+0000","last_undegraded":"2024-02-05T07:43:29.912742+0000","last_fullsized":"2024-02-05T07:43:29.912742+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'27","last_scrub_stamp":"2024-02-05T07:43:04.866562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:04.866562+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:17.895184+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913801+0000","last_change":"2024-02-05T07:42:41.124845+0000","last_active":"2024-02-05T07:43:29.913801+0000","last_peered":"2024-02-05T07:43:29.913801+0000","last_clean":"2024-02-05T07:43:29.913801+0000","last_became_active":"2024-02-05T07:41:53.751189+0000","last_became_peered":"2024-02-05T07:41:53.751189+0000","last_unstale":"2024-02-05T07:43:29.913801+0000","last_undegraded":"2024-02-05T07:43:29.913801+0000","last_fullsized":"2024-02-05T07:43:29.913801+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:41.124746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:23:23.435135+0000","last_clean_scrub_stamp":"2024-02-05T07:42:41.124746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:09.119048+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1487,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742711+0000","last_change":"2024-02-05T07:43:15.180126+0000","last_active":"2024-02-05T07:43:29.742711+0000","last_peered":"2024-02-05T07:43:29.742711+0000","last_clean":"2024-02-05T07:43:29.742711+0000","last_became_active":"2024-02-05T07:41:49.750878+0000","last_became_peered":"2024-02-05T07:41:49.750878+0000","last_unstale":"2024-02-05T07:43:29.742711+0000","last_undegraded":"2024-02-05T07:43:29.742711+0000","last_fullsized":"2024-02-05T07:43:29.742711+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:15.180043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:32:48.345965+0000","last_clean_scrub_stamp":"2024-02-05T07:43:15.180043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:20.403842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1043,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055491+0000","last_change":"2024-02-05T07:42:09.551986+0000","last_active":"2024-02-05T07:43:36.055491+0000","last_peered":"2024-02-05T07:43:36.055491+0000","last_clean":"2024-02-05T07:43:36.055491+0000","last_became_active":"2024-02-05T07:41:49.762328+0000","last_became_peered":"2024-02-05T07:41:49.762328+0000","last_unstale":"2024-02-05T07:43:36.055491+0000","last_undegraded":"2024-02-05T07:43:36.055491+0000","last_fullsized":"2024-02-05T07:43:36.055491+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:09.551894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:35.120512+0000","last_clean_scrub_stamp":"2024-02-05T07:42:09.551894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"615'21","reported_seq":1122,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742270+0000","last_change":"2024-02-05T07:42:31.165246+0000","last_active":"2024-02-05T07:43:29.742270+0000","last_peered":"2024-02-05T07:43:29.742270+0000","last_clean":"2024-02-05T07:43:29.742270+0000","last_became_active":"2024-02-05T07:36:32.710090+0000","last_became_peered":"2024-02-05T07:36:32.710090+0000","last_unstale":"2024-02-05T07:43:29.742270+0000","last_undegraded":"2024-02-05T07:43:29.742270+0000","last_fullsized":"2024-02-05T07:43:29.742270+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'21","last_scrub_stamp":"2024-02-05T07:42:31.165083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:42:31.165083+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:54.549680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":137,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898545+0000","last_change":"2024-02-05T07:43:19.777472+0000","last_active":"2024-02-05T07:43:29.898545+0000","last_peered":"2024-02-05T07:43:29.898545+0000","last_clean":"2024-02-05T07:43:29.898545+0000","last_became_active":"2024-02-05T07:41:53.753389+0000","last_became_peered":"2024-02-05T07:41:53.753389+0000","last_unstale":"2024-02-05T07:43:29.898545+0000","last_undegraded":"2024-02-05T07:43:29.898545+0000","last_fullsized":"2024-02-05T07:43:29.898545+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:19.777339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:43:19.777339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:33.432123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1027,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.056163+0000","last_change":"2024-02-05T07:42:30.589866+0000","last_active":"2024-02-05T07:43:36.056163+0000","last_peered":"2024-02-05T07:43:36.056163+0000","last_clean":"2024-02-05T07:43:36.056163+0000","last_became_active":"2024-02-05T07:27:16.764393+0000","last_became_peered":"2024-02-05T07:27:16.764393+0000","last_unstale":"2024-02-05T07:43:36.056163+0000","last_undegraded":"2024-02-05T07:43:36.056163+0000","last_fullsized":"2024-02-05T07:43:36.056163+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:30.589751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:26:31.506641+0000","last_clean_scrub_stamp":"2024-02-05T07:42:30.589751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":382,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053341+0000","last_change":"2024-02-05T07:43:10.734795+0000","last_active":"2024-02-05T07:43:36.053341+0000","last_peered":"2024-02-05T07:43:36.053341+0000","last_clean":"2024-02-05T07:43:36.053341+0000","last_became_active":"2024-02-05T07:41:49.724979+0000","last_became_peered":"2024-02-05T07:41:49.724979+0000","last_unstale":"2024-02-05T07:43:36.053341+0000","last_undegraded":"2024-02-05T07:43:36.053341+0000","last_fullsized":"2024-02-05T07:43:36.053341+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:10.734709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:43:10.734709+0000","last_clean_scrub_stamp":"2024-02-05T07:43:10.734709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:30.426933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"86'128","reported_seq":1864,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.826740+0000","last_change":"2024-02-05T07:42:46.142513+0000","last_active":"2024-02-05T07:43:35.826740+0000","last_peered":"2024-02-05T07:43:35.826740+0000","last_clean":"2024-02-05T07:43:35.826740+0000","last_became_active":"2024-02-05T07:41:56.731143+0000","last_became_peered":"2024-02-05T07:41:56.731143+0000","last_unstale":"2024-02-05T07:43:35.826740+0000","last_undegraded":"2024-02-05T07:43:35.826740+0000","last_fullsized":"2024-02-05T07:43:35.826740+0000","mapping_epoch":714,"log_start":"86'128","ondisk_log_start":"86'128","created":555,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'128","last_scrub_stamp":"2024-02-05T07:42:46.142430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:16:33.084443+0000","last_clean_scrub_stamp":"2024-02-05T07:42:46.142430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:53.503737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"616'27","reported_seq":1110,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055385+0000","last_change":"2024-02-05T07:43:13.488714+0000","last_active":"2024-02-05T07:43:36.055385+0000","last_peered":"2024-02-05T07:43:36.055385+0000","last_clean":"2024-02-05T07:43:36.055385+0000","last_became_active":"2024-02-05T07:41:45.948277+0000","last_became_peered":"2024-02-05T07:41:45.948277+0000","last_unstale":"2024-02-05T07:43:36.055385+0000","last_undegraded":"2024-02-05T07:43:36.055385+0000","last_fullsized":"2024-02-05T07:43:36.055385+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'27","last_scrub_stamp":"2024-02-05T07:43:13.488630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:13.488630+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:28.999480+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":115,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744939+0000","last_change":"2024-02-05T07:43:17.155583+0000","last_active":"2024-02-05T07:43:29.744939+0000","last_peered":"2024-02-05T07:43:29.744939+0000","last_clean":"2024-02-05T07:43:29.744939+0000","last_became_active":"2024-02-05T07:41:49.736869+0000","last_became_peered":"2024-02-05T07:41:49.736869+0000","last_unstale":"2024-02-05T07:43:29.744939+0000","last_undegraded":"2024-02-05T07:43:29.744939+0000","last_fullsized":"2024-02-05T07:43:29.744939+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:17.155488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:43:17.155488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:40.037279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":129,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897720+0000","last_change":"2024-02-05T07:42:10.745719+0000","last_active":"2024-02-05T07:43:29.897720+0000","last_peered":"2024-02-05T07:43:29.897720+0000","last_clean":"2024-02-05T07:43:29.897720+0000","last_became_active":"2024-02-05T07:41:53.737579+0000","last_became_peered":"2024-02-05T07:41:53.737579+0000","last_unstale":"2024-02-05T07:43:29.897720+0000","last_undegraded":"2024-02-05T07:43:29.897720+0000","last_fullsized":"2024-02-05T07:43:29.897720+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:10.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:58.191815+0000","last_clean_scrub_stamp":"2024-02-05T07:42:10.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:38.258160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1039,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055699+0000","last_change":"2024-02-05T07:42:50.783601+0000","last_active":"2024-02-05T07:43:36.055699+0000","last_peered":"2024-02-05T07:43:36.055699+0000","last_clean":"2024-02-05T07:43:36.055699+0000","last_became_active":"2024-02-05T07:28:26.601487+0000","last_became_peered":"2024-02-05T07:28:26.601487+0000","last_unstale":"2024-02-05T07:43:36.055699+0000","last_undegraded":"2024-02-05T07:43:36.055699+0000","last_fullsized":"2024-02-05T07:43:36.055699+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:50.783496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:42:50.783496+0000","last_clean_scrub_stamp":"2024-02-05T07:42:50.783496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:07.269592+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"615'12","reported_seq":1057,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055461+0000","last_change":"2024-02-05T07:43:11.497632+0000","last_active":"2024-02-05T07:43:36.055461+0000","last_peered":"2024-02-05T07:43:36.055461+0000","last_clean":"2024-02-05T07:43:36.055461+0000","last_became_active":"2024-02-05T07:28:26.601633+0000","last_became_peered":"2024-02-05T07:28:26.601633+0000","last_unstale":"2024-02-05T07:43:36.055461+0000","last_undegraded":"2024-02-05T07:43:36.055461+0000","last_fullsized":"2024-02-05T07:43:36.055461+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'12","last_scrub_stamp":"2024-02-05T07:43:11.497509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:11.497509+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:15.037504+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":400,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.054001+0000","last_change":"2024-02-05T07:42:37.359520+0000","last_active":"2024-02-05T07:43:36.054001+0000","last_peered":"2024-02-05T07:43:36.054001+0000","last_clean":"2024-02-05T07:43:36.054001+0000","last_became_active":"2024-02-05T07:42:05.928456+0000","last_became_peered":"2024-02-05T07:42:05.928456+0000","last_unstale":"2024-02-05T07:43:36.054001+0000","last_undegraded":"2024-02-05T07:43:36.054001+0000","last_fullsized":"2024-02-05T07:43:36.054001+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:37.359439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:34:39.980067+0000","last_clean_scrub_stamp":"2024-02-05T07:42:37.359439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:44.025447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1326,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:32.761407+0000","last_change":"2024-02-05T07:43:32.761430+0000","last_active":"2024-02-05T07:43:32.761407+0000","last_peered":"2024-02-05T07:43:32.761407+0000","last_clean":"2024-02-05T07:43:32.761407+0000","last_became_active":"2024-02-05T07:41:53.745802+0000","last_became_peered":"2024-02-05T07:41:53.745802+0000","last_unstale":"2024-02-05T07:43:32.761407+0000","last_undegraded":"2024-02-05T07:43:32.761407+0000","last_fullsized":"2024-02-05T07:43:32.761407+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:32.761338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:20:09.349342+0000","last_clean_scrub_stamp":"2024-02-05T07:43:32.761338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:51.237793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":129,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.897790+0000","last_change":"2024-02-05T07:42:42.719864+0000","last_active":"2024-02-05T07:43:29.897790+0000","last_peered":"2024-02-05T07:43:29.897790+0000","last_clean":"2024-02-05T07:43:29.897790+0000","last_became_active":"2024-02-05T07:41:53.737953+0000","last_became_peered":"2024-02-05T07:41:53.737953+0000","last_unstale":"2024-02-05T07:43:29.897790+0000","last_undegraded":"2024-02-05T07:43:29.897790+0000","last_fullsized":"2024-02-05T07:43:29.897790+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:42.719750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:36:58.296469+0000","last_clean_scrub_stamp":"2024-02-05T07:42:42.719750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:00.454523+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"616'27","reported_seq":1325,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.765715+0000","last_change":"2024-02-05T07:43:08.721766+0000","last_active":"2024-02-05T07:43:29.765715+0000","last_peered":"2024-02-05T07:43:29.765715+0000","last_clean":"2024-02-05T07:43:29.765715+0000","last_became_active":"2024-02-05T07:41:49.741249+0000","last_became_peered":"2024-02-05T07:41:49.741249+0000","last_unstale":"2024-02-05T07:43:29.765715+0000","last_undegraded":"2024-02-05T07:43:29.765715+0000","last_fullsized":"2024-02-05T07:43:29.765715+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'27","last_scrub_stamp":"2024-02-05T07:43:08.721624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:08.721624+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:35.548763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1035,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055278+0000","last_change":"2024-02-05T07:42:54.586637+0000","last_active":"2024-02-05T07:43:36.055278+0000","last_peered":"2024-02-05T07:43:36.055278+0000","last_clean":"2024-02-05T07:43:36.055278+0000","last_became_active":"2024-02-05T07:27:16.830144+0000","last_became_peered":"2024-02-05T07:27:16.830144+0000","last_unstale":"2024-02-05T07:43:36.055278+0000","last_undegraded":"2024-02-05T07:43:36.055278+0000","last_fullsized":"2024-02-05T07:43:36.055278+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:54.586552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:40:11.849558+0000","last_clean_scrub_stamp":"2024-02-05T07:42:54.586552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:18.516132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":339,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.054246+0000","last_change":"2024-02-05T07:42:32.406206+0000","last_active":"2024-02-05T07:43:36.054246+0000","last_peered":"2024-02-05T07:43:36.054246+0000","last_clean":"2024-02-05T07:43:36.054246+0000","last_became_active":"2024-02-05T07:42:05.921203+0000","last_became_peered":"2024-02-05T07:42:05.921203+0000","last_unstale":"2024-02-05T07:43:36.054246+0000","last_undegraded":"2024-02-05T07:43:36.054246+0000","last_fullsized":"2024-02-05T07:43:36.054246+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:32.406132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:42:32.406132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:58.099585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":896,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:32.316591+0000","last_change":"2024-02-05T07:43:32.316633+0000","last_active":"2024-02-05T07:43:32.316591+0000","last_peered":"2024-02-05T07:43:32.316591+0000","last_clean":"2024-02-05T07:43:32.316591+0000","last_became_active":"2024-02-05T07:42:11.957914+0000","last_became_peered":"2024-02-05T07:42:11.957914+0000","last_unstale":"2024-02-05T07:43:32.316591+0000","last_undegraded":"2024-02-05T07:43:32.316591+0000","last_fullsized":"2024-02-05T07:43:32.316591+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:32.316501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:43:32.316501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:37.541860+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"616'30","reported_seq":1010,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053933+0000","last_change":"2024-02-05T07:42:57.532987+0000","last_active":"2024-02-05T07:43:36.053933+0000","last_peered":"2024-02-05T07:43:36.053933+0000","last_clean":"2024-02-05T07:43:36.053933+0000","last_became_active":"2024-02-05T07:42:05.927928+0000","last_became_peered":"2024-02-05T07:42:05.927928+0000","last_unstale":"2024-02-05T07:43:36.053933+0000","last_undegraded":"2024-02-05T07:43:36.053933+0000","last_fullsized":"2024-02-05T07:43:36.053933+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'30","last_scrub_stamp":"2024-02-05T07:42:57.532884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:42:57.532884+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:11.267326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":37,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"585'2","reported_seq":351,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.826511+0000","last_change":"2024-02-05T07:43:31.005078+0000","last_active":"2024-02-05T07:43:35.826511+0000","last_peered":"2024-02-05T07:43:35.826511+0000","last_clean":"2024-02-05T07:43:35.826511+0000","last_became_active":"2024-02-05T07:41:53.747431+0000","last_became_peered":"2024-02-05T07:41:53.747431+0000","last_unstale":"2024-02-05T07:43:35.826511+0000","last_undegraded":"2024-02-05T07:43:35.826511+0000","last_fullsized":"2024-02-05T07:43:35.826511+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'2","last_scrub_stamp":"2024-02-05T07:43:31.004994+0000","last_deep_scrub":"585'2","last_deep_scrub_stamp":"2024-02-05T07:38:00.631235+0000","last_clean_scrub_stamp":"2024-02-05T07:43:31.004994+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:40.505239+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"713'16","reported_seq":1632,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.765829+0000","last_change":"2024-02-05T07:42:36.667978+0000","last_active":"2024-02-05T07:43:29.765829+0000","last_peered":"2024-02-05T07:43:29.765829+0000","last_clean":"2024-02-05T07:43:29.765829+0000","last_became_active":"2024-02-05T07:41:53.756627+0000","last_became_peered":"2024-02-05T07:41:53.756627+0000","last_unstale":"2024-02-05T07:43:29.765829+0000","last_undegraded":"2024-02-05T07:43:29.765829+0000","last_fullsized":"2024-02-05T07:43:29.765829+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'16","last_scrub_stamp":"2024-02-05T07:42:36.667872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:42:36.667872+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:52.433126+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"116'2","reported_seq":137,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898299+0000","last_change":"2024-02-05T07:42:24.691285+0000","last_active":"2024-02-05T07:43:29.898299+0000","last_peered":"2024-02-05T07:43:29.898299+0000","last_clean":"2024-02-05T07:43:29.898299+0000","last_became_active":"2024-02-05T07:41:49.734786+0000","last_became_peered":"2024-02-05T07:41:49.734786+0000","last_unstale":"2024-02-05T07:43:29.898299+0000","last_undegraded":"2024-02-05T07:43:29.898299+0000","last_fullsized":"2024-02-05T07:43:29.898299+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-02-05T07:42:24.691117+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-02-05T07:21:39.490479+0000","last_clean_scrub_stamp":"2024-02-05T07:42:24.691117+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:38.181577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"616'29","reported_seq":1106,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.056257+0000","last_change":"2024-02-05T07:42:48.671635+0000","last_active":"2024-02-05T07:43:36.056257+0000","last_peered":"2024-02-05T07:43:36.056257+0000","last_clean":"2024-02-05T07:43:36.056257+0000","last_became_active":"2024-02-05T07:41:53.751859+0000","last_became_peered":"2024-02-05T07:41:53.751859+0000","last_unstale":"2024-02-05T07:43:36.056257+0000","last_undegraded":"2024-02-05T07:43:36.056257+0000","last_fullsized":"2024-02-05T07:43:36.056257+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'29","last_scrub_stamp":"2024-02-05T07:42:48.671504+0000","last_deep_scrub":"321'17","last_deep_scrub_stamp":"2024-02-05T07:28:47.062618+0000","last_clean_scrub_stamp":"2024-02-05T07:42:48.671504+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:02.519893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913487+0000","last_change":"2024-02-05T07:43:14.840381+0000","last_active":"2024-02-05T07:43:29.913487+0000","last_peered":"2024-02-05T07:43:29.913487+0000","last_clean":"2024-02-05T07:43:29.913487+0000","last_became_active":"2024-02-05T07:41:53.736908+0000","last_became_peered":"2024-02-05T07:41:53.736908+0000","last_unstale":"2024-02-05T07:43:29.913487+0000","last_undegraded":"2024-02-05T07:43:29.913487+0000","last_fullsized":"2024-02-05T07:43:29.913487+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:14.840237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:30:39.874081+0000","last_clean_scrub_stamp":"2024-02-05T07:43:14.840237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:32.460118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913687+0000","last_change":"2024-02-05T07:42:03.128705+0000","last_active":"2024-02-05T07:43:29.913687+0000","last_peered":"2024-02-05T07:43:29.913687+0000","last_clean":"2024-02-05T07:43:29.913687+0000","last_became_active":"2024-02-05T07:41:53.766217+0000","last_became_peered":"2024-02-05T07:41:53.766217+0000","last_unstale":"2024-02-05T07:43:29.913687+0000","last_undegraded":"2024-02-05T07:43:29.913687+0000","last_fullsized":"2024-02-05T07:43:29.913687+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:03.128612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:42:03.128612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:31.984685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":415,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053817+0000","last_change":"2024-02-05T07:43:06.897431+0000","last_active":"2024-02-05T07:43:36.053817+0000","last_peered":"2024-02-05T07:43:36.053817+0000","last_clean":"2024-02-05T07:43:36.053817+0000","last_became_active":"2024-02-05T07:42:05.931798+0000","last_became_peered":"2024-02-05T07:42:05.931798+0000","last_unstale":"2024-02-05T07:43:36.053817+0000","last_undegraded":"2024-02-05T07:43:36.053817+0000","last_fullsized":"2024-02-05T07:43:36.053817+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:06.897325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:41:52.942441+0000","last_clean_scrub_stamp":"2024-02-05T07:43:06.897325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:08.569154+0000","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"615'39","reported_seq":513,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.826857+0000","last_change":"2024-02-05T07:43:09.996344+0000","last_active":"2024-02-05T07:43:35.826857+0000","last_peered":"2024-02-05T07:43:35.826857+0000","last_clean":"2024-02-05T07:43:35.826857+0000","last_became_active":"2024-02-05T07:36:32.744301+0000","last_became_peered":"2024-02-05T07:36:32.744301+0000","last_unstale":"2024-02-05T07:43:35.826857+0000","last_undegraded":"2024-02-05T07:43:35.826857+0000","last_fullsized":"2024-02-05T07:43:35.826857+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'39","last_scrub_stamp":"2024-02-05T07:43:09.996257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:50.421025+0000","last_clean_scrub_stamp":"2024-02-05T07:43:09.996257+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:37.694273+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":425,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:35.826695+0000","last_change":"2024-02-05T07:42:42.160383+0000","last_active":"2024-02-05T07:43:35.826695+0000","last_peered":"2024-02-05T07:43:35.826695+0000","last_clean":"2024-02-05T07:43:35.826695+0000","last_became_active":"2024-02-05T07:41:53.739157+0000","last_became_peered":"2024-02-05T07:41:53.739157+0000","last_unstale":"2024-02-05T07:43:35.826695+0000","last_undegraded":"2024-02-05T07:43:35.826695+0000","last_fullsized":"2024-02-05T07:43:35.826695+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:42.160302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:42:42.160302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:59.700319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1470,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.743749+0000","last_change":"2024-02-05T07:42:56.170275+0000","last_active":"2024-02-05T07:43:29.743749+0000","last_peered":"2024-02-05T07:43:29.743749+0000","last_clean":"2024-02-05T07:43:29.743749+0000","last_became_active":"2024-02-05T07:41:53.747506+0000","last_became_peered":"2024-02-05T07:41:53.747506+0000","last_unstale":"2024-02-05T07:43:29.743749+0000","last_undegraded":"2024-02-05T07:43:29.743749+0000","last_fullsized":"2024-02-05T07:43:29.743749+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:56.170120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:41:46.786494+0000","last_clean_scrub_stamp":"2024-02-05T07:42:56.170120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:15.811099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913186+0000","last_change":"2024-02-05T07:42:51.109730+0000","last_active":"2024-02-05T07:43:29.913186+0000","last_peered":"2024-02-05T07:43:29.913186+0000","last_clean":"2024-02-05T07:43:29.913186+0000","last_became_active":"2024-02-05T07:41:53.764501+0000","last_became_peered":"2024-02-05T07:41:53.764501+0000","last_unstale":"2024-02-05T07:43:29.913186+0000","last_undegraded":"2024-02-05T07:43:29.913186+0000","last_fullsized":"2024-02-05T07:43:29.913186+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:51.109628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:40:18.398592+0000","last_clean_scrub_stamp":"2024-02-05T07:42:51.109628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:02.490625+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"719'11","reported_seq":123,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.913071+0000","last_change":"2024-02-05T07:43:21.923765+0000","last_active":"2024-02-05T07:43:29.913071+0000","last_peered":"2024-02-05T07:43:29.913071+0000","last_clean":"2024-02-05T07:43:29.913071+0000","last_became_active":"2024-02-05T07:41:53.754051+0000","last_became_peered":"2024-02-05T07:41:53.754051+0000","last_unstale":"2024-02-05T07:43:29.913071+0000","last_undegraded":"2024-02-05T07:43:29.913071+0000","last_fullsized":"2024-02-05T07:43:29.913071+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"719'11","last_scrub_stamp":"2024-02-05T07:43:21.923661+0000","last_deep_scrub":"719'11","last_deep_scrub_stamp":"2024-02-05T07:42:05.182833+0000","last_clean_scrub_stamp":"2024-02-05T07:43:21.923661+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:23.665787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":640,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744815+0000","last_change":"2024-02-05T07:42:41.079220+0000","last_active":"2024-02-05T07:43:29.744815+0000","last_peered":"2024-02-05T07:43:29.744815+0000","last_clean":"2024-02-05T07:43:29.744815+0000","last_became_active":"2024-02-05T07:42:36.735340+0000","last_became_peered":"2024-02-05T07:42:36.735340+0000","last_unstale":"2024-02-05T07:43:29.744815+0000","last_undegraded":"2024-02-05T07:43:29.744815+0000","last_fullsized":"2024-02-05T07:43:29.744815+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:41.079113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:42:41.079113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:05.250301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":137,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.898007+0000","last_change":"2024-02-05T07:43:27.868178+0000","last_active":"2024-02-05T07:43:29.898007+0000","last_peered":"2024-02-05T07:43:29.898007+0000","last_clean":"2024-02-05T07:43:29.898007+0000","last_became_active":"2024-02-05T07:41:53.739127+0000","last_became_peered":"2024-02-05T07:41:53.739127+0000","last_unstale":"2024-02-05T07:43:29.898007+0000","last_undegraded":"2024-02-05T07:43:29.898007+0000","last_fullsized":"2024-02-05T07:43:29.898007+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:27.868035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:27.868035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:51.324777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":115,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744248+0000","last_change":"2024-02-05T07:42:57.168859+0000","last_active":"2024-02-05T07:43:29.744248+0000","last_peered":"2024-02-05T07:43:29.744248+0000","last_clean":"2024-02-05T07:43:29.744248+0000","last_became_active":"2024-02-05T07:41:49.765238+0000","last_became_peered":"2024-02-05T07:41:49.765238+0000","last_unstale":"2024-02-05T07:43:29.744248+0000","last_undegraded":"2024-02-05T07:43:29.744248+0000","last_fullsized":"2024-02-05T07:43:29.744248+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:57.168732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:57.168732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:09.273025+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"615'25","reported_seq":117,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912694+0000","last_change":"2024-02-05T07:43:07.869638+0000","last_active":"2024-02-05T07:43:29.912694+0000","last_peered":"2024-02-05T07:43:29.912694+0000","last_clean":"2024-02-05T07:43:29.912694+0000","last_became_active":"2024-02-05T07:41:53.751157+0000","last_became_peered":"2024-02-05T07:41:53.751157+0000","last_unstale":"2024-02-05T07:43:29.912694+0000","last_undegraded":"2024-02-05T07:43:29.912694+0000","last_fullsized":"2024-02-05T07:43:29.912694+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'25","last_scrub_stamp":"2024-02-05T07:43:07.869497+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-05T07:20:02.798180+0000","last_clean_scrub_stamp":"2024-02-05T07:43:07.869497+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:16.515760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":629,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744798+0000","last_change":"2024-02-05T07:42:40.081185+0000","last_active":"2024-02-05T07:43:29.744798+0000","last_peered":"2024-02-05T07:43:29.744798+0000","last_clean":"2024-02-05T07:43:29.744798+0000","last_became_active":"2024-02-05T07:42:36.741191+0000","last_became_peered":"2024-02-05T07:42:36.741191+0000","last_unstale":"2024-02-05T07:43:29.744798+0000","last_undegraded":"2024-02-05T07:43:29.744798+0000","last_fullsized":"2024-02-05T07:43:29.744798+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":742,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:40.081091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:42:40.081091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:48.604227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":123,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.744485+0000","last_change":"2024-02-05T07:43:18.178931+0000","last_active":"2024-02-05T07:43:29.744485+0000","last_peered":"2024-02-05T07:43:29.744485+0000","last_clean":"2024-02-05T07:43:29.744485+0000","last_became_active":"2024-02-05T07:41:49.732819+0000","last_became_peered":"2024-02-05T07:41:49.732819+0000","last_unstale":"2024-02-05T07:43:29.744485+0000","last_undegraded":"2024-02-05T07:43:29.744485+0000","last_fullsized":"2024-02-05T07:43:29.744485+0000","mapping_epoch":707,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:18.178829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:18.178829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:19.583756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":480,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053497+0000","last_change":"2024-02-05T07:42:11.287040+0000","last_active":"2024-02-05T07:43:36.053497+0000","last_peered":"2024-02-05T07:43:36.053497+0000","last_clean":"2024-02-05T07:43:36.053497+0000","last_became_active":"2024-02-05T07:42:05.921956+0000","last_became_peered":"2024-02-05T07:42:05.921956+0000","last_unstale":"2024-02-05T07:43:36.053497+0000","last_undegraded":"2024-02-05T07:43:36.053497+0000","last_fullsized":"2024-02-05T07:43:36.053497+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:11.286964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:42:11.286964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:38.098619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"752'46","reported_seq":1138,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055282+0000","last_change":"2024-02-05T07:43:19.464717+0000","last_active":"2024-02-05T07:43:36.055282+0000","last_peered":"2024-02-05T07:43:36.055282+0000","last_clean":"2024-02-05T07:43:36.055282+0000","last_became_active":"2024-02-05T07:36:32.741691+0000","last_became_peered":"2024-02-05T07:36:32.741691+0000","last_unstale":"2024-02-05T07:43:36.055282+0000","last_undegraded":"2024-02-05T07:43:36.055282+0000","last_fullsized":"2024-02-05T07:43:36.055282+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"752'46","last_scrub_stamp":"2024-02-05T07:43:19.464585+0000","last_deep_scrub":"752'46","last_deep_scrub_stamp":"2024-02-05T07:43:19.464585+0000","last_clean_scrub_stamp":"2024-02-05T07:43:19.464585+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:36.264092+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1053,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.055803+0000","last_change":"2024-02-05T07:43:21.483047+0000","last_active":"2024-02-05T07:43:36.055803+0000","last_peered":"2024-02-05T07:43:36.055803+0000","last_clean":"2024-02-05T07:43:36.055803+0000","last_became_active":"2024-02-05T07:36:32.746731+0000","last_became_peered":"2024-02-05T07:36:32.746731+0000","last_unstale":"2024-02-05T07:43:36.055803+0000","last_undegraded":"2024-02-05T07:43:36.055803+0000","last_fullsized":"2024-02-05T07:43:36.055803+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:21.482944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:48.410364+0000","last_clean_scrub_stamp":"2024-02-05T07:43:21.482944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:51.309766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912660+0000","last_change":"2024-02-05T07:43:16.844544+0000","last_active":"2024-02-05T07:43:29.912660+0000","last_peered":"2024-02-05T07:43:29.912660+0000","last_clean":"2024-02-05T07:43:29.912660+0000","last_became_active":"2024-02-05T07:41:53.747813+0000","last_became_peered":"2024-02-05T07:41:53.747813+0000","last_unstale":"2024-02-05T07:43:29.912660+0000","last_undegraded":"2024-02-05T07:43:29.912660+0000","last_fullsized":"2024-02-05T07:43:29.912660+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:16.844418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:41:50.825097+0000","last_clean_scrub_stamp":"2024-02-05T07:43:16.844418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:31.959846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1051,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.056072+0000","last_change":"2024-02-05T07:42:36.827332+0000","last_active":"2024-02-05T07:43:36.056072+0000","last_peered":"2024-02-05T07:43:36.056072+0000","last_clean":"2024-02-05T07:43:36.056072+0000","last_became_active":"2024-02-05T07:41:45.939654+0000","last_became_peered":"2024-02-05T07:41:45.939654+0000","last_unstale":"2024-02-05T07:43:36.056072+0000","last_undegraded":"2024-02-05T07:43:36.056072+0000","last_fullsized":"2024-02-05T07:43:36.056072+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:36.827238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:38:58.711041+0000","last_clean_scrub_stamp":"2024-02-05T07:42:36.827238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:58.319667+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"615'25","reported_seq":117,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912672+0000","last_change":"2024-02-05T07:42:49.050344+0000","last_active":"2024-02-05T07:43:29.912672+0000","last_peered":"2024-02-05T07:43:29.912672+0000","last_clean":"2024-02-05T07:43:29.912672+0000","last_became_active":"2024-02-05T07:41:53.747734+0000","last_became_peered":"2024-02-05T07:41:53.747734+0000","last_unstale":"2024-02-05T07:43:29.912672+0000","last_undegraded":"2024-02-05T07:43:29.912672+0000","last_fullsized":"2024-02-05T07:43:29.912672+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"615'25","last_scrub_stamp":"2024-02-05T07:42:49.050261+0000","last_deep_scrub":"450'19","last_deep_scrub_stamp":"2024-02-05T07:34:34.853068+0000","last_clean_scrub_stamp":"2024-02-05T07:42:49.050261+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:55.058453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":456,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-05T07:43:36.053865+0000","last_change":"2024-02-05T07:42:20.360388+0000","last_active":"2024-02-05T07:43:36.053865+0000","last_peered":"2024-02-05T07:43:36.053865+0000","last_clean":"2024-02-05T07:43:36.053865+0000","last_became_active":"2024-02-05T07:42:05.919133+0000","last_became_peered":"2024-02-05T07:42:05.919133+0000","last_unstale":"2024-02-05T07:43:36.053865+0000","last_undegraded":"2024-02-05T07:43:36.053865+0000","last_fullsized":"2024-02-05T07:43:36.053865+0000","mapping_epoch":719,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:42:20.360311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:29:50.274768+0000","last_clean_scrub_stamp":"2024-02-05T07:42:20.360311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:43:50.262787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":890,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.742923+0000","last_change":"2024-02-05T07:43:05.105031+0000","last_active":"2024-02-05T07:43:29.742923+0000","last_peered":"2024-02-05T07:43:29.742923+0000","last_clean":"2024-02-05T07:43:29.742923+0000","last_became_active":"2024-02-05T07:42:12.072167+0000","last_became_peered":"2024-02-05T07:42:12.072167+0000","last_unstale":"2024-02-05T07:43:29.742923+0000","last_undegraded":"2024-02-05T07:43:29.742923+0000","last_fullsized":"2024-02-05T07:43:29.742923+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:05.104913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:52.436288+0000","last_clean_scrub_stamp":"2024-02-05T07:43:05.104913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:28.535307+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":107,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-05T07:43:29.912705+0000","last_change":"2024-02-05T07:43:12.929364+0000","last_active":"2024-02-05T07:43:29.912705+0000","last_peered":"2024-02-05T07:43:29.912705+0000","last_clean":"2024-02-05T07:43:29.912705+0000","last_became_active":"2024-02-05T07:41:53.758815+0000","last_became_peered":"2024-02-05T07:41:53.758815+0000","last_unstale":"2024-02-05T07:43:29.912705+0000","last_undegraded":"2024-02-05T07:43:29.912705+0000","last_fullsized":"2024-02-05T07:43:29.912705+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T07:43:12.929256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T07:18:54.446999+0000","last_clean_scrub_stamp":"2024-02-05T07:43:12.929256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T07:44:13.065451+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":755,"num_read_kb":750,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":797,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1042,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1511,"num_read_kb":1268,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":617,"num_bytes_recovered":13130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1075,"ondisk_log_size":1075,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038063,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":518580,"kb_used_data":4268,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93226572,"statfs":{"total":95995035648,"available":95464009728,"internally_reserved":0,"allocated":4370432,"data_stored":2529346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":526645706},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.85199999999999998,"15min":0.89800000000000002},"min":{"1min":0.33300000000000002,"5min":0.23000000000000001,"15min":0.214},"max":{"1min":0.94599999999999995,"5min":4.7489999999999997,"15min":17.518999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":1.006,"15min":1.069},"min":{"1min":0.34000000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.85099999999999998,"5min":9.9169999999999998,"15min":17.151},"last":0.60199999999999998}]},{"osd":1,"last update":"Mon Feb 5 07:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":1.5109999999999999,"15min":2.411},"min":{"1min":0.36799999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76500000000000001,"5min":36.286000000000001,"15min":65.792000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":1.254,"15min":2.1019999999999999},"min":{"1min":0.36699999999999999,"5min":0.28199999999999997,"15min":0.27500000000000002},"max":{"1min":0.76300000000000001,"5min":36.125999999999998,"15min":65.430000000000007},"last":0.502}]},{"osd":2,"last update":"Mon Feb 5 07:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.84699999999999998,"15min":1.7909999999999999},"min":{"1min":0.38600000000000001,"5min":0.29299999999999998,"15min":0.26900000000000002},"max":{"1min":0.70899999999999996,"5min":4.5839999999999996,"15min":170.59100000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":1.2569999999999999,"15min":1.883},"min":{"1min":0.40000000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":11.401999999999999,"15min":170.25399999999999},"last":0.44800000000000001}]},{"osd":3,"last update":"Mon Feb 5 07:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":1.4550000000000001,"15min":1.948},"min":{"1min":0.39800000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.98799999999999999,"5min":36.673000000000002,"15min":61.079999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.595,"15min":2.0230000000000001},"min":{"1min":0.46999999999999997,"5min":0.28399999999999997,"15min":0.27300000000000002},"max":{"1min":0.84499999999999997,"5min":37.106000000000002,"15min":60.936999999999998},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Feb 5 07:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.433,"15min":0.71499999999999997},"min":{"1min":0.40899999999999997,"5min":0.27900000000000003,"15min":0.23000000000000001},"max":{"1min":0.69199999999999995,"5min":0.92100000000000004,"15min":72.484999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.432,"15min":0.72699999999999998},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.247},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":70.831999999999994},"last":0.35599999999999998}]},{"osd":5,"last update":"Mon Feb 5 07:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.42999999999999999,"15min":0.52600000000000002},"min":{"1min":0.34599999999999997,"5min":0.27200000000000002,"15min":0.248},"max":{"1min":0.75600000000000001,"5min":0.95299999999999996,"15min":4.0839999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.47999999999999998,"15min":0.48899999999999999},"min":{"1min":0.379,"5min":0.318,"15min":0.28999999999999998},"max":{"1min":0.74299999999999999,"5min":0.98199999999999998,"15min":1.8779999999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Mon Feb 5 07:43:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.46800000000000003,"15min":0.85399999999999998},"min":{"1min":0.34499999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72399999999999998,"5min":1.405,"15min":71.091999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.504,"15min":0.83899999999999997},"min":{"1min":0.44500000000000001,"5min":0.314,"15min":0.30299999999999999},"max":{"1min":0.82099999999999995,"5min":1.0029999999999999,"15min":70.977000000000004},"last":0.48099999999999998}]}]},{"osd":6,"up_from":301,"seq":1292785156280,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":795044,"kb_used_data":4572,"kb_used_omap":5,"kb_used_meta":790458,"kb_avail":92950108,"statfs":{"total":95995035648,"available":95180910592,"internally_reserved":0,"allocated":4681728,"data_stored":2857137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5627,"internal_metadata":809429509},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.69199999999999995,"15min":1.022},"min":{"1min":0.32800000000000001,"5min":0.22700000000000001,"15min":0.192},"max":{"1min":0.76400000000000001,"5min":5.1959999999999997,"15min":14.218},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.71699999999999997,"15min":1.0880000000000001},"min":{"1min":0.44700000000000001,"5min":0.23200000000000001,"15min":0.22700000000000001},"max":{"1min":0.73799999999999999,"5min":5.7460000000000004,"15min":14.175000000000001},"last":0.53900000000000003}]},{"osd":1,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":1.548,"15min":1.877},"min":{"1min":0.29799999999999999,"5min":0.222,"15min":0.217},"max":{"1min":0.74399999999999999,"5min":56.325000000000003,"15min":56.325000000000003},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.6970000000000001,"15min":2.1619999999999999},"min":{"1min":0.36899999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.73199999999999998,"5min":60.115000000000002,"15min":61.103000000000002},"last":0.65700000000000003}]},{"osd":2,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":1.2010000000000001,"15min":1.669},"min":{"1min":0.40400000000000003,"5min":0.28299999999999997,"15min":0.249},"max":{"1min":0.67300000000000004,"5min":9.5109999999999992,"15min":46.927},"last":0.63},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.93200000000000005,"15min":1.671},"min":{"1min":0.34899999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.71999999999999997,"5min":7.6479999999999997,"15min":46.506},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":2.012,"15min":2.266},"min":{"1min":0.41099999999999998,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":0.70799999999999996,"5min":56.271000000000001,"15min":56.271000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":2.202,"15min":2.4009999999999998},"min":{"1min":0.46400000000000002,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":0.69599999999999995,"5min":56.235999999999997,"15min":56.235999999999997},"last":0.69499999999999995}]},{"osd":4,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.41699999999999998,"15min":1.0149999999999999},"min":{"1min":0.32200000000000001,"5min":0.26600000000000001,"15min":0.224},"max":{"1min":0.67600000000000005,"5min":0.68899999999999995,"15min":151.02099999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42099999999999999,"15min":1.056},"min":{"1min":0.35099999999999998,"5min":0.26300000000000001,"15min":0.22800000000000001},"max":{"1min":0.60799999999999998,"5min":0.77300000000000002,"15min":149.21799999999999},"last":0.52500000000000002}]},{"osd":5,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.46899999999999997,"15min":0.54600000000000004},"min":{"1min":0.39100000000000001,"5min":0.313,"15min":0.26800000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":2.8919999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.47299999999999998,"15min":0.496},"min":{"1min":0.33000000000000002,"5min":0.28499999999999998,"15min":0.28100000000000003},"max":{"1min":0.72499999999999998,"5min":0.747,"15min":1.526},"last":0.443}]},{"osd":7,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.46300000000000002,"15min":1.286},"min":{"1min":0.379,"5min":0.315,"15min":0.28000000000000003},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":84.158000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.48199999999999998,"15min":1.397},"min":{"1min":0.39600000000000002,"5min":0.318,"15min":0.312},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":85.018000000000001},"last":0.73799999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575181,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1153732,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":1149438,"kb_avail":92591420,"statfs":{"total":95995035648,"available":94813614080,"internally_reserved":0,"allocated":4386816,"data_stored":2530123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1177024965},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":1.51,"15min":1.8919999999999999},"min":{"1min":0.245,"5min":0.192,"15min":0.192},"max":{"1min":0.57699999999999996,"5min":15.327,"15min":29.449999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":1.601,"15min":2.1659999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.23799999999999999},"max":{"1min":0.624,"5min":8.3930000000000007,"15min":29.268999999999998},"last":0.68000000000000005}]},{"osd":2,"last update":"Mon Feb 5 07:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":1.599,"15min":2.2599999999999998},"min":{"1min":0.33100000000000002,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":0.59099999999999997,"5min":12.848000000000001,"15min":42.823999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":1.246,"15min":2.0030000000000001},"min":{"1min":0.25,"5min":0.22800000000000001,"15min":0.20699999999999999},"max":{"1min":0.54900000000000004,"5min":9.8030000000000008,"15min":42.426000000000002},"last":0.58499999999999996}]},{"osd":3,"last update":"Mon Feb 5 07:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.5589999999999999,"15min":2.2149999999999999},"min":{"1min":0.28199999999999997,"5min":0.252,"15min":0.22900000000000001},"max":{"1min":0.63,"5min":9.5519999999999996,"15min":46.777000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.47199999999999998,"5min":1.327,"15min":1.982},"min":{"1min":0.36299999999999999,"5min":0.25,"15min":0.20000000000000001},"max":{"1min":0.59799999999999998,"5min":7.3899999999999997,"15min":50.164999999999999},"last":0.625}]},{"osd":4,"last update":"Mon Feb 5 07:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.79500000000000004,"15min":1.51},"min":{"1min":0.379,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.91800000000000004,"5min":4.0069999999999997,"15min":89.409000000000006},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":1.127,"15min":1.629},"min":{"1min":0.45200000000000001,"5min":0.38900000000000001,"15min":0.28199999999999997},"max":{"1min":0.96899999999999997,"5min":12.743,"15min":89.915000000000006},"last":0.748}]},{"osd":5,"last update":"Mon Feb 5 07:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.78700000000000003,"15min":1.3340000000000001},"min":{"1min":0.436,"5min":0.32200000000000001,"15min":0.309},"max":{"1min":0.86899999999999999,"5min":3.9580000000000002,"15min":46.398000000000003},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":1.6499999999999999,"15min":1.6799999999999999},"min":{"1min":0.42499999999999999,"5min":0.34799999999999998,"15min":0.27600000000000002},"max":{"1min":0.83099999999999996,"5min":14.613,"15min":20.085999999999999},"last":0.88700000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":1.131,"15min":1.2010000000000001},"min":{"1min":0.45900000000000002,"5min":0.36099999999999999,"15min":0.29199999999999998},"max":{"1min":0.88700000000000001,"5min":12.685,"15min":12.685},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.84199999999999997,"15min":1.3029999999999999},"min":{"1min":0.44400000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.81100000000000005,"5min":3.9300000000000002,"15min":8.923},"last":0.78000000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":1.353,"15min":1.8029999999999999},"min":{"1min":0.41899999999999998,"5min":0.38200000000000001,"15min":0.318},"max":{"1min":0.91500000000000004,"5min":8.2439999999999998,"15min":46.351999999999997},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":1.3140000000000001,"15min":1.383},"min":{"1min":0.51200000000000001,"5min":0.40999999999999998,"15min":0.34000000000000002},"max":{"1min":0.98199999999999998,"5min":14.492000000000001,"15min":14.492000000000001},"last":0.82199999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738706,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1127884,"kb_used_data":4608,"kb_used_omap":1,"kb_used_meta":1123262,"kb_avail":92617268,"statfs":{"total":95995035648,"available":94840082432,"internally_reserved":0,"allocated":4718592,"data_stored":2857415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1150220741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 07:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":2.431,"15min":2.371},"min":{"1min":0.21299999999999999,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":0.57399999999999995,"5min":65.799000000000007,"15min":65.799000000000007},"last":0.313},{"interface":"front","average":{"1min":0.373,"5min":2.286,"15min":2.1440000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.192},"max":{"1min":0.57599999999999996,"5min":65.093999999999994,"15min":65.093999999999994},"last":0.33500000000000002}]},{"osd":2,"last update":"Mon Feb 5 07:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":1.353,"15min":1.8560000000000001},"min":{"1min":0.34300000000000003,"5min":0.23000000000000001,"15min":0.191},"max":{"1min":0.65600000000000003,"5min":7.4560000000000004,"15min":47.137999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":1.1339999999999999,"15min":1.6859999999999999},"min":{"1min":0.26500000000000001,"5min":0.222,"15min":0.20300000000000001},"max":{"1min":0.60099999999999998,"5min":7.9039999999999999,"15min":47.170999999999999},"last":0.35499999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":1.4530000000000001,"15min":1.7150000000000001},"min":{"1min":0.26200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.84499999999999997,"5min":22.129000000000001,"15min":29.629000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.38,"5min":1.615,"15min":2.1720000000000002},"min":{"1min":0.251,"5min":0.23200000000000001,"15min":0.20899999999999999},"max":{"1min":0.55300000000000005,"5min":22.155999999999999,"15min":29.463999999999999},"last":0.38}]},{"osd":4,"last update":"Mon Feb 5 07:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":1.353,"15min":1.615},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.26700000000000002},"max":{"1min":0.753,"5min":7.8460000000000001,"15min":42.901000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.111,"15min":1.45},"min":{"1min":0.40799999999999997,"5min":0.28000000000000003,"15min":0.27100000000000002},"max":{"1min":0.78800000000000003,"5min":7.3959999999999999,"15min":42.960999999999999},"last":0.47099999999999997}]},{"osd":5,"last update":"Mon Feb 5 07:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":1.0309999999999999,"15min":1.0349999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.30199999999999999},"max":{"1min":0.89300000000000002,"5min":7.492,"15min":7.492},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":1.1000000000000001,"15min":1.1819999999999999},"min":{"1min":0.38500000000000001,"5min":0.32900000000000001,"15min":0.29299999999999998},"max":{"1min":0.74299999999999999,"5min":7.8399999999999999,"15min":8.7599999999999998},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Feb 5 07:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":2.5990000000000002,"15min":1.8120000000000001},"min":{"1min":0.40999999999999998,"5min":0.312,"15min":0.26300000000000001},"max":{"1min":0.76100000000000001,"5min":141.59899999999999,"15min":141.59899999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":2.391,"15min":1.7110000000000001},"min":{"1min":0.40600000000000003,"5min":0.376,"15min":0.35099999999999998},"max":{"1min":0.78100000000000003,"5min":141.39599999999999,"15min":141.39599999999999},"last":0.56299999999999994}]},{"osd":7,"last update":"Mon Feb 5 07:43:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.379,"15min":1.649},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.28599999999999998},"max":{"1min":0.73299999999999998,"5min":7.8200000000000003,"15min":43.012},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":1.169,"15min":1.524},"min":{"1min":0.441,"5min":0.41699999999999998,"15min":0.34000000000000002},"max":{"1min":0.82399999999999995,"5min":7.883,"15min":43.512},"last":0.58199999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411656,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":996404,"kb_used_data":4516,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92748748,"statfs":{"total":95995035648,"available":94974717952,"internally_reserved":0,"allocated":4624384,"data_stored":2792127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1015675333},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:43:35 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.1339999999999999,"15min":1.6160000000000001},"min":{"1min":0.29599999999999999,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":0.57199999999999995,"5min":17.398,"15min":80.427000000000007},"last":0.314},{"interface":"front","average":{"1min":0.38800000000000001,"5min":1.2869999999999999,"15min":1.8520000000000001},"min":{"1min":0.27400000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.59499999999999997,"5min":11.583,"15min":80.939999999999998},"last":0.27400000000000002}]},{"osd":1,"last update":"Mon Feb 5 07:43:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":1.5780000000000001,"15min":2.6429999999999998},"min":{"1min":0.27900000000000003,"5min":0.221,"15min":0.17899999999999999},"max":{"1min":0.70799999999999996,"5min":18.135999999999999,"15min":110.917},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.45000000000000001,"5min":1.49,"15min":2.0379999999999998},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.20300000000000001},"max":{"1min":0.68700000000000006,"5min":7.9509999999999996,"15min":64.771000000000001},"last":0.41299999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":1.47,"15min":2.9049999999999998},"min":{"1min":0.26100000000000001,"5min":0.247,"15min":0.214},"max":{"1min":0.625,"5min":18.338999999999999,"15min":110.773},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":1.6200000000000001,"15min":2.3279999999999998},"min":{"1min":0.30599999999999999,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":0.67400000000000004,"5min":19.640000000000001,"15min":63.521000000000001},"last":0.37}]},{"osd":4,"last update":"Mon Feb 5 07:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.85299999999999998,"15min":2.2029999999999998},"min":{"1min":0.42099999999999999,"5min":0.28699999999999998,"15min":0.24299999999999999},"max":{"1min":0.81399999999999995,"5min":4.0759999999999996,"15min":168.11699999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.875,"15min":2.0009999999999999},"min":{"1min":0.433,"5min":0.33800000000000002,"15min":0.27200000000000002},"max":{"1min":0.83499999999999996,"5min":17.43,"15min":168.589},"last":0.496}]},{"osd":5,"last update":"Mon Feb 5 07:43:35 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.98799999999999999,"15min":1.0149999999999999},"min":{"1min":0.47999999999999998,"5min":0.35899999999999999,"15min":0.315},"max":{"1min":0.85899999999999999,"5min":17.347999999999999,"15min":17.347999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.222,"15min":1.4830000000000001},"min":{"1min":0.375,"5min":0.36399999999999999,"15min":0.30499999999999999},"max":{"1min":0.89400000000000002,"5min":5.601,"15min":47.761000000000003},"last":0.65600000000000003}]},{"osd":6,"last update":"Mon Feb 5 07:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":1.2490000000000001,"15min":1.7210000000000001},"min":{"1min":0.49299999999999999,"5min":0.35999999999999999,"15min":0.28799999999999998},"max":{"1min":0.95499999999999996,"5min":5.2750000000000004,"15min":47.726999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.94499999999999995,"15min":1.252},"min":{"1min":0.44600000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.99199999999999999,"5min":17.457999999999998,"15min":40.872999999999998},"last":0.53600000000000003}]},{"osd":7,"last update":"Mon Feb 5 07:43:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.874,"15min":1.3220000000000001},"min":{"1min":0.45100000000000001,"5min":0.36399999999999999,"15min":0.30499999999999999},"max":{"1min":0.873,"5min":4.0510000000000002,"15min":37.573999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.26,"15min":2.056},"min":{"1min":0.51000000000000001,"5min":0.42299999999999999,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":5.6369999999999996,"15min":110.86799999999999},"last":0.69099999999999995}]}]},{"osd":3,"up_from":26,"seq":111669150019,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1127704,"kb_used_data":4876,"kb_used_omap":1,"kb_used_meta":1122814,"kb_avail":92617448,"statfs":{"total":95995035648,"available":94840266752,"internally_reserved":0,"allocated":4993024,"data_stored":3126267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1149761991},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":1.335,"15min":1.6379999999999999},"min":{"1min":0.28899999999999998,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.56399999999999995,"5min":12.092000000000001,"15min":48.688000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.372,"5min":1.5109999999999999,"15min":2.2010000000000001},"min":{"1min":0.23599999999999999,"5min":0.20799999999999999,"15min":0.185},"max":{"1min":0.495,"5min":8.4930000000000003,"15min":99.989999999999995},"last":0.51100000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":2.6200000000000001,"15min":2.6080000000000001},"min":{"1min":0.26400000000000001,"5min":0.23400000000000001,"15min":0.20399999999999999},"max":{"1min":0.61899999999999999,"5min":61.941000000000003,"15min":102.20399999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.433,"5min":2.556,"15min":2.5},"min":{"1min":0.309,"5min":0.28699999999999998,"15min":0.20999999999999999},"max":{"1min":0.624,"5min":61.902000000000001,"15min":102.286},"last":0.497}]},{"osd":2,"last update":"Mon Feb 5 07:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":1.4910000000000001,"15min":2.468},"min":{"1min":0.29899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.66800000000000004,"5min":12.6,"15min":145.94999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.40400000000000003,"5min":1.3839999999999999,"15min":2.5270000000000001},"min":{"1min":0.26300000000000001,"5min":0.21299999999999999,"15min":0.20200000000000001},"max":{"1min":0.55600000000000005,"5min":9.7370000000000001,"15min":146.006},"last":0.438}]},{"osd":4,"last update":"Mon Feb 5 07:43:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.997,"15min":1.7949999999999999},"min":{"1min":0.41799999999999998,"5min":0.28799999999999998,"15min":0.27000000000000002},"max":{"1min":0.82099999999999995,"5min":5.992,"15min":100.024},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":1.2669999999999999,"15min":2.0590000000000002},"min":{"1min":0.42299999999999999,"5min":0.28899999999999998,"15min":0.27300000000000002},"max":{"1min":0.81299999999999994,"5min":6.3200000000000003,"15min":99.859999999999999},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Feb 5 07:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":1.306,"15min":1.5489999999999999},"min":{"1min":0.435,"5min":0.313,"15min":0.308},"max":{"1min":0.92500000000000004,"5min":11.507999999999999,"15min":99.911000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":1.04,"15min":1.514},"min":{"1min":0.46400000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.76100000000000001,"5min":5.9640000000000004,"15min":100.053},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Feb 5 07:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":1.3440000000000001,"15min":1.722},"min":{"1min":0.47199999999999998,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":1.034,"5min":11.563000000000001,"15min":99.807000000000002},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":1.137,"15min":1.752},"min":{"1min":0.44700000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.80600000000000005,"5min":5.9199999999999999,"15min":101.489},"last":0.65000000000000002}]},{"osd":7,"last update":"Mon Feb 5 07:43:19 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":1.2889999999999999,"15min":2.3679999999999999},"min":{"1min":0.41299999999999998,"5min":0.38200000000000001,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":6.3609999999999998,"15min":99.893000000000001},"last":0.625},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.333,"15min":2.3149999999999999},"min":{"1min":0.52000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.97699999999999998,"5min":11.457000000000001,"15min":99.879000000000005},"last":0.68799999999999994}]}]},{"osd":4,"up_from":32,"seq":137438953790,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":825400,"kb_used_data":4272,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92919752,"statfs":{"total":95995035648,"available":95149826048,"internally_reserved":0,"allocated":4374528,"data_stored":2528790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:43:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.0589999999999999,"15min":1.4319999999999999},"min":{"1min":0.40500000000000003,"5min":0.23499999999999999,"15min":0.20300000000000001},"max":{"1min":13.606999999999999,"5min":13.606999999999999,"15min":101.212},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":1.0529999999999999,"15min":1.4379999999999999},"min":{"1min":0.41299999999999998,"5min":0.26100000000000001,"15min":0.222},"max":{"1min":0.754,"5min":6.7350000000000003,"15min":102.07599999999999},"last":0.63300000000000001}]},{"osd":1,"last update":"Mon Feb 5 07:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":1.115,"15min":1.909},"min":{"1min":0.38300000000000001,"5min":0.25800000000000001,"15min":0.253},"max":{"1min":0.72499999999999998,"5min":5.601,"15min":67.838999999999999},"last":0.499},{"interface":"front","average":{"1min":0.63500000000000001,"5min":1.845,"15min":2.327},"min":{"1min":0.38100000000000001,"5min":0.26500000000000001,"15min":0.23300000000000001},"max":{"1min":0.877,"5min":32.598999999999997,"15min":68.108999999999995},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon Feb 5 07:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":1.7689999999999999,"15min":2.532},"min":{"1min":0.47399999999999998,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":0.80600000000000005,"5min":22.738,"15min":180.40199999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":1.6259999999999999,"15min":2.528},"min":{"1min":0.37,"5min":0.26200000000000001,"15min":0.24299999999999999},"max":{"1min":0.76800000000000002,"5min":22.407,"15min":180.381},"last":0.60099999999999998}]},{"osd":3,"last update":"Mon Feb 5 07:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":2.0819999999999999,"15min":2.3879999999999999},"min":{"1min":0.40799999999999997,"5min":0.27500000000000002,"15min":0.23799999999999999},"max":{"1min":0.72599999999999998,"5min":35.484000000000002,"15min":66.019999999999996},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":1.4870000000000001,"15min":2.335},"min":{"1min":0.41799999999999998,"5min":0.29799999999999999,"15min":0.26400000000000001},"max":{"1min":0.78700000000000003,"5min":33.344000000000001,"15min":101.001},"last":0.56899999999999995}]},{"osd":5,"last update":"Mon Feb 5 07:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.42099999999999999,"15min":0.51400000000000001},"min":{"1min":0.372,"5min":0.27800000000000002,"15min":0.24399999999999999},"max":{"1min":0.625,"5min":0.71399999999999997,"15min":10.917999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.434,"15min":0.55500000000000005},"min":{"1min":0.376,"5min":0.29699999999999999,"15min":0.24199999999999999},"max":{"1min":0.60499999999999998,"5min":0.82399999999999995,"15min":15.105},"last":0.32100000000000001}]},{"osd":6,"last update":"Mon Feb 5 07:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.44800000000000001,"15min":0.61599999999999999},"min":{"1min":0.35399999999999998,"5min":0.26100000000000001,"15min":0.245},"max":{"1min":0.70399999999999996,"5min":0.92200000000000004,"15min":14.606},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.48099999999999998,"15min":0.66900000000000004},"min":{"1min":0.41399999999999998,"5min":0.27400000000000002,"15min":0.254},"max":{"1min":0.71799999999999997,"5min":0.90200000000000002,"15min":13.114000000000001},"last":0.38600000000000001}]},{"osd":7,"last update":"Mon Feb 5 07:43:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.48399999999999999,"15min":0.625},"min":{"1min":0.41099999999999998,"5min":0.316,"15min":0.25700000000000001},"max":{"1min":0.69099999999999995,"5min":0.877,"15min":15.073},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.5,"15min":0.61599999999999999},"min":{"1min":0.39300000000000002,"5min":0.33900000000000002,"15min":0.28999999999999998},"max":{"1min":0.745,"5min":0.76500000000000001,"15min":14.581},"last":0.43099999999999999}]}]},{"osd":5,"up_from":41,"seq":176093659449,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":436908,"kb_used_data":4516,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93308244,"statfs":{"total":95995035648,"available":95547641856,"internally_reserved":0,"allocated":4624384,"data_stored":2791609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 07:43:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.92700000000000005,"15min":1.415},"min":{"1min":0.314,"5min":0.22900000000000001,"15min":0.219},"max":{"1min":0.81999999999999995,"5min":11.586,"15min":45.518000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.77100000000000002,"15min":1.329},"min":{"1min":0.45000000000000001,"5min":0.248,"15min":0.24099999999999999},"max":{"1min":0.71299999999999997,"5min":11.234999999999999,"15min":49.914000000000001},"last":0.56699999999999995}]},{"osd":1,"last update":"Mon Feb 5 07:43:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":2.6899999999999999,"15min":2.4279999999999999},"min":{"1min":0.40200000000000002,"5min":0.26400000000000001,"15min":0.23699999999999999},"max":{"1min":0.82899999999999996,"5min":77.477999999999994,"15min":77.477999999999994},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":2.6640000000000001,"15min":2.363},"min":{"1min":0.35199999999999998,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.752,"5min":79.176000000000002,"15min":79.176000000000002},"last":0.61499999999999999}]},{"osd":2,"last update":"Mon Feb 5 07:43:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":1.0760000000000001,"15min":1.649},"min":{"1min":0.39200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.80400000000000005,"5min":35.546999999999997,"15min":74.126999999999995},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":1.198,"15min":1.669},"min":{"1min":0.39200000000000002,"5min":0.312,"15min":0.25},"max":{"1min":0.79000000000000004,"5min":35.896999999999998,"15min":73.688999999999993},"last":0.629}]},{"osd":3,"last update":"Mon Feb 5 07:43:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":4.0229999999999997,"15min":3.0459999999999998},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.27400000000000002},"max":{"1min":0.749,"5min":97.192999999999998,"15min":97.192999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":3.7669999999999999,"15min":2.8079999999999998},"min":{"1min":0.36799999999999999,"5min":0.32400000000000001,"15min":0.28499999999999998},"max":{"1min":0.73299999999999998,"5min":97.206999999999994,"15min":97.206999999999994},"last":0.60099999999999998}]},{"osd":4,"last update":"Mon Feb 5 07:43:35 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.442,"15min":1.2470000000000001},"min":{"1min":0.29199999999999998,"5min":0.26500000000000001,"15min":0.252},"max":{"1min":0.749,"5min":0.76000000000000001,"15min":118.986},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.46300000000000002,"15min":1.276},"min":{"1min":0.33100000000000002,"5min":0.318,"15min":0.25900000000000001},"max":{"1min":0.71299999999999997,"5min":0.72099999999999997,"15min":120.893},"last":0.498}]},{"osd":6,"last update":"Mon Feb 5 07:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.45600000000000002,"15min":1.2949999999999999},"min":{"1min":0.375,"5min":0.253,"15min":0.253},"max":{"1min":0.747,"5min":0.747,"15min":122.346},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.48599999999999999,"15min":1.3009999999999999},"min":{"1min":0.46700000000000003,"5min":0.28799999999999998,"15min":0.28599999999999998},"max":{"1min":0.749,"5min":0.89800000000000002,"15min":120.462},"last":0.57899999999999996}]},{"osd":7,"last update":"Mon Feb 5 07:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.496,"15min":1.0840000000000001},"min":{"1min":0.372,"5min":0.30399999999999999,"15min":0.222},"max":{"1min":0.76300000000000001,"5min":0.79700000000000004,"15min":125.23699999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.50900000000000001,"15min":1.103},"min":{"1min":0.40100000000000002,"5min":0.312,"15min":0.27600000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":124.72},"last":0.66200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T07:43:38.434 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-05T07:43:38.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-02-05T07:43:38.679 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T07:43:39.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:38 smithi007 ceph-mon[29098]: pgmap v1399: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:39.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:38 smithi007 ceph-mon[29098]: from='client.16278 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:43:39.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:38 smithi007 ceph-mon[29098]: 5.1c scrub starts 2024-02-05T07:43:39.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:38 smithi007 ceph-mon[29098]: 5.1c scrub ok 2024-02-05T07:43:39.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:38 smithi007 ceph-mon[29098]: 1.b scrub starts 2024-02-05T07:43:39.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:38 smithi007 ceph-mon[29098]: 1.b scrub ok 2024-02-05T07:43:39.112 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:38 smithi007 ceph-mon[37158]: pgmap v1399: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:39.112 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:38 smithi007 ceph-mon[37158]: from='client.16278 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:43:39.112 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:38 smithi007 ceph-mon[37158]: 5.1c scrub starts 2024-02-05T07:43:39.112 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:38 smithi007 ceph-mon[37158]: 5.1c scrub ok 2024-02-05T07:43:39.112 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:38 smithi007 ceph-mon[37158]: 1.b scrub starts 2024-02-05T07:43:39.112 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:38 smithi007 ceph-mon[37158]: 1.b scrub ok 2024-02-05T07:43:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:38 smithi090 ceph-mon[33213]: pgmap v1399: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T07:43:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:38 smithi090 ceph-mon[33213]: from='client.16278 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T07:43:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:38 smithi090 ceph-mon[33213]: 5.1c scrub starts 2024-02-05T07:43:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:38 smithi090 ceph-mon[33213]: 5.1c scrub ok 2024-02-05T07:43:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:38 smithi090 ceph-mon[33213]: 1.b scrub starts 2024-02-05T07:43:39.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:38 smithi090 ceph-mon[33213]: 1.b scrub ok 2024-02-05T07:43:40.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:39 smithi007 ceph-mon[29098]: 5.d scrub starts 2024-02-05T07:43:40.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:39 smithi007 ceph-mon[29098]: 5.d scrub ok 2024-02-05T07:43:40.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:39 smithi007 ceph-mon[29098]: purged_snaps scrub starts 2024-02-05T07:43:40.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:39 smithi007 ceph-mon[29098]: purged_snaps scrub ok 2024-02-05T07:43:40.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:39 smithi007 ceph-mon[29098]: osdmap e758: 8 total, 8 up, 8 in 2024-02-05T07:43:40.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:39 smithi007 ceph-mon[29098]: 1.11 scrub starts 2024-02-05T07:43:40.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:39 smithi007 ceph-mon[29098]: 1.11 scrub ok 2024-02-05T07:43:40.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:39 smithi007 ceph-mon[37158]: 5.d scrub starts 2024-02-05T07:43:40.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:39 smithi007 ceph-mon[37158]: 5.d scrub ok 2024-02-05T07:43:40.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:39 smithi007 ceph-mon[37158]: purged_snaps scrub starts 2024-02-05T07:43:40.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:39 smithi007 ceph-mon[37158]: purged_snaps scrub ok 2024-02-05T07:43:40.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:39 smithi007 ceph-mon[37158]: osdmap e758: 8 total, 8 up, 8 in 2024-02-05T07:43:40.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:39 smithi007 ceph-mon[37158]: 1.11 scrub starts 2024-02-05T07:43:40.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:39 smithi007 ceph-mon[37158]: 1.11 scrub ok 2024-02-05T07:43:40.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:39 smithi090 ceph-mon[33213]: 5.d scrub starts 2024-02-05T07:43:40.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:39 smithi090 ceph-mon[33213]: 5.d scrub ok 2024-02-05T07:43:40.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:39 smithi090 ceph-mon[33213]: purged_snaps scrub starts 2024-02-05T07:43:40.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:39 smithi090 ceph-mon[33213]: purged_snaps scrub ok 2024-02-05T07:43:40.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:39 smithi090 ceph-mon[33213]: osdmap e758: 8 total, 8 up, 8 in 2024-02-05T07:43:40.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:39 smithi090 ceph-mon[33213]: 1.11 scrub starts 2024-02-05T07:43:40.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:39 smithi090 ceph-mon[33213]: 1.11 scrub ok 2024-02-05T07:43:40.843 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:43:40] ENGINE Bus STOPPING 2024-02-05T07:43:40.843 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:43:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T07:43:40.843 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:43:40] ENGINE Bus STOPPED 2024-02-05T07:43:40.843 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:43:40] ENGINE Bus STARTING 2024-02-05T07:43:40.844 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:43:40] ENGINE Serving on http://:::9283 2024-02-05T07:43:40.844 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:43:40] ENGINE Bus STARTED 2024-02-05T07:43:41.157 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[29098]: pgmap v1401: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:41.157 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[29098]: 6.18 scrub starts 2024-02-05T07:43:41.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[29098]: 6.18 scrub ok 2024-02-05T07:43:41.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[29098]: 1.13 scrub starts 2024-02-05T07:43:41.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[29098]: 1.13 scrub ok 2024-02-05T07:43:41.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[29098]: from='client.? 172.21.15.7:0/4221160246' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T07:43:41.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T07:43:41.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[29098]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-05T07:43:41.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:43:41.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:43:41.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[29098]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:43:41.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[29098]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:43:41.158 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[37158]: pgmap v1401: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:41.159 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[37158]: 6.18 scrub starts 2024-02-05T07:43:41.159 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[37158]: 6.18 scrub ok 2024-02-05T07:43:41.159 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[37158]: 1.13 scrub starts 2024-02-05T07:43:41.159 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[37158]: 1.13 scrub ok 2024-02-05T07:43:41.159 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[37158]: from='client.? 172.21.15.7:0/4221160246' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T07:43:41.159 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T07:43:41.159 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[37158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-05T07:43:41.159 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:43:41.159 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:43:41.159 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[37158]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:43:41.159 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:40 smithi007 ceph-mon[37158]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:43:41.185 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-05T07:43:41.196 INFO:tasks.cephadm:Teardown begin 2024-02-05T07:43:41.197 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-05T07:43:41.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-05T07:43:41.273 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:40 smithi090 ceph-mon[33213]: pgmap v1401: 152 pgs: 152 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T07:43:41.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:40 smithi090 ceph-mon[33213]: 6.18 scrub starts 2024-02-05T07:43:41.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:40 smithi090 ceph-mon[33213]: 6.18 scrub ok 2024-02-05T07:43:41.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:40 smithi090 ceph-mon[33213]: 1.13 scrub starts 2024-02-05T07:43:41.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:40 smithi090 ceph-mon[33213]: 1.13 scrub ok 2024-02-05T07:43:41.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:40 smithi090 ceph-mon[33213]: from='client.? 172.21.15.7:0/4221160246' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T07:43:41.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:40 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T07:43:41.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:40 smithi090 ceph-mon[33213]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-05T07:43:41.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:40 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:43:41.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:40 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T07:43:41.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:40 smithi090 ceph-mon[33213]: from='mgr.24473 172.21.15.7:0/3688988294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T07:43:41.274 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:40 smithi090 ceph-mon[33213]: from='mgr.24473 ' entity='mgr.y' 2024-02-05T07:43:41.301 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-05T07:43:41.302 DEBUG:teuthology.orchestra.run.smithi007:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-05T07:43:41.340 DEBUG:teuthology.orchestra.run.smithi090:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-05T07:43:41.364 INFO:tasks.cephadm:Stopping all daemons... 2024-02-05T07:43:41.364 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-05T07:43:41.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mon.a 2024-02-05T07:43:41.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:41 smithi007 systemd[1]: Stopping Ceph mon.a for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:43:41.773 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:43:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug there is no tcmu-runner data available 2024-02-05T07:43:42.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:41 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a[29094]: 2024-02-05T07:43:41.701+0000 7fe35bcde700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:43:42.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:41 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a[29094]: 2024-02-05T07:43:41.701+0000 7fe35bcde700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-05T07:43:42.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:41 smithi007 podman[211430]: 2024-02-05 07:43:41.885297071 +0000 UTC m=+0.277788142 container died 0693f30f1833f925d469b3e73e4fbf8ca84c0dd9c2efcb09d41965be0d1662b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, io.buildah.version=1.33.2) 2024-02-05T07:43:42.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:42 smithi007 podman[211430]: 2024-02-05 07:43:42.352246272 +0000 UTC m=+0.744737372 container cleanup 0693f30f1833f925d469b3e73e4fbf8ca84c0dd9c2efcb09d41965be0d1662b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T07:43:42.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Feb 05 07:43:42 smithi007 bash[211430]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-a 2024-02-05T07:43:42.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mon.a.service' 2024-02-05T07:43:42.851 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:43:42.851 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-05T07:43:42.851 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-05T07:43:42.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mon.c 2024-02-05T07:43:43.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:42 smithi007 systemd[1]: Stopping Ceph mon.c for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:43:43.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-c[37154]: 2024-02-05T07:43:43.147+0000 7fb09853a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:43:43.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-c[37154]: 2024-02-05T07:43:43.147+0000 7fb09853a700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-02-05T07:43:43.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:43 smithi007 podman[211550]: 2024-02-05 07:43:43.352807168 +0000 UTC m=+0.318747816 container died 5ab95d0c35ea1725621fd116554fbbf6f7592a8f64a66d8c734ea45039ad4f74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-c, ceph=True, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T07:43:43.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:43:43] ENGINE Bus STOPPING 2024-02-05T07:43:43.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:43:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T07:43:43.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:43:43] ENGINE Bus STOPPED 2024-02-05T07:43:43.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:43:43] ENGINE Bus STARTING 2024-02-05T07:43:43.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:43:43] ENGINE Serving on http://:::9283 2024-02-05T07:43:43.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: [05/Feb/2024:07:43:43] ENGINE Bus STARTED 2024-02-05T07:43:43.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:43 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: ::ffff:172.21.15.90 - - [05/Feb/2024:07:43:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T07:43:43.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:43 smithi007 podman[211550]: 2024-02-05 07:43:43.69897149 +0000 UTC m=+0.664912138 container cleanup 5ab95d0c35ea1725621fd116554fbbf6f7592a8f64a66d8c734ea45039ad4f74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-05T07:43:43.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Feb 05 07:43:43 smithi007 bash[211550]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-c 2024-02-05T07:43:44.169 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mon.c.service' 2024-02-05T07:43:44.207 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:43:44.207 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-05T07:43:44.208 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-05T07:43:44.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mon.b 2024-02-05T07:43:44.523 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:44 smithi090 systemd[1]: Stopping Ceph mon.b for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:43:44.871 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:44 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-b[33208]: 2024-02-05T07:43:44.567+0000 7f96365cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:43:44.871 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:44 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-b[33208]: 2024-02-05T07:43:44.567+0000 7f96365cd700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-02-05T07:43:44.871 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:44 smithi090 podman[158067]: 2024-02-05 07:43:44.618969778 +0000 UTC m=+0.182534627 container died f4c0e28b73aeff5d516794b188f91ef8e84a102d54f9fb8cdc06fdca33ca57cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-b, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-05T07:43:45.255 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:45 smithi090 podman[158067]: 2024-02-05 07:43:45.027951577 +0000 UTC m=+0.591516416 container cleanup f4c0e28b73aeff5d516794b188f91ef8e84a102d54f9fb8cdc06fdca33ca57cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0) 2024-02-05T07:43:45.255 INFO:journalctl@ceph.mon.b.smithi090.stdout:Feb 05 07:43:45 smithi090 bash[158067]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mon-b 2024-02-05T07:43:45.477 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mon.b.service' 2024-02-05T07:43:45.520 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:43:45.520 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-05T07:43:45.520 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-05T07:43:45.521 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mgr.y 2024-02-05T07:43:45.828 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:45 smithi007 systemd[1]: Stopping Ceph mgr.y for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:43:46.138 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:45 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:43:45.827+0000 7fdd539a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:43:46.138 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:45 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y[29328]: 2024-02-05T07:43:45.827+0000 7fdd539a0700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-05T07:43:46.138 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:45 smithi007 podman[211685]: 2024-02-05 07:43:45.858973885 +0000 UTC m=+0.144199446 container died 763e883e587bb6d98d1552f781ef34b72d20855a080e9c9d8e0c89a94ab6f86e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_CLEAN=True, ceph=True) 2024-02-05T07:43:46.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:46 smithi007 podman[211685]: 2024-02-05 07:43:46.303875529 +0000 UTC m=+0.589101090 container cleanup 763e883e587bb6d98d1552f781ef34b72d20855a080e9c9d8e0c89a94ab6f86e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.2) 2024-02-05T07:43:46.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Feb 05 07:43:46 smithi007 bash[211685]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-y 2024-02-05T07:43:46.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mgr.y.service' 2024-02-05T07:43:46.752 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:43:46.753 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-05T07:43:46.753 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-05T07:43:46.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mgr.x 2024-02-05T07:43:47.122 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:43:46 smithi090 systemd[1]: Stopping Ceph mgr.x for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:43:47.410 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:43:47 smithi090 podman[158189]: 2024-02-05 07:43:47.122315998 +0000 UTC m=+0.181850447 container died e809b7c6434d4b19e9aff1e760bfcc7ed08aff88fa65f79e76ed62bddecdce50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T07:43:47.774 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:43:47 smithi090 podman[158189]: 2024-02-05 07:43:47.500118971 +0000 UTC m=+0.559653415 container cleanup e809b7c6434d4b19e9aff1e760bfcc7ed08aff88fa65f79e76ed62bddecdce50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0) 2024-02-05T07:43:47.774 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Feb 05 07:43:47 smithi090 bash[158189]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-mgr-x 2024-02-05T07:43:47.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@mgr.x.service' 2024-02-05T07:43:47.970 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:43:47.970 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-05T07:43:47.970 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-05T07:43:47.970 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.0 2024-02-05T07:43:48.267 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:43:48 smithi007 systemd[1]: Stopping Ceph osd.0 for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:43:48.673 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:43:48 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-0[41180]: 2024-02-05T07:43:48.266+0000 7fc74d641700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:43:48.673 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:43:48 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-0[41180]: 2024-02-05T07:43:48.266+0000 7fc74d641700 -1 osd.0 758 *** Got signal Terminated *** 2024-02-05T07:43:48.673 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:43:48 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-0[41180]: 2024-02-05T07:43:48.266+0000 7fc74d641700 -1 osd.0 758 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T07:43:54.134 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:43:53 smithi007 podman[211807]: 2024-02-05 07:43:53.868955434 +0000 UTC m=+5.699370284 container died a450e3f3ef1be19c71efe4fcdfa9ad901dbf82f5ec8eb3e46bea2e0badbdb8e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_BRANCH=HEAD) 2024-02-05T07:43:54.423 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:43:54 smithi007 podman[211807]: 2024-02-05 07:43:54.350419387 +0000 UTC m=+6.180834280 container cleanup a450e3f3ef1be19c71efe4fcdfa9ad901dbf82f5ec8eb3e46bea2e0badbdb8e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-0, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-05T07:43:54.423 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:43:54 smithi007 bash[211807]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-0 2024-02-05T07:43:54.922 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:43:54 smithi007 podman[211823]: 2024-02-05 07:43:54.635428769 +0000 UTC m=+0.763383050 container remove a450e3f3ef1be19c71efe4fcdfa9ad901dbf82f5ec8eb3e46bea2e0badbdb8e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-05T07:43:55.264 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:43:54 smithi007 podman[211917]: 2024-02-05 07:43:54.8517554 +0000 UTC m=+0.017629543 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T07:43:55.264 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:43:54 smithi007 podman[211917]: 2024-02-05 07:43:54.969827938 +0000 UTC m=+0.135702121 container create 0862d7695e83fa5717c487304312edd0795aa39641f252e6ea5c6e9517971c63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True) 2024-02-05T07:43:55.264 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:43:55 smithi007 podman[211917]: 2024-02-05 07:43:55.181055793 +0000 UTC m=+0.346929931 container init 0862d7695e83fa5717c487304312edd0795aa39641f252e6ea5c6e9517971c63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-0-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T07:43:55.264 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:43:55 smithi007 podman[211917]: 2024-02-05 07:43:55.184243943 +0000 UTC m=+0.350118087 container start 0862d7695e83fa5717c487304312edd0795aa39641f252e6ea5c6e9517971c63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T07:43:55.264 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:43:55 smithi007 podman[211917]: 2024-02-05 07:43:55.264114228 +0000 UTC m=+0.429988376 container attach 0862d7695e83fa5717c487304312edd0795aa39641f252e6ea5c6e9517971c63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-05T07:43:56.168 INFO:journalctl@ceph.osd.0.smithi007.stdout:Feb 05 07:43:55 smithi007 podman[211917]: 2024-02-05 07:43:55.815292793 +0000 UTC m=+0.981166930 container died 0862d7695e83fa5717c487304312edd0795aa39641f252e6ea5c6e9517971c63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-02-05T07:43:56.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.0.service' 2024-02-05T07:43:56.599 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:43:56.599 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-05T07:43:56.600 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-05T07:43:56.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.1 2024-02-05T07:43:56.921 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:43:56 smithi007 systemd[1]: Stopping Ceph osd.1 for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:43:57.173 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:43:56 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-1[47043]: 2024-02-05T07:43:56.920+0000 7f21ad71c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:43:57.173 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:43:56 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-1[47043]: 2024-02-05T07:43:56.920+0000 7f21ad71c700 -1 osd.1 758 *** Got signal Terminated *** 2024-02-05T07:43:57.174 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:43:56 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-1[47043]: 2024-02-05T07:43:56.920+0000 7f21ad71c700 -1 osd.1 758 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T07:44:02.639 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:44:02 smithi007 podman[212024]: 2024-02-05 07:44:02.349296642 +0000 UTC m=+5.527169543 container died 91d007caefbae71e7e1b20f6694497affe6c2877aef3e394ae2fe19146b1e309 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-1, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T07:44:02.923 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:44:02 smithi007 podman[212024]: 2024-02-05 07:44:02.80451228 +0000 UTC m=+5.982385181 container cleanup 91d007caefbae71e7e1b20f6694497affe6c2877aef3e394ae2fe19146b1e309 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T07:44:02.923 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:44:02 smithi007 bash[212024]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-1 2024-02-05T07:44:03.415 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:44:03 smithi007 podman[212041]: 2024-02-05 07:44:03.006001373 +0000 UTC m=+0.651727031 container remove 91d007caefbae71e7e1b20f6694497affe6c2877aef3e394ae2fe19146b1e309 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-1, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux ) 2024-02-05T07:44:03.673 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:44:03 smithi007 podman[212131]: 2024-02-05 07:44:03.414996673 +0000 UTC m=+0.113924237 container create 6e500e482f78d9eea39d04063557831ae9d19ce2d2808a4d185d951db0304dbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-05T07:44:03.673 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:44:03 smithi007 podman[212131]: 2024-02-05 07:44:03.319806604 +0000 UTC m=+0.018734169 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T07:44:03.673 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:44:03 smithi007 podman[212131]: 2024-02-05 07:44:03.625811244 +0000 UTC m=+0.324738809 container init 6e500e482f78d9eea39d04063557831ae9d19ce2d2808a4d185d951db0304dbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-05T07:44:03.673 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:44:03 smithi007 podman[212131]: 2024-02-05 07:44:03.628932863 +0000 UTC m=+0.327860428 container start 6e500e482f78d9eea39d04063557831ae9d19ce2d2808a4d185d951db0304dbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-05T07:44:04.173 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:44:03 smithi007 podman[212131]: 2024-02-05 07:44:03.717464178 +0000 UTC m=+0.416391747 container attach 6e500e482f78d9eea39d04063557831ae9d19ce2d2808a4d185d951db0304dbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-05T07:44:04.173 INFO:journalctl@ceph.osd.1.smithi007.stdout:Feb 05 07:44:03 smithi007 podman[212131]: 2024-02-05 07:44:03.921008016 +0000 UTC m=+0.619935583 container died 6e500e482f78d9eea39d04063557831ae9d19ce2d2808a4d185d951db0304dbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-1-deactivate, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_CLEAN=True) 2024-02-05T07:44:04.540 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.1.service' 2024-02-05T07:44:04.585 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:44:04.585 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-05T07:44:04.585 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-05T07:44:04.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.2 2024-02-05T07:44:04.923 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:44:04 smithi007 systemd[1]: Stopping Ceph osd.2 for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:44:05.423 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:44:04 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-2[52839]: 2024-02-05T07:44:04.923+0000 7fab32a04700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:44:05.423 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:44:04 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-2[52839]: 2024-02-05T07:44:04.923+0000 7fab32a04700 -1 osd.2 758 *** Got signal Terminated *** 2024-02-05T07:44:05.423 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:44:04 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-2[52839]: 2024-02-05T07:44:04.923+0000 7fab32a04700 -1 osd.2 758 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T07:44:10.825 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:44:10 smithi007 podman[212237]: 2024-02-05 07:44:10.575388349 +0000 UTC m=+5.765005356 container died 7bcc379742c1582ad6a1259dd33aa8b9f60d010311d46042451070447cfd0ccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-05T07:44:11.173 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:44:10 smithi007 podman[212237]: 2024-02-05 07:44:10.974065485 +0000 UTC m=+6.163682537 container cleanup 7bcc379742c1582ad6a1259dd33aa8b9f60d010311d46042451070447cfd0ccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-05T07:44:11.173 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:44:10 smithi007 bash[212237]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-2 2024-02-05T07:44:11.518 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:44:11 smithi007 podman[212254]: 2024-02-05 07:44:11.213744218 +0000 UTC m=+0.638810402 container remove 7bcc379742c1582ad6a1259dd33aa8b9f60d010311d46042451070447cfd0ccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True) 2024-02-05T07:44:11.837 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:44:11 smithi007 podman[212345]: 2024-02-05 07:44:11.517813666 +0000 UTC m=+0.103056333 container create 57a40a5d19f87d6f05f6510d51cfae21801c850eb75e7f43a353a8244fd02827 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-05T07:44:11.837 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:44:11 smithi007 podman[212345]: 2024-02-05 07:44:11.434254292 +0000 UTC m=+0.019496922 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T07:44:11.837 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:44:11 smithi007 podman[212345]: 2024-02-05 07:44:11.762035628 +0000 UTC m=+0.347278257 container init 57a40a5d19f87d6f05f6510d51cfae21801c850eb75e7f43a353a8244fd02827 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T07:44:11.837 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:44:11 smithi007 podman[212345]: 2024-02-05 07:44:11.764990011 +0000 UTC m=+0.350232634 container start 57a40a5d19f87d6f05f6510d51cfae21801c850eb75e7f43a353a8244fd02827 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-05T07:44:11.837 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:44:11 smithi007 podman[212345]: 2024-02-05 07:44:11.837012605 +0000 UTC m=+0.422255243 container attach 57a40a5d19f87d6f05f6510d51cfae21801c850eb75e7f43a353a8244fd02827 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-05T07:44:12.173 INFO:journalctl@ceph.osd.2.smithi007.stdout:Feb 05 07:44:12 smithi007 podman[212345]: 2024-02-05 07:44:12.059524609 +0000 UTC m=+0.644767234 container died 57a40a5d19f87d6f05f6510d51cfae21801c850eb75e7f43a353a8244fd02827 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-05T07:44:12.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.2.service' 2024-02-05T07:44:12.734 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:44:12.734 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-05T07:44:12.734 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-05T07:44:12.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.3 2024-02-05T07:44:13.043 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:12 smithi007 systemd[1]: Stopping Ceph osd.3 for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:44:13.423 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3[58793]: 2024-02-05T07:44:13.042+0000 7fba2a537700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:44:13.423 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3[58793]: 2024-02-05T07:44:13.042+0000 7fba2a537700 -1 osd.3 758 *** Got signal Terminated *** 2024-02-05T07:44:13.423 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:13 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3[58793]: 2024-02-05T07:44:13.042+0000 7fba2a537700 -1 osd.3 758 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T07:44:15.444 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:15 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:15.156+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:49.872658+0000 front 2024-02-05T07:43:49.872756+0000 (oldest deadline 2024-02-05T07:44:14.472935+0000) 2024-02-05T07:44:15.444 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:15 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:15.039+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:16.456 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:16 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:16.050+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:16.456 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:16 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:16.158+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:16.457 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:16 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:16.169+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:49.872658+0000 front 2024-02-05T07:43:49.872756+0000 (oldest deadline 2024-02-05T07:44:14.472935+0000) 2024-02-05T07:44:17.173 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:16 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3[58793]: 2024-02-05T07:44:16.686+0000 7fba22b2a700 -1 osd.3 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.207495+0000 front 2024-02-05T07:43:51.207631+0000 (oldest deadline 2024-02-05T07:44:15.907620+0000) 2024-02-05T07:44:17.274 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:17 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:17.194+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:49.872658+0000 front 2024-02-05T07:43:49.872756+0000 (oldest deadline 2024-02-05T07:44:14.472935+0000) 2024-02-05T07:44:17.274 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:17 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:17.181+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:17.274 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:17 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:17.010+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:17.923 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:17 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3[58793]: 2024-02-05T07:44:17.667+0000 7fba22b2a700 -1 osd.3 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.207495+0000 front 2024-02-05T07:43:51.207631+0000 (oldest deadline 2024-02-05T07:44:15.907620+0000) 2024-02-05T07:44:18.274 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:18 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:18.151+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:49.872658+0000 front 2024-02-05T07:43:49.872756+0000 (oldest deadline 2024-02-05T07:44:14.472935+0000) 2024-02-05T07:44:18.274 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:18 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:18.156+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:18.274 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:17 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:17.982+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:19.068 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:18 smithi007 podman[212452]: 2024-02-05 07:44:18.677090679 +0000 UTC m=+5.717678957 container died 61e468aab9a42ec005566e8baea2f709bfc57bad95f4484ba6abc0a9c786b329 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T07:44:19.132 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:19 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:19.029+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:19.132 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:18 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:18.818+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:19.344 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:19 smithi007 podman[212452]: 2024-02-05 07:44:19.067871378 +0000 UTC m=+6.108459654 container cleanup 61e468aab9a42ec005566e8baea2f709bfc57bad95f4484ba6abc0a9c786b329 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux ) 2024-02-05T07:44:19.344 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:19 smithi007 bash[212452]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3 2024-02-05T07:44:19.523 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:19 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:19.131+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:49.872658+0000 front 2024-02-05T07:43:49.872756+0000 (oldest deadline 2024-02-05T07:44:14.472935+0000) 2024-02-05T07:44:19.524 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:19 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:19.160+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:19.645 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:19 smithi007 podman[212469]: 2024-02-05 07:44:19.344086936 +0000 UTC m=+0.667238373 container remove 61e468aab9a42ec005566e8baea2f709bfc57bad95f4484ba6abc0a9c786b329 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-05T07:44:19.923 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:19 smithi007 podman[212564]: 2024-02-05 07:44:19.644937809 +0000 UTC m=+0.102397992 container create 651ac8e30af893e0f06a923a5891a1aaddf795a8713aec3dbe295729aef9e764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3-deactivate, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T07:44:19.923 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:19 smithi007 podman[212564]: 2024-02-05 07:44:19.560455415 +0000 UTC m=+0.017915554 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T07:44:19.923 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:19 smithi007 podman[212564]: 2024-02-05 07:44:19.847314208 +0000 UTC m=+0.304774347 container init 651ac8e30af893e0f06a923a5891a1aaddf795a8713aec3dbe295729aef9e764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3-deactivate, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD) 2024-02-05T07:44:19.923 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:19 smithi007 podman[212564]: 2024-02-05 07:44:19.851101783 +0000 UTC m=+0.308561925 container start 651ac8e30af893e0f06a923a5891a1aaddf795a8713aec3dbe295729aef9e764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True) 2024-02-05T07:44:20.178 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:19 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:19.861+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:20.178 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:20.023+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:20.423 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:19 smithi007 podman[212564]: 2024-02-05 07:44:19.955376081 +0000 UTC m=+0.412836234 container attach 651ac8e30af893e0f06a923a5891a1aaddf795a8713aec3dbe295729aef9e764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True) 2024-02-05T07:44:20.423 INFO:journalctl@ceph.osd.3.smithi007.stdout:Feb 05 07:44:20 smithi007 podman[212564]: 2024-02-05 07:44:20.14123231 +0000 UTC m=+0.598692449 container died 651ac8e30af893e0f06a923a5891a1aaddf795a8713aec3dbe295729aef9e764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-3-deactivate, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T07:44:20.524 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:20.192+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:20.524 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:20.177+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:49.872658+0000 front 2024-02-05T07:43:49.872756+0000 (oldest deadline 2024-02-05T07:44:14.472935+0000) 2024-02-05T07:44:20.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.3.service' 2024-02-05T07:44:20.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:44:20.907 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-05T07:44:20.907 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-05T07:44:20.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.4 2024-02-05T07:44:21.159 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:20.895+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:21.160 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:20 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:20.998+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:21.160 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:20 smithi090 systemd[1]: Stopping Ceph osd.4 for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:44:21.160 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:21.130+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:49.872658+0000 front 2024-02-05T07:43:49.872756+0000 (oldest deadline 2024-02-05T07:44:14.472935+0000) 2024-02-05T07:44:21.524 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:21.158+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:21.524 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:21.231+0000 7f006d175700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:44:21.525 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:21.231+0000 7f006d175700 -1 osd.4 758 *** Got signal Terminated *** 2024-02-05T07:44:21.525 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:21.231+0000 7f006d175700 -1 osd.4 758 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T07:44:22.157 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:21.978+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:22.157 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:21 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:21.876+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:22.524 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:22.156+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:49.872658+0000 front 2024-02-05T07:43:49.872756+0000 (oldest deadline 2024-02-05T07:44:14.472935+0000) 2024-02-05T07:44:22.524 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:22.167+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:23.170 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:23.120+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:23.171 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:22.987+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:23.171 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:22.910+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:23.171 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:22 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:22.910+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:23.523 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:23.169+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:49.872658+0000 front 2024-02-05T07:43:49.872756+0000 (oldest deadline 2024-02-05T07:44:14.472935+0000) 2024-02-05T07:44:24.139 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:24 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:24.024+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:24.140 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:23.871+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:24.140 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:23 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:23.871+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:24.523 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:24 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:24.177+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:49.872658+0000 front 2024-02-05T07:43:49.872756+0000 (oldest deadline 2024-02-05T07:44:14.472935+0000) 2024-02-05T07:44:24.524 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:24 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:24.177+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:57.973820+0000 front 2024-02-05T07:43:57.973902+0000 (oldest deadline 2024-02-05T07:44:23.273859+0000) 2024-02-05T07:44:24.524 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:24 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:24.138+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:25.168 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:25 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:25.145+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:25.170 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:24 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:24.986+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:25.170 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:24 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:24.912+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:25.170 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:24 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:24.912+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:25.524 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:25 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:25.167+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:49.872658+0000 front 2024-02-05T07:43:49.872756+0000 (oldest deadline 2024-02-05T07:44:14.472935+0000) 2024-02-05T07:44:25.524 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:25 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:25.167+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:57.973820+0000 front 2024-02-05T07:43:57.973902+0000 (oldest deadline 2024-02-05T07:44:23.273859+0000) 2024-02-05T07:44:26.202 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:26 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:26.193+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:26.203 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:26 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:26.030+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:26.203 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:25 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:25.950+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:26.203 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:25 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:25.950+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:26.524 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:26 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:26.201+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:49.872658+0000 front 2024-02-05T07:43:49.872756+0000 (oldest deadline 2024-02-05T07:44:14.472935+0000) 2024-02-05T07:44:26.524 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:26 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4[38160]: 2024-02-05T07:44:26.201+0000 7f0065768700 -1 osd.4 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:57.973820+0000 front 2024-02-05T07:43:57.973902+0000 (oldest deadline 2024-02-05T07:44:23.273859+0000) 2024-02-05T07:44:27.136 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:26 smithi090 podman[158326]: 2024-02-05 07:44:26.869321117 +0000 UTC m=+5.765854267 container died a5f6b0447815eca96df604e9d55bb41a7873646c56ad813a7f68d31822e047ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-05T07:44:27.136 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:26 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:26.981+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:27.136 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:26 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:26.981+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:27.136 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:27 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:27.031+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:27.137 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:27 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:27.031+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:27.437 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:27 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:27.238+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:27.437 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:27 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:27.238+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:59.367450+0000 front 2024-02-05T07:43:59.367477+0000 (oldest deadline 2024-02-05T07:44:26.367976+0000) 2024-02-05T07:44:27.437 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:27 smithi090 podman[158326]: 2024-02-05 07:44:27.225759521 +0000 UTC m=+6.122292671 container cleanup a5f6b0447815eca96df604e9d55bb41a7873646c56ad813a7f68d31822e047ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.2) 2024-02-05T07:44:27.437 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:27 smithi090 bash[158326]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4 2024-02-05T07:44:27.713 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:27 smithi090 podman[158344]: 2024-02-05 07:44:27.436841577 +0000 UTC m=+0.552200759 container remove a5f6b0447815eca96df604e9d55bb41a7873646c56ad813a7f68d31822e047ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-05T07:44:27.972 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:27 smithi090 podman[158441]: 2024-02-05 07:44:27.871361198 +0000 UTC m=+0.110008373 container create 6256ad13de8f8ab4e5584285a0bdc6550fa060731f365f21bb3d4e6aa7ce73e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T07:44:27.972 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:27 smithi090 podman[158441]: 2024-02-05 07:44:27.779203837 +0000 UTC m=+0.017851017 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T07:44:28.225 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:28 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:28.016+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:28.225 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:28 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:28.016+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:28.226 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:28 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:28.081+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:28.226 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:28 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:28.081+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:28.226 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:28 smithi090 podman[158441]: 2024-02-05 07:44:28.099160374 +0000 UTC m=+0.337807549 container init 6256ad13de8f8ab4e5584285a0bdc6550fa060731f365f21bb3d4e6aa7ce73e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-05T07:44:28.226 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:28 smithi090 podman[158441]: 2024-02-05 07:44:28.102189763 +0000 UTC m=+0.340836939 container start 6256ad13de8f8ab4e5584285a0bdc6550fa060731f365f21bb3d4e6aa7ce73e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-05T07:44:28.523 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:28 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:28.224+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:28.524 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:28 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:28.224+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:59.367450+0000 front 2024-02-05T07:43:59.367477+0000 (oldest deadline 2024-02-05T07:44:26.367976+0000) 2024-02-05T07:44:28.524 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:28 smithi090 podman[158441]: 2024-02-05 07:44:28.23257256 +0000 UTC m=+0.471219751 container attach 6256ad13de8f8ab4e5584285a0bdc6550fa060731f365f21bb3d4e6aa7ce73e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-02-05T07:44:29.240 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 05 07:44:28 smithi090 podman[158441]: 2024-02-05 07:44:28.975472718 +0000 UTC m=+1.214119894 container died 6256ad13de8f8ab4e5584285a0bdc6550fa060731f365f21bb3d4e6aa7ce73e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.license=GPLv2) 2024-02-05T07:44:29.240 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:29 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:29.065+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:29.240 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:29 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:29.065+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:29.241 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:29 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:29.052+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:29.241 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:29 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:29.052+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:29.523 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:29 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:29.239+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:29.524 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:29 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:29.239+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:59.367450+0000 front 2024-02-05T07:43:59.367477+0000 (oldest deadline 2024-02-05T07:44:26.367976+0000) 2024-02-05T07:44:29.553 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.4.service' 2024-02-05T07:44:29.595 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:44:29.595 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-05T07:44:29.595 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-05T07:44:29.595 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.5 2024-02-05T07:44:29.816 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:29 smithi090 systemd[1]: Stopping Ceph osd.5 for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:44:30.206 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:30.016+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:30.206 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:30.016+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:30.206 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:30.041+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:30.206 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:30.041+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:30.207 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:29 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:29.942+0000 7ff1730b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:44:30.207 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:29 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:29.942+0000 7ff1730b9700 -1 osd.5 758 *** Got signal Terminated *** 2024-02-05T07:44:30.207 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:29 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:29.942+0000 7ff1730b9700 -1 osd.5 758 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T07:44:30.207 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:30.205+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:30.207 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:30 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:30.205+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:59.367450+0000 front 2024-02-05T07:43:59.367477+0000 (oldest deadline 2024-02-05T07:44:26.367976+0000) 2024-02-05T07:44:31.524 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:31 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:31.248+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:31.524 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:31 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:31.248+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:59.367450+0000 front 2024-02-05T07:43:59.367477+0000 (oldest deadline 2024-02-05T07:44:26.367976+0000) 2024-02-05T07:44:31.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:31 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:31.088+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:31.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:31 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:31.088+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:31.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:31 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:31.040+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:31.525 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:31 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:31.040+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:31.525 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:31 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:31.040+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:04.836513+0000 front 2024-02-05T07:44:04.836326+0000 (oldest deadline 2024-02-05T07:44:30.736375+0000) 2024-02-05T07:44:32.523 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:32 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:32.262+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:32.524 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:32 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:32.262+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:59.367450+0000 front 2024-02-05T07:43:59.367477+0000 (oldest deadline 2024-02-05T07:44:26.367976+0000) 2024-02-05T07:44:32.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:32 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:32.120+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:32.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:32 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:32.120+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:32.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:32 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:32.120+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:32.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:32 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:32.077+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:32.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:32 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:32.077+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:32.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:32 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:32.077+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:04.836513+0000 front 2024-02-05T07:44:04.836326+0000 (oldest deadline 2024-02-05T07:44:30.736375+0000) 2024-02-05T07:44:33.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:33 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:33.042+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:33.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:33 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:33.042+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:33.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:33 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:33.042+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:04.836513+0000 front 2024-02-05T07:44:04.836326+0000 (oldest deadline 2024-02-05T07:44:30.736375+0000) 2024-02-05T07:44:33.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:33 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:33.077+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:33.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:33 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:33.077+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:33.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:33 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:33.077+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:33.525 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:33 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:33.279+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:33.525 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:33 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:33.279+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:59.367450+0000 front 2024-02-05T07:43:59.367477+0000 (oldest deadline 2024-02-05T07:44:26.367976+0000) 2024-02-05T07:44:33.525 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:33 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:33.279+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.368440+0000 front 2024-02-05T07:44:06.368474+0000 (oldest deadline 2024-02-05T07:44:32.268578+0000) 2024-02-05T07:44:34.523 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:34 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:34.059+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:34.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:34 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:34.059+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:34.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:34 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:34.059+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:04.836513+0000 front 2024-02-05T07:44:04.836326+0000 (oldest deadline 2024-02-05T07:44:30.736375+0000) 2024-02-05T07:44:34.524 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:34 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:34.246+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:51.865983+0000 front 2024-02-05T07:43:51.866110+0000 (oldest deadline 2024-02-05T07:44:15.965856+0000) 2024-02-05T07:44:34.524 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:34 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:34.246+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:59.367450+0000 front 2024-02-05T07:43:59.367477+0000 (oldest deadline 2024-02-05T07:44:26.367976+0000) 2024-02-05T07:44:34.524 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:34 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5[43818]: 2024-02-05T07:44:34.246+0000 7ff16f6b4700 -1 osd.5 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.368440+0000 front 2024-02-05T07:44:06.368474+0000 (oldest deadline 2024-02-05T07:44:32.268578+0000) 2024-02-05T07:44:34.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:34 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:34.064+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:34.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:34 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:34.065+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:34.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:34 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:34.065+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:35.495 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:35 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:35.063+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:35.495 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:35 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:35.063+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:35.495 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:35 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:35.063+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:35.495 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:35 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:35.044+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:35.495 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:35 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:35.044+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:35.495 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:35 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:35.044+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:04.836513+0000 front 2024-02-05T07:44:04.836326+0000 (oldest deadline 2024-02-05T07:44:30.736375+0000) 2024-02-05T07:44:35.773 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:35 smithi090 podman[158546]: 2024-02-05 07:44:35.494786525 +0000 UTC m=+5.672686009 container died 3dc2e86c7d9b8cdc579608e2f549c005e27b0373f75cdcfbfb2fc88c2db7d753 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-05T07:44:36.091 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:35 smithi090 podman[158546]: 2024-02-05 07:44:35.900592161 +0000 UTC m=+6.078491647 container cleanup 3dc2e86c7d9b8cdc579608e2f549c005e27b0373f75cdcfbfb2fc88c2db7d753 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True) 2024-02-05T07:44:36.091 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:35 smithi090 bash[158546]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5 2024-02-05T07:44:36.446 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:36 smithi090 podman[158574]: 2024-02-05 07:44:36.152565588 +0000 UTC m=+0.655747020 container remove 3dc2e86c7d9b8cdc579608e2f549c005e27b0373f75cdcfbfb2fc88c2db7d753 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T07:44:36.447 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:36.090+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:36.447 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:36.090+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:36.447 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:36.090+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:04.836513+0000 front 2024-02-05T07:44:04.836326+0000 (oldest deadline 2024-02-05T07:44:30.736375+0000) 2024-02-05T07:44:36.447 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:36.113+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:36.447 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:36.113+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:36.447 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:36.113+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:36.447 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:36 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:36.113+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:36.749 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:36 smithi090 podman[158666]: 2024-02-05 07:44:36.446510672 +0000 UTC m=+0.111265005 container create 4138e257d97ccb879e6f39433ac524145f06fd3aebe6162d7937b269519df87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-05T07:44:36.750 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:36 smithi090 podman[158666]: 2024-02-05 07:44:36.353033987 +0000 UTC m=+0.017788283 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T07:44:36.750 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:36 smithi090 podman[158666]: 2024-02-05 07:44:36.674105378 +0000 UTC m=+0.338859669 container init 4138e257d97ccb879e6f39433ac524145f06fd3aebe6162d7937b269519df87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2) 2024-02-05T07:44:36.750 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:36 smithi090 podman[158666]: 2024-02-05 07:44:36.677150213 +0000 UTC m=+0.341904505 container start 4138e257d97ccb879e6f39433ac524145f06fd3aebe6162d7937b269519df87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0) 2024-02-05T07:44:36.750 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:36 smithi090 podman[158666]: 2024-02-05 07:44:36.7491639 +0000 UTC m=+0.413918194 container attach 4138e257d97ccb879e6f39433ac524145f06fd3aebe6162d7937b269519df87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2) 2024-02-05T07:44:37.024 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 05 07:44:36 smithi090 podman[158666]: 2024-02-05 07:44:36.966802278 +0000 UTC m=+0.631556579 container died 4138e257d97ccb879e6f39433ac524145f06fd3aebe6162d7937b269519df87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux ) 2024-02-05T07:44:37.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:37.078+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:37.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:37.078+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:37.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:37.078+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:04.836513+0000 front 2024-02-05T07:44:04.836326+0000 (oldest deadline 2024-02-05T07:44:30.736375+0000) 2024-02-05T07:44:37.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:37.114+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:37.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:37.114+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:37.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:37.114+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:37.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:37.114+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:37.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.5.service' 2024-02-05T07:44:37.642 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:44:37.642 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-05T07:44:37.643 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-05T07:44:37.643 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.6 2024-02-05T07:44:37.832 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:37 smithi090 systemd[1]: Stopping Ceph osd.6 for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:44:38.097 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:37.958+0000 7fb0f36ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:44:38.097 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:37.958+0000 7fb0f36ee700 -1 osd.6 758 *** Got signal Terminated *** 2024-02-05T07:44:38.098 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:37.958+0000 7fb0f36ee700 -1 osd.6 758 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T07:44:38.098 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:44:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:44:37.831Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-02-05T07:44:38.098 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:44:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:44:37.833Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-02-05T07:44:38.098 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:44:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:44:37.835Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-02-05T07:44:38.098 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:44:37 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:44:37.836Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-02-05T07:44:38.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:38 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:38.104+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:38.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:38 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:38.104+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:38.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:38 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:38.104+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:04.836513+0000 front 2024-02-05T07:44:04.836326+0000 (oldest deadline 2024-02-05T07:44:30.736375+0000) 2024-02-05T07:44:38.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:38 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:38.097+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:38.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:38 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:38.097+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:38.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:38 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:38.097+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:38.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:38 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:38.097+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:39.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:39.144+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:39.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:39.144+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:39.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:39.144+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:04.836513+0000 front 2024-02-05T07:44:04.836326+0000 (oldest deadline 2024-02-05T07:44:30.736375+0000) 2024-02-05T07:44:39.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:39.145+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:39.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:39.145+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:39.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:39.145+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:39.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:39.145+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-02-05T07:44:17.587744+0000 front 2024-02-05T07:44:17.587825+0000 (oldest deadline 2024-02-05T07:44:38.687792+0000) 2024-02-05T07:44:39.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:39 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:39.145+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:40.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:40.112+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:40.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:40.112+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:40.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:40.112+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:04.836513+0000 front 2024-02-05T07:44:04.836326+0000 (oldest deadline 2024-02-05T07:44:30.736375+0000) 2024-02-05T07:44:40.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:40.150+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:40.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:40.150+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:40.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:40.150+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:40.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:40.150+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-02-05T07:44:17.587744+0000 front 2024-02-05T07:44:17.587825+0000 (oldest deadline 2024-02-05T07:44:38.687792+0000) 2024-02-05T07:44:40.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:40 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:40.150+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:41.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:41.159+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:41.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:41.159+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:41.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:41.159+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:04.836513+0000 front 2024-02-05T07:44:04.836326+0000 (oldest deadline 2024-02-05T07:44:30.736375+0000) 2024-02-05T07:44:41.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:41.159+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-02-05T07:44:15.838197+0000 front 2024-02-05T07:44:15.838212+0000 (oldest deadline 2024-02-05T07:44:40.538087+0000) 2024-02-05T07:44:41.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:41.128+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:41.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:41.128+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:41.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:41.128+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:41.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:41.128+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-02-05T07:44:17.587744+0000 front 2024-02-05T07:44:17.587825+0000 (oldest deadline 2024-02-05T07:44:38.687792+0000) 2024-02-05T07:44:41.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:41 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:41.128+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:42.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:42 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:42.185+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:52.535164+0000 front 2024-02-05T07:43:52.535221+0000 (oldest deadline 2024-02-05T07:44:18.434977+0000) 2024-02-05T07:44:42.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:42 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:42.185+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:43:58.435470+0000 front 2024-02-05T07:43:58.435385+0000 (oldest deadline 2024-02-05T07:44:22.535433+0000) 2024-02-05T07:44:42.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:42 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:42.185+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:04.836513+0000 front 2024-02-05T07:44:04.836326+0000 (oldest deadline 2024-02-05T07:44:30.736375+0000) 2024-02-05T07:44:42.524 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:42 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6[99501]: 2024-02-05T07:44:42.185+0000 7fb0ebce1700 -1 osd.6 758 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-02-05T07:44:15.838197+0000 front 2024-02-05T07:44:15.838212+0000 (oldest deadline 2024-02-05T07:44:40.538087+0000) 2024-02-05T07:44:42.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:42 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:42.115+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:42.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:42 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:42.115+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:42.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:42 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:42.115+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:42.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:42 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:42.115+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-02-05T07:44:17.587744+0000 front 2024-02-05T07:44:17.587825+0000 (oldest deadline 2024-02-05T07:44:38.687792+0000) 2024-02-05T07:44:42.525 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:42 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:42.115+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:43.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:43 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:43.150+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:43.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:43 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:43.150+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:43.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:43 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:43.150+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:43.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:43 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:43.150+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-02-05T07:44:17.587744+0000 front 2024-02-05T07:44:17.587825+0000 (oldest deadline 2024-02-05T07:44:38.687792+0000) 2024-02-05T07:44:43.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:43 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:43.150+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:43.826 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:43 smithi090 podman[158771]: 2024-02-05 07:44:43.543785095 +0000 UTC m=+5.674596795 container died d164ba3156345f5d9b1a905782af7988560137fefbd132c44879882d55f63b68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=) 2024-02-05T07:44:44.135 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:43 smithi090 podman[158771]: 2024-02-05 07:44:43.916955608 +0000 UTC m=+6.047767296 container cleanup d164ba3156345f5d9b1a905782af7988560137fefbd132c44879882d55f63b68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T07:44:44.135 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:43 smithi090 bash[158771]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6 2024-02-05T07:44:44.473 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:44 smithi090 podman[158788]: 2024-02-05 07:44:44.176715815 +0000 UTC m=+0.628704197 container remove d164ba3156345f5d9b1a905782af7988560137fefbd132c44879882d55f63b68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True) 2024-02-05T07:44:44.474 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:44 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:44.134+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:44.474 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:44 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:44.134+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:44.474 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:44 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:44.134+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:44.474 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:44 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:44.134+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-02-05T07:44:17.587744+0000 front 2024-02-05T07:44:17.587825+0000 (oldest deadline 2024-02-05T07:44:38.687792+0000) 2024-02-05T07:44:44.474 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:44 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:44.135+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:44.732 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:44 smithi090 podman[158881]: 2024-02-05 07:44:44.376099789 +0000 UTC m=+0.018163797 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T07:44:44.733 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:44 smithi090 podman[158881]: 2024-02-05 07:44:44.494728805 +0000 UTC m=+0.136792813 container create 2632dfbfde360696c9e5302fe6fc01afd3b47635ebfc31c20d213e6c519303f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-05T07:44:44.733 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:44 smithi090 podman[158881]: 2024-02-05 07:44:44.731991765 +0000 UTC m=+0.374055773 container init 2632dfbfde360696c9e5302fe6fc01afd3b47635ebfc31c20d213e6c519303f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2) 2024-02-05T07:44:45.021 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:44 smithi090 podman[158881]: 2024-02-05 07:44:44.735844862 +0000 UTC m=+0.377908878 container start 2632dfbfde360696c9e5302fe6fc01afd3b47635ebfc31c20d213e6c519303f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131) 2024-02-05T07:44:45.021 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:44 smithi090 podman[158881]: 2024-02-05 07:44:44.832137373 +0000 UTC m=+0.474201395 container attach 2632dfbfde360696c9e5302fe6fc01afd3b47635ebfc31c20d213e6c519303f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-05T07:44:45.273 INFO:journalctl@ceph.osd.6.smithi090.stdout:Feb 05 07:44:45 smithi090 podman[158881]: 2024-02-05 07:44:45.020842519 +0000 UTC m=+0.662906534 container died 2632dfbfde360696c9e5302fe6fc01afd3b47635ebfc31c20d213e6c519303f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-05T07:44:45.274 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:45 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:45.101+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:45.274 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:45 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:45.101+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:45.274 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:45 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:45.101+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:45.274 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:45 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:45.101+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-02-05T07:44:17.587744+0000 front 2024-02-05T07:44:17.587825+0000 (oldest deadline 2024-02-05T07:44:38.687792+0000) 2024-02-05T07:44:45.274 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:45 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:45.102+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:45.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.6.service' 2024-02-05T07:44:45.646 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:44:45.646 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-05T07:44:45.647 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-05T07:44:45.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.7 2024-02-05T07:44:45.975 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:45 smithi090 systemd[1]: Stopping Ceph osd.7 for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:44:46.247 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:45 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:45.973+0000 7f8a2c6fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:44:46.247 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:45 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:45.973+0000 7f8a2c6fa700 -1 osd.7 758 *** Got signal Terminated *** 2024-02-05T07:44:46.247 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:45 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:45.973+0000 7f8a2c6fa700 -1 osd.7 758 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T07:44:46.247 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:46 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:46.059+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:46.247 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:46 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:46.059+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:46.247 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:46 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:46.059+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:46.248 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:46 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:46.059+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-02-05T07:44:17.587744+0000 front 2024-02-05T07:44:17.587825+0000 (oldest deadline 2024-02-05T07:44:38.687792+0000) 2024-02-05T07:44:46.248 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:46 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:46.059+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:47.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:47 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:47.103+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:47.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:47 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:47.103+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:47.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:47 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:47.103+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:47.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:47 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:47.103+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-02-05T07:44:17.587744+0000 front 2024-02-05T07:44:17.587825+0000 (oldest deadline 2024-02-05T07:44:38.687792+0000) 2024-02-05T07:44:47.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:47 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:47.103+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:48.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:48 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:48.074+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:48.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:48 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:48.074+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:48.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:48 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:48.074+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:48.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:48 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:48.074+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-02-05T07:44:17.587744+0000 front 2024-02-05T07:44:17.587825+0000 (oldest deadline 2024-02-05T07:44:38.687792+0000) 2024-02-05T07:44:48.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:48 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:48.074+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.90:6802 osd.4 since back 2024-02-05T07:44:25.089163+0000 front 2024-02-05T07:44:25.089175+0000 (oldest deadline 2024-02-05T07:44:47.989089+0000) 2024-02-05T07:44:48.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:48 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:48.074+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:49.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:49 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:49.098+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:49.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:49 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:49.098+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:49.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:49 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:49.098+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:49.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:49 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:49.098+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-02-05T07:44:17.587744+0000 front 2024-02-05T07:44:17.587825+0000 (oldest deadline 2024-02-05T07:44:38.687792+0000) 2024-02-05T07:44:49.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:49 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:49.098+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.90:6802 osd.4 since back 2024-02-05T07:44:25.089163+0000 front 2024-02-05T07:44:25.089175+0000 (oldest deadline 2024-02-05T07:44:47.989089+0000) 2024-02-05T07:44:49.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:49 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:49.099+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:50.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:50.137+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-02-05T07:43:48.683106+0000 front 2024-02-05T07:43:48.683155+0000 (oldest deadline 2024-02-05T07:44:14.583063+0000) 2024-02-05T07:44:50.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:50.137+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-02-05T07:44:01.885706+0000 front 2024-02-05T07:44:01.885804+0000 (oldest deadline 2024-02-05T07:44:26.585651+0000) 2024-02-05T07:44:50.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:50.137+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-02-05T07:44:06.586254+0000 front 2024-02-05T07:44:06.586221+0000 (oldest deadline 2024-02-05T07:44:35.386537+0000) 2024-02-05T07:44:50.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:50.137+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-02-05T07:44:17.587744+0000 front 2024-02-05T07:44:17.587825+0000 (oldest deadline 2024-02-05T07:44:38.687792+0000) 2024-02-05T07:44:50.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:50.137+0000 7f8a28cf5700 -1 osd.7 758 heartbeat_check: no reply from 172.21.15.90:6802 osd.4 since back 2024-02-05T07:44:25.089163+0000 front 2024-02-05T07:44:25.089175+0000 (oldest deadline 2024-02-05T07:44:47.989089+0000) 2024-02-05T07:44:50.524 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:50 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7[55146]: 2024-02-05T07:44:50.137+0000 7f8a28cf5700 -1 osd.7 758 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1494 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e758) 2024-02-05T07:44:51.684 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:51 smithi090 podman[158997]: 2024-02-05 07:44:51.399001039 +0000 UTC m=+5.528990311 container died 4fcffcdfc407e3ff57f37bb9e99faf35297ebf273c87739849b3b0cd46822d3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-05T07:44:52.024 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:51 smithi090 podman[158997]: 2024-02-05 07:44:51.769720637 +0000 UTC m=+5.899709910 container cleanup 4fcffcdfc407e3ff57f37bb9e99faf35297ebf273c87739849b3b0cd46822d3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T07:44:52.024 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:51 smithi090 bash[158997]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7 2024-02-05T07:44:52.469 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:52 smithi090 podman[159015]: 2024-02-05 07:44:52.167692475 +0000 UTC m=+0.759695997 container remove 4fcffcdfc407e3ff57f37bb9e99faf35297ebf273c87739849b3b0cd46822d3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-05T07:44:52.755 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:52 smithi090 podman[159108]: 2024-02-05 07:44:52.373540675 +0000 UTC m=+0.018985919 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T07:44:52.755 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:52 smithi090 podman[159108]: 2024-02-05 07:44:52.476881177 +0000 UTC m=+0.122326417 container create 753f3fae569598d65e65c2bec1707cf18610decc2fed3632eb53affdea00ff4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-05T07:44:52.755 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:52 smithi090 podman[159108]: 2024-02-05 07:44:52.679788583 +0000 UTC m=+0.325233824 container init 753f3fae569598d65e65c2bec1707cf18610decc2fed3632eb53affdea00ff4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True) 2024-02-05T07:44:52.755 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:52 smithi090 podman[159108]: 2024-02-05 07:44:52.682736691 +0000 UTC m=+0.328181932 container start 753f3fae569598d65e65c2bec1707cf18610decc2fed3632eb53affdea00ff4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-05T07:44:52.755 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:52 smithi090 podman[159108]: 2024-02-05 07:44:52.754622964 +0000 UTC m=+0.400068216 container attach 753f3fae569598d65e65c2bec1707cf18610decc2fed3632eb53affdea00ff4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-05T07:44:53.024 INFO:journalctl@ceph.osd.7.smithi090.stdout:Feb 05 07:44:52 smithi090 podman[159108]: 2024-02-05 07:44:52.972274309 +0000 UTC m=+0.617719550 container died 753f3fae569598d65e65c2bec1707cf18610decc2fed3632eb53affdea00ff4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-osd-7-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-05T07:44:53.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@osd.7.service' 2024-02-05T07:44:53.613 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:44:53.614 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-05T07:44:53.614 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-02-05T07:44:53.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@rgw.foo.a 2024-02-05T07:44:53.908 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Feb 05 07:44:53 smithi007 systemd[1]: Stopping Ceph rgw.foo.a for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:44:54.173 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Feb 05 07:44:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-rgw-foo-a[63940]: 2024-02-05T07:44:53.906+0000 7f3f27b1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:44:54.173 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Feb 05 07:44:53 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-rgw-foo-a[63940]: 2024-02-05T07:44:53.906+0000 7f3f36389ac0 -1 shutting down 2024-02-05T07:45:04.173 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Feb 05 07:45:03 smithi007 bash[212670]: time="2024-02-05T07:45:03Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-02-05T07:45:04.173 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Feb 05 07:45:03 smithi007 podman[212670]: 2024-02-05 07:45:03.930973707 +0000 UTC m=+10.122737546 container died d567dde6e018f96ce00a18475ade441157746120b71cec33ef59b883376ea8e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-05T07:45:04.573 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Feb 05 07:45:04 smithi007 podman[212670]: 2024-02-05 07:45:04.3390489 +0000 UTC m=+10.530812733 container cleanup d567dde6e018f96ce00a18475ade441157746120b71cec33ef59b883376ea8e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-05T07:45:04.574 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Feb 05 07:45:04 smithi007 bash[212670]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-rgw-foo-a 2024-02-05T07:45:04.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@rgw.foo.a.service' 2024-02-05T07:45:04.750 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:45:04.750 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-02-05T07:45:04.750 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-05T07:45:04.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@prometheus.a 2024-02-05T07:45:05.038 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:04 smithi090 systemd[1]: Stopping Ceph prometheus.a for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:45:05.297 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:45:05.038Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-05T07:45:05.298 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:45:05.038Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-05T07:45:05.298 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:45:05.038Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-05T07:45:05.298 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:45:05.038Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-05T07:45:05.298 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:45:05.038Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-05T07:45:05.298 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:45:05.038Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-05T07:45:05.298 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:45:05.038Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-05T07:45:05.298 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:45:05.038Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-05T07:45:05.298 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:45:05.042Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-05T07:45:05.298 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:45:05.042Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-05T07:45:05.298 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:45:05.043Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-05T07:45:05.298 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a[68741]: ts=2024-02-05T07:45:05.043Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-05T07:45:05.298 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 podman[159216]: 2024-02-05 07:45:05.048395357 +0000 UTC m=+0.107537844 container died aca460d64221bfdc00e9cba9e129339e83cae80387660bf0959d7e283e48e4cb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T07:45:05.589 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 podman[159216]: 2024-02-05 07:45:05.379624617 +0000 UTC m=+0.438767099 container cleanup aca460d64221bfdc00e9cba9e129339e83cae80387660bf0959d7e283e48e4cb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T07:45:05.589 INFO:journalctl@ceph.prometheus.a.smithi090.stdout:Feb 05 07:45:05 smithi090 bash[159216]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-prometheus-a 2024-02-05T07:45:05.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@prometheus.a.service' 2024-02-05T07:45:05.835 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T07:45:05.835 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-05T07:45:05.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 --force --keep-logs 2024-02-05T07:45:06.034 INFO:teuthology.orchestra.run.smithi007.stdout:Deleting cluster with fsid: e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:45:08.423 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:45:08 smithi007 systemd[1]: Stopping Ceph alertmanager.a for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:45:08.879 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:45:08 smithi007 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a[80747]: ts=2024-02-05T07:45:08.568Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-05T07:45:08.879 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:45:08 smithi007 podman[213165]: 2024-02-05 07:45:08.609677641 +0000 UTC m=+0.171470195 container died c0a0ce033b5b8ff0ba54aba37e3075e41e7fb5e7198bf1b5e0d44f0eb4528bbe (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T07:45:09.173 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:45:09 smithi007 podman[213165]: 2024-02-05 07:45:09.04414624 +0000 UTC m=+0.605938796 container cleanup c0a0ce033b5b8ff0ba54aba37e3075e41e7fb5e7198bf1b5e0d44f0eb4528bbe (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T07:45:09.173 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:45:09 smithi007 bash[213165]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a 2024-02-05T07:45:09.673 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:45:09 smithi007 podman[213179]: 2024-02-05 07:45:09.370801382 +0000 UTC m=+0.760867481 container remove c0a0ce033b5b8ff0ba54aba37e3075e41e7fb5e7198bf1b5e0d44f0eb4528bbe (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T07:45:09.673 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:45:09 smithi007 podman[213179]: 2024-02-05 07:45:09.464775731 +0000 UTC m=+0.854841830 volume remove cc89949747cd04ff9fcc0aaf67f9de1e2a560dd7a215618084c9692ed8cdd910 2024-02-05T07:45:10.168 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:45:09 smithi007 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-05T07:45:10.168 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:45:09 smithi007 systemd[1]: Stopped Ceph alertmanager.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:45:10.168 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Feb 05 07:45:09 smithi007 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@alertmanager.a.service: Consumed 1.759s CPU time. 2024-02-05T07:45:10.168 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Feb 05 07:45:09 smithi007 systemd[1]: Stopping Ceph node-exporter.a for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:45:10.422 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Feb 05 07:45:10 smithi007 podman[213293]: 2024-02-05 07:45:10.167987662 +0000 UTC m=+0.110103045 container died 339bd22ee8613d57b78888bbb648af1ce01210d252588a1086736877d85b0de4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T07:45:10.918 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Feb 05 07:45:10 smithi007 podman[213293]: 2024-02-05 07:45:10.507479852 +0000 UTC m=+0.449595235 container cleanup 339bd22ee8613d57b78888bbb648af1ce01210d252588a1086736877d85b0de4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T07:45:10.918 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Feb 05 07:45:10 smithi007 bash[213293]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-a 2024-02-05T07:45:10.918 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Feb 05 07:45:10 smithi007 podman[213307]: 2024-02-05 07:45:10.683584683 +0000 UTC m=+0.520035360 container remove 339bd22ee8613d57b78888bbb648af1ce01210d252588a1086736877d85b0de4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T07:45:10.918 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Feb 05 07:45:10 smithi007 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-05T07:45:11.218 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Feb 05 07:45:10 smithi007 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-05T07:45:11.218 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Feb 05 07:45:10 smithi007 systemd[1]: Stopped Ceph node-exporter.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:45:11.218 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Feb 05 07:45:10 smithi007 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@node-exporter.a.service: Consumed 4.919s CPU time. 2024-02-05T07:45:11.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 --force --keep-logs 2024-02-05T07:45:11.813 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T07:45:13.396 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:45:13 smithi090 systemd[1]: Stopping Ceph iscsi.iscsi.a for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:45:13.773 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:45:13 smithi090 ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a[60026]: debug Shutdown received 2024-02-05T07:45:23.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:45:23 smithi090 bash[159628]: time="2024-02-05T07:45:23Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-02-05T07:45:23.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:45:23 smithi090 podman[159628]: 2024-02-05 07:45:23.409290443 +0000 UTC m=+10.159490350 container died 6e24d6dcfa5ca989848e9135ddf5b58fc8b1a325044083bee7a790bdc3503f65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.schema-version=1.0) 2024-02-05T07:45:24.023 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:45:23 smithi090 podman[159628]: 2024-02-05 07:45:23.786616855 +0000 UTC m=+10.536816754 container cleanup 6e24d6dcfa5ca989848e9135ddf5b58fc8b1a325044083bee7a790bdc3503f65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T07:45:24.023 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:45:23 smithi090 bash[159628]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a 2024-02-05T07:45:24.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:45:24 smithi090 podman[159643]: 2024-02-05 07:45:24.221416305 +0000 UTC m=+0.812795095 container remove 6e24d6dcfa5ca989848e9135ddf5b58fc8b1a325044083bee7a790bdc3503f65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-iscsi-iscsi-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.2) 2024-02-05T07:45:24.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:45:24 smithi090 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-02-05T07:45:24.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:45:24 smithi090 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-02-05T07:45:24.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:45:24 smithi090 systemd[1]: Stopped Ceph iscsi.iscsi.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:45:24.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi090.stdout:Feb 05 07:45:24 smithi090 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@iscsi.iscsi.a.service: Consumed 3.529s CPU time. 2024-02-05T07:45:26.118 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:45:25 smithi090 systemd[1]: Stopping Ceph grafana.a for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:45:26.456 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:45:26 smithi090 podman[159914]: 2024-02-05 07:45:26.116482611 +0000 UTC m=+0.114603426 container died dfa4c185051b59779667b9443621e705ee93d165c0a39e0a11739a5fb29d764c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, vendor=Red Hat, Inc., io.buildah.version=1.32.0, maintainer=Paul Cuzner , distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, description=Ceph Grafana Container, io.openshift.tags=base rhel8) 2024-02-05T07:45:26.456 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:45:26 smithi090 podman[159914]: 2024-02-05 07:45:26.455936745 +0000 UTC m=+0.454057569 container cleanup dfa4c185051b59779667b9443621e705ee93d165c0a39e0a11739a5fb29d764c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , architecture=x86_64, version=8.5, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, distribution-scope=public, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, description=Ceph Grafana Container, vcs-type=git, io.buildah.version=1.32.0) 2024-02-05T07:45:26.706 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:45:26 smithi090 bash[159914]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a 2024-02-05T07:45:26.706 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:45:26 smithi090 podman[159926]: 2024-02-05 07:45:26.615802165 +0000 UTC m=+0.501645271 container remove dfa4c185051b59779667b9443621e705ee93d165c0a39e0a11739a5fb29d764c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., version=8.5, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, name=ubi8, io.openshift.tags=base rhel8, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, maintainer=Paul Cuzner , vcs-type=git, com.redhat.component=ubi8-container, io.buildah.version=1.32.0) 2024-02-05T07:45:26.706 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:45:26 smithi090 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-02-05T07:45:27.024 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:45:26 smithi090 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-05T07:45:27.024 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:45:26 smithi090 systemd[1]: Stopped Ceph grafana.a for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:45:27.024 INFO:journalctl@ceph.grafana.a.smithi090.stdout:Feb 05 07:45:26 smithi090 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@grafana.a.service: Consumed 13.061s CPU time. 2024-02-05T07:45:27.296 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:45:27 smithi090 systemd[1]: Stopping Ceph node-exporter.b for e3e7059e-c3f5-11ee-95b6-87774f69a715... 2024-02-05T07:45:27.653 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:45:27 smithi090 podman[160044]: 2024-02-05 07:45:27.296403386 +0000 UTC m=+0.109812976 container died 7d496c9d2a54e69cdbe652d1d2447c2de25a03dd54e1d5a3fd88bde7ced5dd00 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T07:45:27.653 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:45:27 smithi090 podman[160044]: 2024-02-05 07:45:27.653576347 +0000 UTC m=+0.466985934 container cleanup 7d496c9d2a54e69cdbe652d1d2447c2de25a03dd54e1d5a3fd88bde7ced5dd00 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T07:45:27.922 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:45:27 smithi090 bash[160044]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b 2024-02-05T07:45:28.204 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:45:27 smithi090 podman[160058]: 2024-02-05 07:45:27.921983238 +0000 UTC m=+0.623211398 container remove 7d496c9d2a54e69cdbe652d1d2447c2de25a03dd54e1d5a3fd88bde7ced5dd00 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T07:45:28.204 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:45:27 smithi090 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-05T07:45:28.204 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:45:28 smithi090 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-05T07:45:28.205 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:45:28 smithi090 systemd[1]: Stopped Ceph node-exporter.b for e3e7059e-c3f5-11ee-95b6-87774f69a715. 2024-02-05T07:45:28.205 INFO:journalctl@ceph.node-exporter.b.smithi090.stdout:Feb 05 07:45:28 smithi090 systemd[1]: ceph-e3e7059e-c3f5-11ee-95b6-87774f69a715@node-exporter.b.service: Consumed 4.234s CPU time. 2024-02-05T07:45:28.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-05T07:45:28.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-05T07:45:28.844 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-05T07:45:28.847 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546729/remote/smithi007/crash 2024-02-05T07:45:28.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/crash -- . 2024-02-05T07:45:28.908 INFO:teuthology.orchestra.run.smithi007.stderr:tar: /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/crash: Cannot open: No such file or directory 2024-02-05T07:45:28.908 INFO:teuthology.orchestra.run.smithi007.stderr:tar: Error is not recoverable: exiting now 2024-02-05T07:45:28.910 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546729/remote/smithi090/crash 2024-02-05T07:45:28.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/crash -- . 2024-02-05T07:45:28.951 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/crash: Cannot open: No such file or directory 2024-02-05T07:45:28.951 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2024-02-05T07:45:28.952 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-05T07:45:28.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-05T07:45:29.113 INFO:teuthology.orchestra.run.smithi007.stdout:2024-02-05T07:40:00.000704+0000 mon.a (mon.0) 2097 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-02-05T07:45:29.113 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-05T07:45:29.113 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[SEC\]' /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-05T07:45:29.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]' /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-05T07:45:29.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[WRN\]' /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-05T07:45:29.288 INFO:teuthology.orchestra.run.smithi007.stdout:2024-02-05T07:40:00.000704+0000 mon.a (mon.0) 2097 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-02-05T07:45:29.288 INFO:tasks.cephadm:Compressing logs... 2024-02-05T07:45:29.288 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-05T07:45:29.330 DEBUG:teuthology.orchestra.run.smithi090:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-05T07:45:29.381 INFO:teuthology.orchestra.run.smithi007.stderr:find: ‘/var/log/rbd-target-api’gzip -5: No such file or directory --verbose 2024-02-05T07:45:29.381 INFO:teuthology.orchestra.run.smithi007.stderr: -- /var/log/ceph/cephadm.log 2024-02-05T07:45:29.390 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mon.a.log 2024-02-05T07:45:29.392 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.audit.log 2024-02-05T07:45:29.392 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-client.rgw.foo.a.log 2024-02-05T07:45:29.392 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.1.log 2024-02-05T07:45:29.392 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.3.log 2024-02-05T07:45:29.392 INFO:teuthology.orchestra.run.smithi007.stderr: 57.7% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-02-05T07:45:29.392 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mgr.y.log 2024-02-05T07:45:29.392 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.1.log: gzip -5/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.3.log: --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.0.log 2024-02-05T07:45:29.392 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.cephadm.log 2024-02-05T07:45:29.392 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.log 2024-02-05T07:45:29.393 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.2.log 2024-02-05T07:45:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-volume.log 2024-02-05T07:45:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mon.c.log 2024-02-05T07:45:29.394 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-05T07:45:29.394 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-05T07:45:29.402 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.6.log 2024-02-05T07:45:29.406 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.audit.log 2024-02-05T07:45:29.406 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.4.log 2024-02-05T07:45:29.406 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.6.log: /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.7.log 2024-02-05T07:45:29.406 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.5.log 2024-02-05T07:45:29.406 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mgr.x.log 2024-02-05T07:45:29.406 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/tcmu-runner.log 2024-02-05T07:45:29.406 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.cephadm.log 2024-02-05T07:45:29.406 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.log 2024-02-05T07:45:29.406 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-volume.log 2024-02-05T07:45:29.406 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mon.b.log 2024-02-05T07:45:29.406 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/objectstore_tool.2.log 2024-02-05T07:45:29.440 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-volume.log: /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mon.c.log: 91.8% 95.2% -- replaced with /var/log/ceph/cephadm.log.gz -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.log.gz 2024-02-05T07:45:29.440 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-02-05T07:45:29.512 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mon.b.log: /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/objectstore_tool.2.log: 91.0% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-volume.log.gz 2024-02-05T07:45:29.554 INFO:teuthology.orchestra.run.smithi090.stderr: 67.1% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/tcmu-runner.log.gz 2024-02-05T07:45:29.619 INFO:teuthology.orchestra.run.smithi090.stderr: 94.4% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/objectstore_tool.2.log.gz 2024-02-05T07:45:29.619 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.cephadm.log.gz 2024-02-05T07:45:29.659 INFO:teuthology.orchestra.run.smithi090.stderr: 92.2% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mgr.x.log.gz 2024-02-05T07:45:29.660 INFO:teuthology.orchestra.run.smithi007.stderr: 91.1% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-volume.log.gz 2024-02-05T07:45:29.726 INFO:teuthology.orchestra.run.smithi090.stderr: 89.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-05T07:45:29.792 INFO:teuthology.orchestra.run.smithi090.stderr: 81.5% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.cephadm.log.gz 2024-02-05T07:45:29.808 INFO:teuthology.orchestra.run.smithi007.stderr: 94.1% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.audit.log.gz 2024-02-05T07:45:29.823 INFO:teuthology.orchestra.run.smithi090.stderr: 85.5% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.log.gz 2024-02-05T07:45:29.858 INFO:teuthology.orchestra.run.smithi090.stderr: 90.1% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph.audit.log.gz 2024-02-05T07:45:31.831 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mgr.y.log.gz 2024-02-05T07:45:32.792 INFO:teuthology.orchestra.run.smithi090.stderr: 91.1% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mon.b.log.gz 2024-02-05T07:45:47.829 INFO:teuthology.orchestra.run.smithi007.stderr: 91.1% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mon.c.log.gz 2024-02-05T07:45:56.817 INFO:teuthology.orchestra.run.smithi007.stderr: 90.4% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-mon.a.log.gz 2024-02-05T07:48:11.518 INFO:teuthology.orchestra.run.smithi090.stderr: 92.8% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.5.log.gz 2024-02-05T07:48:54.398 INFO:teuthology.orchestra.run.smithi090.stderr: 92.5% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.7.log.gz 2024-02-05T07:49:05.233 INFO:teuthology.orchestra.run.smithi090.stderr: 93.0% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.4.log.gz 2024-02-05T07:49:14.415 INFO:teuthology.orchestra.run.smithi090.stderr: 93.7% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.6.log.gz 2024-02-05T07:49:14.417 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-02-05T07:49:14.417 INFO:teuthology.orchestra.run.smithi090.stderr:real 3m45.071s 2024-02-05T07:49:14.417 INFO:teuthology.orchestra.run.smithi090.stderr:user 4m34.818s 2024-02-05T07:49:14.417 INFO:teuthology.orchestra.run.smithi090.stderr:sys 0m11.066s 2024-02-05T08:09:49.039 INFO:teuthology.orchestra.run.smithi007.stderr: 92.8% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.2.log.gz 2024-02-05T08:09:56.635 INFO:teuthology.orchestra.run.smithi007.stderr: 92.4% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.0.log.gz 2024-02-05T08:12:35.092 INFO:teuthology.orchestra.run.smithi007.stderr: 92.4% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.3.log.gz 2024-02-05T08:12:43.819 INFO:teuthology.orchestra.run.smithi007.stderr: 92.4% -- replaced with /var/log/ceph/e3e7059e-c3f5-11ee-95b6-87774f69a715/ceph-osd.1.log.gz 2024-02-05T08:12:43.821 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-02-05T08:12:43.821 INFO:teuthology.orchestra.run.smithi007.stderr:real 27m14.479s 2024-02-05T08:12:43.821 INFO:teuthology.orchestra.run.smithi007.stderr:user 9m19.312s 2024-02-05T08:12:43.821 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m26.111s 2024-02-05T08:12:43.822 INFO:tasks.cephadm:Archiving logs... 2024-02-05T08:12:43.822 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546729/remote/smithi007/log 2024-02-05T08:12:43.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-05T08:14:07.332 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546729/remote/smithi090/log 2024-02-05T08:14:07.333 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-05T08:14:47.476 INFO:tasks.cephadm:Removing cluster... 2024-02-05T08:14:47.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 --force 2024-02-05T08:14:47.730 INFO:teuthology.orchestra.run.smithi007.stdout:Deleting cluster with fsid: e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T08:14:48.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e3e7059e-c3f5-11ee-95b6-87774f69a715 --force 2024-02-05T08:14:48.817 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: e3e7059e-c3f5-11ee-95b6-87774f69a715 2024-02-05T08:14:49.395 INFO:tasks.cephadm:Removing cephadm ... 2024-02-05T08:14:49.395 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-05T08:14:49.414 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-05T08:14:49.430 INFO:tasks.cephadm:Teardown complete 2024-02-05T08:14:49.430 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-05T08:14:49.439 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-05T08:14:49.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-05T08:14:49.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-05T08:14:49.551 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-05T08:14:49.560 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-05T08:14:49.560 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T08:14:49.563 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T08:14:49.576 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-02-05T08:14:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-02-05T08:14:49.606 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T08:14:49.606 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-02-05T08:14:49.606 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 +931us[ +919us] +/- 101ms 2024-02-05T08:14:49.606 INFO:teuthology.orchestra.run.smithi090.stdout:^* hv02.front.sepia.ceph.com 3 6 377 17 -726us[ -738us] +/- 59ms 2024-02-05T08:14:49.606 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 +261us[ +249us] +/- 68ms 2024-02-05T08:14:49.607 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T08:14:49.610 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T08:14:49.610 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-02-05T08:14:49.610 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 87 +886us[ +903us] +/- 100ms 2024-02-05T08:14:49.610 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -754us[ -762us] +/- 59ms 2024-02-05T08:14:49.610 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +232us[ +224us] +/- 68ms 2024-02-05T08:14:49.611 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T08:14:49.611 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-05T08:14:49.620 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-05T08:14:49.620 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-05T08:14:49.628 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-05T08:14:49.762 DEBUG:teuthology.orchestra.run.smithi090:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-05T08:14:49.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T08:14:49.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:49.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T08:14:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:49.929 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-05T08:14:49.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-05T08:14:49.951 INFO:teuthology.task.internal:Duration was 4160.426799 seconds 2024-02-05T08:14:49.951 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-05T08:14:49.959 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-05T08:14:49.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-05T08:14:49.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-05T08:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T08:14:50.042 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T08:14:50.486 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-05T08:14:50.486 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2024-02-05T08:14:50.486 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-05T08:14:50.507 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2024-02-05T08:14:50.508 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-05T08:14:50.541 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-05T08:14:50.541 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-05T08:14:50.550 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-05T08:14:50.612 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-05T08:14:50.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T08:14:50.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T08:14:52.179 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-05T08:14:52.449 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-05T08:14:52.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-05T08:14:52.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-05T08:14:52.523 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-05T08:14:52.612 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-05T08:14:52.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-05T08:14:52.649 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2024-02-05T08:14:52.653 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2024-02-05T08:14:52.679 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-05T08:14:52.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:52.733 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-05T08:14:52.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:52.748 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-05T08:14:52.758 INFO:teuthology.task.internal:Transferring archived files... 2024-02-05T08:14:52.760 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546729/remote/smithi007 2024-02-05T08:14:52.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-05T08:14:52.848 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546729/remote/smithi090 2024-02-05T08:14:52.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-05T08:14:52.892 INFO:teuthology.task.internal:Removing archive directory... 2024-02-05T08:14:52.892 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-05T08:14:52.894 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-05T08:14:52.942 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-05T08:14:52.955 INFO:teuthology.task.internal:Not uploading archives. 2024-02-05T08:14:52.956 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-05T08:14:52.965 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-05T08:14:52.965 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-05T08:14:52.968 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-05T08:14:52.979 INFO:teuthology.orchestra.run.smithi007.stdout: 265293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 08:14 /home/ubuntu/cephtest 2024-02-05T08:14:52.998 INFO:teuthology.orchestra.run.smithi090.stdout: 265582 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 08:14 /home/ubuntu/cephtest 2024-02-05T08:14:53.000 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-05T08:14:53.011 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-05T08:14:53.035 INFO:teuthology.nuke:Checking targets against current locks 2024-02-05T08:14:53.062 DEBUG:teuthology.nuke:shortname: smithi007 2024-02-05T08:14:53.062 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T08:14:53.087 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546729', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 06:55:56.963327', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCz1R3OHxhVp2mKR6RqPhAvbhtI32CP2dgQ5Xhmx2AFPKED9mGGBpiR+tejO5ppiDdHVd52nqyNd9wPwDeU958k='} 2024-02-05T08:14:53.088 DEBUG:teuthology.nuke:shortname: smithi090 2024-02-05T08:14:53.089 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T08:14:53.114 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546729', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 06:55:56.961345', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN7CVL30sw/Qt233Zvb9eaojHn2grchuZh5xuuFs7Du40CDTCrKcdHjnGtVfbawCn+dmelh5N7JuCk51oGqODcI='} 2024-02-05T08:14:53.140 INFO:teuthology.orchestra.console.smithi007:Power off 2024-02-05T08:14:53.140 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-05T08:14:53.165 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2024-02-05T08:14:53.165 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T08:14:53.179 INFO:teuthology.orchestra.console.smithi090:Power off 2024-02-05T08:14:53.179 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-05T08:14:53.189 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-02-05T08:14:53.201 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off 2024-02-05T08:14:53.202 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T08:14:53.224 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-02-05T08:14:57.191 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T08:14:57.225 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T08:14:57.301 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-02-05T08:14:57.335 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-02-05T08:15:01.304 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T08:15:01.336 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T08:15:01.414 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-02-05T08:15:01.445 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-02-05T08:15:01.516 INFO:teuthology.orchestra.console.smithi007:Power off completed 2024-02-05T08:15:05.450 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T08:15:05.559 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off 2024-02-05T08:15:05.660 INFO:teuthology.orchestra.console.smithi090:Power off completed 2024-02-05T08:15:05.764 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} duration: 4160.426799297333 failure_reason: '"2024-02-05T07:40:00.000704+0000 mon.a (mon.0) 2097 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log' flavor: default owner: scheduled_gabrioux@teuthology success: false 2024-02-05T08:15:05.764 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-05T08:15:05.843 INFO:teuthology.run:FAIL